How to create a Spark Job Notebook in Microsoft Fabric: step by step

João Barros 13 de August de 2026 5 min read

This tutorial shows how to create a Notebook and publish it as a Spark Job in Microsoft Fabric to schedule recurring PySpark execution. It is useful to automate ETL/ELT, transformations and pipelines without manual intervention — for example, processing 1–10 GB of daily files or aggregating millions of rows for reports. I will explain the reasons behind the choices and provide concrete examples for testing and production.

Prerequisites

An account with access to a workspace in Microsoft Fabric with author permissions (or higher).
A Lakehouse or OneLake to read/write files (e.g.: CSV/Parquet). Ideally you have a folder for raw and another for processed.
Basic knowledge of Python/PySpark and the Fabric Notebook environment. Being able to interpret logs and execution metrics is a plus.

Step 1: Create a new Notebook in the workspace

Open your workspace in Microsoft Fabric and create a new Notebook. Choose the appropriate PySpark kernel: Serverless Spark for small/occasional workloads or a Spark Pool associated with the Warehouse/Lakehouse for larger workloads. Give it an identifiable name, for example: process_sales_notebook, along with a brief description: "Aggregates daily sales". Maintain a naming standard (prefix, purpose, environment) to manage multiple jobs easily.

Step 2: Write minimal PySpark code ready for production

Write code that reads data from the Lakehouse, performs a simple transformation and writes the result. Keep the code modular and use paths relative to OneLake/Lakehouse for portability between dev and prod. Examples of best practices: schema validation, partitioning by date and using coalesce when writing to control the number of files.

from pyspark.sql import functions as F

# Caminhos no OneLake/Lakehouse
input_path = 'one:///lakehouse/sales/raw/sales.csv'
output_path = 'one:///lakehouse/sales/processed/sales_agg.parquet'

# Ler CSV (ajusta opções conforme necessário)
df = spark.read.option('header', 'true').option('inferSchema', 'true').csv(input_path)

# Exemplo de transformação: agregação por dia
df2 = (df
  .withColumn('date', F.to_date('order_date'))
  .groupBy('date')
  .agg(F.sum(F.col('amount')).alias('total_amount'), F.count('*').alias('orders'))
)

# Gravar em Parquet (substituir)
df2.write.mode('overwrite').parquet(output_path)

For larger datasets, consider partitioning by date and using repartition(10) or a number of partitions equivalent to the number of available cores (e.g.: 8–32). If you process 5 GB daily, 8–16 cores are often sufficient to keep runtimes under 10–20 minutes, depending on the transformations.

Step 3: Test the Notebook manually

Run the cells in the Notebook to validate that the code runs without errors and that files are read/written in OneLake/Lakehouse. Test with a subset of 1–10% of the data to reduce time and cost. Fix common issues: credentials, paths, read options (delimiter/header), permissions and type incompatibilities (string vs numeric).

Step 4: Adjust settings for running as a Spark Job

Before publishing, ensure the Notebook does not depend on interactive variables. Replace hardcoded values with parameters when necessary. At the top of the Notebook, add a block to read parameters when executed as a job. This allows reusing the same Notebook across environments (dev/prod) and daily datasets.

import os
# Parâmetros com valores por defeito
input_path = os.environ.get('INPUT_PATH', input_path)
output_path = os.environ.get('OUTPUT_PATH', output_path)

Also check the kernel and desired Spark version. Confirm dependencies (external libraries) and add instructions to install them in the job environment if needed.

Step 5: Publish the Notebook as a Spark Job

In the Notebook menu, choose "Publish as Spark Job" (Publicar como Spark Job). Define a Job name, description and select the Notebook bundle. Configure the cluster type (Serverless Spark or a Spark Pool/Warehouse) and the size (CPU/memory). Example: for 5 GB daily, choose 8 cores and 32 GB RAM; for small workloads use 2 cores and 8 GB. These choices directly impact cost and execution time.

Step 6: Define Job parameters and variables

In the Spark Job configuration, add environment parameters or arguments that the Notebook expects (for example INPUT_PATH, OUTPUT_PATH). This makes the job reusable for different environments (dev/prod) and dates. Use naming conventions and include default values for quick testing.

{
  "INPUT_PATH": "one:///lakehouse/sales/raw/sales_2026-08-01.csv",
  "OUTPUT_PATH": "one:///lakehouse/sales/processed/sales_2026-08-01.parquet"
}

Step 7: Schedule and configure retries/alerts

In the Spark Job scheduling section, create a recurrence (daily, hourly or CRON). For example, use daily at 02:00 UTC for nightly workloads. Define retry policies (e.g.: 3 attempts with exponential backoff) and notifications (email or integration with monitoring tools). Confirm the timezone and execution window to avoid overlap with other maintenance windows.

Verify the result

After the Spark Job runs, verify: 1) the Job status in the Jobs panel (Succeeded/Failed), 2) run logs for messages, number of tasks and execution time (for example: 12 min, 8 executors), 3) presence and integrity of the output file in OneLake/Lakehouse. Open the Parquet with a Notebook or use the Lakehouse view to confirm schema and counts (count() / sample()).

Conclusion

You now have a Notebook converted into a Spark Job in Microsoft Fabric, ready to run automatically with parameters and scheduling. Recommended next steps: add data unit tests, use secrets for credentials, version Notebooks and integrate with a Pipeline for orchestration. Practical tip: start by scheduling reduced runs (1% sample) to validate costs and performance before moving to full production.