How to Create a Spark Job Schedule in Microsoft Fabric: step by step

Automating the execution of Spark Jobs in Microsoft Fabric is useful for ETL pipelines, data preparation and recurring tasks. This tutorial shows how to create a Spark Job, configure it with parameters, schedule it and manage errors to ensure reliable runs. We will cover why each option is used and provide practical examples with plausible numbers (memory, cores, retries) so you can apply them directly.

Prerequisites

• An account with access to a workspace in Microsoft Fabric and contributor permissions. Without this permission you cannot create Jobs or schedule triggers.
• A working Spark Notebook in the workspace (PySpark or Spark SQL). If you already have a notebook that reads from OneLake/Lakehouse and writes parquet, you are almost ready.
• Basic knowledge of PySpark / Spark SQL and the Fabric UI. Being able to read logs and adjust memory/cores helps avoid OOM.
• Practical recommendation: test with a sample dataset (10k–100k rows) before going to production with millions of rows.

Step 1: Prepare a Notebook with parameters

Why: using parameters allows reusing the same Notebook for different inputs, dates or environments (dev/prod). It also makes debugging simpler — you only change the arguments in the Job UI, not the code.

# Exemplo PySpark no Notebook
from pyspark.sql import SparkSession
from datetime import datetime

# widgets para parametrização no Fabric
dbutils.widgets.text("input_path", "/lakehouse/default/mydata")
input_path = dbutils.widgets.get("input_path")

dbutils.widgets.text("output_root", "/lakehouse/default/output")
output_root = dbutils.widgets.get("output_root")

spark = SparkSession.builder.getOrCreate()
df = spark.read.format("parquet").load(input_path)
# pequena transformação
df2 = df.filter("value IS NOT NULL")

out_path = f"{output_root}/{datetime.now().strftime('%Y%m%d_%H%M%S')}"
df2.write.mode("overwrite").parquet(out_path)
print(f"Wrote to {out_path}")

Concrete example: in a test environment set input_path = /lakehouse/dev/sample (≈50k rows) and output_root = /lakehouse/dev/out. In a production environment use paths partitioned by date.

Step 2: Create a Spark Job in the workspace

Simple explanation: a Spark Job is a managed resource that runs a Notebook with an execution pool. In the Fabric UI, open the Jobs / Spark section and choose "Create new job". Specify the Notebook and the Spark pool (e.g., a pool with 2 workers, each with 4 vCPU and 8 GB RAM).

Practical example: for a medium dataset (1–10M rows) start with driverMemory=4g, executorCores=2 and 4 executors; then adjust according to actual usage. Record the initial configuration to compare costs.

Step 3: Define Job parameters and configurations

Why: parameterizing the Job lets you change input_path or other options without editing the Notebook. In the Job form, add the corresponding widgets/args — for example input_path and output_root. Here you also set Spark configs (driverMemory, executorMemory, executorCores) and tags for billing.

# Exemplo de parâmetros no Job UI
input_path = /lakehouse/default/mydata
output_root = /lakehouse/default/output
-- Spark configs --
driverMemory = 4g
executorMemory = 8g
executorCores = 2
numExecutors = 4

Note: document the choices (e.g., "executorMemory 8g for 2 vCPU per executor"), because this affects costs and performance. If you see excessive GC, increase memory or reduce shuffle partitions.

Step 4: Configure the schedule

Explanation: define when the Job runs automatically — daily, hourly or via cron. In the Job, choose "Schedule" and configure a recurring trigger. To avoid overlap, enable "Max concurrent runs = 1" or set retry and backoff policies.

# Exemplos de opções comuns no UI
Schedule: Recurring daily at 02:00
Timezone: Europe/Lisbon
Retry policy: 2 retries, backoff 5 minutes (exp. backoff opcional)
Max concurrent runs: 1

Concrete example: schedule a daily run at 02:00 for ETL jobs that process the previous day's data. For hourly pipelines use cron (e.g., "0 * * * *" for at the start of each hour). If the expected runtime per run is 30–45 min, avoid triggering every 15 minutes.

Step 5: Add notifications and failure strategies

Why: receiving alerts and having retries improves operational reliability. In the Job, configure e‑mail/webhook in Notifications and define the retry policy. Also record the job state in OneLake for auditing and reconciliation.

# Boas práticas
- Enable email on failure para a equipa de operações
- Set 2 retries com exponential backoff (ex.: 5m, 15m)
- Write job status to /lakehouse/default/job_status as parquet/json

Example of writing status in the Notebook (simple): write a JSON file with status, start_time, end_time and rows_processed to enable monitoring dashboards.

Step 6: Test manually before scheduling

Explanation: run the Job manually with test parameters to confirm the Notebook and connections to OneLake/Lakehouse work. Observe the runtime (for example 12 min on the first run, 8 min on rerun) and adjust resources as needed.

Verify the result

Confirm the Job ran and produced output:

1. In the Jobs section view the run history and status (Success/Failed). Check start/end times and duration (useful for SLAs).
2. Check the logs for messages, warnings and exceptions; copy relevant stack traces to the incident system.
3. Confirm the parquet/files were written to the specified path in OneLake/Lakehouse and validate size/partitions (e.g., 3 parquet files, 120 MB total).
4. Check failure notifications/e‑mail if configured and the record in /lakehouse/default/job_status.

Conclusion

By creating a scheduled Spark Job in Microsoft Fabric with parameters, scheduling and notifications, you automate repetitive ETL tasks robustly. Next steps: integrate the Job into a broader pipeline, add unit tests in the Notebook and use execution metrics to optimize costs. Practical tip: start by scheduling in off-peak hours, monitor the first 7–14 runs and adjust resources and retries according to the observed failure and latency patterns.