How to Use PySpark in a Microsoft Fabric Notebook

João Barros 15 de July de 2026 4 min read

Microsoft Fabric brings data engineering, data science, and Power BI together in one place, and the Notebook is the ideal tool for transforming data with code. Knowing how to use PySpark in a Microsoft Fabric Notebook lets you read, transform, and save data as a Delta table in the Lakehouse in a reproducible way — and this guide shows you how, step by step, from scratch. PySpark is the Python API for Apache Spark, the distributed engine Fabric uses to process data at scale.

Prerequisites

A Microsoft Fabric workspace backed by an active capacity (the free trial works).
A Lakehouse you can write to (Contributor role or higher).
Basic Python knowledge — no prior Spark experience required.

Step 1: Create the Notebook

In your workspace, select + New item and choose Notebook. Alternatively, open your Lakehouse and use Open notebook > New notebook. The Notebook opens with an empty cell and uses PySpark as the default language — exactly what we need.

There is no cluster to configure: Fabric provides a starter Spark pool that spins up in a few seconds the moment you run your first cell.

Step 2: Attach the Lakehouse to the Notebook

On the left, in the Explorer, select Add (or Add Lakehouse) and pick your Lakehouse. Once it becomes the default Lakehouse, you can use relative paths such as Files/ and Tables/ and write tables straight into it. This step is essential: without an attached Lakehouse, the command that saves the table later has nowhere to write.

Step 3: Read data with PySpark

So the example is reproducible without uploading files, we'll build a small in-memory DataFrame. Paste this into the first cell and run it with Shift+Enter:

from pyspark.sql import Row

sales = [
    Row(product="Keyboard", category="Accessories", quantity=3, price=25.0),
    Row(product="Monitor", category="Screens", quantity=1, price=180.0),
    Row(product="Mouse", category="Accessories", quantity=5, price=15.0),
    Row(product="Laptop", category="Computers", quantity=2, price=950.0),
]
df = spark.createDataFrame(sales)
display(df)

The display() function renders the DataFrame as an interactive table. If instead you want to read a file already in the Lakehouse Files section, the pattern is: spark.read.format("csv").option("header", "true").load("Files/sales.csv").

Step 4: Transform the data

With the DataFrame loaded, we'll add a calculated column for the per-row total and keep only sales above 50. PySpark uses the col() function to reference columns:

from pyspark.sql.functions import col

df_total = df.withColumn("total", col("quantity") * col("price"))
df_filtered = df_total.filter(col("total") > 50)
display(df_filtered)

Next, we aggregate the total by category — the kind of summary that typically feeds a report:

from pyspark.sql.functions import sum as sum_

sales_by_category = df_filtered.groupBy("category").agg(sum_("total").alias("total_by_category"))
display(sales_by_category)

Step 5: Save as a Delta table

In Fabric, Delta is the default table format. Delta adds a transaction log on top of Parquet files, bringing ACID reliability and version history to your tables. To save the result in the Lakehouse Tables section, use saveAsTable with overwrite mode (it replaces the table if it already exists):

sales_by_category.write.format("delta").mode("overwrite").saveAsTable("sales_by_category")

If you'd rather append rows to an existing table instead of replacing it, swap overwrite for append.

Check the result

Refresh the Lakehouse Explorer (three-dot menu > Refresh) and confirm that the sales_by_category table appears under Tables. To verify the data in code, query the table with Spark SQL:

result = spark.sql("SELECT * FROM sales_by_category ORDER BY total_by_category DESC")
display(result)

You should see one row per category with its total. The same table is also available through the Lakehouse SQL analytics endpoint, ready to use in a Power BI report.

Conclusion

In just a few steps you created a Notebook, attached a Lakehouse, and used PySpark to read, transform, and save data as a Delta table — the foundation of any data engineering pipeline in Fabric. From here, try reading real data from the Files section, scheduling the Notebook inside a Data Pipeline, or applying the Medallion architecture (Bronze, Silver, Gold). What is the first transformation you need to automate in your project?