How to Generate PySpark Code with Copilot in Fabric

João Barros 06 de July de 2026 4 min read

Copilot in Fabric can turn a natural-language sentence into a block of PySpark code that is ready to run inside a Notebook. For anyone starting out with Spark, this speeds up the work and, at the same time, helps you learn the syntax through real examples. The goal of this guide is clear: generate PySpark code that reads and summarizes data from a Lakehouse, using the Copilot chat pane and the %%code magic command, with a step-by-step example.

Prerequisites

A Fabric F2 or higher capacity (or P1 or higher) assigned to the workspace.
The tenant setting "Users can use Copilot and other features powered by Azure OpenAI" turned on — your Fabric admin can confirm this in the admin portal.
A Notebook inside a workspace with an attached Lakehouse and at least one sample table (here we use a table called sales).
Basic Python knowledge. You don't need to master PySpark to follow this example.

Step 1: Attach the Lakehouse to the Notebook

Copilot uses the Notebook context to give useful answers: the workspace, the attached Lakehouse, and the available schemas, tables, and files. So the first step is to make sure a Lakehouse is connected. In the Notebook, open the Explorer pane on the left and choose Add Lakehouse to connect the Lakehouse that holds your data.

This step matters more than it seems: without an attached data source, Copilot doesn't know which tables exist, and its suggestions stay generic and less accurate.

Step 2: Open the Copilot chat pane

On the Notebook ribbon, select the Copilot button. The chat pane opens on the right side of the screen. This is where you write natural-language requests and hold a multi-step conversation, which is ideal for building the solution gradually.

The first time you use the magic commands (chat-magics), the Notebook may ask you to run an initialization cell. Run that cell once and carry on as usual.

Step 3: Describe the task in natural language

In the Copilot chat pane, write a clear, specific request. The more concrete it is — the table name, the columns, and the filters — the better the generated code. A good example request is:

Read the sales table from the Lakehouse and calculate
total sales by region.

Copilot replies with a block of PySpark that you can copy into a cell. Always read the code carefully before running it: Copilot is an assistant, not a replacement for your judgment.

Step 4: Generate code inline with %%code

If you prefer to generate code without leaving the cell, use the %%code magic command. Type %%code at the top of a cell, followed by your natural-language request, and run the cell:

%%code
Load the sales table from the Lakehouse into a Spark DataFrame
and show the first 10 rows.

Copilot generates the matching PySpark code, which you can review and insert into a cell to run. The result will look similar to this:

df = spark.read.table("sales")
df.show(10)

Step 5: Refine the code with a follow-up conversation

The first result is rarely perfect, and that's fine. Because Copilot keeps the conversation context, you can ask for improvements without repeating everything. For example, type "group by region and sort from highest to lowest" and the code is adjusted:

from pyspark.sql.functions import sum, col

summary = (df
    .groupBy("region")
    .agg(sum("amount").alias("total_sales"))
    .orderBy(col("total_sales").desc()))

summary.show()

Verify the result

Run the final cell and confirm the output shows one row per region, with total sales sorted from highest to lowest. To be sure the numbers are right, compare them with a quick SparkSQL query on the same table:

%%sql
SELECT region, SUM(amount) AS total_sales
FROM sales
GROUP BY region
ORDER BY total_sales DESC

If both results match, the code generated by Copilot is correct and you can rely on it for the next step.

Conclusion

With the Copilot chat pane and the %%code magic command, you turned a natural-language description into working PySpark code without memorizing the syntax. A good next step is to ask Copilot to add comments to the code or to explain each line — it's an excellent way to learn while you work. Which repetitive task in your daily data work would you automate first with Copilot?