How to create and use a Synapse Notebook to explore data in Azure Synapse Analytics

João Barros 27 de August de 2026 5 min read

This tutorial shows, step by step, how to create and use a Synapse Notebook in Azure Synapse Analytics to read Parquet files stored in a Data Lake, explore data, apply simple transformations and write results. It is a hands-on approach for exploratory data analysis (EDA), schema validation and data preparation for pipelines or machine learning models. In real scenarios, a Notebook is very useful for interactively working with datasets from a few megabytes up to hundreds of gigabytes.

Prerequisites

Azure account with permissions for an Azure Synapse Analytics workspace.
A Synapse Workspace configured and a Spark Pool available (for example, 3–8 nodes depending on data volume).
Parquet files in a container in Azure Data Lake Storage Gen2 (access via Linked Service, SAS or managed identity).
Basic knowledge of PySpark and Python; minimal experience with SQL is useful.

Practical suggestion: for datasets < 1 GB, a Spark Pool with 3 nodes (each with ~8–16 GB) is sufficient; for 50–200 GB consider 8 nodes with 32 GB each. Estimating correctly helps avoid out-of-memory failures or unnecessary costs.

Step 1: Open Synapse Studio and create a Notebook

Open Synapse Studio in the Azure portal, click Develop and create a new Notebook. Choose a descriptive name, for example Exploracao_Parquet. Associate the Notebook with a Spark Pool (in the Notebook's top bar choose the pool). If you are testing, use a small pool; for production, prefer pools whose nodes have at least 16–32 GB of RAM per node.

Best practices: when working with large files, start the cluster only when needed and stop it outside work hours to control costs. Use notes inside the Notebook to describe each step and the parameters used.

Step 2: Configure access to the Data Lake

Check how you will access the storage: Linked Service (recommended for production), SAS token (useful for quick tests) or managed identity (more secure when configured). In the Notebook you can mount the ABFSS path directly or use the az storage APIs. If the Workspace has access configured, use ABFSS to avoid sharing keys.

# Path para o ficheiro Parquet
path = "abfss://container@storageaccount.dfs.core.windows.net/pasta/exemplo.parquet"

Example: if your container has 12 Parquet files of 1 GB each, the path can point to the folder (e.g. /pasta/*.parquet) to read them all at once.

Step 3: Read the Parquet file with PySpark

Use the Spark session available in the Notebook to read the Parquet into a DataFrame. This allows inspecting the schema, counting records quickly and measuring read times.

# Ler ficheiro Parquet
df = spark.read.parquet(path)

# Ver schema e primeiras linhas
df.printSchema()
df.show(10)

Useful indicator: after reading, run df.count() (only for small datasets) or use df.rdd.getNumPartitions() to see the initial partitioning. Reading 10 million rows can take anywhere from seconds to minutes, depending on the network and number of partitions.

Step 4: Explore and clean the data (practical example)

Explore statistics, null values and apply transformations. Here we show how to get statistics, count nulls per column and perform a simple cleanup: fill nulls and create a derived column with the year from the timestamp.

# Estatísticas básicas
from pyspark.sql.functions import col, when

df.describe().show()

# Contar valores nulos por coluna
nulls = {c: df.filter(col(c).isNull()).count() for c in df.columns}
print(nulls)

# Exemplo de limpeza: preencher nulos e criar coluna derivada
df2 = df.fillna({ 'payment_amount': 0 })
df2 = df2.withColumn('year', df2['event_timestamp'].cast('timestamp').substr(1,4))

df2.select('payment_amount','year').show(10)

Performance tip: if you will reuse df2 in multiple operations, do df2.cache() to avoid re-reads. For write workloads, use df2.repartition(10) or another number based on the desired number of output files. For example, to generate ~10 output files for 100 GB, use repartition(10) to distribute ~10 GB per file.

Step 5: Write results as Parquet or temporary tables

After transforming the data you can write the result back to the Data Lake as Parquet (compact and efficient) or register a temp view for interactive SQL queries in the same Notebook.

# Gravar como Parquet num caminho de saída
output_path = "abfss://container@storageaccount.dfs.core.windows.net/pasta/out/resultado.parquet"
df2.write.mode('overwrite').parquet(output_path)

# Criar uma view temporária para usar Spark SQL
df2.createOrReplaceTempView('vw_exploracao')

# Exemplo de query SQL
spark.sql("SELECT year, AVG(payment_amount) as avg_pay FROM vw_exploracao GROUP BY year").show()

Check the number of files written and the average size — if many small files appear, consider coalescing before writing: df2.coalesce(10).write(...).

Verify the result

Confirm that the output file exists in the Data Lake (via Azure Portal or Storage Explorer) and validate the schema. In the Notebook use spark.read.parquet(output_path).show() for a quick inspection. If permissions are missing you will see access errors (403) — check the Linked Service or managed identity. If data types are unexpected, review the transformations and checkpoints.

Conclusion

You now have a Synapse Notebook that reads Parquet, explores, cleans and writes results using Spark. Practical next steps: automate this Notebook with a Pipeline in Synapse, parameterize paths (for example, passed as widgets) and optimize the Spark Pool according to the load. Always start with small datasets to validate logic and only then scale to larger volumes — typically testing with 1–10% of the final dataset gives a good idea of performance. Would you like to see an example of parameterizing the Notebook or scheduling it with a Pipeline?