How to add columns with withColumn in PySpark

João Barros 04 de July de 2026 4 min read

Adding new columns to a DataFrame is one of the most common tasks when transforming data in PySpark. The withColumn method lets you create calculated columns, convert data types and apply conditional rules, all in a distributed way that scales to large volumes. Below you will see, with simple examples, how to add columns with withColumn — from the most basic case to a conditional one — and how to confirm the result is correct.

Prerequisites

An environment with PySpark available (for example, Databricks, Microsoft Fabric or a local Spark install).
Basic Python knowledge and familiarity with the DataFrame concept.
A sample DataFrame to practise on — we create one in the first step.

Step 1: Create a sample DataFrame

Before adding columns, we need some data. The code below creates a SparkSession (the entry point of any Spark application) and a small DataFrame with three products, their prices and the quantities sold.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("colunas").getOrCreate()

dados = [
    ("Teclado", 25.0, 4),
    ("Rato", 15.0, 10),
    ("Monitor", 120.0, 2),
]
df = spark.createDataFrame(dados, ["produto", "preco", "quantidade"])
df.show()

The show() method prints the first rows, so you can confirm the data loaded as expected.

Step 2: Add a calculated column

The most frequent use of withColumn is to build a column from others. The method takes two arguments: the name of the new column and the expression that computes it. To refer to existing columns we import the col function. In the example, we multiply price by quantity to get each row's total.

from pyspark.sql.functions import col

df = df.withColumn("total", col("preco") * col("quantidade"))
df.show()

One important detail: withColumn always returns a new DataFrame instead of changing the original, which is why we store the result back in the df variable. This immutability is intentional and helps Spark optimise execution.

Step 3: Add a column with a fixed value

Sometimes you just need a constant column, such as the currency or the data source. In that case the expression cannot be a plain string — it must be a literal value created with the lit function.

from pyspark.sql.functions import lit

df = df.withColumn("moeda", lit("EUR"))
df.show()

If you forgot lit and passed just "EUR", Spark would try to read it as a column name and raise an error.

Step 4: Create a conditional column

To apply business rules, combine the when and otherwise functions, which work like an "if… else". In the example, we tag products whose total is above 100 as "alto" and all others as "normal". You can chain several when calls before otherwise to cover more cases.

from pyspark.sql.functions import when

df = df.withColumn(
    "categoria",
    when(col("total") > 100, "alto").otherwise("normal")
)
df.show()

Step 5: Convert a column's type

You can also use withColumn to change a column's data type with cast. Here we convert the quantity from an integer to text.

df = df.withColumn("quantidade", col("quantidade").cast("string"))
df.printSchema()

Because we gave the column the same name it already had (quantidade), withColumn replaces the original column instead of creating a new one. That is how you update a column in place.

Tip: to add many columns at once there is withColumns (plural), which accepts a dictionary of several name/expression pairs. Avoid chaining dozens of withColumn calls inside a loop, though, as it can make the execution plan slower.

Verify the result

Run df.show() to see the rows with the new columns and df.printSchema() to confirm names and types. You should see the total, moeda and categoria columns, with quantidade now of type string. If a column comes out all null, check that the name used inside col("...") exists and is written exactly as in the schema — capitalisation and accents included.

Conclusion

You can now use withColumn to create calculated, constant and conditional columns, and to convert types — the basis of almost every transformation in PySpark. As a next step, try combining these techniques to prepare a real DataFrame and save it in Parquet format. What will be the first column you need to create in your pipeline?