How to remove duplicate rows with pandas in Python

Duplicate rows in a DataFrame are one of the most common data-cleaning problems: they appear when you concatenate several files, import faulty exports, or collect the same records twice. Left untouched, they distort counts, averages, and reports. Fortunately, removing duplicate rows with pandas in Python is fast and safe thanks to the drop_duplicates method, and below you will see how to detect, count, and remove them with full control.

Prerequisites

• Python 3 installed and pandas available (pip install pandas).
• Knowing how to create or load a DataFrame (for example, from a CSV).
• An editor or notebook, such as Jupyter or VS Code, to run the code.

Step 1: Create a sample DataFrame

So you can follow along without external files, start by creating a small DataFrame with a few rows repeated on purpose. Notice that the row for "Ana" in "Porto" appears exactly the same twice.

import pandas as pd

df = pd.DataFrame({
    "nome": ["Ana", "Bruno", "Ana", "Carla", "Bruno"],
    "cidade": ["Porto", "Lisboa", "Porto", "Braga", "Faro"],
    "vendas": [100, 200, 100, 300, 250],
})

print(df)

When you print the DataFrame you will see five rows, two of which are identical. A row only counts as a duplicate when every value matches another row: a single different cell is enough for pandas to treat it as unique.

Step 2: Detect and count the duplicates

Before removing anything, it is good practice to check how many rows are repeated. The duplicated() method returns a series of True and False values, marking as True every repetition from the second occurrence onward. Detecting before deleting avoids surprises, because you learn how many rows you will lose and can inspect them if needed.

# Marca cada linha repetida (exceto a primeira ocorrência)
print(df.duplicated())

# Conta quantas linhas duplicadas existem no total
print(df.duplicated().sum())

In our example, only the second "Ana" row is fully identical to an earlier one, so the count returns 1.

Step 3: Remove the duplicate rows

To drop the repetitions, apply drop_duplicates(). By default, pandas keeps the first occurrence and discards the following ones. The method returns a new DataFrame and does not change the original, so store the result in a variable.

df_limpo = df.drop_duplicates()
print(df_limpo)

Tip: avoid inplace=True. Reassigning the result (as in df_limpo = df.drop_duplicates()) keeps the code clearer and easier to debug.

Here the DataFrame goes from five to four rows, because the repeated "Ana" was removed while the rest stayed intact.

Step 4: Choose which occurrence to keep

The keep parameter decides which repetition stays. Use "first" (the default) to keep the first, "last" to keep the last, or False to remove every row that has duplicates.

# Mantém a última ocorrência de cada duplicado
df.drop_duplicates(keep="last")

# Remove TODAS as linhas que aparecem mais do que uma vez
df.drop_duplicates(keep=False)

Step 5: Duplicates in selected columns only

Often a row should count as duplicated only when certain columns match, ignoring the rest. The subset parameter takes a list of columns. In the next example, two rows are equal if they share the same nome, even if the vendas value differs.

# Considera duplicado quando o "nome" se repete
df.drop_duplicates(subset=["nome"])

# Também podes comparar várias colunas ao mesmo tempo
df.drop_duplicates(subset=["nome", "cidade"])

After removing rows, the index is left with gaps in its numbering. To renumber it from 0 onward, add ignore_index=True.

df.drop_duplicates(subset=["nome"], ignore_index=True)

Verify the result

To confirm there are no more repetitions, count the duplicates again on the cleaned DataFrame: the total should be 0. Comparing the number of rows before and after also helps you see how many were removed. This small test is especially handy when you run the same script several times or when data arrives from different sources.

print("Antes:", len(df))
print("Depois:", len(df_limpo))
print("Duplicados restantes:", df_limpo.duplicated().sum())

If you see Duplicados restantes: 0, the cleanup worked as expected.

Conclusion

With duplicated() and drop_duplicates() you have everything you need to find and remove repeated rows in pandas, choosing which occurrence to keep and which columns to compare. The next step is to fold this cleanup into your usual routine, before grouping data or computing metrics, to guarantee reliable numbers. And in your case: which columns truly define a "duplicate" row?