How to extract named entities in Python for data: step by step

Extracting named entities in Python for data allows you to transform free text into structured columns (people, organizations, locations, dates). With this approach you get columns that can feed reports, create features for Machine Learning models or automate labeling in ETL pipelines. We will explain the reason for each step and show how to apply the technique to real examples, with performance and validation tips.

Prerequisites

• Python 3.8+ installed
• pip to install packages
• Basic knowledge of Python and pandas
• Internet connection to install spaCy and download a model

I recommend a virtual environment (venv) to avoid conflicts with other libraries. For small datasets (100–1,000 texts) a common laptop is enough; for thousands or millions of texts you should use a machine with more CPU/RAM or parallelize processing.

Step 1: Install spaCy and pandas

We will use spaCy for entity recognition and pandas to manipulate the data. Install the packages and download a pre-trained model. The en_core_web_sm model is a good starting point for English; for Portuguese use pt_core_news_sm.

pip install spacy pandas
python -m spacy download en_core_web_sm
# para português, usa:
# python -m spacy download pt_core_news_sm

Note: small models are faster and lighter, but tend to be less accurate in specialized domains (e.g., legal or medical). If you need higher accuracy, consider larger models or fine-tuning.

Step 2: Prepare an example DataFrame

Create a DataFrame with texts you want to process — for example, news descriptions or comments. Keep the text in a column here called text. It is useful to have a unique id per row for traceability.

import pandas as pd

data = {
    'id': [1, 2, 3],
    'text': [
        'Apple lançará o novo iPhone em Setembro em Cupertino.',
        'O primeiro-ministro visitou Lisboa e o Museu Nacional ontem.',
        'Tesla anuncia fábrica em Berlim prevista para 2024.'
    ]
}

df = pd.DataFrame(data)
print(df)

In real scenarios you can load the CSV file with pd.read_csv() or extract from a database. If you have 10,000+ records, avoid using apply row by row without optimization (see Step 4).

Step 3: Load the spaCy model and create extraction function

Load the en_core_web_sm model (or another appropriate one). Create a function that receives a text and returns a list of entities with type and text. Keep the function simple to ease later transformation into columns.

import spacy
nlp = spacy.load('en_core_web_sm')  # usa um modelo pequeno; muda se precisares de outro idioma

def extract_entities(text):
    doc = nlp(text)
    ents = []
    for ent in doc.ents:
        ents.append({'text': ent.text, 'label': ent.label_})
    return ents

# Teste rápido
print(extract_entities('Barack Obama visited Berlin in 2013.'))

Performance tip: if you only need the NER component, you can load with nlp = spacy.load('en_core_web_sm', exclude=['parser','tagger']) to speed up. For large volumes use nlp.pipe in the next step.

Step 4: Apply extraction to the DataFrame

Use pandas.apply to create new columns with the entities. For medium to large datasets it is more efficient to use nlp.pipe which processes in batches and reduces Python overhead.

def ents_to_dict(ents):
    # transforma lista de entidades em dicionário com tipos como chaves
    out = {'PERSON': [], 'ORG': [], 'GPE': [], 'DATE': []}
    for e in ents:
        if e['label'] in out:
            out[e['label']].append(e['text'])
    return out

# Extrai todas as entidades
df['entities'] = df['text'].apply(extract_entities)
# Converte para dicionário de tipos
df_types = df['entities'].apply(ents_to_dict).apply(pd.Series)
# Junta ao DataFrame original
df = pd.concat([df, df_types], axis=1)
print(df[['id','text','entities','PERSON','ORG','GPE','DATE']])

Example with nlp.pipe (more efficient for 1k+ texts):

docs = list(nlp.pipe(df['text'].astype('str')))
# depois iteras sobre docs para construir a coluna entities

Step 5: Normalize and clean entities

Entities can repeat or have variants (e.g., "Lisboa" vs "Lisboa, Portugal"). Normalize by removing duplicates, trimming whitespace and turning lists into comma-separated strings for analysis/storage. You can also apply lemmatization or use rules to unify forms.

def normalize_list(lst):
    seen = []
    for item in lst:
        item = item.strip()
        if item not in seen:
            seen.append(item)
    return ', '.join(seen) if seen else ''

for col in ['PERSON','ORG','GPE','DATE']:
    df[col] = df[col].apply(normalize_list)

print(df[['id','PERSON','ORG','GPE','DATE']])

To remove more complex variations, consider using fuzzy matching (fuzzywuzzy) or rule-based normalization (regex) before aggregating results.

Verify the result

Confirm that the new columns contain the expected entities. Perform manual validation on samples (e.g., 100 rows) to estimate precision and recall. Look for common errors: empty entities, wrong labels due to language or weak models.

# Visualizar linhas onde GPE (local) não está vazio
print(df[df['GPE'] != ''][['id','text','GPE']])

# Ver exemplos sem entidades detetadas
print(df[df['entities'].apply(len) == 0][['id','text']])

If you find many false positives, adjust the model, use a domain-specific model or apply post-processing rules to filter irrelevant results.

Conclusion

You have completed named entity extraction in Python for data using spaCy and pandas, transforming text into structured columns ready for analysis or ML features. Next steps: test models in Portuguese (e.g., pt_core_news_sm), adjust the pipeline for normalization (lemmatization, coreference resolution) or store results in CSV/SQL with df.to_csv('entities.csv', index=False) or via SQLAlchemy. Final tip: start by validating with manual samples (e.g., 100–300 examples) to measure quality before automating in production.