How to create text featurization in Python for data: step by step

João Barros 04 de August de 2026 5 min read

Transforming text into numerical features is essential for Machine Learning and data analysis applications. This tutorial shows how to perform text featurization in Python for data, including simple cleaning, TF‑IDF, n‑grams and dimensionality reduction, useful for classification or clustering. We will explain why each step is taken and provide concrete examples that work on small datasets (10–1 000 rows) and medium datasets (up to 100 000 rows).

Prerequisites

Python 3.8+ and pip
Libraries: pandas, scikit‑learn, nltk (install with pip)
Basic knowledge of pandas and Python

For large‑scale text processing also consider tools like spaCy or distributed solutions, but for prototypes and feature engineering pipelines scikit‑learn + nltk are sufficient.

Step 1: Install and import dependencies

Install the required libraries and import the modules. We use nltk for tokenization and scikit‑learn for TF‑IDF and dimensionality reduction. Downloading nltk resources is a one‑time step.

pip install pandas scikit-learn nltk

# Código Python
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.pipeline import make_pipeline
import nltk
nltk.download('punkt')

Why: TfidfVectorizer converts text into a sparse matrix, memory‑efficient. TruncatedSVD works well on sparse matrices (unlike PCA).

Step 2: Load and inspect text data

Load a dataset with a text column. Here we create a minimal example with sentences to test featurization. In real scenarios, the column may be comments, product descriptions or review texts (e.g., 10k–100k records).

# Exemplo mínimo de DataFrame
data = {
    'id': [1, 2, 3, 4],
    'texto': [
        'O produto é excelente e muito rápido',
        'Entrega atrasada, mau atendimento',
        'Qualidade boa, preço razoável',
        'Produto com defeito, devolvi e reembolsaram'
    ]
}
df = pd.DataFrame(data)
print(df.head())

Check for null values, duplicates or texts that are too short — these cases often require specific handling (for example, removing or imputing).

Step 3: Basic text cleaning (normalization)

Normalizing text reduces noise: lowercasing, removing punctuation and irrelevant tokens. We keep the process simple so it is easy to adapt. In Portuguese, be mindful of accents; we kept accented characters in the regex to preserve meaning.

import re
from nltk.tokenize import word_tokenize

def limpar_texto(s):
    s = s.lower()
    s = re.sub(r'[^a-záâãàçéêíóôõúü0-9\s]', ' ', s)
    tokens = word_tokenize(s)
    tokens = [t for t in tokens if len(t) > 2]
    return ' '.join(tokens)

df['texto_limpo'] = df['texto'].apply(limpar_texto)
print(df[['texto','texto_limpo']])

Tip: for larger datasets add stopword removal (for example stop_words='portuguese' in TfidfVectorizer) and more advanced normalization (stemming/lemmatization) if needed.

Step 4: Create features with TF‑IDF and n‑grams

TF‑IDF weights terms by document frequency vs corpus frequency — useful to downweight very common terms. Important parameters include ngram_range, max_df and min_df.

vectorizer = TfidfVectorizer(ngram_range=(1,2), max_df=0.8, min_df=1)
X_tfidf = vectorizer.fit_transform(df['texto_limpo'])
print('TF‑IDF shape:', X_tfidf.shape)
print('Algumas features:', vectorizer.get_feature_names_out()[:10])

Parameter explanation: ngram_range=(1,2) captures unigrams and bigrams; max_df=0.8 ignores terms present in >80% of documents (possible noise); min_df=1 includes terms that appear at least once — in production consider using min_df=2 or min_df=0.01 (1% of the corpus) to reduce dimensionality.

Step 5: Reduce dimensionality with TruncatedSVD

Dimensionality reduction helps models and visualizations. TruncatedSVD (LSA) preserves important variation. Choose n_components according to the case: 2 for visualization, 20–100 for modeling. In datasets with 10k documents it's common to use 50–200 components.

n_components = 2
svd = TruncatedSVD(n_components=n_components, random_state=42)
X_reduced = svd.fit_transform(X_tfidf)
# Adicionar ao DataFrame para utilização em modelos
for i in range(n_components):
    df[f'feature_text_{i+1}'] = X_reduced[:, i]
print(df.head())

Note: the TF‑IDF matrix is sparse, so TruncatedSVD is efficient. Check the percentage of explained variance (svd.explained_variance_ratio_.sum()) to see how much information was preserved.

Step 6: Complete and reusable pipeline

Creating a pipeline makes it easy to apply the same process to new data without repeating code. This ensures reproducibility — important in production and A/B tests.

pipeline = make_pipeline(
    TfidfVectorizer(ngram_range=(1,2), max_df=0.8, min_df=1),
    TruncatedSVD(n_components=50, random_state=42)
)

# Treinar pipeline e transformar
X_features = pipeline.fit_transform(df['texto_limpo'])
print('Features shape (pipeline):', X_features.shape)

Example: with 1 000 documents and 20 000 TF‑IDF features, TruncatedSVD with 50 components reduces to (1000, 50) — much more manageable for models like LogisticRegression or RandomForest.

Verify the result

Verify you obtained a numerical matrix with expected dimensions and that the features make sense: inspect outliers, similarity between records (cosine similarity) and highlight principal components. Test the pipeline with a simple classifier and validate via cross‑validation.

# Verificar shape e algumas linhas
print('DataFrame final:\n', df.head())
print('Matriz reduzida exemplo (primeiras 3 linhas):\n', X_reduced[:3])

# Erro comum: ter muitas features vazias - ajustar min_df/max_df ou remover stopwords

Conclusion

You have implemented text featurization in Python for data: cleaning, TF‑IDF with n‑grams and dimensionality reduction. Experiment with parameters like ngram_range, min_df, max_df and n_components for your case; for example, in reviews with 50k records try min_df=5 and n_components=100. Next steps include training a classifier (e.g., LogisticRegression) and measuring metrics (accuracy, F1), or using pre‑trained embeddings if you need greater capacity without manual feature engineering. Practical tip: keep the pipeline versioned to reproduce results in production.