How to add logging to an ETL pipeline: step by step

An ETL pipeline that runs silently is a problem waiting to happen: when something fails in the middle of the night, you have no idea where or why. Adding logging to an ETL pipeline gives you a clear record of every stage — extract, transform and load — so you can catch errors early, measure how long it takes and know how many rows were processed. Unlike a simple print(), logging gives you severity levels, automatic timestamps and the option to write to several destinations at once. You will build, step by step, a simple logging and monitoring setup in Python using only the standard library.

Prerequisites

• Python 3.9 or later installed.
• An ETL pipeline, even a small one (read data, transform, write).
• Basic knowledge of Python functions.
• A code editor, such as VS Code.

Step 1: Set up basic logging

The logging module is part of the Python standard library, so there is nothing to install. Start by defining the message format and the minimum level you want to record. The INFO level is ideal for day-to-day progress; keep DEBUG for when you are actually investigating a problem.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

logger = logging.getLogger("etl")
logger.info("Pipeline ETL iniciado")

Step 2: Log every stage of the pipeline

Log the start of each stage and, above all, the number of rows processed. These counts are the simplest form of monitoring: if one day you extract 0 rows where you used to get thousands, the log flags the problem straight away. Use the INFO level for the normal flow and WARNING when something is odd but does not stop the pipeline.

def extract():
    logger.info("Extração: a ler dados da origem")
    rows = read_source()          # a tua função de leitura
    logger.info("Extração: %d linhas lidas", len(rows))
    return rows

def transform(rows):
    logger.info("Transformação: a aplicar regras")
    clean = [r for r in rows if r["valor"] is not None]
    logger.info("Transformação: %d válidas de %d", len(clean), len(rows))
    return clean

Step 3: Save logs to a file with rotation

Seeing messages on screen helps during development, but in production you need to store them. The RotatingFileHandler writes to a file and automatically creates a new one when the current file reaches a size limit, avoiding giant files that fill up the disk.

from logging.handlers import RotatingFileHandler

file_handler = RotatingFileHandler(
    "etl.log", maxBytes=1_000_000, backupCount=5
)
file_handler.setFormatter(
    logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")
)
logger.addHandler(file_handler)

Step 4: Catch and log errors

Wrap each stage in a try/except block. The logger.exception() method records the message and the full traceback, which makes analysing the error much faster. Re-raise the exception afterwards so the pipeline stops in a controlled way instead of carrying on with incomplete data. That way errors are never missed and are stored with all their context.

def load(rows):
    try:
        logger.info("Carga: a gravar %d linhas", len(rows))
        write_target(rows)        # a tua função de escrita
        logger.info("Carga: concluída com sucesso")
    except Exception:
        logger.exception("Carga: falhou ao gravar no destino")
        raise

Step 5: Measure how long each run takes

Duration is a very useful monitoring metric: a pipeline that usually takes 2 minutes and suddenly takes 30 is a clear warning sign. Use time.perf_counter() to measure and log the total time of each run.

import time

def run_pipeline():
    start = time.perf_counter()
    logger.info("=== Execução iniciada ===")
    rows = transform(extract())
    load(rows)
    elapsed = time.perf_counter() - start
    logger.info("=== Execução concluída em %.1f s ===", elapsed)

if __name__ == "__main__":
    run_pipeline()

Check the result

Run the pipeline and open the etl.log file. You should see one line per stage, with date, time, level and message — including the row counts and the final duration. To test error handling, force a failure on purpose (for example, point the read to a file that does not exist) and confirm that an ERROR line appears with the traceback. Then filter just the problems with a quick command:

grep ERROR etl.log

Conclusion

With just a few lines of code you have gone from a "black box" pipeline to an observable process: you know what runs, how long it takes and where it fails. The natural next step is to send these logs to a central tool, such as Azure Monitor or Application Insights, and create automatic alerts whenever an ERROR appears. Which metric do you think is more important to watch in your pipeline: the number of rows processed or the run time?