AI-901: how to set up continuous evaluation pipelines in Microsoft Foundry

I will teach you the skill of setting up continuous evaluation pipelines in Microsoft Foundry — a capability relevant for the AI-901 because it demonstrates practical understanding of how to ensure AI models maintain performance and compliance in production. In practice, this helps detect data drift, performance regression and quality issues before they affect users.

What you need to know

A continuous evaluation pipeline automates the collection of metrics, comparison with baselines and actions (alerts, retraining, human review) when anomalies arise. The essential components are:

• Production data: records of model inputs and outputs for evaluation.
• Metrics: accuracy, recall, F1, mean error, calibration, latency, field coverage, etc.
• Baselines and thresholds: reference values that define acceptability.
• Drift detection: statistical tests or metrics (e.g. PSI, KL divergence) to compare distributions.
• Actions: alerts, triggers for retraining, routes for human validation.

Example: imagine a customer support classification model. The pipeline collects predictions and true labels (when available), computes daily F1, compares it with the baseline (e.g. F1>=0.78) and sends an alert to the ML team if it falls below the threshold or if the PSI of the "texto" field indicates drift.

How it works

In practical terms in Microsoft Foundry, a continuous evaluation pipeline follows these steps:

1. Instrumentation: instrument the service that serves the model to log inputs, outputs and metadata (timestamps, model version, context).
2. Ingestion to OneLake: route those logs to an appropriate zone in OneLake for processing and auditing.
3. Processing and aggregation: use Foundry pipelines (Spark/Notebooks/Jobs) to clean, join with delayed labels and compute metrics.
4. Comparison with baselines: store baselines (in a config file or table) and compare current metrics with those values.
5. Drift detection: apply statistical tests or anomaly detection models over features or embeddings.
6. Automated actions: when thresholds are exceeded, publish events to the monitoring system, create tickets or start a retraining workflow.

A simple schema in pseudo-code for the pipeline logic:

# Pseudo-código de um job de avaliação contínua
# 1. Carregar dados de previsões e rótulos
preds = read_from_onelake('predictions/stream')
labels = read_from_onelake('labels/delayed')

# 2. Juntar e calcular métricas
data = join(preds, labels, on='id')
metrics = compute_metrics(data, metrics=['precision','recall','f1','latency'])

# 3. Calcular deriva para uma feature text
psi = compute_psi(reference_feature_dist, current_feature_dist)

# 4. Comparar com baseline
if metrics['f1'] < baseline['f1_threshold'] or psi > baseline['psi_threshold']:
    publish_alert({ 'metrics': metrics, 'psi': psi, 'model_version': model_version })
    trigger_retrain_if_configured(model_version)

In practice

Concrete steps you can follow in Foundry to implement this:

1. Create a logging scheme: define the tables/paths in OneLake for predictions, inputs, labels and metadata. Use clear schemas (id, timestamp, model_version, features, prediction, probability, label).
2. Instrument the inference endpoint: when the service responds, post to the "predictions" path with a mini-batch to reduce cost and latency.
3. Define baselines under version control: store a JSON file or table with thresholds and reference versions of the training/validation set.
4. Create a recurring job in Foundry (e.g. daily) that runs the pseudo-code above. Use notebooks with Spark for scalability and store results in metrics tables.
5. Configure alerts: connect the job to a notification system (e-mail, Teams, ServiceNow) and to a workflow for human review or automatic retraining.
6. Log everything for audit: store model versions, configurations and alert events to explain decisions and meet Responsible AI requirements.

Common mistakes

• Not collecting delayed labels: many pipelines fail because there are no processes to collect the true labels that allow calculating real metrics. Plan how to obtain and join those labels.
• Using static thresholds without validation: arbitrary thresholds can generate noisy alerts. Define them based on historical analysis and re-evaluate periodically.
• Ignoring costs and privacy: logging full inputs (e.g. sensitive text) can violate policies. Anonymize/mask data and consider storage and processing costs.

How to practice

To consolidate this skill, use Microsoft's official resources. Take the official and free Microsoft Practice Assessment for AI-901 and consult the official study guide (both are free). Practice with a small project in Microsoft Foundry: build a simple model (e.g. classification), serve it, instrument logs in OneLake and implement a daily evaluation job that computes F1 and PSI. Try varying thresholds and simulating drift to see alerts in action.

In summary

• Continuous evaluation pipelines automate metrics, drift detection and actions to keep models reliable in production.
• In Microsoft Foundry, use OneLake for storage, jobs/notebooks for processing and configure alerts/workflows for automated responses.
• Avoid pitfalls: ensure collection of labels, set data-driven thresholds and protect privacy/costs.
• Practice with the official Practice Assessment and Microsoft study guide; implement a small end-to-end workflow in Foundry.