AI-901: validate and evaluate AI models with Microsoft Foundry

I will teach how to validate and evaluate AI models inside Microsoft Foundry — an essential skill for AI-901 because it ensures the model meets quality requirements before being used in production. Knowing how to measure performance, choose metrics and create validation sets is critical both for the exam and for real-world scenarios.

What you need to know

Validating and evaluating a model means measuring its behavior with data that was not used for training, identifying failures and quantifying its usefulness for the task. In Foundry, this involves: preparing a validation set, running controlled inference and calculating appropriate metrics (for example, accuracy, precision/recall, F1, AUC for classification; MAE, RMSE for regression). It also includes assessing robustness, bias and temporal/latency performance.

Simple example: you have a binary classifier to detect spam messages. After training in Foundry, you prepare a validation set with labeled messages (spam/not spam) that were not in the training. You run an inference job in Foundry that applies the model to the validation set and generate metrics such as accuracy, recall and F1. If recall is very low, many spam messages slip through — it is a signal to adjust the threshold, collect more examples or review preprocessing.

How it works (step-by-step)

Follow a practical workflow applicable in Foundry:

1. Define evaluation objectives:

   Determine what matters: minimize false positives, maximize recall, reduce latency, or ensure fairness. That decision guides the metrics and the test design.

2. Prepare validation and test sets:

   Separate data that was not used for training. Ideally you have three sets: training, validation (for tuning) and test (final evaluation). In Foundry, use files in OneLake or versioned tables.

3. Run controlled inference:

   # Exemplo conceptual de pipeline de inferência
   # 1. Carregar dados de validação
   # 2. Aplicar transformações idênticas às do treino
   # 3. Chamar o modelo para previsões
   # 4. Guardar previsões para análise
   

   In Foundry, this is typically a Job that reads data, applies the model and writes versioned results.

4. Compute relevant metrics:

   For classification: confusion matrix, accuracy, precision, recall, F1, AUC. For regression: MAE, RMSE, R². For language models: perplexity, BLEU (for generation) or task-specific metrics. Implement the calculation logic as part of the pipeline or use Foundry components/visualizations.

5. Additional tests:

   • Robustness: test with noise/adversarial inputs.
   • Temporal generalization: validate with recent data to detect degradation.
   • Fairness and bias: analyze metrics by demographic subgroups.

6. Document and version results:

   Store metrics, datasets and the model artifact (version) in Foundry for audit and result regression.

In practice — quick example

Imagine a classification pipeline that already exists in Foundry. To add an evaluation step you can:

1. Create a dataset validation_set in OneLake with N labeled records.
2. Create an inference Job that:

# Pseudocódigo do Job
validation = read_table('oneLake://project/validation_set')
preprocessed = apply_transform(validation)
preds = model.predict(preprocessed)
results = join(validation.labels, preds)
save_table(results, 'oneLake://project/eval_results')
# Calcular métricas (no mesmo Job ou num Job separado)
metrics = compute_metrics(results)
save_table(metrics, 'oneLake://project/eval_metrics')

Then, create a dashboard in Foundry that shows the evolution of metrics by model version and by date. This helps detect drift (data drift) or performance drops quickly.

Common mistakes

• Using the same data for training and validation: leads to overly optimistic metrics. Always separate sets or use cross-validation.
• Choosing metrics without connection to business requirements: for example, focusing on accuracy when the cost of false negatives is high. Define metrics aligned with real impact.
• Ignoring subgroups and fairness: global average performance can hide severe failures in specific groups.

How to practice

To practice this skill, use the OFFICIAL and free Microsoft Practice Assessment and the official study guide (also free). These resources help you verify general knowledge for the AI-901 exam and guide practical study on topics like model validation. Additionally, in Foundry, create small end-to-end projects: train a simple model, version it, build an evaluation Job and record metrics over time.

In summary

• Validating models requires separate sets, appropriate metrics and robustness tests to ensure quality before production.
• In Microsoft Foundry, build inference and evaluation pipelines that version data, results and artifacts for traceability.
• Choose metrics aligned with business risk and analyze performance by subgroups to detect bias.
• Practice with Microsoft official resources (Practice Assessment and study guide free) and implement simple pipelines in Foundry to gain practical experience.