(+351) 21 24 10006  ·  info@bconcepts.pt
Carnaxide, Lisbon

AI-901: how to configure inference monitoring mechanisms in Microsoft Foundry

João Barros 15 de November de 2026 4 min read

I will show how to configure an inference monitoring mechanism in Microsoft Foundry — a practical skill valuable for the AI-901. This capability is important because, in practice, it ensures that production models continue to deliver correct results and meet performance requirements.

What you need to know

Inference monitoring consists of collecting signals from production model calls — for example, latency, error rate, input distribution, output distributions and confidence metrics. In Microsoft Foundry, this involves configuring pipelines that collect telemetry from inference calls, store samples (inputs/outputs) in OneLake, and generate alerts or reports when anomalies occur (for example, concept drift or increased latency).

Example: you have an image classification endpoint. Monitoring should record response time per request, the proportion of labels with low confidence (< 0.6), and the distribution of predicted classes. If a previously rare class suddenly spikes, that may indicate a change in production data or an error in the acquisition pipeline.

How it works (step-by-step)

The typical configuration follows these steps:

  1. Endpoint instrumentation: enable logging on inference calls to capture timestamp, payload (or a summary/embedding), latency and score/confidence.
  2. Routing to OneLake: configure Foundry to send inference events and samples to a controlled location in OneLake, with an appropriate retention policy.
  3. Building aggregation pipelines: create pipelines in Foundry that compute aggregated metrics (e.g. latency percentiles, error rate, class distribution) over time windows.
  4. Anomaly detection: apply simple rules (thresholds) and/or anomaly detection models that notify when metrics deviate from normal.
  5. Alerts and dashboards: configure alerts (email, webhook) and dashboards (Power BI or integrated tooling) for continuous visualization.
  6. Response workflow: define playbooks that describe steps to take — for example, collect more samples, retrain the model, roll back to the previous version, or scale the infrastructure.

Example of instrumentation setup (simple pseudocode for the client calling the endpoint):

// Pseudocódigo: chamar endpoint e enviar evento de telemetria
start = now()
response = call_model_endpoint(input_payload)
latency = now() - start
telemetry = {
  "timestamp": start,
  "latency_ms": latency,
  "confidence": response.confidence,
  "predicted_label": response.label,
  "input_summary": summarize(input_payload) // p.ex. hashes, features, embedding
}
send_to_foundry_telemetry(telemetry)
store_sample_in_onelake(input=maybe_redact(input_payload), output=response)

Practical notes: redact sensitive data before storing, and sample to avoid high costs. Instead of saving the entire payload, store summaries (hashes) or embeddings.

In practice: examples of metrics to monitor

  • Latency: mean, p95, p99 per endpoint/version.
  • Error rate: percentage of responses with HTTP error or exception.
  • Confidence: percentiles of probabilities/score, and proportion below the threshold.
  • Class/output distribution: comparison between training and production (drift detection).
  • Request volume: to detect unexpected spikes or drops.

Common mistakes

1) Collecting too much information without anonymizing: storing full input files without redaction can violate privacy policies and make the system unusable for legal reasons. Instead, redact or sample.

2) Not defining thresholds or reference metrics: monitoring without a baseline makes it hard to distinguish normal fluctuations from real anomalies. Define baselines during an initial stability window.

3) Forgetting version instrumentation: if you don’t log the model version in telemetry, it’s hard to correlate behavior changes with model updates.

How to practice

Practice with official and free Microsoft resources: take the OFFICIAL Practice Assessment for AI-901 and review the Microsoft study guide for AI-901. Use Foundry test environments (or Azure subscriptions with quotas) to create a simple endpoint, instrument it as described, and build a pipeline that computes basic metrics and triggers alerts. These official resources are free for evaluation and study. Do not use dumps or unofficial materials.

In summary

  • Inference monitoring collects telemetry (latency, errors, confidence, distribution) to watch production models.
  • In Foundry, send events to OneLake, aggregate in pipelines and apply anomaly detection/alerts.
  • Protect sensitive data: redaction and sampling are essential for privacy and cost control.
  • Define baselines and log model versions to quickly diagnose regressions and drift.