AI-901: how to set up and use human annotation pipelines in Microsoft Foundry

I will show how to set up and use human-in-the-loop annotation pipelines in Microsoft Foundry — a practical skill useful for the AI-901, because high-quality labeled data are essential for robust AI models in practice and are often covered in the exam both as a concept and as an implementation scenario.

What you need to know

A human annotation pipeline coordinates the input of human annotators to label or correct data that will be used to train AI models. In Foundry, this involves creating annotation tasks, defining clear instructions, managing review workflows, and integrating the annotated data back into OneLake or the project's data repository. The goal is to ensure consistency, quality, and traceability of annotations.

Simple example: you have industrial inspection images and need bounding boxes for anomalies. The pipeline asks annotators to draw boxes, assigns priorities, validates quality, and returns JSON files with the coordinates for model training.

How it works (step by step)

The general steps to create an annotation pipeline in Foundry are:

1. Prepare the data: organize files in OneLake with minimal metadata (IDs, timestamps, file type).
2. Define the annotation schema: which labels to use, output format (COCO, Pascal VOC, JSON), and level of detail.
3. Create annotation tasks in Foundry: specify instructions, examples of good/bad annotation, and acceptance criteria.
4. Assign annotators and review levels: for example, 2 independent annotators + 1 reviewer.
5. Monitor quality: metrics such as inter-annotator agreement (IoU for bounding boxes, F1 for labels) and average time per task.
6. Ingest results: validate and transform formats for the training pipeline and store versions with audit metadata.

Example of a simplified JSON schema for an annotated image (COCO-like):

{
  "image_id": "img_0001.jpg",
  "annotations": [
    {"label": "anomaly", "bbox": [x_min, y_min, width, height], "confidence": 0.0, "annotator": "user123"}
  ],
  "annotation_version": "v1",
  "review_status": "approved"
}

In practice: configuring in Foundry

1) Prepare a dataset in OneLake: create the project folder and upload images/files with minimal metadata (CSV or parquet with paths and ids). 2) In the Foundry workspace, create a new annotation task template: choose the type (image, text, audio), define labels and rules. 3) Write clear instructions: include positive/negative examples, tolerances (e.g. minimum IoU to consider agreement) and edge cases. 4) Configure the review flow: for example, each item goes through two annotators; if they disagree, send to a reviewer. 5) Launch the job and monitor the dashboard: see throughput, time per task and discrepancy percentage. 6) Export annotations in a format understood by the training pipeline (convert to COCO/JSON/CSV), version in Foundry and record quality metadata.

Conceptual snippet (pseudo-Foundry command):

# Criar task template (conceitual)
foundry annotation create-template --name "inspecao_anomalias" \
  --type image --labels "anomaly,normal" \
  --output-format coco --instructions "Desenhe bounding boxes nas anomalias"

# Iniciar job de anotação
foundry annotation start-job --template "inspecao_anomalias" --dataset /OneLake/ProjectX/images/ --assign group:annotators

Common mistakes

1) Vague instructions: annotators interpret labels differently. Always provide clear examples and decision rules. 2) Lack of review/quality control: not implementing reviews leads to noisy data that harms training. 3) Forgetting metadata and versioning: without traceability it is hard to reproduce or fix dataset issues.

How to practice

Practice by creating a small project in Foundry (or a sandbox environment if available): upload 50 images, define 2–3 labels and run an annotation job with at least two annotators. Measure agreement and adjust instructions until you reach good consistency. For exam preparation, use the OFFICIAL and free Microsoft Practice Assessment and the Microsoft study guide (both free) — they guide the measured skills and help validate knowledge without exposing real questions.

In summary

• A human annotation pipeline ensures high-quality labeled data, essential for training reliable models.
• In Foundry, you configure a task template, annotation rules, review levels and export/versioning of results.
• Good instructions, quality metrics (e.g.: IoU, agreement) and audit trails are critical to reduce data noise.
• Practice with small sets, monitor metrics and use Microsoft’s official free resources to complement study.