How to build a text semantic annotation service in Azure AI & Machine Learning

This guide shows how to build a simple text semantic annotation service in Azure AI & Machine Learning to identify entities, categories and relationships in a collection of documents — useful for search, knowledge extraction and preparing data for models.

Prerequisites

• Azure account with permissions to create resources (Cognitive Services, Azure Machine Learning).
• Azure CLI and Azure ML SDK for Python installed.
• Python 3.8+ and pip; text editor (VS Code suggested).
• Set of documents in .txt or .csv format for annotation.
• Basic knowledge of JSON and HTTP (to test the API).

Step 1: Concept and data preparation

Explain what semantic annotation is: tagging tokens with labels (entities) and defining relationships (for example, Person–Organization). Prepare a CSV file with columns: id, text. If you want relationships, create a JSON file with examples of spans and relations.

# Exemplo mínimo CSV (documents.csv)
id,text
1,"João trabalha na Contoso desde 2020."
2,"A Maria visitou o Porto em Junho."

Step 2: Create Azure resources

Create an Azure Cognitive Services or Azure OpenAI resource if needed, and an Azure Machine Learning workspace. The Cognitive Services resource gives you language APIs; Azure ML helps manage pipelines and custom models.

# Exemplo Azure CLI (resumo)
az login
az group create -n rg-annotation -l westeurope
az cognitiveservices account create -n cs-annotation -g rg-annotation -l westeurope --kind TextAnalytics --sku S0 --yes
az ml workspace create -w aml-annotation -g rg-annotation

Step 3: Explore Language APIs for initial annotation

Use Azure AI Language (Text Analytics) for automatic entity extraction as a starting point. This speeds up human annotations: correct and add relationships.

from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://.cognitiveservices.azure.com/"
key = ""
client = TextAnalyticsClient(endpoint=endpoint, credential=AzureKeyCredential(key))

docs = ["João trabalha na Contoso desde 2020."]
resp = client.recognize_entities(docs)
for doc in resp:
    for ent in doc.entities:
        print(ent.text, ent.category, ent.offset, ent.length)

Step 4: Build an annotation schema and a simple tool

Define labels (e.g.: Person, Organization, Location, Date) and relationship types (e.g.: works_at). To annotate as a team, create a minimal Flask script to display text and save annotations as JSON. This standardizes the format for training.

# app.py (exemplo mínimo Flask)
from flask import Flask, request, jsonify, render_template_string
import json

app = Flask(__name__)

TEMPLATE = """


{{text}}



Guardar


"""

@app.route('/', methods=['GET','POST'])
def index():
    if request.method == 'POST':
        text = request.form['text']
        anns = request.form['annotations']
        with open('annotations.json','a') as f:
            f.write(json.dumps({'text':text,'annotations':json.loads(anns)})+'\n')
        return 'Saved'
    return render_template_string(TEMPLATE, text='')

if __name__ == '__main__':
    app.run(debug=True)

Step 5: Train a simple model in Azure Machine Learning

Convert the annotations into a training format (conll/JSONL). Create an experiment in Azure ML that runs a local training script using a library like spaCy or Hugging Face Transformers for NER and relations.

# train.py (esqueleto)
import json
# carregar annotations.json, converter para formato spaCy/HF
# treinar modelo NER e salvar em outputs/model

if __name__=='__main__':
    # código de treino mínimo
    pass

Step 6: Deploy the model as an inference service

After training, register the model in Azure ML and create an Endpoint for inference. This allows sending text via HTTP and receiving predicted entities and relationships.

# Exemplo de publicação conceptual (Azure ML SDK)
from azureml.core import Workspace, Model
ws = Workspace.from_config()
model = Model.register(ws, model_path='outputs/model', model_name='ner-annotator')
# cria image e deploy para ACI ou AKS (resumido)

Step 7: Post-processing pipeline and integration

Implement a wrapper that combines model predictions with simple rules (dictionaries, regex) to improve accuracy and extract relationships. Save results in JSONL or in a table in Azure SQL/Storage for consumption.

# pseudo-código de pós-processamento
preds = model.predict(text)
# aplicar regras: normalizar datas, mapear abreviaturas
# ligar entidades por proximidade ou heurísticas

Verify the result

Test the API by sending example texts and confirm whether the returned entities and relationships make sense. Compare with human annotations on a validation set and compute NER metrics (Precision, Recall, F1) or perform a manual review.

# ex. testar endpoint (curl)
curl -X POST https:///score -H "Content-Type: application/json" -d '{"text":"João trabalha na Contoso."}'

# Espera JSON com spans: [{"start":0,"end":4,"label":"Pessoa"}, ...]

Conclusion

You have completed an end-to-end flow: data preparation, human annotation assisted by Azure AI Language, training in Azure ML and deployment of a semantic annotation service. Next steps: add active learning to collect automatic corrections, improve relationships with relation models, or integrate with Power BI to explore results. Tip: start with a few well-defined labels to reduce ambiguity in annotations — which label would you prioritize first?