How to do object detection in video with Azure AI Vision: step by step

This tutorial shows how to do object detection in video using Azure AI Vision to identify and annotate objects in video frames. It is useful for surveillance, traffic analysis or quality control, when you need to automate object identification in streams or video files.

Prerequisites

• Azure account with a subscription and permission to create resources.
• An Azure AI Vision instance (resource key and endpoint) or Azure Cognitive Services with Vision enabled.
• Python 3.8+ installed and pip.
• Libraries: azure-ai-vision, opencv-python, numpy.
• Video file (MP4) or camera available for testing.

Step 1: Create the Azure AI Vision resource

In the Azure portal create an Azure AI Vision resource (or Cognitive Services if you prefer). Save the endpoint and the key (resource key) — they will be needed to authenticate inference calls.

Step 2: Install dependencies and configure environment

Install the required libraries in a virtual environment. Here we use the azure-ai-vision SDK, OpenCV and numpy to read video and draw boxes.

python -m venv venv
source venv/bin/activate   # ou venv\Scripts\activate no Windows
pip install azure-ai-vision opencv-python numpy

Step 3: Basic code to read video and send frames to Azure AI Vision

We split the video into frames (for example, 1 frame per second) and call the object detection API. This example assumes you use the Vision service endpoint with the detectObjects operation.

import cv2
import numpy as np
from azure.ai.vision import VisionClient, VisionServiceOptions
from azure.core.credentials import AzureKeyCredential

# Configurar credenciais
endpoint = "https://SEU_ENDPOINT"
key = "SUA_CHAVE"
client = VisionClient(endpoint=endpoint, credential=AzureKeyCredential(key))

# Abrir vídeo
cap = cv2.VideoCapture('exemplo.mp4')
fps = cap.get(cv2.CAP_PROP_FPS)
frame_interval = int(fps)  # 1 frame por segundo
frame_idx = 0

while True:
    ret, frame = cap.read()
    if not ret:
        break
    if frame_idx % frame_interval == 0:
        # Converter frame para JPEG em memória
        ret2, buf = cv2.imencode('.jpg', frame)
        jpg_bytes = buf.tobytes()

        # Chamada de deteção de objetos (exemplo conceptual)
        response = client.detect_objects(image=jpg_bytes)
        # A resposta terá uma lista de boxes: cada box tem left, top, width, height e tagName/confidence
        for obj in response.objects:
            x = int(obj.bounding_box.x)
            y = int(obj.bounding_box.y)
            w = int(obj.bounding_box.w)
            h = int(obj.bounding_box.h)
            label = f"{obj.tag_name}: {obj.confidence:.2f}"
            cv2.rectangle(frame, (x, y), (x+w, y+h), (0,255,0), 2)
            cv2.putText(frame, label, (x, y-10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0,255,0), 1)

        # Mostrar frame anotado
        cv2.imshow('Deteção', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    frame_idx += 1

cap.release()
cv2.destroyAllWindows()

Step 4: Adjust frequency, resolution and class filtering

To optimize cost and latency, send fewer frames, resize to a lower resolution and filter by classes of interest (for example, "car" or "person"). You can also use a confidence threshold to reduce false positives.

# Exemplo de filtragem e resize antes da chamada
target_w = 640
h, w = frame.shape[:2]
scale = target_w / w
frame_small = cv2.resize(frame, (target_w, int(h*scale)))
# converter e enviar frame_small em vez do original

# Ao processar resposta:
min_conf = 0.6
for obj in response.objects:
    if obj.confidence < min_conf:
        continue
    if obj.tag_name not in ('person','car'):
        continue
    # desenhar box como antes

Step 5: Save results and generate reports

Save annotations to CSV or JSON with the frame time, class and confidence for later analysis or integration with alerts. This enables creating dashboards in Power BI or ETL pipelines.

import csv
with open('detecoes.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    writer.writerow(['frame_time_s','class','confidence','x','y','w','h'])
    # para cada deteção: writer.writerow([time_sec, obj.tag_name, obj.confidence, x, y, w, h])

Verify the result

Run the script and verify that boxes appear on the expected objects, that the false positive rate is acceptable and that the CSV file has coherent records. Test with validation videos and adjust min_conf, resolution and frame intervals to tune performance and cost.

Conclusion

With Azure AI Vision and a simple Python client it is possible to create an object detection pipeline in video for surveillance, traffic analysis or quality control. Next steps: integrate real-time streaming (Azure Media Services), use Custom Vision models for specific classes, or expose results via Azure Functions. Tip: record metrics (latency, detection rate) early to optimize cost and accuracy.