How to implement audio classification with Azure AI Speech: step by step

This tutorial shows how to create a simple audio classification pipeline with Azure AI Speech to differentiate commands or sounds (for example: ‘clap’, ‘applause’, ‘speech’). Having an audio classifier is useful for automation, accessibility and monitoring in real-world applications.

Prerequisites

• Azure account with an active subscription and Azure AI Speech configured (key and endpoint).
• Python 3.8+ installed and pip.
• WAV audio files organized in a folder structure (one folder per class).
• Libraries: azure-cognitiveservices-speech, librosa, scikit-learn, soundfile.
• Text editor/IDE and basic command line skills.

Step 1: Why and choice of approach

Audio classification can be done with pre-trained models or by creating a simple classifier based on features (MFCCs) followed by a classical classifier (e.g.: RandomForest). For an introductory tutorial we use MFCC extraction with librosa and an sklearn classifier — lighter and more explainable before migrating to deep learning or Custom Speech.

Step 2: Prepare environment and install dependencies

Install the required libraries. This includes the Azure Speech SDK if you intend to integrate recording or transcription, but the core of the classification uses librosa and scikit-learn.

python -m pip install azure-cognitiveservices-speech librosa scikit-learn soundfile numpy

Step 3: Structure the audio data

Organize the WAV files like this: dataset/clap/*.wav, dataset/applause/*.wav, dataset/speech/*.wav. Make sure the files have the same sample rate (e.g.: 16 kHz) or they will be converted when loading.

Step 4: Extract features (MFCC) from audio

We extract MFCCs because they summarize timbral characteristics well. We will load each file, compute time-averaged MFCCs and build X (features) and y (labels).

import os
import numpy as np
import librosa

def extract_features(file_path, n_mfcc=13):
    y, sr = librosa.load(file_path, sr=16000)
    mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=n_mfcc)
    return np.mean(mfcc, axis=1)

def load_dataset(root):
    X, y = [], []
    classes = sorted(os.listdir(root))
    for label in classes:
        folder = os.path.join(root, label)
        if not os.path.isdir(folder):
            continue
        for f in os.listdir(folder):
            if f.lower().endswith('.wav'):
                path = os.path.join(folder, f)
                features = extract_features(path)
                X.append(features)
                y.append(label)
    return np.array(X), np.array(y)

# Example usage
# X, y = load_dataset('dataset')

Step 5: Train and evaluate a classifier

We use a RandomForest to start. We do a train/test split and measure accuracy and the confusion matrix to see common errors (confusion between similar classes).

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix

X, y = load_dataset('dataset')
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)

pred = clf.predict(X_test)
print('Accuracy:', accuracy_score(y_test, pred))
print('Confusion matrix:\n', confusion_matrix(y_test, pred))

Step 6: Integrate with Azure AI Speech (optional)

If you want to capture audio in real time or use the recording service, use azure-cognitiveservices-speech to record and then apply the extraction+classification pipeline to the captured buffer.

import azure.cognitiveservices.speech as speechsdk

speech_key = 'SUA_SPEECH_KEY'
service_region = 'SUA_REGION'

speech_config = speechsdk.SpeechConfig(subscription=speech_key, region=service_region)
audio_config = speechsdk.audio.AudioConfig(use_default_microphone=True)
rec = speechsdk.AudioConfig(use_default_microphone=True)

# Simple example: record 3s and save as local WAV (can be used for classification)
recorder = speechsdk.audio.AudioRecorder(audio_config)
# Notes: the SDK has classes for recording; here we refer to general integration.

Check the result

Confirm that the accuracy is acceptable (e.g.: >70% on a simple dataset). Check the confusion matrix for highly confused classes. Test with new unseen WAV files and see if the model predicts correctly. If using the microphone via Azure AI Speech, record short audio and pass it through the same extract_features -> clf.predict pipeline.

Conclusion

You have just built a simple audio classifier with MFCC extraction and RandomForest, and you know how to integrate recording via Azure AI Speech. Next steps: use data augmentation, try deep learning models (CNN/LSTM) or experiment with Custom Speech for pre-trained models. Tip: if classes are similar, increase the number of MFCCs or try additional features (chroma, spectral contrast).