DP-700: how to implement auditing and logging in analytics solutions on Fabric
I will teach the skill of implementing auditing and logging in an analytics solution on Microsoft Fabric — a key area of the DP-700 that matters both for compliance and for operational diagnosis. Knowing how to configure logs and audits allows you to respond to incidents, prove compliance and optimize ingestion and query processes.
What you need to know
Auditing and logging consist of recording relevant events (accesses, schema changes, pipeline failures, administrative operations) and making them available for analysis. In the context of Fabric, this involves several levels: service diagnostics (for example, diagnostic settings in Fabric or Azure), workspace activity logs, pipeline telemetry (Dataflows, Spark, Synapse), and OneLake events. Typical destinations are Log Analytics, Event Hubs and storage accounts for long-term retention.
Practical example: when an ingestion pipeline fails, we want a record that includes the pipeline id, timestamp, error message and input data. Those records should be sent to a Log Analytics workspace for alerts and investigation with Kusto Query Language (KQL).
How it works (step-by-step)
Practical steps to implement a basic auditing/logging solution in Fabric:
- Inventory the logging surfaces — identify the components that generate events: Workspaces, Dataflow/Synapse pipelines, Spark jobs, OneLake operations and control plane (administrative operations).
- Enable Diagnostic Settings — for each Fabric resource that supports diagnostic settings, configure the sending of logs to appropriate destinations (Log Analytics, Event Hub, Storage). This allows centralizing events and retaining data for audit.
- Define which categories to record — choose categories (AuditLogs, OperationalLogs, PipelineRuns, SparkExecution). Only record what is necessary to reduce costs and noise.
- Structure the destination — generally: use Log Analytics for ad‑hoc queries/alerts; Event Hub for streaming to SIEM; Storage account for immutable retention (e.g., legal retention policies).
- Normalize and enrich events — add useful fields (workspaceId, userPrincipal, pipelineId, dataSource, region). If you use Azure Monitor or custom functions, transform events before storing.
- Create KQL queries and alerts — in Log Analytics write queries to detect repeated failures, unauthorized access or latency spikes and create alerts that trigger notifications or playbooks.
- Automate retention and export — define retention policies in Storage/Log Analytics and export periodically to an immutable file if required for compliance.
Example of Diagnostic Setting configuration (concept):
1. No portal Azure / Fabric resource -> Diagnostic settings
2. Add diagnostic setting -> select categories: AuditLogs, PipelineRuns
3. Choose destinations: Log Analytics workspace (MyLA), Storage account (audit-archive)
4. Save
Example KQL query to identify pipeline failures in the last 24h:
FabricPipelineLogs
| where TimeGenerated >= ago(24h)
| where Status == "Failed"
| summarize Count = count(), FirstSeen = min(TimeGenerated), LastSeen = max(TimeGenerated) by PipelineName, ErrorCode
| order by Count desc
Common mistakes
- Enabling all log types without filtering: leads to very high costs and lots of noise. Select relevant categories and adjust retention.
- Not enriching events with context: logs with little information (only error messages) make investigation slow. Include pipeline IDs, workspace and userPrincipal.
- Storing logs only locally: if you only have files in the same subscription without exporting to a long-term destination, you risk loss in case of incidents. Use storage with retention policies and, when relevant, export to a SIEM tenant.
How to practice
Practice these tasks in a lab environment: create a Fabric workspace, implement a simple pipeline that intentionally generates errors and configure diagnostic settings to a Log Analytics workspace. Then write KQL queries to analyze the events and create an alert that triggers a Logic App or action group.
For exam preparation, use the OFFICIAL and free Microsoft Practice Assessment for DP-700 (the official Practice Assessment helps evaluate skills; it is free). Also consult the official Microsoft study guide for DP-700, which explains the measured areas and provides study resources (also free).
In summary
- Auditing and logging in Fabric involve capturing events from multiple components and sending them to destinations like Log Analytics, Event Hubs and Storage.
- Configure diagnostic settings selectively, enrich events and structure retention to reduce costs and meet legal requirements.
- Create KQL queries and alerts to detect incidents and automate responses.
- Avoid recording everything without filtering and ensure proper export/retention for investigation and compliance.