Data Factory Lineage and Run Audit Log

Overview

This flow gives Azure Data Factory pipeline runs a governed audit trail. At the end of each run, Data Factory calls the flow (an HTTP request trigger) with the run id. The flow reads the run's metadata from ADF, reads the activity-level lineage (source, target, rows read/copied) from the ADF REST API, records a complete lineage and audit entry in a Dataverse table, classifies the outcome, and emails the data team when a run fails or shows an abnormal row-count deviation. A correlation id is minted per run for cross-system tracing. When data is wrong, you need to know which run produced it, from what source, to what target, and how many rows moved — a governed run/lineage log makes audits and impact analysis fast and surfaces silent data loss as it happens. Ships Off (a FlowLibs reference flow); going live requires only connection authorization and environment-variable configuration.

Use Case

A data team needs an auditable, searchable record of every Data Factory pipeline run and its lineage — source, target, row counts, parameters, duration, outcome, and the identity/event that triggered it — for compliance and impact analysis.

The flow is ideal for teams that:

• Event-driven: logs each pipeline run the moment it finishes
• Captures full lineage — source, target, rows read/copied, duration, invoked-by
• Governed Dataverse audit table for compliance and impact analysis
• Classifies and emails on failures and abnormal row-count deviations

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Wire the trigger

Add a Web activity at the end of each ADF pipeline that POSTs { runId, pipelineName } to the flow URL. Turn the flow On to get the URL.

Multi-activity lineage

The flow parses the first activity's row counts. For pipelines with several copy activities, swap the Set-Variable expressions for an Apply-to-each over body('Query_Activity_Runs')?['value'] and aggregate.

Impact graph

Add columns linking source -> target chains, or push lineage to Microsoft Purview.

Anomaly logic

flowlibs_RowAnomalyPct controls the read-vs-copied tolerance; extend Compose Outcome to compare against a historical average from prior rows in the table.

Retention

Add a scheduled companion flow to prune flowlibs_adflineage rows per your audit-retention policy.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

@if(not(equals(body('Get_Pipeline_Run_Details')?['status'],'Succeeded')), 'Failed', if(and(greater(variables('varRowsCopied'),0), greater(abs(sub(variables('varRowsRead'),variables('varRowsCopied'))), mul(variables('varRowsCopied'), div(float(variables('varAnomalyPct')),100)))), 'Anomaly', 'Succeeded'))

@int(coalesce(first(body('Query_Activity_Runs')?['value'])?['output']?['rowsRead'], 0))

@string(coalesce(first(body('Query_Activity_Runs')?['value'])?['input']?['source']?['type'], 'Unknown'))

https://management.azure.com/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.DataFactory/factories/{df}/pipelineruns/{runId}/queryActivityruns?api-version=2018-06-01