Data Factory Self-Service Backfill Requests

Overview

Governed self-service historical backfills for Azure Data Factory. An approved user submits a backfill request (start date, end date, reason) through a manual/Teams-style request. The flow validates the range against a guardrail, routes a Basic approval, then runs the ADF pipeline once per day across the requested window with a throttle pause between runs, tracks per-day status, and reports completion to Teams.

Why it matters: backfills are common, risky, and engineering-heavy. A controlled self-service path - guardrail + approval + throttling - removes the engineering ticket while protecting production loads from being starved by a runaway reprocess.

Ships Off (demo).

Use Case

Analysts and operations need historical reprocessing (a late-arriving source, a corrected transform, a missed window) without filing an engineering ticket. They request a date range; an owner approves; ADF reprocesses each day in the range with throttling so it never overwhelms the integration runtime or source.

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Window size

Change windowStart/windowEnd to a multi-day window per run (e.g. weekly) and adjust the cursor increment.

Resume

Persist the loop cursor to a store and seed it on restart to continue from the last completed window.

Priority

Point the backfill pipeline at a low-priority integration runtime so backfills never contend with production loads.

Guardrail/throttle

Tune the max backfill days and throttle seconds variables.

Progress notify

Add a percentage-progress Teams update inside the loop using succeeded+failed over the day count.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

@add(div(sub(ticks(triggerBody()?['endDate']),ticks(triggerBody()?['startDate'])),864000000000),1)

@and(greaterOrEquals(variables('varDayCount'),1),lessOrEquals(variables('varDayCount'),variables('varMaxBackfillDays')))

@formatDateTime(addDays(variables('varCurrentDate'),1),'yyyy-MM-dd')

@equals(outputs('Request_Backfill_Approval')?['body/outcome'],'Approve')