Data Factory Self-Service Backfill Requests
Via a Teams/form request, the flow lets approved users request a historical backfill for a date range; it validates the range, runs the Azure Data Factory pipeline parameterized per day/window with throttling, tracks progress, and reports completion. Provides governed self-service backfills.
Provided as-is, without warranty of any kind. Review and test each pattern in a non-production environment before deploying it to live automations. See our Terms.
Overview
Governed self-service historical backfills for Azure Data Factory. An approved user submits a backfill request (start date, end date, reason) through a manual/Teams-style request. The flow validates the range against a guardrail, routes a Basic approval, then runs the ADF pipeline once per day across the requested window with a throttle pause between runs, tracks per-day status, and reports completion to Teams.
Why it matters: backfills are common, risky, and engineering-heavy. A controlled self-service path - guardrail + approval + throttling - removes the engineering ticket while protecting production loads from being starved by a runaway reprocess.
Ships Off (demo).
Use Case
Analysts and operations need historical reprocessing (a late-arriving source, a corrected transform, a missed window) without filing an engineering ticket. They request a date range; an owner approves; ADF reprocesses each day in the range with throttling so it never overwhelms the integration runtime or source.
Flow Architecture
Manually Trigger a Flow
Request / ButtonCaptures the backfill startDate, endDate, and reason from the requester.
Initialize Trace, Config & Guardrails
Initialize variableMints a correlation id, binds the seven env vars, and sets the max-days guardrail (31), throttle seconds (15), result tallies, day count, and loop cursor.
Validate Range
ConditionRequires 1 <= dayCount <= max; notifies and terminates empty, reversed, or oversized ranges before approval.
Request Backfill Approval
Approvals - StartAndWaitForAnApprovalGovernance gate to the configured approver; only an explicit Approve proceeds.
Run Backfill Per Day
Do UntilIterates day-by-day to the end date inclusive: starts the pipeline for the daily window, throttle-delays, reads run status, and tallies succeeded/failed.
Report Completion
Microsoft Teams - PostMessageToConversationPosts a summary with succeeded/failed counts and the correlation id, or a rejection notice with the requested range and reason.
Environment Variables
| Schema name | Type | Default | Description |
|---|---|---|---|
| flowlibs_AdfSubscriptionId | String | <your-subscription-id> | Azure subscription hosting the Data Factory. |
| flowlibs_AdfResourceGroup | String | rg-data-platform | Resource group of the Data Factory. |
| flowlibs_AdfFactoryName | String | adf-analytics | Target Azure Data Factory name. |
| flowlibs_BackfillPipelineName | String | pl_DailyBackfill | Pipeline executed once per day in the range. |
| flowlibs_ApproverEmail | String | you@yourcompany.com | Approver for the backfill gate. |
| flowlibs_TeamsGroupId | String | <your-team-id> | Teams team (group) id for notifications. |
| flowlibs_TeamsChannelId | String | <your-channel-id> | Teams channel id for notifications. |
Connectors & Connections
| Connector | API name | Actions used |
|---|---|---|
| Azure Data Factory | shared_azuredatafactory | CreatePipelineRun GetPipelineRun |
| Approvals | shared_approvals | StartAndWaitForAnApproval |
| Microsoft Teams | shared_teams | PostMessageToConversation |
Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.
Customization Guide
Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.
- Window size
- Change windowStart/windowEnd to a multi-day window per run (e.g. weekly) and adjust the cursor increment.
- Resume
- Persist the loop cursor to a store and seed it on restart to continue from the last completed window.
- Priority
- Point the backfill pipeline at a low-priority integration runtime so backfills never contend with production loads.
- Guardrail/throttle
- Tune the max backfill days and throttle seconds variables.
- Progress notify
- Add a percentage-progress Teams update inside the loop using succeeded+failed over the day count.
Key Expressions
The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.
EXPR.01Inclusive day count
Days in the requested range, inclusive.
EXPR.02Range valid
Guardrail on the requested range size.
EXPR.03Advance cursor one day
Moves the loop cursor forward one normalized day.
EXPR.04Approved?
Only an explicit Approve proceeds to the backfill loop.
Comments
Sign in to join the conversation.
Sign inNo comments yet. Be the first to share your experience with this flow.