Queue-Based Bulk Data Import
Break large CSV imports into individual messages on an Azure Queue, process each row into Dataverse to handle volume without timeout.
Provided as-is, without warranty of any kind. Review and test each pattern in a non-production environment before deploying it to live automations. See our Terms.
Overview
Large CSV imports often exceed cloud-flow timeout budgets when written one row at a time directly into Dataverse. This flow solves that by fanning the import across two flows joined by an Azure Storage Queue: a producer flow reads the CSV from SharePoint, splits it into per-row JSON messages, and enqueues each one; a consumer flow drains the queue on its own schedule and writes each row into the target Dataverse table.
This page documents the producer side of the pattern. It is the flow that ships in the FlowLibs solution FlowLibsQueueBasedBulkDataImport.
Use Case
This pattern fits scenarios where a single, monolithic import flow would time out or be hard to retry — the Azure Queue gives you a durable buffer between the CSV source and the Dataverse sink, plus per-row retry granularity.
The flow is ideal for teams that:
- A nightly system export drops a .csv in a SharePoint document library and you need each row landed in a Dataverse table.
- A user-triggered bulk upload of 5,000+ records would otherwise time out a single flow.
- You need backpressure / throttling control between the source (SharePoint file drop) and the sink (Dataverse table) — Azure Queues give you a durable buffer between the two.
- You want to retry individual failed rows without re-processing the whole file.
Flow Architecture
Manual (Power Apps button trigger)
Manually trigger a flowInput schema: filePath (string, required) — server-relative path of the CSV file in the SharePoint library, e.g. /Shared Documents/BulkImports/contacts.csv. Replace this trigger with OnNewFile against the import library to make the flow fully event-driven.
Initialize varSiteUrl
Initialize variableString variable seeded from the flowlibs_SharePointSiteURL environment variable.
Initialize varLibraryPath
Initialize variableString variable seeded from the flowlibs_BulkImportLibrary environment variable.
Initialize varStorageAccount
Initialize variableString variable seeded from the flowlibs_AzureQueueStorageAccount environment variable.
Initialize varQueueName
Initialize variableString variable seeded from the flowlibs_BulkImportQueueName environment variable.
Initialize varTargetTable
Initialize variableString variable seeded from the flowlibs_BulkImportTargetTable environment variable.
Initialize varNotifyEmail
Initialize variableString variable seeded from the flowlibs_BulkImportNotifyEmail environment variable.
Initialize varRowsQueued
Initialize variableInteger accumulator seeded to 0; incremented after each successful enqueue.
Initialize varRowsFailed
Initialize variableInteger accumulator seeded to 0; incremented when an enqueue fails, times out, or is skipped.
Environment Variables
| Schema name | Type | Default | Description |
|---|---|---|---|
| flowlibs_SharePointSiteURL | String | https://your-tenant.sharepoint.com | Root SharePoint site that hosts the CSV drop library. Reused across FlowLibs solutions. |
| flowlibs_BulkImportLibrary | String | /Shared Documents/BulkImports | Server-relative path of the document library where new CSVs land. |
| flowlibs_AzureQueueStorageAccount | String | <configure> | Azure Storage account name hosting the queue. Reused across FlowLibs queue solutions. Set to the lowercase storage account name (no URL). |
| flowlibs_BulkImportQueueName | String | bulk-import-rows | Queue that receives one message per CSV data row. |
| flowlibs_BulkImportTargetTable | String | flowlibs_contacts | Dataverse table logical name (entity set name) the consumer flow writes to. Embedded in every queue message. |
| flowlibs_BulkImportNotifyEmail | String | you@yourcompany.com | Recipient of the post-run summary email. |
Connectors & Connections
| Connector | API name | Actions used |
|---|---|---|
| SharePoint | shared_sharepointonline | GetFileContentByPath (reads the CSV file) |
| Azure Queues | shared_azurequeues | PutMessage_V2 (enqueues the per-row JSON message) |
| Office 365 Outlook | shared_office365 | SendEmailV2 (delivers the summary email) |
Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.
Customization Guide
Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.
- Switching to an event-driven SharePoint trigger
- Replace the manual trigger with OnNewFile (operationId OnNewFile on shared_sharepointonline) bound to the varLibraryPath folder, then change Get_CSV_File_Content to GetFileContent and bind its id parameter to triggerOutputs()?['body/Id']. Filter to .csv extensions with an If guard wrapping the parse pipeline using @endsWith(toLower(triggerOutputs()?['headers']?['x-ms-file-name']), '.csv').
- Tuning row throughput
- Default concurrency on Apply_To_Each_Row is 1 so the row counters stay accurate. If you don't need exact counts you can raise it to 20-50 for faster enqueueing, but Azure Queues throttles enqueue at the storage-account scope so very high concurrency is rarely worth the loss of counter accuracy.
- Schema discipline of the queued message
- Every message carries sourceFile, targetTable, headerRow, rowText, and enqueuedAt. The consumer flow can therefore (1) map fields by header position without re-reading SharePoint, (2) write to targetTable (lets one consumer service many imports), and (3) skip stale messages older than X hours via enqueuedAt.
- Pairing with the consumer flow
- Build a separate consumer flow with a Recurrence trigger (every 5 minutes), GetMessages_V2 (numofmessages: 32, visibilitytimeout: 60), Foreach over body('Get_Messages_From_Queue')?['QueueMessagesList']?['QueueMessage'] wrapped in coalesce(..., createArray()), parse the JSON message text, split rowText and headerRow on commas, build a Dataverse CreateRecord payload, and DeleteMessage_V2 only on success. Use the same flowlibs_AzureQueueStorageAccount and flowlibs_BulkImportQueueName env vars so the producer and consumer stay paired.
- Multi-table fan-out
- Promote flowlibs_BulkImportTargetTable to a column inside the CSV (or compute it from a folder convention like /BulkImports/Contacts/*.csv -> flowlibs_contacts), and stamp that value into the targetTable field of Compose_QueueMessage. The same producer can then drive imports into multiple tables.
Key Expressions
The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.
EXPR.01Decode CSV binary to string
Decodes the binary CSV payload returned by SharePoint into a string.
EXPR.02Split CSV into rows (CRLF-safe)
Strips carriage returns and splits on newline — handles CRLF and LF line endings without regex.
EXPR.03Separate header from data rows
First() keeps the header for embedding in every queue message; skip(...,1) yields just the data rows for the loop.
EXPR.04Empty-row guard
Guards against trailing empty rows from the final newline.
EXPR.05Serialize queue message body
Serializes the per-row JSON object to a string for the queue body — PutMessage_V2 only accepts plain string in message.
EXPR.06Per-row error counter runAfter
Per-row error counter pattern so a single bad row doesn't kill the loop.
Comments
Sign in to join the conversation.
Sign inNo comments yet. Be the first to share your experience with this flow.