Cosmos DB Change Feed to Downstream Sync

Overview

This flow turns Azure Cosmos DB changes into reliable downstream updates. On a 5-minute schedule it polls the Cosmos change feed using a stored _ts watermark, projects each new or updated document into a Microsoft Dataverse mirror table with an idempotent upsert (keyed on the Cosmos document id), advances the watermark for near exactly-once processing, and posts a run summary to a Microsoft Teams channel. Apps write to Cosmos but analytics and operations often live elsewhere; consuming the change feed keeps downstream systems current without dual-writes or brittle export jobs. The solution ships Off (demo) — going live requires only authorizing the three connections and setting the environment variable values.

Use Case

A Cosmos-backed application needs its document changes mirrored into Dataverse for reporting and automation, while operations wants a heartbeat in Teams each cycle showing how many documents moved. The continuation watermark guarantees each run only reads what changed since the last successful run, so the flow is cheap and resumable. Two Dataverse tables back the flow: a continuation-token state table (one row per feed) and a downstream projection table mirroring each Cosmos document.

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Per-type routing

Branch on a docType field to project different document types into different Dataverse tables or downstream systems.

Deletes

The change feed surfaces upserts only; adopt a soft-delete flag in source documents and mirror it as a status, since hard deletes are not emitted.

Partition key

The demo stores coalesce(partitionKey, id); set this to your container's actual partition-key path for accurate projection.

Throughput

For high-volume containers, shorten the recurrence or hand off to an Azure Function change-feed processor and have this flow orchestrate.

Payload size

flowlibs_payload is a Memo column storing the full document JSON; project only the fields you need if documents are large.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

concat('SELECT * FROM c WHERE c._ts > ', variables('varLastTimestamp'), ' ORDER BY c._ts ASC')

coalesce(first(outputs('Get_Continuation_State')?['body/value'])?['flowlibs_lastts'], '0')

if(greater(length(outputs('Read_Change_Feed')?['body/value']),0), string(last(outputs('Read_Change_Feed')?['body/value'])?['_ts']), variables('varLastTimestamp'))

flowlibs_documentid eq '{document id}'

string(items('Apply_to_each_Changed_Document'))