Cosmos DB TTL and Archival Automation

Overview

This flow automates Azure Cosmos DB data lifecycle (TTL + archival). On a daily schedule it finds documents whose business timestamp is older than a configurable retention window, writes a full JSON copy of each document to Azure Blob Storage, deletes the document from Cosmos, and posts a run summary to Microsoft Teams. Old documents bloat RU/storage cost and slow queries; archiving cold data to cheap blob storage before pruning keeps the operational container lean and fast while preserving a complete history for compliance and restore. Ships Off (demo) — going live requires only authorizing the three connections and setting the environment variable values.

Use Case

A team running a globally distributed or high-write Cosmos container wants cold documents moved to inexpensive storage on a retention schedule, with an auditable archive and a delete step that respects the container's partition key — fully unattended and reported to a Teams channel each run. Age is measured against a configurable business timestamp field (e.g. createdDate), not the system _ts, so retention reflects business meaning.

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

TTL combo

Set the Cosmos container's native TTL instead of (or alongside) the explicit delete; the flow still produces the blob archive and TTL prunes automatically.

Restore / rehydrate

Build a companion flow that reads a {id}.json from the archive folder and re-creates the document with Cosmos CreateDocument_V3.

Partitioned archive

Extend flowlibs_ArchiveBlobFolder or the blob name to write a folder per partition key or per date for easier lookup.

Retention granularity

Point flowlibs_RetentionTimestampField at any ISO-8601 field; change flowlibs_CosmosRetentionDays to tune the window without editing logic.

Throughput

For very large sweeps, raise the loop concurrency (watch Cosmos RU/429s) or page the query with continuationToken.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

addDays(utcNow(), mul(variables('varRetentionDays'), -1))

concat('SELECT * FROM c WHERE c.', variables('varRetentionField'), ' < "', outputs('Compose_Retention_Cutoff'), '"')

concat('["', string(items('Apply_to_each_Expired_Document')?[variables('varPartitionKeyField')]), '"]')

concat(items('Apply_to_each_Expired_Document')?['id'], '.json')

length(outputs('Query_Expired_Documents')?['body/value'])