Transform Once, Deliver Many: transformation, apidelivery, and webhookdelivery

Partners still drop CSVs. Downstream still wants JSON records, paged REST, or a webhook that never skips a batch. I do not hide that work in one mega Function. I keep azure integration architecture as three connectors: transformation runs a named transformer from a shared registry; apidelivery pages the JSON array into a REST API as a Durable orchestration; webhookdelivery posts ordered batches and stops at the first failure.

That is the whole job in one sentence: transform once against config, then pick the egress that matches the contract — paged HTTP with a singleton instance id, or an ordered webhook with a ledger duplicate check.

Think of a campus print shop. The copier has four named presets: CSV to JSON, JSON remap, XSLT, or “send the file to a sidecar.” Clubs do not fork the copier. They pick a preset on a sheet of YAML. Then they choose a dock. The registrar wants numbered boxes of 500 records and one tracking number per file so a second drop-off does not start a second courier. The department notice board wants sheets posted in order — if sheet 3 is rejected, sheet 4 never goes up.

This is part 5 of the Azure integration series. The catalog lives in microservices integration patterns on Azure Functions. Earlier deep dives: facade, SFTP publish / sender / delivery, and PGP at the pipeline boundary. Names and behaviour come from the public repo github.com/fransiscuss/integration-azure (commit 2539242).

What azure integration architecture looks like after crypto

Inbound decrypt and outbound encrypt already sit at the edges. In the middle I still need a shape change, then a way out that is not SFTP. That is still azure integration architecture: connector code stays generic, and config owns transformer name, output path, page size, auth mode, and success codes. You do not fork C# because partner B wants XSLT and partner C wants a Teams-shaped webhook. The transformation microservice is one Function artifact; webhook delivery azure is another; both are the same enterprise integration patterns as facade and SFTP — shared packages, per-partner YAML.

Connector	Trigger	Job	Terminal?
transformation	Service Bus queue	Named transform → blob → next queue	No — forwards envelope
apidelivery	HTTP POST /deliveries	Durable paged REST POSTs	Yes — ledger + orchestration status
webhookdelivery	Service Bus queue	Ordered JSON webhook batches	Yes — ledger only

Both egress connectors read Output when set, otherwise Source. That is why transform-once works: the next hop does not care which blob field you filled, only that the payload is a JSON array (or, for webhook, an array that will be split, or a single object posted as one part).

transformation: four names, one registry

The trigger is %Transformation:InputQueue% on the SERVICE_BUS_NAMESPACE binding, with AutoCompleteMessages = false. Body is a PipelineEnvelope v1. ConfigBlobUri points at instance YAML. After work, Output points at the transformed blob and the envelope is sent to output.queue with MessageId = envelope.EventId so duplicate detection on the next queue absorbs redelivery-after-send. Same peek-lock instinct as my Service Bus peek-lock guide: poison is dead-lettered with a reason; transient faults are abandoned.

The important order is in the processor, not the trigger glue. Resolve the transformer before any blob I/O so a typo in YAML is cheap:

// functions/transformation/Processing/TransformationProcessor.cs
var config = await configLoader.LoadAsync(envelope, ct);
var transformer = registry.Resolve(config.Transformer.Name); // before download

using var input = new MemoryStream();
foreach (var location in InputLocations(envelope, config))
    await payloadStore.DownloadIntoAsync(location, input, ct);

The registry is a name lookup over ITransformer implementations in shared/Integration.Transform:

// shared/Integration.Transform/TransformerRegistry.cs
public ITransformer Resolve(string name) => _byName.TryGetValue(name, out var transformer)
    ? transformer
    : throw new KeyNotFoundException(
        $"no transformer registered with name '{name}'. Known: {string.Join(", ", _byName.Keys)}");

KeyNotFoundException becomes dead-letter reason UnknownTransformer. Missing input blob (storage 404) is InputNotFound. Corrupt CSV/JSON/XSLT throws into TransformFailed. Unknown exceptions still propagate so the host retries them.

Name	Input	Settings that matter	Honest limit
CsvToJson	CSV with header row	none in the sample	Quote-aware commas; no embedded newlines inside quoted fields
JsonToJson	JSON array or one object	map.field = expression	Same expression language as facade CloudEvent mapping
Xslt	XML	xsltContent (compiled stylesheet text)	Missing setting is InvalidOperationException → TransformFailed
Http	opaque stream	endpoint plus optional header.*	Escape hatch. Missing endpoint is TransformFailed. Non-success HTTP is HttpRequestException → abandon, not DLQ

# functions/transformation/config/transformation.sample.yaml
transformer:
  name: CsvToJson
  settings: {}
inputs:
  - "${$.source.path}"
output:
  container: transformed
  pathTemplate: "${$.publisher}/${$.interface}/${$.tokens.ts ?? 'noversion'}.json"
  queue: transformation-out

Empty inputs means “just the envelope source.” Extra templates are resolved against envelope metadata (${$.publisher}, ${$.fileName}, named-group ${$.tokens.*}) and concatenated in order. Output account defaults to the source account. One deployed Function, many instance blobs — that is the Lego rule from the overview post.

apidelivery: azure functions durable, one file, one orchestration

apidelivery is the odd one in the catalog: HTTP start, not a queue trigger. POST /deliveries (function key) takes a PipelineEnvelope. The blob must be a JSON array of records. Status does not go to another queue. It goes to Cosmos and to Durable custom status at /runtime/webhooks/durabletask/instances/{instanceId}.

Idempotency is a deterministic instance id, not a lock sleep:

// functions/apidelivery/Orchestration/DeliveryLogic.cs
public static class InstanceIdFactory
{
    public static string Create(string publisher, string interfaceName, string fileName) =>
        $"apidelivery:{publisher}:{interfaceName}:{fileName}";
}

A repeated POST for the same file returns 202 with alreadyRunning: true and the existing runtime status. It does not start a second delivery. That is the pattern I want when an upstream retry and a human “send it again” land on the same file name.

The orchestration is boring on purpose. Mark DeliveryInProgress, count records, POST each page in order, then Delivered — or Failed with the offending page and status:

// functions/apidelivery/Functions/ApiDeliveryOrchestrator.cs
for (var page = 0; page < pages; page++)
{
    var result = await context.CallActivityAsync<PostPageResult>(
        "ApiDelivery_PostPage", new PostPageInput(envelope, page), taskOptions);

    if (StatusClassifier.IsSuccess(result.StatusCode, options.Value.SuccessStatusCodes))
        continue;

    await context.CallActivityAsync("ApiDelivery_UpdateStatus",
        new UpdateStatusInput(envelope, LedgerStatus.Failed,
            $"page {result.Page} returned {result.StatusCode}"), taskOptions);
    return;
}

The Durable retry policy is config, not magic numbers. Sample: 3 attempts, 5s initial, backoff coefficient 2, applied to every activity. That policy retries thrown activities (network, 5xx from the HTTP client pipeline). It does not retry a completed page POST that returned a non-success status. The orchestrator only checks StatusClassifier.IsSuccess. Anything else — including the sample terminal list 400/401/403/404/409/422 — marks Failed and returns. PostPageResult.Terminal exists on the activity result and is unused. The README line about “terminal codes skip retries, other non-success consume retries first” is not what the orchestration does today. I would rather quote the C# than the README.

Replay safety is the part people skip. Durable activities may run more than once. Page POSTs rely on the target API’s idempotency contract. Ledger updates use ETag optimistic concurrency. If the ERP treats “same page twice” as a duplicate insert, this connector cannot save you. I would rather document that than pretend Azure Functions Durable is a free exactly-once bus.

# functions/apidelivery/config/apidelivery.sample.yaml
ApiDelivery:
  Endpoint: "https://erp.example.com/api/v2"
  PagePath: /records
  PageSize: 500
  SuccessStatusCodes: [200, 201]
  TerminalStatusCodes: [400, 401, 403, 404, 409, 422]
  Retry:
    MaxAttempts: 3
    InitialInterval: "00:00:05"
    BackoffCoefficient: 2
  Auth:
    Mode: AadClientCredentials
    TokenScope: "api://erp-integration/.default"

webhookdelivery: order means stop

Webhook delivery is a queue worker, not Durable. Trigger is %INPUT_QUEUE%. It is a terminal connector. Happy path: ledger duplicate-check → download payload → post each part in order → mark Delivered → complete the message.

If the ledger row is already Delivered, it completes without posting. If FindAsync returns no row, the processor still posts and completes — it only calls UpdateAsync(Delivered) when a row existed. That is cheaper than Durable custom status when the contract is “a channel, not an ERP,” and it is also a real gap: missing ledger is not a hard fail here. apidelivery is stricter; its status activity throws if the row is missing.

A JSON array is split; anything else is one part. BatchSize chunks parts. Optional WrapProperty wraps each POST as { "<name>": "<part as text>" } — the sample uses text because that is how the original Teams-shaped config looked. Posting stops at the first failure so a later part never overtakes an earlier one.

Outcome	When	Settlement
Complete	All parts succeeded, or already Delivered	Complete
Abandon	429 / 5xx, network, Cosmos ETag conflict	Host redelivery up to maxDeliveryCount
Dead-letter	InvalidEnvelope, SourceReadFailed, EmptyPayload, non-success 4xx WebhookRejected	DLQ with reason

# functions/webhookdelivery/config/webhookdelivery.sample.yaml
INPUT_QUEUE: webhookdelivery-in
WebhookDelivery:
  Endpoint: "https://contoso.webhook.office.com/webhookb2/..."
  WrapProperty: text
  BatchSize: 1
  SuccessStatusCodes: [200, 202]
  Auth:
    Mode: None

Auth modes match apidelivery: None / ApiKey / Hmac / AadClientCredentials, with secrets as Key Vault references in deployed environments. HMAC is a commented variant in the sample (HmacHeaderName + secret setting). I do not invent extra modes.

Which dock I pick

If the downstream…	I use	Because
Wants record arrays, paging, and a status URL	apidelivery	Durable custom status + singleton instance id
Wants ordered POSTs and can live with SB redelivery	webhookdelivery	Stop-on-failure is the ordering contract
Wants files on a disk	sender or delivery	That is the SFTP pair, not this post
Needs a new mapping	New instance YAML, maybe a new ITransformer	Not a new Function project per partner

I still compose these with integrations/*.yaml rather than forking connector code. That capstone is the next Azure slot after subscription + sequence.

FAQ

Does transformation write the ledger?
No. It forwards an envelope. Ledger Ready / Delivered / Failed belong to later connectors. apidelivery writes DeliveryInProgress then Delivered or Failed (and throws if the row is missing). webhookdelivery writes Delivered only when a ledger row already existed; a missing row still completes after a successful post. Cosmos partition key is /interface.

What if I POST /deliveries twice for the same file?
You get 202 and alreadyRunning: true. The instance id is apidelivery:{publisher}:{interface}:{fileName}. A second orchestration is not scheduled.

Is page POST exactly-once?
No. Activities can replay. The connector assumes the target API is idempotent for a repeated page. Ledger updates use ETag concurrency; conflicts are not a second insert of records.

Why is apidelivery HTTP while webhookdelivery is a queue?
Paged REST needs an orchestration lifetime longer than a single Function invocation and a status URL humans and other systems can poll. A webhook batch is a short, ordered loop that fits peek-lock + abandon. I would not put a 40-page ERP load on a 30-second queue lock.

Can JsonToJson call Http?
No. One name per instance. Chain connectors instead: transform with JsonToJson, forward to another transformation instance with Http, or go straight to egress. The registry is a dictionary, not a pipeline inside one message.

Next on this track is subscription (declarative router) plus sequence (ordering gate without sleeping on the lock). Until then, the source of truth remains the repo — not a slide with boxes that never compiled.