SFTP In and Out on Azure Functions: publish, sender, and sequenced delivery

Partners still drop files on SFTP. Azure still wants blobs, queues, and a ledger you can audit. SFTP integration on this stack is not “one Function that does everything.” It is three small, config-driven connectors: publish for inbound poll, then either sender (any order) or delivery (strict order) for outbound upload. Same problem space as classic file moves on Azure Integration Services — here the building blocks are Azure Functions microservices you compose with YAML, not a single designer canvas.

That is the whole job in one sentence: pull files in without mid-write corruption, land them in Blob, and push results out with atomic renames — choosing ordered vs unordered egress on purpose.

Think of a warehouse night dock. Inbound is a lockable drop-box you empty on a schedule. Outbound is either “ship whenever a box is ready” or “load truck lanes in sequence so carton 2 never leaves before carton 1.” Same forklift motion; different traffic rules.

This post is the SFTP deep dive in my Azure integration series. The overview lives in microservices integration patterns on Azure Functions; the HTTP edge pattern is in the facade API microservice. Source of truth for names and behaviour is the public repo github.com/fransiscuss/integration-azure (commit 2539242 when I wrote this).

What SFTP integration means here

I treat SFTP as a transport boundary, not a business process. Config owns host, path, pattern, and destinations. Connector code stays generic. That is the shape of azure integration architecture I actually ship: shared packages, per-partner config, managed identity. Integrations are composed later in YAML under integrations/ — you do not fork C# per partner. For searchers comparing stacks, this is SFTP Azure Functions done as reusable connectors, not a one-off script in a Function App.

Connector	Trigger	Direction	Ordering	Terminal?
publish	Timer (SCHEDULE)	SFTP → Blob	N/A (batch poll)	No messages out; blobs drive downstream
sender	Service Bus queue	Blob → SFTP	Unsequenced	Yes (optional Event Grid notify)
delivery	Service Bus queue	Blob → SFTP	Sequenced per interface	Yes (requeues on own input)

Shared spine matches the rest of the catalog: Blob payloads, Cosmos ledger where needed, Key Vault references for credentials, managed identity RBAC. Peek-lock style settlement on Service Bus still matters on the egress side — same instincts as my Azure Service Bus peek-lock guide.

Inbound: publish as a scheduled drop-box

The timer function does one important thing before any SFTP I/O: take a blob lease. If another scaled instance already holds it, this tick logs and exits. That keeps “list + download + archive” singleton without relying on hope.

// functions/publish/Functions/PublishFunction.cs
[Function("PublishPoll")]
public async Task Run([TimerTrigger("%SCHEDULE%")] TimerInfo timer, CancellationToken ct)
{
    await using var lease = await pollLock.TryAcquireAsync(ct);
    if (lease is null)
    {
        LogPollSkipped(timer.ScheduleStatus?.Next);
        return;
    }

    await poller.RunAsync(ct);
}

Once the lease is held, the poller lists the remote in-directory with a compiled regex and a minAge floor (default five minutes). Young files stay put so you do not yank a zip while the partner is still writing it.

// functions/publish/Polling/PublishPoller.cs (happy path core)
files = await sftp.ListAsync(opts.InDirectory, pattern, opts.MinAge, ct);
// ...
foreach (var destination in opts.Destinations)
{
    var fileName = evaluator.EvaluateTemplate(destination.FileNameTemplate, context);
    var blobPath = string.IsNullOrEmpty(destination.PathPrefix)
        ? fileName
        : $"{destination.PathPrefix.TrimEnd('/')}/{fileName}";
    var blob = blobs.GetBlobContainerClient(destination.Container).GetBlobClient(blobPath);
    await using var stream = await sftp.OpenReadAsync(file.FullPath, ct);
    await blob.UploadAsync(stream, overwrite: true, ct);
}
// then archive or delete remote per onSuccess

Design choices I care about in production tickets:

• Fresh stream per destination — no whole-file buffer; a mid-fan-out failure does not force a fake “all destinations” assumption.
• overwrite: true — retries after partial fan-out stay idempotent.
• Per-file failure leaves the remote file — one bad name does not block the batch; next poll retries.
• No output queue from publish — downstream reacts to blob-created events (or a later stage you wire). The connector stays a pure file bridge.

Config is the contract. Sample YAML from the repo:

# functions/publish/config/publish.sample.yaml
schedule: "0 */5 * * * *"
publisher: acme-erp
interfaceName: invoices-outbound
sftp:
  host: sftp.partner.example.com
  port: 22
  username: integration
  privateKey: "@Microsoft.KeyVault(SecretUri=https://kv-integration.vault.azure.net/secrets/sftp-private-key)"
inDirectory: /outbound/invoices
filePattern: "^INV_(?<ts>\\d{8}_\\d{6})\\.zip$"
minAge: "00:05:00"
onSuccess: archive
archiveDirectory: /outbound/invoices/archive
destinations:
  - container: invoices-raw
    pathPrefix: inbound
    fileNameTemplate: "INV_${$route['ts']}.zip"
  - container: audit
    pathPrefix: erp
    fileNameTemplate: "${$route['ts']}_invoice.zip"

Named regex groups become template tokens. That is how INV_20260827_010000.zip becomes a stable blob name without custom C#.

Ops note from the README I keep in runbooks: even with the lease, set WEBSITE_MAX_DYNAMIC_SCALE_OUT_SIZE = 1 for this app. Extra instances never do useful poll work; they only burn cold starts.

Outbound: sender when order does not matter

sender is the unsequenced SFTP upload. Trigger is a Service Bus queue with explicit settlement. Input is a PipelineEnvelope v1. Payload is Output when set, otherwise Source.

Happy path, in order:

1. Ledger lookup — if already Delivered, complete without re-uploading.
2. Stream blob → SFTP via UploadAtomicAsync (temp name, then rename).
3. Optional flag file (partners that watch .flg sidecars).
4. Optional Event Grid completion event.
5. Ledger Delivered last so a crash earlier is recovered by redelivery; completed uploads stay idempotent on deterministic names.

// functions/sender/Delivery/SenderProcessor.cs
if (record?.Status == LedgerStatus.Delivered)
{
    return SenderResult.DuplicateSkipped;
}

var tokens = TokenTemplate.TokensWithFile(envelope.FileNameTokens, source.Path);
var remoteName = TokenTemplate.Render(opts.FileNameTemplate, tokens);

await using var content = await blobClientFor(source.ToUri()).OpenReadAsync(cancellationToken: ct);
await sftp.UploadAtomicAsync(content, opts.RemoteDirectory, remoteName, ct);
if (opts.FlagFile.Enabled)
{
    await sftp.PutFlagFileAsync(opts.RemoteDirectory, remoteName, opts.FlagFile.Suffix, ct);
}
// optional Event Grid...
// then ledger → Delivered

Poison goes to dead-letter with reasons like InvalidEnvelope or UploadFailed. Transient SFTP/blob/Cosmos issues abandon for host redelivery up to maxDeliveryCount. If the ledger row is missing on sender, upload can still complete — the Delivered stamp runs only when a record exists. delivery is stricter: a missing ledger row is poison (LedgerRecordMissing).

# functions/sender/config/sender.sample.yaml
INPUT_QUEUE: sender-in
Sender:
  RemoteDirectory: /inbound/invoices
  FileNameTemplate: "{fileName}"
  Sftp:
    Host: sftp.erp.example.com
    Username: integration-svc
    PrivateKey: "-----BEGIN OPENSSH PRIVATE KEY-----..."
  FlagFile:
    Enabled: true
    Suffix: .flg

Outbound: delivery when order is the product

Some partners treat file order as business correctness — day-end packs, sequential acknowledgements, “process 001 before 002.” That is delivery, not a sleep loop inside the Function.

A file uploads only when the ledger says Ready and it is the oldest pending record for that interface. Otherwise the connector schedules a delayed copy on its own input queue and completes the current message. The wait never holds a Service Bus lock. That is the intentional fix called out in the processor docs versus older “sleep while locked” patterns.

// functions/delivery/Sequenced/DeliveryProcessor.cs
if (record.Status != LedgerStatus.Ready || oldest?.Id != record.Id)
{
    await requeue(envelope, opts.RetryDelay, ct);
    return DeliveryResult.Requeued;
}

await using var content = await blobClientFor(source.ToUri()).OpenReadAsync(cancellationToken: ct);
await sftp.UploadAtomicAsync(content, opts.RemoteDirectory, remoteName, ct);
// flag file optional...
await ledger.UpdateAsync(record with { Status = LedgerStatus.Delivered, UpdatedAt = DateTimeOffset.UtcNow }, ct);

Requeue message IDs use {eventId}:r{deliveryCount} so duplicate detection does not collide with the just-completed original. Cosmos needs a composite index on (interface, fileTimestamp) for FindOldestPendingAsync. Metrics: watch messages.requeued as your “waiting for its turn” signal.

# functions/delivery/config/delivery.sample.yaml
INPUT_QUEUE: delivery-in
Delivery:
  RemoteDirectory: /inbound/invoices
  FileNameTemplate: "{fileName}"
  RetryDelay: "00:00:30"
  Sftp:
    Host: sftp.erp.example.com
    Username: integration-svc
    PrivateKey: "-----BEGIN OPENSSH PRIVATE KEY-----..."
  FlagFile:
    Enabled: true
    Suffix: .flg

Decision guide: sender vs delivery

Question	Prefer sender	Prefer delivery
Partner care about arrival order?	No	Yes — strict per interface
Latency if file N waits on N−1?	Want lowest lag	Accept RetryDelay polling
Ledger role	Dedupe + Delivered stamp	Dedupe + Ready/oldest gate
SB permissions	Receiver (+ Event Grid sender optional)	Receiver and Sender (scheduled requeue)
Failure mode to monitor	UploadFailed / DLQ	Requeue rate stuck high = head-of-line blocker

I do not put both on the same interface “just in case.” Pick one egress mode in the integration YAML and keep the other connector for different partners.

How this composes with the rest of the pipeline

Typical happy spine after inbound land:

1. publish → blobs in invoices-raw (and optional audit copy).
2. Optional crypto/transform stages (preprocessing, decompress, transformation, …) on Service Bus — covered in later series posts.
3. Ledger moves toward Ready.
4. sender or delivery pushes to the partner’s inbound directory with atomic rename + optional flag.

Nothing in these three folders knows partner invoice rules. That is the point of the Lego model from the series overview: CI/CD ships shared connector packages; each integration is config.

Failure modes I design for

Mode	What happens	What you do
Partner still writing the zip	minAge skips young files	Tune minAge to partner write time; never disable blindly
Two publish instances	Lease loser skips tick	Keep scale-out = 1; alert if skips spike with no winner work
One destination upload fails	Remote left in place; next poll retries with overwrite	Fix container RBAC / path; do not manual-delete mid-retry
Egress already delivered	DuplicateSkipped	Expected after abandon-before-ledger-update races
Sequenced head blocked	Requeue storm on older file	Inspect oldest ledger row; poison permanent SFTP errors
Secrets wrong	Auth failures on list/upload	Key Vault refs + managed identity Secrets User — not app settings plaintext

What I want junior engineers to copy

• Separate ingress poll from egress upload. Different triggers, different failure domains.
• Put ordering policy in a connector choice, not in scattered if flags inside one mega-Function.
• Make uploads atomic and ledger updates last.
• Never sleep on a Service Bus lock to wait for order — schedule a requeue instead.
• Prove behaviour with real repo paths, not slideware connector names.

If you are wiring this for real, start from the READMEs under functions/publish, functions/sender, and functions/delivery, then compose instances in integrations/*.yaml. Next Azure deep dive in this series is the PGP boundary pair (preprocessing / postprocessing).

FAQ

What is SFTP integration on Azure Functions in this design?
Config-driven microservices that move files between partner SFTP and Azure Blob/Service Bus without partner-specific code. Inbound uses timer-based publish; outbound uses queue-triggered sender or sequenced delivery.

Why not one Function for both directions?
Different triggers, scale rules, and failure semantics. A timer poll with a lease is not the same reliability story as a competing-consumer queue with dead-letter reasons.

When should I choose delivery over sender?
When the partner requires strict per-interface order. If files can land in any order, sender is simpler and lower latency.

How does minAge prevent corrupt downloads?
Files newer than minAge are skipped on list. You only stream objects that have sat long enough to be fully written by the partner process.

Where do credentials live?
In deployed environments, SFTP password or private key app settings are Key Vault references (managed identity as Secrets User). The publish.sample.yaml shows that @Microsoft.KeyVault(SecretUri=…) shape. The sender / delivery sample files use PEM placeholders for local readability — production still wires Key Vault the same way as publish.

Does publish write Service Bus messages?
No. It writes blobs (optionally many destinations). Downstream stages subscribe to blob activity or your own wiring. That keeps the poll connector narrow.

Where is the code?
https://github.com/fransiscuss/integration-azure — folders functions/publish, functions/sender, functions/delivery, plus shared/Integration.Connectors.Sftp.