Workflow engine recovery

Use this guide when durable work stops advancing in DALP. The Workflow Engine runs inside the same backend workload as the Platform API and Ledger Index. Recovery therefore starts with backend health and persisted transaction state, not with a separate workflow service or deployment registry.

The safe order is inspect, reconcile, repair, and resume. Never delete durable history or send a replacement transaction only because an HTTP request timed out or an indexed view is stale.

Current recovery model

Durable execution persists workflow, activity, and keyed-entity state in PostgreSQL. Healthy backend replicas share ownership. If one replica exits, another can reclaim its shards and continue from persisted state. Recovery does not require service registration, stale-deployment cleanup, or a virtual-object reset.

Prerequisites

Set the example variables:

export DALP_API_URL="https://platform.example.com"
export DALP_API_KEY="sm_dalp_operator_1234567890"
export TRANSACTION_ID="01934567-89ab-7def-8123-456789abcdef"

Quickstart: inspect one durable transaction

Read the transaction before taking any recovery action:

curl -sS "$DALP_API_URL/api/v2/transaction-requests/$TRANSACTION_ID" \
  -H "X-Api-Key: $DALP_API_KEY"

{
  "data": {
    "transactionId": "01934567-89ab-7def-8123-456789abcdef",
    "kind": "token.mint",
    "status": "FAILED",
    "subStatus": "UNKNOWN_ERROR",
    "transactionHash": null,
    "blockNumber": null,
    "errorMessage": "The signer dependency was unavailable.",
    "createdAt": "2026-08-01T08:00:00.000Z",
    "updatedAt": "2026-08-01T08:01:00.000Z"
  }
}

This response is only the starting point. Before retrying a failed or dead-letter entry, check whether a transaction hash, nonce reservation, custody approval, signed payload, or provider-side operation already exists.

Decide from the persisted state

Repair by failure domain

Force retry a failed transaction

The transaction administration route accepts only FAILED or DEAD_LETTER entries. It creates a fresh native workflow attempt and records the state transition in the transaction audit trail. DALP rejects the request when another attempt is active.

Before forcing a retry, export the failed transaction's signed payload, transaction hash, receipt, nonce, gas price, and failure details as incident evidence. A successful force retry clears attempt-specific execution fields from the transaction row before the new attempt starts; the audit trail records the transition and override metadata, not a copy of those cleared fields.

curl -sS -X POST "$DALP_API_URL/api/v2/transaction-requests/$TRANSACTION_ID/retries" \
  -H "X-Api-Key: $DALP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

{
  "data": {
    "transactionId": "01934567-89ab-7def-8123-456789abcdef",
    "previousStatus": "FAILED",
    "status": "QUEUED"
  }
}

An operator can supply a gas-price or nonce override only when incident evidence justifies it. Those fields change chain execution behavior and require nonce and replacement-transaction reconciliation before use.

After the route returns, poll the original transaction ID. The transaction row remains the stable operator-facing identity even though DALP starts a new durable execution attempt.

Workflow-specific recovery

Not every workflow is a transaction queue entry. Organization deployment, invitation acceptance, identity recovery, settlement, and other long-running operations publish their own typed status or retry surface. Use that domain route and its stable deployment or operation ID.

The same rules still apply:

1. Read the domain status and the related transaction requests.
2. Distinguish an active wait from a terminal failure.
3. Repair the failed dependency before retrying.
4. Reuse the documented idempotency or deployment identity.
5. Confirm both the durable outcome and indexed visibility.

Do not invent a generic workflow reset when the owning domain exposes no recovery route. Preserve the evidence and escalate the typed failure.

Replica restart and shutdown

A backend restart is a workload recovery operation, not a workflow reset. Readiness removes the replica from traffic. Shutdown drains HTTP, releases runner and Ledger Index ownership, and flushes telemetry. Persisted work remains in PostgreSQL and can move to another healthy replica.

If every backend replica is unavailable, durable work pauses. Restore the merged backend and its direct PostgreSQL path. There is no independent workflow deployment to register first.

Legacy identifiers

Some frozen API error codes and debug-bundle category names retain legacy prefixes for compatibility. Treat those strings as stable wire identifiers. They do not indicate that the retired workflow runtime is still a deployed production dependency.

Close the incident

Record evidence for each boundary before closing recovery:

• backend readiness and healthy durable ownership;
• transaction request state and all known transaction hashes;
• receipt and nonce reconciliation;
• signer, custody, or provider outcome where involved;
• Ledger Index checkpoint and read visibility;
• logs and traces for the recovered attempt;
• the operator decision that made retry safe.

Related operations

• Transaction tracking for queue state, receipts, finality, and read visibility
• Transaction queue lifecycle for the force-retry API contract
• Platform status for backend and dependency health
• Blockchain monitoring for RPC, Ledger Index, and reindex signals
• Workflow Engine for durability, ownership, and failure semantics