Workflow Engine

The Workflow Engine turns an accepted lifecycle request into durable platform work. It coordinates operations that cannot safely fit inside one HTTP request, including approvals, signing, transaction submission, confirmation, scheduled work, and webhook delivery. This page explains the execution model, recovery boundary, keyed serialization, and operating signals. It does not define custody policy or guarantee that an external dependency succeeds.

Place in the platform

The Workflow Engine runs in the DAPI runner role. The Platform API runs in the api role and the Ledger Index runs in the indexer role. The three roles have separate process lifecycles and scale independently, but they share one image, release, and PostgreSQL data plane. Every process sets its role explicitly; there is no combined role.

Effect Cluster carries workflow submission from API clients to runner owners through PostgreSQL-backed messages and direct runner notification. The responsibility boundaries remain explicit: each Platform API implementation package owns its endpoint schemas and handlers together, the Workflow Engine owns durable coordination, the persistence boundary owns durable state, and the Ledger Index owns event interpretation. The server package only composes those implementation-owned API groups into the public transport surface; it does not maintain a parallel contract model.

All three roles also move through environments as one versioned backend release. An upgrade cannot replace the workflow runner independently from the API implementations or Ledger Index that exchange state with it. Database migrations run before that release becomes ready, and a rollback restores the same complete runtime boundary. This keeps code version, durable state expectations, and indexed-read behaviour aligned during recovery.

Native runtime boundary

The Workflow Engine is the native Effect Cluster runner role. The Effect layer and PostgreSQL journal own execution state; the Platform API submits work and the Ledger Index reports committed results. This boundary keeps recovery based on persisted workflow state and explicit runtime ownership rather than process-local coordination.

The release compiles the role-selecting application with Bun bytecode into one architecture-specific executable. The distroless DAPI image starts the executable directly with DAPI_ROLE=runner, without a package install, JavaScript source tree, or separate process manager. Bun owns HTTP serving and signal delivery; the Effect scope still owns workflow acquisition and release. A container restart therefore re-enters the persisted cluster state instead of attempting to recover process-local work.

Typed value boundary

Values that carry a unit cross the Platform API, workflow journal, and chain boundary through an Effect Schema. The schema validates the external encoding and constructs an opaque workflow value. Domain code receives the typed value, not the string, tuple, bigint, or hexadecimal field that carried it across the boundary. Encoding follows the same path in reverse.

This distinction prevents a token amount from being added to an amount in a different denomination, or a gas price from one chain from being reused on another. It also keeps rounding visible. A calculation that produces a finite decimal or integral quantity names the output role, scale, and rounding policy instead of relying on an ambient decimal setting.

The three external boundaries have separate encodings:

1. The Platform API decodes documented decimal and integer strings before a request reaches workflow code, and encodes typed response values through the response schema.
2. The durable journal uses canonical JSON-safe encodings. Decimal values and bigint quantities are stored as schema-defined strings rather than runtime objects.
3. The chain boundary decodes RPC and ABI values into chain-specific gas, native-value, and token-quantity types. Only the chain encoder turns those values back into hexadecimal or ABI fields.

Replay compatibility belongs to the durable schema. When an established journal record has an older supported decimal representation, the schema can decode that representation into the current opaque value. New journal writes use the canonical string representation. This preserves the wire contract for recorded work without allowing the old representation to spread through new workflow code.

Workflow handlers do not construct unit-bearing values through generic decimal converters or mutate their hidden scalar representation. They use the named schema, arithmetic, comparison, and formatting operations for the value kind. Exact rational factors remain an internal calculation tool and never become an API, journal, or chain payload.

Journals record facts; the workflow body decides

A journaled step records what was observed, not what was concluded from it. This matters because the journal collapses every failure raised inside a step into one generic non-retryable outcome: a step that both reads and decides loses the decision's specific meaning the moment the decision is a refusal. Callers see "this step failed" where they needed "the organisation already has a system at this address that the index cannot confirm".

So a step that must both consult an external source and choose between outcomes is split. The step does the read and journals a value that names which case holds. A chain read belongs in that step too, because chain answers are point-in-time facts rather than lagging projections. The workflow body then branches on that recorded value and raises its own typed outcomes. Transient failures of the read retry inside the step, before the journal's error mapping; a persistent failure surfaces as the step failing, which is the correct result, because "unable to determine" is never the same as "nothing is there".

Two properties follow. A replay reproduces the branch from the recorded facts rather than re-deriving it from a source that has since moved. And an outcome that would destroy state can be refused with diagnostics precise enough for an operator to act on, instead of being flattened into a generic failure or, worse, into a silent success.

Extending such a step's recorded shape is additive. New cases are added alongside the shapes the previous code wrote, so a deployment already in flight when the change lands still decodes and replays.

Why operations are durable

A regulated asset operation often crosses systems with different response times and failure modes. A signer can require approval. A transaction can be accepted by an RPC endpoint before its response reaches the caller. A receipt can arrive before the indexed read model catches up. A process can restart between any two of those points.

The Workflow Engine records enough execution state to resume the operation instead of rebuilding it from process memory. Domain workflows divide external work into named activities, persist results that later steps depend on, and use durable waits for approvals, retries, child work, and scheduled wake-ups.

Durability preserves orchestration state. It does not turn an unsafe provider or blockchain call into an exactly-once side effect. Workflows still use idempotency keys, persisted transaction material, receipt reconciliation, and domain-specific duplicate protection where the external boundary requires them.

Execution primitives

Scheduled ownership

Recurring platform work is hosted by Effect Cluster, not by an interval owned by each backend process. Cron and singleton definitions run only on runner-role pods and elect one active owner across healthy runner replicas for monitoring rollups, account-abstraction runway maintenance, nonce recovery, transaction confirmation and reconciliation, bundler watchdogs, market-data refresh, indexer usage aggregation and wallet classification, and durable-state retention. When ownership moves, the next replica continues the cadence; stable cycle keys prevent the same scheduled window from starting duplicate durable work.

The cadence itself is deployment configuration rather than code. The blockchain-health collector cron uses monitoring.blockchainHealthCollector.intervalSeconds (default 60 seconds). The default uses a minute cron. Values below 60 must divide 60 because a cron seconds field restarts at zero each minute. This rule prevents an uneven rhythm. Those probes read the process network after control-plane saved RPC upstream pools overlay the chart baseline. The Platform API applies the same saved pools for admin serving. The transaction confirmation watcher spaces each chain's tick by durable.confirmationWatcher.intervalMs (default 250 ms, bounded between 50 ms and 60 s), so a misconfigured value can neither hot-loop the per-chain database poll nor push confirmations out of near-real-time. Catch-up after owner downtime is bounded the same way: one monitoring rollup cycle rebuilds at most 24 missed hour buckets and logs whether it capped the backfill and how many gap hours remain, so a long outage drains as visible increments across successive cycles rather than one unbounded first pass.

Recurring passes that reconcile derived state are written to converge rather than to succeed once. A pass reads its inputs fresh, writes idempotently, and leaves anything it could not resolve for a later pass, so a temporarily missing input costs a delay instead of requiring a restart to retry. That is why this work is owned by a singleton on a cadence rather than run as a startup step.

Single ownership is the right shape for work that must not run twice, and the wrong shape for work that is safe to divide. The Ledger Index uses both. Its per-chain ingest loop is a singleton, because the range near the chain tip can reorganise. Its historical catch-up is not: the chain's owner records leased ranges of settled history in the database, and every indexer replica claims one, works it, and marks it complete. Ownership there is a lease with an expiry rather than an elected singleton, so a replica that stops has its range offered to another instead of stalling the queue behind it.

The daily retention schedule keeps a 30-day operational window for completed cluster messages and terminal workflow status projections. Its repositories lock and delete bounded batches with skipped locked rows, so cleanup makes progress without one unbounded transaction or a process-local lock token.

A separate daily cluster cron owns webhook audit retention. Its 30-day window starts only after an event reaches dispatch_settled_at. The repository selects bounded, skip-locked batches and deletes receipt rows, then delivery rows, then the settled event. A null settled marker, a pending or provisional state, or a test-delivery marker keeps the row outside the policy. Concurrent dispatch and receipt work therefore wins its lock and leaves the event for a later pass. The owner reports deleted child and event rows, remaining eligible rows, failures, and pass duration so operators can distinguish a quiet pass from a stalled backlog.

How ownership is held and refreshed

Shard ownership is a lock PostgreSQL enforces, not a lease a timer enforces. Each runner or indexer replica reserves one session-stable database connection for the life of the process and takes a session advisory lock on it for every shard it owns. A replica that stops for any reason, including a crash, ends that session, and PostgreSQL releases its locks immediately, so a surviving owning replica can take the shards without waiting for a timer to expire.

That design makes the reserved connection the ownership itself. The engine re-asserts its locks and its runner heartbeat on a refresh interval, and gives every lock operation a deadline derived from the refresh interval and the liveness window together. If a lock operation overruns the deadline, the engine treats shard-lock storage as unhealthy, releases every shard the replica owns, and rebuilds the reserved connection. That is the recovery path working as intended, but its cost is real: in-flight work on those shards is abandoned and re-elected elsewhere, including a chain's active ingest loop.

The refresh cadence and the liveness window that derive that deadline are fixed by the platform, not exposed as environment settings, because they are only safe in combination with PostgreSQL holding the lock. What is configurable is the shard count under durable.sharding, which sets how much work a replica gives up when it does lose a deadline. It has to be identical on every DAPI role, because it is the modulus that maps a piece of work to a shard: a process using a different count addresses shards no owner recognizes.

Keyed serialization

Keyed entities serialize work for one domain identity. The key can represent a sender on one chain, a queue, a workflow coordinator, or another resource whose commands must not race. Unrelated keys continue in parallel.

Serialization is scoped to the selected key. It does not replace database constraints, custody approval rules, tenant authorization, or on-chain finality.

Transaction lifecycle

The shared transaction path combines durable orchestration with explicit status projection.

The transaction construction boundary owns the fixed ERC-8021 schema-0 attribution suffix. It computes the suffix once with the platform's retained EVM hex primitives, then reuses the exact bytes for every submission attempt. The attribution is not persisted workflow state and performs no external I/O, so it does not change retry or recovery semantics. This ownership also avoids a separate production library edge solely for suffix encoding.

The workflow can distinguish four milestones:

1. The request is accepted and assigned an execution identity.
2. The transaction is signed and submitted.
3. The configured receipt or finality condition is met.
4. The Ledger Index reaches the transaction block when the response needs read-after-write visibility.

On-chain confirmation and indexed visibility are separate facts. A completed chain operation can be final before every read surface shows its effects.

Multi-broadcast creation phases

Some creation flows are not one transaction. Asset creation runs a feed phase that can broadcast three writes in sequence: create the price feed, submit its first observation, and install that feed's staleness bound on the price resolver, all under one durable workflow key. Each broadcast is a separately-keyed step, so a replay resumes at the first one that has no recorded result instead of re-broadcasting the whole phase.

The boundary that matters is where the parameters come from. The resolver address and the staleness value are resolved from the asset template at the request boundary, never carried in caller input, and the write targets only the feed the same phase just created. Broad template-authoring authority therefore cannot be laundered into an arbitrary write against someone else's feed.

The invariant is fail-closed ordering: the staleness bound is the terminal step, and a failure there leaves the asset paused. An asset whose freshness bound was never installed must not be mintable, so a partially-completed phase ends in a state that blocks issuance rather than one that silently permits an unbounded price. The operating consequence is that a stuck creation shows up as a paused asset with a recorded partial hash list, and is resolved by resuming the same workflow key, not by creating a second asset.

Provider-managed submission has an additional evidence boundary. The Workflow Engine checks the provider's current status before waiting for approval. A reported rejection becomes a terminal transaction outcome with its available reason; it is not allowed to sit in an approval wait or remain labelled as an in-flight broadcast. If the provider confirms a transaction but the response carrying its hash is lost, the engine can recover the hash through the operation's stable external identifier and persist it before reporting success. An unavailable provider or an unsafe write-back leaves the original failure unchanged.

A custody provider can also silently drop an accepted intent: it reports acceptance but never produces an on-chain transaction. For an organization-scoped custody write, the Workflow Engine detects that stall from the poll pattern alone and recovers it automatically. A platform-scoped write, which has no organization to recover on behalf of, falls back to the standard poll-budget timeout instead. The recovery runs inside the same durable execution as the write it is recovering, so its own retries and waits replay correctly if the process restarts mid-recovery.

Account abstraction address resolution

Sponsored operations use an ERC-4337 EntryPoint contract. The address that each chain's on-chain directory records for the EntryPoint is a configured address. It is not always the EntryPoint contract itself. On a public chain that already hosts the standard deployment, the platform records a small reference contract that points at the standard one, because the directory must be able to introspect the address it holds.

Off chain, the reference contract is the wrong target. It holds no deposits, it does not define the signature domain, and it does not emit the events that a receipt is read for. So the platform resolves the recorded address to the real EntryPoint at one shared boundary, and every account-abstraction step uses the resolved result: the operation hash and its signature domain, nonce reads, validation simulation, submission, paymaster deposits, and receipt parsing. No workflow keeps a separate lookup of its own.

That single boundary also sets the dependency direction. The bundler workflow reads the resolved EntryPoint through it and holds no build-time dependency on the account-abstraction projection tables. Only its tests name the canonical EntryPoint constant directly. An operator reading a stale EntryPoint therefore looks at the resolution boundary and the directory record behind it, never at indexed factory rows, because the workflow never consulted them.

The indexed rows those tables hold record their own removals rather than deleting them. A validator module that a factory removed keeps its row with the removal height stamped on it, so a later replay of chain history cannot restore a module the factory no longer offers, and a reader that asks for the modules a factory offers receives only live ones. An operator who sees a module missing from that list should read it as removed on chain at a known height, not as an indexing gap.

Resolution has two outcomes and one refusal. A recorded reference resolves to the contract it points at. A recorded address that is not a reference passes through unchanged. A network failure during the check is neither outcome, so it is reported as a retryable error instead of being treated as a pass through. A network blip that was read as a pass through would let one operation use two different signature domains, which fails validation later and is much harder to diagnose.

One boundary keeps both values. The client-facing bundler interface accepts either address for gas estimation and for the paymaster methods, because the platform published the configured one. Submission is different. The signature of an operation is bound to the address that the operation was hashed over, so a submission must name the resolved address. A submission that names the configured address is refused, and the operation must be built and signed again. The supported-EntryPoint list reports only the resolved address.

Durable replay makes this resolution non-retroactive, and operators need to plan for that. An operation that recorded an address before a resolution change replays the address it recorded. Operations in flight across such a change are drained rather than expected to heal, and collected multi-signature approvals gathered under the earlier domain are cancelled and re-gathered.

Synchronous and asynchronous calls

Synchronous, asynchronous, and hybrid responses attach to the same durable execution.

Retry a state-changing API call only according to its idempotency and status contract. A lost HTTP response does not prove that the durable operation failed to start.

Persistence and recovery

PostgreSQL stores workflow messages, runner ownership, replies, schedules, and domain status projections. Ordinary repositories use each role pod's bounded application pool, connected directly to PostgreSQL. Runner and indexer pods also use the native Effect Cluster pool because a session holds each pod's advisory ownership lock; that lock's session identity is why the ownership path can never cross a transaction-mode pooler. API pods use only the cluster client and do not pay the runner ownership cost. Completed transport records and terminal status projections are operational data rather than an audit ledger; cluster-owned retention removes them after 30 days in bounded batches.

When a runner pod leaves:

1. Readiness marks the pod unavailable.
2. Workflow and entity handlers receive a bounded termination period.
3. The pod releases its shard ownership and runner listener.
4. Persisted work remains available for another healthy runner.
5. The new owner continues from recorded workflow and domain state.

Database availability is therefore part of workflow availability. Work that cannot persist or read its durable state does not advance as if it succeeded.

Steps that write state the workflow does not own

A durable step may write state that outlives its own workflow. Organization deployment does this once: the step that saves the organization's system settings also puts the new system in the indexing scope the Ledger Index reads. That step is the earliest point at which the system address exists, so no earlier step can name it.

The order inside such a step is the design, not an implementation detail. The step reads the address the organization carries now, takes that address out of scope when the deployment replaces it, and only then writes the new one. A durable step that fails part way is re-run from its first line, so the read has to happen before the write that overwrites what it reads. Repairing the older address in a failure handler would not survive the same event, because a runner that is killed skips failure handlers.

The workflow cannot cover the case where it disappears before it reaches that step. A separate scheduled sweep declares a deployment failed when its execution is gone, and it is that sweep, not the workflow, that takes the abandoned system out of scope. Both writers are automatic. An operator only intervenes to turn a known system on or off, and does not have to clean up after a failed deployment.

Scaling model

Work is partitioned into shard groups and keys. Healthy runner replicas share the default and workflow groups, while indexer replicas share only the indexer group. Each execution or entity key has one active owner at a time. Adding API replicas does not add runner capacity; increasing runnerDeployment.replicaCount does, subject to database connections, chain RPC limits, signer throughput, and external provider limits. The Helm capacity gate prices API, runner, and indexer pods separately, including rollout overlap, the non-DAPI reserve, and required PostgreSQL headroom.

The Ledger Index and recurring schedules use the same cluster ownership model. Each enabled chain has one active ingest owner, while each scheduled responsibility has one active cadence owner. The indexer adds a database owner epoch to fence late writes after ownership moves between replicas; scheduled durable launches use stable cycle keys to deduplicate handoffs.

Shard ownership itself is a PostgreSQL advisory lock held on a session-stable connection, not a lease that expires on a timer. Mutual exclusion is therefore enforced by the database rather than by a clock, and it holds regardless of how a replica fails: a crashed or gracefully stopped runner drops its lock within seconds. The deliberate cost is that a network-partitioned runner keeps its lock until its backend is reaped, so the affected shard stalls rather than moving. That trade is chosen rather than incidental. Shard configuration is process-wide and cannot be scoped per shard group, so switching to an expiring lease would apply to every group at once. While the Ledger Index survives a brief double-owner on its epoch fence, the workflow and default groups carry nonce reservations and workflow executions that have no equivalent fence. A stalled shard is recoverable; duplicate nonces and duplicated side effects are not. Shortening dead-peer detection on the cluster connection is the correct remedy for the stall, because it shrinks the window without ever allowing two owners.

Failure semantics

DALP distinguishes business outcomes from infrastructure failures.

A related failure mode is bounded to owning-role startup. A starting runner or indexer pod claims its share of a shard group as soon as its cluster runtime opens, which happens before its capabilities have registered. Until registration completes, the pod holds keys it cannot serve. The engine tolerates a bounded amount of that gap by retrying, and past it records a registration defect against the claimed work, so a caller sees a failed operation rather than a slow one. Each role narrows the gap to the runtime's own bring-up by registering every owned capability before any startup work that reaches the database or a chain endpoint. Registration cannot precede the runtime that hosts it, so the gap is made small rather than removed. API pods never enter this window because they use the client-only layer and own no shard.

A second startup bound covers dependency loss during a rolling upgrade. Before a role's health server first binds, a brief connection drop during a rollout can relaunch that role graph in process for a duration budget inside the startup probe. The bound is time-based rather than a fixed retry count because those failures often die in well under a second and would otherwise exhaust a count budget while the probe still had minutes left. After the server has bound, and once the budget is exhausted, the process exits and the platform restarts the pod, so a persistent outage still fails fast. The relaunch rebinds every context-scoped singleton for that role, so a replacement scope never uses resources from a failed earlier attempt.

One failure mode sits outside that table because it produces no outcome at all. Each capability declares the workflows and entities it serves, and a runner only serves what a capability declared. A call to a workflow no capability declared waits indefinitely instead of failing, so it surfaces as an operation that never leaves its initial state and never reports an error, not as a terminal or retryable failure. Operators should read an operation that is accepted, never progresses, and produces no failure signal as a registration gap rather than a slow dependency, and escalate it rather than retrying. The platform's own startup checks assert that every capability present is also registered, so this state indicates a build that never should have shipped.

Observability and status

The Workflow Engine emits traces, metrics, and structured logs for operation boundaries, execution outcomes, retries, queue and runner health, and completed or stalled work. Platform Status derives workflow health from persisted execution projections and configured pending and no-mutation thresholds.

Those signals use stable operation names and bounded lifecycle labels rather than workflow keys, organisation identifiers, transaction hashes, or payloads. The deployment identity is carried on the telemetry resource, so a central dashboard can separate two installations without turning one workflow execution into a metric series. Scheduled work records attempt, outcome, duration, and last success independently: a genuine zero, an overdue schedule, and a collector that stopped reporting therefore remain different states. Applications export complete workflow traces; any sampling decision happens at the trace-affine collector after the full tree and its error or critical class are available.

Those execution projections are written by the capability packages, not by the engine. A workflow publishes its own status transitions as it runs, and a long-running workflow may publish incremental progress snapshots between them. The engine guarantees that those writes survive a restart; it does not decide what they contain. That ownership split is why a workflow can be durably alive and still report no progress: the projection only advances when the capability publishes.

Runtime-level faults reach the same place. Warnings raised by the Bun process that hosts the engine, such as dependency deprecations and experimental-feature notices, are republished through the structured logger under the dapi.process category rather than left as unparsed stderr text. This matters for an engine whose durability depends on its database driver: a driver deprecation is emitted once at startup, and without that republishing it never appears in a log query and no alert can be written against it.

Use these signals together:

Architecture boundaries

Related architecture

• Platform API for request contracts and transport boundaries.
• Signing flow for approval, signing, broadcast, and confirmation.
• Ledger Index for indexed visibility, reorg recovery, and reindexing.
• Database for the shared PostgreSQL data plane.
• Failure modes for dependency outages and recovery behavior.
• Observability for traces, metrics, logs, and operational evidence.