Deployment topology

DALP deployments separate public access, backend processing, durable data, chain connectivity, signing, storage, and observability into explicit trust and recovery domains. The backend uses three required DAPI process roles: API, Workflow Engine runner, and Ledger Index. All three use one image and release, but each workload sets one role explicitly. There is no combined role or default. Use this page to place those workloads and their dependencies in a managed, customer-hosted, hybrid, or restricted environment.

The topology does not choose custody policy, EVM governance, recovery targets, data retention, or provider service levels for your organisation. Those decisions remain part of the deployment design and operating model.

Startup and recovery ordering

The deployment applies versioned database migrations and verifies database readiness before API replicas accept traffic. Only then do the API, Workflow Engine runner, and Ledger Index roles share the PostgreSQL data plane. This ordering keeps migration ownership, runtime recovery, and indexed-read availability explicit across managed and self-hosted topologies.

A failed migration halts the release before DAPI role pods roll, and the failed migration job is retained rather than cleaned up with the release, so its logs remain available for diagnosis after the deployment has stopped. Retention is bounded and the next upgrade reclaims the previous job, which keeps failure evidence available without leaving finished workloads in the namespace indefinitely. The same retention is what allows a failed migration to be alerted on, because the job has to outlive its own failure to be observed.

Each application image is an immutable release unit containing the complete runtime needed by that workload. Images do not install application dependencies or assemble the backend during cluster startup. The same image digest progresses between environments, while environment-specific configuration and secrets are supplied at deployment time. This makes provenance, rollback, and restricted environment mirroring properties of the image rather than of a live package installation.

The DAPI image is the narrowest runtime form: the build produces a bytecode-enabled Bun executable for Linux AMD64 or ARM64, and a non-root distroless image starts that executable directly. Release publishing combines both variants under one multi-architecture manifest. The required DAPI_ROLE selects only api, runner, or indexer at startup. Console, documentation, migration, and contract workloads also run on Bun, but retain their runtime or toolchain files where server-side rendering, migration loading, or contract deployment requires them.

Startup also installs the process diagnostics listener before any other work. Bun's Node-compatible process.emitWarning path writes warnings to stderr as free text, so a deprecation or experimental-feature notice otherwise arrives with no level, category, or timestamp and is skipped by structured log queries. The listener republishes each warning under the dapi.process category with its name and stack, so these records aggregate and alert like any other. Process- fatal conditions are not intercepted: Bun terminates the executable and lets the deployment restart it.

Each DAPI role binds a health port before it builds its role graph. The Helm workloads therefore use /healthz as a startup probe; liveness begins only after that probe succeeds, while /readyz reports role-local dependency readiness. Only API pods are selected by the public DAPI Service. A pod that exceeds the startup budget should be investigated through its logs and role dependencies rather than treated as a routing failure.

The boot window before first bind is also where a rolling upgrade can sever the process from its dependencies, most visibly when a brief connection drop during an autoscaler surge or a dependency roll in the same release drops open connections at once. A failure in that window often dies in well under a second, so the role application relaunches in process for a duration budget inside the startup probe rather than after a fixed retry count; a failure after first bind, or one that outlives the budget, still exits so the platform restarts the pod. Deploys therefore survive a transient connection loss without every new role pod CrashLooping through the release wait.

Readiness latency belongs to the role pod's own database path. Each DAPI role connects directly to PostgreSQL through bounded pools, and /readyz uses a dedicated direct health session. A database that is unreachable fails the check immediately. The backend bounds the database leg at eight seconds and the in-memory host leg at one second, reports an expiry as an unavailable dependency, and the readiness probe allows 10 seconds over a longer period so the two checks cannot overlap it. The eight-second bound is deliberately the saturation threshold: it decides how much pool queue wait becomes a structured 503, not how fast an outage is detected, and a tighter bound converts transient queue waits into readiness flapping across every role pod at once. The bound itself is what preserves the structured 503: a check that never answers looks identical to a dead process, so a single slow but reachable database would otherwise remove every API replica from the Service and stall a rolling update that is not allowed to run below full capacity. Size the connection budget for every role and read repeated readiness timeouts with the role-specific database gauges rather than as a fault in the pods being ejected.

Current runtime topology

The three role graphs are explicit and have separate scale and recovery domains:

The DAPI subchart always renders all three role Deployments. Scaling API replicas does not add shard owners or chain listeners. Runner and indexer replica counts are fixed independently, and database connections, RPC limits, signer throughput, and provider limits still bound useful scale.

What the platform chart actually deploys

The platform chart is an umbrella over a small, fixed set of workloads, each behind its own enable flag. The deployed set is therefore an environment decision rather than a fixed list, and reading it is the fastest way to check a running deployment against this page:

There is deliberately no separate workflow or indexer subchart and no RPC gateway chart. The DAPI subchart owns the three mandatory role Deployments. Outbound EVM traffic is routed by the backend's own RPC control plane using endpoints configured after deployment. A topology diagram that combines the roles or omits one does not describe this platform and should be corrected rather than reconciled.

Control-plane RPC upstream pools reach that process network through one overlay path. Native entity handlers, including the blockchain-health collector, resolve RPC endpoints from the process network after those saved pools apply. The Platform API applies the same saved pools for admin serving. Endpoint URLs that embed credentials, a query string, or a fragment are refused before they enter a pool.

The version stamp itself does not restart the pooler. The pooler workload rolls only when the rendered content of its configuration or user list changes, because its restart-triggering checksums hash that rendered data rather than whole templates whose labels carry the release version. A pooler restart severs every pooled connection at once, so restarting it on every release would turn each upgrade into a platform-wide connection loss event; with content-keyed checksums the pooler rides through releases whose pooling configuration is unchanged.

Every chart in a release carries the same version as the application it deploys. The platform chart, its subcharts, the support and observability groups, and the environment wrapper charts are all stamped from one release number, so a deployed chart version identifies the platform version directly and a mismatch between the two is a finding rather than an expected difference. The chart groups still install and upgrade as separate units, which is what makes a partial upgrade visible as a version difference between groups instead of hidden inside one release.

Deployment paths

The runtime shape stays the same across hosting models. Ownership and network paths change.

Bring-your-own-cloud and on-premises installations are customer-hosted paths. A private EVM network changes the node and governance owner, not the DALP backend structure.

Runtime zones

Combining zones for a small environment changes the risk profile. It does not remove the responsibility boundary.

The durable data zone owns the trust anchor for its own transport, not only the credential. Enabling TLS for the cache selects an encrypted connection, and the platform then verifies the endpoint certificate against the trust store the container image carries. An endpoint whose certificate comes from a private authority is therefore refused until that authority's bundle is supplied with the cache settings, because no credential or network control substitutes for the missing anchor. Verification stays enabled by default, so an environment that runs its own authority declares it rather than turning the check off.

DAPI role and shard model

API replicas serve HTTP and use the Effect Cluster client; they own no shard, schedule, or fleet listener. Runner replicas own the default and workflow shard groups plus recurring maintenance. Indexer replicas own only the indexer shard group and the chain listeners. Durable work is partitioned by shard group and stable domain key. Recurring maintenance registers cluster cron or singleton owners, so monitoring, runway, nonce, transaction, bundler, market-data, and retention passes run once per cluster rather than once per runner replica. The Ledger Index registers one singleton per enabled chain, so only one indexer replica owns a chain's active ingest loop at a time.

Webhook audit retention is one of those deployment-scoped owners. Each daily occurrence keeps 30 days from the event's settled marker and drains bounded repository batches in foreign-key order: receipts, deliveries, then events. Skip-locked selection lets active dispatch or receipt work keep the row for a later occurrence. A failed occurrence leaves committed batches intact and the next owner resumes the same idempotent cutoff policy, so recovery does not need a manual delete or a replica-local cursor.

Ledger Index work that is not about one chain gets its own deployment-scoped owner rather than riding along with a chain: usage aggregation and operator and bundler wallet classification each run as a single cluster owner covering every chain the deployment indexes. Keeping them off the chain singletons means their cost does not multiply with the chain count, and their cadence does not change when chain ownership moves between replicas.

The cadences these owners run at are deployment configuration rather than code. monitoring.blockchainHealthCollector.intervalSeconds (default 60) paces the chain health polls. The default uses a minute cron. Values below 60 must divide 60 because the seconds-step cron restarts each minute. The per-network RPC probes inside one collector pass run concurrently against the process network. The control-plane overlay is applied first. A pass therefore costs the slowest probe rather than the sum across networks. It reports health for the endpoints operators saved rather than only the chart floor. durable.confirmationWatcher.intervalMs (default 250 ms, bounded 50 ms to 60 s) spaces each chain's transaction confirmation tick, and indexer.derivedMeasures.intervalMs and rowCeiling (defaults 1000 ms and 500 rows) pace the derived-measures drain each chain owner runs. Monitoring rollup catch-up after owner downtime is capped at 24 hour buckets per cycle, with the cap and the remaining gap hours logged, so a replica that returns from a long outage backfills across successive cycles instead of issuing one unbounded pass.

Singleton ownership bounds live following, not historical catch-up. A deployment with more enabled chains than replicas leaves some replicas owning no chain, and that is intended for the ingest loop, because the range near the chain tip can reorganise and only one writer may hold it. Settled history has no such constraint. The chain's owner divides it into leased ranges below the finalized watermark, and every indexer replica claims and processes them, so the work of a reindex or a cold start is spread across the indexer pool. Adding indexer replicas therefore shortens historical catch-up even though it does not add chain owners.

Two limits bound that fan-out, and neither is the replica count. Chain reads pass through the same per-upstream admission control as forward sync, so a rate-limited RPC endpoint governs request pressure however many replicas claim ranges. Database pressure is bounded by a configured ceiling on concurrent claims per chain rather than by the number of workers a replica runs, because the pooled server connections a claim consumes are bounded per indexer pod and do not grow within that pod. Raise indexer.backfill.maxInFlightClaimsPerChain to give catch-up a larger share of that pool, and re-check the connection budget when indexer capacity changes.

Ownership itself is held as a session advisory lock on one reserved PostgreSQL connection per owning role pod, which is why the backend needs a session-stable path to PostgreSQL rather than a transaction-pooled one. Plan capacity for that connection separately: it is held for the life of the process and never serves query traffic. Because PostgreSQL releases the locks the moment the session ends, an owning pod that crashes hands its shards back without waiting for a timer, and a pod that shuts down gracefully unregisters itself first. The shard count under durable.sharding sets how much work a pod gives up when it does lose its locks, so it is sized to the owning replica counts rather than left at the engine default, and it must be identical on every DAPI role because it is the modulus that maps work to a shard.

The internal runner port must be reachable among DAPI role pods. PostgreSQL remains the durable recovery path when direct delivery is interrupted. The cluster-wide shard count must match on every role.

Bun workloads use JavaScriptCore rather than V8. The charts therefore rely on each workload's Kubernetes resources.limits.memory and do not inject Node-only heap flags. Keep memory limits on every service and migration job so an out-of-budget process is contained by its own cgroup.

Database paths

Every DALP service connects directly to PostgreSQL. The DAPI roles use two connection classes:

1. Request and repository queries use each role pod's bounded application pool.
2. Owning runner and indexer roles use dedicated durable connections, and every role uses direct notification and health sessions where required. Advisory ownership and LISTEN cannot survive a connection swap.

The chart's connection budget gate prices API, runner, and indexer pods separately, then applies each role's replica ceiling, rollout surge, and terminating-pod overlap. It also prices the block explorer's eager pools when enabled and reserves capacity for migrations and administration. Run the gate before increasing any role replica count, pool cap, or rollout overlap.

Single-generation Deployments and Helm upgrades

Some workloads must never run two pod generations at once. The block explorer opens its database pools eagerly on boot, so a rolling surge would double that demand during every platform upgrade. The PostgreSQL metrics exporter is a single-replica scrape surface (the chart and budget gate reject replicaCount above 1). Bundled PostgreSQL, Redis with persistence, and standalone object storage use Recreate so a second writer cannot attach the same volume. Recreate answers generation overlap only; multi-replica capacity still needs an explicit budget term or a fail-closed replica ceiling. The connection budget gate prices those single-generation workloads once; keeping Recreate is what makes that price honest.

Helm upgrades keep Recreate on those single-generation Deployments. When a cluster still holds an older rolling strategy beside Recreate, the upgrade clears that legacy strategy state while preserving single-generation rollout. For the block explorer, a pre-upgrade job performs that clear against the live workload before the chart applies the Recreate strategy.

That job needs to call the cluster API, and it authenticates with the projected service account credentials the platform grants it. If those credentials are unavailable, the job cannot reach the API, it ends failed, and the upgrade stops before the block explorer is changed. Read that outcome as a blocked upgrade, not as a step that was safely skipped: the legacy rolling parameters remain on the live workload, so the next upgrade attempt fails the same admission check until the cause is fixed. The platform keeps the credential grant on the job's own service account rather than on its pod, because cluster policy that requires pods to opt out of API access explicitly will reject a pod that opts in. Do not switch those Deployments to RollingUpdate to silence an upgrade failure: that would reopen the connection-budget and volume-attach questions the gate already answers.

Platform API and Console Deployments default to RollingUpdate for zero-downtime upgrades. If you set their update strategy to Recreate, the chart ships a Recreate-only strategy so leftover rolling parameters cannot fail admission. The connection budget gate prices Recreate as maxSurge: 0 so a leftover surge default cannot reject a valid Recreate topology.

Treat per-pod worker counts, including indexer.backfill.workersPerChain, as concurrency hints and never as the safety bound. The indexer enforces its own pool-wide limit inside the claim table, so adding indexer replicas cannot widen backfill's total claim concurrency; heavy claim activity on one pod shows up as saturation on that pod's own pool gauges.

Routing is the platform chart's decision for every service in the release; no individual service opts in or out on its own. Confirm after each install that the backend connection URL names the PostgreSQL host with the datastore's configured TLS mode.

Observing the connection path

Two observation points own the evidence for this boundary, because application traces cannot supply it. A backend request blocked waiting for a pool connection produces a span that looks exactly like a slow query, since the statement has not reached PostgreSQL yet. Distinguishing the two requires reading the pool and the server directly.

Each DAPI role pod samples its own pool: connection usage by state and requests pending a connection, labelled by pool name and role identity. Saturation is a per-pod fact, so dashboards aggregate the gauges with a sum over the pool label, and sustained pending requests on one pod mean that pod's pool is the bottleneck, not the database.

The PostgreSQL exporter is a separate workload that reads the managed server's catalog views for backend counts against max_connections. It holds a direct session. It ships disabled: every session it opens is charged against the shared connection ceiling through the non-DALP-backend reserve, so enabling it is a connection budget decision that the chart budget check must re-validate. The chart and budget gate fail closed above one replica: Recreate only prevents two generations overlapping, and does not price replicaCount * maxConnections. Keep replicaCount at 1. When the exporter is enabled, the gate also compares replicaCount × maxConnections to postgresConnectionBudget.nonDapiReserve so an oversized maxConnections cannot pass while exceeding the entire reserve. Raise maxConnections only after sizing that reserve and re-running the gate.

Operationally, treat pending pool requests above zero as backend queries that PostgreSQL never received, and revisit the pool cap and replica ceiling through the budget gate rather than tuning queries first. On clusters that source credentials from an external secret manager, the exporter requires an existing Secret because the chart renders no plaintext credentials there.

Telemetry paths

A single-cluster deployment collects and stores its own metrics, logs, traces, and profiles. A deployment that spans clusters does not: only one cluster runs the storage backends, and every other cluster runs a collector that forwards to it through an authenticated OTLP receiver published on the ingress. Each signal and destination has an independent queue, so a central outage cannot block the local backend and a local outage cannot be reported as central receipt. That forwarding hop is the only place where telemetry crosses a cluster boundary, and it is where signal identity is either preserved or lost.

Workloads export complete traces. Tail sampling belongs after identity and safety processing at a trace-affine collector, where error, measured-slow, and critical-operation policies can retain the entire span tree. Node profiles use the supported profiler and the same bounded deployment and service identity. Both features stay disabled until enabled-versus-disabled resource evidence meets the deployment's approved budget; receiver readiness alone does not authorize either one.

Identity travels as OTLP resource attributes. Loki promotes resource attributes to stream index labels, so the originating cluster, namespace, pod, and container must be set there before the record leaves the sending cluster. The same values carried alongside the record instead of on its resource are still stored and still searchable, but they no longer define a stream, so the logs cannot be selected by cluster or by pod and are reachable only by scanning text.

The receiving collector enriches in-cluster telemetry with Kubernetes metadata by looking up the pod that owns the connection's source address. That lookup is correct only for senders inside the same cluster. Forwarded telemetry arrives through the ingress, so the address it presents belongs to the gateway proxy, and the same enrichment would relabel every remote record with the proxy's identity. The receiver that accepts remote traffic therefore performs no such enrichment and treats the sender as authoritative.

Three operating consequences follow:

1. Telemetry that arrives under the receiving cluster's own identity is a collector configuration fault, not a sender fault. The sender's export counters read healthy because delivery did succeed, so restarting it changes nothing.
2. Per-pod stream identity keeps a collector's startup backlog acceptable. A collector reads each container from its start time, and Loki rejects entries that fall too far behind the head of the stream they land in. Correct per-pod labels give a new pod an empty stream with no head to fall behind; incorrect ones collapse every pod onto one shared stream and its replayed history is discarded.
3. Rejected entries are counted where they are discarded. Treat the storage-side discard counters, which name a reason, as the measure of telemetry loss; the sending collector's failure counter charges an entire batch when only part of it was refused, and overstates it.

Verify after connecting a cluster that its logs and metrics carry that cluster's own name as a label, not the receiving cluster's, and that a pod identifier resolves to a real workload in the sending cluster.

The forwarding hop is authenticated, and the collector reads both the receiver credential and the sending credential from environment variables backed by Kubernetes Secrets. The chart creates no Secret and writes no credential into its ConfigMap. Each cluster supplies its own: one Secret for the receiver on the cluster that stores, one for forwarding on each cluster that sends, each holding a single password key. Both are optional, so a cluster that enables neither receiver nor forwarding authentication does not need them.

A receiver without its credential fails closed. It does not fall back to accepting traffic, and it does not accept an empty password, so a cluster that enables the receiver but never supplies the Secret rejects every forwarder rather than admitting any. Read a receiving cluster that answers 401 invalid credentials for a sender you believe is configured as a missing or misnamed Secret on the receiving side, not as a bad password on the sending side.

Two operating consequences follow. A cluster that sets a username without supplying the matching Secret still starts and still reads as fully configured on both sides, so the 401 is the only signal that anything is wrong. Rotating either credential needs an Alloy pod restart, because the config reloader watches only the ConfigMap and an environment variable is fixed for the life of a container. Plan a credential change as a rollout on the sending and the receiving cluster together.

Alerts about pod terminations are evaluated against the same central store, and an out-of-memory alert is bounded by the termination's own timestamp. The alert therefore carries a time claim you can act on: the termination it names occurred inside the rule's recent window, so it remains accurate while platform components restart around it.

That bound asserts recency, not current pod existence. A controller can replace an out-of-memory pod inside the same window, so a correct alert may name a pod that is already gone. Read the alert by its owning workload, the replacement pod, and the termination timestamp rather than by whether the named pod is still present.

Inside a cluster, the export hop crosses a namespace boundary. The collector belongs to the observability release, which installs in its own namespace, while the application workloads run in theirs. Where NetworkPolicy is enforced, each application policy must admit that specific namespace, so the platform charts build the collector peer from global.observability.namespace. A deployment that runs the collector elsewhere sets that value, and the same value selects the query endpoints the platform reads back.

Two peer shapes fail in opposite directions, and both are silent. A peer that carries only a pod selector is scoped to the policy's own namespace, so it matches no collector and the workload exports nothing. A peer that matches every namespace admits any pod that carries the collector's name label, so a co-tenant can present itself as a collector.

The operating consequence is the same in both directions: in the shipped configuration the application containers reach the log store only through this export path, because the collector does not also scrape their console output. A blocked or misdirected egress therefore removes logs, metrics, and traces together, while the pod stays ready and keeps writing records to its console. Read that state as a loss of telemetry, not as a quiet service. The log-ingest absence alert is the signal that reports it.

One deliberate exception matters during an incident. Restoring the console scrape for those containers, through alloy.dalp.logIngest.dropScrapedAppLogs, brings their logs back while the export path stays blocked. Logs then arrive without metrics or traces, and they arrive degraded, because the collector infers severity from the text rather than reading it from the record. Treat that combination as a recovery posture, and as evidence the export path is still broken, not as the deployment returning to health.

Grafana reads only the native V2 dashboard estate provisioned by the chart. The Start Here dashboard, folder navigation, alert links, and legacy-resource deletion list are reconciled as one manifest. Rules remain shadow-owned until navigation and four-backend readback pass; notification ownership then changes separately so exactly one ruleset can page. A rollback restores the prior chart release and continues querying the preserved Loki, metrics, Tempo, and Pyroscope storage.

EVM and custody paths

Transaction execution and event ingestion share the configured EVM network identity but have different traffic patterns.

A public-chain deployment can use approved managed RPC endpoints. A private-chain deployment must also assign ownership for validators, membership changes, node recovery, block history, and archive access.

Provider-result recovery depends on both the custody and RPC paths. The backend uses a stable operation identifier to reconcile a provider-confirmed write whose hash response was lost, and persists that hash before treating the write as successful. A provider rejection can also trigger a bounded call replay through the configured RPC path to recover contract-revert evidence. Operators should monitor provider-status latency, rejected broadcasts, hash write-back failures, and replay timeouts together; loss of either dependency keeps the original failure visible rather than authorising a duplicate submission.

The EntryPoint address differs by chain, and the backend resolves it

Sponsored transactions need an ERC-4337 EntryPoint. Which contract fills that role is a deployment fact, not a constant, and it splits by chain type.

On a private chain the platform deploys its own EntryPoint and the on-chain directory records that address. On a public chain the standard EntryPoint is already deployed at a fixed address, so the platform deploys a small reference contract that points at the standard one and records the reference instead. The directory needs an address it can introspect, and the standard contract does not answer that introspection.

The backend resolves the recorded address before it uses it. Resolution is one read-only call on the broadcast RPC path, and the result is cached per chain and per recorded address, so it does not add a call per request. A resolved reference is cached for the life of the process, because the pointer cannot change. A pass through, meaning a recorded address that is not a reference, is cached for a short period so that a wrong answer cannot become permanent.

Two operator consequences follow. First, this resolution depends on the broadcast RPC path: while that path is unavailable, sponsored operations report a retryable error rather than proceeding with an unverified address. Second, paymasters record the EntryPoint they were deployed against, so a fleet deployed across a resolution change holds two different values. Both work. The difference is deploy history, not drift.

The served network set is deploy-time plus runtime

The networks a deployment serves are not fixed by the chart alone. Two writers feed one catalog: the values baseline shipped in the deployed configuration, and a database registry a platform administrator writes at runtime through the Platform API. Endpoints follow the same pattern in a sibling table. The registry is an overlay merged onto the baseline by name, so an entry that shadows a configured network can be removed to fall back to the configured definition, while an entry for a network the values never declared is removed outright.

Operators should read four consequences from that split.

The admin list reports each network's origin, configured or registered, so a drift review can tell them apart without reading the database. A network that still reads as configured while a registry entry exists for it is the platform reporting a rejected entry, not a stale display: the merge dropped the entry and the baseline is still serving.

Serialization is a single catalog-wide transaction-scoped advisory lock. The transaction scope binds the lock to whichever pooled connection runs the write, so it holds on the direct application pool and under any future pooler. Two administrators writing at once are ordered rather than raced: the second re-validates against what the first committed and either succeeds or receives the same typed refusal it would have received alone.

Indexing scope is written by the backend, not by the index

Which systems the Ledger Index follows is held in a platform ledger in the shared database, outside the index deployment schema. The DAPI roles share the repository for that ledger with the other domain repositories, so every writer reaches it over the ordinary pooled database path. The index itself only reads the flag and acts on it.

Three writers exist, and they have different owners.

Operators should read two consequences from that placement. The ledger survives a reindex, because a reindex rebuilds the index deployment schema and the ledger is not in it, so no operator decision is lost when the index version moves. And a scope change needs no rollout: the writing replica commits to the shared database, and the index applies the change on its next pass over that chain.

A network entry names one Directory, and the chain may hold several

contracts.directory on a network entry is the address the environment resolves everything else through. It is a single-valued decision on purpose: a chain can hold more than one Directory generation at a time, and the index records each of them, so an environment that does not name exactly one has no defined answer to "which contracts are mine".

Two ownership lines follow, and both are enforced rather than documented.

An entry that names no Directory does not borrow one. Earlier behaviour scanned sibling entries on the same chain and used the first Directory it found, which made an unrelated network entry a silent input to the environment's contract resolution. Absence is now absence, and resolution fails where it is required instead of succeeding against someone else's world. Two enabled entries on one chain naming different Directory addresses is a configuration defect and fails naming both, rather than one of them winning by ordering.

Changing that address to a Directory at a new address is a data migration, not a configuration edit: every system, registry, and identity already deployed resolves through the old address, and the config change moves none of them. The in-place proxy upgrade is the normal path and needs none of this. A re-point needs a staged cutover with reader and writer quiescence and a convergence check before write traffic resumes. Follow the directory re-point runbook.

Operationally, a chain serving more than one indexed Directory generation is alertable in its own right, so the cutover itself is visible rather than only its consequences. The second alert on the same fault line fires when an identity is registered that the platform cannot attribute to any indexed factory, which is what the two-generation state eventually produces. Both are noDataState: OK while the emitting metric is still reaching an install; a monitored environment should flip them once the series is confirmed exporting, or the quiet reads as healthy.

Readiness and shutdown

Readiness is role-local, not HTTP alone. The API role opens the cluster client, binds the serving read schema, and only then accepts traffic. The runner role registers workflow and entity definitions before it can own the default or workflow shard groups. The indexer role registers its chain capabilities before it owns the indexer shard group, then initializes the index deployment and seeds progress. Registration order is load bearing: an owning role joins its work partition as soon as the runner opens, so later registration would create a window in which it holds work it cannot serve. Lengthening the startup budget does not substitute for that order. Schema reconciliation and reindex decisions use transaction-scoped advisory locks, so concurrent indexer replicas wait for one owner and then inspect its committed result. PostgreSQL or role dependency failure produces a structured, non-cacheable 503 from /readyz rather than a generic application error. Staging boot and scale settings stabilize the topology and do not replace its connection budget.

During shutdown a role pod first becomes not ready. An API pod drains HTTP; a runner or indexer pod releases only the ownership and listeners for its role. Every role then flushes telemetry and closes its resources within the termination budget. Another healthy owning replica can acquire released work; stable cycle keys ensure a handoff joins already-started durable work. The deployment grace period must cover those phases.

Failure domains

Do not use a reindex as a generic recovery response. First distinguish chain execution, RPC health, index ownership, checkpoint progress, handler failures, and active reindex state.

Recovery posture

Set recovery targets per failure domain. One platform-wide target hides the dependencies that determine actual recovery.

Chain state can rebuild supported indexed projections after an indexer outage, subject to retained RPC history and processing time. It cannot rebuild off-chain application, approval, identity-provider, custody, or document records. Back up those data classes according to their owner and retention policy.

Alert evaluation locality

Alert evaluation is a separate placement decision from workload placement. Each cluster's collector forwards metrics to a central metric store, and rules evaluate wherever the dashboard layer reads that store. A cluster can be alerted on without hosting any part of the observability stack, and the observability install can evaluate rules for clusters it does not otherwise manage.

Two consequences follow for environment planning.

Cluster identity travels as a metric label, not as a deployment boundary. A rule that groups results on a label its metrics do not carry will merge distinct clusters into one result, so a fault on one can be masked by healthy activity on another. Treat the grouping keys of a rule as part of its correctness when you review a ruleset.

Silence is not health. Rules resolve to a healthy state when no matching series exist, which is correct for a rule that simply does not apply to an install, but it means a cluster that stops reporting suppresses its own alerts instead of raising them. Name every cluster expected to report into an install, so the platform can alert on the absence. A cluster left off that list is unmonitored in a way that looks identical to a cluster that is fine.

A rule that alerts on absence reads an empty result in the opposite way, and its no-data setting must follow the query. An absence query returns a result only when the stream it names has stopped. An empty result is therefore the healthy answer for that rule, and the rule must treat no data as healthy. A rule that alerts on absence but treats no data as undetermined inverts its no-data signal. It then reports a data problem on every healthy evaluation, so it sends a no-data notification continuously while ingest works. In the one case the rule exists to catch, that no-data notification stops, because the query returns a result. The absence alert itself is not affected and still fires at that moment. Only the no-data notification is inverted, and it is the surface that misleads. Read a continuous no-data notification from an absence rule as a fault in the rule, not as a fault in the data source. A data source that truly fails is reported through the separate execution-error state.

An absence check written for the whole fleet removes cluster identity from the rule altogether, which is the sharpest form of the masking described above. Such a rule reports only when nothing anywhere in the store matches it, so a single cluster that still reports keeps it silent no matter what the other clusters have lost, and when it does report it cannot say which cluster is affected. Use it only where one named producer is expected and its silence is itself the incident. Where the thing being watched exists separately on each cluster, and can therefore be removed from one of them, compare the two sets per cluster instead: the clusters reporting the subsystem at all, less the clusters reporting the specific object. That form names the affected cluster in the alert, and it also fixes what the rule ignores, because a cluster whose collector has stopped disappears from both sets at once and cannot raise the alert. Deciding to route a collector or metrics outage to its own rule is part of designing the guard, not a detail left to the alerting layer.

Choose the baseline set with care, because it must outlive whatever the rule watches. A baseline drawn from the same source as the thing being checked disappears together with it, so the most complete failure produces the quietest alert, which is the opposite of what a guard is for. Take the baseline from the layer beneath instead, and give that layer its own rule with a baseline beneath it again. The lowest rule in such a chain must rest on a signal that a different component reports, so that the subsystem cannot mask its own absence.

A baseline must also hold steady when the system is idle. Prefer a measure that exists because a component is running, such as a liveness or version gauge, over a counter that exists only because the component was used. A request counter reports nothing until the first request arrives, so a healthy but quiet component looks exactly like a missing one, and every rule built on it goes quiet at the same time.

Rules whose metrics exist on only one cluster stay disabled by default and are switched on per install. Without that gate, every other environment carries a rule that can never fire, which erodes the signal value of the alert list.

Network responsibility lines

The telemetry connection carries logs in every environment. The backend writes each log record to two independent sinks: the OTLP exporter that feeds the collector, and the pod's own console output. Choosing one has never been an option and is no longer expressible in configuration, so an environment cannot be quiet on the collector because of how its console reads. Whether the exporter runs at all is the single telemetry.logs.exporter decision, and the minimum severity in telemetry.logs.level applies to both sinks together. Owners planning a log budget should size the collector for the full record stream of every environment that points at it, not only for production.

Client-address trust is also environment-specific. The configured edge profile accepts only x-envoy-external-address from Envoy, the NGINX-replaced x-forwarded-for value, the terminal hop appended by OpenShift Router, or no forwarding signal for none. The Console's server-side proxy removes every inbound forwarding variant and passes only that profile-owned signal; DAPI converts it to the sole internal x-real-ip value at the authentication boundary. The Envoy policy derives its signal from the preserved downstream TCP source. Network policies admit only the selected edge-controller namespace to the Console and only the Console plus that namespace to DAPI.

Cluster DNS is the one path those policies do not restrict by destination. Every policy that enforces egress carries a port-53 rule that opens UDP and TCP 53 and names no destination at all. A policy that admits only ingress restricts no outbound traffic, so it carries no DNS rule and needs none. The reason is that some clusters run a node-level DNS cache, such as GKE NodeLocal DNSCache. That cache answers pod lookups from a host-network daemon without rewriting the packet, so the packet still carries the cluster DNS service address and belongs to no pod and no namespace. A rule that names a pod, a namespace, or an IP range cannot match it, and the network plugin then drops every lookup the workload makes. A destination-scoped DNS rule therefore does not tighten this path; it only makes the platform fail on the clusters that cache DNS locally.

Two consequences follow for a reviewer. The charts own this rule, so it is not an environment value and no overlay can narrow it. Do not expect a per-cluster DNS destination to configure. And the port itself stays the control: only 53 is open, and every other egress rule keeps naming what it may reach.

Environment model

Environment promotion moves code, configuration intent, and verified procedures. It does not move test chain state, test keys, indexer checkpoints, or test identity data into production.

Deployment identity is declared, never defaulted

Each installation carries two bounded coordinates: a stable slug in global.dalpInstance and one of the environments above in global.deploymentEnvironment. Together they are the deployment identity stamped on every metric, log, trace, and profile the installation exports.

The Observability chart requires both and ships no default for either. It refuses to render when one is missing, and it refuses the reserved local slug outside the local environment. The identity is therefore a deploy-time decision that belongs with the values, not a value the platform infers. It is never derived from a cluster, namespace, pod, or process name, because those are ephemeral and several of them can belong to one deployment.

This boundary exists because the failure it prevents is silent. Deployment identity is the dimension the converged store groups by, so two installations that share a slug become one identity after their signals arrive: instance selectors offer a single entry, and alerts group unrelated deployments together. Every signal still flows and every export counter still reads healthy, so the loss surfaces only when an operator notices the estate looks smaller than it is. A refused render moves that failure to install time, where it costs a values edit.

The ownership line follows the same split as the rest of the chart. Whoever supplies the values owns the identity, and it must be unique across every installation that reports into the same Observability store, including installations on other clusters and other release lines.

Architecture decisions to record

Before deployment approval, record these decisions and their owners:

Limits

Where to go next

• Architecture one-pager for the eight product systems.
• Workflow Engine for durable execution and keyed ownership.
• Ledger Index for checkpoints, reorgs, and reindexing.
• Database for PostgreSQL pools, TLS, backup, and restore.
• Observability for logs, metrics, traces, and incident evidence.
• High availability for recovery patterns and topology choices.
• Self-hosting prerequisites for installation inputs.