Platform API not ready

Kubernetes routes traffic only to pods that report Ready. A pod that is Running but not Ready is alive and deliberately out of rotation: its Service sends it nothing, so Console screens and API calls fail while kubectl get pods looks healthy. This runbook covers that state on the three backend roles of a DALP instance: the Platform API, the Ledger Data indexer, and the Workflow Engine. The Workflow Engine carries multi-step operations across crashes and restarts instead of running them as one call that has to finish in one pass.

kubectl get pods does not print those role names. The chart labels each role's Deployment with app.kubernetes.io/component, so find the pod a check below names by that label:

• Platform API: kubectl get pods -l app.kubernetes.io/component=api
• Indexer: kubectl get pods -l app.kubernetes.io/component=indexer
• Workflow Engine: kubectl get pods -l app.kubernetes.io/component=runner

The pod publishes its own diagnosis, but the contract depends on its role. Platform API and Workflow Engine pods can report boot phases or dependency verdicts on /readyz. An indexer pod answers /healthz and /readyz with 200 while its process runs; its chain state is not a readiness verdict. A non-reorg chain ingest failure logs one Chain <verdict>; exiting: <failure> error in category didx.host, with chain_id and outcome annotations, then exits with status 1. Kubernetes restarts the pod. Read the indexer pod's restart count and that log line. A pod held out by chain indexing is therefore an indexing problem; Ledger indexing stalled owns that diagnosis.

SymptomsCopy Anchor Link

• kubectl get pods shows a Platform API, indexer, or Workflow Engine pod Running but not 1/1 Ready.
• A Helm install or upgrade hangs waiting for the rollout, then fails with a progress deadline error.
• Console or API requests return connection errors or 503 responses while every pod shows Running.

ChecksCopy Anchor Link

1. Rule out a crash loop first. Signal: the RESTARTS column of kubectl get pods. Healthy value: a count that stays put. The process polices its own boot. A lightweight boot shell binds the HTTP port at start and answers the startup probe within milliseconds, so the probe never kills a slow boot. Instead the process enforces internal deadlines: up to 10 minutes to build its application graph, then 150 seconds for the two gated steps after the build. The first step binds the servable schema, the current version of the indexed database tables a Platform API or indexer pod reads and writes through. The second catches up any outstanding watermark lag, the gap between the chain blocks already applied to that schema and the chain's real head. A pod that exceeds either deadline retries in process, then exits so the failure becomes a visible restart. A climbing count therefore means a boot failure: read kubectl logs --previous <pod> for the exit, and stop investigating readiness until the pod stops crash looping.

2. Read the pod's own answer. Signal: the JSON body of /readyz on the pod's HTTP port, reached with kubectl port-forward or a curl from inside the cluster. For Platform API and Workflow Engine pods, the body routes you to the remaining checks: draining is check 3, starting with a reason field is check 4, and degraded with per-dependency fields is check 5. An indexer pod is different: /readyz returns 200 while the process runs, so read its restart count and exit log for chain failures.

3. Decide whether the pod is arriving or leaving. Signal: the pod's age next to a draining status. A pod that is shutting down flips not-Ready on purpose and skips its dependency checks rather than racing a closing connection pool; every rolling restart produces this state briefly. Platform API rollouts start one new pod before taking any old one away by default, so the old replicas keep serving while the new one boots. Indexer rollouts replace one pod at a time instead, holding each replacement 90 seconds past Ready before moving to the next. Workflow Engine rollouts also replace one pod at a time, but without that held pause: the next replacement starts as soon as the previous one reports Ready. Healthy value: at most one replica in transition at a time. Investigate when rollout progress stops or capacity falls below the deployment's needs. Losing every Ready replica is a complete serving outage.

4. Place a starting pod in its boot phase. Signal: the reason field beside status: "starting". Three phases can appear, and each has a different clock. application-building is the graph build, bounded at 10 minutes; if it wedges, the pod restarts itself, which sends you back to check 1. serving-schema-unbound covers two waits behind one name. It first waits, indefinitely by design, for a servable schema to appear. Producing that schema, including on a fresh install, is background work the indexer pod does inside its own process; it is not a separate Kubernetes Job, so kubectl get jobs finds nothing for it. There is no deadline on this part, and a pod can sit here for a long stretch and still be healthy while that background work runs. Once a schema is published, the pod moves on to bind it, and that bind step shares the 150-second budget with the watermark catch-up (catchup-pending); if it wedges, the pod restarts itself, sending you back to check 1. Healthy value: the phase advances and reaches ready within minutes of a schema appearing. A pod parked in serving-schema-unbound for a long stretch is almost always waiting on the indexer, not wedged. Read the indexer pod's own logs and readiness before treating this pod as stuck; see Actions for what to do when the indexer pod itself shows no progress.

5. Name the verdict holding a running Platform API or Workflow Engine pod out. Signal: the postgres and durable fields in a degraded body, each with a reason. These verdicts do not describe an indexer pod. An indexer pod answers /readyz with 200 while its process runs; a non-reorg chain ingest failure exits the process and Kubernetes restarts the pod.

   • postgres. The probe dials a dedicated client against a PgBouncer alias reserved for it, an isolated one-connection pool that application traffic never shares, so a saturated app pool cannot make a healthy database look down. A failing verdict names a real problem, but not always Postgres itself. The pooler in front of it is deliberately given a shorter rollout deadline than this pod's own, precisely so a stuck pooler fails first and shows up here rather than being mistaken for a Platform API problem. Check the pooler alongside the database.
   • durable. This is the work-scheduling cluster the Workflow Engine runs on. It lives inside the same pod and stores its state in the same Postgres, so there is no separate service to check; an unhealthy verdict means the cluster is shutting down or has not come up.

ActionsCopy Anchor Link

• If the restart count is climbing, fix the boot failure from kubectl logs --previous first. Readiness is downstream of a boot that completes.
• If the postgres verdict is failing, check both the database and the pooler in front of it. The probe reaches Postgres through PgBouncer, and the pooler's own rollout is tuned to fail before this pod's does, so a stuck pooler surfaces here first. Do not restart the pod. On a Platform API pod the liveness probe reads a heartbeat the main event loop stamps into shared memory every quarter second, watched by a separate thread with its own event loop. Only a heartbeat gone stale for 30 seconds, three liveness probe periods, calls that pod wedged. The indexer and Workflow Engine pods answer liveness from their own main loop instead, and four failed probes ten seconds apart call one of them wedged. Neither probe touches a dependency, so a database or pooler outage cannot trip either one. Short of a wedge, restarting only forces the reconnect the pod would already make on its own. The pod rejoins the Service once the database and pooler both answer again. The default probe settings pull a pod from rotation after about 45 seconds of sustained readiness failure (three failed probes, 15 seconds apart), and return it after a single successful probe, up to 15 seconds later.
• If the indexer pod restarts, read the one Chain <verdict>; exiting: <failure> error in its logs. It is categorized didx.host and carries chain_id and outcome. A non-reorg chain ingest failure exits with status 1, so Kubernetes restarts the pod. Follow Ledger indexing stalled. Restarting or resizing the Platform API deployment changes nothing about indexing.
• If a pod sits in serving-schema-unbound (check 4) with no schema ever appearing, the wait itself is not the problem: diagnose the indexer pod, whose own background schema work is what every other pod is waiting on. Run this same runbook against it (kubectl get pods -l app.kubernetes.io/component=indexer): read its restart count and logs. A restarted indexer explains the wait; restarting the waiting pod changes nothing, because the wait resumes identically on the next boot.
• If the durable verdict is failing while postgres is healthy, and the pod still sits not-Ready past every budget above, capture kubectl describe pod and its logs first, then delete the pod. A durable failure outside a drain you already expect (check 3) means the wedge case applies, and the wedged state does not survive the restart, so the evidence has to come first.
• If the same pod wedges again after a restart, escalate with the logs, the describe output, and which verdict was failing. A repeating wedge is a defect in the platform, not a transient condition, and the failing verdict is the fact the escalation needs most.

Frequently asked questionsCopy Anchor Link