Ledger indexing stalled

Ledger DataLedger Data lets an integration answer a holder list, a transfer history, or a compliance check with one query, instead of decoding chain events itself.Read the concept is the platform's own record of the chains a DALP InstanceA DALP Instance is one self-contained copy of the platform, deployed inside a boundary you choose and connected to ledger networks you approve.Read the concept follows. It reads each block once, decodes the events into business records, and Console, the API, and reports answer from that record rather than from the live network. When indexing stops advancing on a chain, those surfaces freeze at the last indexed block while the network moves on. This runbook is for that condition, on one chain or several, whether or not anything else looks unhealthy.

The whole diagnosis hangs on one distinction. A chain whose data is behind leaves one of three signatures: an indexed position that is still advancing, a position that stopped moving, or no position at all. An advancing position indicates progress, a stopped position points at an indexer pod that failed on that chain's ingest, and an absent position calls for a startup and ownership check. The checks below separate them in order, cheapest first.

This page assumes the indexer pod itself is running and passing its probes. For a pod that Kubernetes holds out of Ready, start with Platform API not ready instead; it covers the readiness legs, and its indexer leg hands back here.

SymptomsCopy Anchor Link

• Console screens, API responses, or reports stop reflecting new on-chain activity while the underlying network keeps producing blocks.
• The Console's Platform status screen shows a chain out of sync in its Indexing section, a growing worst sync-lag, or a degraded data-freshness panel.
• The gap between the chain's head and the indexed position grows instead of shrinking.
• The indexer pod may still report Ready, and that is expected: its probes answer for the process, not for chain state, so a chain that is merely lagging leaves the pod green. A chain whose ingest fails does not leave it green, because the process exits on the failure and Kubernetes restarts it (Check 2). A green pod with a frozen position therefore means an ingest that is running but waiting, not an ingest that stopped.

ChecksCopy Anchor Link

1. Confirm the chain is frozen, not catching up. The signal is the indexed position over time: the Console's Platform status screen shows chains in sync and the worst sync-lag in blocks, and the indexer exports the same lag per chain as the dalp.didx.watermark_lag_blocks gauge. Healthy catch-up means the indexed position advances and the lag shrinks. A constant lag can mean ingestion matches block production but is not catching up.

   The platform's own monitor marks a chain's indexing degraded once it is 10 or more blocks behind or has gone 5 minutes without progress, and critical past 100 blocks or 15 minutes.

   Two shapes look alarming and are not. A rebuild after a platform upgrade reports as reindexing rather than degraded; that is the indexer working, so let it drain. A chain that is fully caught up on a quiet network is idle, not stalled: the stall clock only runs while the chain is behind. Chains that mint blocks only when a transaction is submitted skip the no-progress rule entirely.

2. On a chain whose position stopped moving, read the indexer pods' restart counts and their crashed containers' logs. An indexer pod whose chain ingest fails exits and is restarted by Kubernetes: a failure the process cannot continue past ends it, and the container log ends with exactly one exit line naming the chain and the failure.

   That line is the searchable record, and it is the finding: filter the indexer pods' logs on the didx.host category's outcome annotation or on the ; exiting: suffix every exit line carries. Two of the verdicts keep their own richer text, Handler failure; exiting: and Batch refused; exiting: ; the rest read Chain <verdict>; exiting: <failure>, for example Chain Resume; exiting: upstream refused the range. Each line carries the chain's chain_id.

   A position that stopped moving and a pod that restarted at the same moment are one event, not two. The restart replans from the last committed watermark, so a failure whose cause is already fixed clears itself on the next restart, and one that is still present keeps the pod crash-looping on the same line. A pod that restarted once and settled was transient; a pod whose restart count keeps climbing is not.

3. For a chain with no indexed position at all from the moment the instance started, and no restarting indexer pod, check the ownership claim rather than waiting for catch-up. Each chain's ingest claims exclusive ownership of that chain before its first write. A claim the database refuses for a missing schema object or a revoked permission is permanent.

   The process logs a warning at startup, categorized didx.chain-ownership and carrying the chain's chain_id, network, and terminal: true, saying the ownership claim failed permanently and the chain will not start. Not advancing, not stalled, only absent, on every replica.

   Operators can mistake this for slow catch-up, because both look like "no data yet". The test is whether a position exists at all: a position that never existed is this check, a position that moved and stopped is the check above.

4. For positions that froze or flapped around a redeploy, a replica restart, or a replica-count change, look at the ownership handoff. Ownership of each chain is fenced with a monotonically increasing epoch: the later claimant always wins, and a superseded owner's position write is rejected as a conflict rather than silently accepted. A healthy handoff therefore shows the departing owner's writes failing briefly while the new owner continues from the stored cursor; owners keep no state of their own, so nothing is lost in the transfer.

   Each pod carries only the chains it currently owns, so a redeploy or a replica-count change moves chains between pods and a chain can legitimately have no owner on the pod you happened to read. Check every replica before treating a frozen chain as one nobody is working on.

   Inside the managed deployment only one process can own a chain at a time, which means check for a process outside the deployment using the same database, for example an operator's ad hoc instance. Such a process claims a newer epoch, wins the fence, and starves the managed owner while reporting itself healthy.

ActionsCopy Anchor Link

• If the position is behind but advancing, this is backlog, not a stall: monitor rather than intervene. Divide the lag in blocks (dalp.didx.watermark_lag_blocks) by the observed rate at which that lag decreases to size the window it should close in; revisit your RPC provider limits or the chain's scope if it is not closing within that window. Do not add indexer replicas to speed one chain up: each chain's live ingest is a fenced singleton owned by one process, so extra replicas spread chains across pods and buy failover, never per-chain throughput.
• If the chain's position stopped moving and the pod restarted, read the exit line for that chain's id: the failure it names decides the fix. A failure the database refused (a missing table or column, a constraint violation, a revoked permission) is fixed in the database, and the refused transaction rolled back whole, so nothing already indexed is lost and the checkpoint never moved. A failure naming the chain's configuration is fixed in the instance's networks entry and needs a rollout rather than a restart.
• If an RPC provider is unavailable or rate-limiting, the chain's ingest fails on it and the pod restarts until the provider recovers. Check the provider's status and your configured endpoint first: it is quicker to rule out than duplicate ownership. The chain needs nothing from you while you investigate; its next restart replans from the last committed watermark.
• If the failure is a handler failure, escalate as a platform defect with the held block, the event name, and the exit line's text; do not keep restarting the pod for it. Nothing already indexed is lost while the chain is stopped: once the release carrying the fix deploys, the pod's restart resumes from the held checkpoint on its own. A release whose fix changes what earlier data should have looked like declares its own rebuild, which reports as reindexing while it drains.
• If a configured chain never claimed ownership, treat the startup claim warning as the finding: it means the database refused the claim for a reason no retry fixes, such as a missing schema object or a revoked permission. Fix that refusal, then restart the indexer deployment; the claim does not retry a permanent refusal on its own.
• If a process outside the managed deployment holds chain ownership, stop that process. The managed owner cannot win it back by capacity: the fence always favors the most recent claimant, so as long as the outside process keeps claiming, the two trade ownership and neither makes steady progress.
• Escalate with the chain id, the exit line or the startup claim warning, the last position that advanced, and which of the three signatures you observed. The signature is the fastest path to root cause, because each one points at a different subsystem.

Frequently asked questionsCopy Anchor Link