Webhook deliveries failing

A webhook endpoint is a URL you register with a DALP instanceA DALP Instance is one self-contained copy of the platform, deployed inside a boundary you choose and connected to ledger networks you approve.Read the concept so the platform pushes a signed HTTP request to it whenever a relevant event happens, instead of your systems polling for changes. This runbook is for when a registered endpoint stops receiving those pushes, whether that is every event or only some. It covers outbound webhook deliveryWebhook Delivery retries a signed notification for three days and records whether a counterparty was told, so an integrator skips its own retry logic.Read the concept from the platform to your consumer; webhooks your instance receives from compliance providers are a different surface with their own management screens.

Every check reads from the endpoint's delivery history: one row per attempt, with its status, its failure reason, and the response your consumer returned. In Console, open Platform settings, then Webhooks, then the endpoint; the Deliveries tab lists every attempt, and clicking a row shows the failure class, the signed request headers, the response, and a ready-made repro command. The same history is available from the CLI as dalp get webhooks deliveries <id>, and over the webhook management endpoints.

Whatever the cause, the history shows one of three shapes: attempts that keep failing, attempts that have stopped entirely, or attempts that succeeded while your consumer disagrees. Each shape has its own check below.

SymptomsCopy Anchor Link

• A registered webhook endpoint stops receiving events it previously received.
• Deliveries to one endpoint fail while other endpoints on the same instance keep receiving events normally.
• The endpoint's status in Console changed to Disabled without anyone disabling it.
• Your consumer reports missing events for a time window, then receiving them all at once, or not at all.

ChecksCopy Anchor Link

The checks form a short decision tree. Read the delivery history first; its shape tells you which branch you are on.

1. Confirm whether the endpoint's failures are transient or sustained. The signal is the Deliveries tab: a healthy endpoint shows mostly delivered rows, with occasional failed attempts followed by a delivered one for the same event. A single failed attempt retries automatically on an exponential schedule that starts near 30 seconds and roughly doubles after each attempt, with random jitter so many endpoints failing at once do not retry in lockstep. Retrying continues until 3 days after the platform recorded the event, not 3 days after the first attempt. The next attempt after expiry records abandonment. An open breaker prevents attempts, so that log can appear after the event has already aged out (check 2). A handful of retries in the history is normal operation, not an incident. Not every failure retries, though. A delivery your consumer rejects with an ordinary 4xx response, other than a request timeout or a rate-limit signal, is permanent for that event; a fix made afterward recovers future events but never that one. Each attempt itself times out after 20 seconds if the consumer never responds.

2. Check whether the platform has stopped attempting deliveries entirely, not just failing them. The signal is the Circuit breaker card on the endpoint's Overview tab: Healthy means deliveries flow, Disabled means the breaker tripped and the platform is skipping deliveries to this endpoint rather than attempting them.

   • Tripping. An endpoint trips the breaker after 30 consecutive failed deliveries; missed receipt deadlines count toward that streak, manually sent test events never do.
   • Recovery. Recovery is automatic: the platform probes an open endpoint by re-sending its most recent live event, each probe at least a minute apart with 10 seconds to answer. A failed probe resets the success count, so a consumer that flaps between working and failing stays paused until it answers twice in a row. Two consecutive probe successes move the endpoint to Recovering, and live deliveries resume immediately; the next delivered event returns it to Healthy.
   • The Recovering trap. Entering Recovering does not reset the failure streak that tripped the breaker in the first place. One failed live delivery there reopens the breaker immediately, before a second probe or a delivered event ever gets the chance. A consumer that is only half fixed can loop through probe, Recovering, and Disabled without a single live delivery landing.

3. Verify the consumer's endpoint is reachable and answering correctly. The signal is the Failure reason on each row.

   • Reading the taxonomy. It separates the ways a request can die: the name never resolved, the TLS handshake failed, the connection never opened, the response never finished, or the consumer answered with an error status. The sixth value, an unparseable response, also includes the two policy rejections described below. Check the policy reason before assuming the consumer returned an invalid response; the request might not have reached it. The healthy value is a delivered row, which the platform records only for a 2xx response; accepting the connection is not enough.
   • Network failures. All of them retry on the normal schedule, so fixing the consumer is enough and no platform-side change is needed.
   • Platform policy. The platform checks endpoint address and HTTPS policy at registration and before each dispatch. A URL that fails those checks cannot be registered. Operator-configured exceptions are described below. The endpoint URL must resolve to a publicly routable address; a receiver on private address space fails that check until the operator adds its range to webhooks.allowedTargetCidrs in the deployment's Helm values, or the equivalent DALP_WEBHOOKS_ALLOWED_TARGET_CIDRS environment variable. The URL must also use https. Both rules share one exception: an origin the operator lists in DALP_WEBHOOKS_TRUSTED_DESTINATIONS, for a private-network install, is exempt from the https requirement and the public-address check alike, so an endpoint on that list can legitimately use http:// or a private address. Tenant-facing APIs cannot add an origin to that list; only an operator changing the deployment's configuration can.

4. If the integration verifies signed payloads, confirm the consumer holds the current signing secret. The signal is a run of terminal 4xx rows that started when a secret changed on either side. A consumer that rejects a signature usually does so with an ordinary 4xx, which the platform treats as permanent for that event. Fixing the secret recovers deliveries from that point forward but does not resurrect the events rejected while it was wrong; those need a replay, described in the next section. Two behaviors keep rotation itself from causing this. After a rotation the previous secret stays valid for 24 hours, and the platform signs with both secrets during the overlap. An event that waited in retry backoff is re-signed with a fresh timestamp before the next attempt, inside the Standard Webhooks tolerance window. A long network delay or clock mismatch can still exceed the receiver's timestamp tolerance. Check receiver time and the verified delivery headers.

5. If the endpoint requires counter-signed receipts, confirm the consumer responds inside its receipt window. That window is 30 seconds in every configuration reachable from Console, the CLI, or the API; nothing tenant-facing changes it. The signal is rows that reached the consumer but still count as failures because the receipt never arrived in time. A late receipt is treated like any other failed delivery and retried on the normal schedule. A receipt that fails its signature or hash check is different: that is permanent for the event, on the same footing as a terminal 4xx, because re-sending cannot make a forged or mismatched receipt valid.

ActionsCopy Anchor Link

• If the breaker tripped, fix the underlying reachability or response problem first. The platform resumes on its own once its probes confirm recovery: two consecutive successes, each at least a minute apart, so recovery typically takes close to two minutes after your fix, rarely less. When that wait matters, recover immediately instead: on the endpoint's Overview tab, the Circuit breaker card offers Reactivate manually after you type the endpoint's name, which re-enables the endpoint and closes the breaker at once. Use Send test event to confirm your fix either way; a test event delivers even while the breaker is open and never counts toward tripping or clearing it, so testing is free but does not by itself reopen anything.
• If the endpoint was down long enough that events aged past the 3-day retry window, or lost events to terminal failures, replay them. The webhook replay call re-queues either one event by its id or every matching event in a block range. Each replayed delivery is flagged as a replay in its envelope, so your consumer can tell it from a first delivery. Replay is refused while the endpoint is disabled or the breaker is open, so recover the endpoint first, then replay. No log entry marks an event abandoned while the breaker stays open and no attempt runs against it; that entry appears only once the next attempt evaluates the event, typically after recovery. Check the delivery history directly for events approaching the 3-day window rather than waiting for an alert.
• If signature verification was the failure, correct the consumer's secret, then replay. A rejected signature is permanent per event, so the events that failed while the secret was wrong stay failed. The same replay and retry calls cover both a window of them and a single delivery you want to re-queue on its own. Deliveries themselves are otherwise working, so no platform-side change is needed.
• If your consumer sees a duplicate event, treat it as normal at-least-once delivery rather than a defect. The platform journals delivery attempts, and a retry reuses the same event id as its original delivery throughout. Deduplicate retries on the webhook-id header, which carries that id on every attempt. A replay is different: it mints a fresh event id for each event it re-queues, so the webhook-id header on a replayed delivery will not match the original and deduplicating on it does not work. Use is_replay to recognize the replay, then use stable domain identifiers from the event payload to avoid applying the same outcome twice.
• Escalate with the endpoint id, the failure window, whether the breaker had tripped, and the failure reason shown in the delivery history. That last field distinguishes a consumer-side outage from a platform delivery defect faster than any retry count. A name that never resolved points at DNS or egress, a terminal 4xx points at the consumer's validation, and a run of timeouts points at the consumer's capacity.

Frequently asked questionsCopy Anchor Link