Observability
The OpenTelemetry pipeline a DALP instance exports by default: traces, metrics, and logs, and how the admin debug bundle auto-wires to an in-cluster stack.
DALP exports metrics, logs, and sampled traces over the OpenTelemetry Protocol (OTLP). Configure each signal's endpoint to use your collector or the bundled observability stack. The admin debug bundle reads stored logs and traces through a separate query path.
Every deployment declares a stable dalpInstance slug and deploymentEnvironment value, stamped on every signal it exports, so a central view can group a whole fleet by instance identity. Each Platform API role process owns one observability instance, with a separate exporter for each signal, so replica identity remains a reliable grouping key.
What exports by default
| Signal | Enabled by default | Config key | Notes |
|---|---|---|---|
| Traces | Yes | telemetry.traces.otlp.endpoint | Batches export every 5 seconds (telemetry.traces.exportIntervalMs) by default. telemetry.traces.sampleRatio defaults to 0.1 (10%), a source-side safety guard against overloading your backend with unsampled volume. Raise it toward 1 (100%) only behind a measured Alloy tail sampler. |
| Metrics | Yes | telemetry.metrics.otlp.endpoint | Pushed every 60 seconds (telemetry.metrics.exportIntervalMs) by default. |
| Logs | Yes | telemetry.logs.otlp.endpoint | info level by default. The console sink renders one logfmt line per record in kubectl logs whenever telemetry.logs.enabled stays true. Set telemetry.logs.exporter to "none" to stop OTLP export alone and keep kubectl logs output; set telemetry.logs.enabled to false to stop both. |
All three default to the same in-cluster Alloy OTLP receiver at obs-alloy.monitoring.svc.cluster.local:4318. Point these at a different collector by overriding each signal's otlp.endpoint key individually; there is no single global override for all three. This exporter configuration is a separate concern from the admin debug bundle below: the bundle reads telemetry back out of a VictoriaLogs and Tempo query API, while the keys above control what the Platform API pushes in.
Every role process exports OTLP on its own timer, one stream each for traces, metrics, and logs, all defaulting to the in-cluster Alloy receiver; the admin debug bundle is a separate, request-driven read from a VictoriaLogs and Tempo query API, wired automatically only when the observability stack is enabled.
Point the exporters at the bundled in-cluster collector when you want zero-configuration observability and are comfortable keeping telemetry inside the cluster. Point them at an external OTLP endpoint instead when a compliance requirement keeps observability data outside the deployment's own cluster, or when you already run a fleet-wide observability backend and want this instance folded into it rather than siloed. Overriding the three endpoints independently is a supported shape, not a misconfiguration: a deployment that sends traces to one backend and logs to another is a normal split, not something to reconcile.
During shutdown, the process stops accepting new work and makes bounded attempts to release role-owned resources and flush its telemetry exporters before forced exit. The flush runs last so it can report the shutdown itself. That flush is one bounded delivery attempt rather than a delivery guarantee, so the terminal log record remains the second carrier of any final condition.
Extending the database metrics exporter
The PostgreSQL exporter ships enabled by default: it opens one direct session to managed Postgres and reports the aggregate server backend count, on top of the metrics its own upstream collectors define. The platform adds no custom queries of its own. Set postgres-exporter.enabled to false if that direct session is not something your deployment wants.
If you extend it, do not give a custom query a metric name a built-in collector already publishes. A duplicated name does not shadow one series or drop it: the exporter rejects the whole scrape and serves nothing. The pod stays ready while that happens, because readiness no longer reads /metrics, so watch up and pg_up for the gap rather than the pod status.
Treat the exporter's own metric names as a fixed contract, and prefer widening what it already collects over declaring a metric again.
Neither probe queries the database. Both hit /, the exporter's static landing page, on a 15-second period with a 5-second timeout. Only the Prometheus scrape of /metrics runs the collection, so the database pays for one collection per scrape interval and for nothing else. A readiness probe on /metrics would double that cost and would also mark the pod unready when the database is slow, which drops the scrape that carries pg_up and the wait metrics exactly when they answer the question.
The per-table collectors, stat_user_tables and statio_user_tables, ship off. stat_user_tables calls pg_indexes_size and pg_table_size once per user table, so one scrape stats every fork and segment of every relation and index in the database. On a schema with about a thousand tables that is seconds of server time per scrape and tens of thousands of series. Set postgres-exporter.perTableCollectors to true on a small schema where per-table vacuum, bloat and scan counts are worth that price.
Which signal answers which operational question
The three signals answer different questions, and reaching for the wrong one during an incident wastes the minutes that matter most.
| What you are asking | Reach for | Why |
|---|---|---|
| Which downstream call is responsible when one operation is slow or failing | A trace | Shows the causal path of one request, call by call |
| Whether a rate, count, or duration is trending worse, staying in range, or shifted since a deploy | A metric | Aggregates over time on a 60-second export interval, with up to one export interval of delay before ingestion |
| Exactly what one component reported, once a trace or metric has narrowed the search to a service and a time window | A log | A structured, point-in-time record carrying the identifiers that pivot back to the trace or metric around it |
Traces show the causal path of one request: which service called which, in what order, and where the time went. Reach for a trace when a single operation is slow or failing and you need to know which downstream call is responsible. A webhook delivery attempt that times out, or a Console request that hangs waiting on a database query, are both trace questions. A trace answers "what happened during this one request," not "how often does this happen."
Metrics show a rate, count, or duration aggregated over time, exported every 60 seconds (telemetry.metrics.exportIntervalMs) by default. Reach for a metric to answer a trend question: is this getting worse, is it within a normal range, has it changed since a deploy. Because the export interval is 60 seconds, a metrics dashboard is with up to one export interval of delay before ingestion. Treat a metric reading during an active incident as slightly stale, and confirm a fast-moving condition against a trace or a direct check instead.
Logs are structured, point-in-time records, one per notable event, carrying whatever identifiers, such as a request id or a trace id, let you pivot from a log line back to the trace or metric around it. Reach for a log once a trace or metric has narrowed the search to a service and a time window; a log answers "what exactly did this component report," which a trace's span names alone usually cannot.
Kubernetes health probes (/healthz and /readyz) are separate from this OTLP pipeline: fast checks the cluster itself polls to decide whether to route traffic to a pod or restart it. They are not exported anywhere and do not appear in VictoriaLogs or Tempo. See Platform API not ready for what each probe checks and what a failure means.
The sampling trade-off
The default 10% trace sample ratio is a head sampler: the platform decides whether to keep a given trace before that request finishes, with no knowledge yet of whether it will fail or run slow. Raising the ratio toward 100% keeps every trace, including the large majority that succeed quickly and tell you nothing new, and ingestion and storage cost rises in direct proportion to the ratio you set.
A downstream tail sampler can keep traces according to their outcome or latency. It can only inspect traces that pass the source sampler. To evaluate every trace, raise source sampling to 100% and size the collector for that incoming load before applying selective retention. Measure exporter, collector, network, and storage costs when changing the ratio.
Retention is not controlled from this page
How long a trace, metric, or log line survives in your backend is a property of that backend's own configuration, not of anything in the table above. If your deployment uses the bundled in-cluster stack, its defaults keep logs and traces for 168 hours and metrics for one month; Observability stack covers the values that raise them. An investigation that starts days after an incident can fail because retention already expired, not because anything is misconfigured; if that risk matters to your deployment, raise retention deliberately rather than assume a default depth holds.
The admin debug bundle
Console's admin menu carries an "Export debug information" item, visible only to a platform administrator or a system manager, the same gate the Platform Status page enforces. Selecting it calls a Platform API route that assembles recent logs and traces for the instance into a point-in-time zip and downloads it, without giving the requester direct access to the observability cluster. Set global.observability.enabled: true when your deployment includes the bundled LGTM/Grafana observability stack, and the debug bundle's in-cluster VictoriaLogs and Tempo query endpoints are derived automatically from that stack's release service DNS: no per-URL configuration needed. global.observability.namespace (default monitoring) controls the namespace those derived endpoints target, and has no effect on where the traces, metrics, and logs exporters above send signals: those always target the fixed obs-alloy.monitoring.svc.cluster.local host unless you override each otlp.endpoint key yourself.
Two settings stay explicit regardless of auto-wiring:
dapi.observabilityLogsUrl/dapi.observabilityTempoUrloverride the derived endpoints, for an out-of-cluster stack or renamed services.dapi.observabilityBaseUrlsets the Grafana deep-link base, an external route host never auto-derived from in-cluster service DNS.
Alert on duration, not connection state
A degraded dependency needs a duration, not a state. A gauge that reports whether an input is connected reads the same for a two-second blip and a two-hour outage. Both show the disconnected value, and only the second is an operator's problem. Alert on how long the input has been unavailable, not on whether it is unavailable right now.
The platform's own signals follow this pattern. dalp.didx.ws.unavailable_seconds tracks how long the chain's newHeads WebSocket has been continuously down. dalp_redis_client_unavailable_seconds tracks the same for a Redis client. Both sit at 0 while connected. Both escalate their log line from WARN to ERROR once the outage passes 30 seconds, The log threshold is separate from an alert's evaluation and hold periods. The bundled sustained-outage alerts take about six minutes to fire; tune them for your operating requirements. Reconnection can still recover without operator action after either threshold.
Record the duration from when the outage started, not from the most recent reconnect attempt. A duration that resets on every attempt stays near zero for exactly the outage that will not resolve itself, which is the case the alert exists to catch. Clear the gauge once the dependency reconnects, so the alert resolves itself instead of paging on a stale reading.
The dashboards and alert rules the bundled stack ships with, and its default retention windows, live with the stack itself: see Observability stack. This page covers export configuration and this alerting shape, not the receiving stack's content or storage policy.
Frequently asked questions
What happens if I set global.observability.enabled to true but never install the observability stack?
The debug bundle still derives the in-cluster VictoriaLogs and Tempo URLs from that flag, whether or not the stack is actually running. Each query carries its own five-second timeout, and an unreachable or erroring endpoint is reported as an unavailable artifact rather than failing the whole export, so the rest of the bundle still lands. The exporters above are unaffected either way, since they send OTLP to obs-alloy independently of this flag.
Can I disable trace export while keeping metrics and logs exporting?
Yes. Traces, metrics, and logs each carry their own enabled key under telemetry, defaulting to true, so turning one off leaves the other two untouched. No single switch disables all three at once: disabling logs, for instance, does nothing to the trace or metric exporters, which keep running on their own schedules and their own endpoints.
Do the three signals authenticate to their collector the same way?
No, not necessarily. Each signal's OTLP block carries its own headers key alongside its endpoint, so traces, metrics, and logs can each present a different bearer token or API key, or none at all, to whatever collector they target. That matters most once you split signals across backends: a managed metrics vendor and an in-cluster log store rarely share one credential.
If I set dapi.observabilityLogsUrl explicitly, do I still need global.observability.enabled: true?
No. The debug bundle reads that value directly and uses it whenever it is a non-empty, valid HTTP or HTTPS URL, independent of the flag. global.observability.enabled only controls whether the chart derives a default for you; an explicit URL is used as given, whether the flag is on, off, or the bundled stack was never installed at all.
Can the debug bundle's incident window run past my backend's retention?
Yes, if your backend retains less history than the requested window. The bundle's window defaults to the six hours before the request and is capped at 24 hours of lookback regardless of what a caller asks for, well inside the bundled stack's 168-hour default for logs and traces. Retention only becomes a real risk for a direct backend query days or weeks later, which is a different question from this bundle's own short window.
Deposit funding instructions
How an operator chooses which tokens investors can deposit and how the instructions API tells an investor where to send them.
Upgrades and migrations
How the database migration Job runs inside a Helm upgrade, which hook type to choose, and what a failed migration leaves behind for you to inspect.