Install the observability stack
Install the observability Helm chart a DALP instance exports to: an Alloy collector, VictoriaLogs, Tempo, VictoriaMetrics, and Grafana, then verify telemetry.
The observability chart is the receiving side of the platform's telemetry. A DALP instanceA DALP Instance is one self-contained copy of the platform, deployed inside a boundary you choose and connected to ledger networks you approve.Read the concept exports traces, metrics, and logs over OTLP (the OpenTelemetry Protocol) whether or not anything is listening, as Observability describes. This chart is the one Helm release that catches that export and makes it usable, routing each signal to its own store and surfacing all three through Grafana, with DALP dashboards provisioned and wired in. Install it when you want the deployment observable without assembling a stack yourself. Skip it when a fleet-wide backend already exists: in that case point each signal's export endpoint at your collector instead, and nothing on this page applies.
Installing it changes two things beyond the new pods. First, the platform's zero-configuration path starts working: the dalp chart's export defaults already target this stack's collector in the monitoring namespace, so a platform installed after this chart ships telemetry from its first pod onward with no export values set. Second, you commit to a telemetry identity. The chart refuses to render until you name the deployment, and that name is stamped on every signal the stack handles, so choosing it is the one decision here that is expensive to change later.
The middle ground is normal too. Every component sits behind its own enable flag, so a deployment that already runs, say, a fleet-wide log store can disable the bundled one and keep the rest. Disabling a component alone is not enough, though: Alloy and Grafana still point at the bundled service names, so keeping your own store also means repointing the matching alloy.endpoints.internal.<signal> endpoint and the corresponding Grafana datasource at it.
The platform's OTLP-exporting services export to the Alloy collector, which also scrapes the cluster, nodes, and non-OTLP workloads directly; Alloy routes logs to VictoriaLogs, traces to Tempo, and metrics to VictoriaMetrics, and Grafana reads all three, with DALP dashboards provisioned by default and alert rules opt in.
Steps
Name the deployment's telemetry identity
Set global.dalpInstance to a stable slug for this deployment: 1 to 63 characters, lowercase letters, digits, and hyphens, starting and ending with a letter or digit. Set global.deploymentEnvironment to one of local, development, test, staging, demo, or production. Both are required, both are validated at render time, and neither has a default, deliberately. The pair is stamped on every metric, log, trace, and profile the stack handles, and two deployments sharing one slug merge silently into a single identity in every fleet-level dashboard and alert group, while every signal still flows. Pick a slug you can keep, because changing it later splits the deployment's history into two identities. The render also refuses the slug local in any environment other than local, which catches the common copy-paste of a local values file into a staging install.
Write the values file
Start the values file with the identity pair, then configure credentials, routing, storage, and retention for your deployment:
global:
dalpInstance: "example-primary"
deploymentEnvironment: "production"On a restricted or multi-tenant cluster, scope Alloy before you install rather than after. Set alloy.rbac.namespaces to the namespaces you want the ClusterRole to cover. That list limits pod discovery for stdout scraping. It does not restrict OTLP logs, which can arrive from any pod that can reach the receiver Service. The platform's dapi, dapp, and docs containers export OTLP, so the stdout scrape excludes them. Set alloy.podDiscovery.namespaces instead when the scrape needs a different scope than the RBAC namespaces.
Add to it deliberately rather than at incident time. To keep a store you already run elsewhere, disable the bundled component and repoint Alloy and Grafana at your own. Each component has its own enable flag: victoria-logs-single.enabled, tempo.enabled, victoria-metrics-single.enabled, grafana.enabled, alloy.enabled. Disabling one alone leaves Alloy writing to a Service that no longer exists. Set the matching alloy.endpoints.internal.<signal> URL (prometheus, logs, or otel) to your store's address, and point the corresponding Grafana datasource at the same place. To hold telemetry longer than the defaults, raise the retention values now: victoria-logs-single.server.retentionPeriod and tempo.tempo.retention both default to seven days, and victoria-metrics-single.server.retentionPeriod to 1 month. A seven-day window cannot answer an investigation into older data, so size retention for the questions you are obliged to answer, not for the disk you happen to have. Supply unique Grafana administrator credentials through your deployment's secret mechanism: the chart ships working defaults for both, and a default admin login on the surface that can read every log line is the first thing a review flags.
Install the release
helm install observability oci://pkg.distribution.settlemint.com/settlemint/observability \
--version 3.1.5 \
--namespace monitoring --create-namespace \
--values my-values.yamlThe chart fixes its service names regardless of the release name you pick, matching the ones the diagram above already names (obs-alloy, obs-logs, obs-tempo, obs-metrics, obs-grafana). The namespace is the one placement choice that matters. The dalp chart's traces, metrics, and logs exporters all default to the Alloy receiver in monitoring, so installing there means the platform needs no export configuration at all. Installing anywhere else works, and costs two sets of overrides: each signal's otlp.endpoint key on the platform release, and global.observability.namespace on the same release if you enable the debug bundle wiring in the next step.
Install the receiving stack before the platform on a new deployment to avoid losing early telemetry. When upgrading both releases, check compatibility between the collector configuration and application exporters before deciding the order.
Connect the platform release
If the platform runs with default export settings and this stack sits in monitoring, telemetry flows with no further change. One connection is opt-in: set global.observability.enabled: true on the dalp release and the admin debug bundle derives its VictoriaLogs and Tempo query endpoints from this stack's service DNS. The Platform API admin interface's "Export debug information" capability then pulls raw logs and traces without cluster access. It defaults to off because the platform cannot assume this chart exists. Explicit dapi.observabilityLogsUrl and dapi.observabilityTempoUrl values override the derivation for a renamed or out-of-cluster stack. For export-side choices, the sample ratio, a different collector per signal, or an external backend, follow Observability.
What the chart deploys
| Component | Role | Enable flag (default) |
|---|---|---|
| Alloy | OTLP collector and router; scrapes cluster and pods | alloy.enabled (on) |
| VictoriaLogs | Log store | victoria-logs-single.enabled (on) |
| Tempo | Trace store | tempo.enabled (on) |
| VictoriaMetrics | Metrics store | victoria-metrics-single.enabled (on) |
| Grafana | Dashboards, datasources, alerting | grafana.enabled (on) |
| kube-state-metrics | Cluster metric source | kube-state-metrics.enabled (on) |
| node exporter | Node metric source | prometheus-node-exporter.enabled (on) |
| metrics-server | Cluster resource metrics API | metrics-server.enabled (off) |
Grafana's three datasources come provisioned and cross-linked: a log line carries a link to its trace, and a trace links back to the logs in its time window, so an investigation moves between signals without hand-building queries.
Alert rules are a deliberate second step
Dashboards arrive working; alerting does not, by design. The rule estate ships in the chart but stays dormant until you set grafana.sidecar.alerts.enabled: true, and enabling it forces a second choice: grafana.sidecar.alerts.notificationOwner must be active or shadow. Exactly one Grafana in an estate should be active and deliver notifications. A shadow install evaluates every rule and shows its state, but an always-on mute route stops it paging anyone, which is how a second stack observes the same fleet without doubling every page.
What you get once enabled spans twenty-two rule groups. Grouped by what they watch:
| Area | Rule groups | What they watch |
|---|---|---|
| Application and ledger | Application, Blockchain, BlockchainHealth | Log volume and absence, route errors and latency, ledger heap, block lag, sync, and peers |
| Data layer | Database, PgBouncer, PostgresAvailability, RedisConnectivity | Pool exhaustion, PgBouncer and Postgres availability, replica lag, Redis connectivity |
| Indexer | DidxBackfill, DidxDirectoryLineage, DidxLive, DidxNativeBalance | Backfill health, directory conflicts, indexing lag, wallet balance |
| Delivery and identity | Webhooks, OnboardingDeploy, EstateIdentity, RpcPool | Webhook backlog, stuck deploys, slug conflicts, RPC health |
| Settlement | Settlement | Failed XvP and token-sale workflows |
| Platform guardrails | Security, Migrations | Kyverno quorum, latency, and denials; migration failures |
| Infrastructure | Infrastructure, Velero, Heartbeat | Crash loops, replica mismatches, resource pressure, cert expiry, backups, heartbeat |
| CI runners (opt-in) | CIRunners | Runner scale-set health; needs the runner controller's own cluster |
Two families of absence rules fire on silence instead of a threshold: per-service log-ingest rules for streams that stop arriving, and a per-cluster heartbeat for a whole cluster that stops reporting. They exist because a broken telemetry path silences every log-based alert without any signal of its own.
Delivery goes to Slack in one of two modes. Webhook mode posts everything to one channel through grafana.sidecar.alerts.slackUrl. Bot mode routes critical and warning alerts to separate channels with runbook and dashboard links; it reads its token from the grafana-alerting-slack-token Secret, which the chart only renders from a chart value as a development convenience. Supply the token through your own secret machinery in a real environment, and never through a values file in version control.
Tune before trusting: the thresholds under dalp.observability are starting points. nativeBalance.lowThreshold defaults to 0.1 whole units of the chain's native currency for LowOperatorWalletBalance, regardless of that currency's decimal precision. webhooks.queueDepthThreshold defaults to 500, and webhooks.overdueMsThreshold defaults to 300000 milliseconds (5 minutes). These control the backlog and stalled-delivery alerts. kyverno.admissionLatencyHighSeconds defaults to 7 seconds, below the admission webhook's 10-second timeout. webhooks.queueDepthThreshold also carries a ceiling the others do not: the gauge it compares against saturates at 50000, so a threshold at or above that value could never fire, and the chart refuses to render one that high.
Two lists must also match your deployment. The log-ingest rules generate one rule per entry of dalp.observability.logIngest.services, whose default names the platform's own service containers: dapi (the Platform API), dapp (the Console), and docs (this documentation site). An install that does not deploy one of them must trim the entry, or that rule fires forever. The heartbeat rules generate one per entry of dalp.observability.heartbeat.clusters and default to none, because the chart cannot infer which clusters are supposed to report into it.
One stack for several clusters
The bundled stack is sized for one instance's telemetry in one cluster. To aggregate a fleet, run one central stack and put edge installs into forwarding mode. Set alloy.forwardingMode.enabled: true and alloy.forwardingMode.otel.url to the central Alloy's endpoint, and every signal copies there while the local stores keep working, so an edge cluster stays diagnosable on its own even when the link is down. Signals a cluster does not need locally can be switched off individually under alloy.endpoints.internal.
The central side does not authenticate its inbound traffic by default: the standard OTLP receiver on 4317 (gRPC) and 4318 (HTTP) takes unauthenticated traffic from anywhere that can reach it, in-cluster or not. Setting alloy.receiver.auth.enabled: true and alloy.receiver.auth.username adds a second, authenticated OTLP receiver on ports 4320 (gRPC) and 4321 (HTTP) with basic auth; the standard 4317 and 4318 stay open and unauthenticated alongside it. Put an ingress or network policy in front of the standard ports if this cluster is reachable from outside, since enabling the auth receiver does not close them. Both ends read their passwords from Secrets the chart deliberately does not create, observability-receiver-basic-auth on the receiving side and observability-forwarding-basic-auth on the sending side; create each Secret, holding a single password key, before the install that enables it. Rotating either password needs an Alloy pod restart, because the container reads it from its environment, and the config reloader only watches configuration.
Troubleshooting
helm install fails at render time asking for identity values
Four distinct messages come from the same guard, and each names its own fix. A required error naming
global.dalpInstance or global.deploymentEnvironment means the value is missing; set it. global.dalpInstance must be a 1-63 character lowercase slug means the slug is set but shaped wrong; it must be lowercase letters, digits, and
hyphens, starting and ending with a letter or digit. global.deploymentEnvironment must be local, development, test, staging, demo, or production means the environment value is set but does not match one of those six; typing prod
instead of production is the common miss. A message naming the slug and the environment together, ending "reserved
for local installs", means the slug local reached a non-local environment; give the deployment its own stable slug
instead. Fix the value the message names rather than working around it: every one of these failures is a signal that
would otherwise surface as corrupted fleet dashboards weeks later.
helm install fails at render time complaining about webhooks.queueDepthThreshold
dalp.observability.webhooks.queueDepthThreshold must be a whole number from 0 up to, but not including, 50000. The
gauge it compares against saturates at 50000, so a threshold at or above that cap could never fire. A non-numeric
value would otherwise coerce to 0 and page on the first pending row that lingers 15 minutes, so the chart fails the
render instead. Set the threshold to a whole number below 50000.
The render fails after I supplied my own alloy.alloy.extraEnv
Helm replaces list values wholesale, so your list dropped the chart's Secret-backed password entries,
RECEIVER_PASSWORD and FORWARDING_PASSWORD. The chart fails the render when receiver auth or forwarding auth is
enabled and its entry is missing, because catching the values mistake at deploy time beats discovering it as
authentication failures on a live receiver. Repeat the chart's entries inside your own list alongside your additions.
Grafana is up but the service dashboards are empty
The stack shows what reaches it. Before the dalp release lands, empty is correct. Once it is installed, check
placement and endpoints: the platform's export defaults target obs-alloy in monitoring, so a stack in a different
namespace receives nothing until each signal's export endpoint on the platform release is overridden to match. Cluster
dashboards are the control here; they draw on Alloy's own scrapes and fill in regardless of the platform. Cluster data
with no service data points at the export path, while no data at all points at the stack itself.
A log-ingest absence alert fires and never resolves
Two different causes generate this alert, and each needs a different fix. dalp.observability.logIngest.services
lists the services the rule expects to hear from; a service the deployment never runs produces a stream that never
arrives, so trim the list to the services actually deployed. Separately, dalp.observability.logIngest.namespaces
(falling back to alloy.podDiscovery.namespaces, then alloy.rbac.namespaces) names the namespaces each service must
report from, and the rule emits one series per namespace, so it fires when a single namespace goes dark rather than
only when every namespace does. A namespace in that list that never runs the service fires forever regardless of the
services list. Do not confuse either key with alloy.dalp.logIngest.services, which drops the stdout scrape for the
same container names and must never list a workload that does not export OTLP, the ledger node, proxies, and policy
controllers included. The fix in every case is a values change and an upgrade, not a silence.
Alerts are enabled but nothing reaches Slack
Check ownership first: a notificationOwner of shadow mutes all delivery by design, and only an active owner
pages. Then check the delivery mode: webhook mode needs grafana.sidecar.alerts.slackUrl set, and bot mode needs the
token present in the grafana-alerting-slack-token Secret. A rule visibly firing in Grafana with nothing delivered is
almost always one of these two configurations rather than the rule itself.
Frequently asked questions
Can several DALP deployments share one stack?
Yes. Every signal carries its global.dalpInstance slug, so two deployments in the same Grafana stay on separate dashboards and alert groups as long as neither reuses the other's value. A deployment in the same cluster can point straight at this stack's receiver. A deployment in another cluster reaches it through forwarding mode instead: it keeps running its own local stack and relays a copy here, with the authenticated receiver turned on this side.
What does changing the slug cost once the deployment has history?
The stamped global.dalpInstance value is how every earlier signal ties back to this deployment. Changing the slug starts a second identity. Existing history retains the old slug; new signals use the replacement. Treat the slug as fixed for the life of the deployment rather than something a later values change can correct.
Can I install this chart after the dalp chart instead of before?
Yes. A DALP instance exports whether or not anything is listening, so the platform just drops telemetry until this chart's receiver exists, then picks up on its own once obs-alloy starts answering in the monitoring namespace, no restart of the platform release required. Installing this chart first only avoids that gap; it does not change whether the eventual state is correct. On a new deployment, install it first anyway, since there is no reason to accept the gap on purpose.
Do I need to enable alerting to get value from this chart?
No. Dashboards and their cross-linked datasources come up as soon as the release installs: cluster panels show data immediately, and service panels fill in once the dalp release starts exporting to it. Alerting is a separate, deliberate step. It stays dormant until grafana.sidecar.alerts.enabled is set, and turning it on immediately forces a second choice, active or shadow, so a working install and a paging install happen on purpose, not as one bundled default.
What happens to the 30Gi of stored telemetry when I uninstall?
It stays. VictoriaLogs, Tempo, and VictoriaMetrics each use a stateful workload with its own persistent volume claim. helm uninstall removes the release's pods and services but leaves those claims. Kubernetes retains their storage after the workload is removed. The three 10Gi claims stick around, still billed, still bound, until you delete them yourself with kubectl delete pvc. A reinstall with matching claim names can reuse the retained data. A complete teardown also requires a deliberate decision about those claims and their underlying volumes.