Backup and disaster recovery
What a DALP instance backs up inside the cluster, what your managed dependencies must back up themselves, and how to restore or plan a failover.
DALP's Helm charts can schedule Velero backups of Kubernetes resources and volumes. Managed databases and object stores need their own backup and recovery plans. Configure both paths before production and test a restore in an isolated environment. This page explains schedule scope, retention, restore verification, and the limits of the bundled stateful services.
What gets backed up, and what does not
The platform backs up what runs inside your Kubernetes cluster through Velero, the Kubernetes-native backup tool the charts drive. Anything a managed dependency owns outside the cluster, your PostgreSQL instance, your Redis instance, your object storage bucket, is backed up by that service's own mechanism, not by DALP. Prerequisites already states this as a checklist item for production: a tested backup and restore approach for PostgreSQL and object storage, decided and in place before go-live, is your responsibility to arrange with whichever provider hosts them. This page does not restate that checklist; it covers the piece that sits on the other side of the boundary.
| What | Backed up by | How |
|---|---|---|
| Kubernetes resources and volumes in the release namespace, by default | The dalp chart's Velero schedule | A file-system backup, using Kopia (Velero's file-level backup engine), of every labeled resource, once a day by default; backup.namespaces.additional widens the schedule to more namespaces |
| Externally managed PostgreSQL (the recommended production path) | Your database provider | Not orchestrated by these charts |
| Externally managed Redis | Your provider, if it offers one | Not orchestrated by these charts; Redis here holds cache and queue state, not a system of record |
| Externally managed object storage | Your storage provider's versioning or replication | Not orchestrated by these charts |
The support chart's bundled PostgreSQL, Redis, or RustFS, if you run them instead | The dalp chart's Velero schedule | A raw file-system snapshot of the volume, not a database-native backup, and only when the resource sits in a scheduled namespace and carries the schedule's label; see Two chart pieces, one capability below |
That last row carries a real caveat. No chart configures a Velero backup hook against the bundled PostgreSQL or Redis, so a Velero backup of their volumes is a crash-consistent file-system copy, not a pg_dump or an application-quiesced snapshot. Use a database-native backup and verify recovery for production data.
Two backup paths split by who owns the data. The dalp chart's own Velero schedule backs up Kubernetes resources and
persistent volumes in the release namespace by default, as a crash-consistent file-system copy, with no database-aware
hook. An externally managed PostgreSQL, Redis, or object storage service backs up through its own provider mechanism,
entirely outside these charts. Every cluster backup lands in the configured backup store, RustFS by default if you use
the bundled support chart. On that schedule, only resources still carrying the kots.io/app-slug: settlemint-dalp
label by default fall inside its scope.
Two chart pieces, one capability
The backup capability spans two charts, and each does something different.
The support chart can deploy the Velero server, gated behind velero.enabled, off by default. It brings the server, a node-agent DaemonSet for file-system backups, and the AWS, GCP, and Azure plugin containers, and nothing else: no BackupStorageLocation, no Schedule, and no credentials Secret. The server mounts no credentials file, so a workload identity (IRSA, Azure managed identity, GCP workload identity) takes effect once you bind it to the server's service account through velero.serviceAccount.server.annotations, plus whatever pod settings your provider's Velero plugin documents.
The dalp chart never deploys a Velero server, and it is the one chart that decides what gets backed up and where. Its backup block, off by default, creates the BackupStorageLocation and Schedule. These custom resources need a running Velero server: the support chart's, or an installation such as OpenShift API for Data Protection (OADP). Velero only reconciles resources in its own namespace, so the chart renders them in the release namespace unless you set backup.veleroNamespace, for example to openshift-adp for OADP. An existingSecret you name must live in that same namespace. What the block does with the credentials Secret depends on what you set.
| You set | The chart creates |
|---|---|
backup.storage.aws.useIRSA, azure.useManagedIdentity, or gcp.useWorkloadIdentity | No Secret, and no credential reference on the BackupStorageLocation |
backup.storage.s3.existingSecret (or the equivalent field for your provider) | No Secret; the BackupStorageLocation references the one you already manage |
accessKeyId and secretAccessKey inline (storageAccountKey for Azure; not offered for GCP) | The Secret, populated from those values |
| None of these, the shipped default | Nothing: the render fails and names the missing field, so no location ever points at a Secret that does not exist |
The chart only decides what it renders; whether a Velero server exists depends on whether you enabled support.velero, or already run one another way.
Enabling support.velero alone backs up nothing: turn on the dalp chart's backup block too. No values file, the support chart's own or an environment overlay, may set velero.schedules, velero.configuration.backupStorageLocation, or velero.backupsEnabled: true. Before this split, the support chart also rendered a dalp-backup schedule every 4 hours and a second default location on the same bucket name. An upgrade removes both. Velero's garbage collection skips a backup whose storage location no longer exists, so the sup-velero-dalp-backup-* backups that schedule already wrote never expire on their own: keep them as recovery points, or delete the Backup objects and their bucket data yourself.
What the default schedule captures
These are the dalp chart's own backup.* defaults, the ones a production install is most likely to run.
| Setting | Default | What it means |
|---|---|---|
backup.schedule.cron | 0 2 * * * | Daily, at 02:00 |
backup.retention.days | 30 | Sets the backup's TTL to 720 hours; Velero deletes the recovery point, and the objects it wrote, once the TTL expires |
backup.includeAllPVCs | true | Turns on defaultVolumesToFsBackup, the Kopia file-system backup mode, for every included volume |
backup.labelSelector.matchLabels | kots.io/app-slug: settlemint-dalp | The dapp, dapi, and blockscout subcharts carry this label on their resources by default, so the default selector matches them without extra configuration |
| Cluster-level resources | Excluded | includeClusterResources is hardcoded false; the schedule only ever backs up namespaced resources |
| Excluded resource types | events, events.events.k8s.io | Hardcoded, not configurable |
| Storage location name | <fullname>-rustfs | Fixed suffix regardless of provider; an Azure- or GCP-backed location still shows -rustfs in its resource name |
| Schedule name | <fullname>-daily | The name to look for with kubectl get schedules.velero.io |
Decisions to make before you need a restore
Make these choices deliberately, before an incident forces them.
Decide your storage target and who owns its credentials. backup.storage.provider accepts s3, aws, azure, or gcp; the default points at the bundled RustFS endpoint. Store production recovery points outside the failure boundary they must recover. The chart creates no inline credentials secret for GCP; supply either backup.storage.gcp.existingSecret or set backup.storage.gcp.useWorkloadIdentity and skip the secret entirely.
Decide your retention window against your compliance or evidentiary needs, not the chart default. The 30-day default TTL is the only retention control; Velero keeps no minimum count of backups once older ones expire. If a regulator or an internal policy expects a longer recovery window, raise retention.days deliberately rather than assume a floor exists underneath it.
Decide whether the default label scope covers what you need backed up. A resource that never carried kots.io/app-slug: settlemint-dalp, or that loses it later, silently falls outside the schedule; the schedule keeps completing successfully, it just backs up less than you think. Two subcharts already fall outside it by default: pgbouncer, the mandated path in front of every configured PostgreSQL connection, and postgres-exporter. Both are on by default, and neither sets global.labels, so neither one ever carries the label. Confirm the scope with kubectl get all -l kots.io/app-slug=settlemint-dalp -n <namespace> and compare it against what you expect to matter in a restore.
Decide whether you are running the support chart's bundled PostgreSQL, Redis, and RustFS, or externally managed services. This choice changes what a restore, and a failover, can promise; see Failing over below.
Confirm the backup alerts reach someone. Three Velero alert rules ship in the observability stack, and all three default to warning severity, not critical.
| Alert | Condition | For |
|---|---|---|
VeleroBackupFailure | A backup failed in the last hour | 5 minutes |
VeleroBackupPartialFailure | A backup partially failed in the last hour | 5 minutes |
VeleroBackupStale | No successful backup for a schedule in over 24 hours | 15 minutes |
A stale schedule can run silently broken for close to a full day before VeleroBackupStale fires at all. If your operating process needs faster notice than that, do not assume the default severity routes to an on-call page; confirm it does, or raise it, deliberately.
Verify a backup is usable
The Grafana dashboard backups-policy gives you the automated signal: its "Backup failures" panel and "Last successful backup" panel tell you a schedule ran and did not error, and the three alerts above tell you when it stops. Neither one tells you the data that came back is usable; only a test restore does that.
Run a test restore on a schedule that matches how much you trust the last one you ran, at minimum before your first production go-live and after any change to backup.storage, backup.labelSelector, or backup.schedule. Restore into a namespace other than the one the backup came from, so you can inspect the result without touching a live workload. Confirm the resource and volume counts match what you expected. For anything that held database data, confirm the application starts cleanly against it, rather than only checking that files arrived. Delete the test restore's namespace once you have confirmed it, so a stale test copy does not linger as a second, unmanaged instance.
Restoring from a backup
-
Confirm which backup you are restoring from. A backup's phase must read
Completed, notPartiallyFailed, before you trust it as your restore source; a partial failure means some resources or volumes did not make it in.kubectl get backups.velero.io -n <namespace>This lists the backups the
<fullname>-dailyschedule created. Use the Velero namespace, the release namespace unless you setbackup.veleroNamespace. -
Decide your target before you create the restore. Restoring into the same namespace where the original resources still exist behaves differently from restoring into an empty namespace or a new cluster. Existing-resource handling depends on Velero's configured restore policy. Do not assume a restore overwrites live resources. For a test restore, set
spec.namespaceMappingon theRestore(step 3) to send the original namespace into empty space instead of overwriting anything live. -
Create the restore from the chosen backup by applying a
Restoreresource, then watch its phase rather than assuming it finished because the command returned.apiVersion: velero.io/v1 kind: Restore metadata: name: <restore-name> namespace: <namespace> spec: backupName: <backup-name> namespaceMapping: <original-namespace>: <target-namespace>kubectl get restores.velero.io <restore-name> -n <namespace> -w -
Wait for volume restores to finish separately from the resources that reference them. The default schedule backs up volumes at the file-system level with Kopia, so a restored pod can stay pending after its other resources already exist, while its volume's own restore is still copying data in the background. Watch the
PodVolumeRestoreresources Velero creates for this restore, not only the pod's own status:kubectl get podvolumerestores.velero.io -n <namespace> -
Confirm what came back before you call the restore done. A
Completedrestore proves Velero moved the bytes it had; it does not prove the data is transactionally consistent, since no backup hook runs against the bundled PostgreSQL or Redis before the snapshot is taken. Start the application against the restored data and confirm it comes up cleanly, the same check you already ran the last time you tested a restore.
Failing over
Failover splits into two different jobs, because they answer to two different owners.
Failing over the compute layer has one answer. Every dapi pod is stateless, and so is the Console (dapp); neither one holds state locally that a failover would need to carry over. Recovering compute also requires reachable datastores, secrets, registry access, network endpoints, and routing in the target environment. Validate those dependencies before sending traffic there.
The stateful layer is where failover is decided, and what it looks like depends entirely on what backs it. If your PostgreSQL, Redis, and object storage are externally managed, the production path Prerequisites recommends, failover is your provider's own replication and promotion mechanism, not something these charts orchestrate. A provider running a live standby promotes it, and your recovery point tracks that standby's replication lag, not the age of your last Velero backup.
If you are instead running the support chart's bundled PostgreSQL and Redis, there is no live standby to fail over to: both deploy as a single, unreplicated instance by design (see Deployment topologies). A cluster or regional loss under that topology means restoring your most recent Velero backup into new infrastructure, not promoting a replica. Recovery reaches the latest usable backup. Successful daily backups can leave a 24-hour gap; the support schedule reduces that to four hours. Missed or unusable backups make the gap longer. Treat the bundled stateful services as a development and evaluation topology, not a production failover story.
Consequences and reversibility
Once retention.days elapses, 30 days by default, Velero deletes that recovery point and the objects it wrote from your bucket, with no undo. Deleting the bucket a BackupStorageLocation points at destroys every recovery point that location owns, not just the newest one.
A successful backup or restore phase does not prove complete application recovery. Check resource scope, inspect warnings, and validate the restored application and data. Measure recovery time with your own volumes; the chart does not guarantee a restore duration.
For restores into a namespace with existing resources, check the behavior of your installed Velero version and its resource policy first. An isolated test target avoids relying on overwrite behavior.
Frequently asked questions
Will a restore bring back cluster-scoped resources like ClusterRoles?
No. includeClusterResources is hardcoded false in the default schedule, so nothing cluster-scoped ever enters the backup, and a restore has nothing to bring back for it. If your workload depends on a ClusterRole, a ClusterRoleBinding, or another cluster-scoped object, reapply it separately, from your chart's rendered manifests or your own record of it, before or after the restore runs.
Does raising backup.retention.days extend the life of backups that already exist?
No. Velero sets a backup's time-to-live from retention.days at the moment that backup is created, not on an ongoing basis, so a change to the value only lengthens the life of backups taken afterward. A recovery point already deleted stays deleted, and one still alive under a shorter TTL from before your change keeps that original TTL rather than adopting the new one.
If I switch backup.storage.provider away from the bundled RustFS default, do my existing backups move with it?
No. The dalp chart's BackupStorageLocation keeps one fixed name, <fullname>-rustfs, regardless of provider, and changing backup.storage.provider repoints that same location at a different bucket or account rather than copying objects between them. Your existing recovery points stay in the old bucket and become unreachable through this location once you repoint it, so migrate or archive anything you still need before you switch.