workflow engine operator API

These routes let platform operators inspect Workflow Engine health and run narrow recovery steps without touching tenant-facing product flows. Start with the doctor route, choose one write route that matches the failing component, then run doctor again to confirm the result.

Call these endpoints from an authenticated account with the system operate permission. Use them from operator accounts and trusted automation only, not tenant-facing product flows.

Recovery order

When a dependency outage pauses durable work, the engine parks the affected invocations instead of failing them. After the dependency recovers, use the resume routes below to release those paused invocations: resume one invocation by id, or bulk-resume every paused invocation on the current deployment behind a confirmation gate.

The exact route path contains the current public API service segment. Treat that segment as an endpoint path, not as product terminology. Do not use it as a product name.

Component statuses

Doctor responses use the same status vocabulary for each component.

A degraded sub-check does not fail the whole doctor route. Read every check result before you choose a recovery step.

Doctor

doctor is the read-only entry point. It probes the workflow engine ingress URL, health API, deployment list, service list, and invocation table. Send an empty JSON object as the request body.

POST /api/v2/admin/operator/workflow-engine/doctor

Success response

{
  "ingress": { "status": "ok", "latencyMs": 12 },
  "admin": { "status": "ok", "latencyMs": 9, "version": "1.4.0" },
  "deployments": {
    "status": "ok",
    "items": [
      {
        "id": "dp_01j8m7k2q3r4s5t6u7v8w9x0y1",
        "serviceUrl": "https://workflow-service.example.com",
        "createdAt": "2026-05-09T10:00:00.000Z"
      }
    ],
    "error": null
  },
  "services": {
    "status": "ok",
    "items": [{ "name": "IdentityRecoveryWorkflow", "revision": 3 }],
    "error": null
  },
  "invocations": {
    "status": "ok",
    "byStatus": { "invoked": 2, "suspended": 1 },
    "recentFailures": [],
    "error": null
  }
}

recentFailures returns up to 20 recent failed invocations. Each item includes id, serviceName, serviceKey, failedAt, and errorMessage.

Force redeploy

force-redeploy registers the workflow service URL with the workflow engine health API. It does not remove old deployments. Run stale deployment cleanup when you see old deployment records in the doctor output.

POST /api/v2/admin/operator/workflow-engine/force-redeploy

Request body

{
  "serviceUrl": "https://workflow-service.example.com",
  "force": true
}

Success response

{
  "acknowledged": true,
  "deploymentId": "dp_01j8m7k2q3r4s5t6u7v8w9x0y1"
}

Cleanup stale deployments

cleanup-stale-deployments keeps the deployment matching serviceUrl and drains every other registered deployment. Use this route after doctor or force redeploy confirms your active service URL.

POST /api/v2/admin/operator/workflow-engine/cleanup-stale-deployments

Request body

{
  "serviceUrl": "https://workflow-service.example.com",
  "forceDrain": false
}

Success response

{ "acknowledged": true }

If DALP cannot list deployments or reach the health API, the route returns a workflow-engine-unreachable error instead of acknowledging cleanup.

Recover stuck workflow

recover-stuck-workflow prepares one workflow key for retry. DALP kills and purges prior invocations for the supplied (serviceName, serviceKey) pair, then clears keyed workflow state so the next submission starts from a blank state.

POST /api/v2/admin/operator/workflow-engine/recover-stuck-workflow

Request body

{
  "serviceName": "IdentityRecoveryWorkflow",
  "serviceKey": "invitation_01j8m7k2q3r4s5t6u7v8w9x0y1"
}

Success response

{ "acknowledged": true }

DALP refuses to clear a workflow when an active invocation is still running or when the previous invocation already succeeded. In that case, the route returns a structured retry-blocked error with reason and invocationIds. Inspect those fields before you retry or escalate.

Resume one paused invocation

When a dependency outage exhausts an invocation's retries, the Workflow Engine pauses that invocation rather than failing it. The paused invocation keeps its journal and can continue once the dependency recovers. resume-invocation releases one paused invocation by id, and the engine retries from the last journal checkpoint. Reach for this route when doctor or your alerting points to a single parked invocation.

POST /api/v2/admin/operator/workflow-engine/resume-invocation

Request body

{
  "invocationId": "inv_01j8m7k2q3r4s5t6u7v8w9x0y1"
}

Success response

{
  "acknowledged": true,
  "invocationId": "inv_01j8m7k2q3r4s5t6u7v8w9x0y1"
}

DALP resumes only invocations that are currently paused. If the id is unknown, the route returns invocation not found. If the invocation exists but sits in another state, the route returns invocation not paused so you do not resume work that is already running or finished.

Resume all paused invocations

After a shared dependency recovers, many invocations can sit paused at once. resume-paused-invocations clears them in one call, scoped to the live deployment you name. The route runs in two stages: preview with dryRun, then resume with confirm. The two-stage flow exists so you review the list before releasing a batch of durable work.

POST /api/v2/admin/operator/workflow-engine/resume-paused-invocations

Request body

{
  "serviceUrl": "https://workflow-service.example.com",
  "dryRun": true
}

Preview response

A dry-run lists what a resume would touch and changes nothing. pausedCount reports how many invocations match, resumedCount stays 0, and invocations previews each one.

{
  "acknowledged": true,
  "dryRun": true,
  "pausedCount": 2,
  "resumedCount": 0,
  "invocations": [
    {
      "id": "inv_01j8m7k2q3r4s5t6u7v8w9x0y1",
      "serviceName": "IdentityRecoveryWorkflow",
      "serviceKey": "invitation_01j8m7k2q3r4s5t6u7v8w9x0y1",
      "modifiedAt": "2026-05-09T10:00:00.000Z"
    }
  ]
}

Resume response

Review the preview, then call again with confirm: true and dryRun: false. The route resumes each matched invocation in turn and skips any that another operator already resumed.

{
  "acknowledged": true,
  "dryRun": false,
  "pausedCount": 2,
  "resumedCount": 2,
  "invocations": [
    {
      "id": "inv_01j8m7k2q3r4s5t6u7v8w9x0y1",
      "serviceName": "IdentityRecoveryWorkflow",
      "serviceKey": "invitation_01j8m7k2q3r4s5t6u7v8w9x0y1",
      "modifiedAt": "2026-05-09T10:00:00.000Z"
    }
  ]
}

When dryRun is false and one or more invocations match, the route refuses to act without confirm: true and returns bulk resume not confirmed. If serviceUrl does not match a registered deployment, the route returns deployment not found.

Error conditions

Related pages

• workflow engine recovery
• API monitoring
• API error reference
• Authorization