Queues and crons

Symptom: a step never runs

Diagnose in this order, because each is cheaper than the next:

1. Is it switched off? Check the step at /system/crons. Every step is seeded off, so a newly added one has never run.
2. Does it have a schedule line? A job with no line never runs and nothing reports it.
3. Does it have an entry in the cron inventory? Without one it runs unswitchable, and again nothing says so.

Repair: switch it on at /system/crons. For one run without switching it on, pass force after the filename.

Verify: the batch moves todo to in progress to finished, and the healthcheck goes green.

Symptom: a batch sits in progress

The job may still be running, or may have died before marking the batch finished. Inspect its process and output first. The retry job reopens aged batches up to each row's retry limit; age alone is not proof that the original worker stopped.

Repair: do not start a second worker while the first is running. Once it has stopped, diagnose the failure and allow or request a retry. A batch at its limit needs a person, at /system/batches/problems, where you can reopen it or raise its limit.

Verify: read a real row that the batch should have written. A finished batch is not evidence.

Symptom: a batch cannot be made to work and holds up everything behind it

An open batch counts as a blocker whether or not it will ever succeed, so one hopeless batch stops every job that waits on its type.

Repair: select it at /system/batches/running and mark it won't do. That takes it out of the queue and out of the blocker chain, and the work is not lost: the next reset queues the same type and settings again, because queueing only looks at what is still todo. A batch whose worker is genuinely alive records that it finished when it gets there, so the mark is for what is queued or dead.

Verify: it appears at /system/batches/wontdo and the jobs behind it run again. Reopening it there puts it back on todo with its retry count cleared, so it gets a full set of retries.

Symptom: one type never finishes while its siblings do

Two workers are hitting one upstream slot that serves a single caller at a time. The loser is refused immediately, dies after claiming its batch, and the retry job feeds it back into the same collision until it burns its retry limit.

Repair: give the jobs offset schedule minutes and have each check whether its own type is already running, so a second worker exits before claiming anything.

Symptom: a whole chain stops while only one step is paused

Expected. A queued batch of the paused type counts as a blocker, and many jobs depend on the sales order import.

Repair: switch the type back on, or accept that its dependants wait. A pause defers work rather than losing it; the queue drains when you switch it back on, with no backfill.

Symptom: nothing runs on a brand new environment

Every step is seeded switched off by design. Run the install jobs with force, then turn the steps on.

Related

• The pipeline
• Missing or stale data
• Verification

Evidence

Switch semantics, the blocker chain, retry behaviour and the concurrency failure mode from AGENTS.md, Jobs and the batches queue and What fails silently. Basis: code-checked against the current implementation.