Cadence
Concepts

What moves a task without a person

What advances a task with no one in the loop, what still needs a person, and the conditions each depends on.

Cadence can move work without a person: a pool leases a task and a worker works it, a rule advances it to the next stage, another rule holds it. You read this page to decide what you can leave to run and what you must still watch. Each thing that happens on its own is stated here with the conditions it depends on, and each thing that still needs a person is stated plainly.

What advances a task on its own

A pool leases and works a task

When a task enters a stage that an assignment rule routes to a pool, the pool leases the task to a worker and the worker works it — no one dispatches it by hand. This happens only when there is an assignment rule on the stage, the pool has a non-zero size and is not paused, there is room to run the lease, nothing is suppressing dispatch — the task has no blocking hold, no uncleared dependency, and no unpassed require check at the stage — and, for an agent pool, there is an LLM token to serve the worker.

An agent pool's worker runs a model sold by the LLM vendor named by the worker version its lease runs — the version the pool pinned when the lease was created. Cadence looks for one registration of that vendor: your space's, or, only when your space has no registration of that vendor at all, the one registered for the whole instance. The worker starts only if that one registration has a token that is live, can be read, and is not held back by its limit or by the vendor. If your space has its own registration of the vendor, its tokens are the only ones considered: when they are all held back or cannot be read, the worker does not start, even if the instance's registration has tokens to spare.

When no token can be served, the lease is created but its worker does not start: Cadence records worker_spin_up_failed on the task, ends the lease as failed, puts a blocking hold on the task, and does not retry. Cadence does not get the task moving again on its own. A person must, unless a worker whose lease reaches the task replaces the hold with an advisory one and clears it. A worker that moves the task does not get it moving: a move keeps the blocking hold, so no pool starts on the task at its new stage until the hold is cleared (see A blocking hold waits for a person to clear it). First fix the cause — register the vendor, add a token, raise or clear a token's limit, top up the account or raise the key's cap with the vendor, replace a key the vendor rejected, or repin the pool to a version whose vendor has a token to serve. Then clear the hold, with tasks_clear_hold or the Clear hold button in the task drawer in the board view. Each task whose lease failed this way carries its own hold, so clear each one. Clearing dispatches the task again at its stage, which starts the pool again through the stage's assignment rule when the conditions above hold; clear before you fix the cause and the next lease fails and holds the task again. See Hold a task and clear a hold § 3 and LLM providers § How a token reaches a worker.

This condition applies to every agent pool whose version names a vendor. A version published now must name one, but a version made before vendors were required may name none; a lease on such a version starts with no token, and this condition does not apply to it.

Room means three ceilings, each of which must have space: the pool (its size, a count of leases), the space (its Max vCPU ceiling, if set), and the whole instance (a ceiling the operator running your instance may set). The space and instance ceilings are counted in vCPU, not in a count of leases: room there means enough free vCPU for the lease's whole size — the agent it runs plus every sidecar alongside it — so a large lease can wait while smaller ones keep taking the vCPU it needs. A task that meets every other condition above — assignment rule, a non-paused non-zero pool, no hold, no uncleared dependency, no unpassed check — can still sit pending when the pool, the space, or the instance is at its ceiling. Meeting the conditions is necessary, not sufficient: it does not guarantee the task is leased right away.

An assignment rule with target: parent is the exception to where those conditions are read. It sits at a stage a child enters and leases the child's parent, not the child. Cadence checks the hold, dependency and require-check conditions against the child that entered, not against the parent it leases. So a parent with an uncleared dependency, or an unpassed require check at its own stage, is still leased, and its worker runs. A parent with a blocking hold is still leased too, but its worker is refused when it tries to start: over MCP lease_claim answers task is on hold, and over the HTTP API the claim answers 409 task_on_hold. A parent rule needs no assignment rule at the parent's own stage. See Stage rules § Assignment rules.

A lease held back by a ceiling is not dropped. It stays pending, and the space admits waiting leases oldest-first as running work finishes and frees room — with no one intervening. Oldest-first is the order, not a reservation: because the space and instance ceilings count vCPU, a large lease can be passed over while smaller leases behind it keep taking the vCPU it is waiting for.

Every lease held back at a ceiling now leaves a trail. The first time a lease is held back, it records a one-time dispatch_waiting_on_ceiling activity naming which ceiling bound it, so you can see why a lease is queued rather than running:

  • A pool ceiling — the pool at its size — carries the pool's limit, its size as a count of leases.
  • A space or instance ceiling — no free vCPU for the lease's whole size — carries limit_vcpus (the ceiling in vCPU), needed_vcpus (what this lease needs), and in_flight_vcpus (what is already running against that ceiling).

Only one such entry is written per lease, even while it waits across many dispatch passes. A lease held back only by a full pool records this activity too — the pool no longer queues its overflow silently, so an unexplained pending lease is not something you have to guess about.

Two more dispatch waits, previously silent, now leave a one-time trail as well — and both are conditions you must clear, not ones that free themselves:

  • A pending lease behind a paused pool records a dispatch_waiting_on_paused_pool activity. Nothing dispatches from that pool until you resume it.
  • A pending lease in an archived space records a dispatch_waiting_on_archived_space activity. Nothing dispatches until you unarchive the space.

Each of these names the pool waiting and a short summary of the quiet. So a pending lease that is not moving now always accounts for itself on the task's timeline: which ceiling it is behind, or that it is waiting on a pool you paused or a space you archived.

A scheduled rule starts a worker on a clock

A scheduled rule mints a lease on a pool at a subject — a task, a workflow, or the whole space — on a schedule, with no event and no person. It does this independently of any stage or assignment rule: where a pool leases a task because the task entered a stage, a scheduled rule leases because the clock came round.

This happens only when the rule is enabled, its schedule is due, the pool's space is not archived, and the rule's pool and subject both still exist. Only one worker runs at a time for a given subject and pool: if a worker from that pool is already live on that subject when the rule is due, the run starts nothing and waits for a later tick.

One case runs off schedule. When a worker at a workflow or space subject is lost, Cadence starts the same pool at that subject again on its own, on its sweep about once a minute, whether or not the rule is still enabled or due. Disabling the rule does not stop this. It happens only when the lease was lost less than 24 hours ago, the pool is not archived, no lease of that pool is active at that subject, and no later lease of that pool at that subject ended in a state other than lost. After three lost leases of the pool at that subject, Cadence stops and places no hold, because there is no task to hold. That count is never reset. Nothing in your task list points you to these leases: find them with leases_list space_key=OPS state=lost, whose subject_kind is workflow or space. See Leases.

At a task subject, a scheduled rule does not wait for the task's dependencies or required checks, and it still creates its lease while the task has a blocking hold. The worker is then refused when it tries to start. See Scheduled rules § A scheduled rule at a held or blocked task.

A scheduled rule starts a worker — it does not move a task by itself. A scheduled lease at a workflow or space subject runs a worker that answers for that workflow or space; it advances nothing on its own. Even at a task subject, minting the lease runs the worker — what happens to the task afterwards is up to what the worker does. The worker can move the task itself (below); otherwise a move to another stage still takes a sentinel rule or a person.

An inbound rule starts a worker when an event arrives

An inbound dispatch-worker rule starts the pool it names at a task when a matching event arrives from outside Cadence, at whatever stage the task is in. No assignment rule and no person is involved.

It checks the same conditions as an assignment rule, but it does not wait for them. When the match's turn comes, if the task has a blocking hold, an uncleared dependency, or an unpassed require check at its stage, the match is recorded failed on the delivery, with the reason, and is never tried again. Clearing the hold or the dependency later does not bring it back. See Inbound § When a dispatch mints nothing.

The one thing a match does wait for is a lease of the same pool on the task that is still active: it runs once that lease has ended, with matches handled in the order their events arrived. It still runs then if the rule has been disabled in the meantime; see Inbound § The dispatch queue to remove it first. A pool that is paused or has no room does not fail the match either. The lease is created and waits, not started, until the pool is unpaused and has room.

A person or a worker assigns a task to a pool

Assigning a task to a pool starts that pool's worker on it, with no assignment rule involved. A person does this with the task's Assignee field in the browser, and anyone, including a worker whose lease reaches the task, does it with tasks_assign and assignee: "pool:<id>". See Tasks § Assign a task.

The assignment creates the lease straight away. It does not check a hold, a dependency or a require check. If a lease of that pool on the task is still active, it reuses that lease instead of creating another. While the task has a blocking hold, the worker is refused when it tries to start; otherwise it runs. Whether a worker assigns tasks to pools depends on its worker version and the agent skills it pins.

A check rule prepares or gates the work

On entering a stage, a create check rule materialises a pending check on the task automatically. A require check rule gates entry to the stage automatically — the task cannot enter until that check has passed.

The gate is automatic; passing the check is not. Cadence never passes a check of its own judgement. A check moves to passed only when a worker or an operator sets it (see Checks) — including a classifier whose output you configured to set it. Until something sets it, the gate holds.

A sentinel rule advances or holds the task

A sentinel rule can move a task to another stage, or set a hold, once the checks at its stage reach a state — with no person involved. It does this only when all of these hold:

  • The task's workflow has auto_advance_enabled set. On a workflow without it, no sentinel rule moves a task, and Cadence places none of the holds that come from the sentinel sweep (below). A worker can still move a task (below), and pools still start work, under the conditions in A pool leases and works a task, A scheduled rule starts a worker on a clock, An inbound rule starts a worker when an event arrives and A person or a worker assigns a task to a pool.
  • The space is not archived, the stage is not archived, and the task is not archived. A task resolved since it entered its current stage is also passed over: no sentinel rule moves or holds it until it is moved to another stage or unresolved, by tasks_unresolve or by a resolution rule that unresolves it.
  • The rule applies to the task by its scope, including its parent_state condition.
  • The task has no blocking hold.
  • No work at the stage is still running: every lease of the pools named by the stage's assignment rules that apply to the task, created since the task entered the stage, has ended. A lease of any other pool does not hold the sentinel rule back, for example one started by a scheduled rule, an inbound rule, an assignment to a pool, or a target: parent rule at a child's stage. The sentinel rule can move the task while that worker is still running on it, and can act when no lease ran at the stage at all.
  • Its predicate over the stage's checks is true: all_passed means every such check has passed; any_failed means every such check is resolved (none still pending) and at least one failed; none_required means the stage has no such checks.
  • For a transition, the target stage's own gates pass.

Sentinel rules are evaluated on a short recurring tick and whenever a check changes, so a passing check can trigger a move within moments — but only under the conditions above.

A worker moves a task

A worker can move a task itself, with no rule and no person involved. A lease can call tasks_transition to move a task to another stage on any task its lease scope reaches. Auto-advance does not affect this, and a blocking hold does not stop it: any worker whose lease reaches the task can move it, including one started after the hold was placed. The move must pass the target stage's gates, and a worker can never force one.

A lease can also call tasks_transfer to move a task it reaches to another workflow, but only while no other lease on that task is active. A worker that reaches the task from its parent, a child or a sibling cannot transfer it while the task's own lease is still running. Whether a worker moves tasks at all depends on its worker version and the agent skills it pins. Each move is recorded on the task's timeline. The entry's by names the worker's pool, as pool:<id>, not the individual lease.

A classifier writes checks and properties

A classifier is a worker that asks a question-answering provider a fixed list of questions about a task, then turns the answers into checks and properties on that task — with no person in the loop. When a classifier pool's lease reports success, Cadence renders the pinned classifier version's output document against the worker's reported result — the answers it reported when it completed the lease — and writes what it names: each properties entry as a task property, each checks entry as a check result. How the output expressions render over that result is covered in How an output expression is rendered.

The conditions are these, and they hold in the same breath as the claim:

  • It happens only on a success report. A classifier lease that fails writes nothing.
  • It is all-or-nothing. If any property or check the output names cannot be rendered or applied, Cadence writes none of them and completes the lease as failure instead — which holds the task for an operator, the same as any failed lease. There is no half-applied outcome to discover.

Cadence records a classifier_outcome_applied activity on the task naming the properties and checks it wrote, so the outcome is readable from the timeline. A classifier can therefore pass a check without a person — but only a check its own output names, and only because you configured that version to. It is a worker setting the check, decided in advance, not Cadence judging the work itself.

What still needs a person

A check no one sets holds the task forever

Because nothing auto-passes a check, a stage can wait indefinitely:

  • A require check that no worker or operator ever sets to passed gates entry forever.
  • A sentinel all_passed predicate never becomes true while any required check stays pending — a worker that sets one check and leaves a sibling pending stalls the task until someone sets the rest.

If you rely on a stage to advance on its own, make sure something — a worker on the stage, or you — actually sets the checks its rule reads.

A blocking hold waits for a person to clear it

A blocking hold stops the task: it stops the task's assignment rules and inbound rules from starting a pool for it, it refuses every worker that tries to start on it, and it excludes the task from sentinel evaluation. Cadence never clears one on its own: neither the sentinel nor dispatch ever clears a blocking hold. The tools that clear a hold (tasks_clear_hold, tasks_nudge and tasks_dispatch_pool) refuse a worker or other non-human principal on a blocking hold. (An advisory hold suppresses nothing and is retired when the task moves to another stage on its workflow. A move to another workflow with tasks_transfer keeps it.)

A blocking hold is still not a guarantee that the task waits for you. A worker whose lease reaches the task can call tasks_set_hold with kind: "advisory", which replaces the blocking hold with an advisory one that the worker may then clear. Any worker whose lease reaches the task can also move it, whenever its lease started. The task's timeline records each of these, naming who did it: a person as user:<id>, a worker by its pool as pool:<id>. See Web interface § Making a task wait for a person.

Cadence itself sets a blocking hold in several cases. These three apply on any workflow, whether auto-advance is on or off:

  • A worker's lease failed. A lease on the task failed, was stopped for running past its pool's time limit, or stayed deferred for too long. This is the hold you are most likely to meet. It includes a worker that could not start at all — for an agent pool, because no LLM token of its version's vendor could be served. That hold's reason reads dispatch blocked — manual intervention needed: followed by the pool's name and (lease failed: spin-up failed: …), naming the vendor and the pool.

  • A pool could not be dispatched. When a lease of a pool on the task is lost, Cadence starts that pool on the task again, and holds the task instead once three of the pool's leases on it have been lost. It does either only when all of these are true:

    • the lease was lost less than 24 hours ago
    • the lease was created after the task entered its current stage
    • an assignment rule at the task's current stage names the pool and applies to the task
    • the pool is not archived and has no lease active on the task
    • no later lease of that pool on the task ended in a state other than lost

    Starting the pool again, but not the hold, also needs the task to have no blocking hold, no unresolved dependency that blocks at its current stage, and no unpassed required check. While one of those stops it, Cadence tries again about once a minute and starts the pool once all of them are true, as long as the conditions above still hold. Once the lease was lost more than 24 hours ago, nothing starts the pool again on its own. The count of lost leases covers every stage the task has been at since the count was last reset, so the hold can come before three losses at the current stage.

    A pool that no assignment rule at the stage names, such as one started only by an assignment, a scheduled rule or an inbound rule, is never started again this way and never gets this hold. The hold's reason reads dispatch blocked — manual intervention needed: followed by the pool's name and its number of attempts, such as (3 attempts), and names every pool that reached that point for the task. The (3 attempts) ending is how you tell it from a failed or stopped lease, whose reason ends in (lease failed: …) or (lease killed: …).

    Once this hold is placed, the pool stays blocked until a clear of the task's hold (Clear hold, tasks_clear_hold, or Dispatch or tasks_dispatch_pool while the task has a hold), a move to another stage on the workflow while the task has an advisory hold (Cadence clears that hold on the move), or tasks_dispatch_unblock for that pool resets it. A move while the task has a blocking hold or no hold, and a move to another workflow, do not reset it. tasks_nudge does not reset it. After a nudge while the pool is still blocked, whatever hold the task then carries, Cadence does not place this hold again for that pool: the next time the retries run out, it stops retrying and places no hold, and the task sits idle. A worker may nudge a task with an advisory hold or no hold, so this can happen with no person involved. On a workflow with auto-advance on, the abandoned-work hold below may catch the task, under the conditions listed there; otherwise nothing points you to it. Clear the hold with Clear hold or tasks_clear_hold instead of nudging, or call tasks_dispatch_unblock after the nudge, and the hold comes back when it is needed. Dispatch and tasks_dispatch_pool reset the count of lost leases only for the pool you dispatch. If the hold names several pools, the task may be held again for one of the others on Cadence's next lost-lease sweep, which runs about once a minute, separately from the 30-second sentinel sweep, but only if that pool's last lost lease still meets every condition above. To find and restart a task already left idle by a nudge, see Tasks § Find and restart a task left idle after a nudge.

  • A stage loops. A stage holds a task that keeps returning to it, rather than let it cycle forever. reentry_limit (three by default) counts the task's first arrival along with every return, so with the default a stage tolerates two returns and holds the task on the third — its fourth entry overall.

The next three come from the sentinel sweep, so they happen only on a workflow with auto_advance_enabled set, in a space that is not archived, at a stage that is not archived, and only to a task that is not archived and has not been resolved since it entered its current stage. On a workflow with auto-advance off, Cadence never places them, and a task stuck there stays unflagged until you notice it:

  • Work looks abandoned. Cadence holds the task rather than leave it silently stuck, but only when all of these are true:
    • the stage has abandoned_work_detection on (the default)
    • no sentinel rule at the stage moved or held the task on that pass
    • the task is not waiting on an uncleared dependency. That task is blocked, not abandoned.
    • at least one lease of a pool named by an assignment rule at the stage that applies to the task (see Stage rules § How a rule's scope is chosen) has been created since the task entered the stage. A task that no assignment rule at its stage applies to never trips this hold, and leases of pools that no assignment rule applying to the task names do not count.
    • none of those leases is still active, and the newest of them finished more than 3 minutes ago, so no retry is still in flight
    • a check the stage expects is still missing or pending
  • Evaluation keeps failing. On a workflow with auto-advance on, a task whose sentinel evaluation errors 3 times during its stay at the stage is held rather than retried without end.
  • A hold rule matches. A sentinel rule whose action is to hold the task sets a blocking hold.

The hold's reason says which case it was. The full list is also in Web interface § Holds on a task's page.

While any of these holds stands, no sentinel rule moves the task until the hold is cleared (see Tasks § Holds). A person can still move the task, and so can a worker, as described above. A worker can also change the hold to advisory and then clear it.

A limit and a dependency pause the task

A WIP constraint at its limit refuses a task entering the capped stages. This clears on its own as other tasks leave — but it will not move until they do, unless an admin forces the transition.

An uncleared dependency stops the blocked task's own assignment rules and inbound dispatch-worker rules from starting a pool for it until the blocking task reaches the state the edge waits for — resolved or completed. Three things still start a worker on it before that: a scheduled rule that has the task as its subject, a parent-target assignment rule at a stage one of its children enters (above), and an assignment of the task to a pool by a person or a worker (above). If you need the task to wait, check that none of these applies to it.

The short version

Cadence will lease, work, and advance a task on its own when a stage is configured for it and its conditions are met — and an agent pool runs work only while its version's LLM vendor has a token Cadence can serve. It will not invent a passing check or clear a blocking hold, and no sentinel rule moves a task on a workflow that has auto-advance turned off. A worker, though, can move a task or change its hold whenever its lease reaches the task, whatever the workflow's settings. Where a step needs a person, Cadence holds the task and waits — it does not pretend the work is done. Whether a worker waits too depends on what you tell it to do.

On this page