Upgrading the agent on ZHZN cabinets, remotely

For whoever publishes and aims an agent release: a platform operator or whoever holds the release-signing key. This is the most dangerous write surface in the product. Everything on it ends with code running on machines that take card payments, in venues nobody is standing in, and there is no undo button that does not involve a van.

So the mechanism is built to make the catastrophic action awkward and the safe one automatic, and this runbook is written in that order: the key material first, then publishing, then aiming, then reading whether it worked, then stopping it. Read §8 Aborting before you need it.

The cabinet-side half — installing the key and the boot guard in the first place — is First install at the factory §2.2 step 4 and §2.5. A cabinet built without those two files can never be upgraded remotely, and no amount of publishing changes that.

Everything here is traceable to code, and every claim in it is either asserted by tests/test_zhzn_agent_upgrade.py / tests/test_zhzn_agent_ota_install.py or demonstrated end to end in docs/evidence/zhzn-agent-ota/, against a real signed channel with a real broken release forced onto a real cabinet.

1. What the channel is, in one picture

your laptop, offline          CI                    kiosk-x cloud            the cabinet
─────────────────────         ──                    ─────────────            ───────────
release.pem  ──────────────►  Actions secret
  (never leaves)                │
                                ├─ build_agent_release.py
                                │    → zhzn-agent-1.4.0.tar.gz
                                │    → zhzn-agent-1.4.0.release.json (signed)
                                ├─ upload tarball to S3 ──────────────────────┐
                                └─ POST /api/v1/agent/releases ──► registry   │
                                     (signature verified here)       │        │
                                                                     │        │
release.pub ────────────────────────────────► cluster secret ─────────┘        │
     │                                                               │        │
     └──────────────────────── /etc/zhzn-agent-release.pub ──────────────────► verify
                                        (at the factory)             │        then fetch
                                                                     │
                                   PUT /api/v1/agent/rollout ────────┤
                                     (who may install it)            │
                                                                GET /zhzn/upgrade

Two separations carry the whole safety story, and both are deliberate:

• Signing is separate from hosting. A digest published beside an artifact proves the bytes were not corrupted; it proves nothing about who made them, because whoever can rewrite the manifest rewrites the digest with it. Here the cloud, CI and the bucket are three places where "can substitute the tarball" and "can substitute the digest" are the same capability — so the trust anchor is a detached signature verified on the cabinet against a key whose private half never enters any of them (app/agent_releases.py).
• Publishing is separate from deploying. A published release is offered to nobody until a rollout names it. The build that lands on main is never, by itself, the build the fleet installs. This is asserted directly (test_publishing_a_release_offers_it_to_nobody).

2. The signing key

2.1 Generating the fleet key — once, offline

Do this on a machine that is not a CI runner and not a cluster node.

openssl genrsa -out zhzn-agent-release.pem 4096
openssl rsa -in zhzn-agent-release.pem -pubout -out zhzn-agent-release.pub

# the fingerprint. Write it down; it is what a factory checks a batch against.
openssl pkey -pubin -in zhzn-agent-release.pub -outform DER | sha256sum

2.2 Where each half goes, and where it must never go

The private half must not be in the repo, in the cluster, or in the artifact bucket. That is not hygiene, it is the entire point: an attacker who owns any one of those three still cannot make a cabinet run their code.

2.3 Both ends fail closed, in opposite directions

A cabinet in the second state is invisible from the fleet view except as a version that never moves — which is why the factory runbook makes installing the key a numbered step rather than a note.

2.4 Rotating it

Rotation is a file write and a restart on the cabinet, not a reinstall: the agent reads the key fresh on every check rather than caching it at import (agent/ota.py public_key). But releases signed with the old key stop verifying, so the order matters: distribute the new public key to every cabinet first, then switch the cluster secret and the Actions secret, then re-publish anything you still need offerable under the new key. A cabinet missed in step one is a cabinet that has silently stopped accepting upgrades.

3. Building the artifact

CI does this. A developer runs the identical command to reproduce a release and check its digest against what was published — which is the only cheap way anyone ever verifies an artifact, and the reason the build is deterministic.

# in the zhzn-vending-kiosk repo
python3 scripts/build_agent_release.py --out dist --key-file release.pem

It writes two files and prints what you need for the publish call and for the upload:

artifact=dist/zhzn-agent-1.4.0.tar.gz
manifest=dist/zhzn-agent-1.4.0.release.json
version=1.4.0
sha256=12ca365d92307f3d…
size=184320
url=https://media.intelli-verse-x.ai/docs/agent/zhzn-agent-1.4.0.tar.gz
s3_uri=s3://ivx-kiosk-x-docs/agent/zhzn-agent-1.4.0.tar.gz

3.1 There is no artifact location to pass in, and that is the fix

--url-prefix used to be a required argument, and that is how the release channel broke while every test was green. The prefix is a contract between three parties who never speak to each other:

The cabinet and the backend had drifted onto kiosk-x-docs.s3.us-east-1.amazonaws.com/agent/ — a bucket in the AWS account being decommissioned, whose surviving copy (ivx-kiosk-x-docs) is not readable over the S3 REST endpoint at all — and CI stated a fourth spelling of the same string, without the trailing slash. Nothing compared them, because there was nothing that could: they lived in three repos.

Now the publisher imports the cabinet's constant instead of taking one, derives both the signed url and the s3_uri from it, and refuses to build if a caller passes a --url-prefix that disagrees. tests/test_zhzn_release_channel.py builds a release with the real script, publishes it through the real route, and hands it to the real cabinet check — so an edit to any one of the three sides goes red there instead of stranding the fleet.

The trailing slash is part of the value, not cosmetic: both checks are a startswith, so …/agent without it also admits …/agent-anything.

Things worth knowing before you run it:

• The version comes from the tree, agent/version.py AGENT_VERSION, unless you override it with --version. Bump it in the tree; do not rename a tarball.
• The tarball is byte-reproducible — sorted member order, zeroed mtimes, no owner names, no gzip timestamp. Same tree in, same digest out (test_the_release_packaging_script_produces_the_same_bytes_twice).
• It packages agent/ and emulator/, and emulator is not optional despite the name. agent/dispenser.py imports the CSM protocol library out of it, so an agent-only tarball produces a cabinet that cannot import its own dispenser — a failure invisible until after the swap has happened.
• The private key is read from a file and never placed on a command line, so ps on the runner does not show it.
• --migration and --min-from belong here, not on the publish call: they are inside the signature (§5, §6).

Upload the tarball to the s3_uri the script printed, then publish the record. Confirm the url is anonymously fetchable before you aim anything at it — the cabinet fetches it with no credentials at all (it strips its own gateway credential from that request on purpose), so an object only a signed-in caller can read is an object the fleet cannot update from:

aws s3 cp dist/zhzn-agent-1.4.0.tar.gz \
    s3://ivx-kiosk-x-docs/agent/zhzn-agent-1.4.0.tar.gz \
    --content-type application/gzip \
    --cache-control "public, max-age=31536000, immutable"
curl -fsSI https://media.intelli-verse-x.ai/docs/agent/zhzn-agent-1.4.0.tar.gz

4. Publishing

export BASE=https://api.kiosk-x.ai
export ADMIN_KEY=<full-admin API key>

curl -sX POST -H "X-API-Key: $ADMIN_KEY" -H 'content-type: application/json' \
  "$BASE/api/v1/agent/releases" \
  -d @dist/zhzn-agent-1.4.0.release.json

The manifest the packaging script wrote is the request body: version, sha256, size, url, signature, signatureAlg, and migration / minFrom when you used them. Full-admin only (app/routes/agent_releases.py:79-96) — an operator is refused even for their own cabinets, because publishing is remote code execution on machines that are not only theirs.

Then read the registry back:

curl -sH "X-API-Key: $ADMIN_KEY" "$BASE/api/v1/agent/releases"

4.1 What is signed, and what is not

The split is the rule the whole channel runs on: the signing key attests to what the artifact is, never to who should run it. What the bytes are, where they come from, and whether a migration may be stepped over are the publisher's assertions and are signed. Which cabinet gets an offer and when is cloud policy and is not — the agent treats those fields as a permission and never as a source of code or digests.

sequence is deliberately outside the signature: it is publish order, which depends on what else has been published and so cannot be known by an offline signer at build time. Signing a value the signer has to guess produces signatures that fail for reasons unrelated to authenticity.

Adding a field to SIGNED_FIELDS invalidates every previously published release. That is the correct and loud failure rather than a silently unverified new field — and the cloud's construction of the signed bytes and the agent's are asserted byte-identical (test_the_cabinet_and_the_cloud_sign_and_verify_the_identical_bytes), because two implementations of one canonical form is a fleet-wide outage waiting for someone to add a field.

4.2 The refusals, and what each one is protecting

Every one of these is a 400 unless noted, and the message is the remedy.

The prefix check is enforced twice, here and again on the cabinet against its own ZHZN_OTA_URL_PREFIX, so a cloud misconfiguration cannot aim the fleet elsewhere either (test_an_artifact_hosted_off_the_trusted_prefix_is_refused).

5. Migration releases and the ladder

The artifact is a whole tree, not a delta, so any version installs directly over any other and a cabinet offline for two months reaches current in one hop. That is the default because it is almost always right: each intermediate hop would be another restart and another chance to fail in a venue nobody is at.

The exception is a release that does something irreversible to on-disk state. Publish it with --migration and cabinets below it may not skip over it: the offer for such a cabinet becomes that release rather than the target, and the cabinet climbs the ladder one rung at a time through the migrations and jumps the rest (app/agent_releases.py _next_step).

Every offer says where it is in the climb:

So a cabinet's own log reads "1 of 3 on the way to 1.6.0" rather than looking like a rollout that keeps changing its mind. A cabinet handed isFinalStep: false will install, restart, prove itself, and be offered the next rung on its following check — you do not stage the intermediate hops yourself, and you should not aim a rollout at one.

A cabinet that has never reported a version (an agent build too old to say) is offered the target directly. Nothing can be concluded about what it may skip, and guessing an intermediate would be a fabricated answer — so its migrations are the publisher's to know about (test_the_version_the_cabinet_states_on_the_poll_decides_its_rung).

6. minFrom: the floor a release installs over

minFrom is the lowest version a release may be installed over. It is a floor the release needs but cannot create for itself — some other release has to put the cabinet on a rung at or above it first.

It is checked in three places, and that is not redundancy:

1. At publish time, because that is the only moment with a person present and the whole release set in view (§6.1).
2. When an offer is chosen, so a cabinet below the floor is stepped to a rung it can install from instead (_reachable_step). This applies to whichever rung is being offered, not only the last one — an intermediate migration carrying its own floor is the case that makes it matter, because it cannot be skipped.
3. On the cabinet, after the signature verifies and never before, because minFrom is inside the signed fields precisely so it is the publisher's assertion. A floor read from an unverified offer would turn this safeguard into a way for anyone who can rewrite an offer to talk a cabinet out of every upgrade it is ever sent (agent/ota.py check_offer).

Comparison is semantic, so 1.10.0 is above 1.9.0 — a string compare would wave that through — and a cabinet standing exactly on the floor installs normally. A refusal is only correct when the floor is genuinely above the running version.

6.1 The two shapes refused at publish time

Both come back 400, and both messages name the fix.

A floor at or above the release's own version — 2.0.0 with minFrom: 2.0.0, or minFrom: 2.4.0:

minFrom 2.0.0 is not below 2.0.0. A release that can only be installed from
itself or later is one no cabinet can ever be stepped up to, whatever else gets
published.

Refused unconditionally, in any publishing order: the ladder would have to stand the cabinet on a rung in the empty interval [minFrom, version), so nothing published later can rescue it.

A floor with a real gap below it — 2.0.0 with minFrom: 1.5.0, over a registry holding 1.3.0 and nothing at or above 1.5.0:

minFrom 1.5.0 has no published release between it and 2.0.0, and 1.3.0 is
published below it. Cabinets on the rungs beneath 1.5.0 could never climb to it,
so the offer would be an install guaranteed to fail in a venue — publish the rung
at or above 1.5.0 first.

Do exactly what it says: publish the rung, then re-publish this release.

The reason the decision lives at publish and not at offer time is that at offer time both available answers are bad — hand the cabinet an install guaranteed to fail on the box, or hand it nothing and leave it silently stuck with no signal. Neither is something a person can act on at 3am.

6.2 What still publishes, so this is not an outage

The check is deliberately narrow. All of these are ordinary releases:

• No floor at all. The common case.
• A floor at or below the oldest published release. Also common, because a floor usually names a build already in the field rather than a rung the ladder needs to insert.
• A floor naming a factory-installed build that was never published to the registry at all.
• Publishing out of ascending order — 1.5.0 with minFrom: 1.4.0 goes through before 1.4.0 exists. Until something is published below the floor, "not yet reachable" is a different claim from "unreachable". CI re-arming the registry after a redeploy publishes into an empty registry and is never caught.

The rule guards the publish path and nothing else; it is not an invariant over the registry's lifetime. A release that was reachable when published becomes unreachable if the rung beneath it is later removed (the peer-sync delete path and a sandbox reset can both do that). That residue is exactly why the cabinet checks the floor too, and why a below-floor refusal shows up as blocked in the fleet view (§7.2) rather than as a failed install.

7. Staging a rollout

7.1 Aim it, narrowly, then widen it

# 1. prove it on three cabinets. percent: 0 means nobody else.
curl -sX PUT -H "X-API-Key: $ADMIN_KEY" -H 'content-type: application/json' \
  -d '{"targetVersion":"1.4.0","phase":"rolling",
       "canary":["VM-2026-0001","VM-2026-0002","VM-2026-0003"],
       "percent":0,"note":"1.4.0 canary — lane-7 timing fix"}' \
  "$BASE/api/v1/agent/rollout"

# 2. wait for the canary to come up on the target (§7.2), then widen
curl -sX PUT -H "X-API-Key: $ADMIN_KEY" -H 'content-type: application/json' \
  -d '{"targetVersion":"1.4.0","phase":"rolling","percent":25}' \
  "$BASE/api/v1/agent/rollout"

A rollout with no canary and no percent reaches nobody, and that is the default on purpose. The dangerous instruction — every cabinet at once — has to be typed out as percent: 100; the safe one is what you get by leaving fields off (test_a_rollout_with_no_cohort_named_reaches_no_cabinet).

There is one rollout at a time, fleet-wide. Two concurrent rollouts of the same agent to overlapping cohorts is a state nobody can reason about at 2am, and the pin covers every "except this machine" case.

Percentage membership is stable: the bucket is derived from the machine code, not from a random draw, so a cabinet in the first 10% stays in the first 10% across pod restarts. A cabinet that flapped in and out of the cohort would be offered an upgrade, then not, then again — and every offer is a restart (test_a_cabinet_stays_in_or_out_of_a_percentage_cohort_across_polls).

A cabinet outside the cohort is told nothing at all rather than told about a version it may not have. An offer a cabinet must refuse is an offer an attacker can make it accept.

7.2 Reading coverage

Coverage comes from what cabinets report — software.agentVersion, written on register and on every heartbeat — not from tracking adoption separately. A second source for "what version is on that machine" is a second source that can be wrong.

curl -sH "X-API-Key: $ADMIN_KEY" "$BASE/api/v1/agent/fleet" | jq '.data | {
  target, total, onTarget, eligible, pinned, deferring, blocked, byVersion }'

deferring and blocked are separate columns because they need opposite responses, and folding them together is the conflation both were added to remove. Both look identical from the cloud otherwise: a version that does not move — which is also what a cabinet nobody aimed a rollout at looks like.

A deferring cabinet is doing the right thing. A cabinet only restarts for an upgrade when it is genuinely quiet: the vend lock free, nothing leased in the dispatch queue, an empty report spool, and 180 seconds since the last activity (ZHZN_OTA_QUIESCE_SECONDS). A busy machine therefore declines every offer, and reports what it is declining, for how long, and why:

curl -sH "X-API-Key: $ADMIN_KEY" "$BASE/api/v1/agent/fleet" \
  | jq '.data | {deferring, machines: [.machines[] | select(.deferring)]}'

{"version":"1.4.0","count":41,"seconds":88400,
 "reason":"only 12s since the last activity (need 180s)",
 "reportedAt":"…","firstReportedAt":"…"}

The remedy is a quieter window or a visit, never an override — there is deliberately no way to make a cabinet restart anyway. A restart between taking money and dropping product either keeps the money or loses the sale. A row that has been deferring for days is the signal to choose one of the two remedies.

A blocked cabinet is stuck and only you can unstick it:

curl -sH "X-API-Key: $ADMIN_KEY" "$BASE/api/v1/agent/fleet" \
  | jq '.data | {blocked, stuck: [.machines[] | select(.blocked)
      | {machineNo, agentVersion, needs: .blocked.minFrom,
         offered: .blocked.version, refusals: .blocked.count,
         since: .blocked.firstReportedAt}]}'

A row reads: this cabinet is on agentVersion, it is being offered blocked.version, and that release will not install below blocked.minFrom. The fix is a publish or a rollout change, never a visit.

Which duration to trust. count and seconds are measured in the agent process and restart with it, so a small number means "since this agent last started", not "since this began". firstReportedAt — on both blocks — is held by the cloud from the first beat carrying that block and kept for as long as the version and the floor do not change. That is the restart-proof answer to "how long has this cabinet been stuck", and it costs the cabinet nothing: persisting the counter on the machine would have put a durable write on an eMMC board to recover a number the cloud can hold for free.

Both blocks clear the moment the condition ends, so a stale warning never outlives the rollout that caused it (test_a_cabinet_stops_reporting_a_deferral_once_nothing_is_on_offer).

7.3 One cabinet, in detail

curl -sH "X-API-Key: $ADMIN_KEY" "$BASE/api/v1/machines/VM-2026-0001/agent" \
  | jq '.data | {agentVersion, pin, offer: .offer.version,
                 isFinalStep: .offer.isFinalStep, blocked, deferring}'

offer runs the same computation GET /zhzn/upgrade performs, so you can see what a cabinet will be offered without waiting for it to poll — which is roughly every six hours by default (ZHZN_OTA_CHECK_INTERVAL, 21600s). An operator can read this and /api/v1/agent/fleet for their own cabinets without asking us; the write surface is admin-only.

Which rung the cabinet is judged to be on comes from the agentVersion on its poll, with the version it last reported on a heartbeat as the fallback — so a cabinet whose build is too old to send the query parameter still gets a correct answer as long as it has ever reported one (app/routes/zhzn.py upgrade).

7.4 What the cabinet does with an offer, in order

Refusals above the download are the ones that cost nothing.

The journal is written before the swap, so a power cut between the two renames leaves an unproven upgrade the guard knows to reconcile rather than a half-swapped tree nobody knows to repair.

8. Aborting a rollout, and rolling back

Three tools, in increasing order of how much they undo. The first two are the cheapest calls on this surface, on purpose — an emergency brake nobody can find is not a brake.

8.1 Hold: stop the bleeding

curl -sX POST -H "X-API-Key: $ADMIN_KEY" -H 'content-type: application/json' \
  -d '{"reason":"canary 0002 is not reporting"}' \
  "$BASE/api/v1/agent/rollout/hold"

Cabinets already on the target keep it; nobody else is offered it. It deliberately does not roll anything back: whoever reaches for this at 2am has one fact — this rollout is going badly — and deciding what to do next is a separate act. It also does not need the target restating, so you do not have to look up a version number to act.

POST /api/v1/agent/rollout/resume puts it back to rolling. With no rollout in flight either call is 409 There is no rollout in flight to change the phase of.

8.2 Pin: exempt or walk back one cabinet

curl -sX POST -H "X-API-Key: $ADMIN_KEY" -H 'content-type: application/json' \
  -d '{"version":"1.3.0","reason":"1.4.0 stalls lane 7 on rev-B boards"}' \
  "$BASE/api/v1/machines/VM-2026-0001/agent/pin"

curl -sX DELETE -H "X-API-Key: $ADMIN_KEY" \
  "$BASE/api/v1/machines/VM-2026-0001/agent/pin"

A pin outranks the rollout in both directions: the cabinet is neither dragged forward past its pin nor left behind it. Pinning below what a cabinet is running is the deliberate rollback — the boot guard catches a build that will not start, and only a person can catch one that starts fine and behaves badly.

The reason is mandatory and must be at least four characters (A version pin needs a reason — it overrides the fleet rollout for this cabinet indefinitely), and the version has to be published (404).

8.3 Roll the fleet back

A fleet-wide walk-back is a rollout at the older version with allowDowngrade:

curl -sX PUT -H "X-API-Key: $ADMIN_KEY" -H 'content-type: application/json' \
  -d '{"targetVersion":"1.3.0","phase":"rolling","percent":100,
       "allowDowngrade":true,"note":"1.4.0 stalls lane 7 — reverting"}' \
  "$BASE/api/v1/agent/rollout"

allowDowngrade is required and is not inside the signature: it is cloud policy, and the agent treats it as a permission. Without it every published release would be a permanently live downgrade offer, which is the replay attack the flag bounds (test_an_older_release_is_never_offered_unless_a_downgrade_was_asked_for).

What a signature deliberately does not solve, stated rather than assumed: an attacker who controls the cloud response can still replay a genuine, older signed release to walk a cabinet onto a build with a known bug. allowDowngrade bounds that to "revert the fleet to a build we ourselves published and signed" — bad, visible in the coverage view, and recoverable. Closing it completely needs a monotonic counter with a trust anchor the cloud cannot rewrite, which is a larger change than this channel warrants today.

8.4 The rollback you do not have to ask for

A cabinet is never left running a version it could not prove, and this is the row to internalise because it happens with nobody in the venue.

systemd runs the boot guard as ExecStartPre before every agent start, including every Restart=always relaunch. It lives at /opt/zhzn-agent-guard/ota_guard.py — outside the install root, because the thing that reverts a broken tree cannot be inside the tree it reverts — reads the journal in the state directory, which the swap never touches, and:

It counts starts rather than trusting the process to report, which is the whole point: a crash loop increments the counter without the agent's cooperation, and the case that strands a cabinet — a version that dies on import — is exactly the case where the code that would perform the revert is the code that is broken.

What this looks like from where you sit: software.agentVersion on that cabinet goes back down. It is not a cabinet that lost its upgrade; it is a cabinet that rejected one. The reason is in journalctl -u zhzn-agent | grep ota:

[ota-guard] 9.2.0 failed to start 3 times — restored 9.1.0 from /opt/zhzn-vending-kiosk.old

Do not reinstall by hand. Hold the rollout, pin the cabinet, and fix the release.

Retention is bounded, which matters because these are 8–16 GB eMMC boards and a filled disk is its own outage, arriving months later on the machines nobody visits. Exactly one previous tree is kept — a second-oldest release is not a better fallback than the current one, only an older guess — the failed tree is deleted after a revert, staging directories are cleaned, and every downloaded tarball is removed from the state directory, including after a download that died mid-stream. Demonstrated after four installs and a revert in docs/evidence/zhzn-agent-ota/upgrade-and-revert.txt step 24: /opt holds only the guard and the live tree, and no artifact survived the install that used it.

9. A release, start to finish

The sequence with nothing skipped, for the first one you publish.

1. Once, ever: generate the fleet keypair, distribute the public half to the cluster and every cabinet, record the fingerprint (§2).
2. Bump AGENT_VERSION in the agent tree; merge.
3. CI builds the tarball and the signed manifest, uploads the tarball to the prefix, and POSTs the manifest to /api/v1/agent/releases (§3, §4).
4. GET /api/v1/agent/releases — confirm it is there with the digest CI printed.
5. GET /api/v1/agent/fleet — note byVersion before you aim anything, so you have something to compare against.
6. PUT /api/v1/agent/rollout with three named canaries and percent: 0 (§7.1).
7. Watch those three reach the target. Give them longer than you think: a cabinet in a busy venue legitimately defers, and deferring says so (§7.2).
8. Widen: percent: 25, then 50, then 100. Re-read coverage at each step and check blocked is 0 — a nonzero blocked means some cabinets can never take this release and need a rung published first (§6).
9. If anything looks wrong: hold first (§8.1), then decide.
10. When onTarget == eligible and deferring is 0, set phase: "complete".

10. What is not yet automated

Honest gaps, each one something this runbook would otherwise want to tell you to do and cannot.

1. Nothing promotes a green canary. There is no automatic widening from a healthy canary to the fleet, deliberately — nothing should widen a rollout at 3am without a person looking at the coverage view first. Every percent step in §9 is a human typing a number.
2. There is no route that deletes a release. Immutability is the point, but it means a release published in error stays in the registry; the remedy is to never aim a rollout at it. (The peer-sync delete path and a sandbox reset can remove one, and doing so can strand a floor — see §6.2.)
3. Coverage is only as current as the last heartbeat. A cabinet that is dark reports nothing, so it shows its last known version indefinitely. Read status and agentReportedAt beside agentVersion, or an offline cabinet reads as one that has not taken the upgrade yet.
4. A cabinet with no signing key installed is invisible as such. It refuses every offer with a local log line and reports no blocked block, because the refusal is above the point where a floor is known. It looks exactly like a version that will not move (§2.3).
5. Replaying an older signed release is bounded, not closed (§8.3).
6. There is a legacy fleet-wide manifest, and you should not use it. KIOSKX_ZHZN_OTA_MANIFEST (app/config.py:108) serves one hand-written offer to every cabinet, and exists only for a deployment with no registry at all. A registry with any release published in it wins outright — the fallback is never consulted, because falling through to it would hand a held rollout, or a pinned cabinet, exactly the upgrade those mechanisms exist to withhold (app/routes/zhzn.py upgrade). One manifest for everyone means the only rollout available is "all cabinets at once", which is the case this whole channel is built to make hard.

Also: First install at the factory · Receiving a cabinet · APK rollouts for the Reyeah/Android half · Vending manufacturers manual