released 2026-08-09
changelog
Every hal0 release, newest first — breaking changes, new features, and fixes, straight fromCHANGELOG.md. No hand-maintained second copy to drift.
Get hal0
hal0's 1.0 line is at release-candidate stage — RCs ship on the preview channel ahead of GA.
released 2026-07-13
One command on a fresh Linux x86_64 host (Python ≥3.12). AMD (ROCm/Vulkan) is the first-class path; NVIDIA/CUDA is experimental.
curl -fsSL https://hal0.dev/install.sh | bashSelf-update verifies a cosign-signed tarball and swaps the current symlink atomically. Three channels — the 1.0 release candidates ship on preview:
hal0 update --channel stablehal0 update --channel previewhal0 update --channel nightlyReverts in one step: hal0 update --rollback. See the update & rollback guide.
- Highlights
- Privileged-seam hardening. The
hal0-systemctlroot wrapper now allow-lists the content of everything it writes — systemd drop-ins (#1718), quadlet bodies with the rawwrite-unitverb removed (#1748), andPodmanArgs=pinned to the flags providers emit, closing a--runtimehost-exec path (#1759). The update staging path closes a verify→extract TOCTOU (root-only stage dir + digest pinned to the extraction handle, #1745), reaps orphaned staging/quarantine trees (#1755), never dereferences symlinks when applying ownership (#1743), and only trusts a root-owned config for the releases-URL override (#1750). Slot flag denylisting canonicalizes the--flag=valueform (#1746) and rejects malformed shell quoting at save time (#1737/#1749). - Fresh-install & update robustness. The installer waits on the dpkg/apt
lock instead of dying (#1733), runs AppArmor remediation before the podman
preflight gate (#1728), preflights git for Hermes (#1727), and applies the
model-layout migration as its final model step (#1732). An up-to-date
hal0 updatenow converges an outstanding v1.0 profile-catalog reset instead of going silent (#1585/#1757); staged trees get their perms normalized after extraction (#1725); everyhal0.tomlwrite goes through one serialized read-modify-write path (#1724). - Capabilities & settings UI. A unified AI Capabilities page (TTS, STT, embeddings, reranking, image generation, NPU) with a canonical rerank slot (#1747), a memory reranker model picker (#1781), and per-panel probe-failure surfacing instead of a silent grey Save (#1774).
- Dashboard redesign. The main and Benchmarks dashboards now share the hal0.dev design-system chrome (#1764/#1768), with benchmark trigger/config/ outcomes/regressions surfaced (#1744) and assorted tile/row polish (#1729/#1779/#1780/#1782).
- Benchmark system overhaul. Adopts GuideLLM, llama-benchy and
tool-eval-bench (#1765), adds record telemetry + regression journaling
(#1766) and shareable result bundles (#1758), folds the shell harness into
Python behind a validate-and-exec
benchctlshim (#1761), and makes the measurements truthful and the pipeline reliable (#1736 and follow-ups). - Slots. Per-slot profile-flag divergence overlay with a cross-device picker (#1639), and a live slot now restarts when apply changes config but not the model (#1770).
- Docs. Quick-start and migration guide rewritten for v1.0 (#1731).
Added- Settings: new unified AI Capabilities page (TTS, STT, embeddings, reranking, image generation, NPU anchor) replaces the Voice / Image Generation / NPU pages; old
#settings/voice|imagegen|npulinks redirect. First UI for the embed/rerank capability selections.
Security- The
hal0-systemctl write-quadletallow-list now pinsPodmanArgs=to the exact flags hal0's providers emit —--group-add <gid>,--security-opt <token>(GPU/llama-server),--ipc <mode>(comfyui),--ulimit <name=v[:v]>(flm/NPU) — instead of accepting any non-empty value (#1759). Podman's quadlet generator copiesPodmanArgs=verbatim into the generated root unit'spodman runargv, so a persistent flag like--runtime <path>or--hooks-dir <dir>made podman exec an attacker-named binary as root with no container involved — a direct host-exec primitive reachable from the unprivilegedhal0account, exactly what #1740 set out to close. All shipped providers keep loading; only out-of-list flags are refused. Minor breaking (deprecated feature): the free-formextra_argsescape hatch (already loggingcontainer.extra_args_deprecated) is honoured on a hal0-service install only for flags in that list — an out-of-list flag now makes the slot refuse at the root seam. Move those to typed Quadlet keys in ahal0-slot@<token>.container.d/drop-in. - The privileged update stage no longer downloads, verifies and extracts the
release tarball out of the service-writable
/var/lib/hal0/cache/<version>/(#1738). That directory ishal0:hal00o2775with no sticky bit, and the post-stage ownership restore hands it back to the service account, so a process compromised ashal0could substitute its own tarball in the window betweencosign verify-blobexiting and root callingtarfile.open— and root would extract andpip installit. Staging now happens in a root-only0700directory created under the install root and destroyed when the stage ends, and the authenticateddigest_sha256is re-derived from the very file object the archive is read out of, so the bytes cosign accepted are provably the bytes that land on disk. The cache directory now holds only the verified manifest thatcommit()re-reads. - An interrupted extraction no longer wedges a version permanently. A killed
extractallused to leave the destination holding just the un-flattenedhal0-<version>/prefix, which the "is this a prior hal0 install" check did not recognise, so every retry refused to extract over a non-empty directory — unrecoverable for the unprivileged daemon, since the wedged tree is root-owned. Extraction now drops a.hal0-stagingsentinel for its duration and the check also recognises an un-flattened prefix directory, so an incomplete tree is quarantined and retried. The.stale-<ts>quarantine directories, which accumulated one whole install tree per retry and were never cleaned up, are now reaped: the newest three are kept for recovery and anything older than 30 days goes. - The root-only
.stage-<version>-XXXXXXstaging directory (#1738) is now also reaped on the next apply. Its teardown runs in afinally, which a SIGKILL / OOM / power-cut skips, leaking a full release tarball under the root filesystem per killed stage. Orphans older than an hour — comfortably above the 30-minute privileged stage timeout, so a live concurrent stage is never touched — are swept. (#1754) - The privileged
hal0-systemctlwrapper now allow-lists the content of the two systemd drop-ins it writes as root (write-gateway-dropin,write-hindsight-dropin). Both verbs take their whole payload on stdin, and the sudoers grant lets the unprivilegedhal0service account run the wrapper as root — so a process compromised ashal0could previously supply a[Service]fragment withUser=rootand a replacedExecStart=, then use the wrapper's owndaemon-reload+svc-restartverbs to run it as root. Validation is parsed on the root side of that boundary: only#comments, blank lines, a single[Service]header and a closed set of directives per verb (Environment=HINDSIGHT_API_LLM_MODEL/…_TIMEOUT; anEnvironmentFile=confined to the hal0 secrets vault) are accepted, each with a pinned value charset. Anything else — any other section or directive, a line continuation, a control byte, leading whitespace — is rejected with a loud error and nothing is written. The wrapper persists the validated reconstruction rather than raw stdin, and a new side-effect-freecheck-dropin <gateway|hindsight>verb dry-runs the allow-list. - The same wrapper's
write-quadletverb now allow-lists its body too, and the raw-stdinwrite-unitverb is removed. A.containerfile is not just a container spec — podman's quadlet generator copies its[Unit],[Service]and[Install]sections verbatim into the generated system unit — so an unvalidated body meant a process compromised as the unprivilegedhal0account could write[Service] ExecStartPre=/bin/sh -c '…', then use the wrapper's owndaemon-reload+startverbs for an unconditional root exec. The root side now accepts only the sections, directive keys and value shapes the one renderer (_render_quadlet_from_plan) emits, in that order, each at most once, and writes its own validated reconstruction.write-unithad no producer left after the Quadlet migration and no render contract to allow-list against, so it was deleted rather than guessed at. A new side-effect-freecheck-quadlet [<slot-id>]verb dry-runs the allow-list. Scope, stated honestly: this closes the direct host-side exec primitive; it does not make anhal0-account compromise non-root-equivalent, because slots run under rootful podman and a slot's[Container]section legitimately carries config-derivedImage=/Volume=/AddDevice=/PodmanArgs=/Exec=values. Containing that means running slots rootless or pinning mount roots.
Fixed- Enabling the rerank capability now creates/loads the
rerankslot the dispatcher actually routes/v1/rerankingsto (wasembed-rerank, which nothing routed to);embed-rerankresolves as an alias. hal0 updateon an already-current box now converges an outstanding one-shot v1.0 profile-catalog reset instead of printing "nothing to apply" and hiding it (#1585). The reset ridescommit(), but a box updated 0.9.8→1.0 ran commit under the old daemon, which had no reset — so it landed converged-except-for-this and then went silent./api/updates/checknow carries a read-onlyprofile_resetsnapshot, and a new localPOST /api/updates/converge-profilesruns the reset with no download or swap. The up-to-date CLI path consults the snapshot: it converges (prompting or honoring--yesfor the consent-needing case), converges silently when there's nothing to lose, and otherwise reports the reset as outstanding and exits 2 (up to date, convergence outstanding) rather than exiting 0 in silence.
AudiencePreview-channel operators validating the 1.0 line ahead of GA, and fresh installs that want the current build. This is the release-candidate carrying the privileged-seam hardening cluster (#1738/#1740/#1750/#1759); it is the recommended pre-GA validation target. Boxes on the stable channel are not offered this tag.
Supported upgrades1.0.0-rc.3→1.0.0-rc.4viahal0 update(preview channel). The GitHub release asset URL (https://github.com/Hal0ai/hal0/releases/download/v1.0.0-rc.4/preview.json) works end-to-end for install and update.1.0.0-rc.2/1.0.0-rc.1→1.0.0-rc.4directly, same mechanism.0.9.8→1.0.0-rc.4in place — re-run the installer orhal0 updateon the preview channel; the R5 migration set applies (see the 1.0.0 changelog section). The profile-catalog reset that used to defer silently on this transition now converges on the first post-updatehal0 update(#1585).- Older than 0.9.8: step through 0.9.8 first.
Known issuesThe 0.9.8 CLI's spurious end-of-update
ConnectError(the pre-1.0 client dies when the apply restarts hal0-api under its status poll, even though the update succeeds — verify withhal0 --version) and the first-bootunattended-upgradesdpkg race (#1584) carry forward. A box still on rc.1/rc.2 whose venv predates #1663 is not offered a newer tag by the passive check —hal0 update --target <version>is the recovery path (#1715). The rc.1 profile-catalog-defer item is resolved this release (#1585).Operator migrations- Releases-URL override (#1750): a
file://HAL0_RELEASES_URLis no longer accepted from the service-owned/etc/hal0/api.env— put it in the root-owned/etc/hal0/update.conf(root:root 0644) instead.https://overrides inapi.envare unchanged, so boxes using the GitHub asset URL need no action; onlyfile://-staging boxes move the line. - Deprecated
extra_argsat the seam (#1759): a slot whose deprecated free-formextra_argscarries apodman runflag outside--group-add/--security-opt/--ipc/--ulimitis now refused at the root seam on a hal0-service install. Move those flags to typed Quadlet keys in ahal0-slot@<token>.container.d/drop-in. Shipped providers are unaffected. - The R5 operator-run migrators (slot-flag fold, id-keying) are unchanged from rc.1.
RollbackRelease tarballs are immutable and cosign-signed; roll back by re-installing the previous tag (
v1.0.0-rc.3) from its GitHub release. No rc.4 change writes a state shape rc.3 cannot read — the profile-catalog reset stamps the sameschema_version = 2rc.3's commit path already used. - Privileged-seam hardening. The
The hardening candidate: rc.2 plus the full yield of the pre-1.0 review — 38 changes, all defect fixes, dependency bumps, and doc corrections; no new features and no breaking changes. Every fix landed with a red-first test through its own reviewed PR.
Highlights- The update path works end-to-end on real hosting. The release-manifest fetch follows redirects, so
HAL0_RELEASES_URLcan point straight at a GitHub release asset (#1692); the privileged stage seam resolves the same operator-configured URL as the daemon instead of silently falling back to the default (#1700);hal0 updaterefreshes the privileged sudo wrappers on every activate, so seam fixes now reach existing boxes without aninstall.shre-run (#1720); and a signal-killed self-restart reports as ambiguous instead of a spurious failure (#1713). - Id-keyed boxes are first-class. The capability enable/disable lane resolves slots bilingually, so a box migrated with
hal0 slot migrate-id-keyingno longer strands a disabled slot bound-and-routable (#1681); the NPU trio's shadow drawer and scalar config writes are shape-aware to match (#1708, #1719). - Memory surfaces tell the truth. The Operations panel reads the engine's real envelope (#1674), the bank-delete preview reports the real blast radius (#1678), per-agent stats are scoped to the agent (#1679), list pagination no longer skips rows under ACL filtering (#1697), and extraction-slot changes actually reach the hindsight-api daemon through the privileged seam (#1682).
Fixed- Updater: release-manifest fetch follows redirects (#1692); the root-side
stage seam honours
HAL0_RELEASES_URLfrom root-owned config (#1700); privileged wrappers refresh on every activate (#1720); config migrations no longer TOML-dumpNone(#1686); a signal-killed self-restart is reported as ambiguous rather than failed (#1713). - Slots: capability lane resolves id-keyed and name-keyed layouts (#1681);
renaming a static seed tombstones the vacated name instead of re-seeding
a duplicate (#1698);
install.sh's seed loop honours.seed-tombstones(#1693);merge_slot_configwrites scalars into[slot]on nested-shape files instead of a root key nobody reads (#1719); legacyload_asr/load_embedkeys fold into the lifted[npu]table (#1710). - Models: FLM probe capabilities survive dedupe (#1695) and the legacy
<tag>-FLMid stays resolvable (#1706); a re-pull'scontext_lengthwins over its own prior stamp (#1707); ROCmFPX quant detection runs before the generic regex on the lazy path (#1694). - Memory: operations envelope (#1674), bank-delete preview keys (#1678),
per-agent stats scoping (#1679), ACL-aware list pagination (#1697),
project-scope dedupe on both sides (#1702),
bank=private__<agent>resolves through the read resolver and fails closed (#1711); extraction propagation reconciles drop-in drift on already-broken hosts, survives request cancellation without a stale clobber, and shares one hal0.toml write lock with the settings route (#1717). - Settings/config:
[brain_chat] tool_model = "off"(#1672) andactivity.max_rows = null(#1704) survive the save/load round-trip; the 20 registry-gap keys are classified in the apply plan (#1712). - Brain: native tool attach checks the serving slot's runner first and reroutes instead of 500ing on a pre-tool-support image (#1699).
- UI: MTP eligibility mirrors
defaults.mtppost-tag-retirement (#1671); the MOE filter chip keys on a real signal (#1701); NPU STT/Embed pills reflectnpu_modality_active(#1696); field-info popups portal-render to escape clipped panels (#1709); dead #1632 tag helpers removed (#1705). - Release tooling:
update-toolbox-digests.shhonours digest-pinned refs instead of nulling them (#1703). - Deps: cryptography 50.0.0, js-yaml 4.3.1 (both dependabot highs, #1684,
#1680). Docs:
docs/README.mdcanonical-home flip (#1714) and thehal0 update --targetrecovery path for rc.1 boxes jumping to GA (#1715). - Field-info popups (the
(i)descriptions in the slot/model drawers and settings pages) no longer collapse to one word per line — the popup was shrink-to-fit against its tiny icon wrapper; it now sizes to its text up to the 280px wrap width. The slot drawer's Auto-load and Pin descriptions now also spell out the difference: Auto-load only controls boot start, Pin only controls residency (eviction exemption + guarded unload/delete). - Slot drawer lifecycle controls consolidated: Auto-Load moved from the Model section up into the drawer header next to the Pinned toggle (same instant-apply style — the pair now reads as one story: Auto-Load = when it starts, Pin = whether it may be stopped), and Eviction priority moved into the Advanced disclosure. Two fewer always-visible rows in the drawer body, and NPU slots — which have no Model section — gain the Auto-Load toggle.
- Memory extraction-slot / LLM-timeout changes now actually reach the
hindsight-api daemon (#1641). The propagation wrote
/etc/systemd/system/hindsight-api.service.d/extraction-model.confdirectly and called baresystemctl, but hal0-api runs as the unprivilegedhal0user — the write wasEPERMand the restart would have hit polkit, so on every standard install the drop-in was never created whilehal0.toml(and the dashboard) reported the new slot as applied. The write,daemon-reloadand restart all route through the existinghal0-systemctlseam now (newwrite-hindsight-dropinverb — fixed literal path, body on stdin, no sudoers change: the grant is pinned to the wrapper binary). The propagation also runs off the event loop, so a hindsight-api cold start no longer blocks the API for the length of the restart;hal0.tomlis persisted before it, and the whole read-modify-write is serialised, so a disconnect or a concurrent save can no longer leave the recorded slot and the running daemon disagreeing. Upgrading in place refreshes the wrapper only on aninstall.shre-run; until then the failure is loud (propagation.error) instead of silent.
AudiencePreview-channel operators validating the 1.0 line ahead of GA, and fresh installs that want the current build. Boxes on the stable channel are not offered this tag.
Supported upgrades1.0.0-rc.2→1.0.0-rc.3viahal0 update(preview channel). With #1692/#1700 in rc.3, this is the last upgrade that needs a directly-servable manifest URL workaround on the rc.2 side.1.0.0-rc.1→1.0.0-rc.3directly, same mechanism.0.9.8→1.0.0-rc.3in place — re-run the installer orhal0 updateon the preview channel; the R5 migration set applies (see the 1.0.0 changelog section).- Older than 0.9.8: step through 0.9.8 first.
Known issuesThe three rc.1 items carry forward unchanged (profile-catalog reset defers when coming from 0.9.8, #1585; the 0.9.8 CLI's spurious end-of-update error; the first-boot
unattended-upgradesdpkg race, #1584). Additionally: a box still on rc.1 or rc.2 whose venv predates #1663 will not be offered a stable tag by the passive check —hal0 update --target <version>is the recovery path (documented in #1715).Operator migrationsNone. All rc.3 changes are self-applying fixes; the R5 operator-run migrators (slot-flag fold, id-keying) are unchanged from rc.1.
RollbackRelease tarballs are immutable and cosign-signed; roll back by re-installing the previous tag (
v1.0.0-rc.2) from its GitHub release. No rc.3 change writes a state shape rc.2 cannot read.- The update path works end-to-end on real hosting. The release-manifest fetch follows redirects, so
Second — and intended final — release candidate on the road to 1.0.0. This tag is the preview-channel snapshot of everything documented in the 1.0.0 section of the changelog; a box on
1.0.0-rc.1picks up all of it viahal0 update.Highlights- Preview snapshot of the full 1.0.0 content: 180-tool admin MCP catalog, memory MCP parity with Hindsight 0.8.4, Moonshine CPU STT reinstated, slot
autoload+ evictionpriority, on-demand capability slots, per-slot profiles, and the hardened OpenWebUI/secrets posture. Full detail in the 1.0.0 changelog section.
BreakingNew since
1.0.0-rc.1(a box upgrading from 0.9.8 also gets the R5 set — see the 1.0.0 changelog section):- The experimental standalone browser MCP server is removed. (
hal0.mcp.browser_server, port 9178,HAL0_BROWSER_*env; detail in the 1.0.0 changelog section linked above.) - The
lru = trueeviction opt-in is retired — every non-pinned resident slot is now an eviction candidate, ordered by the newpriorityfield. On a box that never setlru = true, pressure and pre-load eviction go from inert to active;pinnedis the only exemption. (Detail in the 1.0.0 changelog section linked above.)
Migrations- Nothing new for a box already on
1.0.0-rc.1— slots with a bound model migrate toautoload = trueautomatically, so boot behaviour is unchanged until toggled. Boxes coming from 0.9.8 follow the 1.0.0 migration set (one-shotenabledsweep at first boot; operator-run slot-flag fold and id-keying migrators).
AudiencePreview-channel operators validating the 1.0 line ahead of GA, and fresh installs that want the current build. Boxes that track the stable channel are not offered this tag.
Supported upgrades1.0.0-rc.1→1.0.0-rc.2viahal0 update(preview channel).0.9.8→1.0.0-rc.2in place — re-run the installer orhal0 updateafter switching to the preview channel; the R5 migration set applies (see Migrations above).- Older than 0.9.8: step through 0.9.8 first.
Known issuesThe three known issues carry forward from
1.0.0-rc.1unchanged: the profile-catalog reset defers to the next v1.0-applied update when coming from 0.9.8 (#1585); the 0.9.8 CLI reports a spurious error at the end of a successful 0.9.8 → 1.0 update (verify withhal0 --versionandcurl http://127.0.0.1:8080/api/health); and first-boot installs can lose the dpkg lock race tounattended-upgrades(#1584, degrades with a remediation line). Detail in the 1.0.0 changelog section linked above.Operator migrationsNone for a box already on
1.0.0-rc.1. Coming from 0.9.8, the one-shotenabledsweep runs automatically at first boot; the slot-flag fold and slot id-keying migrators remain operator-run and unchanged from rc.1 (dry-run by default — back uphal0.dband the slot dirs first).RollbackRelease tarballs are immutable and cosign-signed; roll back by re-installing the previous tag (
v1.0.0-rc.1) from its GitHub release. Note theenabled-sweep caveat from the R5 set: pre-R5 code reads a missingenabledas true, so a rollback past R5 needs the config backup taken before upgrading.- Preview snapshot of the full 1.0.0 content: 180-tool admin MCP catalog, memory MCP parity with Hindsight 0.8.4, Moonshine CPU STT reinstated, slot
- Highlights
- The whole platform is agent-reachable. The admin MCP catalog went from 92 to 180 tools — services, ComfyUI, updater/doctor/health, hardware and request telemetry, slots and models long-tail, bench, activity, approvals, runner images, NPU load/unload — plus the full 26-tool memory surface (was 5) at feature parity with Hindsight 0.8.4.
- Slots start when you say so. New
autoloadsetting: binding a model no longer implies boot start. New evictionpriority(0-100) replaces the inertlru = trueopt-in, so memory-pressure eviction actually works on a stock box. - Voice is a device-keyed switch. Moonshine is back as the CPU STT engine in its own toolbox image;
cpuruns Moonshine,npuruns whisper-v3:turbo, GPU resolves to no STT engine instead of silently taking a llama chat profile. - Image Gen and Slots panes got their lifecycle right — state-typed engine indicators, a Stop that drives the GPU arbiter back to inference mode, dropdown-driven runner image/binary selection, and a
GET /api/slotspath that no longer multipliespodman inspectfan-out on wide boxes. - Documentation moved into this repo.
docs/is the source of truth and publishes to hal0.dev through a mirror workflow, with a restored, v1.0-reconciledgetting-started/section.
BreakingThe experimental standalone browser MCP server is removed. (
hal0.mcp.browser_server, port 9178,HAL0_BROWSER_*env.) It was never mounted or registered as a bundled server, and its shipped unit pointed at a retired path. Browser tooling is the agent's own concern — Hermes brings its own.The
lru = trueeviction opt-in is retired — every non-pinned resident slot is now an eviction candidate, ordered by the newpriorityfield. Memory-pressure and pre-load eviction used to only ever touch a slot that explicitly setlru = true; now every non-pinned resident slot is a candidate, ordered by the newpriorityfield. The key is still accepted in slot TOML but ignored, with a one-time deprecation warning — remove it and usepriority/pinnedinstead. Practically: on a stock box that never setlru = trueon anything, pressure and pre-load eviction go from inert (nothing was ever eligible) to ACTIVE the moment host memory gets tight. Also noteidle_timeout_s = 0never exempted a slot from pressure or pre-load eviction — it only ever disabled that one slot's idle-TTL path — and that distinction now matters more than it used to.pinned(plus the built-inagent/utility/npuanchors) is the only exemption from pressure and pre-load eviction.
The R5 breaking changes below shipped in
1.0.0-rc.1and apply equally to a box coming straight from 0.9.8 — the expected upgrade path into 1.0. Full rationale and code-path detail in the 1.0.0-rc.1 changelog section.- Launch flags, device, and chat-template moved off slots onto models.
A slot is just
(id, name, model, port, state);model.defaultscarries the materialized tune. Slot TOMLs with the old fields still load but are ignored at launch until you run the fold migrator (see Migrations). - The Honcho memory engine is removed; memory is Hindsight-only.
hal0_memory_*tools are renamed tohindsight_*(old names kept as aliases this release). No data carry-over path exists. SlotConfig.enabledis gone — a bound model is the activation signal.PUT /api/slots/{name}/config {"enabled": …}now returns 400slot.removed_key_denied; the boot migration sweeps the key from slot TOMLs (see Migrations).PUT /api/slots/{name}/configno longer has a lifecycle side effect. Stopping a slot isPOST /api/slots/{name}/unload(409slot.pinnedon pinned slots,?force=truebypass).- The NPU-exclusivity 409 moved from the toggle to the model write.
Configuring a model on a second
device=npuLLM anchor returns 409slot.npu_exclusivity_violation; model-less NPU LLM slots coexist freely. - NPU trio dispatch reads the anchor's
[npu]table, not the shadow slots' own flags.flm-stt/flm-embedare display+dispatch records for the anchor's singleflm serveprocess; a modality that was never launched is no longer routable. [brain_chat] tool_modelis removed — it was never read. A config that sets it explicitly now fails validation with a clear error; the live[brain_chat] modeloverride is the real steering knob.- Deprecated surfaces are
HAL0-SUNSET-stamped for scheduled removal: the--backendflag (use--provider),SlotConfig.runtime/workers, thecogneeengine literal, and several legacy CLI aliases.
MigrationsThese carry forward from
1.0.0-rc.1for every box upgrading from 0.9.8; full detail in the 1.0.0-rc.1 changelog section.- Upgrade in place — re-run the installer (or
hal0 update); idempotent, non-destructive, never clobbers existing config. No reinstall. - Slot-flag fold (operator-run): the fold migrator moves slot tunes into model defaults — dry-run by default; back up
hal0.db+ slot dirs before applying. It refuses the whole run (no partial write) if slots share a model with divergent tunes — resolve each shared model (canonicalize or split) first. - The one-shot
enabledsweep runs automatically at first v1.0 boot;hal0 slot migrate-enabled-removalruns the same sweep on demand. (hal0.config.migrations.slot_enabled_removal.)enabled = falsewith a bound model → the model is cleared so the slot stays off; every other shape just loses the key. Idempotent; the CLI form is dry-run by default and safe live. - Slot id-keying (operator-run, optional): run
hal0 slot migrate-id-keyingin a downtime window (takes a pre-flight backup). The runtime reads either layout; the flip is deliberate and reversible. - A stale
[brain_chat] tool_modelkey inhal0.tomlno longer breaks config load.load_hal0_configdrops it before validation on every load path. - Disabling a capability now clears the slot's model instead of writing
enabled = false. The pick survives incapabilities.tomland a re-enable rebinds it. - Honcho → Hindsight (only boxes that ran Honcho): no migration step — Honcho support was removed outright and Hindsight starts fresh.
- Rollback is one-way — restore from a config backup if you need the prior state.
Pre-R5 code reads a missing
enabledasTrue, so slots the sweep cleared come back model-less rather than re-enabled.
AddedAdmin MCP catalog expanded from 92 to 180 tools — the full platform management surface is now agent-reachable: services lifecycle (
service_list/service_health+ gatedservice_action), ComfyUI (status/workflows reads, gated switchover/pin/launch/cancel/restart), updater/doctor/health/features reads (updater_state,updater_check,doctor_report,health_system, …), hardware and request telemetry (slot_stats,request_metrics,npu_occupancy,power_stats,throughput_history, …), slots and models long-tail (slot_config,slot_voices,hf_search,model_validate,model_health_check), bench plan/results reads + gatedbench_run,journal_snapshotand activity reads,approval_list(the brain can now tell the operator what's pending), runner images, NPU backend load/unload, and MCP self-management.slot_create/slot_editparam hints now advertiseautoloadandpriority. The admin mount also delegates the full 26-tool memory surface (previously 5). Brain chat reaches every non-excluded new tool through the shared dispatch core.Profiles:
POST /api/profiles/generate(admin MCP:profile_generate) drafts a profile from a registered model or a HuggingFace repo — capability classification, device fit for the local host, and seed selection reuse the install-time heuristics, with an optionaluse_llmpass that summarizes the model card through the local utility slot and degrades to heuristics when inference is down. The draft is a portable envelope ready for the existing create/import flow; the catalog is never written.hal0 doctor allgains two MCP preflights:mcp_mounts(liveinitialize+tools/listagainst/mcp/adminand/mcp/memorywith the configured agent token) andhermes_mcp_auth(the rendered Hermes config carriesAuthorizationwhenever the box requires auth). A 401 now fails doctor with the repair command instead of silently breaking every agent.Memory MCP surface brought to feature parity with the live Hindsight 0.8.4 server:
memory_reflect(LLM-backed synthesis over memory),memory_curate/memory_history(the non-destructive "this is wrong" correction path — edit or reversibly invalidate a single fact), mental model tools (memory_mental_model_list/_get/_create/_update/_delete/_refresh), directive tools (memory_directive_list/_get/_create/_update/_delete), async-operation tools (memory_operation_list/_get/_cancel/_retry— retain is async by default, so these are the poll target formemory_add'soperation_id), and bank introspection (memory_tags_list,memory_bank_stats,memory_bank_consolidate).memory_addgainedentities/observation_scopes/strategy/update_mode/sync(Hindsight's fullRetainRequestitem shape) and now surfacesoperation_ids/items_countinstead of dropping them.memory_recallgainedtag_groups/budget/prefer_observations/include/query_timestamp/min_scores, and its results now carry the engine's native per-result relevance score plus optionalentities/chunks/source_factsresponse enrichment — a stale comment claiming "Hindsight recall returns no numeric score" is fixed.memory_searchgainedtag_groups/min_scores. Every new destructive tool (memory_mental_model_delete,memory_directive_delete) goes through the same operator-approval gate as a bulkmemory_delete; no destructive bank-level operation (delete/clear a whole bank) is exposed over MCP.Moonshine reinstated as hal0's CPU STT engine, packaged as its own toolbox image (
hal0-toolbox-moonshine:v1).voice.sttis now a device-keyed engine switch exactly likevoice.tts:cpuruns Moonshine,npuruns whisper-v3:turbo via the FLM trio, and GPU devices resolve to no STT engine (previously a fall-through bug handed thesttslot the wrong llama chat profile). Moonshine's weights are operator-staged under the model store and preflighted at slot spawn, failing loudly by name (slot.weights_missing) instead of 500ing on first request. This supersedes the[v0.2.0]"Moonshine STT retired in favour ofwhisper.cpp" entry below — that justification never held, sincewhisper.cppnever shipped as a standalone CPU service. Seedocs/adr/0001-moonshine-cpu-stt-reinstatement.md.Image Gen pane: proper engine lifecycle controls and running indicators. The header pill is state-typed off the live engine (stopped / starting / running / generating·% / error) with matching colors, and a Stop button appears while the engine is up — it drives the GPU-arbiter switchover back to inference mode (restoring the LLM slots) and then unloads the img slot so the container actually goes down. Start/Restart/Logs unchanged.
Slots: explicit
autoloadsetting — a slot starts at boot only whenautoload = true(slot drawer toggle). Binding a model no longer implies boot start; existing slots with a bound model migrate astrue, so upgrade changes nothing until toggled.Slots: eviction
priority(0–100, default 50, drawer field) — memory pressure and pre-load eviction unload the lowest-priority slot first (least-recently-used as the tie-break within a tier).pinnedstill exempts a slot entirely.
ChangedSlot drawer: the Profile field moved into the Model group, directly under the model select (it rides the model choice). NPU slots keep a standalone Profile group since the capability matrix replaces the Model group there.
Slot drawer: the Runner Image field is a dropdown of the runner-image catalog (the same registry the Runtimes page shows) instead of a free-text input; a "Custom image ref…" option keeps the debug/A-B/rollback escape hatch. Picking an image repopulates the Runner Binary dropdown with the binaries that image ships — dual-binary images (e.g. the shared ROCm/Vulkan image) offer both, single-binary images hop the selection to their sole binary.
FixedHermes bootstrap MCP wiring actually works now: the seed TOML never declared the builtin
[mcp.servers.*]blocks so the allow-list silently skipped wiring both servers, and the post-wire live probe double-appended/mcpand 404ed — together the provisioning-time handshake had never succeeded. Bootstrap also injectsHAL0_MCP_TOKEN(0600) into the agent driver env and rendersAuthorization: Bearerinto the Hermes MCP client config whenever the box has auth enabled, and refreshes it on--repairafter a key rotation.ComfyUI img slot reliability: the provider now creates its bind-mount data dirs before spawn (a missing tree crash-looped the container with podman exit 125), slot readiness waits on ComfyUI's real health probe (
GET /system_stats) instead of 404-polling the llama-style/healthfor the full 180 s deadline and wedging the slot in WARMING, and the fail-watcher's health probe delegates to the ComfyUI provider so a READY img slot is no longer struck to ERROR seconds after coming up.GET /api/slotslatency cut sharply on wide boxes (#1507 follow-up): the per-slot probe no longer runspodman inspectfor stopped slots,podman image inspectanswers are TTL-cached per image ref, and the whole snapshot is served single-flight with a 2 s TTL (any slot mutation invalidates it immediately) so overlapping dashboard polls stop multiplying the subprocess fan-out.Slots page no longer stalls while the activity log backfills: the SSE stream replayed the full 1000-row durable backlog one frame at a time on every fresh connect, and the pane re-rendered once per frame. The stream now takes a
limit(the pane asks for its 200-row ring cap) and the client coalesces frame bursts into one render per 50ms window.hal0-brain tool-bearing requests no longer 500 with "Unknown (built-in) filter 'min'": the GGUF-embedded chat template uses the
|minfilter, which llama-server's jinja engine (minja) lacks. A corrected template (hal0-brain-sft.jinja) now ships bundled, and the curated catalogue stamps it into the model'sdefaults.chat_templateat pull time so fresh installs launch with--chat-template-fileinstead of the broken embedded template.
Known IssuesThese carry forward from
1.0.0-rc.1. The first two concern upgrading from 0.9.8 — the path most existing boxes take into 1.0; the third affects fresh installs; the fourth affects boxes that skipped straight fromrc.1to this release.- The profile-catalog reset does not fire during the 0.9.8 → 1.0 update itself (#1585, still open). The update's commit phase runs inside the old (0.9.8) daemon, which predates the reset — so an upgraded box keeps its
profiles.tomlandmeta.schema_version = 1until the next update applied by v1.0 code. Nothing is lost (the reset is biased against deletion), buthal0 updateon such a box reports "nothing to apply" without mentioning the outstanding reset. - Updating from 0.9.8 ends with a spurious error from the old client. The 0.9.8 CLI polls job status through the API it is restarting, treats the mid-restart connection refusal as fatal, and exits 1 after the update has in fact applied. The fix (#1540) ships in the v1.0 CLI, but the client driving a 0.9.8 → 1.0 update is by definition the old one. Verify the real outcome with
hal0 --versionandcurl http://127.0.0.1:8080/api/health. - First-boot installs can lose the dpkg lock race to
unattended-upgrades(#1584, still open) — the hermes-agent provisioning step degrades gracefully with a remediation line (hal0 agent install hermes) rather than failing the install. - A box still on
1.0.0-rc.1sees no update available for this GA release (#1663/#1640, fixed going forward but not retroactively — a running rc.1 daemon can't rerun its own fix). Its venv predates the prerelease-aware version comparison, so the old naive tuple fallback ranks1.0.0rc1above1.0.0andhal0 update//api/updates/checkboth report nothing to do. Boxes that updatedrc.1→rc.2first are unaffected — the rc-vs-rc comparison orders correctly even on the old fallback. If you're still onrc.1, pull this release directly withhal0 update --target 1.0.0, which bypasses theupdate_availablegate entirely.
Security- The
/mcp/memorymount is now CLIENT-tier (was ADMIN): memory-only agents no longer need the platform-admin key; the fail-closed namespace ACL and the operator-approval gate on destructive tools bound its blast radius./mcp/adminand any future/mcp/*mount remain ADMIN. Docs that still claimed "no built-in network auth" (pre-KB-1) are corrected everywhere.
The R5 rework puts the platform back together as a genuine 1.0: memory and Hermes finished, dead surface swept, launch flags re-homed onto models, the boot path split into named observable phases, and the installer hardened against real-world hosts (validated live on privileged/podman-4.9.3 and unprivileged/podman-5.7 substrates — clean install and in-place upgrade, exit 0). Full operator walkthrough:
docs/hal0-install-migration-guide.html.Highlights- FLAGS-own — flags belong to models. Launch flags, device, and chat-template now live on the model; a slot is just
(id, name, model, port, state). Profiles became copy-on-stamp templates. The argv resolver stops reading profile/slot overrides at launch;model.defaultscarries the materialized tune. The managed-arg denylist now also screens a model'sdefaults.extra_args(closing a bypass where a denied flag reached the container). - Memory is Hindsight-only. Honcho removed. Tools renamed to the upstream surface —
hindsight_recall/hindsight_retain/hindsight_reflect(oldhal0_memory_*kept as aliases);reflectimplemented; config moved to~/.hermes/hindsight/config.json(local_external). - Hermes brain-lane relocated into the api boot lifespan — persona seed + identity/brain-profile registration + self-report now run on every restart (the one hook fresh/update/dev share), reached in-process rather than over loopback HTTP.
- Install & runtime hardening — accurate container-runtime preflight diagnostics (keyring-quota exhaustion surfaced instead of a misleading nesting/keyctl message); GPU group resolution derives the render GID from the device node's owner (not the group name); slot units emit
StartLimit*in[Unit]so restart limiting applies; Hermes gateway install drops to thehal0user (no stray root-owned~/.hermes);hal0 agent status --jsonemits real JSON;reconcile_listenerswired into/api/ports. - Vulkan is the default for shared-APU models — benchmark-backed (RADV +40% prefill / +16% gen / −30% TTFT vs ROCm on Strix Halo).
- Boot split into 14 named, observable phases with a typed
BootState; drift-watch fixtures + ahermes-bumprunbook added. - Self-update works on the shipped
User=hal0posture — the new narrowhal0-updatesudo seam (check/stage/activate/discard, root-side cosign verification) makeshal0 updatestructurally possible on a hardened install for the first time (#1464), and theHAL0_UPDATE_SKIP_COSIGNescape hatch is hard-disabled on v1+ stable builds. - One-shot v1.0 profile-catalog reset — the updater converges pre-v1.0 boxes to the tuning-only catalog exactly once, gated on a
meta.schema_versionwatermark, with operator consent, a timestamped backup, and headless runs never deleting operator-authored profiles (#1574; see Known Issues for the trigger-point caveat on 0.9.8 upgrades). - Write-boundary enforcement — every in-process slot-TOML writer (stack apply,
SlotManager.create) now passes the same key-partition and hardware-flag guards the HTTP layer enforces, so a stack apply can no longer persist whatPUT /api/slots/{name}/configwould refuse. - OpenWebUI honors the box's bind choice — the second web surface on
:3001followsHAL0_BIND_HOSTinstead of hardcoding0.0.0.0, and hand edits toopenwebui.envsurvive installer re-runs (#1515/#1514/#1568).
Breaking- Launch flags/device/chat-template moved off slots onto models. Existing slot TOMLs with those fields still load (config is
extra="allow") and are ignored at launch until you run the fold migrator — but the slot-level surface is deprecated andHAL0-SUNSET-stamped for removal. - Honcho removed as a memory engine;
hal0_memory_*tools renamed tohindsight_*(aliases retained this release). - Deprecated surfaces machine-stamped
HAL0-SUNSET: v1.0.0for scheduled removal: the--backendflag (use--provider),SlotConfig.runtime/workers, thecogneeengine literal, and several legacy CLI aliases. SlotConfig.enabledis gone — a bound model is the activation signal (#1369, follows #1367). The field claimed to control "whether this slot is started on hal0 startup" and never did: boot autostart is the Quadlet[Install] WantedBy=hal0.targetstanza, which only exists becauseSlotManager.load()refuses to write a unit for a model-less slot — so[model].defaultwas always the real gate. Every one of the ~8 routability checks that consultedenabled(slots/routing.py, the four/v1listing helpers inapi/__init__.py,slot_view,api/routes/v1.py,dispatcher/_npu_common.py,npu_swap_status.py,omni_router/{filter,route_to_chat}.py) was immediately followed by anif not model_idgate; the shipped seeds had already hand-maintained the two signals 1:1 (all ten shipenabled = false+ no model pin, exceptbrain, which ships both). Two truths for one question is what forced the UI to render a "disabled but running" escape hatch and letPUT /configcarry a hiddenunload(). The single predicate now lives inhal0.slots.activation(is_activated/claims_npu_anchor/npu_modality_active).enabledis dropped from theGET /api/slotspayload and from theSlotTS type;PUT /api/slots/{name}/config {"enabled": …}is now a 400slot.removed_key_deniednaming the replacement, rather than silently persisting inert debris (the same treatmentbackendgot). Untouched: theenabledfields onUpstream, MCP clients, providers, metrics, memory, auth, and the[npu]modality toggles — different models entirely.CapabilitySelection.enabledalso survives incapabilities.toml; only its projection onto the slot changed (see Migrations).PUT /api/slots/{name}/configno longer has a lifecycle side effect. It used to unload a running slot on anenabled: falsebody so the faded card matched reality. Config edits and lifecycle are now separate verbs: stopping a slot isPOST /api/slots/{name}/unload, which #1367 pin-gates (409slot.pinned,?force=truebypass).- The NPU-exclusivity 409 moved from the toggle to the model write. The AMDXDNA chat context still admits exactly one
device=npu, type=llmslot, but the discriminator is now "has a model configured":PUT /config {"model":{"default":X}}on a second NPU LLM anchor returns 409slot.npu_exclusivity_violation("only one NPU LLM slot may have a model configured at a time"; hint: clear the incumbent's model first). Model-less NPU LLM slots — including the shippedflmseed — coexist freely. - NPU trio dispatch now reads the anchor's
[npu]table, not the shadow's own flag.flm-stt/flm-embedare display+dispatch records for the anchor's singleflm serveprocess and always carry a placeholder[model].default, so they never had activation state of their own to express._is_npu_trio_requestgates onnpu.asr/npu.embedon thedevice=npu, type=llmanchor — which is literally the flag set FLM was launched with, so a modality that was never launched can no longer advertise itself as routable and then 503 on the liveness probe. The slot payload gainsnpu_modality_active(resolved server-side) for the shadow card's ON/OFF pill. [brain_chat] tool_modelis gone — it was never read (#1453). The field was documented (schema docstring), defaulted ("hal0/agent"), and promised by the shippedbrainseed profile and slot docstring as "the escape hatch for boxes whose model can't emit tool calls the local runtime parses natively", buthal0.brain.chat._chat_stream's model precedence was alwayspayload.model or cfg.model or default_model—cfg.tool_modelhad zero consumers. A documented no-op steering knob on the steward's tool loop was worse than no knob, so it's deleted rather than wired:BrainChatConfigisextra="forbid", so a config that settool_modelexplicitly now gets a clear validation error naming the field instead of silently doing nothing. The still-live[brain_chat] modeloverride is the real way to point the whole steward chat at a tool-capable model (e.g.hal0/agent) — brain.toml's docstring now says so. Follow-up design question (wire real per-turn tool-model routing, or leavemodelas the only knob) is tracked in #1480.
Migrations- Upgrade in place — re-run the installer (or
hal0 update); it is idempotent + non-destructive and never clobbers existing config. No reinstall. - Honcho → Hindsight (only boxes that ran Honcho): no migration step is needed — Honcho support was removed outright and Hindsight starts fresh; there is no data carry-over command (#1463 corrected the guide that claimed one existed).
- Slot-flag fold (operator-run): the migrator folds slot tunes into model defaults; it refuses the whole run (no partial write) if slots share a model with divergent tunes — resolve each shared model (canonicalize or split) first. Dry-run by default; back up
hal0.db+ slot dirs before applying. - Slot id-keying (operator-run, optional):
hal0 slot migrate-id-keyingin a downtime window (takes a pre-flight backup). The runtime reads either layout; the flip is deliberate and reversible. - A
[brain_chat] tool_modelkey left over from before its removal no longer breaks config load (#1453).load_hal0_configdrops it from the raw dict before validation — every load path (not just the packagedhal0 update, which already runshal0.config.migrationsfirst) is forgiving of a config written by an older build.HAL0-SUNSET: v1.1— the shim inhal0.config.loader._DEAD_KEYScomes out once no box in the field can still be running a pre-#1453hal0.toml. - One-shot
enabledsweep, boot-integrated (hal0.config.migrations.slot_enabled_removal). Runs first in theslot_reconcileboot phase, before every pass that readsmodel.defaultto decide what is configured. Rules:enabled = falsewith a[model].default→ the model is cleared (under the new rules a bound model reads as "on", so an in-place upgrade would otherwise silently activate a slot the operator had switched off — this is the only shape that needed more than a key drop); every other shape → the key is dropped and nothing else changes. NPU trio shadows (device=npu+typetranscription/embedding) keep their placeholder model — it is structural, not an operator pick — and only lose the key. Sibling[model]keys (context_size,labels, …) survive a clear; a slot that doesn't carry the key is left byte-identical, so a second run is a genuine no-op. Idempotent ("enabled" in rawis the check), best-effort per file (one corrupt TOML is logged and skipped, never fatal at boot).hal0 slot migrate-enabled-removalruns the same sweep on demand, dry-run by default; unlike the otherslot migrate-*commands it is safe to run live and needs no deploy window. - Disabling a capability now clears the slot's model instead of writing
enabled = false(SlotConfigStore._reconciled_slot, SC-1). The operator's pick is not lost — it stays incapabilities.tomlasCapabilitySelection.model, so a re-enable rebinds it. A pure disable still leaves device/provider/profile untouched. - Guided install gates on the model id, not a flag.
_build_slot_cfgcreates slots with an emptymodel.defaultandrun_pull_and_activatestamps the id only after the bytes land (_activate_slot_model); a failed pull leaves the slot model-less and marks[meta].pull_failed. Withholding the id is the start gate now — pre-stamping it was the one way a slot could have started before its model existed. - Rollback is one-way. Pre-#1369 code reads a missing
enabledasTrue, so slots this migration cleared come back model-less rather than re-enabled. Restore from a config backup if you need the prior state.
Known Issues- The profile-catalog reset does not fire during the 0.9.8 → 1.0 update itself (#1585). The update's commit phase runs inside the old (0.9.8) daemon, which predates the reset — so an upgraded box keeps its
profiles.tomlandmeta.schema_version = 1until the next update applied by v1.0 code. Nothing is lost (the reset is biased against deletion), buthal0 updateon such a box reports "nothing to apply" without mentioning the outstanding reset. Live-validated: the reset mechanism itself (consent gate, timestamped backup,schema_version = 2stamp, idempotence, virtual reseed) works correctly on real data. - Updating from 0.9.8 ends with a spurious error from the old client — the 0.9.8 CLI polls job status through the API it is restarting, treats the mid-restart connection refusal as fatal, and exits 1 after the update has in fact applied. Fixed in the v1.0 CLI (#1540), but the client driving a 0.9.8 → 1.0 update is by definition the old one. Verify with
hal0 --versionandcurl /api/health. - First-boot installs can lose the dpkg lock race to
unattended-upgrades(#1584) — the hermes-agent provisioning step degrades gracefully with a remediation line (hal0 agent install hermes) rather than failing the install.
Security- OpenWebUI's second, unauthenticated web surface now follows the box's bind choice, and the hardening guide admits it exists (#1515, with #1514).
hal0-openwebui.servicepublishes a complete chat UI — model access plus every stored conversation in/var/lib/hal0/openwebui— withWEBUI_AUTH=Falseand a hardcoded-p 0.0.0.0:3001:8080. Three things made that a posture gap rather than a documented trade-off. The bind ignored the operator:hal0.install.network's own docstring states the rule ("OneHAL0_BIND_HOSTdrives BOTH…"), and someone who setHAL0_BIND_HOST=127.0.0.1to keep hal0 off the LAN got exactly that on:8080and a wide-open chat UI on:3001regardless. The documented mitigation had no caller:env_writer's docstring told operators to passWEBUI_AUTH=True+WEBUI_AUTH_TRUSTED_EMAIL_HEADER"via theoverridesparameter", and nothing — notinstall.sh, notinstall_openwebui(), no CLI flag, no route — ever passed a non-emptyoverrides; the one remaining path, hand-editing/etc/hal0/openwebui.env, was erased on the next installer run (#1514), so the instruction was self-defeating rather than merely awkward. Anddocs/operate/auth.mdx, the canonical "Securing hal0" page, documentedhal0-apion:8080and stopped, so following the guide to completion still left the chat UI open. The publish address is now${HAL0_OWUI_BIND_HOST}, rendered byenv_writerfrom the sameHAL0_BIND_HOSTthe API uses and threaded throughinstall.sh; settingHAL0_OWUI_TRUSTED_EMAIL_HEADER(installer env or the file) is the single opt-in that turns OpenWebUI's auth on and points it at the header, since auth without a header is a login page with no identity behind it and a header without auth is ignored. The unit carriesEnvironment=HAL0_OWUI_BIND_HOST=0.0.0.0ahead of itsEnvironmentFile=— required, not defensive: systemd has no${VAR:-default}inExecStart, so an unset variable would hand podman-p :3001:8080, and this is what carries a box whoseopenwebui.envpredates the key. The value is sourced fromopenwebui.envrather thanapi.envdeliberately — reachingHAL0_BIND_HOSTdirectly would mean sourcing the file that carries every provider token and pushing them into the podman process environment, re-spreading what #1466 just contained. The default posture is unchanged (0.0.0.0,WEBUI_AUTH=False): this is "stop ignoring the operator's choice", not a silent flip that would strand every existing LAN user mid-release. The security guide gains a "second listener" section naming the port, both knobs, and the reason trusted-header auth is worthless while the port stays directly reachable. install.shstops erasing/etc/hal0/openwebui.envon every run (#1514).installer/README.mdpromises existing config files are "never clobbered on re-run", and the siblings in the same block keep it —hal0.tomlandupstreams.tomlare[[ ! -f ]]-guarded,api.envrewrites only a marker-delimited network block.openwebui.envwas regenerated wholesale from defaults every time, so a changedAUDIO_TTS_ENGINE, a repointedOPENAI_API_BASE_URLS, or the trusted-header pair #1515 tells operators to set was gone on the next repair or upgrade. The installer path now merges rather than skipping the write: every key already in the file keeps its value — including keys hal0 does not ship — and only genuinely new defaults are added, so a box installed before a key existed still receives it on upgrade instead of silently running a half-configured OpenWebUI. Precedence is shipped default < value already on disk < explicitoverrides, because an override is a caller stating intent while a preserved value is merely an absent one.write_env_atomicgrows aheaderparameter so the file stops carrying the slot-env boilerplate "Do not edit manually; changes will be overwritten on next slot load" — wrong on both counts for this file now; slot envs, which genuinely are regenerated every load, keep the original wording.DELETE /api/memory/banks/{id}/memoriesis no longer a one-call bank wipe (#1457, the #1024 incident class on the sibling route). #1024 was a bank delete reached with one unauthenticatedcurl; its hardening — the echoed-?confirm=<bank_id>gate and the dry-run blast-radius preview (#1028), plus arecord_actionaudit row (#1030) — was applied by lifting that one path out of the generic_FORWARDSpassthrough table into a hand-written handler. The bank-memories wipe stayed in the table. Upstream documents it as "Delete memory units for a memory bank … a destructive operation that cannot be undone", so it had the same blast radius with none of the friction: the audit row recorded a wipe that had already happened, on a call anyone could make. Classification did not save it — the route was already pinned ADMIN inexposure.DESTRUCTIVE_MEMORY_ROUTES, and in the shipped default posture (auth_required=false,has_admin_key=false) an ADMIN classification decides nothing; the audit that found this reachedGET /api/memory/banks/shared/memoriesunauthenticated against a bank holding 1629 nodes across 315 documents. It now takes the same echoed id (query string or body, like its sibling), returns the same preview payload on refusal, forwards nothing until confirmed, and still passes upstream filters through —confirmis hal0's own gate and is stripped so it never reaches the engine as an unrecognised filter. The two handlers share one_require_echoed_confirm, because duplicating the check per route is how they came to differ. Newexposure.CONFIRM_GUARDED_MEMORY_ROUTESnames the routes that must carry the echo, andtests/security/test_memory_bank_wipe_guard.pydrives every entry against the live app — so the next bank-scoped delete added to_FORWARDScannot inherit the passthrough silently.- An unaddressable memory namespace no longer collapses into a shared-bank sweep (#1451).
resolve_read_datasetsfiltered a caller'sdatasetlist against the spec §3 closed table and documented the result as "fail-open-empty" — right for a partial drop (["agents", "nope"]→["agents"]), catastrophic when the last entry was dropped, because[]is falsy and every consumer downstream read it back as "nothing requested":list(requested or [_SHARED])andreturn out or [_SHARED]in both providers,_requested_scope, anddelete()'s owndataset or _SHARED. Sodataset=["bogus-bank"]resolved to[]at the front door and to["shared"]inside the executor. The operator-visible shape is the one that matters: a bulkmemory_deleteis approval-gated on its arguments (mcp/admin.pygates any list-valued dataset), so the operator was shown a call naming a bank that does not exist, approved it, and watched it delete live documents out ofshared— on the deployed default (unified_bank = true) that is the only bank there is. Reads had the quieter half: a search or recall scoped exclusively to namespaces the caller may not address returned shared rows.[]now means no banks and onlyNonemay expand to the default: a non-empty request that resolves to nothing is aMemoryNamespaceError(400 /mcp.memory_schema) at the front door, and — because the REST delete route and the providers are both reachable without the resolver — an empty list handed to a provider sweeps nothing rather than everything.POST /api/memory/deletealso stops hand-rolling its list branch ([str(d) for d in requested]) and goes throughresolve_read_datasetslike the MCP surface, closing the two-surface drift thehal0.memory.namespacemodule exists to prevent.tests/security/test_memory_namespace_fail_closed.pypins the all-foreign list on the resolver, both providers, the three MCP handlers and the REST route; the pre-existing partial-drop behaviour is unchanged and covered by its own negative control. /api/memory/listandPOST /api/memory/deletefinally speak the same id (#1456).MemoryProviderpinsMemoryItem.idas "the document_id — idempotent, recall-visible, delete-addressable, NOT a per-fact id", and recall honoured it (document_id or id) while list inverted it (id or document_id). On a real Hindsight 0.8.x those are different UUIDs on the same item, so the round trip the API advertises — list, then delete what you listed — handeddelete_documenta fact id, 404-swept every bank, and returned{"deleted": 0}with a 200. Unified mode (the deployed default) failed in the other direction too:_deletable_idsmatched caller ids against the same fact-id field, so even a correct document_id never matched and was fail-closed withheld — the endpoint deleted nothing at all, for anyone. Every test agreed with the bug because every fakelist_memoriesreturned{"id": document_id}, a shape the engine never emits.idis now the document_id on every surface; the per-fact id moves tometadata.fact_idrather than disappearing (it is the only handle on an individual extracted fact); and_deletable_idsreturns a caller-id → document_id mapping, so a caller holding either handle resolves to the owning document while the engine is still addressed by document_id. Widening the id match did not widen the ACL — an unresolvable id and another agent'svisibility:privatedoc are still withheld, with negative controls intests/memory/test_memory_id_contract.py, whose fake deliberately returns a distinct fact id and document_id./etc/hal0/api.envis owner-only, and stays that way through an upgrade (#1466). The live box carried644 hal0:hal0on the file holdingHF_TOKEN,MINIMAX_API_KEY,OPENROUTER_API_KEY,HERMES_SESSION_TOKENandHAL0_TURNSTONE_TOKEN— every local account could read them. Four writers held three opinions, so the strictest always lost:installer/install.shseeded it 0644 and re-chmod 0644'd it on the network-block refresh, which #1375 made run on every re-run over an existing file, so any upgrade or repair re-published the secrets;_env_storewrote 0600 and had it undone;service_identitywrote 0640 for key rotation whileroutes/auth.pypromised "a never-world-readable 0640"; andinstall/perms.py— the engine whose job is converging the filesystem — pinned the row at0o644behind aFIXME(phase4)reading "may carry tokens", so it independently reverted every tightening the other three applied. There is now one constant,hal0.config.paths.API_ENV_MODE(0600), plus a singlepaths.api_env()resolver; the_env_storewriter, the rotation writer and the perms row all read it, and because the mode rides the temp file through the rename, the next dashboard write repairs an already-widened file rather than preserving it. The installer sets 0600 on both the initial write and the refresh, and no longer advertises a world-readable file in the comment that points operators at the dashboard Secrets path. 0600 rather than 0640: systemd readsEnvironmentFile=as the service manager before dropping privileges, so nothing needs the group bit, and a group-readable file becomes world-readable the moment a second account joins the group. A newhal0 doctorrow (check_secret_file_modes) fails critical on any group- or world-readableapi.env/openwebui.env— deliberately asserting the property rather than reading the perms table, because a check generated from that table would have agreed with the bug.- Settings ▸ Secrets stops offering hal0's own service config and auth keys as one-click removable secrets (#1450).
api.envis two stores in one file, and the route treated it as one:list_secretsenumerated every uncommentedKEY=line with no filter, anddelete_secretremoved anything matching^[A-Z][A-Z0-9_]{0,63}$— atomically, poppingos.environlive, returning an idempotent 204. So the dashboard renderedHAL0_ADMIN_KEY(written there byservice_identityon rotation, and whatroutes/auth.pyvalidates every login against) as a Remove button whose real effect is locking every new session out, andHAL0_PORT/HAL0_UI_DISTas buttons that break the service on next restart — with no confirmation dialog anywhere in the page, and captioned "Custom key · exported to hal0 services and slot containers as an env var".HAL0_is now a reserved namespace: those keys still list — an operator should be able to see what the service is configured with, and hiding them trades one lie for another — but carryprotected: true, and set/delete on them is a 403secret.protected. The gate is server-side because the UI is not the only caller. A prefix rule rather than an enumerated name list on purpose: a list goes stale the moment a newHAL0_*var entersinstall.sh, and it is the unlisted one that stays deletable. The page renders protected rows locked with no mutating control, and every remaining Remove now goes through a type-the-nameConfirmDialog— the value is never stored anywhere else and never shown again after saving, so there is nothing to undo with. - Pin the destructive
/api/memoryroutes as ADMIN, above the generic prefix rule (closes the last open slice of #1024). #1024's incident was a single unauthenticatedDELETE /api/memory/banks/{bank_id}that cascade-deleted ~632 live records; its echoed-?confirm=guard (#1028) and audit row (#1030) landed, but the classification itself was only ever ADMIN by generic prefix and nothing asserted it. That matters because #1024's own follow-up proposes "keep reads open if desired", and the natural expression of that — a_prefix("/api/memory")CLIENT rule — silently takes the bank wipe with it under first-match-wins. Two narrow rules (any DELETE under /api/memory,POST /api/memory/delete) now sit above the generic memory row, andexposure.DESTRUCTIVE_MEMORY_ROUTESenumerates the irreversible surface so a new memory delete route, or a reclassification, has to touch the constant in the same diff.tests/security/test_memory_delete_auth.pyasserts the constant tracks the live route table, that each route resolves via a pinned rule (not the ADMIN fallback), that all of them survive a simulated reads-are-CLIENT widening, and — against the realAuthEnforcementMiddleware, armed — that an anonymous call gets401, a client/inference key gets403, and the operator's admin key clears the gate.
Added- The dashboard Services page can actually manage services under the shipped
User=hal0posture, and ComfyUI gains a Start (#1590, #1591). Every mutating verb on the Services page (start/stop/restart on all four cards) died with polkit's "Interactive authentication required":hal0.services.systemdinvoked systemctl directly under a stale "hal0-api runs as root" assumption, and thehal0-systemctlseam never covered companion units. The wrapper gainsstart-agent/restart-agent/enable-agentarms plus a closedsvc-<verb>family for the openwebui/hindsight units, the seam routes agent + companion units through it, andunit_actionexecutes via the seam. ComfyUI's new Start — on the service card and the Image-Gen header — deliberately drives the GPU-arbiter switchover (drain LLM slots, hand the iGPU over) rather than a rawsystemctl start, which would boot ComfyUI under the resident LLM stack; stop stays arbiter-only. The services cards also adopt the slot-card design system (container, typography, ok/err tokens,btn ghost smbuttons). - Delete in the model row's "⋯" menu (#1593). Deleting a model no longer requires selecting the row and reaching the detail pane — a danger item behind a divider opens the same
DeleteModelDialog(type-the-name confirm, blast-radius warning, refcounted blob release), offered only for installed local models. - Superset workspace lifecycle scripts (
.superset/setup.sh,run.sh,teardown.sh). Superset gives each task its own git worktree and keeps several alive at once, which breaks every default in the dev stack:scripts/dev-bootstrap.shbinds8080/5173/3001and names its OpenWebUI containerhal0-openwebui-dev, so the second workspace's startupdocker stopkilled the first workspace's container, and its dev servers either failed to bind or — with the UI — silently attached to a server running a different branch. That last failure mode is not hypothetical: it is exactly #1399, whichui/tests/e2e/port.tsalready fixed for the Playwright suite by hashing the worktree path into a stable, distinct port..superset/ports.shapplies the same rule to the dev servers, in windows that don't overlap the e2e one (api 18000–18499, UI 6100–6599, OpenWebUI 3300–3799), plus a per-workspaceHAL0_HOMEand container name.run.shis a thin wrapper that assigns those and hands off todev-bootstrap.shrather than forking a second launcher — one owner for "start hal0 locally" (CONTRIBUTING rule 11).VITE_API_TARGETis exported rather than written toui/.envbecause a Vite config file readsprocess.envand.envfiles never reach it; writing the file would have looked right and proxied/apito a dead127.0.0.1:8080.setup.shinstalls viauv sync --frozen --extra dev— the same lockfile path CI uses, so a workspace can't skew from it (CONTRIBUTING rule 10).teardown.shstops the workspace's services and reclaims the rebuildable artifacts, but exits non-zero when the worktree still holds uncommitted changes or commits reachable from no remote, which Superset surfaces as an error toast with a Delete Anyway button — a speed bump the operator can override, not a lock. Its "is this pushed?" test isgit log HEAD --not --remotes, not a comparison againstorigin/main: hal0 has noorigin(its remotes aregithubandhal0), so the obvious version would have reported a clean slate for every branch and discarded the work silently. hal0 slot migrate-flags— the flags-fold migrator finally has an operator entry point (#1396).hal0.config.migrations.slot_flags_foldhas existed since the flags-ownership lane, but nothing ever exposed it: no CLI, no installer hook, no boot wiring — only tests referenced it. Its sibling folds both had commands (slot migrate-hw,slot migrate-caps). Meanwhile the launch-side readers were already deleted (providers.containerdropsprofile_flags/slot_parallel/extra_args;resolve_chat_templateno longer consults the slot), so an upgraded box with a bench-tuned slot silently launched without that tune and had no supported way to recover it — the exact ordering hazard spec-flags-ownership §5.4 flagged ("readers become expired shims with a sunset"). The new command mirrorsmigrate-hw: dry-run by default,--applytakes a timestamped backup and passes thedeploy_window=Trueack, refuses to run while any hal0 unit is live (--stop-servicesto stop them), and is never wired into an automatic path. Divergent-share conflicts (two slots folding different tunes onto one model) are surfaced as a clean non-zero exit listing every conflict — on the dry-run path too, which previously would have raised an unhandledRuntimeErrorat an operator merely previewing a conflicted box.- Field-wiring and rejected-write contract specs for the slot + model drawers (#1371). Every editable drawer field routes its value somewhere specific — the batched
PUT /config, its own POST, or an instant-apply write — and several of those hops had no assertion that the value reached the wire with the right key, so a silent rewiring regression was invisible. Three new e2e specs close that:slot-drawer-field-wiring-v3(model swap →POST /swap {model_id}and its live-container confirm gate;parallel→PUT /config {parallel}incl. empty→null, untouched-never-rides, and sub-1 validation; the NPU Chat/Embed modality writes; the NPU chat-model pick; and the not-installed pull-then-apply flow, which must not write config while the download is in flight),model-drawer-duplicate-v3(POST /api/models/{id}/duplicatebody with and without a device template, id suggestion and pinning, invalid-id block, and the 409 path), anddrawer-save-errors-v3. That last one closes a whole missing axis: a grep forstatus: 4/status: 5across everyslot-*spec previously returned nothing, so nothing covered what happens when the server says no — precisely where an operator loses work. It pins that a rejected write never closes the drawer, the operator's edits survive, the backend envelope message is surfaced verbatim, a failedPATCH /defaultsshort-circuits the/configPUT rather than leaving a half-applied save, and a server-rejected NPU toggle reverts to server truth instead of displaying a state the backend refused.
Changed- The slot drawer's header toggle is now Pinned/Unpinned, and an explicit
pinned = falseun-pins a default anchor (#1367). The §21.10 operator pin (SlotConfig.pinned) shipped with no UI surface at all, while the drawer header exposed the enable/disable flag — the toggle an operator actually reaches for when they mean "keep this slot resident". The header toggle now reads/writespinned(instant-applyPUT /config {pinned}, same shape as the old enable wire). Two backend halves make it honest:reaper.is_pinned()treated an authoredpinned = falseas indistinguishable from absent (is TrueOR anchor-set), soagent/utility/npucould never be un-pinned by config — the raw-TOML key now wins in both directions and the anchor set only applies when the key is absent; and the slot list (GET /api/slots) lifts the effective pin per entry, so a fresh install'sutilityrenders Pinned without a per-slot/configfetch. The unload/delete guards are unchanged (409slot.pinned,?force=truebypass) and are now covered by route tests, including the un-pinned-anchor path. Enable/disable is untouched backend-side; its drawer toggle is gone (removal ofenableditself is staged separately).
Fixed- Saving slot settings could crash the whole Slots view to the error boundary (#1588). The save invalidates the slots query; on a box with a slow
GET /api/slotsthe next poll can briefly return a list without the edited slot, andEditSlotDrawer'suseRefSM(baseline ? slot.name : null)initializer argument was evaluated on that render — before the cleanup effect that drops the baseline could run. Regression from the #1447 frozen-baseline rework; only the save path triggered it (cancel/close keeps the slot in the list). Reproduced and verified fixed live. - Settings info icons sit inline with their labels (#1592).
.s-row .kstill carriedflex-direction: columnfrom the era when the sub-text rendered under the label, so everyFieldInfoIcondropped below its label — same defect and same fix as the model/slot drawers'.form-lbl. - The slots-page activity sidebar no longer stretches past the main column (#1594).
height: 100%on the log card fills the stretched grid track but does not stop the log's intrinsic content height from driving track sizing, so a long feed extended the row thousands of px past the NPU pane. The side column's child is now absolutely filled into the stretched track (contributing nothing to sizing); single-column breakpoints revert to in-flow with a 70vh cap. - A FastAPI minor bump silently removed the entire MCP surface. FastAPI 0.138 stopped flattening
include_router:app.routesnow holds onefastapi.routing._IncludedRouterper included router instead of that router'sAPIRouteobjects, and the wrapper carries neither.pathnor.methods.build_admin_route_map's flat one-level walk skipped every one of them, returned an empty map, andinstall_admin_route_mapraised "catalog drift" for ~85 classified routes — whichcreate_appcaught and logged, so/mcp/adminand/mcp/memorynever mounted while the API kept reporting healthy. Found live on halo: 21 boots, 0 successful mounts. CI could not see it, because the test venv resolvesfastapifromuv.lock(0.136.1, still flat) while the installer resolvespyproject.toml, which pinnedfastapi>=0.115with no upper bound — so every fresh install got the restructured version and lost its agent control surface. Three-part fix: the walker descends through wrappers viaeffective_candidates()(duck-typed, depth-bounded, verified against real 0.138 — 315 route-map entries, both servers mounted);pyproject.tomlgains<0.140; and a mount failure is now anerror-level log recorded onapp.state.mcp_mount_error. - A failed MCP mount is now visible instead of only logged.
create_appstill survives the failure (a serving API with no MCP beats no API), but/api/health/systemreports amcp_mountcheck that names the reason and flips the whole payload todegraded. The blocker above hid behind a single warning line for 21 boots with every health endpoint sayingok; this makes that class of failure surface within a boot. - An in-place upgrade left the OLD code serving. The installer ended with
systemctl enable --now hal0-api;--nowstarts a stopped unit but is a no-op on an active one. So an upgrade over a live box replaced the venv, swapped/usr/lib/hal0/current, printed its success banner and exited 0 while the running process kept serving the code it had already imported —hal0 --versionand/api/healthdisagreed until someone restarted by hand.start_or_restart_apinow enables for boot either way and issues an explicitrestartwhen the unit is already active. Deliberately the opposite policy from slot units, which are still not bounced (a slot restart costs a model reload; hal0-api is the thing being replaced). - Slot drawer — restore the per-slot
ReasoningandMTPcontrols (fixes γ-suite, #1333). Thefeat(ui): slot drawer & model drawer reworklanded with Reasoning + MTP removed from the slot drawer under "now model-owned"; the model drawer carries model-level defaults but operators need a per-slot override. Re-added the Reasoning pill (llm slots only, instant-apply viaPUT /config { enable_thinking }) and the MTP pill (llm slots only, tri-state Auto/On/Off, instant-apply viaPUT /config { mtp }+ non-blocking cold restart) into theInferenceFieldGroup. Renamed the HW gridHardwareFieldGroup →Slot(the drawer is now Slot / Model / Inference) and the runner-binary select labelProfile→Binary(the bound runner stays on the slot card chip — no editable profile select in the drawer). The 12 γ-suite slot-drawer tests that were red on main since 2026-07-20 (#1333) all pass. - Hermes gateway install runs as root — fix installer
sudo -u hal0→ root call (PR #1337). The installer was dropping to thehal0user before runninghermes gateway install --system, which checksos.geteuid() == 0and refuses non-root. The--run-as-user hal0flag already tells hermes which runtime user to bake into the systemd unit. Validated on halo150 (10.0.1.150, podman 4.9.3). /mcp/memorybulk delete was an ungated bypass of the admin approval queue (#1302).mcp__hal0-admin__memory_deletegateslen(ids) > 1through the approval queue, but the standalone/mcp/memorymount ran the same tool against the same provider with no gate — an agent holding only the narrow memory surface could bulk-delete without an operator ever seeing a prompt. The memory dispatcher now takes the process-wideApprovalQueueand enqueues bulk deletes itself, delegating the classification tohal0.mcp.admin.is_gatedso the two mounts cannot drift. The dispatcher handed to the admin server deliberately stays ungated (admin gates first, then runs the approved call through it — double-gating would re-enqueue an approved call forever). Single-id deletes stay autonomous./api/memory/*auth posture ratified as perimeter-only (#1302).X-hal0-Agentis self-asserted and hal0 does not authenticate it — there is no credential to check it against (ADR-0012 removed auth platform-wide). hal0 validates the header's shape and rejects a body-suppliedsource, so audit can never disagree with the namespace a write landed in, butprivate:<agent>is an isolation boundary between cooperating agents, not a security boundary against a hostile LAN caller. Documented for operators indocs/concepts/security.mdx(with the reverse-proxy header-injection pattern for multi-tenant deployments) and next to the code inapi/routes/memory.py.- Hindsight→pgvector degrade ladder was inert;
degradedreported healthy while memory was broken (#1301)._build_hindsight_clientonly calledHindsightRestClient.from_env(), which builds an httpx client and does no I/O — so a daemon that was down never raised at boot andprovider_from_config's degrade branch never fired. Operators got a live-but-brokenHindsightProviderwithdegraded=False: failures surfaced only as empty recalls whileGET /api/status.memory_degradedandhal0 memory statusboth said healthy. Construction now probes/health(the same endpointinstall.shwaits on), timeout-bounded viaHAL0_HINDSIGHT_PROBE_TIMEOUT_S(default 2s). A401/403passes — that proves the daemon is answering, and a wrong API key is not a reason to silently drop the durable engine;5xx, connect errors, and timeouts degrade. project:<id>isolation was a silent no-op inunified_bankmode (#1300). Under the default config every namespace exceptagentscollapses onto the singlesharedbank.private:survives that collapse becauseaddstampsvisibility:private+agent:<id>;project:<id>got no compensating marker, so aproject:foowrite became an ordinary shared write and aproject:barrecall returned it — project scoping did not exist. Writes now stamp aproject:<id>tag on collapse, and recall / list / delete filter by it, reproducing the bank isolation legacy multi-bank mode gets for free: a project read sees only that project, asharedread does not see project docs, and the filter composes with (rather than shadows) private visibility. Delete is gated on the same predicate, so a project-scoped delete cannot reach outside its scope. Legacy multi-bank mode is untouched — no tag, isolation still by bank.slot loadno-op'd on a live slot, so config edits never converged (#1224 part 2). The unit file claims to be "regenerated on every slot load", but a load on a ready/serving/idle slot short-circuited to a status snapshot.PUT /api/slots/ops/config {"port": 8091}→slot load opsreturned the stale snapshot with the container still on--port 8089, and the next implicit reload cycled warming → error with nothing listening on either port; recovery neededsystemctl reset-failedplus a second load from the error state — the only path that regenerated. An explicit load now compares the running argv against what a restart would render (reusing the drift comparator, which cannot be lost across an api restart and self-heals a unit drifted by any route) and converges when they disagree. Unchanged slots stay a no-op, and an unreadable comparison counts as "no drift" rather than bouncing a healthy container.--portjoins the compared keys — it is the field the issue was reported against.slot restartcould hang forever on a wedged unit (#1224 part 1, completing the earlier fix).terminateransystemctl stopas a blocking executor call with no timeout; against an already-failedunit that never returns, sorestart's best-effortsuppress(Exception)could never fire and the CLI ReadTimeout'd with the unit never relaunched. The stop is now bounded (SlotManager._terminate_timeout_s, default 30s) and raisesSlotTerminateTimeouton expiry. We cannot kill the executor thread, so this does not cancel the stop — it hands control back so the caller converges instead of hanging; the abandoned thread retires on its own.hal0 update --rollbackreported success while the version it rolled away from kept serving every request (#1541). Rollback reverts thecurrentsymlink and re-pips the venv but never bounceshal0-api, so after a rollback the box sits in a split state: disk, venv andhal0 --versionall say the old version while the running process is still the new one. The banner said onlyrolled back (<channel>)and exited 0. This is the emergency path — it is what an operator reaches for when the new version is actively misbehaving — and every signal they would naturally check confirmed the rollback and reinforced the wrong conclusion. The CLI now names the version it reverted to, compares it against the version/api/healthreports as actually running, and when they differ says so and printssystemctl restart hal0-api; the warning is suppressed when the service is unreachable, since nothing stale is serving then. The running version is read from the API response rather than the CLI's own in-processhal0.__version__, which was imported before the swap and would have compared the old version against itself. Separately, the route was discarding everythingUpdater.rollback()computed —rolled_back_to,previous_nowandschema_warning— and now returns all three; the droppedschema_warningflags a forward-only migration the reverted tree may not understand, which is a data hazard that was never reaching an operator.- A successful
hal0 updatereported an error and exited 1 (#1540). Applying an update restartshal0-api, and the CLI polls job status through that same API — so the connection-refused it gets back mid-restart is the expected response, not a failure._poll_jobtreated the first transport error as fatal and calleddie(), so every successful update ended in a red error and a non-zero exit; automation reading that exit code would see a good release as a failed deploy. Two further defects sat behind it. The restart was a blockingsystemctl restartissued from insidehal0-api.service's own cgroup, so systemd SIGTERM'd thesystemctlcall and the calling process together — the wrapper now passes--no-block. And the job's terminalappliedstate was written only after that restart returned, so on a real box it frequently never reached disk at all; the terminal state is now persisted before the restart is attempted, withrestarted=Nonemeaning "applied, bounce still in flight". Retrying the poll alone would not have fixed this: the CLI would have re-attached after the restart, read a stalerunningsnapshot and still failed, just ten minutes later on the poll timeout. Only transport failures are retried — a live API answering 4xx/5xx still fails immediately, so a genuine commit failure cannot turn into a hang. - A fresh install could not register a single model: the primary database was root-owned and no ownership row covered it (#1546).
hal0-apiruns asUser=hal0, but/var/lib/hal0/hal0.dbis created by root during install (the schema migrator runs before the daemon first starts) and was never chowned. SQLite needs write access to the file, not just its directory, so every registry write failed withattempt to write a readonly database— and because registration happens after the download,hal0 model pullfetched the entire artifact (532 MB in the report) before failing, leaving unregistered weights orphaned on disk. The ownership table had a row forregistry/hal0.db, which only ever matches aSqliteModelRegistry(registry_dir=...)override used for test/dev isolation; production resolves throughpaths.db_path()tovar_lib/"hal0.db", which had no row at all. Soinstall.shprintedownership table applied (37 path(s) reconciled)andhal0 doctor permsreported the box clean while the registry was unwritable — the same "the check agrees with the bug" shape as #1466. The primary database and its-wal/-shmWAL siblings now have explicithal0:hal0rows, so bothdoctor permsandperms --fixcover them. - The Connections endpoints pane claimed "auth none · open on lan" and emitted a cURL with no Authorization, regardless of the live auth toggle (#1467). The copy and the generated command were static leftovers from "no inbound auth in v0.3", while Settings ▸ Security ships a working
PUT /api/auth/requireand/v1/*is CLIENT-classified — so with enforcement armed the pane told an operator the box was open and handed them a command that 401s. Both are keyed onuseAuthStatusnow: the auth row and pane footer read "client key required", and the cURL gains-H "Authorization: Bearer $HAL0_CLIENT_KEY". - The Security page's client-key action was disabled with a reason that was no longer true (#1467). It read "Setting/clearing the client key has no route yet — keys are configured via HAL0_*_KEY env today", but
POST /api/auth/rotateacceptstier: "client"(Literal["admin", "client"]) and mints the key live. The dead button is replaced with a working Rotate action reusing the existing dialog, which already parameterised tier;lastRotatedis now tracked per tier so both rows show their own fingerprint and timestamp. The matching stale comments inuseAuthStatus.ts("there is likewise no key-rotation route") andendpoints.ts("ExposureTable currently ships a stub-with-reason") are corrected too. - Saving an HF token swallowed every failure (#1467).
setSecret.mutatecarried noonErrorand the component never renderedsetSecret.isError, so a backend rejection — e.g.400 secret.value_invalidfor a non-printable value — was invisible: the field kept the typed token and the status stayed "not set" while the operator believed it had saved. Now surfaces the error the way the Remove action on the same page already did. - The Voice and Image-Gen panes rendered blank controls with no error state when the capability probe failed (#1467). Both gated only on
capsQuery.isLoading, so a failedGET /api/capabilitiespainted unchecked, empty controls as though nothing were configured — with Save still clickable, inviting a write of model/enabled selections against unknown live state. Both now follow theAdvancedPagepattern: an error banner, and Save disabled while the probe is failing. - Upstreams mutations failed silently (#1467). The enabled and advertise toggles, the filter Apply/Clear, Delete, and the per-row key save all called
mutatewith noonError, so a failed PATCH just snapped the checkbox back on the next refetch with no explanation. Worse,AddUpstreamFormclosed ononSettled, so a failed credential write left the upstream created but keyless with only an "OPENROUTER_API_KEY unset" chip as evidence. All now surface errors, and the add form usesonSuccess/onErrorso a failed key write keeps the form open with the reason shown. - The Updates page's "Auto-check" row could never read anything but enabled (#1467).
updater.pyreturned a literal"autoCheck": Truederived from nothing — no timer, no config knob — and the UI rendered it as "Background update checks by the daemon · enabled", so an operator who masked the check timer still saw green. The tree was searched for any real signal (systemd units, config keys,registry/update_check.py— that one is the model-registry checker, unrelated) and none exists, so the row is removed rather than fabricated: a permanently-green evidence row is worse than no row. The wire field itself is left in place; retiring it is a separate contract change. - A second toast replaced the first instead of queueing, and any toast fired before React mounted was silently dropped (chrome GA polish, #1473).
globals-install.tswireswindow.__hal0Toastto theuseToastStorezustand store before React mounts (so bundle-init and the AuthGate login screen can toast), but nothing ever rendered the store'squeue—dash/main.jsxkept its own single-slotuseStateand unconditionally overwrotewindow.__hal0Toastwith a wrapper around it on every App mount, so anything already queued in the store was orphaned and a second toast while a first was showing clobbered it rather than stacking. AddedinstallToastQueueHook()(window.__hal0UseToastQueue, same window-hook-bridge patternboard-hook-bridge.tsuses for the no-ES-importsdash/*.jsxprototype files) and aToastHost-style render inmain.jsxthat maps the real queue; deleted the shadowuseState. CSS gained a.hal0-toast-stackfixed-position flex wrapper —position: fixedon each.hal0-toastdirectly (the old rule) doesn't respect DOM nesting, so multiple simultaneous toasts rendered on top of each other regardless of queue order. ApprovalModalfabricated acapabilityandpolicyrow, and a "Deny + remember" button that behaved identically to plain Deny (chrome GA polish, #1473).ApprovalEntry.as_dict()(src/hal0/mcp/approval_queue.py) carries nocapability/policyfield — the rows were synthesized client-side from data that doesn't exist server-side, and the "remember" button'sonClickwas byte-identical to the plain Deny handler, so it never remembered anything. Both rows and the dead button are gone; the footer's dangling "Configure auto-approve rules in the agent view." (no such surface exists) is gone too.- The dashboard throughput card's footer crashed to
"undefined slots serving"when the latest sample had nototal_tps(chrome GA polish, #1473).RDThroughputCardrendered`${serving} slot${...} serving`unconditionally onceisPendingcleared, butservingis derived from the last sample'stotal_tps, which can benull/missing on a freshly-warming slot. The footer now reads "no samples yet" in that case instead of the literal string "null slots serving". - MCP tool errors had two different shapes depending on which path failed, so
error.codewas unreachable for every REST-forwarded failure (#1468)._call_restreturned the hal0 REST envelope verbatim under its ownerrorkey, yieldingerror.error.code, while every in-process path —mcp.unknown_tool,mcp.missing_arg, the memory dispatcher — returns a flaterror.code. An MCP client or the hal0-brain toolloop branching onresult["error"]["code"]therefore readNonefor every forwarded 4xx/5xx, andhttp_statusexisted on only one of the two paths._rest_error_payloadnow lifts the inner object (message and details included) so both paths answer to one accessor; a non-hal0 body — FastAPI's bare{"detail": …}, a reverse proxy's HTML 502, an empty body — keeps all of its diagnostic content under a syntheticmcp.rest_errorcode rather than being dropped, because that body is often the only evidence of what actually broke. - The Connections blast-radius manifest showed
args: objectfor all ~92 hal0-admin tools, making the tool-detail feature dead for the entire admin surface (#1468).build_serverdeliberately advertises each admin tool as a single object-typedargsproperty wrapping the real per-tool schema (a flat signature would make FastMCP silently drop undeclared body fields), but_args_signatureonly walked top-level properties — so the renderer that exists precisely to show "the contract … an args signature" rendered the wrapper instead of the contract. The memory server's flat schemas proved the renderer itself was fine._unwrap_args_envelopenow descends through exactly that wrapper shape — a loneargsproperty that is itself an object with its own properties — leaving genuinely flat schemas untouched, and leaving a tool that really does take one opaqueargsobject rendered as such rather than as "no args". - The MCP install catalog shipped v0.3-alpha mock data at GA: invented popularity metrics, and five
verified: trueentries that could not install (#1468). Checked against the live npm registry:@modelcontextprotocol/server-puppeteer,-gdriveand-slackare retired upstream, and@modelcontextprotocol/server-sqlite,@linear/mcp-serverandhomeassistant-mcpreturn 404 — so every entry wearing the curated badge was a guaranteed failed-or-unsupported install, while three of the four unbadged community entries resolved fine. Sincehal0 mcp catalog installfeedsspecstraight to the npm resolver, the badge was actively inverted from reality. The dead entries are replaced with first-party servers verified to resolve (server-filesystem,server-memory,server-sequential-thinking,server-everything, and@playwright/mcpas the maintained successor to the archived puppeteer server);starsandtoolsare removed rather than re-sourced, because a popularity figure an operator might weigh a trust decision on has to come from somewhere real and nothing here can supply one.verifiednow carries one enforced meaning — first-party publisher, not a security review — a newadvisoryfield states that in the payload so no consumer invents its own wording, andcategoriesis derived from the items instead of being a hand-maintained Title-case list that matched none of the lowercase item categories. The CLI table drops the two invented columns in favour of the installspec. - A
<tool_call>wrapper holding more than one<function>ran only the first — the rest were silently dropped (#1509). The nested attribute-XML branch acceptednested[0]and nothing else, while still recording the whole wrapper's span, so the siblings were lost twice over: never accepted there, and then skipped by the later bare-<function>pass as already-consumed. No error, no log, no trace in the cleaned text — a model that asked for two tools and got one is indistinguishable, from the operator's side, from a model that only asked for one. That is the same silent-divergence class as #1419, which is why this came out of the #1477 LOW sweep and was fixed on its own branch. Every nested call is now accepted, with the wrapper span recorded exactly once regardless of how many calls it held:cleaneddeletes each recorded span in turn, so a duplicated span would have cut a second, unrelated slice out of the reply. Same fix run: a mangled call with no</function>(the wire form #1434 targets) bounded its final parameter at end-of-turn, so the tool received the model's closing prose appended to its last argument and the span covering it stripped that prose from the visible reply — two failures from one bound. It now stops at the first paragraph break, which is the cheapest signal that the value ended and the model went back to talking; bounding at the first newline instead would truncate the many legitimately multi-line values (file bodies, patches, prompts), so a multi-line value with no blank line still arrives whole. The #1419 live-captured tag-stripped form (name="get_weather"> name="city">Paris) is pinned as a regression guard. - Six memory-bank panels rendered an engine outage as an empty bank; one made its card vanish entirely (#1539, first tranche). Every panel read
query.data?.<list> || []and rendered an empty-state when the list came back short — but a failed query hasdata === undefined, so the fallback fired and a 503, a dropped connection or a restarting hindsight-api was indistinguishable from a healthy quiet bank, which is exactly what a fresh install has. This is the #1471 defect (the graph explorer) repeated across the bank surface, except here there was no branch to get wrong: none of these panels consultedisErrorat all. Documents, mental models, directives and the retain timeseries said "No documents in this bank." / "No mental models defined." / "No directives." / "No retain activity in this window."; the operations panelreturn nulled on an empty list, so the whole card silently disappeared; and bank cards showed every count as0, so an unreachable engine read as a bank with nothing in it. All six now announce the outage with a retry, and the bank card carries a compact "stats unavailable" chip rather than a banner per card. None of this was testable before #1538 made a non-ok response representable under forced-mock — which is why it shipped unnoticed. - Settings ▸ Updates displayed a release channel it had failed to read, and kept "Roll back" armed against the version it never got (#1539, second tranche).
/api/updates/statefeeds seven call sites and none of them consultedisError; six were honest about a failed read anyway — the footer chip and the update banner stay hidden, About renders "—", the notification bell drops the row, andmain.jsxleaves the tab title on its build-time stamp. The Updates page was the exception, and in the #1467 shape exactly: the channel<select>readu.hal0?.channel || 'stable', so any outage displayedstable— a specific, plausible, persisted-looking value the page never actually read. A box onnightlywas shownstablewith nothing on the page to suggest the read had failed, and the picker stayed live, so "switching" away from that phantom baseline was one click. The hal0 row fell through tocurrent {u.hal0?.current}withcurrentundefined, rendering the bare word "current" followed by nothing. The tell that this is a defect class and not a missing banner: the Auto-check row in the same panel already guarded onstateQuery.dataand rendered "—" (that guard is #1467's fix) — one row was honest and the one next to it fabricated a value. The panel now announces the failed read with a retry, the version renders "—", the channel select holds an explicit empty option and goes inert, and roll back — the one irreversible control here — is disarmed until a real payload lands. Driven through #1538's__hal0MockPassthrough, sinceupdatesStateis a plain allowlist row that forced-mock substitutes before any fetch is issued. - Every UI error state behind a mocked route was untestable by construction; the e2e suite could not tell a working error path from a broken one (#1498, #1527). Forced-mock (
VITE_MOCK_HAL0=1, always on for Playwright) guaranteed a successful response for all 30 allowlisted GETs, by two separate mechanisms inmockFetch: 24 plain rows were substituted before any fetch was issued — so apage.routeoverride never even saw the URL — and the 6networkFirstrows reached the network but had any non-ok response replaced with the baked payload. An error response was therefore unrepresentable anywhere in that class. This was found twice independently, from opposite ends: #1471's "memory engine unreachable" branch and #1467's capability-probeisErrorbanner each had to ship uncovered, and in both cases the red-first attempt did not go red, which is what exposed the harness rather than the code.window.__hal0MockPassthroughnow lets a spec claim paths (string prefix or RegExp) and drive its own responses. It gatessubstitutable, which all four substitution branches already consult — pre-fetch, 404 fallback, network-error fallback and thenetworkFirstrescue — so one flag covers every branch instead of leaving a hole in whichever one a future comment forgets. Scoped rather than global, so claiming one path leaves the rest of the surface mocked and an opted-in spec needn't re-stub everything; it can only ever disable substitution, so it is inert in dev and production. The three memory-graph outage assertions dropped from #1471 are restored, and the mechanism is pinned by 9 cases inui/src/api/mock.test.ts— including that a malformed value is ignored rather than taking down every request on the page. - Settings section navigation now tracks the URL hash both ways (#1438).
SettingsShellseeded a localuseState(initialSection)from theparamprop once at mount; since the outer router keepsrouteat"settings"across every#settings/<section>hash (only the sub-path changes), the component was never remounted — only re-rendered with a newparam, which the local state ignored. Deep links landed on the wrong section, and browser back/forward did nothing.sectionis now a pure derivation ofparam(no local state), and the nav'sonSelectwrites the hash instead of local state, so the existinghashchangelistener is the single feedback loop for both directions. - The console no longer logs a 404 for
GET /api/migrations/flag-reporton every route (#1439). The backend route doesn't exist yet (flagged — it lands with the migration lane), butuseMigrationReportpolled it every 60s from the app root (MigrationBannermounts inmain.jsx) regardless of page. The hook's try/catch prevented a crash but not the browser logging the failed request. The query is nowenabled: falseuntil the real endpoint ships;report/count/hasWorkalready resolved to the same empty values against a 404, so this is a pure noise fix. - The models catalog list now shows which model is the default for its type (#1440).
model.defaultalready drove the drawer's "Default for {type}" badge, but the list row never read it — an operator had to open every row's drawer one at a time to find the current default.ModelRownow renders a✓ defaultchip whenmodel.defaultis true. - The duplicate-model dialog no longer claims a false "You can undo this later." (#1442). Duplication has no undo — delete is the (refcounted) inverse. The shared
ConfirmDialogprimitive's default non-destructive footer copy is accurate for most of its callers but not this one; it now accepts an optionalfooterNoteoverride, andDuplicateModelDialogstates the real contract: "The duplicate can be deleted at any time; weights are shared." /v1/images/generationsaccepted curated entries that are not renderable checkpoints and silently pinned a LoRA or a.pthupscaler in as the workflow checkpoint (#1470). The gate was a barecurated.capability != "image"check, but six curated entries carrycapability="image"and two are not checkpoint files:esrgan-4xis a RealESRGAN.pthupscaler (comfyui_subdir="upscale_models") andsdxl-lightningis a LoRA whose own notes say it needs the SDXL base already loaded. Both passed, andtemplate_for_model_classdeliberately falls back tosdxl_turbo_simplefor any unrecognisedmodel_class— so the request was rendered with the wrong file as the checkpoint, yielding a ComfyUI node failure or garbage instead of a clean 4xx. The gate now requires both that the entry is staged as a real checkpoint (comfyui_subdir == "checkpoints") and that itsmodel_classresolves to an explicitly-declared template. Theandmatters:sdxl-lightningis already taggedcomfyui_subdir="checkpoints"— that mistagging is part of the bug — so the more obvious "checkpoint or has-a-template" rule would have let the LoRA straight back through. One further entry changes behaviour as a result:SD-Turbo-GGUF(model_class="sd-turbo", absent from the template map) now returns a clean 4xx where it used to be rendered through the sdxl-turbo template with a.ggufpinned as a safetensors checkpoint — the same defect as the two named ones, previously unnoticed.Flux-2-Klein-9B-GGUFstill passes, since itsflux-kleinclass is a deliberately declared soft-fallback. The 404's built-ins list is now generated from the curated table through the same predicate the gate uses, so message and gate cannot drift.- The ComfyUI pane rendered a fictional job whenever the status fetch failed (#1470).
paneDatafell back toCOMFYUI_V2_MOCK— "generating · 72%", a wan2.2-i2v render at KSampler step 3/4, a queue holdingqwen-imageandsdxl, GTT 54/80 — on first paint and for as long as/api/comfyui/statuserrored, so an API outage or auth failure looked like a busy GPU.COMFYUI_FALLBACK, the hook's purpose-built neutral "stopped shell", was imported and referenced nowhere: the intended wiring had been dropped. The fallback is nowtransformComfyuiStatus(COMFYUI_FALLBACK), with the mock reachable only through thewindow.__comfyuiV2MockOverridee2e seam. - Dead controls removed from the ComfyUI pane (#1470). "Stop container" rendered an
onClick={onStop}that the sole call site never passed, and no stop mutation exists inuseComfyui.ts— a silently inert stop button on a GPU-exclusive engine is worse than none. Pending queue rows rendered Logs and Remove buttons with no handler at all, and the no-URL branches rendered a plain "Open ComfyUI ↗" button that did nothing. All are dropped rather than wired to speculative endpoints; the rest of the app already treats ComfyUI as restart-only (the GPU arbiter owns start/stop). - The Voice page presented Kokoro's behaviour as engine truth while the shipped default is qwen3tts (#1470). The default-voice option advertised
af_bella, the speed hint claimed "Kokoro clamps to 0.5–2.0", the sample-rate row read "fixed by the Kokoro engine · 24 kHz", and the seed-pack fallback keyed onttsModel.toLowerCase().includes("kokoro")— all rendered unchanged for a liveqwen3ttsselection. The STT half hardcoded a "Language: English" row from moonshine's English-only limitation while the live catalogue is multilingual Whisper-Large-v3-Turbo. Every one of those is now keyed on the selection's actualprovider, which both the catalogue rows and the persisted selection already carried. Checked against the engine rather than guessed:qwen3tts_server.pydoes clamp speed to 0.5–2.0 (so that hint survives for both bundled engines) but its default voice isRyan, notaf_bella, and its sample rate is whatever the loaded model's codec reports at startup, not a fixed 24 kHz. doctor allreported "Slot ports — slots endpoint unreachable" on a box that was serving every slot (#1501). The check never probed reachability; it inferred it._get_anyswallows aCliApiErrortoNone,check_portsrendered anyNoneas "unreachable", andapi_get's budget is 10.0s — butGET /api/slotsis the slowest read-only route hal0 has, because the aggregator merges SlotManager entries with upstream-backed ones and container-probes each, so its cost scales with slot count. Measured on lxc105 (19 slots): 11.2–14.6s.httpx.TimeoutExceptionsubclasseshttpx.HTTPError, so the timeout became aCliApiError, becameNone, and became a false negative on the operator's first-line diagnostic — the failure mode that teaches people to ignoredoctorright up until it is telling the truth. The slots probe now gets its own 30s budget (SLOTS_PROBE_TIMEOUT_S), and_get_anytakes a per-call timeout so one slow route doesn't loosen the budget for every other probe in the roll-up.check_portsalso stops conflating two different faults the waycheck_model_storenever did: aNonebody ("no answer — down, or slower than the probe budget") is a distinct row from a non-list body ("unexpected slots payload"), because a shape regression being reported as a connectivity problem sends the operator to the wrong place. Both rows now name a follow-up, and the command they name exists: a newhal0 doctor portslists each slot's bound port and fails on a collision — previously that row was the only failing row in the table pointing nowhere (No such command 'ports'. Did you mean 'perms'?). Note this makesdoctorhonest about a slow endpoint; it does not make the endpoint fast, and the dashboard polls that same route.- The per-slot log stream had no keepalive, so a quiet slot's SSE connection was reaped and every reconnect duplicated up to 400 lines (#1472).
tail_journalyields only journalctl output, so a slot that iswarming, idle, or simply not logging emitted zero bytes — verified on the live box, where a 4-secondcurl -sNagainst a warming slot returned nothing at all. Any proxy idle timeout then dropped the stream while the client still reporteddisconnected=false; the client's own reconnect re-opened the same URL, the server replayed its 400-line default backfill, and the slot-log ring appends without content-dedup (correctly — raw journald repeats progress-bar lines legitimately), so each drop could duplicate 400 lines. Both sibling SSE routes already pulsed at 15 s; this one was the exception.tail_journal_keepalivenow yields an idle tick on the same cadence, and reconnects requestbackfill=0. The wrapper pumps through a queue rather than the obviousasyncio.wait_for(agen.__anext__(), …): that cancels a half-executed async-generator frame on every idle tick, and an async generator is not re-entrant across a cancelled step — on a quiet slot that failure mode would be the normal path, not an edge case. The stale comment inuseLogs.tsjustifying dedup-free append ("the backend backfills once with no replay on reconnect") is corrected; it was false for self-initiated reconnects, which is precisely when it mattered. GET /api/slots/{name}/logsreturned a bare empty string with no explanation (#1472).read_tail's contract promises("", <hint>)when a unit has never started, and the route only attaches ahintkey when one comes back — but the success path returned(text, None)unconditionally, so a journalctl that ran fine and printed nothing produced{"logs": ""}and a blank pane with no reason. Now hinted, including whenquiet=Truefilters a noisy-but-real tail down to nothing, since the operator sees the same blank pane either way.- Retired
lemondreferences removed from live dashboard copy and gating (#1472). The lemonade daemon was removed in #687, but the board task drawer still told operators "worker streaming · tail attached to lemond journal",chrome.jsxcarried a colour token for a source that can no longer appear, and the Scheduler card justified its permanent gate with "lemond dispatcher is stateless". The card stays gated — checked rather than assumed:/api/stats/requestsis a throughput rollup, not a queue, andsingle_flight.in_flight_keys()is in-process and unrouted, so there is still no scheduler telemetry to render and ungating it would mean inventing a source. Only the rationale was wrong, and it now names the real reason. - The Agents overview claimed Pi and Turnstone stream live status; neither card can ever have a record (#1472). The header read "Hermes, Pi, and Turnstone are live — their cards stream real install/endpoint status", and both cards looked themselves up in
GET /api/agents. That lookup can never hit:BUNDLED_AGENTSis("hermes",)andPOST /api/agents/install404s any other name, so both rendered a permanent "not installed" — which reads as installable, just not installed yet, contradicting the page's own legend. Both are labelled as roadmap entries now and the dead lookup is dropped rather than left implying a signal exists; the copy says only what is true. - Board mutations toasted success before the request resolved, with no failure feedback (#1472). Comment, link, dep-removal, specify, decompose, status change, reassign, dispatcher nudge and lane drag all fired their toast synchronously with
.mutate(), and none of the board hooks surfaced errors — so a 4xx/5xx (notably the 409 If-Match conflictboard.pydocuments) still read "posted"/"linked"/"queued" while the change silently vanished on the next refetch, defeating the one job the toast has. Every one now settles first and reports which way it went. On multi-id operations success is announced once but failures are announced per id, because "which one didn't move" is the detail worth having. - The dashboard's Operator Board chat hardcoded
model: 'hal0/brain'on every turn, permanently defeating[brain_chat].modeland the persona'spreferred_model(v1.0 GA polish, chat sweep #1469).board_chat.py(a transparentsys.modulesalias ofbrain/chat.py) resolvespayload.get("model") or cfg.model or default_model— an explicit client-sent model always wins, so the dashboard's hardcoded constant made the config override and the persona default dead for the primary chat surface, and its own comment claiming the model "routes to the agent slot" contradicted the constant it sent (hal0/brain). The dashboard now omitsmodelentirely and lets the server-side precedence chain decide. - The omni router picked a caller slot by declaration order, not health, when two slots bound the same model id (v1.0 GA polish, chat sweep #1469).
_maybe_run_omni_loopmatched the request's model against configured slots'model.defaultand took the first match — the same class of bug #1418 already fixed for the backend-aware load path, and observed for real on lxc105 (two slots sharing one checkpoint). Now reuseshal0.dispatcher.lane_pin.preferred_slot/lane_slot_pin, so an ERROR-parked slot declared first no longer wins over a healthy sibling, and an explicit lane pin from upstream resolution is honoured. - A failed omni tool-calling loop returned HTTP 200 with a bare
{"error": "<string>"}body (v1.0 GA polish, chat sweep #1469).OmniRouter.run_loop's failure contract (transport/HTTP/non-JSON failures, and the loop-budget-exhausted fallback) is a plain{"error": ...}dict —_maybe_run_omni_loopjson-dumped that straight into a 200Response, indistinguishable from success on status code alone. It now raises a typedHal0Error(omni.loop_failed, 502), rendered through the same structured{"error": {"code","message","details"}}envelope every other/api/*failure uses. - 7 of the omni router's 8 tools were permanently inert:
required_model_labelsgated on[model].labels, which nothing writes (v1.0 GA polish, chat sweep #1469). The label overlay (resolve_for_request's step 2) required an exact hand-authored TOML label match with no fallback — every live slot TOML carries zero labels, sogenerate_image/text_to_speech/transcribe_audio/analyze_image/embed_text/rerank_documentscould never find an eligible slot even when a correctly-typed, correctly-capable one existed.LoadedSlotnow also carriesmodalities, the registry's already-populated fact-derived capability signal (hal0.model_meta.modality.derive_modalities_from_model_info— mmproj presence,pooling_type, backend family; the exact §7.1d fallback patterntool_callingalready got). The label overlay falls back to it for any label that folds onto the closedModalitytaxonomy (vision/tts/image, and the tool-taxonomy aliasestranscription/embeddings/reranking). Struck from the checklist, not implemented:edit_image'seditlabel has noModalityequivalent — nothing in the registry distinguishes an image model that can edit from one that can only generate, and inventing that schema is a real design decision out of scope for this sweep;editstill only matches an explicit hand-authored TOML label (documented + regression-tested, not silently over-matched to any image-capable slot). - QuickChat discarded the operator's message on a failed send, with no way to retry (v1.0 GA polish, chat sweep #1469).
send()cleared the input unconditionally before the async stream could possibly fail, so a rate-limited or crash-looping slot lost the typed message for good — despite the backend already shipping aretry_after_shint in the structured error envelope for exactly this case. A failed send now restores the message to the input box, surfacesretry_after_swhen present, and a Retry control resends without retyping. - Duplicate CHANGELOG subsection headings silently dropped entries from
release.json(#1499).extract_structuredassigned per heading instead of accumulating, so when one version section carried the same### Breaking(or### Migrations) heading twice — the routine result of union-resolving a CHANGELOG merge conflict — only the last block survived. That digest is whathal0 updaterenders as its breaking-change and migrations callout before an operator confirms, so a dropped bullet meant the confirm banner under-reported what the update actually does: a quieter safety prompt, with no error and no warning. Repeated headings now accumulate in document order. - The slot drawer stops presenting three launch controls that had no launch effect (#1379).
spec-flags-ownership§1/§4 moved launch flags to the model tier — "Slots carry no user flag overrides" — andspec-hw-slot-ownership§8 prescribes the slot editor exhaustively as the 4-field HW grid plusimage_pin. Template (chat_templateoverride), Parallel and Extra Args were never removed from the drawer, so all three kept editing and persisting slot TOML keys the launch path had already stopped reading (providers/container.pydoesdel profile_flags, slot_parallel, extra_args;resolve_chat_templatedocuments the per-slot key as "no longer consulted";ServerConfig.extra_argsandSlotConfig.parallelboth describe themselves as "INERT at launch … Retained for TOML round-trip"). Worse than dead UI: Template also firedPOST /restart, so picking a chat format took the slot down for a model-load and changed no argv — the operator reads the unchanged behaviour as "it didn't work" and tries again; Parallel printed a confident explainer ("N slots share the {ctx}-token context pool (--kv-unified) …") describing argv never emitted; and Extra Args' Regenerate cleared its own stale-command overlay because the baseline then matched the typed value, while the resolved command came back byte-identical — the overlay disappearing was the only feedback, and it was a false positive. All three are removed outright rather than left read-only, the same call made for Reasoning/MTP/Vision underspec-hw-slot-ownership§1, and the Model group now signposts where the launch tune actually lives.chat_templateis dropped from the restart trigger with them. The removal is deliberately non-destructive: the three keys are absent from the drawer's frozen baseline and its derived change-set (#1447), so a slot TOML that still carries them round-trips untouched — this drawer neither reads nor clears config it no longer displays. Folding an already-persisted slot tune into the bound model remainshal0 slot migrate-flags, which shipped first for exactly this sequencing (#1396/#1397). Theextra_argsshlex gate goes too, which makes #1389 — a validator seeded from persisted state vetoing saves from an unmounted subtree — unrepresentable rather than merely fixed; its contract test is kept as a regression guard.tests/e2e/specs/slot-drawer-sunset-removal-v3.spec.tspins the absence AND the wire-level guarantee that no Save can putchat_template,parallelorserver.extra_argson the wire even when the slot TOML carries all three; the nine tests that asserted these writes (field-wiring W3/W7,slot-edit-controlsextra_args + Parallel,slot-drawer-profileC7k) are removed with the surface they covered. scripts/dev-bootstrap.shleft the Vite dev server running after shutdown, holding the UI port. The cleanup trap killed$!for each service, but the UI job is a subshell that runsnpm run dev, which forksnode; killing the wrapper reaped neither. So Ctrl-C (or any TERM) took down the API and left Vite alive onUI_PORT— and becauseplaywright.config.tssetsreuseExistingServer, the next run could attach to that orphan and report results for the code it was started with. The script now enables job control (set -m) so each service is its own process-group leader, and cleanup TERMs the whole group before escalating to KILL. Verified by starting.superset/run.sh, tearing it down, and confirming both ports free with no survivingvite/uvicornin the worktree. The container name is nowHAL0_OWU_CONTAINER-overridable for the same reason, andHAL0_DEV_SKIP_OPENWEBUI=1drops the hard Docker requirement for checkouts that don't need it.search'sbefore/aftertime-window filters were accepted and silently dropped on the durable engine, while the degrade fallback honoured them (#1471).HindsightProvider.searchtookbefore/after/modeand delegated torecall()without them; the REST route documents and forwards them and the MCP tool schema exposes them, so a caller filtering by time window got unfiltered results with no error — and got correctly filtered ones fromPgVectorProviderwhen memory was degraded. Two engines, two behaviours, one advertised contract. The window is now applied to the merged result, with the token budget over-fetched so a filtered page can still filllimitinstead of silently returning short. Deliberately not mapped onto Hindsight'squery_timestamp: that is an "as of when" for temporal reasoning, not a range filter, so folding a window into it would quietly change what the query means.mode("vector"/"graph"/"hybrid") was inert everywhere it was declared; a non-default value is now a clear error naming the engine rather than a silent downgrade to vector recall, since a caller explicitly asking for graph traversal getting plain recall with no indication is the same class of bug.memory_listpagination was dead on Hindsight — only the first page of a bank was reachable (#1471).list_itemsaccepted acursorand never used it, always callinglist_memories(offset=0)and returningnext_cursor: None, while the MCP tool advertises "Page through long-term memory items" and the live shared bank holds 1629 facts — so everything past the firstlimit(max 200) was silently unreachable. Cursors now encodebank@offset, which is all Hindsight's/memories/listneeds since it already supportsoffset. Two ordering bugs went with it: the visibility filter now runs before the page is cut (the loop used to break on the raw count, so a unified-mode reader whose page was mostly other agents' private docs received fewer thanlimititems even when more visible ones existed), and each bank is over-fetched so a heavily-filtered page still fills. A cursor naming a bank the caller may no longer read returns empty rather than walking it — a paging token minted while access was granted must not outlive the grant.- The memory graph explorer rendered an engine outage as an empty bank (#1471). The stage showed "No graph data for this bank/filter." whenever
!loading && nodeCount === 0, and never consultedisError— so a 503memory.unavailable/memory.engine_unreachablewas indistinguishable from a genuinely empty graph, which is exactly what a new install has. It now branches to an explicit "Memory engine unreachable — {message}" state with a retry affordance, covering the subgraph and ego queries too (on a big bank the stage renders their payload, so their failure was equally invisible). Mirrors the treatment the Overview engine card and the settings panel already had. - Graph-extraction ON/OFF is labelled honestly instead of implying it stops the work (#1471). Nothing consumes
[memory.graph].enabledbeyond reporting and extraction-slot routing — Hindsight builds its graph natively inside the daemon and keeps extracting either way, which is why/graph/statuscounters (real Hindsight operations, read back by_augment_build_counters) keep rising after a disable. The surfaces claimed otherwise: the CLI printed "In-flight builds cancelled." (nothing was cancelled), the Agent panel said the graph "isn't built" when off (it is), and Settings described the checkbox as gating background extraction. All three now say what the flag actually does. Not enforced into a real gate: doing that means reaching into the daemon's extraction pipeline, which is a behavioural change that wants its own issue rather than a GA polish pass — the reporting lie is what shipped, and that is what is fixed. The CLI docstring's stale--route/--provider/--modeloptions are dropped too. - Bumped
postcsspast the path-traversal.map-disclosure advisory (v1.0 GA polish, #1476).ui/package-lock.jsonlockedpostcss@8.5.15(<= 8.5.17vulnerable, dependabot alert #32, high). Build-time only — postcss runs in the UI build, not at runtime — but it was the sole open HIGH dependabot alert onmaingoing into GA.npm update postcssinui/bumps to8.5.25(well past the8.5.18fix); UI build verified clean. - The login/rotate rate limiter (and caller-IP attribution) can now honour
X-Forwarded-Forbehind a reverse proxy, opt-in (v1.0 GA polish, #1476)._client_ipread the raw TCP peer unconditionally, so behind hal0's first-class reverse-proxy deployment (HAL0_PUBLIC_URL) every caller resolved to the proxy's own IP — a single remote guesser exhausted the shared per-"IP" budget and 429-locked out the operator too. New[security].trust_forwarded_for/HAL0_TRUST_FORWARDED_FORtoggle (same precedence and persisted-config pattern as[security].require_auth), OFF by default since a client can forge the header — only enable it once the reverse proxy is known to strip/overwrite any client-suppliedX-Forwarded-For(Traefik/nginx/Caddy do this by default). When enabled, the leftmostX-Forwarded-Forentry is used instead of the raw peer. hal0 benchis now documented incli.mdx, and the docs-parity test can no longer be fooled by a substring match (v1.0 GA polish, #1474). All 11 argparse verbs (plan/run/status/worker/results/history/reindex/devices/publish/eval/import-v1) shipped with zero doc coverage —tests/cli/test_cli_docs_parity.py'spath not in textcheck treated the substringbenchinsidebench-tunedas a match, so the parity guard passed while the command was fully undocumented. Matching is now word-boundary (bench-tunedno longer satisfiesbench), and a new parity walk overhal0.bench.cli's argparse subparsers — the surface_walk()'s typer traversal can't see past the single top-levelbenchpassthrough command — catches a future verb shipping the same way. Also dropped the test docstring's stale claim of a curated "Planned" section check that didn't exist in either the doc or the test.hal0 slot migrate-id-keyingnow matches its four siblings' dry-run contract (v1.0 GA polish, #1474).migrate-id-keyingapplied its destructive rename by default (--dry-runopt-in), whilemigrate-hw/migrate-caps/migrate-flags/migrate-enabled-removalare all dry-run by default (--applyopt-in) — an operator habituated to "run bare to preview" by the other four got a real migration from the fifth.--applynow gates the write;--dry-runstill parses (a no-op alias, since dry-run is the new default) so existing scripts keep working.cli.mdx'shal0 setupsection now documents all 8 flags (v1.0 GA polish, #1474) —--no-slots,--answers,--emit-answers, and--plan/--dry-run(shipped via #1115/#1116/#1117) had no doc line; the parity test can't catch missing flags, only missing commands. Also reconciled the--autoslot-seeding prose with whatinstall.shactually invokes (--auto --no-pull --no-extensions— slots yes, models no) rather than the stale premise that--no-slotsis the installer's default call.- install.sh's repair/upgrade-in-place path now runs the same post-activation migrations
hal0 updatedoes (v1.0 GA polish, #1475).Updater.commit()runs five passes after a self-update swap (schema migration, seed-profile pruning, stale-MTP clearing, runner-image retagging,defaults.extra_argssanitizing); install.sh's re-run path called only two of them directly, so a box upgraded by re-running install.sh kept a stalemeta.schema_version, stale runner-image pins, and unsanitiseddefaults.extra_argsthathal0 updatewould have fixed — two boxes on the same version, different on-disk state. Extracted the sequence into one sharedrun_post_activation_migrations()(hal0.updater.updater), called from bothUpdater.commit()and install.sh's venv-python block, so the two upgrade paths converge. Updater.rollback()now re-renders slot units, mirroringcommit()'s own step (v1.0 GA polish, #1475).commit()re-renders every slot unit through a fresh interpreter after the venv re-pip so any subsequent start (systemctl restart, crash-restart, reboot) uses current argv;rollback()did the symlink swap and the #980 re-pip of the prior tree and then stopped — a rolled-back box kept units carrying argv rendered by the version it had just rolled away from. The re-render logic is now a shared_rerender_units_after_swap()helper used by both paths.- Curated static slot seeds (agent's
chadrock-moeprofile,brain's tuned profile,embed's 4096 context, ...) now reach a fresh install (v1.0 GA polish, #1475). install.sh ranhal0 setup --auto— which scaffolds the same slot names with generic derived profiles — BEFORE copying the curated seed TOMLs; both sides are never-overwrite, so--auto's generic profiles always won the race and the curated seeds never landed on the standard install path. Moved the seed-copy loop to run first. - The
brainseed no longer pins an unresolvable model id (v1.0 GA polish, #1475).brain.tomlshippeddefault = "MiniCPM5-1B-Agentic-Tooluse", an id in neither shipped catalog — the operator could not pull it from any dashboard surface, and post-#1408 a bound model is the sole activation signal, so the slot showed "activated" while unable to actually serve. Reversed the #1258 "brain ships ready" exception:brainnow ships model-less like every other seed (the steward chat already falls back to theagentslot until a real model is bound). - The boot-time static-slot seeder now registers a newly-seeded slot's identity in the same boot (v1.0 GA polish, #1475).
seed_static_slotsruns in a later boot phase thanfold_identity, so a slot it adds fresh (e.g. a new release'scoder/embed/qwen3ttsseed reaching an upgraded box) sat name-keyed with no identity row for the rest of that boot — the exact name+id coexistence #1422 reports as duplicate/api/slotsentries. A re-fold (idempotent, additive — no artefact/unit rename) now runs immediately after a non-empty seed batch. ProfileCatalog.delete's in-use guard now also catches modeldefaults.profilereferences (HAL0-41, #1437). The 409profiles.in_useguard scanned slot TOMLs (slots_using) but never the model registry, so a model that preferred a profile viadefaults.profile— without yet being bound to any slot — could not block the delete, leaving it with a dangling reference the moment the profile disappeared.ProfileCatalog.deletenow also scans the model registry (ProfileCatalog.models_using) and folds both slot and model hits into one conflict; the errordetailsgains amodelskey alongside the existingslotskey, so existing callers readingdetails["slots"]are unaffected.POST /api/profiles/importnow returns 201 on commit, matchingPOST /api/profiles(HAL0-41, #1437). Both routes create a profile, but import fell through to FastAPI's default 200 — an inconsistent envelope for the same "resource created" outcome. Thedry_runbranch creates nothing and still returns 200.- Dropped the vestigial
enabledcolumn surface from the slot identity table (#1383). #1369 removedSlotConfig.enabledfrom the TOML schema (model-presence is the sole activation signal), but the SQLiteslotidentity table (hal0.slots.identity) still carried the flag:SlotRow.enabled,create()'senabledparameter,set_enabled(), andlist_by_type(enabled_only=...). Since #1369 every row is registeredenabled = 1in perpetuity (migrate_id_keying.py'sslot_tbl.get("enabled", True)can only ever fall through to its default, as no slot TOML carries the key anymore), soset_enabled()had zero production callers and theAND enabled = 1filter inlist_by_typewas a no-op predicate — dead code that read as meaningful and invited someone to "wire it back up," reintroducing the two-sources-of-truth problem #1369 exists to delete. All of it is gone:SlotRowno longer carriesenabled,create()no longer accepts it,set_enabled()is deleted,list_by_type()no longer acceptsenabled_only, andmigrate_id_keying.pyno longer feeds the column from TOML. The SQLiteslot.enabledcolumn itself stays on disk (additive schema,DEFAULT 1) — no destructive migration; simply nothing reads or writes it anymore. - A model row could still advertise
visionwith no projector — the fix for #1380 was client-side only, and two other doors were wide open (#1393, #1394, plus an unfiled third). PR #1392 gated the model drawer's Save on thevision capability requires an mmproj sidecar patherror, so the invariant held in exactly one of the several places a registry row can be written;screen_model_write— the shared screen forPOST /api/models,PUT /api/models/{id}andPOST /api/models/validate, whose docstring enumerates what it checks — looked only atdefaults.extra_args, soPUT {"capabilities":["chat","vision"],"mmproj":null}via curl, the CLI, or MCPmodel_editpersisted the same broken row and the launch path got no--mmprojto hand llama-server, loading text-only under a vision alias. The invariant now lives in one predicate,models_service.screen_vision_mmproj, raisingmodel.vision_requires_mmprojin the envelope the dashboard already keys on. Two decisions the #1393 write-up flagged: sparse PUTs are screened body-over-stored (the route hoists its pre-update snapshot above the screen and passes it asexisting), so a body that addsvisionwithout mentioningmmprojis checked against the persisted projector rather than an absent key — and/validateresolves the same row from an optional bodyidso the drawer's dry run matches the save; pre-existing rows are not retro-broken, because the screen fires only on a write that touchescapabilitiesormmproj— a rename on a legacy projector-less vision row still saves, while every write that could create or preserve the pairing is refused. Also new at that boundary: anmmprojpath that isn't a file on this host is nowmodel.mmproj_not_foundat SAVE instead of a slot that fails at load — safe to check eagerly because the pull worker writes the post-download path straight throughregistry.update, never through this screen, so a still-downloading sidecar can't trip it. - The Add-model-from-HF modal shipped the identical decorative error, and its Pull didn't go through the model-write screen at all (#1394).
model-modals.jsxrenderedvision label requires an mmproj file — pick one belowwhilecanPull = inspected && variant && name && !pullJob.inFlightcarried nommprojterm, so tickingvisionwith nothing selected left Pull live andonPullsentmmproj_filename: undefined. The modal writes its registry row throughPOST /api/models/{id}/pull→seed_registry_from_body, a different door entirely — the #1393 screen would never have seen it, so the pull route now screens the bodylabelsagainst the sidecar the job will install (require_on_disk=False:mmproj_fileis an HF filename inside the repo, not a host path) and 400s before seeding or scheduling anything. Client-side, the error folds into the singlecanPullgate the way #1392 did for the drawer, and the aggravating case gets an honest escape hatch: when the inspected repo ships no mmproj at all, "pick one below" is unactionable, so the message becomes untick vision to pull — the pull itself was always fine, only the label is unsupportable. The error div and the projector<select>gaindata-testids (their absence is why this twin went unnoticed when #1380 was fixed). - The Voice/Image-gen model pickers never populated — the pages indexed an
.items/.modelswrapper that/api/capabilitiesdoesn't ship (#1454).GET /api/capabilitiesreturnscatalogs.voice.stt,catalogs.voice.tts, andcatalogs.img.imgas bare arrays of picker rows ({id, capabilities, size_gb, backends}—mockFixtures.tsbuildCapabilities()already documented the real shape);VoicePage.jsxandImageGenPage.jsxreadvoiceCatalogs.stt?.items || voiceCatalogs.stt?.models || [], and an Array has neither property, socatalogItemswas always[]. The<select>branch was dead code — every model field fell back to a free-text input, and the "no installed STT/TTS/image models — install one in the Models view" hint rendered even when models were installed and serving. The three catalog reads now use the array directly (voiceCatalogs.stt || [], etc.);useCapabilities.ts'sCapabilitiesBagtype is corrected from the obsolete pre-orchestrator{capabilities: Record<...>}envelope to the real{backends, catalogs, selections}shape./api/capabilitiesalso gainsnetworkFirst: truein the mock allowlist so a spec'spage.routeoverride is authoritative in forced-mock e2e, matching the siblingprofiles/stacks/chat-templatesrows. - The ComfyUI pane's "models" block rendered a fabricated hardcoded inventory (6 checkpoints, 11 loras,
'Wan 2.2 · Qwen-Image · HunyuanVideo 1.5 · LTX-2') and never consumed the backend's verified counts (#1455).comfyui-pane.jsxmounted<ModelsBlock />with no props, so itsINV_DEFAULT/MODELS_DEFAULTfallback rendered unconditionally — even though_model_inventory()(src/hal0/api/routes/comfyui.py) counts real weight files per category on the model share and/api/comfyui/statusalready carried the result asinventory.transformComfyuiStatus(useComfyui.ts) never mapped that field into the pane's data shape, so the wiring gap was silent.inventorynow flowsstatus.inventory → transformComfyuiStatus → ImageGenCard → ModelsBlock, rendered as<count> <category>pills (checkpoints/diffusion/loras/vae/controlnet/upscale/text_encoders); the block is omitted entirely — not rendered with zeros — wheninventoryisnull(the backend's fresh-install contract, model share root absent). Also removed two secondary fabrications the same audit flagged: the propless<BarSpark />sparkline in the system-ram metric tile (no real per-tick RAM history exists behind it) and the non-functional "manager ↗" note on the models block header (linked to nothing). - The model drawer reverted other people's edits, and its Save button fired with nothing to save (#1441 — closes the drawer dirty-tracking class filed as #1398, whose slot half landed just before). Same shape as the slot half: the drawer seeds its form once (effect keyed on
[open, model?.id]) but answered "did this field change?" against the livemodelprop. Both callers hand it a live-polled value —models.jsxdoesmodelList.find(m => m.id === selId)offmodelsQuery.data(useModels: 30s poll plus an invalidation on every model mutation), and the slot drawer's stacked editor does(modelsQuery.data ?? []).find(m => m.id === curModelId)— so the in-file comment claiming the prop was "a SNAPSHOT captured when the drawer opened" described neither. Three consequences: (a) a concurrent write (another operator, a slot-drawer save, the CLI) moveddefaults.context_sizeunder the open drawer, the untouched field went dirty, and Save wrote the drawer's stale seed back over it — a lost update caused by doing nothing; (b) Save was never gated on the dirty aggregate (#1441), and sinceonSaverebuilds the entiredefaultsblock a zero-edit Save was not the harmless no-op the issue assumed — it rewrotecontext_size,extra_args,chat_template,profile,n_gpu_layersand all four tri-state caps; (c)onSavestarted from{ ...init }— read live — so the keys the drawer never renders (rope_freq_base, …) rode along as a mid-edit poll left them rather than as the operator saw them. Fixed with the seam the slot drawer got:modelBaselinesnapshots the row once at open (including the wholedefaultsblock for pass-through, and the already-canonicalised capability list so a late/api/meta/enumsload can't fabricate a diff),deriveModelChangesderives the comparison once, and both the unsaved-changes guard and the save body read it — no predicate computed twice, nothing reading the live prop. Save is now disabled when nothing changed. Unlike the slot half there is deliberately no degraded-payload guard:useModelshas no union/soft-fail fallback — the/api/statusleg offetchSlotsUnionis what made the slot-side degradation (#1391) possible — so nothing can hand this drawer a shape-degraded row.tests/e2e/specs/model-drawer-dirty-baseline-v3.spec.tspins all three on the wire. - The slot drawer wrote fields nobody touched, because its dirty-tracking compared a once-seeded form against a live-polled prop (#1390, #1391 — the structural fix for the class filed as #1398). The drawer seeded its form once (effect keyed on
[slot?.name]) but answered "did this field change?" against theslotprop, whichuseSlotsre-derives every 5s. The two sides drifted with no operator input, and the drawer read that drift as an edit. Two live consequences: (a)ctxBaselinefell back toslot.metrics.ctx, a runtime metric — open the drawer on a cold slot (field seeds to the 8192 floor), let it start serving, touch nothing, click Save, and it wrotePATCH /defaults {"ctx_size": 8192}, turning a transient observation into a persisted--ctx-sizethat pins the slot's context window from then on; (b) one dropped/api/slotspoll degrades the entry to the bare/api/statusshape, which carries no config enrichment at all (config_enrichmentruns only in the/api/slotsbuilder), so for that interval every batched field read dirty and an idle Save rewrotechat_template/binary/n_gpu_layersand firedPOST /restart— a network blip became a cold model reload. Fixed by freezing the baseline rather than patching the two symptoms: the drawer snapshots the persisted config once, from the enriched payload it opened with (configBaseline), and both the unsaved-changes guard and the Save body now read a single derived comparison against that frozen snapshot (deriveChanges) — no predicate is computed twice, and nothing reads the live prop. The snapshot refreshes only on slot identity change, drawer close, or an explicit save success.useSlotsadditionally tags each entry with_configEnriched— provenance (did this come from/api/slots?), not a key sniff, since every config key is legitimately absent when the slot has no override on disk — and the drawer refuses to compute dirtiness while it reads false, disabling Save behind a "Slot data degraded — reconnecting…" hint instead of guessing. In-flight edits and the last good baseline survive the blip and the drawer recovers intact on the next good poll.tests/e2e/specs/slot-drawer-dirty-baseline-v3.spec.tspins both instances on the wire (zero writes) — a spurious write is indistinguishable from an intentional one in the API log, which is how the class went unnoticed. The degraded path is now reachable from the harness at all viawindow.__hal0MockSlotsDegraded, which failsGET /api/slotsand strips the config keys off the/api/statusunion entries. - Self-update works on the shipped
User=hal0posture — it was structurally impossible before (#1464). P3-perms flippedhal0-api.servicetoUser=hal0and pinned/usr/lib/hal0toroot:root 0755, declared ininstall/perms.pyas "never service-writable at any point". The updater never got the matching seam:Updater.prepare/commit/rollbackran entirely in-process insidehal0-apiand wrote that tree directly — extract to<lib>/hal0-<version>/, atomic-swap<lib>/current,pip install --force-reinstallinto the root-owned venv. Every one of those isEACCESfor the service account, so a v1.0 install could never take v1.0.1, and the failure landed only after a full download + sha256 +cosign verify-blobpass, surfacing as a rawUpdateExtractError('Permission denied')in the dashboard. There was no escape hatch: the CLI never invokes the updater directly (it POSTs to the API), the UI hits the same route, and thehal0-systemctlwrapper has no staging/swap/pip verbs. The three privileged phases now route throughhal0-update— a new narrow sudo seam built exactly likehal0-systemctl(installer/wrappers/hal0-update+packaging/sudoers/hal0-update, grant pinned to the binary, every argument validated, no shell, no wildcards), fronted byhal0.updater.privileged.UpdateSeam, which mirrorsSystemCtlSeam's gating: it engages only when the process runs as the literalhal0service account, so a dev shell, a CI runner and every unit test stay on the pre-existing in-process path. The grant surface is four verbs —check(non-mutating probe),stage <channel> [version],activate <hal0-VERSION>,discard <hal0-VERSION>— and the only things that cross it are a channel from a three-value allow-list, an optional exact version pin, and ahal0-<version>directory basename (never a path, never a file body, never a URL).stagedeliberately performs the manifest fetch, digest check and cosign verification root-side: the smaller-looking split — unprivileged download, privileged install — would let a compromised hal0-api skip verification and have rootpip installan attacker-supplied tree, i.e. turn the grant into arbitrary root code execution.activateadditionally refuses any tree that is not root-owned and free of group/other write. Everything that writes hal0-owned state — config migrations, the seed-profile/mtp/extra-args/image sweeps, thehal0.previousbreadcrumb, the slot-unit re-render — stays unprivileged on purpose, so root never re-owns/etc/hal0or the SQLite files out from under the service.prepare/commit/rollbacknow also open with a privilege preflight (system.update_privilege_denied) that fails in milliseconds with remediation text, instead of after a multi-hundred-megabyte download. Two behaviour notes:rollbackrefuses ahal0.previousbreadcrumb pointing outside the install root (the seam only takes a basename), and on a box wherecurrentpoints at a non-root-owned tree — a hand-deployed git checkout, not a released install —activaterefuses with thechowncommand to fix it. hal0 doctornow verifies the sudo seams every slot op and update depends on; a failed grant install aborts the installer (#1465). Every privileged operation post-P3-perms goes throughsudo -n /usr/lib/hal0/bin/hal0-*, andinstall.shinstalled each wrapper +/etc/sudoers.ddrop-in best-effort: avisudo -cffailure or a missing source produced only a mid-logwarn, after which the run proceeded to its success box. Nothing verified the result afterwards —preflight_all's 15 checks never touched sudoers,doctor verifycomposes only live-API rows,doctor all's extras were auth/model-store/migrations/ports/hal0.target, anddoctor permshad a row for/usr/lib/hal0but none for the wrappers themselves. So a box where that warn fired reported all green from every doctor surface while every slot start, unit write anddaemon-reloadfailed undiagnosably. Three changes:hal0.system.seam_checkis the new predicate (wrapper present +root:root 0755; drop-in present +root 0440— sudo silently ignores a wrongly-moded one, so that is a total failure, not a nit; and the fact that actually matters,sudo -n <seam> <probe>exiting 0 as the hal0 user, viasudo -n -u hal0 sudo -n …when run from root).hal0 doctor allgains a Privileged seams row wired to it — a broken required seam (hal0-systemctl,hal0-update) is an actionablefail(exit 1) naming what breaks and how to fix it; an optional seam or a grant that cannot be tested from the current account is an advisorywarn, never a silent pass. Andinstall.shnow dies rather than warns when a required wrapper or grant fails to install, requiresvisudoon a real install (hal0 cannot run withoutsudo), and runspreflight_seamsas a post-install assertion once the wrappers are down — deliberately not part ofpreflight_all, which runs before the seams exist.doctor permsalso gains a row for<lib>/bin/hal0-*so a drifted wrapper mode is repairable by--fix.PUT /api/models/{id}merges the nesteddefaultsone level deep instead of replacing it wholesale (#1413, server-side sibling of #1378/#1381). The route documents itself as "body accepts any subset of …defaults", butregistry/store.py::merge_updatewas flat —{**existing, **updates}— so a body that so much as nameddefaultsreplaced the entireModelDefaultstable. Every sibling the client didn't resend (context_size,rope_freq_base,chat_template,profile,mtp,jinja,enable_thinking,vision) reset tonullbehind an HTTP 200, withchanged_fields: ["defaults"]as the only trace and a response body that looked correct because it was the new (lossy) state. Following the documented contract destroyed config; the dashboard escaped it only by accident, because its drawer resends the whole object.defaultsandcapability_flags— the two schema'd subtables_model_to_tomlalready special-cases — now merge per sub-key with the tri-state semantics those fields already carry: absent keeps the stored value, an explicitnullclears that one value, and{"defaults": null}still drops the whole table.metadatais deliberately excluded (freeform bag, no schema, so "absent = keep" would leave no way to drop a key) and every other field stays a flat replace. The three UI writers that cleared a field bydelete-ing its key (model drawer, recipe editor, MODELS ▸ Defaults) now send an explicitnull, since under the new merge an omitted key means "keep", not "clear".- A malformed
defaults.context_sizereturns 400, and 0 / negative / absurd values are rejected (#1414, server half of #1378).{"context_size": "abc"}escaped as an HTTP 500model.registry_errorcarrying a raw pydantic traceback indetails.reason—merge_updatewraps anyModel.model_validatefailure inRegistryError, so plain client garbage produced a 5xx. Meanwhile0,-1and99999999all persisted with a 200 and round-tripped throughGETinto the database, andPOST /api/models/validatereported{"ok": true}for every one of them, becausescreen_model_writeonly ever inspectedpreferred_runneranddefaults.extra_args. The client-side guard was therefore the only guard: the CLI, a script, or a stale dashboard build could still write a row guaranteed to fail at launch.screen_model_writenow parses the incomingdefaultsthroughModelDefaults(→400 model.defaults_invalid) and range-checkscontext_sizeagainst[128, 2**24](→400 model.context_size_out_of_range); the floor mirrors the model and slot drawers'≥ 128, the ceiling is an absurdity screen ~16× the largest advertised window on any shipping model. Screened at the write boundary rather than as aField(ge=…, le=…)constraint on purpose — a model-level constraint also applies on the READ path, so any already-persisted out-of-range row would stop loading, which is the exact "policy applied to data that predates it" trap #1411 documents on the profile side. Create, update and/validateshare the one screen so the three cannot drift, the wayextra_argsalready couldn't; MODELS ▸ Defaults gains the matching inline≥ 128gate. - Every container slot reported offline/stopped while it was serving — the status probe asked systemd about a name-keyed unit on an id-keyed box (#1417).
slot_instance_tokenis the §11.1 id-flip seam: post-migration it returns the durablecfg["id"], soload_synccreateshal0-slot@2.service/hal0-slot-2for the slot displayed asbrain. The lifecycle half resolved that token first; the query half —ContainerProvider.is_active/running_image/running_argv, plusslot_view's raw stopped-vs-crashedsystemctl is-active, which hardcodedf"hal0-slot@{name}.service"outside the seam entirely — passed the mutable slot NAME straight into the pure formatters.slot_unit_name()/slot_container_name()are formatters over a token, so a name silently yielded the pre-migration artefact:systemctl is-active hal0-slot@brain.service→inactivewhilehal0-slot@2.servicewasactive,podman inspect hal0-slot-brain→no such object. Becauseis_activeis the drift reconciler's only source of truth (SlotManager.status), every healthy slot was force-transitioned to OFFLINE on every poll — the dashboard showed threeUp (healthy)containers as not-loaded, the dispatcher readiness gate answered503 slot.loading/502 slot.load_failedfor slots that were generating and then tried to "recover" them,running_imagereturned None for every slot so the #663 image-is-the-backend record was inert, and Hindsight fact extraction against theutilityslot failed 170 retains with "slot 'utility' is offline". All three probes now take the slot config and derive the token through one new chokepoint (providers.container._artefact_token, back-compatible with an already-resolved token string), and all five callers hand the config down (slots/watchdog.pyis_active+readiness_check,slots/manager.pyreconcile_container_upstreams,slots/drift.py,slot_view— each already had the config in hand one line earlier). A name-keyed (pre-migration) box is bit-for-bit unchanged: with noid, the token IS the name.tests/slots/test_query_path_token_keying.pypins the invariant the seam exists for — given{"name": "brain", "id": 2}the probe half and the teardown half target the samehal0-slot@2.service. - Two slots on one model id: selection now prefers a slot that is actually serving, so the brain lane stops 502ing at an ERROR-parked sibling (#1418).
brain(id 2) andnano(id 11) both boundhal0-brain-sft-fpx8.brainwas loaded, healthy and generating (532 ms, 90.8 tok/s straight to its port);nano's restart failed and it sat in ERROR. Every path that maps a model id back to a slot took the FIRST candidate and health was an input at none of them: the route layer's backend-aware load (#430) reversed the alias map and droveload("nano"), and the dispatcher committed to the registry bindingmodel → nanobecause it was "online", after which the readiness gate raisedslot.load_failed. Two compounding facts made it unrecoverable: thehal0/<slot>rewrite collapses the lane to a bare model id before dispatch, so the dispatcher could not even tell which of the two slots the caller asked for; and the existing"registry binding offline; falling through"escape only fires on an offline upstream, never on an ERROR one that still advertises a warm model cache. Net effect:POST /api/brain/chat— the first-class hal0-brain steward surface, SPEC §G / R4 — was completely unreachable (502 slot.load_failed, upstreamnano), as was the model over/v1(503 slot.loadingafter 28 s). Three changes, one concern:hal0.slots.state.slot_selection_rankis the single ordering (dispatchable → loading → offline → ERROR last, unknown states ranked with ERROR);hal0.dispatcher.lane_pincarries the resolver's matched slot forward onrequest.state(set only when the chain matched a LIVE slot, so a pin can never point at a slot that cannot serve) and owns the sharedpreferred_slot/rank_slot_namehelpers; and the dispatcher orders slot-backed upstreams by that rank within the positions they already occupy — a genuine remote (OpenAI, OpenRouter…) never moves, so this can only change which of two slots wins — plus falls through a registry binding that a healthier slot outranks on warm-cache evidence alone (no new probe on the hot path). A model with one candidate slot is unchanged in every path, ERROR included: the readiness gate keeps owning the retry/recover envelope.tests/dispatcher/test_same_model_slot_health.pypins the lxc105 shape (nano declared first, in ERROR; brain second, READY) at both layers, plus the lane pin, the health tie → declaration order, and the single-slot no-ops. - The single-file pull path now writes
store_blob/model_filerows, so refcounting and store GC are no longer inert (#1412).run_pullhas two branches, and only the fileset one (_run_pull_fileset) did the blob accounting —_register_blob_after_installper file plus onemodel_filerow each in_register_pulled_fileset. The single-file / mmproj-pair branch — the one the Add-by-HF-coords modal andPOST /api/models/{id}/pullwithhf_repo+hf_filenameactually take — went straight to_register_pulled, which only upserts themodelrow. On a box whose models all came through that path both tables were empty box-wide (63 registered models, zero rows on lxc105), which silently disabled every consumer of them:store_blob.refcountnever existed, so the store GC had no record of any pulled bytes;_maybe_hardlink_from_blobcould never hit, so an identical re-pull always re-streamed the whole file; andduplicate_model's documented safety — "bumps each shared blob's refcount, so a later delete of either row never orphans bytes the other still uses" — was a no-op returningfiles_refcounted: 0with no signal. Worse,gc.reconcile_store_treeclassifies anything under the store root that nostore_blob/model_filerow tracks as reap-eligible bare bytes, so a live pulled model looked like debris to a real (dry_run=False) GC pass. Pull completion now registers the blob and the file row for the main file and the WS-11 mmproj sidecar, keyed on the repo-relative filename, through one new_register_installed_fileshelper. It holds the invariant refcount == number ofmodel_filerows referencing the blob, which makes the write idempotent — a re-pull of unchanged bytes leaves the count alone rather than ratcheting it upward forever (an inflated count never falls back to 0, so the bytes would become unreclaimable), and a re-pull whose digest changed drops the superseded blob's reference first. No-op on the TOML registry escape hatch, same as the fileset path. Not included: a backfill for models pulled before this fix — the GC stays blind to those until one lands. add-from-pathno longer rejects every.ggufsymlink into the HuggingFace hub cache (#1415). The[models].file_extensionsallow-list was applied topath.resolve(), and the hub stores bytes as extensionless sha-named blobs with the real filename existing only as a symlink (snapshots/<rev>/<name>.gguf -> ../../blobs/<sha256>). So "Add by path" failed with400 model.unsupported_format— "file extension''not in [models].file_extensions" — for a path that visibly ends in.gguf, quoting an opaque sha blob indetails.paththe operator never typed. It fired on hal0's own conventions: the curated/mnt/ai-models/local/*symlink farm and the/mnt/ai-models/huggingface/hubtree thehftooling writes into — i.e. the most obvious "register the model I already have" case. The check now runs on the literal path the operator supplied, falling back to the resolved suffix only when the literal has none (so pointing straight at a bare blob behaves exactly as before), and the rejection echoes the operator's path.detect()and the storedModel.pathstill use the resolved target; the derived id and display name now come from the literal stem too, so a hub symlink no longer registers under 64 hex characters. The scan walker shares the allow-list but already tests the un-resolved walk entry (registry.discover.find_candidates), so it needed no matching change.parse_text_tool_callsunderstands the attribute-XML tool-call dialect, so hal0-brain's tool loop can actually fire (#1419).hal0-brain-sft-fpx8's chat template documents its contract as<function name="X"><param name="K">V</param></function>. llama.cpp's--jinjaparsers do not recognise that shape — the slot returnsfinish_reason: "stop"with notool_callskey at all and the markup inmessage.content— and hal0's own text fallback missed it too, because_FUNCTION_TAG_REonly accepted the equals form<function=NAME>{json}</function>. SoPOST /api/brain/chatandPOST /api/board/chatcould never execute a tool on this model, and the documented promise that "matched spans are removed … so the raw tool syntax is never shown to the operator" inverted: unmatched spans were shown. The parser now recognises the attribute form, collecting<param>children (CDATA unwrapped,{...}/[...]bodies JSON-decoded, everything else kept verbatim so"007"can't silently become7) — including llama.cpp's mangled rendering of it, which is what the live slot actually emitted: its partial-tool-call scanner eats the<function/<paramopeners before giving up, leavingname="get_weather"> name="city">Paris. Matching only the well-formed shape would have left the reported failure unfixed on the wire, so the tag openers are optional and the existingknown_namesgate — a token whose name is not a surfaced tool never starts a call — is what keeps prose out. Gemma-4's<|tool_call>call:NAME{json}<tool_call|>(pipes inside both delimiters, so it never matched<tool_call>…</tool_call>either) is covered too, per the issue's follow-up: three local models, three unparseable dialects, so the durable lever is the parser rather than the weights. A Gemma-4 call whose body uses the template's bespokekey:<|"|>value<|"|>encoding is deliberately not synthesized — inventing a call with silently empty arguments is worse than not firing. Not included: making an unrecognised tool-syntax fragment scrub itself fromcontent(the issue raises it as a separate decision), and the registry-sidetool_calling: true/chat_templateentry for the brain model./api/statusnow reports whether memory WRITES are landing, not just whether the daemon answers (#1420). On lxc105 no memory write had succeeded in hours — fact extraction 503'd against an offlineutilityslot, 170 operations sat failed, and/api/memory/listshowed nothing newer than 8 days ago — while/api/statusreportedmemory_degraded: falseandhal0 memory statusprintedState ON / Provider durable. Both were correct about the thing they measure:HindsightProvider.degraded(#1301) tracks daemon reachability, and the daemon was reachable. It accepted every retain with a200+operation_id, passed/health, and served recalls in 11s using only the local embedder and cross-encoder. The subsystem was half alive — reads fine, writes silently dropped — which is the worst possible shape for a green flag, and an operator, dashboard, or agent checkingmemory_degradedbefore trusting memory gotfalseand proceeded. Rather than wideningdegraded(which would break #1301's contract and misreport the read path, which genuinely worked), this adds a distinct signal:memory_write_degradedplus amemory_write_healthdetail object on/api/status, and matchingWrites/Operationsrows inhal0 memory status. It is fed by two observations of the write path itself — a retain that raises, and the engine's ownfailedoperation counter increasing between two samples, which is the half that catches the reported box (there, no retain ever raises). The delta comparison is deliberate: the counter is cumulative, so an absolute threshold would never return to green after a historic backlog. An observed failure is held for 10 minutes rather than cleared by the next accepted retain, since accepting a retain into a failing queue is precisely the evidence that was already misleading; the probe is TTL-cached at 30s so the dashboard's/api/statuspoll doesn't hammer the daemon, and it is fail-soft — an engine that can't answer reportsreason: "unknown", never a false green. Fields areNonefor a provider with no retain pipeline (the volatile PgVector fallback). Not included: signalling atPOST /api/memory/addaccept time that the extraction target is down (the issue's option 3), and the underlyingutility-slot 503 (#1417).- The model drawer's Context size field validates instead of corrupting or deleting the stored value (#1378). The field was a plain text input run through a bare
parseInt, with no error slot and no save gate — andparseIntis lenient in both directions. Typing32kPUTdefaults.context_size: 32, a 1000× context collapse;8.9landed as8and16384abcas16384. Typingabcwas worse:parseIntreturnedNaN, the codedeleted the key, and sincePUT /api/models/{id}mergesdefaultswholesale (registry/store.pymerge_update) an absent key is a deletion, not "unchanged" — the storedcontext_sizewas destroyed,--ctx-sizevanished from the launch line, and every slot bound to the model silently fell back to the llama-server default. All four cases finished with a green "Updated" toast. Save now demands a clean integer (/^\d+$/on the trimmed value) at or above the≥ 128floor the slot drawer already enforced, surfacing the reason inline (model-ctx-error) and disabling Save so no PUT fires — derived from the field like the existingflagsErrorgate, so a correction releases it on the next keystroke. Empty stays an explicit clear of the override; only malformed text is an error.model-ctx-validation-v3pins the wire for32k,abc,8.9, sub-floor64, a valid16384, and the empty-clear path. - The slot drawer's "Clear override" never removed a persisted
chat_template(#1372). Clearing an override issued no request at all: the template stayed on disk and kept feedingllama-server, while the drawer rendered the model default as though the removal had taken. It was not even counted as an unsaved change, so the discard guard stayed silent too — and there was no other way to drop a per-slot override from the dashboard. Both the save body and the dirty aggregate gated onoverrideOpen, which the Clear button itself sets tofalsebefore either predicate runs, so both evaluated false. Both predicates now share one normalized baseline-vs-desired comparison (landed with the #1401 drawer rework), so they cannot drift apart again — duplication is how they diverged. The removal rides asnull, never"":reconcile_slot_updatesimplements None-means-delete, sonulldrops the key from the slot TOML while""would persist an empty-string override (a different, still-broken state). Verified end-to-end at the route level — no test covered None-deletion of any slot key before, sotests/api/test_slot_config_validation.pygains two:nullremoves the key and leaves siblings ([model].default,port) intact, and""is explicitly not a removal. - Board/brain chat no longer swallows a pre-stream or network failure in silence (#1452).
useBoardChat'ssend()opened the SSE POST withif (!res.ok || !res.body) { setStreaming(false); return }, and the fetch.catchdid the same for any non-abort error — so a503 slot.loadingwhile the brain slot warmed, a502on a crash-looped backend, a401, or a dead gateway all vanished with no bubble, no toast, and the composed message gone as far as the operator could tell. Only an in-stream SSEerrorframe ever rendered anything. Both paths now lift the backend's{error:{code,message,details}}envelope — via a new sharedreadErrorEnvelopehelper insrc/api/client.ts(the same liftapi()already did inline) — and append an assistant bubble mirroring the existing SSEerror-frame path, including aretry_after_shint ("… — retry in 15s") when the envelope carries one. The operator's turn was never actually lost (the user bubble was already appended before the fetch fired); the fix makes the failure visible and adds a Retry button on the error bubble that resends the original text verbatim, so recovering doesn't mean retyping. An aborted turn (operator hits Stop) is left untouched — that is an intentional cancellation, not a failure, and must not produce an error bubble. - Dashboard layout persistence works: the backend speaks v3, and a rejected save stops pretending it succeeded (#1460). The UI moved to the fixed-band v3 schema in #1061 and has been PUTting
{v:3, cells, quickActions}ever since;routes/dashboard_layout.pystill requiredv == 2plusorder/enabled/spans/pinned, so every save 422'd withlayout.invalid.useSaveDashLayoutswallowed it ("Backend not yet shipping this endpoint — silently swallow") whileonMutateoptimistically updated the cache, so a widget swap or a quick-actions toggle appeared to take and then reverted on reload, with nothing on screen to explain it.GET /api/user/dashboard-layoutreturned{}on a live box — nothing had ever persisted. v3 is now the canonical backend schema, with v2 tolerated rather than rejected. PUT validates a v3 body and reconciles it server-side against theCELL_DEFSwhitelists (src/hal0/dashboard/layout_v3.py, mirroringuseDashLayout.reconcile): every cell must exist and hold a widget from its ownacceptslist that is actually built, otherwise it falls back to that cell'sdefaultWidget— shape-only validation would let a stale client persist a cell the dashboard can't render.layout_store.reconcilenow dispatches on the STORED payload's version instead of assuming v2, which matters most on the read path: the v2 pin/span rules would have graftedorder/spans/pinnedonto a v3 file and handed back something no client can parse. A pre-#1061 v2 file on disk is preserved, not erased — GET still returns it reconciled under the v2 rules, and the FE'sreconcile()already fail-softs an unrecognised payload toDEFAULT_LAYOUT, so an operator holding one sees defaults rather than an error. On the client, write-path fail-soft is narrowed to a 404 (a backend that genuinely lacks the route); any other rejection now toasts through the dashboard's existingwindow.__hal0Toastchannel and re-reads server truth, so a lost customization can't keep looking like a saved one.ui/CONTRACTS.md§4 rewritten from the dead v2 grid contract to the v3 cell + widget registries. GET /api/agentsreports systemd unit liveness, so an inactive Hermes stops rendering as running (#1459).AgentManager.list()is a filesystem read —status: "installed"means a bundle exists on disk and says nothing abouthal0-agent@<name>.service.list_agents()returned those records verbatim with no liveness field at all, and the dashboard mapped install-state straight onto liveness (useAgents.ts:if (first.status === 'installed') agentStatus = 'running';agents-overview.jsx_derive(), whose "down" branch needed abrokenstatus literallist_agentsnever derives). On a box wheresystemctl is-active hal0-agent@hermes.servicesaid inactive andis-enabledsaid disabled, the Agents card showed Hermes ready — while the box's own/api/doctorfeed simultaneously reportedHAL0-HERMES-DOWN. Records now carryunit_active, probed through the systemctl seam that already exists for restart (api/agents/restart.py, reusing_systemctl_path/_unit_namerather than growing a second wrapper):trueactive,falseinstalled-but-down,nullunknown. The probe is read-onlysystemctl is-active— no polkit, no unit work — run concurrently off the event loop with a 2s ceiling, because this is a route the dashboard polls; a host without systemd, a spawn failure, a timeout, or an agent with no template unit all degrade tonull.nullis never treated as healthy: the card renders it as "unknown" (grey),falseas "down" (red) with the existing Restart action as the way out, and onlytrueunlocks ready/serving.tests/api/test_agents_unit_liveness.pycovers active / inactive / probe-unavailable.- Settings ▸ Diagnostics ▸ Doctor reads the live
GET /api/doctorfeed instead of reporting "all clear" (#1458). The route has been live since D6 —routes/doctor.pycomposes the same typedDiagnosisrowshal0 doctor verify --jsonprints, andui/src/api/endpoints.tsalready documenteddoctor: '/api/doctor'as LIVE — but nothing inui/srcever called it.useDiagnosesimported onlyuseSystemInfo, synthesised a single info-severityHAL0-SYS-INFOcard from the hardware probe, and hardcodeddoctorFeedPending: truebehind aDOCTOR_FEED_REASONstring still claiming "there is no HTTP route yet". Because the only card wasinfo,overallVerdict()could never return anything butok, whichDiagnosisPanelrenders as the chip label "all clear" — on a box whose live verdict waswarnwithHAL0-RUNNERS-NONE-HEALTHYandHAL0-HERMES-DOWNrows waiting to be shown. The panel was already a generic Diagnosis renderer, so the server rows drop straight in: the hook now pollsENDPOINTS.doctor, maps the response faithfully againstDoctorResponse/DiagnosisOut(verdict+id/severity/confidence/summary/detail/fixable/evidence[]/next_steps[]), and reports the SERVER's verdict. The synthesis survives only as a fallback for a backend that predates the route — a 404, or a payload with nodiagnosesarray — and the panel's stub now says exactly that; a 5xx or network failure surfaces as an error rather than being downgraded to "ok". Stale comments inDoctorPage.jsx,DiagnosisPanel.jsx,useDiagnoses.tsandendpoints.tscorrected. - The footer's "degraded" tooltip names the check that actually failed, not all five (#1461).
failingChecksinui/src/api/hooks/useRuntime.tsfilteredc.status !== 'ok'and theHealthCheckinterface declared astatusfield, but/api/health/systemhas never emitted one:routes/health.pyreports a booleanokper check (disk_state,disk_config,slot_manager,event_bus,mcp_mount).undefined !== 'ok'is true for every check, so a box with one broken subsystem rendereddegraded — disk_state; disk_config; slot_manager; event_bus; mcp_mountin both the runtime chip and thehal0service pip — five alarms for one fault, and no way to tell which. The real reason was dropped too: the backend setsdetailon some failure paths (mcp_mount, andslot_managerwhensm.list()raised or was never wired), but the live slot_manager failure carries its cause in anerroredlist of slot names ({'ok': false, 'errored': ['flm']}), which the tooltip never read. The filter now testsc.ok === falseand each failing check renders its own reason fromdetailanderrored, so the tooltip readsslot_manager: errored: flm.HealthCheckis retyped to the shape the route actually returns, andtests/api/test_health_degraded.pypins that per-check contract — booleanok, no per-checkstatus, stringdetailwhere present — so the two halves can't drift apart again. - Slot drawer Save is no longer silently dead on an NPU slot with a malformed persisted
extra_args(#1389). The freeform extra_args field and its error surface live in the Model group, which is unmounted fordevice === "npu"— but the Save validator still computedextraArgsErrfrom the persistedllamacpp_args. A slot whose stored override carried an unbalanced quote blocked every Save on the slot with zero feedback: no request fired, no error rendered, the drawer just did nothing. The validator now only vetoes Save when the field is actually mounted; an operator who flips the device offnpustill sees (and must fix) the error before the field's value can ride a write. - Every pre-existing custom profile was un-editable — the §5 hardware screen fired on the profile's own stored flags (#1411, follows #1404).
_screen_profile_flagsrejects-ngl/-dev/--device/--threads/-ton bothPOSTandPUT /api/profiles/{name}, and shipped with no data migration — so a profile authored before spec-hw-slot-ownership §5 failed its own round-trip: load it in the drawer, press Save without changing anything,400 slot.hardware_flag_deniedon the flag stringGEThad just returned. On lxc105 that was 10 of 10 pre-existing custom profiles, five of them bound to live slots, so retuningintent/quant/mtpon a running slot's profile had no path through the API at all and the Profiles page saved nothing. #1404's load-path sanitizer had deliberately left the slot-hardware flags (they are functional at launch), which is exactly why they were still there to trip the guard. The screen now judges what an update introduces, not what it inherits: a hardware flag already present in the stored text passes (logged asprofile.hardware_flags_grandfathered), adding a new one still hard-rejects, and dropping the inherited one is a normal save after which re-adding it is a new reach.POSTand the import path keep the strict reject — they have no baseline to inherit from. Extending the sanitizer to strip these instead was rejected: it would silently rewrite an operator's working device selection out from under a live slot. Implementation collapses the route's_screen_profile_flagsinto the catalog'sscreen_profile_flags(they were token-for-token duplicates, and the grandfather rule would otherwise have had to be learned twice), moves the update screen inside the catalog's lock soexisting.flagsis the baseline, and mirrors the rule client-side viafindNewSlotHardwareFlags— the profile drawer's inline guard would otherwise have kept blocking a save the API now accepts. POST /api/profiles/importverifies the envelope checksum on commit, not just ondry_run(#1416).verify_checksumwas referenced only inside theif dry_run:branch, surfaced aschecksum_ok; the commit branch calledimport_profile(...)two lines away and never checked it. A.hal0profile.jsonwhosechecksumhad been altered — or whoseprofilebody had drifted from the checksum covering it — imported with HTTP 200, no warning, and persisted toprofiles.toml. The stamp exists precisely so a hand-edited or transport-corrupted envelope is detectable, and a profile is a launch-flag template that gets stamped into a slot's argv (and, viaPOST /api/models/{id}/duplicate?profile=…, into a model'sdefaults.extra_args), so importing one unverified was the wrong default. The commit path now parses first (so a structurally wrong envelope keeps its more actionableprofiles.bad_envelope) then verifies, raising400 profiles.checksum_mismatch.{"force": true}is the documented escape hatch for a deliberately hand-edited envelope — it waives the integrity check only and logsprofile.import_checksum_forced; the §5/§21.7 flag screen still applies, so an envelope can never back-door a hardware or managed flag past the guardsPOST/PUTenforce (that half was already covered byProfileCatalog.create, and now has regression fences).- NPU modality toggles rewrote the configured chat tag with a stale
model_id(#1388). Every Chat/ASR/Embed toggle routes throughapplyNpu, which attaches[model].defaultalongside the[npu]table — and it sourced that tag fromslot.model_id.useSlots.tsdocumentsmodel_idas stale for exactly this slot class ("trio slots never load as their own process, somodel_idnever reconciles" off the pre-trio GGUF) and already exposes the configured value asmodelDefault, lifted from[model].defaultbyconfig_enrichment— the drawer never read it. So flipping ASR or Embed, controls with no business touching the chat model, rewrote the slot's configured FLM tag to an unrelated GGUF id and cold-restarted it: silent config corruption on NPU boxes, with the operator unable to see the value being sent because the chat<select>has no out-of-vocabulary option for it. The seed (and re-seed) now prefermodelDefault, keeping the live id only as a fallback for a slot with no configured default on disk. - The flags fold silently dropped every slot's
[server].extra_args(#1396).collect_inputsfeeds the plannerSlotConfig.model_dump(by_alias=True), and SlotConfig's_tuck_server_into_extramodel_serializer re-parks the server sub-table underextra["server"]so the loader round-trips a proper[server]TOML table._slot_flag_tokensread only a top-levelserverkey, so against real input the freeform tune — the single value the migrator exists to preserve — never entered the fold; onlyparalleland the typed ngl/ctx survived. The same gap defeated the divergent-share guard: two slots differing only inextra_argsfolded to an identical tune, so the planner saw no conflict and would have silently picked a winner instead of refusing. The token reader now accepts both shapes. Caught by the new CLI tests before the command shipped, so the unreachable migrator was never made reachable-and-wrong. - Parallel-worktree e2e runs silently tested the wrong branch (#1399).
ui/playwright.config.tsdefaulted every run to port 5173 withreuseExistingServer: !CI, so a second git worktree running Playwright locally attached to the first worktree's Vite server and exercised its code. The results were not flaky but confidently wrong in both directions — a real verification reported7 failedthat became34 passedon a unique port with identical commits, and a second run of the same tree disagreed with itself (50 passedvs42 passed / 6 skipped). CI was never affected (CI=1disables reuse, and CI has one checkout); every local parallel run was. The default port is now derived from the worktree's own path — stable per worktree, soreuseExistingServerkeeps its iteration speedup, but distinct across worktrees so cross-contamination is structurally impossible.HAL0_E2E_PORTstill wins explicitly (and now ignores an unparseable value instead of passingNaNtovite --port); CI keeps the fixed 5173. memory_degradednow tracks the live engine, not just the boot probe (#1301, runtime half). The boot/healthprobe added in 1.0.0-rc.1 makes the hindsight→pgvector ladder fire when the daemon is down at boot, but it answers one question once and the answer goes stale immediately: a daemon that dies afterwards leaves theHindsightProviderin place, and/api/status.memory_degradedplushal0 memory statuswent back to reporting healthy while every recall came back empty and every retain raised.HindsightProvider.degradedis now a live property fed by a single_callwrapper every engine round-trip funnels through — it flips on an observed transport failure and, unlike a boot probe, clears itself when the daemon comes back. A 4xx does not degrade (the daemon answered; the delete sweep's routine per-bank 404s must not flap it), a 5xx does — the same ruleprobe_healthuses, so boot and runtime cannot disagree. Thehal0 memory statusline no longer claims "in-memory fallback" for what may be a failing durable engine.- Config drift false-warns forever when the TOML id and the registry key are spelled differently (fixes #1226). The drift comparator substitutes a model's on-disk path for a bare registry id before comparing, but matched the two ids with
==. The slot TOML keeps the catalog spelling (Qwopus3.5-4B-Coder-MTP-Q6_K) while the registry key — and the running container's--alias— is the slug (qwopus3-5-4b-coder-mtp-q6-k), so the substitution never fired and every status read printedconfig drift: --model: running=/mnt/ai-models/….gguf rendered=Qwopus3.5-4B-Coder-MTP-Q6_K. The comparison now normalises both ids. Separately,compute_config_driftlooked up the raw TOML id whileload()launches the container from_resolve_servable_model(...)(a catalog id that landed locally under a different id), so for exactly the slots this matters for the registry lookup missed, the renderer emitted the bare id, and the warning was permanent; the drift check now resolves the model the same way the launch path does. A genuinely different running--modelpath is still flagged. - Bound the slot teardown on the seamed (hal0-service-user) route too (#1224, worker side).
SlotManager.terminate's timeout releases the caller, but it cannot touch the executor thread sitting onsystemctl stop; on a real install that thread stayed blocked andunload_syncnever reached the Quadlet-source removal +daemon-reloadthat lets the next load converge.SystemCtlSeam.systemctlnow forwards atimeouton both the direct and the seamed route (defaultNone= unbounded, unchanged), andContainerProvider.unload_syncbounds the stop at 20s — under the caller's budget so the worker unwinds first — logscontainer.unit_stop_timeout, and continues the teardown rather than aborting it. - The model drawer's "vision requires an mmproj sidecar" error now blocks the save instead of decorating it (#1380). Toggling the
visioncapability on a model with no projector rendered a red inline message and then let the PUT through, so the registry row advertisedvisionwith nothing for the launch path to hand--mmproj. The invariant now lives in ammprojErrormemo folded into a singlesaveBlockedgate alongside the existingflagsError— the one gate bothonSave's early return and the Save button'sdisabledconsult — and the message carries adata-testidso it is assertable. Nothing else in the drawer's validation surface moved; the flags editor'sinvalidstyling still keys onflagsErroralone. - An emptied Display name now clears the stored name instead of being silently dropped (#1381).
if (trimmedName && trimmedName !== model.name)collapsed "unchanged" and "deliberately emptied" into the same skip branch, so thenamekey never reached the PUT: the old name survived whiledirtyarmed the discard guard and the drawer closed with a success toast naming the value you had just tried to remove. The guard now diffs on the value alone and sendsname: "", matching themmproj/hf_repo/hf_filenamefields beside it —Model.nameisstrwithdefault="", andnormalizeApiModelalready falls back tomodel.id, which is exactly the affordance the field's "empty keeps the model id" help text advertises. cli.mdxand the update-and-rollback guide taught ahal0 update --source {release|git}flag that no longer exists (#1462). The git-based update path was removed in 4eb9376f, andhal0 updatenow calls_refuse_if_editable()outright on an editable/dev install rather than offering any git-clone fallback —hal0 is installed in editable mode from {path}. Install from release wheel with pip install hal0.The "Update from git (local dev)" section and the phantom--sourcecell are gone from both docs;docs/reference/cli.mdxnow states the real refusal instead. Addedtest_cli_mdx_options_table_has_no_phantom_flagstotests/cli/test_cli_docs_parity.py— it parses the "## Top-level" table's "Key options" cells and checks every documented flag against the live Typer/Click command's realOption.opts, so a doc/CLI flag drift like this one fails CI instead of shipping quietly; verified TDD-style against the pre-fix doc.docs/hal0-install-migration-guide.htmlinstructed three commands that don't exist (#1463).hal0 memory migrate --from honcho --to hindsightwas never real post-Honcho-removal (71fc255d) — the onlymemory migratesubcommand isunify, which folds Hindsight banks and has nothing to do with Honcho; both the migration table row and the "Honcho → Hindsight"<pre>block now say plainly that Honcho-era boxes need no migration step.hal0 doctor --jsondoesn't exist on the baredoctorgroup (only--plain/--portsdo) — replaced with the realhal0 doctor verify --json(Quick Start sanity check) andhal0 doctor all --json(the "Always start here" troubleshooting note, matched to that note's broader "slots, memory banks, Hermes, services" claim, whichdoctor allcomposes fromdoctor verify's report card plus auth/model-store/migrations/ports/hal0.targetrows —permsaudit is a separate command and the note was adjusted accordingly).hal0 chat --slot <s>doesn't exist — the real option is--model; fixed in both the quick-reference table and the first-chat example. Every command in this pass was re-verified against live--helpoutput before writing it.
- FLAGS-own — flags belong to models. Launch flags, device, and chat-template now live on the model; a slot is just
Turnstone lands as a second heavyweight bundled agent alongside Hermes and becomes a first-class companion service; the unified hal0-rocmfpx runner becomes the default image for AMD GPUs (with an automatic slot migration on update); and a memory security fix stops one agent deleting another's private memories.
Highlights- Turnstone — a second heavyweight bundled agent joins Hermes: a native
turnstone-serveron loopback :9129, installed into its own managed PyPI venv, coexisting with Hermes via relaxed single-pick (#1299). - Turnstone is a first-class companion service — it shows in the Services pane and the Overview health card with start/stop/restart controls, next to Hermes/Hindsight/OpenWebUI.
- hal0-rocmfpx is now the universal default runner for AMD GPUs — one unified image (Vulkan/RADV + HIP) replaces the per-lane toolboxes; CUDA and CPU-only lanes keep their lean images (#1297).
- Memory: private-visibility is now enforced on delete — in unified-bank mode one agent could delete another agent's
visibility:privatememory by id;deletenow applies the same fail-closed ACL as read/search/list.
Migrationshal0 updateautomatically re-pins existing AMD-GPU slots from the oldamd-strix-halo-toolboxesimages to the unifiedhal0-rocmfpxrunner (no-op on CUDA/CPU lanes) (#1297).
Added- Turnstone bundled agent — provisioning pipeline (managed PyPI venv, JWT-secret generation, model automap) plus hal0 provider/memory wiring, coexisting with Hermes (#1299).
- Turnstone companion-service registration — a
ServiceDef+ systemd health probe, so turnstone appears in/api/services(Services pane, full lifecycle actions) and/api/services/health(Overview card + sidebar status).
Changed- hal0-rocmfpx as the default AMD-GPU image — basic seed profiles defer to a manifest-driven resolver that returns the unified runner for AMD lanes, the CUDA image for NVIDIA, and the lean toolbox for CPU-only (#1297).
- PyPI distribution is published under the name
hal0ai(the import package andhal0/hal0-agentconsole scripts are unchanged) (#1298).
Fixed- Memory delete ACL —
deleteenforces thevisibility:privateowner check in unified-bank mode, so an agent can no longer delete another agent's private memory by (guessable) document id; unresolved ids are withheld fail-closed (#1302).
- Turnstone — a second heavyweight bundled agent joins Hermes: a native
A robustness release. Fresh installs now adapt to the host — provisioning their own Python and Node, tolerating podman or docker, and surviving hardened umasks and non-root operation — and the self-hosted Honcho memory stack stands up cleanly alongside Hindsight. Plus the pi-coder and opencode bundled agents, the unified per-agent memory model, and a large batch of install/setup/agent fixes surfaced by end-to-end reinstall testing on a clean box.
Highlights- Installs self-heal across environments — auto-provision Python 3.12 and Node 20 LTS when missing, tolerate podman or docker, and survive hardened/root umasks (#1291, #1289).
hal0 doctorand bundled-agent installs work on packaged installs — FHS-aware resolution fixes the "packaged without scripts" and "could not locate preflight.sh" failures on non-editable installs (#1284, #1285).- Crash-safe agent switching — a failed
agent install --switchno longer bricks the running agent; it verifies the target first and rolls back on failure (#1285). - Self-hosted Honcho as a per-agent memory provider — unified memory, swappable and migratable with Hindsight per agent, with a clean opt-in standup (#1243, #1294, #1295).
- pi-coder and opencode join Hermes as bundled, single-pick agents (#1254, #1271).
Added- Node.js LTS auto-provisioning in the installer (the dashboard build and pi-coder/opencode all need npm) plus a curated
qwen3-embedding-0-6bmodel for the memory pipeline (#1291, #1294). - Unified per-agent memory — self-hosted Honcho v3 provider and a unified-bank model with server-side tagging, plus
hal0 memory bank/ops/mm/recallandmigrate unifyCLI (#1243, #1244, #1257). - pi-coder agent (provisioning, hal0 provider/memory plugins, live dashboard card) and the opencode bundled agent (#1254, #1271).
- hal0-brain as a first-class profile (#1258), upstream controls CLI and UI (#1279), and a bench queue dropdown with lane/tool-eval/tune options (#1255).
Fixed- Installer/preflight robustness — Python floor raised to 3.12 with auto-install, hardened-umask permissions, a container-runtime smoke test that no longer false-fails, and Node/disk/graphroot preflight gaps (#1291, #1292).
- Honcho standup — unbound-var abort, pgvector embedding-dim reconcile, compose-provider and migration ordering, AppArmor-in-LXC, and full
--purgeteardown (#1293, #1294, #1295, #1287). - docker/podman portability — runtime-appropriate slot units and a PATH-based runtime probe (#1289).
- Slots — errored-slot restart recovery and the WARMING watchdog, plus drift/image_status false positives (#1278, #1269).
- Memory — private-visibility enforcement on read, migrate-unify retag scoping, and Hermes plugin/bank identity consolidation (#1260, #1262, #1245).
- CLI — editable-install
hal0 updaterefusal, footgun confirmation gates, and command consolidation (#1274).
Changed- comfyui, doctor, and agent helpers resolve bundled scripts and assets FHS-aware so they work on packaged (non-editable) installs (#1286, #1284, #1285).
- Docs — ADR/Cognee→Hindsight sweep,
hal0/chat→hal0/agentalias, fresh-box first-run journey, and a loud LAN-only bind warning (#1280, #1276, #1275).
Hotfix for a fresh-install blocker — Hermes never auto-provisioned on a clean
curl | bashinstall — bundled with the dashboard, provider, and memory improvements merged since 0.9.7.Highlights- Hermes now auto-provisions on a fresh install — no more manual
--adopt/--repair; all bootstrap phases complete and the gateway comes up managed, idle until a bot token is added (#1239). - MiniMax and DeepSeek are now one-click options in the upstream provider catalog (#1236).
- Keyboard-driven
hal0 setup— the guided setup TUI gets real arrow-key navigation and a clearer model picker (#1237). - Memory subsystem toggle —
hal0 memory enable/disable/statusreplace the old install-time env flag (#1240).
Added- MiniMax + DeepSeek upstream catalog entries (OpenAI-compatible, bearer auth) — selectable in Slots ▸ Endpoints ▸ Add upstream with prefilled URL and auth (#1236).
- Arrow-key navigation in the guided-setup TUI: ↑↓/jk move, space toggles apps/agents, enter selects; scaffold/skip are clean navigable rows; numbered entry still works over a pipe or in CI (#1237).
hal0 memory enable/disable/statuscommands, plus the[memory].enabledconfig flag (#1240).- Retry failed graph extractions — a Memory-tab button that re-runs Hindsight's failed extraction operations, plus mental-model delete (#1235).
Fixed- Hermes fresh-install provisioning (#1239, closes #1238). Two chained bugs aborted provisioning on every clean install: the api-lifespan seed populated
HERMES_HOMEbefore the bootstrap could claim it ("unclaimed HERMES_HOME"), and the gateway installer started a unit that hal0 then flagged as its own "foreign" gateway. The lifespan now stamps.hal0-managedbefore seeding and pre-writes the gateway secrets drop-in before the unit starts. - Model-filters spacing in the Endpoints upstream pane — the filter inputs no longer sit flush against the panel border (#1236).
- Graph-extraction consolidation reliability in the Memory tab (#1235).
Changed- The memory subsystem is now gated by
[memory].enabledinhal0.toml(default on) instead of the installer-writtenHAL0_MEMORY_ENABLEDenv var; toggle it withhal0 memory enable/disable(#1240).
Migrations- If you had disabled memory with
HAL0_MEMORY_ENABLED=0, that env var is now ignored (memory defaults on) — runhal0 memory disableto keep it off (#1240).
- Hermes now auto-provisions on a fresh install — no more manual
The steward release. The dashboard's agent chat graduates from a side-panel into a real control surface for the whole platform, external LLM providers get a first-class management surface, and the FLM/NPU stack settles onto canonical names with an automatic migration.
Highlights- hal0-brain steward. The top-bar agent chat now drives every platform surface: it runs the full 74-tool
hal0-adminMCP catalog under a per-persona tool policy, pauses turns on gated tools for inline approve/deny, replays tool history across turns, and renders reasoning + tool cards inline (#1208, #1215, #1221, #1222, #1223). - Upstream model controls. A full management surface for external providers (OpenRouter, Anthropic, OpenAI, Google AI Studio, Ollama, custom) — reactive CRUD, per-upstream model filters, an
enabledkill-switch, and CLI + MCP + dashboard parity (#1228). - Graceful restarts keep your downloads. Model pulls no longer block a clean
hal0-apishutdown, so restarting mid-download no longer trips the 90s SIGKILL that was killing in-flight pulls (#1225). - FLM / NPU canonicalization. The NPU trio's shadow slots settle onto
flm-stt/flm-embed,/v1/modelsalias routing is finalized, and a naming migration +hal0 doctoraudits move existing installs onto the new scheme (#1210, #1229, #1231, #1214). - Agent is the new anchor. Seeded
agentandbrainslots replacechatas the default LLM anchor, and are seeded on every startup — not just fresh installs (#1204, #1217, #1218, #1230). hal0 updateverification actually works again. Release signing now dual-emits a Sigstore bundle (with an embedded Rekor timestamp) so cosign verification survives the short-lived Fulcio cert's expiry —curl … | bashinstalls and in-app updates were failing verification 100% of the time on v0.9.2/0.9.3/0.9.5. Fully back-compatible with already-deployed clients (#1159).
Added- hal0-brain steward / agent chat as a control surface. The dashboard's top-bar agent chat becomes a first-class operator for the whole platform:
- Full
hal0-adminplatform surface (#1208). The admin MCP catalog grew from 28 to 74 tools, replacing the slide-out's old hardcoded ~25-tool list, so the Brain can drive every surface end-to-end:- Models —
model_inspect(read an HF repo before pulling), register / add-from-path, metadata edit (PUT), in-place HF re-pull (model_update), pull status/cancel, scan (+preview), catalogue, update-check, and store show/set/migrate. Pulls always land in the operator's configured[models].store(re-read per call; the tool descriptions state the contract). - Slots — load / unload / edit-config (PUT) / metrics / capacity / logs, on top of create / delete / restart / swap.
- Stacks — create / update / export / snapshot, joining apply / import / delete.
- Profiles — create / update, joining import / export / delete (author a profile straight from a model card).
- Settings & platform —
settings_get/schema/apply_plan/reload,upstream_list, and benchmark runs/status/queue reads plus gated enqueue/control. - Two new guards make catalog drift impossible to reintroduce: import-time catalog validation (classification ↔ REST-map ↔ annotations ↔ path-args must cohere) and a
build_serverregistration-completeness check.
- Models —
- Per-persona tool policy (#1215). The persona TOML's
tools_allowed+[persona.approval]tables — previously decorative on the sidebar path — become an enforced server-side overlay (admin.ToolPolicy, fnmatch globs over tool names):tools_allowedhides tools from the surface entirely;require_approvaltightens an autonomous tool behind the approval queue;auto_approve/default_policy=auto-approvegrants standing approval to gated tools;default_policy=neverrefuses gated calls outright. Precedence is hide > tighten > loosen > server verdict. APOLICY_NO_LOOSENfloor meansmodel/slot/stack/profile_delete, bulkmemory_delete,config_write, andprovider_credential_writecan never be loosened by a persona edit; denials are typed (mcp.tool_not_allowed/mcp.gated_tool_refused) with audit rows. - Tool-use hardening (#1222) — fixes for four failure modes seen live:
- Runaway generation — every round now carries
max_tokens(default 4096, payload-overridable); an uncapped completion against a slow local slot used to burn ~25k tokens and the 300s transport window, killing the turn before the first tool call. - Invisible approvals — gated calls no longer park silently on the queue; the loop emits an
approval_requiredSSE frame and pauses (with keepalive pings) until the operator approves/denies, then streams a secondtool_resultso the same turn continues (timeout falls back to the pending result). - Guessed argument names — high-traffic tools ship real parameter schemas + explicit descriptions (no more
model_inspectwithouthf_repo→ 400, ormodel_pullwithmodel_id='org/repo'→ 405); path args containing/are rejected with an actionable hint. - Round budget —
_MAX_ROUNDSraised 8 → 90 (with per-round caps it's now a runaway backstop, not a working limit multi-step sessions kept hitting). - UI —
useBoardChat.send()now replays full tool history (calls + results) in the outgoing conversation; previously it rebuilt from user/assistant text only, so after one tool chain the model saw no tool calls in its own history and learned to skip tools and hallucinate results. Adds inline approval cards, a Stop button, an auto-approve toggle, and new-session. - A global port-claim registry — the single authority for which slot owns which port — lands alongside in the same PR.
- Runaway generation — every round now carries
[brain_chat]server-side guardrails (#1221), enforced independently of the persona TOML (a persona edit can loosen the persona, never these):enabledis a hard kill switch (the endpoint refuses every turn — no LLM call, no board request);read_onlylets reads through but refuses every mutating/admin-write tool at the single_dispatch_toolchokepoint (unknown tools fail closed);max_rounds/completion_timeout_smove the loop budget + per-round transport timeout out of module constants into config, with the schema as the single source of truth.- Agents / Brain settings + slot override (#1223). A dashboard Settings → "Agents / Brain" section surfaces
[brain_chat](enabled + read-only toggles,max_rounds/completion_timeout_sinputs) plus a slot-override picker built from the live slots list — point the steward at any slot (e.g.hal0/nputo run it on the NPU chat slot). Model precedence is explicit: per-request model >[brain_chat]override > persona/default; the rows carry a live badge (brain_chat.*is apply-planimmediate).
- Full
- Upstream model controls — full management surface for external LLM providers (OpenRouter, Anthropic, OpenAI, Google AI Studio, Ollama, custom):
- Reactive CRUD:
POST/PATCH/DELETE /api/upstreams(create prefills from the provider catalog viacatalog_id);upstreams.tomlstays canonical — every write rewrites it atomically before touching the running registry. hal0 upstreamCLI group (list/show/create/update/delete/test/set-credentials);create --catalog openrouter --api-keywires a provider end-to-end in one command.- MCP admin tools
upstream_create/upstream_update/upstream_delete(gated) +upstream_test. - Dashboard Upstream providers panel (Slots → Endpoints / Connections): add-from-catalog form, write-only key entry, test-connection with latency + model count, enable/advertise toggles, filter editor with live preview, delete-with-confirm.
- Per-upstream
model_filters(modelsallowlist +include/excludeglobs, exclude wins) curating/v1/modelsand/api/modelsadvertising — dispatch stays unfiltered so hidden models remain addressable by name (per the 2026-07-06 upstream-model-filters spec). enabledkill-switch on every upstream:falseremoves it from dispatch routing and the model catalog while retaining config + credentials.auth_key_presentin upstream serializations — whether the declared env-var actually holds a key (drives the dashboard auth badge), distinct fromauth_configured.
- Reactive CRUD:
- Install / lifecycle:
- Safe capture of an existing Hermes install —
--adopt, a fatal claim abort, foreign-gateway preflight, and ownership reconcile so hal0 can take over an already-running gateway without clobbering it (#1220). - Static slot TOMLs plus the
agent+brainslots are now seeded on everyhal0-apistartup, not only on fresh install (#1217, #1218, #1230).
- Safe capture of an existing Hermes install —
hal0 doctorgrows FLM / migration / profile audits and a--forcedelete for seeded slots (#1214).- Dashboard / UI — slots-page polish: unified headings + status indicators, an NPU activity tint, a full-height slot-logs drawer surfaced from the Logs page, an image-gen header pass, and a Documentation button in the topbar (#1205, #1207, #1209, #1213, #1216).
scripts/push-dev.sh— a push-based inner-loop deploy onto an editable box (#1193).
Changed- The default LLM anchor is the seeded
agentslot; thecoderseed is retired andhal0/chatresolves tohal0/agent, on every install path (#1204, #1217). - The NPU trio's shadow slots are canonicalized to
flm-stt/flm-embed, with a startup reconcile that migrates legacy names (#1210). - FLM host pulls land in the operator's resolved model store (#1211).
Fixed- Model pulls no longer block a graceful
hal0-apirestart — the shutdown path drains pulls instead of being SIGKILLed at the 90s deadline, which was killing in-flight downloads (#1225). - FLM model pulls that died instantly in production — uvloop rejects the user/group spawn kwargs the pull worker passed (#1192).
- FLM host pulls now land in the resolved store with robust 0-byte progress reporting (#1211).
- NPU-trio alias routing is finalized in the
/v1/modelsroute (#1231). - Static slot seeding no longer pollutes every test's zero-slot baseline (#1219).
- Dashboard: real update-banner actions, NPU-grid colours keyed to the primary FLM slot, and a single unified "Needs attention" + bell notifications source (#1194, #1195).
upstreams.tomlschema drift:auth_style = "anthropic"/"google_query"(long implemented by the dispatcher) now validate;auth_headeris a real schema field soauth_style = "header"works; warmup vocabulary canonicalized tonone|ondemand|alwayswithlazy/eageraccepted as normalizing aliases — a TOML authored in the runtime vocabulary no longer fails validation and silently empties the upstream registry./api/modelsno longer stamps slot-backed advertisements asorigin="upstream": the compositehal0aggregate and container slots serve local models, but a raw GGUF id whose casing differed from the registry id (e.g.Qwopus3.5-4B-Coder-MTP-Q6_K) surfaced in the Models page Upstream tab as "via hal0". Only genuine remotes contribute upstream rows, honoringenabled/advertise_models/model_filters.hal0 upstreamfirst cut sent field names the API rejects (openai_base_url,kind,allow/deny,api_key) — every write 422'd; request bodies now match the route contracts exactly and are pinned by tests.- Release signing survives Fulcio cert expiry (#1159). Keyless cosign signing issues a ~10-minute Fulcio certificate; the old detached
.sig+.crtcarried no Rekor Signed Entry Timestamp, socurl … | bashinstalls and in-apphal0 update— which run hours or days after signing — had no trusted timestamp to anchor the signature and failedverify-blob100% of the time on v0.9.2/0.9.3/0.9.5 (the in-CI self-verify passed only because it ran seconds after signing). Releases now dual-emit a Sigstore bundle (.tar.gz.bundle, embedding the cert + signature + Rekor SET) alongside the legacy pair; the manifest carriesbundle_urlnext tosig_url/cert_url, andinstaller/bootstrap.sh+ the updater prefercosign verify-blob --bundle, falling back to the legacy pair on manifests without it — so already-deployed (≤ v0.9.6.1) clients keep verifying through the transition. The legacy fields drop once fleet adoption is confirmed.
Migrations- FLM / NPU naming migration (#1229). Existing NPU-trio installs move from the legacy shadow-slot names to canonical
flm-stt/flm-embed. The migration runs automatically on startup, andhal0 doctoraudits + repairs any slot left on the old scheme. Customprofiles.tomloverrides that reference the old names should be updated. - Agent replaces chat as the LLM anchor (#1204, #1217). The
coderseed is retired andhal0/chatnow resolves tohal0/agent. Operators who pinnedhal0/chatin a custom config should confirm theagentslot is seeded (it is, on startup) or repoint tohal0/agent.
- hal0-brain steward. The top-bar agent chat now drives every platform surface: it runs the full 74-tool
- Added
- Model updates surface in the topbar notification bell: the update check now runs app-level, so "N model updates available" appears (with an Update all action) without opening the Models page, and the row self-clears once updates land. The Models page gains an always-visible "Check updates" button that bypasses the check's TTL cache when nothing is currently flagged.
hal0-brainagent profile: a third seeded persona (alongsidehermesandcoder) that stewards the platform from the dashboard's agent-chat slide-out — its own memory namespace (private:hal0-brain), a hal0-heavy system prompt (slot lifecycle, model setup, benchmarking), and the dedicatedbrainslot (hal0/brain) as the default model.- Board/agent chat streams a
{type:"thinking"}SSE frame: explicitreasoning_contentand inline<think>…</think>blocks are split out of the reply and rendered as a folded "thinking" section instead of raw tags.
Changed- The top-bar agent chat now embodies the
hal0-brainprofile: it runs onhal0/brain(falls back to theagentslot via the resolver chain), and an operator-editedhal0-brainpersona TOML overrides its system prompt/model without a code change. - Agent-chat suggestion chips are now platform-steward starters ("Help me create a new slot", "Download and set up a model", "Benchmark the model on a slot", "How's the hardware doing?").
- Agent-chat replies render markdown (fences, lists, headings, bold/italic/inline code, links); tool calls render as structured cards with args, live status, and a folded result — replacing the raw
→ tool({json})text rows.
Fixed- FLM NPU slots no longer wedge in
warmingforever: the warm→ready inference sentinel now probes the slot's assigned model instead ofmodels[0]from FLM's full catalogue. Probing an arbitrary other model forced FLM to reload the wrong weights onto its single NPU context mid-gate and deadlocked the load (#1171). hal0 setupno longer aborts with a rawHTTPStatusErrortraceback whenapply-selectionsreturns409 Conflict: the setup CLI now treats a 409 (install already applied / a concurrent apply in flight) as a recoverable no-op with a clean message, and still raises on genuine errors (#1158).- Unified the dashboard warming-state color to a single canonical
--warn: #f2792btoken indashboard.css, removing the divergent local redefinitions inengine-panes.css/overhaul.cssso every warming indicator renders the same orange-yellow (#1156, #1155). - Added an ESLint
no-undefguard (enforced in CI) over the dash.jsxprototype so an undefined identifier like the one behind the Create Slot crash can't ship again (#1170).
- Highlights
- FLM NPU trio: the edit-slot drawer's Chat/ASR/Embed toggles now drive the running
flm serveprocess, with a full-catalogue chat model picker that downloads on demand. - FLM can run embed- or STT-primary (chat disabled) — a modality-aware readiness gate promotes the slot instead of wedging on a chat probe.
- Real FastFlowLM v0.9.44 toolbox image (
ghcr.io/hal0ai/hal0-toolbox-flm:0.9.44), rebuilt from the actual v0.9.44 binary. - NPU occupancy cards now glow purple when a slot is running (they used to read flat green regardless of state).
- hal0-bench is now in-tree: the
hal0.benchengine,/api/benchmarks, and a Benchmarks dashboard page.
Added- NPU trio drawer wiring: ASR/Embed on-off toggles + a Chat model picker that lists the full FLM catalogue and pulls a not-yet-downloaded model on select (auto-applies on completion).
FLMProvider.verify_embed— a one-shot/v1/embeddingsreadiness sentinel used when a slot serves embeddings without chat.- hal0-bench in-tree port:
hal0.benchengine,/api/benchmarksroutes, and the Benchmarks dashboard tab (roster / runs / evals / run-queue).
Changed- FLM toolbox pinned to
0.9.44acrossmanifest.json, the flm seed profile, and the capabilities catalog (contains FastFlowLM binary v0.9.44). - The warm→ready gate for FLM slots picks its sentinel by served modality: chat →
/v1/chat/completions, chat-off+embed →/v1/embeddings, ASR-only →/v1/modelsliveness. /api/slots/flm/modelsreturns the full FLM catalogue (installed + downloadable) with accurateinstalledflags, container-exec first with a host-probe fallback.
Fixed- Chat toggle now actually gates the container:
container_specno longer passes the positional chat tag when[npu].chat=false(it was cosmetic on the container path). - Editing an NPU modality no longer clobbers
[model].default/context_size— the drawer sends the model as a nested[model]table so the backend merge preserves sibling keys. /api/npu/occupancyno longer 500s while the FLM slot is offline (the degraded single-tenant fallback'szip(..., strict=True)mismatchedslots_outvsflm_slots).- NPU cards read a clear purple "running" glow for up/resident slots and dim for offline; the coresident STT/embed sub-cards reflect the anchor's
[npu]toggles instead of the legacy shadow-slotenabledflag.
- FLM NPU trio: the edit-slot drawer's Chat/ASR/Embed toggles now drive the running
- Fixed
- Models → Downloads tab: React invariant 310 ("Rendered more hooks than during the previous render"). The Downloads pane in the Models view called
useStateM(false)inside ajobs.map((j) => { … })callback. React counts hook calls per render, so the moment the downloads list changed length (a pull started, finished, failed, or was cleared) the hook count differed across renders and the entire<ModelsView>crashed behind the v3 error boundary — masking the rest of the dashboard. Fix: extract the per-row logic into aDownloadRowcomponent that owns its owncancellingstate, so the hook lives at row-component top level and its count is stable across renders. Pinned byui/src/dash/__tests__/react-hooks-order.test.mjs, a new AST-based static check that walks every JSX/JS file undersrc/dash/and flags any useXxx (incl. theuseStateM/useEffectM/etc. aliases used in this dashboard) called inside an array-iteration callback (.map/.forEach/.filter/.reduce/.some/.every/.flatMap/.find/.findIndex).
- Models → Downloads tab: React invariant 310 ("Rendered more hooks than during the previous render"). The Downloads pane in the Models view called
- Highlights
- First green build on the 0.9.5 line — v0.9.5 shipped with a red CI (stale tests + lint); this hotfix greens the gate and closes v0.9.5's incomplete consolidations.
Added- HF inspect recognizes FastFlowLM (NPU) repos.
POST /api/models/inspectnow detects the FLM model shape (aconfig.json+ tokenizer +…nxNPU-quant weight directory, e.g.model.q4nx) and surfaces one whole-repo variant flaggedflmrouted to theflm pullpath — previously such repos inspected as "no variants" because the filter only admitted.gguf/.mmproj. Detection is shape-based (requires thenxweight blob) so a plain safetensors/GGUF repo is not misread as FLM.
Fixed- Profile test suite realigned to the 2×2 seed grid. v0.9.5 consolidated the retired
rocmfpx-rocm/vkfpx-*slugs into the{rocm,vulkan} × {dense,moe}grid but left the tests asserting the old names, somainwas red. Tests now assert the shipped grid. - Role-retirement cleanup finished. v0.9.5 retired
SlotConfig.rolebut left stale tests/docstrings referencing it (test_slot_role,test_chat_normalization,test_llm_slot_views,hal0_llm_slot_views). Removed/updated — slot identity is the name. - Curated catalogue guard for FLM/NPU entries. FLM-served curated models (empty
hf_repo) no longer surface in thepullablecatalogue bucket (nothing to HF-pull), and a newCuratedModelvalidator requires every entry to be deployable — HF coords (hf_repo+hf_file) or annputag withrecommended_slot="flm". - FLM provider
load_chatUnboundLocalError. The legacydefaultsbranch no longer leavesload_chatunset; chat defaults on (NpuConfig.chat=True). - CI lint gate. Cleared 12
ruff checkerrors + reformatted 7 files (import sorting, duplicate imports, placeholder-less f-strings, deadresultbinding,SIM103/SIM110/RUF034) that had blocked release CI since v0.9.5.
- Added
- NPU chat-first seed (FLM container shape).
NpuConfig.chatnow defaults toTrueso a bare[npu]section in a slot TOML is a chat-ready NPU slot out of the box. Operators opt out by settingchat = falsewhen they want an asr-only or embed-only NPU slot. A new seedinstaller/etc-hal0/slots/npu.tomlis included sohal0 setupcan register a clean (no FLM model pinned,enabled = false)hal0/nputile on fresh boxes. The EditSlotDrawer now hides the Model field fordevice = "npu"slots (the NPU capability matrix replaces it).
Changed- Slot routing key is now the slot
name, notrole(ADR-0023 §2.1). The legacyrolefield onSlotConfigis gone — slot identity IS the routing key forhal0/<slot>aliases. Resolution chains insrc/hal0/normalize/resolver.pynow use_slot_matches_name(case- insensitive exact match onname), with one silicon-class escape hatch: the special namenpuadditionally matches any slot withdevice == "npu", so a container that calls the trio's chat edge by a different slot name (flm,npuchat, …) still answershal0/npu. Operator-custom seeds (saber-fpx,deckard-fpx*) and live slot TOMLs that used the oldroletag now resolve by name. Live installs with the oldrolefield in any slot TOML continue to parse (the field is parked underextravia Pydantic) and resolve to the slot by itsname.
Highlights- New canonical ROCmFPX runner:
ghcr.io/hal0ai/hal0-rocmfpx:vulkan-minicpm5— built fromHal0ai/Hal0_ROCmFPX@5b395660plus a 4-line upstream cherry-pick that wires the minicpm5 pre-tokenizer (upstream PR #23384). Unblocks loading the BRAINTRAIN-1B GGUFs on Strix Halo (thetokenizer.ggml.pre = minicpm5field now resolves correctly). The new image serves both ROCm/HIP and Vulkan backends from a single artifact; this consolidates the previously-3-tag ROCmFPX image family into one. - Slot-level image override —
slot.image(top-level string in the slot TOML) now overrides the profile'simage. Resolution order isslot.image→profile.image→DEFAULT_ROCMFPX_IMAGE. Future image bumps will be a code-only release (one constant inschema.py); operators no longer have to chase profile clones to keep their stack current. The EditSlotDrawer now exposes an Image form-row (free-form text input with a livewill use:preview of the effective ref, plus a Reset button) so operators can pin/clear the override without editing TOMLs.
BreakingSEED_PROFILES reshuffled to a clean 2x2 grid. The three old ROCmFPX runner slugs (
rocmfpx-rocm,vkfpx-moe,vkfpx-dense) are replaced by four explicitly-named profiles that match the 2x2 (backend x {dense,moe}) matrix. The names are self-describing — the backend is the prefix, the weight format is the suffix:Old slug New slug Backend Format rocmfpx-rocmrocm-denseROCm0 ROCmFP4 (new) rocm-moeROCm0 ROCmFPX vkfpx-densevulkan-denseVulkan0 ROCmFP4 vkfpx-moevulkan-moeVulkan0 ROCmFPX All four reference
ghcr.io/hal0ai/hal0-rocmfpx:vulkan-minicpm5(the new canonical runner).rocm-moeis a NEW profile — pre-0.9.5 the ROCmFPX MoEQuality format was Vulkan-only; this adds the ROCm0/HIP variant for operators who want a single-backend fleet.profile.imageis now deprecated for new code paths. It still works as a fallback when no slot override is set, but the plan is to drop it fromSEED_PROFILESentirely in 0.9.6. Operator-custom profiles that overrideimage(e.g.saber-fpx,rocmfpx-rocm-custom,vkfpx-moe-custom,deckard-fpx*) should migrate to per-slotimageinstead — a single slot TOML is a single grep hit, a profile clone is a maintenance liability.
Migrations- Re-pin your custom profiles' images via the slot, not the profile. Edit
/etc/hal0/slots/<name>.tomland add a top-levelimage = "ghcr.io/hal0ai/ hal0-rocmfpx:vulkan-minicpm5"line. The slot-level value wins on nexthal0 slot <name> restart. Once your slots are migrated, you can remove theimageline from your operator-custom profile inprofiles.toml. - Rename operator-custom profile references in slot TOMLs. If your slot
points at one of the old slugs (
profile = "rocmfpx-rocm","vkfpx-moe","vkfpx-dense", or a-customclone of those), update it to the matching new name ("rocm-dense","vulkan-moe","vulkan-dense", or your custom variant of the new name). Example:nano.toml'sprofilefield should change from"vkfpx-moe-custom"→"vulkan-moe-custom"(and yourvulkan-moe-customprofile inprofiles.tomlshould mirror the newvulkan-moeseed's image + flags). - Verify the new image pulls cleanly on a non-production slot first:
hal0 slot <name> image ghcr.io/hal0ai/hal0-rocmfpx:vulkan-minicpm5 && hal0 slot <name> restart. Thencurl :PORT/v1/modelsto confirm the slot launched the new image (system_fingerprintcarries the upstream commit hash for traceability). - The
image_mismatchwarning clears naturally — it was a symptom of the oldprofiles.tomlcarrying stale image refs; once slots own their image, the mismatch can only happen on a manualimageedit.
- NPU chat-first seed (FLM container shape).
- Changed
- Idempotent
hal0 setup— removed the install-closed guard that blocked re-runninghal0 setupafter the first-run sentinel was written (#1161). The three provisioning endpoints (/apply,/apply-selections,/complete) are now idempotent and can be invoked at any time, fixing a Python traceback wheninstall.shre-launched the interactive setup after the--autoseed.
Docs- README updated — corrected version, slot list, setup flow description, and agent selection docs to match v0.9.4 reality.
- Idempotent
- Added
- Downloads pane in footer with pull job tracking (#1165).
- Settings page reorganisation with improved layout and navigation (#1163).
- Configurable
direct_read_timeoutvia[dispatcher]TOML section (#1160).
Fixed- NPU double-free: stop FLM health poll from double-freeing the NPU slot (#1077).
- Vision models: add missing
mmproj_fileto curated vision model entries (#1162). - Memory tools UI: make documents card full-width (#1166).
- CI repair: fix test failures post-settings-reorg (#1163).
- Lint: ruff formatting and
__all__ordering across several files.
The Guided Setup. hal0 setup grows a review-gated TUI, headless answer files, hardware preflight, and a pick-free first-run workflow. The installer no longer ships model recommendations — every box starts clean and the operator chooses models interactively.
Highlights- Guided Stage-2 setup TUI with a review gate and two-stage handoff: hardware detection, model selection, capability scaffolding — all reviewed before writing config.
- Headless answer files —
hal0 setup --emit-answerscaptures selections as JSON;hal0 setup --answers <file>replays them on another box. Combine with--yesfor fully automated provisioning. - Pick-free install. The installer no longer seeds model slots. Instead, the guided TUI (or headless answers) scaffolds capability slots and walks the operator through model selection interactively.
- Hardware preflight. GPU detection runs during install with an LXC smart-block and dev0 remedy; NPU functional state is persisted in a hardware.json written at install time.
- hal0 doctor --verify — a structured report card checking config, slots, models, and service health.
Added- Guided Stage-2 setup TUI with review gate + two-stage handoff (#1144).
- Headless answer files —
hal0 setup --answers/--emit-answersround-trip (#1119, #1120). hal0 setup --plan/--dry-runpreview (#1121).- GPU preflight during install — LXC smart-block + dev0 remedy (#1135).
- Authoritative hardware.json persisted at install incl. NPU functional result (#1118).
- Network coherence — one
HAL0_BIND_HOST+ seeded origins (#1130). - HF_TOKEN gathered + persisted to
secrets/EnvironmentFile, threaded through in-process apply_setup (#1136, #1122). hal0 doctor --verifyreport card — config, slots, models, service health (#1145).- Post-update drift surfacing +
hal0 update --restart-slots(#1142). - Slot-render reconcile seam shared by install + update (#1138).
- Safe slot activation — enable-on-pull-success + clamp context (#1137).
- Clean seeded slots — no model pins, derived device (#1140).
- NPU opt-in threaded through suggest+apply; NPU introduction gated on hardware present + healthy (#1134).
- Capability + NPU slot scaffolding — pick-free, guide models interactively (#1091).
- ComfyUI gen branch — scaffold-only default + per-variant download picker; repaired model fetch + shipped curated workflow JSONs (#1146, #1128).
- First-run model workflow —
hal0 models scan/add/store/run+hal0 doctor models. - Pick-free install — installer seeds zero model slots; the operator chooses via the guided TUI or headless answers (#1104, #1140).
- Co-locate
[models].flm_store+ free-space validation (#1132). - Honor
Selections.storage_dir— thread to[models].store(#1127). - Close
/api/install/*provisioning after first-run sentinel (#1126). - Apps skip/defer parity —
openwebuiinstall verb + gateway in deferred hermes (#1125). - Converge
/applyonto/apply-selections+ write first-run sentinel from endpoint (#1124). - Platform-gate hardening — bootstrap-prereq parity, disk-on-store, early hal0 user (#1139).
- Runtime
advertise_modelstoggle for upstream catalog entries (#1152).
Fixed- Dashboard URL no longer leaks
hal0.thinmint.devas default (#1092). - Advertise all enabled LLM slots in
/v1/modelsdiscovery, not just loaded ones (#1153). - Support flat slot TOML shape in profile in-use scanning (#1129).
- Don't persist hardware.json on
--plan/--emitpreview paths (#1131). - Delete stale
avahi/hal0.servicesystemd unit (#1123). - HF vision mmproj sidecars (
.mmproj) + remove backend-switch surface (#1089). - Seed
[models].store, create FLM cache dir, add GPU/NPU preflight. - Rescan models immediately when the store path changes.
- Honor
[models].flm_storeconfig + make NPU slot bind source reboot-durable. - Auto-resolve a Hindsight-compatible Python instead of a raw pip wall.
- Don't write through the seeded uname symlink in prereq-parity tests (#1143).
- HERMES.md.j2 rendering — fix crash on empty env snapshots (Jinja2
Undefined.__getattr__raises immediately, bypassing|defaultfilters).
Migrations- First-run sentinel closes
/api/install/*— a one-way gate at install time. The guided TUI or headless answer files are the canonical path for future configuration changes.
Hotfix: restore the full model listing in every slot's model picker.
Highlights- Fix collapsed slot model dropdowns —
/api/modelsrows again advertise the dispatcher-vocabtype(llm/embedding/reranking) the pickers join on.
Fixed- Slot model dropdowns (and the model→slot compatibility list) showed only the currently-assigned model.
/api/modelsstamped local- and upstream-registry rows'typewithclassify()'s coarse modality bucket (chat/embed/rerank) instead of the dispatcher vocabulary (llm/embedding/reranking) that the FLM path already emitted and that the UI joins on (model.type === slot.type), so every local model failed the picker filter and each dropdown collapsed to its default. Map the modality bucket → dispatcher type at both stamp sites; adds a local-row regression test (the FLM path was already covered, which is how this slipped through).
- Fix collapsed slot model dropdowns —
ROCmFPX llama.cpp runner support, plus a safer notes-aware self-update.
Highlights- ROCmFPX runner support: GGUF quant-family detection,
rocmfpx-rocm/vkfpx-moeseed profiles, and a build/quantize agent skill. hal0 updatenow shows cosign-verified release notes and asks before applying (staged prepare → commit).
Added- Updater
prepare/commitsplit.hal0 updatedownloads + cosign-verifies + extracts a release and shows its notes — with breaking/migration callouts — before activating anything;--yesskips the prompt for headless/cron. AddsPOST /api/updates/prepare+/commit(#1075). - Release notes in the update. The release build bundles
RELEASE_NOTES.md+release.jsoninto the cosign-verified tarball; a CHANGELOG section's### Highlights/### Breaking/### Migrationssubsections become thehal0 updatecallouts (#1078). - ROCmFPX quant detection — the registry classifies the
ROCmFPX/ROCmFP{3,4,6,8}quant family from a GGUF filename so FPX slots resolve their launch command (#1068). - ROCmFPX seed profiles
rocmfpx-rocm(ROCm0 dense) andvkfpx-moe(Vulkan0 MoE) for the custom ROCmFPX runner (#1069, renamed in #1076). vkfpx-denseseed profile — the Vulkan0 lane for DENSE ROCmFP4, for prefill-bound dense workloads (Vulkan wins prompt-processing); complements the decode-optimal ROCm0rocmfpx-rocm.hal0-quantizeagent skill — build the ROCmFPX toolchain and quantize a model to ROCmFP4/FPX (#1071).- ROCmFPX bench tooling: server-ab (MTP / concurrency) aggregation in the benchmark SUMMARY, FPX sweep cells, run provenance (#1072).
- Continuous batching — per-slot
parallelfield. A slot can now set llama-server's--parallel/-npsequence-slot count so concurrent requests share the once-loaded weights instead of serializing through a single sequence and thrashing one prompt cache (the win the shared-slot architecture already earns but never harvested — every seed profile pins--parallel 1).Noneinherits the profile; a value >1 also emits--kv-unifiedso--ctx-sizestays a SHARED pool (each request may use the full context) rather than being silently split to ctx/N per slot. Emitted as a slot override (beats the profile, loses to hand-authoredextra_args); surfaced in the slot drawer with a shared-pool hint. The dead haloaiworkersfield is deprecated (inert; a non-default value now logs at launch). MTP x batching runs but logsmtp.batched_speculation(unproven on gfx1151, bench-gated). Seed-profile defaults stay--parallel 1pending the on-box-npsweep (server_ab.py --mode batch). See the concurrency-batching plan handoff.
Changed- The plain
rocm/vulkanseed profiles are reduced to basic flags (-ngl 999 -fa on --jinja); per-model KV/batch tuning now lives in the model'sdefaults.extra_args(#1076). - Seed-profile
intentlabels normalised to terse structural tags (e.g.ROCmFPX · DENSE · MTP,VULKFPX · MOE · MTP,ROCm,Embeddings) — no served-model names, no filler. - Slot units are re-rendered through the new code during an update's
commitstep, so a subsequent restart uses current argv (#1075).
Removed- Legacy MTP toolbox seed profiles
rocm-moeandrocm-dnse(superseded by the ROCmFPX profiles);rocmfpx-moerenamed tovkfpx-moeto indicate its Vulkan lane (#1076).
Migrations- Slots pinned to a removed/renamed seed profile (
rocm-moe,rocm-dnse,rocmfpx-moe) auto-fall-back to the backend's basic profile (rocm/vulkan) on launch — existing slots keep working with no operator action (#1076).
- ROCmFPX runner support: GGUF quant-family detection,
The first public-beta cut. hal0 graduates from the b-tagged 0.8.x line: the dashboard gets its redesigned fixed-band layout and a live telemetry header, seed profiles carry flags from a measured Strix Halo bench matrix (plus a new per-model-family override layer), MCP servers are manageable from the CLI, and the Memory view becomes a per-bank workspace. Safe upgrade from v0.8.5b2 — no on-disk migrations in this cut. Profile flag changes (re-tune,
FAMILY_DEFAULTS) land on each slot's next restart; thanks to the v0.8.5b2 auto unit re-render, any restart path picks them up.Added- Dashboard redesign — fixed-band layout with swap-in-place widgets
(#1061). The free-form drag/resize grid is replaced by a fixed
vertical band stack: hero strip (steady-on + quick actions), 5-cell
health strip, a full-width Unified Memory hero (slot allocations
drawn inside the pool bar, striped system block, Proxmox-host block
when configured), Throughput / Utilization / Requests band, locked
dense slot rows, and an Activity / Services / Needs-Attention band
with inline actions. Customization is swap-in-place per cell (layout
v3 via the existing
PUT /api/user/dashboard-layout, fail-soft to defaults on old payloads). Ships a new Requests & Latency widget against a new/api/stats/requestsendpoint (gates to "source pending" until the dispatcher rollup ships). - Telemetry header on the Slots page (#1059, #1062, #1064). One combined live-metrics card replaces the old hero band: throughput hero + 20-bucket spark, GPU semicircle gauge (sclk/temp/watts), CPU + memory gauge, and the NPU 4×8 occupancy grid with per-slot owner hues — above a full-width memory rack ruler in the #1061 memory-hero style (in-bar allocations, live tok/s on serving segments, click-through to the slot). Honest-data rules throughout: missing metric → em-dash, measured zero renders as 0.0 with the serving count, GPU util captioned "pinned" when forced high. Container queries keep the 4→3→2→1 column wrap gap-free.
hal0 mcpCLI surface (#504) —hal0 mcp {list,status,install,uninstall,restart,catalog}backed by the existing/api/mcp/*routes. Rich tables with--jsonflag for machine output. Therestartsubcommand surfaces the 501 supervisor-stub gracefully until ADR-0015 lands.FAMILY_DEFAULTS— per-model-family launcher-flag overrides. A new resolution layer between a profile's generic flags and a slot's own[model].defaults, keyed on model family (matched from the id/filename). Applied automatically at slot resolution (launch + preview parity) and collapsed bynormalize_argvlast-wins, so a family override beats the profile but a per-slot[server].extra_argsstill beats the family. First tenant: gemma →-ctk f16 -ctv f16 --cache-reuse 0— any gemma model on any q8 profile is pinned back to f16 KV (gemma iSWA regresses on quantized KV: measured -28.5% pp on RADV / -10% tg on rocm, plus SWA+cache-reuse bugs #21468/#21749). This fixes the live gemma-on-rocm-dnseregression and makes adopting Vulkan q8 KV safe as a follow-up.- Configurable slot publish host —
[slots].publish_host(#1058). Slot containers published on 127.0.0.1 only; raw slot ports were reachable solely through hal0-api/Traefik. A first-class, UI-settable config key (default127.0.0.1, unchanged behavior) lets an operator widen to0.0.0.0or a specific interface IP. Fail-soft: an unresolvable value falls back to loopback, never opens the box. Baked into ExecStart, so live slots re-bind on their next restart; the Settings row carries a loud LAN-exposure warning.
Changed- Memory Overview is a per-bank workspace (#1057). Selecting a bank
drives one combined primary card — retained-memories spark, graph
extraction panel, embedded Tools (recall · reflect · documents ·
mental models · directives), async operations, danger zone. The
standalone
#memory/toolsroute is retired; documents paginate and show composed titles instead of raw UUIDs; mental models and directives gain create forms. - The MTP control is always visible on llm slots (#1054). Hiding the row for ineligible models made the tri-state undiscoverable. The slot drawer now always renders it, with a reason line for Auto·off ("model has no MTP heads" / "profile doesn't enable MTP") and a launch-will-fail warning when forcing On for a model without advertised heads. Stack editor rows follow the same contract.
- Activity sidebar rework (#1063). Taller pane, stacked timestamp/actor meta reclaiming horizontal space, and the free-text search replaced by a slot filter dropdown (exact target match, server-side pre-narrowed).
- Seed profiles: bench-driven flag re-tune (Strix Halo matrix, 2026-07-04).
rocm-moemicro-batch-ub 2048→-ub 1024(+30% prompt-processing on Qwen3.6-35B-A3B-MTP: 1165 vs 895 t/s pp2048, consistent across all-b; token-gen flat ~47).vulkan-ub 512→-ub 256(+5.4% pp; the reported 1024 sweet spot measured worse). Dropped--threads-batch 32and--poll 100 --poll-batch 1from the rocm chat profiles (measured within noise at full offload — simpler flags win ties). Added explicit-ngl 999to all GPU LLM profiles (GTT/unified free-mem autodetect is unreliable) and--jinjato all LLM profiles. Decode throughput is unchanged (all wins are prefill), soPROFILE_BENCHhero numbers stand. MTP draft depth measuredn-max 4optimal (+23% decode vs n-max 2 on dense MTP) — seeded default kept. - Vulkan seed adopts symmetric q8 KV (
-ctk q8_0 -ctv q8_0): +45% pp at 32k depth on qwen (168 vs 116 t/s) and halves KV memory. It is the mirror image on gemma (gemma-4-12B @32k: q8 costs -28.5% pp on RADV), so the vulkan profile is no longer intrinsically gemma-safe — it relies onFAMILY_DEFAULTS["gemma"]pinning gemma slots back to f16 KV. (The upstream "~10x pp cliff" did NOT reproduce on this fork.)
Fixed- OpenRouter OAuth callback route is gated behind
HAL0_OPENROUTER_OAUTH_ENABLED(#775). The callback endpoint no longer registers unless the flow is explicitly enabled, closing an unauthenticated surface on installs that never use OpenRouter OAuth. - Capability slot mini-cards regained Logs/Edit buttons (#1055). The utility-tier (embedding/reranking/tts/transcription) cards rendered compact controls with no way to edit or tail a capability slot created via the UI.
- Hermes memory identity defaults to
hermes(#1056). The upstream plugin base defaulted tohermes-agenton env-less code paths, spawning a stray duplicateprivate:hermes-agentbank alongside the correctprivate:hermes. Reads/writes now consistently targetprivate:hermes. - Telemetry throughput no longer flaps to "source pending" on an idle box (#1062). An empty 100-second history window is a measurement (0.0 with the live serving count), and the spark keeps the last 20 measured buckets on screen; "source pending" is reserved for a genuinely missing source.
- Removed a dead duplicate
SELF_MANAGED_PROVIDERSconstant fromkokoro.py(#982).
- Dashboard redesign — fixed-band layout with swap-in-place widgets
(#1061). The free-form drag/resize grid is replaced by a fixed
vertical band stack: hero strip (steady-on + quick actions), 5-cell
health strip, a full-width Unified Memory hero (slot allocations
drawn inside the pool bar, striped system block, Proxmox-host block
when configured), Throughput / Utilization / Requests band, locked
dense slot rows, and an Activity / Services / Needs-Attention band
with inline actions. Customization is swap-in-place per cell (layout
v3 via the existing
Hotfix over v0.8.5b1, closing the three findings from the first live update+verification pass on Strix Halo hardware (CT105): stale
mtp = trueoverrides crashing slots on re-render, slot units not re-rendering on update, and the gateway missingPOST /v1/rerank. Safe upgrade from v0.8.5b1 — the one migration (crash-only MTP override defuse) touches exactly the slot configs that could not have loaded anyway, with a loud per-slot log.Added- Slot units re-render automatically on update. A slot's systemd unit
bakes the launch argv at load time, so updating hal0 changed the code that
WOULD render but not the file that DID —
systemctl restart, crash restarts, and reboots kept running pre-update flags until an operator did a hal0-level slot restart (field finding). The updater (post venv-reinstall, via a fresh interpreter so the NEW code renders) andinstall.shnow rewrite every existing unit through current code plus onedaemon-reload— running services are never bounced; fresh argv applies on each slot's next start from any path. Per-slot failures log and skip. The dashboard drift indicator still covers the "process running old argv until next restart" window.
Fixed- Gateway now serves
POST /v1/rerank. The dispatcher's capability path map already resolved/rerankto the rerank slot, but the gateway only registered/v1/rerankings— so clients using llama-server's / Jina-style/v1/rerankgot 405 and had to hit the slot port directly (field finding)./v1/rerankis now an alias of/v1/rerankingsthrough the same dispatch path. - Crash-only
mtp = trueoverrides are defused automatically. A forced MTP override pointing at a model with no MTP heads crashes llama-server at load once the slot's unit re-renders under the v0.8.5b1 MTP separation (field-confirmed: pre-separationmtp = truedebris on a headless MoE model). Two mechanisms now clear exactly that combination: an updater migration over the slot TOMLs (updater.mtp_force_on_clearedlog; force-off, eligible force-on, and unresolvable models untouched) and a swap-path guard (swapping onto an ineligible model drops a forcedtrue→ AUTO, so the staleness can't regenerate). The false-negative escape hatch — forcing MTP on for an untagged-but-capable model — is preserved.
- Slot units re-render automatically on update. A slot's systemd unit
bakes the launch argv at load time, so updating hal0 changed the code that
WOULD render but not the file that DID —
Everything landed on
mainsince the v0.8.4b1 cut. The headlines: hal0 generalizes beyond the Strix Halo iGPU (experimental CUDA + multi-GPU pinning), companion services get one management surface (registry +/api/services+ mDNS + dashboard page), the settings-completeness plan finishes (phases 3–5 + an Advanced section with fullhal0.tomlparity), and seed profiles go virtual with dedicated embed/rerank lanes and a proper model×profile×slot MTP decision (#1045). Safe upgrade from v0.8.4b1 — the one on-disk migration (virtual-seed prune) backs upprofiles.tomlfirst and rescues divergent operator content to-customnames; note the MTP auto behaviour below if you run untagged local MTP builds.Upgrade note — re-render slot units after updating. A container slot's systemd unit bakes the launch argv at load time, so after an update the running slots (and their unit files) still carry the PRE-update flags. A bare
systemctl restart hal0-slot@<name>re-runs the stale ExecStart — restart slots through hal0 (dashboard restart, or unload→load) so the unit re-renders through the new code. The dashboard's resolved-command drift indicator shows which slots are stale. Automatic unit re-rendering on update (without bouncing serving) is planned as the follow-up.Upgrade note — stale
mtp = trueslot overrides crash on re-render. An explicitmtp = truein a slot TOML is honored literally (it is the escape hatch for MTP-capable models the eligibility heuristics miss). If a stale override — typically left behind by the old binary MTP pill or a pre-#1045 stack apply, surviving a later model swap — points at a model with NO MTP layers, llama-server exits at load ("context type MTP requested but model doesn't contain MTP layers") once the unit re-renders. Fix: set the slot's MTP to Auto ({"mtp": null}) or Off in the drawer and restart. An updater migration that clears provably-stale force-ons (with a loud log) ships in the follow-up.Added- GPU generalization — experimental CUDA + multi-GPU. A dedicated
cudaseed profile (upstreamllama.cpp:server-cudaimage, preferred by the installer when NVIDIA CDI is present, Vulkan fallback otherwise) and per-slotgpu_indexpinning for multi-GPU hosts. Ships alongside dead-path retirement and multi-file pulls (a model's mmproj/vision sidecars download with the main GGUF in one job). - Unified companion-service management (#1037). A code-level service
registry (Open WebUI, ComfyUI, Hermes, Hindsight, n8n),
GET /api/services+ allow-listed lifecycle actions, mDNS advertisement of addon services (hal0-addon-<id>.servicefiles, avahi inotify pickup,HAL0_HOSTNAMEprecedence), and a dashboard Services page (cards, logs drawer, ComfyUI queue drawer, fail-soft probes). - Settings completeness, phases 3–5. TTS request defaults
(
default_voice/default_speed/default_response_format) seeded into/v1/audio/speechplus a live voice list proxied from thettsslot (#1038); a Settings NPU section (#1040); ComfyUIidle_restore_minuteshot-reload (no API restart) and a workflow listing endpoint + dynamic strip (#1043). - Advanced settings section. Full
hal0.tomlparity in the dashboard (every config key editable, grouped, with descriptions), a memory-graph panel, and an API restart button; an AWS secret-pair preset and a reload-config-from-disk button; previously-inert config keys wired through, and the memory schema aligned to the Hindsight era. - Dedicated
embedandrerankseed profiles (#1045). GPU llama-server templates that bake in the serving flags (--embedding/--reranking,-ub 8192so a full input fits one physical batch) so an embedding or reranking slot no longer hand-wires them inextra_args. On agpu-rocmbox, embed/rerank capabilities derive onto these lanes automatically (install path and picker/apply fit path both updated); Vulkan/CPU boxes keep falling back to thevulkan/cpu-llmprofile until backend-specific variants ship. - Profile bench matrix tooling (#1045).
installer/bench/profile-matrix.shscripts the seed-profile re-tune matrix ashal0-benchctlseam sweeps, andinstaller/bench/server_ab.pymeasures the server-level levers llama-bench can't see — MTP draft depth (with acceptance %),--cache-reuseon a shared-prefix trace, poll, and embed/rerank endpoint sanity — via hal0-api as the unprivileged user, always restoring the slot's original config. Supersedes the ad-hoc/root/bench_mtp.py; an on-box runbook ships athandoffs/bench-profile-matrix-local-session-2026-07-04.md. - Catalog UX finish (#1042). Sort / tag-filter / quant chip wiring in the Models view, and a chat-template pick at pull time.
- Canonical device/backend taxonomy. One enum source at
GET /api/meta/enums(ETag/cache-friendly), consumed by the dashboard — plus stacks fixes and dialog guards that rode the same change.
Changed- Seed profiles are now virtual (#1045). The built-in profile catalog
(
SEED_PROFILES) is overlaid from code on every load and never persisted to/etc/hal0/profiles.toml. Previously the installer materialised every seed inline and the loader only injected missing seeds, so a re-tuned seed (new flags, a bumped toolbox image) never reached an existing install. Now the code definition always wins:load_profiles_configoverlays seeds over any on-disk copy,save_profiles_configstrips seeds before writing, and the updater'sensure_seed_profiles()prunes any materialised seeds left by an older install (self-heal on upgrade). Seed profiles remain immutable — clone to customise. Operator (non-seed) profiles are untouched. Data-safe migration: the pre-prune file is backed up once (profiles.toml.pre-virtual-seeds.bak) and any seed-named entry whose content differs from the code seed (a hand-edited seed table, or an operator profile whose name only became a seed in this release, e.g.embed) is rescued to<name>-custominstead of deleted, with a loud log. - MTP is now a model × profile × slot decision (#1045). Model
eligibility (
mtpregistry tag or name marker) × profile opt-in (profile.mtpnow means "enable for eligible models", not "append the bundle regardless") × a tri-state per-slot override (Auto/On/Off, Auto = profile opts in AND model eligible). A non-MTP model on an MTP profile no longer launches with dead--spec-draft-*flags, and the draft device tracks the profile backend (ROCm/Vulkan/CUDA) instead of hardcoded ROCm. The slot drawer swaps the binary MTP pill for the tri-state control with a live "Auto · active/inactive" hint; stack editor rows default to Auto, and an Auto row now clears a forced override on apply (the config write layer treats an explicitnullas delete-key — TOML has no null — which is also what makes "back to Auto" work from the dashboard instead of 500ing). Behaviour note: an MTP-capable model that carries neither the registrymtptag nor an MTP name marker stops speculating under Auto after this upgrade — tag the model or force the slot On; the launch log saysmtp.auto_off_model_ineligiblewhen this bites.
Fixed- MTP auto-off breadcrumb is launch-gated. The
mtp.auto_off_model_ineligiblehint lives inside the shared launch/preview scalar resolver, so it fired on every dashboardGET /api/slotspoll (~0.4/s per client, forever) for any AUTO slot pairing an MTP profile with a non-MTP model. It now logs only on a real container launch; preview/status renders stay silent, and launch/preview argv parity is unchanged. - Upstream-advertised models are clearly identified as remote (#1035), not local, across the dashboard model surfaces.
- Operator Board (#1032). Hermes-contract repairs, honest UI state, and platform-assistant chat.
- Slot pipeline hardening. API boundary validation, backend-switch completion, manager guards, and guarded stack writes; a single argv assembler with model-defaults wiring and provider fixes; normalizer bug, dead-code removal, and a11y quick wins in the slot drawers.
- Settings polish. Truthful apply plan, safe engine picker, secret descriptions, rollback behaviour, and palette ghosts.
Docs- README re-baselined to v0.8.4b1 + full accuracy pass (#1044, #1046):
canonical
agent/utilityseeded slots, real backend-profile and hardware-tier tables (experimental CUDA row), removed the no-longer-shippedHAL0_USERunprivileged mode, added the Discord invite (header + Contributing). - hal0.dev docs mirror refreshed (#1044): new
operate/servicespage,operate/authrewritten to the real ADR-0012 reverse-proxy model (the fictional--auth=basic/ managed-Caddy docs from 3e056de removed site-wide), plus the full v0.8.x feature-doc sweep. - Handoffs: platform reliability/config/UI review (#1010), ONNX / Strix Halo NPU research and integration plan (#1034), llama.cpp seed-profile evaluation + consolidation proposal (#1041).
- GPU generalization — experimental CUDA + multi-GPU. A dedicated
A models, logs & memory follow-up to v0.8.3b1. Models can now carry a preferred runtime profile that loads with them, image-gen/ComfyUI models get their own properly-tagged surface, the slot context window finally persists across reloads, the logs/events system is unified, and the memory subsystem gains a destructive-op audit trail plus console shape guards. Safe upgrade from v0.8.3b1 — no breaking changes; all additions are additive.
Added- Model preferred profile. A registry model can declare
defaults.profile— the runtime profile it wants loaded with it. A slot adopts it on create (when it has no explicit profile) and on every model swap, gated on device/type compatibility (an incompatible preference is ignored and the slot keeps its device-default profile; slot hardware is never flipped to satisfy a model). Surfaced as a Preferred profile selector in the model recipe editor. - ComfyUI / image-gen model surface. The Models view gains a Models |
Image/ComfyUI segmented toggle; image-gen models are grouped by their
models-tree category (checkpoints/loras/vae/upscale_models/…) and kept out of
the dispatcher list. ComfyUI models are correctly tagged
image/comfyuiat every registration path, and/api/modelsself-heals rows an older pull mis-tagged by deriving the ComfyUI category from the on-disk path (no migration needed). - Memory: audit trail for destructive ops (#1024). Every destructive
/api/memory/*op — bank delete, and memories/config/document/directive/ operation/mental-model deletes, plus the namespacePOST /api/memory/delete— now records a durable audit row (actor + target + truthful outcome) via the sharedrecord_actionfacility, so a memory wipe is attributable after the fact. Complements the bank-DELETE?confirm=guard shipped in #1028. - Memory: response-shape guard on the cognition consoles (#1026). The
recall/reflect/directives passthroughs now validate the load-bearing envelope
key (
results/text/items); upstream Hindsight shape drift surfaces as a loudmemory.engine_shape502 instead of a silently-blank console panel. The two collidingrecallcontracts (namespace{items}vs bank{results}) are now documented at both call sites.
Changed- Unified logs/events. Restores per-slot model-load logs, real source/slot attribution, and a channel selector across the logs/events surface.
- Memory Overview UI. The graph-extraction gate now sits beside a shrunk
"memories retained" spark in the top row; card headings share one unified
"eyebrow" style; dropped the stray
ADR-0023label from the extraction title.
Fixed- Persistent slot context. The slot edit drawer seeds the context field
from the persisted
[model].context_size(not the live runtime metric or a hardcoded 16384) and only writesctx_sizewhen it actually changed, so an unrelated save no longer clobbers the stored context window with 16k on a cold reload or swap. - Don't surface invisible models. FLM tags advertised by the composite upstream before their weights are pulled are dropped from the catalog (the dedicated probe still surfaces the genuinely installed ones); freshly-pulled ComfyUI checkpoints are no longer mis-filed as chat models.
- Memory tab functional; bank delete guarded (#1028). The Memory tab renders
its graph status/slot UI and consolidate/list actions correctly, and a bank
DELETEnow requires an explicit?confirm=guard. - Stack edit drawer (#1023). Slot cards render as labeled multi-line entries and the escaped toggle-knob glyph is fixed.
- Board: Hermes kanban task-detail drawer (#1014). The task-detail envelope from Hermes is unwrapped so the board drawer renders instead of showing empty.
- Installer: cosign optional for the one-line install. The bootstrap no
longer hard-requires
cosign, so the one-line installer runs on hosts without it (signature verification still applies wherecosignis present).
- Model preferred profile. A registry model can declare
A large reliability and UI-completeness release. The headline is a 72-finding platform-review remediation delivered as eight verified waves (each finding regression-tested and gated by CI + Playwright), landing alongside earlier staged fixes. It retires several silent-failure bugs, adds cross-process safety and pull resumability, makes every backend-supported value editable in the slot/model/profile drawers, and surfaces live telemetry on the dashboard. Safe upgrade from v0.8.2b4 — no breaking changes; all config/UI additions are additive.
The most user-visible behaviour changes: the dashboard now shows Power & Thermal (live GPU clock/temp/power) and Per-Slot Throughput cards by default; disabling a capability now genuinely stops it serving; and the
bgereranker is now classified and routed as a reranker.Added- Dashboard live telemetry, on by default. The Power & Thermal card (GPU clock MHz, temp, power) and the Per-Slot Throughput card are now default-on, and the Utilization card shows a live clock/temp caption. (#1019)
- Edit-drawer completeness. The model editor now exposes
capabilities,backends,rope_freq_base,mmproj, andhf_repo/hf_filename; slots gain a per-slotvisiontoggle and NPUasr/embedmodality toggles — all fields the API already accepted but no drawer surfaced. (#1020) - Settings. An opt-in anonymous telemetry toggle and the image-gen
defaults (
default_size,default_steps,idle_restore_minutes). (#1021) - Interrupted pulls resume via HTTP
Range(withIf-Range) instead of re-downloading from zero; the on-disk prefix is re-hashed so the final SHA-256 stays exact. (#1017) - Disk-space preflight before multi-GB pulls fails fast with a
structured
model.insufficient_diskerror instead of filling the disk. (#1013) - Host-memory-pressure LRU eviction of idle slots. (#1003)
Changed- Retired duplicated logic that had silently drifted: one device→profile derivation, one filename→capability classifier, one dispatchable-state predicate, and one slot-projection reconcile. (#1015)
- Rebuilt the dash editing drawers on a shared
FormDrawer+useFormwith an unsaved-changes dirty guard, onecompatibleModelsfilter, a focus trap and real<label>wiring (a11y), plus honest, confirmed destructive actions (styled type-to-confirm deletes; no more "Pause" that silently cancels; real "fits in memory" check). (#1016, #1018) - Cross-process safety: advisory file locks around registry and
capabilities writes; parent-directory
fsyncafter atomic writes. (#1017) - Housekeeping: startup GC of stale pull-job snapshots and orphaned
.partpartials. (#1017) - Extracted the capability-resolution heuristics out of
dispatcher/router.py. (#1017)
Fixed- Disabling a capability now sticks. The disable is written through to the slot config, so a later request can no longer wake a "disabled" slot and serve from it. (#1011)
- The NPU trio is advertised on podman-only hosts — the picker probed
dockerand never offered it on the reference platform. (#1011) - GPU-less installs get a chat-capable primary slot instead of one
bound to the Kokoro TTS engine (
cpunow defaults tocpu-llm). (#1011) - Idle-evicted embed/rerank/tts slots wake on request instead of 404'ing until a manual load. (#1011)
- The
bgereranker is classified as a reranker (was mislabeledchat) and is routable as one. (#1015) - Installer/bundle pulls survive an api restart — status/stream polls no longer 404 mid-install. (#1012)
- A completed pull is no longer reported "failed" after a restart. (#1012)
- The GpuArbiter drain no longer unloads a slot under an in-flight request (image-mode switch race). (#1012)
- Slots are no longer advertised READY on a health-probe timeout. (#1012)
- Write-time validation: a second
default=trueslot of a type is refused at save;create()no longer clobbers an existing custom slot; stack apply flags unresolved profile/model refs and reports degraded (not "clean") when slots fail to load. (#1013, #1018) - The model editor no longer silently wipes unshown launcher defaults on
Save, and the ComfyUI image-profile control no longer corrupts
device_classon edit. (#1020) - Chat requests with mis-positioned or stacked
role='system'messages no longer 500 the Qwen3.6-35B-A3B upstream. The OpenAI-compat normaliser now collapses every system entry into one and hoists it to position 0 (matching the OpenAI/Anthropic convention), so a stale mid-array system message from the SPA/Open WebUI/LibreChat — or two deliberately-stacked system blocks — no longer trips the Qwen3 Jinja template'sSystem message must be at the beginning. No-system payloads skip the copy. (#992) - Additional staged fixes from the assessment sweep: dead-port guard for
container slots (#1001),
events.gapon subscriber-queue overflow (#1000), non-retryableSlotLoadFailedfor ERROR slots (#999), bounded httpx pool + tighter read timeout (#998), cold-slot 404 ordering (#996), container slots entering SERVING and bumpinglast_used_at(#995), installer port reachability when Docker is co-installed (#990), updater rollback re-pip (#994), an in-memory PgVector write warning (#1008), additive seed-profile merge (#1007), durable pull-job persistence (#1006), canonical slot fields on npu/load + install model-update (#1009), and the yellow-halo favicon restore (#991).
Documentation and installer hygiene — no runtime behaviour change. This release re-baselines the engineering docs to the current v0.8.x reality (container runtime, Hindsight memory, agent/utility roles) and fixes a handful of installer drift bugs surfaced by a codebase assessment sweep. Safe upgrade from v0.8.2b3.
Changed- Docs re-baselined to v0.8.x. Swept the Lemonade→container-runtime and
Cognee→Hindsight terminology out of
AGENTS.md,ARCHITECTURE.md,CONTEXT.md,PLAN.md, andREADME.md; corrected the Hermes provisioner to its real 15-phase pipeline; refreshed version/status lines; and rewrote the (largely fictional)hal0-service-managementcodebase-map reference against the currentsrc/hal0/tree. Marked the shipped Stacks and voice-stack superpowers plans as completed. - Dead ADR links fixed. Tracked docs no longer link into the gitignored
docs/internal/adr/tree (#638); surviving decisions are inlined and ADRs are referenced by number. (In-code citation sweep tracked in #984.)
Fixed- qwen3tts migration script aligned with the
tts-slot model. The standalone-to-slot migration script's Guard 1 checked a non-existentqwen3ttsslot (always 404) and Guard 3 checked a removed kokoro:8084fallback; both are corrected to the deployed design where Qwen3-TTS serves from the canonicalttsslot viavoice.tts. (#979) uninstall.shnow removes all three sudoers grants. It only removedhal0-benchctl, leakinghal0-agentenvandhal0-comfyuibehind on uninstall.- Hermes private memory bank seeded under its canonical name.
install.shseededprivate__hermes-agent, which the server (which derives the bank from the agent-id viaPRIVATE_PREFIX="private:") never matches — corrected toprivate:hermesso the pre-seeded retain-mission/dispositions actually apply. - Dropped the obsolete Lemonade boot-contention comment from
installer/comfyui/scripts/comfy-up.sh.
- Docs re-baselined to v0.8.x. Swept the Lemonade→container-runtime and
Cognee→Hindsight terminology out of
GPU text-to-speech lands: Qwen3-TTS now runs as a hal0-native slot with a one-switch swap between the Kokoro (CPU) and Qwen3-TTS (GPU) engines, plus a GPU benchmark harness and dashboard polish. Safe upgrade from v0.8.2b2.
Added- GPU Qwen3-TTS as a hal0-native slot. New
Qwen3TTSProviderserves Qwen3-TTS from thettsslot, with avoice.ttscapability switch that swaps the engine between Kokoro (CPU) and Qwen3-TTS (GPU) without reconfiguring the slot. (#972, #976) The toolbox image builds + pushes to ghcr.io and its digest is pinned inmanifest.json. (#975, #977) Ships a standalone-to-slot migration runbook + guarded script. (#974) - GPU benchmark harness. A GPU benchmarking toolbox with the
hal0-benchctlseam and accompanying agent skills. (#967) Dashboard gains an iGPU usage gauge and a prefill TTFT readout. (#968) Topbar adds Kanban / Agent-Chat launchers, a global agent chat, and an Archived lane. (#966)
Fixed- Dispatcher injects upstream auth headers for remote providers, so requests routed to authenticated remote upstreams carry their credentials. (#973)
- UI builds land reliably on deploy (no-cache index + install to the served
dist). (#969)
Docs- Benchmarking toolbox UI/feature handoff. (#971)
- GPU Qwen3-TTS as a hal0-native slot. New
Two fixes on the 0.8.2 beta line — profile MTP tuning now actually takes effect, and stacks can pull their referenced models. Safe upgrade from v0.8.2b1.
Fixed- Explicit profile spec flags win over the MTP bundle.
resolve_profile_flagsappendedMTP_FLAG_BUNDLEafter a profile's own flags, so the bundle's spec-draft defaults (--spec-draft-type-k q8_0,--spec-draft-p-min 0.0) silently clobbered any--spec-draft-*a profile pinned — there was no way to tune the MTP draft through a profile. The bundle is now merged as defaults that the profile's explicit flags override (gap-filled only). (#963) - Absent stack models can be pulled. Custom GGUF builds referenced by the
seed stacks (saber, pi-agent, qwopus coders, halostrix, gemma, …) were
auto-scanned with empty
hf_repo/hf_filename, so on stack import/apply they classified "unresolvable" with no download URL. Registered their public HF coordinates (jcbtc/ + Jackrong/ + unsloth/ repos) in the curated catalogue;embed_referencesfalls back to curated on export; and a newbackfill_coordless()repairs existing coord-less registry rows on rescan. (#964)
Changed- CI: cancel superseded PR runs (never
main); PRs test Python 3.12 only whilemainruns 3.12/3.13/3.14 (3.14 non-blocking); least-privilege workflow permissions; Node 20 → 22. (#898)
- Explicit profile spec flags win over the MTP bundle.
Profiles gain the same portable export/import/sharing model stacks already have. Safe upgrade from v0.8.1-beta.2.
Added- Portable profile export/import. A profile can now be exported to a
self-contained, checksummed
.hal0profile.jsonenvelope and imported on another host — the same file-based sharing model stacks use. The envelope carries the profile template plus a sha256 content checksum and an independentschema_version; no secrets or host paths are serialized. New routesGET /api/profiles/{name},POST /api/profiles/{name}/export, andPOST /api/profiles/import(dry-run reports checksum validity + name collision; commit creates under a chosen name,409 profiles.existson a duplicate). New MCP tools mirror thestack_*set:profile_list/profile_status/profile_export(autonomous read) andprofile_import/profile_delete(gated). The dashboard adds an Export button to every profile card and an Import dialog (file → dry-run preview → commit). (#962)
- Portable profile export/import. A profile can now be exported to a
self-contained, checksummed
Bugfix on the 0.8.1 beta line — restores fleet auto-update. Safe upgrade from v0.8.1-beta.1.
Fixed- Updater version comparison uses PEP 440.
hal0 updatecompared versions with a digit-tuple parser that split on.and stripped non-digits per segment, so the pip-normalised installed beta0.8.0b3parsed to(0, 8, 3)and the tag-form manifest0.8.1-beta.1to(0, 8, 1, 1)—(0,8,1,1) > (0,8,3)is false, so every box on a0.8.0bNbeta saw the new release as "not newer" andhal0 updatereported nothing to apply (the installed beta number was misread as the patch component). The comparator now usespackaging.version.Versionin both the updater and the API route, falling back to the digit-tuple only for non-PEP-440 nightly tags (whose timestamp ordering still relies on it). (#957)
- Updater version comparison uses PEP 440.
Installer/privilege simplification + Hermes durable memory on by default. The hal0-api privilege seams (hardened unprivileged mode, the slot privilege seam) are gone, and a fresh
hal0 agent bootstrap hermesnow provisions a working durable-memory provider out of the box.Added- Hermes durable memory enabled by default. Provisioning now ships a working
hal0-memoryprovider and setsmemory.provider=hal0-memory, so Hermes gets cross-session recall with no manual config. Two banks —private:hermes(default) andshared(cross-agent) — backed by hal0's Hindsight engine via the hal0-api REST front door; reads union both banks. The provider exposeshal0_memory_{search,recall,add}(withshared=trueto write the shared bank) and auto-injects recalled context each turn. (#955)
Changed- Hermes agent identity is
hermes(washermes-agent), matching thehal0 agentregistry. The agent-id is the single source for theX-hal0-AgentMCP headers, the persona memory namespace, and the prelude. (#955) - Memory plugin install path fixed. The plugin is copied to
$HERMES_HOME/plugins/hal0-memory/(a direct child ofplugins/, which the Hermes loader actually scans) instead of the nestedplugins/memory/…that never loaded. (#955)
Removed- Hardened (unprivileged hal0-api) mode removed; hal0-api runs as root. Dropped live-hello; fixed ready-summary IPs. (#953)
- Dormant slot privilege seam removed (
hal0-slotctl+ euid routing). (#954)
Breaking- Hermes memory namespace renamed
private:hermes-agent→private:hermes. Existingprivate:hermes-agentdata is not auto-migrated; reprovisioned agents start recalling fromprivate:hermes+shared. (#955)
- Hermes durable memory enabled by default. Provisioning now ships a working
Canonical LLM roles + Hindsight-native memory extraction (ADR-0023). The two canonical LLM roles are now
agent(the capable default + fallback anchor, replacingchat) andutility(the cheap helper, now seeded on every install).chatandprimaryare retired as slot/role names.Changed- Canonical roles are
agent+utility.agentreplaceschatas the default/anchor everywhere (seeded slots, dispatch rule-9 fallback, the default pin set,_configured_primary).utilityjoinsSEEDED_SLOTSso a fresh box never silently falls back to a heavy model for cheap extraction. - Generalized virtual addressing. Any enabled
type=llmslotXis now addressable ashal0/X(chain(X, agent)); the advertised canonical virtuals arehal0/agent,hal0/utility,hal0/npu. - Memory graph extraction is operator-selectable and actually wired.
[memory.graph].extraction_slotnames the local llm slot Hindsight uses for graph extraction; hal0 propagates it tohindsight-apivia a systemd drop-in (HINDSIGHT_API_LLM_MODEL=hal0/<slot>) + restart.hal0 memory graph enabletakes--slot <name>(validated against the live enabled-llm-slot set). - Cognee fully removed. The Cognee engine + wrapper are deleted; Hindsight is
the platform engine (with a PgVector boot-degrade fallback).
MemoryRecordsurvives as an alias ofMemoryItem.
Breakinghal0/chatis no longer advertised. Clients pinned tohal0/chat(Hermes, OpenWebUI, any custom consumer) must repoint tohal0/agent. Hermesmodel.defaultis nowhal0/agent.memory.graph.route/memory.graph.upstreamremoved, replaced bymemory.graph.extraction_slot(default"utility"). Oldroute/upstreamkeys inhal0.tomlare silently dropped on load (no hard-fail on upgrade). Thehal0 memory graph enable --route/--provider/--modeloptions are gone — use--slot.primaryis no longer a slot alias.SLOT_ALIASESis{"agent-hermes": "agent"}.
- Canonical roles are
Bugfix release on the 0.8.0 beta line. No behaviour changes beyond the two fixes below — a safe upgrade from v0.8.0-beta.1. First release to carry #948.
Fixed- Operator Board live updates restored. The board's events-WS proxy
(
/api/board/events) resolved its upstream Hermes session token from theHERMES_SESSION_TOKENenv var only, while the REST path harvests the rotating per-process token from the dashboard HTML. With no env pin (the default), the WS connected upstream with no token, Hermes rejected the upgrade (403), and the browser socket died with 1011 — so tasks created in Hermes loaded on refresh but never pushed live to the board. The WS bridge now shares the REST client's token resolution (env-pin → HTML-harvest → rotation cache) and re-harvests + retries once on connect failure. (#949) - Hermes privileged env seam. A privileged env-write seam lets the
unprivileged provisioner write
root:root.envfiles, so Hermes agent config provisioning works under the dropped-roothal0-api. (#948)
- Operator Board live updates restored. The board's events-WS proxy
(
First beta of the 0.8.0 line — the model-config, Hermes, and permissions overhaul. Configuration becomes a declarative single source of truth (Stacks + single-source launch argv), Hermes consolidates onto its own config ownership, and
hal0-apican finally drop root. Voice (TTS + STT) lands end-to-end. One behaviour change to be aware of: thehal0/primaryandhal0/flmvirtual aliases are gone — see Changed.Added- Stacks — declarative config SSOT. A
StackConfigschema plus aStackApplyEnginethatplan()s a Stack into a ChangeSet,apply_config()s it as an atomic commit with rollback, andconverge()s the live slot set (primary-slot load/swap/skip + capability-child routing through the orchestrator). Content-hash drift detection and an active-stack pointer, export/import via a checksummed.hal0stack.jsonenvelope, snapshot of live config into a Stack, and aStacksCatalogCRUD with seed guards. Seed stacks (saber / forge / pi) derived from the roster bench. (#921, #923, #925, #926) - Voice stack — TTS + STT. Brought up and verified end-to-end:
voice_wirefixed, Open WebUI Call mode wired, and the NPU-trio facade auto-provisions STT. (#924, #928) - Single-source slot argv (overhaul stream A). A resolver dedups the launch
flag soup down to a last-wins canonical command; per-flag provenance is
exposed at
GET /api/slots/{name}/resolved, and the slot Edit drawer renders the resolved command with per-flag source badges (base / profile / extra_args). (#929, #930, #931, #932) - Capability-based slot fallback. When a slot's
model.defaultisn't locally servable (registered-but-no-file, or pulled-away),load()falls back to the best locally-registered model matching the slot's capability — excluding diffusion / image / video models and preferring name-similarity to the configured id. (#940, #942) - Hardened permissions — run
hal0-apiunprivileged (opt-in). SetHAL0_USER=hal0and the installer drops the API off root: a declarative ownership table (audited read-only byhal0 doctor perms), a narrow privileged seam (hal0-slotctl+ a no-wildcard sudoers grant) so the unprivileged API can still write per-slot units and drivehal0-slot@*, and a codified flip (run-as drop-in + recursive chown of config + state, pruningagents/+secrets/+ the models dir). Slot containers stay rootful — the container remains the sandbox boundary. DefaultHAL0_USER=rootis byte-for-byte unchanged. (#929, #943, #944, #945) - Hermes owns its own config. The runtime is unpinned with a real upgrade
path, and a config-set overlay replaces the whole-file
config.yamlrender — Hermes owns and self-migrates its config while hal0 layers only its keys. (#934, #938) - Chat-template render-validation. The template catalog is render-validated so a broken template can no longer ship silently. (#917)
Changed- Breaking —
hal0/primaryandhal0/flmvirtual aliases removed. Virtual model names now map 1:1 to their resolution chains;hal0/primaryno longer resolves (usehal0/chat) andhal0/flmis gone (usehal0/npu). Slot-name back-compat is intentionally kept, and the Hermes overlay now emitsmodel.default: hal0/chat. (#939) - q8_0 KV cache, universally. Main and MTP-draft KV caches are now
q8_0across slots — near-lossless and keeps fused FlashAttention on AMD HIP. (#933) - Profiles lift bench-tuned MTP config into
rocm-moe/rocm-dnse. (#922) - Open WebUI disables PersistentConfig so the env prewire wins the chat connection. (#927)
- Docs mirrored from hal0-web.
Fixed- Installer —
setup --storage-diris passed as a separate argv token so fresh--models-dirinstalls seed slots correctly (#946); the hardened-perms flip chowns config + state recursively so a root→hal0 upgrade doesn't strand root-owned state subdirs (#945);hermes gateway installruns non-interactively and skipsenablewhen the unit is absent (#941); a lemonade-team PPA is added so the FLM/NPU.debresolves on a fresh Ubuntu box (#937); the registry scans the effective store / pull_root rather than only declared roots (#935). - Hermes gateway marks its
EnvironmentFileoptional (-) so fresh installs don't crash-loop on a missing secrets vault. (#936) - Dependencies — bump vulnerable deps flagged by Dependabot. (#919)
- Stacks — declarative config SSOT. A
Second beta in the 0.7.3 line. Vision lands on the chat slot, idle slots finally give their RAM back, and the Operator Board stops crashing on task creation.
Added- Chat-slot vision — mmproj sidecars are now associated with their parent
model in the registry, the container provider emits
--mmprojfrom that sidecar, and vision auto-surfaces as a capability with a per-slot toggle. (#899, #900, #901) - TTL-driven hard eviction — idle slots are now unloaded after their timeout, freeing resident RAM instead of merely relabelling READY→IDLE. (#902)
- Hermes memory authorship — writes from Hermes are stamped with an
agent:hermesauthor tag. (#912)
Fixed- Operator Board no longer black-screens (React #31) when adding a task; modal styling, drag-to-delete, and the agent-chat drawer are reworked, and board chat now runs on the agent slot instead of the (wedged) chat slot. (#905, #914, #916)
- Slot MTP gate hardened on the backend and NUL bytes stripped from model-modal inputs. (#918)
- Memory rejects anonymous private writes (stops
private__anonymousmisrouting). (#915) - Memory banks grid fills its width again, unnested from the section title row. (#911)
- ComfyUI reads slot logs from journald and reworks its card layout. (#909)
Changed- Docs mirrored from hal0-web.
- Chat-slot vision — mmproj sidecars are now associated with their parent
model in the registry, the container provider emits
First Beta. The dashboard becomes a full operations console — ComfyUI image generation, an agent task board, NPU/FLM slots, and a unified profile-card layout — on top of honest slot health and per-slot context derivation.
Added- ComfyUI generation engine — full platform integration (model store, capability picker, installer wiring, V2 Image-Gen pane). The Image-Gen tab collapses its queue/workflows and an Inference-tab dot tracks live state; image generation flips the GPU into exclusive image mode via the iGPU switchover. (#878, #890, #881)
- NPU occupancy — a living occupancy grid with per-slot accents and activity-driven breathing, replacing the NpuFlmStack/trio picker. (#859, #861, #860)
- Operator Board — a hal0-skinned kanban wired to Hermes (
/api/board/*) with a live agent-chat drawer and working task creation. (#852, #858) - Agents page — an agent-card library with a live Hermes card. (#848)
- Dashboard overhaul — inference/NPU/ComfyUI cards unified to the profile-card style; Profiles given an engine-style section header and moved into the Slots tab; inference-pane living-grid redesign; sidebar nav accordion + bottom Services launch zone; a live-journal footer with runtime + service health groups; the memory+throughput band lifted above the tabs and slot cards freed from the accordion. (#888, #889, #879, #894, #867, #853)
- Editable per-slot
extra_argswith a Regenerate overlay. (#854) - Qwen3.6 MTP chat template + slot rails.
- A generated changelog is now included in every release (nightly + stable). (#842)
Changed- Slot health-probe honesty — a slot is marked ready only once its real
/healthpasses, not on a systemd snapshot. (#866) - Slot context is derived per-slot and never silently inherits llama-server's 4096; the edit-drawer default is 16k. (#862, #850)
- Disabled-but-running slots are surfaced; the enable toggle moved into the drawer. (#856)
- The runtime indicator split into a sidebar launcher + a footer health chip. (#864)
- Durable group-shared model ownership for an editable
/opt/hal0. (#843, #857) - Nightly versions carry a sub-day timestamp so same-day re-cuts stay monotonic. (#841)
Fixed- Hardware: report the live GTT total instead of a stale cached probe value. (#891)
- NPU: probe AIE columns via a temp file, not
-o /dev/stdout. (#893) - Slots: harden container config-drift comparisons and warn on drift. (#880, #869)
- Routing: translate FLM
<tag>-FLMids to served tags in the chat-slot rewrite. (#840) - Hermes: run-as-
hal0guard + ownership handover prevents root-clobber; corrected env arg order and dashboard TUI argv order. (#844, #847) - Dashboard: dedup the journal SSE ring; chyron/timestamp polish; grid alignment; empty memory-bank graph no longer locks the dashboard; stray dev-test slots removed from the persona UI; responsive sizing + chrome cleanup. (#868, #871, #870, #845, #855, #846, #851)
Docs- Restored
doctor perms+migrate model-layoutto the CLI reference; added the deploy + PR workflow for parallel teammate sessions. (#849, #865)
Pre-Alpha. Retires the web FirstRun picker in favour of a terminal
hal0 setupTUI, and adds Ubuntu 26.04 / Python 3.14 install support.Addedhal0 setupTUI — replaces the web FirstRun picker with a rich two-column terminal setup (storage → Extensions → Main model → Agent model → NPU) over an always-on context pane. Hybrid apply (in-process at install time, via the API when it's up — roster coherence),--auto/--storage-dir/--no-pull/--no-extensionsflags, and a tier-lessPOST /api/install/apply-selectionsendpoint (#833).- Extensions — selectable, auto-wired Apps (Open WebUI) + Agents (Hermes, Pi),
a growing registry surfaced in
hal0 setup(#833). - Ubuntu 26.04 / Python 3.14 install support — per-distro FLM
.debselection, hindsight--ignore-requires-python, py-version-agnostic Hermes web_dist (#829).
Changed- A fresh install seeds the hardware-recommended Main slot non-destructively
(only slots whose config is absent) and writes the first-run sentinel via
hal0 setup --auto --no-pull— sohal0 update/re-install never overwrites a customised slot. The web bundle-tier picker is retired; the bundle backend is kept dormant for the future Stacks feature (#833).
Removed- Web FirstRun picker (
firstrun.jsx+ hooks), the v1/api/bundlessurface,bundles/store.py, and the legacy/api/install/pick-defaultroute (#833).
Pre-Alpha. Zero-boot install + FirstRun v2: a fresh install now stands up the memory engine, agents, and Hermes with no manual steps, and the FirstRun wizard orchestrates a full multi-slot bring-up from a single bundle pick.
Added- FirstRun v2 — quick-path wizard + orchestrated multi-slot install from a single bundle/kit pick (#809), with an Advanced drawer exposing per-slot model/profile overrides (#812).
- Slot config UX — Phase 2 per-slot MTP override + capability-gated MTP pill (#800); Phase 3 non-manual chat templates, model-level and per-slot (#802).
- Zero-boot installer — stands up a local Hindsight memory engine + seed banks (#806), ships the hal0 agent skills + drop-in dirs (#805), and provisions Hermes on a fresh install (#804).
- NPU telemetry — live column / duty / tok-s / KV surface, repointed to
hal0-toolbox-flm:0.9.43(#813). - Settings — HuggingFace token field +
api.envhint for gated pulls (#816); standalone/pulluses capability-grouped paths (#815). - Dashboard overhaul — the home page becomes a customizable operator widget board: drag/resize/pin-to-home slot cards, live memory-map, throughput, utilization and power monitors, a quick-chat tester, and a live ComfyUI job-queue widget; layout persists per operator (#814).
- v0.5 navigation — Connections dissolves into Slots/Agent tabs with sidebar sub-links; Memory + MCP unify under a tabbed Agent shell (#817).
Fixed- Non-blocking slot controls + NPU/image-gen toggles; cancel mid-load (#801).
- Slot edit drawer shows profile intent in its dropdown (#811).
- Enforce device↔profile backend coherence on slot create/update (#807).
- Drop the unimplemented
memory migrate --applyflag (#820).
Internal- Recolor the device palette — free red for errors/stop (#803).
- CI tests against the latest supported Python (3.12) only (#808).
- gitignore
.superpowers/brainstorm scratch (#810).
Pre-Alpha. First release carrying the clean-install hardening proven end-to-end on fresh Ubuntu 24.04 containers:
- Bundled-agent install converges on the hal0-managed venv —
hal0 agent install hermesprovisions toolchain → venv → wrapper → unit in one foreground command, and the API path becomes a thin register-or-hint (#766). - Installer auto-installs the python venv stdlib on clean Debian/Ubuntu instead of aborting at preflight (#778); NPU host-lib prereqs (ffmpeg6/XRT) are now best-effort, not fatal (#779).
/var/lib/hal0permissions let thehal0agent refresh the shared STATE.md the session hook reads (#777).- Slot config UX Phase 1: grouped drawer, reasoning pill, type-default pane, reactive model dropdown (#796).
- Bundled-agent install converges on the hal0-managed venv —
End-of-stream cut for v0.3. Bundles MCP-completion, memory-map redesign, the Settings → Updates fix, the silent-eviction dispatcher recovery, ADR-0020 OpenRouter callback skeleton, the persona spending-cap primitive, and the docs/internal pin + dashboard-v3 walkthrough.
After this tag, active scope rolls to v0.4 (install-mode reconciliation, UI polish, fully-implemented Agents/UI/Install bootstrapped) and v0.5 (MCP admin + memory wiring across UI and agents).
Added- Per-persona spending-cap primitive (#411 — Phase 0 OpenRouter
prereq).
[persona.budget]TOML sub-table + pure-Python budget enforcement layer landing BEFORE the V1 OpenRouter upstream provider and V2hal0-fusionMCP server. DA review of the OpenRouter integration plan flagged this as P0 must-fix #3 — without a spending-cap envelope, fusion (4.4× cost vs single-model) plus a recursing Hermes loop could drain a $200/credit pool overnight.src/hal0/agents/budget.py—Budgetdataclass, append-onlyBudgetLedger, purecheck_budget/record_charge, daily / monthly / lifetime aggregation + per-call max.- REST surface under
/api/agents/{id}/personas/{pid}/budget—GET(caps + spend + headroom),PUT(replace; round-trip preserves),POST /check(dry-run pre-call gate),POST /charge(post-response recorder). - Ledger at
/var/lib/hal0/agents/{agent_id}/personas/{persona_id}/spend.jsonl— append-only JSON-lines, fsync per write,tail -f | jqfriendly. PersonaBudgetPaneldashboard editor under Personas tab.- Persona seed (hermes + coder) ships with empty budget block;
operators opt in.
hal0 agent reprovision hermespreserves operator-set budgets (idempotent seed,overwrite=False). - Scope: per-persona only in v0.3.2; per-agent and platform-wide scopes deferred to v0.4. No provider charges this primitive yet — V1 OpenRouter wires the pre-call gate and post-response record.
- ADR-0020 + OpenRouter callback skeleton + loopback guard (#409,
Phase 0 OpenRouter prereq). Documents why the future OAuth PKCE
callback URL is constrained to
127.0.0.1so ADR-0012's LAN-trust posture survives the V1 OpenRouter integration. Ships a registeredGET /api/openrouter/auth/callbackroute returning HTTP 501 with a per-route loopback guard so V1 inherits a baseline that respects the constraint from day 1. No live behaviour change. - Dashboard v3
/agentreal-backend wiring (#364, closes #207 #228 #227 #226).useAgents()hook against/api/agents; live Memory tab against/api/memory/graph/status; live Skills tab against newGET /api/agents/skills; PersonaEditModal hydrated from newGET /api/agents/persona-enums. Server-side TONES + TOOLS- skill catalog moved to
src/hal0/agents/persona.py.
- skill catalog moved to
- Embedding model pinning + rerank wiring (#365, closes #116).
New
[memory.embedding]config block —model,rerank_enabled,rerank_url,rerank_over_fetch_factor,rerank_max_candidates, splitrerank_connect_timeout_s/rerank_read_timeout_s. Defaults preserve v0.3.0 semantics (rerank off, embedding model unchanged). - Private namespace contract for REST + read path (#366 + #369,
closes #317 #367).
X-hal0-Agent+X-hal0-Privateheader contract on/api/memory/{add,search,list,delete}— shared ADR-0005 §3 resolver insrc/hal0/memory/namespace.py. Wrapperadd/search/list_items/deleteaccept per-callclient_id;_allowed_read_datasetshonors it so per-agent reads work end-to-end. Audit rows now stamp the resolved per-call identity instead of the singleton wrapper's anonymous default. Identity hardening: regex on agent id (path-traversal blocked), rejection ofprivate:*agent values, rejection of bodydataset=private:*when the private toggle is off. - Dashboard v3
/mcpinstall/uninstall/config + real audit stream (#368, closes #305 #224 #222). Newsrc/hal0/mcp/installed.pyregistry +src/hal0/mcp/manifest.pyresolver (oci/npm/uvx/git/httpspecs). 501 stubs for install / uninstall / config replaced with real impls;/api/mcp/resolveadded;/api/mcp/serversmerges bundled (live FastMCP introspection) + installed (registry). Real audit stream consumed byuseMcpServerLogs. SSRF guard on URL fetch (loopback / RFC-1918 / link-local / 169.254.169.254 / mDNS / CGNAT / unspecified all blocked; redirects disabled). Registry files at/etc/hal0/mcp-servers/<id>.tomlwritten 0o600 inside a 0o700 directory.
Fixed- Settings → Updates: Install update silently no-op'd (#386).
The dashboard's Install button hit
POST /api/updates/apply, received 202 with ajob_id, toasted "Update started", and never polled the job — so when the background apply hitUpdateExtractErrorfrom a leftover/usr/lib/hal0/hal0-<v>/the user saw nothing. Three fixes:- UI:
useUpdateApplysignature corrected (version?, not misnamedchannel);useUpdateCheckGETs/api/updates/check(was POSTing to a GET-only route → silent 405); newuseUpdateJob(jobId)poller surfacesrunning/applied/failedto inline progress + toasts. - Backend:
Updater._extract_tarballnow quarantines a prior hal0 extraction at the same path to<dest>.stale-<unix-ts>instead of refusing, so a retry after a half-failed apply isn't permanently wedged. Foreign non-empty dirs are still refused — heuristic recognises hal0 installs byVERSIONfile orpyproject.tomlname="hal0". - Deduped the non-empty check in
Updater.apply(); the extract step is the single source of truth.
- UI:
- Dispatcher silent-eviction recovery (#392). When Lemonade silently evicts a model mid-stream the dispatcher now catches the upstream 502, refreshes slot state, and retries once before surfacing — turning a user-visible 502 into a transparent recovery.
Tests- δ-harness coverage of Hermes
delegate_taskfor 3 backends (Phase 0 OpenRouter prereq — DA must-fix #2). New δ-tier pytest suite attests/harness/integration/test_delegate_task_*.pyproves thedelegate_task → execution-backenddispatch hop works end-to-end for local + docker + modal with mockedBaseEnvironmentsubclasses (no Modal credits, no docker pulls in CI). The matrix test fans out one call across all three backends and asserts each was invoked exactly once with a per-backend-shaped payload. Findings catalogued attests/harness/FINDINGS.md§46 including the upstream audit (R7's "7 backends" claim corrected to 6 — local/docker/singularity/modal/daytona/ssh; Vercel Sandbox not present in upstream pin0554ef1a). Gates V3a Hermes observability peropenrouter-research-2026-05-28/PLANNING.md§3 Phase 0.
Docs- Internal docs pin + ADR-0017 + release-manifest refresh (#389).
- Operate + dashboard + installer sweep (#390): Lemonade reference page, dashboard v3 walkthrough, installer auth section gutted to match the ADR-0012 post-Caddy reality.
- PLAN §9 async-job polling contract (#387). Codifies that any
202+job_id endpoint requires UI polling of
GET /status/{id}until terminal state — the underlying pattern behind the #386 fix.
Deferred- MCP-installed-server supervisor: start / stop / restart still
return 501; installed servers report
state=stopped. Dashboard buttons disabled with tooltip pending the supervisor design. - AgentInbox / AgentOverview hero strip / Recent records pane / Skills "calls" column / per-store DB tile breakdown — adjacent hardcoded surfaces in dashboard v3 (filed as #374-#380).
- Manifest fetcher streaming + size guard, patch_config R-M-W lock, bundled-id shadow defense, dev-host worktree disk footprint (filed as #381-#384).
- Install-mode reconciliation (#406, HITL→AFK) and hal0-test-template CT 200 + clone harness (#407, AFK) — both filed against v0.4 scope.
- Per-persona spending-cap primitive (#411 — Phase 0 OpenRouter
prereq).
Hermes-and-Cognee + dashboard v3 polish release. v0.3 stream work that landed on
mainbetween 2026-05-23 and 2026-05-27 — 64 PRs — packaged into the first patch tag after the v0.3.0-alpha.1 auth/Caddy cut.Added- Hermes-Agent bootstrap pipeline (PRs #279, #284, #286, #289, #291,
#292, #295, #296, #298, #316). 12-phase pipeline (
preflight,install,home_init,env_probe,config_write,mcp_wire,namespace_register,context_link,model_automap,voice_wire,smoke_tests,self_report). Plugin model (Hal0Profile,Hal0MemoryProvider).hal0 agent {status,log,upgrade}CLI. - MCP host: per-agent client allow-list (ADR-0013 — PRs #278, #293,
#295, #300, #304).
mcp_client.py, host-introspection probe tools forhal0-admin, per-agent MCP-clients view in the dashboard, full read-only introspection + audit-log SSE on the MCP page. - Memory graph extraction (ADR-0014 — PRs #287, #290, #294, #297,
#303).
[memory.graph]schema + cognify gate on Cognee. New/api/memory/{add,search,list,delete}REST shims (closes #302).hal0 memory graph {status,enable,disable}CLI. Graph-extraction panel in dashboard Memory tab. - Agents > Peers tab (PR #299) — identity cards from agents dataset.
- Models surface (PRs #313, #319, #343, #353) — scan +
add-by-path + model-dir setting, single
[models].storesetting with firstrun + migration, default scan/preview recursive with UI toggle, model.type derived at theuseModelshook. - Chat surface in dashboard (PRs #309, #314, #315, #356, #357,
#358) — real chat against the primary slot, slot indicator dots +
warming pulse, collapsible reasoning above the answer, chat moves
to its own
/chatroute, snapshot/memmap/throughput sidebar mirrored onto/slots. - Footer journal + update banner (Epic #322 — PRs #321, #328,
#329, #330, #332).
/api/journal+/api/journal/streammerged log surface; Settings → Updates wired to the real backend. - Slot UX bundle (PRs #281, #282, #283, #342, #344, #351) — POST
normalizes Lemonade-shape model + auto-assigns port,
hal0 slot create --typederives Lemonade device, max_loaded_models 4→8, swap-arrow affordance, zero-red-dots bundle, swap popover reads live/api/models. - One-line Proxmox VE LXC installer (PR #341).
Fixed- Slot backend update now invalidates state.json (PR #360, issue
#359). Previously
POST /api/slots/{name}/backendrewrote the TOML butextra.backendin state.json stuck at the boot-time adoption value forever, so the snapshot lied even though inference itself ran on the new backend. - Dispatcher fall-through to Lemonade proxy (PR #277) and drift to OFFLINE not ERROR when lemond evicts a model (PR #276).
- Hermes uninstall — registry coherence + state-dir cleanup (#352), venv + context_link teardown (#354), memory teardown failure surfacing (#355).
/v1/health.last_usetreated as an opaque counter (PR #307); removes spurious "idle since the unix epoch" rendering.- Live sidebars + memory map + throughput (PRs #306, #308, #328) finally read the real backend instead of HAL0_DATA seed fixtures.
Changed- Bundle name rendered from manifest instead of placeholder text across install banners + progress (#214 / #331).
- MCP page moved from mock to real backend introspection (#304).
- Settings → Updates moved from mock to real backend (#321).
- UpdateBanner wired to live update state (#324 / #329).
- HAL0_DATA fixtures further retired — multiple dash surfaces now
read
/api/models(#345 / #351).
NotesThis is a patch-level tag (
0.3.0 → 0.3.1) by SemVer convention, but the scope is closer to a minor release — Hermes, memory graph, and the MCP host surface are all new user-facing systems. Future patch tags- Hermes-Agent bootstrap pipeline (PRs #279, #284, #286, #289, #291,
#292, #295, #296, #298, #316). 12-phase pipeline (
End-to-end Hermes-Agent integration lands. The 12-PR master-plan (
docs/internal/scratch/hermes-research-2026-05-28/MASTER-PLAN.md) ships as one mergeable surface: provisioner overhaul, persona TOML, hal0-cognee memory plugin, <hal0-agent@.service> template, chat WS proxy, plugin host, SidebarAgentBlock, v3 dashboard refactor, HermesChat composer/transcript, the missing endpoints (restart,skills,memory/stats), tests + docs sweep, and the upstream pin / weekly drift CI job.Decision record consolidated in ADR-0019; upstream pin process in ADR-0018.
New / improvedhermes_provisionoverhaul (#393, #396) — 12-phase orchestrator (preflight → install → env_probe → home_init → config_write → mcp_wire → context_link → namespace_register → model_automap → voice_wire → smoke_tests → self_report). Idempotent + checkpointed. Compositehal0upstream + MCP registration + system-prompt addendum + persona seed all happen during bootstrap.- hal0-cognee MemoryProvider (#394) —
src/hal0/agents/hermes/plugins/memory_cognee/wraps/api/memory/*so memory is part of the prompt (system_prompt_block), not a tool the agent has to remember to call. Locks the #317 dataset-namespace contract. hal0-agent@.servicetemplate (#395) — sandboxed systemd instance template (NoNewPrivileges,ProtectSystem=strict,ProtectHome=yes,Type=notify,WatchdogSec=60). Soft-link to lemonade (Wants=, notRequires=/BindsTo=) so the agent survives a lemonade GPU-cleanup hang. CLI shim at/usr/local/bin/hal0-agent.- Persona TOML store + endpoints (#399) —
GET/POST /api/agents/{id}/personas[/{pid}/activate]. Hot-reload nudge over JSON-RPC swaps system-prompt scope on the next turn without restart. Seeded personas:hermes,coder. - Plugin host (#397) — manifest proxy at
/api/dashboard/plugins; per-plugin static-asset surface at/dashboard-plugins/{name}/...; shadow-DOM SDK shim. Lets the v3 dashboard mount upstream Hermes plugin bundles (kanban today) inside an<AgentView>tab. - Chat WS proxy + session REST shim (#398) —
/api/agents/{id}/{events, submit,session/*}. Origin allowlist + HMAC session cookie on every WS upgrade; embed token inAuthorization: Bearer(never the query string).tool.progressserver-side coalesced at 100ms; ordering invariant (progress before complete) preserved. - SidebarAgentBlock (#400) — service/persona/approvals/skills/
memory chips +
[Open chat]button. Parameterised byagent_idso v0.4 pi-coder lights up by adding a row. - Dashboard v3 agents refactor (#401) —
<AgentView>monolith split into Composer, Transcript, Sidecar; Inbox tab dropped; Peers tab folded into Memory. - HermesChat composer + transcript (#404) — React composer (Enter submits, Shift+Enter newline); zustand transcript with WebSocket reconnect (250ms → 4s jittered backoff); inline tool-call cards.
- ADR-0018 upstream Hermes pin + weekly hermes-sdk-diff CI (#403)
—
pyproject.toml [tool.hal0.upstream-hermes]is the machine-readable pin;.github/workflows/hermes-sdk-diff.ymlopens a drift issue weekly when any tracked file changes between pin and upstream HEAD. - PR-11 sweep — tests + docs + final missing endpoints:
POST /api/agents/{id}/restart— systemctl restart wrapper for the SidebarAgentBlock service chip. Audit-logged viahal0.agents.audit. Subprocess-level timeout + spawn-failure envelopes.GET /api/agents/skills— replaces the static catalog the SidebarAgentBlock used during build-out. Returns the v0.3 catalog (hermes-core+hal0-admin+hal0-memory). Bumps ride ADR-0018 drift PRs.GET /api/agents/{id}/memory/stats— per-agent counts the sidebar memory chip renders; pulls from the in-process Cognee wrapper. Gracefulavailable=falsefallback when memory isn't configured.- δ-harness
tests/harness/integration/— full chat round-trip + persona activate round-trip against aFakeWsServermock hermes (no GGUF download required). AGENTS.md,ARCHITECTURE.md,CONTEXT.mdglossary refresh (composer, transcript, plugin host, sidecar agent block, persona TOML, hal0-cognee, hermes-sdk-diff, HMAC session cookie, X-hal0-Agent, composite hal0 upstream).- ADR-0019 consolidates the master-plan decisions.
Internal contractsX-hal0-Agent(NOT Bearer) is the identity claim on hal0-api per ADR-0012; the chat-proxy injects it on outbound hops, the browser never sees it./api/agents/{id}/*is the v0.4-ready shape — every endpoint is parameterised by agent id; v0.3 only resolves"hermes".- Bundled agents follow single-pick (ADR-0004): installing one uninstalls any other.
Known follow-up- hal0-web
public/CONTENT_BRIEF.md+src/pages/agents.astroupdate lands in a sibling PR on theHal0ai/hal0-webrepo.
Caddy and the auth surface are removed. PLAN.md v0.3 stream 4 ("Admin / auth simplification") lands as a hard cut rather than the softer "reduce/keep simplified password auth" originally planned in ADR-0001. Architecture in ADR-0012, which supersedes ADR-0001.
Breaking- Auth is gone. A fresh install is open on
0.0.0.0:8080. There is no password, no Bearer-token store, no/api/auth/*router, no first-run claim OTP, no session cookie. If hal0 is reachable from a hostile network, you must front it with an upstream reverse proxy that owns auth (Traefik / nginx / Cloudflare Tunnel; seedocs/operate/auth.mdx). - Caddy is gone. The installer no longer installs Caddy or renders
a Caddyfile. The
hal0-caddy.serviceunit is no longer shipped.uninstall.shstill tears down legacyhal0-caddy.serviceand/var/lib/hal0/.first-run.lockartifacts from older installs. --no-tlsinstall flag is gone (now the only path).HAL0_AUTH_ENABLED/HAL0_AUTH_DISABLEDenv vars are no-ops. Both are unread by any hal0 process post-upgrade.- Bearer tokens minted under v0.2.x stop working — there's no token
store to validate them against. Programmatic clients that hit
/v1/*no longer need (or are even able to use) an Authorization header.
New / improved- v3 React dashboard on
main(#235), with the deferred slot-metrics normalizer (#249) and the slot type/group inference + hardware shape normalizer (#253) that took the sparse Lemonade payloads to a rendered state. /v1/*reverse-proxy to Lemonade (#248, closes #212). hal0-api catches every un-routed/v1/{path:path}and forwards to127.0.0.1:13305. Sidebarlemondstatus chip now updates from/v1/healthinstead of permanently reading "down."- Footer chips honor backend null (#252, closes #221).
queued/coresidentrender as—when Lemonade hasn't surfaced them. - Settings → default landing tab is now "Secrets" (was "Auth"; the panel is gone).
Removed codesrc/hal0/api/auth/(4 files, 712 lines) — first-run lockfile, password hash/verify, OTP rate-limitersrc/hal0/auth/(3 files, 646 lines) — token store, password helpers,auth_enabled()src/hal0/api/middleware/auth.py(508 lines) —require_token,require_writer,require_admindeps +AuthIdentityresolversrc/hal0/api/routes/auth.py(33 KB) —/api/auth/{status,login, logout,password,me,tokens,tokens/{id}/rotate}ui/src/api/hooks/useAuth.ts(58 lines) — token reveal/rotate hooksui/src/dash/settings.jsx::AuthSection(~60 lines)tests/api/test_auth_*+tests/auth/— ~2,500 lines of test coverage for moot architecturepackaging/caddy/Caddyfile.template+packaging/systemd/hal0-caddy.service- ~135 lines of
install_caddy_tls()+--no-tlshandling ininstall.sh - ~110 lines of first-run-lockfile + OTP minting + password-claim
banner in
install.sh
Upgrade notes- An existing v0.2.x install will lose its password + tokens on the next
install.
uninstall.shcleans up the legacy Caddy unit + lockfile if you want a clean slate first. - If you were relying on
--no-tls, drop the flag — the installer no longer accepts it (and no longer needs it).
- Auth is gone. A fresh install is open on
The Lemonade Server adoption release. AMD's Lemonade Server replaces the six per-modality toolbox containers and the
hal0-slot@.servicetemplate as the unified inference runtime; onehal0-lemonade.servicesupervises a singlelemonddaemon. Architecture recorded in ADR-0008, ADR-0009, ADR-0010; locked implementation contract atdocs/internal/lemonade-adoption-plan-2026-05-22.md.Breaking- v0.1.x → v0.2 is a clean break — no auto-migration.
install.shdetects v0.1.x state (presence of/etc/hal0/slots/*.tomlAND absence of/var/lib/hal0/lemonade/config.json) and refuses to overwrite it, printing explicit backup + wipe instructions and exiting non-zero. See https://hal0.dev/docs/v0.2-upgrade for the user-facing procedure. - Per-modality toolbox containers retired.
hal0-toolbox-vulkan/rocm/flm/moonshine/kokoro/comfyuiare no longer built or pulled. Their dispatch responsibilities consolidate into Lemonade'sllamacpp/flm:npu/whisper.cpp/kokoro:cpu/sd-cpprecipes. hal0-slot@.servicesystemd template retired. Per-slot units no longer exist.hal0-lemonade.serviceis the new daemon supervisor — one process serving every slot via Lemonade's per-type LRU.- Model layout reorganised to the canonical
/var/lib/hal0/models/<recipe>/<capability>/tree. PR-7's migration script reorganises/mnt/ai-models/{local,flm-ubuntu,moonshine_voice,voices,comfyui}into the same shape with per-leaf symlinks back to the canonical path. Lemonade'sextra_models_dirpoints at the canonical tree. /etc/hal0/slots/*.tomlremoved as a persistence surface;capabilities.tomlis now the single source of truth for slot selections. The slot lifecycle state machine insrc/hal0/slots/state.pysurvives; per-slot Provider classes and the slot-systemd-template do not.- Moonshine STT retired in favour of
whisper.cppvia Lemonade. More accurate but heavier on weak CPUs; lite-tier users may notice. - ComfyUI workflows lost.
sd-cppcovers the 90% case; power users are directed to external ComfyUI installations for advanced workflow graphs. HAL0_BACKEND=lemonadeenv flag introduced in PR-8 and removed in PR-10 — Lemonade is now the unconditional runtime.
Features- Lemonade Server unified inference runtime (PR-3 #156 through
PR-22). One
lemondprocess per host on127.0.0.1:13305, cache- config at
/var/lib/hal0/lemonade/, supervised byhal0-lemonade.service.
- config at
LemonadeProvideris the onlyProviderin v0.2's dispatch path. Capability dispatcher reads/v1/healthfor slot state and routes through Lemonade's/v1/chat/completions//v1/embeddings//v1/rerank//v1/audio/*//v1/images/*endpoints.- FLM trio NPU packing (PR-19 #201, PR-20 #202). Lemonade's
flm.args = "--asr 1 --embed 1"packs chat + transcription + embedding into oneflm serveprocess sharing the single AMDXDNA hardware context. hal0 exposes three slots (agent,stt-npu,embed-npu); the capability dispatcher reads/v1/health.loaded[].backend_urlfor the FLM model and routesstt-npu/embed-npurequests directly to the child's port (Lemonade only knows about the chat role). NPU exclusivity (onedevice = "npu", type = "llm"slot enabled at a time) is enforced incapabilities.tomlvalidation; chat-model swap surfaces a "swap incoming, voice + embed paused" UX. See ADR-0009. - OmniRouter client-side tool-calling (PR-16 #189). 8 tools — 5
upstream-mirrored (
generate_image,edit_image,text_to_speech,transcribe_audio,analyze_image) + 3 hal0-custom (embed_text,rerank_documents,route_to_chat). Dynamic per-request filtering: a tool is included in the LLM prompt only if at least one enabled slot of its target type exists AND (for label-gated tools) at least one of those slots has a model with the required labels. LLMs without thetool-callinglabel receive no tools.route_to_chatis one-shot delegation, blocked at depth=1, blocked across NPU LLM slots. - First-run bundle picker (PR-17 #196, PR-18 #198).
capabilities.tomlships empty by design; the dashboard's first load renders four hardware-anchored tiers (hal0-Lite≥16 GB /Default≥32 GB /Pro≥64 GB /Max≥100 GB Strix Halo) plus the AMD-curatedLMX-Omni-52B-Halokit, with a "Skip — configure manually" path. Tiers that don't fit detected unified RAM grey out with a tooltip. Bundle manifests live at/var/lib/hal0/models/collections/omni/. The NPU trio is opt-in even at Pro and Max tiers. See ADR-0010. - Settings → Lemonade admin panel (PR-13 #183). Surfaces
/internal/configsnapshot +/internal/setatomic writes for a curated subset of keys. Guards against overridingllamacpp.argsto an unbounded value (would cause the multi-LLM CPU oversubscription deadlock). - Journal panel folded into Logs tab (PR-14 #184). Lemonade's
/logs/streamWebSocket streams into the dashboard's event ring, alongside hal0's own structured journal. - Metrics shim (PR-12 #179). Per-slot TTFT + tok/s +
prompt_tokens scraped from
/v1/stats. FLM-native KV% (kv_token_occupancy_rate_percentage) on NPU slots. See known limitations below for the GPU-slot KV% gap. [CPU]chip + tooltip on the voice slot card (PR-15 #186) disclosing that kokoro is CPU-only in v0.2. GPU TTS deferred to v0.3.- Dashboard reads
/v1/healthfor slot state (PR-11 #163); surfaces NPU exclusivity, FLM trio coresident marker, and the nuclear-evict banner via/logs/streamline parsing. - Mandatory
llamacpp.args = "--parallel 1 --threads N"in thelemondconfig baseline (PR-5 #159). N is computed at install time as(cores − 2) / 4, min 2. Without this, two concurrent child llama-servers oversubscribe the CPU and freeze the Vulkan dispatch — a hard install-time requirement, not a tunable. - Per-type LRU concurrency. Six independent type budgets
(
llm,embedding,reranking,transcription,tts,image) reported by/v1/health.max_models; default global budget set to- Nuclear evict-all only fires when a
/v1/loaderrors AND the error message does NOT substring-match "not found" / "does not exist" / "No such file" — common failure modes (bad path, missing variant, mistyped name) return graceful errors and leave the loaded pool intact.
- Nuclear evict-all only fires when a
- Slot model: bare-name identity +
type(Lemonade vocab:llm | embedding | reranking | transcription | tts | image) +device(gpu-rocm | gpu-vulkan | cpu | npu) +model+enabled- optional
default+groupfor dashboard rollup. User-added slots viahal0 slot add NAME --type TYPE --model MODEL. Exactly onedefault = trueper type enforced at save / load.
- optional
- Canonical model namespace.
registered(no prefix, fromregistry.toml→ Lemonade'sserver_models.json) vsuser.*(on-demand pulls viaPOST /v1/pull).extra.*auto-discovery unused. Dashboard surfaces two badges:blessedandpulled. hal0 registry sync(PR-6 #141 → #151) — regenerates/var/lib/hal0/lemonade/resources/server_models.jsonfromregistry.tomland restartslemond. Hourly drift detector surfaces a dashboard banner whenregistry.tomlis newer thanserver_models.json.hal0 registry import(PR-21 #203) — single command, restoresregistry.tomlfrom a v0.1.x backup tarball. Slot selections must be redone via the bundle picker.hal0 doctorextended to probelemondreachability + FLM.debpresence (Linux NPU path).
Internal- 22 implementation PRs landed across 6 sub-phases. Foundation (PR-2 #137, PR-3 #156), install + registry (PR-4 #157, PR-5 #159, PR-6 #141 → #151, PR-7 #158), slot layer rewrite (PR-8 #161, PR-9 #160, PR-10 #162), UI + metrics (PR-11 #163, PR-12 #179, PR-13 #183, PR-14 #184, PR-15 #186), OmniRouter + bundles (PR-16 #189, PR-17 #196, PR-18 #198), NPU + close-out (PR-19 #201, PR-20 #202, PR-21 #203, PR-22 — this PR).
SlotManagersimplified ~358 LOC in PR-10 (#162) — provider ABC dispatch + per-slot systemd adoption logic deleted.- Legacy provider classes preserved as code (used by image-gen / hardware-probe / catalog non-slot consumers) but no longer in the Lemonade dispatch path.
SlotConfig.devicerefactor +capabilities.tomlschema_version=2migration (#143 → #153).- Preload validation + idle-unload driver (#144 → #152) shipped
ahead of ADR-0007 supersession; preload validation removed per
ADR-0008 §3 in
e660fa3. src/hal0/lemonade/— HTTP client +catalog_sync.py+metrics_shim.py+log_proxy.py.src/hal0/omni_router/— client + tool definitions (checksum-pinned mirror of Lemonade upstream'stoolDefinitions.json; CI scriptscripts/check-tool-definitions.shfails on drift).- NPU FLM trio dispatch carve-out documented in
ADR-0009 — narrow exception to ADR-0008's "Lemonade owns
inference lifecycle" thesis; scoped to the two endpoint paths
(
/v1/audio/transcriptions,/v1/embeddings) that Lemonade doesn't know exist on the FLM child. - v0.2.1 dashboard rewrite (slice #176, PR #199) cut over on
mainin parallel; PR #197 carries v2 polish work and remains open at v0.2 ship.
Known limitations- KV% for GPU slots reads
—. Lemonade's bundledllama-server(b9253 Vulkan, b1274 ROCm) returnsnullforn_past/n_prompt_tokens/promptin/slotsresponses, even during active inference. PR #124's KV%-from-/slotsstrategy did not survive the migration. FLM/NPU slots get KV% native from thekv_token_occupancy_rate_percentagefield in/v1/chat/completionsresponses. v0.2.x patch path: hal0 builds its own llama-server and swaps vialemonade config set llamacpp.{rocm_bin,vulkan_bin}if upstream doesn't populate the fields within ~6 weeks. See ADR-0008 §Costs. - Kokoro TTS is CPU-only in v0.2. No upstream GPU-Kokoro on
Linux at v0.2 ship. UI surfaces a
[CPU]chip + tooltip on the voice slot card. GPU-accelerated TTS deferred to v0.3. - Performance: parity-to-regression vs the v0.1 hal0-Vulkan baseline (-13% to -18% on tested models in spike #1; hermes-14b at parity). Accepted in exchange for the six-toolbox-to-one-runtime maintenance collapse.
- NPU LLM swap is slow (~14s). Changing the
agentslot's chat model tears down the FLM trio (stt + embed go with it) and restartsflm serve <new-chat-model> --asr 1 --embed 1. UI surfaces "swap incoming, voice + embed paused". - FLM .deb install is manual on Linux. Lemonade's
flm:npuauto-installer is Windows-only as of v0.2. Linux install procedure is PPAlemonade-team/stable+ libxrt-npu2 + ffmpeg6- boost1.83 + fftw3 + FastFlowLM
.deb. The hal0 installer handles this end-to-end; users running off-script need thehal0_lemonade_flm_npu_installrecipe.
- boost1.83 + fftw3 + FastFlowLM
- Ongoing pin maintenance for two upstream artifacts (the
Lemonade embeddable tarball + the FastFlowLM
.deb). Each hal0 release manually bumps both pins, sha256-verifies, and CI-smokes the install + a triple-concurrency probe before tagging.
- v0.1.x → v0.2 is a clean break — no auto-migration.
learn more
Read the docs
Every guide, concept, and reference page for running hal0.