fix(hub): reconcile completed updates after polling stalls

The full-system updater runs as a detached systemd service and can finish
successfully even when the browser loses its status connection. In that
case the update log and status file correctly report REBOOT_REQUIRED, but
the Hub modal can remain on "Updating..." with its controls disabled.

There were four independent ways for the frontend to get stuck:

* update status fetches had no deadline, so a request that stayed pending
  never rejected and never advanced the existing failure counter;
* setInterval started async polls without waiting for the previous poll,
  allowing slow requests to overlap and responses to arrive out of order;
* each log chunk used textContent +=, replacing the complete and growing
  Nix build log every two seconds, which could stall browser rendering and
  was especially visible over RDP; and
* page reload, tab resume, and RDP reconnect did not reattach the modal to
  the update status persisted by the backend.

This produced a dangerous UX mismatch: the machine had a fully staged
NixOS generation and was ready to reboot, while the Hub continued telling
the user that the update was still running.

Bound status requests with AbortController, prevent overlapping polls, and
replace the endless spinner after sustained failures with an explicit
"Update status unavailable" state and Retry Status action. Reconcile state
immediately on focus, visibility, online, page startup, and before starting
a new update. Use no-store requests and render verbose logs incrementally
with a bounded visible tail while retaining the complete report in memory.
Apply the same timeout and single-flight protection to rebuild polling.

Record the exact generation produced by `nixos-rebuild boot`. The Hub now
keeps REBOOT_REQUIRED visible until that generation matches
/run/current-system, then clears the marker after reboot. For an update
started by an older updater that did not write the marker, recover the
staged generation from the final nixos-rebuild log line. The dashboard
sidebar also distinguishes update-in-progress and restart-required states.

Regression coverage verifies generation marker/log recovery, pre- versus
post-reboot detection, request timeout wiring, single-flight polling,
connection-loss UX, RDP/tab resume reconciliation, bounded log rendering,
page-reload recovery, and JavaScript syntax.

Validation:
* python3 -m unittest discover -s tests -p 'test_*.py' -v (170 passed)
* node --check app/sovran_systemsos_web/static/js/*.js
* python3 -m py_compile for changed Python modules
* git diff --check

A Nix evaluation was not available in the development sandbox; the NixOS
module should still be evaluated and built in CI or on a test machine before
release.
This commit is contained in:
naturallaw777
2026-08-18 10:31:28 -05:00
committed by naturallaw777
parent 1ccce429a5
commit 64624002bb
13 changed files with 530 additions and 52 deletions
+81
View File
@@ -0,0 +1,81 @@
"""Persistent update-state helpers for the Sovran Hub.
The full-system updater stages a NixOS generation with ``nixos-rebuild boot``.
That generation is not active until the machine reboots. These helpers let the
Hub distinguish a genuinely pending reboot from an old REBOOT_REQUIRED marker
that survived the reboot.
This module deliberately has no FastAPI or systemd dependencies so its state
reconciliation can be tested without importing the Hub server.
"""
from __future__ import annotations
import os
import re
# Nix store hashes use the lower-case Nix base32 alphabet. Keep the output
# name deliberately conservative: a system generation has no path separators.
_SYSTEM_GENERATION_RE = re.compile(
r"^/nix/store/[0-9a-z]{32}-nixos-system-[A-Za-z0-9._+\-]+$"
)
_LOG_GENERATION_RE = re.compile(
r"The new configuration is "
r"(/nix/store/[0-9a-z]{32}-nixos-system-[A-Za-z0-9._+\-]+)"
)
def _valid_generation(value: str) -> str | None:
"""Return a normalized NixOS generation path, or ``None`` if invalid."""
candidate = value.strip()
if _SYSTEM_GENERATION_RE.fullmatch(candidate):
return candidate
return None
def read_staged_generation(marker_path: str, log_path: str) -> str | None:
"""Read the generation staged by the last successful Hub update.
New updater versions write ``marker_path`` explicitly. For an update that
started with an older updater, recover the same value from the final
``nixos-rebuild`` log line. Only the tail is needed and bounding the read
avoids loading a potentially large build log during every status poll.
"""
try:
with open(marker_path, "r", encoding="utf-8") as marker:
generation = _valid_generation(marker.read())
if generation:
return generation
except OSError:
pass
try:
with open(log_path, "rb") as log:
log.seek(0, os.SEEK_END)
size = log.tell()
log.seek(max(0, size - 131_072), os.SEEK_SET)
tail = log.read().decode("utf-8", errors="replace")
except OSError:
return None
matches = list(_LOG_GENERATION_RE.finditer(tail))
if not matches:
return None
return _valid_generation(matches[-1].group(1))
def staged_generation_is_active(
marker_path: str,
log_path: str,
current_system_path: str = "/run/current-system",
) -> bool:
"""Return whether the staged update generation is now the running system."""
staged = read_staged_generation(marker_path, log_path)
if not staged:
return False
try:
current = os.path.realpath(current_system_path)
except OSError:
return False
return current == staged