chore(release): prepare v1.1.2

The module mixed the `options` keyword attribute with a bare top-level
`system.*` setting. Once a module declares `options` (or `config`), every
other top-level attribute must be a reserved module keyword — the nixpkgs
unifyModuleSyntax check rejects anything else, so every nixos-rebuild
aborted at evaluation time with:

  error: Module '.../modules/core/public-ip.nix' has an unsupported
  attribute `system'. ... move all of them (namely: system) into the
  `config' attribute.

Prefix the activation script with `config.` (equivalent to wrapping it in
`config = { ... };`) so the module evaluates again. The detector script
itself is unchanged.

Fixes: ac6c498615 (-feat(public-ip): unify public-IP detection into one privacy-first script-)

CHANGELOG.md

@@ -7,6 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [1.1.2] - 2026-08-21
+
+### Added
+- Nixpkgs update with Bisq update
+- Unify public-IP detection into one privacy-first script
+- Fix Element X discovery and harden federated calling
+- Nixpkgs update
+- Add tor-browser to the package list
+
+### Changed
+- Use pkgs.lndinit, drop local packages/lndinit
+- Update RTL to 0.15.10
+
+### Fixed
+- Move system.activationScripts under the config attribute
+- Reuse Hub external IP for LiveKit, drop egress service calls
+- Derive Restart required from boot default vs running system
+- Reconcile completed updates after polling stalls
+- Migrate i2pd SAM settings for nixpkgs 26.11
+- Order turn-setup after network-online to fix boot-time red dot
+- Restore a Zeus-scannable LND REST connect QR
+
+### Security
+- Prevent LND admin macaroon exposure in curl argv
+
+### Documentation
+- Update desktop and Sovran Hub screenshots
+[1.1.2]: https://git.sovransystems.com/Sovran_Systems/Sovran_SystemsOS/releases/tag/v1.1.2
+
+
 ## [1.1.1] - 2026-08-15
 
 ### Added
@@ -75,6 +105,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     polling failures instead of hanging indefinitely.
   - Rebuild/update scripts stream `nixos-rebuild` output into the live log
     (it was buffered until completion, making long rebuilds look frozen).
+  - The update modal now reconciles persisted update state after a page/RDP
+    reconnect, bounds every status request with a timeout, prevents overlapping
+    async polls, and reports an explicit "status unavailable" state with a
+    Retry Status action instead of spinning forever. Verbose Nix logs are
+    rendered incrementally and bounded so they cannot stall the browser UI.
+  - Successful staged updates now record the exact NixOS generation. The Hub
+    keeps showing "Restart required" until that generation is active, then
+    clears the marker after reboot (with log-based recovery for older updates).
+  - Pending-reboot state is now derived from NixOS itself — the boot default
+    (`/nix/var/nix/profiles/system`) versus the running `/run/current-system`
+    — instead of from Hub-written marker files. Updates performed from a
+    terminal or support session (which never touch the Hub's status files)
+    previously left a stale `REBOOT_REQUIRED` marker the Hub could never
+    clear, pinning the "Restart required" badge forever even after many
+    reboots. The marker self-heals to `IDLE` on the next status read whenever
+    boot default and running system agree, and recording the informational
+    `.generation` marker can no longer fail an otherwise successful update.
 
 ---
 

README.md

@@ -21,9 +21,9 @@ Lightning infrastructure, private cloud, and communications platform when you
 are ready.
 
 [Visit the Website](https://sovransystems.com) ·
-[Download the ISO](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso) ·
+[Download the ISO](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.2.iso) ·
 [Try it safely in a VM](#try-it-first-in-a-virtual-machine) ·
-[Verify the Download](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso.sha256) ·
+[Verify the Download](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.2.iso.sha256) ·
 [Build from Source](#build-from-source)
 
 <img src="assets/desktop-screenshot.webp" alt="Sovran_SystemsOS private Bitcoin desktop" width="800" />
@@ -345,8 +345,8 @@ with an imaging application such as [Balena Etcher](https://etcher.balena.io).
 
 ### 1. Download the ISO and checksum
 
-- [Download Sovran_SystemsOS-1.1.1.iso](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso)
-- [Download Sovran_SystemsOS-1.1.1.iso.sha256](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso.sha256)
+- [Download Sovran_SystemsOS-1.1.2.iso](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.2.iso)
+- [Download Sovran_SystemsOS-1.1.2.iso.sha256](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.2.iso.sha256)
 
 The download may take some time. Do not rename or modify the ISO before
 verifying it, and keep both files in the same folder.
@@ -364,16 +364,16 @@ checksum exactly.
 Open a terminal in the download folder and run:
 
 ```bash
-sha256sum --check Sovran_SystemsOS-1.1.1.iso.sha256
+sha256sum --check Sovran_SystemsOS-1.1.2.iso.sha256
 ```
 
 A successful comparison reports:
 
 ```text
-Sovran_SystemsOS-1.1.1.iso: OK
+Sovran_SystemsOS-1.1.2.iso: OK
 ```
 
-You can also run `sha256sum Sovran_SystemsOS-1.1.1.iso` and compare the output
+You can also run `sha256sum Sovran_SystemsOS-1.1.2.iso` and compare the output
 against the checksum file manually.
 
 </details>
@@ -384,11 +384,11 @@ against the checksum file manually.
 Open Terminal in the download folder and run:
 
 ```bash
-shasum -a 256 Sovran_SystemsOS-1.1.1.iso
+shasum -a 256 Sovran_SystemsOS-1.1.2.iso
 ```
 
 Compare the value shown in Terminal with the value inside
-`Sovran_SystemsOS-1.1.1.iso.sha256`.
+`Sovran_SystemsOS-1.1.2.iso.sha256`.
 
 </details>
 
@@ -398,7 +398,7 @@ Compare the value shown in Terminal with the value inside
 Open PowerShell in the download folder and run:
 
 ```powershell
-Get-FileHash .\Sovran_SystemsOS-1.1.1.iso -Algorithm SHA256
+Get-FileHash .\Sovran_SystemsOS-1.1.2.iso -Algorithm SHA256
 ```
 
 Compare the value under `Hash` with the published checksum.
@@ -413,7 +413,7 @@ match exactly.
 
 1. Download and install [Balena Etcher](https://etcher.balena.io), then
    connect the USB drive.
-2. Choose **Flash from file** and select `Sovran_SystemsOS-1.1.1.iso`.
+2. Choose **Flash from file** and select `Sovran_SystemsOS-1.1.2.iso`.
 3. Choose **Select target**, select the USB drive, and review your selection
    carefully.
 4. Choose **Flash** and wait for the writing and verification process to
@@ -856,7 +856,7 @@ primary location for collaboration. Please read our
 ## Privacy. Sovereignty. Bitcoin.
 
 [Visit Sovran Systems](https://sovransystems.com) ·
-[Download Sovran_SystemsOS](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso) ·
+[Download Sovran_SystemsOS](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.2.iso) ·
 [View the License](LICENSE)
 
 </div>

VERSION

@@ -1 +1 @@
-1.1.1
+1.1.2

app/sovran_systemsos_web/server.py

@@ -21,6 +21,7 @@ import subprocess
 import tempfile
 import threading
 import time
+import sys
 import urllib.error
 import urllib.parse
 import urllib.request
@@ -55,6 +56,7 @@ from .security_helpers import (
     load_session_store,
     save_session_store,
 )
+from .update_state import effective_update_status
 
 logger = logging.getLogger(__name__)
 
@@ -64,9 +66,10 @@ FLAKE_LOCK_PATH = "/etc/nixos/flake.lock"
 FLAKE_INPUT_NAME = "Sovran_Systems"
 GITEA_API_BASE = "https://git.sovransystems.com/api/v1/repos/Sovran_Systems/Sovran_SystemsOS/commits"
 
-UPDATE_LOG    = "/var/log/sovran-hub-update.log"
-UPDATE_STATUS = "/var/log/sovran-hub-update.status"
-UPDATE_UNIT   = "sovran-hub-update.service"
+UPDATE_LOG        = "/var/log/sovran-hub-update.log"
+UPDATE_STATUS     = "/var/log/sovran-hub-update.status"
+UPDATE_GENERATION = "/var/log/sovran-hub-update.generation"
+UPDATE_UNIT       = "sovran-hub-update.service"
 
 REBUILD_LOG    = "/var/log/sovran-hub-rebuild.log"
 REBUILD_STATUS = "/var/log/sovran-hub-rebuild.status"
@@ -132,6 +135,7 @@ _SERVICE_DOMAIN_KEYS = frozenset([
 ])
 
 INTERNAL_IP_FILE = "/var/lib/secrets/internal-ip"
+EXTERNAL_IP_FILE = "/var/lib/secrets/external-ip"
 ZEUS_CONNECT_FILE = "/var/lib/secrets/zeus-connect-url"
 
 ONBOARDING_FLAG = "/var/lib/sovran/onboarding-complete"
@@ -950,21 +954,43 @@ def _save_internal_ip(ip: str):
             pass
 
 
-def _get_external_ip() -> str:
-    MAX_IP_LENGTH = 46
-    for url in [
-        "https://api.ipify.org",
-        "https://ifconfig.me/ip",
-        "https://icanhazip.com",
-    ]:
+def _save_external_ip(ip: str):
+    """Write the external IP to a file so other services (e.g. LiveKit) can
+    reference it without running their own detection."""
+    if ip and ip != "unavailable":
         try:
-            req = urllib.request.Request(url, method="GET")
-            with urllib.request.urlopen(req, timeout=8) as resp:
-                ip = resp.read().decode().strip()
-                if ip and len(ip) < MAX_IP_LENGTH:
-                    return ip
-        except Exception:
-            continue
+            os.makedirs(os.path.dirname(EXTERNAL_IP_FILE), exist_ok=True)
+            with open(EXTERNAL_IP_FILE, "w") as f:
+                f.write(ip)
+        except OSError:
+            pass
+
+
+def _get_external_ip() -> str:
+    """Public IP via the shared detector (/var/lib/sovran/public-ip.py).
+
+    The detector owns discovery (STUN -> DNS -> opt-in HTTPS echo), caches the
+    result in /var/lib/secrets/external-ip, and contacts at most one third
+    party per refresh interval. This function only reads the cache and asks
+    the detector to refresh when it is missing or stale — it performs no
+    per-call external queries of its own.
+    """
+    try:
+        r = subprocess.run(
+            [sys.executable, "/var/lib/sovran/public-ip.py", "check"],
+            capture_output=True, text=True, timeout=20,
+        )
+        if r.returncode == 0 and r.stdout.strip():
+            return r.stdout.strip().splitlines()[0]
+    except Exception:
+        pass
+    try:
+        with open(EXTERNAL_IP_FILE) as f:
+            ip = f.read().strip()
+        if ip:
+            return ip
+    except OSError:
+        pass
     return "unavailable"
 
 
@@ -1634,15 +1660,6 @@ def _nwc_lnurl_bech32(alias: str, domain: str) -> str:
 
 # ── Update helpers (file-based, no systemctl) ────────────────────
 
-def _read_update_status() -> str:
-    """Read the status file. Returns RUNNING, SUCCESS, REBOOT_REQUIRED, FAILED, or IDLE."""
-    try:
-        with open(UPDATE_STATUS, "r") as f:
-            return f.read().strip()
-    except FileNotFoundError:
-        return "IDLE"
-
-
 def _write_update_status(status: str):
     """Write to the status file."""
     try:
@@ -1652,6 +1669,36 @@ def _write_update_status(status: str):
         pass
 
 
+def _read_update_status() -> str:
+    """Read and reconcile the persistent update status.
+
+    ``REBOOT_REQUIRED`` survives Hub/browser restarts before the reboot, but
+    it is a CLAIM about live NixOS state, not the source of truth: the boot
+    default (``/nix/var/nix/profiles/system``) versus the running
+    ``/run/current-system``.  Re-validating on every read keeps the Hub
+    correct when the system was updated from a terminal or support session
+    (which never writes Hub markers), and lets an old marker that predates
+    the reconciliation feature self-heal instead of demanding reboots
+    forever.  The stale marker file is removed once cleared.
+    """
+    try:
+        with open(UPDATE_STATUS, "r") as f:
+            status = f.read().strip()
+    except FileNotFoundError:
+        return "IDLE"
+
+    effective = effective_update_status(status)
+    if effective != status:
+        _write_update_status(effective)
+        try:
+            os.remove(UPDATE_GENERATION)
+        except OSError:
+            pass
+        return effective
+
+    return status
+
+
 def _read_log(offset: int = 0) -> tuple[str, int]:
     """Read the update log file from the given byte offset.
     Returns (new_text, new_offset)."""
@@ -3695,6 +3742,9 @@ async def api_network():
     # Keep the internal-ip file in sync for credential lookups
     _save_internal_ip(internal)
     _cached_external_ip = external
+    # Persist the external IP so other services (e.g. LiveKit) can reuse the
+    # Hub's detection instead of running their own.
+    _save_external_ip(external)
     return {"internal_ip": internal, "external_ip": external}
 
 
@@ -3832,9 +3882,15 @@ async def api_ports_health():
 @app.get("/api/updates/check")
 async def api_updates_check():
     loop = asyncio.get_event_loop()
+    status = await loop.run_in_executor(None, _read_update_status)
+    if status in {"RUNNING", "REBOOT_REQUIRED"}:
+        # Avoid a slow remote update check when there is already an operation
+        # the dashboard needs to surface.
+        return {"available": True, "status": status.lower()}
+
     available = await loop.run_in_executor(None, check_for_updates)
     # None means inconclusive (check failed) — report as available so the UI doesn't block
-    return {"available": available is not False}
+    return {"available": available is not False, "status": status.lower()}
 
 
 @app.get("/api/ping")
@@ -6181,10 +6237,19 @@ async def _startup_migrate_deprecated_features():
 
 async def _background_domain_reachability_checker():
     """Periodically curl configured domains and cache reachability results."""
+    global _cached_external_ip
     await asyncio.sleep(_DOMAIN_REACHABILITY_STARTUP_DELAY)
     consecutive_failures = 0
     while True:
         try:
+            # Keep the persisted external IP fresh (dynamic WAN IPs), so
+            # services like LiveKit can read /var/lib/secrets/external-ip.
+            loop = asyncio.get_event_loop()
+            external = await loop.run_in_executor(None, _get_external_ip)
+            if external != "unavailable":
+                _cached_external_ip = external
+                _save_external_ip(external)
+
             cfg = load_config()
             services = cfg.get("services", [])
 
@@ -6194,7 +6259,6 @@ async def _background_domain_reachability_checker():
                 if unit is not None
             }
 
-            loop = asyncio.get_event_loop()
             overrides, *_ = await loop.run_in_executor(None, _read_hub_overrides)
 
             domains_to_check: list[str] = []

app/sovran_systemsos_web/static/js/constants.js

@@ -4,11 +4,17 @@
 
 const POLL_INTERVAL_SERVICES    = 5000;
 const POLL_INTERVAL_UPDATES     = 1800000;
-const UPDATE_POLL_INTERVAL      = 2000;
-// Max consecutive failed rebuild/update status polls before the page gives up
-// waiting and reloads to re-sync (2s interval → ~2 minutes of failures).
-// A brief Hub restart during activation only causes a handful of failures.
-const STATUS_POLL_MAX_FAILURES  = 60;
+const UPDATE_POLL_INTERVAL          = 2000;
+// A pending fetch never rejects by itself. Bound every status request so a
+// wedged browser connection cannot leave the modal spinning forever.
+const STATUS_POLL_FETCH_TIMEOUT      = 15000;
+// Eight timed-out requests plus the poll interval is a little over two minutes.
+// A brief Hub restart or a heavily loaded Nix build remains well inside this.
+const STATUS_POLL_MAX_FAILURES       = 8;
+// Keep verbose Nix output from making textContent updates quadratic and
+// freezing the Hub renderer (especially noticeable over RDP).
+const UPDATE_VISIBLE_LOG_MAX_CHARS   = 250000;
+const UPDATE_VISIBLE_LOG_TRIM_CHARS  = 200000;
 const REBOOT_CHECK_INTERVAL     = 5000;
 const REBOOT_FETCH_TIMEOUT      = 12000;
 const REBOOT_REQUEST_TIMEOUT    = 4000;

app/sovran_systemsos_web/static/js/events.js

@@ -6,6 +6,16 @@
 if ($btnCloseModal) $btnCloseModal.addEventListener("click", closeUpdateModal);
 if ($btnReboot) $btnReboot.addEventListener("click", doReboot);
 if ($btnSave) $btnSave.addEventListener("click", saveErrorReport);
+if ($btnRetryUpdate) $btnRetryUpdate.addEventListener("click", retryUpdateStatus);
+
+// Browser timers and requests may be suspended while an RDP session/tab is in
+// the background. Reconcile immediately when the user returns instead of
+// waiting for the next interval.
+window.addEventListener("focus", resumeUpdateStatusAfterInterruption);
+window.addEventListener("online", resumeUpdateStatusAfterInterruption);
+document.addEventListener("visibilitychange", function() {
+  if (document.visibilityState === "visible") resumeUpdateStatusAfterInterruption();
+});
 if ($credsCloseBtn) $credsCloseBtn.addEventListener("click", closeCredsModal);
 if ($supportCloseBtn) $supportCloseBtn.addEventListener("click", closeSupportModal);
 
@@ -248,6 +258,11 @@ async function init() {
     setInterval(checkUpdates, POLL_INTERVAL_UPDATES);
     loadAutolaunchToggle();
   }
+
+  // If the page was reloaded or the RDP/browser session resumed during an
+  // update, reopen the modal from the persisted backend state. This also
+  // surfaces a completed update that is waiting for its activation reboot.
+  await restoreUpdateModalIfNeeded();
 }
 
 document.addEventListener("DOMContentLoaded", init);

app/sovran_systemsos_web/static/js/helpers.js

@@ -144,3 +144,22 @@ async function apiFetch(path, options) {
   }
   return res.json();
 }
+
+async function apiFetchWithTimeout(path, options, timeoutMs) {
+  var controller = new AbortController();
+  var fetchOptions = Object.assign({}, options || {});
+  fetchOptions.signal = controller.signal;
+  var timer = setTimeout(function() { controller.abort(); }, timeoutMs);
+  try {
+    return await apiFetch(path, fetchOptions);
+  } catch (err) {
+    if (controller.signal.aborted) {
+      var timeoutError = new Error("Request timed out");
+      timeoutError.name = "TimeoutError";
+      throw timeoutError;
+    }
+    throw err;
+  } finally {
+    clearTimeout(timer);
+  }
+}

app/sovran_systemsos_web/static/js/rebuild.js

@@ -8,6 +8,7 @@ function openRebuildModal() {
   _rebuildLogOffset = 0;
   _rebuildServerDown = false;
   _rebuildFinished = false;
+  _rebuildPollInFlight = false;
   _rebuildPollFailures = 0;
   if ($rebuildLog) { $rebuildLog.textContent = ""; $rebuildLog.style.display = "none"; }
   var action = _rebuildIsEnabling ? "Enabling" : "Disabling";
@@ -34,6 +35,7 @@ function appendRebuildLog(text) {
 }
 
 function startRebuildPoll() {
+  if (_rebuildPollTimer) clearInterval(_rebuildPollTimer);
   pollRebuildStatus();
   _rebuildPollTimer = setInterval(pollRebuildStatus, UPDATE_POLL_INTERVAL);
 }
@@ -43,9 +45,14 @@ function stopRebuildPoll() {
 }
 
 async function pollRebuildStatus() {
-  if (_rebuildFinished) return;
+  if (_rebuildFinished || _rebuildPollInFlight) return;
+  _rebuildPollInFlight = true;
   try {
-    var data = await apiFetch("/api/rebuild/status?offset=" + _rebuildLogOffset);
+    var data = await apiFetchWithTimeout(
+      "/api/rebuild/status?offset=" + _rebuildLogOffset,
+      { cache: "no-store" },
+      STATUS_POLL_FETCH_TIMEOUT
+    );
     _rebuildPollFailures = 0;
     if (_rebuildServerDown) { _rebuildServerDown = false; }
     if (data.log) appendRebuildLog(data.log);
@@ -61,10 +68,8 @@ async function pollRebuildStatus() {
   } catch (err) {
     _rebuildPollFailures += 1;
     // The Hub restarts itself during activation, which briefly drops this poll.
-    // If polling stays broken long past a normal restart, the page's session
-    // almost certainly no longer matches the server (e.g. the Hub restarted
-    // and sessions were not recovered).  Reload to re-authenticate and show
-    // the real feature state instead of spinning forever.
+    // If polling stays broken long past a normal restart, reload to
+    // re-authenticate and show the resulting feature state.
     if (_rebuildPollFailures >= STATUS_POLL_MAX_FAILURES) {
       _rebuildFinished = true;
       stopRebuildPoll();
@@ -72,6 +77,8 @@ async function pollRebuildStatus() {
       return;
     }
     if (!_rebuildServerDown) { _rebuildServerDown = true; if ($rebuildStatus) $rebuildStatus.textContent = "Applying changes…"; }
+  } finally {
+    _rebuildPollInFlight = false;
   }
 }
 

app/sovran_systemsos_web/static/js/state.js

@@ -6,9 +6,12 @@ let _servicesCache    = [];
 let _categoryLabels   = {};
 let _updateLog        = "";
 let _updatePollTimer  = null;
+let _updatePollInFlight = false;
 let _updateLogOffset  = 0;
+let _updateVisibleLogChars = 0;
 let _serverWasDown    = false;
 let _updateFinished   = false;
+let _updateStatusUnavailable = false;
 let _updatePollFailures = 0; // consecutive failed update-status polls
 let _supportTimerInt      = null;
 let _supportEnabledAt     = null;
@@ -24,6 +27,7 @@ let _featuresData         = null;
 let _rebuildLog           = "";
 let _rebuildLogOffset     = 0;
 let _rebuildPollTimer     = null;
+let _rebuildPollInFlight  = false;
 let _rebuildFinished      = false;
 let _rebuildServerDown    = false;
 let _rebuildPollFailures  = 0; // consecutive failed rebuild-status polls
@@ -48,6 +52,7 @@ const $modalStatus    = document.getElementById("modal-status");
 const $modalLog       = document.getElementById("modal-log");
 const $btnReboot      = document.getElementById("btn-reboot");
 const $btnSave        = document.getElementById("btn-save-report");
+const $btnRetryUpdate = document.getElementById("btn-retry-update-status");
 const $btnCloseModal  = document.getElementById("btn-close-modal");
 
 const $rebootOverlay  = document.getElementById("reboot-overlay");

app/sovran_systemsos_web/static/js/tiles.js

@@ -271,10 +271,19 @@ async function checkUpdates() {
   try {
     var data = await apiFetch("/api/updates/check");
     var hasUpdates = !!data.available;
+    var updateStatus = data.status || "idle";
     var sidebarUpdateBtn = document.getElementById("sidebar-btn-update");
     var sidebarUpdateHint = document.getElementById("sidebar-update-hint");
     if (sidebarUpdateBtn) {
-      if (hasUpdates) {
+      if (updateStatus === "reboot_required") {
+        sidebarUpdateBtn.style.borderColor = "#e5a50a";
+        sidebarUpdateBtn.style.backgroundColor = "rgba(229, 165, 10, 0.10)";
+        if (sidebarUpdateHint) sidebarUpdateHint.textContent = "Restart required";
+      } else if (updateStatus === "running") {
+        sidebarUpdateBtn.style.borderColor = "#3584e4";
+        sidebarUpdateBtn.style.backgroundColor = "rgba(53, 132, 228, 0.10)";
+        if (sidebarUpdateHint) sidebarUpdateHint.textContent = "Update in progress…";
+      } else if (hasUpdates) {
         sidebarUpdateBtn.style.borderColor = "#2ec27e";
         sidebarUpdateBtn.style.backgroundColor = "rgba(46, 194, 126, 0.08)";
         if (sidebarUpdateHint) sidebarUpdateHint.textContent = "Updates available!";

app/sovran_systemsos_web/static/js/update.js

@@ -2,20 +2,45 @@
 
 // ── Update modal ──────────────────────────────────────────────────
 
-function openUpdateModal() {
+async function openUpdateModal() {
   if (!$modal) return;
-  apiFetch("/api/updates/check")
+
+  // Reattach before checking for new updates. This makes a browser reload,
+  // RDP reconnect, or suspended tab recover the authoritative systemd-backed
+  // state instead of starting over or claiming the system is merely up to date.
+  try {
+    var current = await apiFetchWithTimeout(
+      "/api/updates/status?offset=0",
+      { cache: "no-store" },
+      STATUS_POLL_FETCH_TIMEOUT
+    );
+    if (current.running || current.result === "reboot_required") {
+      showExistingUpdate(current);
+      return;
+    }
+  } catch (_) {
+    // The normal start path below has its own visible error handling.
+  }
+
+  apiFetchWithTimeout(
+    "/api/updates/check",
+    { cache: "no-store" },
+    STATUS_POLL_FETCH_TIMEOUT
+  )
     .then(function(data) {
       if (!data.available) {
         stopUpdatePoll();
         _updateLog = "";
         _updateLogOffset = 0;
+        _updateVisibleLogChars = 0;
         _updateFinished = true;
+        _updateStatusUnavailable = false;
         if ($modalLog) $modalLog.textContent = "";
         if ($modalStatus) $modalStatus.textContent = "✓ System is already up to date";
         if ($modalSpinner) $modalSpinner.classList.remove("spinning");
         if ($btnReboot) $btnReboot.style.display = "none";
         if ($btnSave) $btnSave.style.display = "none";
+        if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
         if ($btnCloseModal) $btnCloseModal.disabled = false;
         $modal.classList.add("open");
         return;
@@ -27,23 +52,69 @@ function openUpdateModal() {
     });
 }
 
-function _doOpenUpdateModal() {
+function prepareUpdateModal() {
   if (!$modal) return;
+  stopUpdatePoll();
   _updateLog = "";
   _updateLogOffset = 0;
+  _updateVisibleLogChars = 0;
+  _updatePollInFlight = false;
   _serverWasDown = false;
   _updateFinished = false;
+  _updateStatusUnavailable = false;
   _updatePollFailures = 0;
   if ($modalLog) $modalLog.textContent = "";
   if ($modalStatus) $modalStatus.textContent = "Starting update…";
   if ($modalSpinner) $modalSpinner.classList.add("spinning");
   if ($btnReboot) $btnReboot.style.display = "none";
   if ($btnSave) $btnSave.style.display = "none";
+  if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
   if ($btnCloseModal) $btnCloseModal.disabled = true;
   $modal.classList.add("open");
+}
+
+function _doOpenUpdateModal() {
+  prepareUpdateModal();
   startUpdate();
 }
 
+function showExistingUpdate(data) {
+  prepareUpdateModal();
+  if (data.log) appendLog(data.log);
+  _updateLogOffset = Number(data.offset) || 0;
+
+  if (data.running) {
+    if ($modalStatus) $modalStatus.textContent = "Updating…";
+    startUpdatePoll();
+    return;
+  }
+
+  _updateFinished = true;
+  if (data.result === "reboot_required") {
+    onUpdateDone("reboot_required");
+  } else if (data.result === "success") {
+    onUpdateDone(true);
+  } else {
+    onUpdateDone(false);
+  }
+}
+
+async function restoreUpdateModalIfNeeded() {
+  if (!$modal || $modal.classList.contains("open")) return;
+  try {
+    var data = await apiFetchWithTimeout(
+      "/api/updates/status?offset=0",
+      { cache: "no-store" },
+      STATUS_POLL_FETCH_TIMEOUT
+    );
+    if (data.running || data.result === "reboot_required") {
+      showExistingUpdate(data);
+    }
+  } catch (_) {
+    // Dashboard startup must remain usable when status cannot be reached.
+  }
+}
+
 function closeUpdateModal() {
   if (!$modal) return;
   $modal.classList.remove("open");
@@ -53,21 +124,36 @@ function closeUpdateModal() {
 function appendLog(text) {
   if (!text) return;
   _updateLog += text;
-  if ($modalLog) { $modalLog.textContent += text; $modalLog.scrollTop = $modalLog.scrollHeight; }
+  if ($modalLog) {
+    // Appending a text node avoids reparsing/replacing the complete log on
+    // every two-second poll. Trim only occasionally once the visible log is
+    // large; the complete _updateLog remains available for error reports.
+    if (_updateVisibleLogChars + text.length > UPDATE_VISIBLE_LOG_MAX_CHARS) {
+      var tail = _updateLog.slice(-UPDATE_VISIBLE_LOG_TRIM_CHARS);
+      var notice = "[Earlier update output hidden from this view; it remains in the saved report.]\n\n";
+      $modalLog.textContent = notice + tail;
+      _updateVisibleLogChars = notice.length + tail.length;
+    } else {
+      $modalLog.appendChild(document.createTextNode(text));
+      _updateVisibleLogChars += text.length;
+    }
+    $modalLog.scrollTop = $modalLog.scrollHeight;
+  }
 }
 
 function startUpdate() {
-  fetch("/api/updates/run", { method: "POST" })
-    .then(function(response) {
-      if (!response.ok) return response.text().then(function(t) { throw new Error(t); });
-      return response.json();
-    })
+  apiFetchWithTimeout(
+    "/api/updates/run",
+    { method: "POST" },
+    STATUS_POLL_FETCH_TIMEOUT * 2
+  )
     .then(function(data) {
       if (data.status === "no_updates") {
         if ($modalStatus) $modalStatus.textContent = "✓ System is already up to date";
         if ($modalSpinner) $modalSpinner.classList.remove("spinning");
         if ($btnReboot) $btnReboot.style.display = "none";
         if ($btnSave) $btnSave.style.display = "none";
+        if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
         if ($btnCloseModal) $btnCloseModal.disabled = false;
         _updateFinished = true;
         return;
@@ -83,6 +169,7 @@ function startUpdate() {
 }
 
 function startUpdatePoll() {
+  if (_updatePollTimer) clearInterval(_updatePollTimer);
   pollUpdateStatus();
   _updatePollTimer = setInterval(pollUpdateStatus, UPDATE_POLL_INTERVAL);
 }
@@ -92,33 +179,45 @@ function stopUpdatePoll() {
 }
 
 async function pollUpdateStatus() {
-  if (_updateFinished) return;
+  // setInterval does not wait for an async callback. The guard prevents a slow
+  // request from creating overlapping, out-of-order status polls.
+  if (_updateFinished || _updatePollInFlight) return;
+  _updatePollInFlight = true;
   try {
-    var data = await apiFetch("/api/updates/status?offset=" + _updateLogOffset);
+    var data = await apiFetchWithTimeout(
+      "/api/updates/status?offset=" + _updateLogOffset,
+      { cache: "no-store" },
+      STATUS_POLL_FETCH_TIMEOUT
+    );
     _updatePollFailures = 0;
     if (_serverWasDown) {
       _serverWasDown = false;
       if (!data.running) {
-        // The update finished while the server was restarting.  Reset to
-        // offset 0 and re-fetch so the complete log is shown from the top.
+        // The update finished while the server or browser connection was away.
+        // Re-fetch from offset 0 so the final result and complete tail agree.
         _updateLog = "";
         _updateLogOffset = 0;
+        _updateVisibleLogChars = 0;
         if ($modalLog) $modalLog.textContent = "";
         try {
-          var fullData = await apiFetch("/api/updates/status?offset=0");
+          var fullData = await apiFetchWithTimeout(
+            "/api/updates/status?offset=0",
+            { cache: "no-store" },
+            STATUS_POLL_FETCH_TIMEOUT
+          );
           if (fullData.log) appendLog(fullData.log);
           _updateLogOffset = fullData.offset;
-        } catch (e) {
-          // If the re-fetch fails, fall through with whatever we have.
+          data = fullData;
+        } catch (_) {
           if (data.log) appendLog(data.log);
           _updateLogOffset = data.offset;
         }
         if (data.result === "reboot_required") {
-          appendLog("[Server restarted — update completed, reboot required.]\n");
+          appendLog("[Reconnected — update completed, reboot required.]\n");
         } else if (data.result === "success") {
-          appendLog("[Server restarted — update completed successfully.]\n");
+          appendLog("[Reconnected — update completed successfully.]\n");
         } else {
-          appendLog("[Server restarted — update encountered an error.]\n");
+          appendLog("[Reconnected — update encountered an error.]\n");
         }
         _updateFinished = true;
         stopUpdatePoll();
@@ -129,7 +228,7 @@ async function pollUpdateStatus() {
         }
         return;
       }
-      appendLog("[Server reconnected]\n");
+      appendLog("[Update status reconnected]\n");
       if ($modalStatus) $modalStatus.textContent = "Updating…";
     }
     if (data.log) appendLog(data.log);
@@ -146,21 +245,57 @@ async function pollUpdateStatus() {
     }
   } catch (err) {
     _updatePollFailures += 1;
-    // Same guard as the rebuild modal: if polling stays broken long past a
-    // normal Hub restart, reload to re-authenticate and show the real state
-    // instead of spinning forever.
     if (_updatePollFailures >= STATUS_POLL_MAX_FAILURES) {
-      _updateFinished = true;
-      stopUpdatePoll();
-      window.location.reload();
+      showUpdateStatusUnavailable();
       return;
     }
-    if (!_serverWasDown) { _serverWasDown = true; appendLog("\n[Server restarting — waiting for it to come back…]\n"); if ($modalStatus) $modalStatus.textContent = "Server restarting…"; }
+    if (!_serverWasDown) {
+      _serverWasDown = true;
+      appendLog("\n[Update status connection interrupted — retrying…]\n");
+      if ($modalStatus) $modalStatus.textContent = "Reconnecting to update…";
+    }
+  } finally {
+    _updatePollInFlight = false;
+  }
+}
+
+function showUpdateStatusUnavailable() {
+  _updateFinished = true;
+  _updateStatusUnavailable = true;
+  stopUpdatePoll();
+  if ($modalSpinner) $modalSpinner.classList.remove("spinning");
+  if ($modalStatus) $modalStatus.textContent = "Update status unavailable — update may still be running";
+  appendLog("\n[The Hub could not confirm update status. The background update was not stopped. Select Retry Status after reconnecting.]\n");
+  if ($btnRetryUpdate) $btnRetryUpdate.style.display = "inline-flex";
+  if ($btnCloseModal) $btnCloseModal.disabled = false;
+}
+
+function retryUpdateStatus() {
+  if (!$modal) return;
+  _updateFinished = false;
+  _updateStatusUnavailable = false;
+  _updatePollFailures = 0;
+  _serverWasDown = true;
+  if ($modalSpinner) $modalSpinner.classList.add("spinning");
+  if ($modalStatus) $modalStatus.textContent = "Reconnecting to update…";
+  if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
+  if ($btnCloseModal) $btnCloseModal.disabled = true;
+  startUpdatePoll();
+}
+
+function resumeUpdateStatusAfterInterruption() {
+  if (!$modal || !$modal.classList.contains("open")) return;
+  if (_updateStatusUnavailable) {
+    retryUpdateStatus();
+  } else if (!_updateFinished) {
+    pollUpdateStatus();
   }
 }
 
 function onUpdateDone(result) {
+  _updateStatusUnavailable = false;
   if ($modalSpinner) $modalSpinner.classList.remove("spinning");
+  if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
   if ($btnCloseModal) $btnCloseModal.disabled = false;
   if (result === true) {
     if ($modalStatus) $modalStatus.textContent = "✓ Update complete";
@@ -187,6 +322,7 @@ function saveErrorReport() {
   URL.revokeObjectURL(url);
 }
 
+
 // ── Reboot ────────────────────────────────────────────────────────
 
 var _rebootStartTime = 0;

app/sovran_systemsos_web/templates/index.html

@@ -74,6 +74,7 @@
       <div class="modal-log" id="modal-log" aria-live="polite"></div>
       <div class="modal-footer">
         <button class="btn btn-save" id="btn-save-report" style="display:none">Save Error Report</button>
+        <button class="btn btn-save" id="btn-retry-update-status" style="display:none">Retry Status</button>
         <button class="btn btn-reboot" id="btn-reboot" style="display:none">Restart Entire System</button>
         <button class="btn btn-close-modal" id="btn-close-modal" disabled>Close</button>
       </div>

app/sovran_systemsos_web/update_state.py

@@ -0,0 +1,88 @@
+"""Update-state helpers for the Sovran Hub.
+
+The full-system updater stages a NixOS generation with ``nixos-rebuild boot``.
+That generation is not active until the machine reboots — and the same is true
+for updates started from a terminal or an SSH support session, which never go
+near the Hub's status files.
+
+The ONLY reliable indicator that a reboot is pending is NixOS itself: the
+system profile (``/nix/var/nix/profiles/system``), which ``nixos-rebuild``
+points at the newest generation on every ``boot`` AND every ``switch``, versus
+``/run/current-system``, the generation actually running since the last boot.
+When the two differ, a staged generation has not been booted yet.
+
+Earlier revisions reconstructed this from a marker file and log tails written
+by the Hub's own updater.  Any system updated by other means — or whose
+``REBOOT_REQUIRED`` status was written by an updater older than the marker
+feature — left the Hub showing "Restart required" forever: the recorded
+generation could never equal the (since advanced) running one, so the marker
+could never be cleared.
+
+This module has no FastAPI or systemd dependencies so the policy can be tested
+without importing the Hub server.
+"""
+
+from __future__ import annotations
+
+import os
+
+# The NixOS system profile.  ``nixos-rebuild boot`` and ``nixos-rebuild
+# switch`` both add a generation here; ``boot`` additionally makes it the
+# bootloader default.  The path is a symlink chain (``system`` ->
+# ``system-N-link`` -> ``/nix/store/...-nixos-system-...``).
+BOOT_PROFILE_PATH = "/nix/var/nix/profiles/system"
+CURRENT_SYSTEM_PATH = "/run/current-system"
+
+
+def reboot_is_pending(
+    boot_profile_path: str = BOOT_PROFILE_PATH,
+    current_system_path: str = CURRENT_SYSTEM_PATH,
+) -> bool:
+    """Return whether a staged NixOS generation has not been booted yet.
+
+    This is deliberately independent of how the update was started — Hub
+    "Update System", terminal ``nixos-rebuild boot``, or a support session all
+    move the system profile the same way:
+
+    * after ``nixos-rebuild boot``:  profile -> new, current -> old  → pending
+    * after rebooting:               both -> new                     → cleared
+    * after ``nixos-rebuild switch``: both move together             → no reboot
+      ever needed (switch activates immediately)
+    * after a rollback:              both point at the rollback target → cleared
+
+    Unreadable or missing paths are treated as "not pending": the Hub must
+    never demand a reboot it cannot substantiate.
+    """
+    try:
+        boot_default = os.path.realpath(boot_profile_path)
+        current = os.path.realpath(current_system_path)
+    except OSError:
+        return False
+    if not os.path.exists(boot_default) or not os.path.exists(current):
+        return False
+    return boot_default != current
+
+
+def effective_update_status(
+    status: str,
+    boot_profile_path: str = BOOT_PROFILE_PATH,
+    current_system_path: str = CURRENT_SYSTEM_PATH,
+) -> str:
+    """Map a persisted Hub status to the one that reflects live NixOS state.
+
+    Only ``REBOOT_REQUIRED`` is re-validated: it means "the update staged a
+    generation the machine has not booted into", a claim that must stay true
+    no matter which tool performed the last update.  When the boot default IS
+    the running system the claim is stale — the staged generation booted, was
+    superseded by a newer update, or the marker was written by an updater that
+    could never clear it — so the effective status is ``IDLE``.
+
+    All other statuses (``RUNNING``, ``FAILED``, ``SUCCESS``, ``IDLE``) pass
+    through unchanged; RUNNING staleness is handled separately against the
+    systemd unit itself.
+    """
+    if status == "REBOOT_REQUIRED" and not reboot_is_pending(
+        boot_profile_path, current_system_path
+    ):
+        return "IDLE"
+    return status

app/sovran_systemsos_web/versions.json

@@ -5,7 +5,7 @@
   "bitcoind.service": "27.1.0",
   "electrs.service": "0.10.6",
   "lnd.service": "0.18.0",
-  "rtl.service": "0.15.8",
+  "rtl.service": "0.15.10",
   "btcpayserver.service": "2.4.2",
   "albyhub.service": "1.8.0",
   "mempool.service": "3.2.1",

flake.lock

@@ -5,11 +5,11 @@
         "nixpkgs": "nixpkgs"
       },
       "locked": {
-        "lastModified": 1785778881,
-        "narHash": "sha256-yHJJTm3o7ZhiVp68G+hh+dz1GKV+ACOT6mQj+eYvJLQ=",
+        "lastModified": 1787246616,
+        "narHash": "sha256-TTbXIBwoaPbzk2o+rGEayqOvrXqKo9JdYJAMDwqjCQ4=",
         "owner": "emmanuelrosa",
         "repo": "btc-clients-nix",
-        "rev": "fc1aca94d839f82e7501fe0d01279d5c1fba6e63",
+        "rev": "8b10c40cb13bac5d100ae1d1fb42eccc0d9c3223",
         "type": "github"
       },
       "original": {
@@ -26,11 +26,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1782949081,
-        "narHash": "sha256-vp6Y/Grm98ESt6ceOkWiHWyZRDV3J1RID4w+6NWK9yA=",
+        "lastModified": 1785627969,
+        "narHash": "sha256-4dtXQk/NMePegK/nWp5NSeuZKLATItOq61lpEvmXqGw=",
         "owner": "hercules-ci",
         "repo": "flake-parts",
-        "rev": "17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e",
+        "rev": "427bf4bd9435fdf21321c8cc628c24efc14c0f7a",
         "type": "github"
       },
       "original": {
@@ -56,11 +56,11 @@
     },
     "nixpkgs-stable": {
       "locked": {
-        "lastModified": 1786201459,
-        "narHash": "sha256-CiOTEjmwAmG2AWnaIno9YaCJJmpca2FXPhMAsnrolCg=",
+        "lastModified": 1787101114,
+        "narHash": "sha256-gwrPcFf/rDjHPaVflbDZ040ZDmBTRj/7+s8ZmE2SaIM=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "8b8c811c7c2541c30382c5de7ed26be055569c60",
+        "rev": "b18a4b905f8d028dc4476412e6d6891728695379",
         "type": "github"
       },
       "original": {
@@ -72,11 +72,11 @@
     },
     "nixpkgs_2": {
       "locked": {
-        "lastModified": 1786106723,
-        "narHash": "sha256-zDSUbpoeo/9ZmD2+wXnzxoo1+uhL8vxc0b8yuYMKYq0=",
+        "lastModified": 1787135253,
+        "narHash": "sha256-RD2kNWCG+Bjo6h+JVjWVNntZs2GtRoeY2xHjts/FNkA=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "f13ff45afd1bb73e640eaa08a7066dbed07e3238",
+        "rev": "ffb3c9b700e759be2ef13237c9d8f953b32a1e46",
         "type": "github"
       },
       "original": {
@@ -88,11 +88,11 @@
     },
     "nixpkgs_3": {
       "locked": {
-        "lastModified": 1784555310,
-        "narHash": "sha256-/FCliTPgiuV1owejZFNx3Ch9irdvkOfOFl+HHZ+DrtM=",
+        "lastModified": 1787111413,
+        "narHash": "sha256-sFosWtq21eHGJRnTc/hvf4M1obRgLEUMNm/IzllkHMA=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "421eebfd0ec7bccd4abe826ce62d7e6e83129493",
+        "rev": "afe3d8ac4395617bdcdac9f188ac8717a062e014",
         "type": "github"
       },
       "original": {
@@ -109,11 +109,11 @@
         "systems": "systems"
       },
       "locked": {
-        "lastModified": 1785763201,
-        "narHash": "sha256-wA373y/B9orM3HatLu9oS+Ke5lmdZyBl/bdjd2gLMq4=",
+        "lastModified": 1787151631,
+        "narHash": "sha256-EblMdrDFBFNNlUPm5zUQdh0j6gDx+OFQPv7LFE4B5AA=",
         "owner": "nix-community",
         "repo": "nixvim",
-        "rev": "c7be49306b23a952c0151cf4bbeacd944ed82f2a",
+        "rev": "d0d62a2b5027da689b4e8d5ee43f1cf83f2e975d",
         "type": "github"
       },
       "original": {
@@ -131,6 +131,7 @@
       }
     },
     "systems": {
+      "flake": false,
       "locked": {
         "lastModified": 1774449309,
         "narHash": "sha256-brhZ8DmuGtzkCYHJg4HEd602amKm89Y9ytsFZ5uWD1w=",

modules/bitcoin/bitcoind.nix

@@ -284,7 +284,9 @@ let
   nbLib = config.nix-bitcoin.lib;
   secretsDir = config.nix-bitcoin.secretsDir;
 
-  i2pSAM = config.services.i2pd.proto.sam;
+  # nixpkgs 26.11 moved i2pd's protocol configuration from
+  # `services.i2pd.proto` to the RFC42-style `services.i2pd.settings`.
+  i2pSAM = config.services.i2pd.settings.sam;
 
   configFile = builtins.toFile "bitcoin.conf" ''
     # We're already logging via journald
@@ -374,7 +376,11 @@ in {
 
     services.i2pd = mkIf (cfg.i2p != false) {
       enable = true;
-      proto.sam.enable = true;
+      settings.sam = {
+        enabled = true;
+        address = "127.0.0.1";
+        port = 7656;
+      };
     };
 
     systemd.tmpfiles.rules = [

modules/bitcoin/lnd.nix

@@ -150,7 +150,7 @@ let
   nbLib = config.nix-bitcoin.lib;
   secretsDir = config.nix-bitcoin.secretsDir;
   runAsUser = config.nix-bitcoin.runAsUserCmd;
-  lndinit = "${(pkgs.callPackage ../../packages/lndinit {})}/bin/lndinit";
+  lndinit = "${pkgs.lndinit}/bin/lndinit";
 
   bitcoind = config.services.bitcoind;
 

modules/core/njalla.nix

@@ -33,7 +33,7 @@
       # /var/lib/njalla/ddns_urls.json.
       NoNewPrivileges    = true;
       ProtectSystem      = "strict";
-      ReadWritePaths     = [ "/var/lib/njalla" ];
+      ReadWritePaths     = [ "/var/lib/njalla" "/var/lib/secrets" ];
       ReadOnlyPaths      = [ "/etc/sovran" ];
       ProtectHome        = true;
       PrivateTmp         = true;
@@ -88,12 +88,15 @@ try:
 except Exception:
     sys.exit(0)  # no URLs configured — nothing to do
 
-# Resolve current public IP once
+# Resolve current public IP via the shared detector — one script, one cache
+# (STUN -> DNS -> opt-in HTTPS echo; see /var/lib/sovran/public-ip.py).
+# The detector refreshes /var/lib/secrets/external-ip, which the Hub and
+# LiveKit read as well, so the whole system shares a single detected value.
 public_ip = ""
 try:
     r = subprocess.run(
-        ["dig", "@resolver4.opendns.com", "myip.opendns.com", "+short", "-4"],
-        capture_output=True, text=True, timeout=10,
+        [sys.executable, "/var/lib/sovran/public-ip.py", "check"],
+        capture_output=True, text=True, timeout=20,
     )
     raw = r.stdout.strip().splitlines()[0] if r.stdout.strip() else ""
     ipaddress.ip_address(raw)  # validates — raises if not a real IP
@@ -101,6 +104,16 @@ try:
 except Exception:
     pass
 
+if not public_ip:
+    # Last resort: the shared cache file, if the detector is unavailable.
+    try:
+        with open("/var/lib/secrets/external-ip") as f:
+            raw = f.read().strip()
+        ipaddress.ip_address(raw)
+        public_ip = raw
+    except Exception:
+        pass
+
 if not public_ip:
     sys.exit(0)  # no IP resolved — skip to avoid sending bare ''${IP}
 

modules/core/public-ip.nix

@@ -0,0 +1,323 @@
+# ── Unified public-IP detection (privacy-first) ─────────────────────────────
+#
+# One script, one cache file, every consumer on the system reads the same
+# value. Previously the public IP was detected independently in three places,
+# each phoning home to a different third party:
+#   * the Hub (server.py _get_external_ip) → api.ipify.org / ifconfig.me /
+#     icanhazip.com over HTTPS on every /api/network call and every
+#     background-loop tick
+#   * DDNS (ddns-update.py) → myip.opendns.com via OpenDNS
+#   * LiveKit → STUN (its own embedded detection)
+#
+# This module replaces all of that with a single script
+# (/var/lib/sovran/public-ip.py) that detects the IP once per TTL using the
+# least-exposing mechanism available, and caches it in
+# /var/lib/secrets/external-ip. Consumers (Hub, DDNS, LiveKit) read the cache
+# and only invoke the script when it is missing or stale.
+#
+# Detection chain (first success wins, stops immediately):
+#   1. pin         — sovran_systemsOS.elementCalling.externalIP (baked in)
+#   2. cache       — /var/lib/secrets/external-ip if newer than cacheTTL
+#   3. STUN        — UDP binding request (one packet, no application data,
+#                    no HTTP metadata; the same protocol every WebRTC client
+#                    uses). Server configurable via publicIP.stunServer.
+#   4. DNS         — "myip.opendns.com" A query via publicIP.dnsResolver
+#                    (single DNS query, no HTTP headers)
+#   5. HTTPS echo  — ONLY endpoints listed in publicIP.httpsEcho (empty by
+#                    default → never contacted)
+#
+# Privacy property: while the cache is fresh, zero third parties are
+# contacted. When detection runs, at most ONE party learns the IP per
+# refresh interval (default 5 minutes), and the STUN/DNS mechanisms expose
+# nothing beyond the bare address.
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}:
+
+let
+  stunServer  = config.sovran_systemsOS.publicIP.stunServer;
+  stunPort    = config.sovran_systemsOS.publicIP.stunPort;
+  dnsResolver = config.sovran_systemsOS.publicIP.dnsResolver;
+  httpsEcho   = config.sovran_systemsOS.publicIP.httpsEcho;
+  cacheTTL    = config.sovran_systemsOS.publicIP.cacheTTL;
+
+  # Optional pin shared with element-calling (baked in at build time).
+  pin = if config.sovran_systemsOS.elementCalling.externalIP != null then config.sovran_systemsOS.elementCalling.externalIP else "";
+
+  echoList = lib.concatStringsSep "," (map (u: "'${u}'") httpsEcho);
+in
+{
+  options.sovran_systemsOS.publicIP = {
+    stunServer = lib.mkOption {
+      type = lib.types.str;
+      default = "stun.l.google.com";
+      description = ''
+        STUN server used to discover the public IP over UDP. STUN is the most
+        privacy-preserving detection mechanism: a single stateless packet,
+        no HTTP metadata. Only used when the cache is stale.
+      '';
+    };
+    stunPort = lib.mkOption {
+      type = lib.types.port;
+      default = 19302;
+    };
+    dnsResolver = lib.mkOption {
+      type = lib.types.str;
+      default = "resolver4.opendns.com";
+      description = ''
+        DNS resolver used as fallback (myip.opendns.com trick) when STUN is
+        unavailable (e.g. ISP blocks UDP egress). A single DNS query, no
+        HTTP headers.
+      '';
+    };
+    httpsEcho = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ ];
+      example = [ "https://api.ipify.org" ];
+      description = ''
+        OPT-IN HTTPS endpoints that return the caller's public IP as a bare
+        IPv4 literal. Each listed endpoint observes this server's public IP
+        and HTTP metadata every time detection runs. Empty by default — no
+        HTTPS echo service is ever contacted unless you add one here. This is
+        the last-resort fallback after STUN and DNS.
+      '';
+    };
+    cacheTTL = lib.mkOption {
+      type = lib.types.int;
+      default = 300;
+      description = "Seconds the detected public IP is cached before re-detection.";
+    };
+  };
+
+  # ── Install the unified detector ──────────────────────────────────────────
+  # This module declares `options` above, so ALL configuration must go under
+  # the `config` attribute: NixOS forbids mixing bare top-level settings
+  # (like `system.*`) with the `options`/`config` keyword attributes in the
+  # same module. (Fixes: "Module ... has an unsupported attribute `system'".)
+  config.system.activationScripts.sovranPublicIpInstall = lib.stringAfter [ "users" ] ''
+    install -d -m 0755 /var/lib/sovran
+    cat > /var/lib/sovran/public-ip.py <<'PYEOF'
+#!/usr/bin/env python3
+"""sovran-public-ip — one detector, one cache, every consumer reads the same IP.
+
+Privacy-first detection chain (first success wins):
+  1. pin     — baked in from sovran_systemsOS.elementCalling.externalIP
+  2. cache   — /var/lib/secrets/external-ip if newer than CACHE_TTL seconds
+  3. STUN    — UDP binding request (one packet, no application data)
+  4. DNS     — myip.opendns.com A query via the configured resolver
+  5. HTTPS   — ONLY endpoints baked in from publicIP.httpsEcho (opt-in)
+
+Usage:
+  public-ip.py check     print current public IP (cache first; refresh if stale)
+  public-ip.py refresh   force re-detection, update the cache file, print IP
+
+Exit status: 0 with the IP on stdout on success; 1 if no IP is available
+(cached value, if any, is still printed to stdout with a warning on stderr).
+"""
+import ipaddress
+import os
+import random
+import socket
+import struct
+import sys
+import time
+import urllib.request
+
+CACHE_FILE = "/var/lib/secrets/external-ip"
+PIN        = "${pin}"
+STUN_SERVER = "${stunServer}"
+STUN_PORT   = ${toString stunPort}
+DNS_RESOLVER = "${dnsResolver}"
+DNS_HOST     = "myip.opendns.com"
+ECHO_URLS    = [ ${echoList} ]
+CACHE_TTL    = ${toString cacheTTL}
+TIMEOUT      = 3.0
+
+# ---------------------------------------------------------------------------
+# Detection primitives
+# ---------------------------------------------------------------------------
+
+def is_usable_ip(text: str) -> bool:
+    """True if text is a globally routable IPv4 that LiveKit may advertise."""
+    try:
+        ip = ipaddress.ip_address(text)
+    except ValueError:
+        return False
+    if ip.version != 4:
+        return False
+    if (ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_multicast
+            or ip.is_reserved or ip.is_unspecified or not ip.is_global):
+        return False
+    # RFC 6598 shared (CGNAT) space — not reachable from the internet.
+    if ip in ipaddress.ip_network("100.64.0.0/10"):
+        return False
+    return True
+
+
+def stun_public_ip() -> str | None:
+    """RFC 5389 Binding request over UDP; returns the mapped (public) IPv4."""
+    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    sock.settimeout(TIMEOUT)
+    try:
+        txid = random.randbytes(12)
+        req = struct.pack("!HHI", 0x0001, 0, 0) + txid  # Binding request
+        sock.sendto(req, (STUN_SERVER, STUN_PORT))
+        data, _ = sock.recvfrom(2048)
+    except OSError:
+        return None
+    finally:
+        sock.close()
+
+    if len(data) < 20:
+        return None
+    mtype, _mlen = struct.unpack("!HH", data[:4])
+    if mtype != 0x0101:  # Binding success response
+        return None
+
+    cookie = data[4:8]
+    i = 20
+    while i + 4 <= len(data):
+        atype, alen = struct.unpack("!HH", data[i : i + 4])
+        aval = data[i + 4 : i + 4 + alen]
+        if atype in (0x0001, 0x0020) and len(aval) >= 8:  # MAPPED / XOR-MAPPED
+            family = aval[1]
+            if family == 0x01:  # IPv4
+                raw = aval[4:8]
+                if atype == 0x0020:  # XOR with magic cookie + txid prefix
+                    raw = bytes(b ^ c for b, c in zip(raw, cookie + txid[:4]))
+                return socket.inet_ntop(socket.AF_INET, raw)
+        i += 4 + ((alen + 3) // 4) * 4
+    return None
+
+
+def dns_public_ip() -> str | None:
+    """Minimal DNS A query for myip.opendns.com against the given resolver."""
+    qid = random.randint(0, 0xFFFF)
+    qname = b"".join(bytes([len(p)]) + p.encode() for p in DNS_HOST.split(".")) + b"\x00"
+    query = struct.pack("!HHHHHH", qid, 0x0100, 1, 0, 0, 0) + qname + struct.pack("!HH", 1, 1)
+    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    sock.settimeout(TIMEOUT)
+    try:
+        sock.sendto(query, (DNS_RESOLVER, 53))
+        data, _ = sock.recvfrom(4096)
+    except OSError:
+        return None
+    finally:
+        sock.close()
+
+    try:
+        if len(data) < 12:
+            return None
+        rid, _flags, _qd, an, _ns, _ar = struct.unpack("!HHHHHH", data[:12])
+        if rid != qid or an == 0:
+            return None
+        i = 12
+        for _ in range(_qd):  # skip question
+            while data[i] != 0:
+                i += 1 + data[i]
+            i += 5
+        for _ in range(an):
+            if data[i] & 0xC0 == 0xC0:
+                i += 2
+            else:
+                while data[i] != 0:
+                    i += 1 + data[i]
+                i += 1
+            rtype, _rclass, _ttl, rdlen = struct.unpack("!HHIH", data[i : i + 10])
+            i += 10
+            if rtype == 1 and rdlen == 4:
+                return socket.inet_ntop(socket.AF_INET, data[i : i + 4])
+            i += rdlen
+    except (IndexError, struct.error):
+        return None
+    return None
+
+
+def echo_public_ip() -> str | None:
+    """Opt-in HTTPS echo endpoints (baked in at build time; empty by default)."""
+    for url in ECHO_URLS:
+        try:
+            req = urllib.request.Request(url, headers={"User-Agent": "sovran-public-ip"})
+            with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
+                text = resp.read().decode().strip()
+            if is_usable_ip(text):
+                return text
+        except Exception:
+            continue
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Cache handling
+# ---------------------------------------------------------------------------
+
+def read_cache() -> str:
+    try:
+        with open(CACHE_FILE) as f:
+            return f.read().strip()
+    except OSError:
+        return ""
+
+
+def write_cache(ip: str) -> None:
+    try:
+        os.makedirs(os.path.dirname(CACHE_FILE), exist_ok=True)
+        tmp = f"{CACHE_FILE}.tmp"
+        with open(tmp, "w") as f:
+            f.write(ip + "\n")
+        os.replace(tmp, CACHE_FILE)
+    except OSError:
+        pass
+
+
+def cache_fresh() -> bool:
+    try:
+        return time.time() - os.path.getmtime(CACHE_FILE) < CACHE_TTL
+    except OSError:
+        return False
+
+
+def detect() -> str:
+    """Run the chain; returns usable IP or an empty string."""
+    if PIN and is_usable_ip(PIN):
+        return PIN
+    for fn in (stun_public_ip, dns_public_ip, echo_public_ip):
+        try:
+            cand = fn()
+        except Exception:
+            continue
+        if cand and is_usable_ip(cand):
+            return cand
+    return ""
+
+
+def main() -> int:
+    force = len(sys.argv) > 1 and sys.argv[1] == "refresh"
+    ip = ""
+    if not force and cache_fresh():
+        ip = read_cache()
+    if not ip:
+        ip = detect()
+        if ip:
+            write_cache(ip)
+        else:
+            stale = read_cache()
+            if stale:
+                print(stale)
+                print("WARNING: detection failed; using last known public IP", file=sys.stderr)
+                return 0
+            print("ERROR: could not determine a public IP (STUN/DNS unreachable)", file=sys.stderr)
+            return 1
+    print(ip)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
+PYEOF
+    chmod 0555 /var/lib/sovran/public-ip.py
+  '';
+}

modules/core/roles.nix

@@ -79,6 +79,41 @@
       };
     };
 
+    # ── Element Calling (video/audio) tuning ──────────────────
+    elementCalling = {
+      fullAccessHomeservers = lib.mkOption {
+        type = lib.types.listOf lib.types.str;
+        default = [ ];
+        example = [ "matrix.peer.example.com" ];
+        description = ''
+          Additional Matrix server_names (beyond this server itself) that may
+          trigger LiveKit room creation on this server's SFU via lk-jwt-service.
+
+          Not needed for the common federated setup: each participant's client
+          always obtains its token from its own homeserver's JWT service and
+          publishes to its own SFU, and the participant who starts a call
+          creates the room on their own SFU — the remote user merely joins
+          (joining does not require full access).
+
+          Only set this for asymmetric cases: e.g. a peer homeserver that has
+          no focus of its own, or calls whose first participant lands on this
+          server's SFU but belongs to the peer.
+        '';
+      };
+      externalIP = lib.mkOption {
+        type = lib.types.nullOr lib.types.str;
+        default = null;
+        example = "203.0.113.10";
+        description = ''
+          Optional pin: force LiveKit to advertise this public IPv4 in its
+          host/TURN ICE candidates. Not required in normal operation — the
+          module auto-detects the public IP at runtime (HTTPS egress
+          detection, falling back to STUN). Set it only to override a
+          mis-detected address (e.g. multi-WAN/VPN setups).
+        '';
+      };
+    };
+
     # ── Domain setup registry ─────────────────────────────────
     domainRequirements = lib.mkOption {
       type = lib.types.listOf (lib.types.submodule {

modules/core/sovran-hub.nix

@@ -136,9 +136,9 @@ let
     "electrs.service" = if pkgs ? electrs then pkgs.electrs.version else "0.10.6";
     "lnd.service" = if pkgs ? lnd then pkgs.lnd.version else "0.18.0";
     # Keep the fallbacks aligned with the vendored packages used by the
-    # service modules (RTL 0.15.8 and Mempool 3.2.1).  The nixpkgs attrs are
+    # service modules (RTL 0.15.10 and Mempool 3.2.1).  The nixpkgs attrs are
     # optional because these packages are built locally in this repository.
-    "rtl.service" = if pkgs ? clightning-rtl then pkgs.clightning-rtl.version else (if pkgs ? rtl then pkgs.rtl.version else "0.15.8");
+    "rtl.service" = if pkgs ? clightning-rtl then pkgs.clightning-rtl.version else (if pkgs ? rtl then pkgs.rtl.version else "0.15.10");
     # BTCPay Server is intentionally sourced from pkgs.stable by the service
     # module.  Read the configured package here rather than pkgs.btcpayserver
     # (unstable), otherwise the Hub can advertise a version that is not running.
@@ -160,8 +160,10 @@ let
 
     LOG="/var/log/sovran-hub-update.log"
     STATUS="/var/log/sovran-hub-update.status"
+    GENERATION="/var/log/sovran-hub-update.generation"
 
     echo "RUNNING" > "$STATUS"
+    rm -f "$GENERATION"
     : > "$LOG"
     exec > >(tee -a "$LOG") 2>&1
 
@@ -196,6 +198,12 @@ let
       if [ "$BOOT_RC" -ne 0 ]; then
         echo "[ERROR] nixos-rebuild boot failed"
         RC=1
+      elif ! readlink -f /nix/var/nix/profiles/system > "$GENERATION"; then
+        # The marker is informational only.  The Hub derives pending-reboot
+        # state from the NixOS system profile itself, so failing to record
+        # the marker must not fail an otherwise successful update.
+        echo "[WARNING] update succeeded but its staged generation could not be recorded"
+        rm -f "$GENERATION"
       fi
       echo ""
     fi

modules/element-calling.nix

@@ -136,9 +136,10 @@ EOF
     unitConfig = {
       ConditionPathExists = "/var/lib/domains/element-calling";
     };
-    path = [ pkgs.coreutils pkgs.findutils pkgs.iproute2 pkgs.gawk ];
+    path = [ pkgs.coreutils pkgs.findutils pkgs.iproute2 pkgs.gawk pkgs.python3 ];
     script = ''
       MATRIX=$(cat /var/lib/domains/matrix)
+      ELEMENT_CALLING=$(cat /var/lib/domains/element-calling)
 
       mkdir -p /run/livekit
 
@@ -188,7 +189,57 @@ EOF
       # rtc.interfaces.includes are only known at runtime, so they are
       # substituted here. The cert/key paths point at the LoadCredential-staged
       # copies under /run/credentials.
-      cat > /run/livekit/livekit.yaml <<EOF
+      #
+      # Determine the public IPv4 to advertise in LiveKit ICE candidates.
+      # Remote peers must be able to reach this address, so it must be the
+      # server's public IP — or the router's WAN IP when the server is behind
+      # NAT with port-forwarding. It does not need to be assigned to this box,
+      # and it may be dynamic.
+      #
+      # Reuse the shared detector (/var/lib/sovran/public-ip.py — see
+      # modules/core/public-ip.nix) instead of running our own: one script,
+      # one cache, privacy-first (STUN -> DNS -> opt-in HTTPS echo). Priority:
+      #   1. sovran_systemsOS.elementCalling.externalIP (explicit pin, if set)
+      #   2. /var/lib/secrets/external-ip (the shared cache)
+      #   3. run the detector now (it refreshes the cache)
+      #   4. STUN auto-detection (use_external_ip) as the fallback, with a
+      #      warning — this is where broken installs used to silently end up
+      #      advertising a private IP, causing "call connects but no video".
+      EXTERNAL_IP='${if config.sovran_systemsOS.elementCalling.externalIP != null then config.sovran_systemsOS.elementCalling.externalIP else ""}'
+
+      PUBLIC_IP="$EXTERNAL_IP"
+      if [ -z "$PUBLIC_IP" ] && [ -f /var/lib/secrets/external-ip ]; then
+        PUBLIC_IP=$(tr -d '[:space:]' < /var/lib/secrets/external-ip 2>/dev/null)
+      fi
+      if [ -z "$PUBLIC_IP" ] && [ -x /var/lib/sovran/public-ip.py ]; then
+        PUBLIC_IP=$(python3 /var/lib/sovran/public-ip.py check 2>/dev/null | head -n1)
+      fi
+
+      # Reject non-routable addresses (loopback, private, link-local, CGNAT).
+      # A detected/pinned address like this must never be advertised.
+      if [ -n "$PUBLIC_IP" ] && printf '%s' "$PUBLIC_IP" | grep -qE \
+        '^(0\.|127\.|10\.|100\.64\.|169\.254\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)'; then
+        echo "WARNING: external IP '$PUBLIC_IP' is not routable; falling back to STUN auto-detection." >&2
+        PUBLIC_IP=""
+      fi
+
+      if [ -n "$PUBLIC_IP" ]; then
+        cat > /run/livekit/livekit.yaml <<EOF
+port: 7880
+rtc:
+  use_external_ip: false
+  node_ip: $PUBLIC_IP
+  tcp_port: 7881
+  udp_port: 7882
+  port_range_start: 30000
+  port_range_end: 40000
+  interfaces:
+    includes:
+      - $IFACE
+EOF
+        echo "LiveKit will advertise public IP: $PUBLIC_IP"
+      else
+        cat > /run/livekit/livekit.yaml <<EOF
 port: 7880
 rtc:
   use_external_ip: true
@@ -200,6 +251,20 @@ rtc:
   interfaces:
     includes:
       - $IFACE
+EOF
+        echo "WARNING: could not determine a public IP for LiveKit; using STUN auto-detection. If calls connect without media, check STUN egress or set sovran_systemsOS.elementCalling.externalIP." >&2
+      fi
+
+      # Webhooks → lk-jwt-service. The JWT service validates the HMAC
+      # signature against the same key file it issues tokens with, and uses
+      # the events (participant_left / room_finished) to detect abruptly
+      # disconnected participants instead of waiting for the delayed-event
+      # timeout. The URL hits local Caddy via the /etc/hosts loopback
+      # override and is routed to the JWT service by the element-calling
+      # vhost (/livekit/jwt/sfu_webhook → 8073).
+      LK_KEY=$(cut -d: -f1 < ${livekitKeyFile} | tr -d '[:space:]')
+
+      cat >> /run/livekit/livekit.yaml <<EOF
 room:
   auto_create: false
 turn:
@@ -209,6 +274,10 @@ turn:
   udp_port: 3478
   cert_file: /run/credentials/livekit.service/turn-cert
   key_file: /run/credentials/livekit.service/turn-key
+webhook:
+  api_key: $LK_KEY
+  urls:
+    - https://$ELEMENT_CALLING/livekit/jwt/sfu_webhook
 EOF
 
       chmod 644 /run/livekit/livekit.yaml
@@ -216,24 +285,17 @@ EOF
   };
 
   ####### LIVEKIT SERVICE #######
+  # NOTE: the runtime config (rtc ports, TURN, webhook, node_ip) is generated
+  # by livekit-turn-setup and delivered via LoadCredential; the upstream
+  # module's `settings` block is therefore intentionally NOT used (it would
+  # be dead config that silently diverges from what LiveKit actually loads).
+  # The firewall ports are opened explicitly below; openFirewall is left off
+  # so the upstream module does not also open 7880/tcp publicly (Caddy fronts
+  # the SFU on this host).
   services.livekit = {
     enable = true;
-    openFirewall = true;
+    openFirewall = false;
     keyFile = livekitKeyFile;
-    settings = {
-      rtc.use_external_ip = true;
-      rtc.skip_external_ip_validation = true;
-      rtc.tcp_port = 7881;
-      rtc.udp_port = 7882;
-      rtc.port_range_start = 30000;
-      rtc.port_range_end = 40000;
-      room.auto_create = false;
-      turn = {
-        enabled = true;
-        tls_port = 5349;
-        udp_port = 3478;
-      };
-    };
   };
 
   # Override ExecStart to load the runtime-generated config (which carries the
@@ -277,12 +339,26 @@ EOF
     script = ''
       ELEMENT_CALLING=$(cat /var/lib/domains/element-calling)
       MATRIX=$(cat /var/lib/domains/matrix)
+      FULL_ACCESS_HOMESERVERS="$MATRIX"
+
+      # Federated peers may also be granted LiveKit room-creation (full access)
+      # on this SFU via sovran_systemsOS.elementCalling.fullAccessHomeservers.
+      # Without this, remote users can join existing calls but cannot be the
+      # first to start one on your SFU.
+      EXTRA_HS='${lib.concatStringsSep "," config.sovran_systemsOS.elementCalling.fullAccessHomeservers}'
+      if [ -n "$EXTRA_HS" ]; then
+        FULL_ACCESS_HOMESERVERS="$FULL_ACCESS_HOMESERVERS,$EXTRA_HS"
+      fi
 
       mkdir -p /run/lk-jwt-service
 
       cat > /run/lk-jwt-service/env <<EOF
 LIVEKIT_URL=wss://$ELEMENT_CALLING
-LIVEKIT_FULL_ACCESS_HOMESERVERS=$MATRIX
+LIVEKIT_FULL_ACCESS_HOMESERVERS=$FULL_ACCESS_HOMESERVERS
+# Re-check, every 60s, that connected participants are still on the SFU;
+# guards against missed SFU webhooks (e.g. an SFU restart) leaving stale
+# call members in Matrix rooms.
+LIVEKIT_SANITY_CHECK_INTERVAL_SECONDS=60
 EOF
 
       chmod 640 /run/lk-jwt-service/env
@@ -294,6 +370,9 @@ EOF
     enable = true;
     port = 8073;
     keyFile = livekitKeyFile;
+    # Required by the upstream module's option type, but overridden at runtime
+    # by EnvironmentFile (/run/lk-jwt-service/env, generated above from the
+    # element-calling domain). Kept as a harmless placeholder.
     livekitUrl = "wss://placeholder.local";
   };
 
@@ -301,6 +380,126 @@ EOF
     "/run/lk-jwt-service/env"
   ];
 
+  # Restart LiveKit / lk-jwt-service when a rebuild regenerates their runtime
+  # configs (new domains, externalIP, full-access list), mirroring the domain
+  # change flow.
+  # Re-run the config generator and restart LiveKit when a rebuild regenerates
+  # the runtime config, or when the Hub persists a new external IP (dynamic
+  # WAN IPs), so the advertised ICE candidate stays current without a manual
+  # restart. The trigger chain: external-ip change → livekit-turn-setup
+  # re-runs → rewrites livekit.yaml → livekit restarts with the new config.
+  systemd.services.livekit-turn-setup.restartTriggers = [ "/var/lib/secrets/external-ip" ];
+  systemd.services.livekit.restartTriggers = [ "/run/livekit/livekit.yaml" ];
+  systemd.services.lk-jwt-service.restartTriggers = [ "/run/lk-jwt-service/env" ];
+
+  ####### PUBLIC REACHABILITY SELF-CHECK #######
+  # Diagnostic only — never a hard dependency of livekit/caddy. Catches the
+  # classic "call connects but no media" setup errors at boot instead of at
+  # call time:
+  #   * the element-calling domain having no public records (or resolving to
+  #     loopback/link-local/CGNAT for remote peers),
+  #   * the lk-jwt-service being unreachable through Caddy,
+  #   * the MatrixRTC transports endpoint being absent (Element X cannot
+  #     discover calling and shows MISSING_MATRIX_RTC_TRANSPORT).
+  # The check queries only the operator's own DNS provider (the domain's
+  # authoritative nameservers, resolved via the local resolver) plus the
+  # server's own Caddy and public IP — no third-party resolver or service is
+  # contacted. dig queries resolvers directly, so the /etc/hosts loopback
+  # overrides (modules/core/local-domain-loopback.nix) do not influence the
+  # result.
+  systemd.services.element-calling-public-check = {
+    description = "Verify Element Calling domain, JWT service and MatrixRTC transports endpoint are publicly reachable";
+    after = [ "network-online.target" "caddy.service" "livekit.service" "lk-jwt-service.service" ];
+    wants = [ "network-online.target" ];
+    wantedBy = [ "multi-user.target" ];
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+    };
+    unitConfig = {
+      ConditionPathExists = "/var/lib/domains/element-calling";
+    };
+    path = [ pkgs.coreutils pkgs.gawk pkgs.dnsutils pkgs.curl ];
+    script = ''
+      ELEMENT_CALLING=$(cat /var/lib/domains/element-calling)
+      MATRIX=$(cat /var/lib/domains/matrix)
+      FAIL=0
+
+      echo "── Element Calling public reachability self-check ──"
+
+      # 1) Authoritative DNS view (bypassing the /etc/hosts loopback
+      #    overrides). Resolve the domain's own nameservers via the local
+      #    resolver, then query those nameservers directly — the only party
+      #    that sees the query is the DNS provider the operator already uses
+      #    for the domain.
+      NS_LIST=$(dig +short NS "$ELEMENT_CALLING" 2>/dev/null | tr '\n' ' ')
+      if [ -n "$NS_LIST" ]; then
+        IPS=""
+        for NSRV in $NS_LIST; do
+          IPS=$( { dig +short A "$ELEMENT_CALLING" "@$NSRV" 2>/dev/null; dig +short AAAA "$ELEMENT_CALLING" "@$NSRV" 2>/dev/null; } | tr '\n' ' ' )
+          [ -n "$IPS" ] && break
+        done
+        echo "Authoritative nameservers for $ELEMENT_CALLING: $NS_LIST"
+      else
+        echo "WARNING: could not resolve nameservers for $ELEMENT_CALLING via the local resolver; using the local resolver's answer instead." >&2
+        IPS=$( { dig +short A "$ELEMENT_CALLING" 2>/dev/null; dig +short AAAA "$ELEMENT_CALLING" 2>/dev/null; } | tr '\n' ' ' )
+      fi
+
+      if [ -z "$IPS" ]; then
+        echo "ERROR: no A/AAAA records for $ELEMENT_CALLING at its authoritative nameservers. Remote peers cannot reach this LiveKit; calls will connect without media." >&2
+        FAIL=1
+      else
+        echo "Public DNS for $ELEMENT_CALLING: $IPS"
+        for IP in $IPS; do
+          case "$IP" in
+            0.*|127.*|169.254.*|100.64.*|::1|fe80:*|fc*:*|fd*:*)
+              echo "ERROR: $ELEMENT_CALLING resolves to $IP (loopback/link-local/CGNAT). Remote peers cannot reach it." >&2
+              FAIL=1 ;;
+          esac
+        done
+      fi
+
+      # 2) lk-jwt-service healthz through Caddy (validates the proxy chain).
+      if curl -fsS --max-time 10 "https://$ELEMENT_CALLING/livekit/jwt/healthz" >/dev/null 2>&1; then
+        echo "OK: https://$ELEMENT_CALLING/livekit/jwt/healthz responds"
+      else
+        echo "ERROR: https://$ELEMENT_CALLING/livekit/jwt/healthz not reachable through Caddy." >&2
+        FAIL=1
+      fi
+
+      # 3) Same healthz via the first public IP (tests the full NAT path).
+      #    NOTE: if this box is behind the same NAT you are testing through,
+      #    routers without hairpin NAT will fail this step — the warning is
+      #    then expected and harmless; verify from an external device instead.
+      if [ -n "$IPS" ]; then
+        PUBIP=$(echo "$IPS" | awk '{print $1}')
+        if curl -fsS --max-time 15 --resolve "$ELEMENT_CALLING:443:$PUBIP" "https://$ELEMENT_CALLING/livekit/jwt/healthz" >/dev/null 2>&1; then
+          echo "OK: healthz reachable via public IP $PUBIP (NAT path works)"
+        else
+          echo "WARNING: healthz NOT reachable via public IP $PUBIP — check router port-forwarding (443/TCP) and NAT hairpin. Expected if the router lacks hairpin NAT; verify from an external device." >&2
+        fi
+      fi
+
+      # 4) MatrixRTC transports registry (MSC4519) — required by Element X.
+      CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "https://$MATRIX/_matrix/client/unstable/org.matrix.msc4143/rtc/transports")
+      case "$CODE" in
+        401|200)
+          echo "OK: MatrixRTC transports endpoint present (HTTP $CODE; auth required is expected)" ;;
+        404)
+          echo "ERROR: /_matrix/client/unstable/org.matrix.msc4143/rtc/transports missing (HTTP 404) — Element X cannot discover calling. Enable msc4143_enabled and matrix_rtc.transports in Synapse." >&2
+          FAIL=1 ;;
+        *)
+          echo "WARNING: transports endpoint returned HTTP $CODE" >&2 ;;
+      esac
+
+      if [ "$FAIL" -eq 1 ]; then
+        echo "── Element Calling self-check FAILED — see errors above ──" >&2
+        exit 1
+      fi
+      echo "── Element Calling self-check passed ──"
+    '';
+  };
+
   ####### SYNAPSE RUNTIME CONFIG (element-calling additions) #######
   systemd.services.element-calling-synapse-config = {
     description = "Generate Synapse runtime config for Element Calling";
@@ -317,6 +516,7 @@ EOF
     path = [ pkgs.coreutils ];
     script = ''
       MATRIX=$(cat /var/lib/domains/matrix)
+      ELEMENT_CALLING=$(cat /var/lib/domains/element-calling)
 
       mkdir -p /run/matrix-synapse
 
@@ -326,7 +526,17 @@ public_baseurl: "https://$MATRIX"
 serve_server_wellknown: true
 experimental_features:
   msc3266_enabled: true
+  # MSC4143: enables the MatrixRTC transports registry endpoint
+  # (/_matrix/client/unstable/org.matrix.msc4143/rtc/transports, MSC4519).
+  # Element X requires this endpoint to discover the LiveKit focus; without it
+  # mobile clients fail with MISSING_MATRIX_RTC_TRANSPORT / cannot start calls.
+  msc4143_enabled: true
   msc4222_enabled: true
+# MSC4519: advertise this site's LiveKit focus via the transports registry.
+matrix_rtc:
+  transports:
+    - type: livekit
+      livekit_service_url: "https://$ELEMENT_CALLING/livekit/jwt"
 max_event_delay_duration: "24h"
 rc_message:
   per_second: 0.5

modules/modules.nix

@@ -17,6 +17,7 @@
     ./core/no-sleep.nix
     ./core/cpu-performance.nix
     ./core/local-domain-loopback.nix
+    ./core/public-ip.nix
 
     # ── Always on (no flag) ───────────────────────────────────
     ./php.nix

packages/lndinit/default.nix

@@ -1,19 +0,0 @@
-{ lib, buildGoModule, fetchFromGitHub }:
-
-buildGoModule rec {
-  pname = "lndinit";
-  version = "0.1.3-beta";
-  src = fetchFromGitHub {
-    owner = "lightninglabs";
-    repo = pname;
-    rev = "v${version}";
-    sha256 = "sha256-sO1DpbppCurxr9g9nUl9Vx82FJK1mTcUw3rY1Fm1wEU=";
-  };
-  vendorHash = "sha256-El44BS5Bu0K/klMxkajciU/R6uqiXBMOiLN536QztbE=";
-  subPackages = [ "." ];
-  meta = with lib; {
-    description = "Wallet initializer for lnd (from nix-bitcoin)";
-    homepage = "https://github.com/lightninglabs/lndinit";
-    license = licenses.mit;
-  };
-}

packages/rtl/default.nix

@@ -10,11 +10,11 @@
 }:
 let self = stdenvNoCC.mkDerivation {
   pname = "rtl";
-  version = "0.15.8";
+  version = "0.15.10";
 
   src = fetchurl {
     url = "https://github.com/Ride-The-Lightning/RTL/archive/refs/tags/v${self.version}.tar.gz";
-    hash = "sha256-8XdGyORxB2dkZRB/Yl7zh+Quqo4L/Y0VmC6Brbr/hqU=";
+    hash = "sha256-r5riYV2FN0OKi0mwj9I1jBeeU1LOv2HVB6CEovPlUuY=";
   };
 
   passthru = {
@@ -26,7 +26,7 @@ let self = stdenvNoCC.mkDerivation {
       # TODO-EXTERNAL: Remove `npmFlags` when no longer required
       # See: https://github.com/Ride-The-Lightning/RTL/issues/1182
       npmFlags = "--legacy-peer-deps";
-      hash = "sha256-oMqd6nLzS6iQ9w4z2yzpR2unA5qhOq5YdvfoS8IgYLY=";
+      hash = "sha256-NKiWcjqYcHBVIB+vbF3aKXLe2fJRmh/quu8obztP3TA=";
     };
   };
 

tests/test_update_state.py

@@ -0,0 +1,222 @@
+"""Regression tests for Hub update completion and polling recovery."""
+
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+
+_REPO_ROOT = Path(__file__).resolve().parents[1]
+_APP_PARENT = _REPO_ROOT / "app"
+if str(_APP_PARENT) not in sys.path:
+    sys.path.insert(0, str(_APP_PARENT))
+
+from sovran_systemsos_web.update_state import (  # noqa: E402
+    effective_update_status,
+    reboot_is_pending,
+)
+
+
+# Real store paths observed on the incident machine that prompted this rework.
+RUNNING_GENERATION = (
+    "84rsiqi66nc68jbikd26ms50ap831xf8-nixos-system-nixos-26.11.20260817.ec2d622"
+)
+PREVIOUS_GENERATION = (
+    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-nixos-system-nixos-26.11.20260816.old"
+)
+# What the stale Hub log still claimed was staged: a two-week-old update cycle.
+HUB_LOG_GENERATION = (
+    "yis0saq6p8fhqcaii0h2yzqf0blhdwns-nixos-system-nixos-26.11.20260804.e72e4f2"
+)
+
+
+class TestRebootPendingState(unittest.TestCase):
+    """Reboot-pending state is derived from NixOS, not from Hub marker files.
+
+    The system profile (what boots next) is compared against
+    /run/current-system (what is running).  This stays correct no matter
+    which tool performed the update: Hub "Update System", a terminal
+    ``nixos-rebuild``, or an SSH support session.
+    """
+
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        root = Path(self.tmp.name)
+        self.store = root / "store"
+        self.store.mkdir()
+        # /nix/var/nix/profiles/system is a two-hop chain: the diagnostics on
+        # the incident box showed ``system`` -> ``system-148-link`` -> store.
+        self.profile = root / "system"
+        self.profile_entry = root / "system-148-link"
+        # /run/current-system links straight to the running store path.
+        self.current = root / "current-system"
+
+    def _store_dir(self, name: str) -> str:
+        path = self.store / name
+        path.mkdir(exist_ok=True)
+        return str(path)
+
+    def _stage(self, booted: str, running: str) -> None:
+        """Stage ``booted`` as the boot default with ``running`` live."""
+        os.symlink(self._store_dir(booted), self.profile_entry)
+        os.symlink(self.profile_entry, self.profile)
+        os.symlink(self._store_dir(running), self.current)
+
+    def test_hub_update_staged_and_not_yet_rebooted_is_pending(self):
+        self._stage(booted=RUNNING_GENERATION, running=PREVIOUS_GENERATION)
+        self.assertTrue(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_staged_generation_booted_is_no_longer_pending(self):
+        self._stage(booted=RUNNING_GENERATION, running=RUNNING_GENERATION)
+        self.assertFalse(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_terminal_switch_needs_no_reboot(self):
+        # nixos-rebuild switch moves the profile AND current-system together.
+        self._stage(booted=RUNNING_GENERATION, running=RUNNING_GENERATION)
+        self.assertFalse(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_rollback_leaves_nothing_pending(self):
+        # nixos-rebuild switch --rollback points both at the rollback target.
+        self._stage(booted=PREVIOUS_GENERATION, running=PREVIOUS_GENERATION)
+        self.assertFalse(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_unverifiable_state_is_never_a_reboot_demand(self):
+        # No profile and no running system readable: the Hub must not nag
+        # about a reboot it cannot substantiate.
+        self.assertFalse(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_stale_hub_marker_clears_after_terminal_updates(self):
+        """The exact incident: terminal-updated machine, frozen Hub marker.
+
+        The user's last Hub update (old updater, weeks prior) left
+        REBOOT_REQUIRED behind.  Every update since ran in a terminal and
+        never touched the Hub's files, so the log still records a staged
+        generation from 2026-08-04 while the machine runs a 2026-08-17
+        build.  With the profile and the running system in agreement, the
+        stale claim must reconcile to IDLE regardless of anything the old
+        marker/log files say.
+        """
+        root = Path(self.tmp.name)
+        log = root / "sovran-hub-update.log"
+        log.write_text(
+            "Done. The new configuration is "
+            f"/nix/store/{HUB_LOG_GENERATION}\n"
+            "✓ Update staged successfully\n",
+            encoding="utf-8",
+        )
+        # Deliberately no sovran-hub-update.generation marker: the updater
+        # that produced this state predates the marker feature.
+        self.assertFalse(
+            (root / "sovran-hub-update.generation").exists()
+        )
+
+        self._stage(booted=RUNNING_GENERATION, running=RUNNING_GENERATION)
+        self.assertEqual(
+            effective_update_status(
+                "REBOOT_REQUIRED", str(self.profile), str(self.current)
+            ),
+            "IDLE",
+        )
+
+    def test_genuine_pending_reboot_claim_survives(self):
+        # A staged generation that has NOT been booted yet: the claim is
+        # true and must keep surfacing until the reboot really happens.
+        self._stage(booted=RUNNING_GENERATION, running=PREVIOUS_GENERATION)
+        self.assertEqual(
+            effective_update_status(
+                "REBOOT_REQUIRED", str(self.profile), str(self.current)
+            ),
+            "REBOOT_REQUIRED",
+        )
+
+    def test_other_statuses_pass_through_unchanged(self):
+        self._stage(booted=RUNNING_GENERATION, running=RUNNING_GENERATION)
+        for status in ("RUNNING", "FAILED", "SUCCESS", "IDLE"):
+            self.assertEqual(
+                effective_update_status(
+                    status, str(self.profile), str(self.current)
+                ),
+                status,
+            )
+
+
+class TestUpdatePollingWiring(unittest.TestCase):
+    """Guard the browser failure modes that caused a permanent spinner."""
+
+    @classmethod
+    def setUpClass(cls):
+        js_dir = _REPO_ROOT / "app" / "sovran_systemsos_web" / "static" / "js"
+        cls.update_js = (js_dir / "update.js").read_text(encoding="utf-8")
+        cls.rebuild_js = (js_dir / "rebuild.js").read_text(encoding="utf-8")
+        cls.helpers_js = (js_dir / "helpers.js").read_text(encoding="utf-8")
+        cls.events_js = (js_dir / "events.js").read_text(encoding="utf-8")
+        cls.template = (
+            _REPO_ROOT / "app" / "sovran_systemsos_web" / "templates" / "index.html"
+        ).read_text(encoding="utf-8")
+
+    def test_status_fetches_have_an_abort_timeout(self):
+        self.assertIn("function apiFetchWithTimeout", self.helpers_js)
+        self.assertIn("new AbortController()", self.helpers_js)
+        self.assertIn("controller.abort()", self.helpers_js)
+        self.assertIn("apiFetchWithTimeout(", self.update_js)
+        self.assertIn("STATUS_POLL_FETCH_TIMEOUT", self.update_js)
+
+    def test_async_update_polls_cannot_overlap(self):
+        self.assertIn("_updatePollInFlight", self.update_js)
+        self.assertIn(
+            "if (_updateFinished || _updatePollInFlight) return;", self.update_js
+        )
+        self.assertIn("finally", self.update_js)
+
+    def test_connection_failure_has_explicit_non_running_ui(self):
+        self.assertIn("showUpdateStatusUnavailable", self.update_js)
+        self.assertIn("Update status unavailable", self.update_js)
+        self.assertIn("Retry Status", self.template)
+        self.assertIn("retryUpdateStatus", self.events_js)
+
+    def test_rdp_or_tab_resume_forces_reconciliation(self):
+        self.assertIn("resumeUpdateStatusAfterInterruption", self.events_js)
+        self.assertIn('window.addEventListener("focus"', self.events_js)
+        self.assertIn('document.addEventListener("visibilitychange"', self.events_js)
+
+    def test_verbose_log_rendering_is_bounded(self):
+        self.assertIn("UPDATE_VISIBLE_LOG_MAX_CHARS", self.update_js)
+        self.assertIn("document.createTextNode(text)", self.update_js)
+        self.assertNotIn("$modalLog.textContent += text", self.update_js)
+
+    def test_page_reload_restores_running_or_completed_update(self):
+        self.assertIn("restoreUpdateModalIfNeeded", self.update_js)
+        self.assertIn("await restoreUpdateModalIfNeeded();", self.events_js)
+        self.assertIn('current.result === "reboot_required"', self.update_js)
+
+    def test_all_javascript_remains_syntax_valid(self):
+        node = shutil.which("node")
+        if not node:
+            self.skipTest("node is not available in this test environment")
+        for script in (
+            _REPO_ROOT / "app" / "sovran_systemsos_web" / "static" / "js"
+        ).glob("*.js"):
+            result = subprocess.run(
+                [node, "--check", str(script)], capture_output=True, text=True
+            )
+            self.assertEqual(result.returncode, 0, result.stderr)
+
+
+if __name__ == "__main__":
+    unittest.main()