chore(release): prepare v1.1.3

Fix the version metadata generation in sovran-hub.nix for Alby Hub, RTL, and Mempool. Previously, the build would incorrectly fall back to older upstream nixpkgs versions because the package names exist upstream, despite the OS deploying custom vendored forks locally. This replaces the fragile checks with direct evaluations of the local packages.

Also updates the development fallback versions.json to reflect the current vendored Alby Hub version (1.24.0).

CHANGELOG.md

@@ -7,6 +7,54 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [1.1.3] - 2026-08-27
+
+### Added
+- Sparrow update
+- Nixpkgs update with bisq and bisq2
+- Albyhub: re-harvest hashes for fixed sovran tag
+- Albyhub default.nix hash update
+- Albyhub default.nix fix
+- Albyhub: build from sovran fork tag (LND-only, no frontend)
+- Vendor v1.24.0 LND-only, no-frontend build
+- Ux: simplify domain and router setup guidance
+
+### Fixed
+- Evaluate local vendored package versions directly
+- Use proxyVendor to preserve secp256k1-zkp cgo headers
+[1.1.3]: https://git.sovransystems.com/Sovran_Systems/Sovran_SystemsOS/releases/tag/v1.1.3
+
+
+## [1.1.2] - 2026-08-21
+
+### Added
+- Nixpkgs update with Bisq update
+- Unify public-IP detection into one privacy-first script
+- Fix Element X discovery and harden federated calling
+- Nixpkgs update
+- Add tor-browser to the package list
+
+### Changed
+- Use pkgs.lndinit, drop local packages/lndinit
+- Update RTL to 0.15.10
+
+### Fixed
+- Move system.activationScripts under the config attribute
+- Reuse Hub external IP for LiveKit, drop egress service calls
+- Derive Restart required from boot default vs running system
+- Reconcile completed updates after polling stalls
+- Migrate i2pd SAM settings for nixpkgs 26.11
+- Order turn-setup after network-online to fix boot-time red dot
+- Restore a Zeus-scannable LND REST connect QR
+
+### Security
+- Prevent LND admin macaroon exposure in curl argv
+
+### Documentation
+- Update desktop and Sovran Hub screenshots
+[1.1.2]: https://git.sovransystems.com/Sovran_Systems/Sovran_SystemsOS/releases/tag/v1.1.2
+
+
 ## [1.1.1] - 2026-08-15
 
 ### Added
@@ -75,6 +123,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     polling failures instead of hanging indefinitely.
   - Rebuild/update scripts stream `nixos-rebuild` output into the live log
     (it was buffered until completion, making long rebuilds look frozen).
+  - The update modal now reconciles persisted update state after a page/RDP
+    reconnect, bounds every status request with a timeout, prevents overlapping
+    async polls, and reports an explicit "status unavailable" state with a
+    Retry Status action instead of spinning forever. Verbose Nix logs are
+    rendered incrementally and bounded so they cannot stall the browser UI.
+  - Successful staged updates now record the exact NixOS generation. The Hub
+    keeps showing "Restart required" until that generation is active, then
+    clears the marker after reboot (with log-based recovery for older updates).
+  - Pending-reboot state is now derived from NixOS itself — the boot default
+    (`/nix/var/nix/profiles/system`) versus the running `/run/current-system`
+    — instead of from Hub-written marker files. Updates performed from a
+    terminal or support session (which never touch the Hub's status files)
+    previously left a stale `REBOOT_REQUIRED` marker the Hub could never
+    clear, pinning the "Restart required" badge forever even after many
+    reboots. The marker self-heals to `IDLE` on the next status read whenever
+    boot default and running system agree, and recording the informational
+    `.generation` marker can no longer fail an otherwise successful update.
 
 ---
 

README.md

@@ -21,9 +21,9 @@ Lightning infrastructure, private cloud, and communications platform when you
 are ready.
 
 [Visit the Website](https://sovransystems.com) ·
-[Download the ISO](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso) ·
+[Download the ISO](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.3.iso) ·
 [Try it safely in a VM](#try-it-first-in-a-virtual-machine) ·
-[Verify the Download](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso.sha256) ·
+[Verify the Download](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.3.iso.sha256) ·
 [Build from Source](#build-from-source)
 
 <img src="assets/desktop-screenshot.webp" alt="Sovran_SystemsOS private Bitcoin desktop" width="800" />
@@ -345,8 +345,8 @@ with an imaging application such as [Balena Etcher](https://etcher.balena.io).
 
 ### 1. Download the ISO and checksum
 
-- [Download Sovran_SystemsOS-1.1.1.iso](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso)
+- [Download Sovran_SystemsOS-1.1.3.iso](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.3.iso)
-- [Download Sovran_SystemsOS-1.1.1.iso.sha256](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso.sha256)
+- [Download Sovran_SystemsOS-1.1.3.iso.sha256](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.3.iso.sha256)
 
 The download may take some time. Do not rename or modify the ISO before
 verifying it, and keep both files in the same folder.
@@ -364,16 +364,16 @@ checksum exactly.
 Open a terminal in the download folder and run:
 
 ```bash
-sha256sum --check Sovran_SystemsOS-1.1.1.iso.sha256
+sha256sum --check Sovran_SystemsOS-1.1.3.iso.sha256
 ```
 
 A successful comparison reports:
 
 ```text
-Sovran_SystemsOS-1.1.1.iso: OK
+Sovran_SystemsOS-1.1.3.iso: OK
 ```
 
-You can also run `sha256sum Sovran_SystemsOS-1.1.1.iso` and compare the output
+You can also run `sha256sum Sovran_SystemsOS-1.1.3.iso` and compare the output
 against the checksum file manually.
 
 </details>
@@ -384,11 +384,11 @@ against the checksum file manually.
 Open Terminal in the download folder and run:
 
 ```bash
-shasum -a 256 Sovran_SystemsOS-1.1.1.iso
+shasum -a 256 Sovran_SystemsOS-1.1.3.iso
 ```
 
 Compare the value shown in Terminal with the value inside
-`Sovran_SystemsOS-1.1.1.iso.sha256`.
+`Sovran_SystemsOS-1.1.3.iso.sha256`.
 
 </details>
 
@@ -398,7 +398,7 @@ Compare the value shown in Terminal with the value inside
 Open PowerShell in the download folder and run:
 
 ```powershell
-Get-FileHash .\Sovran_SystemsOS-1.1.1.iso -Algorithm SHA256
+Get-FileHash .\Sovran_SystemsOS-1.1.3.iso -Algorithm SHA256
 ```
 
 Compare the value under `Hash` with the published checksum.
@@ -413,7 +413,7 @@ match exactly.
 
 1. Download and install [Balena Etcher](https://etcher.balena.io), then
    connect the USB drive.
-2. Choose **Flash from file** and select `Sovran_SystemsOS-1.1.1.iso`.
+2. Choose **Flash from file** and select `Sovran_SystemsOS-1.1.3.iso`.
 3. Choose **Select target**, select the USB drive, and review your selection
    carefully.
 4. Choose **Flash** and wait for the writing and verification process to
@@ -856,7 +856,7 @@ primary location for collaboration. Please read our
 ## Privacy. Sovereignty. Bitcoin.
 
 [Visit Sovran Systems](https://sovransystems.com) ·
-[Download Sovran_SystemsOS](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.1.iso) ·
+[Download Sovran_SystemsOS](https://downloads.sovransystems.com/Sovran_SystemsOS-1.1.3.iso) ·
 [View the License](LICENSE)
 
 </div>

VERSION

@@ -1 +1 @@
-1.1.1
+1.1.3

app/sovran_systemsos_web/server.py

@@ -21,6 +21,7 @@ import subprocess
 import tempfile
 import threading
 import time
+import sys
 import urllib.error
 import urllib.parse
 import urllib.request
@@ -55,6 +56,7 @@ from .security_helpers import (
     load_session_store,
     save_session_store,
 )
+from .update_state import effective_update_status
 
 logger = logging.getLogger(__name__)
 
@@ -66,6 +68,7 @@ GITEA_API_BASE = "https://git.sovransystems.com/api/v1/repos/Sovran_Systems/Sovr
 
 UPDATE_LOG        = "/var/log/sovran-hub-update.log"
 UPDATE_STATUS     = "/var/log/sovran-hub-update.status"
+UPDATE_GENERATION = "/var/log/sovran-hub-update.generation"
 UPDATE_UNIT       = "sovran-hub-update.service"
 
 REBUILD_LOG    = "/var/log/sovran-hub-rebuild.log"
@@ -132,6 +135,7 @@ _SERVICE_DOMAIN_KEYS = frozenset([
 ])
 
 INTERNAL_IP_FILE = "/var/lib/secrets/internal-ip"
+EXTERNAL_IP_FILE = "/var/lib/secrets/external-ip"
 ZEUS_CONNECT_FILE = "/var/lib/secrets/zeus-connect-url"
 
 ONBOARDING_FLAG = "/var/lib/sovran/onboarding-complete"
@@ -242,8 +246,8 @@ CATEGORY_ORDER = [
 FEATURE_REGISTRY = [
     {
         "id": "rdp",
-        "name": "Remote Desktop (RDP)",
+        "name": "Remote Desktop",
-        "description": "Access your desktop remotely via RDP client",
+        "description": "Access your desktop from a device on your home network — no router setup needed",
         "category": "infrastructure",
         "needs_domain": False,
         "domain_name": None,
@@ -358,8 +362,8 @@ FEATURE_REGISTRY = [
     },
     {
         "id": "btcpay-web",
-        "name": "BTCPay Server Web Access",
+        "name": "Put BTCPay Server Online",
-        "description": "Expose BTCPay Server to the internet via your domain. When disabled, BTCPay Server still runs locally but is not accessible from the web.",
+        "description": "Let people donate Bitcoin, pay you, or visit your online store. BTCPay Server stays private when this is turned off.",
         "category": "bitcoin",
         "needs_domain": True,
         "domain_name": "btcpayserver",
@@ -541,9 +545,9 @@ SERVICE_DESCRIPTIONS: dict[str, str] = {
         "Addresses on your Sovran_SystemsOS node."
     ),
     "gnome-remote-desktop.service": (
-        "Access your server's full desktop environment from anywhere using any RDP client. "
+        "Access your desktop from a device on your home network — no router setup needed. "
-        "Manage your system visually without being physically present. "
+        "Sovran_SystemsOS creates the address, username, and password for you: "
-        "Sovran_SystemsOS sets up secure remote access with generated credentials — connect and go."
+        "just open an RDP app and connect."
     ),
     "sshd.service": (
         "Secure Shell (SSH) remote access. When enabled, authorized users can connect "
@@ -950,21 +954,43 @@ def _save_internal_ip(ip: str):
             pass
 
 
-def _get_external_ip() -> str:
+def _save_external_ip(ip: str):
-    MAX_IP_LENGTH = 46
+    """Write the external IP to a file so other services (e.g. LiveKit) can
-    for url in [
+    reference it without running their own detection."""
-        "https://api.ipify.org",
+    if ip and ip != "unavailable":
-        "https://ifconfig.me/ip",
-        "https://icanhazip.com",
-    ]:
         try:
-            req = urllib.request.Request(url, method="GET")
+            os.makedirs(os.path.dirname(EXTERNAL_IP_FILE), exist_ok=True)
-            with urllib.request.urlopen(req, timeout=8) as resp:
+            with open(EXTERNAL_IP_FILE, "w") as f:
-                ip = resp.read().decode().strip()
+                f.write(ip)
-                if ip and len(ip) < MAX_IP_LENGTH:
+        except OSError:
-                    return ip
+            pass
+
+
+def _get_external_ip() -> str:
+    """Public IP via the shared detector (/var/lib/sovran/public-ip.py).
+
+    The detector owns discovery (STUN -> DNS -> opt-in HTTPS echo), caches the
+    result in /var/lib/secrets/external-ip, and contacts at most one third
+    party per refresh interval. This function only reads the cache and asks
+    the detector to refresh when it is missing or stale — it performs no
+    per-call external queries of its own.
+    """
+    try:
+        r = subprocess.run(
+            [sys.executable, "/var/lib/sovran/public-ip.py", "check"],
+            capture_output=True, text=True, timeout=20,
+        )
+        if r.returncode == 0 and r.stdout.strip():
+            return r.stdout.strip().splitlines()[0]
     except Exception:
-            continue
+        pass
+    try:
+        with open(EXTERNAL_IP_FILE) as f:
+            ip = f.read().strip()
+        if ip:
+            return ip
+    except OSError:
+        pass
     return "unavailable"
 
 
@@ -1634,15 +1660,6 @@ def _nwc_lnurl_bech32(alias: str, domain: str) -> str:
 
 # ── Update helpers (file-based, no systemctl) ────────────────────
 
-def _read_update_status() -> str:
-    """Read the status file. Returns RUNNING, SUCCESS, REBOOT_REQUIRED, FAILED, or IDLE."""
-    try:
-        with open(UPDATE_STATUS, "r") as f:
-            return f.read().strip()
-    except FileNotFoundError:
-        return "IDLE"
-
-
 def _write_update_status(status: str):
     """Write to the status file."""
     try:
@@ -1652,6 +1669,36 @@ def _write_update_status(status: str):
         pass
 
 
+def _read_update_status() -> str:
+    """Read and reconcile the persistent update status.
+
+    ``REBOOT_REQUIRED`` survives Hub/browser restarts before the reboot, but
+    it is a CLAIM about live NixOS state, not the source of truth: the boot
+    default (``/nix/var/nix/profiles/system``) versus the running
+    ``/run/current-system``.  Re-validating on every read keeps the Hub
+    correct when the system was updated from a terminal or support session
+    (which never writes Hub markers), and lets an old marker that predates
+    the reconciliation feature self-heal instead of demanding reboots
+    forever.  The stale marker file is removed once cleared.
+    """
+    try:
+        with open(UPDATE_STATUS, "r") as f:
+            status = f.read().strip()
+    except FileNotFoundError:
+        return "IDLE"
+
+    effective = effective_update_status(status)
+    if effective != status:
+        _write_update_status(effective)
+        try:
+            os.remove(UPDATE_GENERATION)
+        except OSError:
+            pass
+        return effective
+
+    return status
+
+
 def _read_log(offset: int = 0) -> tuple[str, int]:
     """Read the update log file from the given byte offset.
     Returns (new_text, new_offset)."""
@@ -3695,6 +3742,9 @@ async def api_network():
     # Keep the internal-ip file in sync for credential lookups
     _save_internal_ip(internal)
     _cached_external_ip = external
+    # Persist the external IP so other services (e.g. LiveKit) can reuse the
+    # Hub's detection instead of running their own.
+    _save_external_ip(external)
     return {"internal_ip": internal, "external_ip": external}
 
 
@@ -3832,9 +3882,15 @@ async def api_ports_health():
 @app.get("/api/updates/check")
 async def api_updates_check():
     loop = asyncio.get_event_loop()
+    status = await loop.run_in_executor(None, _read_update_status)
+    if status in {"RUNNING", "REBOOT_REQUIRED"}:
+        # Avoid a slow remote update check when there is already an operation
+        # the dashboard needs to surface.
+        return {"available": True, "status": status.lower()}
+
     available = await loop.run_in_executor(None, check_for_updates)
     # None means inconclusive (check failed) — report as available so the UI doesn't block
-    return {"available": available is not False}
+    return {"available": available is not False, "status": status.lower()}
 
 
 @app.get("/api/ping")
@@ -6181,10 +6237,19 @@ async def _startup_migrate_deprecated_features():
 
 async def _background_domain_reachability_checker():
     """Periodically curl configured domains and cache reachability results."""
+    global _cached_external_ip
     await asyncio.sleep(_DOMAIN_REACHABILITY_STARTUP_DELAY)
     consecutive_failures = 0
     while True:
         try:
+            # Keep the persisted external IP fresh (dynamic WAN IPs), so
+            # services like LiveKit can read /var/lib/secrets/external-ip.
+            loop = asyncio.get_event_loop()
+            external = await loop.run_in_executor(None, _get_external_ip)
+            if external != "unavailable":
+                _cached_external_ip = external
+                _save_external_ip(external)
+
             cfg = load_config()
             services = cfg.get("services", [])
 
@@ -6194,7 +6259,6 @@ async def _background_domain_reachability_checker():
                 if unit is not None
             }
 
-            loop = asyncio.get_event_loop()
             overrides, *_ = await loop.run_in_executor(None, _read_hub_overrides)
 
             domains_to_check: list[str] = []

app/sovran_systemsos_web/static/js/constants.js

@@ -5,10 +5,16 @@
 const POLL_INTERVAL_SERVICES    = 5000;
 const POLL_INTERVAL_UPDATES     = 1800000;
 const UPDATE_POLL_INTERVAL          = 2000;
-// Max consecutive failed rebuild/update status polls before the page gives up
+// A pending fetch never rejects by itself. Bound every status request so a
-// waiting and reloads to re-sync (2s interval → ~2 minutes of failures).
+// wedged browser connection cannot leave the modal spinning forever.
-// A brief Hub restart during activation only causes a handful of failures.
+const STATUS_POLL_FETCH_TIMEOUT      = 15000;
-const STATUS_POLL_MAX_FAILURES  = 60;
+// Eight timed-out requests plus the poll interval is a little over two minutes.
+// A brief Hub restart or a heavily loaded Nix build remains well inside this.
+const STATUS_POLL_MAX_FAILURES       = 8;
+// Keep verbose Nix output from making textContent updates quadratic and
+// freezing the Hub renderer (especially noticeable over RDP).
+const UPDATE_VISIBLE_LOG_MAX_CHARS   = 250000;
+const UPDATE_VISIBLE_LOG_TRIM_CHARS  = 200000;
 const REBOOT_CHECK_INTERVAL     = 5000;
 const REBOOT_FETCH_TIMEOUT      = 12000;
 const REBOOT_REQUEST_TIMEOUT    = 4000;

app/sovran_systemsos_web/static/js/domain-prereqs.js

@@ -1,22 +1,14 @@
-/* Sovran_SystemsOS Hub — Shared domain-prerequisite instructions.
+/* Sovran_SystemsOS Hub — Shared domain and router instructions.
 
-   SINGLE SOURCE OF TRUTH for the "you need a Njal.la domain + router port
+   This is the single source of truth for the simple setup guidance shown in:
-   forwarding" guidance that must read identically everywhere it appears:
 
-     • First-boot onboarding wizard (Server + Desktop role)
+     • First-boot onboarding for Server + Desktop
-       — onboarding.js step 3
+     • Feature setup in the Hub
-     • Feature-enable domain modal (Node role, and any role)
+     • Domain reconfiguration and troubleshooting
-       — features.js openDomainSetupModal()
-       (Lightning Wallet Connections / NWC, BTCPay Server, Haven, …)
-     • Domain reconfigure / troubleshooting modal
-       — features.js openDomainReconfigureModal()
 
-   Keep these three surfaces word-for-word consistent: always edit this
+   Keep the wording consistent by editing it here. */
-   file, never fork the wording inline. Plain classic script (no modules) —
-   both templates load it with a plain <script> tag. */
 "use strict";
 
-/* Escape helper local to this file so it is self-contained on both pages. */
 function dpEsc(str) {
   return String(str)
     .replace(/&/g, "&amp;")
@@ -26,105 +18,71 @@ function dpEsc(str) {
     .replace(/'/g, "&#39;");
 }
 
-/* ── "What you'll need" intro ──────────────────────────────────────
+/* Explain only what the user needs for the step in front of them. */
-   opts.serviceName — e.g. "Lightning Wallet Connections", or null for
-   the multi-service (onboarding) wording.
-   opts.hostExample — e.g. "lightning" → lightning.yourdomain.com */
 function renderDomainNeedsHtml(opts) {
+  opts = opts || {};
   var serviceName = opts.serviceName || null;
-  var hostExample = dpEsc(opts.hostExample || "myservice");
+  var purpose = opts.purpose || "";
   var html = "";
 
-  if (serviceName) {
+  if (purpose === "btcpay") {
-    html += "<p>To enable <strong>" + dpEsc(serviceName) + "</strong>, you'll need two things first:</p>";
+    html += "<p><strong>Put your BTCPay Server online</strong></p>";
-    html += '<ol style="margin:8px 0 0 16px;padding:0;line-height:1.7;">';
+    html += "<p>Let people donate Bitcoin, pay you, or visit your online store.</p>";
-    html += "<li><strong>A domain of your own from <a href=\"https://njal.la\" target=\"_blank\" rel=\"noopener noreferrer\" style=\"color:var(--accent-color);\">Njal.la</a></strong> "
+    html += "<p>BTCPay Server needs a domain so people can reach it. "
-      + "— privacy-friendly, no personal details required, accepts Bitcoin. "
+      + "Sovran_SystemsOS walks you through getting your domain from "
-      + dpEsc(serviceName) + " gets its own hostname: a subdomain (e.g. <code>" + hostExample + ".yourdomain.com</code>) "
+      + '<a href="https://njal.la" target="_blank" rel="noopener noreferrer" style="color:var(--accent-color);">Njal.la</a>'
-      + "or a separate domain — your choice. Subdomains are free, and one domain can have many.</li>";
+      + " and connecting it. Just follow the steps below.</p>";
-    html += "<li><strong>Access to your router</strong> — you'll forward ports <strong>80</strong> and <strong>443</strong> (TCP) "
+  } else if (serviceName) {
-      + "to this computer once. All domain-based services share these two ports; "
+    html += "<p>To make <strong>" + dpEsc(serviceName) + "</strong> available outside your home, it needs a domain.</p>";
-      + "they're required for HTTPS and SSL certificates.</li>";
+    html += "<p>Sovran_SystemsOS walks you through getting your domain from "
-    html += "</ol>";
+      + '<a href="https://njal.la" target="_blank" rel="noopener noreferrer" style="color:var(--accent-color);">Njal.la</a>'
+      + " and connecting it. Just follow the steps below.</p>";
   } else {
-    html += "<p><strong>Each service below needs two things set up:</strong></p>";
+    html += "<p>Your domain is the address people use to reach your services on the internet.</p>";
-    html += '<ol style="margin:8px 0 0 16px;padding:0;line-height:1.7;">';
+    html += "<p>Sovran_SystemsOS walks you through getting your domain from "
-    html += "<li><strong>A domain of your own from <a href=\"https://njal.la\" target=\"_blank\" rel=\"noopener noreferrer\" style=\"color:var(--accent-color);\">Njal.la</a></strong> "
+      + '<a href="https://njal.la" target="_blank" rel="noopener noreferrer" style="color:var(--accent-color);">Njal.la</a>'
-      + "— privacy-friendly, no personal details required, accepts Bitcoin. "
+      + " and connecting your services. Just follow the steps below.</p>";
-      + "Each service gets its own hostname: its own subdomain (e.g. <code>" + hostExample + ".yourdomain.com</code>) "
-      + "or a separate domain — your choice. Subdomains are free, and one domain can have many, "
-      + "so a single domain can serve every service.</li>";
-    html += "<li><strong>Access to your router</strong> — you'll forward ports <strong>80</strong> and <strong>443</strong> (TCP) "
-      + "to this computer once. All domain-based services share these two ports; "
-      + "they're required for HTTPS and SSL certificates.</li>";
-    html += "</ol>";
   }
   return html;
 }
 
-/* ── "How to set it up at Njal.la" steps ───────────────────────────
+/* The shortest Njal.la steps that still tell the user exactly what to do. */
-   opts.hostExample — host part used in the examples (default "call").
-   opts.pasteHint   — where the curl command goes:
-                      "below" (single service) or "next to its service below"
-                      (onboarding, many services). */
 function renderNjallaStepsHtml(opts) {
   var hostExample = dpEsc((opts && opts.hostExample) || "call");
   var pasteHint = (opts && opts.pasteHint) || "below";
   var html = "";
 
-  html += "<p style=\"margin-top:12px;\"><strong>How to set it up at Njal.la:</strong></p>";
+  html += '<p style="margin-top:12px;"><strong>Get and connect your domain:</strong></p>';
   html += '<ol style="margin:8px 0 0 16px;padding:0;line-height:1.7;">';
-  html += "<li>Create an account at <a href=\"https://njal.la\" target=\"_blank\" rel=\"noopener noreferrer\" style=\"color:var(--accent-color);\">https://njal.la</a> and buy a domain.</li>";
+  html += '<li>Open <a href="https://njal.la" target="_blank" rel="noopener noreferrer" style="color:var(--accent-color);">Njal.la</a>, create an account, and get a domain.</li>';
-  html += "<li>Add a <strong>Dynamic</strong> record for the hostname:"
+  html += "<li>Open your domain and add a <strong>Dynamic</strong> record for this service. "
-    + '<ul style="margin:4px 0 0 16px;padding:0;line-height:1.7;">'
+    + "In the <strong>Name</strong> box, enter only the first part of the address. "
-    + "<li>In the Njal.la <strong>Name</strong> field, type ONLY the host part — the word before your domain.<br>"
+    + "For <code>" + hostExample + ".yourdomain.com</code>, enter <code>" + hostExample + "</code>. "
-    + "(For &quot;" + hostExample + ".yourdomain.com&quot; you&apos;d type just: <code>" + hostExample + "</code>.)<br>"
+    + "If you are using the whole domain, leave the box blank or enter <code>@</code>.</li>";
-    + "&#9888; Do NOT type the full domain here — Njal.la adds it automatically.</li>"
+  html += "<li>Copy the update command Njal.la gives you and paste it " + pasteHint + ".</li>";
-    + "<li>Dedicating a whole separate domain to this service? Leave the Name field blank or use <code>@</code>.</li>"
-    + "</ul>"
-    + "</li>";
-  html += "<li>A Dynamic record has <strong>NO IP field</strong> — you don&apos;t enter an IP anywhere. "
-    + "It auto-fills once Sovran_SystemsOS runs the update command (on save, and again after every reboot).</li>";
-  html += "<li>Njal.la gives you a curl command, e.g.:<br>"
-    + '<code style="font-size:0.8em;">curl &quot;https://njal.la/update/?h=' + hostExample + '.yourdomain.com&amp;k=abc123&amp;auto&quot;</code><br>'
-    + "Copy it and paste it " + pasteHint + ".</li>";
   html += "</ol>";
   return html;
 }
 
-/* ── "One router task" port-forwarding box ─────────────────────────
+/* One router task shared by all normal domain-based services. */
-   opts.internalIp  — LAN IP string, or empty/null for generic wording.
-   opts.plural      — true for multi-service (onboarding) wording.
-   opts.includeSsh  — also mention port 22 for SSH (onboarding).
-   opts.extraNote   — extra sentence appended at the end (optional). */
 function renderRouterPortsHtml(opts) {
   opts = opts || {};
   var ipPart = opts.internalIp
-    ? " to this computer&rsquo;s internal IP <strong>" + dpEsc(opts.internalIp) + "</strong>"
+    ? " to this computer at <strong>" + dpEsc(opts.internalIp) + "</strong>"
-    : " to this computer&rsquo;s internal IP";
+    : " to this computer";
-  var serviceWord = opts.plural ? "services" : "service";
 
   var html = "";
-  html += "🔌 <strong>One router task:</strong> in your router&rsquo;s <strong>port forwarding</strong> settings, "
+  html += "🔌 <strong>One router task:</strong> in your router&rsquo;s <strong>Port Forwarding</strong> settings, "
-    + "forward port <strong>80 (TCP)</strong> and port <strong>443 (TCP)</strong>"
+    + "forward <strong>80 (TCP)</strong> and <strong>443 (TCP)</strong>"
-    + ipPart + ". Use the <strong>same number for the internal and external port</strong>. "
+    + ipPart + ". If your router asks for outside and inside port numbers, use the same number for both. "
-    + "This only needs to be done once — all domain services share these ports — "
+    + "You only need to do this once. All your services share these two ports.";
-    + "but HTTPS and SSL certificates won&rsquo;t work without them, so your " + serviceWord + " "
-    + "can&rsquo;t be reached from outside your home network. "
-    + "You&rsquo;ll need normal access to your router&rsquo;s settings with working port forwarding — "
-    + "if your ISP blocks it (e.g. CGNAT), domain-based services can&rsquo;t be reached from the internet. ";
-  if (opts.includeSsh) {
-    html += "Add port <strong>22 (TCP)</strong> as well if you want remote SSH access. ";
-  }
   if (opts.extraNote) {
-    html += dpEsc(opts.extraNote);
+    html += " " + dpEsc(opts.extraNote);
   }
   return html;
 }
 
-/* ── Async helper: fill a router-box container with the internal IP ──
+/* Show generic wording immediately, then add the computer's address when found. */
-   Renders generic text immediately, then upgrades to the concrete LAN IP
-   if /api/network provides one. Best-effort — never blocks the UI. */
 function renderRouterPortsBox(elId, opts) {
   var el = document.getElementById(elId);
   if (!el) return;
@@ -142,5 +100,5 @@ function renderRouterPortsBox(elId, opts) {
         target.innerHTML = renderRouterPortsHtml(next);
       }
     })
-    .catch(function() { /* generic wording already shown — fine */ });
+    .catch(function() { /* Generic wording is already shown. */ });
 }

app/sovran_systemsos_web/static/js/events.js

@@ -6,6 +6,16 @@
 if ($btnCloseModal) $btnCloseModal.addEventListener("click", closeUpdateModal);
 if ($btnReboot) $btnReboot.addEventListener("click", doReboot);
 if ($btnSave) $btnSave.addEventListener("click", saveErrorReport);
+if ($btnRetryUpdate) $btnRetryUpdate.addEventListener("click", retryUpdateStatus);
+
+// Browser timers and requests may be suspended while an RDP session/tab is in
+// the background. Reconcile immediately when the user returns instead of
+// waiting for the next interval.
+window.addEventListener("focus", resumeUpdateStatusAfterInterruption);
+window.addEventListener("online", resumeUpdateStatusAfterInterruption);
+document.addEventListener("visibilitychange", function() {
+  if (document.visibilityState === "visible") resumeUpdateStatusAfterInterruption();
+});
 if ($credsCloseBtn) $credsCloseBtn.addEventListener("click", closeCredsModal);
 if ($supportCloseBtn) $supportCloseBtn.addEventListener("click", closeSupportModal);
 
@@ -248,6 +258,11 @@ async function init() {
     setInterval(checkUpdates, POLL_INTERVAL_UPDATES);
     loadAutolaunchToggle();
   }
+
+  // If the page was reloaded or the RDP/browser session resumed during an
+  // update, reopen the modal from the persisted backend state. This also
+  // surfaces a completed update that is waiting for its activation reboot.
+  await restoreUpdateModalIfNeeded();
 }
 
 document.addEventListener("DOMContentLoaded", init);

app/sovran_systemsos_web/static/js/features.js

@@ -60,6 +60,7 @@ function openDomainSetupModal(feat, onSaved) {
   if ($domainSetupTitle) $domainSetupTitle.textContent = "🌐 Domain Setup — " + feat.name;
 
   var isWalletConnections = (feat.id === "nwc-wallets" || feat.domain_name === "lightning");
+  var isBtcpay = feat.id === "btcpay-web";
 
   var npubField = "";
   if (feat.id === "haven") {
@@ -77,30 +78,30 @@ function openDomainSetupModal(feat, onSaved) {
 
   var nwcWarning = isWalletConnections
     ? '<div class="domain-nwc-warning">' +
-      '<strong>⚠ Lightning Wallet Connections requires its own unique hostname.</strong> ' +
+      '<strong>Lightning Wallet Connections needs its own address.</strong> ' +
-      'Use a new subdomain such as <code>lightning.yourdomain.com</code>, or a separate domain. ' +
+      'Use a new address such as <code>lightning.yourdomain.com</code>. Do not use an address already connected to another service.' +
-      'Do not reuse a domain already assigned to Matrix, Nextcloud, WordPress, BTCPay Server, Vaultwarden, Haven, or another Caddy site.' +
       '</div>'
     : '';
 
-  var domainPlaceholder = isWalletConnections ? "lightning.yourdomain.com" : "myservice.example.com";
+  var domainPlaceholder = isWalletConnections ? "lightning.yourdomain.com" : (isBtcpay ? "btcpay.yourdomain.com" : "myservice.example.com");
-  var domainLabelExample = isWalletConnections ? "lightning.yourdomain.com" : "call.yourdomain.com";
+  var domainLabelExample = isWalletConnections ? "lightning.yourdomain.com" : (isBtcpay ? "btcpay.yourdomain.com" : "call.yourdomain.com");
 
   // Shared instructions (single source of truth: static/js/domain-prereqs.js) —
   // identical wording to the Server + Desktop onboarding wizard and every other
   // domain-based feature, regardless of role (Node / Desktop / Server+Desktop).
-  var hostExample = isWalletConnections ? "lightning" : "call";
+  var hostExample = isWalletConnections ? "lightning" : (isBtcpay ? "btcpay" : "call");
+  var purpose = isBtcpay ? "btcpay" : "";
 
   $domainSetupBody.innerHTML =
     '<div class="domain-setup-intro">' +
     nwcWarning +
-    renderDomainNeedsHtml({ serviceName: feat.name, hostExample: hostExample }) +
+    renderDomainNeedsHtml({ serviceName: feat.name, hostExample: hostExample, purpose: purpose }) +
     renderNjallaStepsHtml({ hostExample: hostExample, pasteHint: "below" }) +
     '<div class="onboarding-port-warn" id="domain-router-box" style="margin-top:12px;"></div>' +
-    '<p style="margin-top:10px;">Below, enter the full domain for this service — a subdomain (e.g. ' + domainLabelExample + ') or a separate domain — and paste its curl command.</p>' +
+    '<p style="margin-top:10px;">Enter the address for this service and paste the update command from Njal.la.</p>' +
     '</div>' +
-    '<div class="domain-field-group"><label class="domain-field-label" for="domain-subdomain-input">Service domain (e.g. ' + domainLabelExample + '):</label><input class="domain-field-input" type="text" id="domain-subdomain-input" placeholder="' + domainPlaceholder + '" /></div>' +
+    '<div class="domain-field-group"><label class="domain-field-label" for="domain-subdomain-input">Service address (e.g. ' + domainLabelExample + '):</label><input class="domain-field-input" type="text" id="domain-subdomain-input" placeholder="' + domainPlaceholder + '" /></div>' +
-    '<div class="domain-field-group"><label class="domain-field-label" for="domain-ddns-input">Njal.la Dynamic DNS Update Command:</label><input class="domain-field-input" type="text" id="domain-ddns-input" placeholder="curl &quot;https://njal.la/update/?h=' + domainPlaceholder + '&amp;k=abc123&amp;auto&quot;" /><p class="domain-field-hint">ℹ Paste the full curl command from your Njal.la dashboard\'s Dynamic record</p></div>' +
+    '<div class="domain-field-group"><label class="domain-field-label" for="domain-ddns-input">Njal.la Update Command:</label><input class="domain-field-input" type="text" id="domain-ddns-input" placeholder="curl &quot;https://njal.la/update/?h=' + domainPlaceholder + '&amp;k=abc123&amp;auto&quot;" /><p class="domain-field-hint">Paste the update command from Njal.la</p></div>' +
     npubField +
     '<div class="domain-field-actions"><button class="btn btn-close-modal" id="domain-setup-cancel-btn">Cancel</button><button class="btn btn-primary" id="domain-setup-save-btn">Save &amp; Enable</button></div>';
 
@@ -169,9 +170,8 @@ function openDomainReconfigureModal(feat, existingDomain, onSaved) {
 
   var nwcWarning = isWalletConnections
     ? '<div class="domain-nwc-warning">' +
-      '<strong>⚠ Lightning Wallet Connections requires its own unique hostname.</strong> ' +
+      '<strong>Lightning Wallet Connections needs its own address.</strong> ' +
-      'Use a new subdomain such as <code>lightning.yourdomain.com</code>, or a separate domain. ' +
+      'Use a new address such as <code>lightning.yourdomain.com</code>. Do not use an address already connected to another service.' +
-      'Do not reuse a domain already assigned to Matrix, Nextcloud, WordPress, BTCPay Server, Vaultwarden, Haven, or another Caddy site.' +
       '</div>'
     : '';
 

app/sovran_systemsos_web/static/js/helpers.js

@@ -144,3 +144,22 @@ async function apiFetch(path, options) {
   }
   return res.json();
 }
+
+async function apiFetchWithTimeout(path, options, timeoutMs) {
+  var controller = new AbortController();
+  var fetchOptions = Object.assign({}, options || {});
+  fetchOptions.signal = controller.signal;
+  var timer = setTimeout(function() { controller.abort(); }, timeoutMs);
+  try {
+    return await apiFetch(path, fetchOptions);
+  } catch (err) {
+    if (controller.signal.aborted) {
+      var timeoutError = new Error("Request timed out");
+      timeoutError.name = "TimeoutError";
+      throw timeoutError;
+    }
+    throw err;
+  } finally {
+    clearTimeout(timer);
+  }
+}

app/sovran_systemsos_web/static/js/rebuild.js

@@ -8,6 +8,7 @@ function openRebuildModal() {
   _rebuildLogOffset = 0;
   _rebuildServerDown = false;
   _rebuildFinished = false;
+  _rebuildPollInFlight = false;
   _rebuildPollFailures = 0;
   if ($rebuildLog) { $rebuildLog.textContent = ""; $rebuildLog.style.display = "none"; }
   var action = _rebuildIsEnabling ? "Enabling" : "Disabling";
@@ -34,6 +35,7 @@ function appendRebuildLog(text) {
 }
 
 function startRebuildPoll() {
+  if (_rebuildPollTimer) clearInterval(_rebuildPollTimer);
   pollRebuildStatus();
   _rebuildPollTimer = setInterval(pollRebuildStatus, UPDATE_POLL_INTERVAL);
 }
@@ -43,9 +45,14 @@ function stopRebuildPoll() {
 }
 
 async function pollRebuildStatus() {
-  if (_rebuildFinished) return;
+  if (_rebuildFinished || _rebuildPollInFlight) return;
+  _rebuildPollInFlight = true;
   try {
-    var data = await apiFetch("/api/rebuild/status?offset=" + _rebuildLogOffset);
+    var data = await apiFetchWithTimeout(
+      "/api/rebuild/status?offset=" + _rebuildLogOffset,
+      { cache: "no-store" },
+      STATUS_POLL_FETCH_TIMEOUT
+    );
     _rebuildPollFailures = 0;
     if (_rebuildServerDown) { _rebuildServerDown = false; }
     if (data.log) appendRebuildLog(data.log);
@@ -61,10 +68,8 @@ async function pollRebuildStatus() {
   } catch (err) {
     _rebuildPollFailures += 1;
     // The Hub restarts itself during activation, which briefly drops this poll.
-    // If polling stays broken long past a normal restart, the page's session
+    // If polling stays broken long past a normal restart, reload to
-    // almost certainly no longer matches the server (e.g. the Hub restarted
+    // re-authenticate and show the resulting feature state.
-    // and sessions were not recovered).  Reload to re-authenticate and show
-    // the real feature state instead of spinning forever.
     if (_rebuildPollFailures >= STATUS_POLL_MAX_FAILURES) {
       _rebuildFinished = true;
       stopRebuildPoll();
@@ -72,6 +77,8 @@ async function pollRebuildStatus() {
       return;
     }
     if (!_rebuildServerDown) { _rebuildServerDown = true; if ($rebuildStatus) $rebuildStatus.textContent = "Applying changes…"; }
+  } finally {
+    _rebuildPollInFlight = false;
   }
 }
 

app/sovran_systemsos_web/static/js/state.js

@@ -6,9 +6,12 @@ let _servicesCache    = [];
 let _categoryLabels   = {};
 let _updateLog        = "";
 let _updatePollTimer  = null;
+let _updatePollInFlight = false;
 let _updateLogOffset  = 0;
+let _updateVisibleLogChars = 0;
 let _serverWasDown    = false;
 let _updateFinished   = false;
+let _updateStatusUnavailable = false;
 let _updatePollFailures = 0; // consecutive failed update-status polls
 let _supportTimerInt      = null;
 let _supportEnabledAt     = null;
@@ -24,6 +27,7 @@ let _featuresData         = null;
 let _rebuildLog           = "";
 let _rebuildLogOffset     = 0;
 let _rebuildPollTimer     = null;
+let _rebuildPollInFlight  = false;
 let _rebuildFinished      = false;
 let _rebuildServerDown    = false;
 let _rebuildPollFailures  = 0; // consecutive failed rebuild-status polls
@@ -48,6 +52,7 @@ const $modalStatus    = document.getElementById("modal-status");
 const $modalLog       = document.getElementById("modal-log");
 const $btnReboot      = document.getElementById("btn-reboot");
 const $btnSave        = document.getElementById("btn-save-report");
+const $btnRetryUpdate = document.getElementById("btn-retry-update-status");
 const $btnCloseModal  = document.getElementById("btn-close-modal");
 
 const $rebootOverlay  = document.getElementById("reboot-overlay");

app/sovran_systemsos_web/static/js/tiles.js

@@ -271,10 +271,19 @@ async function checkUpdates() {
   try {
     var data = await apiFetch("/api/updates/check");
     var hasUpdates = !!data.available;
+    var updateStatus = data.status || "idle";
     var sidebarUpdateBtn = document.getElementById("sidebar-btn-update");
     var sidebarUpdateHint = document.getElementById("sidebar-update-hint");
     if (sidebarUpdateBtn) {
-      if (hasUpdates) {
+      if (updateStatus === "reboot_required") {
+        sidebarUpdateBtn.style.borderColor = "#e5a50a";
+        sidebarUpdateBtn.style.backgroundColor = "rgba(229, 165, 10, 0.10)";
+        if (sidebarUpdateHint) sidebarUpdateHint.textContent = "Restart required";
+      } else if (updateStatus === "running") {
+        sidebarUpdateBtn.style.borderColor = "#3584e4";
+        sidebarUpdateBtn.style.backgroundColor = "rgba(53, 132, 228, 0.10)";
+        if (sidebarUpdateHint) sidebarUpdateHint.textContent = "Update in progress…";
+      } else if (hasUpdates) {
         sidebarUpdateBtn.style.borderColor = "#2ec27e";
         sidebarUpdateBtn.style.backgroundColor = "rgba(46, 194, 126, 0.08)";
         if (sidebarUpdateHint) sidebarUpdateHint.textContent = "Updates available!";

app/sovran_systemsos_web/static/js/update.js

@@ -2,20 +2,45 @@
 
 // ── Update modal ──────────────────────────────────────────────────
 
-function openUpdateModal() {
+async function openUpdateModal() {
   if (!$modal) return;
-  apiFetch("/api/updates/check")
+
+  // Reattach before checking for new updates. This makes a browser reload,
+  // RDP reconnect, or suspended tab recover the authoritative systemd-backed
+  // state instead of starting over or claiming the system is merely up to date.
+  try {
+    var current = await apiFetchWithTimeout(
+      "/api/updates/status?offset=0",
+      { cache: "no-store" },
+      STATUS_POLL_FETCH_TIMEOUT
+    );
+    if (current.running || current.result === "reboot_required") {
+      showExistingUpdate(current);
+      return;
+    }
+  } catch (_) {
+    // The normal start path below has its own visible error handling.
+  }
+
+  apiFetchWithTimeout(
+    "/api/updates/check",
+    { cache: "no-store" },
+    STATUS_POLL_FETCH_TIMEOUT
+  )
     .then(function(data) {
       if (!data.available) {
         stopUpdatePoll();
         _updateLog = "";
         _updateLogOffset = 0;
+        _updateVisibleLogChars = 0;
         _updateFinished = true;
+        _updateStatusUnavailable = false;
         if ($modalLog) $modalLog.textContent = "";
         if ($modalStatus) $modalStatus.textContent = "✓ System is already up to date";
         if ($modalSpinner) $modalSpinner.classList.remove("spinning");
         if ($btnReboot) $btnReboot.style.display = "none";
         if ($btnSave) $btnSave.style.display = "none";
+        if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
         if ($btnCloseModal) $btnCloseModal.disabled = false;
         $modal.classList.add("open");
         return;
@@ -27,23 +52,69 @@ function openUpdateModal() {
     });
 }
 
-function _doOpenUpdateModal() {
+function prepareUpdateModal() {
   if (!$modal) return;
+  stopUpdatePoll();
   _updateLog = "";
   _updateLogOffset = 0;
+  _updateVisibleLogChars = 0;
+  _updatePollInFlight = false;
   _serverWasDown = false;
   _updateFinished = false;
+  _updateStatusUnavailable = false;
   _updatePollFailures = 0;
   if ($modalLog) $modalLog.textContent = "";
   if ($modalStatus) $modalStatus.textContent = "Starting update…";
   if ($modalSpinner) $modalSpinner.classList.add("spinning");
   if ($btnReboot) $btnReboot.style.display = "none";
   if ($btnSave) $btnSave.style.display = "none";
+  if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
   if ($btnCloseModal) $btnCloseModal.disabled = true;
   $modal.classList.add("open");
+}
+
+function _doOpenUpdateModal() {
+  prepareUpdateModal();
   startUpdate();
 }
 
+function showExistingUpdate(data) {
+  prepareUpdateModal();
+  if (data.log) appendLog(data.log);
+  _updateLogOffset = Number(data.offset) || 0;
+
+  if (data.running) {
+    if ($modalStatus) $modalStatus.textContent = "Updating…";
+    startUpdatePoll();
+    return;
+  }
+
+  _updateFinished = true;
+  if (data.result === "reboot_required") {
+    onUpdateDone("reboot_required");
+  } else if (data.result === "success") {
+    onUpdateDone(true);
+  } else {
+    onUpdateDone(false);
+  }
+}
+
+async function restoreUpdateModalIfNeeded() {
+  if (!$modal || $modal.classList.contains("open")) return;
+  try {
+    var data = await apiFetchWithTimeout(
+      "/api/updates/status?offset=0",
+      { cache: "no-store" },
+      STATUS_POLL_FETCH_TIMEOUT
+    );
+    if (data.running || data.result === "reboot_required") {
+      showExistingUpdate(data);
+    }
+  } catch (_) {
+    // Dashboard startup must remain usable when status cannot be reached.
+  }
+}
+
 function closeUpdateModal() {
   if (!$modal) return;
   $modal.classList.remove("open");
@@ -53,21 +124,36 @@ function closeUpdateModal() {
 function appendLog(text) {
   if (!text) return;
   _updateLog += text;
-  if ($modalLog) { $modalLog.textContent += text; $modalLog.scrollTop = $modalLog.scrollHeight; }
+  if ($modalLog) {
+    // Appending a text node avoids reparsing/replacing the complete log on
+    // every two-second poll. Trim only occasionally once the visible log is
+    // large; the complete _updateLog remains available for error reports.
+    if (_updateVisibleLogChars + text.length > UPDATE_VISIBLE_LOG_MAX_CHARS) {
+      var tail = _updateLog.slice(-UPDATE_VISIBLE_LOG_TRIM_CHARS);
+      var notice = "[Earlier update output hidden from this view; it remains in the saved report.]\n\n";
+      $modalLog.textContent = notice + tail;
+      _updateVisibleLogChars = notice.length + tail.length;
+    } else {
+      $modalLog.appendChild(document.createTextNode(text));
+      _updateVisibleLogChars += text.length;
+    }
+    $modalLog.scrollTop = $modalLog.scrollHeight;
+  }
 }
 
 function startUpdate() {
-  fetch("/api/updates/run", { method: "POST" })
+  apiFetchWithTimeout(
-    .then(function(response) {
+    "/api/updates/run",
-      if (!response.ok) return response.text().then(function(t) { throw new Error(t); });
+    { method: "POST" },
-      return response.json();
+    STATUS_POLL_FETCH_TIMEOUT * 2
-    })
+  )
     .then(function(data) {
       if (data.status === "no_updates") {
         if ($modalStatus) $modalStatus.textContent = "✓ System is already up to date";
         if ($modalSpinner) $modalSpinner.classList.remove("spinning");
         if ($btnReboot) $btnReboot.style.display = "none";
         if ($btnSave) $btnSave.style.display = "none";
+        if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
         if ($btnCloseModal) $btnCloseModal.disabled = false;
         _updateFinished = true;
         return;
@@ -83,6 +169,7 @@ function startUpdate() {
 }
 
 function startUpdatePoll() {
+  if (_updatePollTimer) clearInterval(_updatePollTimer);
   pollUpdateStatus();
   _updatePollTimer = setInterval(pollUpdateStatus, UPDATE_POLL_INTERVAL);
 }
@@ -92,33 +179,45 @@ function stopUpdatePoll() {
 }
 
 async function pollUpdateStatus() {
-  if (_updateFinished) return;
+  // setInterval does not wait for an async callback. The guard prevents a slow
+  // request from creating overlapping, out-of-order status polls.
+  if (_updateFinished || _updatePollInFlight) return;
+  _updatePollInFlight = true;
   try {
-    var data = await apiFetch("/api/updates/status?offset=" + _updateLogOffset);
+    var data = await apiFetchWithTimeout(
+      "/api/updates/status?offset=" + _updateLogOffset,
+      { cache: "no-store" },
+      STATUS_POLL_FETCH_TIMEOUT
+    );
     _updatePollFailures = 0;
     if (_serverWasDown) {
       _serverWasDown = false;
       if (!data.running) {
-        // The update finished while the server was restarting.  Reset to
+        // The update finished while the server or browser connection was away.
-        // offset 0 and re-fetch so the complete log is shown from the top.
+        // Re-fetch from offset 0 so the final result and complete tail agree.
         _updateLog = "";
         _updateLogOffset = 0;
+        _updateVisibleLogChars = 0;
         if ($modalLog) $modalLog.textContent = "";
         try {
-          var fullData = await apiFetch("/api/updates/status?offset=0");
+          var fullData = await apiFetchWithTimeout(
+            "/api/updates/status?offset=0",
+            { cache: "no-store" },
+            STATUS_POLL_FETCH_TIMEOUT
+          );
           if (fullData.log) appendLog(fullData.log);
           _updateLogOffset = fullData.offset;
-        } catch (e) {
+          data = fullData;
-          // If the re-fetch fails, fall through with whatever we have.
+        } catch (_) {
           if (data.log) appendLog(data.log);
           _updateLogOffset = data.offset;
         }
         if (data.result === "reboot_required") {
-          appendLog("[Server restarted — update completed, reboot required.]\n");
+          appendLog("[Reconnected — update completed, reboot required.]\n");
         } else if (data.result === "success") {
-          appendLog("[Server restarted — update completed successfully.]\n");
+          appendLog("[Reconnected — update completed successfully.]\n");
         } else {
-          appendLog("[Server restarted — update encountered an error.]\n");
+          appendLog("[Reconnected — update encountered an error.]\n");
         }
         _updateFinished = true;
         stopUpdatePoll();
@@ -129,7 +228,7 @@ async function pollUpdateStatus() {
         }
         return;
       }
-      appendLog("[Server reconnected]\n");
+      appendLog("[Update status reconnected]\n");
       if ($modalStatus) $modalStatus.textContent = "Updating…";
     }
     if (data.log) appendLog(data.log);
@@ -146,21 +245,57 @@ async function pollUpdateStatus() {
     }
   } catch (err) {
     _updatePollFailures += 1;
-    // Same guard as the rebuild modal: if polling stays broken long past a
-    // normal Hub restart, reload to re-authenticate and show the real state
-    // instead of spinning forever.
     if (_updatePollFailures >= STATUS_POLL_MAX_FAILURES) {
-      _updateFinished = true;
+      showUpdateStatusUnavailable();
-      stopUpdatePoll();
-      window.location.reload();
       return;
     }
-    if (!_serverWasDown) { _serverWasDown = true; appendLog("\n[Server restarting — waiting for it to come back…]\n"); if ($modalStatus) $modalStatus.textContent = "Server restarting…"; }
+    if (!_serverWasDown) {
+      _serverWasDown = true;
+      appendLog("\n[Update status connection interrupted — retrying…]\n");
+      if ($modalStatus) $modalStatus.textContent = "Reconnecting to update…";
+    }
+  } finally {
+    _updatePollInFlight = false;
+  }
+}
+
+function showUpdateStatusUnavailable() {
+  _updateFinished = true;
+  _updateStatusUnavailable = true;
+  stopUpdatePoll();
+  if ($modalSpinner) $modalSpinner.classList.remove("spinning");
+  if ($modalStatus) $modalStatus.textContent = "Update status unavailable — update may still be running";
+  appendLog("\n[The Hub could not confirm update status. The background update was not stopped. Select Retry Status after reconnecting.]\n");
+  if ($btnRetryUpdate) $btnRetryUpdate.style.display = "inline-flex";
+  if ($btnCloseModal) $btnCloseModal.disabled = false;
+}
+
+function retryUpdateStatus() {
+  if (!$modal) return;
+  _updateFinished = false;
+  _updateStatusUnavailable = false;
+  _updatePollFailures = 0;
+  _serverWasDown = true;
+  if ($modalSpinner) $modalSpinner.classList.add("spinning");
+  if ($modalStatus) $modalStatus.textContent = "Reconnecting to update…";
+  if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
+  if ($btnCloseModal) $btnCloseModal.disabled = true;
+  startUpdatePoll();
+}
+
+function resumeUpdateStatusAfterInterruption() {
+  if (!$modal || !$modal.classList.contains("open")) return;
+  if (_updateStatusUnavailable) {
+    retryUpdateStatus();
+  } else if (!_updateFinished) {
+    pollUpdateStatus();
   }
 }
 
 function onUpdateDone(result) {
+  _updateStatusUnavailable = false;
   if ($modalSpinner) $modalSpinner.classList.remove("spinning");
+  if ($btnRetryUpdate) $btnRetryUpdate.style.display = "none";
   if ($btnCloseModal) $btnCloseModal.disabled = false;
   if (result === true) {
     if ($modalStatus) $modalStatus.textContent = "✓ Update complete";
@@ -187,6 +322,7 @@ function saveErrorReport() {
   URL.revokeObjectURL(url);
 }
 
+
 // ── Reboot ────────────────────────────────────────────────────────
 
 var _rebootStartTime = 0;

app/sovran_systemsos_web/static/onboarding.js

@@ -364,17 +364,12 @@ async function loadStep3() {
       + renderDomainNeedsHtml({ serviceName: null, hostExample: "call" })
       + renderNjallaStepsHtml({ hostExample: "call", pasteHint: "next to its service below" })
       + '</div>';
-    html += '<p class="onboarding-hint">Enter each service\'s full domain — a subdomain (e.g. <code>call.yourdomain.com</code>) or a separate domain (e.g. <code>call.com</code>) — and its Njal.la DDNS curl command.</p>';
+    html += '<p class="onboarding-hint">Enter the address for each service and paste its update command from Njal.la.</p>';
 
     // Router note (the same wording is shown again, per-service, whenever a
     // domain-based feature is enabled, and lives on each service tile afterwards)
     html += '<div class="onboarding-port-warn" style="margin-bottom:16px;">'
-      + renderRouterPortsHtml({
+      + renderRouterPortsHtml({ internalIp: internalIp })
-          internalIp: internalIp,
-          plural: true,
-          includeSsh: true,
-          extraNote: "Element Call needs a few extra ports (some UDP), and you’ll be shown exactly which when you enable it.",
-        })
       + '</div>';
 
     relevantDomains.forEach(function(d) {
@@ -382,20 +377,20 @@ async function loadStep3() {
       html += '<div class="onboarding-domain-group">';
       html += '<label class="onboarding-domain-label">' + escHtml(d.label) + '</label>';
       html += '<input class="onboarding-domain-input domain-field-input" type="text" id="domain-input-' + escHtml(d.name) + '" data-domain="' + escHtml(d.name) + '" placeholder="e.g. ' + escHtml(d.name) + '.yourdomain.com" value="' + escHtml(currentVal) + '" />';
-      html += '<label class="onboarding-domain-label onboarding-domain-label--sub">Njal.la DDNS Curl Command</label>';
+      html += '<label class="onboarding-domain-label onboarding-domain-label--sub">Njal.la Update Command</label>';
       html += '<input class="onboarding-domain-input domain-field-input" type="text" id="ddns-input-' + escHtml(d.name) + '" data-ddns="' + escHtml(d.name) + '" placeholder="curl &quot;https://njal.la/update/?h=...&amp;k=...&amp;auto&quot;" />';
-      html += '<p class="onboarding-hint" style="margin-top:4px;">ℹ Paste the curl URL from your Njal.la dashboard\'s Dynamic record</p>';
+      html += '<p class="onboarding-hint" style="margin-top:4px;">Paste the update command from Njal.la</p>';
       html += '<button type="button" class="btn btn-primary onboarding-domain-save-btn" data-save-domain="' + escHtml(d.name) + '" style="align-self:flex-start;margin-top:8px;font-size:0.82rem;padding:6px 16px;">Save</button>';
       html += '<span class="onboarding-domain-save-status" id="domain-save-status-' + escHtml(d.name) + '" style="font-size:0.82rem;min-height:1.2em;"></span>';
       html += '</div>';
     });
   }
 
-  // SSL email section
+  // Contact email section
   var emailVal = domainValues["sslemail"] || "";
   html += '<div class="onboarding-domain-group onboarding-domain-group--email">';
-  html += '<label class="onboarding-domain-label">📧 SSL Certificate Email</label>';
+  html += '<label class="onboarding-domain-label">📧 Email Address</label>';
-  html += '<p class="onboarding-hint onboarding-hint--inline">Let\'s Encrypt uses this for certificate expiry notifications.</p>';
+  html += '<p class="onboarding-hint onboarding-hint--inline">Used to send important notices about your services.</p>';
   html += '<input class="onboarding-domain-input domain-field-input" type="email" id="ssl-email-input" placeholder="you@example.com" value="' + escHtml(emailVal) + '" />';
   html += '<button type="button" class="btn btn-primary onboarding-domain-save-btn" data-save-email="true" style="align-self:flex-start;margin-top:8px;font-size:0.82rem;padding:6px 16px;">Save</button>';
   html += '<span class="onboarding-domain-save-status" id="domain-save-status-email" style="font-size:0.82rem;min-height:1.2em;"></span>';

app/sovran_systemsos_web/templates/index.html

@@ -74,6 +74,7 @@
       <div class="modal-log" id="modal-log" aria-live="polite"></div>
       <div class="modal-footer">
         <button class="btn btn-save" id="btn-save-report" style="display:none">Save Error Report</button>
+        <button class="btn btn-save" id="btn-retry-update-status" style="display:none">Retry Status</button>
         <button class="btn btn-reboot" id="btn-reboot" style="display:none">Restart Entire System</button>
         <button class="btn btn-close-modal" id="btn-close-modal" disabled>Close</button>
       </div>
@@ -196,22 +197,14 @@
           encrypted messaging, password management, cloud storage, website hosting, and more.
         </p>
         <div class="upgrade-info-box">
-          <p class="upgrade-info-title">⚠ What you should know:</p>
+          <p class="upgrade-info-title">Simple, guided setup:</p>
           <ul class="upgrade-info-list">
-            <li>You will need to purchase domains for your services — <a href="https://njal.la" target="_blank" rel="noopener noreferrer">Njal.la</a> is the only supported domain provider</li>
+            <li>Sovran_SystemsOS walks you through getting your domain from <a href="https://njal.la" target="_blank" rel="noopener noreferrer">Njal.la</a> and connecting your services</li>
-            <li>Some services require ports to be opened on your router</li>
+            <li>To make your services available outside your home, complete one router task: forward ports <strong>80 and 443</strong> to this computer</li>
-          </ul>
-        </div>
-        <div class="upgrade-info-box">
-          <p class="upgrade-info-title">ℹ️ Good to know:</p>
-          <ul class="upgrade-info-list">
-            <li>Your services will be accessible via your home internet connection. Your approximate geographic area may be visible through DNS records — this is normal for all self-hosted services</li>
-            <li>Your Bitcoin node remains fully private over Tor</li>
           </ul>
         </div>
         <p class="support-desc">
-          <strong>Don't worry</strong> — the Hub will walk you through every step after the upgrade.
+          The Hub guides you through every step.
-          Domain setup, port forwarding, and configuration are all guided.
         </p>
         <p class="support-desc upgrade-rebuild-note">
           The system will rebuild after upgrading. This may take several minutes.

app/sovran_systemsos_web/templates/onboarding.html

@@ -123,10 +123,10 @@
       <div class="onboarding-panel" id="step-3" style="display:none">
         <div class="onboarding-step-header">
           <span class="onboarding-step-icon">🌐</span>
-          <h2 class="onboarding-step-title">Domain Configuration</h2>
+          <h2 class="onboarding-step-title">Connect Your Services</h2>
           <p class="onboarding-step-desc">
-            Sovran_SystemsOS uses <strong><a href="https://njal.la" target="_blank" style="color: var(--accent-color);">Njal.la</a></strong> for domains and Dynamic DNS, and your router needs ports <strong>80</strong> and <strong>443</strong> (TCP) forwarded to this computer.
+            Sovran_SystemsOS walks you through getting your domain from <strong><a href="https://njal.la" target="_blank" style="color: var(--accent-color);">Njal.la</a></strong> and connecting your services.
-            Everything you need — Njal.la account, Dynamic records, and the one router task — is laid out step by step below.
+            To make your services available outside your home, you’ll complete one router task: forward ports <strong>80</strong> and <strong>443</strong> to this computer.
           </p>
         </div>
         <div class="onboarding-card" id="step-3-body">

app/sovran_systemsos_web/update_state.py

@@ -0,0 +1,88 @@
+"""Update-state helpers for the Sovran Hub.
+
+The full-system updater stages a NixOS generation with ``nixos-rebuild boot``.
+That generation is not active until the machine reboots — and the same is true
+for updates started from a terminal or an SSH support session, which never go
+near the Hub's status files.
+
+The ONLY reliable indicator that a reboot is pending is NixOS itself: the
+system profile (``/nix/var/nix/profiles/system``), which ``nixos-rebuild``
+points at the newest generation on every ``boot`` AND every ``switch``, versus
+``/run/current-system``, the generation actually running since the last boot.
+When the two differ, a staged generation has not been booted yet.
+
+Earlier revisions reconstructed this from a marker file and log tails written
+by the Hub's own updater.  Any system updated by other means — or whose
+``REBOOT_REQUIRED`` status was written by an updater older than the marker
+feature — left the Hub showing "Restart required" forever: the recorded
+generation could never equal the (since advanced) running one, so the marker
+could never be cleared.
+
+This module has no FastAPI or systemd dependencies so the policy can be tested
+without importing the Hub server.
+"""
+
+from __future__ import annotations
+
+import os
+
+# The NixOS system profile.  ``nixos-rebuild boot`` and ``nixos-rebuild
+# switch`` both add a generation here; ``boot`` additionally makes it the
+# bootloader default.  The path is a symlink chain (``system`` ->
+# ``system-N-link`` -> ``/nix/store/...-nixos-system-...``).
+BOOT_PROFILE_PATH = "/nix/var/nix/profiles/system"
+CURRENT_SYSTEM_PATH = "/run/current-system"
+
+
+def reboot_is_pending(
+    boot_profile_path: str = BOOT_PROFILE_PATH,
+    current_system_path: str = CURRENT_SYSTEM_PATH,
+) -> bool:
+    """Return whether a staged NixOS generation has not been booted yet.
+
+    This is deliberately independent of how the update was started — Hub
+    "Update System", terminal ``nixos-rebuild boot``, or a support session all
+    move the system profile the same way:
+
+    * after ``nixos-rebuild boot``:  profile -> new, current -> old  → pending
+    * after rebooting:               both -> new                     → cleared
+    * after ``nixos-rebuild switch``: both move together             → no reboot
+      ever needed (switch activates immediately)
+    * after a rollback:              both point at the rollback target → cleared
+
+    Unreadable or missing paths are treated as "not pending": the Hub must
+    never demand a reboot it cannot substantiate.
+    """
+    try:
+        boot_default = os.path.realpath(boot_profile_path)
+        current = os.path.realpath(current_system_path)
+    except OSError:
+        return False
+    if not os.path.exists(boot_default) or not os.path.exists(current):
+        return False
+    return boot_default != current
+
+
+def effective_update_status(
+    status: str,
+    boot_profile_path: str = BOOT_PROFILE_PATH,
+    current_system_path: str = CURRENT_SYSTEM_PATH,
+) -> str:
+    """Map a persisted Hub status to the one that reflects live NixOS state.
+
+    Only ``REBOOT_REQUIRED`` is re-validated: it means "the update staged a
+    generation the machine has not booted into", a claim that must stay true
+    no matter which tool performed the last update.  When the boot default IS
+    the running system the claim is stale — the staged generation booted, was
+    superseded by a newer update, or the marker was written by an updater that
+    could never clear it — so the effective status is ``IDLE``.
+
+    All other statuses (``RUNNING``, ``FAILED``, ``SUCCESS``, ``IDLE``) pass
+    through unchanged; RUNNING staleness is handled separately against the
+    systemd unit itself.
+    """
+    if status == "REBOOT_REQUIRED" and not reboot_is_pending(
+        boot_profile_path, current_system_path
+    ):
+        return "IDLE"
+    return status

app/sovran_systemsos_web/versions.json

@@ -5,9 +5,9 @@
   "bitcoind.service": "27.1.0",
   "electrs.service": "0.10.6",
   "lnd.service": "0.18.0",
-  "rtl.service": "0.15.8",
+  "rtl.service": "0.15.10",
   "btcpayserver.service": "2.4.2",
-  "albyhub.service": "1.8.0",
+  "albyhub.service": "1.24.0",
   "mempool.service": "3.2.1",
   "matrix-synapse.service": "1.115.0",
   "livekit.service": "1.5.2",

flake.lock

@@ -5,11 +5,11 @@
         "nixpkgs": "nixpkgs"
       },
       "locked": {
-        "lastModified": 1785778881,
+        "lastModified": 1787839684,
-        "narHash": "sha256-yHJJTm3o7ZhiVp68G+hh+dz1GKV+ACOT6mQj+eYvJLQ=",
+        "narHash": "sha256-xTafqsVs2/zyGsaCel41QT5Lpbs/rG56Kp+YsmLXL8I=",
         "owner": "emmanuelrosa",
         "repo": "btc-clients-nix",
-        "rev": "fc1aca94d839f82e7501fe0d01279d5c1fba6e63",
+        "rev": "ef81159876930802ef9861aa5c039cec6886a30f",
         "type": "github"
       },
       "original": {
@@ -26,11 +26,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1782949081,
+        "lastModified": 1785627969,
-        "narHash": "sha256-vp6Y/Grm98ESt6ceOkWiHWyZRDV3J1RID4w+6NWK9yA=",
+        "narHash": "sha256-4dtXQk/NMePegK/nWp5NSeuZKLATItOq61lpEvmXqGw=",
         "owner": "hercules-ci",
         "repo": "flake-parts",
-        "rev": "17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e",
+        "rev": "427bf4bd9435fdf21321c8cc628c24efc14c0f7a",
         "type": "github"
       },
       "original": {
@@ -56,11 +56,11 @@
     },
     "nixpkgs-stable": {
       "locked": {
-        "lastModified": 1786201459,
+        "lastModified": 1787753485,
-        "narHash": "sha256-CiOTEjmwAmG2AWnaIno9YaCJJmpca2FXPhMAsnrolCg=",
+        "narHash": "sha256-BZWCi9ZRJiARTuKTbbtvFTj7t1TK4G3UEckT3HyNfRg=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "8b8c811c7c2541c30382c5de7ed26be055569c60",
+        "rev": "062346a6d85bc4b49dfaa61c986e9c5be21217d1",
         "type": "github"
       },
       "original": {
@@ -72,11 +72,11 @@
     },
     "nixpkgs_2": {
       "locked": {
-        "lastModified": 1786106723,
+        "lastModified": 1787498568,
-        "narHash": "sha256-zDSUbpoeo/9ZmD2+wXnzxoo1+uhL8vxc0b8yuYMKYq0=",
+        "narHash": "sha256-9i/VTdusq/+NM/tz+J1Re+ojkMB8MBf0QshnYfzHz30=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "f13ff45afd1bb73e640eaa08a7066dbed07e3238",
+        "rev": "56c02bc00adcf003215cc4bd996d6efaf4cff188",
         "type": "github"
       },
       "original": {
@@ -88,11 +88,11 @@
     },
     "nixpkgs_3": {
       "locked": {
-        "lastModified": 1784555310,
+        "lastModified": 1787364730,
-        "narHash": "sha256-/FCliTPgiuV1owejZFNx3Ch9irdvkOfOFl+HHZ+DrtM=",
+        "narHash": "sha256-NcYt9QJfpJiF1lAyN8BDPB4EeScbPU+EwQqPiBElrpU=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "421eebfd0ec7bccd4abe826ce62d7e6e83129493",
+        "rev": "a831408e6378bc02ebf8cc09b52c96ca86f6bab4",
         "type": "github"
       },
       "original": {
@@ -109,11 +109,11 @@
         "systems": "systems"
       },
       "locked": {
-        "lastModified": 1785763201,
+        "lastModified": 1787691219,
-        "narHash": "sha256-wA373y/B9orM3HatLu9oS+Ke5lmdZyBl/bdjd2gLMq4=",
+        "narHash": "sha256-CjeJTOjrluiDuL0W/YBFkSBOQjBYmdKwj7hWAJ4PIh0=",
         "owner": "nix-community",
         "repo": "nixvim",
-        "rev": "c7be49306b23a952c0151cf4bbeacd944ed82f2a",
+        "rev": "ebec1ae277a50b16e7d6682edd009ef585d68261",
         "type": "github"
       },
       "original": {
@@ -131,6 +131,7 @@
       }
     },
     "systems": {
+      "flake": false,
       "locked": {
         "lastModified": 1774449309,
         "narHash": "sha256-brhZ8DmuGtzkCYHJg4HEd602amKm89Y9ytsFZ5uWD1w=",

iso/installer.py

@@ -423,40 +423,6 @@ class InstallerWindow(Adw.ApplicationWindow):
         sep.set_margin_end(40)
         outer.append(sep)
 
-        notice_frame = Gtk.Frame()
-        notice_frame.add_css_class("card")
-        notice_frame.set_margin_start(40)
-        notice_frame.set_margin_end(40)
-        notice_frame.set_margin_top(20)
-        notice_frame.set_margin_bottom(4)
-
-        notice_box = Gtk.Box(orientation=Gtk.Orientation.HORIZONTAL, spacing=12)
-        notice_box.set_margin_top(12)
-        notice_box.set_margin_bottom(12)
-        notice_box.set_margin_start(16)
-        notice_box.set_margin_end(16)
-
-        notice_icon = symbolic_icon("dialog-information-symbolic")
-        notice_icon.set_valign(Gtk.Align.START)
-        notice_box.append(notice_icon)
-
-        notice_lbl = Gtk.Label()
-        notice_lbl.set_use_markup(True)
-        notice_lbl.set_wrap(True)
-        notice_lbl.set_xalign(0)
-        notice_lbl.set_halign(Gtk.Align.FILL)
-        notice_lbl.set_markup(
-            "<span weight='bold'>Heads up — Server + Desktop prerequisites</span>\n"
-            "• A domain or subdomain from <span weight='bold'>https://njal.la</span>\n"
-            "• The ability to open / forward ports on your router\n\n"
-            "Don't worry — after install, the onboarding wizard walks you through every step.\n"
-            "<span size='small'>Desktop Only and Node Only do not require a domain or port forwarding.</span>"
-        )
-        notice_box.append(notice_lbl)
-
-        notice_frame.set_child(notice_box)
-        outer.append(notice_frame)
-
         if self.virtualization:
             vm_frame = Gtk.Frame()
             vm_frame.add_css_class("card")
@@ -505,7 +471,7 @@ class InstallerWindow(Adw.ApplicationWindow):
         # Role cards
         roles = [
             ("Server + Desktop",
-             "Full sovereign experience: beautiful desktop, your own cloud, secure messaging, Bitcoin node, and more.",
+             "Full sovereignty: host your own websites, cloud, chat, passwords, and Bitcoin services instead of relying on Big Tech. Sovran_SystemsOS walks you through getting your domain from Njal.la and connecting everything. One router task is required: forward ports 80 and 443 to this computer.",
              "Server+Desktop"),
             ("Desktop Only",
              "A beautiful, easy-to-use desktop without the background server applications.",

modules/bitcoin/bitcoind.nix

@@ -284,7 +284,9 @@ let
   nbLib = config.nix-bitcoin.lib;
   secretsDir = config.nix-bitcoin.secretsDir;
 
-  i2pSAM = config.services.i2pd.proto.sam;
+  # nixpkgs 26.11 moved i2pd's protocol configuration from
+  # `services.i2pd.proto` to the RFC42-style `services.i2pd.settings`.
+  i2pSAM = config.services.i2pd.settings.sam;
 
   configFile = builtins.toFile "bitcoin.conf" ''
     # We're already logging via journald
@@ -374,7 +376,11 @@ in {
 
     services.i2pd = mkIf (cfg.i2p != false) {
       enable = true;
-      proto.sam.enable = true;
+      settings.sam = {
+        enabled = true;
+        address = "127.0.0.1";
+        port = 7656;
+      };
     };
 
     systemd.tmpfiles.rules = [

modules/bitcoin/lnd.nix

@@ -150,7 +150,7 @@ let
   nbLib = config.nix-bitcoin.lib;
   secretsDir = config.nix-bitcoin.secretsDir;
   runAsUser = config.nix-bitcoin.runAsUserCmd;
-  lndinit = "${(pkgs.callPackage ../../packages/lndinit {})}/bin/lndinit";
+  lndinit = "${pkgs.lndinit}/bin/lndinit";
 
   bitcoind = config.services.bitcoind;
 

modules/core/njalla.nix

@@ -33,7 +33,7 @@
       # /var/lib/njalla/ddns_urls.json.
       NoNewPrivileges    = true;
       ProtectSystem      = "strict";
-      ReadWritePaths     = [ "/var/lib/njalla" ];
+      ReadWritePaths     = [ "/var/lib/njalla" "/var/lib/secrets" ];
       ReadOnlyPaths      = [ "/etc/sovran" ];
       ProtectHome        = true;
       PrivateTmp         = true;
@@ -88,12 +88,15 @@ try:
 except Exception:
     sys.exit(0)  # no URLs configured — nothing to do
 
-# Resolve current public IP once
+# Resolve current public IP via the shared detector — one script, one cache
+# (STUN -> DNS -> opt-in HTTPS echo; see /var/lib/sovran/public-ip.py).
+# The detector refreshes /var/lib/secrets/external-ip, which the Hub and
+# LiveKit read as well, so the whole system shares a single detected value.
 public_ip = ""
 try:
     r = subprocess.run(
-        ["dig", "@resolver4.opendns.com", "myip.opendns.com", "+short", "-4"],
+        [sys.executable, "/var/lib/sovran/public-ip.py", "check"],
-        capture_output=True, text=True, timeout=10,
+        capture_output=True, text=True, timeout=20,
     )
     raw = r.stdout.strip().splitlines()[0] if r.stdout.strip() else ""
     ipaddress.ip_address(raw)  # validates — raises if not a real IP
@@ -101,6 +104,16 @@ try:
 except Exception:
     pass
 
+if not public_ip:
+    # Last resort: the shared cache file, if the detector is unavailable.
+    try:
+        with open("/var/lib/secrets/external-ip") as f:
+            raw = f.read().strip()
+        ipaddress.ip_address(raw)
+        public_ip = raw
+    except Exception:
+        pass
+
 if not public_ip:
     sys.exit(0)  # no IP resolved — skip to avoid sending bare ''${IP}
 

modules/core/public-ip.nix

@@ -0,0 +1,323 @@
+# ── Unified public-IP detection (privacy-first) ─────────────────────────────
+#
+# One script, one cache file, every consumer on the system reads the same
+# value. Previously the public IP was detected independently in three places,
+# each phoning home to a different third party:
+#   * the Hub (server.py _get_external_ip) → api.ipify.org / ifconfig.me /
+#     icanhazip.com over HTTPS on every /api/network call and every
+#     background-loop tick
+#   * DDNS (ddns-update.py) → myip.opendns.com via OpenDNS
+#   * LiveKit → STUN (its own embedded detection)
+#
+# This module replaces all of that with a single script
+# (/var/lib/sovran/public-ip.py) that detects the IP once per TTL using the
+# least-exposing mechanism available, and caches it in
+# /var/lib/secrets/external-ip. Consumers (Hub, DDNS, LiveKit) read the cache
+# and only invoke the script when it is missing or stale.
+#
+# Detection chain (first success wins, stops immediately):
+#   1. pin         — sovran_systemsOS.elementCalling.externalIP (baked in)
+#   2. cache       — /var/lib/secrets/external-ip if newer than cacheTTL
+#   3. STUN        — UDP binding request (one packet, no application data,
+#                    no HTTP metadata; the same protocol every WebRTC client
+#                    uses). Server configurable via publicIP.stunServer.
+#   4. DNS         — "myip.opendns.com" A query via publicIP.dnsResolver
+#                    (single DNS query, no HTTP headers)
+#   5. HTTPS echo  — ONLY endpoints listed in publicIP.httpsEcho (empty by
+#                    default → never contacted)
+#
+# Privacy property: while the cache is fresh, zero third parties are
+# contacted. When detection runs, at most ONE party learns the IP per
+# refresh interval (default 5 minutes), and the STUN/DNS mechanisms expose
+# nothing beyond the bare address.
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}:
+
+let
+  stunServer  = config.sovran_systemsOS.publicIP.stunServer;
+  stunPort    = config.sovran_systemsOS.publicIP.stunPort;
+  dnsResolver = config.sovran_systemsOS.publicIP.dnsResolver;
+  httpsEcho   = config.sovran_systemsOS.publicIP.httpsEcho;
+  cacheTTL    = config.sovran_systemsOS.publicIP.cacheTTL;
+
+  # Optional pin shared with element-calling (baked in at build time).
+  pin = if config.sovran_systemsOS.elementCalling.externalIP != null then config.sovran_systemsOS.elementCalling.externalIP else "";
+
+  echoList = lib.concatStringsSep "," (map (u: "'${u}'") httpsEcho);
+in
+{
+  options.sovran_systemsOS.publicIP = {
+    stunServer = lib.mkOption {
+      type = lib.types.str;
+      default = "stun.l.google.com";
+      description = ''
+        STUN server used to discover the public IP over UDP. STUN is the most
+        privacy-preserving detection mechanism: a single stateless packet,
+        no HTTP metadata. Only used when the cache is stale.
+      '';
+    };
+    stunPort = lib.mkOption {
+      type = lib.types.port;
+      default = 19302;
+    };
+    dnsResolver = lib.mkOption {
+      type = lib.types.str;
+      default = "resolver4.opendns.com";
+      description = ''
+        DNS resolver used as fallback (myip.opendns.com trick) when STUN is
+        unavailable (e.g. ISP blocks UDP egress). A single DNS query, no
+        HTTP headers.
+      '';
+    };
+    httpsEcho = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ ];
+      example = [ "https://api.ipify.org" ];
+      description = ''
+        OPT-IN HTTPS endpoints that return the caller's public IP as a bare
+        IPv4 literal. Each listed endpoint observes this server's public IP
+        and HTTP metadata every time detection runs. Empty by default — no
+        HTTPS echo service is ever contacted unless you add one here. This is
+        the last-resort fallback after STUN and DNS.
+      '';
+    };
+    cacheTTL = lib.mkOption {
+      type = lib.types.int;
+      default = 300;
+      description = "Seconds the detected public IP is cached before re-detection.";
+    };
+  };
+
+  # ── Install the unified detector ──────────────────────────────────────────
+  # This module declares `options` above, so ALL configuration must go under
+  # the `config` attribute: NixOS forbids mixing bare top-level settings
+  # (like `system.*`) with the `options`/`config` keyword attributes in the
+  # same module. (Fixes: "Module ... has an unsupported attribute `system'".)
+  config.system.activationScripts.sovranPublicIpInstall = lib.stringAfter [ "users" ] ''
+    install -d -m 0755 /var/lib/sovran
+    cat > /var/lib/sovran/public-ip.py <<'PYEOF'
+#!/usr/bin/env python3
+"""sovran-public-ip — one detector, one cache, every consumer reads the same IP.
+
+Privacy-first detection chain (first success wins):
+  1. pin     — baked in from sovran_systemsOS.elementCalling.externalIP
+  2. cache   — /var/lib/secrets/external-ip if newer than CACHE_TTL seconds
+  3. STUN    — UDP binding request (one packet, no application data)
+  4. DNS     — myip.opendns.com A query via the configured resolver
+  5. HTTPS   — ONLY endpoints baked in from publicIP.httpsEcho (opt-in)
+
+Usage:
+  public-ip.py check     print current public IP (cache first; refresh if stale)
+  public-ip.py refresh   force re-detection, update the cache file, print IP
+
+Exit status: 0 with the IP on stdout on success; 1 if no IP is available
+(cached value, if any, is still printed to stdout with a warning on stderr).
+"""
+import ipaddress
+import os
+import random
+import socket
+import struct
+import sys
+import time
+import urllib.request
+
+CACHE_FILE = "/var/lib/secrets/external-ip"
+PIN        = "${pin}"
+STUN_SERVER = "${stunServer}"
+STUN_PORT   = ${toString stunPort}
+DNS_RESOLVER = "${dnsResolver}"
+DNS_HOST     = "myip.opendns.com"
+ECHO_URLS    = [ ${echoList} ]
+CACHE_TTL    = ${toString cacheTTL}
+TIMEOUT      = 3.0
+
+# ---------------------------------------------------------------------------
+# Detection primitives
+# ---------------------------------------------------------------------------
+
+def is_usable_ip(text: str) -> bool:
+    """True if text is a globally routable IPv4 that LiveKit may advertise."""
+    try:
+        ip = ipaddress.ip_address(text)
+    except ValueError:
+        return False
+    if ip.version != 4:
+        return False
+    if (ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_multicast
+            or ip.is_reserved or ip.is_unspecified or not ip.is_global):
+        return False
+    # RFC 6598 shared (CGNAT) space — not reachable from the internet.
+    if ip in ipaddress.ip_network("100.64.0.0/10"):
+        return False
+    return True
+
+
+def stun_public_ip() -> str | None:
+    """RFC 5389 Binding request over UDP; returns the mapped (public) IPv4."""
+    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    sock.settimeout(TIMEOUT)
+    try:
+        txid = random.randbytes(12)
+        req = struct.pack("!HHI", 0x0001, 0, 0) + txid  # Binding request
+        sock.sendto(req, (STUN_SERVER, STUN_PORT))
+        data, _ = sock.recvfrom(2048)
+    except OSError:
+        return None
+    finally:
+        sock.close()
+
+    if len(data) < 20:
+        return None
+    mtype, _mlen = struct.unpack("!HH", data[:4])
+    if mtype != 0x0101:  # Binding success response
+        return None
+
+    cookie = data[4:8]
+    i = 20
+    while i + 4 <= len(data):
+        atype, alen = struct.unpack("!HH", data[i : i + 4])
+        aval = data[i + 4 : i + 4 + alen]
+        if atype in (0x0001, 0x0020) and len(aval) >= 8:  # MAPPED / XOR-MAPPED
+            family = aval[1]
+            if family == 0x01:  # IPv4
+                raw = aval[4:8]
+                if atype == 0x0020:  # XOR with magic cookie + txid prefix
+                    raw = bytes(b ^ c for b, c in zip(raw, cookie + txid[:4]))
+                return socket.inet_ntop(socket.AF_INET, raw)
+        i += 4 + ((alen + 3) // 4) * 4
+    return None
+
+
+def dns_public_ip() -> str | None:
+    """Minimal DNS A query for myip.opendns.com against the given resolver."""
+    qid = random.randint(0, 0xFFFF)
+    qname = b"".join(bytes([len(p)]) + p.encode() for p in DNS_HOST.split(".")) + b"\x00"
+    query = struct.pack("!HHHHHH", qid, 0x0100, 1, 0, 0, 0) + qname + struct.pack("!HH", 1, 1)
+    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    sock.settimeout(TIMEOUT)
+    try:
+        sock.sendto(query, (DNS_RESOLVER, 53))
+        data, _ = sock.recvfrom(4096)
+    except OSError:
+        return None
+    finally:
+        sock.close()
+
+    try:
+        if len(data) < 12:
+            return None
+        rid, _flags, _qd, an, _ns, _ar = struct.unpack("!HHHHHH", data[:12])
+        if rid != qid or an == 0:
+            return None
+        i = 12
+        for _ in range(_qd):  # skip question
+            while data[i] != 0:
+                i += 1 + data[i]
+            i += 5
+        for _ in range(an):
+            if data[i] & 0xC0 == 0xC0:
+                i += 2
+            else:
+                while data[i] != 0:
+                    i += 1 + data[i]
+                i += 1
+            rtype, _rclass, _ttl, rdlen = struct.unpack("!HHIH", data[i : i + 10])
+            i += 10
+            if rtype == 1 and rdlen == 4:
+                return socket.inet_ntop(socket.AF_INET, data[i : i + 4])
+            i += rdlen
+    except (IndexError, struct.error):
+        return None
+    return None
+
+
+def echo_public_ip() -> str | None:
+    """Opt-in HTTPS echo endpoints (baked in at build time; empty by default)."""
+    for url in ECHO_URLS:
+        try:
+            req = urllib.request.Request(url, headers={"User-Agent": "sovran-public-ip"})
+            with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
+                text = resp.read().decode().strip()
+            if is_usable_ip(text):
+                return text
+        except Exception:
+            continue
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Cache handling
+# ---------------------------------------------------------------------------
+
+def read_cache() -> str:
+    try:
+        with open(CACHE_FILE) as f:
+            return f.read().strip()
+    except OSError:
+        return ""
+
+
+def write_cache(ip: str) -> None:
+    try:
+        os.makedirs(os.path.dirname(CACHE_FILE), exist_ok=True)
+        tmp = f"{CACHE_FILE}.tmp"
+        with open(tmp, "w") as f:
+            f.write(ip + "\n")
+        os.replace(tmp, CACHE_FILE)
+    except OSError:
+        pass
+
+
+def cache_fresh() -> bool:
+    try:
+        return time.time() - os.path.getmtime(CACHE_FILE) < CACHE_TTL
+    except OSError:
+        return False
+
+
+def detect() -> str:
+    """Run the chain; returns usable IP or an empty string."""
+    if PIN and is_usable_ip(PIN):
+        return PIN
+    for fn in (stun_public_ip, dns_public_ip, echo_public_ip):
+        try:
+            cand = fn()
+        except Exception:
+            continue
+        if cand and is_usable_ip(cand):
+            return cand
+    return ""
+
+
+def main() -> int:
+    force = len(sys.argv) > 1 and sys.argv[1] == "refresh"
+    ip = ""
+    if not force and cache_fresh():
+        ip = read_cache()
+    if not ip:
+        ip = detect()
+        if ip:
+            write_cache(ip)
+        else:
+            stale = read_cache()
+            if stale:
+                print(stale)
+                print("WARNING: detection failed; using last known public IP", file=sys.stderr)
+                return 0
+            print("ERROR: could not determine a public IP (STUN/DNS unreachable)", file=sys.stderr)
+            return 1
+    print(ip)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
+PYEOF
+    chmod 0555 /var/lib/sovran/public-ip.py
+  '';
+}

modules/core/roles.nix

@@ -79,6 +79,41 @@
       };
     };
 
+    # ── Element Calling (video/audio) tuning ──────────────────
+    elementCalling = {
+      fullAccessHomeservers = lib.mkOption {
+        type = lib.types.listOf lib.types.str;
+        default = [ ];
+        example = [ "matrix.peer.example.com" ];
+        description = ''
+          Additional Matrix server_names (beyond this server itself) that may
+          trigger LiveKit room creation on this server's SFU via lk-jwt-service.
+
+          Not needed for the common federated setup: each participant's client
+          always obtains its token from its own homeserver's JWT service and
+          publishes to its own SFU, and the participant who starts a call
+          creates the room on their own SFU — the remote user merely joins
+          (joining does not require full access).
+
+          Only set this for asymmetric cases: e.g. a peer homeserver that has
+          no focus of its own, or calls whose first participant lands on this
+          server's SFU but belongs to the peer.
+        '';
+      };
+      externalIP = lib.mkOption {
+        type = lib.types.nullOr lib.types.str;
+        default = null;
+        example = "203.0.113.10";
+        description = ''
+          Optional pin: force LiveKit to advertise this public IPv4 in its
+          host/TURN ICE candidates. Not required in normal operation — the
+          module auto-detects the public IP at runtime (HTTPS egress
+          detection, falling back to STUN). Set it only to override a
+          mis-detected address (e.g. multi-WAN/VPN setups).
+        '';
+      };
+    };
+
     # ── Domain setup registry ─────────────────────────────────
     domainRequirements = lib.mkOption {
       type = lib.types.listOf (lib.types.submodule {

modules/core/sovran-hub.nix

@@ -135,16 +135,12 @@ let
     "bitcoind.service" = if pkgs ? bitcoind then pkgs.bitcoind.version else "27.1.0";
     "electrs.service" = if pkgs ? electrs then pkgs.electrs.version else "0.10.6";
     "lnd.service" = if pkgs ? lnd then pkgs.lnd.version else "0.18.0";
-    # Keep the fallbacks aligned with the vendored packages used by the
+    # Evaluate locally vendored packages directly so we don't accidentally
-    # service modules (RTL 0.15.8 and Mempool 3.2.1).  The nixpkgs attrs are
+    # pick up older/different versions from upstream nixpkgs.
-    # optional because these packages are built locally in this repository.
+    "rtl.service" = (pkgs.callPackage ../../packages/rtl { fetchNodeModules = pkgs.callPackage ../../packages/build-support/fetch-node-modules.nix {}; }).version;
-    "rtl.service" = if pkgs ? clightning-rtl then pkgs.clightning-rtl.version else (if pkgs ? rtl then pkgs.rtl.version else "0.15.8");
-    # BTCPay Server is intentionally sourced from pkgs.stable by the service
-    # module.  Read the configured package here rather than pkgs.btcpayserver
-    # (unstable), otherwise the Hub can advertise a version that is not running.
     "btcpayserver.service" = lib.getVersion config.services.btcpayserver.package;
-    "albyhub.service" = if pkgs ? albyhub then pkgs.albyhub.version else "1.8.0";
+    "albyhub.service" = (pkgs.callPackage ../../packages/albyhub {}).version;
-    "mempool.service" = if pkgs ? mempool then pkgs.mempool.version else "3.2.1";
+    "mempool.service" = (pkgs.callPackage ../../packages/mempool { fetchNodeModules = pkgs.callPackage ../../packages/build-support/fetch-node-modules.nix {}; }).version;
     "matrix-synapse.service" = if pkgs ? matrix-synapse then pkgs.matrix-synapse.version else "1.115.0";
     "livekit.service" = if pkgs ? livekit then pkgs.livekit.version else "1.5.2";
     "vaultwarden.service" = if pkgs ? vaultwarden then pkgs.vaultwarden.version else "1.32.0";
@@ -160,8 +156,10 @@ let
 
     LOG="/var/log/sovran-hub-update.log"
     STATUS="/var/log/sovran-hub-update.status"
+    GENERATION="/var/log/sovran-hub-update.generation"
 
     echo "RUNNING" > "$STATUS"
+    rm -f "$GENERATION"
     : > "$LOG"
     exec > >(tee -a "$LOG") 2>&1
 
@@ -196,6 +194,12 @@ let
       if [ "$BOOT_RC" -ne 0 ]; then
         echo "[ERROR] nixos-rebuild boot failed"
         RC=1
+      elif ! readlink -f /nix/var/nix/profiles/system > "$GENERATION"; then
+        # The marker is informational only.  The Hub derives pending-reboot
+        # state from the NixOS system profile itself, so failing to record
+        # the marker must not fail an otherwise successful update.
+        echo "[WARNING] update succeeded but its staged generation could not be recorded"
+        rm -f "$GENERATION"
       fi
       echo ""
     fi

modules/element-calling.nix

@@ -136,9 +136,10 @@ EOF
     unitConfig = {
       ConditionPathExists = "/var/lib/domains/element-calling";
     };
-    path = [ pkgs.coreutils pkgs.findutils pkgs.iproute2 pkgs.gawk ];
+    path = [ pkgs.coreutils pkgs.findutils pkgs.iproute2 pkgs.gawk pkgs.python3 ];
     script = ''
       MATRIX=$(cat /var/lib/domains/matrix)
+      ELEMENT_CALLING=$(cat /var/lib/domains/element-calling)
 
       mkdir -p /run/livekit
 
@@ -188,6 +189,56 @@ EOF
       # rtc.interfaces.includes are only known at runtime, so they are
       # substituted here. The cert/key paths point at the LoadCredential-staged
       # copies under /run/credentials.
+      #
+      # Determine the public IPv4 to advertise in LiveKit ICE candidates.
+      # Remote peers must be able to reach this address, so it must be the
+      # server's public IP — or the router's WAN IP when the server is behind
+      # NAT with port-forwarding. It does not need to be assigned to this box,
+      # and it may be dynamic.
+      #
+      # Reuse the shared detector (/var/lib/sovran/public-ip.py — see
+      # modules/core/public-ip.nix) instead of running our own: one script,
+      # one cache, privacy-first (STUN -> DNS -> opt-in HTTPS echo). Priority:
+      #   1. sovran_systemsOS.elementCalling.externalIP (explicit pin, if set)
+      #   2. /var/lib/secrets/external-ip (the shared cache)
+      #   3. run the detector now (it refreshes the cache)
+      #   4. STUN auto-detection (use_external_ip) as the fallback, with a
+      #      warning — this is where broken installs used to silently end up
+      #      advertising a private IP, causing "call connects but no video".
+      EXTERNAL_IP='${if config.sovran_systemsOS.elementCalling.externalIP != null then config.sovran_systemsOS.elementCalling.externalIP else ""}'
+
+      PUBLIC_IP="$EXTERNAL_IP"
+      if [ -z "$PUBLIC_IP" ] && [ -f /var/lib/secrets/external-ip ]; then
+        PUBLIC_IP=$(tr -d '[:space:]' < /var/lib/secrets/external-ip 2>/dev/null)
+      fi
+      if [ -z "$PUBLIC_IP" ] && [ -x /var/lib/sovran/public-ip.py ]; then
+        PUBLIC_IP=$(python3 /var/lib/sovran/public-ip.py check 2>/dev/null | head -n1)
+      fi
+
+      # Reject non-routable addresses (loopback, private, link-local, CGNAT).
+      # A detected/pinned address like this must never be advertised.
+      if [ -n "$PUBLIC_IP" ] && printf '%s' "$PUBLIC_IP" | grep -qE \
+        '^(0\.|127\.|10\.|100\.64\.|169\.254\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)'; then
+        echo "WARNING: external IP '$PUBLIC_IP' is not routable; falling back to STUN auto-detection." >&2
+        PUBLIC_IP=""
+      fi
+
+      if [ -n "$PUBLIC_IP" ]; then
+        cat > /run/livekit/livekit.yaml <<EOF
+port: 7880
+rtc:
+  use_external_ip: false
+  node_ip: $PUBLIC_IP
+  tcp_port: 7881
+  udp_port: 7882
+  port_range_start: 30000
+  port_range_end: 40000
+  interfaces:
+    includes:
+      - $IFACE
+EOF
+        echo "LiveKit will advertise public IP: $PUBLIC_IP"
+      else
         cat > /run/livekit/livekit.yaml <<EOF
 port: 7880
 rtc:
@@ -200,6 +251,20 @@ rtc:
   interfaces:
     includes:
       - $IFACE
+EOF
+        echo "WARNING: could not determine a public IP for LiveKit; using STUN auto-detection. If calls connect without media, check STUN egress or set sovran_systemsOS.elementCalling.externalIP." >&2
+      fi
+
+      # Webhooks → lk-jwt-service. The JWT service validates the HMAC
+      # signature against the same key file it issues tokens with, and uses
+      # the events (participant_left / room_finished) to detect abruptly
+      # disconnected participants instead of waiting for the delayed-event
+      # timeout. The URL hits local Caddy via the /etc/hosts loopback
+      # override and is routed to the JWT service by the element-calling
+      # vhost (/livekit/jwt/sfu_webhook → 8073).
+      LK_KEY=$(cut -d: -f1 < ${livekitKeyFile} | tr -d '[:space:]')
+
+      cat >> /run/livekit/livekit.yaml <<EOF
 room:
   auto_create: false
 turn:
@@ -209,6 +274,10 @@ turn:
   udp_port: 3478
   cert_file: /run/credentials/livekit.service/turn-cert
   key_file: /run/credentials/livekit.service/turn-key
+webhook:
+  api_key: $LK_KEY
+  urls:
+    - https://$ELEMENT_CALLING/livekit/jwt/sfu_webhook
 EOF
 
       chmod 644 /run/livekit/livekit.yaml
@@ -216,24 +285,17 @@ EOF
   };
 
   ####### LIVEKIT SERVICE #######
+  # NOTE: the runtime config (rtc ports, TURN, webhook, node_ip) is generated
+  # by livekit-turn-setup and delivered via LoadCredential; the upstream
+  # module's `settings` block is therefore intentionally NOT used (it would
+  # be dead config that silently diverges from what LiveKit actually loads).
+  # The firewall ports are opened explicitly below; openFirewall is left off
+  # so the upstream module does not also open 7880/tcp publicly (Caddy fronts
+  # the SFU on this host).
   services.livekit = {
     enable = true;
-    openFirewall = true;
+    openFirewall = false;
     keyFile = livekitKeyFile;
-    settings = {
-      rtc.use_external_ip = true;
-      rtc.skip_external_ip_validation = true;
-      rtc.tcp_port = 7881;
-      rtc.udp_port = 7882;
-      rtc.port_range_start = 30000;
-      rtc.port_range_end = 40000;
-      room.auto_create = false;
-      turn = {
-        enabled = true;
-        tls_port = 5349;
-        udp_port = 3478;
-      };
-    };
   };
 
   # Override ExecStart to load the runtime-generated config (which carries the
@@ -277,12 +339,26 @@ EOF
     script = ''
       ELEMENT_CALLING=$(cat /var/lib/domains/element-calling)
       MATRIX=$(cat /var/lib/domains/matrix)
+      FULL_ACCESS_HOMESERVERS="$MATRIX"
+
+      # Federated peers may also be granted LiveKit room-creation (full access)
+      # on this SFU via sovran_systemsOS.elementCalling.fullAccessHomeservers.
+      # Without this, remote users can join existing calls but cannot be the
+      # first to start one on your SFU.
+      EXTRA_HS='${lib.concatStringsSep "," config.sovran_systemsOS.elementCalling.fullAccessHomeservers}'
+      if [ -n "$EXTRA_HS" ]; then
+        FULL_ACCESS_HOMESERVERS="$FULL_ACCESS_HOMESERVERS,$EXTRA_HS"
+      fi
 
       mkdir -p /run/lk-jwt-service
 
       cat > /run/lk-jwt-service/env <<EOF
 LIVEKIT_URL=wss://$ELEMENT_CALLING
-LIVEKIT_FULL_ACCESS_HOMESERVERS=$MATRIX
+LIVEKIT_FULL_ACCESS_HOMESERVERS=$FULL_ACCESS_HOMESERVERS
+# Re-check, every 60s, that connected participants are still on the SFU;
+# guards against missed SFU webhooks (e.g. an SFU restart) leaving stale
+# call members in Matrix rooms.
+LIVEKIT_SANITY_CHECK_INTERVAL_SECONDS=60
 EOF
 
       chmod 640 /run/lk-jwt-service/env
@@ -294,6 +370,9 @@ EOF
     enable = true;
     port = 8073;
     keyFile = livekitKeyFile;
+    # Required by the upstream module's option type, but overridden at runtime
+    # by EnvironmentFile (/run/lk-jwt-service/env, generated above from the
+    # element-calling domain). Kept as a harmless placeholder.
     livekitUrl = "wss://placeholder.local";
   };
 
@@ -301,6 +380,126 @@ EOF
     "/run/lk-jwt-service/env"
   ];
 
+  # Restart LiveKit / lk-jwt-service when a rebuild regenerates their runtime
+  # configs (new domains, externalIP, full-access list), mirroring the domain
+  # change flow.
+  # Re-run the config generator and restart LiveKit when a rebuild regenerates
+  # the runtime config, or when the Hub persists a new external IP (dynamic
+  # WAN IPs), so the advertised ICE candidate stays current without a manual
+  # restart. The trigger chain: external-ip change → livekit-turn-setup
+  # re-runs → rewrites livekit.yaml → livekit restarts with the new config.
+  systemd.services.livekit-turn-setup.restartTriggers = [ "/var/lib/secrets/external-ip" ];
+  systemd.services.livekit.restartTriggers = [ "/run/livekit/livekit.yaml" ];
+  systemd.services.lk-jwt-service.restartTriggers = [ "/run/lk-jwt-service/env" ];
+
+  ####### PUBLIC REACHABILITY SELF-CHECK #######
+  # Diagnostic only — never a hard dependency of livekit/caddy. Catches the
+  # classic "call connects but no media" setup errors at boot instead of at
+  # call time:
+  #   * the element-calling domain having no public records (or resolving to
+  #     loopback/link-local/CGNAT for remote peers),
+  #   * the lk-jwt-service being unreachable through Caddy,
+  #   * the MatrixRTC transports endpoint being absent (Element X cannot
+  #     discover calling and shows MISSING_MATRIX_RTC_TRANSPORT).
+  # The check queries only the operator's own DNS provider (the domain's
+  # authoritative nameservers, resolved via the local resolver) plus the
+  # server's own Caddy and public IP — no third-party resolver or service is
+  # contacted. dig queries resolvers directly, so the /etc/hosts loopback
+  # overrides (modules/core/local-domain-loopback.nix) do not influence the
+  # result.
+  systemd.services.element-calling-public-check = {
+    description = "Verify Element Calling domain, JWT service and MatrixRTC transports endpoint are publicly reachable";
+    after = [ "network-online.target" "caddy.service" "livekit.service" "lk-jwt-service.service" ];
+    wants = [ "network-online.target" ];
+    wantedBy = [ "multi-user.target" ];
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+    };
+    unitConfig = {
+      ConditionPathExists = "/var/lib/domains/element-calling";
+    };
+    path = [ pkgs.coreutils pkgs.gawk pkgs.dnsutils pkgs.curl ];
+    script = ''
+      ELEMENT_CALLING=$(cat /var/lib/domains/element-calling)
+      MATRIX=$(cat /var/lib/domains/matrix)
+      FAIL=0
+
+      echo "── Element Calling public reachability self-check ──"
+
+      # 1) Authoritative DNS view (bypassing the /etc/hosts loopback
+      #    overrides). Resolve the domain's own nameservers via the local
+      #    resolver, then query those nameservers directly — the only party
+      #    that sees the query is the DNS provider the operator already uses
+      #    for the domain.
+      NS_LIST=$(dig +short NS "$ELEMENT_CALLING" 2>/dev/null | tr '\n' ' ')
+      if [ -n "$NS_LIST" ]; then
+        IPS=""
+        for NSRV in $NS_LIST; do
+          IPS=$( { dig +short A "$ELEMENT_CALLING" "@$NSRV" 2>/dev/null; dig +short AAAA "$ELEMENT_CALLING" "@$NSRV" 2>/dev/null; } | tr '\n' ' ' )
+          [ -n "$IPS" ] && break
+        done
+        echo "Authoritative nameservers for $ELEMENT_CALLING: $NS_LIST"
+      else
+        echo "WARNING: could not resolve nameservers for $ELEMENT_CALLING via the local resolver; using the local resolver's answer instead." >&2
+        IPS=$( { dig +short A "$ELEMENT_CALLING" 2>/dev/null; dig +short AAAA "$ELEMENT_CALLING" 2>/dev/null; } | tr '\n' ' ' )
+      fi
+
+      if [ -z "$IPS" ]; then
+        echo "ERROR: no A/AAAA records for $ELEMENT_CALLING at its authoritative nameservers. Remote peers cannot reach this LiveKit; calls will connect without media." >&2
+        FAIL=1
+      else
+        echo "Public DNS for $ELEMENT_CALLING: $IPS"
+        for IP in $IPS; do
+          case "$IP" in
+            0.*|127.*|169.254.*|100.64.*|::1|fe80:*|fc*:*|fd*:*)
+              echo "ERROR: $ELEMENT_CALLING resolves to $IP (loopback/link-local/CGNAT). Remote peers cannot reach it." >&2
+              FAIL=1 ;;
+          esac
+        done
+      fi
+
+      # 2) lk-jwt-service healthz through Caddy (validates the proxy chain).
+      if curl -fsS --max-time 10 "https://$ELEMENT_CALLING/livekit/jwt/healthz" >/dev/null 2>&1; then
+        echo "OK: https://$ELEMENT_CALLING/livekit/jwt/healthz responds"
+      else
+        echo "ERROR: https://$ELEMENT_CALLING/livekit/jwt/healthz not reachable through Caddy." >&2
+        FAIL=1
+      fi
+
+      # 3) Same healthz via the first public IP (tests the full NAT path).
+      #    NOTE: if this box is behind the same NAT you are testing through,
+      #    routers without hairpin NAT will fail this step — the warning is
+      #    then expected and harmless; verify from an external device instead.
+      if [ -n "$IPS" ]; then
+        PUBIP=$(echo "$IPS" | awk '{print $1}')
+        if curl -fsS --max-time 15 --resolve "$ELEMENT_CALLING:443:$PUBIP" "https://$ELEMENT_CALLING/livekit/jwt/healthz" >/dev/null 2>&1; then
+          echo "OK: healthz reachable via public IP $PUBIP (NAT path works)"
+        else
+          echo "WARNING: healthz NOT reachable via public IP $PUBIP — check router port-forwarding (443/TCP) and NAT hairpin. Expected if the router lacks hairpin NAT; verify from an external device." >&2
+        fi
+      fi
+
+      # 4) MatrixRTC transports registry (MSC4519) — required by Element X.
+      CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "https://$MATRIX/_matrix/client/unstable/org.matrix.msc4143/rtc/transports")
+      case "$CODE" in
+        401|200)
+          echo "OK: MatrixRTC transports endpoint present (HTTP $CODE; auth required is expected)" ;;
+        404)
+          echo "ERROR: /_matrix/client/unstable/org.matrix.msc4143/rtc/transports missing (HTTP 404) — Element X cannot discover calling. Enable msc4143_enabled and matrix_rtc.transports in Synapse." >&2
+          FAIL=1 ;;
+        *)
+          echo "WARNING: transports endpoint returned HTTP $CODE" >&2 ;;
+      esac
+
+      if [ "$FAIL" -eq 1 ]; then
+        echo "── Element Calling self-check FAILED — see errors above ──" >&2
+        exit 1
+      fi
+      echo "── Element Calling self-check passed ──"
+    '';
+  };
+
   ####### SYNAPSE RUNTIME CONFIG (element-calling additions) #######
   systemd.services.element-calling-synapse-config = {
     description = "Generate Synapse runtime config for Element Calling";
@@ -317,6 +516,7 @@ EOF
     path = [ pkgs.coreutils ];
     script = ''
       MATRIX=$(cat /var/lib/domains/matrix)
+      ELEMENT_CALLING=$(cat /var/lib/domains/element-calling)
 
       mkdir -p /run/matrix-synapse
 
@@ -326,7 +526,17 @@ public_baseurl: "https://$MATRIX"
 serve_server_wellknown: true
 experimental_features:
   msc3266_enabled: true
+  # MSC4143: enables the MatrixRTC transports registry endpoint
+  # (/_matrix/client/unstable/org.matrix.msc4143/rtc/transports, MSC4519).
+  # Element X requires this endpoint to discover the LiveKit focus; without it
+  # mobile clients fail with MISSING_MATRIX_RTC_TRANSPORT / cannot start calls.
+  msc4143_enabled: true
   msc4222_enabled: true
+# MSC4519: advertise this site's LiveKit focus via the transports registry.
+matrix_rtc:
+  transports:
+    - type: livekit
+      livekit_service_url: "https://$ELEMENT_CALLING/livekit/jwt"
 max_event_delay_duration: "24h"
 rc_message:
   per_second: 0.5

modules/modules.nix

@@ -17,6 +17,7 @@
     ./core/no-sleep.nix
     ./core/cpu-performance.nix
     ./core/local-domain-loopback.nix
+    ./core/public-ip.nix
 
     # ── Always on (no flag) ───────────────────────────────────
     ./php.nix

modules/nwc-wallets.nix

@@ -3,13 +3,8 @@
 let
   albyHubPort = 18080;
   albyHubApiBase = "http://127.0.0.1:${toString albyHubPort}";
-  patchedAlbyHub = pkgs.albyhub.overrideAttrs (old: {
+  vendoredAlbyHub = pkgs.callPackage ../packages/albyhub {};
-    patches = (old.patches or []) ++ [
+  patchedAlbyHub = vendoredAlbyHub;
-      ../packages/albyhub/0001-private-route-hints.patch
-      ../packages/albyhub/0002-isolated-invoice-app-id.patch
-      ../packages/albyhub/0003-loopback-bind-host.patch
-    ];
-  });
 
   lndRpcAddress = lib.attrByPath [ "services" "lnd" "rpcAddress" ] "127.0.0.1" config;
   lndRpcPort = toString (lib.attrByPath [ "services" "lnd" "rpcPort" ] 10009 config);

packages/albyhub/0001-private-route-hints.patch

@@ -1,13 +0,0 @@
-diff --git a/lnclient/lnd/lnd.go b/lnclient/lnd/lnd.go
-index bad532f..ba35b7b 100644
---- a/lnclient/lnd/lnd.go
-+++ b/lnclient/lnd/lnd.go
-@@ -708,7 +708,7 @@ func (svc *LNDService) MakeInvoice(ctx context.Context, amountMsat int64, descri
- 		DescriptionHash: descriptionHashBytes,
- 		Expiry:          expiry,
- 		RouteHints:      hints,
--		Private:         !hasPublicChannels, // use private channel hints in the invoice
-+		Private:         true, // always include private channel hints in the invoice
- 	}
- 
- 	resp, err := svc.client.AddInvoice(ctx, addInvoiceRequest)

packages/albyhub/0002-isolated-invoice-app-id.patch

@@ -1,71 +0,0 @@
-diff --git a/api/models.go b/api/models.go
-index 28bd117..dceb79a 100644
---- a/api/models.go
-+++ b/api/models.go
-@@ -46,7 +46,7 @@ type API interface {
- 	ListTransactions(ctx context.Context, appId *uint, limit uint64, offset uint64) (*ListTransactionsResponse, error)
- 	ListOnchainTransactions(ctx context.Context) ([]OnchainTransaction, error)
- 	SendPayment(ctx context.Context, invoice string, amountMsat *uint64, metadata map[string]interface{}, fromAppId *uint) (*SendPaymentResponse, error)
--	CreateInvoice(ctx context.Context, amountMsat uint64, description string) (*MakeInvoiceResponse, error)
-+	CreateInvoice(ctx context.Context, amountMsat uint64, description string, appId *uint) (*MakeInvoiceResponse, error)
- 	LookupInvoice(ctx context.Context, paymentHash string) (*LookupInvoiceResponse, error)
- 	SetTransactionUserLabels(ctx context.Context, id uint, labels map[string]string) error
- 	RequestMempoolApi(ctx context.Context, endpoint string) (interface{}, error)
-@@ -586,6 +586,7 @@ type MakeInvoiceRequest struct {
- 	AmountSat   *uint64 `json:"amountSat"`
- 	AmountMsat  *uint64 `json:"amountMsat"`
- 	Description string  `json:"description"`
-+	AppId       *uint   `json:"appId"`
- }
- 
- type ResetRouterRequest struct {
-diff --git a/api/transactions.go b/api/transactions.go
-index aaf9380..3f9850a 100644
---- a/api/transactions.go
-+++ b/api/transactions.go
-@@ -12,12 +12,12 @@ import (
- 	"github.com/sirupsen/logrus"
- )
- 
--func (api *api) CreateInvoice(ctx context.Context, amountMsat uint64, description string) (*MakeInvoiceResponse, error) {
-+func (api *api) CreateInvoice(ctx context.Context, amountMsat uint64, description string, appId *uint) (*MakeInvoiceResponse, error) {
- 	lnClient := api.svc.GetLNClient()
- 	if lnClient == nil {
- 		return nil, ErrLNClientNotStarted
- 	}
--	transaction, err := api.svc.GetTransactionsService().MakeInvoice(ctx, amountMsat, description, "", 0, nil, lnClient, nil, nil, nil)
-+	transaction, err := api.svc.GetTransactionsService().MakeInvoice(ctx, amountMsat, description, "", 0, nil, lnClient, appId, nil, nil)
- 	if err != nil {
- 		return nil, err
- 	}
-diff --git a/http/http_service.go b/http/http_service.go
-index bf650b9..cf61250 100644
---- a/http/http_service.go
-+++ b/http/http_service.go
-@@ -687,7 +687,12 @@ func (httpSvc *HttpService) makeInvoiceHandler(c echo.Context) error {
- 		amountMsat = *resolvedAmountMsat
- 	}
- 
--	invoice, err := httpSvc.api.CreateInvoice(c.Request().Context(), amountMsat, makeInvoiceRequest.Description)
-+	invoice, err := httpSvc.api.CreateInvoice(
-+		c.Request().Context(),
-+		amountMsat,
-+		makeInvoiceRequest.Description,
-+		makeInvoiceRequest.AppId,
-+	)
- 
- 	if err != nil {
- 		return c.JSON(http.StatusInternalServerError, ErrorResponse{
-diff --git a/wails/wails_handlers.go b/wails/wails_handlers.go
-index d25259b..c9fe4e8 100644
---- a/wails/wails_handlers.go
-+++ b/wails/wails_handlers.go
-@@ -600,7 +600,7 @@ func (app *WailsApp) WailsRequestRouter(route string, method string, body string
- 		if resolvedAmountMsat != nil {
- 			amountMsat = *resolvedAmountMsat
- 		}
--		invoice, err := app.api.CreateInvoice(ctx, amountMsat, makeInvoiceRequest.Description)
-+		invoice, err := app.api.CreateInvoice(ctx, amountMsat, makeInvoiceRequest.Description, nil)
- 		if err != nil {
- 			return WailsRequestRouterResponse{Body: nil, Error: err.Error()}
- 		}

packages/albyhub/0003-loopback-bind-host.patch

@@ -1,37 +0,0 @@
-diff --git a/cmd/http/main.go b/cmd/http/main.go
-index ed31f96..4d31212 100644
---- a/cmd/http/main.go
-+++ b/cmd/http/main.go
-@@ -2,8 +2,8 @@ package main
- 
- import (
- 	"context"
--	"fmt"
- 	nethttp "net/http"
-+	"net"
- 	"os"
- 	"os/signal"
- 	"syscall"
-@@ -51,8 +51,9 @@ func main() {
- 	httpSvc := http.NewHttpService(svc, svc.GetEventPublisher())
- 	httpSvc.RegisterSharedRoutes(e)
- 	//start Echo server
-+	bindAddress := net.JoinHostPort(svc.GetConfig().GetEnv().Host, svc.GetConfig().GetEnv().Port)
- 	go func() {
--		if err := e.Start(fmt.Sprintf(":%v", svc.GetConfig().GetEnv().Port)); err != nil && err != nethttp.ErrServerClosed {
-+		if err := e.Start(bindAddress); err != nil && err != nethttp.ErrServerClosed {
- 			logger.Logger.WithError(err).Error("echo server failed to start")
- 			cancel()
- 		}
-diff --git a/config/models.go b/config/models.go
-index 0fb8870..dc11afd 100644
---- a/config/models.go
-+++ b/config/models.go
-@@ -24,6 +24,7 @@ type AppConfig struct {
- 	LNDAddress                         string `envconfig:"LND_ADDRESS"`
- 	LNDCertFile                        string `envconfig:"LND_CERT_FILE"`
- 	LNDMacaroonFile                    string `envconfig:"LND_MACAROON_FILE"`
-+	Host                               string `envconfig:"HOST" default:"127.0.0.1"`
- 	Workdir                            string `envconfig:"WORK_DIR"`
- 	Port                               string `envconfig:"PORT" default:"8080"`
- 	DatabaseUri                        string `envconfig:"DATABASE_URI" default:"nwc.db"`

packages/albyhub/default.nix

@@ -0,0 +1,78 @@
+{ lib
+, buildGoModule
+, fetchFromGitHub
+, stdenv
+}:
+
+buildGoModule rec {
+  pname = "albyhub";
+  version = "1.24.0";
+
+  # sovran fork = upstream v1.24.0 + LND-only, no frontend, HOST bind,
+  # always-private route hints (the customization lives in the fork's commits)
+  src = fetchFromGitHub {
+    owner = "naturallaw777";
+    repo = "hub";
+    tag = "sovran-1.24.0";
+    # round 1: copy the "got: sha256-..." from the build error
+    hash = "sha256-9l7xK7iA1p9qQRClgNV+eYg/vYJxbwBiRObeLRpyfDo=";
+  };
+
+  # `go mod vendor` strips the secp256k1-zkp cgo headers (include/), so use
+  # the full module cache. Round 2: copy the "got: sha256-..." from the error
+  proxyVendor = true;
+  vendorHash = "sha256-U0NLVV0gH2IERqaGsJwE2f+iZQ35D+czH8mUKXSM4Wc=";
+
+  subPackages = [ "cmd/http" ];
+
+  # LND client needs cgo (secp256k1-zkp)
+  buildInputs = [ (lib.getLib stdenv.cc.cc) ];
+
+  # pin module downloads to a hermetic proxy and the local toolchain
+  # (otherwise GOPROXY/GOTOOLCHAIN can leak into the builder env and the
+  # cache comes back incomplete — the original failure)
+  overrideModAttrs = (finalAttrs: previousAttrs: {
+    modBuildPhase = ''
+      runHook preBuild
+
+      export GIT_SSL_CAINFO=$NIX_SSL_CERT_FILE
+      export GOPROXY=https://proxy.golang.org,direct
+      export GOSUMDB=sum.golang.org
+      export GOTOOLCHAIN=local
+
+      mkdir -p "$GOPATH/pkg/mod/cache/download"
+      go mod download all
+
+      export GOPROXY="file://$GOPATH/pkg/mod/cache/download"
+
+      go list ./cmd/http
+
+      mkdir -p vendor
+
+      runHook postBuild
+    '';
+  });
+
+  ldflags = [
+    "-X github.com/getAlby/hub/version.Tag=${version}"
+    "-s"
+    "-w"
+  ];
+
+  postInstall = ''
+    mv $out/bin/http $out/bin/albyhub
+  '';
+
+   preFixup = ''
+    # Go's external linker bakes the sandbox build dir into the RPATH;
+    # rewrite it to the real store paths before the fixup audit
+    patchelf --set-rpath ${
+      lib.makeLibraryPath [
+        (lib.getLib stdenv.cc.cc)
+        (lib.getLib stdenv.cc.libc)
+      ]
+    } $out/bin/albyhub
+  '';
+
+  meta.mainProgram = "albyhub";
+}

packages/lndinit/default.nix

@@ -1,19 +0,0 @@
-{ lib, buildGoModule, fetchFromGitHub }:
-
-buildGoModule rec {
-  pname = "lndinit";
-  version = "0.1.3-beta";
-  src = fetchFromGitHub {
-    owner = "lightninglabs";
-    repo = pname;
-    rev = "v${version}";
-    sha256 = "sha256-sO1DpbppCurxr9g9nUl9Vx82FJK1mTcUw3rY1Fm1wEU=";
-  };
-  vendorHash = "sha256-El44BS5Bu0K/klMxkajciU/R6uqiXBMOiLN536QztbE=";
-  subPackages = [ "." ];
-  meta = with lib; {
-    description = "Wallet initializer for lnd (from nix-bitcoin)";
-    homepage = "https://github.com/lightninglabs/lndinit";
-    license = licenses.mit;
-  };
-}

packages/rtl/default.nix

@@ -10,11 +10,11 @@
 }:
 let self = stdenvNoCC.mkDerivation {
   pname = "rtl";
-  version = "0.15.8";
+  version = "0.15.10";
 
   src = fetchurl {
     url = "https://github.com/Ride-The-Lightning/RTL/archive/refs/tags/v${self.version}.tar.gz";
-    hash = "sha256-8XdGyORxB2dkZRB/Yl7zh+Quqo4L/Y0VmC6Brbr/hqU=";
+    hash = "sha256-r5riYV2FN0OKi0mwj9I1jBeeU1LOv2HVB6CEovPlUuY=";
   };
 
   passthru = {
@@ -26,7 +26,7 @@ let self = stdenvNoCC.mkDerivation {
       # TODO-EXTERNAL: Remove `npmFlags` when no longer required
       # See: https://github.com/Ride-The-Lightning/RTL/issues/1182
       npmFlags = "--legacy-peer-deps";
-      hash = "sha256-oMqd6nLzS6iQ9w4z2yzpR2unA5qhOq5YdvfoS8IgYLY=";
+      hash = "sha256-NKiWcjqYcHBVIB+vbF3aKXLe2fJRmh/quu8obztP3TA=";
     };
   };
 

tests/test_update_state.py

@@ -0,0 +1,222 @@
+"""Regression tests for Hub update completion and polling recovery."""
+
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+
+_REPO_ROOT = Path(__file__).resolve().parents[1]
+_APP_PARENT = _REPO_ROOT / "app"
+if str(_APP_PARENT) not in sys.path:
+    sys.path.insert(0, str(_APP_PARENT))
+
+from sovran_systemsos_web.update_state import (  # noqa: E402
+    effective_update_status,
+    reboot_is_pending,
+)
+
+
+# Real store paths observed on the incident machine that prompted this rework.
+RUNNING_GENERATION = (
+    "84rsiqi66nc68jbikd26ms50ap831xf8-nixos-system-nixos-26.11.20260817.ec2d622"
+)
+PREVIOUS_GENERATION = (
+    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-nixos-system-nixos-26.11.20260816.old"
+)
+# What the stale Hub log still claimed was staged: a two-week-old update cycle.
+HUB_LOG_GENERATION = (
+    "yis0saq6p8fhqcaii0h2yzqf0blhdwns-nixos-system-nixos-26.11.20260804.e72e4f2"
+)
+
+
+class TestRebootPendingState(unittest.TestCase):
+    """Reboot-pending state is derived from NixOS, not from Hub marker files.
+
+    The system profile (what boots next) is compared against
+    /run/current-system (what is running).  This stays correct no matter
+    which tool performed the update: Hub "Update System", a terminal
+    ``nixos-rebuild``, or an SSH support session.
+    """
+
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        root = Path(self.tmp.name)
+        self.store = root / "store"
+        self.store.mkdir()
+        # /nix/var/nix/profiles/system is a two-hop chain: the diagnostics on
+        # the incident box showed ``system`` -> ``system-148-link`` -> store.
+        self.profile = root / "system"
+        self.profile_entry = root / "system-148-link"
+        # /run/current-system links straight to the running store path.
+        self.current = root / "current-system"
+
+    def _store_dir(self, name: str) -> str:
+        path = self.store / name
+        path.mkdir(exist_ok=True)
+        return str(path)
+
+    def _stage(self, booted: str, running: str) -> None:
+        """Stage ``booted`` as the boot default with ``running`` live."""
+        os.symlink(self._store_dir(booted), self.profile_entry)
+        os.symlink(self.profile_entry, self.profile)
+        os.symlink(self._store_dir(running), self.current)
+
+    def test_hub_update_staged_and_not_yet_rebooted_is_pending(self):
+        self._stage(booted=RUNNING_GENERATION, running=PREVIOUS_GENERATION)
+        self.assertTrue(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_staged_generation_booted_is_no_longer_pending(self):
+        self._stage(booted=RUNNING_GENERATION, running=RUNNING_GENERATION)
+        self.assertFalse(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_terminal_switch_needs_no_reboot(self):
+        # nixos-rebuild switch moves the profile AND current-system together.
+        self._stage(booted=RUNNING_GENERATION, running=RUNNING_GENERATION)
+        self.assertFalse(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_rollback_leaves_nothing_pending(self):
+        # nixos-rebuild switch --rollback points both at the rollback target.
+        self._stage(booted=PREVIOUS_GENERATION, running=PREVIOUS_GENERATION)
+        self.assertFalse(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_unverifiable_state_is_never_a_reboot_demand(self):
+        # No profile and no running system readable: the Hub must not nag
+        # about a reboot it cannot substantiate.
+        self.assertFalse(
+            reboot_is_pending(str(self.profile), str(self.current))
+        )
+
+    def test_stale_hub_marker_clears_after_terminal_updates(self):
+        """The exact incident: terminal-updated machine, frozen Hub marker.
+
+        The user's last Hub update (old updater, weeks prior) left
+        REBOOT_REQUIRED behind.  Every update since ran in a terminal and
+        never touched the Hub's files, so the log still records a staged
+        generation from 2026-08-04 while the machine runs a 2026-08-17
+        build.  With the profile and the running system in agreement, the
+        stale claim must reconcile to IDLE regardless of anything the old
+        marker/log files say.
+        """
+        root = Path(self.tmp.name)
+        log = root / "sovran-hub-update.log"
+        log.write_text(
+            "Done. The new configuration is "
+            f"/nix/store/{HUB_LOG_GENERATION}\n"
+            "✓ Update staged successfully\n",
+            encoding="utf-8",
+        )
+        # Deliberately no sovran-hub-update.generation marker: the updater
+        # that produced this state predates the marker feature.
+        self.assertFalse(
+            (root / "sovran-hub-update.generation").exists()
+        )
+
+        self._stage(booted=RUNNING_GENERATION, running=RUNNING_GENERATION)
+        self.assertEqual(
+            effective_update_status(
+                "REBOOT_REQUIRED", str(self.profile), str(self.current)
+            ),
+            "IDLE",
+        )
+
+    def test_genuine_pending_reboot_claim_survives(self):
+        # A staged generation that has NOT been booted yet: the claim is
+        # true and must keep surfacing until the reboot really happens.
+        self._stage(booted=RUNNING_GENERATION, running=PREVIOUS_GENERATION)
+        self.assertEqual(
+            effective_update_status(
+                "REBOOT_REQUIRED", str(self.profile), str(self.current)
+            ),
+            "REBOOT_REQUIRED",
+        )
+
+    def test_other_statuses_pass_through_unchanged(self):
+        self._stage(booted=RUNNING_GENERATION, running=RUNNING_GENERATION)
+        for status in ("RUNNING", "FAILED", "SUCCESS", "IDLE"):
+            self.assertEqual(
+                effective_update_status(
+                    status, str(self.profile), str(self.current)
+                ),
+                status,
+            )
+
+
+class TestUpdatePollingWiring(unittest.TestCase):
+    """Guard the browser failure modes that caused a permanent spinner."""
+
+    @classmethod
+    def setUpClass(cls):
+        js_dir = _REPO_ROOT / "app" / "sovran_systemsos_web" / "static" / "js"
+        cls.update_js = (js_dir / "update.js").read_text(encoding="utf-8")
+        cls.rebuild_js = (js_dir / "rebuild.js").read_text(encoding="utf-8")
+        cls.helpers_js = (js_dir / "helpers.js").read_text(encoding="utf-8")
+        cls.events_js = (js_dir / "events.js").read_text(encoding="utf-8")
+        cls.template = (
+            _REPO_ROOT / "app" / "sovran_systemsos_web" / "templates" / "index.html"
+        ).read_text(encoding="utf-8")
+
+    def test_status_fetches_have_an_abort_timeout(self):
+        self.assertIn("function apiFetchWithTimeout", self.helpers_js)
+        self.assertIn("new AbortController()", self.helpers_js)
+        self.assertIn("controller.abort()", self.helpers_js)
+        self.assertIn("apiFetchWithTimeout(", self.update_js)
+        self.assertIn("STATUS_POLL_FETCH_TIMEOUT", self.update_js)
+
+    def test_async_update_polls_cannot_overlap(self):
+        self.assertIn("_updatePollInFlight", self.update_js)
+        self.assertIn(
+            "if (_updateFinished || _updatePollInFlight) return;", self.update_js
+        )
+        self.assertIn("finally", self.update_js)
+
+    def test_connection_failure_has_explicit_non_running_ui(self):
+        self.assertIn("showUpdateStatusUnavailable", self.update_js)
+        self.assertIn("Update status unavailable", self.update_js)
+        self.assertIn("Retry Status", self.template)
+        self.assertIn("retryUpdateStatus", self.events_js)
+
+    def test_rdp_or_tab_resume_forces_reconciliation(self):
+        self.assertIn("resumeUpdateStatusAfterInterruption", self.events_js)
+        self.assertIn('window.addEventListener("focus"', self.events_js)
+        self.assertIn('document.addEventListener("visibilitychange"', self.events_js)
+
+    def test_verbose_log_rendering_is_bounded(self):
+        self.assertIn("UPDATE_VISIBLE_LOG_MAX_CHARS", self.update_js)
+        self.assertIn("document.createTextNode(text)", self.update_js)
+        self.assertNotIn("$modalLog.textContent += text", self.update_js)
+
+    def test_page_reload_restores_running_or_completed_update(self):
+        self.assertIn("restoreUpdateModalIfNeeded", self.update_js)
+        self.assertIn("await restoreUpdateModalIfNeeded();", self.events_js)
+        self.assertIn('current.result === "reboot_required"', self.update_js)
+
+    def test_all_javascript_remains_syntax_valid(self):
+        node = shutil.which("node")
+        if not node:
+            self.skipTest("node is not available in this test environment")
+        for script in (
+            _REPO_ROOT / "app" / "sovran_systemsos_web" / "static" / "js"
+        ).glob("*.js"):
+            result = subprocess.run(
+                [node, "--check", str(script)], capture_output=True, text=True
+            )
+            self.assertEqual(result.returncode, 0, result.stderr)
+
+
+if __name__ == "__main__":
+    unittest.main()