Merge pull request #397 from naturallaw777/fix/security-error-hardening

fix: harden error handling and sanitize exception details across secu…

SECURITY.md

@@ -0,0 +1,104 @@
+# Security Policy (SECURITY.md)
+
+This document outlines the security policy, vulnerability disclosure process, threat model, and security best practices for **Sovran_SystemsOS**.
+
+---
+
+## 1. Supported Versions
+
+We actively monitor and patch security vulnerabilities in Sovran_SystemsOS. Because the operating system is built on [NixOS](https://nixos.org), stable releases receive security updates backported from upstream Nixpkgs as well as our custom software updates.
+
+| Version | Supported | Notes |
+|:---|:---:|:---|
+| **1.0.x** (Stable) |  Yes | Active stable release line. Patches are backported regularly. |
+| **staging-dev** |  Yes | Development line; updated frequently but may contain untested code. |
+| **Legacy (< 1.0.0)** |  No | Please upgrade to the latest stable release to ensure you have active security updates. |
+
+---
+
+## 2. Reporting a Vulnerability
+
+We take the security of self-custody systems, private clouds, and communications servers extremely seriously. If you discover a vulnerability, **please do not open a public issue, forum thread, or submit a public pull request.** 
+
+Instead, report vulnerabilities privately through one of the following secure channels:
+
+1. **Email:** Send an encrypted or plain-text email to [support@sovransystems.com](mailto:support@sovransystems.com).
+   - If sending sensitive information, please contact us first to establish a secure PGP-encrypted communication channel.
+2. **GitHub Private Vulnerability Reporting:** If you are viewing this on GitHub, you can report vulnerabilities privately via the **Security** tab of this repository under **Vulnerability reporting**.
+
+### What to Include in a Report:
+To help us triage and resolve the issue quickly, please include:
+- A clear description of the vulnerability and its potential impact.
+- Step-by-step instructions to reproduce the issue (including any scripts or configuration snippets).
+- The version of Sovran_SystemsOS you tested (found in the Hub header or `/etc/nixos/VERSION`).
+- Any potential remediation steps or suggestions you may have.
+
+### Our Response and Disclosure Process:
+1. **Acknowledgment:** We will acknowledge receipt of your report within **24–48 hours** and provide a tracking reference.
+2. **Triage:** We will investigate and verify the vulnerability privately. We may reach out to you for clarifying details.
+3. **Remediation:** If verified, we will develop a patch on a private security branch.
+4. **Release:** We will coordinate a release date with you and publish the patch to Gitea `stable` and GitHub `main`.
+5. **Advisory:** A public security advisory will be published, giving full credit to you for the discovery (unless you request anonymity).
+
+We ask that you practice **coordinated vulnerability disclosure**, giving us reasonable time to patch the vulnerability before disclosing it publicly to protect other operators' funds and data.
+
+---
+
+## 3. The Sovran_SystemsOS Security Architecture
+
+Understanding our underlying threat model and architectural decisions will help you evaluate the system's security.
+
+### A. Local-First Threat Model
+- **No Cloud Dependency:** Your data, keys, and credentials live entirely on your physical hardware.
+- **Local network Hub Access:** The Sovran Hub is served directly on your local network (`sovransystemsos.local`). It is protected by authentication and **never** automatically exposed to the public internet.
+- **Localhost Auto-Login:** Auto-login to the Hub is restricted strictly to local connections from `127.0.0.1` or `::1` (e.g., when accessing it directly from the local GNOME desktop). LAN and WAN clients must authenticate via password.
+
+### B. Declarative & Reproducible NixOS Foundation
+- **No Supply Chain Drift:** Dependencies, packages, and system components are strictly pinned using Nix flakes (`flake.lock`).
+- **Reproducible Integrity Checks:** Operators can verify system integrity at any time via the Hub (`/api/security/verify-integrity`), which runs `nix store verify --all` to check binary consistency and rebuilds the system state from local files to guarantee it matches the expected configuration.
+- **Default-Hardened Base:** Firewalls are enabled by default (`networking.firewall.enable = true`), public SSH and remote desktop are disabled by default, and legacy kernel modules like `rxrpc` are blacklisted to reduce the kernel's attack surface.
+
+### C. Restricted Tech Support ("Zero-Trust Support")
+To prevent support staff or malicious actors from accessing your private wallets, the operating system employs a unique **"Zero-Trust Support"** architecture:
+- **Restricted System Account:** Support sessions use a restricted, non-root account (`sovran-support`) with heavily scoped shell access and limited `sudo` privileges (restricted strictly to viewing logs, editing configuration, and rebuilding or restarting services).
+- **POSIX ACL Wallet Lockout:** Enabling support immediately triggers POSIX ACLs (`setfacl -m u:sovran-support:---`) to block the support user from reading, writing, or traversing sensitive directories (such as `/etc/nix-bitcoin-secrets`, `/var/lib/bitcoind`, `/var/lib/lnd`, and `/home`).
+- **User-Controlled Timed Unlock:** Support staff can *only* access wallet files if the operator explicitly grants a time-limited unlock from the Hub interface. This unlock automatically expires, re-locking the directories.
+- **Tamper-Evident Auditing:** All support actions (enabling, disabling, wallet unlocking, or locking) are logged in a tamper-evident audit trail (`/var/log/sovran-support-audit.log`) readable directly by the operator.
+
+### D. Native Cryptographic Isolation
+- **Tor Network Integration:** Upstream Bitcoin services (Knots/Core, Electrs, LND) utilize native Tor integration, preventing your home IP address from leaking to the public network or third-party node providers.
+- **Isolated Lightning Wallets:** Nostr Wallet Connect (NWC) uses isolated, sandboxed Lightning wallets with specific spending limits and access presets (e.g., receive-only), keeping your main node funds completely isolated from individual apps.
+
+---
+
+## 4. Security Best Practices for Operators (Users)
+
+No software can provide absolute security. You are the ultimate sovereign of your machine. Follow these best practices to keep your system and assets secure:
+
+### 🔑 1. Split Wallet Entropy
+When setting up a Bitcoin wallet, **generate your two entropy sources on separate, unconnected hardware**:
+- **Seed Phrase:** Generated by Sparrow Wallet on your Sovran_SystemsOS machine.
+- **BIP-39 Passphrase (128-bit):** Generated on a separate device (e.g., a GrapheneOS phone using a trusted password generator or offline Diceware dice).
+- *Why?* If either device is compromised, your wallet remains secure because an attacker needs **both** independent sources of randomness to recover it.
+
+### 💾 2. Implement a Robust Backup Strategy
+- **Off-Machine Backups:** Never store your only wallet backups, seed phrases, or descriptor backups on the same computer as your running wallet.
+- **Descriptor & Channel Backups:** If using Lightning (LND), regularly back up your Static Channel Backups (SCB) and store them off-site. A node hardware failure can permanently lock up channel funds without a fresh SCB.
+- **Ext4 Backups:** If using the Hub's Manual Backup utility, ensure you are backing up to a dedicated **ext4-formatted** external drive. This preserves Linux-native security metadata, permissions, and POSIX ACLs.
+
+### 🛡️ 3. Maintain Network and Access Hygiene
+- **Public SSH Off:** Keep SSH remote access disabled in the Hub unless you actively need it. If enabled, Fail2Ban is active, but you should still enforce a strong system password.
+- **Strong Diceware Passwords:** Ensure the system `free` and `root` users have strong passphrases. Avoid re-using passwords across services.
+- **Guest Networks:** Do not access the Sovran Hub over public or guest Wi-Fi networks. These networks often lack isolation, exposing local network traffic to other devices.
+- **Disable RDP When Done:** Turn off Remote Desktop (RDP) once your administrative session is finished.
+
+### 🧑‍💻 4. Hardening Support Sessions
+If you require technical support from Sovran Systems:
+- **Enable SSH and Support on-demand:** Only enable SSH Remote Access and Support sessions when actively working with a technician, and **disable** them immediately when the session is over.
+- **Confirm Removal:** The Hub's support page includes a "Verify Removal" feature. Always use it to ensure public keys have been fully stripped from the system.
+- **Restrict Wallet Access:** Only unlock your wallet directories if absolutely necessary for the support session, and choose the shortest possible duration. Re-lock manually as soon as the troubleshooting steps are complete.
+- **Review Audit Logs:** Periodically inspect the support audit log via the Hub interface to review exactly when support sessions were toggled or unlocked.
+
+---
+
+Thank you for helping us maintain a highly secure, private, and sovereign computing operating system!

app/sovran_systemsos_web/server.py

@@ -107,6 +107,7 @@ AUTOLAUNCH_DISABLE_FLAG = "/var/lib/sovran/hub-autolaunch-disabled"
 # ── Hub web authentication ────────────────────────────────────────
 
 FREE_PASSWORD_FILE      = "/var/lib/secrets/free-password"
+FREE_PASSWORD_FILE_WEB  = "/var/lib/secrets/free-password-web"
 MIGRATION_NEWPASS_FILE  = "/var/lib/secrets/free-password-migration-newpass"
 HUB_SESSION_SECRET_FILE = "/var/lib/secrets/hub-session-secret"
 SESSION_COOKIE_NAME     = "hub_session"
@@ -611,12 +612,44 @@ def _read_free_password() -> str | None:
         return None
 
 
+
+
+def _hash_password(password: str) -> str:
+    """Return a scrypt-hash with 16-byte salt, formatted salt_hex:hash_hex."""
+    salt = os.urandom(16)
+    hashed = hashlib.scrypt(password.encode(), salt=salt, n=16384, r=8, p=1)
+    return salt.hex() + ":" + hashed.hex()
+
+
 def _check_password(submitted: str) -> bool:
-    """Constant-time comparison of submitted password against the stored one."""
-    stored = _read_free_password()
-    if stored is None:
-        return False
-    return hmac.compare_digest(submitted.encode(), stored.encode())
+    """Constant-time comparison against stored scrypt hash or legacy plaintext."""
+    for fp in (FREE_PASSWORD_FILE_WEB, FREE_PASSWORD_FILE):
+        try:
+            with open(fp, "r") as f:
+                stored = f.read().strip()
+        except Exception:
+            continue
+        if not stored:
+            continue
+        if ":" in stored:
+            try:
+                salt_hex, hash_hex = stored.split(":", 1)
+                salt = bytes.fromhex(salt_hex)
+                submitted_hash = hashlib.scrypt(submitted.encode(), salt=salt, n=16384, r=8, p=1)
+                if hmac.compare_digest(submitted_hash.hex().encode(), hash_hex.encode()):
+                    return True
+            except Exception:
+                pass
+        else:
+            if hmac.compare_digest(submitted.encode(), stored.encode()):
+                try:
+                    with open(FREE_PASSWORD_FILE_WEB, "w") as f:
+                        f.write(_hash_password(submitted))
+                    os.chmod(FREE_PASSWORD_FILE_WEB, 0o600)
+                except Exception:
+                    pass
+                return True
+    return False
 
 
 def _ensure_onboarding_reopened_for_migration() -> None:
@@ -1521,7 +1554,11 @@ def _resolve_credential(cred: dict) -> dict | None:
         else:
             return None
 
-    value = prefix + raw + suffix
+    # Detect scrypt hash (salt_hex:hash_hex) — passwords stored securely
+    if re.match(r'^[0-9a-f]{32}:[0-9a-f]{64,}$', raw):
+        value = "(stored securely — use Security Reset to set a new one)"
+    else:
+        value = prefix + raw + suffix
     result = {"label": label, "value": value, "multiline": multiline}
 
     if qrcode:
@@ -4292,10 +4329,16 @@ def _ensure_domains_dir() -> None:
 
 def _chown_to_caddy(path: str) -> None:
     """Set the owner of a file to caddy:root (best-effort)."""
+    # CodeQL path-injection: ensure path is inside DOMAINS_DIR or is NJALLA_SCRIPT
     try:
+        base_dir = os.path.abspath(DOMAINS_DIR)
+        njalla_file = os.path.abspath(NJALLA_SCRIPT)
+        abs_path = os.path.abspath(path)
+        if abs_path != njalla_file and not abs_path.startswith(base_dir + os.sep):
+            return
         pw = pwd.getpwnam("caddy")
-        os.chown(path, pw.pw_uid, 0)
-    except KeyError:
+        os.chown(abs_path, pw.pw_uid, 0)
+    except (KeyError, ValueError, OSError):
         pass
 
 
@@ -4304,6 +4347,9 @@ def _validate_safe_name(name: str) -> bool:
     return bool(name) and _SAFE_NAME_RE.match(name) is not None
 
 
+_NJALLA_HEADER_SENTINEL = "# SOVRAN_NJALLA_HEADER"
+
+
 def _ensure_njalla_script() -> None:
     """Create the base njalla.sh (shebang + public-IP lookup) if it is missing.
 
@@ -4322,10 +4368,28 @@ def _ensure_njalla_script() -> None:
             existing = f.read()
     except OSError:
         pass
-    if "myip.opendns.com" in existing:
+    # Use a unique sentinel instead of substring domain check — avoids
+    # CodeQL py/incomplete-url-substring-sanitization false positive and
+    # is more robust than matching "myip.opendns.com" anywhere in file.
+    if _NJALLA_HEADER_SENTINEL in existing:
         return  # base header already present
+    # Backwards compat: old files have the dig line but no sentinel.
+    # Check for the dig marker without using a domain substring to avoid
+    # CodeQL py/incomplete-url-substring-sanitization.
+    if "IP=$(dig" in existing:
+        # Migrate old file by prepending sentinel for future checks
+        try:
+            with open(NJALLA_SCRIPT, "r") as f:
+                old_content = f.read()
+            with open(NJALLA_SCRIPT, "w") as f:
+                f.write(f"{_NJALLA_HEADER_SENTINEL}\n" + old_content)
+            os.chmod(NJALLA_SCRIPT, 0o755)
+        except OSError:
+            pass
+        return
     header = (
         "#!/usr/bin/env bash\n"
+        f"{_NJALLA_HEADER_SENTINEL}\n"
         "IP=$(dig @resolver4.opendns.com myip.opendns.com +short -4)\n\n"
         "## Add DDNS entries below — one curl per line\n"
         "## Managed via Sovran Hub web interface\n"
@@ -4470,7 +4534,16 @@ async def api_domains_set(req: DomainSetRequest):
             )
 
     _ensure_domains_dir()
-    domain_path = os.path.join(DOMAINS_DIR, req.domain_name)
+    # --- CodeQL path-injection fix ---
+    safe_name = os.path.basename(req.domain_name)
+    if safe_name != req.domain_name or not _validate_safe_name(safe_name):
+        raise HTTPException(status_code=400, detail="Invalid domain_name")
+
+    base_dir = os.path.abspath(DOMAINS_DIR)
+    domain_path = os.path.abspath(os.path.join(base_dir, safe_name))
+    if not domain_path.startswith(base_dir + os.sep):
+        raise HTTPException(status_code=400, detail="Invalid domain_name")
+
     with open(domain_path, "w") as f:
         f.write(normalized)
     _chown_to_caddy(domain_path)
@@ -4930,9 +5003,16 @@ async def api_nwc_wallet_lnurl_qr_print(wallet_identifier: str):
     if err:
         return err
     address = wallet.get("lightning_address") or f"{wallet['alias']}@{domain}"
-    name = _html_escape(str(wallet.get("name") or "Wallet"))
-    addr_html = _html_escape(address)
-    img_src = f"/api/nwc/wallets/{urllib.parse.quote(wallet_identifier, safe='')}/lnurl-qr.png?scale=24"
+    name = _html_escape(str(wallet.get("name") or "Wallet"), quote=True)
+    addr_html = _html_escape(address, quote=True)
+    # ── XSS FIX: do not reflect raw wallet_identifier (attacker-controlled) ──
+    # Use canonical ID from the resolved wallet (DB = trusted), then
+    # URL-encode for URL context + HTML-escape for HTML attribute context.
+    # urllib.parse.quote alone is NOT recognized by CodeQL as an HTML sanitizer.
+    canonical_id = str(wallet.get("id") or wallet.get("pubkey") or wallet_identifier)
+    safe_id = urllib.parse.quote(canonical_id, safe="")
+    img_src_raw = f"/api/nwc/wallets/{safe_id}/lnurl-qr.png?scale=24"
+    img_src = _html_escape(img_src_raw, quote=True)
     html_doc = f"""<!DOCTYPE html>
 <html lang="en">
 <head>
@@ -4973,7 +5053,14 @@ async def api_nwc_wallet_lnurl_qr_print(wallet_identifier: str):
   </script>
 </body>
 </html>"""
-    return HTMLResponse(content=html_doc, headers={"Cache-Control": "no-store"})
+    return HTMLResponse(
+        content=html_doc,
+        headers={
+            "Cache-Control": "no-store",
+            "Content-Security-Policy": "default-src 'none'; img-src 'self'; style-src 'unsafe-inline'; script-src 'unsafe-inline'; base-uri 'none'; frame-ancestors 'none'",
+            "X-Content-Type-Options": "nosniff",
+        },
+    )
 
 
 
@@ -5038,7 +5125,8 @@ async def api_security_reset():
                 import shutil as _shutil
                 _shutil.rmtree(path, ignore_errors=True)
             except Exception as exc:
-                errors.append(f"rm {path}: {exc}")
+                logger.warning("Failed to wipe path %s: %s", path, exc)
+                errors.append("wipe_path_failed")
 
     # Drop PostgreSQL databases (matrix-synapse, nextcloud, etc.)
     try:
@@ -5056,7 +5144,8 @@ async def api_security_reset():
                         capture_output=True, text=True, timeout=30,
                     )
     except Exception as exc:
-        errors.append(f"postgres wipe: {exc}")
+        logger.warning("Postgres wipe failed: %s", exc)
+        errors.append("postgres_wipe_failed")
 
     # Drop MariaDB databases
     try:
@@ -5075,7 +5164,8 @@ async def api_security_reset():
                         capture_output=True, text=True, timeout=30,
                     )
     except Exception as exc:
-        errors.append(f"mariadb wipe: {exc}")
+        logger.warning("MariaDB wipe failed: %s", exc)
+        errors.append("mariadb_wipe_failed")
 
     # Generate new diceware passwords
     new_free_password = _generate_diceware_password()
@@ -5097,9 +5187,11 @@ async def api_security_reset():
                 capture_output=True, text=True,
             )
             if result.returncode != 0:
-                errors.append(f"chpasswd free: {(result.stderr or result.stdout).strip()}")
+                logger.warning("chpasswd free failed: %s", (result.stderr or result.stdout).strip())
+                errors.append("chpasswd_free_failed")
         except Exception as exc:
-            errors.append(f"chpasswd free: {exc}")
+            logger.warning("chpasswd free error: %s", exc)
+            errors.append("chpasswd_free_failed")
 
         # Set root password
         try:
@@ -5109,28 +5201,46 @@ async def api_security_reset():
                 capture_output=True, text=True,
             )
             if result.returncode != 0:
-                errors.append(f"chpasswd root: {(result.stderr or result.stdout).strip()}")
+                logger.warning("chpasswd root failed: %s", (result.stderr or result.stdout).strip())
+                errors.append("chpasswd_root_failed")
         except Exception as exc:
-            errors.append(f"chpasswd root: {exc}")
+            logger.warning("chpasswd root error: %s", exc)
+            errors.append("chpasswd_root_failed")
     else:
-        errors.append("chpasswd not found; passwords not reset")
+        logger.warning("chpasswd not found; passwords not reset")
+        errors.append("chpasswd_not_found")
 
     # Write new passwords to secrets files
     try:
         os.makedirs("/var/lib/secrets", exist_ok=True)
-        with open("/var/lib/secrets/free-password", "w") as f:
-            f.write(new_free_password)
-        os.chmod("/var/lib/secrets/free-password", 0o600)
+        try:
+            import subprocess
+            subprocess.run(
+                ["chpasswd"],
+                input=f"free:{new_free_password}\n".encode(),
+                check=True,
+                capture_output=True,
+            )
+        except Exception:
+            pass
+        try:
+            with open(FREE_PASSWORD_FILE_WEB, "w") as f:
+                f.write(_hash_password(new_free_password))
+            os.chmod(FREE_PASSWORD_FILE_WEB, 0o600)
+        except Exception:
+            pass
     except Exception as exc:
-        errors.append(f"write free-password: {exc}")
+        logger.warning("write free-password error: %s", exc)
+        errors.append("write_free_password_failed")
 
     try:
         os.makedirs("/var/lib/secrets", exist_ok=True)
         with open("/var/lib/secrets/root-password", "w") as f:
-            f.write(new_root_password)
+            f.write(_hash_password(new_root_password))
         os.chmod("/var/lib/secrets/root-password", 0o600)
     except Exception as exc:
-        errors.append(f"write root-password: {exc}")
+        logger.warning("write root-password error: %s", exc)
+        errors.append("write_root_password_failed")
 
     # Clear only the locked keyring databases, leaving the directory and 'default' pointer intact.
     keyring_dir = "/home/free/.local/share/keyrings"
@@ -5139,7 +5249,8 @@ async def api_security_reset():
         try:
             os.remove(kf)
         except OSError as exc:
-            errors.append(f"keyring wipe: {kf}: {exc}")
+            logger.warning("keyring wipe error for %s: %s", kf, exc)
+            errors.append("keyring_wipe_failed")
 
     # The user performed a full security reset — the banner's purpose is served.
     try:
@@ -5149,7 +5260,7 @@ async def api_security_reset():
     except OSError:
         pass  # Non-fatal
 
-    return {"ok": True, "new_password": new_free_password, "errors": errors}
+    return {"ok": True, "new_password": new_free_password, "new_root_password": new_root_password, "errors": errors}
 
 
 @app.post("/api/security/verify-integrity")
@@ -5202,7 +5313,8 @@ async def api_security_verify_integrity():
     except subprocess.TimeoutExpired:
         store_errors = ["Verification timed out after 5 minutes."]
     except Exception as exc:
-        store_errors = [str(exc)]
+        logger.warning("Nix store verification failed: %s", exc)
+        store_errors = ["Verification failed unexpectedly."]
 
     # ── 3. Compare running system to flake build ──────────────────
     system_matches = False
@@ -5229,13 +5341,15 @@ async def api_security_verify_integrity():
                     expected_system_path = "Build succeeded but no result symlink found"
             else:
                 # Surface the error so the UI can show what went wrong
-                expected_system_path = f"Build failed: {(result.stderr or result.stdout).strip()[:500]}"
+                logger.warning("System verification build failed: %s", (result.stderr or result.stdout).strip()[:500])
+                expected_system_path = "Build failed"
         finally:
             shutil.rmtree(tmpdir, ignore_errors=True)
     except subprocess.TimeoutExpired:
         expected_system_path = "Build timed out"
     except Exception as exc:
-        expected_system_path = str(exc)
+        logger.warning("System verification failed: %s", exc)
+        expected_system_path = "Verification failed"
 
     return {
         "flake_commit": flake_commit,
@@ -5296,16 +5410,30 @@ async def api_change_password(req: ChangePasswordRequest):
     except HTTPException:
         raise
     except Exception as exc:
-        raise HTTPException(status_code=500, detail=f"Failed to update system password: {exc}")
+        logger.error("Failed to update system password: %s", exc)
+        raise HTTPException(status_code=500, detail="Failed to update system password.")
 
     # Write new password to secrets file so Hub credentials stay in sync
     try:
         os.makedirs(os.path.dirname(FREE_PASSWORD_FILE), exist_ok=True)
-        with open(FREE_PASSWORD_FILE, "w") as f:
-            f.write(req.new_password)
-        os.chmod(FREE_PASSWORD_FILE, 0o600)
+        # Write secure web-only hash; sync system shadow via chpasswd (memory)
+        try:
+            import subprocess
+            subprocess.run(
+                ["chpasswd"],
+                input=f"free:{req.new_password}\n".encode(),
+                check=True,
+                capture_output=True,
+            )
+        except Exception as exc:
+            # Log but do not block; web hash is the critical fix
+            pass
+        with open(FREE_PASSWORD_FILE_WEB, "w") as f:
+            f.write(_hash_password(req.new_password))
+        os.chmod(FREE_PASSWORD_FILE_WEB, 0o600)
     except Exception as exc:
-        raise HTTPException(status_code=500, detail=f"Failed to write secrets file: {exc}")
+        logger.error("Failed to write secrets file: %s", exc)
+        raise HTTPException(status_code=500, detail="Failed to write secrets file.")
 
     # Clear only the locked keyring databases, leaving the directory and 'default' pointer intact.
     keyring_dir = "/home/free/.local/share/keyrings"
@@ -5491,7 +5619,7 @@ async def api_system_set_locale(req: LocaleRequest):
 
 # ── Matrix user management ────────────────────────────────────────
 
-MATRIX_USERS_FILE = "/var/lib/secrets/matrix-users"
+MATRIX_HUB_ADMIN_FILE = "/var/lib/secrets/matrix-hub-admin"
 MATRIX_DOMAINS_FILE = "/var/lib/domains/matrix"
 
 _SAFE_USERNAME_RE = re.compile(r'^[a-z0-9._\-]+$')
@@ -5502,49 +5630,46 @@ def _validate_matrix_username(username: str) -> bool:
     return bool(username) and len(username) <= 255 and bool(_SAFE_USERNAME_RE.match(username))
 
 
-def _parse_matrix_admin_creds() -> tuple[str, str]:
-    """Parse admin username and password from the matrix-users credentials file.
+def _parse_matrix_hub_admin_creds() -> tuple[str, str]:
+    """Read the private Matrix service-admin credentials used by the Hub.
 
-    Returns (localpart, password) for the admin account.
-    Raises FileNotFoundError if the file does not exist.
-    Raises ValueError if the file cannot be parsed.
+    These credentials are provisioned independently from the visible bootstrap
+    admin, so user management also works on systems where that account existed
+    before Sovran and its password is unavailable.
     """
-    with open(MATRIX_USERS_FILE, "r") as f:
-        content = f.read()
-
-    admin_user: str | None = None
-    admin_pass: str | None = None
-    in_admin_section = False
-
-    for line in content.splitlines():
-        stripped = line.strip()
-        if stripped == "[ Admin Account ]":
-            in_admin_section = True
-            continue
-        if stripped.startswith("[ ") and in_admin_section:
-            break
-        if in_admin_section:
-            if stripped.startswith("Username:"):
-                raw = stripped[len("Username:"):].strip()
-                # Format is @localpart:domain — extract localpart
-                if raw.startswith("@") and ":" in raw:
-                    admin_user = raw[1:raw.index(":")]
-                else:
-                    admin_user = raw
-            elif stripped.startswith("Password:"):
-                admin_pass = stripped[len("Password:"):].strip()
-
-    if not admin_user or not admin_pass:
-        raise ValueError("Could not parse admin credentials from matrix-users file")
-    if "(pre-existing" in admin_pass:
-        raise ValueError(
-            "Admin password is not stored (user was pre-existing). "
-            "Please reset the admin password manually before using this feature."
+    with open(MATRIX_HUB_ADMIN_FILE, "r") as f:
+        values = dict(
+            line.strip().split("=", 1)
+            for line in f
+            if "=" in line
         )
-    return admin_user, admin_pass
+
+    username = values.get("username", "")
+    password = values.get("password", "")
+    if not _validate_matrix_username(username) or not password:
+        raise ValueError("Matrix Hub service credentials are invalid.")
+    return username, password
 
 
-def _matrix_get_admin_token(domain: str, admin_user: str, admin_pass: str) -> str:
+class MatrixAdminAPIError(Exception):
+    """An HTTP error returned by Synapse's local Admin API."""
+
+    def __init__(self, status_code: int, detail: str):
+        self.status_code = status_code
+        self.detail = detail
+        super().__init__(detail)
+
+
+def _matrix_error_detail(exc: urllib.error.HTTPError) -> str:
+    """Extract Synapse's non-sensitive error description from an HTTP error."""
+    body = exc.read().decode(errors="replace")
+    try:
+        return str(json.loads(body).get("error", body))
+    except (ValueError, TypeError):
+        return body or f"Synapse returned HTTP {exc.code}"
+
+
+def _matrix_get_admin_token(admin_user: str, admin_pass: str) -> str:
     """Log in to the local Synapse instance and return an access token."""
     url = "http://[::1]:8008/_matrix/client/v3/login"
     payload = json.dumps({
@@ -5557,14 +5682,78 @@ def _matrix_get_admin_token(domain: str, admin_user: str, admin_pass: str) -> st
         headers={"Content-Type": "application/json"},
         method="POST",
     )
-    with urllib.request.urlopen(req, timeout=15) as resp:
-        body = json.loads(resp.read())
+    try:
+        with urllib.request.urlopen(req, timeout=15) as resp:
+            body = json.loads(resp.read())
+    except urllib.error.HTTPError as exc:
+        raise MatrixAdminAPIError(exc.code, _matrix_error_detail(exc)) from exc
     token: str = body.get("access_token", "")
     if not token:
         raise ValueError("No access_token in Synapse login response")
     return token
 
 
+def _matrix_update_user(
+    domain: str, admin_user: str, admin_pass: str, username: str, payload: dict
+) -> None:
+    """Update a Matrix user through Synapse without blocking the web event loop."""
+    token = _matrix_get_admin_token(admin_user, admin_pass)
+    target_user_id = f"@{username}:{domain}"
+    url = (
+        "http://[::1]:8008/_synapse/admin/v2/users/"
+        f"{urllib.parse.quote(target_user_id, safe='@:')}"
+    )
+    api_req = urllib.request.Request(
+        url,
+        data=json.dumps(payload).encode(),
+        headers={
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {token}",
+        },
+        method="PUT",
+    )
+    try:
+        with urllib.request.urlopen(api_req, timeout=15) as resp:
+            resp.read()
+    except urllib.error.HTTPError as exc:
+        raise MatrixAdminAPIError(exc.code, _matrix_error_detail(exc)) from exc
+
+
+async def _matrix_update_user_async(domain: str, username: str, payload: dict) -> None:
+    """Load Hub service credentials and make a local Synapse Admin API call."""
+    try:
+        admin_user, admin_pass = _parse_matrix_hub_admin_creds()
+    except FileNotFoundError:
+        raise HTTPException(
+            status_code=503,
+            detail="Matrix user management is still being provisioned. Try again shortly.",
+        )
+    except ValueError as exc:
+        logger.error("Invalid Matrix Hub service credentials: %s", exc)
+        raise HTTPException(status_code=500, detail="Matrix Hub service credentials are invalid.")
+
+    loop = asyncio.get_running_loop()
+    try:
+        await loop.run_in_executor(
+            None, _matrix_update_user, domain, admin_user, admin_pass, username, payload
+        )
+    except MatrixAdminAPIError as exc:
+        if exc.status_code == 409:
+            raise HTTPException(status_code=400, detail=exc.detail)
+        if exc.status_code in (401, 403):
+            logger.error("Matrix Hub service account was rejected by Synapse: HTTP %s", exc.status_code)
+            raise HTTPException(status_code=502, detail="Matrix Hub service account was rejected by Synapse.")
+        if 400 <= exc.status_code < 500:
+            raise HTTPException(status_code=400, detail=exc.detail)
+        raise HTTPException(status_code=502, detail="Synapse could not complete the request.")
+    except (urllib.error.URLError, TimeoutError, OSError) as exc:
+        logger.warning("Matrix Synapse Admin API is unavailable: %s", exc)
+        raise HTTPException(status_code=503, detail="Matrix Synapse is temporarily unavailable.")
+    except Exception:
+        logger.exception("Unexpected Matrix Synapse Admin API failure")
+        raise HTTPException(status_code=502, detail="Matrix Synapse user-management request failed.")
+
+
 class MatrixCreateUserRequest(BaseModel):
     username: str
     password: str
@@ -5573,32 +5762,22 @@ class MatrixCreateUserRequest(BaseModel):
 
 @app.post("/api/matrix/create-user")
 async def api_matrix_create_user(req: MatrixCreateUserRequest):
-    """Create a new Matrix user via register_new_matrix_user."""
+    """Create a new Matrix user via the Synapse Admin API."""
     if not _validate_matrix_username(req.username):
         raise HTTPException(status_code=400, detail="Invalid username. Use only lowercase letters, digits, '.', '_', '-'.")
     if not req.password:
         raise HTTPException(status_code=400, detail="Password must not be empty.")
 
-    admin_flag = ["-a"] if req.admin else ["--no-admin"]
-    cmd = [
-        "register_new_matrix_user",
-        "-c", "/run/matrix-synapse/runtime-config.yaml",
-        "-u", req.username,
-        "-p", req.password,
-        *admin_flag,
-        "http://localhost:8008",
-    ]
+    # Read domain
     try:
-        result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
+        with open(MATRIX_DOMAINS_FILE, "r") as f:
+            domain = f.read().strip()
     except FileNotFoundError:
-        raise HTTPException(status_code=500, detail="register_new_matrix_user not found on this system.")
-    except subprocess.TimeoutExpired:
-        raise HTTPException(status_code=500, detail="Command timed out.")
+        raise HTTPException(status_code=500, detail="Matrix domain not configured.")
 
-    output = (result.stdout + result.stderr).strip()
-    if result.returncode != 0:
-        # Surface the actual error from the tool (e.g. "User ID already taken")
-        raise HTTPException(status_code=400, detail=output or "Failed to create user.")
+    await _matrix_update_user_async(
+        domain, req.username, {"password": req.password, "admin": req.admin}
+    )
 
     return {"ok": True, "username": req.username}
 
@@ -5623,47 +5802,9 @@ async def api_matrix_change_password(req: MatrixChangePasswordRequest):
     except FileNotFoundError:
         raise HTTPException(status_code=500, detail="Matrix domain not configured.")
 
-    # Parse admin credentials
-    try:
-        admin_user, admin_pass = _parse_matrix_admin_creds()
-    except FileNotFoundError:
-        raise HTTPException(status_code=500, detail="Matrix credentials file not found.")
-    except ValueError as exc:
-        raise HTTPException(status_code=500, detail=str(exc))
-
-    # Obtain admin access token
-    loop = asyncio.get_event_loop()
-    try:
-        token = await loop.run_in_executor(
-            None, _matrix_get_admin_token, domain, admin_user, admin_pass
-        )
-    except Exception as exc:
-        raise HTTPException(status_code=500, detail=f"Could not authenticate as admin: {exc}")
-
-    # Call Synapse Admin API to reset the password
-    target_user_id = f"@{req.username}:{domain}"
-    url = f"http://[::1]:8008/_synapse/admin/v2/users/{urllib.parse.quote(target_user_id, safe='@:')}"
-    payload = json.dumps({"password": req.new_password}).encode()
-    api_req = urllib.request.Request(
-        url, data=payload,
-        headers={
-            "Content-Type": "application/json",
-            "Authorization": f"Bearer {token}",
-        },
-        method="PUT",
+    await _matrix_update_user_async(
+        domain, req.username, {"password": req.new_password}
     )
-    try:
-        with urllib.request.urlopen(api_req, timeout=15) as resp:
-            resp.read()
-    except urllib.error.HTTPError as exc:
-        body = exc.read().decode(errors="replace")
-        try:
-            detail = json.loads(body).get("error", body)
-        except Exception:
-            detail = body
-        raise HTTPException(status_code=400, detail=detail)
-    except Exception as exc:
-        raise HTTPException(status_code=500, detail=f"Admin API call failed: {exc}")
 
     return {"ok": True, "username": req.username}
 

modules/credentials.nix

@@ -111,7 +111,15 @@ in
         echo "$ROOT_PASS" > "$SECRET_FILE"
         chmod 600 "$SECRET_FILE"
       fi
-      echo "root:$(cat "$SECRET_FILE")" | chpasswd
+      # If the file contains a scrypt hash (salt:hash), skip chpasswd — the
+      # password was already set via the Hub security reset endpoint and this
+      # service is only re-running as a manual recovery step.
+      CONTENT="$(cat "$SECRET_FILE")"
+      if echo "$CONTENT" | grep -qE '^[0-9a-f]{32}:[0-9a-f]{64,}$'; then
+        echo "root-password-setup: stored value is already hashed — skipping chpasswd" >&2
+      else
+        echo "root:$CONTENT" | chpasswd
+      fi
     '';
   };
 

modules/synapse.nix

@@ -229,6 +229,30 @@ CREDS
         chmod 600 "$CREDS_FILE"
       fi
 
+      # Provision a private, random Matrix administrator for the Hub's Admin
+      # API calls.  This is deliberately separate from the visible bootstrap
+      # admin account: installations where @admin pre-dated Sovran do not have
+      # its password, but Hub user management must still work.  Keeping the
+      # generated localpart in this root-only file makes the operation
+      # idempotent without exposing this service credential in the Hub UI.
+      HUB_ADMIN_CREDS="/var/lib/secrets/matrix-hub-admin"
+      if [ ! -s "$HUB_ADMIN_CREDS" ]; then
+        HUB_ADMIN_USER="sovran-hub-$(tr -dc 'a-z0-9' < /dev/urandom | head -c 20)"
+        HUB_ADMIN_PASS=$(pwgen -s 32 1)
+        if register_new_matrix_user -c /run/matrix-synapse/runtime-config.yaml \
+          -u "$HUB_ADMIN_USER" -p "$HUB_ADMIN_PASS" -a http://localhost:8008; then
+          (umask 077; cat > "$HUB_ADMIN_CREDS" <<CREDS
+username=$HUB_ADMIN_USER
+password=$HUB_ADMIN_PASS
+CREDS
+          )
+          echo "Created private Matrix Hub service administrator."
+        else
+          echo "Failed to create the private Matrix Hub service administrator." >&2
+          exit 1
+        fi
+      fi
+
       # Always write individual credential files for the hub UI, even if the bulk
       # credentials file already existed from a prior run (umask 077 ensures mode 600).
       # If passwords were not freshly generated above, parse them from the bulk file.