The full-system updater runs as a detached systemd service and can finish
successfully even when the browser loses its status connection. In that
case the update log and status file correctly report REBOOT_REQUIRED, but
the Hub modal can remain on "Updating..." with its controls disabled.
There were four independent ways for the frontend to get stuck:
* update status fetches had no deadline, so a request that stayed pending
never rejected and never advanced the existing failure counter;
* setInterval started async polls without waiting for the previous poll,
allowing slow requests to overlap and responses to arrive out of order;
* each log chunk used textContent +=, replacing the complete and growing
Nix build log every two seconds, which could stall browser rendering and
was especially visible over RDP; and
* page reload, tab resume, and RDP reconnect did not reattach the modal to
the update status persisted by the backend.
This produced a dangerous UX mismatch: the machine had a fully staged
NixOS generation and was ready to reboot, while the Hub continued telling
the user that the update was still running.
Bound status requests with AbortController, prevent overlapping polls, and
replace the endless spinner after sustained failures with an explicit
"Update status unavailable" state and Retry Status action. Reconcile state
immediately on focus, visibility, online, page startup, and before starting
a new update. Use no-store requests and render verbose logs incrementally
with a bounded visible tail while retaining the complete report in memory.
Apply the same timeout and single-flight protection to rebuild polling.
Record the exact generation produced by `nixos-rebuild boot`. The Hub now
keeps REBOOT_REQUIRED visible until that generation matches
/run/current-system, then clears the marker after reboot. For an update
started by an older updater that did not write the marker, recover the
staged generation from the final nixos-rebuild log line. The dashboard
sidebar also distinguishes update-in-progress and restart-required states.
Regression coverage verifies generation marker/log recovery, pre- versus
post-reboot detection, request timeout wiring, single-flight polling,
connection-loss UX, RDP/tab resume reconciliation, bounded log rendering,
page-reload recovery, and JavaScript syntax.
Validation:
* python3 -m unittest discover -s tests -p 'test_*.py' -v (170 passed)
* node --check app/sovran_systemsos_web/static/js/*.js
* python3 -m py_compile for changed Python modules
* git diff --check
A Nix evaluation was not available in the development sandbox; the NixOS
module should still be evaluated and built in CI or on a test machine before
release.
nixpkgs commit c8f9654 refactored the services.i2pd module to use
an RFC42-style settings attribute set and removed services.i2pd.proto.
After updating the root nixpkgs input from f13ff45 to ec2d622, the
vendored bitcoind module failed evaluation on the obsolete
services.i2pd.proto.sam.enable definition.
The error occurred even with services.bitcoind.i2p at its false default:
bitcoind was enabled, so NixOS still validated the obsolete option path
inside the conditional i2pd integration.
Read the SAM endpoint from services.i2pd.settings.sam and configure its
new upstream-style fields explicitly. Keep 127.0.0.1:7656, matching the
old typed option defaults that bitcoind uses to generate its i2psam
setting.
This preserves optional I2P support without activating it by default.
i2pd remains disabled until services.bitcoind.i2p is set to true or
"only-outgoing".
Nixpkgs migration: https://github.com/NixOS/nixpkgs/commit/c8f965411e812060a9377fa4c2d7d0f84e8b10e0
livekit-turn-setup.service detects the primary interface from the IPv4
default route, but had no ordering against network-online.target. With
NetworkManager+DHCP the default route is applied late at boot, so the
oneshot could run before it existed, exit 1, and — being a hard
dependency of livekit.service — take livekit down with it. The Hub then
showed a 'failed' red dot until livekit was restarted manually.
Order both livekit.service and livekit-turn-setup.service after
network-online.target. Also add a bounded retry when copying Caddy's ACME
cert so we never write an empty turn.crt/turn.key on a fresh boot.
The LND-only rewrite of lndconnect.nix shipped a wrapper Zeus cannot
use: unknown flags (--cert/--macaroon), a non-existent onion path
(free/lnd.onion), and a REST hidden service that collided with LND's
P2P onion. Restore the nix-bitcoin contract — dedicated lnd-rest
onion on port 8080, --nocert over Tor, admin macaroon in the URI —
and only persist a valid lndconnect:// URI for the Hub QR.
The Hub launcher used an ephemeral /tmp profile deleted on exit, which
wiped the hub_manual_logout marker cookie. On reopen, /auto-login minted a
new session and logged the user straight back in without a password.
Use a persistent per-user profile under XDG_STATE_HOME and drop the
deletion trap so the logout marker survives close/reopen. Keep
--skip-origin-startup-dialog. Adds regression tests.
- modules/synapse.nix: replace tr|head pipeline (causes SIGPIPE under
set -euo pipefail) with pwgen -sA0 20 1 which is already in PATH
- modules/bitcoin/rtl.nix: lowercase Authentication->authentication and
Settings->settings per RTL v0.15.8 schema; add lnServerUrl pointing
to LND REST endpoint; move swapServerUrl/boltzServerUrl inside
settings; apply same fixes to CLN branch
Co-authored-by: naturallaw777 <99053422+naturallaw777@users.noreply.github.com>
- Add btcexplorercookiefile to BTCPay deterministic config so NBXplorer
cookie authentication succeeds (fixes 401 Unauthorized)
- Set WorkingDirectory to package lib dir so ASP.NET Core can locate
wwwroot and LanguageService.ctor does not throw ArgumentNullException
- Update regression test to assert cookie file path and WorkingDirectory
Co-authored-by: naturallaw777 <99053422+naturallaw777@users.noreply.github.com>
ensureClauses generates ALTER ROLE clauses, not GRANT statements, so
'"DATABASE btcpayserver" = "ALL PRIVILEGES"' produced invalid SQL.
ensureDBOwnership = true makes each user own its database, which is
the intended effect.
The services.lnd.macaroons submodule has no 'enable' option — macaroons
are implicitly enabled by being defined. Replace 'enable = true' with
'user = cfg.btcpayserver.user' which is the correct option for
controlling macaroon file ownership.
The option services.postgresql.ensureUsers.*.ensurePermissions was renamed
to ensureClauses in nixpkgs unstable. Update both btcpayserver and nbxplorer
database user configurations.
Replace all pkgs-stable references with pkgs.stable (from overlay-stable in flake.nix):
- Remove pkgs-stable from function arguments
- Use pkgs.stable.nbxplorer (2.6.10 from nixos-26.05)
- Use pkgs.stable.btcpayserver (2.4.2 from nixos-26.05)
1. flake.nix: Add pkgs-stable as specialArg for modules
2. btcpayserver.nix:
- Fix preStart script syntax error (closing brace on new line)
- Use pkgs-stable.btcpayserver (2.4.2) instead of pkgs.btcpayserver
- Use pkgs-stable.nbxplorer (2.6.10) instead of pkgs.nbxplorer
- Remove clightning references (Sovran is LND-only)
- Create packages/rtl/default.nix to build RTL from source
- Update modules/bitcoin/rtl.nix to use vendored package instead of pkgs.rtl
- Remove clightning/lightning-loop references (Sovran is LND-only)
- Simplify to LND-only configuration
The joinmarket-ob-watcher module was not imported from nix-bitcoin,
so referencing it in the onion-services defaults caused an
"attribute 'joinmarket-ob-watcher' missing" evaluation error.
- Previous lnd-only refactor removed the '++ optional clightning' line
but left 'extraGroups = [ cfg.nbxplorer.group ]' without trailing ';'
-> syntax error: unexpected '=', expecting ';' at home = ...
- Add ';' to fix build on both f13ff45 and 8b8c811
- lndconnect.nix: rewrite to lnd-only (320 lines removed) - remove
services.clightning.plugins.clnrest.lnconnect and
services.clightning-rest.lndconnect (Sovran only uses services.lnd.lndconnect for Zeus)
Fixes: 'services.clightning.plugins.clnrest.address does not exist' on f13ff45
and 'attribute enable missing' at lndconnect.nix:216
- btcpayserver.nix: remove liquidd (lbtc) dead branch - Sovran uses lbtc=false
* remove inherit (config.services) bitcoind liquidd -> just bitcoind
* lbtc chains/rpc handling is dead code, keep guards but no need for liquidd service
- mempool.nix: remove fulcrum branch - Sovran uses electrs only
- rtl.nix: remove lightning-loop block and clightning service block (lnd only)
- Delete modules/bitcoin/stubs.nix entirely - no dead service references left
* liquidd, fulcrum, lightning-loop, joinmarket now not referenced at all
* clightning already handled via guards, not stubs
- Result: modules/bitcoin/ is now 100% tailored to Sovran (bitcoind+knots, lnd, electrs, rtl, btcpayserver, mempool)
No more stubs whack-a-mole on any nixos-unstable
- Move modules/vendor/nix-bitcoin/* -> modules/bitcoin/* (clean, Sovran-owned)
* modules/bitcoin/default.nix imports the 6 tailored services
* modules/bitcoin/common.nix bundles secrets/onion/lib
* modules/bitcoin/stubs.nix kept minimal (no clightning)
* packages/lndinit/default.nix replaces pkgs/sovran-overlay.nix
(lnd.nix now uses pkgs.callPackage ../../packages/lndinit {})
- Remove pkgs/sovran-overlay.nix, pkgs/nbxplorer.nix, pkgs/README.md
* No global overlay - lndinit is a normal package in packages/
- Remove modules/vendor/ entirely
- Update flake.nix: drop overlay-sovran, import ./modules/bitcoin
instead of ./modules/vendor/nix-bitcoin/modules.nix
- No more random vendor/ or pkgs/ dirs - follows Sovran convention:
modules/ for NixOS modules, packages/ for packages
- f13ff45 HAS services.clightning but only as { plugins = ... }, no enable
-> 'option does not exist, did you mean plugins' (your error)
- Sovran never uses lightningBackend == clightning (uses lnd),
so just don't set services.clightning.enable at all
- Also make lnd port conflict check always true for clightning case
- Fixes both f13ff45 (has clightning/plugins) and 8b8c811 (removed)
- f13ff45 (staging-dev) HAS both services -> unconditional stubs duplicate
- Remove both from stubs.nix, keep only liquidd/fulcrum/etc.
- lndconnect.nix left as is (defines clightning-rest.lndconnect) - safe on f13ff45 where base exists
- On 8b8c811 where clightning is removed, Sovran doesn't use it anyway (lnd only), so guards in btcpayserver/lnd prevent use
nixpkgs 8b8c811 (2026-08-08) removed services.clightning.enable,
causing btcpayserver.nix:124 to throw 'option does not exist' on
nixos-rebuild (your error). Sovran uses lnd only, never clightning,
but evaluation still throws.
- Add modules/vendor/nix-bitcoin/stubs.nix to provide missing options
as false stubs: clightning, clightning-rest, liquidd, fulcrum,
lightning-loop/pool, joinmarket
- Guard btcpayserver/rtl/lnd/mempool clightning/liquidd references
with config.services ? X checks
- Trim enable-tor.nix onionServices for removed services
- Remove inputs.nix-bitcoin (fort-nix/nix-bitcoin/release) from flake.nix
- Vendor only 6 services actually used by Sovran: bitcoind, electrs,
lnd (+lndconnect), rtl, btcpayserver, mempool + supporting infra:
secrets, onion-services/addresses, operator, nodeinfo, security,
versioning
- All packages now from nixpkgs directly (pkgs.*) — no pinned pkgs
- Keep nix-bitcoin.* option namespace for compatibility
- backups.nix removed: Sovran uses rsnapshot to Second_Drive
(configuration.nix: hourly/daily to BTCEcoandBackup) — duplicity
remote backup not needed
- netns-isolation.nix replaced with stub (5 lines): original 365-line
bridge/iptables/ip-netns broke Caddy/AlbyHub/RTL a year ago and
is incompatible with nwc-wallets (requires enable=false). Stub
keeps option valid but warns if enabled.
- Add pkgs/sovran-overlay.nix for gaps only: lndinit + netns-exec stub
- Replace plain-text write of new_root_password in api_security_reset()
with scrypt-hashed storage via _hash_password(), matching how the free
password is already handled.
- Return new_root_password in the API response so the user sees it once
before it is irreversibly hashed on disk.
- Teach _resolve_credential() to detect scrypt hashes and display a
human-readable placeholder instead of raw hex in the Hub credentials UI.
- Harden root-password-setup systemd service: if the secrets file already
contains a hash, skip chpasswd so a manual restart never sets the hash
as the literal login password.
naturallaw777
naturallaw777
Sovran Systems
copilot-swe-agent[bot]andnaturallaw777
Arena.ai Agent
Syntax Fix
Sovran Tailor
Sovran Clean
Sovran Fix
Sovran PR Bot
Contributor