The reported bug was the Hub being reachable from outside the local
network when Server + Desktop is active. c33457f guards the Caddy site
for sovransystemsos.local, but the Hub is an application that also
listens on a port of its own (8937), and a check in Caddy does nothing
for a client that never goes through Caddy. Whether a client could reach
the Hub depended on which door it used.
The Hub now asks the question itself. LanOnlyMiddleware is registered
outermost, so a client that is not on this computer or the local network
gets a bare 403 before authentication is considered; it never sees the
login page.
- Local means loopback, 10/8, 172.16/12, 192.168/16, 100.64/10
(Tailscale and other CGNAT/VPN ranges) and 169.254/16 over IPv4, and
::1, fc00::/7 and fe80::/10 over IPv6.
- IPv6 global addresses are not on the list. A global address belonging
to a laptop on the LAN cannot be told apart from a stranger's by the
address alone, and 2000::/3 is every public IPv6 address there is.
- ::ffff:a.b.c.d is read as the IPv4 address inside it.
- sovran_systemsOS.hub.extraLanNetworks adds networks (IPv4 or IPv6 CIDR)
for setups whose own devices use addresses outside those ranges. It is
checked at build time. The app ignores an entry it cannot parse and
refuses 0.0.0.0/0 and ::/0: it must never widen its policy by
guessing, and "everyone" is hub.lanOnly = false, asked for by name.
- sovran_systemsOS.hub.lanOnly (default true) turns the check off.
- The first refusal from each address is logged, naming the option to
change, so an operator whose own device is refused can find out why.
The list is capped so a scanner cannot fill the journal or memory.
Behind Caddy the policy applies to the real client, not to Caddy: uvicorn
takes the address from X-Forwarded-For only when the peer is 127.0.0.1.
Checked, not only reviewed:
- The real app with the config.json the module really generates (nix
eval on the nixpkgs revision flake.lock pins, read back from the
derivation), on a real socket with the source address chosen per
request: 127.0.0.1, 192.168/16, 100.64/10 and a declared extra
network are served; 203.0.113.9, 8.8.4.4 and an address just outside
the declared /28 get 403 on /login, / and /api/ping. /auto-login still
answers 303 to loopback and 403 to everyone else.
- With c33457f's Caddy guard in front, an IPv6 client at 2001:db8::9
passes Caddy (it is inside 2000::/3) and is refused by the Hub; an
IPv4 stranger has the connection closed by Caddy; a LAN client is
served.
- hub.extraLanNetworks accepts 203.0.113.0/28, 2001:db8:abcd::/48 and
bare hosts, and fails the build for /33, 300.1.1.1/8, 0.0.0.0/0, ::/0,
2001:db8::/129 and junk, with a message that says what to write.
Add tests/test_lan_policy.py and tests/test_hub_lan_only.py, and a note
in SECURITY.md.
LOGIN_FAIL_MAX was declared as "max failures in window before extra
delay" and then never read anywhere in the file. _record_failure() only
ever slept a flat LOGIN_FAIL_DELAY. So the Hub's entire defence against
online guessing was a constant 2 second pause per wrong password: no
escalation, no lockout, no ban, and fail2ban is configured for SSH only.
Worse, the old 60 second window could not have worked even if the
constant had been wired up. With a delay per attempt, reaching 10
failures takes about 80 seconds, so the earliest failures aged out of the
window before the count could ever reach the limit.
- security_helpers.py: new LoginThrottle. The delay ramps with the
failure count (2s, 4s, ... capped at 10s), and once LOGIN_FAIL_MAX
failures land inside the window the address is refused outright for
LOGIN_LOCKOUT_SECONDS (5 minutes). A successful login clears the
address, so an operator who fumbles a password is not penalised later.
The sleep is never taken under the lock, so one slow client cannot
stall every other login. Tracked addresses are evicted, so a
distributed sweep cannot grow the table without bound.
The window moves from 60s to 900s so the whole ramp fits inside it.
clock and sleep are injectable, which is what makes it testable.
- server.py: /api/login checks the lockout before the scrypt hash, so a
locked-out client costs almost nothing to reject, and answers 429 with
a human-readable wait instead of a bare 401.
- tests/test_login_throttle.py: covers the ramp, the cap, the lockout
firing and expiring, per-address isolation, clearing on success,
eviction, and that the limit is actually reachable inside the window.
Verified against the real app with TestClient: 10 wrong passwords return
401 and the 11th returns 429 "Too many failed attempts. Try again in
about 5 minute(s)." A correct password clears the counter.
The Hub no longer asks a STUN server, a public DNS resolver or a "what
is my IP" service for the home IP address. The DDNS update asks Njal.la
to use the address the request comes from ("&auto"), reads back the
address Njal.la says it recorded and saves it to
/var/lib/secrets/external-ip. Njal.la is the only third party that
learns the address; it has to, to publish it.
- Add app/sovran_systemsos_web/ddns_update.py, installed as
/etc/sovran/ddns-update.py and run by sovran-ddns-update.service. It
runs curl --ipv4 without redirects, accepts only a public IPv4
address, and rewrites the file atomically and only when the address
changes. Stored "&a=${IP}" URLs are converted when they are used and
"&quiet" is dropped.
- Rewrite modules/core/njalla.nix around that runner and delete
modules/core/public-ip.nix. Setting a sovran_systemsOS.publicIP.*
option now fails with a message that says where the address comes
from. Activation removes the old scripts in /var/lib/sovran. The
existing external-ip file keeps working.
- server.py reads the saved address and starts
sovran-ddns-update.service after a domain is saved, instead of looking
the address up itself.
- Element calling uses sovran_systemsOS.elementCalling.externalIP if
set, otherwise the saved address, and fails with a clear message when
neither exists or the address is not public. It no longer falls back
to STUN. livekit-external-ip.path re-runs livekit-turn-setup and
starts LiveKit when the address changes.
- Add tests/test_ddns_update.py.
- One "Bitcoin" category: the service catalog distinguishes bitcoin-base
from bitcoin-apps, which surfaced as two sidebar menus and two tile
sections. The Hub now normalizes both into a single "Bitcoin" category
(nav item with combined count, one tile section). CATEGORY_ALIASES in
constants.js maps the catalog keys; server config sends one
("bitcoin", "Bitcoin") entry and the node role allowlist is updated
to match. The nix catalog is unchanged.
- "Self-Hosted Apps" is now "Personal Apps" — every service here is
self-hosted, so the label added no distinction.
Updater/rebuild self-heal:
- Add a shared run_step wrapper used by both the update and rebuild
scripts. On the first failure matching a transient fetch/cache signature
(truncated tarball, corrupt NAR, hash mismatch, network timeout,
interrupted download), clear Nix's fetch caches and repair the store,
then retry once. Real config errors do not match and still fail loudly.
- The kernel-change boot fallback in the rebuild path is also wrapped.
- Fixes the reported 'cannot read file from tarball: Truncated tar archive
detected' failure, which a plain re-run cannot clear because Nix reuses
the corrupt cached archive.
Failed-update recovery / reporting:
- check_for_updates() now compares the running Hub version against the
branch VERSION, so a failed 'nix flake update' (lock advanced but no
generation staged) can no longer masquerade as 'up to date' and block
retries.
- /api/updates/check surfaces a persistent 'failed' state; /api/updates/run
never blocks a retry after a failure.
- Dashboard shows a red 'Update failed - click to retry' tile; the modal
offers a Retry Update button and stops offering a reboot on failure.
User-facing polish to match the element-calling port changes:
- helpers.js: port-forward guide example now shows the 40000-40099 range
instead of the removed 30000-40000.
- versions.json: livekit dev fallback 1.5.2 -> 1.13.6 (the Nix-generated
version in sovran-hub.nix already derives from pkgs.livekit.version).
- server.py: docstring/comment examples reference 40000-40099.
- CHANGELOG.md: record the calling changes under [Unreleased].
The tile/modal port tables themselves are data-driven from server.py's
port_requirements / _PORTS_ELEMENT_CALLING (updated in the previous commit),
so no further UI changes are needed.
Root cause of 'calls fail on Wi-Fi but work on mobile data': LiveKit only
advertised the public/WAN IP (rtc.node_ip), so LAN clients had to hairpin
through the router for media. Fixes and cleanup:
- rtc.advertise_internal_ip: true — also advertise the primary interface's
LAN host candidate, so Wi-Fi callers connect directly (no hairpin).
- Drop rtc.port_range_start/end (30000-40000) and keep the single UDP mux
(udp_port: 7882). In LiveKit 1.13.x the range takes precedence over
udp_port, so media was actually spread over 10000 ports.
- Drop turn.tls_port: 5349 — LiveKit advertises turns:<domain>:443 to clients
regardless of tls_port, so a 5349 TURN/TLS listener was unreachable dead
config (and needless attack surface).
- Pin TURN relay allocation to 40000-40099 (disjoint from the media mux) and
open/forward that range; the old default overlapped RTC media.
- turn.allow_restricted_peer_cidrs with the LAN subnet derived from the
primary interface: without it the relay refuses to deliver to the private
LAN host candidate and its final hop would fall back to WAN hairpin.
- Update Hub port guidance (server.py) to the new list.
The Hub vendored a second copy of the NWC stack
(app/sovran_systemsos_web/nwc_hub_manager.py, nwc_audit.py,
nwc_lnurl_service.py, nwc_wallet_cli.py) and built its own nwc-wallet /
nwc-lnurl binaries from it. That copy drifted from the pinned Alby Hub
API contract (appId vs toAppId) and duplicated code that Sovran_Bitcoin
already ships and fixes.
Changes:
- Delete the four vendored modules; server.py now imports the canonical
implementation directly (from sovran_nwc import nwc_hub_manager) from
the sovran-nwc package (pkgs.sovran-bitcoin.nwc). API fixes in
Sovran_Bitcoin now propagate to the Hub web app automatically.
- sovran-hub-web launcher: add <sovran-nwc>/lib/sovran-nwc to
sys.path so the import resolves.
- Stop shipping nwc-wallet / nwc-lnurl binaries from sovran-hub-web:
the flake already provides them (env-wrapped nwc-wallet with
NWC_* vars via albyhub.nix, and nwc-lnurl.service via lnurl.nix).
Requires a Sovran_Bitcoin rev containing the toAppId fix (and the
LNURL module audit-log fix); bump the flake input afterwards:
nix flake update sovran-bitcoin
Test:
- nixos-rebuild switch
- Hub Wallet Connections tab still lists/creates wallets
- nwc-wallet list works from the operator shell
- journalctl -u nwc-lnurl shows no import/contract errors
The public IP was previously detected independently in three places,
each contacting a different third party: the Hub (HTTPS echo via
api.ipify.org / ifconfig.me / icanhazip.com on every API call and
background tick), DDNS (myip.opendns.com via OpenDNS), and LiveKit
(embedded STUN). Consolidate into a single detector with one shared
cache so every consumer reads the same value with minimal exposure.
- add modules/core/public-ip.nix: installs /var/lib/sovran/public-ip.py
(pure Python stdlib, no new deps) writing /var/lib/secrets/external-ip
- detection chain (first success wins): explicit pin, fresh cache
(default TTL 300s), STUN binding request over UDP (one packet, no
metadata), DNS myip.opendns.com query, then OPT-IN HTTPS echo
(publicIP.httpsEcho, empty by default — never contacted unless listed)
- privacy: while the cache is fresh zero third parties are contacted;
at most one party learns the IP per refresh interval, via the least
exposing mechanism available
- hub (server.py): _get_external_ip() now reads the shared detector /
cache instead of calling ipify/ifconfig/icanhazip directly
- ddns (njalla.nix): use the shared detector instead of a separate
OpenDNS dig; allow the hardened service to write /var/lib/secrets
- element-calling: livekit-turn-setup falls back to the shared
detector on cold boot; add LiveKit webhooks to lk-jwt-service
(sfu_webhook) so abrupt disconnects are cleaned up immediately;
set LIVEKIT_SANITY_CHECK_INTERVAL_SECONDS=60 as a missed-webhook
guard; drop the dead services.livekit.settings block and set
openFirewall=false (Caddy fronts the SFU; no public 7880/tcp)
- new options: sovran_systemsOS.publicIP.{stunServer,stunPort,
dnsResolver,httpsEcho,cacheTTL}
NixOS already knows whether a reboot is pending: /nix/var/nix/profiles/
system vs /run/current-system. Marker files only the Hub's own updater
wrote desynced for terminal-updated machines (and markers from older
updaters could never clear), pinning the badge on forever. Reconcile
REBOOT_REQUIRED against live state on every read; the stale marker
self-heals to IDLE. The .generation marker write is now informational.
The full-system updater runs as a detached systemd service and can finish
successfully even when the browser loses its status connection. In that
case the update log and status file correctly report REBOOT_REQUIRED, but
the Hub modal can remain on "Updating..." with its controls disabled.
There were four independent ways for the frontend to get stuck:
* update status fetches had no deadline, so a request that stayed pending
never rejected and never advanced the existing failure counter;
* setInterval started async polls without waiting for the previous poll,
allowing slow requests to overlap and responses to arrive out of order;
* each log chunk used textContent +=, replacing the complete and growing
Nix build log every two seconds, which could stall browser rendering and
was especially visible over RDP; and
* page reload, tab resume, and RDP reconnect did not reattach the modal to
the update status persisted by the backend.
This produced a dangerous UX mismatch: the machine had a fully staged
NixOS generation and was ready to reboot, while the Hub continued telling
the user that the update was still running.
Bound status requests with AbortController, prevent overlapping polls, and
replace the endless spinner after sustained failures with an explicit
"Update status unavailable" state and Retry Status action. Reconcile state
immediately on focus, visibility, online, page startup, and before starting
a new update. Use no-store requests and render verbose logs incrementally
with a bounded visible tail while retaining the complete report in memory.
Apply the same timeout and single-flight protection to rebuild polling.
Record the exact generation produced by `nixos-rebuild boot`. The Hub now
keeps REBOOT_REQUIRED visible until that generation matches
/run/current-system, then clears the marker after reboot. For an update
started by an older updater that did not write the marker, recover the
staged generation from the final nixos-rebuild log line. The dashboard
sidebar also distinguishes update-in-progress and restart-required states.
Regression coverage verifies generation marker/log recovery, pre- versus
post-reboot detection, request timeout wiring, single-flight polling,
connection-loss UX, RDP/tab resume reconciliation, bounded log rendering,
page-reload recovery, and JavaScript syntax.
Validation:
* python3 -m unittest discover -s tests -p 'test_*.py' -v (170 passed)
* node --check app/sovran_systemsos_web/static/js/*.js
* python3 -m py_compile for changed Python modules
* git diff --check
A Nix evaluation was not available in the development sandbox; the NixOS
module should still be evaluated and built in CI or on a test machine before
release.
The LND-only rewrite of lndconnect.nix shipped a wrapper Zeus cannot
use: unknown flags (--cert/--macaroon), a non-existent onion path
(free/lnd.onion), and a REST hidden service that collided with LND's
P2P onion. Restore the nix-bitcoin contract — dedicated lnd-rest
onion on port 8080, --nocert over Tor, admin macaroon in the URI —
and only persist a valid lndconnect:// URI for the Hub QR.
- Replace plain-text write of new_root_password in api_security_reset()
with scrypt-hashed storage via _hash_password(), matching how the free
password is already handled.
- Return new_root_password in the API response so the user sees it once
before it is irreversibly hashed on disk.
- Teach _resolve_credential() to detect scrypt hashes and display a
human-readable placeholder instead of raw hex in the Hub credentials UI.
- Harden root-password-setup systemd service: if the secrets file already
contains a hash, skip chpasswd so a manual restart never sets the hash
as the literal login password.
The Caddyfile is generated at runtime by caddy-generate-config.service
from /var/lib/domains/*, but the generator only re-runs when caddy.service
starts fresh. Saving a domain while Caddy is already running therefore
never seats the new virtual host — no proxying and no ACME cert — and the
Hub's reachability check shows a misleading 'ports 80/443' router error
until the next reboot or rebuild.
api_domains_set and api_domains_set_email now restart the generator and
reload Caddy (ExecReload: caddy reload --force, no dropped connections)
right after saving. Entirely skipped when Caddy is inactive — e.g. Node
role before its first domain-based service is enabled — because the
rebuild that enables the service starts caddy.service for the first time,
runs the generator first (requiredBy), and seats the already-saved domain
on its own. Best-effort throughout: a domain save never fails because of
a Caddy reload issue.
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
- The Hub header badge could render 'vdev' because the runtime
/etc/nixos/VERSION lookup fell back to the literal string 'dev' when
the file was missing (e.g. dev/test environments, or before the
Nix-generated config carried a version at all).
- modules/core/sovran-hub.nix now reads the repo's VERSION file at
Nix eval time and bakes a real semantic version (sovran_version)
into the generated config.json and a VERSION file shipped with the
package, so the Hub always has a solid value to display.
- server.py's _get_sovran_version() now reads that baked-in
sovran_version first, and explicitly rejects a literal 'dev' value
from any of its file-based fallbacks so the badge never shows
'vdev' again.
- templates/index.html + header.css: wrapped the title and the
version badge in a '.title-group' column so the version badge sits
directly underneath 'Sovran_SystemsOS Hub', left-aligned with it.
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
- Add release-stable.sh script with automatic tagging, CHANGELOG updates,
GitHub/Gitea release creation, and VERSION file management
- Update iso/common.nix to include version from VERSION file in ISO filename
- Add VERSION file (current: 1.0.3)
- Polish OS version badge in Sovran Hub header (top-right)
- Update README.md download link to versioned ISO
- Update CHANGELOG.md with existing tags (v1.0.0 – v1.0.3)
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Previously only bitcoind.service showed a version on the tile.
Now every service with a parsable Nix store path gets its version
displayed (electrs, btcpayserver, lnd, rtl, mempool, vaultwarden,
matrix-synapse, caddy, tor, etc.).
The existing _get_service_version() helper reads systemctl show
<unit> --property=ExecStart and extracts the version from the Nix
store path. Results are cached for 5 minutes per unit.
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Naming: user-facing 'Wallet Connections' -> 'Lightning Wallet Connections'
across the Hub, feature registry, tile, and NixOS modules. Internal ids
(nwc-wallets, albyhub.service, /api/nwc/*) are unchanged.
UX: the service-detail modal put status, domain diagnostics, router ports,
the enable/disable toggle, restart, the liquidity guide and the whole wallet
manager in one cramped scrolling column. For this feature the modal is now
980px wide and split into two tabs:
- Wallets: wallet grid, create/share/verify flows, collapsible liquidity guide
- Service & Setup: description, status, domain checklist, ports, enable, restart
A status dot and domain chip sit in the tab bar so state is visible from both
tabs, and the modal opens on Setup when the service is off or the Lightning
Address domain is unconfigured. Wallet cards gain a balance chip, pending
badge, a prominent address row, and separated destructive actions.
Non-NWC services keep the original single-column layout and width.
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Security Fix
Arena.ai Agent
naturallaw777
Sovran Systems
Sovran_SystemsOS
naturallaw777
Arena Agent
copilot-swe-agent[bot]andnaturallaw777
Contributor
