diff --git a/docs/manual-backup.md b/docs/manual-backup.md deleted file mode 100644 index 1f608f4..0000000 --- a/docs/manual-backup.md +++ /dev/null @@ -1,93 +0,0 @@ -# Sovran Hub — Manual Backup - -The manual backup service copies critical system data from your Sovran Pro to an external USB drive, providing a third copy of your data (your Sovran Pro already maintains an automatic internal backup on its second drive). - -Backups are written to: - -``` -/Sovran_SystemsOS_Backup// -``` - -where `` is formatted as `YYYYMMDD_HHMMSS`. - ---- - -## Backup Stages - -The script always attempts all four stages, but skips stages that are irrelevant to the system's configured role (see [Per-Role Breakdown](#per-role-breakdown) below). - -| Stage | Directory | Contents | -|-------|-----------|----------| -| **1/4 — NixOS config** | `/etc/nixos/` | Full NixOS system configuration: `role-state.nix`, `custom.nix`, flake files, and any other config managed by the Hub | -| **2/4 — Secrets** | `/etc/nix-bitcoin-secrets` | Bitcoin/LND secrets stored under `/etc/` | -| **3/4 — Home directory** | `/home/` | All user home directories (`.cache/` and Trash are excluded) | -| **4/4 — System data** | `/var/lib/` | Full service data tree, including Vaultwarden, bitcoind, LND, sovran-hub config, domains, secrets, and other `/var/lib` service directories (logs excluded as appropriate) | - ---- - -## Per-Role Breakdown - -The script detects the system role at runtime by reading `/var/lib/sovran-hub/config.json` (falling back to `/etc/nixos/role-state.nix`) and adjusts its behaviour accordingly. - -### Server + Desktop (default) - -All services are enabled: Bitcoin, Matrix Synapse, Vaultwarden, WordPress, Nextcloud. - -| Stage | Status | Notes | -|-------|--------|-------| -| Stage 1 — NixOS config | ✅ Backed up | Full server configuration | -| Stage 2 — Secrets | ✅ Backed up | `/etc/nix-bitcoin-secrets` | -| Stage 3 — Home directory | ✅ Backed up | Desktop user data | -| Stage 4 — System data (`/var/lib`) | ✅ Backed up | Includes Vaultwarden, bitcoind, LND, sovran-hub config, domains, secrets, and all other service data under `/var/lib` (logs excluded) | - -This produces the largest backup. All four stages generate meaningful data. - -### Desktop Only - -All server services are disabled (`bitcoin = false`, `synapse = false`, `vaultwarden = false`, `wordpress = false`, `nextcloud = false`). Only GNOME desktop is active. - -| Stage | Status | Notes | -|-------|--------|-------| -| Stage 1 — NixOS config | ✅ Backed up | Simpler config (no server services) | -| Stage 2 — Secrets | ⏭️ Skipped | `/etc/nix-bitcoin-secrets` is not applicable for Desktop Only role | -| Stage 3 — Home directory | ✅ Backed up | **The most important data for this role** | -| Stage 4 — System data (`/var/lib`) | ✅ Backed up | Full `/var/lib` backup with `/var/lib/lnd` excluded for Desktop Only role | - -This produces the smallest and fastest backup. Stages 1 and 3 are the primary sources of meaningful data. - -### Node (Bitcoin-only) - -Only the Bitcoin ecosystem is active: `bitcoind`, `electrs`, `lnd`, `rtl`, `btcpay`, `mempool`, and `bip110`. All other server services are disabled. - -| Stage | Status | Notes | -|-------|--------|-------| -| Stage 1 — NixOS config | ✅ Backed up | Node-specific configuration | -| Stage 2 — Secrets | ✅ Backed up | `/etc/nix-bitcoin-secrets` | -| Stage 3 — Home directory | ✅ Backed up | User data | -| Stage 4 — System data (`/var/lib`) | ✅ Backed up | **Critical** — includes Lightning wallet/channel data plus all other `/var/lib` service data | - -All four stages run, matching Server + Desktop behaviour. Some non-Bitcoin service directories under `/var/lib` may be sparse or absent depending on role. - ---- - -## Backup Manifest - -After all stages complete, the script writes a `BACKUP_MANIFEST.txt` file inside the timestamped backup directory. This file records the date, hostname, detected role, target drive, and a directory listing of everything that was backed up. - ---- - -## Running the Backup - -The backup is triggered from the Sovran Hub web UI. You can also run it directly: - -```bash -# Auto-detect the first external USB drive -sudo bash /path/to/sovran-hub-backup.sh - -# Specify a target drive explicitly -sudo BACKUP_TARGET=/run/media// bash /path/to/sovran-hub-backup.sh -``` - -The script requires at least **10 GB** of free space on the target drive and will refuse to write to internal system drives. - -Logs are written to `/var/log/sovran-hub-backup.log` and the current status (`RUNNING`, `SUCCESS`, or `FAILED`) is tracked in `/var/log/sovran-hub-backup.status`. diff --git a/docs/remote-deploy-headscale.md b/docs/remote-deploy-headscale.md deleted file mode 100644 index 83769bb..0000000 --- a/docs/remote-deploy-headscale.md +++ /dev/null @@ -1,472 +0,0 @@ -# Remote Deployment via Headscale (Self-Hosted Tailscale) - -This guide covers the Sovran Systems remote deployment system built on [Headscale](https://headscale.net) — a self-hosted, open-source implementation of the Tailscale coordination server. Freshly booted ISOs automatically join a private WireGuard mesh VPN without any per-machine key pre-generation. - ---- - -## Architecture Overview - -``` -┌─────────────────────────────────────────────────────────┐ -│ Internet │ -└────────────┬─────────────────────┬──────────────────────┘ - │ │ - ▼ ▼ -┌────────────────────┐ ┌─────────────────────────────────┐ -│ Admin Workstation │ │ Sovran VPS │ -│ │ │ ┌─────────────────────────────┐ │ -│ tailscale up │ │ │ Headscale (port 8080) │ │ -│ --login-server │◄──┼─►│ Coordination server │ │ -│ hs.example.com │ │ ├─────────────────────────────┤ │ -│ │ │ │ Provisioning API (9090) │ │ -└────────────────────┘ │ │ POST /register │ │ - │ │ GET /machines │ │ - │ │ GET /health │ │ - │ ├─────────────────────────────┤ │ - │ │ Caddy (80/443) │ │ - │ │ hs.example.com → :8080 │ │ - │ │ prov.example.com → :9090 │ │ - │ └─────────────────────────────┘ │ - └─────────────────────────────────┘ - ▲ - │ WireGuard mesh (Tailnet) - ▼ - ┌─────────────────────────────────┐ - │ Deploy Target Machine │ - │ │ - │ Boot live ISO → │ - │ sovran-auto-provision → │ - │ POST /register → │ - │ tailscale up --authkey=... │ - └─────────────────────────────────┘ -``` - -**Components:** -- **`sovran-provisioner.nix`** — NixOS module deployed on a separate VPS; runs Headscale + provisioning API + Caddy. -- **Live ISO** (`iso/common.nix`) — Auto-registers with the provisioning server and joins the Tailnet on boot. -- **`remote-deploy.nix`** — Post-install NixOS module that uses Tailscale/Headscale for ongoing access. - ---- - -## Part 1: VPS Setup — Deploy `sovran-provisioner.nix` - -### Prerequisites - -- A NixOS VPS (any provider) with a public IP -- Two DNS A records pointing to your VPS: - - `hs.yourdomain.com` → VPS IP (Headscale coordination server) - - `prov.yourdomain.com` → VPS IP (Provisioning API) -- Ports 80, 443 (TCP) and 3478 (UDP, STUN/DERP) open in your VPS firewall - -### DNS Records - -| Type | Name | Value | -|------|-----------------------|------------| -| A | `hs.yourdomain.com` | `` | -| A | `prov.yourdomain.com` | `` | - -### NixOS Configuration - -Add the following to your VPS's `/etc/nixos/configuration.nix`: - -```nix -{ config, lib, pkgs, ... }: - -{ - imports = [ - ./hardware-configuration.nix - /path/to/sovran-provisioner.nix # or fetch from the repo - ]; - - sovranProvisioner = { - enable = true; - domain = "prov.yourdomain.com"; - headscaleDomain = "hs.yourdomain.com"; - - # Optional: customise defaults - headscaleUser = "sovran-deploy"; # namespace for deploy machines - adminUser = "admin"; # namespace for your workstation - keyExpiry = "1h"; # pre-auth keys expire after 1 hour - rateLimitMax = 10; # max registrations per window - rateLimitWindow = 60; # window in seconds - }; - - # Required for Caddy ACME (Let's Encrypt) - networking.hostName = "sovran-vps"; - system.stateVersion = "24.11"; -} -``` - -### Deploy - -```bash -nixos-rebuild switch -``` - -Caddy will automatically obtain TLS certificates via Let's Encrypt. - -### Retrieve the Enrollment Token - -```bash -cat /var/lib/sovran-provisioner/enroll-token -``` - -Keep this token secret — it is used to authenticate ISO registrations. The token is auto-generated on first boot and stored at this path. You never need to set it manually. Just `cat` it from the VPS and copy it to `iso/secrets/enroll-token` before building the ISO. - ---- - -## Part 2: Admin Workstation Setup - -Join your Tailnet as an admin so you can reach deployed machines: - -### Install Tailscale - -Follow the [Tailscale installation guide](https://tailscale.com/download) for your OS, or on NixOS: - -```nix -services.tailscale.enable = true; -``` - -### Join the Tailnet - -```bash -sudo tailscale up --login-server https://hs.yourdomain.com --accept-dns=false -``` - -> **Note:** The `--accept-dns=false` flag prevents Tailscale from taking over your system DNS resolver. This is important if you are behind a VPN (see [Troubleshooting](#troubleshooting) below). - -Tailscale prints a URL. Open it and copy the node key (starts with `mkey:`). - -### Approve the Node in Headscale - -On the VPS, first find the numeric user ID for the `admin` user, then register the node: - -```bash -# Look up the numeric ID for the admin user (Headscale 0.28.0 requires -u ) -headscale users list -o json - -# Register the node using the numeric user ID -headscale nodes register -u --key mkey:xxxxxxxxxxxxxxxx -``` - -Your workstation is now on the Tailnet. You can list nodes: - -```bash -headscale nodes list -``` - ---- - -## Part 3: Building the Deploy ISO - -### Add Secrets (gitignored) - -The secrets directory `iso/secrets/` is gitignored. Populate it before building: - -```bash -# Copy the enrollment token from the VPS -ssh root@ cat /var/lib/sovran-provisioner/enroll-token > iso/secrets/enroll-token - -# Set the provisioner URL -echo "https://prov.yourdomain.com" > iso/secrets/provisioner-url -``` - -These files are baked into the ISO at build time. If the files are absent the ISO still builds — the auto-provision service exits cleanly with "No enroll token found, skipping auto-provision", leaving DIY users unaffected. - -### Build the ISO - -```bash -nix build .#nixosConfigurations.sovran_systemsos-iso.config.system.build.isoImage -``` - -The resulting ISO is in `./result/iso/`. - ---- - -## Part 4: Deployment Workflow - -### Step-by-Step - -1. **Hand the ISO to the remote person** — they burn it to a USB drive and boot. - -2. **ISO boots and auto-registers** — `sovran-auto-provision.service` runs automatically: - - Reads `enroll-token` and `provisioner-url` from `/etc/sovran/` - - `POST https://prov.yourdomain.com/register` with hostname + MAC - - Receives a Headscale pre-auth key - - Runs `tailscale up --login-server=... --authkey=...` - - The machine appears in `headscale nodes list` within ~30 seconds - -3. **Approve the node (if not using auto-approve)** — on the VPS: - ```bash - headscale nodes list - # Note the node key for the new machine - ``` - -4. **SSH from your workstation** — once the machine is on the Tailnet: - ```bash - # Get the machine's Tailscale IP - headscale nodes list | grep sovran-deploy- - - # SSH in - ssh root@100.64.x.x # password: sovran-remote (live ISO default) - ``` - -5. **Run the headless installer**: - - The `--deploy-key` is your SSH public key that gets injected into `root`'s `authorized_keys` on the deployed machine. This grants full root access for initial setup. Generate it once on your workstation if you haven't already: - ```bash - ssh-keygen -t ed25519 -f ~/.ssh/sovran-deploy -C "sovran-deploy" - ``` - After deployment is complete and you disable deploy mode, this key is removed. - - ```bash - sudo sovran-install-headless.sh \ - --disk /dev/sda \ - --role server \ - --deploy-key "$(cat ~/.ssh/sovran-deploy.pub)" \ - --headscale-server "https://hs.yourdomain.com" \ - --headscale-key "$(headscale preauthkeys create -u $(headscale users list -o json | jq -r '.[] | select(.name=="sovran-deploy") | .id') -e 2h -o json | jq -r '.key')" - ``` - -6. **Machine reboots into Sovran_SystemsOS** — `deploy-tailscale-connect.service` runs: - - Reads `/var/lib/secrets/headscale-authkey` - - Joins the Tailnet with a deterministic hostname (`sovran-`) - -7. **Post-install SSH and RDP**: - ```bash - # SSH over Tailnet - ssh root@ - - # RDP over Tailnet (desktop role) — Sovran_SystemsOS uses GNOME Remote Desktop (native Wayland RDP) - # Retrieve the auto-generated RDP password: - ssh root@ cat /var/lib/gnome-remote-desktop/rdp-password - # Then connect with any RDP client (Remmina, GNOME Connections, Microsoft Remote Desktop): - # Host: :3389 User: sovran Password: - ``` - -8. **Disable deploy mode** — edit `/etc/nixos/custom.nix` on the target, set `enable = false`, then: - ```bash - sudo nixos-rebuild switch - ``` - ---- - -## Part 5: Post-Install Access - -### SSH - -```bash -# Over Tailnet -ssh root@100.64.x.x -``` - -### RDP (desktop/server roles) - -Sovran_SystemsOS uses **GNOME Remote Desktop** (native Wayland RDP — not xfreerdp). The RDP service auto-generates credentials on first boot. - -**Username:** `sovran` -**Password:** auto-generated — retrieve it via SSH: -```bash -ssh root@ cat /var/lib/gnome-remote-desktop/rdp-password -``` - -Connect using any RDP client (Remmina, GNOME Connections, Microsoft Remote Desktop) to `:3389`. - ---- - -## Security Model - -| Concern | Mitigation | -|---------|-----------| -| Enrollment token theft | Token only triggers key generation; it does not grant access to the machine itself | -| Rogue device joins Tailnet | Visible in `headscale nodes list`; removable instantly with `headscale nodes delete` | -| Pre-auth key reuse | Keys are ephemeral and expire in 1 hour (configurable via `keyExpiry`) | -| Rate limiting | Provisioning API limits to 10 registrations/minute by default (configurable) | -| SSH access | Requires ed25519 key injected at install time; password authentication disabled | -| Credential storage | Auth key written to `/var/lib/secrets/headscale-authkey` (mode 600) on the installed OS | - -### Token Rotation - -To rotate the enrollment token: - -1. On the VPS: - ```bash - openssl rand -hex 32 > /var/lib/sovran-provisioner/enroll-token - chmod 600 /var/lib/sovran-provisioner/enroll-token - ``` - -2. Update `iso/secrets/enroll-token` and rebuild the ISO. - -Old ISOs with the previous token will fail to register (receive 401). - ---- - -## Monitoring - -### List Active Tailnet Nodes - -```bash -# On the VPS -headscale nodes list -``` - -### List Registered Machines (Provisioning API) - -```bash -curl -s -H "Authorization: Bearer $(cat /var/lib/sovran-provisioner/enroll-token)" \ - https://prov.yourdomain.com/machines | jq . -``` - -### Health Check - -```bash -curl https://prov.yourdomain.com/health -# {"status": "ok"} -``` - -### Provisioner Logs - -```bash -journalctl -u sovran-provisioner -f -``` - -### Headscale Logs - -```bash -journalctl -u headscale -f -``` - ---- - -## Cleanup - -### Remove a Machine from the Tailnet - -```bash -headscale nodes list -headscale nodes delete --identifier -``` - -### Disable Deploy Mode on an Installed Machine - -Edit `/etc/nixos/custom.nix`: - -```nix -sovran_systemsOS.deploy.enable = false; -``` - -Then rebuild: - -```bash -nixos-rebuild switch -``` - -This stops the Tailscale connect service. - -### Revoke All Active Pre-Auth Keys - -```bash -# List pre-auth keys (Headscale 0.28.0: no --user flag on list) -headscale preauthkeys list - -# Expire a specific key — use numeric user ID (-u ) -# First find the user ID: -headscale users list -o json -# Then expire the key: -headscale preauthkeys expire -u --key -``` - ---- - -## Troubleshooting - -### VPN Conflicts (Mullvad, WireGuard, etc.) - -**Symptom:** `tailscale up` hangs or fails with `connection refused` on port 443, even though `curl https://hs.yourdomain.com/health` works fine. - -**Cause:** VPNs like Mullvad route all traffic — including Tailscale's control-plane connections — through the VPN tunnel. Additionally, Tailscale's DNS handler (`--accept-dns=true` by default) hijacks DNS resolution and may prevent correct resolution of your Headscale server even when logged out. - -**Solution:** -1. Disconnect your VPN temporarily and retry `tailscale up`. -2. If you need the VPN active, use split tunneling to exclude `tailscaled`: - ```bash - # Mullvad CLI - mullvad split-tunnel add $(pidof tailscaled) - ``` - Or in the Mullvad GUI: **Settings → Split tunneling → Add tailscaled**. -3. Always pass `--accept-dns=false` when enrolling to avoid DNS hijacking: - ```bash - sudo tailscale up --login-server https://hs.yourdomain.com --authkey --accept-dns=false - ``` - ---- - -### "RATELIMIT" in tailscaled Logs - -**Symptom:** `journalctl -u tailscaled` shows lines like: -``` -[RATELIMIT] format("Received error: %v") -``` - -**Cause:** This is **NOT** a server-side rate limit from Headscale. It is tailscaled's internal log suppressor de-duplicating repeated connection-refused error messages. The real underlying error is `connection refused`. - -**What to check:** -1. Is Headscale actually running? `curl https://hs.yourdomain.com/health` -2. Is your VPN blocking the connection? (see VPN Conflicts above) -3. Is there a firewall blocking port 443? - ---- - -### "connection refused" on Port 443 - -If `tailscale up` fails but `curl` works, the issue is usually DNS or VPN: - -```bash -# Does curl reach Headscale successfully? -curl -v https://hs.yourdomain.com/health - -# Force IPv4 vs IPv6 to identify if it's an address-family issue -curl -4 https://hs.yourdomain.com/health -curl -6 https://hs.yourdomain.com/health - -# Check what IP headscale resolves to -dig +short hs.yourdomain.com - -# What resolver is the system using? -cat /etc/resolv.conf -``` - -If curl works but tailscale doesn't, tailscaled may be using a different DNS resolver (e.g. its own `100.100.100.100` stub resolver). Fix: pass `--accept-dns=false`. - ---- - -### Headscale User ID Lookup (0.28.0) - -Headscale 0.28.0 removed `--user ` in favour of `-u `. To find the numeric ID for a user: - -```bash -headscale users list -o json -# Output: [{"id": "1", "name": "sovran-deploy", ...}, ...] - -# One-liner to get the ID for a specific user -headscale users list -o json | jq -r '.[] | select(.name=="sovran-deploy") | .id' -``` - -Then use the numeric ID in subsequent commands: -```bash -headscale preauthkeys create -u 1 -e 1h -o json -headscale nodes register -u 1 --key mkey:xxxx -``` - ---- - -## Reference - -| Component | Port | Protocol | Description | -|-----------|------|----------|-------------| -| Caddy | 80 | TCP | HTTP → HTTPS redirect | -| Caddy | 443 | TCP | HTTPS (Let's Encrypt) | -| Headscale | 8080 | TCP | Coordination server (proxied by Caddy) | -| Provisioner | 9090 | TCP | Registration API (proxied by Caddy) | -| DERP/STUN | 3478 | UDP | WireGuard relay fallback | -| Tailscale | N/A | WireGuard | Mesh VPN between nodes | diff --git a/docs/tech-support-security.md b/docs/tech-support-security.md deleted file mode 100644 index e0610a5..0000000 --- a/docs/tech-support-security.md +++ /dev/null @@ -1,259 +0,0 @@ -# Tech Support: Security Design, User Flow, and Incident Response - -## Overview - -The Sovran Hub includes a **Tech Support** feature that lets Sovran Systems -staff remotely diagnose and fix issues on a user's machine via SSH — without -ever having access to private keys or wallet funds. - -Wallet protection is the default. The user must make an active, time-limited -choice to grant support staff access to wallet files, and can revoke that -access at any time. - ---- - -## Implementation Details - -### Restricted User Instead of Root - -When a user enables support access the Hub: - -1. Ensures the `sovran-support` system user exists (declared declaratively in - `modules/core/tech-support.nix`; the Hub also provisions it on demand as a - fallback on non-NixOS systems). -2. Writes the Sovran Systems public SSH key **only** to - `/var/lib/sovran-support/.ssh/authorized_keys`, not to root's - `authorized_keys`. -3. Applies POSIX ACLs (`setfacl -R -m u:sovran-support:---`) to every wallet - directory that exists on disk, denying all access by the support user. -4. Records a timestamped `SUPPORT_ENABLED` event in the audit log at - `/var/log/sovran-support-audit.log`. - -When the session ends (or if the Hub cannot create the restricted user), the -key is removed and all ACLs are revoked immediately. - -### Protected Wallet Paths - -The following directories are locked by default when a support session starts: - -| Path | Contents | -|------|----------| -| `/etc/nix-bitcoin-secrets` | nix-bitcoin generated secrets | -| `/var/lib/bitcoind` | Bitcoin Core chainstate and wallet | -| `/var/lib/lnd` | LND wallet and channel database | -| `/home` | User home directories | - -Paths are only locked if they exist on disk at the time the session starts. - -### POSIX ACL Mechanics - -POSIX ACLs on Linux handle access checks in this order: - -1. If the process UID matches the file owner UID → use owner permissions -2. **If there is a matching named-user ACL entry → use that entry's - permissions** (clamped by the mask entry) -3. If any group matches → use group permissions -4. Otherwise → use "other" permissions - -Setting `u:sovran-support:---` creates a named-user ACL entry with no -permissions. Because the named-user entry is checked before the group/other -entries, the support user cannot access those directories regardless of the -"other" permission bits. - -`setfacl` and `getfacl` are provided by the `acl` package, which is added to -`environment.systemPackages` by `modules/core/tech-support.nix`. - -### Fallback to Root (When Restricted User Cannot Be Created) - -If the `sovran-support` user does not exist and cannot be created (e.g., -`users.mutableUsers = false` and the declarative module has not been deployed -yet), the Hub falls back to adding the support key to root's -`authorized_keys`. The modal prominently warns the user when this has happened -so they can decide whether to end the session. - -### Audit Log - -Every session event is appended to `/var/log/sovran-support-audit.log`: - -``` -[2025-01-15 14:32:01 UTC] SUPPORT_ENABLED: restricted_user=True acl_applied=True protected_paths=4 -[2025-01-15 14:45:00 UTC] WALLET_UNLOCKED: duration=3600s expires=2025-01-15 15:45:00 UTC -[2025-01-15 15:45:00 UTC] WALLET_RELOCKED: auto-expired -[2025-01-15 16:01:22 UTC] SUPPORT_DISABLED -``` - -The last 100 lines of this log are accessible from the Hub UI while a session -is active (or after it ends, until the page is refreshed). - ---- - -## Security Tradeoffs - -### What This Protects Against - -- **Accidental wallet exposure** — support staff cannot read wallet files - during a normal session; they must ask the user to explicitly grant access. -- **Credential theft** — private keys in the wallet directories are not - visible to the `sovran-support` user by default. -- **Scope creep** — the restricted user account limits the blast radius of an - SSH session compared to direct root access. - -### Known Limitations - -| Limitation | Mitigation | -|------------|------------| -| Support user still has system-wide bash access | Restrict with `ForceCommand` or AppArmor in the NixOS config if a narrower scope is required | -| ACLs apply only to directories that exist at session start | If new wallet directories are created during a session, they are not auto-protected. Re-lock and re-enable support to pick up new paths | -| Root fallback grants full access | The Hub UI warns the user prominently; users should end the session if they are uncomfortable | -| `setfacl` / ACL filesystem support required | The `acl` package is declared in `tech-support.nix`; most Linux filesystems (ext4, btrfs, xfs) support ACLs by default | -| Wallet access grant is time-limited but lazy-expired | Expiry is checked on the next `/api/support/status` poll (every 10 seconds in the UI); there is a small window after expiry | - -### Defense-in-Depth Recommendations - -For environments that require stronger isolation, consider layering one or -more additional controls: - -- **`ForceCommand`** in `sshd_config` (or `~/.ssh/authorized_keys` command - prefix) to restrict the support user to a specific diagnostic script. -- **`ChrootDirectory`** in the `sshd_config` `Match User sovran-support` block - to confine the session to a prepared directory tree. -- **AppArmor or SELinux** profiles that deny the support process read access - to wallet paths at the kernel level. -- **Namespace/bind-mount overlays** (e.g., via a wrapper systemd unit) to - present a sanitized filesystem view. - ---- - -## User Flow - -``` -User opens Hub → Clicks "Tech Support" in sidebar - │ - ▼ -Modal: "Need help from Sovran Systems?" - • Explains what will happen - • Shows Wallet Protection notice - • User clicks "Enable Support Access" - │ - ▼ -Hub: 1. Creates / verifies sovran-support user - 2. Writes SSH key to that user's authorized_keys - 3. Applies POSIX ACL deny on all existing wallet paths - 4. Saves session metadata + writes SUPPORT_ENABLED to audit log - │ - ▼ -Modal: "Support Access is Active" - • Live session duration timer - • Wallet Files: Protected panel - – Optional: "Grant Wallet Access" (time-limited, user-chosen) - • "End Support Session" button - • "View Audit Log" button - │ - (User grants wallet access) - │ - ▼ -Hub: • Removes ACL deny entries - • Records WALLET_UNLOCKED event with expiry time - • Starts countdown timer in UI - │ - (Timer expires or user clicks "Re-lock Wallet Now") - │ - ▼ -Hub: • Re-applies ACL deny entries - • Removes WALLET_UNLOCK_FILE - • Records WALLET_RELOCKED event - │ - (User clicks "End Support Session") - │ - ▼ -Hub: 1. Removes SSH key from sovran-support authorized_keys - 2. Removes SSH key from root authorized_keys (legacy cleanup) - 3. Revokes any wallet unlock, re-applies ACL deny - 4. Verifies key is gone - 5. Records SUPPORT_DISABLED event - │ - ▼ -Modal: "Support Session Ended — SSH key removed" - • Shows verified removal status -``` - ---- - -## Incident Response - -### Scenario 1 — You accidentally granted wallet access and are unsure what was copied - -**Immediate steps:** - -1. Click **"Re-lock Wallet Now"** in the Hub modal, or click - **"End Support Session"** to simultaneously revoke SSH access and wallet - access. -2. Open the **Audit Log** from the Hub modal and note the timestamps of - `WALLET_UNLOCKED` and `WALLET_RELOCKED` events. -3. Check `/var/log/auth.log` (or `journalctl -u sshd`) for SSH login events - by `sovran-support` during the unlocked window. - -**Assessment:** - -- If no SSH login occurred during the wallet-unlocked window, your keys are - safe. -- If an SSH login did occur, treat private keys as potentially compromised. - -**Recovery if keys may be compromised:** - -| Wallet | Recovery action | -|--------|----------------| -| LND | Move all funds out using `lncli sendcoins` to a freshly generated on-chain address; close channels; recreate wallet | -| Sparrow | Sweep funds to a new wallet generated on an air-gapped device | -| Bisq | Withdraw all BSQ and BTC to external wallets; delete the Bisq data directory and recreate | -| nix-bitcoin secrets | Rotate all secrets with `nix-bitcoin-secrets generate` and redeploy | - -**Report the incident:** - -Contact Sovran Systems immediately at support@sovransystems.com with: -- The audit log output (`/var/log/sovran-support-audit.log`) -- The SSH auth log for the affected time window -- A description of what you were troubleshooting - ---- - -### Scenario 2 — Support session cannot be ended (button fails or server is unresponsive) - -**Manual key removal (run as root on the device):** - -```bash -# Remove from support user's authorized_keys -rm -f /var/lib/sovran-support/.ssh/authorized_keys - -# Remove from root's authorized_keys (fallback / legacy) -sed -i '/sovransystemsos-support/d' /root/.ssh/authorized_keys - -# Remove wallet unlock state -rm -f /var/lib/secrets/support-wallet-unlock - -# Re-apply wallet ACL protections -setfacl -R -m u:sovran-support:--- /etc/nix-bitcoin-secrets \ - /var/lib/bitcoind /var/lib/lnd /home 2>/dev/null || true - -# Restart sshd to drop any active connections -systemctl restart sshd -``` - ---- - -### Scenario 3 — You see an unexpected SUPPORT_ENABLED in the audit log - -This should never happen without physical or remote access to the Hub web -interface. If you see an unexpected entry: - -1. Immediately run the manual key removal commands above. -2. Change the Sovran Hub web interface password. -3. Check `/var/log/nginx/access.log` (or Caddy access logs) for unexpected - requests to `/api/support/enable`. -4. Consider rebooting the device to clear any in-memory state. -5. Report the incident to Sovran Systems. - ---- - -*This document is part of the Sovran_SystemsOS repository. For the -authoritative and up-to-date version, see the repository.*