Sovran_Systems/Sovran_SystemsOS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'Delete docs directory' (#3) from staging-dev into stable

Browse Source 
Reviewed-on: #3
This commit was merged in pull request #3.
This commit is contained in:
naturallaw777
2026-05-21 16:52:57 -05:00
parent 629f5f76c2 342f60ce0d
commit 51609a1217
3 changed files with 0 additions and 824 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/manual-backup.md
-93
View File
@@ -1,93 +0,0 @@
# Sovran Hub — Manual Backup
The manual backup service copies critical system data from your Sovran Pro to an external USB drive, providing a third copy of your data (your Sovran Pro already maintains an automatic internal backup on its second drive).
Backups are written to:
```
<USB drive>/Sovran_SystemsOS_Backup/<timestamp>/
```
where `<timestamp>` is formatted as `YYYYMMDD_HHMMSS`.
---
## Backup Stages
The script always attempts all four stages, but skips stages that are irrelevant to the system's configured role (see [Per-Role Breakdown](#per-role-breakdown) below).
| Stage | Directory | Contents |
|-------|-----------|----------|
| **1/4 — NixOS config** | `/etc/nixos/` | Full NixOS system configuration: `role-state.nix`, `custom.nix`, flake files, and any other config managed by the Hub |
| **2/4 — Secrets** | `/etc/nix-bitcoin-secrets` | Bitcoin/LND secrets stored under `/etc/` |
| **3/4 — Home directory** | `/home/` | All user home directories (`.cache/` and Trash are excluded) |
| **4/4 — System data** | `/var/lib/` | Full service data tree, including Vaultwarden, bitcoind, LND, sovran-hub config, domains, secrets, and other `/var/lib` service directories (logs excluded as appropriate) |
---
## Per-Role Breakdown
The script detects the system role at runtime by reading `/var/lib/sovran-hub/config.json` (falling back to `/etc/nixos/role-state.nix`) and adjusts its behaviour accordingly.
### Server + Desktop (default)
All services are enabled: Bitcoin, Matrix Synapse, Vaultwarden, WordPress, Nextcloud.
| Stage | Status | Notes |
|-------|--------|-------|
| Stage 1 — NixOS config | ✅ Backed up | Full server configuration |
| Stage 2 — Secrets | ✅ Backed up | `/etc/nix-bitcoin-secrets` |
| Stage 3 — Home directory | ✅ Backed up | Desktop user data |
| Stage 4 — System data (`/var/lib`) | ✅ Backed up | Includes Vaultwarden, bitcoind, LND, sovran-hub config, domains, secrets, and all other service data under `/var/lib` (logs excluded) |
This produces the largest backup. All four stages generate meaningful data.
### Desktop Only
All server services are disabled (`bitcoin = false`, `synapse = false`, `vaultwarden = false`, `wordpress = false`, `nextcloud = false`). Only GNOME desktop is active.
| Stage | Status | Notes |
|-------|--------|-------|
| Stage 1 — NixOS config | ✅ Backed up | Simpler config (no server services) |
| Stage 2 — Secrets | ⏭️ Skipped | `/etc/nix-bitcoin-secrets` is not applicable for Desktop Only role |
| Stage 3 — Home directory | ✅ Backed up | **The most important data for this role** |
| Stage 4 — System data (`/var/lib`) | ✅ Backed up | Full `/var/lib` backup with `/var/lib/lnd` excluded for Desktop Only role |
This produces the smallest and fastest backup. Stages 1 and 3 are the primary sources of meaningful data.
### Node (Bitcoin-only)
Only the Bitcoin ecosystem is active: `bitcoind`, `electrs`, `lnd`, `rtl`, `btcpay`, `mempool`, and `bip110`. All other server services are disabled.
| Stage | Status | Notes |
|-------|--------|-------|
| Stage 1 — NixOS config | ✅ Backed up | Node-specific configuration |
| Stage 2 — Secrets | ✅ Backed up | `/etc/nix-bitcoin-secrets` |
| Stage 3 — Home directory | ✅ Backed up | User data |
| Stage 4 — System data (`/var/lib`) | ✅ Backed up | **Critical** — includes Lightning wallet/channel data plus all other `/var/lib` service data |
All four stages run, matching Server + Desktop behaviour. Some non-Bitcoin service directories under `/var/lib` may be sparse or absent depending on role.
---
## Backup Manifest
After all stages complete, the script writes a `BACKUP_MANIFEST.txt` file inside the timestamped backup directory. This file records the date, hostname, detected role, target drive, and a directory listing of everything that was backed up.
---
## Running the Backup
The backup is triggered from the Sovran Hub web UI. You can also run it directly:
```bash
# Auto-detect the first external USB drive
sudo bash /path/to/sovran-hub-backup.sh
# Specify a target drive explicitly
sudo BACKUP_TARGET=/run/media/<user>/<drive> bash /path/to/sovran-hub-backup.sh
```
The script requires at least **10 GB** of free space on the target drive and will refuse to write to internal system drives.
Logs are written to `/var/log/sovran-hub-backup.log` and the current status (`RUNNING`, `SUCCESS`, or `FAILED`) is tracked in `/var/log/sovran-hub-backup.status`.
docs/remote-deploy-headscale.md
-472
View File
@@ -1,472 +0,0 @@
# Remote Deployment via Headscale (Self-Hosted Tailscale)
This guide covers the Sovran Systems remote deployment system built on [Headscale](https://headscale.net) — a self-hosted, open-source implementation of the Tailscale coordination server. Freshly booted ISOs automatically join a private WireGuard mesh VPN without any per-machine key pre-generation.
---
## Architecture Overview
```
┌─────────────────────────────────────────────────────────┐
│ Internet │
└────────────┬─────────────────────┬──────────────────────┘
│ │
▼ ▼
┌────────────────────┐ ┌─────────────────────────────────┐
│ Admin Workstation │ │ Sovran VPS │
│ │ │ ┌─────────────────────────────┐ │
│ tailscale up │ │ │ Headscale (port 8080) │ │
│ --login-server │◄──┼─►│ Coordination server │ │
│ hs.example.com │ │ ├─────────────────────────────┤ │
│ │ │ │ Provisioning API (9090) │ │
└────────────────────┘ │ │ POST /register │ │
│ │ GET /machines │ │
│ │ GET /health │ │
│ ├─────────────────────────────┤ │
│ │ Caddy (80/443) │ │
│ │ hs.example.com → :8080 │ │
│ │ prov.example.com → :9090 │ │
│ └─────────────────────────────┘ │
└─────────────────────────────────┘
▲
│ WireGuard mesh (Tailnet)
▼
┌─────────────────────────────────┐
│ Deploy Target Machine │
│ │
│ Boot live ISO → │
│ sovran-auto-provision → │
│ POST /register → │
│ tailscale up --authkey=... │
└─────────────────────────────────┘
```
**Components:**
- **`sovran-provisioner.nix`** — NixOS module deployed on a separate VPS; runs Headscale + provisioning API + Caddy.
- **Live ISO** (`iso/common.nix`) — Auto-registers with the provisioning server and joins the Tailnet on boot.
- **`remote-deploy.nix`** — Post-install NixOS module that uses Tailscale/Headscale for ongoing access.
---
## Part 1: VPS Setup — Deploy `sovran-provisioner.nix`
### Prerequisites
- A NixOS VPS (any provider) with a public IP
- Two DNS A records pointing to your VPS:
- `hs.yourdomain.com` → VPS IP (Headscale coordination server)
- `prov.yourdomain.com` → VPS IP (Provisioning API)
- Ports 80, 443 (TCP) and 3478 (UDP, STUN/DERP) open in your VPS firewall
### DNS Records
| Type | Name | Value |
|------|-----------------------|------------|
| A | `hs.yourdomain.com` | `<VPS IP>` |
| A | `prov.yourdomain.com` | `<VPS IP>` |
### NixOS Configuration
Add the following to your VPS's `/etc/nixos/configuration.nix`:
```nix
{ config, lib, pkgs, ... }:
{
imports = [
./hardware-configuration.nix
/path/to/sovran-provisioner.nix # or fetch from the repo
];
sovranProvisioner = {
enable = true;
domain = "prov.yourdomain.com";
headscaleDomain = "hs.yourdomain.com";
# Optional: customise defaults
headscaleUser = "sovran-deploy"; # namespace for deploy machines
adminUser = "admin"; # namespace for your workstation
keyExpiry = "1h"; # pre-auth keys expire after 1 hour
rateLimitMax = 10; # max registrations per window
rateLimitWindow = 60; # window in seconds
};
# Required for Caddy ACME (Let's Encrypt)
networking.hostName = "sovran-vps";
system.stateVersion = "24.11";
}
```
### Deploy
```bash
nixos-rebuild switch
```
Caddy will automatically obtain TLS certificates via Let's Encrypt.
### Retrieve the Enrollment Token
```bash
cat /var/lib/sovran-provisioner/enroll-token
```
Keep this token secret — it is used to authenticate ISO registrations. The token is auto-generated on first boot and stored at this path. You never need to set it manually. Just `cat` it from the VPS and copy it to `iso/secrets/enroll-token` before building the ISO.
---
## Part 2: Admin Workstation Setup
Join your Tailnet as an admin so you can reach deployed machines:
### Install Tailscale
Follow the [Tailscale installation guide](https://tailscale.com/download) for your OS, or on NixOS:
```nix
services.tailscale.enable = true;
```
### Join the Tailnet
```bash
sudo tailscale up --login-server https://hs.yourdomain.com --accept-dns=false
```
> **Note:** The `--accept-dns=false` flag prevents Tailscale from taking over your system DNS resolver. This is important if you are behind a VPN (see [Troubleshooting](#troubleshooting) below).
Tailscale prints a URL. Open it and copy the node key (starts with `mkey:`).
### Approve the Node in Headscale
On the VPS, first find the numeric user ID for the `admin` user, then register the node:
```bash
# Look up the numeric ID for the admin user (Headscale 0.28.0 requires -u <id>)
headscale users list -o json
# Register the node using the numeric user ID
headscale nodes register -u <admin-user-id> --key mkey:xxxxxxxxxxxxxxxx
```
Your workstation is now on the Tailnet. You can list nodes:
```bash
headscale nodes list
```
---
## Part 3: Building the Deploy ISO
### Add Secrets (gitignored)
The secrets directory `iso/secrets/` is gitignored. Populate it before building:
```bash
# Copy the enrollment token from the VPS
ssh root@<VPS> cat /var/lib/sovran-provisioner/enroll-token > iso/secrets/enroll-token
# Set the provisioner URL
echo "https://prov.yourdomain.com" > iso/secrets/provisioner-url
```
These files are baked into the ISO at build time. If the files are absent the ISO still builds — the auto-provision service exits cleanly with "No enroll token found, skipping auto-provision", leaving DIY users unaffected.
### Build the ISO
```bash
nix build .#nixosConfigurations.sovran_systemsos-iso.config.system.build.isoImage
```
The resulting ISO is in `./result/iso/`.
---
## Part 4: Deployment Workflow
### Step-by-Step
1. **Hand the ISO to the remote person** — they burn it to a USB drive and boot.
2. **ISO boots and auto-registers** — `sovran-auto-provision.service` runs automatically:
- Reads `enroll-token` and `provisioner-url` from `/etc/sovran/`
- `POST https://prov.yourdomain.com/register` with hostname + MAC
- Receives a Headscale pre-auth key
- Runs `tailscale up --login-server=... --authkey=...`
- The machine appears in `headscale nodes list` within ~30 seconds
3. **Approve the node (if not using auto-approve)** — on the VPS:
```bash
headscale nodes list
# Note the node key for the new machine
```
4. **SSH from your workstation** — once the machine is on the Tailnet:
```bash
# Get the machine's Tailscale IP
headscale nodes list | grep sovran-deploy-
# SSH in
ssh root@100.64.x.x # password: sovran-remote (live ISO default)
```
5. **Run the headless installer**:
The `--deploy-key` is your SSH public key that gets injected into `root`'s `authorized_keys` on the deployed machine. This grants full root access for initial setup. Generate it once on your workstation if you haven't already:
```bash
ssh-keygen -t ed25519 -f ~/.ssh/sovran-deploy -C "sovran-deploy"
```
After deployment is complete and you disable deploy mode, this key is removed.
```bash
sudo sovran-install-headless.sh \
--disk /dev/sda \
--role server \
--deploy-key "$(cat ~/.ssh/sovran-deploy.pub)" \
--headscale-server "https://hs.yourdomain.com" \
--headscale-key "$(headscale preauthkeys create -u $(headscale users list -o json | jq -r '.[] | select(.name=="sovran-deploy") | .id') -e 2h -o json | jq -r '.key')"
```
6. **Machine reboots into Sovran_SystemsOS** — `deploy-tailscale-connect.service` runs:
- Reads `/var/lib/secrets/headscale-authkey`
- Joins the Tailnet with a deterministic hostname (`sovran-<hostname>`)
7. **Post-install SSH and RDP**:
```bash
# SSH over Tailnet
ssh root@<tailscale-ip>
# RDP over Tailnet (desktop role) — Sovran_SystemsOS uses GNOME Remote Desktop (native Wayland RDP)
# Retrieve the auto-generated RDP password:
ssh root@<tailscale-ip> cat /var/lib/gnome-remote-desktop/rdp-password
# Then connect with any RDP client (Remmina, GNOME Connections, Microsoft Remote Desktop):
# Host: <tailscale-ip>:3389 User: sovran Password: <from above>
```
8. **Disable deploy mode** — edit `/etc/nixos/custom.nix` on the target, set `enable = false`, then:
```bash
sudo nixos-rebuild switch
```
---
## Part 5: Post-Install Access
### SSH
```bash
# Over Tailnet
ssh root@100.64.x.x
```
### RDP (desktop/server roles)
Sovran_SystemsOS uses **GNOME Remote Desktop** (native Wayland RDP — not xfreerdp). The RDP service auto-generates credentials on first boot.
**Username:** `sovran`
**Password:** auto-generated — retrieve it via SSH:
```bash
ssh root@<tailscale-ip> cat /var/lib/gnome-remote-desktop/rdp-password
```
Connect using any RDP client (Remmina, GNOME Connections, Microsoft Remote Desktop) to `<tailscale-ip>:3389`.
---
## Security Model
| Concern | Mitigation |
|---------|-----------|
| Enrollment token theft | Token only triggers key generation; it does not grant access to the machine itself |
| Rogue device joins Tailnet | Visible in `headscale nodes list`; removable instantly with `headscale nodes delete` |
| Pre-auth key reuse | Keys are ephemeral and expire in 1 hour (configurable via `keyExpiry`) |
| Rate limiting | Provisioning API limits to 10 registrations/minute by default (configurable) |
| SSH access | Requires ed25519 key injected at install time; password authentication disabled |
| Credential storage | Auth key written to `/var/lib/secrets/headscale-authkey` (mode 600) on the installed OS |
### Token Rotation
To rotate the enrollment token:
1. On the VPS:
```bash
openssl rand -hex 32 > /var/lib/sovran-provisioner/enroll-token
chmod 600 /var/lib/sovran-provisioner/enroll-token
```
2. Update `iso/secrets/enroll-token` and rebuild the ISO.
Old ISOs with the previous token will fail to register (receive 401).
---
## Monitoring
### List Active Tailnet Nodes
```bash
# On the VPS
headscale nodes list
```
### List Registered Machines (Provisioning API)
```bash
curl -s -H "Authorization: Bearer $(cat /var/lib/sovran-provisioner/enroll-token)" \
https://prov.yourdomain.com/machines | jq .
```
### Health Check
```bash
curl https://prov.yourdomain.com/health
# {"status": "ok"}
```
### Provisioner Logs
```bash
journalctl -u sovran-provisioner -f
```
### Headscale Logs
```bash
journalctl -u headscale -f
```
---
## Cleanup
### Remove a Machine from the Tailnet
```bash
headscale nodes list
headscale nodes delete --identifier <id>
```
### Disable Deploy Mode on an Installed Machine
Edit `/etc/nixos/custom.nix`:
```nix
sovran_systemsOS.deploy.enable = false;
```
Then rebuild:
```bash
nixos-rebuild switch
```
This stops the Tailscale connect service.
### Revoke All Active Pre-Auth Keys
```bash
# List pre-auth keys (Headscale 0.28.0: no --user flag on list)
headscale preauthkeys list
# Expire a specific key — use numeric user ID (-u <id>)
# First find the user ID:
headscale users list -o json
# Then expire the key:
headscale preauthkeys expire -u <user-id> --key <key>
```
---
## Troubleshooting
### VPN Conflicts (Mullvad, WireGuard, etc.)
**Symptom:** `tailscale up` hangs or fails with `connection refused` on port 443, even though `curl https://hs.yourdomain.com/health` works fine.
**Cause:** VPNs like Mullvad route all traffic — including Tailscale's control-plane connections — through the VPN tunnel. Additionally, Tailscale's DNS handler (`--accept-dns=true` by default) hijacks DNS resolution and may prevent correct resolution of your Headscale server even when logged out.
**Solution:**
1. Disconnect your VPN temporarily and retry `tailscale up`.
2. If you need the VPN active, use split tunneling to exclude `tailscaled`:
```bash
# Mullvad CLI
mullvad split-tunnel add $(pidof tailscaled)
```
Or in the Mullvad GUI: **Settings → Split tunneling → Add tailscaled**.
3. Always pass `--accept-dns=false` when enrolling to avoid DNS hijacking:
```bash
sudo tailscale up --login-server https://hs.yourdomain.com --authkey <key> --accept-dns=false
```
---
### "RATELIMIT" in tailscaled Logs
**Symptom:** `journalctl -u tailscaled` shows lines like:
```
[RATELIMIT] format("Received error: %v")
```
**Cause:** This is **NOT** a server-side rate limit from Headscale. It is tailscaled's internal log suppressor de-duplicating repeated connection-refused error messages. The real underlying error is `connection refused`.
**What to check:**
1. Is Headscale actually running? `curl https://hs.yourdomain.com/health`
2. Is your VPN blocking the connection? (see VPN Conflicts above)
3. Is there a firewall blocking port 443?
---
### "connection refused" on Port 443
If `tailscale up` fails but `curl` works, the issue is usually DNS or VPN:
```bash
# Does curl reach Headscale successfully?
curl -v https://hs.yourdomain.com/health
# Force IPv4 vs IPv6 to identify if it's an address-family issue
curl -4 https://hs.yourdomain.com/health
curl -6 https://hs.yourdomain.com/health
# Check what IP headscale resolves to
dig +short hs.yourdomain.com
# What resolver is the system using?
cat /etc/resolv.conf
```
If curl works but tailscale doesn't, tailscaled may be using a different DNS resolver (e.g. its own `100.100.100.100` stub resolver). Fix: pass `--accept-dns=false`.
---
### Headscale User ID Lookup (0.28.0)
Headscale 0.28.0 removed `--user <name>` in favour of `-u <numeric-id>`. To find the numeric ID for a user:
```bash
headscale users list -o json
# Output: [{"id": "1", "name": "sovran-deploy", ...}, ...]
# One-liner to get the ID for a specific user
headscale users list -o json | jq -r '.[] | select(.name=="sovran-deploy") | .id'
```
Then use the numeric ID in subsequent commands:
```bash
headscale preauthkeys create -u 1 -e 1h -o json
headscale nodes register -u 1 --key mkey:xxxx
```
---
## Reference
| Component | Port | Protocol | Description |
|-----------|------|----------|-------------|
| Caddy | 80 | TCP | HTTP → HTTPS redirect |
| Caddy | 443 | TCP | HTTPS (Let's Encrypt) |
| Headscale | 8080 | TCP | Coordination server (proxied by Caddy) |
| Provisioner | 9090 | TCP | Registration API (proxied by Caddy) |
| DERP/STUN | 3478 | UDP | WireGuard relay fallback |
| Tailscale | N/A | WireGuard | Mesh VPN between nodes |
docs/tech-support-security.md
-259
View File
@@ -1,259 +0,0 @@
# Tech Support: Security Design, User Flow, and Incident Response
## Overview
The Sovran Hub includes a **Tech Support** feature that lets Sovran Systems
staff remotely diagnose and fix issues on a user's machine via SSH — without
ever having access to private keys or wallet funds.
Wallet protection is the default. The user must make an active, time-limited
choice to grant support staff access to wallet files, and can revoke that
access at any time.
---
## Implementation Details
### Restricted User Instead of Root
When a user enables support access the Hub:
1. Ensures the `sovran-support` system user exists (declared declaratively in
`modules/core/tech-support.nix`; the Hub also provisions it on demand as a
fallback on non-NixOS systems).
2. Writes the Sovran Systems public SSH key **only** to
`/var/lib/sovran-support/.ssh/authorized_keys`, not to root's
`authorized_keys`.
3. Applies POSIX ACLs (`setfacl -R -m u:sovran-support:---`) to every wallet
directory that exists on disk, denying all access by the support user.
4. Records a timestamped `SUPPORT_ENABLED` event in the audit log at
`/var/log/sovran-support-audit.log`.
When the session ends (or if the Hub cannot create the restricted user), the
key is removed and all ACLs are revoked immediately.
### Protected Wallet Paths
The following directories are locked by default when a support session starts:
| Path | Contents |
|------|----------|
| `/etc/nix-bitcoin-secrets` | nix-bitcoin generated secrets |
| `/var/lib/bitcoind` | Bitcoin Core chainstate and wallet |
| `/var/lib/lnd` | LND wallet and channel database |
| `/home` | User home directories |
Paths are only locked if they exist on disk at the time the session starts.
### POSIX ACL Mechanics
POSIX ACLs on Linux handle access checks in this order:
1. If the process UID matches the file owner UID → use owner permissions
2. **If there is a matching named-user ACL entry → use that entry's
permissions** (clamped by the mask entry)
3. If any group matches → use group permissions
4. Otherwise → use "other" permissions
Setting `u:sovran-support:---` creates a named-user ACL entry with no
permissions. Because the named-user entry is checked before the group/other
entries, the support user cannot access those directories regardless of the
"other" permission bits.
`setfacl` and `getfacl` are provided by the `acl` package, which is added to
`environment.systemPackages` by `modules/core/tech-support.nix`.
### Fallback to Root (When Restricted User Cannot Be Created)
If the `sovran-support` user does not exist and cannot be created (e.g.,
`users.mutableUsers = false` and the declarative module has not been deployed
yet), the Hub falls back to adding the support key to root's
`authorized_keys`. The modal prominently warns the user when this has happened
so they can decide whether to end the session.
### Audit Log
Every session event is appended to `/var/log/sovran-support-audit.log`:
```
[2025-01-15 14:32:01 UTC] SUPPORT_ENABLED: restricted_user=True acl_applied=True protected_paths=4
[2025-01-15 14:45:00 UTC] WALLET_UNLOCKED: duration=3600s expires=2025-01-15 15:45:00 UTC
[2025-01-15 15:45:00 UTC] WALLET_RELOCKED: auto-expired
[2025-01-15 16:01:22 UTC] SUPPORT_DISABLED
```
The last 100 lines of this log are accessible from the Hub UI while a session
is active (or after it ends, until the page is refreshed).
---
## Security Tradeoffs
### What This Protects Against
- **Accidental wallet exposure** — support staff cannot read wallet files
during a normal session; they must ask the user to explicitly grant access.
- **Credential theft** — private keys in the wallet directories are not
visible to the `sovran-support` user by default.
- **Scope creep** — the restricted user account limits the blast radius of an
SSH session compared to direct root access.
### Known Limitations
| Limitation | Mitigation |
|------------|------------|
| Support user still has system-wide bash access | Restrict with `ForceCommand` or AppArmor in the NixOS config if a narrower scope is required |
| ACLs apply only to directories that exist at session start | If new wallet directories are created during a session, they are not auto-protected. Re-lock and re-enable support to pick up new paths |
| Root fallback grants full access | The Hub UI warns the user prominently; users should end the session if they are uncomfortable |
| `setfacl` / ACL filesystem support required | The `acl` package is declared in `tech-support.nix`; most Linux filesystems (ext4, btrfs, xfs) support ACLs by default |
| Wallet access grant is time-limited but lazy-expired | Expiry is checked on the next `/api/support/status` poll (every 10 seconds in the UI); there is a small window after expiry |
### Defense-in-Depth Recommendations
For environments that require stronger isolation, consider layering one or
more additional controls:
- **`ForceCommand`** in `sshd_config` (or `~/.ssh/authorized_keys` command
prefix) to restrict the support user to a specific diagnostic script.
- **`ChrootDirectory`** in the `sshd_config` `Match User sovran-support` block
to confine the session to a prepared directory tree.
- **AppArmor or SELinux** profiles that deny the support process read access
to wallet paths at the kernel level.
- **Namespace/bind-mount overlays** (e.g., via a wrapper systemd unit) to
present a sanitized filesystem view.
---
## User Flow
```
User opens Hub → Clicks "Tech Support" in sidebar
│
▼
Modal: "Need help from Sovran Systems?"
• Explains what will happen
• Shows Wallet Protection notice
• User clicks "Enable Support Access"
│
▼
Hub: 1. Creates / verifies sovran-support user
2. Writes SSH key to that user's authorized_keys
3. Applies POSIX ACL deny on all existing wallet paths
4. Saves session metadata + writes SUPPORT_ENABLED to audit log
│
▼
Modal: "Support Access is Active"
• Live session duration timer
• Wallet Files: Protected panel
– Optional: "Grant Wallet Access" (time-limited, user-chosen)
• "End Support Session" button
• "View Audit Log" button
│
(User grants wallet access)
│
▼
Hub: • Removes ACL deny entries
• Records WALLET_UNLOCKED event with expiry time
• Starts countdown timer in UI
│
(Timer expires or user clicks "Re-lock Wallet Now")
│
▼
Hub: • Re-applies ACL deny entries
• Removes WALLET_UNLOCK_FILE
• Records WALLET_RELOCKED event
│
(User clicks "End Support Session")
│
▼
Hub: 1. Removes SSH key from sovran-support authorized_keys
2. Removes SSH key from root authorized_keys (legacy cleanup)
3. Revokes any wallet unlock, re-applies ACL deny
4. Verifies key is gone
5. Records SUPPORT_DISABLED event
│
▼
Modal: "Support Session Ended — SSH key removed"
• Shows verified removal status
```
---
## Incident Response
### Scenario 1 — You accidentally granted wallet access and are unsure what was copied
**Immediate steps:**
1. Click **"Re-lock Wallet Now"** in the Hub modal, or click
**"End Support Session"** to simultaneously revoke SSH access and wallet
access.
2. Open the **Audit Log** from the Hub modal and note the timestamps of
`WALLET_UNLOCKED` and `WALLET_RELOCKED` events.
3. Check `/var/log/auth.log` (or `journalctl -u sshd`) for SSH login events
by `sovran-support` during the unlocked window.
**Assessment:**
- If no SSH login occurred during the wallet-unlocked window, your keys are
safe.
- If an SSH login did occur, treat private keys as potentially compromised.
**Recovery if keys may be compromised:**
| Wallet | Recovery action |
|--------|----------------|
| LND | Move all funds out using `lncli sendcoins` to a freshly generated on-chain address; close channels; recreate wallet |
| Sparrow | Sweep funds to a new wallet generated on an air-gapped device |
| Bisq | Withdraw all BSQ and BTC to external wallets; delete the Bisq data directory and recreate |
| nix-bitcoin secrets | Rotate all secrets with `nix-bitcoin-secrets generate` and redeploy |
**Report the incident:**
Contact Sovran Systems immediately at support@sovransystems.com with:
- The audit log output (`/var/log/sovran-support-audit.log`)
- The SSH auth log for the affected time window
- A description of what you were troubleshooting
---
### Scenario 2 — Support session cannot be ended (button fails or server is unresponsive)
**Manual key removal (run as root on the device):**
```bash
# Remove from support user's authorized_keys
rm -f /var/lib/sovran-support/.ssh/authorized_keys
# Remove from root's authorized_keys (fallback / legacy)
sed -i '/sovransystemsos-support/d' /root/.ssh/authorized_keys
# Remove wallet unlock state
rm -f /var/lib/secrets/support-wallet-unlock
# Re-apply wallet ACL protections
setfacl -R -m u:sovran-support:--- /etc/nix-bitcoin-secrets \
/var/lib/bitcoind /var/lib/lnd /home 2>/dev/null || true
# Restart sshd to drop any active connections
systemctl restart sshd
```
---
### Scenario 3 — You see an unexpected SUPPORT_ENABLED in the audit log
This should never happen without physical or remote access to the Hub web
interface. If you see an unexpected entry:
1. Immediately run the manual key removal commands above.
2. Change the Sovran Hub web interface password.
3. Check `/var/log/nginx/access.log` (or Caddy access logs) for unexpected
requests to `/api/support/enable`.
4. Consider rebooting the device to clear any in-memory state.
5. Report the incident to Sovran Systems.
---
*This document is part of the Sovran_SystemsOS repository. For the
authoritative and up-to-date version, see the repository.*