Void-Homelab/docs/superpowers/specs/2026-06-08-lan-device-discovery-design.md

# Design: LAN Device Discovery (MAC inventory + review/name)

**Date:** 2026-06-08
**Status:** Approved (brainstorm), pending implementation plan
**Repo:** void-v2 (CT 311 / `void-app`)

## Summary

Replace the static, hand-maintained `public/devices.json` with a **persistent,
MAC-keyed device store** fed by a recurring ARP scan. Each scan **logs MACs to
the DB and diffs against what's known** — new devices land in a review queue;
known devices just get their IP / `last_seen` / presence updated. The owner can
**add a discovered device, edit it, and give it a name** for reference (mirrors
the Void's existing services "discovered → promote" pattern).

## Background

- **Static today:** `public/views/devices_band.js` does `fetch('/devices.json')`
  — a curated, manually-edited list (IP/MAC/vendor/group/flag). It re-reads a
  static file; nothing is persisted or diffed.
- **Existing precedent to mirror:** `monitored_services` uses
  `source='discovered' AND NOT enabled` as a review queue; `PATCH /services/:id`
  promotes + edits. We reproduce this shape for devices.
- **Separate from `network_hosts`:** that table is the homelab-guest inventory
  (Proxmox `BC:24:11:*` LXCs, infra_audit). The devices band is IoT / personal /
  unknown LAN gear — kept separate.
- **Scan engine:** the Void host (CT 311) has `ip`/`arp` but **not**
  `nmap`/`arp-scan`. We add `arp-scan` (chosen for reliable L2 ARP sweeps that
  ICMP-blocking devices can't dodge, plus a built-in OUI vendor DB).

### Lessons borrowed from Scanopy (self-hosted discovery tool)

- **Decouple scanner from storage/UI** — the scanner just scans and reports; the
  server owns dedup + persistence. → isolated `lib/infra/scan.js`.
- **MAC is the identity, IP is a mutable attribute** — key on MAC, update IP each
  scan (handles DHCP churn). → `mac` primary key.
- **Scheduled rescans + timestamp inventory** — periodic batch with
  `first_seen`/`last_seen`/`present`, diff by "MAC seen before?". → hourly cron.
- **Vendor via OUI** — `arp-scan` ships an OUI database; vendor is free.
- **Randomized MACs are an open problem** even for Scanopy — so we at least
  **flag** locally-administered MACs so the user knows OUI can't ID them.

## Decisions

| Decision | Choice | Rationale |
|---|---|---|
| Scan engine | **`arp-scan --localnet` on CT 311**, hourly cron | Reliable L2 sweep + built-in OUI; self-contained (no external scanner dep). |
| Cadence | **Hourly** (staggered, e.g. `7 * * * *`) | "No rush"; device drift is slow. |
| DB growth | **Upsert by MAC — one row per device, no per-scan history** | Table is bounded by distinct devices ever seen (dozens–hundreds), not scan count → no bloat. |
| Identity | **MAC primary key**; IP a mutable column | Survives DHCP IP changes. |
| Review flow | Mirror services `discovered → promote` | New MAC → `status='new'`; owner names/edits → `status='known'`. |
| Source of truth | **DB** (`lan_devices`); `devices.json` becomes the one-time migration seed, then removed | Single source of truth. |
| Randomized-MAC bloat | **Auto-prune unreviewed + absent rows** (randomized >24h, others >14d); keep `known`/`ignored` forever | Rotated randomized MACs never accumulate; the table stays bounded. |

## Architecture

scan (arp-scan) → `parseArpScan` (+randomized flag) → `upsertScan` by MAC →
`markAbsent` for unseen → review queue (`status='new'`) → owner names/groups/promotes
→ known devices render in the band.

## Components

### Migration `024_lan_devices.sql`
Table `lan_devices`:
- `mac text PRIMARY KEY`
- `ip text`, `vendor text`
- `name text` (owner-given reference name, null until named)
- `grp text` (Smart Home | Entertainment | Personal | Network | Flagged)
- `note text`
- `status text NOT NULL DEFAULT 'new'` (`new` | `known` | `ignored`)
- `randomized boolean NOT NULL DEFAULT false` (locally-administered MAC)
- `flagged boolean NOT NULL DEFAULT false`
- `first_seen timestamptz NOT NULL DEFAULT now()`
- `last_seen timestamptz NOT NULL DEFAULT now()`
- `present boolean NOT NULL DEFAULT true`

**Seed (embedded SQL, from the current curated `devices.json`):**
- Devices **with a MAC**: non-flagged → `status='known'` with their name/group;
  flagged (e.g. `.15` ASUS) → `status='new'`, `flagged=true`.
- The `.13` Orbi satellite and `.171` Galaxy Tab S4 fixes carry over as `known`.
- MAC-less curated entries (`.21/.22/.34/.35/.51`, currently offline) are **not
  seeded** — they reappear as `new` (with a real MAC) the first time they're seen
  online. (Documented so it's expected, not a gap.)

### `lib/infra/scan.js` (decoupled scanner)
- `parseArpScan(text) -> [{ ip, mac, vendor, randomized }]` — **pure** parser of
  `arp-scan` tab-separated output (skips banner/footer); `randomized` = first
  octet has the locally-administered bit (`& 0x02`).
- `isRandomizedMac(mac) -> boolean` — pure helper.
- `runScan({ exec }) -> rows` — shells `arp-scan --localnet -x` (interface
  auto/`-I eth0`), returns `parseArpScan(stdout)`. `exec` injected for tests.

### `lib/db/repos/lan_devices.js`
- `upsertScan(rows)` — insert unseen MACs as `status='new'`; for existing, update
  `ip`, `vendor`, `last_seen=now()`, `present=true` (never overwrite owner
  `name`/`grp`/`status`).
- `markAbsent(seenMacs)` — `present=false` for MACs not in the latest scan.
- `listKnown()` (`status='known'`, grouped by `grp`), `listDiscovered()`
  (`status='new'`), `get(mac)`, `update(mac, {name, grp, status, note, flagged})`,
  `remove(mac)`. (`ignored` devices show in neither.)
- `prune()` — delete unreviewed + absent rows past their TTL: `status='new' AND
  present=false AND ((randomized AND last_seen < now()-'24h') OR (NOT randomized
  AND last_seen < now()-'14d'))`. Never touches `known`/`ignored`.

### Cron (`lib/cron/index.js`)
Add hourly (`7 * * * *`): `runScan()` → `upsertScan` → `markAbsent` → `prune()`.
Wrapped in try/catch — a scan failure logs and never crashes the cron, and
`prune()` only runs after a *successful* scan (so a failed scan can't reap rows).

### API `lib/api/routes/devices.js` (mount `/api/devices`, owner-gated)
- `GET /` — known devices grouped for the band.
- `GET /discovered` — `status='new'` review queue.
- `PATCH /:mac` — set `name`/`grp`/`status`/`note`/`flagged` (this is "add from
  discovered" + "edit" + "name"); promoting = `status:'known'`.
- `DELETE /:mac` — remove.
- `POST /scan` — run a scan immediately (owner).
- `:mac` param validated against a MAC regex.

### Frontend
- `public/views/devices_band.js` — fetch `/api/devices` (grouped) instead of the
  static file; render the MAC (existing `.dv-mac` style from today's change).
- **Discovered review** — a section/panel listing `/api/devices/discovered`, each
  with an **Add / Edit** form (name + group select + notes) that `PATCH`es to
  promote; plus inline edit for known devices and an Ignore/Delete action.
- **Randomized devices** get a small "randomized MAC" badge (with a tooltip:
  naming pins it only until the MAC rotates; disable SSID randomization for
  stable tracking). A `known` device that's been `present=false` for ≥30d shows
  an "absent Nd" marker for easy manual cleanup (never auto-deleted).
- Remove `public/devices.json` (superseded by the DB).

## Infra setup (one-time, on CT 311)
`apt install arp-scan` + grant the binary raw-socket capability so the non-root
`void` service user can run it:
`setcap cap_net_raw,cap_net_admin+eip /usr/sbin/arp-scan`. Captured in
`deploy/README.md`. If the capability/tool is missing, the scan logs a clear
error and the feature degrades to "no new discoveries" (existing data still shows).

## Error handling
- `arp-scan` missing / unprivileged / non-zero exit → `runScan` throws; cron
  catches, logs, leaves the DB untouched (known devices still render).
- Empty/garbled scan output → `parseArpScan` returns `[]`; `markAbsent([])` is a
  no-op guard (never blanket-marks everything absent on a failed scan).
- Bad MAC in PATCH → 400 via zod.

## Testing
- **`parseArpScan` / `isRandomizedMac`** — pure unit tests (sample arp-scan
  output incl. a randomized MAC, banner/footer lines, a malformed line).
- **`lan_devices` repo** (vitest + test DB) — `upsertScan` inserts new vs updates
  existing without clobbering owner fields; `markAbsent` flips presence; promote
  via `update`.
- **API** (supertest) — `/discovered` lists only `new`; `PATCH` promotes/edits;
  owner-gated.
- **Frontend** (jsdom) — band renders groups + MAC from `/api/devices`;
  discovered panel renders the add/edit form.
- **Manual** — `POST /api/devices/scan`, confirm new devices appear, name one,
  see it move to the band.

## Out of scope (YAGNI)
- Service/port fingerprinting, SNMP/LLDP topology (that's Scanopy's job).
- Multi-subnet/VLAN scanning (single `/24`).
- Push notifications on new-device discovery.
- Stable identity for randomized-MAC devices across rotations (not solvable from
  L2 alone; the user-side fix is disabling MAC randomization for the SSID).

## References
- Scanopy — github.com/scanopy/scanopy ; scanopy.net (self-hosted discovery/topology, AGPL-3.0).