Void-Homelab/docs/superpowers/specs/2026-06-04-void-migrate-design.md

# void-migrate — Design (Plan 8a)

**Date:** 2026-06-04 · **Component:** Void 2.0 · **Phase:** Plan 8 (Migration & Cutover), sub-project **8a** of 2 · **Status:** Approved (design)

## Goal

A re-runnable Node CLI that imports the data-bearing Void 1.x sources into Void 2 — preserving everything that would be lost when CT 301 / CT 104 / Karakeep retire — so the **8b cutover** (separate, gated) can swap `void.hynesy.com` to void2 and drop `-alpha`.

## Scope (locked)

- **In:** `void1-sqlite` (Void 1 core), `bookstack`, `karakeep`, `plans`.
- **Out:** schemas (DB-backed service registry already replaced it), personas (now a code module), memory-mirror (separate ongoing-sync subsystem; the memory files stay canonical on disk regardless).
- **8b cutover is out of this spec** — verify → tunnel swap → grace → retire CT 301 → rename → drop `-alpha` → tag `2.0.0`.

## Decisions (locked)

1. **Node CLI in the void-2 repo**, reusing the app's repos + validation (not Python-via-API).
2. **Idempotency via a `migration_map` table** — re-running never duplicates. (Refs already have `upsertByExternal` — bookmarks use it.)
3. **One Space per source:** `void1`, `wiki` (BookStack), `bookmarks` (Karakeep), `plans`.
4. **Read paths:** Void 1 via `node:sqlite` (Node 22 built-in, on a copied `void.db`); BookStack + Karakeep via their REST APIs (tokens); plans via the filesystem.

## Sources (verified)

- **Void 1 (CT 301, `/opt/void/data/void.db`, SQLite):** tables incl. `wiki_pages` (+`wiki_revisions`/`wiki_backlinks`), `projects` (+`project_tasks`, `project_journal`, `project_pages`), `conversations` + `messages`. (`mastra.db` = old agent layer, not migrated.)
- **BookStack (CT 104, MariaDB):** ~17 pages under books/chapters — pulled via the BookStack REST API.
- **Karakeep (CT 100, containerized + REST API + Meilisearch):** bookmarks — pulled via the Karakeep REST API.
- **Plans:** `/root/.claude/plans/*.md` on the dev host.

## Architecture

`migrate/` CLI: `node migrate <source> [--dry-run]` and `node migrate verify`. Each importer reads its origin, maps to Void 2 entities, writes through repos. Before creating a non-ref entity it checks `migration_map(source, source_id, entity_type)`; after creating, it records the new id. `--dry-run` logs intended writes without performing them.

```
node migrate void1   --dry-run     # preview
node migrate void1                 # import → Space 'void1'
node migrate bookstack | karakeep | plans
node migrate verify                # source vs migrated counts per type
```

## Components

| File | Responsibility |
|---|---|
| `lib/db/migrations/018_migration_map.sql` | `migration_map(id, source, source_id, entity_type, entity_id, created_at)`, `UNIQUE(source, source_id, entity_type)`. |
| `lib/db/repos/migration_map.js` | `seen(source, srcId, type)` → existing entity id or null; `record(source, srcId, type, entityId)`. |
| `migrate/cli.js` | Arg parse + dispatch (`void1`/`bookstack`/`karakeep`/`plans`/`verify`), `--dry-run`. |
| `migrate/spaces.js` | `ensureSpace(slug, name)` — idempotent by slug; returns space id. |
| `migrate/sources/void1.js` | Open copied `void.db` (`node:sqlite`); map `wiki_pages`→pages, `projects`→projects (+`project_tasks`→tasks, `project_journal`→pages), `conversations`+`messages`→conversations+messages. → `void1`. |
| `migrate/sources/bookstack.js` | BookStack API (`BOOKSTACK_URL`, `BOOKSTACK_TOKEN_ID`, `BOOKSTACK_TOKEN_SECRET`) → pages; book/chapter become parent pages. → `wiki`. |
| `migrate/sources/karakeep.js` | Karakeep API (`KARAKEEP_URL`, `KARAKEEP_TOKEN`) → refs `kind=url` via `refs.upsertByExternal` (`source_kind='karakeep'`, `external_id`=bookmark id). → `bookmarks`. |
| `migrate/sources/plans.js` | Read `PLANS_DIR/*.md` → pages (title = first `# heading` or filename; body = file). → `plans`. |

## Field mappings

- **void1 `wiki_pages`** → page: `title`, `body_md` (from content/markdown column), `slug` (slugified title), space `void1`. map key `(void1, wiki_pages:<id>, page)`.
- **void1 `projects`** → project (`name`, `description`, `status`); `project_tasks`→tasks (`title`, `status`, `project_id`); `project_journal`→pages (journal entries as pages, linked by title). 
- **void1 `conversations`/`messages`** → conversations (`title`, space `void1`) + messages (`role`, `body`, ordered).
- **bookstack page** → page (`title`, `body_md` from HTML→markdown or the raw markdown field, `parent_id` = the migrated book/chapter page). map key `(bookstack, page:<id>, page)`.
- **karakeep bookmark** → ref (`kind='url'`, `url`, `title`, `source_kind='karakeep'`, `external_id`=bookmark id) via `upsertByExternal`.
- **plan file** → page (`title`, `body_md`=file contents). map key `(plans, <filename>, page)`.

## Idempotency / dry-run / verify

Non-ref entities: `migration_map.seen` short-circuits a re-run. Refs: `upsertByExternal`. `--dry-run` performs reads + mapping but no writes (logs a per-type would-create count). `verify` queries each source for its counts and `migration_map`/refs for migrated counts, printing a per-source table so completeness is auditable before cutover.

## Error handling

A bad row is logged and skipped (migration continues); the summary reports skipped counts. Source connection failures (API/DB) abort that source with a clear message, leaving prior sources' results intact. Nothing partial is left half-mapped (record the map row only after a successful entity create, in the same path).

## Testing (vitest, serial — `fileParallelism:false`)

- **`migration_map`**: `seen` null→id after `record`; unique prevents dupes.
- **`spaces.ensureSpace`**: creates once, reuses.
- **void1**: a fixture `void.db` (built in-test via `node:sqlite`) with a couple of wiki_pages/projects/tasks/conversations → assert mapped entities + **idempotency** (run importer twice → counts unchanged).
- **bookstack/karakeep**: injected fetch returning fixture API payloads → assert pages/refs created + idempotency.
- **plans**: a fixture `.md` dir → assert pages.
- **verify**: returns the expected per-source counts from seeded data.

## Out of scope (YAGNI)

Live two-way sync; migrating FTS/revisions/usage_log/sessions/api_tokens; HTML fidelity beyond a basic BookStack→markdown; the memory mirror; the 8b cutover ops.