docs: void-migrate implementation plan
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
root
550
docs/superpowers/plans/2026-06-04-void-migrate.md
Normal file
550
docs/superpowers/plans/2026-06-04-void-migrate.md
Normal file
@@ -0,0 +1,550 @@
|
||||
# void-migrate Implementation Plan (Plan 8a)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** A re-runnable Node CLI that imports Void 1 (SQLite), BookStack, Karakeep, and plans `.md` into Void 2 — idempotently, one Space per source.
|
||||
|
||||
**Architecture:** A `migrate/` CLI reusing Void 2's repos. Idempotency via a `migration_map` table (refs use `upsertByExternal`). Importers are plain async functions (testable with fixtures); the CLI wires them and supports `--dry-run` + `verify`.
|
||||
|
||||
**Tech Stack:** Node 22 ESM (`node:sqlite` built-in), Postgres, vitest (serial).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-04-void-migrate-design.md`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: `migration_map` table + repo
|
||||
|
||||
**Files:** Create `lib/db/migrations/018_migration_map.sql`, `lib/db/repos/migration_map.js`; Test `tests/db/migration_map.test.js`
|
||||
|
||||
- [ ] **Step 1: Migration**
|
||||
```sql
|
||||
-- 018_migration_map.sql — idempotency ledger for void-migrate. Maps a source row
|
||||
-- to the Void 2 entity it created, so re-running an import never duplicates.
|
||||
CREATE TABLE migration_map (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
source text NOT NULL, -- 'void1' | 'bookstack' | 'karakeep' | 'plans'
|
||||
source_id text NOT NULL, -- e.g. 'wiki_pages:42'
|
||||
entity_type text NOT NULL, -- 'page' | 'project' | 'task' | 'conversation' | 'message'
|
||||
entity_id uuid NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (source, source_id, entity_type)
|
||||
);
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Failing test** `tests/db/migration_map.test.js`
|
||||
```js
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import { resetDb } from '../helpers/db.js';
|
||||
import { migrateUp } from '../../lib/db/migrate.js';
|
||||
import * as map from '../../lib/db/repos/migration_map.js';
|
||||
import { randomUUID } from 'crypto';
|
||||
|
||||
beforeAll(async () => { await resetDb(); await migrateUp(); });
|
||||
|
||||
describe('migration_map', () => {
|
||||
it('seen() is null until record(), then returns the entity id', async () => {
|
||||
const eid = randomUUID();
|
||||
expect(await map.seen('void1', 'wiki_pages:1', 'page')).toBeNull();
|
||||
await map.record('void1', 'wiki_pages:1', 'page', eid);
|
||||
expect(await map.seen('void1', 'wiki_pages:1', 'page')).toBe(eid);
|
||||
});
|
||||
it('record() is idempotent on the unique key', async () => {
|
||||
const eid = randomUUID();
|
||||
await map.record('plans', 'a.md', 'page', eid);
|
||||
await map.record('plans', 'a.md', 'page', randomUUID()); // ignored
|
||||
expect(await map.seen('plans', 'a.md', 'page')).toBe(eid);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run → FAIL.**
|
||||
- [ ] **Step 4: Implement** `lib/db/repos/migration_map.js`
|
||||
```js
|
||||
import { pool } from '../pool.js';
|
||||
|
||||
export async function seen(source, source_id, entity_type) {
|
||||
const { rows: [r] } = await pool.query(
|
||||
`SELECT entity_id FROM migration_map WHERE source=$1 AND source_id=$2 AND entity_type=$3`,
|
||||
[source, source_id, entity_type]);
|
||||
return r ? r.entity_id : null;
|
||||
}
|
||||
export async function record(source, source_id, entity_type, entity_id) {
|
||||
await pool.query(
|
||||
`INSERT INTO migration_map(source, source_id, entity_type, entity_id) VALUES($1,$2,$3,$4)
|
||||
ON CONFLICT (source, source_id, entity_type) DO NOTHING`,
|
||||
[source, source_id, entity_type, entity_id]);
|
||||
}
|
||||
```
|
||||
- [ ] **Step 5: Run → PASS. Commit** `feat(migrate): migration_map idempotency ledger`
|
||||
|
||||
---
|
||||
|
||||
### Task 2: `ensureSpace` helper
|
||||
|
||||
**Files:** Create `migrate/spaces.js`; Test `tests/migrate/spaces.test.js`
|
||||
|
||||
- [ ] **Step 1: Failing test**
|
||||
```js
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import { resetDb } from '../helpers/db.js';
|
||||
import { migrateUp } from '../../lib/db/migrate.js';
|
||||
import { ensureSpace } from '../../migrate/spaces.js';
|
||||
|
||||
beforeAll(async () => { await resetDb(); await migrateUp(); });
|
||||
|
||||
describe('ensureSpace', () => {
|
||||
it('creates a space once and reuses it by slug', async () => {
|
||||
const a = await ensureSpace('void1', 'Void 1');
|
||||
const b = await ensureSpace('void1', 'Void 1');
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** `migrate/spaces.js`
|
||||
```js
|
||||
import { pool } from '../lib/db/pool.js';
|
||||
import * as spaces from '../lib/db/repos/spaces.js';
|
||||
|
||||
const SYS = { kind: 'system', id: null };
|
||||
|
||||
// Returns the id of the space with `slug`, creating it if absent. Idempotent.
|
||||
export async function ensureSpace(slug, name) {
|
||||
const { rows: [r] } = await pool.query(`SELECT id FROM spaces WHERE slug=$1`, [slug]);
|
||||
if (r) return r.id;
|
||||
const created = await spaces.create({ slug, name }, SYS);
|
||||
return created.id;
|
||||
}
|
||||
```
|
||||
- [ ] **Step 4: Run → PASS. Commit** `feat(migrate): ensureSpace helper`
|
||||
|
||||
---
|
||||
|
||||
### Task 3: plans importer (simplest — establishes the pattern)
|
||||
|
||||
**Files:** Create `migrate/sources/plans.js`; Test `tests/migrate/plans.test.js` + `tests/fixtures/plans/*`
|
||||
|
||||
- [ ] **Step 1: Fixtures** — create `tests/fixtures/plans/alpha.md` (`# Alpha Plan\nbody one`) and `tests/fixtures/plans/beta.md` (`# Beta\nbody two`).
|
||||
|
||||
- [ ] **Step 2: Failing test** `tests/migrate/plans.test.js`
|
||||
```js
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { pool } from '../../lib/db/pool.js';
|
||||
import { resetDb } from '../helpers/db.js';
|
||||
import { migrateUp } from '../../lib/db/migrate.js';
|
||||
import { ensureSpace } from '../../migrate/spaces.js';
|
||||
import { importPlans } from '../../migrate/sources/plans.js';
|
||||
|
||||
const DIR = fileURLToPath(new URL('../fixtures/plans', import.meta.url));
|
||||
let spaceId;
|
||||
beforeAll(async () => { await resetDb(); await migrateUp(); spaceId = await ensureSpace('plans', 'Plans'); });
|
||||
|
||||
describe('plans importer', () => {
|
||||
it('imports each .md as a page, idempotently', async () => {
|
||||
const r1 = await importPlans({ dir: DIR, spaceId });
|
||||
expect(r1.created).toBe(2);
|
||||
const { rows } = await pool.query(`SELECT title FROM pages WHERE space_id=$1 ORDER BY title`, [spaceId]);
|
||||
expect(rows.map(r => r.title)).toEqual(['Alpha Plan', 'Beta']);
|
||||
const r2 = await importPlans({ dir: DIR, spaceId }); // re-run
|
||||
expect(r2.created).toBe(0);
|
||||
});
|
||||
it('dry-run writes nothing', async () => {
|
||||
await resetDb(); await migrateUp(); const s = await ensureSpace('plans', 'Plans');
|
||||
const r = await importPlans({ dir: DIR, spaceId: s, dryRun: true });
|
||||
expect(r.created).toBe(2);
|
||||
const { rows } = await pool.query(`SELECT count(*)::int n FROM pages WHERE space_id=$1`, [s]);
|
||||
expect(rows[0].n).toBe(0);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run → FAIL.**
|
||||
- [ ] **Step 4: Implement** `migrate/sources/plans.js`
|
||||
```js
|
||||
import { readdirSync, readFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import * as pages from '../../lib/db/repos/pages.js';
|
||||
import * as map from '../../lib/db/repos/migration_map.js';
|
||||
|
||||
const SYS = { kind: 'system', id: null };
|
||||
const slugify = (s) => s.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/(^-|-$)/g, '').slice(0, 60) || 'page';
|
||||
function titleOf(body, file) {
|
||||
const m = body.match(/^#\s+(.+)$/m);
|
||||
return m ? m[1].trim() : file.replace(/\.md$/, '');
|
||||
}
|
||||
|
||||
export async function importPlans({ dir, spaceId, dryRun = false }) {
|
||||
let created = 0;
|
||||
for (const file of readdirSync(dir).filter(f => f.endsWith('.md')).sort()) {
|
||||
if (await map.seen('plans', file, 'page')) continue;
|
||||
const body = readFileSync(join(dir, file), 'utf8');
|
||||
const title = titleOf(body, file);
|
||||
created++;
|
||||
if (dryRun) continue;
|
||||
const page = await pages.create({ space_id: spaceId, slug: slugify(title), title, body_md: body }, SYS);
|
||||
await map.record('plans', file, 'page', page.id);
|
||||
}
|
||||
return { created };
|
||||
}
|
||||
```
|
||||
- [ ] **Step 5: Run → PASS. Commit** `feat(migrate): plans importer`
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Void 1 SQLite importer
|
||||
|
||||
**Files:** Create `migrate/sources/void1.js`; Test `tests/migrate/void1.test.js`
|
||||
|
||||
Void 1 columns (verified): `wiki_pages(id,slug,title,content,tags,parent_id,created_at,updated_at)`, `projects(id,name,description,status,...)`, `project_tasks(id,project_id,text,done,position,...)`, `project_journal(id,project_id,content,created_at)`, `conversations(id,project_id,title,created_at)`, `messages(id,conversation_id,role,content,created_at)`.
|
||||
|
||||
- [ ] **Step 1: Failing test** `tests/migrate/void1.test.js` (builds a fixture sqlite with `node:sqlite`)
|
||||
```js
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import { DatabaseSync } from 'node:sqlite';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { pool } from '../../lib/db/pool.js';
|
||||
import { resetDb } from '../helpers/db.js';
|
||||
import { migrateUp } from '../../lib/db/migrate.js';
|
||||
import { ensureSpace } from '../../migrate/spaces.js';
|
||||
import { importVoid1 } from '../../migrate/sources/void1.js';
|
||||
|
||||
let dbPath, spaceId;
|
||||
beforeAll(async () => {
|
||||
await resetDb(); await migrateUp(); spaceId = await ensureSpace('void1', 'Void 1');
|
||||
dbPath = join(tmpdir(), `void1-${randomUUID()}.db`);
|
||||
const db = new DatabaseSync(dbPath);
|
||||
db.exec(`CREATE TABLE wiki_pages(id TEXT,slug TEXT,title TEXT,content TEXT,tags TEXT,parent_id TEXT,created_at INT,updated_at INT);
|
||||
CREATE TABLE projects(id TEXT,name TEXT,description TEXT,status TEXT);
|
||||
CREATE TABLE project_tasks(id TEXT,project_id TEXT,text TEXT,done INT,position INT);
|
||||
CREATE TABLE project_journal(id TEXT,project_id TEXT,content TEXT,created_at INT);
|
||||
CREATE TABLE conversations(id TEXT,project_id TEXT,title TEXT,created_at INT);
|
||||
CREATE TABLE messages(id TEXT,conversation_id TEXT,role TEXT,content TEXT,created_at INT);`);
|
||||
db.prepare(`INSERT INTO wiki_pages VALUES('w1','home','Home','welcome','',NULL,0,0)`).run();
|
||||
db.prepare(`INSERT INTO projects VALUES('p1','Homelab','my lab','active')`).run();
|
||||
db.prepare(`INSERT INTO project_tasks VALUES('t1','p1','do a thing',0,1)`).run();
|
||||
db.prepare(`INSERT INTO conversations VALUES('c1','p1','Chat 1',0)`).run();
|
||||
db.prepare(`INSERT INTO messages VALUES('m1','c1','user','hi',0)`).run();
|
||||
db.prepare(`INSERT INTO messages VALUES('m2','c1','assistant','hello',1)`).run();
|
||||
db.close();
|
||||
});
|
||||
|
||||
describe('void1 importer', () => {
|
||||
it('maps wiki/projects/tasks/conversations/messages, idempotently', async () => {
|
||||
const r1 = await importVoid1({ sqlitePath: dbPath, spaceId });
|
||||
expect(r1.pages).toBeGreaterThanOrEqual(1);
|
||||
expect(r1.projects).toBe(1);
|
||||
expect(r1.tasks).toBe(1);
|
||||
expect(r1.conversations).toBe(1);
|
||||
expect(r1.messages).toBe(2);
|
||||
const page = await pool.query(`SELECT title FROM pages WHERE space_id=$1 AND title='Home'`, [spaceId]);
|
||||
expect(page.rows.length).toBe(1);
|
||||
const r2 = await importVoid1({ sqlitePath: dbPath, spaceId }); // re-run
|
||||
expect(r2.pages + r2.projects + r2.tasks + r2.conversations + r2.messages).toBe(0);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** `migrate/sources/void1.js`
|
||||
```js
|
||||
import { DatabaseSync } from 'node:sqlite';
|
||||
import { pool } from '../../lib/db/pool.js';
|
||||
import * as pages from '../../lib/db/repos/pages.js';
|
||||
import * as projects from '../../lib/db/repos/projects.js';
|
||||
import * as tasks from '../../lib/db/repos/tasks.js';
|
||||
import * as messages from '../../lib/db/repos/messages.js';
|
||||
import * as map from '../../lib/db/repos/migration_map.js';
|
||||
|
||||
const SYS = { kind: 'system', id: null };
|
||||
const slugify = (s) => (s || '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/(^-|-$)/g, '').slice(0, 60) || 'item';
|
||||
|
||||
// conversations.create() has no space_id param; insert directly so migrated chats
|
||||
// live in the void1 Space.
|
||||
async function createConversation(title, spaceId) {
|
||||
const { rows: [r] } = await pool.query(
|
||||
`INSERT INTO conversations(title, space_id, metadata) VALUES($1,$2,'{}') RETURNING id`, [title || 'Conversation', spaceId]);
|
||||
return r.id;
|
||||
}
|
||||
|
||||
export async function importVoid1({ sqlitePath, spaceId, dryRun = false }) {
|
||||
const db = new DatabaseSync(sqlitePath, { readOnly: true });
|
||||
const n = { pages: 0, projects: 0, tasks: 0, conversations: 0, messages: 0 };
|
||||
const projIds = new Map(); // void1 project id -> void2 project id
|
||||
|
||||
for (const w of db.prepare(`SELECT * FROM wiki_pages`).all()) {
|
||||
if (await map.seen('void1', `wiki_pages:${w.id}`, 'page')) continue;
|
||||
n.pages++; if (dryRun) continue;
|
||||
const p = await pages.create({ space_id: spaceId, slug: slugify(w.slug || w.title), title: w.title || 'Untitled', body_md: w.content || '' }, SYS);
|
||||
await map.record('void1', `wiki_pages:${w.id}`, 'page', p.id);
|
||||
}
|
||||
for (const pr of db.prepare(`SELECT * FROM projects`).all()) {
|
||||
const existing = await map.seen('void1', `projects:${pr.id}`, 'project');
|
||||
if (existing) { projIds.set(pr.id, existing); continue; }
|
||||
n.projects++; if (dryRun) continue;
|
||||
const created = await projects.create({ space_id: spaceId, slug: slugify(pr.name), name: pr.name || 'Project', description: pr.description || null, status: pr.status === 'archived' ? 'archived' : 'active' }, SYS);
|
||||
projIds.set(pr.id, created.id);
|
||||
await map.record('void1', `projects:${pr.id}`, 'project', created.id);
|
||||
}
|
||||
for (const t of db.prepare(`SELECT * FROM project_tasks`).all()) {
|
||||
if (await map.seen('void1', `project_tasks:${t.id}`, 'task')) continue;
|
||||
n.tasks++; if (dryRun) continue;
|
||||
const created = await tasks.create({ space_id: spaceId, project_id: projIds.get(t.project_id) || null, title: t.text || 'Task', position: t.position ?? 0 }, SYS);
|
||||
await map.record('void1', `project_tasks:${t.id}`, 'task', created.id);
|
||||
}
|
||||
for (const j of db.prepare(`SELECT * FROM project_journal`).all()) {
|
||||
if (await map.seen('void1', `project_journal:${j.id}`, 'page')) continue;
|
||||
n.pages++; if (dryRun) continue;
|
||||
const p = await pages.create({ space_id: spaceId, slug: slugify(`journal-${j.id}`), title: `Journal ${j.id}`, body_md: j.content || '' }, SYS);
|
||||
await map.record('void1', `project_journal:${j.id}`, 'page', p.id);
|
||||
}
|
||||
for (const c of db.prepare(`SELECT * FROM conversations`).all()) {
|
||||
let convId = await map.seen('void1', `conversations:${c.id}`, 'conversation');
|
||||
if (!convId) {
|
||||
n.conversations++; if (dryRun) continue;
|
||||
convId = await createConversation(c.title, spaceId);
|
||||
await map.record('void1', `conversations:${c.id}`, 'conversation', convId);
|
||||
}
|
||||
for (const m of db.prepare(`SELECT * FROM messages WHERE conversation_id=? ORDER BY created_at`).all(c.id)) {
|
||||
if (await map.seen('void1', `messages:${m.id}`, 'message')) continue;
|
||||
n.messages++; if (dryRun) continue;
|
||||
const appended = await messages.append(convId, { role: m.role === 'assistant' ? 'assistant' : 'user', body: m.content || '' });
|
||||
await map.record('void1', `messages:${m.id}`, 'message', appended.id);
|
||||
}
|
||||
}
|
||||
db.close();
|
||||
return n;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run → PASS. Commit** `feat(migrate): Void 1 SQLite importer`
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Karakeep importer (refs via upsertByExternal)
|
||||
|
||||
**Files:** Create `migrate/sources/karakeep.js`; Test `tests/migrate/karakeep.test.js`
|
||||
|
||||
- [ ] **Step 1: Failing test**
|
||||
```js
|
||||
import { describe, it, expect, beforeAll, vi } from 'vitest';
|
||||
import { pool } from '../../lib/db/pool.js';
|
||||
import { resetDb } from '../helpers/db.js';
|
||||
import { migrateUp } from '../../lib/db/migrate.js';
|
||||
import { ensureSpace } from '../../migrate/spaces.js';
|
||||
import { importKarakeep } from '../../migrate/sources/karakeep.js';
|
||||
|
||||
let spaceId;
|
||||
beforeAll(async () => { await resetDb(); await migrateUp(); spaceId = await ensureSpace('bookmarks', 'Bookmarks'); });
|
||||
|
||||
describe('karakeep importer', () => {
|
||||
it('imports bookmarks as url refs, idempotently', async () => {
|
||||
const page = { bookmarks: [
|
||||
{ id: 'b1', content: { type: 'link', url: 'https://a.test', title: 'A' } },
|
||||
{ id: 'b2', content: { type: 'link', url: 'https://b.test', title: 'B' } }
|
||||
], nextCursor: null };
|
||||
const fetchImpl = vi.fn(async () => ({ ok: true, json: async () => page }));
|
||||
const r1 = await importKarakeep({ apiUrl: 'http://k', token: 't', spaceId, fetchImpl });
|
||||
expect(r1.created).toBe(2);
|
||||
const { rows } = await pool.query(`SELECT url FROM refs WHERE space_id=$1 ORDER BY url`, [spaceId]);
|
||||
expect(rows.map(r => r.url)).toEqual(['https://a.test', 'https://b.test']);
|
||||
const r2 = await importKarakeep({ apiUrl: 'http://k', token: 't', spaceId, fetchImpl }); // upsert, no dupes
|
||||
const { rows: again } = await pool.query(`SELECT count(*)::int n FROM refs WHERE space_id=$1`, [spaceId]);
|
||||
expect(again[0].n).toBe(2);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** `migrate/sources/karakeep.js`
|
||||
```js
|
||||
import * as refs from '../../lib/db/repos/refs.js';
|
||||
|
||||
const SYS = { kind: 'system', id: null };
|
||||
|
||||
// Karakeep REST: GET /api/v1/bookmarks?cursor=... → { bookmarks:[{id,content:{url,title}}], nextCursor }.
|
||||
export async function importKarakeep({ apiUrl = process.env.KARAKEEP_URL, token = process.env.KARAKEEP_TOKEN, spaceId, dryRun = false, fetchImpl = fetch }) {
|
||||
let cursor = null, created = 0;
|
||||
do {
|
||||
const u = `${apiUrl}/api/v1/bookmarks?limit=100${cursor ? `&cursor=${encodeURIComponent(cursor)}` : ''}`;
|
||||
const res = await fetchImpl(u, { headers: { Authorization: `Bearer ${token}` } });
|
||||
if (!res.ok) throw new Error(`karakeep ${res.status}`);
|
||||
const body = await res.json();
|
||||
for (const b of (body.bookmarks || [])) {
|
||||
const url = b.content?.url; if (!url) continue;
|
||||
created++;
|
||||
if (dryRun) continue;
|
||||
await refs.upsertByExternal({ space_id: spaceId, kind: 'url', url, title: b.content?.title || url, source_kind: 'karakeep', external_id: String(b.id) }, SYS);
|
||||
}
|
||||
cursor = body.nextCursor || null;
|
||||
} while (cursor);
|
||||
return { created };
|
||||
}
|
||||
```
|
||||
- [ ] **Step 4: Run → PASS. Commit** `feat(migrate): Karakeep bookmarks importer`
|
||||
|
||||
---
|
||||
|
||||
### Task 6: BookStack importer
|
||||
|
||||
**Files:** Create `migrate/sources/bookstack.js`; Test `tests/migrate/bookstack.test.js`
|
||||
|
||||
- [ ] **Step 1: Failing test**
|
||||
```js
|
||||
import { describe, it, expect, beforeAll, vi } from 'vitest';
|
||||
import { pool } from '../../lib/db/pool.js';
|
||||
import { resetDb } from '../helpers/db.js';
|
||||
import { migrateUp } from '../../lib/db/migrate.js';
|
||||
import { ensureSpace } from '../../migrate/spaces.js';
|
||||
import { importBookstack } from '../../migrate/sources/bookstack.js';
|
||||
|
||||
let spaceId;
|
||||
beforeAll(async () => { await resetDb(); await migrateUp(); spaceId = await ensureSpace('wiki', 'Wiki'); });
|
||||
|
||||
describe('bookstack importer', () => {
|
||||
it('imports pages with markdown body, idempotently', async () => {
|
||||
const list = { data: [{ id: 1, name: 'Page One' }, { id: 2, name: 'Page Two' }] };
|
||||
const detail = (id) => ({ id, name: id === 1 ? 'Page One' : 'Page Two', markdown: `# P${id}\nbody`, html: `<p>body</p>` });
|
||||
const fetchImpl = vi.fn(async (u) => {
|
||||
if (u.endsWith('/api/pages')) return { ok: true, json: async () => list };
|
||||
const id = Number(u.split('/').pop());
|
||||
return { ok: true, json: async () => detail(id) };
|
||||
});
|
||||
const r1 = await importBookstack({ apiUrl: 'http://bs', tokenId: 'i', tokenSecret: 's', spaceId, fetchImpl });
|
||||
expect(r1.created).toBe(2);
|
||||
const { rows } = await pool.query(`SELECT title FROM pages WHERE space_id=$1 ORDER BY title`, [spaceId]);
|
||||
expect(rows.map(r => r.title)).toEqual(['Page One', 'Page Two']);
|
||||
const r2 = await importBookstack({ apiUrl: 'http://bs', tokenId: 'i', tokenSecret: 's', spaceId, fetchImpl });
|
||||
expect(r2.created).toBe(0);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** `migrate/sources/bookstack.js`
|
||||
```js
|
||||
import * as pages from '../../lib/db/repos/pages.js';
|
||||
import * as map from '../../lib/db/repos/migration_map.js';
|
||||
|
||||
const SYS = { kind: 'system', id: null };
|
||||
const slugify = (s) => (s || '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/(^-|-$)/g, '').slice(0, 60) || 'page';
|
||||
|
||||
// BookStack REST: list GET /api/pages → {data:[{id,name}]}; detail GET /api/pages/:id
|
||||
// → {id,name,markdown,html}. Auth header: "Token <id>:<secret>".
|
||||
export async function importBookstack({ apiUrl = process.env.BOOKSTACK_URL, tokenId = process.env.BOOKSTACK_TOKEN_ID, tokenSecret = process.env.BOOKSTACK_TOKEN_SECRET, spaceId, dryRun = false, fetchImpl = fetch }) {
|
||||
const headers = { Authorization: `Token ${tokenId}:${tokenSecret}` };
|
||||
const listRes = await fetchImpl(`${apiUrl}/api/pages`, { headers });
|
||||
if (!listRes.ok) throw new Error(`bookstack list ${listRes.status}`);
|
||||
const { data = [] } = await listRes.json();
|
||||
let created = 0;
|
||||
for (const item of data) {
|
||||
if (await map.seen('bookstack', `page:${item.id}`, 'page')) continue;
|
||||
created++; if (dryRun) continue;
|
||||
const dRes = await fetchImpl(`${apiUrl}/api/pages/${item.id}`, { headers });
|
||||
if (!dRes.ok) throw new Error(`bookstack page ${item.id} ${dRes.status}`);
|
||||
const d = await dRes.json();
|
||||
const body = d.markdown && d.markdown.trim() ? d.markdown : (d.html || '');
|
||||
const p = await pages.create({ space_id: spaceId, slug: slugify(d.name), title: d.name || 'Untitled', body_md: body }, SYS);
|
||||
await map.record('bookstack', `page:${item.id}`, 'page', p.id);
|
||||
}
|
||||
return { created };
|
||||
}
|
||||
```
|
||||
- [ ] **Step 4: Run → PASS. Commit** `feat(migrate): BookStack importer`
|
||||
|
||||
---
|
||||
|
||||
### Task 7: CLI + verify
|
||||
|
||||
**Files:** Create `migrate/cli.js`, `migrate/verify.js`; Test `tests/migrate/verify.test.js`
|
||||
|
||||
- [ ] **Step 1: `migrate/verify.js`** — counts migrated rows per source/type from `migration_map` + refs:
|
||||
```js
|
||||
import { pool } from '../lib/db/pool.js';
|
||||
|
||||
export async function verify() {
|
||||
const { rows: map } = await pool.query(
|
||||
`SELECT source, entity_type, count(*)::int n FROM migration_map GROUP BY source, entity_type ORDER BY source, entity_type`);
|
||||
const { rows: [{ n: bookmarks }] } = await pool.query(`SELECT count(*)::int n FROM refs WHERE source_kind='karakeep'`);
|
||||
return { map, bookmarks };
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify test** `tests/migrate/verify.test.js`
|
||||
```js
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import { resetDb } from '../helpers/db.js';
|
||||
import { migrateUp } from '../../lib/db/migrate.js';
|
||||
import * as map from '../../lib/db/repos/migration_map.js';
|
||||
import { verify } from '../../migrate/verify.js';
|
||||
import { randomUUID } from 'crypto';
|
||||
|
||||
beforeAll(async () => { await resetDb(); await migrateUp(); await map.record('plans', 'a.md', 'page', randomUUID()); });
|
||||
|
||||
describe('verify', () => {
|
||||
it('reports per-source/type counts', async () => {
|
||||
const out = await verify();
|
||||
expect(out.map.find(r => r.source === 'plans' && r.entity_type === 'page').n).toBe(1);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run verify test → FAIL then PASS** after creating `verify.js`.
|
||||
|
||||
- [ ] **Step 4: `migrate/cli.js`** (dispatch; manual-run entry point):
|
||||
```js
|
||||
import 'dotenv/config';
|
||||
import { ensureSpace } from './spaces.js';
|
||||
import { importPlans } from './sources/plans.js';
|
||||
import { importVoid1 } from './sources/void1.js';
|
||||
import { importKarakeep } from './sources/karakeep.js';
|
||||
import { importBookstack } from './sources/bookstack.js';
|
||||
import { verify } from './verify.js';
|
||||
|
||||
const [, , cmd, ...rest] = process.argv;
|
||||
const dryRun = rest.includes('--dry-run');
|
||||
|
||||
const run = {
|
||||
async plans() { return importPlans({ dir: process.env.PLANS_DIR || '/root/.claude/plans', spaceId: await ensureSpace('plans', 'Plans'), dryRun }); },
|
||||
async void1() { return importVoid1({ sqlitePath: process.env.VOID1_DB || '/tmp/void1.db', spaceId: await ensureSpace('void1', 'Void 1'), dryRun }); },
|
||||
async karakeep() { return importKarakeep({ spaceId: await ensureSpace('bookmarks', 'Bookmarks'), dryRun }); },
|
||||
async bookstack() { return importBookstack({ spaceId: await ensureSpace('wiki', 'Wiki'), dryRun }); },
|
||||
async verify() { return verify(); }
|
||||
};
|
||||
|
||||
if (!run[cmd]) { console.error(`usage: node migrate/cli.js <plans|void1|karakeep|bookstack|verify> [--dry-run]`); process.exit(1); }
|
||||
run[cmd]().then(r => { console.log(JSON.stringify(r, null, 2)); process.exit(0); })
|
||||
.catch(e => { console.error('migrate failed:', e.message); process.exit(1); });
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run → PASS. Commit** `feat(migrate): CLI dispatch + verify`
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Run the migration (gated execution — owner-authorized)
|
||||
|
||||
**Files:** none. Requires: migration 018 applied to the void2 DB + source access.
|
||||
|
||||
- [ ] **Step 1: Apply 018 to prod** — deploy (`bash deploy/push.sh`; runs `018_migration_map`) OR `npm run migrate` against the void2 DB.
|
||||
- [ ] **Step 2: Stage sources on the run host** (CT 311 or dev box with the void2 `DATABASE_URL`):
|
||||
- `scp` Void 1's `void.db` from CT 301 → set `VOID1_DB`.
|
||||
- Set `PLANS_DIR` to the plans dir.
|
||||
- **BookStack:** create an API token (Settings → API Tokens) → `BOOKSTACK_URL`, `BOOKSTACK_TOKEN_ID`, `BOOKSTACK_TOKEN_SECRET`.
|
||||
- **Karakeep:** create an API key → `KARAKEEP_URL`, `KARAKEEP_TOKEN`.
|
||||
- [ ] **Step 3: Dry-run each** — `node migrate/cli.js <src> --dry-run` → review would-create counts.
|
||||
- [ ] **Step 4: Import** — `node migrate/cli.js plans|void1|karakeep|bookstack`.
|
||||
- [ ] **Step 5: `node migrate/cli.js verify`** — confirm counts vs sources; spot-check a few entities in the Void 2 UI (the 4 new Spaces).
|
||||
- [ ] **Step 6: Memory** — record migrated counts + that Plan 8a is done; note 8b (cutover) is the remaining gate to 2.0.0.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage:** migration_map→T1; ensureSpace/per-source Spaces→T2; plans→T3; void1 (wiki/projects/tasks/journal/conversations/messages)→T4; karakeep (refs upsertByExternal)→T5; bookstack→T6; CLI+dry-run+verify→T7; run/verify→T8. Idempotency tested in T3–T6 (re-run → 0). Out-of-scope (8b cutover) deferred. Covered.
|
||||
|
||||
**Placeholder scan:** T8 env values (tokens, paths) are runtime provisioning inputs, not code gaps. No TBDs in code.
|
||||
|
||||
**Type consistency:** importers return `{created}` (plans/karakeep/bookstack) and `{pages,projects,tasks,conversations,messages}` (void1) — tests match. `map.seen(source,source_id,type)`/`map.record(...)` consistent T1 def + T3–T6 use. `ensureSpace(slug,name)→id` consistent. Repo calls match verified signatures (`pages.create({space_id,slug,title,body_md})`, `tasks.create({space_id,project_id,title,position})`, `refs.upsertByExternal({space_id,kind,url,title,source_kind,external_id})`, `messages.append(convId,{role,body})`).
|
||||
Reference in New Issue
Block a user