The url passed to yt-dlp is user-controllable (via /api/capture). Any
string starting with '-' would be parsed as a flag (e.g.
--config-location=/etc/passwd). Mitigations:
1. Validate scheme is http(s) and hostname is present before subprocess.
2. Pass `--` to yt-dlp so it stops flag parsing before the positional
URL.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
POST /api/capture with a youtube.com / youtu.be / vimeo.com URL
enqueues ingest.video (Python worker) instead of ingest.url
(Node worker). Detection by URL hostname; idempotency_key + response
shape unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
yt-dlp pulls metadata (title, description, uploader, thumbnail) and
bestaudio (opus). faster-whisper transcribes; audio file removed after.
Creates a refs row with kind='video' and source_kind='youtube' for
YouTube URLs, generic 'video' otherwise. Idempotent on
sha256(space_id + url) via refs.external_id.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rsync ran as root over SSH so files landed root-owned, but workers run
as voidworkers — the service couldn't even reach the venv binary.
Now: chown -R voidworkers after rsync, run venv create + pip install
under `su voidworkers -c`. Also excludes .env, .gitignore, .pytest_cache
so they survive across deploys.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After creating a ref, the Node-side ingest.blob worker enqueues a
follow-up job for the Python void-workers (Plan 4) to OCR / extract
text. Other kinds (file) get no follow-up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
pdftotext first; falls back to per-page pdftoppm rasterization +
Tesseract OCR when the extracted text is < 200 chars. Updates
refs.body_text + metadata.extract.{method,chars} via the repo shim;
audit entry emitted with actor_kind='worker'.
born_digital.pdf fixture padded so pdftotext yields > 200 chars and
the test exercises the pdftotext path, not the OCR fallback.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Deploy README extended with workers bootstrap + note on the void2-db
SQL_ASCII cluster requiring client_encoding=UTF8 on Python clients.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the Boss class — SELECT … FOR UPDATE SKIP LOCKED to atomically
claim, UPDATE state on completion. Retry semantics match pg-boss:
exponential backoff via retry_count / retry_delay / retry_backoff.
Forces client_encoding=UTF8 on every connection. The void2-db cluster
was initialized as SQL_ASCII so psycopg refuses to decode text by
default; UTF8 client_encoding works because the data is already UTF-8.
Node's pg lib is more forgiving and didn't surface this.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Plan 4 Phase A scaffolding. void-workers package at /workers/, sibling
of /lib/. pyproject.toml pins Python 3.12 with separate extras for
pdf / image / video / test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
All Void 2.0 superpowers specs and implementation plans now live at
docs/superpowers/{specs,plans}/ inside the repo. Previously they were
at /project/docs/superpowers/ which was not under git.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cb(null, address, family) was returning Invalid IP address: undefined
under undici v6. Returning the full records array (each {address, family})
gives undici what it expects and lets it pick the best family.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drops into #main POST /api/capture/upload one file at a time, with
space_id pre-filled from localStorage.last_space_id (set whenever the
space view renders).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
POST /api/ingest/karakeep accepts Karakeep webhook payloads. HMAC
signature on the raw body captured by express.json's verify hook.
Mounted on app before mountApi so it bypasses agentOrOwner — the
shared secret IS the auth.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces FTS-only /api/search in place. RRF (k=60) fuses ts_rank and
pgvector cosine distance rankings. Vector branch silently skipped when
Ollama times out / errors, keeping search snappy and resilient.
Messages have no embeddings in Plan 3, so they participate in the FTS
branch only.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the validate-then-call-fetch pattern (which left a TOCTOU
window where the OS resolver could return a different IP at connect
time) with an undici Agent dispatcher whose lookup() returns the IP we
already validated. Same hardening on every redirect hop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
create/update on embeddable repos enqueue embed.text with a singleton
key that coalesces rapid edits. No-op when the queue is not running
(server tests construct createApp without booting pg-boss).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pads nomic-embed-text's 768 dims to 1024 zeros so a later 1024-dim model
swap is a re-embed, not a migration (per master spec).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
safe_fetch.js validates URLs before fetch: rejects non-http(s), literal
or DNS-resolved loopback / RFC1918 / link-local / CGNAT / metadata
addresses; follows redirects manually with the same checks on each hop.
Test fixtures gate the check with VOID_INGEST_ALLOW_PRIVATE for offline
fixtures that hit 127.0.0.1.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Unifies pgboss.job (current, per-queue partitioned) and pgboss.archive
under one SELECT for operator views. retry promotes archived rows back
into the active partition.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Job queue starts only in the CLI gate (not inside createApp), so tests
manage their own queue lifecycle. waitForJob() takes a (name, id) pair
to match pg-boss v10's getJobById signature.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-name ensureQueue promise dedup so concurrent enqueue+subscribe
on the same queue do not race createQueue (Postgres deadlock).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Search view: read ?q from hash, call /api/search, group hits by kind
with rank + space_id; sidebar filters for kinds and space_id; updates
on Enter or filter change.
Bumps package.json + server.js VERSION to 2.0.0-alpha.2 and pins the
/health version assertion to match.
CHANGELOG: full Plan 2 entry covering API surface, capability tiering,
audit chain extension (approve/reject events), and the SPA shell.
Security: adds safeHref() to dom.js and applies it everywhere an
API-supplied URL becomes href / src (reference media block + reference
source_url anchor + resource url anchor). javascript: and other
non-http(s)/mailto schemes from agent-suggested content can no longer
execute in the owner's browser.
Plan 2 surface is feature-complete: 22/22 tasks landed, 185 tests
across 43 files, SPA renders end-to-end including the suggest -> approve
agent flow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resource: status header (status + runtime_type + host + clickable URL),
three-column row of Dependencies (with add-by-UUID form) / Source docs /
Runbook pages (filters /api/links/to/resource/:id for from_type=page
and relation=runbook, then fetches each page title). Change history
card pulls /api/resources/:id/changes.
Inbox: groups pending changes by agent. Each row shows entity type
badge + action + reason, with a collapsible payload disclosure.
Approve calls /api/pending-changes/:id/approve and, if the response
carries entity_id, navigates to the resulting detail view. Reject just
re-fetches. Both update the shared pending-count emitted to state.js
so the sidebar and topbar badges drop immediately.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Page view: header + split-pane markdown editor (textarea on left,
marked + DOMPurify rendered preview on right) + backlinks card pulling
/api/pages/:id/backlinks. Save calls PATCH /api/pages/:id with body_md
and surfaces the resulting updated_at as a timestamp.
Reference detail: media block (image preview / YouTube embed via
youtube-nocookie / link fallback), summary card, metadata table, tags
card with attach/detach (creates the tag idempotently then attaches),
linked-from card from /api/links/to/ref/:id.
marked + DOMPurify vendored to public/vendor as ESM. The markdown
editor uses the explicit html: opt-in on dom.js's preview element
only — all other text comes from textContent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Home: recent activity feed from /api/audit/actor?limit=20 with relative
timestamps and entity-typed links into detail views.
Space: header + three-column row of Projects / Open tasks (status=todo) /
Recent pages + refs cards. Status badges on projects and tasks use the
shared .status palette.
Project: header (status + start/complete dates), Tasks card with inline
status badges that cycle todo->doing->blocked->done on click (PATCH
/api/tasks/:id), Pages in space card, Add-task inline form bound to
project_id.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sidebar: Spaces tree with lazy-expand to projects on caret click; bottom
Navigate section with Sacred Valley / Search / Inbox + placeholders for
Agents and Resources greyed out as later. Inbox item carries a
pending-count badge that wires to state.js so the topbar bell and the
sidebar share one poll.
Topbar: brand, + Capture button (modal stub for Plan 3 capture queue),
global search input (Enter -> /search?q=), pending Inbox bell with
matching badge, Owner toggle (stub for agent-switching post-Plan-2).
Rightrail remains the T17 collapsible companion placeholder.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three-column grid (sidebar / main / right rail) with Cradle aesthetic:
blackflame accent on Cinzel display headings + Cormorant Garamond body
in cards, system UI for chrome. Hash-based router covers all entity
routes plus search, inbox, sacred-valley. api.js stores OWNER_TOKEN in
localStorage and prompts via a modal on 401. dom.js provides safe el()
+ mount() builders so no component ever assigns innerHTML from API data
(the only exception is an explicit, scary-named html: opt-in for
sanitizer output, used later by the markdown editor).
state.js is a tiny event bus for shared chrome state (pending count).
Components and views are loaded as ES modules — sidebar / topbar /
rightrail + 9 view stubs that the later Phase E tasks fill in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single GET /api/search?q=&space_id=&kinds=&limit=&offset= unions FTS
hits across pages / refs / source_docs / messages with a `kind`
discriminator and ts_rank ordering. Each branch's to_tsvector matches
the GIN index expression on its source table so indexes are used.
Messages have no space_id and are excluded when a space filter is set.
Hybrid vector / RRF lands in Plan 3.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
