v0.12.1 Migration: Extract Performance + Migration Timeout Fix

This release is a pure performance bug fix. No manual steps are needed for most users — re-run gbrain init --migrate-only and re-run gbrain extract all if desired. Both should now complete successfully on any brain size.

What changed

Two production-blocking bugs are fixed:

gbrain extract no longer hangs on large brains. The N+1 dedup pre-load that ran 47K serial getLinks() calls before any work started is gone. Both engines already enforced uniqueness at the SQL layer; the in-memory dedup was redundant. Combined with new batched 100-row INSERTs, a full re-extract on a 47K-page brain drops from "10+ min hang then ~minutes more" to "immediate work, ~30-60s total."
v0.12.0 schema migration no longer times out on duplicate-heavy brains. Migration v9 (timeline_dedup_index) and v8 (links uniqueness) now pre-create a btree helper index before the DELETE ... USING self-join, then drop it after dedup. Turns O(n²) dedup into O(n log n). On 80K+ duplicate rows the migration completes in under a second instead of timing out at 60 seconds.

What you need to do

If your v0.12.0 upgrade succeeded — you're already done

The migration is idempotent. The fix only matters if your migration FAILED on the v0.12.0 upgrade attempt. Run gbrain init --migrate-only once to confirm your schema is at version 10, then run gbrain extract all --dir <brain> if you want to reuse it now that it's fast.

If your v0.12.0 upgrade FAILED on `idx_timeline_dedup` creation

You may have the brain in a partial-migration state with duplicate rows in timeline_entries (or links). Run:

gbrain init --migrate-only

Migration v9 will pre-create the helper index, dedup any existing duplicates (now sub-second instead of timing out), drop the helper, and create the unique index. Idempotent — safe to re-run.

If you previously ran the manual CREATE TABLE _clean AS SELECT DISTINCT ON ... + table swap workaround Garry posted, your schema should already be at version 10. Confirm with:

gbrain config get version

If it shows 10, you're done.

If you previously did manual SQL surgery on duplicate timeline rows

The unique index idx_timeline_dedup should now be present after the workaround. Re-running gbrain init --migrate-only is a no-op for v9 (uses CREATE UNIQUE INDEX IF NOT EXISTS). The new migration code adds the helper btree on the dedup columns first — but the helper is dropped at the end of the migration, so even if v9 re-ran (it won't, version is already at 10), it would leave your schema unchanged.

Re-running `gbrain extract` on a previously-stuck brain

This is the most common case. Run:

gbrain extract all --dir <brain-dir>
# or for live brains with no local checkout:
gbrain extract all --source db

Expect immediate output (Links: created N from M pages lines streaming as files process), not a 10-minute hang. On a re-run of a fully-extracted brain you should see Done: 0 links, 0 timeline entries from N pages — that's the truthful counter at work, confirming nothing changed.

New engine API (informational, optional)

For plugin authors building integrations on the BrainEngine interface, two new methods are available:

addLinksBatch(LinkBatchInput[]) → Promise<number>
addTimelineEntriesBatch(TimelineBatchInput[]) → Promise<number>

Both return the count of rows actually inserted (excluding ON CONFLICT no-ops and JOIN-dropped rows whose slugs don't exist). Existing per-row addLink / addTimelineEntry are unchanged — no migration required for plugin code.

Verification

After the migration completes:

# Confirm schema version
gbrain config get version
# Expect: 10

# Confirm the unique indexes exist (Postgres / Supabase only)
gbrain stats
# Expect: link_count and timeline_entry_count both populated, no duplicates

If you hit any issue, file at https://github.com/garrytan/gbrain/issues with the output of gbrain init --migrate-only and gbrain config get version.

v0.12.1