garrytan/gbrainmarkdown explorer
garrytan/gbrainmaster
skills / migrations

v0.40.3.0

skills/migrations/v0.40.3.0.md


name: v0.40.3.0-migration description: Post-upgrade actions for v0.40.3.0 — contextual retrieval + cache invalidation gate + mode-switch UX + mount/source verbs + RemediationStep refactor feature_pitch: | Two big things shipped: (1) the contextual retrieval cathedral — every chunk gets its page title prefixed before embedding, so search understands what each chunk is about. Power users on tokenmax get a per-chunk Haiku synopsis (35-49% reduction in retrieval failure rate per Anthropic's published research). (2) Cache invalidation that's actually correct via a two-layer gate (corpus-state bookmark + per-page snapshot), so search results expire the moment a page changes instead of waiting for the TTL. Plus four CLI improvements: mounts enable/disable/trust-frontmatter, sources set-cr-mode, config-set mode-switch UX, and a generic RemediationStep refactor.

v0.40.3.0 Migration

What gbrain upgrade did automatically

  1. Merged master (6 commits: v0.38 ingestion cathedral, v0.38.1 agents, v0.38.2 doctor, v0.39 brainstorm cathedral, v0.39.1 schema packs, v0.39.2 autopilot/cycle-lock).
  2. Renumbered v81 → v90 to clear master's v82-v88 reservations + v0.40.2.0-trajectory-routing's v89 reservation. Same 5 contextual retrieval columns: pages.contextual_retrieval_mode, pages.corpus_generation, sources.contextual_retrieval_mode, sources.trust_frontmatter_overrides, query_cache.page_generations.
  3. Added migration v91 with pages.generation BIGINT, query_cache.max_generation_at_store BIGINT, the bump_page_generation_fn trigger (BEFORE INSERT OR UPDATE on a 10-column content allow-list per codex D6), and the pages_generation_idx btree (CONCURRENTLY on Postgres, plain CREATE INDEX on PGLite).
  4. Bumped chunker version 2 → 3 signaling the post-upgrade reembed sweep that every markdown page needs re-embedding through the new wrapper path.
  5. Bumped KNOBS_HASH_VERSION 3 → 5 (skipping past 4 reserved by salem's v0.40.4 graph-signals work).
  6. Showed the post-upgrade cost prompt for the title-tier reembed.

What you should verify

Run:

gbrain doctor

Expected: status: 'healthy', contextual_retrieval_coverage check passes, no migration-version warnings.

If anything looks wrong:

cat ~/.gbrain/upgrade-errors.jsonl  # if it exists
gbrain apply-migrations --yes        # re-runs the orchestrator

Optional: opt up to per-chunk Haiku synopsis (tokenmax mode)

The premium tier costs ~$1-5 per 10K pages in Anthropic Haiku calls, plus ~17 hours of wallclock at Anthropic's default rate limit. Per Anthropic's published research, document-heavy benchmarks see a 35-49% reduction in retrieval failure rate.

gbrain config set search.mode tokenmax

The mode-switch banner explains the cost and (on TTY + active Minion worker) offers to submit gbrain reindex --markdown as a Minion job. Non-TTY callers get a paste-ready hint to stderr. Suppress entirely with GBRAIN_NO_MODE_SWITCH_UX=1.

If you skipped the prompt, run reindex manually:

gbrain reindex --markdown
# Or as a background job:
gbrain jobs submit reindex --params '{"markdown":true}' --allow-protected

Optional: per-source CR mode overrides

If you have multiple sources and want different CR tiers per source:

gbrain sources set-cr-mode <id> <none|title|per_chunk_synopsis>
gbrain sources set-cr-mode <id> unset  # clears the override (falls through to global)

Optional: mount frontmatter trust

If you mount a team brain (gbrain mounts add team-id ...), per-page contextual_retrieval_mode frontmatter overrides are OFF by default. Opt in:

gbrain mounts trust-frontmatter team-id

Reverse:

gbrain mounts untrust-frontmatter team-id

Soft kill switch

If quality regresses post-deploy (cache hit rate drops, search results look worse), the soft kill switch disables future wrapping while leaving already-wrapped vectors in the DB:

gbrain config set search.contextual_retrieval_disabled true

Existing cache rows continue to serve via the backward-compat path (empty {} snapshot vacuously valid). New embeds skip the wrapper. Recovery is then gbrain reindex --markdown to re-embed un-wrapped.

What changed in the file layout

  • New module: src/core/remediation-step.ts (canonical RemediationStep type + makeRemediationStep factory).
  • New module: src/core/search/query-cache-gate.ts (two-layer cache gate helpers).
  • New module: src/core/search/mode-switch-ux.ts (transition summarizer
    • worker probe + Minion submission).
  • New module: src/core/contextual-retrieval-service.ts (already present from v0.40.3.0; just moved over via the merge).
  • Extended: src/commands/mounts.ts (4 new verbs).
  • Extended: src/commands/sources.ts (set-cr-mode verb).
  • Extended: src/commands/config.ts (mode-switch hook).
  • Extended: src/commands/doctor.ts (integrity + sync_failures checks emit RemediationSteps).
  • Extended: src/commands/jobs.ts (3 new handlers: lint-fix, integrity-auto, sync-retry-failed).

Backward compatibility

  • Pre-v0.40.3.0 cache rows have page_generations = '{}'::jsonb AND max_generation_at_store = 0. The Layer 2 gate treats the empty snapshot as vacuously valid, so legacy rows continue serving until they age out via TTL (default 1h). One-time cache miss spike on upgrade is bounded.
  • Existing pages without generation column default to 1 via the v91 migration's NOT NULL DEFAULT 1.
  • Existing mount entries without trust_frontmatter_overrides default to false via loadMounts() projection.
  • Existing source entries without contextual_retrieval_mode default to NULL (falls through to global mode).
Continue exploring589 Markdown documents in the local repository