garrytan/gbrainmarkdown explorer
garrytan/gbrainmaster
skills / migrations

v0.46.3.0

skills/migrations/v0.46.3.0.md


version: 0.46.3 feature_pitch: headline: "ZeroEntropy shuts down 2026-09-04 — switch embedding + reranker before then. New default: Voyage." one_liner: "One command migrates the brain (embeddings AND reranker): gbrain migrate embeddings --to voyage:voyage-4 --dim 1024" user_action_required: true triggers:

  • "switch embedding provider"
  • "migrate my embeddings"
  • "switch reranker"
  • "ZeroEntropy"
  • "provider_sunset"
  • "search stopped working after a provider shutdown"

v0.46.3 migration — ZeroEntropy shutdown (ACTION REQUIRED before 2026-09-04)

ZeroEntropy's hosted API — the former default for BOTH embedding (zembed-1) and reranking (zerank-2) — stops working on 2026-09-04. A brain still resolving to zeroentropyai:* loses semantic retrieval ENTIRELY on that date (existing vectors become unqueryable, not just new content).

Nothing was changed automatically. v0.46.3 only detects exposure and files this action item. The brain keeps working until the date. The switch itself costs money (a one-time re-embed) and needs a target API key, so it is the user's/agent's explicit call.

Agent action — when to run this migration

Run the steps below when the user says anything like:

  • "ZeroEntropy" / "zembed" / "zerank" anything
  • "migrate my embeddings" / "switch embedding provider" / "switch reranker"
  • "search stopped working" / "embedding provider error" (after 2026-09-04)
  • "doctor says provider_sunset" / "gbrain is warning about a provider"
  • after any gbrain upgrade that printed the ACTION REQUIRED sunset block

Also proactively surface it when ~/.gbrain/migrations/pending-host-work.jsonl contains a "migration": "0.46.3" entry.

Step 0 — env preflight (do this FIRST, it prevents the worst failure mode)

env | grep GBRAIN_EMBEDDING

GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS override the file plane at runtime for every gbrain process. Three cases:

  • Nothing printed — proceed to Step 0.5.

  • Set and equal to the target (e.g. voyage:voyage-4 / 1024): the migration proceeds with a notice and also writes the file plane. Keep the env in sync everywhere gbrain runs (cron, workers, other shells) — or unset it so the file plane is the single source of truth. Env-canonical deployments (containers with no ~/.gbrain/config.json) are supported: the env IS the config there.

  • Set and different from the target: the live run REFUSES (this is the guard against config-says-new/runtime-embeds-old damage). Fix before running:

    unset GBRAIN_EMBEDDING_MODEL GBRAIN_EMBEDDING_DIMENSIONS
    

The command never trusts these vars for its "nothing to migrate" decision — it verifies the database directly — so a pre-set env var can no longer fake a completed migration. But an env var pointing elsewhere WILL poison future embeds in other processes, which is why the mismatch refuses.

Step 0.5 — quiesce embed writers

gbrain jobs list --status running 2>/dev/null; gbrain jobs list --status waiting 2>/dev/null

Stop the minion worker (or let embed/embed-catch-up/embed-backfill jobs drain) before migrating. The migration takes the brain-wide migration lock + every per-source embed lock, but generic embed jobs submitted DURING the run don't take those locks — anything they write in the old space is caught by the final census and re-embedded (costing you twice). The plan output warns when live workers/jobs are detected.

Step 1 — confirm exposure

gbrain doctor --json | jq '.checks[] | select(.name=="provider_sunset")'

warn/fail mentioning zeroentropyai = exposed. ok = likely done — verify with gbrain migrate embeddings --status (Step 5) before clearing the pending-host-work entry; doctor alone can be fooled by env overrides, the status command cannot.

Step 2 — pick the target by which key exists

Preferred — Voyage (VOYAGE_API_KEY in env, or voyage_api_key in ~/.gbrain/config.json). One key covers embedding + reranking + the multimodal model. To set the key: export VOYAGE_API_KEY=... or edit ~/.gbrain/config.json directly — do NOT use gbrain config set voyage_api_key (that writes the DB plane, which the embedding pipeline never reads).

gbrain migrate embeddings --to voyage:voyage-4 --dim 1024 --dry-run   # cost preview — show the user
gbrain migrate embeddings --to voyage:voyage-4 --dim 1024 --yes

Note: 1280 is not a valid Voyage width (valid: 256/512/1024/2048), so this includes a one-time schema/HNSW index rebuild to 1024. The command handles it; it is resumable if killed.

Alternative — OpenAI (OPENAI_API_KEY). No reranker coverage, but OpenAI text-embedding-3 supports flexible widths, so a 1280d brain can keep its column (no schema rebuild):

gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 --dry-run
gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 --yes
# (or --dim 1536 for the model's native width — costs a schema rebuild)

Neither key — get one of the two (Voyage: https://dash.voyageai.com/api-keys), or self-host (below).

Step 3 — reranker (handled IN the same command)

The migration handles the reranker automatically (--reranker auto is the default): when the brain's ACTIVE reranker — including the mode-bundle default zeroentropyai:zerank-2 that most ZE brains ride without any explicit config — is exposed, and the target provider ships a reranker, the run probes it live and switches search.reranker.model in the same consented pass. Overrides:

--reranker off               # disable reranking instead
--reranker keep              # leave reranker config untouched
--reranker voyage:rerank-2.5 # explicit model (validated before anything runs)

Migrating to a provider with no reranker (OpenAI)? The run prints an ACTION line with the exact commands instead of silently enabling a third provider:

gbrain config set search.reranker.model voyage:rerank-2.5   # needs VOYAGE_API_KEY
gbrain config set search.reranker.enabled false             # or turn it off

Why the plane asymmetry: embedding config lives on the FILE/ENV planes (it sizes the schema, so it must be stable across engine connects — never gbrain config set embedding_model), while reranker config lives on the DB plane (gbrain config set search.reranker.* is correct there). The migration writes each to its right plane; you only need to know this when doing it by hand.

Without a working reranker, reranking fails open after the shutdown date — search still works, ordering quality drops, and each search pays the timeout.

Step 4 — custom embedding columns (rare)

If the notice listed ZE-backed embedding_columns entries: there is no automated off-ramp for custom columns yet (migrate embeddings covers the primary column only). Re-declare the column config on the new provider and re-embed its content, or drop the column config. A write-side custom-column migration is a filed follow-up (TODOS.md).

Step 5 — verify (trust the database, not the env)

gbrain migrate embeddings --status

Read the output top to bottom — it shows every config plane (env presence, file, DB), the actual column widths, how many chunks/facts still lack vectors, the page-signature census, and the smoke-check outcome from the completed migration. Converged looks like: column at the target width, 0 chunks missing, signature census all on the target, no migration in flight.

Then:

gbrain doctor --json | jq '.checks[] | select(.name=="provider_sunset") | .status'   # → "ok"
gbrain search "anything you know is in the brain"                                     # sanity check

Facts note: fact vectors regenerate on their next write / gbrain extract pass — --status shows the pending count; it is not a failure.

Then remove/mark the 0.46.3 entry in ~/.gbrain/migrations/pending-host-work.jsonl as done — edit the file and change that entry's "status" to "done" (or delete the line). There is no CLI for this yet (filed follow-up).

Recovery — when a run is killed, fails, or something looks wrong

Exit codes: 0 = completed (or verified nothing-to-do), 1 = incomplete / refused / failed (message says which), 2 = non-TTY without --yes.

  • Killed / crashed mid-run → re-run the SAME command. The NULL-embedding column is the checkpoint; already-migrated chunks are never re-embedded or re-billed. gbrain migrate embeddings --status shows the in-flight marker and prints the exact resume command.
  • "Migration paused ... lock" → another embed backfill holds a per-source lock. Check gbrain jobs list; a hard-killed run's lock expires within 60 minutes — re-run then.
  • "lock was lost mid-drain" → another process stole the lock (mutual exclusion ended); partial progress is banked. Re-run once the other holder finishes.
  • Refused: a migration to X is still in flight → resume THAT target with the printed command, or abandon it deliberately with --retarget.
  • Wrong --dim on the first pass → re-run with the right --dim; the column rebuilds at the new width and the re-embed runs again (vectors at the wrong width are unusable — this re-bill is unavoidable).
  • Deferred the re-embed with --no-embed → finish with: gbrain embed --stale --catch-up --include-null-signature (--background carries all of these flags into the job).
  • Only want status, never mutationgbrain migrate embeddings --status is read-only and spend-free.

Self-hosting (zero re-embed, advanced)

zembed-1's weights are Apache-2.0. Keeping vectors WITHOUT re-embedding requires keeping the zeroentropyai:zembed-1 model id (the embedding signature must not change) and pointing its base URL at your own endpoint:

gbrain config set provider_base_urls.zeroentropyai http://your-host:PORT/v1

Two honest caveats:

  1. The endpoint must speak ZeroEntropy's wire dialect (/models/embed, {results: [...]} responses) — a generic OpenAI-compatible llama-server or Ollama endpoint will NOT work; you need a ZE-wire-compatible proxy in front.
  2. This works only until the September removal release deletes the zeroentropyai recipe. A self-host continuity decision (possibly a compat mode) is tracked in TODOS.md — if you rely on this path, follow that item.

Switching the provider id instead (e.g. llama-server:zembed-1) changes pages.embedding_signature, and the next stale-embed pass re-embeds everything — that path is a full re-embed, not a zero-cost move.

Note: gbrain migrate embeddings refuses --to zeroentropyai:* by default (it protects everyone else from re-embedding onto the dying hosted API). With your base-URL override in place, pass --force-sunset-target to proceed.

When NOT to run this migration

  • The brain is keyless (embedding_disabled: true) with no ZE reranker or custom columns — nothing to migrate.
  • gbrain migrate embeddings --status shows convergence on a non-ZE target AND provider_sunset reports ok — done; just clear the pending entry.
  • You only mounted someone else's brain: the migration is host-scoped; the brain's owner migrates it (their upgrade banner + doctor nag them).
Continue exploring589 Markdown documents in the local repository