garrytan/gbrainmarkdown explorer
garrytan/gbrainmaster
skills / migrations

v0.36.2.0

skills/migrations/v0.36.2.0.md


name: v0.36.2.0 version: 0.36.2.0 headline: "HISTORICAL: ZeroEntropy was briefly the default for embedding + reranker"

v0.36.2.0 Migration: ZeroEntropy as default

HISTORICAL — DO NOT FOLLOW. Do not execute any command in this file. ZeroEntropy's hosted API shuts down 2026-09-04; every ze-switch command below exits 1 with reason: provider_sunset today (the command is a pure refusal/redirect shim), and the benchmark pitch below is obsolete. To LEAVE ZeroEntropy, follow skills/migrations/v0.46.3.0.md.

TL;DR (historical record): v0.36.2.0 made ZeroEntropy (zembed-1 at 1280d via Matryoshka, zerank-2 cross-encoder reranker) the default retrieval stack, on the strength of a real-corpus benchmark that has since been overtaken by events — the provider announced its shutdown. New installs landed on it automatically. Existing brains saw a one-shot upgrade prompt with a cost estimate; default-on-Enter was "stay on current provider".

This migration was opt-in for existing brains. Running gbrain upgrade did not silently re-embed data. The prompt was TTY-only; non-TTY upgrades (CI, cron, docker) printed an informational stderr line and skipped the switch.

What the agent did post-upgrade (v0.36.2 era — every command here exits 1 today)

When a user invoked any gbrain command immediately after gbrain upgrade to v0.36.2.0, the playbook was:

  1. The agent checked planner state via ze-switch --dry-run --json (ze_switch_offered, ze_switch_prompt_shown, the comparison numbers).

  2. It surfaced the era's benchmark numbers (head-to-head query wins, latency, price, reranker reshuffle rate, provider overlap).

  3. It recommended switching unless the user had a constraint making ZE unsuitable, pointing at the interactive prompt and the dry-run plan.

  4. For a prior decline, --force re-opened the prompt (90-day re-ask window).

  5. For regret after switching, --undo restored the prior model + dim + reranker state behind a cost-warning prompt.

None of that flow exists anymore: the recommendation aged into a liability when the shutdown was announced, and today the entire surface refuses.

The CLI surface in full (historical — every row exits 1 with provider_sunset today)

CommandEffect (v0.36.2 era)
gbrain ze-switchInteractive prompt (TTY only). Default-on-Enter = stay.
gbrain ze-switch --dry-runPrint plan as text. Change nothing.
gbrain ze-switch --dry-run --jsonMachine-readable envelope: {status, plan}.
gbrain ze-switch --non-interactiveSwitch without prompting. Errors if ZEROENTROPY_API_KEY missing.
gbrain ze-switch --non-interactive --ignore-missing-keySame, but stage the schema change before the key is ready. Embeddings fail loud until key arrives.
gbrain ze-switch --resumeComplete a half-applied switch (crash recovery).
gbrain ze-switch --forceBypass the prompt-shown gate (re-show after n).
gbrain ze-switch --undoReverse with cost-warning prompt. (Today: prints the return-path gbrain migrate embeddings command instead of acting.)
gbrain ze-switch --undo --non-interactive --confirm-reembedScripted undo. (Today: prints guidance, exit 1.)

Consolidation with the v0.32.7 chunker prompt

If a brain has BOTH a stale chunker version AND the ZE-switch offered, the RetrievalUpgradePlanner consolidates into ONE prompt + ONE re-embed pass. The user is not double-charged. This was a real bug class the original cutover plan had; codex outside-voice review caught it before implementation.

Doctor checks added in v0.36.2.0

gbrain doctor now runs two new ZE-aware checks:

  • ze_embedding_health — warns if embedding_model starts with zeroentropyai: but no key is configured. (At the time the fix hint pointed at the setup URL; today it points at the migration off-ramp.)
  • embedding_width_consistency — asserts the configured embedding_dimensions matches the actual vector(N) width on content_chunks.embedding. Warns on drift. (At the time the fix hint suggested --resume; today the check prints an engine-branched recovery recipe — there is no resume.)

What changed under the hood

  • Default embedding model: openai:text-embedding-3-large (1536d) → zeroentropyai:zembed-1 (1280d via Matryoshka).
  • Default reranker: off in balanced mode bundle → on. (Still off in conservative, still on in tokenmax.)
  • Reranker model: already zeroentropyai:zerank-2 since v0.35.0.0; that doesn't change.
  • Schema transition (when user accepts the switch): DROP indexes → ALTER content_chunks.embedding to vector(1280) → CREATE INDEX. Atomic inside one engine.transaction(). HNSW indexes recreated in the same transaction; no silent slow-search window.
  • Three new config keys: ze_switch_prompt_shown, ze_switch_requested, ze_switch_applied. Plus ze_switch_previous_snapshot (JSON, captures prior config for --undo) and ze_switch_declined_at (ISO timestamp for the 90-day re-ask gate).

What NOT to do (historical guardrails for the era's flow)

  • The --ignore-missing-key staging spelling required explaining that every embed call would fail until the key arrived.
  • The sale price was promotional; the regular price was the cost anchor.
  • The user owned the switch decision; the prompt's default-on-Enter was "stay" for exactly this reason.

Why 1280d, not 1024d

The valid ZE Matryoshka dim steps are 2560, 1280, 640, 320, 160, 80, 40. 1024 (Voyage's step) is NOT on ZE's list — see src/core/ai/dims.ts:ZEROENTROPY_VALID_DIMS. 1280 is the step closest to the prior OpenAI 1536d default and stays in the high-recall zone of the Matryoshka curve.

Verifying the switch (historical — the ze-switch line exits 1 today)

gbrain doctor                          # both new checks should be green
gbrain ze-switch --dry-run             # (era) status skipped_already_applied; (today) refuses
gbrain models                          # confirm embedding + reranker defaults
gbrain search "test query" --limit 5   # confirm the reranker is firing
Continue exploring589 Markdown documents in the local repository