garrytan/gbrainmarkdown explorer
garrytan/gbrainmaster
docs / architecture

type taxonomy

docs/architecture/type-taxonomy.md

Type Taxonomy (gbrain-base-v2)

The 14-canonical-type DRY/MECE taxonomy. Predecessor gbrain-base (24 types) stays bundled for back-compat; fresh installs default to gbrain-base-v2.

Why

A production gbrain brain (186K pages) had accreted 94 distinct pages.type values in 9 clusters of redundancy. The type system is the foundation for schema packs, search filtering, extract behavior, enrichment routing, and expert routing. When types are noisy, every downstream feature degrades:

  • Search filtering is ambiguous--type article misses 2.2K articles typed as media/article, sources/article, etc.
  • Enrichment routing is incompleteenrichable_types could only list a few canonical types; 80+ legacy types meant most pages never got enriched.
  • Agent confusion — when ingesting a new article, should it be article, media/article, sources/article, or source/article? Four reasonable choices, none of them right.
  • Orphan inflation — 5,521 concept-redirect pages inflated orphan counts without adding knowledge value.

Issue #1479 catalogues the 9 clusters with exact counts. This doc is the response: a coherent 14-type taxonomy with subtypes/format/origin pushed to frontmatter, alias-table rows for redirects, real link-table rows for edge-shaped pages.

The 14 canonical types (+ note catch-all)

TypePrimitiveWhat it holdsExamples
personentityPeopleFounders, partners, individuals
companyentityCompanies, products, orgs (subtype-distinguished)Companies, YC-companies, products
mediamediaArticles, videos, essays, books, podcasts (subtype-distinguished)Substack posts, YouTube videos, books
tweetmediaTwitter posts (single/bundle/stub subtype)Single tweets, threads, bundles
social-digesttemporalPeriod-grouped social summaries (daily/monthly)X account daily digests
analysismediaResearch + competitive intelMarket analysis, pricing analysis
atomannotationKnowledge units (extraction/manual/lore subtype)Extracted facts, manual notes, lore
conceptconceptIdeas + reference pagesWiki concepts
sourcemediaTranscripts, referencesInterview transcripts
dealtemporalInvestment dealsTerm sheets, investments
emailtemporalEmail threadsEmail correspondence
slacktemporalSlack messages + threadsSlack conversations
writingmediaOriginal writingDrafts, essays in progress
projectconceptInitiatives, workstreamsInternal projects
noteconceptCatch-all for one-offs (legacy_type preserved)Memos, anecdotes, insights, etc.

15 types total (14 canonical + note). The catch-all retype rule binds any uncovered legacy type to note with frontmatter.legacy_type = <original> preserved for rollback.

Subtypes (declared in frontmatter post-unify)

CanonicalSubtype fieldValues
companysubtypecompany / product / org
mediasubtypevideo / article / essay / book / podcast / blog
tweetsubtypesingle / bundle / stub
social-digestsubtypedaily / monthly
atomsubtypeextraction / manual / lore

subtype_field for retype rules is restricted to an allowlist: {subtype, legacy_type, origin, format, kind, period, domain}. This prevents third-party packs from injecting title, slug, or type via mapping_rules (codex D9 security hardening).

Migration flow

gbrain onboard --check                         # surfaces pack_upgrade_available
        ↓
gbrain onboard --check --explain               # per-cluster narrative dry-run
        ↓
gbrain jobs submit unify-types \               # PROTECTED + manual_only
  --allow-protected \
  --params '{"target_pack":"gbrain-base-v2","apply":true}'
                                               # omit "apply":true → dry-run (default)
        ↓
Handler runs 8 phases:
  ┌─────────────────────────────────────┐
  │ Phase 1: Preflight + lock           │ → gbrain-unify db-lock (60min TTL)
  ├─────────────────────────────────────┤
  │ Phase 2: Retype explicit rules      │ → chunked UPDATE 1000/batch
  ├─────────────────────────────────────┤
  │ Phase 3: Retype catch-all sentinel  │ → 'note' with legacy_type
  ├─────────────────────────────────────┤
  │ Phase 4: Page-to-link conversions   │ → insert links + soft-delete
  ├─────────────────────────────────────┤
  │ Phase 5: Page-to-alias conversions  │ → insert slug_aliases + soft-delete
  ├─────────────────────────────────────┤
  │ Phase 6: Final sync (residual)      │ → path-prefix typing
  ├─────────────────────────────────────┤
  │ Phase 7: Flip active pack (D13)     │ → engine.setConfig + saveConfig
  ├─────────────────────────────────────┤
  │ Phase 8: Verify + celebrate         │ → assert ≤16 types; stderr summary
  └─────────────────────────────────────┘
        ↓
gbrain onboard --check                         # pack_upgrade_available cleared
                                               # type_proliferation cleared

Rollback paths

Every primitive ships with a documented rollback:

OperationRollback
Retypefrontmatter.legacy_type = <original> preserved on every page (D8). One SQL UPDATE restores types: UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'.
Page-to-linkSource page soft-deleted with 72h TTL. gbrain restore <slug> within 72h. Link row stays harmless if source restored.
Page-to-aliasSource page soft-deleted with 72h TTL. gbrain restore <slug> within 72h. Alias row stays harmless (or DELETE FROM slug_aliases WHERE alias_slug = <slug> to clean up).
Active-pack flipgbrain schema use gbrain-base reverses the flip.

What if my brain doesn't fit?

The catch-all retype rule (from_type: '*unknown*') handles long-tail types automatically — any page whose type isn't covered by an explicit rule AND isn't a page_to_link / page_to_alias source gets retyped to note with legacy_type preserved. Guarantees ≤16 distinct types post-unify on ANY brain.

For brains with substantial custom types that deserve their own canonical (e.g. researcher for an academic brain), the right move is:

  1. Fork gbrain-base-v2: gbrain schema fork gbrain-base-v2 my-pack
  2. Edit your fork to add page_types + mapping_rules covering your custom domain.
  3. Target your fork: gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"my-pack","apply":true}' (omit "apply":true for a dry-run preview — that is the default)

Your fork can also declare migration_from: {pack: gbrain-base-v2, version: "1.x"} to register itself as a successor — future agents discovering your pack via pack_upgrade_available will offer the migration.

Wikilink resolution post-unify

The slug_aliases table IS the resolver (D15: codex outside voice — don't rewrite body-text wikilinks; the alias table is the right primitive). Wikilinks like [[old-redirect-slug]] keep working post- unify because:

  1. The wikilink resolver short-circuits through engine.resolveSlugWithAlias(slug, sourceId) BEFORE the existing fuzzy/prefix cascade.
  2. The lookup queries slug_aliases for any matching alias_slug in the provided source(s).
  3. If found, returns the canonical_slug. The renderer then resolves the wikilink to the canonical page.

Multi-source ambiguity (same alias_slug in two registered sources) emits a once-per-process multi_match stderr warning and returns the first match by source array order. Federated reads pass the full allowed-source array.

Search ranking signal: alias_resolved_boost

Post-unify, search results whose slug is a canonical_slug in slug_aliases get a 1.05x score multiplier via the applyAliasResolvedBoost post-fusion stage. Semantic intent: "user explicitly disambiguated this as canonical, so it should outrank fuzzy matches that hit aliases by accident."

SearchResult.alias_resolved_boost is stamped on touched results for --explain formatter visibility. The stage participates in the search cache key (KNOBS_HASH_VERSION in src/core/search/mode.ts is the single source of truth for the current cache-key version), so cache rows written before the stage existed are unreachable.

Reference

  • Issue: https://github.com/garrytan/gbrain/issues/1479
  • Pack file: src/core/schema-pack/base/gbrain-base-v2.yaml
  • Pack-upgrade mechanism: docs/architecture/pack-upgrade-mechanism.md
  • Migration handler: src/core/schema-pack/unify-types-handler.ts
  • Onboard checks: src/core/onboard/checks.ts
  • Skill: skills/schema-unify/SKILL.md
Continue exploring589 Markdown documents in the local repository