garrytan/gbrainmarkdown explorer
garrytan/gbrainmaster
docs / migrations

v0.41.2 markdown greenfield

docs/migrations/v0.41.2-markdown-greenfield.md

Migrating your OpenClaw brain to gbrain v0.41.2.0 (greenfield)

The v0.41.2.0 lens packs ship a one-shot importer that re-ingests your existing OpenClaw brain (~/git/brain/atoms/, concepts/, ideas/) through the new ingestion cathedral. Pages land in gbrain with an imported_from: markdown-greenfield frontmatter marker so the new extract_atoms + synthesize_concepts cycle phases skip them (lossless import with provenance).

Before you migrate

  1. Upgrade gbrain to v0.41.2.0+:

    gbrain upgrade
    gbrain --version  # should be 0.41.2.0 or later
    
  2. Activate the creator pack (or gbrain-everything if you also want investor + engineer lenses on the same brain):

    gbrain config set schema_pack gbrain-creator
    # OR
    gbrain config set schema_pack gbrain-everything
    
  3. Apply schema migration v94 (take_domain_assignments table):

    gbrain apply-migrations --yes
    

The dry-run pass

Always start with --dry-run to see what the importer would do without writing anything:

gbrain capture --source markdown-greenfield \
  --repo ~/git/brain \
  --dry-run \
  --limit 100

The output reports:

  • emitted — atoms/concepts/ideas that would import cleanly
  • skipped_no_type — files without a type: frontmatter (counted as benign skips; no audit appended)
  • skipped_invalid — files that failed validation (these append to ~/.gbrain/audit/markdown-greenfield-failures-YYYY-Www.jsonl)

Inspect the audit JSONL:

ls ~/.gbrain/audit/markdown-greenfield-failures-*.jsonl
cat ~/.gbrain/audit/markdown-greenfield-failures-*.jsonl | jq .

Common failures:

  • Empty frontmatter — file has --- but no fields. Not a real brain page; safe to leave skipped.
  • Malformed YAML — fix the file in your OpenClaw then re-run.
  • Missing required field — usually means the original OpenClaw skill output a partial page; check whether the file is worth preserving.

The actual import

When the dry-run looks clean, drop the --dry-run and --limit flags:

gbrain capture --source markdown-greenfield --repo ~/git/brain

Expect ~30-60 minutes for the full 24K-page set (atoms + concepts + ideas). The importer:

  1. Walks atoms/{YYYY-MM-DD}/*.md, concepts/*.md, ideas/*.md in deterministic alphabetical order so partial-run resumes pick up where they left off.
  2. Stamps imported_from: markdown-greenfield + imported_at: <ISO timestamp> on every page's frontmatter, preserving ALL original fields verbatim under metadata.original_frontmatter.
  3. Emits each as an IngestionEvent with mode: 'migration' (T2), which bypasses the daemon's 24h DedupWindow. The importer owns its own permanent slug-keyed idempotency.
  4. Routes through put_page so pages land with proper FK chains and embedding eligibility.

After the import

Verify counts match:

gbrain stats
# Should show ~24K new pages with type=atom/concept/idea

The next gbrain dream cycle will:

  • Run extract_atoms on NEW transcripts (skips pages with the imported_from marker — your historical atoms are frozen, not re-extracted).
  • Run synthesize_concepts on NEW atoms (skips imported concepts for the same reason).
  • Run extract_facts over the imported pages — facts fences in imported atoms/concepts populate the facts table normally.

Retiring your OpenClaw's parallel crons

After verifying the import, retire your OpenClaw's parallel atom pipeline cron entries. In ~/git/your-openclaw/workspace/cron.json:

  • Remove atom-pipeline-coordinator (every-30-min cron)
  • Remove atom-backfill-coordinator (every-10-min cron)

Replace with nothing — gbrain's autopilot already runs extract_atoms + synthesize_concepts inside every dream cycle when gbrain-creator (or gbrain-everything) is the active pack.

The OpenClaw skills themselves shrink to thin wrappers:

  • content-atom-extractor → calls gbrain dream --phase extract_atoms
  • concept-synthesis → calls gbrain dream --phase synthesize_concepts
  • atom-backfill-coordinator → DELETED (backfill is now part of extract_atoms via the Source Quote + lesson enrichment in one Haiku call per transcript)

Rolling back

The import is fully reversible:

# Soft-delete every page with the marker (recoverable for 72h)
gbrain query "imported_from:markdown-greenfield" --type atom --json | \
  jq -r '.[].slug' | xargs -I{} gbrain pages delete {}
# OR hard-delete past the soft-delete window
gbrain pages purge-deleted --older-than 0h

Your OpenClaw's ~/git/brain/atoms/ + concepts/ + ideas/ directories are untouched by the importer — they remain the source of truth for rollback. The greenfield importer only READS from them.

Re-running after partial failures

The importer is idempotent at the page-slug level: re-running on the same --repo produces zero net-new pages (every page either lands fresh or matches an existing slug). If you fix some validation failures in your OpenClaw and want to retry just those:

gbrain capture --source markdown-greenfield --repo ~/git/brain

Already-imported pages stay; previously-failed pages get a fresh attempt; the audit JSONL accumulates per-week (ISO week file rotation).

Continue exploring589 Markdown documents in the local repository