Test Before Bulk Convention

Never run a batch operation without testing one first.

The Process

1. Read the skill first. Don't write throwaway scripts. If a skill exists, use it.
2. Hone the prompt/logic. Get the output format right before running anything.
3. Test on 3-5 items. Run in --test or --dry-run mode if available. Don't commit or push.
4. Check the work yourself. Read the actual output. Is quality pristine? Titles good? Entities extracted? Back-links created? Format clean?
5. Fix what's wrong. Update the skill, not a one-off script. The skill is the durable artifact.
6. Only then: bulk execute. Through the progressive ramp below — with native pacing, progress reporting, commits every N items, and a kill switch.

Why This Matters

One bad bulk run can write 170 mediocre pages that are harder to fix than to do right the first time. The marginal cost of testing 5 first is near zero. The cost of cleaning up a bad bulk run is enormous.

Quality is only half the failure surface. The other half is the silent zero-output run: an embedding backfill once burned thousands of API calls over half an hour and wrote zero rows — every insert failed on a NOT NULL constraint, and the script's own logging never noticed. Exit code 0, money spent, database unchanged. A 10-item trial with a count-before/count-after check would have caught it in 30 seconds. "The script ran without errors" is not the same as "the output exists."

The Progressive Ramp (10 → 100 → 500 → full)

A 5-item quality test is necessary but not sufficient. For any operation that touches more than ~50 items, calls an external API in a loop, writes to the database in bulk, runs longer than 2 minutes, or costs money per item: ramp up in stages instead of jumping from the small test to the full batch.

Round 1: Trial 10

1. Run on exactly 10 items.
2. Verify output EXISTS — this is the step that gets skipped:
   • Writing to the DB: count rows in the target table before AND after. The delta must equal the expected rows.
   • Writing files: ls <output_dir> | wc -l before and after.
   • Calling APIs: check response codes, not just "no errors."
3. Spot-check 3 random outputs for quality: all expected fields populated? Values in sane ranges? Links and foreign keys resolve?
4. STOP on any failure. Fix the bug. Re-run trial 10.

Round 2: Ramp 100

1. Run on 100 items (skip the 10 already done).
2. Verify output: count check, spot-check 5 random items.
3. Error rate must be below 2%.
4. Note throughput (items/sec) and project the full-batch runtime.
5. STOP if the error rate is 2% or higher, or quality degrades.

Round 3: Ramp 500

1. Run on 500 items; same verification as Round 2.
2. If the items should be searchable, query for a few of them and confirm they come back.
3. Estimate total cost for the full batch (per-item cost x remaining items). Check it against the active spend posture (docs/operations/spend-controls.md).
4. STOP if anything is off.

Round 4: Full Batch

1. Only after three clean rounds.
2. Run with progress reporting and pacing (see below), commits every N items, and a kill switch.
3. Post-batch verification: total count matches expected.

Verification Checklist (copy-paste for each round)

□ Count before: ___
□ Items processed: ___
□ Count after: ___
□ Delta matches expected: yes/no
□ Spot-check 3 outputs: all fields populated? yes/no
□ Error rate: ___% (must be < 2%)
□ Throughput: ___ items/sec
□ Estimated full-batch time: ___
□ Estimated full-batch cost: $___

Use the Native Machinery (don't rebuild it in bash)

gbrain already ships the bulk-run plumbing. Wrapping a bulk command in sleep loops or stop/continue scripts rebuilds worse versions of these:

• Pacing (DB-contention throttling): gbrain embed --stale --pace (bare --pace = balanced; or --pace=gentle|balanced|aggressive), plus --pace-max-concurrency=N. The config key is pace.mode, and GBRAIN_PACE_* env vars override config as the incident escape hatch. gbrain sync reads the same env/config. Details in the Pace Mode section of CLAUDE.md and src/core/pace-mode.ts.
• Progress reporting: the global flags --progress-json, --progress-interval=<ms>, and --quiet work on every bulk command (doctor, embed, import, export, sync, extract, migrate, ...). Progress streams to stderr; stdout stays clean for data. See docs/progress-events.md.
• Dry runs: gbrain embed --stale --dry-run shows what would be embedded without spending anything.

What Silent Failure Looks Like (the ramp catches all of these)

1. Silent INSERT failure — the script runs, counters increment in memory, the DB has 0 new rows.
2. Schema mismatch — a column was renamed or a NOT NULL added; the script writes against the old shape.
3. Credential expiry — the first call works (cached token), the bulk run fails once the token expires.
4. Rate limiting — the trial is fine at low volume, the full batch hits 429s.
5. Memory blow-up — 10 items fit in memory, 10K does not.
6. Wrong target — writing to the wrong source or brain. Check --source / --brain routing before Round 1.

Applies To

• Video/media enrichment batches
• People/company enrichment batches
• Brain backfill operations (embeddings, edges, frontmatter)
• Any cron job being deployed for the first time
• Any new skill being run at scale
• Meeting ingestion batches

Anti-Patterns

• Writing a bash script from scratch instead of using an existing skill
• Running 170 items without testing 5 first
• Jumping from the 5-item test straight to the full batch — ramp 10 → 100 → 500 → full
• Trusting exit code 0 without a count-before/count-after check
• Hand-rolling sleep loops or throttle wrappers instead of --pace / pace.mode
• Skipping entity propagation "as a separate step"
• Committing bulk work without reading the output
• "I'll fix the quality later"