garrytan/gbrainmarkdown explorer
garrytan/gbrainmaster
Repository
skills / migrations

v0.22.14

skills/migrations/v0.22.14.md

feature_pitch: headline: Bare workers now self-monitor and fail-stop into your PM's restart loop body: | Bare gbrain jobs work now ships with the same health protection the supervisor already had: DB liveness probes (with per-probe timeout so a hung connection can't wedge the monitor), stall detection filtered by registered handler names, and an RSS watchdog default of 2048 MB.

When the worker detects it's wedged (stuck pgbouncer connection, hung
event loop, stalled job claim), it emits `'unhealthy'` and the CLI calls
`process.exit(1)`. This is **fail-stop**: it requires an external process
manager (systemd, Docker `restart: always`, launchd `KeepAlive`, cron
watchdog) to bring the worker back. Without one, the process exits and
stays dead — that's a regression from pre-v0.22.14 self-healing.

Pre-v0.22.14 behavior: bare workers had ZERO health monitoring. A wedged
worker stayed alive doing nothing while jobs piled up in `waiting` and
your PM's `pgrep` check happily reported green.

If you're using `gbrain jobs supervisor`, you're already protected — the
supervisor handles spawn-on-crash itself. The fail-stop concern only
applies to direct `gbrain jobs work` invocations.

v0.22.14 — Bare-worker self-health-monitoring

⚠️ Pre-flight: confirm you have a process supervisor

If you run gbrain jobs work directly (NOT under gbrain jobs supervisor), verify your process manager is configured to restart the worker on exit BEFORE upgrading:

ManagerWhat to check
systemdRestart=always (or Restart=on-failure) in the .service unit
Dockerrestart: always / restart: unless-stopped in compose, OR --restart flag
launchd (macOS)<key>KeepAlive</key><true/> in the plist
cron watchdogCron entry that re-spawns when pgrep -f "gbrain jobs work" is empty
supervisordautorestart=true

If your bare worker has no restart loop, the v0.22.14 fail-stop behavior will leave you with a dead worker after the first DB blip. Either add a restart policy OR switch to gbrain jobs supervisor (which spawns its own child + restarts on crash internally).

What ships

  • DB liveness probes inside gbrain jobs work (60s interval, 3 strikes → exit)
  • Stall detection (5min warn / 10min exit when waiting jobs accumulate but in-flight is empty)
  • --max-rss defaults to 2048 MB for bare workers (matches supervisor default; was 0 = disabled)
  • New MinionWorkerOpts.{healthCheckInterval, stallWarnAfterMs, stallExitAfterMs, dbFailExitAfter, dbProbeTimeoutMs} for tuning (5 fields)
  • MinionWorker now extends EventEmitter; emits 'unhealthy' event with a structured reason payload. No-listener fallback: if the caller does not subscribe to 'unhealthy', the worker calls process.exit(1) itself to preserve the pre-refactor fail-stop behavior. The CLI subscribes; direct API consumers without a listener inherit the fail-stop default. Inline paths (jobs submit --follow, jobs smoke) explicitly pass healthCheckInterval: 0 to disable the timer entirely so the fallback cannot trip on a one-shot run.
  • GBRAIN_SUPERVISED=1 env var (set by supervisor on its child) causes the child worker to skip its own health timer (no double-monitoring)
  • Constructor validation: throws if stallExitAfterMs <= stallWarnAfterMs (the contract requires strict greater-than)

⚠️ Breaking change you may need to act on

gbrain jobs work now defaults --max-rss 2048 (MB). Pre-v0.22.14, bare workers had no RSS limit. If you ran bare workers with intentionally large embed/import working sets, your worker may now get killed by the watchdog after upgrade.

You need to act if any of the following is true:

  • You run gbrain jobs work directly under a process manager (systemd, Docker, cron, supervisord, your own bash wrapper)
  • Your worker handles large embed batches, multi-GB import jobs, or memory-intensive subagent tasks that legitimately exceed 2 GB RSS
  • You rely on bare-worker resident memory exceeding 2 GB

You do NOT need to act if:

  • You run gbrain jobs supervisor (the supervisor already defaulted to --max-rss 2048 and has reconnect-then-fail logic on top)
  • Your typical worker stays well under 2 GB RSS

What the agent should do post-upgrade

1. Run the orchestrator (mechanical migration)

gbrain apply-migrations --yes

This is a no-op for v0.22.14 — there's no schema migration. The mechanical work is in the binary; the user-side decision is the breaking-change handling below.

2. Detect bare-worker deployments

If the user runs OpenClaw, Hermes, or any agent fork that uses gbrain jobs work directly (NOT under gbrain jobs supervisor), they need to know. Check:

# Look for bare worker invocations in cron, systemd, Docker, launchctl, etc.
grep -r "gbrain jobs work" ~/.config/systemd/ /etc/systemd/ 2>/dev/null | head
crontab -l 2>/dev/null | grep "gbrain jobs work" | head
launchctl list 2>/dev/null | grep gbrain | head
ps -ef | grep "gbrain jobs work" | grep -v grep | grep -v supervisor

If you find bare-worker invocations, surface them to the user.

3. Ask about RSS profile

Use AskUserQuestion to confirm whether the user's bare workers need >2 GB RSS:

"v0.22.14 changes gbrain jobs work to default --max-rss 2048 (MB). Pre-v0.22.14 bare workers had no limit. If your worker handles large embed batches or multi-GB imports, the watchdog may now kill it. Do you want to keep the new 2 GB default, raise the limit, or opt out entirely?"

Options:

  • A) Keep 2 GB default (recommended for most) — protects against memory leaks; restarts on overflow; matches supervisor behavior.
  • B) Raise to N GB (specify N) — pass --max-rss <N*1024> to the worker invocation.
  • C) Opt out — pass --max-rss 0.

4. Apply the user's choice

For each bare-worker invocation, edit the unit/cron/launchctl/script to add the chosen --max-rss flag.

systemd (~/.config/systemd/user/gbrain-worker.service):

ExecStart=/usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
# Or to opt out: --max-rss 0

Then systemctl --user daemon-reload && systemctl --user restart gbrain-worker.

cron (crontab -e):

@reboot /usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096

Docker compose:

command: ["gbrain", "jobs", "work", "--queue", "default", "--concurrency", "3", "--max-rss", "4096"]

launchctl (~/Library/LaunchAgents/com.user.gbrain-worker.plist):

<key>ProgramArguments</key>
<array>
  <string>/usr/local/bin/gbrain</string>
  <string>jobs</string>
  <string>work</string>
  <string>--max-rss</string>
  <string>4096</string>
</array>

Then launchctl unload ... && launchctl load ....

5. (Optional) Tune health-check thresholds

The new opts default to sensible values (60s probe interval, 5min warn / 10min exit, 3 DB failures). If you have specific SLAs, you can pass --health-interval <ms> to adjust the probe cadence. Stall thresholds are not yet CLI-exposed (only the API; CLI flags coming in a follow-up).

To disable self-monitoring entirely (e.g. you have your own external health checker):

gbrain jobs work --health-interval 0 --max-rss 0

6. Verify

gbrain jobs stats              # queue should be flowing normally
gbrain doctor --json | jq '.'  # no critical warnings
ps -o rss= -p $(pgrep -f "gbrain jobs work") | awk '{print $1/1024 " MB"}'

Worker startup log line should now show health-check status:

Minion worker started (queue: default, concurrency: 3, watchdog: 2048MB, health-check: 60s)

If running under supervisor, you'll see the watchdog but NOT the health-check: 60s segment (because GBRAIN_SUPERVISED=1 skips the child's self-monitor).

7. If anything fails

Open an issue at https://github.com/garrytan/gbrain/issues with:

  • Output of gbrain doctor
  • Contents of ~/.gbrain/upgrade-errors.jsonl if it exists
  • Your bare-worker invocation (systemd unit / cron line / Dockerfile snippet)
  • Which step broke
Continue exploring589 Markdown documents in the local repository