diff --git a/docs/profile-isolation/checklist.md b/docs/profile-isolation/checklist.md index 46c1d6bed..9b0fc80c1 100644 --- a/docs/profile-isolation/checklist.md +++ b/docs/profile-isolation/checklist.md @@ -62,6 +62,14 @@ Verify that global system rules, principal-specific profiles, and session-only c - [ ] Streaming terminal payloads rewrite managed-download URLs consistently with non-streaming payloads - [ ] Any remaining token-level streaming leakage risk is either blocked or explicitly tracked for follow-up +### launchd rollout / restart safety +- [ ] If a LaunchAgent plist changed, reload the job definition with `launchctl bootout ...plist` + `launchctl bootstrap ...plist` instead of assuming `kickstart -k` will apply the new arguments +- [ ] After gateway plist changes, `launchctl print gui/$(id -u)/ai.hermes.gateway` shows live arguments that exactly match the edited plist (`gateway run` without stale `--replace`) +- [ ] After runs-proxy plist or script changes, `launchctl print gui/$(id -u)/ai.hermes.openwebui-runs-proxy` shows the expected program arguments and a fresh PID +- [ ] Gateway health recovery is given enough wait time after reload (real run observed: healthy on try 13 with a 1s poll loop) +- [ ] OpenWebUI runs proxy health recovery is given enough wait time after reload (real run observed: healthy on try 2 with a 1s poll loop) +- [ ] `8642`, `8653`, and `8646` are all re-checked after reload before declaring success + ### OpenWebUI / gateway / email - [ ] OpenWebUI resolves the live caller, not a cached operator default - [ ] Telegram and LINE use the bound principal, not chat labels alone diff --git a/docs/profile-isolation/closing-notes.md b/docs/profile-isolation/closing-notes.md index b3e23105b..378975b3e 100644 --- a/docs/profile-isolation/closing-notes.md +++ b/docs/profile-isolation/closing-notes.md @@ -49,6 +49,12 @@ These changes are operational/local and are not part of the git repo: - removed `--replace` from the steady-state LaunchAgent command line - this change only affects future reloads/restarts; it does not rewrite the currently running process until launchd reloads the job +### Live rollout lesson captured for future debugging +- `launchctl kickstart -k` restarted the already-loaded gateway job but did **not** apply the edited plist definition +- the live gateway arguments stayed on `gateway run --replace` until the job definition was reloaded with `launchctl bootout ...plist` + `launchctl bootstrap ...plist` +- after the controlled plist reload, gateway recovered health on try `13` of a 1-second poll loop and `launchctl print` showed live arguments updated to plain `gateway run` +- the OpenWebUI runs proxy recovered on try `2` of the same 1-second poll loop after its own `bootout + bootstrap` reload + ## Verification performed Verified successfully: @@ -92,8 +98,8 @@ Use the runbook in `restart-recovery-runbook.md`. Short version: 1. ensure tests are green 2. ensure `ai.hermes.gateway.plist` no longer contains `--replace` -3. reload gateway first and wait for `8642/health` -4. reload OpenWebUI runs proxy second and wait for `8653/health` +3. if the plist changed, reload gateway with `bootout + bootstrap` (not only `kickstart -k`) and wait for `8642/health` +4. if the proxy plist or launch context changed, reload OpenWebUI runs proxy with `bootout + bootstrap` and wait for `8653/health` 5. re-check `8646/line/webhook/health` 6. inspect `launchctl print` to confirm gateway live arguments no longer include `--replace` 7. smoke-test streaming + managed-download rewrite immediately after restart diff --git a/docs/profile-isolation/restart-recovery-runbook.md b/docs/profile-isolation/restart-recovery-runbook.md index 0f47486d0..75a88508f 100644 --- a/docs/profile-isolation/restart-recovery-runbook.md +++ b/docs/profile-isolation/restart-recovery-runbook.md @@ -39,6 +39,18 @@ Interpretation: - but the long-lived LaunchAgent configuration is risky because `--replace` can create self-replacement churn in a supervised service - this should be corrected before or as part of the next controlled reload +## Critical launchd rule learned from this rollout + +When the LaunchAgent plist itself changes, `launchctl kickstart -k