docs: capture launchd reload troubleshooting

docs/profile-isolation/checklist.md

@@ -62,6 +62,14 @@ Verify that global system rules, principal-specific profiles, and session-only c
 - [ ] Streaming terminal payloads rewrite managed-download URLs consistently with non-streaming payloads
 - [ ] Any remaining token-level streaming leakage risk is either blocked or explicitly tracked for follow-up
 
+### launchd rollout / restart safety
+- [ ] If a LaunchAgent plist changed, reload the job definition with `launchctl bootout ...plist` + `launchctl bootstrap ...plist` instead of assuming `kickstart -k` will apply the new arguments
+- [ ] After gateway plist changes, `launchctl print gui/$(id -u)/ai.hermes.gateway` shows live arguments that exactly match the edited plist (`gateway run` without stale `--replace`)
+- [ ] After runs-proxy plist or script changes, `launchctl print gui/$(id -u)/ai.hermes.openwebui-runs-proxy` shows the expected program arguments and a fresh PID
+- [ ] Gateway health recovery is given enough wait time after reload (real run observed: healthy on try 13 with a 1s poll loop)
+- [ ] OpenWebUI runs proxy health recovery is given enough wait time after reload (real run observed: healthy on try 2 with a 1s poll loop)
+- [ ] `8642`, `8653`, and `8646` are all re-checked after reload before declaring success
+
 ### OpenWebUI / gateway / email
 - [ ] OpenWebUI resolves the live caller, not a cached operator default
 - [ ] Telegram and LINE use the bound principal, not chat labels alone

docs/profile-isolation/closing-notes.md

@@ -49,6 +49,12 @@ These changes are operational/local and are not part of the git repo:
 - removed `--replace` from the steady-state LaunchAgent command line
 - this change only affects future reloads/restarts; it does not rewrite the currently running process until launchd reloads the job
 
+### Live rollout lesson captured for future debugging
+- `launchctl kickstart -k` restarted the already-loaded gateway job but did **not** apply the edited plist definition
+- the live gateway arguments stayed on `gateway run --replace` until the job definition was reloaded with `launchctl bootout ...plist` + `launchctl bootstrap ...plist`
+- after the controlled plist reload, gateway recovered health on try `13` of a 1-second poll loop and `launchctl print` showed live arguments updated to plain `gateway run`
+- the OpenWebUI runs proxy recovered on try `2` of the same 1-second poll loop after its own `bootout + bootstrap` reload
+
 ## Verification performed
 
 Verified successfully:
@@ -92,8 +98,8 @@ Use the runbook in `restart-recovery-runbook.md`.
 Short version:
 1. ensure tests are green
 2. ensure `ai.hermes.gateway.plist` no longer contains `--replace`
-3. reload gateway first and wait for `8642/health`
+3. if the plist changed, reload gateway with `bootout + bootstrap` (not only `kickstart -k`) and wait for `8642/health`
-4. reload OpenWebUI runs proxy second and wait for `8653/health`
+4. if the proxy plist or launch context changed, reload OpenWebUI runs proxy with `bootout + bootstrap` and wait for `8653/health`
 5. re-check `8646/line/webhook/health`
 6. inspect `launchctl print` to confirm gateway live arguments no longer include `--replace`
 7. smoke-test streaming + managed-download rewrite immediately after restart

docs/profile-isolation/restart-recovery-runbook.md

@@ -39,6 +39,18 @@ Interpretation:
 - but the long-lived LaunchAgent configuration is risky because `--replace` can create self-replacement churn in a supervised service
 - this should be corrected before or as part of the next controlled reload
 
+## Critical launchd rule learned from this rollout
+
+When the LaunchAgent plist itself changes, `launchctl kickstart -k <label>` is **not enough** to guarantee the live job adopts the new `ProgramArguments`.
+
+Use these rules instead:
+- use `kickstart -k` when you only want to restart the already-loaded job definition
+- use `bootout ...plist` + `bootstrap ...plist` when you changed the plist and need launchd to load the new definition
+
+Real evidence from this rollout:
+- after removing `--replace` from `~/Library/LaunchAgents/ai.hermes.gateway.plist`, a plain `kickstart -k gui/$(id -u)/ai.hermes.gateway` still left the live arguments on `gateway run --replace`
+- only after `launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway.plist` plus `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway.plist` did `launchctl print` show live arguments updated to plain `gateway run`
+
 ## Safe rollout sequence
 
 ### Phase A — preflight (do not restart yet)
@@ -77,7 +89,7 @@ Reason:
 ### Phase C — reload in a controlled order
 
 1. Update the LaunchAgent plist if it still contains `--replace`.
-2. Unload / bootstrap or kickstart the gateway service cleanly.
+2. If the plist changed, reload the LaunchAgent definition with `bootout + bootstrap` instead of relying on `kickstart -k`.
 3. Verify the new gateway process command line no longer includes `--replace`.
 4. Re-check `8642/health` before treating OpenWebUI as recovered.
 5. Re-check `8653/health` and run a real OpenWebUI proxy smoke request.
@@ -87,6 +99,48 @@ Completion criteria:
 - runs proxy healthy on `8653`
 - live command line matches expected startup arguments
 
+### Phase C1 — exact commands for plist-changing reloads
+
+Gateway:
+
+```bash
+uid=$(id -u)
+launchctl bootout gui/${uid} ~/Library/LaunchAgents/ai.hermes.gateway.plist
+launchctl bootstrap gui/${uid} ~/Library/LaunchAgents/ai.hermes.gateway.plist
+```
+
+OpenWebUI runs proxy:
+
+```bash
+uid=$(id -u)
+launchctl bootout gui/${uid} ~/Library/LaunchAgents/ai.hermes.openwebui-runs-proxy.plist
+launchctl bootstrap gui/${uid} ~/Library/LaunchAgents/ai.hermes.openwebui-runs-proxy.plist
+```
+
+Recommended health wait loops:
+
+```bash
+for i in {1..30}; do
+  if curl -fsS http://127.0.0.1:8642/health; then
+    echo "gateway healthy on try ${i}"
+    break
+  fi
+  sleep 1
+done
+
+for i in {1..30}; do
+  if curl -fsS http://127.0.0.1:8653/health; then
+    echo "proxy healthy on try ${i}"
+    break
+  fi
+  sleep 1
+done
+```
+
+Observed in this real rollout:
+- gateway recovered on try `13`
+- runs proxy recovered on try `2`
+
 ## Post-restart verification checklist
 
 ### Core health
@@ -179,15 +233,17 @@ Most likely causes:
 - service not actually restarted
 - edited one file, but launchd runs another path
 - stale long-lived proxy/gateway process
+- the job was `kickstart -k` restarted but the changed plist definition was never reloaded, so launchd kept the old arguments
 
 ## Recommended diagnostic order when restart fails
 
 1. Confirm health endpoints and listeners.
 2. Inspect launchd runtime state (`launchctl print`).
-3. Inspect recent gateway stdout/stderr logs with timestamps around the restart.
+3. If the plist changed, confirm whether the job was truly reloaded with `bootout + bootstrap` rather than merely `kickstart -k` restarted.
-4. Verify the exact source file on disk contains the expected fix.
+4. Inspect recent gateway stdout/stderr logs with timestamps around the restart.
-5. If needed, stop auto-restart pressure and run the gateway/proxy once in foreground with the same venv and env vars.
+5. Verify the exact source file on disk contains the expected fix.
-6. Only after the foreground process is healthy, restore launchd supervision.
+6. If needed, stop auto-restart pressure and run the gateway/proxy once in foreground with the same venv and env vars.
+7. Only after the foreground process is healthy, restore launchd supervision.
 
 ## Anti-flap rules