# Gateway / API Server / OpenWebUI Restart & Recovery Runbook

## Purpose

Use this runbook when code changes affecting `gateway`, `api_server`, managed downloads, or the OpenWebUI runs proxy need to go live safely.

This document answers four operator questions:

1. Do these changes need a restart?
2. How do we restart without causing restart loops or half-broken state?
3. What should we verify before and after restart?
4. How do we diagnose and recover when restart fails or keeps flapping?

## Does this work require restart?

Yes for the current workstream.

These changes modify live Python code paths:
- `gateway/platforms/api_server.py`
- `hermes_cli/managed_downloads.py`
- `/Users/hermes/.hermes/scripts/openwebui_runs_proxy.py`

If the gateway and/or the OpenWebUI runs proxy are already running, their existing Python processes will keep serving the old code until they are restarted.

## Live findings from this session

Verified during runtime inspection:
- `http://127.0.0.1:8642/health` returned `200`
- `http://127.0.0.1:8653/health` returned `200`
- `http://127.0.0.1:8646/line/webhook/health` returned `200`
- `launchctl print gui/$(id -u)/ai.hermes.gateway` showed the active LaunchAgent still running:
  - `gateway run --replace`
  - `runs = 41`
  - `forks = 2400`
  - `last terminating signal = Killed: 9`

Interpretation:
- the service is currently healthy
- but the long-lived LaunchAgent configuration is risky because `--replace` can create self-replacement churn in a supervised service
- this should be corrected before or as part of the next controlled reload

## Critical launchd rule learned from this rollout

When the LaunchAgent plist itself changes, `launchctl kickstart -k <label>` is **not enough** to guarantee the live job adopts the new `ProgramArguments`.

Use these rules instead:
- use `kickstart -k` when you only want to restart the already-loaded job definition
- use `bootout ...plist` + `bootstrap ...plist` when you changed the plist and need launchd to load the new definition

Real evidence from this rollout:
- after removing `--replace` from `~/Library/LaunchAgents/ai.hermes.gateway.plist`, a plain `kickstart -k gui/$(id -u)/ai.hermes.gateway` still left the live arguments on `gateway run --replace`
- only after `launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway.plist` plus `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.hermes.gateway.plist` did `launchctl print` show live arguments updated to plain `gateway run`

## Safe rollout sequence

### Phase A — preflight (do not restart yet)

1. Confirm modified code compiles.
2. Run targeted tests for:
   - `tests/gateway/test_api_server.py`
   - `tests/gateway/test_session_api.py`
   - `tests/gateway/test_openwebui_runs_proxy_handoff.py`
3. Confirm current health endpoints before touching launchd:
   - `8642/health`
   - `8653/health`
   - `8646/line/webhook/health`
4. Record current listeners / PIDs:
   - port `8642`
   - port `8653`
5. Confirm whether the gateway LaunchAgent still contains `--replace`.

Completion criteria:
- tests pass
- current runtime is healthy
- we know whether launchd config itself needs correction

### Phase B — fix the launchd steady-state configuration

For a long-lived LaunchAgent, prefer:
- `gateway run`

Avoid in steady-state launchd config:
- `gateway run --replace`

Reason:
- `--replace` is appropriate for explicit operator takeover flows
- but in a persistent `RunAtLoad + KeepAlive` LaunchAgent it increases the risk of self-replacement races, SIGTERM churn, and restart flapping

### Phase C — reload in a controlled order

1. Update the LaunchAgent plist if it still contains `--replace`.
2. If the plist changed, reload the LaunchAgent definition with `bootout + bootstrap` instead of relying on `kickstart -k`.
3. Verify the new gateway process command line no longer includes `--replace`.
4. Re-check `8642/health` before treating OpenWebUI as recovered.
5. Re-check `8653/health` and run a real OpenWebUI proxy smoke request.

Completion criteria:
- gateway healthy on `8642`
- runs proxy healthy on `8653`
- live command line matches expected startup arguments

### Phase C1 — exact commands for plist-changing reloads

Gateway:

```bash
uid=$(id -u)
launchctl bootout gui/${uid} ~/Library/LaunchAgents/ai.hermes.gateway.plist
launchctl bootstrap gui/${uid} ~/Library/LaunchAgents/ai.hermes.gateway.plist
```

OpenWebUI runs proxy:

```bash
uid=$(id -u)
launchctl bootout gui/${uid} ~/Library/LaunchAgents/ai.hermes.openwebui-runs-proxy.plist
launchctl bootstrap gui/${uid} ~/Library/LaunchAgents/ai.hermes.openwebui-runs-proxy.plist
```

Recommended health wait loops:

```bash
for i in {1..30}; do
  if curl -fsS http://127.0.0.1:8642/health; then
    echo "gateway healthy on try ${i}"
    break
  fi
  sleep 1
done

for i in {1..30}; do
  if curl -fsS http://127.0.0.1:8653/health; then
    echo "proxy healthy on try ${i}"
    break
  fi
  sleep 1
done
```

Observed in this real rollout:
- gateway recovered on try `13`
- runs proxy recovered on try `2`

## Post-restart verification checklist

### Core health
- [ ] `8642/health` returns `200`
- [ ] `8653/health` returns `200`
- [ ] `8646/line/webhook/health` returns `200`
- [ ] `lsof` shows one expected listener on `8642`
- [ ] `lsof` shows one expected listener on `8653`

### Launchd / process state
- [ ] `launchctl print gui/$(id -u)/ai.hermes.gateway` shows `gateway run` without `--replace`
- [ ] PID changed only once during the controlled restart
- [ ] no immediate follow-up SIGTERM / Killed:9 churn appears in logs

### Functional checks
- [ ] non-streaming `/v1/responses` still rewrites local artifact paths to managed download URLs
- [ ] streaming `/v1/responses` no longer leaks `/Users/...` or `C:\...` in live deltas
- [ ] `/v1/chat/completions` streaming still emits normal assistant text and keepalives
- [ ] OpenWebUI proxy rewrites local file paths to clickable managed download links
- [ ] `download_link_mode=private/public` still propagates correctly

## Failure modes and how to diagnose them

### 1. Service does not come back at all

Symptoms:
- `8642/health` fails
- no listener on `8642`
- launchd job exists but no stable PID

Check first:
- `gateway.error.log`
- `gateway.log`
- `launchctl print gui/$(id -u)/ai.hermes.gateway`
- command line / working directory / venv path in the plist

Most likely causes:
- syntax/import error in changed Python code
- bad venv path or stale interpreter path
- environment mismatch
- port bind failure

### 2. Service comes back, then keeps restarting

Symptoms:
- `runs` / `forks` climb quickly in `launchctl print`
- repeated `Received SIGTERM`
- `last terminating signal = Killed: 9` or similar

Check first:
- whether the LaunchAgent still contains `--replace`
- whether another helper job is repeatedly kickstarting the service
- whether a watchdog or restart helper is racing the main LaunchAgent

Most likely causes:
- supervised launchd service using `--replace`
- duplicate restart authority (helper + launchd + watchdog)
- repeated crash on startup with KeepAlive immediately reviving it

### 3. Gateway is healthy but OpenWebUI still broken

Symptoms:
- `8642/health` is healthy
- `8653` is unhealthy or returns bad stream behavior

Check first:
- runs proxy process / listener on `8653`
- runs proxy logs
- whether the proxy is still serving old code
- whether it can reach upstream `8642`

Most likely causes:
- proxy not restarted after code change
- upstream base URL mismatch
- missing dependency / import in the proxy process
- old proxy process still running

### 4. Health endpoints are fine but behavior is stale

Symptoms:
- routes work, but old behavior remains
- new download-link mode / streaming rewrite behavior not visible

Check first:
- live PID start times
- whether the file on disk matches the process that launchd is running
- whether only one process instance exists per port

Most likely causes:
- service not actually restarted
- edited one file, but launchd runs another path
- stale long-lived proxy/gateway process
- the job was `kickstart -k` restarted but the changed plist definition was never reloaded, so launchd kept the old arguments

## Recommended diagnostic order when restart fails

1. Confirm health endpoints and listeners.
2. Inspect launchd runtime state (`launchctl print`).
3. If the plist changed, confirm whether the job was truly reloaded with `bootout + bootstrap` rather than merely `kickstart -k` restarted.
4. Inspect recent gateway stdout/stderr logs with timestamps around the restart.
5. Verify the exact source file on disk contains the expected fix.
6. If needed, stop auto-restart pressure and run the gateway/proxy once in foreground with the same venv and env vars.
7. Only after the foreground process is healthy, restore launchd supervision.

## Anti-flap rules

- Do not combine multiple independent restart authorities unless one is clearly primary.
- Do not leave `--replace` inside a steady-state launchd job.
- Do not restart blindly when health endpoints are already good; first confirm whether the issue is stale behavior vs runtime outage.
- Do not declare success from static code inspection alone; always verify the live listener and health endpoint.

## Rollback rule

If restart introduces instability:
1. stop the restart loop source
2. revert to the last known-good commit / script version
3. start the service once in foreground or via a controlled launchd reload
4. verify health and behavior before restoring automatic supervision

## Current recommendation for this environment

Before the next production restart of the gateway:
- remove `--replace` from `~/Library/LaunchAgents/ai.hermes.gateway.plist`
- keep the gateway in `RunAtLoad + KeepAlive`, but with plain `gateway run`
- restart in a controlled sequence
- verify both `8642` and `8653`
- smoke-test streaming and managed-download rewriting immediately after restart
