An aggregator whose pooled credentials are all exhausted/dead still counted as
an authenticated provider during no-provider /model resolution. It then won the
model-name match, was set as the sticky session provider, and poisoned every
later switch with "empty API key" errors while still routing through the dead
aggregator.
list_authenticated_providers now requires a pool to have at least one available
entry (has_available, not has_credentials / bare key presence) at all three
credential-pool gates. Simple token-style entries that don't parse into
exhaustion-tracked entries keep the prior behaviour, so providers whose creds
live only in the auth-store credential_pool still appear.
Fixes#45759
ShellFileOperations builds bash commands (wc/head/sed/cat/tee ...) with the
target path as an argument. On a Windows/Git-Bash host a native `C:\...` path
has its backslashes eaten by bash (and mangled by the msys runtime even when
single-quoted) — the "Directory \drivers\etc does not exist; exiting — update
your msys package" class of failures. Rewrite a native drive path to forward
slashes in `_escape_shell_arg`, reusing the env layer's `_windows_to_msys_path`.
Both `C:/...` and `/c/...` fix the backslash bug (the MSYS coreutils resolve
either via the POSIX API). We emit `/c/...` purely for consistency: it's the
same form `_windows_to_msys_path` already produces for the terminal `cd`
(LocalEnvironment._quote_cwd_for_cd), so shell file ops and `cd` share one
helper and one path form.
Scoped from #55481, which also patched BaseEnvironment._quote_cwd_for_cd — but
LocalEnvironment already overrides that through `_windows_to_msys_path`, so on a
real Windows host the base branch never ran (the cwd is already `/c/...`).
Co-authored-by: konsisumer <der@konsi.org>
Add shared translators in hermes_constants (Windows drive → /mnt, `\\wsl(.localhost|$)\`
UNC → POSIX, gated on is_wsl) and apply them at the gateway session-cwd boundary
so a Windows-host UI can hand the WSL backend a path it can actually chdir into.
De-dups the ACP adapter's private `_win_path_to_wsl` onto the shared helper and
extends it to the UNC spelling.
Co-authored-by: Rage Lopez <VrtxOmega@pm.me>
A session's coarse workspace identity: its git repo root when known, else its
cwd (branch excluded, so switching branches doesn't fragment history). Pure
helper over fields sessions already record — no new columns, no git shelling.
Co-authored-by: Cary Palmer <palmer@dugoutfantasy.com>
Addresses @teknium1's review of #61950:
- The desktop live overlay (workspace-groups.ts) matched cwd membership
case-sensitively, so a fresh mixed-case/separator Windows session missed
its explicit/auto project until the next backend tree refresh. Mirror the
backend identity (isWindowsPath/comparisonSegments/pathKey) in isPathUnder,
liveSessionProjectId, and overlayRepoLanes lane matching. Comparison-only —
emitted ids/labels keep their spelling. POSIX stays case-sensitive.
- Backend _is_windows_path missed root-relative `\wsl.localhost\...` (single
leading backslash), leaving that historical spelling case-sensitive. Classify
any backslash-rooted path as Windows.
Tests: WSL-spelling collapse + explicit-project precedence (project_tree),
Windows/WSL live-overlay membership + POSIX case-sensitivity (workspace-groups).
GatewayRunner._prepare_inbound_message_text's "@" context-reference
block read self._model / self._base_url to resolve the model for
get_model_context_length_async. GatewayRunner never sets either
attribute (copy-pasted from HermesCLI in da44c196b, which does carry
self.model/self.base_url). Every message containing "@" raised
AttributeError inside the try block, silently swallowed by the
surrounding except Exception at debug level, so
preprocess_context_references_async never ran and @file:/@folder:/@diff/
etc. references passed through to the model unexpanded.
Fix: resolve model/provider/base_url via
self._resolve_session_agent_runtime(source=, session_key=,
user_config=), the same session-aware resolution the hygiene
compression block already uses a few hundred lines later in this file.
Also raise the swallow log from debug to warning (with exc_info at
debug) so a future regression here is visible instead of silent.
Third correction, and the load-bearing one. The previous commit put the
"did compaction clear the threshold?" verdict inside should_compress(). But
conversation_loop calls should_compress() TWICE per turn with two different
measures (turn_context.py / conversation_loop.py:1033 and :4789):
* pre-API : request_pressure_tokens -- a rough estimate that can dip BELOW
the threshold
* post-API: real prompt tokens -- which stay above it
So the rough reading reset the strike every turn and the loop never stopped.
Reproduced: 8 compactions in 8 turns under the real two-call pattern, even
with the previous fix applied. (My earlier repro only called should_compress()
once per turn, which is why it looked contained.)
Move the verdict to update_from_response(), the one place that sees the
provider's real prompt_tokens for the just-compacted conversation, guarded by
the existing awaiting_real_usage_after_compression flag so it fires exactly
once per compaction. Real-vs-real: it cannot be fooled by a rough sub-threshold
reading, and (from the previous commit's lesson) never subtracts an estimate
from a real count. should_compress() goes back to a plain threshold test plus
the pre-existing cooldown and anti-thrash guards.
New test test_rough_preflight_reading_does_not_reopen_the_loop drives the real
two-call-per-turn pattern and fails on the prior should_compress()-based commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the previous commit, whose futility check was unsound:
incompressible_floor = max(0, display_tokens - pre_estimate)
`display_tokens` is the provider's real prompt count; `pre_estimate` is
`estimate_messages_tokens_rough(messages)`. Subtracting an estimate from a real
count folds the tokenizer skew into "floor" and misreads it as incompressible
overhead. With a 1.6x skew on a 200K window (threshold 150K, true floor 30K):
rough_msgs=253,804 real_prompt=436,086
computed floor = 182,282 <-- mostly skew; exceeds the threshold
after compaction: 401 -> 77 msgs, real prompt = 106,361 (CLEARS 150,000)
verdict: ineffective_count = 1 <-- false positive
Two such passes would permanently disable compaction on a healthy session --
worse than the loop this PR set out to fix.
Move the check into should_compress(), where both sides of the comparison are
the caller's own token count:
* prompt under the threshold -> not thrashing; reset the counter
* a compaction just ran and we are STILL over -> one strike
Real-vs-real, so tokenizer skew can never be mistaken for a floor, and nothing
subtracts an estimate from a real count. compress() now only ever increments the
counter; the reset lives with the one measure the trigger uses.
Adds `test_no_false_positive_under_tokenizer_skew` (the case above) and
`test_counter_resets_once_the_prompt_fits_again` (one failed pass must not
disable compaction forever). Against upstream, 5 of the 7 cases fail; the 2 that
pass are the regression guards, which is the intended shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`should_compress()` documents anti-thrashing protection ("if the last two
compressions each saved less than 10%, skip compression to avoid infinite
loops"). In practice `_ineffective_compression_count` reset on every pass,
so the guard was dead code and a mis-sized context window presented as a
hung CLI instead of a warning.
Two defects:
1. Mixed measurement bases. Effectiveness was
`(current_tokens - estimate(compressed)) / current_tokens`, where
`current_tokens` is the provider's FULL prompt (system prompt + tool
schemas + messages) but `estimate(compressed)` covers messages only.
Every compaction therefore reported ~96% savings and reset the counter.
Savings is now scored messages-vs-messages.
2. Message shrinkage is the wrong yardstick. `should_compress()` trips on
the full prompt, but compaction can only shrink messages -- the system
prompt and tool schemas are an incompressible floor. When that floor
alone meets the threshold, each pass shrinks messages by a healthy
margin, legitimately resets the counter, and still leaves the prompt
over the line; the next turn compacts again, forever. Observed in the
wild: 45+ consecutive compactions, one auxiliary-LLM call each, zero
progress. Effectiveness is now scored against the goal -- did the
projected prompt get under the threshold? -- and a futile pass warns
with the numbers that prove it.
Also record an ineffective pass on the "only N messages (need > M)" early
return, which previously returned the transcript unchanged without moving
any anti-thrash state -- the same class of bug the neighbouring
"no compressable window" branch was already fixed for.
Tests: 4 of the 5 new cases fail on main and pass here; the fifth pins
that effective compaction still resets the counter (121 -> 15 messages,
88.9% savings).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Preserve deletability, route identity, stored costs, aggregate reconciliation,
and zero-usage Codex route accounting on top of the salvaged per-model usage
work.
The `sessions` table records only the initial (model, billing_provider)
for a session, so when a user switches models mid-session (via `/model`
or programmatically) every token — including the switched model's — is
attributed to the first model. Insights/billing reports then hide the
cost of the new model entirely (e.g. a session that started on deepseek
and switched to opus shows $0 for opus).
Add a `session_model_usage` table keyed (session_id, model,
billing_provider) that accumulates each per-API-call delta under the
model active at the time of the call. `update_token_counts()` is the
single chokepoint every per-call delta flows through (CLI, gateway,
cron, delegated, codex), so recording there captures accurate
attribution on every platform. Only the incremental path records — the
gateway's `absolute=True` summary overwrite is skipped to avoid
double-counting cumulative totals that can't be split per model. When a
call omits the model, it falls back to the session's recorded model,
matching the existing COALESCE-from-session summary behaviour.
Insights `_compute_model_breakdown` now aggregates tokens and cost from
`session_model_usage`, so a switched session splits correctly across
models, with a defensive fallback to the per-session aggregate for any
session lacking usage rows. A v17 migration backfills one usage row per
existing token-bearing session from its aggregate totals (idempotent via
INSERT OR IGNORE), validated lossless against a 1.3 GB production DB.
Tests: per-model recording, mid-session split, model fallback, absolute
no-double-count, v17 backfill, and an insights-level switch breakdown.
Fixes#51607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bundle Fireworks AI as a first-class BYOK provider across the CLI, web/TUI,
and desktop onboarding.
- New model-provider plugin with attribution headers (HTTP-Referer / X-Title)
so Fireworks can attribute Hermes traffic; PAYG-safe default aux + fallback
models (accounts/fireworks/models/...), IDs tracking fw-ai/fireconnect.
- Registered in CANONICAL_PROVIDERS so it appears in the CLI/web/TUI pickers.
- Alias wiring (fireworks-ai, fw) into both CLI resolvers.
- First-class wiring: OPTIONAL_ENV_VARS, HERMES_OVERLAYS (FIREWORKS_BASE_URL
override), doctor env hints. Live catalog + model_metadata are auto-derived.
- doctor: treat Fireworks' native slash-form IDs (accounts/fireworks/...) as
valid, not aggregator vendor prefixes, so it no longer tells Fireworks users
to switch to openrouter or drop the prefix.
- picker: plugin providers with no static curated list now lead with their
profile fallback_models, so the default is an agentic chat model instead of
whatever the live catalog returns first (Fireworks listed an image model,
flux-*, ahead of its chat models).
- Desktop onboarding: Fireworks as a RECOMMENDED hero card with the official
Fireworks logomark and a brand-purple badge, routing to the BYOK key form;
i18n in en/ja/zh/zh-hant.
- Tests: profile contract, first-class wiring (both resolvers, overlay, config,
doctor incl. the slash-form regression, aux headers, credentials), discovery
spot-check, and a live smoke test driven through the Hermes runtime.
Fire Pass (fpk_) support is coming soon; the future wiring is kept as a
commented-out scaffold in the plugin.
`hermes -t web chat` silently dropped the toolset filter (and the same
hold true for `-m`, `--provider`, `--tui`, `--dev` placed before
`chat`). Reported in #28780 for `-t/--toolsets`; the others are sibling
failures with the same root cause.
Root cause: the chat subparser re-declared these flags with `default=None`
(or `default=False` for store_true) on top of the matching top-level
parser flags. When argparse dispatches into the subparser it shares the
namespace via `dest`, so the subparser's default overwrites whatever the
top-level parser parsed before the subcommand. `-s/--skills`, `-r/-c/-w`,
`--yolo`, and `--pass-session-id` already use `default=argparse.SUPPRESS`
for exactly this reason — the chat-subparser action becomes a no-op
unless the user explicitly passes the flag after `chat`, and the parent
value survives.
Reproduction (origin/main, before fix):
>>> parser.parse_known_args(["-t", "web", "chat"]).toolsets
None
>>> parser.parse_known_args(["chat", "-t", "web"]).toolsets
'web'
After fix:
>>> parser.parse_known_args(["-t", "web", "chat"]).toolsets
'web'
>>> parser.parse_known_args(["chat", "-t", "web"]).toolsets
'web'
Sibling flags fixed in the same commit because they share the exact same
argparse pattern bug — verified via a new contract test that scans every
chat-subparser action whose `dest` is also on the top-level parser and
asserts `default is argparse.SUPPRESS`. The test fails on origin/main
listing all five offenders and passes after this fix.
Test additions in tests/hermes_cli/test_argparse_flag_propagation.py:
- TestChatSubparserInheritedValueFlags exercising real `_parser` build
(not the hand-rolled replica) so it catches future drift.
- Parametrized before-chat / after-chat cases for `-t`, `--toolsets`,
`-m`, `--model`, `--provider`.
- Negative case: passing none of the flags leaves attrs at the top-level
parser's `None` default (SUPPRESS does not remove existing attrs).
- Combined case: all three value flags before `chat` simultaneously.
- store_true cases for `--tui` / `--dev`.
- Contract test asserting every shared-`dest` flag on chat uses SUPPRESS.
Fixes#28780.
Review feedback: a same-host redirect to a different port can land on a
different service, which must not inherit the provider API key. Compare
(scheme, hostname, effective port) — with 80/443 defaults — instead of
hostname alone, and add a two-server regression test for the
same-host/different-port case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fetch_models() sends Authorization: Bearer <api_key> plus any
default_headers (x-api-key etc.) via urllib.request.urlopen, and
urllib's redirect handler forwards every header when following a
3xx — including to a different host. A catalog endpoint (or a
compromised/misconfigured proxy in front of it) answering with a
redirect to another origin therefore received the provider API key.
Install an HTTPRedirectHandler that drops authorization, x-api-key,
api-key, x-goog-api-key and cookie when the redirect target hostname
differs from the original request, mirroring the pattern already used
in skills/creative/comfyui/scripts/_common.py. Same-host redirects
keep credentials so legitimate path-level redirects still work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_CodexCompletionsAdapter (agent/auxiliary_client.py) is a second,
independent producer of Codex Responses input — used by auxiliary
calls (context compression, flush_memories, MoA aggregation,
session_search) that route through CodexAuxiliaryClient instead of
the main agent's ResponsesApiTransport.build_kwargs. It calls
_chat_messages_to_responses_input() directly without is_github_responses,
so the previous commit's fix didn't cover it: an auxiliary call made
against a Copilot-backed session could still replay a connection-scoped
codex_message_items id and hit the same HTTP 401.
Detect the Copilot host from the adapter's own client.base_url (same
check the adapter already does further down for prompt_cache_key
opt-out) and pass is_github_responses through, closing the gap.
Still #32716.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copilot (api.githubcopilot.com/responses) binds replayed assistant
codex_message_items ids to a specific backend "connection". Credential-
pool rotation, a gateway restart, or routine load-balancer churn between
turns all invalidate that binding, and Copilot rejects the stale id with
HTTP 401 "input item ID does not belong to this connection" — even for
short ids well under the #27038 64-char length cap, since this is a
connection-scope problem, not a length problem. Once a session captures
one of these ids it is persisted and replayed forever, permanently
bricking the session.
Thread an is_github_responses flag from build_kwargs/convert_messages
into _chat_messages_to_responses_input and drop the id unconditionally
on that path, mirroring how reasoning items already strip id on replay.
phase/status/content are still replayed so cache-relevant signal isn't
lost — only the connection-scoped id is unsafe to reuse.
Written to apply independently of the #27038 length-cap fix so the two
PRs don't block each other; they touch adjacent conditions in the same
block and merge cleanly in either order.
Fixes#32716
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Keep empty-tail recovery scoped to the current stream segment and bound fallback flood retries. Preserve Telegram's server retry hint without blocking final delivery through a long cooldown.
Codex assigns assistant message items server-side ids that can run
400+ chars (base64 encrypted blobs), but the Responses API caps
input[].id at 64 chars and rejects the whole request with a
non-retryable HTTP 400. Once a session captures one of these long
ids, every subsequent turn replays it and 400s forever, since the
history persists it in codex_message_items.
Add a 64-char length guard at both replay sites — the history-to-
input converter and the final preflight gate — so oversized ids are
dropped while short ids (msg_...) are kept for prefix-cache hits.
Mirrors the existing pattern for reasoning items, which already
strip their id before replay because store=False means the API
can't resolve ids server-side anyway.
Fixes#27038
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
finalize passed conversation_history=history aliasing the snapshot so flush
skipped every message and wrote nothing. now flush _session_messages via
marker dedup like gateway shutdown. add real db e2e tests.
force_close_tcp_sockets stayed shutdown-only after #29507 to avoid
cross-thread FD recycle. That left CLOSED sockets unreclaimed when
httpx.close() skipped already-shutdown sockets under long-lived
gateways (~1 CLOSED fd / 6 min via proxy).
Add release_fds= for the owning-thread dispose path only; abort still
defaults to shutdown-only.
get_due_jobs()'s one-shot stale-entry recovery (#38758) treated an
expired run_claim (#59229) as proof the claiming tick died, but a run
stalled on network I/O — or a laptop asleep mid-run — legitimately
outlives the TTL while very much alive. The recovery then deleted the
job record mid-flight: list showed the job gone, and when the run
finished mark_job_run() found nothing to update, so last_run_at /
last_status / last_delivery_error were never recorded.
Two guards, per the liveness signals available:
- Same process (the common single-gateway case): before removing a
dispatch-limit-reached one-shot, consult the scheduler's running set
via a lazy import; if the job is still running here it is slow, not
stale — keep the entry.
- Cross process: run_job's monitor loop now refreshes run_claim.at
every 60s while the run is alive (including under
HERMES_CRON_TIMEOUT=0, which previously blocked without polling), so
an expired claim really does mean the owner died and the TTL stays a
dead-owner detector.
Fixes#62002
When web_search results are passed directly to web_extract, the URLs
field contains dict objects (e.g., {"url": "...", "title": "..."})
rather than plain URL strings. Two code paths assumed URLs were always
strings and crashed:
- agent/display.py get_cute_tool_message for web_extract: tried to call
url.replace() on a dict, causing AttributeError
- tools/web_tools.py web_extract_tool loop: tried regex search on a dict,
causing TypeError
Both now extract the URL string from dict objects (url or href field) or
fall back to empty string, preserving the cosmetic display and allowing
the tool to process the URLs correctly.
Fixes#61693
Add a token-free, curated affection matcher (agent/reactions.py) — the single
source of truth for detecting user "vibes" (ily / <3 / good bot / heart emoji).
No model call, no tokens. Generalized to return a reaction *kind* so future
reactions can ride the same signal.
Wire an opt-in AIAgent.reaction_callback that fires from build_turn_context on
the incoming user message. It never touches the conversation (cache-safe) and
never fatal — a purely cosmetic side-beat each host can consume.
Dashboard Chat is an xterm mirror of a TUI inside the gateway, so
server-side clipboard.paste never sees the browser clipboard. Upload
pasted/dropped images to the profile's images/ dir (same place
clipboard.paste / image.attach use), then drive /image over the PTY.
Uses a dedicated /api/chat/image-upload endpoint (magic-byte check,
25MB cap, profile scope) instead of relative managed-files uploads that
400 on local dashboards without a locked root. Ctrl/Cmd+Shift+V also
tries clipboard.read() for images before falling back to text, since
preventDefault on that chord suppresses the DOM paste event.
Salvages #57912 (client composition + /image PTY drive) and folds in
#48563's upload endpoint + drop path.
Co-authored-by: bird <6666242+bird@users.noreply.github.com>
Co-authored-by: tt-a1i <53142663+tt-a1i@users.noreply.github.com>
The second lock block in get_or_create_session held self._lock during six
blocking operations on every inbound message: _is_session_ended_in_db
(SQLite SELECT), _should_reset (callback), _save (SQLite write + JSON write
+ os.fsync), and _recover_session_from_db (SQLite SELECT + UPDATE).
A code comment at line 1607 claimed 'SQLite calls are made outside the
lock' -- true only for _compression_tip_for_session_id, which was moved
out in a prior fix. The remaining I/O was never addressed.
Restructure into a four-phase lock/no-lock split that mirrors the pattern
already established at the bottom of the function:
Phase 1 (lock) -- read entry + session_id
Phase 1b (no lock) -- stale check + reset policy
Phase 2 (lock) -- apply decisions to _entries, capture snapshot + flags
Phase 3 (no lock) -- recovery DB query, _save from snapshot, end/create
_save_entries(snapshot) replaces _save() to avoid dict-mutation races when
called outside the lock. _query_recoverable_session splits the DB I/O out
of _recover_session_from_db so only the _entries assignment needs the lock.
Three early returns inside the lock block are eliminated in favour of a
unified save + return path.
The sync _session_has_compression_in_flight sat on the message hot path
and blocked the event loop twice: under session_store._lock during
_ensure_loaded_locked (JSON read) and via db.get_compression_lock_holder
(SQLite SELECT). Async-ify the method and offload both sources via
asyncio.to_thread; await the call site in _handle_active_session_busy_message.
Git Bash hands file tools paths like /c/Users/... which Path() on native
Windows treats as relative \\c\\Users\\... under the process cwd. Reuse
local._msys_to_windows_path (extended for /cygdrive and /mnt drive forms)
in _resolve_path_for_task / _resolve_base_dir so read/write/search land on
the real drive. Container/WSL Linux paths are left untouched.
Salvages #50488 (drops unrelated desktop artifact commit); tests adapted
from #46995.
Co-authored-by: Jeff Watts <186512915+lEWFkRAD@users.noreply.github.com>
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
teknium1
AIalliAI
Brooklyn Nicholson
Esthon Jr
Adolanium
kshitijk4poor
TheTom
Igor Ganapolsky
Jan Hermes Integration
Thomas Connally
Alex Jestin Taylor
konsisumer
briandevans
dsad
solyanviktor-star
joaomarcos
Gille
PRATHAMESH75
HexLab98
embwl0x
Wilson Kinyua
giggling-ginger
liuhao1024
infinitycrew39
Sami Rusani
luyifan
kenyonxu