Only restore held verification text when the loop genuinely ends through budget exhaustion. Preserve later interrupts and failures, keep generated-summary fragment explanations, and add regression coverage for both contracts.
Treat intent-ack continuation text as non-final so last-turn exhaustion requests a real summary instead of surfacing a premature promise. Keep iteration-limit fallback text free of the abnormal-fragment explainer.
Name the continuation fallback for its actual verification-only provenance so unrelated continuation paths cannot accidentally inherit its cron-delivery semantics.
Track held-back verification responses explicitly so budget exhaustion returns the composed report without a second model call. Keep unrelated error and recovery exit reasons intact, preserve Kanban timeout accounting, and cover the real run_conversation paths.
Sibling sites of the salvaged #55997 fix, all reading user-editable
config values through .get(key, '').method(): MoA slot provider/model
labels, gateway quick-command alias targets (2 sites), gateway.proxy_url,
and gateway.relay_url. Regression tests for the contributor's two sites
plus the MoA labels.
Without this UA tag the Feishu server does not push group @mention events
over the WebSocket transport. The "channel" tag tells the server to use
the Channel protocol which enables group-message routing in addition to P2P
direct messages.
Root cause: FeishuWSClient was created without any UA signaling tag, so the
server defaulted to the basic DM-only push mode. Group @mention events were
silently dropped before reaching Hermes.
Fixes https://github.com/NousResearch/hermes-agent/issues/50656
Also adds a regression test verifying the UA tag is present in the
FeishuWSClient constructor call.
`_load_web_config()` is typed `-> dict` but returned `load_config().get("web",
{})`, which is `None` when the config has a present-but-null `web:` section
(YAML `web:` with no body). Every caller then does
`_load_web_config().get(...)` and raises `AttributeError: 'NoneType' object
has no attribute 'get'` — this hits `_get_backend`, `check_web_api_key`, and
the extract-char-limit reader.
Separately, `check_web_api_key()` read the backend as
`.get("backend", "").lower()`; a null `web.backend` value yields `None` (the
`""` default only applies when the key is absent), so `None.lower()` raised.
`check_web_api_key` is the `check_fn` gate for `web_search`/`web_extract`, so
this surfaced as an exception during tool-availability checking.
- Make `_load_web_config()` honor its `-> dict` contract (`... or {}`), fixing
the null-`web:`-section crash at every call site.
- Guard the backend value in `check_web_api_key` with `or ""`, mirroring the
existing guard in `_get_backend`.
Adds regression tests for both the null-backend-value and null-web-section
cases.
* test: deflake CI and dev-machine flaky tests in bulk
Fixes ten distinct flake sources found by mining recent CI failures and
running the full suite on a dev machine with real user state:
CI-observed races:
- tests/conftest.py live-system guard: allow signal 0 (pure liveness
probe) through _guarded_kill/_guarded_killpg. psutil.pid_exists()
probes a just-killed grandchild reparented to init; the subtree check
fails for it and the guard RuntimeError'd
test_entire_tree_is_sigkilled_not_just_parent intermittently on
unrelated PRs.
Hermeticity flakes (fail on dev machines with real state, pass on CI):
- agent/coding_context.py: _marker_root() now skips the shared temp
root (tempfile.gettempdir()) like it skips $HOME — a stray
/tmp/package.json flipped every tmp_path test into the coding
posture (9 failures in test_coding_context.py).
- test_agent_guardrails.py: pin MAX_CONCURRENT_CHILDREN=3 via autouse
monkeypatch instead of freezing the user's real config value at
import time (import-time vs call-time config mismatch).
- test_web_tools_config.py: TestCheckWebApiKey now neutralizes the
ddgs package probe and registry providers — the optional ddgs
package in a dev venv lit up the fallback backend.
- test_credential_pool.py: block claude_code/hermes-oauth credential
autodiscovery in the two pool-merge tests that assert exact id
lists (a real ~/.claude/.credentials.json seeded an extra entry).
- test_modal_sandbox_fixes.py: clear _permanent_approved /
_session_approved — the user's real command_allowlist silently
approved the guard-escalation commands under test.
- test_setup_irc.py: stub prompt_checklist to select only the IRC row;
the non-TTY cancel fallback re-ran the real configured platforms'
interactive setup_fn, which hit input() under captured stdin.
- test_doctor.py: TestGitHubTokenCheck now patches the module-level
HERMES_HOME constant (the file's established pattern) instead of
only setenv — doctor was running PRAGMA integrity_check against the
real multi-GB state.db and blowing the 300s per-file budget.
Latent atexit-duplication (same _enter_buffered_busy class as #34217):
- test_undo_command.py: drop importlib.reload(tui_gateway.server) in
fixture teardown; reload re-registers the module's atexit hooks.
- test_session_platform_resolution.py: drop per-test reload of
tui_gateway.server; every resolver reads env at call time.
* test: sentinel model value in ignore-user-config fallback assertion
With HERMES_IGNORE_USER_CONFIG=1, load_cli_config() falls back to the
repo-root cli-config.yaml (untracked, gitignored). On a dev machine that
file can legitimately set the same popular model the test hardcoded
(anthropic/claude-sonnet-4.6), flipping the != assertion locally while
CI (no cli-config.yaml) stayed green. Use an impossible sentinel model
name instead.
skills: null crashed with AttributeError, and a bare scalar
disabled: my-skill was split into a set of characters. Both now
normalize the same way agent.skill_utils._normalize_string_set does:
null -> empty set, scalar -> single-item set. Non-dict skills
sections are ignored.
Closes#13026.
Addresses Copilot review on #61348: the HTML-escaped role, while safe from
injection (quotes are escaped), still contains whitespace when a crafted role
is supplied, which splits the class attribute into several unintended CSS
classes. Keep the escaped role for the display badge, and reduce the raw role
to a single safe CSS token (alnum/-/_) for the class name. Real roles
(user/assistant/system/tool) are unchanged, so the existing .message-<role>
rules still match.
The gateway reconnect watcher forwards is_reconnect=True to every
adapter.connect() call on every retry. Adapters whose signature omits
the kwarg raise TypeError at every reconnect attempt and stay silently
disconnected — the exact bug that shipped for QQAdapter and only
surfaced after messages stopped flowing on the QQ channel for hours.
This test statically parses every adapter.py under gateway/platforms/
and plugins/platforms/ (via AST, so third-party SDKs like slack_sdk,
matrix-nio, aiohttp, telegram, etc. are NOT required in the test env)
and asserts every *Adapter class with an async connect() accepts
is_reconnect — either as a keyword-only argument or absorbed by
**kwargs.
Also fixes plugins/platforms/wecom/callback_adapter.py:WecomCallbackAdapter,
which the new test caught as a second offender. Same class of bug: bare
'async def connect(self)' signature would die on the first reconnect.
Companion to #59429 (which fixed the original QQAdapter offender).
The base adapter's signature was updated to include
, which the reconnect watcher passes as
during reconnection. All other platform adapters were
updated, but QQAdapter was missed, causing:
TypeError: QQAdapter.connect() got an unexpected keyword argument 'is_reconnect'
This leads to an infinite retry loop since every reconnect attempt fails
immediately with the same TypeError.
Fix: add to QQAdapter.connect()'s signature.
QQBot has no server-side update queue, so the flag is accepted only for
interface conformance.
Test: new test_connect_accepts_is_reconnect_param verifies both
adapter.connect() and adapter.connect(is_reconnect=True) succeed without
raising.
The streaming fallback path read yaml_cfg.get("gateway", {}).get("streaming") when top-level streaming was absent or malformed. If a user accidentally set gateway to a scalar value, config loading crashed with AttributeError instead of ignoring the malformed block and using defaults.
Read the gateway block once, verify it is a mapping before accessing nested streaming, and keep the existing gateway.platforms fallback using the same checked value.
Adds a regression test for config.yaml containing gateway: disabled.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
GatewayConfig.from_dict(), PlatformConfig.from_dict(), SessionResetPolicy.from_dict(), and StreamingConfig.from_dict() assumed their input sections were mappings. A malformed scalar from legacy gateway.json or an internal caller could crash config loading with AttributeError before env overrides/defaults had a chance to recover.
Coerce non-mapping sections to empty dicts, skip malformed platform entries, and keep valid sibling platform configs loading normally.
Tests cover scalar platform blocks, scalar nested reset/streaming sections, and malformed PlatformConfig home_channel/extra values.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Sibling site of the load_cli_config fix (#58277): _deep_merge treated a
YAML-null section (terminal: with no value) as an override, replacing
the entire DEFAULT_CONFIG dict for that section with None. Every
downstream consumer expecting a mapping was a latent crash, and default
sub-keys were silently lost. A None override of a dict default is now
ignored, matching the CLI loader's behavior. Scalar-null overrides are
unchanged.
Three tests for the #61099 salvage: OpenRouter attribution headers
present after switching to openrouter.ai, Kimi User-Agent sentinel
present after switching to api.kimi.com, and stale headers cleared
when switching to a provider with no URL-specific headers.
2/3 fail on unpatched main (DID NOT ATTACH), confirming the bug.
Follow-ups for salvaged PR #43819: the registry key was
str(Path(db_path).expanduser()) — a symlinked or relative path to the
same DB file got its own connection, silently reintroducing the exact
multi-writer contention the registry prevents. Key on Path.resolve()
(OSError-tolerant fallback). Adds a symlink regression test and the
AUTHOR_MAP entry for adambiggs.
Every MemoryStore instance opened its own SQLite connection guarded by
its own RLock. Several providers coexist in one process (the main agent
plus every delegate_task subagent), so instances pointing at the same
memory_store.db raced as independent WAL writers. Combined with writes
that were not rolled back on error, one connection could leave an open
write transaction that pinned the write lock and made every other
connection's writes fail with "database is locked" for the full busy
timeout.
Instances for the same database now share ONE process-wide connection
and ONE re-entrant lock, so access is fully serialized and
cross-connection contention is impossible. The shared connection is
refcounted: closing one instance never tears it out from under a live
sibling, and the last close releases it. The connection runs in
autocommit (isolation_level=None) so a write that raises mid-method can
never leave a dangling transaction holding the write lock; the existing
explicit commit() calls become harmless no-ops.
The provider's shutdown() now calls the refcount-guarded close() instead
of just dropping the reference: leaving finalization to GC kept the
connection (and its write lock) alive indefinitely on long-running
gateways, prolonging the exact contention this fix removes. The last
provider now releases the connection deterministically while siblings
stay live; regression tests fail without the wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Flip the two tests that pinned the old buggy behavior (rewrite_transcript
called after in-place compaction) to assert the corrected invariant from
#61145: archive_and_compact() already persisted, so the handler must NOT
call rewrite_transcript — its replace_messages(active_only=False) would
DELETE the just-archived rows.
E2E-verified against a real SessionDB: 6 soft-archived rows are wiped by
replace_messages' default path, confirming the data-loss premise.
test_accepted_at_every_position spawned 11 separate
'python -m hermes_cli.main' subprocesses, each cold-importing the full
CLI module tree under a 15s TimeoutExpired deadline. On a loaded CI
worker the import alone can exceed that (slice 2/8 flaked exactly here
on PR #61726's run, TimeoutExpired at subprocess.py:1253), failing PRs
that never touched the CLI.
Replace with ONE driver subprocess that imports hermes_cli.main once
and parses all 11 argvs in-process (catching SystemExit per argv),
reporting JSON results. Same assertions per argv, identical semantics
(verified the --help-before-unknown-flag exit behavior matches the old
method), ~11x less import work, and the 180s timeout only trips on a
genuine hang.
Structural completion of the malformed-job freeze fixes (#61382 id-less,
#61525 non-dict schedule, #61581 bad next_run_at): wrap the per-job body
of _get_due_jobs_locked in try/except so any FUTURE malformed-field
variant degrades to skipping that one job for the tick instead of
aborting the scan before save_jobs() and freezing the whole profile's
scheduler.
Also: restore test_repeated_concurrent_runs_accumulate_completed_count
to TestMarkJobRunConcurrency (accidentally re-parented by the #61581
diff), add a containment regression test, and AUTHOR_MAP for hydracoco7.
E2E: one jobs.json carrying all five malformed shapes (drifted job_id,
missing id, null schedule, garbage next_run_at, non-string last_run_at)
plus a healthy sibling — single tick contains all five, sibling fires,
repairs persist, second tick stable. 670 cron tests green.
One bad next_run_at value in jobs.json aborts the due-jobs scan with
ValueError from fromisoformat, before any save_jobs, so siblings lose
progress (fast-forwards etc).
Early normalization in _get_due_jobs_locked + defensive parses in
compute_next_run / _recoverable_oneshot_run_at.
Added test_bad_next_run_at_does_not_crash_or_block_sibling_jobs.
A job record in jobs.json can have a non-dict 'schedule' value (null, string,
etc.) from direct edit or old writers.
In _get_due_jobs_locked:
schedule = job.get('schedule', {})
kind = schedule.get('kind')
This (and direct schedule['kind'] in compute_next_run etc.) raises and
aborts the entire due-jobs scan before save_jobs() or advancing next_run_at
for healthy jobs. Exactly the same failure mode as the id-less job P1.
Fix: normalize non-dict schedules to {} early (before any use), matching the
defense added for id-less records. Also added defensive guards in compute
functions.
Added regression test that a bad schedule does not crash and healthy sibling
is still returned.
Refs similar pattern in #61382.
A cron record authored by a direct jobs.json edit that bypassed
add_job() can lack an "id" key (older writers used "job_id"). Every
site in _get_due_jobs_locked indexes job["id"] eagerly — both the
logging helpers (job.get("name", job["id"]) evaluates the default
argument unconditionally) and the 'for rj in raw_jobs: if rj["id"] ==
job["id"]' persistence loops. A single malformed record therefore
raised KeyError mid-tick, aborting the entire scan before save_jobs()
ran. Result: healthy jobs' fast-forwarded next_run_at was computed in
memory then discarded on the exception unwind, freezing the whole
profile's scheduler in a per-minute loop (observed dormant for weeks).
Fix: normalize id-less records at the top of _get_due_jobs_locked before
anything keys off job["id"] — recover the id from a drifted "job_id"
key when present, else synthesize one via uuid4, and persist. This
repairs the whole bug class at the source rather than guarding each of
the ~12 downstream index sites.
Adds a regression test that fails with KeyError on the current code and
passes with the fix, asserting a healthy sibling job is still returned
when an id-less record shares the store.
The HTML session export interpolated the tool-call name into the page
without escaping, while every sibling field went through _escape_html. A
tool-call name is attacker-influenced, so a prompt-injected model can emit
a name containing HTML that executes when the export is opened in a browser.
Escape the tool-call name like the other fields.
switch_model() unconditionally set agent.provider but only set
agent.base_url when the resolved value was truthy. When a real
provider change resolved an empty base_url (e.g. minimax after
copilot), the agent ended up with provider="minimax" but
base_url still pointing at api.githubcopilot.com. That incoherent
pair then got snapshotted into agent._primary_runtime, so it kept
re-applying on every subsequent turn via restore_primary_runtime()
until the process restarted.
try_activate_fallback() and _swap_credential() were audited and
confirmed unaffected: both always derive base_url from an actually
constructed client, never from a possibly-empty resolver hint.
Fix: when base_url is empty AND the provider is genuinely changing,
raise ValueError instead of silently keeping the old provider's URL.
This routes through switch_model()'s existing snapshot/rollback
path, and callers (tui_gateway/server.py's _apply_model_switch)
already catch and surface a clean "switch failed, staying on X"
message. Re-selecting the SAME provider with an empty base_url
(credential-only refresh) still keeps the current URL, unchanged.
Fixes#47828
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Port from openai/codex#31188: a parse failure in a policy-bearing config
file must not silently replace the effective policy with an empty/default
one. Codex's load_exec_policy_with_warning replaced the whole exec policy
with Policy::empty() when a .rules file failed to parse, silently dropping
managed prompt/forbidden rules; the fix preserves the managed policy while
still warning.
Hermes had the same bug shape in load_config(): a YAML parse error made
_load_config_impl() fall through to DEFAULT_CONFIG, dropping every user
override — including approvals.deny rules, which are documented to block
commands even under --yolo. In a long-running gateway, a user mid-editing
config.yaml into broken YAML silently disarmed their own deny rules on the
next load.
Now, when the process has a last successfully loaded config for that path
(_LAST_EXPANDED_CONFIG_BY_PATH), a parse failure keeps serving it (cached
under the corrupt file's signature so the broken file isn't re-parsed) and
the warning says edits are being ignored until the YAML is fixed. Fresh
processes with no last-known-good keep the existing DEFAULT_CONFIG
fallback and warning.
E2E-verified: deny rule 'curl*evil.com*' still blocks after mid-process
corruption; fixed file reloads normally; fresh-process fallback unchanged.
The earlier fix gated _resolve_use_tui, but the EARLY launcher
(_wants_tui_early) decides TUI from display.interface before cmd_chat
runs — so a `display.interface: tui` default still booted the Ink UI for
headless spawns (kanban workers), whose no-TTY bail-out exits 0 →
"protocol violation". Gate the early resolver on a real TTY: headless
stdio never boots the TUI regardless of config; explicit --tui still does.
Dragging a task done→ready did nothing: the respawn guard saw a run that
completed within the success window and deferred forever, unable to tell a
deliberate operator re-run from a status flap. Now a re-queue event
(status change, promote, unblock, reclaim) AFTER the completion bypasses
the recent_success guard, so an explicit done→ready runs again.
A retried task (→ running) kept showing "crashed Nx": the in-flight run
has no outcome yet, so the trailing crash scan skipped it and kept
counting the prior streak, and the consecutive_failures counter lingers.
Exempt `running` from both repeated_failures and repeated_crashes so a
fresh attempt clears the banner until it itself resolves (re-fires if the
new run also fails).
An inherited HERMES_TUI=1 or a `display.interface: tui` config default sent
kanban workers into the Ink TUI, whose no-TTY bail-out exits 0 without doing
the task — every attempt ended in "protocol violation". Two layers:
- _default_spawn pins `--cli` (highest-precedence interface flag) and strips
HERMES_TUI from the child env (covers older builds on PATH).
- _resolve_use_tui gates ambient TUI prefs (env/config) behind a real TTY;
an explicit --tui still wins so the informative bail-out stays reachable.
A manual done (dashboard/desktop drag) runs complete_task but ends no
run, so a trailing crashed/crashed run history never gains the
'completed' outcome that breaks the repeated_crashes streak — the card
kept flagging "needs attention" forever after being finished.
repeated_failures had the same hole via a stale counter. Terminal
statuses are now exempt from both: done means done; the history stays
on the event log for audit. Regression test included.
Adding the 6 gpt-5.6 slugs to DEFAULT_CODEX_MODELS grew the curated codex
catalog to 11, above the test's max_models=10 cap. That truncated the
picker list to 10 while total_models reported 11, breaking the
total_models == len(models) assertion. The cap was an implicit
change-detector on catalog size; raise it to 100 so the list is never
truncated and the count-consistency invariant stays meaningful as new
gpt-5.x slugs land.
Close the remaining end-to-end gaps so the full gpt-5.6 family (sol/
terra/luna + their -pro high-effort modes, 6 slugs) works on every
surface a user can reach them through:
- agent/auxiliary_client.py: the Codex OAuth backend hard-caps context
at 272K for gpt-5.6 exactly as it does for 5.4/5.5, but the default
50% compaction trigger would summarize at ~136K and waste half the
usable window. Extend the existing _is_codex_gpt54_or_gpt55 chokepoint
(single enforced predicate feeding _compression_threshold_for_model)
to match gpt-5.6* on the openai-codex route so those sessions get the
same 0.85 auto-raise. Direct-API/OpenRouter routes (full 1.05M window)
are unaffected; the historical codex_gpt55_autoraise opt-out still
applies. The one-time notice banner is model-dynamic and already
renders the correct slug/cap.
- hermes_cli/config.py, agent/agent_init.py: refresh the autoraise
comments/notice to mention the 5.6 family.
- hermes_cli/codex_models.py: add the -pro variants to DEFAULT_CODEX_MODELS
+ forward-compat so ChatGPT-OAuth (openai-codex) Pro users see the full
family in /model, not just the base tiers.
Supersedes the earlier commit's note that 5.6 was intentionally kept out
of the codex catalog: the slugs are confirmed routable (OpenRouter live
+ codex backend), so they belong there like every other codex-capable
gpt-5.x slug.
E2E verified across all 6 slugs: direct-API ctx 1.05M, codex ctx 272K,
pricing reachable from openai + openai-api routes, codex compaction
override 0.85 (and None on direct-API + when opted out), present in
openai-api picker + codex catalog, /model gpt resolves to sol on both
native routes. Guard tests added for the compaction route matrix.
PR #61587 adds sol-pro/terra-pro/luna-pro to the aggregator lists.
Complete those on the native surfaces the same way this PR completes
the base tiers:
- hermes_cli/models.py: -pro variants in _PROVIDER_MODELS[openai-api].
- agent/usage_pricing.py: alias ("openai", "gpt-5.6-*-pro") onto the
base-tier PricingEntry rows — the -pro high-effort modes bill at the
SAME per-token rates (verified against OpenRouter live pricing
2026-07-09: identical prompt/completion prices for base and -pro);
they cost more per task by consuming more tokens, not a higher rate.
- Context lengths need no new entries: "gpt-5.6-sol" et al. are
substrings of their -pro variants and both lookup tables match
longest-key-first (verified: sol-pro -> 1.05M direct / 272K codex).
- model_switch sort: -pro variants parse as suffix "sol-pro" (rank 1),
so /model gpt still defaults to base sol — pinned by test.
- Not added to DEFAULT_CODEX_MODELS: only confirmed routable via API/
OpenRouter so far; codex live discovery will surface them if ChatGPT
exposes them, same policy as other unconfirmed codex slugs.
Tests: invariant tests extended (pro aliases share base entries, base
sol outranks sol-pro); 191 targeted tests pass.
Phase-2 review findings addressed:
- resolve_billing_route: normalize the "openai-api" picker slug to the
"openai" billing provider — without this the ("openai", <model>)
_OFFICIAL_DOCS_PRICING keys (incl. every pre-existing gpt-4o/gpt-4.1
entry, not just 5.6) were unreachable when the provider is openai-api.
- pricing_version: drop the "preview" tag (GA 2026-07-09 at same rates).
- model_metadata comment: dict order is cosmetic — lookups length-sort
keys at match time; the old comment implied a positional invariant.
- model_switch comment: note "sol" is a series codename, not a generic
quality word.
- tests/hermes_cli/test_gpt56_registration.py: behavior contracts (no
list snapshots) — sol > terra/luna > 5.5 sort invariant, pricing
reachability from both openai and openai-api routes, cache-write
1.25x / cache-read 0.10x input relation.
hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned
bubble instead of a separate text message followed by an uncaptioned bubble.
Root cause: the standalone senders (hermes send / cron / send_message tool)
stripped the MEDIA: tag, sent the remaining text as its own message, and
called the media send with no caption -- even though hermes send's help
advertises the captioned form and the bridges/adapters already support a
caption. Signal already captioned correctly.
- tools/send_message_tool.py: new _media_caption_split() chokepoint decides
caption-vs-separate-body (single captionable non-voice file within the
platform's message-length cap). Wired into the Telegram, WhatsApp and
Discord dispatch paths.
- Telegram/WhatsApp/Discord: when the single captioned file is missing, the
caption text is delivered as a plain message so it is never silently lost.
- Telegram caption send gets a MarkdownV2->plain parse fallback.
- Tests: _media_caption_split unit tests + per-platform caption tests
(ride, multi-file fallback, voice exclusion, over-limit fallback,
missing-file text fallback); updated the 3 tests that asserted the old
text-then-media split.
Closes the gap reported against #58911 (the MEDIA_CAPTION directive PR);
credit to @ferreiraesilva for surfacing the caption behavior.
Review finding: callers mutate the returned dicts in place —
hermes_cli/web_server.py annotates s['enabled']/s['usage'] on the skills
list — so handing out the cached objects poisons the cache for every
subsequent caller (and is a cross-thread shared-mutable hazard in the
gateway). Return [dict(s) for s in cached] on both hit and miss paths;
warm-path cost is negligible (241x speedup retained on a 300-skill
fixture). Regression test mutates a returned list/dict and asserts the
next cached call is clean.
Review findings on the cherry-picked cache (follow-up to #58985):
- The cache key was the max mtime of only the TOP-LEVEL scan dirs.
Adding/removing a skill inside a category subdir bumps the category
dir's mtime, NOT the root's, so the cache served a stale list
indefinitely. Replace with a per-dir signature covering roots +
immediate children (one scandir per dir; mirrors
hermes_cli/profiles.py::_count_skills from d5eee133e).
- The disabled-set is config-driven and changes with no filesystem
mtime bump; fold it into the signature so /skills disable takes
effect without a restart.
- Platform is part of the signature (gateway processes serve multiple
platform scopes; scan results are platform-filtered).
- Add a 30s TTL to bound staleness from in-place SKILL.md edits (file
mtime is invisible to any directory signature).
- The original also keyed dirs off the module-level SKILLS_DIR constant;
the scan itself uses _skills_dir() (live profile HERMES_HOME) — use
the same resolution for the signature.
Mutation-verified: nested-add, disabled-set, and TTL tests fail against
the pre-fix cache and pass with it.
The salvaged PR added a standalone test_reasoning_timeouts.py that duplicated
the structure of the existing parametrized test_reasoning_stale_timeout_floor.py.
Fold the v4-flash/v4-pro/-free positive cases and deepseek-chat negative cases
into the canonical parametrized tables and remove the redundant file.
DeepSeek V4 models (deepseek-v4-flash, deepseek-v4-pro) emit
reasoning_content in a separate delta field before final content,
requiring the same 600s stale timeout floor as R1. Without this,
streams hang for 30–50s with APITimeoutError on providers like
opencode-go while direct calls succeed in ~3s.
Fixes#60338.
Phase-2 review finding: the validation branch expanduser()s the path but
web_server.py reads os.environ['HERMES_WEB_DIST'] raw at import — a
'~/dist' value would validate here and still 404 there. Write the
expanded path back before the web_server import. Adds a regression test
asserting the env var holds the expanded path after cmd_dashboard.
Review finding: PR #40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)
Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by #57229 which
already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
behavior change reverting deliberate e8c3ac2f5; belongs in its own
PR with its own justification if pursued.
Conflict resolution: preserved main's newer timestamp-stripping
(#47868) inside the copy-on-write path.
Mock _preflight_user_systemd and _select_systemd_scope in
test_systemd_start_refreshes_outdated_unit and
test_systemd_restart_refreshes_outdated_unit. These tests target
unit-file refresh logic, not D-Bus reachability, so the preflight
check was causing spurious UserSystemdUnavailableError on macOS,
WSL, and Docker where systemd is unavailable.
(cherry picked from commit 34113300a1cbbecf7c9fa2201ddaf517acc521d3)
A custom HERMES_WEB_DIST without --skip-build skipped BOTH the web UI
build and any validation: cmd_dashboard fell through the build gate and
started the server against a dist that may not exist, serving 404s with
no obvious cause. This is the same failure mode issue #23817 fixed for
the --skip-build branch — the env-var branch was left unvalidated.
Add the missing else-branch: fail fast with actionable guidance when
HERMES_WEB_DIST has no index.html, proceed (still without building) when
it does.
Credit: @Caelier (#17845) originally proposed dist validation for the
dashboard startup path; the --skip-build half of that PR's scope has
since landed via the #23817 fix, this covers the remaining env-var path
on the rewritten cmd_dashboard surface.
Review finding: the substring check ('kimi' or 'moonshot' in model)
under-matches bare release slugs like k2-thinking that the repo's
canonical _model_name_is_kimi_family matcher (anthropic_adapter.py)
already covers. Reuse it instead of a second ad-hoc matcher; add a
regression test for the bare-slug case.
Adapted from PR #26014's test file to the canonical seam
(tests/run_agent/test_anthropic_prompt_cache_policy.py _make_agent
helper) instead of a new top-level file with sys.path manipulation.
Covers: kimi-k2.6 + moonshot-v1 on OpenRouter (envelope layout),
kimi via Nous Portal, and the non-OpenRouter negative case.
Structured review (2a/2b/2c) findings, all fixed:
- MAJOR: detect_local_server_type memo was process-lifetime with no
invalidation, permanently pinning a URL's server type. Now a bounded
1h TTL ((type, monotonic) tuples) so a backend swap on the same port
is re-detected. Test covers ollama->lm-studio swap after expiry.
- MAJOR: legacy disk-row compat was one-way. get_cached_context_length
and _invalidate_cached_context_length now consult the same key-shape
set {canonical, literal, canonical+slash} in both directions, so an
old slashed row is found (and cleared) when the runtime passes the
normalized URL. Tests pin both migration directions.
- MINOR: _localhost_to_ipv4 did whole-string replacement, which could
corrupt a proxy URL embedding http://localhost in its query. Now a
scheme-anchored host-only regex; localhost.example.com and embedded
substrings pass through. Tests added.
- MINOR: _invalidate_cached_context_length now also drops the
in-memory TTL probe rows for the pair, so a resolution inside the
TTL window can't re-persist the value just declared stale.
- Test gaps closed: detect-type cache hit + TTL-expiry re-detection,
ollama-show TTL expiry re-probe, reverse legacy-row lookups.
- Attribution gate: added zhchl@hermes-agent.local -> 8294 (PR #50572
author) to AUTHOR_MAP; the strict CI grep needs bare non-plus emails
literal in release.py.
Gates: ruff clean; targeted suites 202 passed / 0 failed; full
tests/agent 5426 passed with 17 failures identical on clean
upstream/main (pre-existing env-dependent anthropic/bedrock/credpool
tests); mypy delta vs base: 0 new errors; live smoke 6/6 PASS.
#37595 fixed the Windows dual-stack IPv6 timeout only inside
detect_local_server_type. The same 2s-per-probe penalty existed at every
other helper that builds a probe URL from base_url. Extract the rewrite
into _localhost_to_ipv4() and apply it at:
- query_ollama_num_ctx
- query_ollama_supports_vision
- _query_ollama_api_show (server_url derivation)
- _query_local_context_length (server root + LM Studio native URL)
Tests cover the helper's URL forms, non-localhost passthrough, and that
the ollama probes actually POST to 127.0.0.1.
Follow-up hunks completing the probe-cache cluster:
1. _query_ollama_api_show now goes through the existing
_LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) —
it was the one remaining per-resolution POST not covered by the
#56431-era wrapper. Failures are never memoized so a server that
comes up mid-startup is re-probed. Idea credit: #42081 (@Morad37),
reworked to comply with the positive-only rule.
2. Persistent context-cache keys are normalized through
_context_cache_key (trailing-slash strip) so http://host/v1 and
http://host/v1/ share one entry; reads and invalidation honor
legacy un-normalized rows. Idea credit: #37905 (@stevenau21).
Tests: TTL hit collapses to one POST, failure-not-memoized
(mutation-verified: unconditional caching makes it fail), namespace
no-collision vs the sibling probe, slash-variant dedup, legacy-row
read, dual-shape invalidation.
Remaining hunk of the PR-branch fixup commit: the LM Studio first-probe
assertion now expects the IPv4-resolved URL (the code half — applying
the rewrite to `normalized` before deriving lmstudio_url — was folded
into the previous cherry-pick's conflict resolution).
(cherry picked from commit 7d324b0e47444887fc615e1c428aa30c3dfcd2e8)
On Windows, `localhost` resolves to both ::1 (IPv6) and 127.0.0.1 (IPv4).
httpx tries IPv6 first, hanging 2 sec per probe when the server binds IPv4
only. detect_local_server_type() is called 3+ times during init, each with
a new httpx.Client, compounding to ~14s of dead time.
Replace localhost with 127.0.0.1 inside the function before connecting.
The function is only called for local endpoints (callers guard with
is_local_endpoint()), so IPv6 loopback adds no diagnostic value.
Measured: 19.9s → 4.0s on Windows with a local proxy on 127.0.0.1:8317.
(cherry picked from commit a075d3194bba2e0fa1aa13852862190839bfecef)
Follow-up to HexLab98's fix. The sync-before-regenerate invariant was
enforced by convention across 6 callsites (~3 idempotent unit-file reads
per command). Consolidate it into the one function every compare/regenerate
path funnels through — systemd_unit_is_current — and drop the now-redundant
callsite pre-syncs in refresh_systemd_unit_if_needed / systemd_start /
systemd_restart / systemd_status.
Kept the systemd_install pre-sync: the --force path bypasses the
is_current gate and calls generate_systemd_unit() directly, so it needs
its own sync to avoid baking /root/.hermes under sudo.
Reworked the two callsite-ordering tests into a chokepoint-invariant guard
(test_is_current_syncs_before_reading_unit) + a delegation test proving
start/restart no longer pre-sync. Both fail if the chokepoint sync is
removed; the pre-existing behavior test still passes.
Follow-up to the salvaged str(tools) fix. The id()-keyed
_TOOLS_TOKENS_CACHE had no eviction, so a long-lived gateway/desktop
backend could accumulate an unbounded number of stale entries as it
builds transient tool lists. Cap it at 256 with oldest-first eviction
(insertion-ordered dict) and add a regression test asserting the cache
never exceeds the cap.
PR #61281 removed the client-side X-Hermes-Session-Token requirement from the
dashboard OAuth mutation calls so cookie-authenticated hosted/mobile sessions
can start provider logins. That change is safe only because the server still
gates those endpoints (gated_auth_middleware cookie check + _require_token).
The PR's api.test.ts suite mocks fetch and only asserts client behavior, so a
re-break of the gated-mode cookie gate would pass CI unnoticed.
Add gated-mode TestClient tests asserting POST /api/env/reveal and the OAuth
mutation endpoints (disconnect/start/submit/cancel) return 401 with no session
cookie. Mutation-verified: neutering both the middleware gate and _require_token
flips all five to 200.
- Return the boundary snapshot from
_launch_session_boundary_memory_flush as a local value instead of
staging it on self._session_boundary_snapshot. The instance-attr
handoff could leak (no memory manager configured) or mis-fire a
stale snapshot on a later /new if an exception hit between staging
and consumption. A local variable eliminates the class; the helper
also returns None when no memory manager is configured so
new_session takes the inline-switch path.
- Drop the now-dead session_id kwarg from commit_memory_session:
after the redesign no production caller passes it (gateway, TUI,
compression all use the default), and speculative params are
rejected per AGENTS.md. The explicit-old-session need is served by
cli.py's direct engine call + commit_session_boundary_async.
- Drop the dead providers snapshot in commit_session_boundary_async
(only the emptiness check used it).
- Tests updated accordingly (dead-kwarg test removed, snapshot
assertion now covered by return-value contract).
Phase-2 gates: 2a tests/cli 1048 passed + 6 memory files 137 passed;
2b programmatic live smoke 0.38ms non-blocking caller, end→switch→sync
ordering verified; 2c structured 4-angle review — no Criticals, these
warnings fixed.
Deep review of the cherry-picked #16454 found the ad-hoc flush thread
raced new_session()'s inline on_session_switch(reset=True): memory
providers key off internal _session_id state (MemoryManager.on_session_end
takes no session id), so a late off-thread extraction ran against
post-rotation bindings — misattributing the old transcript to the new
session id, double-ingesting the old turn buffer (supermemory), or
double-committing (openviking already async-finalizes in
on_session_switch).
Redesign: new MemoryManager.commit_session_boundary_async queues
on_session_end + on_session_switch as ONE task on the manager's existing
single-worker background executor (the same worker sync_all already
uses). This preserves the strict end→switch ordering providers depend on,
serializes against per-turn syncs FIFO, keeps /new non-blocking, and
degrades to inline (pre-#16454 behavior) when the executor is
unavailable. No ad-hoc threads; no per-provider changes needed.
The context-engine on_session_end half stays synchronous in
_launch_session_boundary_memory_flush (cheap, must land before
reset_session_state rebinds the engine).
Exit durability: _run_cleanup calls the manager's existing
flush_pending(timeout=10) barrier before shutdown, so '/new then quit'
doesn't drop the queued extraction (shutdown_all's own drain is ~5s and
cancels queued tasks). Bounded well inside the 30s exit watchdog.
Tests: ordering invariant with slow (LLM-like) extraction, FIFO
serialization vs sync_all, switch-fires-even-if-end-raises, no-provider
no-op, CLI snapshot handoff + inline-switch fallback, sync engine
boundary, cleanup flush_pending.
Fixes#3356
Build the skills snapshot manifest in one directory walk, avoid importing gateway session context during CLI prompt startup, and reuse direct platform-list matching for snapshot entries.
(cherry picked from commit 1a64c2ed04f739ce03de94ca33d20b34c8d0e3cb)
Follow-ups on the #60987 salvage (review pass):
- _refresh_fallback_model: keep last known-good chain on transient
config.yaml read/parse failure (user mid-edit, torn write) — only a
successful read that lacks the key clears the chain. Previously a
refresh error wiped a cached agent's working fallback for the turn.
- Move the cached-agent refresh+apply OUTSIDE the agent-cache lock:
config.yaml read is disk I/O and the idle-sweep watcher contends on
that lock (same reasoning as #52197). Per-session turn serialization
keeps the post-lock apply safe.
- _apply_fallback_chain_to_agent: clear _unavailable_fallback_keys when
chain content actually changes, so an entry re-configured mid-uptime
(e.g. credentials added) is retried instead of staying suppressed for
the cached agent's lifetime; no-op refreshes keep the memo.
- Tests: cwd-independent source pin (Path(__file__) anchor), pin the
reuse-path apply call, + regression tests for last-known-good, memo
clear-on-change, memo keep-on-unchanged (mutation-verified).
Pin reload + cached-agent apply helpers for #60955 so a mid-uptime
fallback chain change reaches messaging sessions without a restart.
(cherry picked from commit fafb34103509f890cce158bfdbfd4faee8982253)
When an API error carries an httpx.Response whose body was consumed via
iter_bytes() during streaming error handling (e.g. GeminiAPIError from
agent/gemini_native_adapter.py), accessing .text raises
httpx.ResponseNotRead. The secondary exception replaced the real,
already-computed provider error (429 free-tier quota guidance) with the
generic 'Attempted to access streaming response content' message on
every turn.
Guard the .text access so it degrades to an empty snippet and falls
through to the str(error) fallback, which carries the full original
message. Mirrors the existing guards in
agent/error_classifier.py::_extract_error_body() and
agent/gemini_native_adapter.py::gemini_http_error().
Fixes#59769
Salvaged from PR #59868 (guard + regression test); the unrelated
desktop Ctrl-C fix bundled in that PR was intentionally dropped and is
triaged separately.
Rebase reconciliation with #60884: _count_status_active_sessions (from
#58238) now passes compact_rows=True (this branch's #47437 projection),
so the fake asserts both.
tui_gateway session.list/most_recent now pass compact_rows=True
(#47437 salvage); the keyword-only fake signatures in
test_tui_gateway_server.py rejected the new kwarg and CI slice 6/8
failed with TypeError. Other list_sessions_rich fakes use **kwargs and
are unaffected.
Review finding: get_messages(offset=N) with no limit dropped the OFFSET
entirely. SQLite requires a LIMIT clause for OFFSET, so emit LIMIT -1
(unbounded) when only offset is given. Regression test added.
- derive the compact_rows projection from SCHEMA_SQL (parse once, cache)
instead of a hardcoded column list: the original #47437 list was cut
against a June schema and silently dropped session_key/chat_id/chat_type/
thread_id/display_name/origin_json/expiry_finalized/git_branch/
git_repo_root/compression_failure_* — including desktop sidebar fields.
Schema-derived means declaratively reconciled new columns are included
automatically; only system_prompt is excluded.
- guard test pinning the schema<->projection contract (mutation-verified:
dropping a column from the projection fails it)
- wire compact_rows=(not full) into /api/sessions and /api/profiles/sessions
so the SQL projection pairs with the API-level field strip (?full=1 still
returns complete rows end-to-end)
- pass compact_rows at the remaining hot list callers: /api/status active
count, _session_latest_descendant fallback, /api/sessions/stats by-source
- thread compact_rows through the compression-tip projection
(_get_session_rich_row) so projected tips can't reintroduce the blob
- add pagination tests for get_messages (#60347 shipped none): paging order,
offset-past-end, active-flag interaction; add tip-projection compact test
- AUTHOR_MAP entries for mahdiwafy + CodeForgeNet (plain emails)
list_sessions_rich and _get_session_rich_row previously used SELECT s.*,
pulling the system_prompt TEXT blob on every row even for dashboard and
picker callers that never display it. On large databases this blob routinely
runs to tens of kilobytes per session, causing unnecessary B-tree I/O.
Add compact_rows=False param to both functions. When True, an explicit
column list omitting system_prompt is substituted for s.* in both the
simple and the recursive-CTE (order_by_last_active) query paths.
Default is False so all existing callers are unaffected.
Update dashboard and session-picker callers in web_server.py and
tui_gateway/server.py to pass compact_rows=True.
Add seven regression tests covering: omission of system_prompt, presence
of all metadata fields, both query paths, _get_session_rich_row, and
backward-compat default.
(cherry picked from commit c470cbd3042d95a62708cb5572118efd964f7699)
get_status now probes via get_running_pid_cached() (#53511 salvage);
these tests were added on main after that PR was cut and still patched
web_server.get_running_pid, so their fakes were bypassed and CI slice
5/8 failed. Patch the name the handler actually calls.
Review finding: SessionDB(read_only=True) requires the DB file to exist
(its documented contract says callers guard on db_path.exists()); on a
fresh install every /api/status poll paid an OperationalError until the
first session was written. Short-circuit to 0 when state.db is absent.
Tests: fresh-install guard + existing read_only test adjusted.
The #39140 CTE used UNION ALL, which recurses forever if a corrupted
parent chain loops (a -> b -> a) — reproduced: query never returns. The
old Python walk was cycle-safe via a seen-set. UNION dedups the working
set and terminates. Regression test added and mutation-verified (UNION
ALL hangs the test, UNION passes).
Sessions on sub-512K-context models were spending most of their wall-clock
re-summarizing: the 50% trigger left too little post-compaction headroom
(the incompressible floor — system prompt, tool schemas, protected tail,
rolling summary — ate most of the reclaimed space), so compaction re-fired
every 1-2 turns. Three compounding defects fixed:
- Threshold floor: models with context windows below 512K now trigger at
>=75% of the window (raise-only — a higher configured value or per-model
autoraise like Codex gpt-5.5's 85% always wins). Re-derived on
update_model() in both directions.
- No max_tokens on the summary call: the summary budget is prompt guidance
only ("Target ~N tokens"). The wire cap truncated summaries mid-section
on the Anthropic Messages / NVIDIA NIM paths (thinking models burn the
cap on reasoning first), yielding truncated or thinking-only summaries
and compaction loops. Summary token ceiling lowered 12K -> 10K to keep
the guidance within the intended 1K-10K envelope.
- Reasoning traces excluded end-to-end: inline <think>/<reasoning> blocks
are now stripped from assistant content before serialization to the
summarizer, and from the summarizer's own output before the summary is
stored (previously a thinking summarizer model's trace was persisted in
_previous_summary and re-fed into every iterative update, compounding
bloat). Native reasoning fields were already excluded.
Verified E2E with real imports against a temp HERMES_HOME: threshold table
across 64K-1M windows, override interactions (user 0.85 wins, spark 0.70
raised, gpt-5.5 0.85 kept), full compress() round-trip with a thinking
summarizer, and wire-kwargs capture proving no max_tokens is sent.
Completes the session-binding class on the gateway surface (#55578),
matching the TUI rules:
1. Fail-closed pinning: switch_session() re-opens ended sessions, so
pinning a completion to a spawning session that has since ENDED
(user /new, closed rotation) would resurrect a conversation the user
explicitly ended and inject into it. The injection path now checks
the pinned row's ended_at first and drops the injection with a
WARNING when the spawning session is dead or unknown - the result
stays in the delegation records.
2. /new ends the old conversation's delegations: _handle_reset_command
calls interrupt_for_session() with the expiring durable session id
(matching the parent_session_id pin stamped at dispatch) plus the
routing key as fallback, so a reset can't leave dangling subagents
whose completions have no live owner.
interrupt_for_session() gains the parent_session_id selector because a
gateway chat's session_key (the platform conversation key) survives a
reset while the session id rotates - key-based matching alone could
never sever a gateway conversation's delegations.
Background delegate_task completions only carried session_key. When multiple
active sessions shared a routing peer, get_or_create_session could recover the
latest ended_at IS NULL row and inject the subagent result into the wrong
session.
Capture parent_agent.session_id at dispatch time, include it on async-delegation
completion events, and pin gateway routing via switch_session when the
synthetic completion message is handled.
Fixes#57498
In single-query (-q) mode, the assistant's final answer was printed and
then immediately erased by _print_exit_summary() — which unconditionally
called _clear_terminal_on_exit() (ESC[3J ESC[2J ESC[H]). The answer was
present in the session store but invisible in the terminal.
The clear is only needed for interactive TUI teardown (#38928) where
prompt_toolkit chrome must be cleaned up. Add a clear_screen parameter
to _print_exit_summary() (default True, preserving interactive behavior)
and pass False from the single-query call site so the answer stays
visible above the exit summary.
Regression tests cover:
- clear_screen=True (default) calls _clear_terminal_on_exit()
- clear_screen=False skips the clear
- Single-query -q path passes False end-to-end
- Interactive path still clears (preserving #38928)
Extends the salvaged session_key filter with the same fail-closed,
compression-chain-aware ownership gate the poller uses (#55578):
- drain_notifications() accepts an owns_event callback; when provided,
an async-delegation event is consumed ONLY on positive proof of
ownership, and a broken callback re-queues (never leaks). Bare key
equality remains for single-session callers (CLI); no filter remains
legacy behavior.
- The TUI post-turn drain passes _session_owns_notification_event, so
it can't adopt another session's (or an orphan's) delegation payload,
while a post-compression session still claims its own pre-compression
dispatches - the gap bare key equality left open.
The completion event already carries the dispatching session's session_key
(captured at dispatch time in delegate_tool.py:2798), but the delivery
router ignored it — results landed in whatever session was active at
completion time instead of the session that dispatched the subagent.
Changes:
- drain_notifications() in process_registry.py: optional session_key
filter. Non-matching async_delegation events are re-queued instead of
consumed, so they remain available for the correct session's drain.
- cli.py process_loop: passes active session_key to drain_notifications()
- tui_gateway/server.py post-turn drain: passes session_key from the
TUI session dict
- gateway/run.py _build_process_event_source: logs warning when routing
metadata is unresolvable (previously silent drop)
- Regression tests verifying session-scoped drain filtering
Fixes#58684
- OPENROUTER_MODELS: remove openrouter/owl-alpha (free) and
tencent/hy3-preview{,:free}; add tencent/hy3 and tencent/hy3:free
- _PROVIDER_MODELS[nous]: tencent/hy3-preview -> tencent/hy3
- run_agent.py reasoning-prefix list: tencent/hy3-preview -> tencent/hy3
(prefix match still covers -preview if pinned)
- model_metadata: register hy3 context length (262144) alongside hy3-preview
- regenerate website/static/api/model-catalog.json
- update tokenhub curated-list tests to the new IDs
The tencent-tokenhub direct provider still serves hy3-preview and is
intentionally unchanged.
Two invariants layered on the origin-routing commit (#55578):
1. Fail closed on orphaned async-delegation payloads. The poller's
belongs-elsewhere check handles events owned by another LIVE session,
but an event whose owner is gone previously fell through and was
adopted by whichever poller saw it - injecting one chat's delegation
output into another chat. Delegation completions are now injected
only into a session that PROVABLY owns them (origin UI id, or
session-key/lineage match via the compression chain); unowned
payloads are dropped from injection with a WARNING (the subagent's
output is already persisted in the delegation records, so nothing is
lost). The shutdown drain applies the same rule. Non-delegation
events keep the historical adopt-orphans behavior.
2. A session's in-flight async delegations end with the session.
_finalize_session now calls interrupt_for_session(): delegations
commissioned by the closing UI session are interrupted always;
key-matched delegations only when the TUI owns the session lifecycle,
so closing a viewer tab on a live gateway session never kills the
gateway's own background work.
Carry the live TUI session id with async delegation completion events and prefer the commissioning UI session when desktop pollers share the completion queue. Resolve compressed session keys to their continuation before treating events as orphaned, and capture the live parent agent session id for TUI/ACP dispatch.
The desktop app's chat panel reuses tui_gateway as its backend, so every chat session was stamped platform="tui". That made the agent read terminal-specific platform guidance while running in the graphical desktop chat surface.
Resolve the misclassification at its source: tui_gateway now picks platform="desktop" when HERMES_DESKTOP=1 and HERMES_DESKTOP_TERMINAL is unset, and keeps platform="tui" for the embedded terminal pane and standalone TUI. Add a PLATFORM_HINTS["desktop"] entry describing the actual chat surface (full GFM markdown, MEDIA: intercept, inline images). Move the embedded-pane clarifier to the platform-hint resolution site so it appends only to the tui hint under HERMES_DESKTOP_TERMINAL=1. Delete the now-dead desktop-hint block from build_environment_hints() that competed with the platform hint.
Standalone TUI sessions produce byte-identical prompts as before; the new desktop hint and clarifier are assembled once per session in the stable tier, so prompt caching is preserved.
Three fixes for the silent post-restart ticker stall:
1. _jobs_lock() bounds its cross-process flock: LOCK_NB polled against a
30s deadline instead of an unbounded LOCK_EX taken while holding the
process-wide RLock. On timeout it logs at ERROR and degrades to
in-process-only locking (the existing fallback path), so a sibling
process wedged while holding .jobs.lock can no longer freeze every
cron function - including the ticker's get_due_jobs() and thus the
heartbeat - forever with zero logging.
2. fire_claim/run_claim freshness checks are bounded on both sides
(0 <= age < ttl): a claim stamped in the future (clock/TZ skew across
a restart) was previously fresh forever, making the job permanently
unfireable and every manual run report 'already being fired'.
3. _execute_job_now distinguishes paused/disabled/missing jobs from a
genuinely held claim instead of mislabeling them all as 'already
being fired'.
For air-gapped / self-hosted-IdP deploys with NO Nous Portal, let the gateway
obtain its caller-identity bearer from a generic OAuth2 client_credentials grant
against the operator's own IdP (e.g. Microsoft Entra ID) instead of only
resolve_nous_access_token(). The connector's OIDC tenant resolver reads a claim
(default tid) off that token as the tenant.
- gateway/relay: new canonical _resolve_relay_identity_token() — client_credentials
when gateway.idp.token_url (or GATEWAY_RELAY_IDP_* env) is set, else Nous Portal
(unchanged default). Wired into self_provision_relay().
- hermes_cli/gateway_enroll: _resolve_identity_token() delegates to the canonical
resolver so the enroll CLI and the runtime self-provision path share ONE impl.
Config via gateway.idp.{token_url,client_id,client_secret,scope} in config.yaml
(env override GATEWAY_RELAY_IDP_*). No behaviour change when unset.
Tests: tests/gateway/relay/test_identity_token_resolver.py (6 — mode selection,
request shape, config/env precedence, fail-closed). Relay suite 162 pass.
Validated via the cross-repo gateway<->connector live E2E (provision, managed
self-provision, inbound round-trip, /link) against a connector running the OIDC
tenant resolver with zero NAS config.
The salvaged guard used a hand-maintained frozenset of 14 platform names —
several of which (line, wechat, facebook, imessage, googlechat) aren't
actual Hermes Platform values, while real ones (whatsapp_cloud, feishu,
wecom, dingtalk, qqbot, yuanbao, plugin platforms like irc) were missing.
Resolve the source through gateway.config.Platform instead (built-ins +
registered plugin platforms via _missing_), with an explicit exclusion set
for self-owned/local sources. Adds tests for the guard and both reap paths.
Keep #60631's get_running_job_ids() snapshot + _active_cron_job_count()
(import-guarded for minimal test doubles) as the single read path, and
retarget #60612's drain tests at it. Drops the redundant
cron_jobs_in_flight() helper so there is one surface, not two.
Follow-up to the previous commit on #60432. The status-write guard
(_consume_interrupted_flag, checked right before mark_job_run) closes
the false-success bookkeeping gap, but run_one_job delivers its result
BEFORE that check: delivery happens right after run_job() returns,
mark_job_run happens at the very end. A job whose tool subprocess was
killed mid-flight can still produce a plausible-looking final_response
from the truncated output, and that response would reach the user via
_deliver_result before the interrupted flag was ever consulted --
correct status in jobs.json, wrong message already sent.
Adds _is_interrupted(), a non-destructive peek at the same
_interrupted_job_ids set (_consume_interrupted_flag stays as the
consuming, authoritative check right before the status write -- this
needed a peek instead since the flag has to still be visible there).
Checked right after save_job_output, before the deliver_content
decision: if the run looked successful but was flagged interrupted,
force success=False with an explicit interruption message. This
routes delivery through the existing _summarize_cron_failure_for_delivery
path (the same one a real failure already uses) instead of the raw
final_response, so the user gets an honest "this run was interrupted"
instead of a truncated/misleading result.
Testing: 4 new tests in tests/cron/test_shutdown_interrupt.py --
_is_interrupted peek semantics (false/true/does-not-clear, as opposed
to the consuming _consume_interrupted_flag), and the delivery-gate
test itself, which mocks run_job to return a normal-looking success
with a "plausible final response" while the job is pre-marked
interrupted, and asserts _deliver_result receives the failure summary
("This run was interrupted.") instead, with the summarizer's error
argument confirmed to mention the interruption.
Fail-then-pass: reverted cron/scheduler.py only, the 4 new tests fail
(3 on the missing _is_interrupted attribute, 1 -- the delivery-gate
test -- on _summarize_cron_failure_for_delivery never being called,
i.e. the raw response would have gone out); restored, all 16 tests in
the file pass.
Regression: tests/cron/ (683 tests) + test_cron_active_work_drain.py +
test_gateway_shutdown.py + test_shutdown_cache_cleanup.py -- 11
pre-existing failures (Unix file-permission-bit and path-tilde
assertions that don't apply on this Windows dev box), matching the
same set already established as pre-existing in the prior commit's
regression check. Zero new failures.
Continues #60432
Cron jobs run through cron/scheduler.py's own ThreadPoolExecutor via a
standalone AIAgent (run_job/run_one_job), entirely outside
GatewayRunner._running_agents -- the dict _drain_active_agents() and
every other active-work check on that class reads. A gateway shutdown
(/update, /restart, and SIGUSR1 all funnel through the same stop())
could log active_at_start=0 and immediately kill tool subprocesses
while a cron job's terminal command was still running, with no wait
and no indication anything was interrupted.
Real-world impact (from the issue): a scheduled daily briefing cron
job was in flight during /update, its tool subprocess got killed
by the unconditional shutdown cleanup, and the job was never marked
failed -- it simply never completed or delivered, with no error
surfaced anywhere. A repro with a 30-minute `sleep` cron job in flight
during /update reproduced the same pattern: subprocess killed at
+0.22s of drain (active_at_start=0), the job's agent thread continued
in-process and produced a plausible-looking final response from the
truncated tool output, and the scheduler marked the run successful.
Root cause is layered, not a single line:
1. GatewayRunner._drain_active_agents() only waits on _running_agents.
Cron work was invisible to it, so drain returned instantly whenever
the only active work was a cron job.
2. Even with visibility, the shutdown's final tool-subprocess kill
(process_registry.kill_all()) is a global, unconditional sweep with
no per-job targeting -- a long-running cron job that outlives the
drain timeout still gets its subprocess killed.
3. cron/scheduler.py had no way to detect that a job's tool subprocess
was killed out from under it mid-run; the agent thread kept going
and its eventual (often degraded but plausible-looking) response
got reported as a normal successful completion.
Fix, three parts:
- cron/scheduler.py: expose get_running_job_ids() (thread-safe
snapshot of the existing _running_job_ids set, already used to
prevent double-dispatch) so the gateway can read cron's in-flight
state without reaching into private module internals.
- gateway/run.py: GatewayRunner._active_cron_job_count() reads that
snapshot. _drain_active_agents() now waits on
(_running_agents OR active cron jobs), so a cron-only workload gets
the same bounded wait chat sessions already get instead of an
instant active_at_start=0. Shutdown drain logging gains
cron_active_at_start/cron_active_now fields alongside the existing
ones (unchanged, for compat).
- cron/scheduler.py: mark_running_jobs_interrupted(reason), called by
gateway/run.py's _kill_tool_subprocesses() right after
process_registry.kill_all(), marks every job still in
_running_job_ids at that instant as failed/interrupted via the
existing mark_job_run() -- and records the job IDs in
_interrupted_job_ids BEFORE writing, so run_one_job()'s own
eventual completion for the same run (racing in its own thread)
checks that flag and skips its normal write instead of clobbering
the interrupted status with a false "ok" produced from the
now-truncated tool output. This does not attempt to correlate a
killed PID to a specific job ID (process_registry tracks PIDs, not
job IDs) -- any job still dispatched at the moment of a forced kill
is treated as interrupted, matching the existing coarser precedent
set by _interrupt_running_agents(), which interrupts every entry in
_running_agents on a drain timeout without per-agent correlation
either.
Deliberately out of scope (flagged in the issue as a separate,
lower-priority concern): startup-time reconciliation of cron runs that
started but never reached a terminal status.
Testing:
- tests/cron/test_shutdown_interrupt.py (12 tests): get_running_job_ids
snapshot semantics, mark_running_jobs_interrupted marking/no-op/
partial-failure behavior, and -- the core race guard -- run_one_job
skipping its own last_status write (both the success path and the
exception path) when the shutdown path already marked the run
interrupted, with a control test proving ordinary un-interrupted
completions are unaffected.
- tests/gateway/test_cron_active_work_drain.py (9 tests):
_active_cron_job_count reading cron state and failing closed (0) if
the cron module is unavailable; _drain_active_agents waiting for an
in-flight cron job the same way it waits for chat sessions, timing
out if the job outruns the window, and leaving existing chat-session
drain behavior unchanged; a full runner.stop() integration test
(drain-timeout path) proving mark_running_jobs_interrupted actually
fires with the right job ID when a tool subprocess is force-killed,
plus a no-op control when nothing cron-related is in flight.
- tests/gateway/test_shutdown_cache_cleanup.py: added
_active_cron_job_count() to that file's hand-rolled _FakeGateway test
double, which stop() now calls -- without it those 8 pre-existing
tests AttributeError (caught by fail-then-pass below, not a
production bug).
Fail-then-pass: reverted gateway/run.py + cron/scheduler.py, all 21
new tests fail (fixture/attribute errors -- the feature doesn't exist
yet); restored, all 21 pass.
Regression check: ran the full plausibly-affected surface --
tests/gateway/{test_gateway_shutdown,test_restart_drain,
test_restart_notification,test_restart_redelivery_dedup,
test_restart_resume_pending,test_restart_service_detection,
test_shutdown_cache_cleanup,test_stuck_loop,test_clean_shutdown_marker,
test_external_drain_control,test_session_state_cleanup,
test_update_command,test_update_streaming}.py plus tests/cron/ (944
tests) -- against a clean upstream/main checkout and against this
branch. Diffed the two FAILED lists: identical, 20 pre-existing
failures on both sides (Windows-locale/cp1252 file-encoding issues and
Unix-permission-bit assertions that don't apply on this Windows dev
box), zero new failures, zero fixed-by-accident. The 8
test_shutdown_cache_cleanup.py failures found mid-development were
from the _FakeGateway gap above, fixed in the same commit and
confirmed clean on the final rerun (diff against baseline: exit 0).
Fixes#60432
safe_load() raises ComposerError on multi-document streams (k8s manifests)
and ConstructorError on application-defined tags (CloudFormation !Sub,
Ansible !vault) — both valid YAML syntax. Now that the linter's verdict is
a fail-closed write gate, those false positives would refuse legitimate
writes outright. Switch to yaml.parse() (scanner+parser only), which still
catches real syntax failures.
write_file() previously called _atomic_write() first and only ran the
JSON/YAML/TOML/Python syntax check afterward as an informational lint
delta -- a parse failure never set the top-level `error` key, so a
corrupt structured-data write still landed on disk (and file_tools.py's
files_modified gating, which keys off `error`, silently reported it as
a successful modification).
Move the in-process syntax check for JSON/YAML/TOML ahead of
_atomic_write() and refuse the write outright on a parse failure: no
temp file, no rename, nothing touches disk, and the result carries a
top-level `error` so callers correctly see it as unmodified.
Deliberately scoped to _FAIL_CLOSED_INPROC_EXTS (JSON/YAML/TOML), not
all of LINTERS_INPROC -- .py is excluded because this codebase's own
test fixtures (TestPatchReplacePostWriteVerification et al.) write
arbitrary non-Python text through *.py paths purely to exercise
write-mechanics; a hard block there broke 3 previously-passing tests
during development. Python keeps its pre-existing non-blocking
lint-delta report.
Adds tests/tools/test_write_file_syntax_gate.py: invalid JSON/YAML/YML/
TOML refused with nothing written (new file) and nothing modified
(existing file); valid JSON/YAML still written byte-for-byte; a
non-linted extension with garbage content is unaffected; invalid Python
is confirmed NOT hard-refused (still just reported).
* feat(install): warn pip/Homebrew installs are unsupported (CLI, TUI, desktop)
pip and Homebrew are now Unsupported install methods per
website/docs/getting-started/platform-support.md. Surface a
warn-don't-block deprecation notice everywhere the install method is
already shown, pointing at the platform-support docs and noting these
installs will not receive further updates. NixOS (Tier 2) is untouched.
- hermes_cli/config.py: shared is_unsupported_install_method() /
format_unsupported_install_warning() helpers so the wording and docs
link stay consistent across every surface.
- hermes_cli/banner.py: generalize the existing pip-only banner
warning to also cover Homebrew.
- hermes_cli/main.py: hermes update and hermes update --check print
the warning before proceeding (still update; warn, don't block).
- tui_gateway/server.py: session.info gains install_warning.
- ui-tui: SessionPanel renders install_warning alongside the existing
'N commits behind' notice.
- apps/desktop: SessionRuntimeInfo/GatewayEventPayload gain
install_warning; applyRuntimeInfo + the live session.info event fire
a snoozable warning toast via a new reportInstallMethodWarning(),
mirroring the existing backend-contract-skew toast pattern. i18n
strings added for en/zh/zh-hant/ja.
- Tests: updated pip banner assertions for the new wording, added a
Homebrew banner test, and two tui_gateway session_info tests
(install_warning present for pip, absent for git).
* fix(nix): make `hermes` in developement environment actually work
install modules as editable overlay with uv
* feat: print install method when running --version
* fix: correct detect install method when running from a subtree
The connector now depends on the single multiplexed gateway for per-profile
relay routing, so hosted deployments need to FORCE multiplexing on regardless
of the image's config.yaml. gateway.multiplex_profiles was config.yaml-only,
which a user could leave unset or flip off.
Add GATEWAY_MULTIPLEX_PROFILES as a standard operator override on top of the
existing config key — the same 'config.yaml is canonical, env is the operator
override' pattern the Telegram/Signal require_mention bridges use:
env (recognized token) > config.yaml (top-level or nested gateway.*) > False
- gateway/config.py: _env_multiplex_profiles_override() resolves the env var
tri-state — recognized truthy/falsy token → bool; unset/blank/unrecognized
→ None (fall through to config). Blank is deliberately None, not False, so a
provisioned-but-unpopulated Fly secret ('') can't shadow a config.yaml opt-in
(the empty-secret trap). Wired into GatewayConfig.from_dict so every consumer
(run.py, session.py via self.config) sees the resolved value.
- hermes_cli/gateway.py: the named-profile-start guard
(_guard_named_profile_under_multiplexer) reads config.yaml directly, so it
gets the SAME env precedence — otherwise env-forced multiplex would leave the
guard blind and someone could start a conflicting per-profile gateway that
double-binds a bot token. Env-forced-on trips the guard even with no
config.yaml key; env-forced-off disables it over a config opt-in.
Tests: full 3-tier precedence in test_config.py (incl. the discriminating
env-overrides-config cases + the empty/whitespace/unrecognized fall-through
trap + resolver tri-state), mutation-verified (flipping precedence fails
exactly the two env-wins tests); guard env cases in test_multiplex_lifecycle.py.
Force-on is safe on a single-profile instance: session keys stay byte-identical
(agent:main) and the _run_agent wrapper installs the per-turn secret scope, so
the fail-closed get_secret() path is satisfied.
Strict charset allowlist (alnum + - _, max 64) on the {name} path param of
the memory-provider config/setup endpoints. Prevents traversal-shaped names
from reaching find_provider_dir(), and setup now 404s when neither a
loadable provider nor a plugin manifest exists, so the command-running path
is only reachable for discoverable plugins. Adds regression tests.
The multiplex machinery already routes an inbound message to a profile via
SessionSource.profile (build_session_key namespacing + the per-turn
config/credential scope in SessionStore._resolve_profile_for_key). But the
relay path never populated it: _event_from_wire rebuilt the SessionSource
field-by-field and dropped any 'profile' the connector sent, so a
Team-Gateway (connector + relay) message could not be routed to a specific
profile the way the /p/<profile>/ HTTP prefix and per-credential polling
adapters already can.
Stamp source.profile from the wire payload in _event_from_wire. This is the
last missing link for NAS-driven per-profile routing over the relay in
multiplex mode; the connector populating the field ships separately
(gateway-gateway contract adds the optional wire field).
Back-compat: absent 'profile' → None → legacy agent:main namespace,
byte-identical to today for every single-profile gateway.
The profile+gateway topology added in #60537 sits entirely behind the
loopback/--insecure auth gate. But a hosted agent (Hermes Cloud) binds
non-loopback with OAuth, so should_require_auth is True, and NAS reads
/api/status over the network (fly-provider.ts getInstanceRuntimeStatus)
with no session token. On that gated path the whole topology block was
omitted, so the Portal could never render the profile list.
Split the topology readout by sensitivity:
- profile NAMES (profiles) + gateway_mode are low-sensitivity product
surface and now ride the always-public status body, surviving the auth
gate so NAS/the Portal can enumerate profiles.
- the per-gateway detail (gateways[], carrying host ports) is deployment
recon and stays gated alongside hermes_home / config_path / env_path /
gateway_pid / gateway_health_url.
The collector now runs unconditionally (still in the executor, off the
event loop). No new fields; only the gate placement changes.
Headless/hosted deploys run the dashboard server without COLORTERM in
the process environment, so chalk inside the PTY-spawned TUI child
downgraded every skin hex color to the xterm 256 palette — the default
skin's bronze banner border (#CD7F32) snapped to palette 173 (#D7875F,
salmon red) and the gold caduceus rendered red/yellow on fresh cloud
instances. Local launches never reproduced it because the operator's
interactive terminal leaks COLORTERM=truecolor into the server env.
xterm.js always renders 24-bit RGB, so the dashboard PTY child should
always advertise truecolor: backfill COLORTERM=truecolor in
_resolve_chat_argv via setdefault (an explicit operator value wins).
Verified with a clean-env PTY probe of the real TUI binary:
no COLORTERM -> 0 truecolor SGRs / 165 palette-256 (salmon 38;5;173);
with the backfill -> 166 truecolor SGRs, exact bronze 38;2;205;127;50.
Session-based channel discovery resurrected historical origins for
platforms with no connected adapter, exposing stale send_message
targets that can no longer deliver. Gate both the enum loop and the
plugin-registry loop on the live adapter set.
Surgical reapply of the channel-directory portion of PR #25959 (branch
was 6.5k commits stale; the text-batching delay changes bundled there
were dropped - separate concern, defaults have since been retuned on
main).
Co-authored-by: Marco-Olivier Lavoie <marcolivier@gmail.com>
Log a one-shot structured warning when Discord denies traffic because
no allowlist/policy is configured, and correct the setup wizard's
inverted warning text. The fail-closed default itself is unchanged.
Fixes#58682.
When a user explicitly configures a platform with its native composite
(e.g. platform_toolsets.discord: [hermes-discord]), the discord and
discord_admin toolsets were silently stripped by _DEFAULT_OFF_TOOLSETS
even though the composite contains those tools. The strip could not tell
an explicit composite opt-in apart from the unconfigured default.
Track whether the platform was explicitly configured and, when it was,
exempt toolsets that are both default-off and platform-restricted to the
current platform from the strip. Only discord/discord_admin are affected
(the sole entries in both _DEFAULT_OFF_TOOLSETS and
_TOOLSET_PLATFORM_RESTRICTIONS). Unconfigured and empty-list platforms
keep the security default-off behaviour.
/api/status (loopback/insecure binds only) now includes:
- profiles: every profile on the host (default + named)
- gateway_mode: none | single | multiple | multiplex
- gateways: one entry per live gateway with the host ports its
port-binding platforms listen on, plus served_profiles when the
default gateway is multiplexing
Ports resolve from each profile's config.yaml (top-level platforms:
wins over gateway.platforms:, matching load_gateway_config precedence)
with adapter defaults as fallback. Topology enumeration runs in an
executor so the profile scan + process-table probes stay off the event
loop, and the whole block is gated behind the same loopback-only split
as hermes_home/gateway_pid so gated binds leak nothing new.
The salvaged test predates the parked-server self-probe
(_PARKED_RETRY_INTERVAL, landed on main after the PR branched): after the
final failed retry, run() parks in a real asyncio.wait that the patched
asyncio.sleep doesn't cover, stalling the test 300s. Signal shutdown once
the retry budget is exhausted so the park exits immediately.
A stdio MCP server (e.g. `npx -y mcp-remote <url>`) is spawned as a direct
child of the Hermes process. Existing teardown (MCPServerTask.shutdown() /
_kill_orphaned_mcp_children()) reaps it correctly on a clean exit, but a
kill -9 / crash / force-quit of the Hermes process skips that path entirely
-- the child (and its own descendants, e.g. mcp-remote's spawned node
process) is orphaned and keeps running. Repeated ungraceful restarts pile up
N orphaned processes racing to hold the same upstream SSE session, producing
errors like 'Invalid request parameters' on legitimate reconnects.
macOS/Linux have no portable equivalent of prctl(PR_SET_PDEATHSIG) at the
Python subprocess level, so this adds a thin supervisor
(tools/mcp_stdio_watchdog.py) that:
- execs the real command as its own child in its own process group
- passes stdin/stdout/stderr through untouched (MCP stdio protocol
talks directly over those streams)
- polls the original spawning PID with the same orphan-detection
algorithm already proven in tui_gateway/slash_worker.py (ppid
comparison + psutil creation-time guard against PID reuse)
- SIGTERM-then-SIGKILL's the child's process group the moment the
original parent is gone
Wired into _run_stdio via a new _wrap_command_with_watchdog() helper,
POSIX-only (matches the existing killpg-based cleanup's platform scope),
fails open (any error resolving pid/create-time falls back to the
unwrapped command) so this can never be the reason a working MCP server
stops starting.
Verified: reproduced the exact orphan scenario standalone (fake parent
process spawns watchdog + fake long-running MCP child, kill -9 the fake
parent, confirm the watchdog reaps the child within its poll window with
zero leaked processes). Updated test_mcp_tool_issue_948.py's resolved-path
assertion to check the watchdog-wrapped command instead of the raw
resolved binary. Full test_mcp_tool.py + test_mcp_stability.py +
test_mcp_tool_issue_948.py suite: 232 passed. Full -k mcp sweep across the
whole test tree: 1003 passed, 2 skipped, 0 failed.
A stdio MCP server that never completes `initialize` (e.g. emits a
non-JSON-RPC frame and then blocks on stdin) leaks a child process plus its
stdio pipes/pidfd on every discovery-retry cycle — unbounded, until the
gateway hits EMFILE and every new open()/spawn fails (#59349).
Root cause (confirmed by instrumenting the live repro, and different from the
issue's own hypothesis): the spawned child IS captured in `new_pids`, so the
report's "new_pids empty at finally" guess is not it. The real cause is that
`session.initialize()` hangs forever on the garbage stream. `connect_timeout`
only bounds the caller's `.result()` wait on the foreground thread — it does
NOT cancel the `_run_stdio` coroutine on the background MCP loop. So the
coroutine is stuck at `await session.initialize()` permanently, its cleanup
`finally` never runs, the child is never reaped, and it stays invisible to the
orphan-reaper (whose `_orphan_stdio_pids` set never gets populated).
Fix: wrap `session.initialize()` in `asyncio.wait_for(..., connect_timeout)`
so a stalled handshake fails instead of hanging. The TimeoutError unwinds
through the SDK context managers (closing the child's stdin -> EOF -> exit)
and lets the existing `finally` reap any straggler. Cross-platform — no
signals/pgid/proc.
Scope: stdio only. The HTTP path has the same `await session.initialize()`
shape but spawns no subprocess (so it can't cause this leak) and already has
httpx transport timeouts.
Verified: the reporter's repro goes from unbounded growth to draining to zero;
added a hermetic regression test (fake transport whose `initialize()` hangs,
asserts the connect is bounded by connect_timeout) that fails on the pre-fix
code and passes on the fix; 566 existing MCP tests pass; ruff clean.
Repro confirmed on macOS (pipe FDs); the Linux-specific pidfd growth in the
report should be equivalent — the reporter offered to validate on Linux.
Closes#59349
When an MCP stdio subprocess fails to connect (token expiry, port
contention, timeout), the run() reconnect loop retries with backoff.
Each retry calls _run_stdio() which spawns a new process pair, but the
previous failed pair was only detected as orphaned (added to
_orphan_stdio_pids) — never actually killed. This caused rapid zombie
accumulation: 5 failed attempts × 2 procs each = 10 orphans competing
for the same port.
Add a _kill_orphaned_mcp_children() call at the top of _run_stdio(),
before the _snapshot_child_pids() baseline, so any orphans from prior
failed attempts are reaped before a new subprocess is spawned.
Fixes#57355
Keep-alive path when ?attach=<token> is present: PTY outlives the socket
via PTY_REGISTRY, reattaches on reconnect. No token = unchanged legacy
pump (_legacy_pump). detach (not close) on disconnect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up on the #56966 salvage:
- is_provider_explicitly_configured(): an env-seeded credential-pool entry
only counts as explicit while its env var still resolves to a usable
secret. A stale auth.json entry left behind after the user deletes the
var no longer keeps the provider in the picker forever (#55790).
- TUI modelPicker + dashboard ModelPickerDialog/api.getModelOptions pass
include_unconfigured=true explicitly, preserving their full-universe
setup-affordance behavior now that the backend defaults to the
configured subset.
- desktop lib/model-options.ts routes explicit_only through the shared
requestModelOptions() helper (added on main after the PR branched).
- regression tests for ambient (gh_cli) pool sources, explicit manual/
device-code sources, and stale vs live env-seeded entries.
* feat(trace): upload sessions to HF Agent Trace Viewer
Salvage trace upload as a smaller CLI-first feature: deterministic Claude Code JSONL export, fail-closed redaction, lazy Hugging Face dependency, and no gateway slash-command wiring.
* chore(trace): drop external porting references from docstrings
Describe the trace-upload design in Hermes' own terms.
* feat(sessions): fold trace upload into 'sessions export --format trace'
Integrates the HF Agent Trace Viewer exporter (PR #36145) onto the
unified export surface instead of a separate 'hermes trace' subcommand:
- --format trace: Claude Code JSONL to stdout/file, or one
<id>.trace.jsonl per session for filtered bulk export; defaults to
the most recent session when no --session-id/filters given.
- --upload pushes to the user's private HF traces dataset (--public to
opt out of private); reads HF_TOKEN with guided setup when missing.
- traces are secret-redacted by default (force mode); --no-redact opts
out after review; redaction failure blocks export (fail closed).
- hermes_cli/trace.py + subcommands/trace.py removed; agent/trace_upload.py
is the single engine. Docs EN + zh-Hans; 4 new CLI tests.
The salvaged SelfHostedBackend made self-hosted servers reachable via
mem0.json / MEM0_HOST, but the setup wizard still offered only Platform
and OSS — exactly the gap users hit (Discord report: 'At memory setup
there's only 2 options'). Adds a third wizard mode:
- interactive picker: Platform / Self-hosted server / Open Source
- non-interactive: hermes memory setup mem0 --mode selfhosted
--host http://... [--api-key ...] [--dry-run]
- host -> mem0.json (behavioral), API key -> .env as MEM0_API_KEY
(secret), optional key for AUTH_DISABLED servers
- best-effort reachability check against the server, non-fatal
- README + memory-providers docs updated with the wizard path
`_save_anthropic_oauth_creds` wrote the Anthropic OAuth token file with
`os.replace(tmp, path)` followed by a post-hoc `chmod(0o600)`. Between the
rename and the chmod the token file existed at the default umask (0o644 on most
hosts) — a window in which another local user could read the access/refresh
tokens.
Write via `utils.atomic_json_write(..., mode=0o600)`, which creates the temp
with mode 0o600 *before* any content is written, fsyncs, atomically replaces,
preserves the existing file's owner, and cleans up its temp on failure. This
matches the `atomic_json_write(mode=0o600)` call already used elsewhere in this
module for the credential-pool write, and #56644's owner preservation.
Tests updated for the new mechanism, plus a check that the write goes through
`atomic_json_write(mode=0o600)` (mutation-verified).
Opt-in discord.approval_mentions (config.yaml, bridged to
DISCORD_APPROVAL_MENTIONS) prepends <@id> mentions for numeric
allowlist entries to exec-approval prompts, with a scoped
AllowedMentions override (users only). Default off - no surprise
pings. Reapplied onto the content-mirror layout from #60245: mentions
prepend to the visible content block and its truncation budget.
Original implementation from PR #39719; commits arrived bot-authored,
re-attributed to the contributor.
Salvage follow-up integrating PR #30481 (@simplast) and PR #57683
(@catbearlove1-lang) into the unified export surface:
- --format html: standalone self-contained HTML transcript (single
session or multi-session with sidebar), works with all shared filters
and --redact; requires a file output path.
- --only user-prompts: prompt-only export (jsonl records or md sections)
via the shared session_export renderer; the separate export-prompts
subcommand from the original PR is subsumed by this flag.
- AUTHOR_MAP entries for both contributors; docs EN + zh-Hans.
Review follow-ups on the salvage:
- get_all() pruned from the ABC and all three backends: mem0_list (its
only caller) was removed by the recall-tuning commit, leaving new,
tested, unreachable code — including SelfHostedBackend's _MAX_TOP_K
over-fetch workaround. Tests for it dropped; fake-class stubs remain
harmlessly. (The #52921 true-total fix lives on in the PR history if
a lister ever returns.)
- The persisted rerank config key was write-only (setup prompted for it,
nothing read it). initialize() now parses it into _rerank_default and
mem0_search uses it when the model doesn't pass rerank explicitly;
per-call args still win. Guard test added.
- Platform-mode setup now warns when MEM0_HOST is set in the environment:
the json host-clear can't help there (_load_config seeds host from the
env var, docs tell users to put it in .env) — the user would silently
keep routing to the self-hosted server.
- SelfHostedBackend: connect-level retries (httpx.HTTPTransport(retries=2))
so a single transient blip doesn't count toward the provider breaker;
transport now injectable and the test helper uses the real __init__
instead of mirroring it via __new__.
- plugin.yaml description no longer leads with reranking (off by default,
platform-only); docs em-dash typo fixed.
Follow-up on the salvaged #55614. The PR added host-based routing to
_create_backend (precedence: oss > host > platform) but two sibling surfaces
didn't mirror it:
- system_prompt_block() checked host before oss, so an oss+host config ran
OSS but told the model it was self-hosted HTTP. Reordered to match routing.
- Platform setup (hermes memory setup mem0 --mode platform) left a stale host
in mem0.json; since host beats platform, the user kept routing to the
self-hosted server. save_config merges (no delete), so clear host to ""
rather than pop() so the merge actually overwrites it.
Adds regression tests for both (mutation-checked).
Salvage of #55614 by @kartik-mem0 (mem0 maintainer). Adds a SelfHostedBackend
that talks to a self-hosted Mem0 Docker server over httpx (X-API-Key auth,
/search + /memories routes), gated behind `host`. Also folds in the mem0
research-team recall tuning that rides with it: rerank defaults to false across
all modes, the mem0_list tool is removed (5->4 tools), search guidance is
de-shouted, and self-hosted get_all reports the true stored total (#52921).
Supersedes the self-hosted portion of #52487 (@liuhao1024, first-submitted).
Closes#52478Fixes#52921
Review findings on the salvaged shim: (a) OpenAI callers may pass stop as
a bare string but Converse's stopSequences requires a list — normalize;
(b) call_llm(stream=True) (MoA aggregator) can reach this client and the
shim silently returned a complete response — keep that behavior (the
streaming consumer's got-final-object path downgrades gracefully) but log
it, and log dropped tool_choice, instead of silently ignoring both.
+2 regression tests.
Follow-up to the salvage of #60217 by @xxxigm.
Review findings on the salvaged #60332 breaker, fixed as follow-ups:
- restore_primary_runtime() now resets the streak (third provider-swap
path; without it a recovered primary was short-circuited before a
single attempt and could never be re-proven healthy except via /model).
- interruptible_api_call (non-streaming) now carries the same breaker
(guard at entry, bump on stale_call_kill, reset on success). Quiet-mode
/ subagent / headless sessions — the profile most like #58962's
unattended 494-failure session — take this path and had the identical
infinite stale-retry class.
- Partial-stream stub return now resets the streak (chunks were received,
provider demonstrably responsive).
- Consolidated the triple-duplicated counter arithmetic into shared
helpers (_stale_streak/_bump_stale_streak/_reset_stale_streak/
_check_stale_giveup) with one canonical comment block; error message
now says 'consecutive stale attempts' (the counter counts kills, not
turns — a single turn can produce several).
4 new tests (restore resets / no-op restore keeps latch / non-streaming
short-circuit / non-streaming success reset).
Follow-up for the salvaged #60332 circuit breaker. The breaker latches:
once the streak trips, interruptible_streaming_api_call raises before any
stream is attempted, so the on-success reset can never run again. The
error text tells the user to switch models and retry — but neither
switch_model() nor try_activate_fallback() cleared the streak, so a
freshly selected healthy provider kept short-circuiting forever (only
/new recovered), and the automatic fallback chain was wedged the same way.
Reset the streak at both swap sites (after a successful rebuild only;
rollback/exhaustion paths keep the latch). 4 tests.
A session wedged against an unresponsive OpenAI-compatible provider can hit the stale-stream detector on every turn and loop forever, burning the full 180s x retries each turn with no response. Issue #58962 reports 494 consecutive failures over 3+ days on a single session.
The streaming retry path already caps retries WITHIN a turn (HERMES_STREAM_RETRIES, default 2) but has no cross-turn cap. Once a session's conversation state makes every turn stale, it retries indefinitely across turns and never notifies the user.
Add a per-session consecutive-stale-stream counter on the agent:
- incremented on every stale-stream kill in the outer poll loop;
- reset to 0 only when a stream actually completes;
- when it reaches HERMES_STREAM_STALE_GIVEUP (default 5), the next turn aborts immediately with a clear, actionable RuntimeError instead of spending 180s x retries again.
This is distinct from the existing stale-stream work (local-provider hard ceiling #44938, backoff/parse-error #60031): those bound a single hung stream, while this bounds repeated cross-turn staleness and surfaces a user-visible error.
Adds tests/run_agent/test_stream_stale_circuit_breaker.py covering the short-circuit, the success-reset, and the increment.
Same 10062 degrade-gracefully pattern as _run_simple_slash: create the
thread anyway, skip the ephemeral followups that need a live
interaction token. Non-expiry defer errors still raise.
- export now shares _add_session_filter_args / build_prune_filters with
prune/archive: AGE grammar (5h/2d/1w/ISO) on --older-than plus the full
filter set (--model, --provider, --min-messages, --min-cost, --branch,
--chat-id, ...) for both JSONL and md/qmd bulk exports; --dry-run works
on JSONL too; removes the one-off list_export_candidates helper.
- new --redact flag runs exported message content and tool output through
force-mode secret redaction (agent.redact) for jsonl, md, and qmd.
- docs EN + zh-Hans updated; new tests for AGE grammar, extended filters,
filtered JSONL, and redaction.
Replaces the parallel export-md subcommand from the salvaged commit with
a --format jsonl|md|qmd flag on the existing export subcommand, so all
session export formats share one surface. Adds AUTHOR_MAP entry.
Salvage follow-up for PR #59542 by @web3blind.
_handle_active_session_busy_message (the busy_session_handler most
platform adapters register) demotes busy_input_mode='interrupt' to
queue semantics for two reasons: active subagents (#30170) and, as of
this week, context compression in flight (#56391) — interrupting while
compression holds the state.db lock races a new turn against the
pre-rotation parent session, and if that new turn also grows past the
compression threshold it starts its own uncancellable compression on
the same stale parent, forking orphaned compression siblings.
_handle_message has its own, independent inline "PRIORITY" busy-path
(reached directly with a live running agent — see the `if _quick_key in
self._running_agents:` guard, exercised end-to-end by the existing
tests/gateway/test_running_agent_session_toggles.py harness). Its own
comment says it mirrors _handle_active_session_busy_message's subagent-
demotion rationale verbatim, and it does demote for active subagents,
but it never checked _session_has_compression_in_flight, so a plain-text
follow-up landing on this path while compression is mid-flight still
called running_agent.interrupt() unconditionally.
Fix: add the same _session_has_compression_in_flight(session_key) check
before the PRIORITY interrupt call, demoting to queue exactly like the
sibling path.
Tests: tests/gateway/test_priority_path_compression_demotion_56391.py
drives _handle_message end-to-end (reusing the test_running_agent_
session_toggles.py harness pattern) with a live running agent and a
mocked compression lock. Mutation-verified: reverting the fix makes the
demotion test fail (interrupt() gets called) against the pre-fix code;
a control test pins the unchanged default-interrupt behavior when no
compression lock is held.
Extends the send_exec_approval embed-invisibility fix to its three
sibling prompt surfaces — send_slash_confirm, send_clarify, and
send_update_prompt — via a shared _self_contained_prompt_content()
helper. All four interactive views now carry their payload in plain
content next to the buttons; the embed stays as progressive
enhancement for clients that render it. Adds gold to the conftest
discord Color mock (update prompt is the only gold user).
Discord's ExecApprovalView, SlashConfirmView, UpdatePromptView, and
ClarifyChoiceView hardcoded timeout=300, ignoring approval timeout
configuration. All four now read approvals.discord_prompt_timeout from
config.yaml (default 300s, clamped 30-900s — Discord interaction tokens
expire at ~15 min, so values beyond 900s would render dead buttons).
Surgical reapply of the timeout portion of PR #45904; the unrelated
channel-context changes bundled in that PR were intentionally excluded.
Co-authored-by: cruzanstx <cruzanstx@users.noreply.github.com>
BaseEnvironment writes shell snapshots and cwd metadata through the process
umask. With a common 022 umask, snapshot files containing exported environment
state landed at mode 0644 even though they can include env-carried credentials
from the parent process.
Set umask 077 only around Hermes metadata writes: the initial snapshot
bootstrap and the post-command snapshot/cwd refresh. User commands still run
under the caller's original umask, while Hermes-owned snapshot and cwd files
are created owner-only.
This intentionally does not copy the source PR's global orphan sweep; deleting
all matching /tmp snapshot files could interfere with concurrent Hermes
processes. The security-critical local disclosure fix is the file mode clamp.
This is salvageable because the source report still identifies a concrete
credential-disclosure path, but the safe subset is smaller than the original
proposal: clamp only the Hermes-owned snapshot writes and leave process-wide
cleanup, user command umask, and concurrent sessions alone.
Salvages source PR: https://github.com/NousResearch/hermes-agent/pull/20056
Related issue: https://github.com/NousResearch/hermes-agent/issues/48441
Co-authored-by: Andrew Homeyer <andrew@hndl.app>
_SlashWorker call sites now pass profile_home=; the fakes' 2-arg
__init__ raised TypeError inside the spawn guard, leaving
slash_worker=None and failing the orphan-race regression tests.
Profile-local skills are unavailable in Dashboard/TUI/Desktop GUI because the
_SlashWorker subprocess is spawned with os.environ.copy() but does NOT receive
the profile-specific HERMES_HOME from the parent session. This causes the
subprocess to search ~/.hermes instead of the active profile's skills directory.
1. Modify _SlashWorker.__init__ to accept optional profile_home parameter
2. When profile_home is provided, set env['HERMES_HOME'] = profile_home before
spawning the subprocess
3. Update all 4 call sites to pass profile_home=session.get('profile_home')
4. Add regression tests for profile-home propagation
- Full TUI gateway test suite: 107 tests pass
- New tests cover:
- profile_home parameter acceptance
- backward compatibility (None, omitted)
- argv correctness
Fixes#40677
Sessions no longer auto-reset by default. SessionResetPolicy.mode now
defaults to "none" (was "both": 24h idle + daily 4am), matching the
setup wizard's existing no-reset default and community feedback that
surprise context loss hurts more than it helps.
- gateway/config.py: dataclass default + from_dict fallback -> "none";
installs whose config.yaml lacks a session_reset section stop
auto-resetting
- hermes_cli/setup.py: "Never auto-reset" is now the recommended/default
choice in hermes setup agent; stale comment updated
- docs (en + zh-Hans): default is no auto-reset, opt in via
session_reset in config.yaml
Users who explicitly configured idle/daily/both resets keep them.
/new's handler calls _reset_notice_session_info, which resolves live
provider credentials and can probe model context length over HTTP. In
CI there are no credentials, so resolution walks the entire fallback
chain (the failed run's log shows 'Primary provider auth failed ...
trying fallback' captured inside the test) and on a slow runner the
first parametrization can blow past send_and_capture's 2s poll window,
making adapter.send appear never-called.
Stub it to return an empty info block in the e2e runner fixture — these
tests exercise gateway command dispatch, not provider resolution, and no
other network-touching path exists in the /new flow. Flaked in run
28856659216 (telegram param only); tests/e2e now 57/57 locally.
Replace the hand-rolled ensurepip bootstrap (and five other one-off
pip-install code paths) with hermes_cli.tools_config._pip_install, which
prefers the bundled uv (fast, needs no pip in the venv), falls back to
python -m pip, and bootstraps pip via ensurepip only when missing.
Sites unified:
- hermes_cli/setup.py: _install_neutts_deps, _install_kittentts_deps,
modal SDK install, daytona SDK install
- hermes_cli/memory_setup.py: memory-plugin pip deps (previously dead-ended
when uv AND pip binaries were both absent)
- hermes_cli/dingtalk_auth.py: qrcode auto-install (previously invoked
'python -m uv' which is not how uv ships)
- agent/lsp/install.py: --target LSP server installs
- plugins/google_meet/cli.py, plugins/platforms/matrix/adapter.py,
plugins/platforms/google_chat/oauth.py, plugins/memory/honcho/cli.py
Tests updated to assert the ladder behavior (uv-first, pip fallback,
ensurepip bootstrap) instead of the removed bespoke branches.
Follow-up on the salvaged fix, which bounded start_polling() only in
_handle_polling_network_error. The same wedge (#59614) exists at the two
sibling call sites:
1. _start_polling_resilient (bootstrap): an exhausted pool hangs connect()
forever. The TimeoutError from wait_for is a builtins TimeoutError
(OSError subclass), so the existing except classifies it via
_looks_like_network_error and schedules background recovery.
2. _handle_polling_conflict (conflict-retry ladder): identical hang wedges
conflict attempt N forever; timeout now converts to RuntimeError and the
existing except schedules the next attempt.
Tests replaced with a stronger suite: hung-network-ladder repro (RED without
the fix), bootstrap hang schedules recovery, success-path sanity, and a
bug-class contract test asserting EVERY updater.start_polling( call site is
wrapped in wait_for so a new unbounded site can't reintroduce the wedge.
Verified RED (3 failures) with the wrappers removed, GREEN with them.
When the connection pool is in a degraded state after
_drain_polling_connections(), start_polling() can hang indefinitely
when both primary and fallback Telegram endpoints are unreachable. The
httpx client may hold a stale socket that neither connects nor times out
within PTB's internal flow, causing the reconnect ladder to stall at
attempt 1/10 forever.
Wrap start_polling() in asyncio.wait_for() with a 30-second timeout so a
hung call raises asyncio.TimeoutError and feeds back into the existing
retry ladder. This unblocks:
- The 10-retry ladder advances to attempt 2, 3, ...
- The heartbeat loop sees _polling_error_task.done() and can trigger recovery
- The reconnect watcher gets the adapter in _failed_platforms
Fixes#59614
Port from anomalyco/opencode#31877: JSON Schema type arrays like
["number","string"] (common in MCP tool schemas) were collapsed to the
first non-null type, silently dropping every other branch. Several
tool-call backends reject the array form outright — llama.cpp's grammar
generator and Gemini via OpenAI-compatible transports (e.g. GitHub
Copilot proxying to Gemini) 400 on it.
_sanitize_node now mirrors @ai-sdk/google: a single non-null type stays
type:X (+nullable if null was present), multiple non-null types become
an anyOf of single-type schemas so no branch is lost, and an all-null
array becomes type:null. Single-null collapse is unchanged.
Verified nested (object props, array items) survive the full sanitize
pipeline — combinator stripping is top-level-only and nullable-union
collapse only fires on single-survivor unions, so multi-type anyOf is
left intact.
Drop the Responses-API native compaction path and its opt-in umbrella
flag from the salvaged feature. On the Codex OAuth chat route Hermes
owns the message list and the summary compressor works (and stays
provider-portable — encrypted compaction items would lock the session
history to chatgpt.com and break /model switches and provider
fallback). On the app-server runtime (codex CLI/agent) the codex agent
owns the real thread context, so thread/compact/start is the only
mechanism that can actually shrink it (#36801) — that path is now the
default behavior for codex_app_server sessions, controlled by
compression.codex_app_server_auto (native|hermes|off), no umbrella
flag.
Removed: responses.compact() call path, codex_compaction_items replay/
persistence plumbing, codex_native_compaction + codex_responses_threshold
config keys, desktop settings fields, and their tests. Kept: everything
app-server (compact_thread(), compaction notifications, bookkeeping,
docs, tests) plus cache-busting keys for the surviving knobs.
brooklyn!
kshitijk4poor
HexLab98
embwl0x
Teknium
luxuguang-leo
HumphreySun98
Koho Zheng
briandevans
lemonwan
sharziki
yungchentang
Adam Biggs
dsad
bassis ho
chenkun
Adolanium
joaomarcos
Kshitij Kapoor
liuhao1024
pedrommaiaa
Tranquil-Flow
Rod Boev
Umi4Life
infinitycrew39
Shannon Sands
ethernet
Tosko4
Dixing Xu
gauravsaxena1997
CodeForgeNet
Omar Baradei
luyifan
0disoft
墨綠BG
yoma
nullptr0807
Grace
nankingjing
loongfay
Tranquil-Flow
Tony Simons
Dan Schnurbusch
fyzanshaik
waroffchange
giggling-ginger
Ben Barclay
Neo Guyver
BROCCOLO1D
harjoth
Sage
rainbowgits
Benn Denton
doncazper
Ronald Reis
Que0x
alex107ivanov
Xue Li
kartik-mem0
xxxigm
Alix-007
web3blind
pierrenode
Jake Present
Eugeniusz Gilewski
Luke The Dev
JP Lew
Georgio Constantinou
hmirin