fix(cli): declare tiktoken for tool token estimates - #33350
Conversation
lordbuffcloud
left a comment
There was a problem hiding this comment.
Looks right to me. This keeps the graceful runtime fallback for base installs, but makes the CLI extra actually ship tiktoken for tool-token estimates and locks that expectation in with a packaging test.\n\nI reran:\n- python3 -m pytest -q tests/hermes_cli/test_tool_token_estimation.py\n\nPassed locally.
austinpickett
left a comment
There was a problem hiding this comment.
✅ Approved
Dep gap confirmed on main: tiktoken appears nowhere in pyproject.toml — fresh pip install hermes-agent[cli] installs silently lack it, causing _estimate_tool_tokens() to always return {}.
Fix is surgical: "tiktoken==0.12.0" added to the cli extra; uv.lock updated (+164 lines resolving transitive deps); ImportError guard retained for base installs. Test parses pyproject.toml via tomllib — proper packaging regression guard.
CI: all 23 checks green including uv lock --check, osv-scanner, all 6 test shards, both platform builds.
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the missing CLI dependency while preserving the base-install fallback. The premise is still present on current main: hermes_cli/tools_config.py:2088-2094 lazily imports tiktoken, but pyproject.toml:171 declares the cli extra with only simple-term-menu.
Problems
tests/hermes_cli/test_tool_token_estimation.py:28snapshots"tiktoken==0.12.0". A normal dependency-version update would break this test even if the CLI extra continues to declaretiktoken. This conflicts with the repository's change-detector guidance inAGENTS.md:1309-1355.
Suggested changes
- Make the new test assert that the
cliextra contains atiktokenrequirement without hard-coding its version; retain version/pin-policy checks in the general packaging metadata tests. - During salvage, regenerate
uv.lockagainst current main rather than reusing the June lockfile resolution.
This is an automated hermes-sweeper review.
|
|
||
| cli_extra = pyproject["project"]["optional-dependencies"]["cli"] | ||
|
|
||
| assert "tiktoken==0.12.0" in cli_extra |
There was a problem hiding this comment.
Please make this assertion version-agnostic (for example, assert that a CLI-extra requirement has package name tiktoken). The exact version is expected to change during dependency/security updates, so this becomes a change-detector rather than a packaging-contract test.
56646de to
2633981
Compare
Summary
tiktoken==0.12.0to thecliextra so fresh CLI installs include the dependency used byhermes toolstoken estimation.uv.lockfrom current main so[cli],[all],termux, andtermux-allresolve consistently through the existing extra graph.Behavior
This does not make token estimation a hard runtime requirement.
_estimate_tool_tokens()still importstiktokenlazily and returns{}when it is unavailable, so base/fallback installs can still open the tool configuration UI without token estimates.Related: #13481 touches the same dependency area, but this PR is intentionally scoped to the CLI token-estimation dependency declaration and preserves the existing fallback behavior.
Current-main refresh
uv.lockwith currentuv lockoutput.hermes_cli/tools_config.py; the lazy no-tiktokenfallback remains intact.Verification
tiktoken==0.12.0install: 13 passed.uv lock --check: passed.git diff --check: passed.