Live verification against a real workspace, plus the docs the README promised - #8
Merged
Merged
Conversation
Ran the tool against a real Azure Databricks workspace and a real Genie Agent over a synthetic revenue table. agents list, ask with --show-sql, --format json, --format csv, the unknown-agent path, and pack run all exercised end to end. Live verification found two things tests had not. DATABRICKS_TOKEN was documented but never read. The DI always constructed the Databricks CLI broker, so the environment path that docs/limitations.md told unattended users to rely on did not exist. It now takes precedence, which is also what made this verification possible without a browser login. Console output encoding was inherited from the host. On Windows that is a legacy code page that cannot represent most non-ASCII, so a currency symbol in machine-readable output would degrade to '?' silently. Pinned to UTF-8. Live evidence worth keeping: decimals reached CSV, JSON and Markdown byte-identical to what Databricks returned, which is the processing-integrity claim confirmed outside its own tests; column type came back DECIMAL(22,2), so capturing type_text rather than only type_name preserved precision and scale; and 2>/dev/null left pure CSV on stdout, confirming the diagnostics split. Genie also answered one pack question with a clarifying question rather than an answer. That completes successfully and exits 0, so an unattended pack can produce a report whose answer is a question. Recorded in limitations rather than papered over with heuristics that would guess wrong either way. Also: 28 output-fidelity tests covering numeric precision, non-ASCII through every writer, null versus empty string, spreadsheet formula defusal and control-character neutralisation; the command, authentication, Question Pack and troubleshooting guides the README linked to but did not have; and the threat model SECURITY.md promised. Every relative link in the repository now resolves.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs the tool against a real Azure Databricks workspace and a real Genie Agent, and fixes what that found.
Two bugs only live use could find
DATABRICKS_TOKENwas documented but never read. The DI always built the Databricks CLI broker, so the environment pathdocs/limitations.mdtold unattended users to rely on simply did not work. Fixed — and it is what made this verification possible without a browser login.Console encoding was inherited from the host. On Windows that is a legacy code page, so non-ASCII in machine-readable output degraded to
?silently. Pinned to UTF-8.What was verified live
agents list(table + json),askwith--show-sql,--format json,--format csv, the unknown-agent path, andpack run— all end to end against a real Genie Agent.Three claims confirmed outside their own tests:
DECIMAL(22,2)— capturingtype_textrather than onlytype_namepreserved precision and scale.2>/dev/nullleft pure CSV on stdout, confirming the diagnostics split.A new finding
Genie answered one pack question with a clarifying question rather than an answer. That completes successfully and exits 0, so an unattended pack can produce a report whose answer is a question. Recorded in limitations rather than papered over with a heuristic that would guess wrong in one direction or the other.
Also
28 output-fidelity tests (numeric precision, non-ASCII through every writer, null vs empty string, spreadsheet formula defusal, control characters). The command / authentication / Question Pack / troubleshooting guides the README linked to but did not have. The threat model
SECURITY.mdpromised. Every relative link in the repository now resolves.89 tests green.