test: adding a basic manual e2e test for mcp server #320
Conversation
✅ Docs preview has no changesThe preview was not built because there were no changes. Build ID: 01a47c523d5cdcf685c6f73c |
|
❌ Changeset file missing for PR All changes should include an associated changeset file. |
| 5. On script exit the generated config is cleaned up. | ||
|
|
||
| ### Example run: | ||
| To run the tests for `local-operations` simply run `./run_tests.sh local-operations` |
There was a problem hiding this comment.
slight improvement suggestion. It might be nice if ./run_tests.sh by default runs all the tests?
There was a problem hiding this comment.
I could but the problem is the mcp-server-tester sometimes doesn't exit. This would hang the script.
There was a problem hiding this comment.
Linking the issue here for future reference. steviec/mcp-server-tester#30
| @@ -0,0 +1,21 @@ | |||
| endpoint: https://thespacedevs-production.up.railway.app/ | |||
| transport: | |||
| type: stdio | |||
There was a problem hiding this comment.
Will this test suite eventually support other transports?
There was a problem hiding this comment.
I'm not sure, currently it does not work with streamable-http.
| @@ -0,0 +1,65 @@ | |||
| tools: | |||
| expected_tool_list: ['introspect', 'execute', 'search', 'validate', 'SearchUpcomingLaunches', 'ExploreCelestialBodies', 'GetAstronautDetails', 'GetAstronautsCurrentlyInSpace'] | |||
There was a problem hiding this comment.
Would it be feasible to actually call one of the graphql operation tools in this first round of tests?
There was a problem hiding this comment.
Yeah we can add a test for that. It would just be like the tests below, but calling something like SearchUpcomingLaunches instead.
There was a problem hiding this comment.
I'll see if I can add one in a follow up PR
e2e/mcp-server-tester/run_tests.sh
Outdated
| sed "s|<test-dir>|$safe_dir|g" "$TEMPLATE_PATH" > "$GEN_CONFIG" | ||
|
|
||
| # Run the command | ||
| exec npx -y mcp-server-tester tools "$TESTS" --server-config "$GEN_CONFIG" No newline at end of file |
There was a problem hiding this comment.
last question, what does exec provide here when you can also run npx directly in the bash script?
There was a problem hiding this comment.
Actually you'r eright, this should not use exec. the trap will not work if this is exec.
Adding some basic e2e tests using mcp-server-tester. Currently the tool does not always exit (ctrl+c is sometimes needed) so this should be run manually.
How to run tests?
Added a script
run_tests.sh(may need to runchmod +xto run it) to run tests. Basic usage found via./run_tests.sh -h. The script does the following:apollo-mcp-serverbinary exists. If not, it builds the binary viacargo build --release.mcp-server-tester) and replaces all<test-dir>placeholders with the test directory value. Generates this test server config file and places it in a temp location.mcp-server-testervianpx.Example run:
To run the tests for
local-operationssimply run./run_tests.sh local-operations