Add reusable gateway workflow harness with mock LLM server

[ilblackdragon]

Summary

• add reusable in-process OpenAI-compatible mock LLM server for integration tests
• add gateway workflow harness for chat + routine + webhook end-to-end testing
• add integration test exercising webhook workflow through gateway/status APIs

Validation

• cargo test --test gateway_workflow_integration --features libsql

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the testing infrastructure by introducing a robust framework for end-to-end integration tests. It provides a mock OpenAI server to simulate LLM interactions and a comprehensive gateway workflow harness, enabling thorough validation of chat functionalities, routine creation and execution, and webhook processing within the system.

Highlights

• Mock LLM Server: Added a reusable in-process OpenAI-compatible mock LLM server for integration tests, allowing controlled simulation of LLM interactions.
• Gateway Workflow Harness: Introduced a comprehensive gateway workflow harness designed for end-to-end testing of chat, routine creation, and webhook functionalities.
• Integration Test: Implemented a new integration test that thoroughly exercises webhook workflows through the gateway and status APIs, validating the full system path.

Changelog

• tests/gateway_workflow_integration.rs
  • Added a new integration test to validate the end-to-end workflow involving chat, routine creation, event emission, and webhook processing using the new harness and mock LLM server.
• tests/support/gateway_workflow_harness.rs
  • Added a new module defining GatewayWorkflowHarness, a utility for setting up a full gateway environment with an agent, database, and webhook server for testing purposes.
  • Included a MockGithubWebhookTool to simulate GitHub webhook events within the harness.
• tests/support/mock_openai_server.rs
  • Added a new module implementing MockOpenAiServer, an in-process server that mimics the OpenAI API for chat completions, allowing controlled responses for testing.
• tests/support/mod.rs
  • Updated the module to expose the newly added gateway_workflow_harness and mock_openai_server modules.

Activity

• No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive integration test harness for gateway workflows, including a mock OpenAI-compatible server. The changes are well-structured and provide valuable end-to-end testing capabilities. I've identified a couple of minor areas for improvement to enhance test robustness and code clarity.

[gemini-code-assist]

Using a fixed tokio::time::sleep can lead to flaky tests if the asynchronous operation takes longer than the specified duration. It's better to use a polling loop with a timeout to wait for the expected state change. This will make the test more robust.

Suggested change

	tokio::time::sleep(Duration::from_millis(500)).await;
	let runs_after = harness.routine_runs(routine_id).await;
	let after_count = runs_after["runs"]
	.as_array()
	.map(|a| a.len())
	.unwrap_or_default();
	assert!(
	after_count > before_count,
	"expected routine runs to increase after webhook; before={before_count}, after={after_count}"
	);
	let mut after_count = before_count;
	for _ in 0..50 { // 5-second timeout
	let runs_after = harness.routine_runs(routine_id).await;
	after_count = runs_after["runs"]
	.as_array()
	.map(|a| a.len())
	.unwrap_or_default();
	if after_count > before_count {
	break;
	}
	tokio::time::sleep(Duration::from_millis(100)).await;
	}
	assert!(
	after_count > before_count,
	"expected routine runs to increase after webhook; before={before_count}, after={after_count}"
	);

[ilblackdragon]

Fixed in e642e53 — replaced the fixed sleep(500ms) with a polling loop (up to 50 iterations × 100ms), matching the wait_for_turns() pattern already used elsewhere in the test.

[gemini-code-assist]

The _notify_rx variable is unused. It's good practice to rename it to _ to signify that it's intentionally unused and to avoid compiler warnings.

Suggested change

	let (notify_tx, _notify_rx) = tokio::sync::mpsc::channel(16);
	let (notify_tx, _) = tokio::sync::mpsc::channel(16);

[ilblackdragon]

Fixed in e642e53 — removed the entire redundant RoutineEngine block (Agent::run() creates its own), so this variable no longer exists.

[zmanian]

Review: Gateway Workflow Harness + Mock LLM Server

Overall this is solid test infrastructure. The mock OpenAI server is well-designed with a clean builder pattern, the rule-matching system is intuitive, and the harness provides good helper methods for common gateway operations. A few issues to address:

Duplicate TestChannelHandle

gateway_workflow_harness.rs defines its own TestChannelHandle (lines 34-89) that is nearly identical to the one in test_rig.rs (lines 41-96). The only difference is the harness version returns "gateway" from name() instead of delegating to self.inner.name(). This should be extracted into a shared helper in test_channel.rs -- perhaps TestChannel::into_channel_handle(name: &str) or a generic TestChannelHandle that accepts a name override. ~55 lines of pure boilerplate duplication.

Redundant RoutineEngine creation

In gateway_workflow_harness.rs lines 241-253, a temporary RoutineEngine is created with RoutineConfig::default() and passed to register_routine_tools(). But Agent::run() (in agent_loop.rs:442-454) creates its own RoutineEngine with the actual config and re-registers the routine tools, overwriting these registrations. This means:

1. The temporary engine is created and immediately becomes dead weight.
2. _notify_rx is dropped immediately, making the temporary engine's notification channel broken.
3. The config used is RoutineConfig::default() rather than the config.routines that was carefully set up above -- an inconsistency that would matter if the engine weren't replaced.

Consider removing this block entirely (the agent's run() handles it), or add a comment explaining it's needed for some pre-run() setup reason I'm not seeing.

Flaky sleep pattern (agree with Gemini's comment)

Line 130 of the test:

tokio::time::sleep(Duration::from_millis(500)).await;

followed by a single check is a flake risk. The Gemini bot's suggestion to use a polling loop with timeout is correct. The test already has wait_for_turns() which demonstrates the right pattern -- apply the same approach here.

register_job_tools passes a fresh ContextManager

Line 224-232: A new ContextManager is created (ctx_mgr) and passed to register_job_tools, but then components.context_manager (a different instance from AppBuilder) is passed to Agent::new() at line 331. This means job tools reference a different context manager than the agent. If any job tool reads/writes context state, it will be invisible to the agent. Either use components.context_manager for both, or document why the separation is intentional.

Minor items

• #![allow(dead_code)] at file level: Fine for test support modules, but consider scoping it more narrowly if most of the API is actually used. The #[allow(dead_code)] at the top of both files suppresses warnings for genuinely unused code that might indicate over-engineering.

• Error handling in test helpers: Methods like create_thread(), send_chat(), etc. use .expect() chains which is fine for test code. Good pattern.

• Mock server Drop impl: Both MockOpenAiServer and GatewayWorkflowHarness have proper Drop impls that abort spawned tasks -- good defensive design to prevent test leaks.

• ToolOutput::success vs ToolOutput::text: The MockGithubWebhookTool uses ToolOutput::success() -- verify this matches the real GitHub tool's output format so the routine engine processes it correctly (the emit_events key in the output must be parsed somewhere upstream).

Design feedback

The harness is 572 lines for what is essentially "start a gateway + agent + webhook server with a mock LLM." That's a lot of wiring. Some of this complexity is inherent (the real main.rs startup is similarly complex), but it suggests AppBuilder could benefit from a build_test_harness() convenience method that handles the GatewayState/channel/agent wiring internally. Not a blocker for this PR, but worth considering as a follow-up to keep test infrastructure maintainable.

[zmanian]

Review: Gateway Workflow Harness + Mock LLM Server

Overall this is solid test infrastructure. The mock OpenAI server is well-designed with a clean builder pattern, the rule-matching system is intuitive, and the harness provides good helper methods for common gateway operations. A few issues to address:

Duplicate TestChannelHandle

gateway_workflow_harness.rs defines its own TestChannelHandle (lines 34-89) that is nearly identical to the one in test_rig.rs (lines 41-96). The only difference is the harness version returns "gateway" from name() instead of delegating to self.inner.name(). This should be extracted into a shared helper in test_channel.rs -- perhaps TestChannel::into_channel_handle(name) or a generic TestChannelHandle that accepts a name override. ~55 lines of pure boilerplate duplication.

Redundant RoutineEngine creation

In gateway_workflow_harness.rs lines 241-253, a temporary RoutineEngine is created with RoutineConfig::default() and passed to register_routine_tools(). But Agent::run() (in agent_loop.rs:442-454) creates its own RoutineEngine with the actual config and re-registers the routine tools, overwriting these registrations. This means:

1. The temporary engine is created and immediately becomes dead weight
2. _notify_rx is dropped immediately, making the temporary engine's notification channel broken
3. The config used is RoutineConfig::default() rather than the config.routines that was carefully set up above

Consider removing this block entirely (the agent's run() handles it), or add a comment explaining why it is needed for some pre-run() setup reason.

Flaky sleep pattern

Line 130 of the test uses tokio::time::sleep(Duration::from_millis(500)).await followed by a single check -- this is a flake risk. The test already has wait_for_turns() which demonstrates the right polling-with-timeout pattern. Apply the same approach for waiting on routine run counts.

register_job_tools passes a fresh ContextManager

A new ContextManager is created (ctx_mgr) and passed to register_job_tools, but then components.context_manager (a different instance from AppBuilder) is passed to Agent::new(). This means job tools reference a different context manager than the agent. If any job tool reads/writes context state, it will be invisible to the agent. Either use components.context_manager for both, or document why the separation is intentional.

Minor items

• Mock server Drop impls on both MockOpenAiServer and GatewayWorkflowHarness properly abort spawned tasks -- good defensive design
• Error handling via .expect() chains in test helpers is fine and appropriate for test code
• MockGithubWebhookTool uses ToolOutput::success() -- verify the emit_events key in the output matches what the real webhook tool produces so the routine engine processes it correctly
• The harness is 572 lines of wiring. Some complexity is inherent, but it suggests AppBuilder could benefit from a build_test_harness() convenience method as a follow-up

None of these are critical blockers, but the duplicate TestChannelHandle and the redundant RoutineEngine creation should be cleaned up before merge to keep the test infrastructure maintainable.

[ilblackdragon]

All review feedback addressed in e642e53:

1. Duplicate TestChannelHandle — Extracted into test_channel.rs as a shared TestChannelHandle with with_name() constructor for name overrides. Both test_rig.rs and gateway_workflow_harness.rs now import it. Net -65 lines.

2. Redundant RoutineEngine creation — Removed the entire block (lines 240-253). Confirmed that Agent::run() creates its own RoutineEngine, calls register_routine_tools(), and populates the routine_engine_slot — the harness block was pure dead weight.

3. Flaky sleep pattern — Replaced sleep(500ms) + single check with a polling loop (50 iterations × 100ms = 5s timeout), matching the wait_for_turns() pattern.

4. Fresh ContextManager — Now uses components.context_manager instead of creating a separate instance, so job tools and agent share the same context.

5. build_test_harness() follow-up — Good idea, noted for a future PR.