feat: Add batch job submission for task graph

[sitaowang1998]

Description

#75 adds batch job submission for starting a task function as a job. This pr adds batch job submission for starting a task graph as a job.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• GitHub workflows pass
• Unit tests pass in dev container
• Integration tests pass in dev container

Summary by CodeRabbit

• New Features

  • Introduced batch processing for job submissions, enabling more efficient handling of multiple tasks concurrently with robust error reporting.

• Tests

  • Expanded testing scenarios with dedicated functions to validate batch submissions and ensure reliable task outcomes.

[coderabbitai]

Walkthrough

The changes introduce support for batch job submissions in the Driver::start method. A conditional structure checks for the presence of a batch; if it exists, add_job_batch is called, otherwise, it defaults to add_job. Error handling is implemented to throw a ConnectionException upon failure. Additionally, the test suite is refactored to use multiple test functions, enhancing organization and clarity. A new test function for batch submissions is included, ensuring the functionality is thoroughly validated.

Changes

File(s)	Change Summary
src/.../Driver.hpp	Enhanced Driver::start to handle batch submissions by checking for m_batch. Calls add_job_batch if a batch is present; otherwise, it defaults to add_job, including error handling with ConnectionException.
tests/.../client-test.cpp	Refactored the main function into multiple specific test functions for better organization. Added tests for various scenarios, including a new function for testing batch submissions. The original error handling and job execution logic are preserved within these new test functions.

Sequence Diagram(s)

sequenceDiagram
    participant D as Driver::start
    participant MS as MetadataStorage
    D->>D: Check if m_batch is provided
    alt Batch exists
        D->>MS: add_job_batch(job_id, graph, m_batch)
    else No batch available
        D->>MS: add_job(job_id, graph)
    end
    MS-->>D: Return status/result
    alt Error encountered
        D->>D: Throw ConnectionException
    else Successful submission
        D->>D: Continue execution
    end

Loading

sequenceDiagram
    participant CT as Client Test (main)
    participant D as Driver::start
    participant MS as MetadataStorage
    CT->>CT: Create task graph (sum_graph with sum_test binding)
    loop For each job in batch (cBatchSize)
        CT->>D: Start job with sum_graph and parameters
        D->>MS: Process job submission (conditional branch)
        MS-->>D: Return job status/ID
    end
    CT->>CT: Wait for all jobs to complete
    CT->>CT: Verify job status and expected result

Loading

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a6fc40f and 7e2bbdb.

📒 Files selected for processing (1)

• tests/client/client-test.cpp (5 hunks)

⏰ Context from checks skipped due to timeout of 90000ms (3)

• GitHub Check: non-storage-unit-tests (ubuntu-22.04)
• GitHub Check: non-storage-unit-tests (ubuntu-24.04)
• GitHub Check: lint

🔇 Additional comments (5)

tests/client/client-test.cpp (5)

45-66: Well-structured test function extraction.

The extraction of the graph testing logic into a dedicated function improves code organization and readability. Each test now has a clear scope and return value.

---

68-86: Good separation of test cases into individual functions.

Breaking down the monolithic test into separate test functions is an excellent refactoring approach. Each function now has a single responsibility, making the tests more maintainable and easier to understand.

Also applies to: 88-98, 100-115, 117-131

---

133-155: Well-implemented batch submission test.

The function batch submission test properly demonstrates the batch API pattern with begin/end batch calls and appropriate result verification.

---

157-180: Excellent implementation of the new graph batch submission test.

The new test function correctly implements batch job submission for task graphs, following the same pattern as the function batch submission test. The test verifies that:

1. Multiple graph jobs can be submitted in a batch
2. All jobs complete successfully
3. Each job produces the expected result

This test properly covers the new functionality introduced in the PR.

---

184-245: Clean main function refactoring with proper error handling.

The refactored main function maintains all the original functionality while calling each test function sequentially. The error handling is thorough, checking the result of each test and returning immediately if any test fails.

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai plan to trigger planning for file edits and PR creation.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.