reduce list tool context by default

[nabuskey]

🔄 Pull Request

📝 Description

Currently, list_jobs and list_stages returns all available data by default. This could overwhelm the context easily for large jobs. This sets default number of items to be returned by each tool. In addition, list_jobs returns stage IDs grouped by their status.

🎯 Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 📖 Documentation update
• 🧪 Test improvement
• 🔧 Refactoring (no functional changes)

🧪 Testing

• ✅ All existing tests pass (task test)
• 🔬 Tested with MCP Inspector
• 📊 Tested with sample Spark data
• 🚀 Tested with real Spark History Server (if applicable)

[andrewdkim47]

Thanks, i think this is a good addition. How did we come up with 20 as the default? Also what if user wants to increase above 20?

[nabuskey]

Thanks, i think this is a good addition. How did we come up with 20 as the default? Also what if user wants to increase above 20?

From my testing with ~3GB event logs, 20 costs about ~9000 tokens.
As for changing the default value, we have several places where we put default values. We need to think about how best go about that.

[andreyvelich]

/hold

[vara-bonthu]

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vara-bonthu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vara-bonthu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment