Skip to content

feat: add structured output support via json_schema #683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bogini wants to merge 9 commits into from
Closed

feat: add structured output support via json_schema #683

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e600a51
feat: add structured output support
bogini Nov 18, 2025
e935838
fix: address PR #683 review feedback
bogini Nov 18, 2025
dcee434
fix: workaround GitHub Actions composite action output limitation
bogini Nov 18, 2025
8cd2cc1
refactor: remove individual field outputs, keep only structured_outpu…
bogini Nov 18, 2025
ec3a934
docs: update structured output documentation for JSON-only approach
bogini Nov 18, 2025
f551cdf
fix: remove double error reporting in parseAndSetStructuredOutputs
bogini Nov 18, 2025
bf8f85c
docs: fix incorrect field naming restrictions in base-action/action.yml
bogini Nov 18, 2025
9d3bab5
test: add proper test coverage for parseAndSetStructuredOutputs
bogini Nov 18, 2025
84265a4
update
bogini Nov 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
349 changes: 349 additions & 0 deletions .github/workflows/test-structured-output.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,349 @@
name: Test Structured Outputs

on:
push:
branches:
- main
pull_request:
workflow_dispatch:

permissions:
contents: read

jobs:
test-basic-types:
name: Test Basic Type Conversions
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Test with explicit values
id: test
uses: ./base-action
with:
prompt: |
Run this command: echo "test"

Then return EXACTLY these values:
- text_field: "hello"
- number_field: 42
- boolean_true: true
- boolean_false: false
json_schema: |
{
"type": "object",
"properties": {
"text_field": {"type": "string"},
"number_field": {"type": "number"},
"boolean_true": {"type": "boolean"},
"boolean_false": {"type": "boolean"}
},
"required": ["text_field", "number_field", "boolean_true", "boolean_false"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
allowed_tools: "Bash"

@claude claude Bot Nov 18, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configuration Issue: The allowed_tools input is used throughout this test file but is not defined in base-action/action.yml. This means the tests won't work as expected since GitHub Actions won't pass this value to the environment.

Recommendation: Either:

  1. Add allowed_tools input to base-action/action.yml, OR
  2. Use claude_args: "--allowed-tools Bash" instead


- name: Verify outputs
run: |
# Parse the structured_output JSON
OUTPUT='${{ steps.test.outputs.structured_output }}'

# Test string pass-through
TEXT_FIELD=$(echo "$OUTPUT" | jq -r '.text_field')
if [ "$TEXT_FIELD" != "hello" ]; then
echo "❌ String: expected 'hello', got '$TEXT_FIELD'"
exit 1
fi

# Test number → string conversion
NUMBER_FIELD=$(echo "$OUTPUT" | jq -r '.number_field')
if [ "$NUMBER_FIELD" != "42" ]; then
echo "❌ Number: expected '42', got '$NUMBER_FIELD'"
exit 1
fi

# Test boolean → "true" conversion
BOOLEAN_TRUE=$(echo "$OUTPUT" | jq -r '.boolean_true')
if [ "$BOOLEAN_TRUE" != "true" ]; then
echo "❌ Boolean true: expected 'true', got '$BOOLEAN_TRUE'"
exit 1
fi

# Test boolean → "false" conversion
BOOLEAN_FALSE=$(echo "$OUTPUT" | jq -r '.boolean_false')
if [ "$BOOLEAN_FALSE" != "false" ]; then
echo "❌ Boolean false: expected 'false', got '$BOOLEAN_FALSE'"
exit 1
fi

echo "✅ All basic type conversions correct"

test-complex-types:
name: Test Arrays and Objects
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Test complex types
id: test
uses: ./base-action
with:
prompt: |
Run: echo "ready"

Return EXACTLY:
- items: ["apple", "banana", "cherry"]
- config: {"key": "value", "count": 3}
- empty_array: []
json_schema: |
{
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {"type": "string"}
},
"config": {"type": "object"},
"empty_array": {"type": "array"}
},
"required": ["items", "config", "empty_array"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
allowed_tools: "Bash"

- name: Verify JSON stringification
run: |
# Parse the structured_output JSON
OUTPUT='${{ steps.test.outputs.structured_output }}'

# Arrays should be JSON stringified
if ! echo "$OUTPUT" | jq -e '.items | length == 3' > /dev/null; then
echo "❌ Array not properly formatted"
echo "$OUTPUT" | jq '.items'
exit 1
fi

# Objects should be JSON stringified
if ! echo "$OUTPUT" | jq -e '.config.key == "value"' > /dev/null; then
echo "❌ Object not properly formatted"
echo "$OUTPUT" | jq '.config'
exit 1
fi

# Empty arrays should work
if ! echo "$OUTPUT" | jq -e '.empty_array | length == 0' > /dev/null; then
echo "❌ Empty array not properly formatted"
echo "$OUTPUT" | jq '.empty_array'
exit 1
fi

echo "✅ All complex types handled correctly"

test-edge-cases:
name: Test Edge Cases
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Test edge cases
id: test
uses: ./base-action
with:
prompt: |
Run: echo "test"

Return EXACTLY:
- zero: 0
- empty_string: ""
- negative: -5
- decimal: 3.14
json_schema: |
{
"type": "object",
"properties": {
"zero": {"type": "number"},
"empty_string": {"type": "string"},
"negative": {"type": "number"},
"decimal": {"type": "number"}
},
"required": ["zero", "empty_string", "negative", "decimal"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
allowed_tools: "Bash"

- name: Verify edge cases
run: |
# Parse the structured_output JSON
OUTPUT='${{ steps.test.outputs.structured_output }}'

# Zero should be "0", not empty or falsy
ZERO=$(echo "$OUTPUT" | jq -r '.zero')
if [ "$ZERO" != "0" ]; then
echo "❌ Zero: expected '0', got '$ZERO'"
exit 1
fi

# Empty string should be empty (not "null" or missing)
EMPTY_STRING=$(echo "$OUTPUT" | jq -r '.empty_string')
if [ "$EMPTY_STRING" != "" ]; then
echo "❌ Empty string: expected '', got '$EMPTY_STRING'"
exit 1
fi

# Negative numbers should work
NEGATIVE=$(echo "$OUTPUT" | jq -r '.negative')
if [ "$NEGATIVE" != "-5" ]; then
echo "❌ Negative: expected '-5', got '$NEGATIVE'"
exit 1
fi

# Decimals should preserve precision
DECIMAL=$(echo "$OUTPUT" | jq -r '.decimal')
if [ "$DECIMAL" != "3.14" ]; then
echo "❌ Decimal: expected '3.14', got '$DECIMAL'"
exit 1
fi

echo "✅ All edge cases handled correctly"

test-name-sanitization:
name: Test Output Name Sanitization
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Test special characters in field names
id: test
uses: ./base-action
with:
prompt: |
Run: echo "test"
Return EXACTLY: {test-result: "passed", item_count: 10}
json_schema: |
{
"type": "object",
"properties": {
"test-result": {"type": "string"},
"item_count": {"type": "number"}
},
"required": ["test-result", "item_count"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
allowed_tools: "Bash"

- name: Verify sanitized names work
run: |
# Parse the structured_output JSON
OUTPUT='${{ steps.test.outputs.structured_output }}'

# Hyphens should be preserved in the JSON
TEST_RESULT=$(echo "$OUTPUT" | jq -r '.["test-result"]')
if [ "$TEST_RESULT" != "passed" ]; then
echo "❌ Hyphenated name failed: expected 'passed', got '$TEST_RESULT'"
exit 1
fi

# Underscores should work
ITEM_COUNT=$(echo "$OUTPUT" | jq -r '.item_count')
if [ "$ITEM_COUNT" != "10" ]; then
echo "❌ Underscore name failed: expected '10', got '$ITEM_COUNT'"
exit 1
fi

echo "✅ Name sanitization works"

test-execution-file-structure:
name: Test Execution File Format
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Run with structured output
id: test
uses: ./base-action
with:
prompt: "Run: echo 'complete'. Return: {done: true}"
json_schema: |
{
"type": "object",
"properties": {
"done": {"type": "boolean"}
},
"required": ["done"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
allowed_tools: "Bash"

- name: Verify execution file contains structured_output
run: |
FILE="${{ steps.test.outputs.execution_file }}"

# Check file exists
if [ ! -f "$FILE" ]; then
echo "❌ Execution file missing"
exit 1
fi

# Check for structured_output field
if ! jq -e '.[] | select(.type == "result") | .structured_output' "$FILE" > /dev/null; then
echo "❌ No structured_output in execution file"
cat "$FILE"
exit 1
fi

# Verify the actual value
DONE=$(jq -r '.[] | select(.type == "result") | .structured_output.done' "$FILE")
if [ "$DONE" != "true" ]; then
echo "❌ Wrong value in execution file"
exit 1
fi

echo "✅ Execution file format correct"

test-summary:
name: Summary
runs-on: ubuntu-latest
needs:
- test-basic-types
- test-complex-types
- test-edge-cases
- test-name-sanitization
- test-execution-file-structure
if: always()
steps:
- name: Generate Summary
run: |
echo "# Structured Output Tests (Optimized)" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "Fast, deterministic tests using explicit prompts" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Test | Result |" >> $GITHUB_STEP_SUMMARY
echo "|------|--------|" >> $GITHUB_STEP_SUMMARY
echo "| Basic Types | ${{ needs.test-basic-types.result == 'success' && '✅ PASS' || '❌ FAIL' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Complex Types | ${{ needs.test-complex-types.result == 'success' && '✅ PASS' || '❌ FAIL' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Edge Cases | ${{ needs.test-edge-cases.result == 'success' && '✅ PASS' || '❌ FAIL' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Name Sanitization | ${{ needs.test-name-sanitization.result == 'success' && '✅ PASS' || '❌ FAIL' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Execution File | ${{ needs.test-execution-file-structure.result == 'success' && '✅ PASS' || '❌ FAIL' }} |" >> $GITHUB_STEP_SUMMARY

# Check if all passed
ALL_PASSED=${{
needs.test-basic-types.result == 'success' &&
needs.test-complex-types.result == 'success' &&
needs.test-edge-cases.result == 'success' &&
needs.test-name-sanitization.result == 'success' &&
needs.test-execution-file-structure.result == 'success'
}}

if [ "$ALL_PASSED" = "true" ]; then
echo "" >> $GITHUB_STEP_SUMMARY
echo "## ✅ All Tests Passed" >> $GITHUB_STEP_SUMMARY
else
echo "" >> $GITHUB_STEP_SUMMARY
echo "## ❌ Some Tests Failed" >> $GITHUB_STEP_SUMMARY
exit 1
fi
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ A general-purpose [Claude Code](https://claude.ai/code) action for GitHub PRs an
- 💬 **PR/Issue Integration**: Works seamlessly with GitHub comments and PR reviews
- 🛠️ **Flexible Tool Access**: Access to GitHub APIs and file operations (additional tools can be enabled via configuration)
- 📋 **Progress Tracking**: Visual progress indicators with checkboxes that dynamically update as Claude completes tasks
- 📊 **Structured Outputs**: Get validated JSON results that automatically become GitHub Action outputs for complex automations
- 🏃 **Runs on Your Infrastructure**: The action executes entirely on your own GitHub runner (Anthropic API calls go to your chosen provider)
- ⚙️ **Simplified Configuration**: Unified `prompt` and `claude_args` inputs provide clean, powerful configuration aligned with Claude Code SDK

Expand Down
6 changes: 6 additions & 0 deletions action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,10 @@ inputs:
description: "Newline-separated list of Claude Code plugin marketplace Git URLs to install from (e.g., 'https://github.com/user/marketplace1.git\nhttps://github.com/user/marketplace2.git')"
required: false
default: ""
json_schema:
Comment thread
bogini marked this conversation as resolved.
description: "JSON schema for structured output validation. When provided, Claude will return validated JSON matching this schema, and the action will automatically set GitHub Action outputs for each field."

@claude claude Bot Nov 18, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation: Missing Important Limitations

The description should document:

  1. Reserved output field names that will be skipped
  2. Size limits and their behavior

Suggested update:

description: "JSON schema for structured output validation. When provided, Claude will return validated JSON matching this schema, and the action will automatically set GitHub Action outputs for each field. Note: Field names 'conclusion' and 'execution_file' are reserved and will be skipped. Individual output fields are limited to 1MB; objects/arrays exceeding this will be skipped, primitives will be truncated."

required: false
default: ""
Comment on lines +116 to +119

@claude claude Bot Nov 18, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation: The json_schema input description is minimal and doesn't explain critical behavior that users need to know:

  1. Dynamic outputs are strings: All values are converted to strings (booleans → "true"/"false", objects → JSON strings)
  2. 1MB size limit: Large outputs are truncated (primitives) or skipped (objects/arrays)
  3. Name sanitization: Special characters become underscores, must start with letter/underscore
  4. Type conversions: null → "", undefined → ""

Recommend expanding the description:

description: |
  JSON schema for structured output validation. When provided, Claude returns validated JSON 
  matching this schema, and the action automatically sets GitHub Action outputs for each field.
  
  Important: All outputs are strings (access via steps.id.outputs.field_name). Type conversions:
  - Booleans: true → "true", false → "false"
  - Numbers: 42 → "42"
  - Objects/Arrays: JSON stringified
  - null: empty string ""
  
  Limits: 1MB per field. Large objects/arrays are skipped. Field names sanitized to alphanumeric/underscore/hyphen.


outputs:
execution_file:
Expand Down Expand Up @@ -174,6 +178,7 @@ runs:
TRACK_PROGRESS: ${{ inputs.track_progress }}
ADDITIONAL_PERMISSIONS: ${{ inputs.additional_permissions }}
CLAUDE_ARGS: ${{ inputs.claude_args }}
JSON_SCHEMA: ${{ inputs.json_schema }}
ALL_INPUTS: ${{ toJson(inputs) }}

- name: Install Base Action Dependencies
Expand Down Expand Up @@ -228,6 +233,7 @@ runs:
INPUT_SHOW_FULL_OUTPUT: ${{ inputs.show_full_output }}
INPUT_PLUGINS: ${{ inputs.plugins }}
INPUT_PLUGIN_MARKETPLACES: ${{ inputs.plugin_marketplaces }}
JSON_SCHEMA: ${{ inputs.json_schema }}

# Model configuration
GITHUB_TOKEN: ${{ steps.prepare.outputs.GITHUB_TOKEN }}
Expand Down
Loading
Loading