Skip to content

Commit 8913c08

Browse files
chandrasekharan-zipstackclaude
chandrasekharan-zipstack
and
claude
authored
[MISC] Consolidate test reports into single PR comment and document libmagic1 dependency (#1598)
* Update tox.ini to document libmagic1 system dependency requirement * [CI] Consolidate test reports into single PR comment This change reduces noise in PR comments by combining multiple test reports (runner and sdk1) into a single consolidated comment. Changes: - Add combine-test-reports.sh script to merge test reports - Update CI workflow to combine reports before posting - Replace separate runner/sdk1 comment steps with single combined step - Summary section shows test counts (passed/failed/total) collapsed by default - Full detailed reports available in collapsible sections - Simplify job summary to use combined report Benefits: - Single PR comment instead of multiple separate comments - Cleaner PR comment section with less clutter - Easy-to-read summary with detailed inspection on demand - Maintains all existing test information 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * Fix test count extraction in combine-test-reports script The previous version was using text pattern matching which incorrectly extracted test counts. This fix properly parses the pytest-md-report markdown table format by: - Finding the table header row to determine column positions - Locating the TOTAL row (handles both TOTAL and **TOTAL** formatting) - Extracting values from the passed, failed, and SUBTOTAL columns - Using proper table parsing instead of pattern matching This resolves the issue where the summary showed incorrect counts (23 for both runner and sdk1 instead of the actual 11 and 66). Fixes: Test count summary in PR comments now shows correct values 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * Fix LaTeX formatting in pytest-md-report test count extraction pytest-md-report wraps all table values in LaTeX formatting: $$\textcolor{...}{\tt{VALUE}}$$ The previous fix attempted to parse the table but didn't handle the LaTeX formatting, causing it to extract 0 for all counts. Changes: - Add strip_latex() function to extract values from LaTeX wrappers - Update grep pattern to match TOTAL row specifically (not SUBTOTAL in header) - Apply LaTeX stripping to all extracted values before parsing This fix was tested locally with tox-generated reports and correctly shows: Runner=11 passed, SDK1=66 passed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Claude <[email protected]>
1 parent 2bef7d3 commit 8913c08

File tree

3 files changed

+184
-26
lines changed

3 files changed

+184
-26
lines changed

‎.github/scripts/combine-test-reports.sh‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
#!/bin/bash
2+
set -euo pipefail
3+
4+
# Script to combine multiple test reports into a single markdown file
5+
# Usage: ./combine-test-reports.sh
6+
7+
OUTPUT_FILE="combined-test-report.md"
8+
REPORTS=()
9+
10+
# Find all test report files
11+
for report in runner-report.md sdk1-report.md; do
12+
if [ -f "$report" ]; then
13+
REPORTS+=("$report")
14+
fi
15+
done
16+
17+
# Exit if no reports found
18+
if [ ${#REPORTS[@]} -eq 0 ]; then
19+
echo "No test reports found. Skipping report generation."
20+
exit 0
21+
fi
22+
23+
# Function to strip LaTeX formatting from pytest-md-report output
24+
# Converts $$\textcolor{...}{\tt{VALUE}}$$ to just VALUE
25+
strip_latex() {
26+
local text="$1"
27+
# Extract content between \tt{ and }}
28+
if [[ "$text" =~ \\tt\{([^}]+)\} ]]; then
29+
echo "${BASH_REMATCH[1]}"
30+
else
31+
echo "$text"
32+
fi
33+
}
34+
35+
# Function to extract test counts from pytest-md-report markdown table
36+
extract_test_counts() {
37+
local report_file=$1
38+
local passed=0
39+
local failed=0
40+
local total=0
41+
42+
# Find the header row to determine column positions
43+
local header_line=$(grep -E '^\|.*filepath' "$report_file" | head -1)
44+
45+
if [ -z "$header_line" ]; then
46+
echo "0:0:0"
47+
return
48+
fi
49+
50+
# Extract column names and find positions (strip LaTeX from headers)
51+
IFS='|' read -ra headers <<< "$header_line"
52+
local passed_col=-1
53+
local failed_col=-1
54+
local subtotal_col=-1
55+
56+
for i in "${!headers[@]}"; do
57+
local col=$(strip_latex "${headers[$i]}" | tr -d ' ' | tr '[:upper:]' '[:lower:]')
58+
case "$col" in
59+
passed) passed_col=$i ;;
60+
failed) failed_col=$i ;;
61+
subtotal|sub) subtotal_col=$i ;;
62+
esac
63+
done
64+
65+
# Find the TOTAL row (TOTAL appears in first column, not as SUBTOTAL in header)
66+
local total_line=$(grep -E '^\|.*\\tt\{TOTAL\}' "$report_file" | head -1)
67+
68+
if [ -z "$total_line" ]; then
69+
echo "0:0:0"
70+
return
71+
fi
72+
73+
# Parse the TOTAL row values
74+
IFS='|' read -ra values <<< "$total_line"
75+
76+
# Extract passed count (strip LaTeX and get number)
77+
if [ "$passed_col" -ge 0 ] && [ "$passed_col" -lt "${#values[@]}" ]; then
78+
local clean_value=$(strip_latex "${values[$passed_col]}")
79+
passed=$(echo "$clean_value" | tr -d ' ' | grep -oE '[0-9]+' | head -1 || echo "0")
80+
fi
81+
82+
# Extract failed count (strip LaTeX and get number)
83+
if [ "$failed_col" -ge 0 ] && [ "$failed_col" -lt "${#values[@]}" ]; then
84+
local clean_value=$(strip_latex "${values[$failed_col]}")
85+
failed=$(echo "$clean_value" | tr -d ' ' | grep -oE '[0-9]+' | head -1 || echo "0")
86+
fi
87+
88+
# Extract total from SUBTOTAL column (strip LaTeX and get number)
89+
if [ "$subtotal_col" -ge 0 ] && [ "$subtotal_col" -lt "${#values[@]}" ]; then
90+
local clean_value=$(strip_latex "${values[$subtotal_col]}")
91+
total=$(echo "$clean_value" | tr -d ' ' | grep -oE '[0-9]+' | head -1 || echo "0")
92+
fi
93+
94+
# If total is still 0, calculate from passed + failed
95+
if [ "$total" -eq 0 ]; then
96+
total=$((passed + failed))
97+
fi
98+
99+
echo "${total}:${passed}:${failed}"
100+
}
101+
102+
# Initialize the combined report with collapsed summary
103+
cat > "$OUTPUT_FILE" << 'EOF'
104+
# Test Results
105+
106+
<details open>
107+
<summary><b>Summary</b></summary>
108+
109+
EOF
110+
111+
# Extract and display summary for each report
112+
for report in "${REPORTS[@]}"; do
113+
report_name=$(basename "$report" .md)
114+
115+
# Convert report name to title case
116+
if [ "$report_name" = "runner-report" ]; then
117+
title="Runner Tests"
118+
elif [ "$report_name" = "sdk1-report" ]; then
119+
title="SDK1 Tests"
120+
else
121+
title="${report_name}"
122+
fi
123+
124+
# Extract counts
125+
counts=$(extract_test_counts "$report")
126+
IFS=':' read -r total passed failed <<< "$counts"
127+
128+
# Determine status icon
129+
if [ "$failed" -gt 0 ]; then
130+
status="❌"
131+
elif [ "$passed" -gt 0 ]; then
132+
status="✅"
133+
else
134+
status="⚠️"
135+
fi
136+
137+
echo "- ${status} **${title}**: ${passed} passed, ${failed} failed (${total} total)" >> "$OUTPUT_FILE"
138+
done
139+
140+
cat >> "$OUTPUT_FILE" << 'EOF'
141+
142+
</details>
143+
144+
---
145+
146+
EOF
147+
148+
# Combine all reports with collapsible sections
149+
for report in "${REPORTS[@]}"; do
150+
report_name=$(basename "$report" .md)
151+
152+
# Convert report name to title case
153+
if [ "$report_name" = "runner-report" ]; then
154+
title="Runner Tests"
155+
elif [ "$report_name" = "sdk1-report" ]; then
156+
title="SDK1 Tests"
157+
else
158+
title="${report_name}"
159+
fi
160+
161+
echo "<details>" >> "$OUTPUT_FILE"
162+
echo "<summary><b>${title} - Full Report</b></summary>" >> "$OUTPUT_FILE"
163+
echo "" >> "$OUTPUT_FILE"
164+
cat "$report" >> "$OUTPUT_FILE"
165+
echo "" >> "$OUTPUT_FILE"
166+
echo "</details>" >> "$OUTPUT_FILE"
167+
echo "" >> "$OUTPUT_FILE"
168+
done
169+
170+
echo "Combined test report created: $OUTPUT_FILE"
171+
echo "Included reports: ${REPORTS[*]}"

‎.github/workflows/ci-test.yaml‎

Lines changed: 11 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -48,36 +48,21 @@ jobs:
4848
run: |
4949
tox
5050
51-
- name: Render the Runner report to the PR
52-
uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
53-
if: always() && hashFiles('runner-report.md') != ''
54-
with:
55-
header: runner-test-report
56-
recreate: true
57-
path: runner-report.md
51+
- name: Combine test reports
52+
if: always() && (hashFiles('runner-report.md') != '' || hashFiles('sdk1-report.md') != '')
53+
run: |
54+
bash .github/scripts/combine-test-reports.sh
5855
59-
- name: Render the SDK1 report to the PR
56+
- name: Render combined test report to PR
6057
uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
61-
if: always() && hashFiles('sdk1-report.md') != ''
58+
if: always() && hashFiles('combined-test-report.md') != ''
6259
with:
63-
header: sdk1-test-report
60+
header: test-results
6461
recreate: true
65-
path: sdk1-report.md
62+
path: combined-test-report.md
6663

67-
- name: Output reports to the job summary when tests fail
64+
- name: Output combined report to job summary
65+
if: always() && hashFiles('combined-test-report.md') != ''
6866
shell: bash
6967
run: |
70-
if [ -f "runner-report.md" ]; then
71-
echo "<details><summary>Runner Test Report</summary>" >> $GITHUB_STEP_SUMMARY
72-
echo "" >> $GITHUB_STEP_SUMMARY
73-
cat "runner-report.md" >> $GITHUB_STEP_SUMMARY
74-
echo "" >> $GITHUB_STEP_SUMMARY
75-
echo "</details>" >> $GITHUB_STEP_SUMMARY
76-
fi
77-
if [ -f "sdk1-report.md" ]; then
78-
echo "<details><summary>SDK1 Test Report</summary>" >> $GITHUB_STEP_SUMMARY
79-
echo "" >> $GITHUB_STEP_SUMMARY
80-
cat "sdk1-report.md" >> $GITHUB_STEP_SUMMARY
81-
echo "" >> $GITHUB_STEP_SUMMARY
82-
echo "</details>" >> $GITHUB_STEP_SUMMARY
83-
fi
68+
cat combined-test-report.md >> $GITHUB_STEP_SUMMARY

‎tox.ini‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ allowlist_externals=
3939
uv
4040
pytest
4141
commands_pre =
42+
# System dependency required: libmagic1
43+
# Install with: sudo apt-get install -y libmagic1 (Ubuntu/Debian)
4244
# Install dependencies with test group
4345
uv sync --group test
4446
commands =

0 commit comments

Comments
 (0)