feat[notask]: automated OCR performance and quality reporting across all platforms#1625
Merged
Conversation
Made-with: Cursor
Separate user-provided clinical chemistry lab result image from the existing lab_results.png (which is a different document). Adds dedicated test and ground truth file for accurate quality evaluation. Made-with: Cursor
- Added steps to download Device Farm artifacts and extract performance reports for mobile tests in the integration workflow. - Updated performance report generation to include HTML and JSON outputs for mobile tests. - Refactored performance reporter utility to support runtime module configuration for Bare compatibility. This improves the visibility of performance metrics for mobile integration tests and ensures consistent reporting across platforms.
…utomation # Conflicts: # .github/workflows/integration-test-qvac-lib-infer-llamacpp-llm.yml # .github/workflows/integration-test-qvac-lib-infer-nmtcpp.yml # packages/qvac-lib-infer-onnx-tts/test/integration/addon.test.js
- Add liver_function_test.png (Simone's benchmark image) with ground truth - Create doctr-liver-function.test.js integration test - Regenerate integration.auto.cjs to include all 18 tests (was missing 3) - Add liver_function_test.png, clinical_chemistry.png, ct_scan_report.png to mobile CI testAssets copy step
The OIDC token from the initial auth may expire during the 2hr Device Farm test run. Add a fresh configure-aws-credentials step before the artifact download so list-jobs/list-artifacts calls succeed.
OCR tests complete well within the 2hr OIDC token lifetime.
Each test now runs twice — once with useGPU: false [CPU] and once with useGPU: true [GPU] — so the performance report clearly shows side-by-side CPU vs GPU timings per image. Labels include [CPU]/[GPU] tags which the reporter uses to set the execution_provider field in the JSON/HTML output.
- Use dynamic require via path.join for performance-reporter and quality-metrics modules so bare-pack cannot statically resolve them during mobile bundling (fixes MODULE_NOT_FOUND on iOS/Android) - Provide no-op fallbacks when modules are unavailable in mobile bundle - Replace 'liver' with 'pathology' in liver function test assertions since OCR reads the header as 'VER.FUNCTION' not 'LIVER' - Flush performance report from run-with-exit.js before writing exit code, since bare is killed by run-tests.sh before exit handler fires - Add debug logging to CI workflow HTML report generation step Made-with: Cursor
Add more reliably-detected words to each test to strengthen assertions — verified against actual CI OCR output. - liver_function_test: +8 words (biochemistry, hospital, conjugated, unconjugated, ratio, specimen, investigation, total) - lab_results: +10 words (medivista, hospital, biochemistry, department, arterial, gases, oxygen, electrolyte, metabolite, oximetry) - ct_scan: +8 words (allied, medical, center, patient, heart, trachea, vascular, normal) Made-with: Cursor
- Use matrix.os instead of matrix.platform in desktop performance report artifact names to avoid 409 Conflict when linux-x64 and linux-arm64 jobs upload to the same name - Re-encode ct_scan_report.png, liver_function_test.png, and clinical_chemistry.png as actual PNG format — they were JPEG data with .png extensions, causing AAPT2 to fail during Android resource compilation Made-with: Cursor
The mobile no-op fallback silently discarded all metrics because scripts/test-utils/ is outside the bare-pack bundle. Replace with a lightweight inline reporter that records metrics in memory and outputs [PERF_REPORT_START]...[PERF_REPORT_END] markers to console. On mobile, write markers after every test recording so the last (most complete) report is always available in Device Farm logs even if the process is killed before exit handlers fire. Update extract-from-log.js to find the last marker pair instead of the first, so it picks up the fully accumulated report. Made-with: Cursor
- ensureDoctrModels returns null on mobile when downloads fail instead of letting unhandled rejection SIGABRT BareKit - Medical test files (ct-scan, lab-results, clinical-chemistry, liver-function) skip gracefully when models unavailable - Mobile gets 5 retries with 10s backoff (was 3/5s) - downloadDoctrModel checks ocr-model-urls.json on mobile first for alternative URLs (future S3 presigned URL support) - Added DocTR model URLs to generate-ocr-presigned-urls.sh output Made-with: Cursor
The workflow only downloaded --type FILE artifacts (test spec output), but app console.log goes to device logcat which is --type LOG. Performance markers were never found because they live in DEVICE_LOG and LOGCAT artifacts. - Add --type LOG artifact download alongside --type FILE - Update extract-from-log.js to handle JSON logcat format (Device Farm stores logcat as JSON arrays with message fields) Made-with: Cursor
Root cause: console.log from BareKit goes to device logcat/syslog, NOT
to the Appium test spec output. The extract script was scanning
TESTSPEC_OUTPUT which never contained the markers.
Fix: Add wdio after hook that calls browser.getLogs() to pull device
logs into TESTSPEC_OUTPUT where extract-from-log.js can find them.
- Both Android/iOS wdio configs: add after hook using getLogs('logcat')
and getLogs('syslog') to dump perf markers to testspec console
- Android post_test: adb logcat -d backup dump to DEVICEFARM_LOG_DIR
- iOS post_test: search all files in DEVICEFARM_LOG_DIR for markers
- Also download --type LOG artifacts (DEVICE_LOG/LOGCAT) as fallback
Made-with: Cursor
Replace unreliable console.log-to-device-log chain with file-based approach: inline reporter writes perf JSON to disk, wdio after hook pulls it via Appium pullFile API. Android tries multiple sandbox paths, iOS uses known @bundleId:documents/ path. getLogs kept as tertiary fallback. Android post_test adds adb find+cat for path discovery. Made-with: Cursor
The cat of wdio.config.devicefarm.js in the testspec printed the
literal JS code console.log("[PERF_REPORT_START]"+json+"[PERF_REPORT_END]")
to TESTSPEC_OUTPUT. extract-from-log.js picked up "+json+" as valid JSON
(a JSON string literal) and wrote it as the report.
Two fixes:
- Remove cat of wdio config from testspec (eliminates false positive source)
- Add isValidReport() check in extract-from-log.js requiring schema_version
and results array (defense in depth against any future false positives)
Made-with: Cursor
writeReport() was only called inside _flushPerfReport() which runs on
process.on('exit') — unreliable on BareKit. The file never got written,
so pullFile had nothing to retrieve. Now writeReport() is called after
each test alongside writeToConsole(), progressively writing cumulative
results to global.testDir/perf-report.json.
Made-with: Cursor
…multi-device) Three issues caused the Android report to show only 3 of many results: 1. Logcat ~4KB line truncation: writeToConsole included the output field (hundreds of detected text strings per test), causing the JSON to exceed the logcat line limit. Stripped input/output fields from console payload; writeReport file still has the full data. 2. pullFile permission denied: Device Farm adb can't access app sandbox. Replaced adb find+cat with run-as <pkg> cat which executes as the app user and can read private files. Wraps output in PERF markers. 3. Single-device extraction: extract-from-log.js exited after first valid report. Now scans ALL files and picks the report with the most results. Made-with: Cursor
…e Farm
Two issues from run 614:
1. Logcat entries from getLogs contain embedded control characters
(ASCII 0x00-0x1F) that break JSON.parse. Added regex sanitization
in extractFromText to strip control chars before parsing.
2. The multi-line run-as block in post_test didn't expand ${PERF_JSON}
on Device Farm. Replaced with simple single-line commands: run-as
cat to a file in DEVICEFARM_LOG_DIR, then cat to stdout.
Made-with: Cursor
Device Farm organizes artifacts by device (e.g. Apple_iPhone_16_Pro/). Previously the extract script picked only the best single report and device.name was just "ios"/"android". Now when multiple devices are found, each gets its own performance-report.json tagged with the real device name, which the aggregate script discovers and groups by device. Made-with: Cursor
Previously quality evaluation was stubbed out on mobile because quality-metrics.js couldn't be loaded by bare-pack. Now the core algorithms (Levenshtein, CER, WER, keyword detection, KV accuracy) are inlined in the mobile fallback, and findGroundTruth reads .quality.json files from global.assetPaths. The workflow now also copies ground truth JSON files to testAssets for mobile bundling. Made-with: Cursor
…jection Three bugs identified during end-to-end mobile pipeline audit: 1. Workflow checked `if [ -f performance-report.json ]` at the root, but multi-device extraction writes per-device subdirectories only. Changed to `find` so aggregate.js runs for any layout. 2. Upload artifact paths only listed root-level files. Added glob to include per-device subdirectory JSONs. 3. Mobile reports lacked run_number (not available on Device Farm). Added --run-number flag to extract-from-log.js; workflow now passes github.run_number so aggregate HTML shows proper run columns. Made-with: Cursor
- Add test-groups.json to define perf (4 medical tests) and regular groups - Run each perf test 3 times for mean + stddev averaging - Schedule 2 parallel Device Farm runs per platform (perf + regular) - Add __TEST_FILTER__ + __MOCHA_GREP__ for app-level and mocha-level filtering - Monitor both runs concurrently, check both for pass/fail - Download artifacts from perf run only for report extraction - Fix duplicate run_number columns in aggregated reports Made-with: Cursor
…traction - Strip ALL ASCII control characters (0x00-0x1F) from JSON between perf markers, fixing "Bad control character at position 1004" on Android - Add --filter flag to extract-from-log.js to keep only results matching a regex pattern (e.g. medical test labels) - Add perf_report_filter to test-groups.json with medical test label pattern - Workflow passes --filter to extraction step so reports only contain perf test data even if non-perf tests also ran Made-with: Cursor
Report tables now display Run 1, Run 2, Run 3 columns (from the values array) instead of collapsing all iterations into a single Run #NNN column. Header shows CI run numbers and iteration count separately. CER/WER computation now sorts tokens alphabetically before comparison so reading-order differences between platforms (mobile bottom-to-top vs desktop top-to-bottom) do not inflate error rates. Mobile CER drops from ~81% to ~12%, matching desktop. Made-with: Cursor
Three root cause fixes for the Android performance reporting issues: 1. Mocha grep causing WDIO early exit: The grep patterns were function names from test-groups.json, NOT WDIO spec test titles. This caused WDIO to skip all spec tests and exit immediately without waiting for the app to finish running tests — producing incomplete reports (only 4 results captured instead of 15+). Fixed by setting grep to "." (match-all) and relying on post-extraction --filter for test selection. 2. JSON parse errors on Android logcat: When console.log output spans multiple logcat lines, Android injects timestamp/PID/tag prefixes into the middle of the JSON. Added regex to strip these prefixes during extraction. 3. Missing clean extraction source: Added marker-wrapped output in the Android post_test phase using run-as cat, providing a clean secondary extraction source when the WDIO after hook fails (e.g., app crash on Pixel 9 Pro). Made-with: Cursor
…eaving
The JSON parse errors (Expected ':' after property name) are caused by
WDIO debug-level logging interleaving with console.log output when the
JSON string is large. Node.js stdout.write splits large strings across
multiple chunks, and WDIO debug output gets inserted between chunks.
Fix: write pullFile JSON to a local file (perf-report-extract.json)
via fs.writeFileSync in the WDIO after hook, then output it cleanly
from the post_test phase using cat with markers. This completely avoids
the console.log interleaving problem.
Also removes the getLogs('logcat') marker printing from both Android
and iOS WDIO configs — these were a major source of corrupted duplicate
markers. The file-based approach is the primary extraction method now.
Added diagnostic char-level logging to extract-from-log.js to capture
what corruption pattern exists if any marker pairs still fail to parse.
Made-with: Cursor
…oling lab_results.quality.json described a KIMS-ICON Hospital report but the actual test image is from Medivista Central Hospital. Rewrote the entire ground truth (reference_text, keywords, key_values) to match the image. CER drops from 39.6% to ~18%. Added verify-quality.js script for independent metric auditing and expandable diagnostic details in the HTML quality report. Made-with: Cursor
- Escape regex special chars in filter pattern (extract-from-log.js) - Remove unused imports in verify-quality.js - Remove unused QUALITY_LABELS constant in utils.js
The perf_report_filter uses pipe alternation (a|b|c) which was broken by unconditional regex escaping. Now tries the pattern as-is first and only escapes if it's invalid regex.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
olyasir
previously approved these changes
Apr 16, 2026
…lert
Use split('|') + includes() instead of RegExp construction from CLI
argument. Eliminates the regex injection vector entirely while keeping
the same pipe-delimited filter syntax from test-groups.json.
olyasir
approved these changes
Apr 17, 2026
Contributor
Author
|
/review |
iancris
approved these changes
Apr 17, 2026
This was referenced Apr 22, 2026
Contributor
❌ E2E Mobile Test Results - iOSOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - AndroidOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - iOSOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - AndroidOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - iOSOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - AndroidOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - iOSOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - AndroidOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - iOSOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
Contributor
❌ E2E Mobile Test Results - AndroidOverall Status: FAILED Test Summary
Links
Automated E2E mobile testing powered by AWS Device Farm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What problem does this PR solve?
📝 How does it solve it?
Performance & Quality Reporting Pipeline
PerfReporterclass collects per-iteration timing and quality metrics (CER, WER, keyword detection, KV accuracy, word recognition rate) during test runsscripts/perf-report/tooling:aggregate.jscombines multi-device JSON reports,utils.jsgenerates HTML reports with heatmaps + embedded image thumbnails,extract-from-log.jsextracts reports from Device Farm logsscripts/test-utils/quality-metrics.jswith order-independent keyword/KV matchingMedical Image Test Coverage
Android Extraction Reliability
pullFile,mobile:shell, logcat parsing, chunked report reassembly/sdcard/Android/data/<pkg>/files/) for Pixel devices with strict scoped storageSingle Combined Report
combine-reportsjob produces a unified cross-platform summary (single$GITHUB_STEP_SUMMARY)HTML-Report-All-Platforms-{run}artifact: full combined HTML with heatmaps, thumbnails, diagnosticsHTML-Reports-Per-Device-{run}artifact: individual device HTML reports for deep dives🧪 How was it tested?