Skip to content

QVAC-17318 infra: add Device Farm artifact downloads to all mobile integration test pipelines#1638

Merged
tobi-legan merged 3 commits into
mainfrom
QVAC-17318/add-devicefarm-artifact-downloads
Apr 17, 2026
Merged

QVAC-17318 infra: add Device Farm artifact downloads to all mobile integration test pipelines#1638
tobi-legan merged 3 commits into
mainfrom
QVAC-17318/add-devicefarm-artifact-downloads

Conversation

@tobi-legan

@tobi-legan tobi-legan commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • Team members without AWS Device Farm console access cannot view test artifacts (logs, videos, results) from mobile integration test CI runs
  • Only 4 of 9 mobile test workflows had artifact downloads; decoder-audio, llamacpp-embed, diffusion, and ocr-onnx were missing them entirely
  • Video artifacts on Android significantly inflate artifact size (~5 GB uncompressed) without providing much value since C++ logs are available there

📝 How does it solve it?

  • Added Device Farm artifact download steps to 4 workflows that were missing them:
    • integration-mobile-test-qvac-lib-decoder-audio.yml
    • integration-mobile-test-qvac-lib-infer-llamacpp-embed.yml
    • integration-mobile-test-lib-infer-diffusion.yml
    • integration-mobile-test-ocr-onnx.yml
  • Each gets 3 new steps: refresh AWS OIDC credentials, download Device Farm logs (FILE + LOG artifacts from runs → jobs → suites), and upload as GitHub Actions artifacts with 30-day retention
  • Added Android video exclusion to all 9 workflows (including the 5 that already had downloads: LLM, NMT, TTS, Whisper, Parakeet):
    • Skips artifacts named "Video" or with .mp4 extension when matrix.platform == Android
    • Keeps video for iOS since C++ logs are not available there yet
    • Applies to both suite-level and job-level FILE artifacts

🧪 How was it tested?

  • Manually trigger each of the 9 Mobile Integration Tests (*) workflows from the Actions tab using this branch
  • Verify that devicefarm-logs-* artifacts appear in the workflow run
  • Verify that Android runs show "Skipped (video)" messages and no .mp4 files in artifacts
  • Verify that iOS runs still include video artifacts

Workflows changed (9 files, +644 lines)

Workflow Change
Decoder Audio +139 lines — new artifact download steps
LlamaCPP Embed +139 lines — new artifact download steps
Diffusion +139 lines — new artifact download steps
OCR-ONNX +139 lines — new artifact download steps
LLM +13 lines — Android video exclusion
NMTCPP +18 lines — Android video exclusion + info message
TTS +19 lines — Android video exclusion + info message
Whispercpp +19 lines — Android video exclusion + info message
Parakeet +19 lines — Android video exclusion + info message

@tobi-legan
QVAC-17318 infra: add Device Farm artifact downloads to all mobile in…
2da28e9 
…tegration test pipelines

Add automatic downloading and uploading of Device Farm artifacts (logs,
test specs, results) as GitHub Actions artifacts across all mobile
integration test workflows. Excludes video artifacts on Android to reduce
artifact size, keeping video only for iOS where C++ logs are unavailable.

New artifact downloads: decoder-audio, llamacpp-embed, diffusion, ocr-onnx.
Android video exclusion: all 9 workflows including llm, nmt, tts, whisper, parakeet.

Made-with: Cursor
@tobi-legan tobi-legan requested review from a team as code owners April 16, 2026 20:06
@tobi-legan
Merge origin/main — resolve OCR-ONNX conflict
ed30b64 
Keep main's OCR-specific artifact download (perf report extraction,
diagnostics) since it's more tailored than the generic download.

Made-with: Cursor
@tobi-legan
QVAC-17318 infra: add generic log download alongside perf download fo…
b2c16f7 
…r OCR

Keep main's perf-specific artifact download (for perf report extraction)
and add the generic Device Farm log download for both perf + regular runs.
Includes Android video exclusion. Both artifact sets uploaded separately
(df-raw-logs for perf, devicefarm-logs for full logs).

Made-with: Cursor
@github-actions

github-actions Bot commented Apr 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ✅ APPROVED

**Requirements:**
- 1 Team Member approval ✅ (1/1)
- 1 Team Lead OR Management approval ✅ (1/1)



---
*This comment is automatically updated when reviews change.*

@github-actions

github-actions Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

❌ E2E Mobile Test Results - iOS

Overall Status: FAILED
Device Farm Result: UNKNOWN
Platform: iOS
Addon: @qvac/translation-nmtcpp
PR: #1638
Commit: 5a5ea01

Test Summary

Metric Count
Total Tests 0
✅ Passed 0
❌ Failed 0
⏭️ Skipped 0

Links

Automated E2E mobile testing powered by AWS Device Farm
Tests located in: test/mobile/

@github-actions

github-actions Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

❌ E2E Mobile Test Results - Android

Overall Status: FAILED
Device Farm Result: UNKNOWN
Platform: Android
Addon: @qvac/translation-nmtcpp
PR: #1638
Commit: 5a5ea01

Test Summary

Metric Count
Total Tests 0
✅ Passed 0
❌ Failed 0
⏭️ Skipped 0

Links

Automated E2E mobile testing powered by AWS Device Farm
Tests located in: test/mobile/

@github-actions

github-actions Bot commented May 17, 2026

Copy link
Copy Markdown
Contributor

❌ E2E Mobile Test Results - iOS

Overall Status: FAILED
Device Farm Result: UNKNOWN
Platform: iOS
Addon: @qvac/translation-nmtcpp
PR: #1638
Commit: 807d7a8

Test Summary

Metric Count
Total Tests 0
✅ Passed 0
❌ Failed 0
⏭️ Skipped 0

Links

Automated E2E mobile testing powered by AWS Device Farm
Tests located in: test/mobile/

@github-actions

github-actions Bot commented May 17, 2026

Copy link
Copy Markdown
Contributor

❌ E2E Mobile Test Results - Android

Overall Status: FAILED
Device Farm Result: UNKNOWN
Platform: Android
Addon: @qvac/translation-nmtcpp
PR: #1638
Commit: 807d7a8

Test Summary

Metric Count
Total Tests 0
✅ Passed 0
❌ Failed 0
⏭️ Skipped 0

Links

Automated E2E mobile testing powered by AWS Device Farm
Tests located in: test/mobile/

@github-actions

github-actions Bot commented May 17, 2026

Copy link
Copy Markdown
Contributor

❌ E2E Mobile Test Results - iOS

Overall Status: FAILED
Device Farm Result: UNKNOWN
Platform: iOS
Addon: @qvac/translation-nmtcpp
PR: #1638
Commit: 807d7a8

Test Summary

Metric Count
Total Tests 0
✅ Passed 0
❌ Failed 0
⏭️ Skipped 0

Links

Automated E2E mobile testing powered by AWS Device Farm
Tests located in: test/mobile/

@github-actions

github-actions Bot commented May 17, 2026

Copy link
Copy Markdown
Contributor

❌ E2E Mobile Test Results - Android

Overall Status: FAILED
Device Farm Result: UNKNOWN
Platform: Android
Addon: @qvac/translation-nmtcpp
PR: #1638
Commit: 807d7a8

Test Summary

Metric Count
Total Tests 0
✅ Passed 0
❌ Failed 0
⏭️ Skipped 0

Links

Automated E2E mobile testing powered by AWS Device Farm
Tests located in: test/mobile/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Proletter Proletter Proletter approved these changes

@Victor-Rodzko Victor-Rodzko Victor-Rodzko approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tobi-legan @Proletter @Victor-Rodzko