Benchmarking Framework

Benchmarking Framework

Section: Development · CLI-and-Developer-Tools · Home

The Benchmarking Framework measures performance, accuracy, and hardware viability of AI dubbing stages (ASR, TTS, audio prep, and more) across execution providers. Results inform starter-pack presets and regression tracking.

System Architecture and Components

The benchmarking system is divided into functional modules that handle the execution, measurement, and reporting of performance data.

flowchart TD
    CLI[Benchmark CLI] --> Core[Benchmark Core]
    Core --> Metrics[Metrics Engine]
    Core --> Scenarios[Benchmark Scenarios]
    Scenarios --> Pipeline[Application Pipeline]
    Metrics --> Reports[Report Generator]
    
    subgraph Execution
    Scenarios
    Pipeline
    end
    
    subgraph Analysis
    Metrics
    Reports
    end

Loading

The diagram above shows the data flow from user invocation through execution and into the reporting engine.

Core Modules

Component	Responsibility	File Path
Harness Core	Orchestrates the execution of benchmark runs and handles CLI interaction.	src/Trackdub.Benchmarks/
Application Benchmarking	Logic for running specific use cases and persisting results.	src/Trackdub.Application/Benchmarking/
Metrics Engine	Calculates technical performance indicators like RTF, Latency, and Error Rates.	src/Trackdub.Benchmarks/Metrics/
Report Generator	Serializes results into CSV, JSON, or SQLite formats.	src/Trackdub.Benchmarks/Reports/
Test Suite	Validates the correctness of metrics and reporting logic.	tests/Trackdub.Benchmarks.Tests/

Benchmark Scenarios

The framework supports several specific scenarios designed to test different segments of the dubbing pipeline.

Model Benchmarking

This scenario evaluates raw ONNX model performance. It measures cold load times, warmup cycles, and warm latency across different hardware providers. It supports testing specific model variants or running an aggregate test across all discovered variants for a given model scope.

Audio Preparation Benchmarking

Focuses on the accuracy of audio processing. It compares generated output against baseline fixtures to calculate deltas in Word Error Rate (WER) and Character Error Rate (CER), as well as turn fragmentation and speech coverage.

End-to-End Dubbing Benchmarking

A full-pipeline test that simulates a real dubbing job. It breaks down duration by pipeline stage (ASR, Translation, TTS, and Mixing) and provides a total execution time compared to the input media duration.

Metrics and Data Collection

The framework tracks a specific set of technical and qualitative metrics to assess the viability of a model or provider.

Performance Metrics

• Cold Load: The time taken to initialize the inference session for the first time.
• Warm Latency: The average, minimum, and maximum latency of inference after the warmup phase.
• Real-Time Factor (RTF): The ratio of processing time to the duration of the processed audio.
• Model Size: Physical storage size of the ONNX model in bytes.

Accuracy Metrics

Used primarily in "Audio Prep" and ASR scenarios to measure transcription and segmentation quality.

Metric	Description
WER Delta	Change in Word Error Rate compared to the baseline fixture.
CER Delta	Change in Character Error Rate compared to the baseline fixture.
Speaker Drift	Discrepancy in the number of speakers identified compared to the expected count.
Speech Coverage	Delta in the total seconds of speech identified within a clip.

Hardware and Provider Interaction

A key purpose of the framework is to validate "Execution Providers" (EP). On Windows, the framework focuses on Windows ML as the integration layer, testing catalog EPs such as TensorRT RTX and MIGraphX, while treating DirectML as a legacy fallback.

sequenceDiagram
    participant Harness as Benchmark Harness
    participant EP as Execution Provider
    participant Model as ONNX Model
    
    Harness->>EP: Request Session (Provider Preference)
    EP-->>Harness: Session Initialized
    Note right of Harness: Measure Cold Load
    Harness->>Model: Warmup Runs
    Note right of Harness: Measure Warmup Time
    loop Measured Runs
        Harness->>Model: Execute Inference
        Model-->>Harness: Result
    end
    Note right of Harness: Calculate Latency Stats

Loading

Sequence of a standard model benchmark execution loop.

CLI Usage and Configuration

The benchmark harness is primarily interacted with via the Trackdub.Benchmarks project.

# Ingest and report status of a media file
dotnet run --project src/Trackdub.Benchmarks ingest --project <path> --media <file>

# Benchmark a specific ONNX model with a hardware provider
dotnet run --project src/Trackdub.Benchmarks --model <path> --provider trt-rtx --runs 10

# Run audio preparation benchmark against a manifest
dotnet run --project src/Trackdub.Benchmarks audio-prep --manifest <path> --format both

Configuration Options

• Provider: Options include cpu, auto, dml, migraphx, or trt-rtx.
• Windows ML Device Policy: Controls GPU selection behavior (e.g., max-performance, prefer-npu, max-efficiency).
• Runs: Specifies the number of measured runs to perform for statistical averaging.

Conclusion

The Benchmarking Framework is essential for maintaining the high performance standards required for local AI dubbing. By isolating and measuring the latency of individual pipeline stages and the accuracy of audio-to-text transitions, it provides the data necessary to make informed decisions about model selection and hardware optimization. This ensures that Trackdub remains a responsive and viable workstation across diverse platform configurations.