Skip to content

[WIP] Add simulation-based integration test for the correctness of the estimator #3214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
TsafrirA wants to merge 1 commit into
base: main
Choose a base branch
from
+94 −3
Draft

[WIP] Add simulation-based integration test for the correctness of the estimator #3214

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 94 additions & 3 deletions test/unit/executor_estimator/simulations/test_estimator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@

import numpy as np
from ddt import ddt
from qiskit.primitives import StatevectorEstimator
from qiskit import QuantumCircuit
from qiskit.primitives import ObservablesArray, StatevectorEstimator
from qiskit.providers.fake_provider import GenericBackendV2
from qiskit.quantum_info import SparsePauliOp
from qiskit.transpiler.preset_passmanagers import generate_preset_pass_manager

Expand All @@ -35,8 +37,12 @@


@ddt
class TestEstimator(IBMTestCase):
"""Tests Executor based EstimatorV2 implementation using simulator through local mode."""
class TestEstimatorErrorMitigationEfficacy(IBMTestCase):
"""Tests the efficacy of error mitigation in the Executor based EstimatorV2 implementation.

The tests use noisy simulations to verify that expectation values get closer to the ideal
result as the mitigation is applied.
"""

def setUp(self):
"""Test level setup."""
Expand Down Expand Up @@ -109,3 +115,88 @@ def test_result_quality_for_different_resilience_levels(self):

# Resilience level 2 should give very accurate expectation value:
np.testing.assert_array_less(errors[2], 0.025, err_msg=debug_message)


@ddt
class TestEstimatorCorrectness(IBMTestCase):
"""Tests the correctness of Executor based EstimatorV2 implementation.

The tests use noiseless simulations to verify the expectation values against
theoretical results.
"""

def setUp(self):
"""Test level setup."""
super().setUp()
self.backend = GenericBackendV2(5, noise_info=False, seed=972)
# self.backend = AerSimulator()
self.preset_pass_manager = generate_preset_pass_manager(
optimization_level=1, target=self.backend.target
)
self.tolerance = 2 # In terms of stansdard deviations

def test_vanilla_correctness(self):

@SamFerracin SamFerracin Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this test is a bit too optimistic when it comes to testing precision. Regardless of what our docs say or do not say (and we can improve that), given well-established statistical arguments, all we can guarantee is a precision p that is O(n_shots**-0.5). Any attempt at finding a constant C such that the errors are below C/n_shots**0.5 is doomed to fail, either because the test becomes too strict and fails often or because it is too lenient and tests nothing [which echoes things we said today].

In the absence of noise, we can make two promises:

  • Our estimates are unbiased
  • If we increase n_shots, our evs get closer to the ideal values, and our stds shrink.

Therefore, I would split this test into two tests that verify those two concerns. For example, for vanilla you could have:

  • A test where you fix a precision (default?) and try to see that the expectation values are unbiased. [For example, I have tried running your test with a single assert, namely assert np.all(np.array(errors) < 0.02), without fixing the seed. I had to remove that one observable for which we expect 0, which we know is problematic, but other than that, it always passes]
  • Another test where you fix one observable, and run with 2/3 values of precision, such as 0.1, 0.01, 0.001. Then you test that what you call errors go down and that the stds go down, like Rainer did in his other test

If you go this way, an additional bonus is that you will not need seeds for the tests to pass

@SamFerracin SamFerracin Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to this, you could consider parametrizing these tests over resilience levels 0, 1, and 2. So that we end up with 2 tests (times 3 -> 6 tests), and we only need to write two more for PEA and PEC

"""Tests the correctness of vanilla estimator (no error mitigation)."""
target_precision = 0.01

circuit = QuantumCircuit(3)
theta = np.pi / 8
circuit.rx(theta, 0)
circuit.rx(-np.pi / 2, 1)
circuit.ry(np.pi / 2, 2)

isa_circuit = self.preset_pass_manager.run(circuit)

observable_pairs: list[tuple[str, float]] = [
("IIZ", np.cos(theta)),
("IYZ", np.cos(theta)),
("XIZ", np.cos(theta)),
("1IZ", 0.5 * np.cos(theta)),
("IYr", np.sin(np.pi / 4 - theta / 2) ** 2),
("X+I", 0.5),
("0IZ", 0.5 * np.cos(theta)),
("IYl", np.cos(np.pi / 4 - theta / 2) ** 2),
("X-I", 0.5),
("ZXY", 0),
Comment on lines +151 to +160

@SamFerracin SamFerracin Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a great test suite!

]
observables = ObservablesArray.coerce([obs_string for obs_string, _ in observable_pairs])
isa_observables = observables.apply_layout(isa_circuit.layout)
pub = (isa_circuit, isa_observables)

options = EstimatorOptions(
resilience_level=0,
experimental={
"local_mode": True,
# "simulator_options": ExperimentalSimulatorOptions(seed_simulator=42),

@TsafrirA TsafrirA Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a random seed the test fails about half of the times. This is a bit more than I expected, I will go over the math again.

In any case, I think we should pick a passing seed, and aim for a relatively high failure rate with a random seed - it means the test is on the very edge, and would detect regressions well.

},
)
options.twirling.enable_gates = True
options.default_precision = target_precision

estimator = EstimatorV2(mode=self.backend, options=options)
result = estimator.run([pub]).result()
errors = [
abs(expected[1] - res) for expected, res in zip(observable_pairs, result[0].data.evs)
]

# Test the maximal deviation \ reported std
# Base distribution N[0, target_precision]
self.assertLess(max(errors), self.tolerance * target_precision)
self.assertLess(
max(result[0].data.ensemble_standard_error), self.tolerance * target_precision
)

# Test the mean of the deviations \ reported stds
# Distribution of the mean N[0.8 * target_precision, 0.6 * target_precision / np.sqrt(N)]
expected_mean = 0.8 * target_precision
expected_mean_std = 0.6 * target_precision / np.sqrt(len(observable_pairs))
self.assertLess(np.mean(errors), expected_mean + self.tolerance * expected_mean_std)
self.assertGreater(np.mean(errors), expected_mean - self.tolerance * expected_mean_std)
self.assertLess(
np.mean(result[0].data.ensemble_standard_error),
expected_mean + self.tolerance * expected_mean_std,
)
self.assertGreater(
np.mean(result[0].data.ensemble_standard_error),
expected_mean - self.tolerance * expected_mean_std,
)
Loading