Add tests executing a circuit on each fake backend#4018
Merged
Conversation
Right now there is nothing in validating that we're able to use all the fake backends defined in qiskit.test.mock.backends. So when we make changes to the fake backends (or backends in general) we don't know if they work until someone goes to use them. This was especially needed for Qiskit#4016. During the development of that PR all the tests pass but the docs builds failed because a fake backend is used in a jupyter-execute section. This commit addresses the situation by adding a simple test that just runs a simple deterministic circuit that just has some easy 1q gate run optimizations on the fake backend at each optimization level (to ensure we use all the backend properties needed for execution). This will provide us coverage over the fake backends to ensure that they at least are functional as we make changes to terra. As part of the development of these tests, 2 missing properties from the new qobj classes were added because the tests caught that they were missing. Additionally the properties json file for fake yorktown had to be updated, the properties there were incorrect when run with aer (and thus a noise model) all the results were invalid. Fixes Qiskit#3735
mtreinish
requested review from
1ucian0,
ajavadia,
chriseclectic,
ewinston,
jyu00,
kdk and
taalexander
as code owners
To fully test the fake backend properties we need to use aer to build a noise model and run the simulation with that. This commit adds qiskit-aer to the ci configuration so that it is a vailable for these new tests. In general we don't want to rely on aer (because of the potential circular dependency) but for this case it's seems worth the tradeoff.
kdk
reviewed
Mar 25, 2020
kdk
reviewed
Mar 25, 2020
| optimization_level=[0, 1, 2, 3]) | ||
| def test_circuit_on_fake_backend(self, fake_backends, optimization_level): | ||
| backend = getattr(backends, fake_backends)() | ||
| if not HAS_AER and backend.configuration().n_qubits > 20: |
Member
There was a problem hiding this comment.
It's a bit unfortunate that we need to use (and install) Aer for these tests (though I understand why its necessary). Maybe it makes sense to update the BasicAer n_qubits check to be number of used qubits.
Member
Author
There was a problem hiding this comment.
Yeah, I wasn't happy with adding aer to the testing. But to do this well it was needed.
The issue with adjusting basicaer to avoid this is that it's looking at the transpiled qobj to determine the number of qubits required: https://github.com/Qiskit/qiskit-terra/blob/master/qiskit/providers/basicaer/qasm_simulator.py#L622. With the fake backends (and specifically in this case for FakeRochester), the physical circuit will have an ancilla with no operations set for all the unused qubits since it's pulling that from the fake backend. To workaround this we'd have to disassemble the qobj and determine that 51 qubits are idle. It's not too much work, but definitely outside the scope of this PR.
When running simulations with a noise model the tests were failing on fake_openpulse_2q because it was set as being too noisy. This commit drops the error rates set in the properties so we can run a circuit with a single x gate and get the expected result.
The fake_qasm_simulator is set as a simulator, but the base fake_provider class was not setup to handle a backend which was a simulator. This commit handles this by having the fake_qasm_simulator return no properties, and also updates the run() method to not try and build a noise model for a simulator.
This commit adjusts the tests to use the fake provider to get a list of backends instead of scanning the backends module for Fake* objects.
For fake_qasm_simulator's run() method top work when aer is not installed it needs to be usable with the basic aer simulator. However, right now the basis set was closer to aer's. This commit adjusts it to be basic aer's which is LCD for running in a simulator.
kdk
approved these changes
Mar 25, 2020
ajavadia
approved these changes
Mar 25, 2020
faisaldebouni
pushed a commit
to faisaldebouni/qiskit-terra
that referenced
this pull request
Aug 5, 2020
* Add tests executing a circuit on each fake backend Right now there is nothing in validating that we're able to use all the fake backends defined in qiskit.test.mock.backends. So when we make changes to the fake backends (or backends in general) we don't know if they work until someone goes to use them. This was especially needed for Qiskit#4016. During the development of that PR all the tests pass but the docs builds failed because a fake backend is used in a jupyter-execute section. This commit addresses the situation by adding a simple test that just runs a simple deterministic circuit that just has some easy 1q gate run optimizations on the fake backend at each optimization level (to ensure we use all the backend properties needed for execution). This will provide us coverage over the fake backends to ensure that they at least are functional as we make changes to terra. As part of the development of these tests, 2 missing properties from the new qobj classes were added because the tests caught that they were missing. Additionally the properties json file for fake yorktown had to be updated, the properties there were incorrect when run with aer (and thus a noise model) all the results were invalid. Fixes Qiskit#3735 * Add qiskit-aer to ci jobs To fully test the fake backend properties we need to use aer to build a noise model and run the simulation with that. This commit adds qiskit-aer to the ci configuration so that it is a vailable for these new tests. In general we don't want to rely on aer (because of the potential circular dependency) but for this case it's seems worth the tradeoff. * Reduce error rates on fake_openpulse_2q backend When running simulations with a noise model the tests were failing on fake_openpulse_2q because it was set as being too noisy. This commit drops the error rates set in the properties so we can run a circuit with a single x gate and get the expected result. * Handle simulators properly for fake backends The fake_qasm_simulator is set as a simulator, but the base fake_provider class was not setup to handle a backend which was a simulator. This commit handles this by having the fake_qasm_simulator return no properties, and also updates the run() method to not try and build a noise model for a simulator. * Adjust tests to use fake provider This commit adjusts the tests to use the fake provider to get a list of backends instead of scanning the backends module for Fake* objects. * Change fake_qasm_simulator basis to basic aer's For fake_qasm_simulator's run() method top work when aer is not installed it needs to be usable with the basic aer simulator. However, right now the basis set was closer to aer's. This commit adjusts it to be basic aer's which is LCD for running in a simulator. * Cleanup arg name for test
ElePT
pushed a commit
to ElePT/qiskit-ibm-provider
that referenced
this pull request
Oct 9, 2023
* Add tests executing a circuit on each fake backend Right now there is nothing in validating that we're able to use all the fake backends defined in qiskit.test.mock.backends. So when we make changes to the fake backends (or backends in general) we don't know if they work until someone goes to use them. This was especially needed for Qiskit/qiskit#4016. During the development of that PR all the tests pass but the docs builds failed because a fake backend is used in a jupyter-execute section. This commit addresses the situation by adding a simple test that just runs a simple deterministic circuit that just has some easy 1q gate run optimizations on the fake backend at each optimization level (to ensure we use all the backend properties needed for execution). This will provide us coverage over the fake backends to ensure that they at least are functional as we make changes to terra. As part of the development of these tests, 2 missing properties from the new qobj classes were added because the tests caught that they were missing. Additionally the properties json file for fake yorktown had to be updated, the properties there were incorrect when run with aer (and thus a noise model) all the results were invalid. Fixes Qiskit/qiskit#3735 * Add qiskit-aer to ci jobs To fully test the fake backend properties we need to use aer to build a noise model and run the simulation with that. This commit adds qiskit-aer to the ci configuration so that it is a vailable for these new tests. In general we don't want to rely on aer (because of the potential circular dependency) but for this case it's seems worth the tradeoff. * Reduce error rates on fake_openpulse_2q backend When running simulations with a noise model the tests were failing on fake_openpulse_2q because it was set as being too noisy. This commit drops the error rates set in the properties so we can run a circuit with a single x gate and get the expected result. * Handle simulators properly for fake backends The fake_qasm_simulator is set as a simulator, but the base fake_provider class was not setup to handle a backend which was a simulator. This commit handles this by having the fake_qasm_simulator return no properties, and also updates the run() method to not try and build a noise model for a simulator. * Adjust tests to use fake provider This commit adjusts the tests to use the fake provider to get a list of backends instead of scanning the backends module for Fake* objects. * Change fake_qasm_simulator basis to basic aer's For fake_qasm_simulator's run() method top work when aer is not installed it needs to be usable with the basic aer simulator. However, right now the basis set was closer to aer's. This commit adjusts it to be basic aer's which is LCD for running in a simulator. * Cleanup arg name for test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Right now there is nothing in validating that we're able to use all the
fake backends defined in qiskit.test.mock.backends. So when we make
changes to the fake backends (or backends in general) we don't know if
they work until someone goes to use them. This was especially needed
for #4016. During the development of that PR all the tests pass but the
docs builds failed because a fake backend is used in a jupyter-execute
section. This commit addresses the situation by adding a simple test
that just runs a simple deterministic circuit that just has some easy
1q gate run optimizations on the fake backend at each optimization level
(to ensure we use all the backend properties needed for execution). This
will provide us coverage over the fake backends to ensure that they at
least are functional as we make changes to terra.
As part of the development of these tests, 2 missing properties from the
new qobj classes were added because the tests caught that they were
missing. Additionally the properties json file for fake yorktown had to
be updated, the properties there were incorrect when run with aer
(and thus a noise model) all the results were invalid.
Details and comments
Fixes #3735