-
Notifications
You must be signed in to change notification settings - Fork 3k
Base pass manager bug fixes #11178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Base pass manager bug fixes #11178
Changes from 3 commits
bf34b13
908e03d
f623be6
6838e55
846e416
4f4aba8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,77 @@ | ||||||||||||||||||||||||||||
| # This code is part of Qiskit. | ||||||||||||||||||||||||||||
| # | ||||||||||||||||||||||||||||
| # (C) Copyright IBM 2023 | ||||||||||||||||||||||||||||
| # | ||||||||||||||||||||||||||||
| # This code is licensed under the Apache License, Version 2.0. You may | ||||||||||||||||||||||||||||
| # obtain a copy of this license in the LICENSE.txt file in the root directory | ||||||||||||||||||||||||||||
| # of this source tree or at http://www.apache.org/licenses/LICENSE-2.0. | ||||||||||||||||||||||||||||
| # | ||||||||||||||||||||||||||||
| # Any modifications or derivative works of this code must retain this | ||||||||||||||||||||||||||||
| # copyright notice, and modified files need to carry a notice indicating | ||||||||||||||||||||||||||||
| # that they have been altered from the originals. | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| """Pass manager test cases.""" | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| import io | ||||||||||||||||||||||||||||
| import re | ||||||||||||||||||||||||||||
| from itertools import zip_longest | ||||||||||||||||||||||||||||
| from logging import StreamHandler, getLogger | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| from qiskit.passmanager import PassManagerState, WorkflowStatus, PropertySet | ||||||||||||||||||||||||||||
| from qiskit.test import QiskitTestCase | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| class PassManagerTestCase(QiskitTestCase): | ||||||||||||||||||||||||||||
| """Test case for the pass manager module.""" | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| def setUp(self): | ||||||||||||||||||||||||||||
| super().setUp() | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| self.output = io.StringIO() | ||||||||||||||||||||||||||||
| self.state = PassManagerState( | ||||||||||||||||||||||||||||
| workflow_status=WorkflowStatus(), | ||||||||||||||||||||||||||||
| property_set=PropertySet(), | ||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| logger = getLogger() | ||||||||||||||||||||||||||||
| self.addCleanup(logger.setLevel, logger.level) | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| logger.setLevel("DEBUG") | ||||||||||||||||||||||||||||
| logger.addHandler(StreamHandler(self.output)) | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| def assertLogEqual(self, func, expected_lines, *args, exception_type=None): | ||||||||||||||||||||||||||||
| """Execute provided function and verify logger. | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| Args: | ||||||||||||||||||||||||||||
| func (Callable): Function to test. | ||||||||||||||||||||||||||||
| expected_lines (List[str]): Expected log output. | ||||||||||||||||||||||||||||
| args (Any): Arguments to the function. | ||||||||||||||||||||||||||||
| exception_type (Type): Optional. Expected exception for error handling. | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| Returns: | ||||||||||||||||||||||||||||
| Any: Output values from the function. | ||||||||||||||||||||||||||||
| """ | ||||||||||||||||||||||||||||
| if exception_type: | ||||||||||||||||||||||||||||
| self.assertRaises(exception_type, func, *args) | ||||||||||||||||||||||||||||
| out = None | ||||||||||||||||||||||||||||
| else: | ||||||||||||||||||||||||||||
| out = func(*args) | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
| self.output.seek(0) | ||||||||||||||||||||||||||||
| recorded_lines = [line.rstrip() for line in self.output.readlines()] | ||||||||||||||||||||||||||||
| for i, (expected, recorded) in enumerate(zip_longest(expected_lines, recorded_lines)): | ||||||||||||||||||||||||||||
| expected = expected or "" | ||||||||||||||||||||||||||||
| recorded = recorded or "" | ||||||||||||||||||||||||||||
| if not re.fullmatch(expected, recorded): | ||||||||||||||||||||||||||||
| raise AssertionError( | ||||||||||||||||||||||||||||
| f"Log didn't match. Mismatch found at line #{i}.\n\n" | ||||||||||||||||||||||||||||
| f"Expected:\n{self._format_log(expected_lines)}\n" | ||||||||||||||||||||||||||||
| f"Recorded:\n{self._format_log(recorded_lines)}" | ||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||
| return out | ||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this code copied from somewhere? The exception handling within this function shouldn't be necessary; wrapping a "standard" call to this function in the It'd also likely be better if this function was a context-manager wrapper around
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I needed this to check qiskit/qiskit/passmanager/base_tasks.py Lines 103 to 115 in dabc5e6
The
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Right, but if they're both used as context managers, the clean-up code from both (the bit that contains the assertions) will still run in both correctly; we can use the context-manager stack to manage things arbitrarily, rather than needing hard-coded functions.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In very very very rough form, what I'm meaning is something like: import contextlib]
import unittest
class MyTestCas(unittest.TestCase):
@contextlib.contextmanager
def assertLogContains(self, expected_lines):
with self.assertLogs() as cm:
yield cm
recorded_lines = cm.output
for i, (expected, recorded) in enumerate(zip(expected_lines, recorded_lines)):
# ... your existing handlingthen I think it should work in test cases as: def test_logs_while_raising(self):
with self.assertLogContains(["blah blah"]):
getLogger().log("blah blah")
with self.assertRaises(Exception):
raise Exceptionand that test would pass.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (possibly the log-checking code should be in a
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, that's neat. I'll try
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done in 846e416. I needed some ugly hack because of this logic.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmmm it didn't work. The default formatter is bit ugly but maybe okey.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, that's unfortunately gross of the standard library. Unfortunately, I think we're going to have problems with the private method: the Is there any way we can write the test enforcing the default logging format, so we can avoid needing the private access? |
||||||||||||||||||||||||||||
| def _format_log(self, lines): | ||||||||||||||||||||||||||||
| out = "" | ||||||||||||||||||||||||||||
| for i, line in enumerate(lines): | ||||||||||||||||||||||||||||
| out += f"#{i:02d}: {line}\n" | ||||||||||||||||||||||||||||
| return out | ||||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| # This code is part of Qiskit. | ||
| # | ||
| # (C) Copyright IBM 2023 | ||
| # | ||
| # This code is licensed under the Apache License, Version 2.0. You may | ||
| # obtain a copy of this license in the LICENSE.txt file in the root directory | ||
| # of this source tree or at http://www.apache.org/licenses/LICENSE-2.0. | ||
| # | ||
| # Any modifications or derivative works of this code must retain this | ||
| # copyright notice, and modified files need to carry a notice indicating | ||
| # that they have been altered from the originals. | ||
| # pylint: disable=missing-class-docstring | ||
|
|
||
| """Pass manager test cases.""" | ||
|
|
||
| from test.python.passmanager import PassManagerTestCase | ||
|
|
||
| from logging import getLogger | ||
|
|
||
| from qiskit.passmanager import GenericPass | ||
| from qiskit.passmanager.compilation_status import RunState | ||
|
|
||
|
|
||
| class TestGenericPass(PassManagerTestCase): | ||
| """Tests for the GenericPass subclass.""" | ||
|
|
||
| def test_run_task(self): | ||
| """Test case: Simple successful task execution.""" | ||
|
|
||
| class Task(GenericPass): | ||
| def run(self, passmanager_ir): | ||
| return passmanager_ir | ||
|
|
||
| task = Task() | ||
| data = "test_data" | ||
| expected = [r"Pass: Task - (\d*\.)?\d+ \(ms\)"] | ||
| self.assertLogEqual(task.execute, expected, data, self.state) | ||
| self.assertEqual(self.state.workflow_status.count, 1) | ||
| self.assertIn(task, self.state.workflow_status.completed_passes) | ||
| self.assertEqual(self.state.workflow_status.previous_run, RunState.SUCCESS) | ||
|
|
||
| def test_failure_task(self): | ||
| """Test case: Log is created regardless of success.""" | ||
|
|
||
| class TestError(Exception): | ||
| pass | ||
|
|
||
| class RaiseError(GenericPass): | ||
| def run(self, passmanager_ir): | ||
| raise TestError() | ||
|
|
||
| task = RaiseError() | ||
| data = "test_data" | ||
| expected = [r"Pass: RaiseError - (\d*\.)?\d+ \(ms\)"] | ||
| self.assertLogEqual(task.execute, expected, data, self.state, exception_type=TestError) | ||
| self.assertEqual(self.state.workflow_status.count, 0) | ||
| self.assertNotIn(task, self.state.workflow_status.completed_passes) | ||
| self.assertEqual(self.state.workflow_status.previous_run, RunState.FAIL) | ||
|
|
||
| def test_requires(self): | ||
| """Test cate: Dependency tasks are run in advance to user provided task.""" | ||
|
|
||
| class TaskA(GenericPass): | ||
| def run(self, passmanager_ir): | ||
| return passmanager_ir | ||
|
|
||
| class TaskB(GenericPass): | ||
| def __init__(self): | ||
| super().__init__() | ||
| self.requires = [TaskA()] | ||
|
|
||
| def run(self, passmanager_ir): | ||
| return passmanager_ir | ||
|
|
||
| task = TaskB() | ||
| data = "test_data" | ||
| expected = [ | ||
| r"Pass: TaskA - (\d*\.)?\d+ \(ms\)", | ||
| r"Pass: TaskB - (\d*\.)?\d+ \(ms\)", | ||
| ] | ||
| self.assertLogEqual(task.execute, expected, data, self.state) | ||
| self.assertEqual(self.state.workflow_status.count, 2) | ||
|
|
||
| def test_requires_in_list(self): | ||
| """Test cate: Dependency tasks are not executed multiple times.""" | ||
|
|
||
| class TaskA(GenericPass): | ||
| def run(self, passmanager_ir): | ||
| return passmanager_ir | ||
|
|
||
| class TaskB(GenericPass): | ||
| def __init__(self): | ||
| super().__init__() | ||
| self.requires = [TaskA()] | ||
|
|
||
| def run(self, passmanager_ir): | ||
| return passmanager_ir | ||
|
|
||
| task = TaskB() | ||
| data = "test_data" | ||
| expected = [ | ||
| r"Pass: TaskB - (\d*\.)?\d+ \(ms\)", | ||
| ] | ||
| self.state.workflow_status.completed_passes.add(task.requires[0]) # already done | ||
| self.assertLogEqual(task.execute, expected, data, self.state) | ||
| self.assertEqual(self.state.workflow_status.count, 1) | ||
|
|
||
| def test_run_with_callable(self): | ||
| """Test case: Callable is called after generic pass is run.""" | ||
|
|
||
| # pylint: disable=unused-argument | ||
| def test_callable(task, passmanager_ir, property_set, running_time, count): | ||
| logger = getLogger() | ||
| logger.info("%s is running on %s", task.name(), passmanager_ir) | ||
|
|
||
| class Task(GenericPass): | ||
| def run(self, passmanager_ir): | ||
| return passmanager_ir | ||
|
|
||
| task = Task() | ||
| data = "test_data" | ||
| expected = [ | ||
| r"Pass: Task - (\d*\.)?\d+ \(ms\)", | ||
| r"Task is running on test_data", | ||
| ] | ||
| self.assertLogEqual(task.execute, expected, data, self.state, test_callable) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I kind of get why we're doing this, but the need also potentially points to a data-model problem in the idea of "completed passes" to me; the pass still completely after a loop iteration, and it's not immediately clear to me that starting a new loop should throw that information away.
What part of the logic is causing a pass to get skipped if it was completed? Passes aren't required to be idempotent, so a pass having been run shouldn't be sufficient to cause it to be skipped on a second attempt.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line. Change to this logic causes many test failures in
test.python.transpiler.test_pass_schedulerqiskit/qiskit/passmanager/base_tasks.py
Lines 95 to 99 in dabc5e6
I guess this protection is added because of
GenericPass.required. i.e.runs
PassBtwice.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, but I think that's potentially indicative of buggy handling:
pass_ = MyPass(); PassManager([pass_, pass_])should runpass_twice unless the pass specifically marks itself as being idempotent somehow. It's a different story if the pass is inserted into the pipeline byrequires, because that's an implicit add and it's fine to re-use a previous run if we know that the output from it is still valid.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I just tried it on 0.25.3, and I realise now that what I'm complaining about is pre-existing behaviour. I think that's bad behaviour, but it being pre-existing puts it out of scope of this PR. The fix you've got in this comment is fine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't really like the required mechanism. Since dependencies are needed to be instantiated within the target pass, the target pass may require extra constructor args if passes have different interface. If we remove this mechanism, I think we can drop
completed_passesand awkward pass equality check.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We definitely do use the
requiredhandling in some places in the existing pipelines, so it won't be as easy as completely stripping it out. I think the intent of it is good, it's just that it uses referential pass equality to implicitly mean "idempotent", which isn't generally true, and that's the knock-on problem here.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually circuit transform pass is not idempotent (so
pass_ = MyPass(); PassManager([pass_, pass_])should work as you expect), but it also removes all passes because.preservesis usually empty at least in Qiskit passes.qiskit/qiskit/transpiler/basepasses.py
Lines 229 to 237 in dabc5e6
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, that's what I mean - I think
PassManager([pass_, pass_])should runpass_twice in the general case, because there's nothing marking_passas idempotent, but the "has this pass run?" handling that existed in 0.25.3 would mean it only runs once. Since it was in 0.25.3 and the 0.45.0 passmanager just maintains the same behaviour, there's no need to rush in a fix now.