Integration with result database by yaelbh · Pull Request #115 · qiskit-community/qiskit-experiments

yaelbh · 2021-06-17T11:08:19Z

Summary

This PR is based on #113, and is a continuation of #66

Details and comments

Co-authored-by: Jessie Yu <jessieyu@us.ibm.com>

yaelbh · 2021-06-17T11:10:51Z

-        except AnalysisError as ex:
-            analysis_results = [AnalysisResult(success=False, error_message=ex)]
-            figures = None
+        analysis_results, figures = self._run_analysis(experiment_data, **analysis_options)


Is it OK to remove the try-catch here? The difficulty is in populating the parameters of AnalysisResultV1 in case of exception. @jyu00 @nkanazawa1989 @chriseclectic

I think I removed the try/catch, and my reasoning was that we shouldn't be saving a failed analysis result.

yaelbh · 2021-06-17T11:11:40Z

-                analysis_result["raw_data"] = raw_data_dict
+                result_data["raw_data"] = raw_data_dict
+
+        analysis_result = AnalysisResultV1(


Should this line be inside or outside "finally" ?

yaelbh · 2021-06-17T11:12:17Z

+            result_data=result_data,
+            result_type=result_data["analysis_type"],
+            device_components=[
+                Qubit(qubit) for qubit in experiment_data.data(0)["metadata"]["qubits"]


Is there a new way to retrieve the qubit, through experiment metadata?

I don't think qubit is currently added to experiment metadata, but it'd be a good thing to do. It'd certainly make this line easier.

yaelbh · 2021-06-17T11:13:00Z

@jyu00 What happened to ResultQuality ?

yaelbh · 2021-06-17T13:27:49Z

@jyu00 While debugging failing tests, I see that for test_t1_end2end the method run_analysis is never called. In other words, we have in BaseExperiment.run:

experiment_data.add_data(job, post_processing_callback=run_analysis)

But the post-processing callback seems not be called; a print at the beginning of BaseExperiment.run_analysis is not displayed. Do you have an idea how to further debug it?

In the test, we call block_for_results. Could it be that once there are results and everything's unblocked, two things happen simultaneously:

run_analysis.
The rest of the test (which assumes that run_analysis has terminated.

jyu00 · 2021-06-17T20:14:01Z

@yaelbh In my other PR I added a new data_index parameter to the post processing callback (run_analysis), which was an unexpected parameter hence the run_analysis() failed. The reason behind this extra parameter was because waiting for job to finish + calling run analysis happens asynchronously in a different thread. I was worried that something like this could happen

exp_data.add_data(job1)
exp_data.add_data(job2)
job1 finishes and its results are added
job2 finishes and its results are added
job1's run_analysis() gets called and uses exp_data.data(), which includes result data from both job1 and job2

the data_index was supposed to tell run_analysis() to only use data in a certain range.

BUT I realized the error in my logic - it still wouldn't have helped if job2 finishes before job1. So I removed it and just added a warning if add_data() is called while there's a pending run_analysis().

I also added code to log a warning if run_analysis() failed (you otherwise wouldn't see the exception since it's in a different thread).

In the test, we call block_for_results. Could it be that once there are results and everything's unblocked, two things happen simultaneously:

run_analysis.

The rest of the test (which assumes that run_analysis has terminated.

block_for_results() waits for both the job and its post processing (run_analysis) to finish. I'm happy to change the name if it's too confusing.

yaelbh · 2021-06-20T10:48:18Z

@mtreinish @jyu00 stestr is having a hard time with the threading (I think it's OK with pytest):

$ stestr run -n test/test_qubit_spectroscopy.py
/home/yaelbh/work/experiments/C66B113/qiskit_experiments/analysis/curve_analysis.py:323: UserWarning: Starting a Matplotlib GUI outside of the main thread will likely fail.
  figure = plotting.pyplot.figure(figsize=(8, 5))
{0} test.test_qubit_spectroscopy.TestQubitSpectroscopy.test_spectroscopy_end2end_classified [1.439731s] ... ok
/home/yaelbh/work/experiments/C66B113/qiskit_experiments/analysis/curve_analysis.py:323: UserWarning: Starting a Matplotlib GUI outside of the main thread will likely fail.
  figure = plotting.pyplot.figure(figsize=(8, 5))
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7fd6e95f0280>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 351, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7fd6e95f0280>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 351, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7fd6e95f0280>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 351, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7fd6e95f0280>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 351, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7fd6e9608b80>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/YaelEnv/lib/python3.8/tkinter/__init__.py", line 4014, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Tcl_AsyncDelete: async handler deleted by the wrong thread
Aborted (core dumped)
{0} test.test_qubit_spectroscopy.TestQubitSpectroscopy.test_spectroscopy_end2end_kerneled [] ... inprogress

======
Totals
======
Ran: 2 tests in 1.4397 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 1.4397 sec.

==============
Worker Balance
==============
 - Worker 0 (2 tests) => N/A

coruscating · 2021-07-15T18:28:24Z

When I save a T2 Ramsey experiment, the analysis results are in the database but the figure is not. Doesn't necessarily need to be fixed in this PR, but just noting the problem.

yaelbh · 2021-07-15T18:58:35Z

I'm curious about the result themselves for T2Ramsey, because they are non-orthodox, and I expected that they'd need to change in order to display on IQX. There is no field name "value", but two fields named "t2ramsey_value" and "frequency_value". Can you paste what you see in IQX?

coruscating · 2021-07-15T19:14:47Z

@yaelbh @merav-aharoni

jyu00 · 2021-07-16T00:56:40Z

@chriseclectic 09538f7 has code that allows all objects to be "serialized" (it just uses the class name). This should at least allow tomography to be saved in results DB. Perhaps we can do the SerializableClass that we discussed in a separate PR?

nkanazawa1989 · 2021-07-16T04:16:09Z

+                result_data["raw_data"] = raw_data_dict

-        return [analysis_result], figures
+        return [CurveAnalysisResultData(result_data)], figures


Why CurveAnalysisResultData is generated again with nested CurveAnalysisResultData?

Sorry, I thought it was a dictionary, I'll fix. Also need to fix in T1 and T2Ramsey, where it should be CurveAnalysisResultData instead of AnalysisResultData.

chriseclectic

Looks pretty close. I think we can still remove the custom CurveAnalysisResult and CurveAnalysisResultData

chriseclectic · 2021-07-16T12:56:13Z



-class CurveAnalysisResult(AnalysisResult):
+class CurveAnalysisResultData(AnalysisResultData):


Can this class be deleted now and just use AnalysisResultData?

@nkanazawa1989 Are you good with this or do you want to keep CurveAnalysisResultData as type hint?

Yes, I'm fine with removing this specific class if the hint will become other than Dict

chriseclectic · 2021-07-16T12:56:25Z

    bounds: Optional[Union[Dict[str, Tuple[float, float]], Tuple[np.ndarray, np.ndarray]]] = None,
    **kwargs,
-) -> CurveAnalysisResult:
+) -> CurveAnalysisResultData:


Suggested change

) -> CurveAnalysisResultData:

) -> AnalysisResultData:

chriseclectic · 2021-07-16T12:59:20Z

        self, experiment_data: ExperimentData, **options
-    ) -> Tuple[List[AnalysisResult], List["matplotlib.figure.Figure"]]:
+    ) -> Tuple[
+        Union["AnalysisResultData", List["AnalysisResultData"]], List["matplotlib.figure.Figure"]


Suggested change

Union["AnalysisResultData", List["AnalysisResultData"]], List["matplotlib.figure.Figure"]

Tuple[List["AnalysisResultData"], List["matplotlib.figure.Figure"]]

chriseclectic · 2021-07-16T13:01:29Z

        plot=True,
        ax=None,
-    ) -> Tuple[List[AnalysisResult], List["matplotlib.figure.Figure"]]:
+    ) -> Tuple[List[AnalysisResultData], Optional[List["matplotlib.figure.Figure"]]]:


Is Optional correct type hint when it returns either List or None. Maybe we should change to

Suggested change

) -> Tuple[List[AnalysisResultData], Optional[List["matplotlib.figure.Figure"]]]:

) -> Tuple[List[AnalysisResultData], List["matplotlib.figure.Figure"]]:

and return [] instead of None if there is no mpl.

chriseclectic · 2021-07-16T13:06:54Z

-
-        analysis_result["fit"]["circuit_unit"] = unit
+        result_data = {
+            "t2ramsey_value": fit_result["popt"][1],


For a follow up PR. But I think this needs to be changed into two analysis results if you want to show both T1 value and frequency value in DB.

chriseclectic

Lets worry about my previous comment on repr of (Curve)AnalysisResultData objects in a follow up PR.

* move from terra * run black * convert service exception to log * fix lint * Integration with result database Co-authored-by: Jessie Yu <jessieyu@us.ibm.com> * some brief fixes * log post processing failure * remove data index * fix t1 t2 tests * fix test can package typo * fix package typo * fixed errors in conlicts resolve * adjusted a test to the new classes * made test_spectroscopy_end2end_classified pass * adjusted all spectro tests * test_curve_fit passes * adjusted rabi * black * lint * fixed test_source * fix doc build * fix figure thread error * run black * remove ResultDict * remove experiment_class and result_class * fix lint * fixes * adjusted tes_update_library * lint * black * doc update Co-authored-by: Naoki Kanazawa <nkanazawa1989@gmail.com> * rename classes * review comments * fix lint * black * black * catch failed analysis + black * omit prefix computer_ from quality * removed debug print * fix docs build * added meas_level to mock iq backend header * black * removed the call to get_memory * adjusted test_rb_analysis * fix analysis result __str__ * add callable to encoder * fix lint * fix lint again * log all job failure * lint * fix test * return DbAnalysisResultV1 for t1 t2 * review comments * Add `load` method to `DbExperimentDataV1` and `DbAnalysisResultV1` This method parses the dicts currently returned by the IBMQ service to allow loading saved experiment data and analysis results. * Revert "Merge pull request qiskit-community#2 from chriseclectic/deserialize" This reverts commit e150dad, reversing changes made to 0d6e3c9. * updated tomography qubits * Add `load` method to `DbExperimentDataV1` and `DbAnalysisResultV1` This method parses the dicts currently returned by the IBMQ service to allow loading saved experiment data and analysis results. * Rename `save` to `save_metadata`, `save_all` to `save` * changed analysis result classes * black * lint * fixes related to AnalysisResultData * allow all objects in encoder * call save if auto save set * fix tests * fixed test_fine_amplitude * lint * black * adjust to recent changes in main * fixes related to AnalysisResultData Co-authored-by: jessieyu <jessieyu@us.ibm.com> Co-authored-by: Naoki Kanazawa <nkanazawa1989@gmail.com> Co-authored-by: Christopher Wood <cjwood@us.ibm.com>

jyu00 and others added 5 commits June 16, 2021 20:29

move from terra

d3af475

run black

7506759

convert service exception to log

4503267

fix lint

5e97146

Integration with result database

b5abffa

Co-authored-by: Jessie Yu <jessieyu@us.ibm.com>

yaelbh commented Jun 17, 2021

View reviewed changes

yaelbh changed the title ~~Integration with result database~~ [WIP] Integration with result database Jun 17, 2021

some brief fixes

d94668f

jyu00 added 4 commits June 17, 2021 14:41

log post processing failure

6bd5c64

remove data index

1780a37

Merge remote-tracking branch 'jessie/results-db' into integrate_resdb

1d408c9

fix t1 t2 tests

26b76df

jyu00 and others added 9 commits June 17, 2021 16:27

fix test can package typo

e3a2d31

Merge remote-tracking branch 'jessie/results-db' into integrate_resdb

9a0be50

fix package typo

ce1c917

Merge branch 'main' into results-db

b727010

Merge branch 'main' into integrate_resdb

abb6f8d

fixed errors in conlicts resolve

c066274

adjusted a test to the new classes

e4bdbe6

made test_spectroscopy_end2end_classified pass

7100638

adjusted all spectro tests

f031947

yaelbh added 3 commits June 20, 2021 14:06

test_curve_fit passes

e83add3

adjusted rabi

cf57d81

black

a14144c

allow all objects in encoder

09538f7

nkanazawa1989 reviewed Jul 16, 2021

View reviewed changes

chriseclectic reviewed Jul 16, 2021

View reviewed changes

jyu00 and others added 6 commits July 16, 2021 14:24

call save if auto save set

bb19b66

fix tests

793c20e

Merge branch 'main' into integrate_resdb

ee56d6e

fixed test_fine_amplitude

5f300f8

lint

955b7f6

black

f96ce73

chriseclectic approved these changes Jul 19, 2021

View reviewed changes

Merge branch 'main' into integrate_resdb

16d7f6d

nkanazawa1989 approved these changes Jul 19, 2021

View reviewed changes

chriseclectic mentioned this pull request Jul 19, 2021

Tomography options need serialization #181

Closed

yaelbh added 3 commits July 20, 2021 09:10

Merge branch 'main' into integrate_resdb

ce3b30e

adjust to recent changes in main

5f967e9

fixes related to AnalysisResultData

c5a3a64

yaelbh merged commit 70d9b48 into qiskit-community:main Jul 20, 2021

yaelbh deleted the integrate_resdb branch July 20, 2021 06:59

This was referenced Jul 20, 2021

Update the tutorials following the merge of 115 #189

Closed

Fix prints of analysis results #202

Closed

merav-aharoni mentioned this pull request Jul 21, 2021

Fixed t2ramsey_tutorial following recent changes #203

Merged

yaelbh mentioned this pull request Jul 21, 2021

Fix T2Ramsey display on IQX #207

Merged

chriseclectic mentioned this pull request Jul 21, 2021

Fix errors in QST notebook #214

Merged

nkanazawa1989 mentioned this pull request Jul 27, 2021

RB and QV cache generation util is broken #240

Closed

3 tasks

nkanazawa1989 mentioned this pull request Aug 25, 2021

More flexible execution #358

Closed



		class CurveAnalysisResult(AnalysisResult):
		class CurveAnalysisResultData(AnalysisResultData):

	Union["AnalysisResultData", List["AnalysisResultData"]], List["matplotlib.figure.Figure"]
	Tuple[List["AnalysisResultData"], List["matplotlib.figure.Figure"]]

	) -> Tuple[List[AnalysisResultData], Optional[List["matplotlib.figure.Figure"]]]:
	) -> Tuple[List[AnalysisResultData], List["matplotlib.figure.Figure"]]:

Conversation

yaelbh commented Jun 17, 2021

Summary

Details and comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaelbh commented Jun 17, 2021

Uh oh!

yaelbh commented Jun 17, 2021

Uh oh!

jyu00 commented Jun 17, 2021

Uh oh!

yaelbh commented Jun 20, 2021

Uh oh!

coruscating commented Jul 15, 2021

Uh oh!

yaelbh commented Jul 15, 2021

Uh oh!

coruscating commented Jul 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jyu00 commented Jul 16, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chriseclectic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chriseclectic left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

coruscating commented Jul 15, 2021 •

edited

Loading