Pipeline services assume event payload to be Dictionary

[JenySadadia]

Pipeline services such as monitor and scheduler are based on PubSub events published on node channel.
While parsing event data, the services assume it to be a Dict and post-process data accordingly.

Previously, event payload data was Dict and hence the the pipeline services were implemented like that. But now it has changed to be Any type. Thus, the issue with data post-processing in the pipeline has surfaced.

e.g. take a look at the below code section of monitor service:

event = self._api.receive_event(sub_id)
obj = event.data
dt = datetime.datetime.fromisoformat(event['time'])
commit = (obj['data']['kernel_revision']['commit'][:12]
                      if 'kernel_revision' in obj['data']
                      else str(None))

If somehow, str data is sent to node channel, it will end up in error.

Monitor service failure:

today at 12:11:55  File "/home/kernelci/pipeline/base.py", line 69, in run
today at 12:11:55    status = self._run(context)
today at 12:11:55             ^^^^^^^^^^^^^^^^^^
today at 12:11:55  File "/home/kernelci/./pipeline/monitor.py", line 64, in _run
today at 12:11:55    if 'kernel_revision' in obj['data']
today at 12:11:55                            ~~~^^^^^^^^
today at 12:11:55TypeError: string indices must be integers, not 'str'
today at 12:11:55
today at 12:11:55Container stopped

Also, scheduler service with shell runtime:

today at 12:11:5503/18/2024 06:41:55 AM UTC [ERROR] Traceback (most recent call last):
today at 12:11:55  File "/home/kernelci/pipeline/base.py", line 69, in run
today at 12:11:55    status = self._run(context)
today at 12:11:55             ^^^^^^^^^^^^^^^^^^
today at 12:11:55  File "/home/kernelci/./pipeline/scheduler.py", line 189, in _run
today at 12:11:55    for job, runtime, platform, rules in self._sched.get_schedule(event):
today at 12:11:55  File "/usr/local/lib/python3.11/site-packages/kernelci/scheduler.py", line 44, in get_schedule
today at 12:11:55    for config in self.get_configs(event, channel):
today at 12:11:55  File "/usr/local/lib/python3.11/site-packages/kernelci/scheduler.py", line 39, in get_configs
today at 12:11:55    if sched_event.items() <= event.items():
today at 12:11:55                              ^^^^^^^^^^^
today at 12:11:55AttributeError: 'str' object has no attribute 'items'

Scheduler with docker:

today at 12:11:5503/18/2024 06:41:55 AM UTC [ERROR] Traceback (most recent call last):
today at 12:11:55  File "/home/kernelci/pipeline/base.py", line 69, in run
today at 12:11:55    status = self._run(context)
today at 12:11:55             ^^^^^^^^^^^^^^^^^^
today at 12:11:55  File "/home/kernelci/./pipeline/scheduler.py", line 189, in _run
today at 12:11:55    for job, runtime, platform, rules in self._sched.get_schedule(event):
today at 12:11:55  File "/usr/local/lib/python3.11/site-packages/kernelci/scheduler.py", line 44, in get_schedule
today at 12:11:55    for config in self.get_configs(event, channel):
today at 12:11:55  File "/usr/local/lib/python3.11/site-packages/kernelci/scheduler.py", line 39, in get_configs
today at 12:11:55    if sched_event.items() <= event.items():
today at 12:11:55                              ^^^^^^^^^^^
today at 12:11:55AttributeError: 'str' object has no attribute 'items'
today at 12:11:55
today at 12:11:55Traceback (most recent call last):
today at 12:11:55  File "/home/kernelci/./pipeline/scheduler.py", line 218, in <module>
today at 12:11:55    status = opts.command(configs, opts)
today at 12:11:55             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 12:11:55  File "/home/kernelci/./pipeline/scheduler.py", line 211, in __call__
today at 12:11:55    return Scheduler(configs, args).run()
today at 12:11:55           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 12:11:55  File "/home/kernelci/pipeline/base.py", line 76, in run
today at 12:11:55    self._stop(context)
today at 12:11:55  File "/home/kernelci/./pipeline/scheduler.py", line 73, in _stop
today at 12:11:55    self._cleanup_paths()
today at 12:11:55  File "/home/kernelci/./pipeline/scheduler.py", line 58, in _cleanup_paths
today at 12:11:55    job_tmp_dirs = {
today at 12:11:55                   ^
today at 12:11:55  File "/home/kernelci/./pipeline/scheduler.py", line 62, in <dictcomp>
today at 12:11:55    if job.poll() is None
today at 12:11:55       ^^^^^^^^
today at 12:11:55AttributeError: 'Container' object has no attribute 'poll'
today at 12:13:49Container stopped

[mwasilew]

I've seen a similar issue when running a pipeline docker compose:

kernelci-pipeline-trigger             | 05/24/2024 10:10:43 AM UTC [INFO] New revision                     kernelci_staging-mainline        12f30990e9b02966e3742c7c94996d4bae8054c9
kernelci-pipeline-monitor             | 2024-05-24 10:10:43.070971  checkout        12f30990e9b0  665067a373749661c10dd085 Running    None      checkout
kernelci-pipeline-tarball             | 05/24/2024 10:10:43 AM UTC [ERROR] Traceback (most recent call last):
kernelci-pipeline-tarball             |   File "/home/kernelci/pipeline/base.py", line 69, in run
kernelci-pipeline-tarball             |     status = self._run(context)
kernelci-pipeline-tarball             |              ^^^^^^^^^^^^^^^^^^
kernelci-pipeline-tarball             |   File "/home/kernelci/./pipeline/tarball.py", line 188, in _run
kernelci-pipeline-tarball             |     build_config = self._find_build_config(checkout_node)
kernelci-pipeline-tarball             |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kernelci-pipeline-tarball             |   File "/home/kernelci/./pipeline/tarball.py", line 54, in _find_build_config
kernelci-pipeline-tarball             |     revision = node['data']['kernel_revision']
kernelci-pipeline-tarball             |                ~~~~^^^^^^^^
kernelci-pipeline-tarball             | TypeError: tuple indices must be integers or slices, not str
kernelci-pipeline-tarball             | 
kernelci-pipeline-tarball exited with code 1

[JenySadadia]

I've seen a similar issue when running a pipeline docker compose:

kernelci-pipeline-trigger             | 05/24/2024 10:10:43 AM UTC [INFO] New revision                     kernelci_staging-mainline        12f30990e9b02966e3742c7c94996d4bae8054c9
kernelci-pipeline-monitor             | 2024-05-24 10:10:43.070971  checkout        12f30990e9b0  665067a373749661c10dd085 Running    None      checkout
kernelci-pipeline-tarball             | 05/24/2024 10:10:43 AM UTC [ERROR] Traceback (most recent call last):
kernelci-pipeline-tarball             |   File "/home/kernelci/pipeline/base.py", line 69, in run
kernelci-pipeline-tarball             |     status = self._run(context)
kernelci-pipeline-tarball             |              ^^^^^^^^^^^^^^^^^^
kernelci-pipeline-tarball             |   File "/home/kernelci/./pipeline/tarball.py", line 188, in _run
kernelci-pipeline-tarball             |     build_config = self._find_build_config(checkout_node)
kernelci-pipeline-tarball             |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kernelci-pipeline-tarball             |   File "/home/kernelci/./pipeline/tarball.py", line 54, in _find_build_config
kernelci-pipeline-tarball             |     revision = node['data']['kernel_revision']
kernelci-pipeline-tarball             |                ~~~~^^^^^^^^
kernelci-pipeline-tarball             | TypeError: tuple indices must be integers or slices, not str
kernelci-pipeline-tarball             | 
kernelci-pipeline-tarball exited with code 1

Thanks for reporting.
This issue is not related to what originally was reported. It was a side effect of not merging the set of PRs listed here: #337 (comment)
Sorry about that. This issue should be fixed now as I merged all the related PRs.