src/patchset.py: Implement Patchset service #342
Conversation
yurinnick
force-pushed
the
patchset-service
branch
from
00ef8cb
to
3dc76bc
Compare
yurinnick
added
the
staging-skip
Setup nodesPatchset logsTarball logsCheck tarballCheck patchset node |
yurinnick
force-pushed
the
patchset-service
branch
3 times, most recently
from
aa29045
to
f4a6ebe
Compare
There was a problem hiding this comment.
I really appreciate posting such a detailed verification steps and all the log information. These will greatly help verifying this patch and also maybe set up an E2E suite for Patchwork-related pipeline changes. Let me deploy these changes and get back to you on Monday if some other questions come up
yurinnick
force-pushed
the
patchset-service
branch
3 times, most recently
from
cb01cea
to
5301005
Compare
Submit patchset nodePatchset logsCheck tarballCheck node |
yurinnick
force-pushed
the
patchset-service
branch
from
5301005
to
544b19f
Compare
| if checkout_node["state"] == "running": | ||
| self.log.info( | ||
| f"Patchset node {patchset_node['id']} is waiting " | ||
| f"for checkout node {checkout_node['id']} to complete", | ||
| ) | ||
| continue |
There was a problem hiding this comment.
Okay, now I understand why you are finding all the running nodes on every event.
According to the state machine and design, checkout node should only have children after its state changes to available.
Instead of manually creating checkout and patchset nodes, we can listen to available checkout node events in the patchset service and create a node automatically from the service itself.
There was a problem hiding this comment.
So, for every patchset node I should create a new checkout node and wait for it to become available? It will pause the queue for quite some time and seems to be quite inefficient.
However it gave me another idea. Ideally I want to apply patchset to whatever latest commit is available on a tree. Is it possible to query latest available checkout node?
There was a problem hiding this comment.
Yes, that's what I am suggesting. You can listen to available checkout nodes and create patchset nodes from the service directly.
See, for instance, you can use the below filter here to get available checkout node events:
def _setup(self, *args):
return self._api_helper.subscribe_filters({
"kind": "checkout",
"state": "available",
})
Then you can get checkout node from the event just like https://github.com/kernelci/kernelci-pipeline/blob/main/src/tarball.py#L130 and then you can create patchset node from it.
There was a problem hiding this comment.
So it should be done outside of patchset service, right? In an another service that will prepare checkout node and then pass it to patchset?
There was a problem hiding this comment.
So I think the greatest challenge with checkout nodes so is picking the right one. I'm not sure how to determine an appropriate version to apply a patch and then correctly query a right node. If we have checkouts for 6.4, 6.8 etc how to pick the right version. Maybe we should pick the latest stable and next branch and create two patchset nodes based on them.
There was a problem hiding this comment.
In an another service that will prepare checkout node and then pass it to patchset?
The pipeline service trigger is already creating checkout nodes whenever a new commit is detected for a particular kernel revision. As I have not been involved in the patchset-related discussions, I am not aware of all the requirements, but do you think you can use checkout nodes created by the trigger service?
Once the trigger service creates a checkout node, the tarball service will create and push a tarball and change the node's status to available. Then you can use the below filter in the patchset service and create a patchset node from it:
def _setup(self, *args):
return self._api_helper.subscribe_filters({
"op": "updated",
"name": "checkout",
"state": "available",
})
def _run(self, _sub_id):
self.log.info("Listening for new trigger events")
self.log.info("Press Ctrl-C to stop.")
while True:
checkout_node = self._api_helper.receive_event_node(sub_id)
patchset_node = { create_patchset_node_from_checkout() }
patchset_node = self._api.node.add(patchset_node)
self._process_patchset(checkout_node, patchset_node)
There was a problem hiding this comment.
As we discussed with @gctucker in kernelci-pipeline#295, patchset service only processed patchset nodes created by another service (patchwork<>kernelci connector). So patchset service should tream these nodes as created on-demand and process them accordingly.
I am open to suggestions on how to link existing checkout nodes and freshly created patchset node. I think it's a topic for a larger discussion on how we want to test kernel patches in general.
There was a problem hiding this comment.
@JenySadadia @yurinnick I agree it will be an important next step to find the best way to connect patchset nodes with checkout ones created by the trigger service. Let keep the scope of this PR to just processing patchset nodes created by on-demand - with an assumption for an already existing parent checkout node (like in the verification steps posted in this PR).
I created #490 to track this task. @JenySadadia Please let me know if there something else that should be done before starting integration of this service.
There was a problem hiding this comment.
I think we should merge this PR now and improvements can be discussed as follow-ups.
|
@yurinnick Please rebase the PR. |
yurinnick
force-pushed
the
patchset-service
branch
from
544b19f
to
f5936c8
Compare
|
Rebased. |
yurinnick
force-pushed
the
patchset-service
branch
from
f5936c8
to
3629a49
Compare
|
Done:
Todo:
|
|
Sorry for jumping into this discussion with naive questions, but why not extend the As it turns out, This would of course need some thinking around how to configure patchsets in |
@a-wai good point! I think we could fold some parts of the Let's keep Patchwork integration as a separate service for now and revisit this idea once the patchset processing gets stableish - I created #491 to track this. |
Patchset service process patchset nodes: - Wait for parent checkout node to be available - Download checkout node tarball - Apply patches and calculate patchset hash - Upload new tarball Signed-off-by: Nikolay Yurin <[email protected]>
yurinnick
force-pushed
the
patchset-service
branch
from
3629a49
to
0f096bc
Compare
|
Rebased and fixed copyright header |
|
@a-wai this is what we can iterate to. I'd prefer |
|
@JenySadadia Please have a look at merging this PR if there are no blockers (you're the sole Pipeline maintainer) |
Let's give it a go to the staging first. |
JenySadadia
removed
the
staging-skip
|
Updated staging TOML and started |
Patchset service process patchset nodes: