Link produced events to consumed events

[afrittoli]

Feature Description

The CDEvents specification should allow propagating identifiers from consumed events into produced events.
When an activity is directly started as a consequence of a consumed CDEvents, an identifier of the consumed event must be included in the produced events related to the activity.
When an activity is directly started as a consequence of a consumed event (non CDEvent), an identifier of the consumed event may be included in the produced events related to the activity.

Use Cases

Children Pipelines

When a pipeline runs, it generates events as different tasks are started and finished. Children pipelines may be triggered by these events (whether in the same CD system or a different one), and they will in turn send events.
As a DevOps engineer I want to be able to discover children pipelines from the events they generated.

Proposal

These are the spec fields that would be required for CDEvents.

Field Name	Type	Description	CloudEvents Binding	Notes
id	string	Event identifier	id	Unique for a source
source	URI-reference	Producer of the event	source	May be globally unique?
subject	string	Producer of the event	source	May be globally unique?
cdevents_platform	string	Platform namespace for sources	<extension>	Only needed is source is not globally unique
cdevents_source	URI	Globally unique source	<extension>	Only needed is source is not globally unique
cdevents_source-event	string	Globally unique source event identified	<extension>	Serialised source + id / cdevents_platform + source + id / cdevents_source

We already use id and source in our PoC, however we do not specify them in our spec, but we should add them.
The source-event field is a globally unique identified of another event, which was the trigger to the activity identified by "source + subject".

CloudEvents have a Source and ID attached to them. Each instance of each platform is responsible for generating unique Source + IDs for its events, but it's not required for one platform to guarantee global uniqueness.

1. If we enforce global uniqueness, it would responsibility of the administrator of the overall system to ensure each source is configured with a globally unique value (e.g. a DNS name) that can be used to generated the various sources.

2. If we do not enforce global uniqueness, we could introduce the cdevents_platform extension or the cdevents_source extension to provide a globally unique value. Again it would be responsibility of the overall administrator to configure a globally unique seed. Alternatively platform could use some strategy to pick a name that is likely to be globally unique. Examples:

{
    "specversion" : "1.0",
    "type" : "cd.events.taskrun.started",
    "source" : "/apis/tekton.dev/v1beta1/namespaces/default/taskruns/curl-run-6gplk",
    "subject" : "my-taskrun-123",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "cdeventssource" : "tekton_ci_prod/apis/tekton.dev/v1beta1/namespaces/default/taskruns/curl-run-6gplk",
    "cdeventsplatform" : "tekton_node123_clusterABC",
    "cdeventssourceevent":  "A123-5678-9012@jenkins_ci_prod/pipeline/my-source-pipeline",
    "datacontenttype" : "text/json",
    "data" : "{}"
}

Open Questions

It may not be possible for a platform to implement the source-event extension in all outgoing events, because of a few reason:

• if cdevents are implemented via an adaptation layer, the information about the source event may not be there at all
• not all activities that require a cdevent may have a source-event associated. This happens typically with core events, where the end of a task run or pipeline run is not triggered by an event. In that case, when an activity is finished, there will be no source-event associated

This means that a consumer of events that would like to visualise the end to end workflow triggered by an initial event will have to use a combination of source-event, subject and event type to discover all relevant events.

[m-linner-ericsson]

I like this initiative as it I think we need this in the protocol.

Eiffel also have links but there it stores the information as an object see the links object. Eiffel would call this link a CAUSE link and would define it as shown from an example of a ActivityTriggeredEvent.

CAUSE
Required: No
Legal targets: Any
Multiple allowed: Yes
Description: Identifies a cause of the event occurring. SHOULD not be used in conjunction with CONTEXT: individual events providing CAUSE within a larger context gives rise to ambiguity. It is instead recommended to let the root event of the context declare CAUSE.

and the json for a event would look like this:

"links": [
    {
      "type": "CAUSE",
      "target": "aaaaaaaa-bbbb-5ccc-8ddd-eeeeeeeeeee1",
      "domainId": "com.example"
    }
  ]

From Eiffels perspective I would like to also have the link typed if/when we want to add more links to the protocol.

Today I opened #19 to add a note we made on a vocabulary meeting about how we want to handle extensions. Do we want this as an extension or as part of the data field?

[afrittoli]

I like this initiative as it I think we need this in the protocol.

Eiffel also have links but there it stores the information as an object see the links object. Eiffel would call this link a CAUSE link and would define it as shown from an example of a ActivityTriggeredEvent.

CAUSE
Required: No
Legal targets: Any
Multiple allowed: Yes
Description: Identifies a cause of the event occurring. SHOULD not be used in conjunction with CONTEXT: individual events providing CAUSE within a larger context gives rise to ambiguity. It is instead recommended to let the root event of the context declare CAUSE.

Oh, interesting, I will go read more about cause vs. context in Eiffel

and the json for a event would look like this:

"links": [
    {
      "type": "CAUSE",
      "target": "aaaaaaaa-bbbb-5ccc-8ddd-eeeeeeeeeee1",
      "domainId": "com.example"
    }
  ]

From Eiffels perspective I would like to also have the link typed if/when we want to add more links to the protocol.

Today I opened #19 to add a note we made on a vocabulary meeting about how we want to handle extensions. Do we want this as an extension or as part of the data field?

I would not add this to #19 for now. Hopefully we can merge #19 soon and adding more to it might slow the process down.
We can use the cloudevents binding document to track specific extensions that we identify. as we go.

What do you think about the open question on source?
Should we use source from CloudEvents to store a globally unique URI, or define our own extension?

[erkist]

Really nice! Being able to have links to previous event will really help with things like visualizing chains without knowing beforehand which events to expect.

The "cdevents_source-event" is a bit unclear to me.
It is defined as "The source-event field is a globally unique identified of another event, which was the trigger to the activity identified by "source + subject"." in the text, but as "Serialised source + id / cdevents_platform + source + id / cdevents_source" in the table.
And in the JSON example it seems to be on the format "id@subject" or maybe "id@source".

[erkist]

not all activities that require a cdevent may have a source-event associated. This happens typically with core events, where the end of a task run or pipeline run is not triggered by an event. In that case, when an activity is finished, there will be no source-event associated

In such cases it would make sense to me to refer back to the corresponding "XStarted" event. The direct "cause" for an activity ending is that it was started in the first place.

[afrittoli]

Really nice! Being able to have links to previous event will really help with things like visualizing chains without knowing beforehand which events to expect.

The "cdevents_source-event" is a bit unclear to me. It is defined as "The source-event field is a globally unique identified of another event, which was the trigger to the activity identified by "source + subject"."

"Source + Subject" Identifies the activity
"Source + ID" Identifies an event

The "source" part is needed to make the subject / id reference globally unique.

in the text, but as "Serialised source + id / cdevents_platform + source + id / cdevents_source" in the table. And in the JSON example it seems to be on the format "id@subject" or maybe "id@source".

The id@source format is an example of how "source + id" could be serialised.

[afrittoli]

not all activities that require a cdevent may have a source-event associated. This happens typically with core events, where the end of a task run or pipeline run is not triggered by an event. In that case, when an activity is finished, there will be no source-event associated

In such cases it would make sense to me to refer back to the corresponding "XStarted" event. The direct "cause" for an activity ending is that it was started in the first place.

This was my initial thought, but ultimately I decided against it in my proposal.
I understand that it might help correlated things together, but it might also create confusion and it would impose a requirement that might be hard for application to satisfy.

The start event is not the trigger of the stop event, not directly, so I think it should be up to the consumer to correlate start and stop events if they want to. They can do so by filtering all events based on subject.

If we added the start event as a source for the stop we might have to do the same also for all the update events, which may not have a source event either. Granted, we don't really have "update" events in "cdevents" today, but we might in future.

For pipelinerun specifically, the initial event is actually "queued" and that would be the one to include the source event.

[m-linner-ericsson]

So again from an Eiffel perspective if we have links with "types" I think we could solve the problem.

For example Eiffels EiffelActivityFinishedEvent links back to the EiffelActivityTriggeredEvent via the ACTIVITY_EXECUTION link.

In CDEvents we could link our finished to our queued event.

@afrittoli Does this give any clue on the source question?

[m-linner-ericsson]

I would not add this to #19 for now. Hopefully we can merge #19 soon and adding more to it might slow the process down. We can use the cloudevents binding document to track specific extensions that we identify. as we go.

@afrittoli Sorry for my fussy language. I wanted to ask if this initiative described by your issue be a extension or should it be moved to the data part.

[afrittoli]

I would not add this to #19 for now. Hopefully we can merge #19 soon and adding more to it might slow the process down. We can use the cloudevents binding document to track specific extensions that we identify. as we go.

@afrittoli Sorry for my fussy language. I wanted to ask if this initiative described by your issue be a extension or should it be moved to the data part.

Heh, that was me not reading properly, sorry about that.
I lean towards making this an extension - or storing it into an existing extension if there is a relevant one.
The reason being that I can imagine a consumer wanting to filter events based on this information, and it would be great to be able to filter without having to dig into the body.

"Dig into the body" out of context sounds really bad 🧟

[m-linner-ericsson]

I would not add this to #19 for now. Hopefully we can merge #19 soon and adding more to it might slow the process down. We can use the cloudevents binding document to track specific extensions that we identify. as we go.

@afrittoli Sorry for my fussy language. I wanted to ask if this initiative described by your issue be a extension or should it be moved to the data part.

Heh, that was me not reading properly, sorry about that. I lean towards making this an extension - or storing it into an existing extension if there is a relevant one. The reason being that I can imagine a consumer wanting to filter events based on this information, and it would be great to be able to filter without having to dig into the body.

"Dig into the body" out of context sounds really bad 🧟

I do see the benefit for having the links outside of the body but that would limit us to strings (if I understand the CloudEvents spec correct). If we would do as in Eiffel and type them it would make more sense to have the link as an object but then we are forced inside the data part.

[afrittoli]

I would not add this to #19 for now. Hopefully we can merge #19 soon and adding more to it might slow the process down. We can use the cloudevents binding document to track specific extensions that we identify. as we go.

@afrittoli Sorry for my fussy language. I wanted to ask if this initiative described by your issue be a extension or should it be moved to the data part.

Heh, that was me not reading properly, sorry about that. I lean towards making this an extension - or storing it into an existing extension if there is a relevant one. The reason being that I can imagine a consumer wanting to filter events based on this information, and it would be great to be able to filter without having to dig into the body.
"Dig into the body" out of context sounds really bad 🧟

I do see the benefit for having the links outside of the body but that would limit us to strings (if I understand the CloudEvents spec correct). If we would do as in Eiffel and type them it would make more sense to have the link as an object but then we are forced inside the data part.

We could have both. The string is a serialised version, with limited information, used for routing.
The event payload could include the same info in object format, with more data if required.