GTFS Trip-Modifications

[gcamp]

Following the proposal of GTFS Trip-Modifications in February, we are now ready for the pull request stage!

As a reminder, Trip-Modifications are modifications done to a trip to modify its shape, remove stops that are not served anymore, and potentially add temporary stops. Trip-Modifications is mainly used in a detour use case.

Today, we're launching the detour feature in Transit for more agencies with Trip Modifications produced by Swiftly! Now that there is one producer and one consumer, the current PR is not an experimental field and actually removes the experimental tag for Shape, which has been in the spec for 2 years.

Feel free to ask questions, or give feedback!

Edit: see example feeds here.
Latest changes before a potential experimental vote are up

Example of the feature in practice:

[doconnoronca]

Can you provide an example trip updates file with a detour?

[skinkie]

Now that there is one producer and one consumer, the current PR is not an experimental field and actually removes the experimental tag for Shape, which has been in the spec for 2 years.

I don't think that one consumer and one producer justifies it from removing it as experimental, neither it being available for two years. The usage is just too sparse. My expectation as a producer is that more than half of my known consumers would be supporting this.

[leonardehrenfried]

One question: do these detours happen so spontaneously that you need a realtime update for that?

I would have thought that you know them at least a couple of days in advance and could put them in a static GTFS feed. (I don't work for an agency though.)

[skinkie]

@leonardehrenfried we have certainly seen spontaneous reroutes, for example for tram rail blockades.

[willcanderson]

Unfortunately, my agency often has to implement detours with no notice. Medical emergency, police activity, public demonstration--lots of unplanned stuff can clog a street.

Even if we did always have a day or two of advance notice, producing a new GTFS Schedule file, publishing it, and making sure all data consumers ingest it in that span of time is a dicey proposition. Even Google prefers receiving a new GTFS Schedule file several days before the changes go into effect.

So I see being able to publish detours in a realtime way as an acute need.

[skinkie]

So I see being able to publish detours in a realtime way as an acute need.

That does not mean magically journey planners can implement such functionality. Otherwise the delay for processing the actual schedule wouldn't be a thing :-)

[stevenmwhite]

That does not mean magically journey planners can implement such functionality.

I think the fact that this is launched in Transit already proves that it can be implemented in rider-facing apps. I didn't say "journey planners" because I'm not exactly sure how it shows in the planned trip part of Transit app, but I'm sure @gcamp can answer that. And for what it's worth, there's always some differences in what various consumers support... Apple, Transit, Google, and others all launched support for trip cancelations at different times, some show real-time passenger load, some don't, etc...

We're planning to support this as a producer in whatever way it's adopted soon and are in favor of the general concept. Some others on my team are going to review the details and we'll offer specific comments if we have them, but it's a general +1 from GMV.

[doconnoronca]

Most transit trips are taken on routes riders are already familiar with, so they don't need journey planners, but they do use real-time predictions and need to know if their route is detouring and where to get it.

[sam-hickey-arcadis]

We’re still reviewing but the following paragraph stood out during an initial review:

Each affected trip is deemed to be canceled and replaced with a new trip. The affected trip can be omitted from the feed while it is detoured. Each replacement trip has an automatically generated ID and shares all of the properties defined in trips.txt for the affected trip it replaced. Replacement trip IDs can be used in the same way as any other trip_id in GTFS-RT. For example, TripUpdates can be provided using the replacement trip ID.

What schedule_relationship is associated with these trip cancellations? Probably DELETED?

It would be helpful to understand the reasoning behind the decision to treat all trip_ids within a trip modification as cancelled. This has potentially major implications for an agency’s real-time data generation systems as all predictions in the Trip Updates feed and locations in the Vehicle Positions feed and alerts in the Service Alerts feed will need to use the new replacement trip_id (modifications_id + ‘_’ + trip_id) since the original trip_id has been cancelled. This may not be as much of an issue for agencies that use one system for all GTFS-realtime feeds, but for the many agencies that have multiple systems working together for generating and consuming real-time data this will be a heavy lift to keep all trip_ids in sync across all feeds.

[skinkie]

I think the fact that this is launched in Transit already proves that it can be implemented in rider-facing apps.

Oh sure. You can implement many things if the performance does not really matter, and falls completely outside the scope of the journey planner algorithm used ;) So if that algorithm needs the complete network, or would need to recompute in order to facilitate transfers, that has an impact. And sure, you can implement this trivially using CSA. Now lets discuss multi day detours... which is allowed in this pull request.

[gcamp]

Can you provide an example trip updates file with a detour?

@doconnoronca good point, I'll follow up soon with this.

I don't think that one consumer and one producer justifies it from removing it as experimental, neither it being available for two years.

@skinkie From the Specification amendment process, we only need one producer and one consumer to move it to the main spec. And at the opposite, we're supposed to either delete or promote to the main spec any experimental fields after 2 years (that last part hasn't been followed very well). All the non-experimental spec changes that happened since the spec is on Github either have 3 or less producer and consumer combined : #229 #214 #210

That said, I understand this is a big spec change and if the community is more comfortable being an experimental field that can work for us.

I would have thought that you know them at least a couple of days in advance and could put them in a static GTFS feed.

@leonardehrenfried other gave good examples but I'll add that the best practices mention, "If a service modification will go into effect in 7 days or fewer, express this service change through a GTFS-realtime feed (service advisories or trip updates) rather than static GTFS dataset."

Oh sure. You can implement many things if the performance does not really matter, and falls completely outside the scope of the journey planner algorithm used 😉

@skinkie @stevenmwhite our implementation affects our nearby functionality, route detail (where you see the shape, etc.) and also the trip planner. Temporary stops are taken into account, as you would expect. Performance-wise, there's no degradation that anyone would notice. On processing time, we're able to process the new data in a few minutes in the vast majority of the cases. We need to keep in mind that detours don't affect the entire system, but if a Trip-Modification feed affects every single route, our processing time would be similar to a new GTFS (but for us that takes minutes and not hours).

I understand this feature might be hard to implement for some consumer. Happy to take feedback on how to improve this. However, I don't think this would be a reason not to approve Trip-Modifications, especially that Trip-Modifications are strictly improvement for consumers that supports it and doesn't affect the consumer that don't.

What schedule_relationship is associated with these trip cancellations? Probably DELETED?

Yes @sam-hickey-ibigroup, they would be DELETED as not shown in the UI. The idea of the new trip_id was to be able to use stop_sequence and other values that might change in the trip with the modification.

I agree compatibility with Trip-Modification aware systems and non-Trip-Modification aware system might require a bit more thought. Let me know if you have suggestions; otherwise I'll work on improvements around backwards compatibility.

[skinkie]

@skinkie From the Specification amendment process, we only need one producer and one consumer to move it to the main spec. And at the opposite, we're supposed to either delete or promote to the main spec any experimental fields after 2 years (that last part hasn't been followed very well). All the non-experimental spec changes that happened since the spec is on Github either have 3 or less producer and consumer combined : #229 #214 #210

This is something I also want to see addressed in the changes towards active particaption, far more important than mentioning where people work, or prevent trolling the spec. In my perspective a single vendor on either side does not cut it for 'promotion', that could still imply that there is proprietary work being done that no other party is going to be picked up, it being too complex or worse introduce patented processes. I obviously do like a consumer and producer validating the work before we merge it.

Oh sure. You can implement many things if the performance does not really matter, and falls completely outside the scope of the journey planner algorithm used 😉

@skinkie @stevenmwhite our implementation affects our nearby functionality, route detail (where you see the map, etc.) and also the trip planner. Temporary stops are taken into account, as you would expect. Performance-wise, there's no degradation that anyone would notice. On processing time, we're able to process the new data in a few minutes in the vast majority of the cases. We need to keep in mind that detours don't affect the entire system, but if a Trip-Modification feed affects every single route, our processing time would be similar to a new GTFS (but for us that takes minutes and not hours).

I know CSA and RAPTOR are unlikely to have issues. Our planner does not even care about how the detour is provided, as long the stops are known a priori. My case would be: could you do this kind of changes with transfer patterns?

I understand this feature might be hard to implement for some consumer. Happy to take feedback on how to improve this. However, I don't think this would be a reason not to approve Trip-Modifications, especially that Trip-Modifications are strictly improvement for consumers that supports it and doesn't affect the consumer that don't.

I think we also need to start a chat about a producer. Hence some producers can not do this, only operationally implement this via a CANCEL and ADDED (or what the two are called these days).

[sam-hickey-arcadis]

I agree compatibility with Trip-Modification aware systems and non-Trip-Modification aware system might require a bit more thought. Let me know if you have suggestions; otherwise I'll work on improvements around backwards compatibility.

We (Arcadis IBI Group) are going to brainstorm some ideas related to this and review this proposal in more detail from producer and consumer perspectives. We’re planning to share our ideas and feedback towards the end of this week or next week.

[doconnoronca]

Using the existing trip ids should allow consumers who haven't been set up to handle this to work better. The stop sequence could be excluded for the stops on the detour.

[skinkie]

Using the existing trip ids should allow consumers who haven't been set up to handle this to work better. The stop sequence could be excluded for the stops on the detour.

This directly breaks anything that asumes no update still exists.

[mads14]

@sam-hickey-ibigroup
Regarding the compatibility: right now Swiftly allows consumers to request the GTFS-rt trip updates feed with or without trip-modifications. If a consumer requests the feed without trip-modifications then we keep the normal trip ids and simply mark stops skipped by a detour as "skipped".

If a consumer can support trip-modifications then they fetch the version that has this feature enabled and will get more details on the detour shape, temp stops, etc. We're definitely open to ideas if there's a way to better support backward compatibility with the trip-modifications feature enabled. The approach we have now seems to be working well for us so far.

[doconnoronca]

This directly breaks anything that asumes no update still exists.

What is the worst that would happen? That there would be no predictions for that trip? If a different trip id used, would have the same effect. I expect most consumers are focused on getting the prediction for the stop without worrying about extra stops it doesn't yet understand.

[skinkie]

What is the worst that would happen? That there would be no predictions for that trip?

Yes, that the provided that is fully ignored because the provided data is ambigous and the producer thinks everything is a-ok. In 2013 we have made a clear description between directly consumable GTFS-RT and data that still requires integration. Deliberately leaving out a stop sequence number is one of the worst things to do.

[leonardehrenfried]

I generally like this proposal because it introduces clear, usecase-specific updates rather than the TripUpdate over which there is a lot of confusion on how to interpret.

Generally speaking I would be in favour of replacing any implicit fields, like the automatically generated replacement trip ids or a missing time of a stop time, with something explicit as this simplify the implementation of this feature.

Also I'm missing a little bit of language on how to apply TripUpdates to these modified trips.

[sam-hickey-arcadis]

tl;dr The changes proposed here have the potential to allow a huge leap forward in how transit agencies communicate detours to customers, but the way the proposed changes are written now to cancel and replace all trip_ids included in a trip modification will prevent widespread adoption of these enhancements. We propose some changes related to canceling and replacing trips along with changes and comments for other parts of the proposal.

Here is our detailed feedback.

Changes to automatically canceling and replacing trips

The TripModifications section says this:

Each affected trip is deemed to be canceled and replaced with a new trip. The affected trip can be omitted from the feed while it is detoured. Each replacement trip has an automatically generated ID and shares all of the properties defined in trips.txt for the affected trip it replaced. Replacement trip IDs can be used in the same way as any other trip_id in GTFS-RT. For example, TripUpdates can be provided using the replacement trip ID.

The stop times of each replacement trip are created from those of the affected trip, by performing the changes listed in modifications. Stop sequences are assigned from 1 to n, ignoring the stop_sequence values of the affected stop times. If a particular stop time is not affected by the detour, all other values will match those originally defined in stop_times.txt.

We would like to see the portion of this that automatically cancels trips removed. We agree there should be an option for consumers to cancel trips, but requiring cancellation has many major implications for maintaining consistency between all of an agency’s real-time feeds. We expect it will be years, if not longer, before many agencies are able to update their systems that produce and consume GTFS-realtime to support automated cancellation and replacement of trips affected by trip modifications.

While it makes sense to cancel and create replacement trips in cases where the scheduled stop times of trips change, transit customers will still receive significant benefits if an agency publishes trip modifications without any changes to stop times. For example, publishing a TripModifications message with just the required fields + shape_id + replacement_stops (without the associated travel_time_to_stop) would be enough for a consumer to know the new shape and stop locations for the given trip(s). This would not be enough to show stop times for the replacement stops, which would prevent some functionality for some consumers (like trip planners), but it would still communicate valuable information to transit customers.

In cases where an agency wants to cancel and replace trips, we would like to see the following explicitly added to this proposal:

1. Trips are canceled only if (1) the trips are output in a TripUpdates feed with SCHEDULE_RELATIONSHIP = DELETED or (2) a new field like delete_trips is added to TripModifications (0/NOT_DELETED is the default and means trips are not deleted; 1/DELETED is the other option and means all trip_ids are assigned SCHEDULE_RELATIONSHIP = DELETED)

2. Replacement trips could be defined in one of the following ways. We’re interested in feedback from others to narrow this down to one preferred option.

   a. Using GTFS-NewTrips. This will provide more explicit definition of new trips and allow for setting stop time properties to their non-default values. For example, some agencies allow pickup and drop off anywhere along a detour route, so these agencies may want to set non-default values for continuous_pickup and continuous_drop_off along the detour path.

   b. Change the following from this proposal:

   The stop times of each replacement trip are created from those of the affected trip, by performing the changes listed in modifications. Stop sequences are assigned from 1 to n, ignoring the stop_sequence values of the affected stop times. If a particular stop time is not affected by the detour, all other values will match those originally defined in stop_times.txt.

   To the following:

   The scheduled stop times of each replacement trip are created from those of the affected trip, by performing the changes listed in modifications. Stop sequences are assigned from 1 to n, increasing by 1 for each stop in the trip. A TripUpdate message must be provided to publish real-time arrival/departure times for the replacement trip.

   We’re suggesting these changes (1) in order to distinguish between scheduled and real-time data for replacement trips, (2) to handle the case where there are more replacement stops than num_stops_replaced and (3) because stop times not part of the detour may still have their stop_sequence value change.

@mads14 To your points on backward compatibility, yes, your current approach sounds reasonable for maintaining backward compatibility for consumers. But without the changes noted above, we expect producer adoption of GTFS Trip-Modifications will be limited.

Clarify replacement stops do not have to define changes to travel time

In ReplacementStop replace this:

Each ReplacementStop message acts as a template for generating a new stop time in each modified trip. The generated stop time will have the specified stop_id.

With this:

Each ReplacementStop message defines a stop that will now be visited by the trip and optionally acts as a template for generating a new stop time in each modified trip. The replacement stop and optional generated stop time will have the specified stop_id.

Add last_modified_timestamp to TripModifications

This would make it easier for consumers to know which TripModifications have changed.

Clarify the feed where TripModifications are published

Our preference is for TripModifications to be published in a separate feed (TripModifications.pb). This would make it easier for all consumers to know where to find them. Our vote is to make this a “MUST”, but we would be ok with “SHOULD” to match the fact that GTFS-realtime doesn’t currently require separate feeds/files for TU, VP, and alerts.

Grouping trip modifications by stop pattern

We would like to hear input from other producers and consumers about this:

A TripModifications message identifies a list of similar trip_ids from the (CSV) GTFS sharing the same stop pattern, which are all affected by a particular modification, like a detour. The modification is in effect on all of the listed service_dates, until it is removed from the feed. On any given service date, a trip MUST NOT be assigned to more than one TripModifications object.

As noted at #369 (comment):

This has the potential to greatly increase the amount of data needed to communicate a single detour. For example, a detour that applies to 3 stops served by 5 routes that each have 5 stop patterns would require 25 TripModifications that all contain the same data other than the list of trip_ids. A detour on 3rd Avenue in Downtown Seattle is just one example of one detour affecting many routes and stop patterns (see map here. Is there a reason grouping trip_ids by stop pattern was chosen (perhaps to allow more fine-grained (at the stop pattern level) control of adjusted travel times?) rather than other options like defining the affected route, trip, stop, etc. as done in an alert’s informed_entity?

We understand the benefit of grouping by trip_ids by stop patterns as noted in @gcamp’s reply in issue 369, but we have some concern about the inefficiencies or repeated data that will result from this requirement.

Finally, a big thank you to Transit and Swiftly for all the initial work on this. The results of this effort for the first 3 agencies using it is truly impressive. Looking forward to working together to refine these spec changes to encourage and enable more widespread adoption and availability of this data.

[npaun]

Hi @sam-hickey-ibigroup,

Thanks for the detailed feedback. I work with @gcamp at Transit on the GTFS-TripModifications spec, and next week we'll be making some changes to the proposal to clarify and address these comments.

Changes to automatically canceling and replacing trips

The trips created through GTFS-TripModifications modify and replace each specified trip_id, and don't create a copy or additional run. However, it remains possible to provide TripUpdates and VehiclePositions for the original trip.

We'll clarify the spec to provide two explicit options:

1. The producer may provide TUs for the replacement trip ID (modifications_id + _ + trip_id).
2. If none is provided, TUs for the original trip_id shall be used with the limitation that ETAs will not be available at the replacement stops.

Clarify replacement stops do not have to define changes to travel time
Clarify the feed where TripModifications are published

Wording will be improved in both of these areas.

Add last_modified_timestamp to TripModifications

We wouldn't consume this field, as we compute a hash over trip modifications to look for updates instead.

Grouping trip modifications by stop pattern

In our definition, a stop pattern is the broadest possible grouping of trips that allows the start and end of each modification to the trip to be unambiguously identified by start stop sequence + # of stops replaced.

In principle, detours could be defined by route and direction, but then I think the start and end of each modification would need to be identified by stop ID, which adds complexity and won't work on loop routes with repeated stops.

[eliasmbd]

I just want to take the time to thank @gcamp for taking the lead on this. MobilityData is actively analyzing the Pull Request and reaching out to Stakeholders that might be concerned by this change.

We will share our findings with you here and provide a few comments soon.

We also stand ready to support @gcamp 's efforts in reaching out to the community or provide a place for discussion if needed.

We look forward to seeing the outcome of this RP.

[sam-hickey-arcadis]

@npaun Looking forward to seeing your updates next week.

We wouldn't consume this field, as we compute a hash over trip modifications to look for updates instead.

Even if Transit wouldn’t use this field, it would still be valuable to add. Other consumers may want to use it to know which TripModifications have changed and/or display to customers when the modification was last updated.

One other thing that would be good to add within the TripModifications message is the alert ID(s) associated with the modification. This new optional field could be something like associated_alert_ids and would provide one or more alert IDs (FeedEntity.id) that are associated with the modification. This would allow consumers to provide additional context from the alert (like header_text, active_period, cause, etc.) along with the changes communicated by the modification.

[gcamp]

We pushed two updates :

• Mostly editorial changes and creating a trip-modifications.md file like already existing for other types of GTFS-rt updates.
  • @sam-hickey-ibigroup does that handle your comment "Clarify the feed where TripModifications are published"? Trip-Modifications would be handled the same way as other types of feed : A feed may, although not required to, combine entities of different types.
• Clarifications about interaction between systems that handle Trip-Modifications and those that don't
  • @sam-hickey-ibigroup I think that was your main issue. Let me know if that helps
  • @leonardehrenfried also include your proposal about the language on Stops

[gcamp]

Also as requested, here are two example feeds from Swiftly (thanks @mads14 !) :

• Maryland Transit Administration (Baltimore) : Trip-Modifications feed and corresponding GTFS
• Green Mountain Transit : Trip-Modifications feed and corresponding GTFS

On the Trip-Modifications feeds, you can remove the format=json if you want the protobuf version cc @doconnoronca

Finally, an update on existing consumer/producer : we also have the STM using Trip-Modifications on their website. We have some producers at different stages of implementation but nothing to announce yet.

[doconnoronca]

-1 from TransSee

The proposal is inflexible and overly complex compared to adding trip updates to describe detours. The issues with file size can be addressed by putting the upcoming changes in a separate feed, reducing the excessively high period that they are excluded from the GTFS and making the link between the real time data and the schedule clearer.

[leonardehrenfried]

I didn't follow this proposal very closely over the past few months but I'm pleased to see that the concatenated ids have been replaced which was my biggest problem with this.

I have a question about the protobuf schema having lots of optional fields but generally I'm happy with this proposal.

[BodoMinea]

+1 FlashWeb as a producer.

We have encountered several situations in which unplanned utility works, accidents or weather cause short notice detours, and it is definitely something that degrades the experience of users that plan their trips through open data consuming apps. We'd really appreciate a standard way to quickly exchange this kind of information and therefore provide a better overall experience to riders.

We have experimented with canceling affected trips and publishing image links to generated maps of the detour, therefore making use of the existing Trip Updates functionality to handle the mentioned cases, but it is absolutely a subpar solution, in terms of impossibility to filter by accessibility features of new stops or modified trips, navigating to those locations or just comprehension of the message by international users.

This PR solves our issues as is and having it adopted by high profile consumers would constitute the business case presented to operators in order to put in the time and resources to upgrade their systems with this extended functionality.

[sam-hickey-arcadis]

If a provider wanted to make a modification that only added new stops and didn't cancel existing stops, am I correct that the only way to do that is to specify an existing stop with StopSelector and make sure that existing stop is re-included as a replacement stop? I mostly just wanted to make sure I understood the semantics correctly.

My interpretation is that to not cancel any stops the producer needs to provide start_stop_selector and not provide end_stop_selector.

But it's not clear to me how propagated_modification_delay is used in a case like this. Does propagated_modification_delay get applied from start_stop_selector, so this stop is both the start and end of the modification?

And to confirm, when no stop times are missed, the travel_time_to_stop is still calculated from the stop before start_stop_selector?

[gcamp]

Thanks @jfabi @bdferris-v2 @sam-hickey-ibigroup for the comments! I applied directly most of your proposals.

But it's not clear to me how propagated_modification_delay is used in a case like this. Does propagated_modification_delay get applied from start_stop_selector, so this stop is both the start and end of the modification?

That's right that wasn't clearly defined, I added some clarification here.

[leonardehrenfried]

Thanks for the clean up and the clarifications.

+1 OpenTripPlanner

[gcamp]

With the updates I did yesterday, I'm hesitating on cancelling this vote because some of the changes might be not be considered editorial changes. The two main one being : this and this.

If anyone feels we should cancel this vote, please let me know and I'll do so (can be privately too).

[yannick-b]

+ 1 from SEPTA

[eliasmbd]

• 1 from SEPTA

Thank you for taking the time to vote @yannick-b . Unfortunately, we cannot tell wether you are voting + or - 1 with the current format. Please revise your vote to reflect your position on the matter. Thanks

Edit: vote was corrected

[sam-hickey-arcadis]

+1 Arcadis (formerly IBI Group)

[jfabi]

+1 from the @mbta.

[gcamp]

The vote has concluded.

+1

• @evansiroky Caltrans
• @westontrillium Trillium
• @mads14 Swiftly
• @matikin9 LA Metro
• @SteveGladstone Maryland Transit Administration
• @BodoMinea FlashWeb
• @leonardehrenfried OpenTripPlanner
• @yannick-b SEPTA
• @sam-hickey-ibigroup Acardis
• @jfabi MBTA

- 1 :

• @doconnoronca TransSee

Since this is an experimental field, the vote passes. Thanks everyone!

[kylerchin]

When will this be merged?

[eliasmbd]

The vote has passed. GTFS Trip-Modifications will be merged to the spec. Trip-Modifications shall be experimental for at most 2 years.

As stated in the GTFS Realtime Amendment Process:

If the experimental field is not adopted via the Specification amendment process within 2 years of being approved as an experimental field, it will be deprecated by adding [deprecated=true] next to the field value in the .proto file file.

Thank you to everyone that participated in the development and see you in less than 2 years.

[doconnoronca]

His anyone implemented the updated version of Trip-Modifications that uses the ModifiedTripSelector? The regular Maryland MTA feed from Swiftly doesn't include it and the version with "enable-feature=trip-modifications" appears to still use the original method.

[gcamp]

We're working on updated implementation right now