transfer rules for routes and trips, and in-seat transfers

[antrim]

• route to route and trip to trip transfers:
  https://developers.google.com/transit/gtfs/reference/gtfs-extensions#TripToTripTransfers
• allow overrides to in-seat block transfers -
  https://groups.google.com/forum/#!msg/gtfs-changes/PkNAWbN9wcs/LFEqHGemBAAJ
• new field: agency.disallow_block_transfer

Updates [see below discussion]:

• agency.disallow_block_transfer field proposal has been removed.
• change Spec reference so that block transfers are not assumed by default

[antrim]

Notes and justification for proposal:

1. Some feed publishers with to specify trip-to-trip and route-to-route transfers. Google Maps has defined this in a GTFS extension, which is incorporated here: https://developers.google.com/transit/gtfs/reference/gtfs-extensions#TripToTripTransfers
2. Current interpretations of block_id are inconsistent. Real-time prediction systems such as Transitime require block_id, but the current GTFS assumes that in-seat transfers are always allowed. The new field agency.disallow_block_transfer and new values for transfer.transfer_type, 4 and 5 would allow overriding this default. This feature would work with the trip-to-trip and route-to-route transfer capabilities.
3. For backwards-compatibility, block transfers (in-seat transfers) are assumed to be allowed unless overriding parameters are set.

[antrim]

Does anyone have comments on these proposed changes? Since there are changes in a few files addressing a few issues, there is more here to think about than with some other recent proposed changes.

[barbeau]

@antrim Please remind me - do we have existing consumers and producers for this?

[antrim]

We do for route-to-route and trip-to-trip transfers (Google Maps is the consumer). But I don’t believe we do for other components of this proposal. Perhaps we need to put voting aside and recruit some consumers & producers.

[mgilligan]

+1

[abyrd]

OpenTripPlanner uses this GTFS extension (see https://github.com/opentripplanner/OpenTripPlanner/blob/master/src/main/java/org/opentripplanner/routing/core/TransferTable.java#L69), as does Google Maps. @antrim reports that Trillium is generating these transfers and I thought that OpenOV did for the Netherlands, but I don't see them in a recent feed. @skinkie are you still generating transfers with trip/stop/route specificity?

About "disallow block transfers" and the corresponding transfer types, I talked to someone at TriMet about this and they were in fact fine with exhaustively listing out all the allowed block transfers in their feed. It is going to be confusing to have some feeds that assume block transfers are allowed (where you have to individually disallow them) and other feeds that assume they are not allowed (where you have to individually enable them). I'm leaning toward just requiring all block transfers to be allowed individually, and if no such transfers are present in the feed, but block IDs are in use, then the consumer can take a guess at what that means.

[antrim]

Trillium is actually not yet producing these transfer types. We could very quickly add support for transfer_type = {4,5} but it would take more time to add route-to-route and trip-to-trip transfers because we will probably wait a month or two while transitioning to an overhauled version of the GTFS Manager software.

quoting @abyrd:

It is going to be confusing to have some feeds that assume block transfers are allowed (where you have to individually disallow them) and other feeds that assume they are not allowed (where you have to individually enable them).

I proposed a default value of 0 (NO) for the new agency.disallow_block_transfer field. This was designed to support backward compatibility with the spec today -- block transfers are assumed to be allowed unless otherwise specified.

[abyrd]

I should have been more clear. This proposal is really two different things. First, route-to-route and trip-to-trip transfers. Trillium, OTP, Google, and OpenOV (I believe) are already using this. Second, better handling of in-seat transfers, for which nothing has been implemented yet.

After all the mailing list and personal discussion, I would propose doing this a different way than previously suggested. In my view, in-seat transfers should never be assumed, and all in-seat transfers should be enumerated in transfers.txt using the trip-to-trip or route-to-route mechanism. Block_ids would only imply that a vehicle was passing from one trip to the next, and would only be relevant for consumers propagating real-time data. This means we only need one new transfer type (4) for adding an in-seat transfer.

In the event that a feed has no in-seat transfers listed but has block_ids, the GTFS consumer might want to infer that some in-seat transfers exist but this would be a heuristic based on knowledge about a particular agency, not a standardized behavior.

I'm aware that this could be considered breaking backward compatibility, but I don't get the sense that there was ever strong agreement on how this feature was supposed to work.

[DaveBarker]

+1 to the overall proposal and to abyrd's recommendation of approach that in-seat transfers should never be assumed.

[skinkie]

are you still generating transfers with trip/stop/route specificity?

Yes, and also for FlixBus.

[antrim]

I changed the proposal so that in-seat transfers are not assumed -- disallowed by default. This can be set with the new agency.block_transfer_default field. 0e149ca

[abyrd]

@antrim do we ever want to enable blanket block transfers using agency.block_transfer_default ? It doesn't seem clear that this should be enabled at the feed, agency, or route level, and I don't personally think this deserves cluttering the spec with a new field. Any agency technically sophisticated enough to consider using block IDs to communicate in-seat transfer availability can probably also pull off the transfers.txt approach.

[antrim]

@abyrd : We could remove the disallow_block_transfer.

However, my reasoning for block_transfer_default was that it allows any feed publisher that had assumed block transfers are allowed to publish a new feed that is consistent with the amended spec by adding only one field.

In public Trillium feeds that are provided to trip planners, we presently assume that block transfers are always allowed. This is consistent with how the Spec is written. For arrival prediction software, we actually have a function to provide different feeds (technically not Spec compliant) with block_id values for all trips (including where in-seat transfer is not allowed).

Trillium can quickly change the way we are publishing feeds. I worry that some other feed publishers assumed block transfers and it will take time for them to catch up with the Spec. I propose that we provide opportunity for a broader scan of practice.

For convenient reference, this was the proposed added field in agency.txt.

Field Name	Required	Details
disallow_block_transfer	Optional	Specifies whether in-seat transfers are allowed by default for the agency (see block_id field in trips.txt). Overriding parameters are specified in transfers.txt.
		* 0 or (empty) - In-seat transfers are disallowed unless an override is set in transfers.txt. This is the default behavior.
		* 1 - In-seat transfers are allowed unless an override is set in transfers.txt.

[abyrd]

I am mainly concerned about the proliferation of rarely-used fields that exist only for historical reasons, or to preserve backward compatibility when all producers are prepared to accept a small change in semantics to yield a new, clearer specification. Among the main benefits of GTFS are its simplicity and pragmatic approach. There should not be many different ways to represent the same thing, and special fields only used by one operator somewhere.

So ultimately this is going to depend on feedback from feed producers. We should also search for concrete examples of agencies that allow in-seat block transfers across their network. Do we have examples of this in the wild?

[antrim]

@abyrd @mgilligan : Point taken. I removed the proposed agency.block_transfer_default field. This aimed to provide a convenient bridge to forwards-compatibility, but would create Spec baggage, confusion, and burden over the long term.

[antrim]

I suppose we should remove transfer_type=5 because in-seat transfers are disallowed by default with this current version.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please let us know the company's name.

[antrim]

I signed it!

[antrim]

Summary of the latest proposal - 4b77a2d.

Original motivation

It is not presently possible to specify blocks for all trips (useful for arrival prediction software) AND to precisely specify in-seat transfer availability. The original (and currently published) Spec defines that "A block consists of two or more sequential trips made using the same vehicle, where a passenger can transfer from one trip to the next just by staying in the vehicle." (at developers.google.com)

What the proposal accomplishes

• Maintains backwards compatibility with the current Spec: in-seat transfers are allowed by default. (We could therefore remove the "non-backward-compatible" label from this pull request.)
• Allows changing the default of whether in-seat transfers are allowed for an agency
• Provide for granular control of in-seat transfers (by stop, trip, and route)

Why the 360° turn back to the original proposal?

Off-list discussion has brought up the need for backwards compatibility, given the number of feeds that presently follow the spec and are integrated in present systems. This is an attempt to minimize disruption to existing feed producers and consumers.

[Update: Sep 16, 2020] Here is a link to a Slack conversation in 2017 that shows of the research on block_id practices from the time.

Suggested path forward

Let's continue to discuss on GitHub for as long as it takes to arrive at a solution. Given that this has lingered for some time (and the original motivating problem has not been solved), I suggest we attempt to limit discussion to 2 weeks from today before bringing this proposal, or any revised proposal, to a vote.

[barbeau]

I'm a +1 on this proposal as current proposed in 4b77a2d, but this PR needs to be updated before it can be merged.

@antrim Looks like some changes unrelated to this proposal snuck into this pull request - I'm flagging them here so they don't accidentally get merged. You should be able to get rid of these updating your local branch to the google/transit master, creating a new branch temp from the updated master branch, and then git cherry-pick 4b77a2da51a74f1362df577bb6771e46bc04dc4d to pull 4b77a2d into your temp branch and then force pushing to your Github remote branch transfers-inseat_and_trips/routes with git push -f origin temp:transfers-inseat_and_trips/routes (hopefully I got that syntax right). Just be aware that the force-push will destroy whatever is currently in this PR, so keep a local backup if you're unsure of exactly what will happen. We've all been bitten by Git at least once ;).

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[antrim]

Hi @barbeau : The additional transfer types are intended to be part of this proposal/PR. If block transfers are turned off by default for an agency (as many agencies will likely choose), then they will also need a way to override and allow block transfers in specific cases.

[barbeau]

Ok, thanks - makes sense.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[antrim]

Intentional to provide additional capabilities to manage transfers broadly, and block transfers as a subset.

[barbeau]

Ah, right - gotcha.

[barbeau]

Unrelated to this proposal - needs to be removed from PR.

[antrim]

@barbeau Your instructions worked perfectly. Thank you. The proposed changes are now all in 03a4592.
.

[devinbraun]

@antrim We can make the change block_transfer_default immediately as our block_ids are vehicle tasks (for real-time information) and aren't specific to in-seat transfers which we don't allow except if it's another trip on the same route (for example, a looping route). We can add in the route-to-route in-seat transfers to transfers.txt for those few routes where it is allowed.

[davidlattimore]

Something I had been thinking about, that's related to this, is when two trips share a block_id, but the later trip starts from a different stop than the earlier trip finishes. Currently Google still allows an in-seat transfer. I'm wondering however if perhaps it shouldn't. I've observed at least some data providers that have given us such trips who didn't intend for them to be in-seat transfers.

GTFS best practices says:

If in-seat (block) transfers are allowed between trips, then the last stop of the arriving trip must be the same as the first stop of the departing trip.

I wonder what proportion of unwanted in-seat transfers would go away if we only allowed in-seat transfers when the last/first stops matched up.

Regardless of the more explicit rules in this proposal as to whether in-seat transfers are allowed, it'd be good to clarify the expected behavior when the stops don't match. Is it an error? Is it OK, but in-seat transfers are disallowed, even if the default is to allow them? The best practices doc seems to suggest the latter.

Also, there's still a Non-Backwards-Compatible label on this pull request. My impression is that it's no longer required, right?

[antrim]

tl;dr: Should we even allow blocked trips where the last stop of the earlier trip is different than the first stop of the subsequent trip? This strikes me as problematic and unnecessary. Longer discussion below.

Quoting @davidlattimore :

Something I had been thinking about, that's related to this, is when two trips share a block_id, but the later trip starts from a different stop than the earlier trip finishes. Currently Google still allows an in-seat transfer. I'm wondering however if perhaps it shouldn't. I've observed at least some data providers that have given us such trips who didn't intend for them to be in-seat transfers.

Question to all -- particularly feed producers: What would be the purpose of specifying subsequent trips where the later trip starts from a different stop than the earlier trip finishes at? @davidlattimore: Are there particular feeds you can point to that do this?

How should a feed consumer interpret a case where the last stop of the earlier (blocked) trip and first stop of the subsequent trip are different? This strikes me as problematic, especially in cases where a real-time system is consuming the GTFS. Opinions / insights?: @scrudden @mike-swiftly @barbeau @nselikoff

If the vehicle goes into non-revenue service, that can be represented in GTFS (see Best Practices RE deadhead trips: https://gtfs.org/best-practices/#stop_timestxt).

[skinkie]

Should we even allow blocked trips where the last stop of the earlier trip is different than the first stop of the subsequent trip? This strikes me as problematic and unnecessary. Longer discussion below.

If Trip 1 ends at a drop-off only stop, and Trip 2 starts at the pick-up that is, in The Netherlands at least, a very common situation to get disembarking passengers as close to the train station entrance as possible, while picking up new passengers at a dedicated platform. The block will not start and end in the same stop.

[davidlattimore]

Sure. Here's a couple of feeds:
https://bkk.hu/gtfs/budapest_gtfs.zip
http://opendata.tec-wl.be/Current%20GTFS/TEC-GTFS.zip
I have no idea if they're representative. Both have a mix of block transfers where the first and last stop match and ones that don't.

[skinkie]

@antrim a compromise might be to state that a block should end and begin at the same parent station. Given the entrances proposal, every station will get one anyway.

[mike-swiftly]

Sorry for the late reply, but with interlining, where a vehicle might cover multiple routes with a single block could also cause differences between last stop of a trip and first stop of next trip. Interlining is pretty common for larger agencies. *Michael Smith* *CIO, **Swiftly, Inc* (415) 260-4700

…

On Tue, Jul 30, 2019 at 1:39 PM Stefan de Konink ***@***.***> wrote: @antrim <https://github.com/antrim> a compromise might be to state that a block should end and begin at the same parent station. Given the entrances proposal, every station will get one anyway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32?email_source=notifications&email_token=AGROG6MSAIUK7JLPJZNCSD3QCCRI7A5CNFSM4CXTB3UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3FH2AY#issuecomment-516586755>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGROG6KUCJGQ4WVE3MLXW3DQCCRI7ANCNFSM4CXTB3UA> .

[abyrd]

Thanks @mike-swiftly. It also seems completely normal to me to allow in-seat transfers between sequential trips in blocks, independent of whether the last stop of trip 1 is the first stop of trip 2. I would expect this to be very common, and maybe even the most common case (any time route 2 begins at a stop a bit farther down the road from where route 1 ends).

I am not sure what we would accomplish by creating more rules about when in-seat transfers are allowed or not. It seems like any such set of more complicated heuristics would give better results for certain agencies and worse results for others, while requiring every consumer to implement more complex code, so I'd prefer the simplest possible behavior. When the broad backward-compatible default is used to enable in-seat transfers, I would expect in-seat transfers to then exist after every trip within each block. Each consumer could perhaps implement a few sanity-check heuristics of their own design for good measure, but such things should not be part of the spec.

If a producer needs to communicate more nuanced information they can disable in-seat transfers by default and employ transfer_type 4. Based on input in the comments above about existing feeds, we do need a field creating backward-compatible behavior, and allowing producers to switch that behavior off and use transfer_type 4.

[abyrd]

I have just re-read most of the comments and the current content of the pull request. I'm still quite supportive of having a way to individually enable in-seat transfers, using transfers.txt. Some comments on the current state of the proposal:

transfer_type 4 and block_id

The text describing transfer_type 4 should be updated to clarify the reference to block_id. To me, block_ids do not seem relevant when a producer is listing individual transfers instead of using the backward-compatible block-based default in-seat transfer behavior. Also, consider that the block_id field only exists for trips. This PR allows transfer_type 4 transfers to be created between routes or stops, not just trips, in which case that transfer should (?) still be available to the rider even in the absence of block_ids. Alternatively, if you really do want to enforce creation of in-seat transfers only between trips with the same block_id even when the enabling transfers.txt entry is between stops or routes, then this behavior needs to be very clearly defined in the spec, including a description of what happens when trips do not have defined block_ids but are nonetheless tied together by a transfer_type 4 transfer. Some additional information should also be given clarifying whether the transfer_type 4 only applies when it connects the last stop of the first trip to the first stop of the next trip (i.e. what if the transfer is between stops that are not at the end of a trip).

My proposed system (which of course needs scrutiny from other people) is this: When default in-seat transfers are enabled, in-seat transfers are available after every trip in every block, with no restrictions. Whether or not this default is enabled, transfers with the new transfer type have the same function: they allow in-seat transfers from the final stop of all trips that meet its from_stop_id, from_route_id, and from_trip_id criteria, toward the initial stop of all trips that meet its to_stop_id, to_route_id, and to_trip_id criteria, with no regard to block_ids at all. In this way, the new transfer_type can be used to exhaustively specify all in-seat transfers, or to augment default in-seat transfers inferred from block_ids.

It definitely makes sense for a producer using transfer_type 4 to still include block_ids for realtime propagation purposes, but how block_id meaning and effect changes in response to the default in-seat transfer flag must be clearly described.

transfer_type 5

I am thinking transfer_type 5 should be eliminated. This leaves only two modes of operation: default in-seat transfers enabled, augmented by any transfer_type 4 transfers; or default in-seat transfers disabled, with all in-seat transfers exhaustively listed with transfer_type 4. Since the mode with default in-seat transfers enabled exists primarily for backward compatibility, I am hesitant about additional complexity allowing producers to subtractively pare down the full default set of in-seat transfers.

[gcamp]

I tried to read as much as possible from this proposition and comments, but I didn't read everything. I'm sorry if I'm repeating discussion that happened before.

My biggest problem with block_id is how they are hard to interpret since they can change depending on the current service day, which is a pain (see this example). The trip to trip transfer solves this.

My proposition would be to actually deprecate completely block_id and replace it with trip to trip with transfers type 4 and 5. If block_id are deprecated, that also means we can remove the block_transfer_default value. from_route_id and to_route_id are also not required in this PR and can be moved to an other proposition.

If that makes sense for everybody we can implement this fairly quickly in Transit. We can also potentially open source a convertor that transforms GTFSs with block_ids with GTFSs with trip to trip transfers. I know this is a big change, but in my opinion it would really clarify the spec and reduce the amount of potential bugs in both producers and consumers.

For those interested, we came to this conclusion because we are actually fixing bugs in our block_id support (bug that is basically the example from the spec). Our final solution was basically have a first step of transformation of block_id to trip to trip transfers inspired by this proposition to simplify processing later on. Instead of duplicating work, we could just share our transformation step and hopefully become part of the spec.

[skinkie]

@gcamp I consider this certainly a very good direction to go to. Considering how blocks in GTFS never represent what they are in more complex systems downstream unless all data is denormalised. But I do want to suggest something that I hope you could take in a account. Would your consider adding an optional service_id to the trip_to_trip transfers?

[gcamp]

@skinkie could you have the same trip to trip transfer with multiple service_id? You want that to avoid duplication of trips just to create different continuations?

In our case we would want to duplicate the trip anyway to simplify storage of trip to trip transfers, but that can be done on our side. If this reduce size and complexity a lot for producers, that's something we can support.

[skinkie]

@gcamp I indeed want to prevent that new trips are created (while the internal trip identifiers are equal) that on certain dates will not share the same block id therefore would not have that trip-to-trip transfer.

[gcamp]

Bump on proposition above ☝️

[paulswartz]

Would this deprecate block_id only for in-seat transfers, or entirely? I imagine it is still useful for consumers to know that two different trips are scheduled to be provided by the same vehicle.

[gcamp]

@paulswartz I would deprecate entirely. If the producer wants to provide the information about two different trips are provided by the same vehicle could do so with a trip to trip transfer type = 5.

[paulswartz]

@gcamp

@paulswartz I would deprecate entirely. If the producer wants to provide the information about two different trips are provided by the same vehicle could do so with a trip to trip transfer type = 5.

Hmm: our current GTFS has ~85k trips on ~5k blocks; that's a lot of trip-to-trip transfers to add to the file. But if that's what the consensus is, we can make it work.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gcamp]

My proposition above is still valid on my side. It's one of the next thing we want to improve in our backend.

[antrim]

Closing this PR. This function is covered in #278.