Specify the usage of token queues ("streams") for I/O #221
Conversation
This change renames the Encoding-specific concept of "streams", which had been causing confusion with readable/writable streams for years, to "token queues". It also exports the corresponding definitions. Closes #180.
…nces. Also refactors the BOM sniff algorithm to not use conversions.
For a long time, token queues (formerly "streams") had been defined as simple data structures, with no reference to I/O except for a few uses of a "wait until" operation not defined anywhere and a comment on the spec's source saying that the read operation blocks on I/O. This change introduces the concept of "blocking tokens", which are allowed to block when being read from the token queue. This makes it possible for token queues to be used as a direct spec representation of implementation-internal I/O streams, while all other operations done on token queues use immediate tokens. See also #215, which introduced the name "token queues" and defined the implicit conversions with strings and byte sequences.
|
This seems like a good idea, but read shouldn't return the blocking token, right? It should wait and then try again, until there is a non-blocking token to return? |
|
The idea here was to make as few changes as possible to the current model, which already treats a stream with data from I/O as containing the whole content when it's created. Since the only way to tell the current length of the stream is by reading tokens until EOS, I'm treating its contents that come from I/O as spec artifacts unobservable from outside the token queue. But perhaps this model isn't too easy to understand and it might be best to simply have one special blocking token that, when read, might prepend an immediate token to the queue before it. That's probably far simpler and would better reflect what implementations do. |
|
After some discussion on IRC, I'm closing this PR, since there are alternatives that are far easier to understand. Additionally, it's best to not discuss this change while the new name for streams is being bikeshed in #215. |
2 participants
For a long time, token queues (formerly "streams") had been defined as simple data structures, with no reference to I/O except for a few uses of a "wait until" operation not defined anywhere and a comment on the spec's source saying that the read operation blocks on I/O.
This change introduces the concept of "blocking tokens", which are allowed to block when being read from the token queue. This makes it possible for token queues to be used as a direct spec representation of implementation-internal I/O streams, while all other operations done on token queues use immediate tokens.
This pull request depends on #215, which introduces the name "token queues" and defines the implicit conversions with strings and byte sequences.
Although this change isn't editorial, all implementations are in fact using token queues for I/O, but I haven't checked yet if my changes conflict with the implementations.
(See WHATWG Working Mode: Changes for more details.)
Preview | Diff