Prepending or conversion of a sequence possibly not exact enough explained

[wlammen]

In section 3 'Terminology' the idea of prepending a sequence of items to an I/O queue, or converting a list of items into an I/O queue is explained. The described mechanisms work only, if the particular end-of-queue item is not present in the source. Even if we accept for a moment, that an unreliable external source can never accidentally add an end-of-queue item to a sequence or a list, a sequence or a list may still contain this item. This is because an I/O queue itself is a list, which in turn is an ordered sequence per definition. So speaking of a sequence/list of items does not preclude the presence of an end-of-queue.
In

To prepend a sequence of items, insert those items, in the given order, before the first item in the queue.

the possibly present end-of-queue item MUST NOT be inserted into a destination I/O queue, unless it is empty.
Similar for conversion. IMO one should adapt the wording here to account for this exceptional situation.

Wolf Lammen

[andreubotella]

Sure, those definitions for prepend and convert might not be formal enough. #215 was a big change, and some of the minor issues fell through the cracks.

Prepend, though, is meant to be a spec-internal algorithm, since it's explicitly not exported under the Bikeshed/Shepherd data model, which means other specs won't find it as easy to link to it. See also the "implementation considerations" appendix, which lists alternatives to implementing prepend for the uses in the spec, not for arbitrary uses.

[wlammen]

Thanks for providing some context information. For the moment all I do (and can do) is performing some text-immanent analysis (frog's eye). I simply have no clue now (bird's eye) what this bunch of standards will later converge into. Actually, in my opinion, documents like these should be self-containing to the extent possible. Not only, because beginners like me get an easier grasp onto its contents and goals, but also because that way the chance of misinterpreting it is minimized for all.

So even if I had been aware of the context I likely would have opened this issue. The above reasons may be seen as my personal bias, and be dismissed / ignored. As with other issues I already opened in Infra and URL, I simply depict my confusion, sometimes struggle, with what I read. It is a sort of a reader's reaction blog and FYI.

[annevk]

Isn't this all self-contained though? Or are there any callers in the Encoding Standard that do the exceptional thing you describe? (That would be problematic.)

I wouldn't mind adding an assert to the convert operation though. (I still kinda hope someone will eventually refactor things so we can take out prepend entirely.)

[wlammen]

Am I addressed?

There is at least one Reader (me), not a Caller, who tries to learn about the semantics of proposed operations on encoded text. I can envision a situation, where someone uses the standards text as a blue-print for his implementation. Actually the very detailed description of prepending a single character suggests to me that it is important to have a fool-proof implementation.
I do not know anyone else using the Encoding standard for whatever reason.

If certain expressions are fully understandable only, if you know how it is meant (but not explicitly stated) to be used, then I'd say it is not fully self-contained.

[annevk]

I'll rephrase, are there places in the Encoding standard that use the prepend operation in a way that leads to problems?

[wlammen]

Not up to section 3, to my knowledge, I haven't read further by now. So this is not a surprise, because usually a definition prepends its first application.

[wlammen]

Here a real world example outside the standard where this issue matters: switching between an immediate and streaming I/O Queue. Such type of conversions are sometimes needed to adapt to given interfaces or method signatures.

I think I should be more concrete in the convert case. To illustrate one problem, look at

To convert a list, string or byte sequence input into an I/O queue, return an I/O queue containing the items in input, in order, followed by end-of-queue.

If the input list is an I/O queue, it may contain an end-of-queue item, which is copied, and another one is pushed back at the end. So you have a non-trailing end-of-queue in the result, which can be moved further away from the end, in particular by using push operations. As a minimum effect, the output I/O queue's size will not increase any more in that case.

[wlammen]

May I suggest how to get rid of one prepend in section 6 in UTF-8 decode? Do not use read but peek instead.

1. Let buffer be the result of peeking three bytes from ioQueue.
   (My note: buffer contains 0 to 3 bytes afterwards, the ioQueue is left untouched)
2. If buffer matches 0xEF 0xBB 0xBF (U+FFFE BOM), read 3 Bytes from ioQueue and discard the result.
   (My note: this removes a leading BOM sequence, if present. I deliberately use read here, to avoid remove, which is defined for lists. One should look up the comparison procedure for correct wording, the buffer may contain less than 3 bytes, I do not care for now. I don't think that accessing the first three bytes twice is more costly than prepending them again as done in the original process.)
3. Run UTF-8’s decoder with ioQueue and output.
4. Return output.