Changing Decoder trait to be more composable. (#938)

[Narsil]

• Changing Decoder trait to be more composable.

[mishig25]

Got 2 questions:

1. What is the difference between Changing Decoder trait to be more composable. #938 & Changing Decoder trait to be more composable. (#938) #1008 and why was Changing Decoder trait to be more composable. #938 reverted in the first place?
2. Do you need to create tokenizers::decoders::Sequence?

[Narsil]

Got 2 questions:

1. What is the difference between [Changing `Decoder` trait to be more composable. #938](https://github.com/huggingface/tokenizers/pull/938) & [Changing `Decoder` trait to be more composable. (#938) #1008](https://github.com/huggingface/tokenizers/pull/1008) and why was [Changing `Decoder` trait to be more composable. #938](https://github.com/huggingface/tokenizers/pull/938) reverted in the first place?

This one is not breaking

2. Do you need to create `tokenizers::decoders::Sequence`?

Yes.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[mishig25]

will this line cause any problem for the SequenceDecoder?
Specifically, one unlikekly edge-case I'm thinking is: one of the sequence decoders is another sequence decoder, and Ok(results.join("")) will cause an informaition loss (vec of str now just becomes a str)?

as a solution, we can implement fn decode for Sequence Decoder?

[Narsil]

It's OK I think.

The parent Sequence would call the child Sequence.decode_chain NOT the decode function, so we're good, no ?

I could add a test to make sure.

[mishig25]

ooh I see, so it is no problem in that case

[mishig25]

lgtm!