Need tests to check join-piece and join-bpe output.

[philip30]

I investigate this problem and found that recent merge (#477) broke the sentence piece output. This is because of the line number 94 in the xnmt/sent.py

 84   def sent_str(self, custom_output_procs=None, **kwargs) -> str:
 85     """
 86     Return a single string containing the readable version of the sentence.
 87 
 88     Args:
 89       custom_output_procs: if not None, overwrite the sentence's default output processors
 90       **kwargs: should accept arbitrary keyword args
 91 
 92     Returns: readable string
 93     """
 94     out_str = " ".join(self.str_tokens(**kwargs))
 95     pps = self.output_procs
 96     if custom_output_procs is not None:
 97       pps = custom_output_procs
 98     if isinstance(pps, OutputProcessor): pps = [pps]
 99     for pp in pps:
100       out_str = pp.process(out_str)
101       # TODO: change output processor interface accordingly
102     return out_str

This creates a 'space' separated sentence piece output like:
so steve lo pe z , a column of los angeles times , street streets in los angeles , when he hear d a beautiful music .

In the sentence piece, we have to join that by empty string, not space.

How to correctly solve this @msperber, @neubig ?

[philip30]

Hmm, I guess we can just add str.replace(" ", "") at the join-piece processor?

[philip30]

I guess the same problem will occur on BPE processor.

[neubig]

Let me re-open this because we need to:

• Check if BPE is fixed
• Add tests to make sure this doesn't happen again