Handle short sentences

[seominjoon]

No description provided.

[jhyuklee]

Partially resolved using concatenation.

[raman-r-4978]

Hi @jhyuklee May I know what do you mean by concatenation? Is it a text concatenation or vector concatenation?

Please refer #13 for more details

[jhyuklee]

Hi @ramanrajarathinam. We concatenated short sentences into a single paragraph (an input to BERT), then performed the indexing. This resolved the short sentence issues.

[raman-r-4978]

Oh okay.. But may I know why model is not performing as expected when the input text is short?

[jhyuklee]

That's because the model was trained on SQuAD which usually contains a passage longer than a sentence.

[raman-r-4978]

So the only solution is to concatenate different texts? or Is there any other way to solve this?

[raman-r-4978]

Can you also please comment on this issues #9 and #13?

[jhyuklee]

So the only solution is to concatenate different texts? or Is there any other way to solve this?
=> For now, yes but I guess there can be plenty of other solutions that can de-bias the length bias like augmenting short passage QA training pairs.