preserve encoding over temporary storage

[heejaechang]

little bit of refactoring over temporary stroage and text factory service to preserve encoding round trip

this fixes #380 and #353

[pharring]

Let's add another test (or enhance this one) to test non-null encodings too.

foreach(var encoding in new Encoding[] {
    null,
    Encoding.ASCII,
    Encoding.UTF8,
    Encoding.Unicode,
    Encoding.BigEndianUnicode})
{
    var text = Text.SourceText.From("Hello, World!", encoding);
    // etc.

[tmeschter]

Yes, please do that.

[heejaechang]

done

[pharring]

LGTM

[tmeschter]

LGTM

[CyrusNajmabadi]

Can you comment this function? I find it confusing that an Encoding is passed in, but then the code calls "detectEncodingFromBytOrderMarks" and then calls AsRoslynText, not using the encoding passed in, but instead reader.CurrentEncoding or Encoding.UTF8.

It seems like someone can be explicit about the encoding, but still have that overridden by the system.

[heejaechang]

default encoding is something it use if it can't figure out Encoding from stream. that is why it is called default encoding. I will add more comments.

But I agree it is confusing. I took me sometime to figure out what that actually mean.

[heejaechang]

New API I added. CreateText(TextReader reader ... ) is the API people should use if they want to set encoding explicitly. this API doesnt have ambiguity where given Encoding and encoding embedded in the stream is different.

the first API is intended to be used in a situation where you just open a file (FileStream) and want to create a SourceText. the default encoding given is basically saying, if you can't figure out Encoding from the stream I gave you, assume it is the default encoding.