Exception in SafariView on non-ASCII URLs ("unsupported scheme")

[sentry-io]

Sentry handles this especially poorly -- every individual instance of this exception seems to result in a new "issue" in Sentry, disconnected from all the previous instances.

Sentry Issue: ZULIP-MOBILE-10G

(In the URL, sequences *** are redactions.)

RCTFatalException: Exception 'The specified URL has an unsupported scheme. Only HTTP and HTTPS URLs are supported.' was throw...: Application threw exception RCTFatalException: Exception 'The specified URL has an unsupported scheme. Only HTTP and HTTPS URLs are supported.' was thrown while invoking show on target SafariViewManager with params (

        {
        url = "https://*********/user_uploads/8091/ZtZg***/***-r\U00e9\U00e9criture-V3-***.docx";
    },
    2873
)
callstack: (
	0   CoreFoundation                      0x00000001938611d0 <redacted> + 148
	1   libobjc.A.dylib                     0x000000019229855c objc_exception_throw + 56
	2   CoreFoundation                      0x0000000193861100 <redacted> + 0
	3   SafariServices                      0x00000001a1dc692c <redacted> + 488
	4   ZulipMobile                         0x000000010019b5e0 _ZN5folly6detail15str_to_integralIxEENS_8ExpectedIT_NS_14ConversionCodeEEEPNS_5RangeIPKcEE + 356548
	5   CoreFoundation                      0x0000000193867150 <redacted> + 144
	6   CoreFoundation                      0x0000000193759eac <redacted> + 284
	7   CoreFoundation                      0x000000019375e95c <redacted> + 60
	8   ZulipMobile                         0x00000001000de568 ZulipMobile + 337256
	9   ZulipMobile                         0x0000000100125318 RCTFBQuickPerformanceLoggerConfigureHooks + 117088
	10  ZulipMobile                         0x0000000100125074 RCTFBQuickPerformanceLoggerConfigureHooks + 116412
	11  libdispatch.dylib                   0x00000001926ea1fc <redacted> + 24
	12  libdispatch.dylib                   0x00000001926ea1bc <redacted> + 16
	13  libdispatch.dylib                   0x00000001926eed68 _dispatch_main_queue_callback_4CF + 1000
	14  CoreFoundation                      0x000000019380e810 <redacted> + 12
	15  CoreFoundation                      0x000000019380c3fc <redacted> + 1660
	16  CoreFoundation                      0x000000019373a2b8 CFRunLoopRunSpecific + 444
	17  GraphicsServices                    0x00000001951ee198 GSEventRunModal + 180
	18  UIKit                               0x00000001997817fc <redacted> + 684
	19  UIKit                               0x000000019977c534 UIApplicationMain + 208
	20  ZulipMobile                         0x0000000100091674 ZulipMobile + 22132
	21  libdyld.dylib                       0x000000019271d5b8 <redacted> + 4
): Exception 'The specified URL has an unsupported scheme. Only HTTP and HTTPS...
  ?, in <redacted>
  ?, in RCTFBQuickPerformanceLoggerConfigureHooks
  ?, in RCTFBQuickPerformanceLoggerConfigureHooks
  ?, in <redacted>
...
(11 additional frame(s) were not displayed)

[gnprice]

(I've just edited this issue description to redact bits of the URL that could reflect the affected user's private information, and also gone into the edit history to delete the bad version.)

[gnprice]

One observation on the URL in the exception: it has the sequence \U00e9.

Scanning through similar exceptions in Sentry, they all have similar sequences: \U00fc, \U00e1, \U041d\U0435.

I think this is some form of encoding issue -- those look like sequences intended to represent the characters

U+00E9 LATIN SMALL LETTER E WITH ACUTE -> é
U+00FC LATIN SMALL LETTER U WITH DIAERESIS -> ü
U+041D CYRILLIC CAPITAL LETTER EN -> Н
U+0435 CYRILLIC SMALL LETTER IE -> е
-> "Не", Russian for "No"

etc.

One question would be, what values are we actually passing in this url property? Do they have the actual characters "é" etc., or the six-character sequences "\U00e9" etc.?

If the latter, that seems pretty bogus -- and the next question would be, how are we ending up with those?

[jainkuniya]

and the next question would be, how are we ending up with those?

I tried to upload file files whose name includes unicode: temp £ som©˙˚˚˙∫√®∂´ffdsafdsa.md on uploading server is replacing unicode to "" (empty string).

@timabbott Does older server encodes it?

[gnprice]

and the next question would be, how are we ending up with those?

I tried to upload file files whose name includes unicode: temp £ som©˙˚˚˙∫√®∂´ffdsafdsa.md on uploading server is replacing unicode to "" (empty string).

Hmm interesting!

Can you say more concretely exactly what you observed? I.e., where did you look and what did you see there?

[jainkuniya]

When I uploaded file with name as temp £ som©˙˚˚˙∫√®∂´ffdsafdsa.md, all the unicodes in the name are removed from it in the url /user_uploads/2/ac/9R623mOtC-XMimqtLZLmzeYa/temp-som-ffdsafdsa.md

…

On Thu, May 23, 2019, 8:49 AM Greg Price ***@***.***> wrote: and the next question would be, how are we ending up with those? I tried to upload file files whose name includes unicode: temp £ som©˙˚˚˙∫√®∂´ffdsafdsa.md on uploading server is replacing unicode to "" (empty string). Hmm interesting! Can you say more concretely exactly what you observed? I.e., where did you look and what did you see there? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#3315?email_source=notifications&email_token=AENHKSJ32A3MY2WLWK7KZNDPWYEKRA5CNFSM4GSSXHL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWA6L6Y#issuecomment-495052283>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AENHKSIXS3QVY6V2DHSEEH3PWYEKRANCNFSM4GSSXHLQ> .

[gnprice]

When I uploaded file with name as temp £ som©˙˚˚˙∫√®∂´ffdsafdsa.md, all the unicodes in the name are removed from it in the url /user_uploads/2/ac/9R623mOtC-XMimqtLZLmzeYa/temp-som-ffdsafdsa.md

Great -- and where did you observe that URL? It's OK to be somewhat painfully detailed and explicit, especially in an issue thread -- it helps make sure we're exactly on the same page.

Then, please try some more things to debug the issue. I think you can get farther than this. 🙂 Here's a few ideas:

• You suggested a hypothesis that it has something to do with the server version. You can get data on this hypothesis yourself: you have Sentry access, so you can see the exception reports we're getting, and they include the URLs. Most servers are on the Internet -- try fetching their server_settings, and see what versions you find.
  • E.g. you might report back something like: I looked at all the instances of this exception from the past N days, which related to M different servers. For each server, I did .... K of them didn't respond, but of the rest the oldest version was V1 and the newest was V2.
  • Whatever version you see it happening on there, try checking out that version in your dev server and reproducing the issue on that same version.
• What server did you do your test on? If it was on zulipchat.com try chat.zulip.org, and vice versa -- lots of file-upload behavior differs between them, and perhaps this does too. Also say which. :-)
• Maybe the real URL for the file actually has the six-character sequences "\U00fc" etc. Play around and see if you can get a file-upload URL like that, and see what happens.

In general, the main work that'll be needed to fix this issue is to figure out what's going on, and probably most of that will be to find a way to reproduce it. Please use your creativity to think of questions it'd be helpful to answer (like your thought about older server versions above); and then also to think of experiments you can try that might answer those questions.

Then report here on the issue thread copious details about the experiments you try, and what happens. 🙂

[jainkuniya]

Yup! did more investigation on this, here are few observations: (btw got one work around 😅)

Observations:

• Initially we were thinking that this is only related to unicodes files names which are uploaded, so I went and tried to reproduce it by: uploading files with wried unicode chars in its name to both CZO and zulipchat realms, server is handling it nicely. Removing them and doing necessary sanity in the uploaded file file name (url).
• I found the server version on which we are getting this errors (in Sentry) : they are 1.0.8, 2.0.1, 2.0.3
• One thing I suspect, is there anything to do with windows file system. Server is running on windows machine or user is uploading file from windows OS, because windows uses \ in file paths. Still not tested.
• One thing I observed is even browser is replacing \ to / when I am pasting the url which I got from sentry. That's why at that time I opened that PR (safariView: Replace \ with /. #3316).
  This might not be browser who is replacing this. As soon as I paste url, server redirects to login page, where I can see that \ are replaced to / in ?next params https://**/login/?next=/user_uploads/2/ee/1avDGo3f4eP9OCufxyZA1-2R/U......................................
• Then I thought that this are the files which are generated, basically exported from some tools. So this tools are responsible for having unicodes in the file names. Issue like this
• But we are also getting same errors for non file uploads url as well, like https://en.wiktionary.org/wiki/þ this report
• Even I uploaded files who have either \ or / in their names, but generated url doesn't have / or \.
• What we are getting urls in Sentry are not the real one that user is trying to access. Let's say if we are trying to access https://en.wiktionary.org/wiki/þ then sentry will report with url
  
  means sentry is showing us \u... but in real it is þ.
• After all this I started debugging react-native-safari-view, through which we are opening safari view controller. I found that it is calling https://github.com/naoufal/react-native-safari-view/blob/v2.1.0/SafariViewManager.m#L49 to open safari view. My primary suspect was that when url is going from js to native via bridge, it is getting converted to some malformed. But I confirmed it by logging it in native part as soon as it is received from js. It was as expected.
• BTW there is one closed issue in the repo: Crash with unicode url #41 They are suggesting to use encodeURIComponent. I tried with it but one think we need to do is to only encode path, not full url because then / in https:// are also encoded and things doesn't work.
• So the problem is: when ios/safari is creating URL object out of https://en.wiktionary.org/wiki/þ it is throwing nil, and resulting in error.
  
  According to safari it is not a valid URL, that is what I can see.
  But if we are pasting same url which have unicode, safari browser is opening it.
• I even tried sending https://en.wiktionary.org/wiki/þ to safari view with this code https://github.com/naoufal/react-native-safari-view/blob/v2.1.0/SafariViewManager.m#L49 so that we can make sure that react-native-safari-view has nothing to do with it. And yup! it has nothing to do with it. Safari view throws error.
• So most probably there is issue in safari view controller, it throws error while parsing url. And its code is black box to us.
• Then I tested how Slack is handling this: I tried to open https://en.wiktionary.org/wiki/þ in iOS safari view controller and it is working fine. And observed that they have passed https://en.wiktionary.org/wiki/%C3%BE, not the one with unicode to the safari view controller.
• So I went ahead and found that: browser also does something similar, it has URL object
  
• Then I thought we can use similar logic in RN as well 😅, but RN still doesn't support URL Creating an instance of URL like: new URL('http://facebook.com') throws an exception. facebook/react-native#16434.
• Then I tried several other options like: https://medialize.github.io/URI.js/, https://github.com/medialize/URI.js nothing worked.
• Then I got this: Creating an instance of URL like: new URL('http://facebook.com') throws an exception. facebook/react-native#16434 (comment)
  and it is working 🎉 . Next I am submitting PR with this approach and let's discuss over there about it.

[gnprice]

Great - thanks for the debugging, and those detailed notes!

Some useful background on the transformation from https://en.wiktionary.org/wiki/þ to https://en.wiktionary.org/wiki/%C3%BE is here:
https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier

In short it looks like we have a "URL" that is officially not a URL but an IRI - because strictly a URL (or a URI, which is basically the same thing but officially a slightly more general category) is all ASCII. And we need to encode that IRI as a URI in the prescribed way.

We'll definitely want to write the code in a way such that it's clear that that's what's going on.

That term IRI may be a helpful keyword for searching for implementations, too. For example, here's one:
https://www.npmjs.com/package/iri
Found by googling for iri encode js.