behind-the-scene broken at gmail

[uBlock-user]

Prerequisites

• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uMatrix
• I tried to reproduce the issue when...
  • uMatrix is the only extension
  • uMatrix with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uMatrix
• I checked the documentation to understand that the issue I report is not a normal behavior

Description

On gmail, behind-the-scene requests are blocked unconditionally and th refresh button on the for behind-the-scene panel doesn't work. BTS Refresh button broke with v1.3.0 and BTS network requests are being blocked unconditionally since 1.3.8.

A specific URL where the issue occurs

https://mail.google.com/mail/u/0/?tab=wm#inbox

Steps to Reproduce

1. Disable ALL extensions and install uMatrix from https://github.com/gorhill/uMatrix/releases/tag/1.3.15b2

2. Once installed, visit gmail and login with your credentials

3. Open the logger and refresh the page to populate, once loaded, page will appear broken, because behind-the-scene network requests are being blocked unconditionally. even though it's default installation where matrix is set to off for behind-the-scene.

4. Click on a behind-the-scene network request to open up behind-the-scene popup panel and now click on the refresh button, the tab doesn't refresh.

Supporting evidence

Errors collected via Chromium on v1.3.0 regarding the refresh button issue -

Broken refresh button at BTS panel and BTS network requests being blocked unconditionally -

https://i.gyazo.com/014e222d6d50f97d57c3d66869f0b0c8.mp4

Your environment

• uMatrix version: 1.13.15.2
• Browser Name and version: Chromium 71
• Operating System and version: Win 10 x64

[uBlock-user]

Because of this, gmail remains broken for me as in I can't use funcionality such as Add to Drive and download attachments/images don't load etc.. until I disable this extension from Extensions page. Seems to be happening with Firefox. Let me know if you're unable to reproduce.

[gorhill]

th refresh button on the for behind-the-scene panel doesn't work

Of course, "behind-the-scene" means there is no specific tab associated with the request, hence there no tab for uMatrix to refresh. This is not new behavior, this has always been the case.

Open the logger and refresh the page to populate, once loaded, page will appear broken

I only have a "dont-care" Gmail account for testing purpose, and I logged in into it without issue and I could not spot anything broken. I don't have emails with attachment in it or even a Google Drive account, so it would help to tell me what exactly is broken with screenshots, etc. The simpler the better, otherwise this will be a cant-reproduce issue.

[uBlock-user]

Of course, "behind-the-scene" means there is no specific tab associated with the request, hence there no tab for uMatrix to refresh. This is not new behavior, this has always been the case.

So why does v1.3.0/4/6 spew errors like this -

what exactly is broken with screenshots

BTS requests are being blocked unconditionally on my end, because of this, gmail remains broken for me as in I can't use funcionality such as Add to Drive and download attachments/images don't load etc..

I only have a "dont-care" Gmail account for testing purpose, and I logged in into it without issue and I could not spot anything broken. I don't have emails with attachment in it or even a Google Drive account

I feared this would happen, I posted all the information I could gather here, don't know what else I could give you.

[uBlock-user]

I could e-mail you a dummy pdf on your gmail account if you want.

[gorhill]

So why does v1.3.0/4/6 spew errors like this

I don't get that with latest uMatrix. I can't say it didn't happen with 1.3.0, but that is an old version, you shouldn't be reporting issue with older versions unless they also occur in current stable.

[uBlock-user]

The reason I'm reporting from that version is because the latest dev build doesn't throw such StackTrace errors even though when refresh button is clicked from BTS panel and it doesn't anything, but only v 1.3.0/4/6 do.

[uBlock-user]

console errors if they help any -

[gorhill]

the latest dev build doesn't throw such StackTrace

Because the issue is no longer present? That was my point: why report issues from past versions which no longer exist in the latest stable release. This was probably fixed in 1.3.8.

[uBlock-user]

OK, but what about those BTS requests being blocked unconditionally ?

FYI, you may have to enable Collect Errors from chrome://extensions/?id=eckgcipdkhcfghnmincccnhpdmnbefki if you're not seeing any in Extension page.

[gorhill]

This is the ruleset I have when trying to reproduce:

https-strict: behind-the-scene false
matrix-off: about-scheme true
matrix-off: chrome-extension-scheme true
matrix-off: chrome-scheme true
matrix-off: moz-extension-scheme true
matrix-off: opera-scheme true
matrix-off: vivaldi-scheme true
matrix-off: wyciwyg-scheme true
noscript-spoof: * true
referrer-spoof: * true
referrer-spoof: behind-the-scene false
* * * block
* * css allow
* * frame block
* * image allow
* 1st-party * allow
* 1st-party frame allow
google.com gstatic.com * allow

What about yours?

[uBlock-user]

same as yours on default rule set

[gorhill]

Don't you need to set google.com gstatic.com * allow? I needed this -- to un-break logging in.

[uBlock-user]

Yes I did add it as temp rule and it works on default installation, so now I go back to my regular config and the images and thumbnails do load but add to Drive is still stuck, very weird case, spend 4 hours on this already and still can't figure whats going on.

Thing is my regular config rules worked perfectly until today, today it stopped.

[uBlock-user]

Done. Sent my regular ruleset alongwith.

[gorhill]

I could open your attachments, and also download locally, but the later required me to add google.com googleusercontent.com frame allow from the popup panel.

[uBlock-user]

With mail.google.com gstatic.com script allow the images and thumbnails remain broken,

With mail.google.com gstatic.com * allow , the images and thumbnails load fine.

Very weird, I have never needed these two rules for images and thumbnails to load until today.

[uBlock-user]

Another thing mail.google.com gstatic.com * allow doesn't fix the Save To Drive functionality which is still broken.

[gorhill]

I could "Save to Drive" fine. However the "Save to Drive" popup was blank, I fixed by adding the following rules:

mail.google.com content.googleapis.com * allow
mail.google.com googleusercontent.com * allow

The first one might not be needed, it's the one I tried first.

Edit: I didn't need to reload by the way, and I confirm only the 2nd seem necessary.

[uBlock-user]

I don't get any popups, I click on the button and it goes into an infinite loop of Saving to Drive...it's fine I will figure that one out.

Since I don't wanna whitelist the entire gstatic.com on mail.google.com, so I narrowed it down to

mail.google.com ssl.gstatic.com xhr allow
mail.google.com www.gstatic.com xhr allow

These two rules bring back the broken images and thumbnails, but weird thing is there's no XHR being made from those rules or atleast a number doesn't appear on those cells

[gorhill]

As per your per-site switch count, you are blocking workers -- I didn't see this in your ruleset screenshot above. Gmail does use workers.

[uBlock-user]

Okay so XHRs re coming from those ssl.gstatic.com and www.gstatic.com but those are behind-the-scene requests, and since BTS is already whitelisted, so I shouldn't need to add

mail.google.com ssl.gstatic.com xhr allow
mail.google.com www.gstatic.com xhr allow

Right ?

[uBlock-user]

No workers appear blocked on my end

[uBlock-user]

@gorhill If I don't add

mail.google.com ssl.gstatic.com xhr allow
mail.google.com www.gstatic.com xhr allow

BTS XHRs get blocked, why is that, why does BTS is getting affected by those rules ?

[gorhill]

No workers appear blocked on my end

Well you opened this issue a few months ago: #22.

[gorhill]

BTS XHRs get blocked, why is that, why does BTS is getting affected by those rules ?

I can't remember where but we talked about how "behind-the-scene" changed in uMatrix: not all entries marked as "behind-the-scene" are real actual behind-the-scene, because now uMatrix makes use of initiator if present to filter request, and when the initiator is present, the logger still report the request as behind the scene -- there is still no specific tab involved, but there is a specific origin, as opposed to real behind-the-scene where there is no tab and no origin. I just miss the time to overhaul the logger to now account for this. This is also true for uBO.

[uBlock-user]

I disabled all th 4 switches and still no change, so it's not the same, BTS XHRs are getting blocked regardless of workers switch enabled or not.

[uBlock-user]

I can't remember where but we talked about how "behind-the-scene" changed in uMatrix: not all entries marked as "behind-the-scene" are real actual behind-the-scene, because now uMatrix makes use of initiator if present to filter request, and when the initiator is present, the logger still report the request as behind the scene -- there is still no specific tab involved, but there is a specific origin, as opposed to real behind-the-scene where there is no tab and no origin.

I think you nailed it there. That only makes logical sense.

[gorhill]

You could use uBO's logger just to find out the context in which these XHRs are made (uMatrix does not have this information in the logger yet). For the target XHRs, click the cell to create filters in uBO's logger, the context (aka initiator) will be shown.

[uBlock-user]

Context is mail.google.com, so it's not BTS as it's shown by both the extensions.

[gorhill]

With uBO it's merely cosmetic -- one can create proper rules/filters -- just need to have a different icon to convey that a request is not associated with any specific tab but yet its origin is known (with real behind-the-scene the origin aka context is unknown).

[uBlock-user]

I meant do you intend to fix uBO's logger regarding this issue ?

[gorhill]

gorhill/uBlock#3654 (comment)

[gorhill]

A column has been added to the logger: the scope in which network requests are made. Thus, the behind-the-scene scope is now shown just like any other scope: behind-the-scene. The block/allow column is clickable and must be use to set rules specific strictly to the entry.

[uBlock-user]

Still doesn't show number of requests on XHR matrix cell, so the user still won't know which cell to whitelist to stop breakage, Also why are those requests still shown as BTS ?

Fixed in 1.3.15.8

[gorhill]

Can you provide a screenshot of what you are seeing? If a request is shown as behind-the-scene, than it is really behind-the-scene.

[uBlock-user]

Still doesn't show number of requests on XHR matrix cell, so the user still won't know which cell to whitelist to stop breakage,

This

The situation at gmail is still the same, those XHR whitelisted cells from ssl.gstatic.com and www.gstatic.com still don't show number of requests inside them.

If the requests were really BTS, then how come whitelisting those two XHR cells fixes the breakage ?

Fixed in 1.3.15.8

[gorhill]

The fix was in the logger -- you should be able to see in the logger these requests -- they will no longer be reported as "behind-the-scene" unless they really are. uMatrix is not reporting them in the popup panel of that tab probably because there is no tab information provided for these specific requests. In such case, the logger is where ultimately those problems are resolved, hence the new UI in the logger.

This is what can happen:

• tab specified, scope known (all reported in the matrix): use the popup panel or logger
• tab not specified, scope known (probably your case): use the logger
• tab not specified, scope unknown (these are literal behind-the-scene): use the popup panel of behind-the-scene or logger

[uBlock-user]

uMatrix is not reporting them in the popup panel of that tab probably because there is no tab information provided for these specific requests.

Can anything be done about this ?

This is how it looks like via the logger -

[gorhill]

Can anything be done about this ?

No, as said, the request is made outside of any tab context, uMatrix has no idea from which tab the request is originating. As shown in your pic, this is solved via the logger, you can set rules for that one request. That new UI in the logger also solves gorhill/uMatrix#951.

[uBlock-user]

Thanks to this new UI, asset updates on Firefox are now initiated by moz-extension-scheme and on Chromium it's chrome-extension-scheme, those who disable uMatrix on both these schemes won't be affected, but thought I would post for those like me who go with whitelist and blocking the rest, due to this change, the BTS popup panel is misleading and is required to be removed FYI.

[gorhill]

I will need to provide access to the per-scope switches from the new per-entry rule editor UI.

[uBlock-user]

images are not loading again, are being seen by BTS in the logger ? Can you confirm if it's really from BTS ?

[uBlock-user]

Update -

On Chromium -

On Firefox -

Notice the XHR cell count for gstatic.com doesn't appear on Chromium's dev build.

[gwarser]

In Chromium service worker loads all gstatic XHRs using tab-less mail.google.com context.

In Firefox they are not tab-less.

---

Maybe some indicator should be added to the popup (for example to scope selector) when tab-less request occur in current context?

[uBlock-user]

Tabless requests number count should also be shown in the main popup UI to combat this. The cell count number not appearing is really troubling.