URL can get too long for browsers

[Meesch]

What went wrong?

When selecting so many filters that the URL exceeds a certain amount of characters, browsers will throw a 414 - Request-URI Too Long Error. It will not do so immediately upon selecting the filter, the site stays functional until the user reloads the page, or navigates to it using the URL.

These are the character limitations for the browsers (source):

Browser     Address bar   document.location
                          or anchor tag
------------------------------------------
Chrome          32779           >64k
Android          8192           >64k
Firefox          >300k          >300k
Safari           >64k           >64k
IE11             2047           5120
Edge 16          2047          10240

This bug occurs for the following corpora:

• Dutch Annual Reports
• Dutch Newspapers
• DIOPTRA-L
• Periodicals
• ECCO
• Rechtspraak

What did you expect to happen?

this exact thing, just making a note of it here.

Screenshot

No response

Where did you find the bug?

• https://ianalyzer.hum.uu.nl
• https://peopleandparliament.hum.uu.nl
• https://peace.sites.uu.nl
• a server hosted elsewhere (i.e. not by the research software lab)
• a local server

Version

No response

Steps to reproduce

Just select a filter that has a lot of options (or options with very long names) and select all of them, e.g. 'Company' in the Dutch Annual Reports, or 'Newspaper Title' in Dutch Newspapers

[lukavdplas]

Possible solutions that I can think of:

• Disable/refuse long filters in a user-friendly way. This doesn't fix anything, but it looks more professional than a browser error?
• Some method of caching filter data so the URL only needs to display the cache ID rather than the full data?

[BeritJanssen]

NB, this is not only affecting filters, but also the option to compare terms: if somebody wanted to compare many terms here (and there are users who do), the URL would also exceed limits. Perhaps we do indeed need to sacrifice the readability of the URL for being able to bookmark wordy queries as well? Wordy queries would have to be sent as JSON perhaps?

[lukavdplas]

They're always sent as JSON to the backend, so this just affects the URL in the frontend.

That said, any form of caching / saving / bookmarking queries could affect the ability to share links with other users / have permanent links / share links between instances.

[BeritJanssen]

What I mean is that the frontend could provide a text field to post JSON queries directly, instead of having to construct queries through the URL or UI. But that's a new feature.

What we now thought would be a good idea: build in a check for the length of the URL, and if it does get too long, turn it into ?queryModel=id_of_query_object instead.

That being said, the visualization parameters would also need to implement such a mechanism, but aren't now saved to the backend.

[lukavdplas]

Using the search history ID is tricky since having search history is optional.

[jgonggrijp]

Another option: always save queries in the backend, even if not bookmarked. Use a hash (SHA256 or something like that) of the normalized JSON representation as the identifier. If the identifier is already in the database, it does not need to be saved, preventing duplicates. Bookmarks refer to the identifier. The URL includes only the identifier, not a full representation of the parameters.

[lukavdplas]

I do think that saving (anonymous) queries is a step back in terms of privacy and maintainability, but yes, that would be the way to handle it.

The listed browser limits are quite long, so I do think that we could probably do a lot by encoding information more efficiently.

To my knowledge, this can happen in the following scenarios:

• The user has entered an insanely long query. Just reject queries that long, if we don't already do that.
• In practice, this issue is normally caused by the multiple choice filter when the user uses "select all" and then deselects one or two options. Let's see if we can adopt Sheean's solution for filtering in MiMa (https://github.com/UUDigitalHumanitieslab/mima/pull/46 I think?) so those filters are instead encoded as negative filters.
• I think Berit pointed out that this might happen if you enter a lot of queries in the "query comparison" option for some visualisations. It's still a lot of text, so this is a bit of an edge case, I think.

[jgonggrijp]

2 KiB is not that long. I don't think you should tell users they can only make queries that fit in that space. Researchers sometimes have complicated questions, and making intricate queries might sometimes be necessary.

As for the hash-based approach, it should not compromise privacy because you do not need to save any association between a query and a user. The query does not even need a timestamp.

[lukavdplas]

Oh I read that wrong, I thought it was 2k characters!