Support get file(notebook) md5

[Wh1isper]

Motivation

Currently, JupyterLab determines whether a file has been manually modified by another client or user by comparing timestamps.

In this way, JupyterLab can prompt the user whether to overwrite the file on disk with the file on the current page, or discard the file on the current page.

This usually performs well on local storage. However, when users use remote storage such as nfs, samba, the modified time of the file is not reliable due to things such as network latency or storage back-end implementations. To solve this problem, jupyterlab/jupyterlab#15037 tries to compare content by comparing them one by one, but this definitely increases the network overhead and client burden.

What is this PR

I propose that this PR to add md5 parameter to the contents API. Any client can get the md5 value of the current file as it needs it and compare it to the previous value to make sure the file has not been modified by another program.

Performance Impact

The server-side calculation of md5 values has essentially no performance impact and it's optional.

Create a 5M file with random bytes.

dd bs=1024 count=5120 </dev/urandom > 5mfile

Calculate its md5.

import hashlib
def caculate(file):
    md5 = hashlib.md5()
    with open(file, "rb") as f:
        c = f.read()
    md5.update(c)
    return md5.hexdigest()
caculate("5mfile")

Here I'm using hyperfine to help me test it.

$hyperfine "python main.py"
Benchmark 1: python main.py
  Time (mean ± σ):      29.4 ms ±   1.4 ms    [User: 24.6 ms, System: 4.7 ms]
  Range (min … max):    27.6 ms …  34.8 ms    91 runs

[blink1073]

Thanks! Just one minor comment

[blink1073]

Thank you!

[blink1073]

I'll cut a minor release early next week.

[bollwyvl]

In 2023, adding the broken md5, for any reason, is probably not a good play. Even when not used for security purposes, the mere presence of it (or any other known-broken algorithm) causes bells and whistles to go off in downstream's inboxes.

If, for some reason, this algorithm must be used, adding the explicit usedforsecurity=False is a way to quite some of the bells. But better would be to use a not-yet-broken hash, such as sha256 or the most modern blake* available in all supported pythons.

[krassowski]

An extra consideration would be whether frontends can compute the hash too to save on a round trip. Hence my suggestion on using murmur as this is what debugger uses. Does murmur trigger downstreams you are speaking of too?

[bollwyvl]

murmur says "non-cryptographic" right on the tin, but would still require a third party library everywhere. Going with something that's in more stdlibs would make it less weird, and possibly more performant, unless very well tuned for the workload, which is likely not possible to predict in this case. Also, for large files, reading (and sending) in responsibly-sized chunks avoids loading the whole file... In this case, twice.

[Wh1isper]

Sorry for the breaking changes to the API. I have created #1367 to fix this.