Inject session identifier into environment variable. #679

Carreau · 2022-01-25T15:59:29Z

There are many use case where users want to know the current notebook
name/path. This help by adding a session identifier (to not really say
this is the current notebook name), and by default make it the full path
to the notebook document that created the session.

This will of course not work if the notebook get renamed, but we can
tackle this later.

See also jupyter/jupyter_client#656,
jupyter/notebook#6180. It will need to be ported
to jupyter_server as well.

This is the mirror commit of
jupyter/notebook#6279 on main notebook.

codecov-commenter · 2022-01-25T16:03:00Z

Codecov Report

Merging #679 (7abce6b) into main (af1ab9a) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #679      +/-   ##
==========================================
- Coverage   77.81%   77.80%   -0.01%     
==========================================
  Files         110      110              
  Lines       10415    10464      +49     
  Branches     1403     1404       +1     
==========================================
+ Hits         8104     8142      +38     
- Misses       1922     1930       +8     
- Partials      389      392       +3

Impacted Files	Coverage Δ
jupyter_server/services/sessions/sessionmanager.py	`87.50% <ø> (ø)`
jupyter_server/pytest_plugin.py	`84.16% <0.00%> (-1.68%)`	⬇️
jupyter_server/utils.py	`65.90% <0.00%> (-0.57%)`	⬇️
jupyter_server/services/contents/filemanager.py	`70.22% <0.00%> (-0.34%)`	⬇️
jupyter_server/tests/services/kernels/test_cull.py	`100.00% <0.00%> (ø)`
jupyter_server/tests/services/sessions/test_api.py	`96.61% <0.00%> (+0.02%)`	⬆️
jupyter_server/tests/services/kernels/test_api.py	`95.13% <0.00%> (+0.21%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af1ab9a...7abce6b. Read the comment docs.

Wh1isper · 2022-01-26T07:39:52Z

The use of session should solve this problem, as far as I know the name of the session can be used to record the path

Carreau · 2022-01-26T08:19:19Z

The use of session should solve this problem, as far as I know the name of the session can be used to record the path

I'm highly confused by this statement, and what it's trying to convey.
Yes session are the right things, and that's why this PR modify the session manager, and yes I use the session to get the path of the notebook. But even if the session knew the path had change you can't update the environment after the kernel has started....

Wh1isper · 2022-01-26T08:28:04Z

The use of session should solve this problem, as far as I know the name of the session can be used to record the path

I'm highly confused by this statement, and what it's trying to convey. Yes session are the right things, and that's why this PR modify the session manager, and yes I use the session to get the path of the notebook. But even if the session knew the path had change you can't update the environment after the kernel has started....

Oh I see, sorry for not looking at the question carefully, have you considered the effect of root_dir on this path?
The relative path represented by path might be enough?
root_dir has the role of change root, which even allows absolute paths to path, for example
root_dir = '/a'
path = '/b.ipynb'
actually kernel start at /a/b.ipynb, but for front-end and user is b,ipynb only

The current session maintenance for files is probably enough, deleting the session when the file is moved or deleted seems to solve all the problems

Carreau · 2022-01-26T08:37:37Z

Oh I see, sorry for not looking at the question carefully, have you considered the effect of root_dir on this path?

Yes, but I haven't check, my understanding is that the notebook server process CWD is root_dir.
I'm find with absolute or relative path in this env variable, as anyway it could be inconsistent.

Wh1isper · 2022-01-26T08:40:58Z

Oh I see, sorry for not looking at the question carefully, have you considered the effect of root_dir on this path?

Yes, but I haven't check, my understanding is that the notebook server process CWD is root_dir. I'm find with absolute or relative path in this env variable, as anyway it could be inconsistent.

So it may expose the current server's root_dir configuration, and I'm not sure if that's appropriate, especially in jupyterhub, where users have some special security needs and configurations

There are many use case where users want to know the current notebook name/path. This help by adding a session identifier (to not really say this is the current notebook name), and by default make it the full path to the notebook document that created the session. This will of course not work if the notebook get renamed, but we can tackle this later. See also jupyter/jupyter_client#656, jupyter/notebook#6180. It will need to be ported to jupyter_server as well. This is the mirror commit of jupyter/notebook#6279 on main notebook.

Carreau · 2022-01-26T08:48:38Z

Sure, removed the resolve().

maartenbreddels · 2022-01-26T09:12:14Z

Love this PR, I think this is really useful!
In voila we've followed the CGI spec, see:

Passing request URI to kernel env voila-dashboards/voila#414 (following https://tools.ietf.org/html/rfc3875)
http_header_envs option to include http headers as an env var voila-dashboards/voila#835 ( https://datatracker.ietf.org/doc/html/rfc3875#section-4.1.18 )

Maybe it is useful to have this standard in the Jupyter ecosystem, what do you think?

echarles · 2022-01-26T09:12:33Z

What about multiple notebooks connected to a single Kernel?

e.g. in jupyterlab, you can connect multiple notebook, console (console don't have a path) to a single kernel.

What is the intended behavior? Should the JPY_SESSION_NAME be a list that can be updated?

Carreau · 2022-01-26T10:43:55Z

It can't be a list, as it is an env variable, it could be a ; delimited string, sure.
In general I think we should be able to piggyback on the execution request, and for each execution request, say this execution comes from file/frontend XXX. Using for example user_expressions. But as explain in other linked thread, it might take a while to go there.

I'm fine with SCRIPT_NAME as well, or any other name that would get this in, I does not particularly matter to me, but we've been through the past few month/years through various proposal, and roughly:

there might not be a file/script, so having NOTEBOOK, FILE, SCRIPT... in the name was frowned upon.
JPY_ as a prefix seem to be common in Jupyter.

Now my idea is to already get that somehow, somewhere, as one source for the current file name or whatever, and at least in IPykernel, use this (and/or other sources), to set __file__ for each execution requests.

echarles · 2022-01-26T10:54:51Z

It can't be a list, as it is an env variable, it could be a ; delimited string, sure.

yes, this is what I was meaning. However, thinking more, the maintenance of that string would be a pain when notebooks/console/files would connect/disconnect.

In general I think we should be able to piggyback on the execution request, and for each execution request, say this execution comes from file/frontend XXX. Using for example user_expressions. But as explain in other linked thread, it might take a while to go there.

That sounds like a good practice to identity the origin of the kernel message request. IIRC There is already a (unused) username field, not sure if there is a origin.

.... and at least in IPykernel, use this (and/or other sources), to set file for each execution requests.

this would work if the content if collocated to the kernel (same machine, same env), but would not be applicable for remote kernels or remote content (when the content is not colocated with the kernel).

Carreau · 2022-01-26T12:31:13Z

this would work if the content if collocated to the kernel (same machine, same env), but would not be applicable for remote kernels or remote content (when the content is not colocated with the kernel).

Of course, unless you know how to map from one to the other.

echarles · 2022-01-27T16:30:19Z

We have discussed this during the weekly server dev meeting:

@vidartf has suggested that we could consider the environment variable (as proposed in this PR) more as a useful hint that can be used (or not) by the kernel and the notebook user.
@blink1073 has suggested that a second step (enhancement outside of this PR) could be adding the source/origin of the kernel message as metadata to that message. This would not request a JEP. Like the debugger case, later on if we would want to move those details to a standard field, we could have a JEP process.

echarles · 2022-02-14T10:53:24Z

I have relaunched the failing CI (Jupyter Server Tests on windows and Mac OS).

It is still failing with the same error. e.g on Windows jupyter_server/tests/services/sessions/test_api.py::test_restart_kernel[jp_server_config0]

Error: Process completed with exit code 1.

Is this expected due to some flakyness?

blink1073 · 2022-02-14T22:40:13Z

Yes, we've had some flakiness overall in CI recently.

echarles · 2022-02-15T06:39:24Z

Based on the previous discussion, I think we agree to to move forward with this partial solution, and agree that it is already useful a a hint. @Carreau Maybe you could add in the doc a paragraph around this, open a subsequent issue to keep track of the remaining work. Based on that, we could merge. Thx again.

vidartf · 2022-03-17T16:06:50Z

This will get merged tomorrow unless there are any more comments or suggestions!

blink1073 · 2022-03-17T16:16:17Z

Kicking CI since a lot has changed since the last commit.

blink1073 · 2022-03-17T20:00:36Z

Whoa, this is new, on the Windows builds.

Fatal Python error: _Py_HashRandomization_Init: failed to get random numbers to initialize Python
Python runtime state: preinitialized

I just kicked a previously passing build to see if it happens there.

blink1073 · 2022-03-17T20:29:44Z

The other builds passed, kicked this one again.

blink1073 · 2022-03-17T21:30:28Z

Ah, I think what it is is that we are overriding the default env that is passed down here, which is preventing the random number generator from working on Windows.

vidartf · 2022-05-27T13:24:44Z

Merged as failing Windows tests seem to be unrelated.

vidartf · 2022-05-27T13:28:01Z

Actually, they probably were relevant as @blink1073 indicated (why only Windows though?). I conflated it with #672

) * Inject session identifier into environment variable. There are many use case where users want to know the current notebook name/path. This help by adding a session identifier (to not really say this is the current notebook name), and by default make it the full path to the notebook document that created the session. This will of course not work if the notebook get renamed, but we can tackle this later. See also jupyter/jupyter_client#656, jupyter/notebook#6180. It will need to be ported to jupyter_server as well. This is the mirror commit of jupyter/notebook#6279 on main notebook. * Make start_kernel env extend os.environ (#859) * Make start_kernel env extend os.environ * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Matthias Bussonnier <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Zsailer added the enhancement label Jan 25, 2022

Carreau force-pushed the jpy-session-name branch from c4056a5 to 7abce6b Compare January 26, 2022 08:46

Zsailer mentioned this pull request Jan 27, 2022

Meeting Notes 2022 jupyter-server/team-compass#15

Closed

blink1073 closed this Mar 17, 2022

blink1073 reopened this Mar 17, 2022

echarles mentioned this pull request Mar 20, 2022

Inject session identifier into environment variable. jupyter/notebook#6279

Closed

Merge branch 'main' into jpy-session-name

2f41c6d

vidartf merged commit c8cbb2c into jupyter-server:main May 27, 2022

vidartf mentioned this pull request May 27, 2022

Make start_kernel env extend os.environ #859

Merged

This was referenced Jul 13, 2022

[Feature Request] __file__ cell magic jupyterlab/jupyterlab#8012

Closed

Backport Inject session identifier into environment variable #920

Merged

vidartf mentioned this pull request Jul 14, 2022

[Backport] Inject session identifier into environment variable #924

Closed

kevin-bates mentioned this pull request Sep 14, 2022

Make it easier to pass custom env variables to kernel #981

Merged

jtpio mentioned this pull request Aug 1, 2023

Uses jupyter/jupyter_client#656 to set notebook name in env. jupyter/notebook#6180

Closed

astrojuanlu mentioned this pull request Jun 4, 2024

Packaged kedro pipeline does not work well on Databricks kedro-org/kedro#1807

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inject session identifier into environment variable. #679

Inject session identifier into environment variable. #679

Carreau commented Jan 25, 2022

codecov-commenter commented Jan 25, 2022 •

edited

Loading

Wh1isper commented Jan 26, 2022

Carreau commented Jan 26, 2022

Wh1isper commented Jan 26, 2022 •

edited

Loading

Carreau commented Jan 26, 2022

Wh1isper commented Jan 26, 2022

Carreau commented Jan 26, 2022

maartenbreddels commented Jan 26, 2022

echarles commented Jan 26, 2022

Carreau commented Jan 26, 2022

echarles commented Jan 26, 2022

Carreau commented Jan 26, 2022

echarles commented Jan 27, 2022

echarles commented Feb 14, 2022

blink1073 commented Feb 14, 2022

echarles commented Feb 15, 2022

vidartf commented Mar 17, 2022

blink1073 commented Mar 17, 2022

blink1073 commented Mar 17, 2022

blink1073 commented Mar 17, 2022

blink1073 commented Mar 17, 2022

vidartf commented May 27, 2022

vidartf commented May 27, 2022

Inject session identifier into environment variable. #679

Inject session identifier into environment variable. #679

Conversation

Carreau commented Jan 25, 2022

codecov-commenter commented Jan 25, 2022 • edited Loading

Codecov Report

Wh1isper commented Jan 26, 2022

Carreau commented Jan 26, 2022

Wh1isper commented Jan 26, 2022 • edited Loading

Carreau commented Jan 26, 2022

Wh1isper commented Jan 26, 2022

Carreau commented Jan 26, 2022

maartenbreddels commented Jan 26, 2022

echarles commented Jan 26, 2022

Carreau commented Jan 26, 2022

echarles commented Jan 26, 2022

Carreau commented Jan 26, 2022

echarles commented Jan 27, 2022

echarles commented Feb 14, 2022

blink1073 commented Feb 14, 2022

echarles commented Feb 15, 2022

vidartf commented Mar 17, 2022

blink1073 commented Mar 17, 2022

blink1073 commented Mar 17, 2022

blink1073 commented Mar 17, 2022

blink1073 commented Mar 17, 2022

vidartf commented May 27, 2022

vidartf commented May 27, 2022

codecov-commenter commented Jan 25, 2022 •

edited

Loading

Wh1isper commented Jan 26, 2022 •

edited

Loading