completion with ein:use-company-backend doesn't work

[fedeinthemix]

I've installed EIN rom melpa and configured it to use 'ein:use-company-backend' for completion, but when I press C-c C-i nothing happens.

Here my configuration.

("EIN system info"
 :emacs-version
 "GNU Emacs 25.1.1 (x86_64-pc-linux-gnu, GTK+ Version 3.22.11)
 of 2017-09-15, modified by Debian"
 :emacs-bzr-version nil
 :window-system x
 :emacs-variant nil
 :os
 (:uname
  "Linux xxx 4.19.0-0.bpo.4-amd64 #1 SMP Debian 4.19.28-2~bpo9+1 (2019-03-27) x86_64 GNU/Linux
"
  :lsb-release
  "No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.9 (stretch)
Release:	9.9
Codename:	stretch
")
 :notebook nil
 :ipython nil
 :image-types
 (svg imagemagick png gif tiff jpeg xpm xbm pbm postscript)
 :image-types-available
 (svg imagemagick png gif tiff jpeg xpm xbm pbm postscript)
 :request
 (:backend curl)
 :ein
 (:version "20190525.1502"
           :source-dir "/home/XXX/.emacs.d/elpa/ein-20190525.1502/")
 :lib
 ((:name "websocket"
         :path "~/.emacs.d/elpa/websocket-20190408.152/websocket.elc"
         :featurep t
         :version-var websocket-version
         :version "1.10")
  (:name "auto-complete"
         :path "~/.emacs.d/elpa/auto-complete-20170125.245/auto-complete.elc"
         :featurep t
         :version-var nil
         :version nil)
  (:name "request"
         :path "~/.emacs.d/elpa/request-20181129.1138/request.elc"
         :featurep t
         :version-var request-version
         :version "0.3.0")
  (:name "deferred"
         :path "~/.emacs.d/elpa/deferred-20170901.1330/deferred.elc"
         :featurep t
         :version-var deferred:version
         :version "0.5.0")
  (:name "polymode"
         :path "~/.emacs.d/elpa/polymode-20190508.1847/polymode.elc"
         :featurep t
         :version-var nil
         :version nil)
  (:name "markdown-mode"
         :path "~/.emacs.d/elpa/markdown-mode-20190305.319/markdown-mode.elc"
         :featurep nil
         :version-var nil
         :version nil)
  (:name "dash"
         :path "~/.emacs.d/elpa/dash-20190424.1804/dash.elc"
         :featurep t
         :version-var nil
         :version nil)
  (:name "s"
         :path "~/.emacs.d/elpa/s-20180406.808/s.elc"
         :featurep t
         :version-var nil
         :version nil)
  (:name "skewer-mode"
         :path "~/.emacs.d/elpa/skewer-mode-20180706.1807/skewer-mode.elc"
         :featurep t
         :version-var nil
         :version nil)))

[millejoh]

Could you tell me the result of C-h v company-backends? I haven't been able to recreate the events that cause it, but sometimes I notice that company-backends gets set to nil at some point in my session, which breaks company completion in general.

[fedeinthemix]

The value of company-backends is

(ein:company-backend company-bbdb company-nxml company-css company-eclim company-semantic company-clang company-xcode company-cmake company-capf company-files
                     (company-dabbrev-code company-gtags company-etags company-keywords)
                     company-oddmuse company-dabbrev)

By the way, I'm using jupyter 4.2.1 coming with Debian 9.

[millejoh]

Sorry, I should have been more specific. Is this the result when you check the variable's value while in a notebook buffer? Is company-mode active in the notebook buffer? What is C-c C-i bound to?

[fedeinthemix]

Yes, this is the value while in a notebook using python's kernel. The company minor mode is active as indicated by the mode line as well as by checking with C-h m.

The function associated with C-c C-i is ein:completer-complete.

[fedeinthemix]

I experimented a little bit with edebug and it looks like the callbacks registered with the kernel completion queries are never called.

[dickmao]

I would suggest a minimum reproducible example with the exact steps, and the exact code that fails to complete. In other words, p[lease use ein:dev-bug-report-template. The company-mode package is such that a completions pop-up should materialize after a one-second idle time without having to explicitly C-c C-i, a keybinding that should probably be obsolesced.

[millejoh]

@fedeinthemix , Since the completion calls are wrapped with deferred constructs what you see in edebug may not be what is actually happening. As @dickmao suggests, following the instructions in ein:dev-bug-report-template may help build a reproducible case as so far I have been unable to reproduce your problems.

[fedeinthemix]

Fair enough:

Problem description

Code completion in a python notebook doesn't work.

Steps to reproduce the problem

1. Start emacs (I have version 25.1.1 from Debian 9).
2. Start EIN with M-x ein:run. Select a directory where I keep
   .ipynb files. Specifically, for testing I select ~/tmp/.
3. Type impo as the beginning of import and press C-c C-i which
   is bound to the function ein:completer-complete.

My expectation is that the command should be completed to import as
it happens with jupyter in the browser (by pressing TAB). Instead nothing happens.

Note that nothing gets completed, that is, the problem is not just with import.

I hope that this clarify the problem.

System info:

("EIN system info"
 :emacs-version
 "GNU Emacs 25.1.1 (x86_64-pc-linux-gnu, GTK+ Version 3.22.11)
 of 2017-09-15, modified by Debian"
 :emacs-bzr-version nil
 :window-system x
 :emacs-variant nil
 :os
 (:uname
  "Linux XXX 4.19.0-0.bpo.4-amd64 #1 SMP Debian 4.19.28-2~bpo9+1 (2019-03-27) x86_64 GNU/Linux
"
  :lsb-release
  "No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.9 (stretch)
Release:	9.9
Codename:	stretch
")
 :notebook nil
 :ipython nil
 :image-types
 (svg imagemagick png gif tiff jpeg xpm xbm pbm postscript)
 :image-types-available
 (svg imagemagick png gif tiff jpeg xpm xbm pbm postscript)
 :request
 (:backend curl)
 :ein
 (:version "20190525.1502"
           :source-dir "/home/XXX/.emacs.d/elpa/ein-20190525.1502/")
 :lib
 ((:name "websocket"
         :path "~/.emacs.d/elpa/websocket-20190408.152/websocket.elc"
         :featurep t
         :version-var websocket-version
         :version "1.10")
  (:name "auto-complete"
         :path "~/.emacs.d/elpa/auto-complete-20170125.245/auto-complete.elc"
         :featurep t
         :version-var nil
         :version nil)
  (:name "request"
         :path "~/.emacs.d/elpa/request-20181129.1138/request.elc"
         :featurep t
         :version-var request-version
         :version "0.3.0")
  (:name "deferred"
         :path "~/.emacs.d/elpa/deferred-20170901.1330/deferred.elc"
         :featurep t
         :version-var deferred:version
         :version "0.5.0")
  (:name "polymode"
         :path "~/.emacs.d/elpa/polymode-20190508.1847/polymode.elc"
         :featurep t
         :version-var nil
         :version nil)
  (:name "markdown-mode"
         :path "~/.emacs.d/elpa/markdown-mode-20190305.319/markdown-mode.elc"
         :featurep nil
         :version-var nil
         :version nil)
  (:name "dash"
         :path "~/.emacs.d/elpa/dash-20190424.1804/dash.elc"
         :featurep t
         :version-var nil
         :version nil)
  (:name "s"
         :path "~/.emacs.d/elpa/s-20180406.808/s.elc"
         :featurep t
         :version-var nil
         :version nil)
  (:name "skewer-mode"
         :path "~/.emacs.d/elpa/skewer-mode-20180706.1807/skewer-mode.elc"
         :featurep t
         :version-var nil
         :version nil)))

[dickmao]

I'm afraid I can't help you. You can try ein:dev-start-debug, reproduce the error, and switch to the buffer starting with "*websocket". If you see something like:

[WS] Sending frame, opcode: text payload: {msg_type":"complete_request"}...
{"code":"impo\n","cursor_pos":4},"parent_header":{},"channel":"shell"}

then something is amiss at the server. If you don't see that, then it's at the client. Either way, there's an unhandled case somewhere.

[fedeinthemix]

Here is the content of the "*websocket..." buffer after pressing C-c C-i:

[WS] Sending frame, opcode: text payload: {"header":{"msg_id":"9a3e438b-9c28-4160-be78-58e7c622cc4b","username":"username","session":"f3782f18-e8fb-436c-bc7d-9ce99d9a5dce","version":"5.0","date":"2019-05-29T08:12:27","msg_type":"complete_request"},"metadata":{},"content":{"line":"impor\n","cursor_pos":5},"parent_header":{},"channel":"shell"}
[WS] Received: \201~^BX{"header": {"msg_id": "aca152f3-ab63-44b7-8591-710dfd968293", "version": "5.0", "session": "55e231a9-c1d9-4f49-ad75-c6d4167f6ad0", "date": "2019-05-29T08:12:27.765035", "msg_type": "status", "username": "XXX"}, "parent_header": {"msg_id": "9a3e438b-9c28-4160-be78-58e7c622cc4b", "version": "5.0", "session": "f3782f18-e8fb-436c-bc7d-9ce99d9a5dce", "date": "2019-05-29T08:12:27", "msg_type": "complete_request", "username": "username"}, "buffers": [], "channel": "iopub", "msg_id": "aca152f3-ab63-44b7-8591-710dfd968293", "msg_type": "status", "content": {"execution_state": "busy"}, "metadata": {}}
[WS] Received: \201~^BX{"header": {"msg_id": "48ab56c3-5c99-4f11-9008-1089e7b23c43", "version": "5.0", "session": "55e231a9-c1d9-4f49-ad75-c6d4167f6ad0", "date": "2019-05-29T08:12:27.768002", "msg_type": "status", "username": "XXX"}, "parent_header": {"msg_id": "9a3e438b-9c28-4160-be78-58e7c622cc4b", "version": "5.0", "session": "f3782f18-e8fb-436c-bc7d-9ce99d9a5dce", "date": "2019-05-29T08:12:27", "msg_type": "complete_request", "username": "username"}, "buffers": [], "channel": "iopub", "msg_id": "48ab56c3-5c99-4f11-9008-1089e7b23c43", "msg_type": "status", "content": {"execution_state": "idle"}, "metadata": {}}
[WS] Received: \211^@
[WS] Sending frame, opcode: pong payload: 
[WS] Received: \211^@

It looks like the server is not returning the list of possible completions. The strange thing is that in the browser notebook completion does work. I've enable debug information with jupyter-notebook --debug, but see no messages related to completion to compare to.

[dickmao]

:notebook nil
:ipython nil

It's hard to know the details of your server as ein:dev-bug-report-template could not run ipython --version on your system.

If you are still interested in using EIN, and are willing to give up resolving this completions issue and your wolfram changes, you might consider enabling ein:polymode as described in the README. This alternative circumvents both these issues but may raise others.

[fedeinthemix]

I've found the problem.

But first: I'm using Debian 9, and jupyter and python 3.5 as distributed by them. Your bug template procedure doesn't find the version of the notebook and of ipython because in Debian 9 the executables for python version 3.5 have the suffix "3".

So, the notebook is:

Name: notebook
Version: 4.2.3
Summary: A web-based notebook environment for interactive computing
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: [email protected]
License: BSD
Location: /usr/lib/python3/dist-packages
Requires: terminado

and ipython

5.1.0

Now to the actual problem: The problem is with the procedure ein:kernel-complete: In my case (< (ein:$kernel-api-version kernel) returns 4 and for this case the request is constructed as

(list
                           ;; :text ""
                           :line line ;
                           :cursor_pos cursor-pos)

However, the kernel expects :code instead of :line, like for version 5 and above. In fact by changing the keyword in this way completion does work.

[dickmao]

Ah, yes, there are many branches in the code that look like:

(if (< version current-version) (do-legacy) (do-current))

The do-legacy branch is largely a museum piece of historical interest, is untested, and almost certainly doesn't work due to bitrot.

I do find it strange that debian9 (afaict the current release) uses 5.1.0. The oldest we test is 5.8.0, and we already consider that "legacy."

[millejoh]

@fedeinthemix - I merged in your changes. Let me know once you update ein if things are working for you. Thanks for the assist.

[fedeinthemix]

Thanks for accepting the patch. I've updated the `ein` package and I confirm that it solves my problem.

[Jimflip]

"you might consider enabling ein:polymode as described in the README. This alternative circumvents both these issues but may raise others."

Am I right in assuming polymode will not communicate to the Jupyter server, rather (for my default python setup via spacemacs layer) it will just utilise whats on the emacs client machine?

[dickmao]

It will just utilise whats on the emacs client machine?

Yes. I also cannot recommend using EIN in spacemacs until we actually test it. I am aware spacemacs is immensely popular.

[alaskamike]

I have been using EIN in Spacemacs for a couple of months, with no issues, and now have it running with Scimax. The only issue is that I have to use the 20190318.1601 version, as the current version doesn't work for me on Windows..

[dickmao]

ein:use-company-backend is no longer available.