Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade all query-mode kernels to use new PUSH/PULL-based agent protocol #34

Closed
13 tasks done
achimnol opened this issue Feb 2, 2017 · 2 comments
Closed
13 tasks done

Upgrade all query-mode kernels to use new PUSH/PULL-based agent protocol #34

achimnol opened this issue Feb 2, 2017 · 2 comments
Labels
enhancement

Comments

@achimnol
Copy link
Member

achimnol commented Feb 2, 2017 •
edited
Loading

  • Python 3 (this is the reference implementation)
  • Python 3 - TensorFlow
  • Python 3 - TensorFlow GPU
  • Python 3 - Caffe
  • Python 2
  • PHP
  • R
  • Julia
  • Javascript
  • Lua
  • Haskell
  • Octave
  • Git shell (command part)

For Python kernels, we also need to update sorna-media package to v0.3.

Minimum features:

  • Binds PUSH/PULL ZeroMQ sockets on TCP port 2000 and 2001
  • Console outputs should be sent as multipart messages in this format: [b'stdout', b'utf8-encoded-text'] and [b'stderr', b'utf8-encoded-text'].
  • No more separate exception handler. Just print language-native traceback to stderr.
  • Send a multipart message [b'finished', b''] when execution is done.

Optional features:

  • Strip tracebacks to include only user-provided codes.
  • Provide interactive input function using input socket (like self.handle_input in Python 3 impl.)
    • If possible, override the most commonly used, standard input function.

Tips:

  • Copy and modify test_run.py in python3 kernel directory to test the main programs before building docker containers for fast iteration of debugging and development.
@achimnol achimnol changed the title Upgrade all kernels to use new PUSH/PULL-based agent protocol Upgrade all query-mode kernels to use new PUSH/PULL-based agent protocol Feb 2, 2017
achimnol added a commit that referenced this issue Feb 3, 2017
@achimnol
refs #34: Update python2 kernel to support continuation.
221ded0
achimnol added a commit that referenced this issue Feb 3, 2017
@achimnol
refs #34: Fix input() in python2 (we should use raw_input)
a1d06e6 
 * Add python2 version of test_run.py
achimnol added a commit that referenced this issue Feb 4, 2017
@achimnol
refs #34: Add continuation support to python3-caffe kernel.
c31e585 
 * Python is NOT yet upgraded to 3.6 due to dependency to libboost-python.
   (It requires some work to manually compile it...)
achimnol added a commit that referenced this issue Feb 4, 2017
@achimnol
refs #34: Add continuation support to python3-tensorflow kernel.
900fc1b 
 * Python is NOT yet upgraded to 3.6 because we need extra work
   to rebuild tensorflow. Coming soon.
achimnol added a commit that referenced this issue Feb 5, 2017
@achimnol
refs #34: Update/add support for continuation to tensorflow kernels
d96e0ec 
 * Rebuilt TensorFlow v0.12.1 for Python 3.6

 * GPU version info: CUDA 8.0, CuDNN 5, compute-capability 6.1
achimnol added a commit that referenced this issue Feb 5, 2017
@achimnol
refs #34: Add continuation support to php7 kernel.
f9dbd71 
 * Users may use our custom $this->input function to get user inputs.
   (In PHP, there is no standard way to read stdin; often uses fgets
   to STDIN pre-defined file variable.)

   Usage example:

     $v = ($this->input)('prompt> ');
     echo $v;

   Here $this is a dummy scope object that exposes Sorna functionality
   to user codes.
achimnol added a commit that referenced this issue Feb 6, 2017
@achimnol
refs #34: Make R3 kernel to be continuation-compatible
a0608c3 
 * It does NOT support continuation (streaming output and interactive
   input) yet.
achimnol added a commit that referenced this issue Feb 9, 2017
@achimnol
refs #34, #5: Update nodejs kernel to support continuation.
8ec8be8 
 * input() is omitted because it does not work with deasync.
achimnol added a commit that referenced this issue Feb 15, 2017
@achimnol
refs #11, #34: WIP - promoting Julia as primarily supported lang.
80d38f4 
 * Julia will support continuation, interactive input, and native
   multimedia outputs.
achimnol added a commit that referenced this issue Feb 20, 2017
@achimnol
refs #11, #34: Prettify stack traces on exceptions for Julia
a34c9ff 
 * ParseError now has line number information as well.
achimnol added a commit that referenced this issue Feb 21, 2017
@achimnol
refs #11, #34: Improve output handling and pre-installed packages.
a7ad6a2 
 * Pre-installed packages now include widely used plotting libraries
   from http://julialang.org/downloads/plotting.html

   - Depending on plotting libraries, you may try:

     display("image/png", plotobj)
     display("image/svg+xml", gcf())

     for native multimedia output in Sorna.

 * Fixed exception trace formatting for more various exception types.
   (only some have "msg" field...)
achimnol added a commit that referenced this issue Feb 21, 2017
@achimnol
refs #22, #34: Update the lua kernel.
c8fe447 
 * Updated to Lua 5.1 to 5.2 and luarocks 2.4.1 to 2.4.2

 * Now supports new PUSH/PULL agent protocol and interactive input.

 * Corrected handling of syntax errors in user-input codes.
   (Previously it simply died!)
achimnol added a commit that referenced this issue Feb 23, 2017
@achimnol
refs #10, #34: Update Octave kernel to support PUSH/PULL protocol.
f33cb8e 
 * Let .deb files be managed by Git LFS.
achimnol added a commit that referenced this issue Feb 23, 2017
@achimnol
refs #10, #34: WIP - figure/plot support in octave kernel
66755d5 
 * Fixed clear command so that it preserves all sorna-related variables.
achimnol added a commit that referenced this issue Feb 24, 2017
@achimnol
refs #18, #34: Support PUSH/PULL protocol in Haskell.
13dc18b 
 * This stdout/stderr-capturing wrapper kernel will become the base
   for compiled language environments such as C/C++, Go, and Rust.
@achimnol
Copy link
Member Author

achimnol commented Feb 24, 2017 •
edited by inureyes
Loading

Next TODO:

  • gather and clean up test cases scattered around each kernel directories.

achimnol added a commit that referenced this issue Feb 25, 2017
@achimnol
refs #32, #34: Update Theano kernel to use PUSH/PULL protocol.
44231a7
achimnol added a commit that referenced this issue Aug 30, 2017
@achimnol
refs #34, #1, #44, #45, #46: Update base runner and all descendant im…
82d680c 
…ages

 * Base runner now offers the auto-completion handler stub.

 * "python3" kernel now uses the base runner and thus support the batch
   mode with the default build command: "python setup.py develop"
   (skipped with a warning when setup.py is not uploaded together)

 * C/C++ kernels now have the make utility. Ooops.
@achimnol
Copy link
Member Author

Current status: WIP - upgrading all kernels to use the common base runner which provides the base implementation of query/batch modes and skeletons for auto-completion.

@achimnol achimnol closed this as completed Sep 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@achimnol