MLiy Release v.2019.11.08

Release Notes

Features

• We're now on Django 2.2!
• The website is more responsive: users no longer have to wait five minutes for an update.
  • Instance/Cluster information is now updated on-demand rather than asynchronously.
  • Other optimization enhancements have been made.
• Added a better versioning mechanic.
• The DNS link to access the Cluster/Instance will not be functional until the Cluster/Instance reaches a certain state.
• Removed the "Discovery" feature from the application. Instances not spun up by the application will no longer be "discovered".
• The sample Cluster template now detects idleness and automatically terminates the entire cluster after a specified amount of time.
• Model changes have been made to implement new features.

Bug Fixes

• BillingData no longer hogs resources and times out.
• Resolved race conditions between the resource Launch and resource Update that was causing duplicate, orphan Billing Data.

Mliy Platform software updates:

• Django==2.2.6
• mysqlclient==1.3.13
• s3transfer==0.2.1
• New: python-json-logger==0.1.11
• New: mod_wsgi

Sample MLiy Cluster:

• Added ability to install software to all nodes and master

MLiy Release v.2019.04.25

Release Notes

Features

• EMR Cluster support has been added! MLiy can now be used to create Spark clusters and manage them from the MLiy web interface.
• Remote cluster connection capabilities have been added. MLiy instances can connect to MLiy clusters and perform tasks there.
• Push libraries to MLiy clusters. MLiy now provides a sparkmagic library, mliymagic, that uploads library files onto every node on an MLiy cluster from an S3 bucket using SSM.
• Significantly faster MLiy instance startup time! MLiy instances now start up in 10-15 minutes instead of 1-2 hours. Administrators can create one-time snapshots of EBS volumes that have all the software pre-downloaded and installed.
• Improved authorization control for MLiy instances and clusters! Administrators can now specify LDAP groups which control users’ ability to log into a particular MLiy instance or cluster.
• Route 53 DNS option has been added.
• Plugins now have logging.
• Time zone can now be set in the install script.
• Both the MLiy web application and instances installation now accepts an optional proxy script as an input, should they be necessary. These scripts should set environment variables to access the internet and download packages if a proxy is required.

Bug Fixes

• We now have an "Archive" button for instances that went out-of-sync and refuse to leave the UI. This will remove the instance from the UI, delete the Cloudformation stack if it has one, but keep it in the database.
• Instances now properly archive upon termination.
• UI now shows failed instance and cluster attempts that never made it to Cloudformation.
• Identified and fixed an issue where Boto3 "required a region to be set" if the server restarts too quickly. Regions are now explicitly set in every boto3 client instantiation to prevent this.
• On stopped instances, the start button now only appears once.

Mliy Platform software updates:

• Django 1.11.20

Sample MLiy Instance software updates:

• CUDA 9.0
• CuDNN 7.0
• H2O 3.20.0.3
• Jupyter 5.7.4
• Keras 2.1.6
• Python 3.6
• Rstudio 1.1.463
• R 3.5.1
• Tensorflow 1.9.0
• R bsts package installed
• s3fs package added
• Now supports P3 instance types
• DASH support added
• R is now compiled with openBLAS

Sample MLiy Cluster:

• Spark cluster on EMR 5.21.0
• JupyterHub 0.9.4
• Ganglia 3.7.2
• Apache Livy 0.5.0

Known Issues

• On P3 instances, the first time the cuda() function is called, it will take several minutes to execute. Subsequent calls (within the same session) will execute as normal (in approximately 30 seconds).

MLiy Release v.2019.02.08

Release Notes

Features:

• Installation of the webapp automatically adds the MLIY webapp instance's public and private IP to allowed hosts.
• Health check url added at /health.
• Users who are staff can now terminate instances.
• The sample LDAP plugin allows users to skip the Django Admin Login if they're authorized through LDAP.
• The default instance Cloudformation stack names can now be overridden through the Group Config's "Override Instance Name" field.
• The instance launch thread now checks for Complete status in the Cloudformation stack.
• DNS can now optionally be attached to the MLiy web application on Cloudformation stack creation
  • A Hosted Zone must be provided.
  • The ELB must be created through the template.

Bug fixes:

• Name tag for MLiy instances now matches the stack name.
• The IP address check in install.sh now properly checks for an IP.
• Debug is no longer the default logging setting.
• Update thread now properly times out.
• The progress bar update no longer checks for the IP.

Miscellaneous:

• All folders and code now reference mliyweb instead of odapweb.
• UI - Button is now formatted to look more like a button.
• Documentation has been updated.
  • Table of contents added to lengthy pages.
  • The LDAP setup Guide is now separate from the Administrator Guide.

MLiy Web Application software versions:

• Django 1.11.15

  • Security vulnerability patched.

• requests 2.20.0

  • Security vulnerability patched.

• botocore 1.4.46 > 1.12.84

  • Required for setting timeouts using Config.

MLiy Instance software versions:

• H5py is locked to 2.9.0 in py2

  • Set to this version due to importError from missing dependencies.

• pandas locked to 0.19 in py2

  • Set to this version due to importError from missing dependencies.

• pg8000 locked to 1.10.6 in py2

  • Set to this version due to importError from missing dependencies.

• matplot locked to 1.4.3 in py2 and py3

  • Set to this version due to importError from missing dependencies.

MLiy Release v.2018.08.08

Release Notes

List of features

• Updated the version of Django, JQuery and other dependent software
• New deployment script for MLiy application using user provided values in a configuration script
• Support for OpenLdap software in both MLiy web application and image
• Removed nonfunctional placeholder view and settings button from My Dashboard
• Updated documentation explaining donuts in Team Dashboard
• Updated documentation on how to setup a required manager group
• Updated “Current as of” in Team Dashboard to use the same function as My Dashboard does
• Removed views and models associated with ssh keys and Keys in admin view
• Removed Last Refreshed and Billing Data from admin view

List of bug fixes

• Instance state donut in Team Dashboard not functioning correctly
• Default password not set for newly created user.
• Terminate button causes error message to pop up
• iTorch Kernel not installing during bootstrap in MLiy Image
• Tensorboard compatibity notice in MLiy image home page
• MLiy image not sending bootstrap completion signal to correct Cloudformation stack