Add, fix and organize documentation (#11)

- Added README.md
- Added user_guide.md
- Added developer_guide.md
- Added change log
- Added dependencies.md
- Add script for generating dependencies from poetry.lock

Author: tkilias

README.md

@@ -0,0 +1,141 @@
+# Script-Languages-Container-Tool
+
+# Overview
+
+The Script-Languages-Container-Tool (exaslct) is the build tool for the script language container.
+You can build, export and upload script-language container from so-called flavors 
+which are description how to build the script language container. You can find pre-defined flavors 
+in the [script-languages-release](https://github.com/exasol/script-languages-release) repository. 
+There we also described how you could customize these flavors to your needs.
+
+## In a Nutshell
+
+### Prerequisites
+
+#### For installation
+
+In order to install this tool, your system needs to provide 
+the following prerequisites:
+
+* Software
+    * Linux
+    * [bash](https://www.gnu.org/software/bash/)
+    * [coreutils](https://www.gnu.org/software/coreutils/)
+      * sha512sum
+      * sed
+    * [curl](https://curl.se/)
+
+#### For running
+
+In order to use this tool, your system needs to fulfill the following prerequisites:
+
+* Software
+    * Linux
+    * [bash](https://www.gnu.org/software/bash/)
+    * [coreutils](https://www.gnu.org/software/coreutils/)
+      * readlink with -f option (the readlink version of macOS doesn't support -f)
+      * dirname
+    * [Docker](https://docs.docker.com/) >= 17.05 
+      * with support for [multi-stage builds required](https://docs.docker.com/develop/develop-images/multistage-build/)
+      * host volume mounts need to be allowed
+    
+* System Setup  
+    * We recommend at least 50 GB free disk space on the partition 
+      where Docker stores its images, on linux Docker typically stores 
+      the images at /var/lib/docker.
+    * For the partition where the output directory (default: ./.build_output)
+      is located we recommend additionally at least 10 GB free disk space.
+
+Further, prerequisites might be necessary for specific tasks. These are listed under the corresponding section.
+
+### Installation
+
+Download the installation and update script via:
+
+```
+curl -L -o install_or_update_exaslct.sh https://raw.githubusercontent.com/exasol/script-languages-container-tool/main/installer/install_or_update_exaslct.sh
+```
+
+Before you continue with installation, please compute with the following command 
+the sha512 hash sum of the downloaded file and compare it with its 
+[checksum file](installer/checksums/install_or_update_exaslct.sh.sha512sum):
+
+```
+sha512sum install_or_update_exaslct.sh
+```
+
+If the checksums are identical, you can continue with the installation. 
+Per default, the script installs exaslct into the current working directory.
+It creates a script directory `exaslct_scripts` and the symlink `exaslct`
+to the starter script. If you want to change the path to the script directory 
+you can set the environment variable `EXASLCT_INSTALL_DIRECTORY` and 
+if you want to create the symlink somewhere else you can set `EXASLCT_SYM_LINK_PATH`.  
+
+```
+bash install_or_update_exaslct.sh [version|git-commit-id|branch|tag] 
+```
+
+You can use the same script to change the version of your current installation.
+You only need to provide a different version, git-commit-id, branch or tag. 
+
+### Usage
+
+#### How to build an existing flavor?
+
+Create the language container and export it to the local file system
+
+```bash
+./exaslct export --flavor-path=flavors/<flavor-name> --export-path <export-path>
+```
+
+or upload it directly into the BucketFS (currently http only, https follows soon)
+
+```bash
+./exaslct upload --flavor-path=flavors/<flavor-name> --database-host <hostname-or-ip> --bucketfs-port <port> \ 
+                   --bucketfs-username w --bucketfs-password <password>  --bucketfs-name <bucketfs-name> \
+                   --bucket-name <bucket-name> --path-in-bucket <path/in/bucket>
+```
+
+Once it is successfully uploaded, it will print the ALTER SESSION statement
+that can be used to activate the script language container in the database.
+
+#### How to activate a script language container in the database
+
+If you uploaded a container manually, you can generate the language activation statement with
+
+```bash
+./exaslct generate-language-activation --flavor-path=flavors/<flavor-name> --bucketfs-name <bucketfs-name> \
+                                         --bucket-name <bucket-name> --path-in-bucket <path/in/bucket> --container-name <container-name>
+```
+
+where \<container-name> is the name of the uploaded archive without its file extension. To activate the language, execute the generated statement in your database session to activate the container for the current session or system wide.
+
+This command will print a SQL statement to activate the language similar to the following one:
+
+```bash
+ALTER SESSION SET SCRIPT_LANGUAGES='<LANGUAGE_ALIAS>=localzmq+protobuf:///<bucketfs-name>/<bucket-name>/<path-in-bucket>/<container-name>?lang=<language>#buckets/<bucketfs-name>/<bucket-name>/<path-in-bucket>/<container-name>/exaudf/exaudfclient[_py3]';
+```
+
+**Please, refer to the User Guide for more detailed information, how to use exalsct.**
+
+## Features
+
+* Build a script language container as docker images
+* Export a script language container as an archive which can be used for extending Exasol UDFs
+* Upload a script language container as an archive to the Exasol DB's BucketFS
+* Generating the activation command for a script language container
+* Can use Docker registries, such as Docker Hub, as a cache to avoid rebuilding image without changes
+* Can push Docker images to Docker registries
+* Run tests for you container against an Exasol DB (docker-db or external db)
+
+## Table of Contents
+
+### Information for Users
+
+* [User Guide](doc/user_guide/user_guide.md)
+* [Changelog](doc/changes/changelog.md)
+
+## Information for Developers
+
+* [Developer Guide](doc/developer_guide/developer_guide.md)
+* [Dependencies](doc/dependencies.md)

doc/changes/changelog.md

@@ -0,0 +1,3 @@
+# Changes
+
+* [0.1.0](changes_0.1.0.md)

doc/changes/changes_0.1.0.md

@@ -0,0 +1,16 @@
+# Script-Languages-Container-Tool 0.1.0, released XXXX-XX-XX
+
+Code name: Extraction from Script-Languages Repository
+
+## Features / Enhancements
+
+- #3: Add installer scripts which download and install the scripts to run exaslct from a prebuild docker container
+- #1: Add initial version
+
+## Bug Fixes
+
+- #6: Fix docker repository name in construct_docker_runner_image_name.sh
+
+## Documentation
+
+- #9: Add, fix and organize documentation

doc/dependencies.md

@@ -0,0 +1,51 @@
+<!-- @formatter:off -->
+# Dependencies
+ 
+## Compile Dependencies
+
+|Package|Version|
+|---|---|
+|poetry|1.1.6|
+
+## Runtime Dependencies
+
+|Package|Version|
+|---|---|
+|Python|>=3.6|
+|certifi|2020.12.5|
+|chardet|4.0.0|
+|click|7.1.2|
+|decorator|4.4.2|
+|docker|5.0.0|
+|docutils|0.17.1|
+|exasol-integration-test-docker-environment @ git+https://github.com/exasol/[email protected] |
+|gitdb|4.0.7|
+|gitpython|3.1.15|
+|humanfriendly|9.1|
+|idna|2.10|
+|importlib-metadata|4.0.1|
+|importlib-resources|5.1.2|
+|jinja2|2.11.3|
+|jsonpickle|2.0.0|
+|lockfile|0.12.2|
+|luigi|3.0.3|
+|markupsafe|1.1.1|
+|netaddr|0.8.0|
+|networkx|2.5.1|
+|pydot|1.4.2|
+|pyparsing|2.4.7|
+|pyreadline|2.1|
+|python-daemon|2.3.0|
+|python-dateutil|2.8.1|
+|pywin32|227|
+|requests|2.25.1|
+|simplejson|3.17.2|
+|six|1.15.0|
+|smmap|4.0.0|
+|stopwatch.py|1.0.1|
+|tenacity|6.3.1|
+|tornado|6.1|
+|typing-extensions|3.7.4.3|
+|urllib3|1.22|
+|websocket-client|0.58.0|
+|zipp|3.4.1|

docs/README.md renamed to doc/developer_guide/developer_guide.md

@@ -1,13 +1,13 @@
-# EXASLCT (Script Langauge Container Tool)
+# Script-Languages-Container-Tool Developer Guide
+
 EXASLCT is the build tool for the script language container. 
-Its usage is described [here](../README.md). 
-This readme is about the inner working of EXASLCT.
+This document is about the inner working of EXASLCT.
 
 ## About the Script Language Containers
 The Script Language Containers are getting build 
 from several Dockerfiles which depend on each other. 
 These Dockerfiles need to install all necessary 
-dependencies for the [script client](../src), 
+dependencies for the [script client](https://github.com/exasol/script-languages/tree/master/exaudfclient/base), 
 compile the script client and install all necessary dependencies 
 for the flavor and the customizations of the user.
 
@@ -21,7 +21,7 @@ It was actual unclear which of the dependencies were essential
 and which were not. For some flavors it was impossible to run 
 the build on travis, because it exceeded the maximum runtime 
 per job of 50 minutes. 
-The build system and the test runner were bash script 
+The build system and the test runner were both bash scripts 
 which were messy and difficult to maintain. 
 They worked with background jobs to do things in parallel 
 which convoluted the logs which made error analysis difficult.
@@ -55,7 +55,7 @@ Most of these tasks produce some kind of output, for example:
 - docker image
 - a running docker container
 
-Often, other tasks than depend either on the output or 
+Often, other tasks then depend either on the output, or 
 the action of one or more other tasks.
 These dependencies build a direct acyclic graph of tasks, 
 also known as workflow.
@@ -73,7 +73,7 @@ but is suitable for other scenarios, too.
 Luigi describes tasks as subclasses of Luigi.Task 
 which implements the following methods:
 
-```python
+```
 class TaskC(luigi.Task):
 
     def output(self):
@@ -85,20 +85,19 @@ class TaskC(luigi.Task):
         
     def requires(self):
         return [TaskA(),TaskB()]
-
 ```
 
-Here we describe a TaskC which depends of TaskA and TaskB 
-defined in the requires() method. 
+Here we describe a TaskC which depends on TaskA and TaskB 
+defined in the `requires()` method. 
 It does something which is specified in the run() method. 
-Futher, it produces Target() as output. 
-Luigi provides the dependency resolution, scheduling and parallelisation.
+Further, it produces Target() as output. 
+Luigi provides the dependency resolution, scheduling and parallelization.
 
-Besides this static way of describing the dependencies between tasks, 
+Besides, this static way of describing the dependencies between tasks, 
 Luigi also provides so called 
 [dynamic dependencies](https://luigi.readthedocs.io/en/stable/tasks.html#dynamic-dependencies), 
 which allow more flexible patterns in special case. 
-Especially, if the order of execution of dependencies is important or 
+Especially, if the order of execution of dependencies is important, or 
 the dependencies depend on some calculation. The dynamic dependencies 
 allow the implementation of a fork-join pattern.
 
@@ -109,7 +108,7 @@ stops if any other StoppableTask failed in the workflow.
 
 ## Build Steps and their Dependencies
 
-We compose the language container from several different Dockerfiles.
+We compose the language container from several Dockerfiles.
 Each Dockerfile installs dependencies for one specific purpose.
 We also added a separate Dockerfile flavor-customization for user specific changes.
 The user specific changes will be merged on filesystem basis 
@@ -119,7 +118,7 @@ that could prevent the script client from working properly.
 
 The following graph shows the default build steps and their dependencies.
 
-![](docs/image-dependencies.png)
+![](images/image-dependencies.png)
 
 A dependency between build steps can be either a FROM or 
 COPY dependencies. A FROM dependency means that 
@@ -130,7 +129,7 @@ of the source of the arrow.
 
 All steps with the string "build_run" in their name, 
 either run the build for the script client or 
-at least inherit from a images which had build it. 
+at least inherit from an image which had built it. 
 As such these images contain all necessary tools to rebuild 
 the script client for debugging purposes.
 
@@ -178,9 +177,9 @@ class AnalyzeBuildRun(DockerFlavorAnalyzeImageTask):
 ## How does caching work
 
 Exaslct was built with caching in mind, 
-because building a flavor might take very long and 
+because building a flavor might take very long, and 
 many build steps don't change that often.
-Furthermore, a end user most likely only changes the build-step 
+Furthermore, an end user most likely only changes the build-step 
 flavor-customization which is designed to have a minimal impact 
 on all other build steps.
 
@@ -191,11 +190,11 @@ EXASLCT provides three types of caching:
 - file system cache with saved docker images
 - docker registry as a remote cache
 
-All caches can work together, the analyzes phase checks 
-in which cache a images is available. 
+All caches can work together, the analysis phase checks 
+in which cache an images is available. 
 The different type of caches have different precedence 
 which might you override by command line parameters. 
-The precedence is derived by how fast is a image available.
+The precedence is derived by how fast is an image available.
 Docker images managed by the docker daemon are instantaneously available.
 Saved docker images on the filesystem follow next, 
 they need to be loaded by the daemon, 
@@ -211,9 +210,9 @@ Responsible for hashing the build context is the `BuildContextHasher`
 which uses the `FileDirectoryListHasher`.
 
 The `BuildContextHasher` combines the hash values of all directories, 
-files and their executable rights of the build context, 
+files and their executable permissions of the build context, 
 such as the hash values of all images
-the current images depends on and the image changing build arguments 
+the current images depends on, and the image changing build arguments 
 to one hash value for the image.
 
 Other build arguments which only influence the resources