Merge pull request #64 from onepanelio/release-v0.18.0

Release v0.18.0

README.md

@@ -14,31 +14,31 @@ See the following references for more information on how to build these template
 ### Workflow Templates
 Workflow Templates consist of YAML definitions and Docker images that define a DAG in Onepanel.
 
-- [Albumentations data pre-processing](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/albumentations-preprocessing) - This Workflow is included in [TFOD](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/tf-object-detection-training) and [MaskRCNN](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/maskrcnn-training) training Workflows and allows you to apply different augmentations to your data before training.
-- [Auto CVAT](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/auto-cvat) - Allows you to automate your annotation workflow by creating CVAT instances and pre-populating them with data to be annotated.
-- [Hyperparameter tuning](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/hyperparameter-tuning) - Hyperparameter tuning Workflow using [NNI](https://github.com/microsoft/nni). Included in Onepanel deployment.
-- [MaskRCNN training](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/maskrcnn-training) - Workflow for semantic segmentation model training fully integrated with CVAT and included in Onepanel deployment.
-- [PyTorch training](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/pytorch-mnist-training) - Simple MNIST training example using PyTorch.
-- [TensorFlow training](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/tensorflow-mnist-training) - Simple MNIST training example using TensorFlow.
-- [TensorFlow Object Detection training](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/tf-object-detection-training) - Workflow for object detection model training fully integrated with CVAT and included in Onepanel deployment. 
+- [Albumentations data pre-processing](https://github.com/onepanelio/templates/blob/master/workflows/albumentations-preprocessing) - This Workflow is included in [TFOD](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/tf-object-detection-training) and [MaskRCNN](https://github.com/onepanelio/templates/tree/release-v0.18.0/workflows/maskrcnn-training) training Workflows and allows you to apply different augmentations to your data before training.
+- [Auto CVAT](https://github.com/onepanelio/templates/blob/master/workflows/auto-cvat) - Allows you to automate your annotation workflow by creating CVAT instances and pre-populating them with data to be annotated.
+- [Hyperparameter tuning](https://github.com/onepanelio/templates/blob/master/workflows/hyperparameter-tuning) - Hyperparameter tuning Workflow using [NNI](https://github.com/microsoft/nni). Included in Onepanel deployment.
+- [MaskRCNN training](https://github.com/onepanelio/templates/blob/master/workflows/maskrcnn-training) - Workflow for semantic segmentation model training fully integrated with CVAT and included in Onepanel deployment.
+- [PyTorch training](https://github.com/onepanelio/templates/blob/master/workflows/pytorch-mnist-training) - Simple MNIST training example using PyTorch.
+- [TensorFlow training](https://github.com/onepanelio/templates/blob/master/workflows/tensorflow-mnist-training) - Simple MNIST training example using TensorFlow.
+- [TensorFlow Object Detection training](https://github.com/onepanelio/templates/blob/master/workflows/tf-object-detection-training) - Workflow for object detection model training fully integrated with CVAT and included in Onepanel deployment. 
 
 ### Workspace Templates
 Workspace Templates consist of YAML definitions and Docker images that define stateful instances like JupyterLab, CVAT and any other IDE.
 
-- [CVAT](https://github.com/onepanelio/templates/tree/release-v0.18.0/workspaces/cvat) - An interactive video and image annotation tool for computer vision.
-- [JupyterLab](https://github.com/onepanelio/templates/tree/release-v0.18.0/workspaces/jupyterlab) - An extensible environment for interactive and reproducible computing, based on Jupyter Notebook.
-- [Eclipse Theia](https://github.com/onepanelio/templates/tree/release-v0.18.0/workspaces/theia) - An extensible platform to develop multi-language cloud and desktop IDEs with state-of-the-art web technologies.
-- [Ubuntu VNC](https://github.com/onepanelio/templates/tree/release-v0.18.0/workspaces/vnc) (alpha) - A full Ubuntu instance accessible in your web browser.
-- [Visual Studio Code](https://github.com/onepanelio/templates/tree/release-v0.18.0/workspaces/vscode) - A lightweight but powerful source code editor which has support for just about everything. 
+- [CVAT](https://github.com/onepanelio/templates/blob/master/workspaces/cvat) - An interactive video and image annotation tool for computer vision.
+- [JupyterLab](https://github.com/onepanelio/templates/blob/master/workspaces/jupyterlab) - An extensible environment for interactive and reproducible computing, based on Jupyter Notebook.
+- [Eclipse Theia](https://github.com/onepanelio/templates/blob/master/workspaces/theia) - An extensible platform to develop multi-language cloud and desktop IDEs with state-of-the-art web technologies.
+- [Ubuntu VNC](https://github.com/onepanelio/templates/blob/master/workspaces/vnc) (alpha) - A full Ubuntu instance accessible in your web browser.
+- [Visual Studio Code](https://github.com/onepanelio/templates/blob/master/workspaces/vscode) - A lightweight but powerful source code editor which has support for just about everything. 
 
 ### Sidecars
 Sidecars are components that extend your Workspace or Workflow Tasks.
 
-- [FileSyncer](https://github.com/onepanelio/templates/tree/release-v0.18.0/sidecars/filesyncer) - Provides the APIs to sync files between Workspaces and default object storage.
-- [NNI Web UI](https://github.com/onepanelio/templates/tree/release-v0.18.0/sidecars/nni-web-ui) - Provides a proxy to NNI Web UI so that you can see the experiments in your hyperparameter tuning Workflows.
+- [FileSyncer](https://github.com/onepanelio/templates/blob/master/sidecars/filesyncer) - Provides the APIs to sync files between Workspaces and default object storage.
+- [NNI Web UI](https://github.com/onepanelio/templates/blob/master/sidecars/nni-web-ui) - Provides a proxy to NNI Web UI so that you can see the experiments in your hyperparameter tuning Workflows.
 
 ### Tasks
 Tasks are the individual tasks in your Workflow (nodes in your DAG).
 
-- [Metrics writer](https://github.com/onepanelio/templates/tree/release-v0.18.0/tasks/metrics-writer) - Task you can include to any Workflow Template to persist final metrics.
-- [Slack notifications](https://github.com/onepanelio/templates/tree/release-v0.18.0/tasks/slack-notify) - Task you can add to any Workflow or Workspace Template to send notifications to Slack.
+- [Metrics writer](https://github.com/onepanelio/templates/blob/master/tasks/metrics-writer) - Task you can include to any Workflow Template to persist final metrics.
+- [Slack notifications](https://github.com/onepanelio/templates/blob/master/tasks/slack-notify) - Task you can add to any Workflow or Workspace Template to send notifications to Slack.

sidecars/filesyncer/Dockerfile.s3 → sidecars/filesyncer/Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.13.10 AS builder
+FROM golang:1.15 AS builder
 
 WORKDIR /go/src
 COPY . .

sidecars/filesyncer/Makefile

@@ -0,0 +1,11 @@
+ifndef version
+	$(error version is undefined)
+endif
+
+docker-build:
+	docker build -t onepanel/filesyncer:$(version) .
+
+docker-push:
+	docker push onepanel/filesyncer:$(version)
+
+docker: docker-build docker-push

sidecars/filesyncer/README.md

@@ -1,19 +1,32 @@
-# File-Syncer
+# FileSyncer
 
-Sidecar container for syncing files from multiple object storage providers. 
+Sidecar container for syncing files from the storage providers supported by Onepanel.
 
-Currently supports:
+## How to run locally
 
-- S3
-- GCS
-- Azure blob storage
+`filesyncer` is meant to run in Onepanel as a sidecar container. For development, you can run it as follows.
 
+### Server mode
 
-## How to run locally
+In this mode, you can send HTTP requests to `filesyncer` to sync down from an object storage prefix to a local path or sync up from a local path to an object storage prefix.
+
+To run `filesyncer` in server mode:
+
+```bash
+go run main.go server -server-prefix /sys/filesyncer
+```
 
-`file-syncer` is meant to run in Onepanel as a sidecar container. 
+To test the endpoints:
 
-For development, you can run as follows:
+```bash
+# Sync down
+curl localhost:8888/sys/filesyncer/api/sync -X POST --data '{"action": "download", "path":"/tmp", "prefix":"artifacts/my-namespace/"}'
+
+# Sync up
+curl localhost:8888/sys/filesyncer/api/sync -X POST --data '{"action": "upload", "path":"/tmp", "prefix":"artifacts/my-namespace/"}'
+```
+
+### Automatic syncing mode
 
 ```bash
 FS_PATH=./files FS_PREFIX=data go run main.go upload
@@ -28,27 +41,18 @@ FS_PATH=./files FS_PREFIX=data FS_PROVIDER=s3 AWS_ACCESS_KEY_ID=<value> AWS_SECR
 You can also indicate `provider`, `path` and `prefix` as flags:
 
 ```bash
-AWS_ACCESS_KEY_ID=<value> AWS_SECRET_ACCESS_KEY=<value> go run main.go upload --provider s3 --path ./files --prefix data 
+AWS_ACCESS_KEY_ID=<value> AWS_SECRET_ACCESS_KEY=<value> go run main.go upload --path ./files --prefix data
 ```
 
 You can also run via `docker` as follows:
 
 ```bash
-# s3
-docker run -it -v $(pwd)/files:/mnt -e AWS_ACCESS_KEY_ID=<value> -e AWS_SECRET_ACCESS_KEY=<value> file-syncer:latest [upload|download] \
-    --provider s3 --path /mnt --bucket <bucket> --prefix <prefix>
-
-# gcs
-docker run -it -v $(pwd)/files:/mnt -e GOOGLE_APPLICATION_CREDENTIALS=<key.json-path> file-syncer:latest [upload|download] \
-    --provider gcs --path /mnt --bucket <bucket> --prefix <prefix>
-
-# az
-docker run -it -v $(pwd)/files:/mnt -e AZURE_STORAGE_ACCOUNT=<account> -e AZURE_STORAGE_KEY=<key> file-syncer:latest [upload|download] \
-    --provider az --path /mnt --bucket <container> --prefix <prefix>
+docker run -it -v $(pwd)/files:/mnt -e AWS_ACCESS_KEY_ID=<value> -e AWS_SECRET_ACCESS_KEY=<value> filesyncer:0.17.0 [upload|download] \
+    --path /mnt --bucket <bucket> --prefix <prefix>
 ```
 
 Note that also indicate `FS_PROVIDER`, `FS_PATH` and `FS_PREFIX` as environment variables in the Docker commands, or you can mount the secret mocks like so:
 
 ```bash
-docker run -v $(pwd)/files:/mnt -v $(pwd)/onepanel:/etc/onepanel -e FS_PATH=/mnt -e FS_PREFIX=data filesyncer:gcs upload
+docker run -v $(pwd)/files:/mnt -v $(pwd)/onepanel:/etc/onepanel -e FS_PATH=/mnt -e FS_PREFIX=data filesyncer:0.17.0 upload
 ```

sidecars/filesyncer/go.mod

@@ -1,8 +1,9 @@
 module github.com/onepanelio/templates/sidecars/filesyncer
 
-go 1.14
+go 1.15
 
 require (
 	github.com/robfig/cron/v3 v3.0.0
 	sigs.k8s.io/yaml v1.2.0
+	github.com/justinas/alice v1.2.0
 )

sidecars/filesyncer/go.sum

@@ -1,5 +1,7 @@
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/justinas/alice v1.2.0 h1:+MHSA/vccVCF4Uq37S42jwlkvI2Xzl7zTPCN5BnZNVo=
+github.com/justinas/alice v1.2.0/go.mod h1:fN5HRH/reO/zrUflLfTN43t3vXvKzvZIENsNEe7i7qA=
 github.com/onepanelio/templates v0.2.0 h1:1+3QTlKRMQxaxDlYm2NqVDeyYcfwBfSIjFOZ9iJgWqY=
 github.com/robfig/cron/v3 v3.0.0 h1:kQ6Cb7aHOHTSzNVNEhmp8EcWKLb4CbiMW9h9VyIhO4E=
 github.com/robfig/cron/v3 v3.0.0/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=