Skip to content

Commit

Permalink
Merge pull request #1 from arpcard/config
Browse files Browse the repository at this point in the history
Adding configuration file
  • Loading branch information
apetkau authored Oct 8, 2020
2 parents 9baa4de + d7bc944 commit 00dedb0
Show file tree
Hide file tree
Showing 16 changed files with 255 additions and 180 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
/.idea
/config
/cardlive.pid
__pycache__
/data
/card_live_dashboard.egg-info
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
# 0.3.0

* Renamed scripts from `cardlive-dash-X` to `card-live-dash-X`.
* Added configuration file `[cardlive-home]/config/cardlive.yaml` to specify a subpath to run the application.
* Added a gunicorn config file `[cardlive-home]/config/gunicorn.conf.py` to contain all config for the production server.
* Updated `card-live-dash-prod` script to allow start/status/stop commands.

# 0.2.0

* Mapping region **Antarctica** to **N/A** if the date is before *2020-07-20*. This is because before that date CARD:Live defaulted to **Antarctica** (instead of no selected region) and so there were many results showing up as Antarctica when it was likely intended these should be N/A.
Expand Down
3 changes: 2 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@ include LICENSE
include README.md
include MANIFEST.in
include card_live_dashboard/assets/*
include card_live_dashboard/service/data/*
recursive-include card_live_dashboard/test/unit/service/data/ *
recursive-include card_live_dashboard/service/data/ *
recursive-include card_live_dashboard/service/config/ *
79 changes: 67 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# CARD:Live Dashboard
[![pypi](https://badge.fury.io/py/card-live-dashboard.svg)](https://pypi.python.org/pypi/card-live-dashboard/)

This repository contains code for the CARD:Live Dashboard. This is used to summarize and display data from [CARD:Live][] in a dashboard.

Expand All @@ -9,21 +10,22 @@ This repository contains code for the CARD:Live Dashboard. This is used to summa
This application uses [Python Dash][] and so requires Python to be installed (Python 3). It is recommended that you use a Python virtual environment (or conda) to install. To set this up and install the application please run:

```bash
git clone https://github.com/arpcard/card-live-dashboard
cd card-live-dashboard

# Setup virtual environment
virtualenv card-live-venv
source card-live-venv

pip install .
pip install card-live-dashboard
```

## Development

If, instead, you want to install and do development on the code you can instead run (after creating a virtual environment):

```bash
# Clone project
git clone https://github.com/arpcard/card-live-dashboard
cd card-live-dashboard

# Change to main project directory
cd card-live-dashboard

Expand All @@ -36,25 +38,78 @@ This will make the installed application reflect any code changes made within `c

## Create CARD:Live Dashboard home directory

Before running, you will have to create a CARD:Live dashboard home directory. This directory will be used to store the CARD:Live data as well as the NCBI taxnomy database. Please run the below command to create this directory:
Before running, you will have to create a CARD:Live dashboard home directory. This directory will be used to store the CARD:Live data as well as the NCBI taxnomy database and configuration. Please run the below command to create this directory:

```bash
cardlive-dash-init [cardlive-home]
card-live-dash-init [cardlive-home]
```

Once this is created, please copy over the CARD:Live data (JSON files) to `[cardlive-home]/data/card_live`.

## Production

### Running

To run the production server, please run:

```bash
cardlive-dash-prod [cardlive-home]
card-live-dash-prod start [cardlive-home]
```

Where `[cardlive-home]` is the CARD:Live home directory.

This will serve the CARD:Live dashboard on port 8050. Underneath, this runs [gunicorn][]. You can also run the `gunicorn` command directly to adjust the port, number of workers, etc.
This will serve the CARD:Live dashboard on port 8050 by default. Underneath, this runs [gunicorn][].

### Check status

To check the status of the CARD:Live application you can run:

```bash
card-live-dash-prod status [cardlive-home]
```

This will let you know if the application is running.

### Stopping the application

To stop the application you can run:

```
card-live-dash-prod stop [cardlive-home]
```

This will kill the main application and any workers. Note this requires the application to be started in **daemon** mode to work properly.

### Configuration

#### Gunicorn config

The file `[cardlive-home]/config/gunicorn.conf.py` can be used to adjust many configuration options for running the web server. An example of this file can be found [here][gunicorn-prod-conf]. A subset of the options is shown below and a more detailed list can be found in the [gunicorn configuration documentation][gunicorn-conf-doc].

```
bind = '127.0.0.1:8050'
workers = 2
...
```

Please modify this file to adjust configuration.

#### Application config

There also exists a separate YAML configuration file for the application. Right now this is only used to specify a path where the application can run. This will be stored in `[cardlive-home]/config/cardlive.yaml` and will look like:

```yaml
---
## A URL path under which the application should run (e.g., http://localhost/app/).
## Defaults to '/'. Uncomment if you want to run under a new path.
#url_base_pathname: /app/
```

If you wish to run the application under some non-root directory (e.g., under `http://localhost:8050/app`) you can modify the `url_base_pathname` here.

### Running directly using gunicorn

You can also run the `gunicorn` command directly to override configuration settings.

```bash
gunicorn --workers 2 -b 0.0.0.0:8050 "card_live_dashboard.app:flask_app(card_live_home='[cardlive-home]')" --timeout 600 --log-level debug
Expand All @@ -65,19 +120,17 @@ gunicorn --workers 2 -b 0.0.0.0:8050 "card_live_dashboard.app:flask_app(card_liv
To run the development server please run:

```bash
cardlive-dash-dev [cardlive-home]
card-live-dash-dev [cardlive-home]
```

This assumes that you've installed the application with `pip install -e card-live-dashboard`.

**Note**: As per the [Dash documentation][dash-deployment] (which references the Flash documentation) it is not recommended to run the development (built-in) server for a production machine since it doesn't scale well. **Important**: also, since debug mode is turned on this will expose certain information about the underlying server. Please do not use development mode in production.

## Profiling

There is also a server used to profile requests coming to the server (for looking at time of requests). This can be run like:

```bash
cardlive-dash-profiler [cardlive-home]
card-live-dash-profiler [cardlive-home]
```

The same caveats as for the Development server still apply (it also turns on Debug mode and should not be run for a production server).
Expand All @@ -92,6 +145,8 @@ pytest

[dash-deployment]: https://dash.plotly.com/deployment
[gunicorn]: https://docs.gunicorn.org
[gunicorn-prod-conf]: card_live_dashboard/service/config/gunicorn.conf.py
[gunicorn-conf-doc]: https://docs.gunicorn.org/en/latest/configure.html
[CARD:Live]: https://card.mcmaster.ca/live
[Python Dash]: https://plotly.com/dash/
[card-live-overview.png]: images/card-live-overview.png
File renamed without changes.
12 changes: 11 additions & 1 deletion bin/cardlive-dash-init → bin/card-live-dash-init
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ from pathlib import Path
import logging
import ete3.ncbi_taxonomy.ncbiquery

from card_live_dashboard.service.ConfigManager import ConfigManager

script_name = path.basename(path.realpath(sys.argv[0]))
logger = logging.getLogger(__name__)

Expand All @@ -18,6 +20,7 @@ if __name__ == '__main__':
raise Exception('You must specify a valid cardlive_home_dir directory')
else:
cardlive_home_path = Path(args.cardlive_home_dir[0])
cardlive_config_path = cardlive_home_path / 'config'
cardlive_data_path = Path(cardlive_home_path, 'data', 'card_live')
cardlive_db_path = Path(cardlive_home_path, 'db')
cardlive_taxa_file = Path(cardlive_db_path, 'taxa.sqlite')
Expand All @@ -27,6 +30,13 @@ if __name__ == '__main__':

print(f'Initalizing CARD:Live home directory as [{cardlive_home_path}]')

if not cardlive_config_path.exists():
mkdir(cardlive_config_path)

print(f'Writing example configurations to [{cardlive_config_path}]')
config_manager = ConfigManager(cardlive_home_path)
config_manager.write_example_config()

if not cardlive_data_path.exists():
makedirs(cardlive_data_path)
else:
Expand All @@ -45,4 +55,4 @@ if __name__ == '__main__':

print(f'Finished initializing CARD:Live home directory as [{cardlive_home_path}]')
print(f'Please add CARD:Live JSON data to [{cardlive_data_path}]',
f'and start the CARD:Live dashboard by running "cardlive-dash-prod {cardlive_home_path}"')
f'and start the CARD:Live dashboard by running "card-live-dash-prod {cardlive_home_path}"')
59 changes: 59 additions & 0 deletions bin/card-live-dash-prod
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/bin/sh
# Runs CARD:Live dashboard via gunicorn
# This script used to keep track of standard options to gunicorn to run application.

run_method=$1
card_live_home=$2
if [ "${run_method}" = "" ] || [ "${card_live_home}" = "" ]
then
echo "Usage: `basename $0` {start, status, stop} [card_live_home]\n"
echo "Starts, prints status, or stops the CARD:Live Dashboard application"
echo "Requires specifying the [card_live_home] directory created with 'card-live-dash-init'"
exit 0
fi

pidfile="${card_live_home}/cardlive.pid"

if [ "${run_method}" = "start" ]
then
if [ -f ${pidfile} ]
then
echo "CARD:Live already started (pid file [$pidfile] exists). Will not start again."
exit 1
else
echo "Starting CARD:Live using configuration settings from [${card_live_home}/config/gunicorn.conf.py]"
gunicorn -c "${card_live_home}/config/gunicorn.conf.py" "card_live_dashboard.app:flask_app(card_live_home='${card_live_home}')"
fi
elif [ "${run_method}" = "status" ]
then
if [ -f ${pidfile} ]
then
pid=`cat ${pidfile} | tr -d '[:space:]'`
echo "CARD:Live is running. Main process has pid [$pid]"
exit 0
else
echo "CARD:Live is not running. You can start with: `basename $0` start $card_live_home"
exit 0
fi
elif [ "${run_method}" = "stop" ]
then
if [ ! -f ${pidfile} ]
then
echo "CARD:Live not running from [${card_live_home}] (pid file [${pidfile}] does not exist)"
echo "Please try starting first with `basename $0` start ${card_live_home}"
exit 1
fi
pid=`cat ${pidfile} | tr -d '[:space:]'`
if [ "${pid}" = "" ]
then
echo "Cannot stop CARD:Live, pid file ${pidfile} is incorrect. Please kill processes manually."
exit 1
else
# Kill with SIGQUIT to immediatly shutdown <https://docs.gunicorn.org/en/stable/signals.html>
echo "Stopping main process with pid [$pid]"
kill -s 3 ${pid}
fi
else
echo "Usage: `basename $0` {start, stop}\n"
echo "Starts, stops, or restarts the CARD:Live Dashboard application"
fi
File renamed without changes.
15 changes: 0 additions & 15 deletions bin/cardlive-dash-prod

This file was deleted.

2 changes: 1 addition & 1 deletion card_live_dashboard/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '0.3.0.dev0'
__version__ = '0.3.0'
18 changes: 16 additions & 2 deletions card_live_dashboard/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from os import getcwd

from card_live_dashboard.service.CardLiveDataManager import CardLiveDataManager
from card_live_dashboard.service.ConfigManager import ConfigManager
import card_live_dashboard.layouts as layouts
import card_live_dashboard.callbacks as callbacks

Expand All @@ -26,7 +27,12 @@ def build_app(card_live_home: Path = DEFAULT_CARD_LIVE_HOME) -> dash.dash.Dash:
'Please check for this directory or change card_live_home to something '
f'other than [{card_live_home}]'))

app = dash.Dash(name=__name__, external_stylesheets=layouts.external_stylesheets)
config_manager = ConfigManager(card_live_home)
config = config_manager.read_config()

app = dash.Dash(name=__name__,
external_stylesheets=layouts.external_stylesheets,
url_base_pathname=config['url_base_pathname'])

CardLiveDataManager.create_instance(card_live_home)

Expand All @@ -45,4 +51,12 @@ def flask_app(card_live_home: Union[str,Path] = DEFAULT_CARD_LIVE_HOME) -> flask
if isinstance(card_live_home, str):
card_live_home = Path(card_live_home)

return build_app(card_live_home).server
return build_app(card_live_home).server


def read_config(card_live_home: Path):
"""
Reads the config file
:param card_live_home:
:return:
"""
65 changes: 65 additions & 0 deletions card_live_dashboard/service/ConfigManager.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
from pathlib import Path
from typing import Dict, Any
import yaml
import logging
from datetime import datetime
import shutil
from os import path

logger = logging.getLogger(__name__)


class ConfigManager:

def __init__(self, card_live_home: Path):
if card_live_home is None:
raise Exception('Cannot pass None for card_live_home')

self._card_live_home = card_live_home
self._config_file = card_live_home / 'config' / 'cardlive.yaml'

def read_config(self) -> Dict[Any, Any]:
"""
Reads the config file and returns a dictionary of the values.
:return: A dictionary of the values.
"""
if not self._config_file.exists():
raise Exception(f'Config file {self._config_file} does not exist')

with open(self._config_file) as file:
config = yaml.load(file, Loader=yaml.FullLoader)
if config is None:
config = {}

# Some basic syntax checking
if 'url_base_pathname' not in config:
config['url_base_pathname'] = '/'
elif not config['url_base_pathname'].endswith('/'):
config['url_base_pathname'] = config['url_base_pathname'] + '/'

return config

def write_example_config(self):
"""
Writes an example configuration file to the previously set home directory.
:return: None.
"""
backup_timepart = datetime.now().strftime('%Y%m%d-%H%M%S')

if self._config_file.exists():
backup_config = f'{self._config_file}.{backup_timepart}.bak'
logger.warning(f'File [{self._config_file}] exists, backing up to [{backup_config}]')
shutil.copy(self._config_file, backup_config)

shutil.copy(Path(path.dirname(__file__)) / 'config' / 'cardlive.yaml', self._config_file)
logger.info(f'Wrote example configuration to [{self._config_file}]')

gunicorn_conf = self._card_live_home / 'config' / 'gunicorn.conf.py'

if gunicorn_conf.exists():
backup_config = f'{gunicorn_conf}.{backup_timepart}.bak'
logger.warning(f'File [{gunicorn_conf}] exists, backing up to [{backup_config}]')
shutil.copy(gunicorn_conf, backup_config)

shutil.copy(Path(path.dirname(__file__)) / 'config' / 'gunicorn.conf.py', gunicorn_conf)
logger.info(f'Wrote example gunicorn configuration to [{gunicorn_conf}]')
4 changes: 4 additions & 0 deletions card_live_dashboard/service/config/cardlive.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
## A URL path under which the application should run (e.g., http://localhost/app/).
## Defaults to '/'. Uncomment if you want to run under a new path.
#url_base_pathname: /app/
Loading

0 comments on commit 00dedb0

Please sign in to comment.