Skip to content

Hyperparameter Optimization Initialization #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
weiji14 merged 10 commits into from
Feb 21, 2019
Merged

Hyperparameter Optimization Initialization #92

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0f00f3f
:heavy_plus_sign: Add Optuna
weiji14 Feb 12, 2019
e629cb4
:beers: Create objective function for hyperparameter optimization
weiji14 Feb 12, 2019
c65128b
:arrow_up: Bump comet-ml from 1.0.42 to 1.0.45 (#101)
dependabot-support Feb 15, 2019
ace6aa1
:zap: Resume optuna study from sqlite db and prune unpromising trials
weiji14 Feb 17, 2019
d569727
:wrench: Recentre hyperparameter search space on good default values
weiji14 Feb 18, 2019
14f0a30
:recycle: Refactor for optional livelossplot, and prepare for more tests
weiji14 Feb 18, 2019
a2866b6
:zap: Enable dual GPU training by removing hardcoding, bad hardcoding!
weiji14 Feb 18, 2019
2cf27d4
:zap: Use cached data for faster evaluation and prevent 2 GPU collision
weiji14 Feb 19, 2019
9c65fb4
:chart_with_upwards_trend: Tune residual scaling and train 400 times
weiji14 Feb 20, 2019
37b90f6
:white_check_mark: Test objective function with new tuned hyperparams
weiji14 Feb 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Pipfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ name = "pypi"
[packages]
black = "==18.9b0"
chainer = "==6.0.0b1"
comet-ml = "==1.0.42"
comet-ml = "==1.0.45"
cupy-cuda92 = "==6.0.0b1"
cython = "==0.29.2"
descartes = "==1.1.0"
Expand All @@ -20,6 +20,7 @@ matplotlib = "==3.0.2"
netcdf4 = "==1.4.1"
numpy = "==1.14.5"
onnx_chainer = "==1.3.0a1"
optuna = "==0.7.0"
packaging = "==19.0"
pandas = "==0.24.1"
pyproj = "==1.9.6"
Expand Down
112 changes: 92 additions & 20 deletions Pipfile.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

185 changes: 91 additions & 94 deletions deepbedmap.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 5 additions & 3 deletions deepbedmap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,18 +185,20 @@ def plot_3d_view(

# %%
def load_trained_model(
filepath: str = "model/weights/srgan_generator_model_weights.npz"
model=None,
model_weights_path: str = "model/weights/srgan_generator_model_weights.npz",
):
"""
Builds the Generator component of the DeepBedMap neural network.
Also loads trained parameter weights into the model from a .npz file.
"""
srgan_train = _load_ipynb_modules("srgan_train.ipynb")

model = srgan_train.GeneratorModel()
if model is None:
model = srgan_train.GeneratorModel()

# Load trained neural network weights into model
chainer.serializers.load_npz(file=filepath, obj=model)
chainer.serializers.load_npz(file=model_weights_path, obj=model)

return model

Expand Down
46 changes: 26 additions & 20 deletions features/environment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
import nbformat
import pandas as pd
import quilt
import requests


def _load_ipynb_modules(ipynb_path: str):
Expand Down Expand Up @@ -87,34 +86,32 @@ def _quick_download_lowres_misc_datasets():
def _download_deepbedmap_model_weights_from_comet():
"""
Download latest neural network model weights from Comet.ML
Uses their REST API endpoint https://www.comet.ml/docs/rest-api/endpoints/
Uses their Python REST API class at https://www.comet.ml/docs/python-sdk/API/
Requires the COMET_REST_API_KEY environment variable to be set in the .env file
"""
authHeader = {"Authorization": base64.b64decode(s=os.environ["COMET_REST_API_KEY"])}

# Get list of DeepBedMap experiments (projectId a7e4f47215b94cd98d6db8a092d78232)
r = requests.get(
url="https://www.comet.ml/api/rest/v1/experiments",
params={"projectId": "a7e4f47215b94cd98d6db8a092d78232"},
headers=authHeader,
comet_api = comet_ml.API(
rest_api_key=base64.b64decode(s=os.environ["COMET_REST_API_KEY"])
)
df = pd.io.json.json_normalize(r.json()["experiments"])

# Get list of DeepBedMap experiments
project = comet_api.get(workspace="weiji14", project="deepbedmap")
df = pd.io.json.json_normalize(data=project.data["experiments"].values())

# Get the key to the latest DeepBedMap experiment on Comet ML
experiment_key = df.loc[df["start_server_timestamp"].idxmax()].experiment_key

# Use key to access url to the experiment's asset which is the hdf5 weight file
r = requests.get(
url="https://www.comet.ml/api/rest/v1/asset/get-asset-list",
params={"experimentKey": experiment_key},
headers=authHeader,
experiment = comet_api.get(
workspace="weiji14", project="deepbedmap", experiment=experiment_key
)
asset_url = r.json()[0]["link"]

# Download the neural network weight file (hdf5 format) to the right place!
r = requests.get(url=asset_url, headers=authHeader)
# Use key to access url to the experiment's asset which is the npz weight file
assets = experiment.asset_list
for asset in experiment.asset_list:
if asset["fileName"].endswith(".npz"): # make sure we pick the .npz file
asset_id = asset["assetId"]
break
# Download the neural network weight file (npz format) to the right place!
open(file="model/weights/srgan_generator_model_weights.npz", mode="wb").write(
r.content
experiment.get_asset(asset_id=asset_id)
)


Expand All @@ -125,6 +122,13 @@ def fixture_data_prep(context):
return context.data_prep


@fixture
def fixture_srgan_train(context):
# set context.srgan_train to have all the module functions
context.srgan_train = _load_ipynb_modules(ipynb_path="srgan_train.ipynb")
return context.srgan_train


@fixture
def fixture_deepbedmap(context):
# Quickly download all the neural network input datasets
Expand All @@ -139,5 +143,7 @@ def fixture_deepbedmap(context):
def before_tag(context, tag):
if tag == "fixture.data_prep":
use_fixture(fixture_func=fixture_data_prep, context=context)
elif tag == "fixture.srgan_train":
use_fixture(fixture_func=fixture_srgan_train, context=context)
elif tag == "fixture.deepbedmap":
use_fixture(fixture_func=fixture_deepbedmap, context=context)
19 changes: 18 additions & 1 deletion features/srgan_train.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,19 @@
# language: en
Feature: Super Resolution Model
@fixture.srgan_train
Feature: Train Super Resolution Model
In order to have a well performing super resolution model
As a machine learning engineer,
We want to craft and teach the model to do well on a test area

Background: Load the prepared data
Given a prepared collection of tiled raster data

Scenario Outline: Train Super Resolution Model with fixed hyperparameters
Given some hyperparameter settings <num_residual_blocks> <residual_scaling> <learning_rate>
And a compiled neural network model
When the model is trained for a while
Then we know how well the model performs on our test area

Examples: Fixed hyperparameters
| num_residual_blocks | residual_scaling | learning_rate |
| 1 | 0.3 | 5e-4 |
Loading