Skip to content

Commit

Permalink
release: 0.5.0 website build
Browse files Browse the repository at this point in the history
  • Loading branch information
@buchanae
Alex Buchanan authored and buchanae committed Jan 16, 2018
1 parent 9f6ee36 commit 005e18c
Show file tree
Hide file tree
Showing 32 changed files with 2,349 additions and 879 deletions.
219 changes: 96 additions & 123 deletions docs/docs/compute/aws-batch/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,18 +92,18 @@
<li>
<ul class="sidebar-nav sidebar-nav-nested">
<li>
<a href="https://ohsu-comp-bio.github.io/funnel/docs/storage/aws-s3/"
<a href="https://ohsu-comp-bio.github.io/funnel/docs/storage/google-storage/"
class="sidebar-nav-item "
>AWS S3</a></li>
>Google Storage</a></li>
</ul>
</li>

<li>
<ul class="sidebar-nav sidebar-nav-nested">
<li>
<a href="https://ohsu-comp-bio.github.io/funnel/docs/storage/google-storage/"
<a href="https://ohsu-comp-bio.github.io/funnel/docs/storage/http/"
class="sidebar-nav-item "
>Google Storage</a></li>
>HTTP(S)</a></li>
</ul>
</li>

Expand All @@ -116,6 +116,15 @@
</ul>
</li>

<li>
<ul class="sidebar-nav sidebar-nav-nested">
<li>
<a href="https://ohsu-comp-bio.github.io/funnel/docs/storage/s3/"
class="sidebar-nav-item "
>S3</a></li>
</ul>
</li>



<li>
Expand Down Expand Up @@ -191,6 +200,15 @@
</ul>
</li>

<li>
<ul class="sidebar-nav sidebar-nav-nested">
<li>
<a href="https://ohsu-comp-bio.github.io/funnel/docs/databases/datastore/"
class="sidebar-nav-item "
>Datastore</a></li>
</ul>
</li>

<li>
<ul class="sidebar-nav sidebar-nav-nested">
<li>
Expand Down Expand Up @@ -220,6 +238,21 @@



<li>

<span class="intermediate">Events</span></li>

<li>
<ul class="sidebar-nav sidebar-nav-nested">
<li>
<a href="https://ohsu-comp-bio.github.io/funnel/docs/events/kafka/"
class="sidebar-nav-item "
>Kafka</a></li>
</ul>
</li>



<li>

<span class="intermediate">Security</span></li>
Expand Down Expand Up @@ -259,155 +292,95 @@



<li>

<span class="intermediate">Events</span></li>

<li>
<ul class="sidebar-nav sidebar-nav-nested">
<li>
<a href="https://ohsu-comp-bio.github.io/funnel/docs/events/kafka/"
class="sidebar-nav-item "
>Kafka</a></li>
</ul>
</li>




</ul>
</div>

<div class="main col span_8_of_12">


<h1 id="amazon-batch">Amazon Batch</h1>
<h1 id="aws-batch">AWS Batch</h1>

<p>This guide covers deploying a Funnel server that leverages <a href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html">DynamoDB</a> for storage
and <a href="http://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html">Batch</a> for task execution. You&rsquo;ll need to set up several resources
using either the Funnel CLI or through the provided Amazon web console.</p>
and <a href="http://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html">AWS Batch</a> for task execution.</p>

<h3 id="create-required-aws-batch-resources">Create Required AWS Batch Resources</h3>
<h2 id="setup">Setup</h2>

<p>For Funnel to execute tasks on Batch, you must define a Compute Environment,
Job Queue and Job Definition. Additionally, you must define an IAM role for your
Batch Job Definition. The role provides the job container with permissions to call
the API actions that are specified in its associated policies on your behalf. For
this configuration, these jobs need access to S3 and DynamoDB.</p>
<p>Get started by creating a compute environment, job queue and job definition using either
the Funnel CLI or the AWS Batch web console. To manage the permissions of instanced
AWS Batch jobs create a new IAM role. For the Funnel configuration outlined
in this document, this role will need to provide read and write access to both S3 and DynamoDB.</p>

<p>Note, we recommend creating the Job Definition with Funnel by running: <code>funnel aws batch create-job-definition</code>.
Funnel expects the JobDefinition to start a <code>worker</code> with a specific configuration. Only
advanced users should consider making any substantial changes to this Job Definition.</p>
<p><em>Note</em>: We recommend creating the Job Definition with Funnel by running: <code>funnel aws batch create-job-definition</code>.
Funnel expects the JobDefinition to start a Funnel worker process with a specific configuration.
Only advanced users should consider making any substantial changes to this Job Definition.</p>

<h3 id="create-resources-with-aws">Create Resources With AWS</h3>
<p>AWS Batch tasks, by default, launch the ECS Optimized AMI which includes
an 8GB volume for the operating system and a 22GB volume for Docker image and metadata
storage. The default Docker configuration allocates up to 10GB of this storage to
each container instance. <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html">Read more about the default AMI</a>. Due to these limitations, we
recommend <a href="http://docs.aws.amazon.com/batch/latest/userguide/create-batch-ami.html">creating a custom AMI</a>. Because AWS Batch has the same requirements for your
AMI as Amazon ECS, use the default Amazon ECS-optimized Amazon Linux AMI as a base and change it
to better suite your tasks.</p>

<p>Amazon provides a quick start guide with more information <a href="http://docs.aws.amazon.com/batch/latest/userguide/Batch_GetStarted.html#first-run-step-2">here</a>.</p>
<h3 id="steps">Steps</h3>

<ul>
<li>Create a Compute Environment - <a href="https://us-west-2.console.aws.amazon.com/batch/home?region=us-west-2#/compute-environments/new">link</a></li>
<li>Create a Job Queue - <a href="https://us-west-2.console.aws.amazon.com/batch/home?region=us-west-2#/queues/new">link</a></li>
<li>Define an EC2ContainerTaskRole with policies for managing access to S3 and DynamoDB - <a href="https://console.aws.amazon.com/iam/home?region=us-west-2#/roles$new?step=permissions&amp;selectedService=EC2ContainerService&amp;selectedUseCase=EC2ContainerTaskRole">link</a></li>
<li>Create a Job Definition - <a href="https://us-west-2.console.aws.amazon.com/batch/home?region=us-west-2#/job-definitions/new">link</a></li>
<li><a href="https://us-west-2.console.aws.amazon.com/batch/home?region=us-west-2#/compute-environments/new">Create a Compute Environment</a></li>
<li>(<em>Optional</em>) <a href="http://docs.aws.amazon.com/batch/latest/userguide/create-batch-ami.html">Create a custom AMI</a></li>
<li><a href="https://us-west-2.console.aws.amazon.com/batch/home?region=us-west-2#/queues/new">Create a Job Queue</a></li>
<li><a href="https://console.aws.amazon.com/iam/home?region=us-west-2#/roles$new?step=permissions&amp;selectedService=EC2ContainerService&amp;selectedUseCase=EC2ContainerTaskRole">Create an EC2ContainerTaskRole with policies for managing access to S3 and DynamoDB</a></li>
<li><a href="https://us-west-2.console.aws.amazon.com/batch/home?region=us-west-2#/job-definitions/new">Create a Job Definition</a></li>
</ul>

<h3 id="create-resources-with-funnel">Create Resources With Funnel</h3>

<p>Funnel provides a utility to create the resources you will need to get up and running.
You will need to specify the AWS region to create these resources in using the <code>--region</code> flag.</p>

<p><em>Note:</em> this command assumes your environment contains your AWS credentials. These
can be configured with the <code>aws configure</code> command.</p>

<pre><code>$ funnel aws batch create-all-resources

Create a compute environment, job queue and job definition in a specified region

Usage:
funnel aws batch create-all-resources [flags]

Flags:
--ComputeEnv.InstanceTypes strings The instances types that may be launched. You can also choose optimal to pick instance types on the fly that match the demand of your job queues. (default [optimal])
--ComputeEnv.MaxVCPUs int The maximum number of EC2 vCPUs that an environment can reach. (default 256)
--ComputeEnv.MinVCPUs int The minimum number of EC2 vCPUs that an environment should maintain. (default 0)
--ComputeEnv.SecurityGroupIds strings The EC2 security groups that are associated with instances launched in the compute environment. If none are specified all security groups will be used.
--ComputeEnv.Subnets strings The VPC subnets into which the compute resources are launched. If none are specified all subnets will be used.
--ComputeEnv.Name string The name of the compute environment. (default &quot;funnel-compute-environment&quot;)
--JobDef.Image string The docker image used to start a container. (default &quot;docker.io/ohsucompbio/funnel:latest&quot;)
--JobDef.JobRoleArn string The Amazon Resource Name (ARN) of the IAM role that the container can assume for AWS permissions. A role will be created if not provided.
--JobDef.MemoryMiB int The hard limit (in MiB) of memory to present to the container. (default 128)
--JobDef.Name string The name of the job definition. (default &quot;funnel-job-def&quot;)
--JobDef.VCPUs int The number of vCPUs reserved for the container. (default 1)
--JobQueue.Name string The name of the job queue. (default &quot;funnel-job-queue&quot;)
--JobQueue.Priority int The priority of the job queue. Priority is determined in descending order. (default 1)
--config string Funnel configuration file
-h, --help help for create-resources
--region string Region in which to create the Batch resources.
</code></pre>
<p>For more information check out AWS Batch&rsquo;s <a href="http://docs.aws.amazon.com/batch/latest/userguide/Batch_GetStarted.html">getting started guide</a>.</p>

<h2 id="configuring-the-funnel-server">Configuring the Funnel Server</h2>

<p>Since the tasks and logs are stored in DynamoDB the Funnel server can be turned
on and off without data loss.</p>
<h3 id="quickstart">Quickstart</h3>

<p>Start the server:</p>
<pre><code>$ funnel aws batch create-all-resources --region us-west-2

<pre><code>funnel server run --config /path/to/config.yaml
</code></pre>

<p>Below is an example of the configuration you would need for the server had you
run <code>funnel aws batch create-all-resources --region us-west-2</code>. Note that the <code>Key</code>
and <code>Secret</code> fields are left blank in the configuration of the componenets. This is because
<p>This command will create a compute environment, job queue, IAM role and job definition.</p>

<h2 id="configuring-the-funnel-server">Configuring the Funnel Server</h2>

<p>Below is an example configuration. Note that the <code>Key</code>
and <code>Secret</code> fields are left blank in the configuration of the components. This is because
Funnel will, by default, try to will try to automatically load credentials from the environment.
Alternatively, you may explicitly set the credentials in the config.</p>

<pre><code class="language-YAML">Server:
Database: &quot;dynamodb&quot;
Databases:
Dynamodb:
TableBasename: &quot;funnel&quot;
AWS:
Region: &quot;us-west-2&quot;
Key: &quot;&quot;
Secret: &quot;&quot;

Backend: &quot;aws-batch&quot;
Backends:
Batch:
JobDefinition: &quot;funnel-job-def&quot;
JobQueue: &quot;funnel-job-queue&quot;
AWS:
Region: &quot;us-west-2&quot;
Key: &quot;&quot;
Secret: &quot;&quot;

Worker:
TaskReader: &quot;dynamodb&quot;
TaskReaders:
Dynamodb:
TableBasename: &quot;funnel&quot;
AWS:
Region: &quot;us-west-2&quot;
Key: &quot;&quot;
Secret: &quot;&quot;
ActiveEventWriters:
- &quot;log&quot;
- &quot;dynamodb&quot;
EventWriters:
Dynamodb:
TableBasename: &quot;funnel&quot;
AWS:
Region: &quot;us-west-2&quot;
Key: &quot;&quot;
Secret: &quot;&quot;
Storage:
AWS:
Key: &quot;&quot;
Secret: &quot;&quot;
<pre><code class="language-YAML">Database: &quot;dynamodb&quot;
Compute: &quot;aws-batch&quot;
EventWriters:
- &quot;log&quot;

Dynamodb:
TableBasename: &quot;funnel&quot;
Region: &quot;us-west-2&quot;
Key: &quot;&quot;
Secret: &quot;&quot;

Batch:
JobDefinition: &quot;funnel-job-def&quot;
JobQueue: &quot;funnel-job-queue&quot;
Region: &quot;us-west-2&quot;
Key: &quot;&quot;
Secret: &quot;&quot;

AmazonS3:
Key: &quot;&quot;
Secret: &quot;&quot;
</code></pre>

<h3 id="start-the-server">Start the server</h3>

<pre><code>funnel server run --config /path/to/config.yaml
</code></pre>

<h3 id="known-issues">Known issues</h3>

<p>Disk size and host volume management extra setup. The <code>Task.Resources.DiskGb</code> field does not have any effect. See <a href="https://github.com/ohsu-comp-bio/funnel/issues/317">issue 317</a>.</p>
<p>The <code>Task.Resources.DiskGb</code> field does not have any effect. See <a href="https://github.com/ohsu-comp-bio/funnel/issues/317">issue 317</a>.</p>

</div>

Expand Down
Loading

0 comments on commit 005e18c

Please sign in to comment.