Skip to content

Commit 6ab3276

Browse files
jerichtjericht
authored
chore: Created sample Typescript app that deploys everything in RFDK (#21)
1 parent a332f82 commit 6ab3276

33 files changed

+1115
-1021
lines changed

‎examples/kitchen-sink/.gitignore ‎examples/deadline/All-In-AWS-Infrastructure-Basic/ts/.gitignore

+1-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,6 @@
33
cdk.out
44
cdk.context.json
55

6-
# Ignore package tarball
7-
kitchen-sink-*.tgz
8-
96
# Used by nyc
107
.nyc_output
118
coverage
@@ -20,4 +17,4 @@ dist
2017
!license-header.js
2118

2219
# The staged files for Deadline
23-
stage
20+
stage

‎examples/kitchen-sink/.npmignore ‎examples/deadline/All-In-AWS-Infrastructure-Basic/ts/.npmignore

File renamed without changes.

‎examples/deadline/All-In-AWS-Infrastructure-Basic/ts/README.md

+186
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# RFDK Sample Application - Deadline
2+
3+
This is a sample RFDK application that deploys the basic infrastructure for a Deadline render farm. This application is structured in tiers, each representing a CloudFormation stack. The tiers are:
4+
5+
1. **Network Tier** - Foundational networking required by all components.
6+
1. **Security Tier** - Contains resources that keep the render farm secure (e.g. certificates).
7+
1. **Storage Tier** - Persistent storage (e.g. database, file system).
8+
1. **Service Tier** - Business logic (e.g. central server, licensing).
9+
1. **Compute Tier** - Compute power to render jobs.
10+
11+
Each deployment tier is deployed as a separate CloudFormation Stack, and is dependent upon the ones before it.
12+
The main benefit of this deployment structure is that the later tiers can be brought down (i.e. stacks destroyed)
13+
to save costs while keeping the earlier tiers. For instance, we could destroy the Service & Compute tiers to
14+
reduce the cost to maintain the farm when we know it will be idle while retaining all of our data;
15+
we could re-deploy the Service & Compute tiers at a later date to restore service to exactly the same state we left it in.
16+
17+
## Architecture
18+
19+
This sample application deploys a basic Deadline Render farm using Usage-Based Licensing. Below is a diagram of the architecture.
20+
21+
```
22+
+------------------------------------------------------------------------------------------------------------------------+
23+
| |
24+
| Private Hosted Zone |
25+
| |
26+
| +--------------------------------------------------------------------------------------------------------------------+ |
27+
| | | |
28+
| | VPC | |
29+
| | | |
30+
| | +------------------------------+ +-----------------------+ +-----------------------------+ | |
31+
| | | | | | | | | |
32+
| | | Repository | | Render Queue | | Usage-Based Licensing | | |
33+
| | | +--------------> +------------> | | |
34+
| | | +----------+ +-------------+ | Backend | +-------------------+ | Deadline | +-------------------------+ | | |
35+
| | | | | | | | API | | | | API | | | | | |
36+
| | | | Database | | File System | <--------------+ | RCS Fleet | <------------+ | License Forwarder Fleet | | | |
37+
| | | | | | | | | | | | | | | | | |
38+
| | | +----------+ +-----+-------+ | | | +-------+ | | | | +-------+ | | | |
39+
| | | | | | | | ALB | | | | | | ALB | | | | |
40+
| | +------------------------------+ | | +---+---+ | | | | +---+---+ | | | |
41+
| | | | | | | | | | | | | | |
42+
| | | | | +-----------+ | | | | +-----------+ | | | |
43+
| | | | | | | | | | | | | | | | | | |
44+
| | |Mounts | | v v v | | | | v v v | | | |
45+
| | |onto | | +-++ ++-+ | | | | +-++ ++-+ | | | |
46+
| | | | | | | ... | | | | | | | | ... | | | | | |
47+
| | | | | +--+ +--+ | | | | +--+ +--+ | | | |
48+
| | | +----> | | | | | | | | |
49+
| | | | | +-------------------+ | | +-------------------------+ | | |
50+
| | +-----------v--+ | | | | | | |
51+
| | | +----------------+ +-----^--------+--------+ +------^--------+-------------+ | |
52+
| | | Bastion Host | Can connect to | | | | | |
53+
| | | | | | | | | |
54+
| | +------------+-+ |Deadline| | | | |
55+
| | | | API | | | | |
56+
| | | | | | Get | | |
57+
| | | | | |Licenses| | |
58+
| | | +-+--------v--------+ | | | |
59+
| | | | | | | | |
60+
| | | | Worker Fleet | | | | |
61+
| | | Can connect to | | | | | |
62+
| | +---------------------------> +-------+ +-------------------+ | | |
63+
| | | | ALB | | | | |
64+
| | | +---+---+ <----------------------------+ | |
65+
| | | | | | |
66+
| | | +-----------+ | +----------------+ | |
67+
| | | | | | | | | | |
68+
| | | v v v | | Health Monitor | | |
69+
| | | +-++ ++-+ | | | | |
70+
| | | | | ... | | | Monitors | +-------+ | | |
71+
| | | +--+ +--+ <--------------+ | NLB | | | |
72+
| | | | | +-------+ | | |
73+
| | +-------------------+ | | | |
74+
| | +----------------+ | |
75+
| | | |
76+
| +--------------------------------------------------------------------------------------------------------------------+ |
77+
| |
78+
+------------------------------------------------------------------------------------------------------------------------+
79+
80+
```
81+
82+
### Components
83+
84+
All components in the render farm live within a [VPC](https://aws.amazon.com/vpc/), which is within a [Private Hosted Zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-private.html).
85+
86+
#### Repository
87+
88+
The Repository component contains the database and file system that store persistent data used by Deadline. These resources are initialized by the Deadline Repository installer. The database can either be [MongoDB](https://www.mongodb.com/) or [Amazon DocumentDB](https://aws.amazon.com/documentdb/).
89+
90+
#### Render Queue
91+
92+
The Render Queue component contains the fleet of [Deadline Remote Connection Server](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/remote-connection-server.html) instances behind an [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html). This acts as the central service for Deadline applications and is the only component that interacts with the Repository.
93+
94+
#### Usage-Based Licensing
95+
96+
The Usage-Based Licensing component contains the fleet of [Deadline License Forwarder](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/license-forwarder.html) instances behind an [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html). This provides [usage-based licenses](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/licensing-usage-based.html) to Deadline Workers that are rendering jobs and communicates with the Render Queue to store/retrieve licensing information.
97+
98+
#### Worker Fleet
99+
100+
The Worker Fleet component contains the fleet of [Deadline Worker](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/worker.html) instances behind an [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html). These are the compute power of the render farm that perform render jobs. They communicate with the Render Queue to carry out render jobs and with the Usage-Based Licensing component to obtain any licenses required for the jobs.
101+
102+
#### Health Monitor
103+
104+
The Health Monitor component contains a [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) used to perform application-level health checks on the Worker Fleet instances. These health checks ensure that the Deadline Worker application is operating properly. In the event that the Worker Fleet is deemed unhealthy, the Health Monitor will scale down the Worker Fleet to 0 to prevent unnecessary cost accrual.
105+
106+
#### Bastion Host
107+
108+
The Bastion Host is a `BastionHostLinux` construct that allows you to connect to the Render Queue and Worker Fleet if you would like to take a look at the state of these components. It is not an essential component to the render farm, so it can be omitted without side effects, if desired. To connect to it, please refer to [Bastion Hosts CDK documentation](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ec2-readme.html#bastion-hosts).
109+
110+
## Prerequisites
111+
112+
- You have an EC2 Amazon Machine Image (AMI) with the Deadline Worker application to run the worker nodes in the compute tier. Make note of the AMI ID as it will be used in this guide. You can use AWS Portal AMIs which can be found in Deadline's amis.json file (see https://awsportal.s3.amazonaws.com/10.1.9/Release/amis.json). **Note:** The link to the amis.json file contains the Deadline version (10.1.9), which you should change if you are using a different version of Deadline.
113+
- You have setup and configured the AWS CLI
114+
- Your AWS account already has CDK bootstrapped in the desired region by running `cdk bootstrap`
115+
116+
## Instructions
117+
118+
---
119+
**NOTE**
120+
121+
These instructions assume that your working directory is `examples/deadline/All-In-AWS-Infrastructure-Basic-Tiered/ts/` relative to the root of the AWS-RFDK package.
122+
123+
---
124+
125+
1. Install the dependencies of the sample app:
126+
```
127+
yarn install
128+
```
129+
2. Change the value in the `deadlineClientLinuxAmiMap` variable in `bin/config.ts` to include the region + AMI ID mapping of your EC2 AMI(s) with Deadline Worker.
130+
```ts
131+
// For example, in the us-west-2 region
132+
public readonly deadlineClientLinuxAmiMap: Record<string, string> = {
133+
['us-west-2']: '<your-ami-id>',
134+
// ...
135+
};
136+
```
137+
3. Create a binary secret in [SecretsManager](https://aws.amazon.com/secrets-manager/) that contains your [Usage-Based Licensing](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/aws-portal/licensing-setup.html?highlight=usage%20based%20licensing) certificates in a `.zip` file:
138+
```
139+
aws secretsmanager create-secret --name <name> --secret-binary fileb://<path-to-zip-file>
140+
```
141+
4. The output from the previous step will contain the secret's ARN. Change the value of the `ublCertificatesSecretArn` variable in `bin/config.ts` to the secret's ARN:
142+
```ts
143+
public readonly ublCertificatesSecretArn: string = '<your-secret-arn>';
144+
```
145+
5. Choose your UBL limits and change the value of the `ublLicenses` variable in `bin/config.ts` accordingly:
146+
```ts
147+
public readonly ublLicenses: UsageBasedLicense[] = [ /* <your-ubl-limits> (e.g. UsageBasedLicense.forMaya(10)) */ ];
148+
```
149+
---
150+
151+
**Note:** The next two steps are optional. You may skip these if you do not need SSH access into your render farm.
152+
153+
---
154+
6. Create an EC2 key pair to give you SSH access to the render farm:
155+
```
156+
aws ec2 create-key-pair --key-name <key-name>
157+
```
158+
7. Change the value of the `keyPairName` variable in `bin/config.ts` to your value for `<key-name>` in the previous step: <br><br>**Note:** Save the value of the "KeyMaterial" field as a file in a secure location. This is your private key that you can use to SSH into the render farm.
159+
```ts
160+
public readonly keyPairName: string = '<key-name>';
161+
```
162+
8. Choose the type of database you would like to deploy and change the value of the `deployMongoDB` variable in `bin/config.ts` accordingly:
163+
```ts
164+
// true = MongoDB, false = Amazon DocumentDB
165+
public readonly deployMongoDB: boolean = false;
166+
```
167+
9. If you set `deployMongoDB` to `true`, then you must accept the [SSPL license](https://www.mongodb.com/licensing/server-side-public-license) to successfully deploy MongoDB. To do so, change the value of `acceptSsplLicense` in `bin/config.ts`:
168+
```ts
169+
public readonly acceptSsplLicense: MongoDbSsplLicenseAcceptance = <value>;
170+
```
171+
10. Modify the `deadline_ver` field in the `config` block of `package.json` as desired, then stage the Docker recipes for `RenderQueue` and `UBLLicensing`:
172+
```
173+
yarn stage
174+
```
175+
11. Build the sample app:
176+
```
177+
yarn build
178+
```
179+
12. Deploy all the stacks in the sample app:
180+
```
181+
cdk deploy "*"
182+
```
183+
13. Once you are finished with the sample app, you can tear it down by running:
184+
```
185+
cdk destroy "*"
186+
```

‎examples/deadline/All-In-AWS-Infrastructure-Basic/ts/bin/app.ts

+123
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/usr/bin/env node
2+
/**
3+
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
4+
* SPDX-License-Identifier: Apache-2.0
5+
*/
6+
7+
import 'source-map-support/register';
8+
import * as path from 'path';
9+
import * as pkg from '../package.json';
10+
import { config } from './config';
11+
import * as cdk from '@aws-cdk/core';
12+
import { NetworkTier } from '../lib/network-tier';
13+
import { ServiceTier } from '../lib/service-tier';
14+
import {
15+
StorageTier,
16+
StorageTierDocDB,
17+
StorageTierMongoDB,
18+
} from '../lib/storage-tier';
19+
import { SecurityTier } from '../lib/security-tier';
20+
import {
21+
InstanceClass,
22+
InstanceSize,
23+
InstanceType,
24+
MachineImage,
25+
} from '@aws-cdk/aws-ec2';
26+
import { ComputeTier } from '../lib/compute-tier';
27+
28+
// ------------------------------ //
29+
// --- Validate Config Values --- //
30+
// ------------------------------ //
31+
32+
if (!config.ublCertificatesSecretArn) {
33+
throw new Error('UBL certificates secret ARN is required but was not specified.');
34+
}
35+
36+
if (!config.ublLicenses) {
37+
throw new Error('At least one UBL license must be specified');
38+
}
39+
40+
if (!config.keyPairName) {
41+
console.log('EC2 key pair name not specified. You will not have SSH access to the render farm.');
42+
}
43+
44+
if (config.deadlineClientLinuxAmiMap === {['region']: 'ami-id'}) {
45+
throw new Error('Deadline Client Linux AMI map is required but was not specified.');
46+
}
47+
48+
// ------------------- //
49+
// --- Application --- //
50+
// ------------------- //
51+
52+
const env = {
53+
account: process.env.CDK_DEPLOY_ACCOUNT ?? process.env.CDK_DEFAULT_ACCOUNT,
54+
region: process.env.CDK_DEPLOY_REGION ?? process.env.CDK_DEFAULT_REGION,
55+
};
56+
57+
const app = new cdk.App();
58+
59+
// -------------------- //
60+
// --- Network Tier --- //
61+
// -------------------- //
62+
63+
const network = new NetworkTier(app, 'NetworkTier', { env });
64+
65+
// --------------------- //
66+
// --- Security Tier --- //
67+
// --------------------- //
68+
69+
const security = new SecurityTier(app, 'SecurityTier', {
70+
env,
71+
vpc: network.vpc,
72+
});
73+
74+
// -------------------- //
75+
// --- Storage Tier --- //
76+
// -------------------- //
77+
78+
let storage: StorageTier;
79+
if (config.deployMongoDB) {
80+
storage = new StorageTierMongoDB(app, 'StorageTier', {
81+
env,
82+
vpc: network.vpc,
83+
databaseInstanceType: InstanceType.of(InstanceClass.R5, InstanceSize.LARGE),
84+
rootCa: security.rootCa,
85+
dnsZone: network.dnsZone,
86+
acceptSsplLicense: config.acceptSsplLicense,
87+
keyPairName: config.keyPairName ? config.keyPairName : undefined,
88+
});
89+
} else {
90+
storage = new StorageTierDocDB(app, 'StorageTier', {
91+
env,
92+
vpc: network.vpc,
93+
databaseInstanceType: InstanceType.of(InstanceClass.R5, InstanceSize.LARGE),
94+
});
95+
}
96+
97+
// -------------------- //
98+
// --- Service Tier --- //
99+
// -------------------- //
100+
101+
const service = new ServiceTier(app, 'ServiceTier', {
102+
env,
103+
database: storage.database,
104+
fileSystem: storage.fileSystem,
105+
vpc: network.vpc,
106+
dockerRecipesStagePath: path.join(__dirname, '..', pkg.config.stage_path), // Stage directory in config is relative, make it absolute
107+
ublCertsSecretArn: config.ublCertificatesSecretArn,
108+
ublLicenses: config.ublLicenses,
109+
rootCa: security.rootCa,
110+
dnsZone: network.dnsZone,
111+
});
112+
113+
// -------------------- //
114+
// --- Compute Tier --- //
115+
// -------------------- //
116+
117+
new ComputeTier(app, 'ComputeTier', {
118+
env,
119+
vpc: network.vpc,
120+
renderQueue: service.renderQueue,
121+
workerMachineImage: MachineImage.genericLinux(config.deadlineClientLinuxAmiMap),
122+
keyPairName: config.keyPairName ? config.keyPairName : undefined,
123+
});

0 commit comments

Comments
 (0)