Skip to content

Commit da650cb

Browse files
adityahaseadityahase
committed
chore: WIP
1 parent fadc38e commit da650cb

33 files changed

+1090
-5
lines changed

‎pilot/bootstrap/__init__.py

Whitespace-only changes.

‎pilot/pilot/design/concepts.md

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Concepts
2+
---
3+
## Global
4+
5+
### Region
6+
Inter Region Latencies
7+
10 - 100
8+
Maybe 300ms
9+
10+
11+
### Availability Zone
12+
< 10 ms
13+
~3 ms
14+
15+
Some components are only available in the same Zone
16+
- AWS EBS volume
17+
- Subnet
18+
19+
### Node
20+
21+
22+
---
23+
24+
## Orchestration
25+
26+
### Allocation
27+
An Allocation is a mapping between a task group in a job and a client node. A single job may have hundreds or thousands of task groups, meaning an equivalent number of allocations must exist to map the work to client machines. Allocations are created by the Nomad servers as part of scheduling decisions made during an evaluation.

‎pilot/pilot/design/general.md

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
Single Global Control Plane
2+
3+
e.g.
4+
- AWS Dashboard
5+
- DO Dashboard
6+
7+
For handling Auth / Teams / Billing Centrally
8+
9+
### Regions
10+
#### Authority Region
11+
There's still single Primary
12+
Maybe Control Plane resides here
13+
14+
#### Isolated
15+
No shared fault domain
16+
Failures don't impact each other
17+
18+
#### Autonomous ?!?
19+
20+
21+
#### Minimal Communication
22+
Data usually doesn't leave the Region.
23+
Public Internet is slow and Expensive
24+

‎pilot/pilot/design/influences.md

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Nomad
2+
Single Binary
3+
Node
4+
- Drivers
5+
- VMs
6+
- KVM
7+
- FireCracker
8+
- Containers
9+
- Podman
10+
- Docker
11+
- Rkt?
12+
- Metadata
13+
14+
Self Registration
15+
16+
Federation
17+
18+
#
19+

‎pilot/pilot/design/objects.md

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
Region
2+
3+
Cluster?!?! VPC ?!!?! Smaller Blast Redius
4+
5+
Zone
6+
- Name
7+
- Subnet
8+
9+
Node
10+
- Name / UUID
11+
- Version
12+
- Status
13+
- Eligible?
14+
- Drained?
15+
- Address
16+
17+
18+
- Pool?
19+
- Zone
20+
- Drivers (KVM, Firecracker, Doker, Podman)
21+
- Status
22+
- Network (Bridge)
23+
- Version
24+
- Runtime?
25+
26+
- Resources (RAM, CPU)
27+
28+
Attributes
29+
- OS
30+
- Arch
31+
- CPU / Cores / Freq
32+
33+
34+
Node Events
35+
36+
37+
Client??!?!
38+
Server?!?!
39+

‎pilot/pilot/design/provision.md

+53
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Provision
2+
There's no way to do something like tofu.deploy() right now.
3+
4+
This goes through two levels of abstractions?. JSII
5+
6+
### CDKTF
7+
We write TerraformStack subclass and define our configration in init. It should something like this
8+
```python
9+
from cdktf import App, Fn, TerraformStack, TerraformVariable
10+
from cdktf_cdktf_provider_digitalocean.provider import DigitaloceanProvider
11+
from cdktf_cdktf_provider_digitalocean.vpc import Vpc
12+
from constructs import Construct
13+
14+
15+
class MyStack(TerraformStack):
16+
def __init__(self, scope: Construct, id: str):
17+
super().__init__(scope, id)
18+
do_token_variable = TerraformVariable(self,"do_token", type="string")
19+
DigitaloceanProvider(self, "digitalocean", token=do_token_variable.string_value)
20+
vpc = Vpc(self, "example_vpc", name="vpc-1", region="blr-1", ip_range="ip_range")
21+
22+
```
23+
24+
Unfortunately, Pilot config isn't static. But good news is, this is actually implemented as following
25+
26+
1. Define TerraformStack subclass, like we did before
27+
This is equivalent of writing a HCL file
28+
29+
2. Define an app and call `synth` on it
30+
31+
```python
32+
from cdktf import App, Fn, TerraformStack, TerraformVariable
33+
34+
35+
app = App()
36+
MyStack(app, "cdktf-demo")
37+
app.synth()
38+
```
39+
40+
41+
3. Apply generated plan
42+
`cdktf deploy`
43+
We can open up the implemntation and see what happens underneath.
44+
45+
1. If the implementation is complicated then we can run `cdktf deploy` ourselves
46+
47+
48+
---
49+
50+
We need to put some dynamic logic on Step 1.
51+
We can't write a class everytime.
52+
Generating Python code is basically the same as writing HCL
53+
What we can do is build a class implementation and app implementation on the fly

‎pilot/pilot/design/workloads.md

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# System
2+
DaemonSet
3+
4+
Run on all (or matching nodes at all times)
5+
6+
# Service
7+
Typical Container

‎pilot/provision/README.wtf.md

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
We want to do as much work in Python / JSON as possible
2+
3+
Luckily Tofu helps out.
4+
5+
Every plan / state can be stored as JSON or converted to JSON.
6+
7+
Here's the rough idea
8+
9+
```mermaid
10+
stateDiagram
11+
code: Python Declaration
12+
plan: Plan
13+
json: Synthesized JSON
14+
infra: Infrastructure
15+
code --> json: app.synth()
16+
json --> plan: tofu plan
17+
plan --> infra: tofu deploy
18+
```
19+
20+
21+
Same thing in words
22+
23+
1. Write Python code in `TerraformStack.__init__()` that describes the infra we need
24+
2. "Synthesize" this TerraformStack object `app.synth()`
25+
26+
The synth actually executes the `__init__` so we can do whatever we want in Python (loops, conditionals etc)
27+
28+
`synth` generates a `cdktf.out/stacks/<stack-name>/cdktf.json` file in the working directory (this is `frappe-bench/sites`)
29+
30+
We will use `sites/<site>/stacks` for now. So the state moves with the site without any special handling.
31+
32+
TODO: Include this directory in the file backups.
33+
34+
3. Store this synthesized JSON in some DocType Provision
35+
36+
37+
Note: Our Stack can have bugs or the code that defines what we need can have bugs. Have a way to prevent catastrophies at this stage. We need sanity checks in Production to guard against
38+
- Don't trigger anything that can cause data loss
39+
- Don't trigger massive changes ( More than n resources at a time)
40+
- Cross stack changes ?! (Don't delete someone other region?!)

‎pilot/provision/bootstrap.py

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Copyright (c) 2024, Frappe and contributors
2+
# For license information, please see license.txt
3+
4+
import frappe
5+
6+
PROVIDER = "do"
7+
DO_ACCESS_TOKEN = ""
8+
CIDR = "10.10.0.0/24"
9+
REGION = "blr1"
10+
TITLE = "Bangalore"
11+
NAME = "do-bangalore-blr1"
12+
13+
14+
def create():
15+
if frappe.db.exists("Region", NAME):
16+
region = frappe.get_doc("Region", NAME)
17+
else:
18+
region = frappe.new_doc(
19+
"Region",
20+
**{
21+
"name": NAME,
22+
"access_token": DO_ACCESS_TOKEN,
23+
"cloud_provider": PROVIDER,
24+
"region_slug": REGION,
25+
"title": TITLE,
26+
"vpc_cidr_block": CIDR,
27+
},
28+
).insert()
29+
30+
region.provision()
31+
32+
33+
def destroy():
34+
if frappe.db.exists("Region", NAME):
35+
region = frappe.get_doc("Region", NAME)
36+
region.destroy()
37+
38+
39+
def clear():
40+
doctypes = ["Provision Declaration", "Provision Plan", "Provision State", "Provision Action"]
41+
for doctype in doctypes:
42+
frappe.db.delete(doctype)

‎pilot/provision/doctype/provision_action/__init__.py

Whitespace-only changes.

‎pilot/provision/doctype/provision_action/provision_action.js

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
// Copyright (c) 2024, Frappe and contributors
2+
// For license information, please see license.txt
3+
4+
// frappe.ui.form.on("Provision Action", {
5+
// refresh(frm) {
6+
7+
// },
8+
// });

‎pilot/provision/doctype/provision_action/provision_action.json

+105
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
{
2+
"actions": [],
3+
"autoname": "autoincrement",
4+
"creation": "2024-07-29 11:26:21.416387",
5+
"doctype": "DocType",
6+
"engine": "MyISAM",
7+
"field_order": [
8+
"region",
9+
"stack",
10+
"column_break_payx",
11+
"action",
12+
"section_break_tioa",
13+
"parsed_output",
14+
"output",
15+
"error"
16+
],
17+
"fields": [
18+
{
19+
"fieldname": "stack",
20+
"fieldtype": "Data",
21+
"in_filter": 1,
22+
"in_list_view": 1,
23+
"in_standard_filter": 1,
24+
"label": "Stack",
25+
"read_only": 1,
26+
"reqd": 1
27+
},
28+
{
29+
"fieldname": "action",
30+
"fieldtype": "Data",
31+
"in_list_view": 1,
32+
"in_preview": 1,
33+
"in_standard_filter": 1,
34+
"label": "Action",
35+
"read_only": 1,
36+
"reqd": 1
37+
},
38+
{
39+
"default": "{}",
40+
"fieldname": "output",
41+
"fieldtype": "Code",
42+
"label": "Output",
43+
"read_only": 1
44+
},
45+
{
46+
"fieldname": "error",
47+
"fieldtype": "Code",
48+
"label": "Error",
49+
"read_only": 1
50+
},
51+
{
52+
"fieldname": "region",
53+
"fieldtype": "Link",
54+
"in_filter": 1,
55+
"in_list_view": 1,
56+
"in_standard_filter": 1,
57+
"label": "Region",
58+
"options": "Region",
59+
"read_only": 1,
60+
"reqd": 1
61+
},
62+
{
63+
"fieldname": "column_break_payx",
64+
"fieldtype": "Column Break"
65+
},
66+
{
67+
"fieldname": "section_break_tioa",
68+
"fieldtype": "Section Break"
69+
},
70+
{
71+
"default": "{}",
72+
"fieldname": "parsed_output",
73+
"fieldtype": "Code",
74+
"label": "Parsed Output",
75+
"read_only": 1
76+
}
77+
],
78+
"index_web_pages_for_search": 1,
79+
"links": [],
80+
"modified": "2024-08-09 16:33:22.536559",
81+
"modified_by": "Administrator",
82+
"module": "Provision",
83+
"name": "Provision Action",
84+
"naming_rule": "Autoincrement",
85+
"owner": "Administrator",
86+
"permissions": [
87+
{
88+
"create": 1,
89+
"delete": 1,
90+
"email": 1,
91+
"export": 1,
92+
"print": 1,
93+
"read": 1,
94+
"report": 1,
95+
"role": "System Manager",
96+
"share": 1,
97+
"write": 1
98+
}
99+
],
100+
"show_title_field_in_link": 1,
101+
"sort_field": "creation",
102+
"sort_order": "DESC",
103+
"states": [],
104+
"title_field": "stack"
105+
}

0 commit comments

Comments
 (0)