Reorganize packaging

.github/workflows/build.yml

@@ -24,12 +24,18 @@ jobs:
       shell: bash --noprofile --norc -x -eo pipefail {0}
       run: |
         go install github.com/golangci/golangci-lint/cmd/[email protected]
-
+    - 
+      name: Build ui
+      working-directory: ./ui/web
+      run: |
+        curl -fsSL https://get.pnpm.io/install.sh | SHELL=bash sh -
+        ~/.local/share/pnpm/pnpm install
+        ~/.local/share/pnpm/pnpm build
     - 
       name: Run Lint
       shell: bash --noprofile --norc -x -eo pipefail {0}
       run: |
-        golangci-lint run
+        golangci-lint run --timeout 5m
 
   test:
     runs-on: ubuntu-latest
@@ -47,20 +53,27 @@ jobs:
       run: |
         go install github.com/onsi/ginkgo/v2/ginkgo
     - 
-      name: Test nex-agent
-      working-directory: ./nex-agent
+      name: Build ui
+      working-directory: ./ui/web
+      run: |
+        curl -fsSL https://get.pnpm.io/install.sh | SHELL=bash sh -
+        ~/.local/share/pnpm/pnpm install
+        ~/.local/share/pnpm/pnpm build
+    - 
+      name: Test agent
+      working-directory: ./agent
       run: go test -v ./...
     - 
-      name: Test nex-cli
+      name: Test nex cli
       working-directory: ./nex
       run: go test -v ./...
     - 
       name: Test control-api
-      working-directory: ./control-api
+      working-directory: ./internal/control-api
       run: ginkgo
     - 
-      name: Test nex-node
-      working-directory: ./nex-node
+      name: Test nex node
+      working-directory: ./internal/node
       run: go test -v ./...
 
   build:
@@ -74,11 +87,17 @@ jobs:
       with:
         go-version: '1.21'
     - 
-      name: Build nex-agent
-      working-directory: ./nex-agent/cmd/nex-agent
+      name: Build agent
+      working-directory: ./agent/cmd/nex-agent
       run: go build -tags netgo -ldflags '-extldflags "-static"'
     - 
-      name: Build nex-cli
+      name: Build ui
+      working-directory: ./ui/web
+      run: |
+        curl -fsSL https://get.pnpm.io/install.sh | SHELL=bash sh -
+        ~/.local/share/pnpm/pnpm install
+        ~/.local/share/pnpm/pnpm build
+    - 
+      name: Build nex cli
       working-directory: ./nex
       run: go build .
-

.gitignore

@@ -26,5 +26,7 @@ go.work
 
 # stash things locally 
 local/
-
 dist/
+
+# nex binaries
+nex/nex

.goreleaser.yaml

@@ -6,7 +6,7 @@ before:
 
 builds:
   - 
-    id: "nex-cli"
+    id: "nex"
     main: "./nex"
     binary: "nex"
     goos:
@@ -17,8 +17,8 @@ builds:
       - -s -w --X main.VERSION={{.Version}} -X main.COMMIT={{.Commit}} -X main.BUILDDATE={{.Date}}
       - -extldflags "-static"
   -
-    id: "nex-agent"
-    main: "./nex-agent/cmd/nex-agent"
+    id: "agent"
+    main: "./agent/cmd/nex-agent"
     binary: "nex-agent"
     goos:
       - linux

CODE-OF-CONDUCT.md

@@ -1,3 +1,3 @@
 ## Community Code of Conduct
 
-The NATS execution engine follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
+The NATS execution engine follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).

README.md

@@ -4,13 +4,11 @@
 # NATS Execution Engine
 Turn your NATS infrastructure into a distributed workload deployment and execution engine.
 
-* [nex-agent](./nex-agent) - Agent that runs inside a Firecracker VM, responsible for running untrusted workloads. Not something end users need to interact with.
-* [nex-node](./nex-node) - Service running on a NEX node. Exposes a control API, starts/stops firecracker processes, communicates with the agent inside each process.
-* [nex-cli](./nex-cli) - CLI for communicating with NEX nodes
-* [control-api](./control-api/) - The API for communicating with and remotely controlling NEX nodes
-* [agent-api](./agent-api/) - Data types and other API data for protocol used between `nex-node` and `nex-agent` across the firecracker boundary. This is an internal API unlikely to be of interest to anyone 
-* [fc-image](./fc-image/) - Tools for building the rootfs (ext4) file system for use in firecracker VMs
-other than contributors.
+* [agent](./agent) - Agent that runs inside a Firecracker VM, responsible for running untrusted workloads. Not something end users need to interact with.
+* [fc-image](./agent/fc-image/) - Tools for building the rootfs (ext4) file system for use in firecracker VMs
+* [node](./internal/node) - Service running on a NEX node. Exposes a control API, starts/stops firecracker processes, communicates with the agent inside each process.
+* [nex](./nex) - CLI for communicating with NEX nodes
+* [ui](./ui) - User interface for viewing the status of NEX nodes in a web browser
 
 
 ## Quickstart

Taskfile.yml

@@ -9,21 +9,14 @@ vars:
   NAME: "nex"
 
 tasks:  
-  nex-agent:
-    dir: nex-agent/cmd/nex-agent
+  agent:
+    dir: agent/cmd/nex-agent
     sources:
       - "*.go"
     cmds:
       - go build -tags netgo -ldflags '-extldflags "-static"'
 
-  nex-node:
-    dir: nex-node/cmd/nex-node
-    sources:
-      - "*.go"
-    cmds:
-      - go build -tags netgo -ldflags '-extldflags "-static"'
-
-  nex-cli:
+  nex:
     dir: nex
     sources:
       - "*.go"
@@ -37,5 +30,11 @@ tasks:
     cmds:
       - go build -tags netgo -ldflags '-extldflags "-static"'
 
+  ui:
+    dir: ui/web
+    cmds:
+      - curl -fsSL https://get.pnpm.io/install.sh | sh -
+      - pnpm build
+
   build:
-    deps: [nex-agent, nex-node, nex-cli, echo-service]
+    deps: [agent, ui, nex, echo-service]

nex-agent/README.md → agent/README.md

@@ -6,4 +6,4 @@ This process is configured as an OpenRC startup service so it's automatically ru
 
 Once running, the agent process uses the metadata available from MMDS to connect via NATS to the node host, where it can receive workload instructions and publish logs and events.
 
-It's worth noting that every `nex-agent` exists within _one_ firecracker VM and will only ever manager _one_ workload. 
+It's worth noting that every `nex-agent` exists within _one_ firecracker VM and will only ever manager _one_ workload. 

nex-agent/agent.go → agent/agent.go

@@ -9,8 +9,8 @@ import (
 	"path"
 	"time"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
-	"github.com/ConnectEverything/nex/nex-agent/providers"
+	"github.com/ConnectEverything/nex/agent/providers"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 	"github.com/cloudevents/sdk-go/pkg/cloudevents"
 	"github.com/nats-io/nats.go"
 )

agent/cmd/nex-agent/build-static.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+set -xe
+
+export GIN_MODE=release
+go build -tags netgo -ldflags '-extldflags "-static"'

nex-agent/cmd/nex-agent/main.go → agent/cmd/nex-agent/main.go

@@ -8,7 +8,7 @@ import (
 	"os"
 	"syscall"
 
-	nexagent "github.com/ConnectEverything/nex/nex-agent"
+	nexagent "github.com/ConnectEverything/nex/agent"
 )
 
 var (

nex-agent/events.go → agent/events.go

@@ -4,7 +4,7 @@ import (
 	"fmt"
 	"os"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 )
 
 const NexEventSourceNexAgent = "nex-agent"

agent/fc-image/.gitignore

@@ -0,0 +1 @@
+*.ext4

fc-image/README.md → agent/fc-image/README.md

@@ -1,2 +1,2 @@
 # Firecracker Image
-The files in this directory are what are used to generate the default `rootfs.ext4` file that is used by Firecracker as the rootfs image when launched. You can use the files in this directory as inspiration if you intend to build your own root fs. **NOTE**, however, that your custom root FS must still launch the `nex-agent` as a startup process, otherwise it won't work with NEX nodes.
+The files in this directory are what are used to generate the default `rootfs.ext4` file that is used by Firecracker as the rootfs image when launched. You can use the files in this directory as inspiration if you intend to build your own root fs. **NOTE**, however, that your custom root FS must still launch the `nex-agent` as a startup process, otherwise it won't work with NEX nodes.

fc-image/build-rootfs.sh → agent/fc-image/build-rootfs.sh

@@ -9,10 +9,10 @@ mount rootfs.ext4 /tmp/my-rootfs
 
 docker run -i --rm \
     -v /tmp/my-rootfs:/my-rootfs \
-    -v "$(pwd)/../nex-agent/cmd/nex-agent/nex-agent:/usr/local/bin/agent" \
+    -v "$(pwd)/../cmd/nex-agent/nex-agent:/usr/local/bin/agent" \
     -v "$(pwd)/openrc-service.sh:/etc/init.d/agent" \
     alpine sh <setup-alpine.sh
 
 umount /tmp/my-rootfs
 
-# rootfs available under `rootfs.ext4`
+# rootfs available under `rootfs.ext4`

fc-image/setup-alpine.sh → agent/fc-image/setup-alpine.sh

@@ -27,4 +27,4 @@ for dir in dev proc run sys var tmp; do mkdir /my-rootfs/${dir}; done
 
 chmod 1777 /my-rootfs/tmp
 mkdir -p /my-rootfs/home/nex/
-chown 1000:1000 /my-rootfs/home/nex/
+chown 1000:1000 /my-rootfs/home/nex/

nex-agent/logger.go → agent/logger.go

@@ -4,7 +4,7 @@ import (
 	"encoding/json"
 	"fmt"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 )
 
 // logEmitter implements the writer interface that allows us to capture a workload's

nex-agent/metadata.go → agent/metadata.go

@@ -8,7 +8,7 @@ import (
 	"net/http"
 	"time"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 )
 
 // MmdsAddress is the address used by the agent to query firecracker MMDS

nex-agent/providers/api.go → agent/providers/api.go

@@ -3,8 +3,8 @@ package providers
 import (
 	"errors"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
-	"github.com/ConnectEverything/nex/nex-agent/providers/lib"
+	"github.com/ConnectEverything/nex/agent/providers/lib"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 )
 
 // NexExecutionProviderELF Executable Linkable Format execution provider

nex-agent/providers/lib/elf.go → agent/providers/lib/elf.go

@@ -9,7 +9,7 @@ import (
 	"strings"
 	"time"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 )
 
 // ELF execution provider implementation

nex-agent/providers/lib/oci.go → agent/providers/lib/oci.go

@@ -3,7 +3,7 @@ package lib
 import (
 	"errors"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 )
 
 // OCI execution provider implementation

nex-agent/providers/lib/v8.go → agent/providers/lib/v8.go

@@ -6,7 +6,7 @@ import (
 	"os"
 	"time"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 	"github.com/nats-io/nats.go"
 	v8 "rogchap.com/v8go"
 )

nex-agent/providers/lib/wasm.go → agent/providers/lib/wasm.go

@@ -7,7 +7,7 @@ import (
 	"io"
 	"os"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 	"github.com/nats-io/nats.go"
 	"github.com/tetratelabs/wazero"
 	"github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1"

nex-agent/providers/lib/wasm_test.go → agent/providers/lib/wasm_test.go

@@ -3,7 +3,7 @@ package lib
 import (
 	"testing"
 
-	agentapi "github.com/ConnectEverything/nex/agent-api"
+	agentapi "github.com/ConnectEverything/nex/internal/agent-api"
 )
 
 func TestWasmExecution(t *testing.T) {

docs/architecture.md

@@ -1,7 +1,7 @@
 # NATS Execution Engine Architecture
-The following diagram provides a (rough) view of the various components of the system. Here, `nex-node` processes all attach to a NATS system. Each `nex-node` process is responsible for spinning up multiple firecracker VMs. Within each firecracker VM is a `nex-agent` process, and spawned by each `nex-agent` process is a _single_, untrusted workload (executable). 
+The following diagram provides a (rough) view of the various components of the system. Here, `nex node` processes all attach to a NATS system. Each `nex node` process is responsible for spinning up multiple firecracker VMs. Within each firecracker VM is a `nex-agent` process, and spawned by each `nex-agent` process is a _single_, untrusted workload (executable). 
 
 In production, if an untrusted workload has a NATS connection, it should not use the same URL+credentials that the node servers are using. Remember that credentials and URLs are all passed to workloads through encrypted environment variables.
 
 
-![architecture diagram](./nex-arch.png)
+![architecture diagram](./nex-arch.png)

docs/getting_started.md

@@ -2,7 +2,7 @@
 Nex is a suite of tools for deploying applications and services into a NATS environment. Anywhere messages can reach, so too can your deployments. Nex is a completely separate add-on to NATS and doesn't require any custom forks of existing software. 
 
 ## Installation
-The first thing you'll want to do is install the `nex-node` and `nex-cli` binaries and compile the `nex-agent` binary (required to create a Root FS, discussed shortly).
+The first thing you'll want to do is install the `nex` binary and compile the `nex-agent` binary (required to create a Root FS, discussed shortly).
 
 At the moment since Nex is still in its infancy, we don't have an automated release or installation process. This means that you'll want to clone this repository, go to the root directory, and run `task build`. If you don't have (or want to) task installed, you can always read the [Taskfile.yml](../Taskfile.yml) file and manually run each of the build commands, e.g.
 
@@ -15,12 +15,12 @@ Consult the appropriate documentation if you're building workloads in another la
 We will have an installation process available soon. We will also have official release images for the root filesystem, letting you skip the next step.
 
 ### Building a Root File System
-Every firecracker virtual machine is a melding of a Linux kernel binary and a root file system. In our case, the root file system contains an [agent](../nex-agent/) configured to start automatically during boot time. To make sure this agent is in the root file system, follow the instructions in the [fc-image](../fc-image/) directory to create a `rootfs.ext4` file. 
+Every firecracker virtual machine is a melding of a Linux kernel binary and a root file system. In our case, the root file system contains an [agent](../agent/) configured to start automatically during boot time. To make sure this agent is in the root file system, follow the instructions in the [fc-image](../agent/fc-image/) directory to create a `rootfs.ext4` file. 
 
 You should locate and download whichever Linux kernel binary you think you'll want. We do all our testing with the most recent stable Linux kernel configured with the defaults.
 
 ### Install CNI Prerequisites
-The [Nex node](../nex-node/) process creates firecracker virtual machines and provisions CNI networks according to CNI configuration files. You'll need to make sure that whatever plugins you need are installed in the `/opt/cni/bin` directory. We currently use the following plugins in the default CNI configuration:
+The [NEX node](../internal/node/) process creates firecracker virtual machines and provisions CNI networks according to CNI configuration files. You'll need to make sure that whatever plugins you need are installed in the `/opt/cni/bin` directory. We currently use the following plugins in the default CNI configuration:
 
 * `ptp`
 * `tc-redirect-tap`
@@ -29,15 +29,15 @@ The [Nex node](../nex-node/) process creates firecracker virtual machines and pr
 `ptp` and `host-local` are part of the [CNI plugins](https://github.com/containernetworking/plugins) bundle. `tc-redirect-tap` is a plugin managed and released by [AWS](https://github.com/awslabs/tc-redirect-tap)
 
 ### Preflight Checklist
-Nex has a few interesting requirements because it's essentially a distributed scheduling framework. To make this easier for you during setup, you can run `nex-node preflight` and specify the path to an intended machine configuration. It will then go looking for all of the appropriate plugins, config files, etc. If you don't already have a CNI configuration for your device, it'll create one for you.
+Nex has a few interesting requirements because it's essentially a distributed scheduling framework. To make this easier for you during setup, you can run `nex node preflight` and specify the path to an intended machine configuration. It will then go looking for all of the appropriate plugins, config files, etc. If you don't already have a CNI configuration for your device, it'll create one for you.
 
 ## Starting a Nex Node
 Nex is an opt-in, completely separate add-on to NATS. It doesn't require any custom forks, distributions, or servers. All you need to do is start up a nex node process as an "empty vessel" awaiting workloads via remote commands. 
 
 To start a nex node, you just need to run the following command:
 
 ```
-$ sudo nex-node up --config=simple.json
+$ sudo nex node up --config=simple.json
 ```
 This will start a nex node process using the data found in `simple.json` to configure the node host as well as define the cookie cutter template used for producing new firecracker virtual machines.
 
@@ -73,7 +73,7 @@ This will attempt to discover all running nodes. If you don't see your node in t
 ## Running Workloads
 At this point, you should have:
 
-* A version of the [nex-cli](../nex-cli/) binary, `nex`.
+* A version of the [nex binary](../nex/), `nex`.
 * A running node host that you launched in the previous section. 
 * Verified the node host is running via `nex node ls`
 
@@ -124,4 +124,4 @@ This will attempt to run the workload stored in object store `MYFILES` under the
 If you're using the echo service from our examples, then when you run `nats micro ls` you'll actually see the instance of the service running inside a nex node. If you issue another run command (not `devrun`), you'll quickly see a second instance of that service running.
 
 ### Observing Workloads
-You can monitor a stream of logs and events for running workloads. These events and logs are published on `$NEX.events.*` and `$NEX.logs.>` respectively. However, there's a more user-friendly way to monitor this using `nex logs` or `nex events`.
+You can monitor a stream of logs and events for running workloads. These events and logs are published on `$NEX.events.*` and `$NEX.logs.>` respectively. However, there's a more user-friendly way to monitor this using `nex logs` or `nex events`.