244 spicy up readmemd (#245)

* Spice up readme.

* Added example queries.

* Added golem examples.

* Read me assets.

* Update getting started doc.

* Revamping docs.

README.md

@@ -1,10 +1,83 @@
+
+<div align="center">
+ <img src="./docs/assets/img/realm_250px.png">
+</div>
+
+# Realm
 ![test-status](https://github.com/kcarretto/realm/actions/workflows/tests.yml/badge.svg?branch=main)
 [![codecov](https://codecov.io/github/kcarretto/realm/branch/main/graph/badge.svg?token=KSRPHYDIE4)](https://app.codecov.io/github/kcarretto/realm)
 [![Go Report Card](https://goreportcard.com/badge/github.com/kcarretto/realm)](https://goreportcard.com/report/github.com/kcarretto/realm)
 [![Rust Report Card](https://rust-reportcard.xuri.me/badge/github.com/kcarretto/realm)](https://rust-reportcard.xuri.me/report/github.com/kcarretto/realm)
 [![Docs](https://img.shields.io/badge/read%20our-docs-informational)](https://docs.realm.pub/)
 
 
-# Realm
+Realm is a cross platform Red Team engagement platform with a focus on automation and reliability. 
+
+![realm-logo](./docs/assets/img/realm_create_job.png)
+
+## Features
+### Agent (imix)
+- Written in rust with support for MacOS, Linux, and Windows.
+- Supports long running jobs by reading output from jobs in real time.
+- Interval callback times.
+- Simple file based configuration.
+- Embedded files.
+- Built-in interpreter.
+
+### Server (tavern)
+- Web interface.
+- Group actions.
+- graphql backend for easy API access.
+- OAuth login support.
+- Cloud native deployment with pre-made terraform for production deployments.
+
+
+### Built-in interpreter (eldritch)
+- Reflective DLL Loader.
+- Port scanning.
+- Remote execution over SSH.
+- And much much more: https://docs.realm.pub/user-guide/eldritch
+
+## Quickstart guide
+*To deploy a production ready instance see the [tavern setup guide](https://docs.realm.pub/user-guide/tavern).*
+### Start the server
+```bash
+git clone https://github.com/KCarretto/realm.git
+cd realm
+go run ./tavern
+
+# If you'd like to test without deploying an agent use the test data.
+ENABLE_TEST_DATA=1 go run ./tavern
+```
+### Start the agent
+```bash
+git clone https://github.com/KCarretto/realm.git
+cd realm/implants/imix
+
+# Create the config file
+cat <<EOF > /tmp/imix-config.json
+{
+    "service_configs": [],
+    "target_forward_connect_ip": "127.0.0.1",
+    "target_name": "test1234",
+    "callback_config": {
+        "interval": 4,
+        "jitter": 1,
+        "timeout": 4,
+        "c2_configs": [
+        {
+            "priority": 1,
+            "uri": "http://127.0.0.1/graphql"
+        }
+        ]
+    }
+}
+EOF
+
+cargo run -- -c /tmp/imix-config.json
+
+```
+
 
-Realm is a Red Team engagement platform. Check out the [docs](https://docs.realm.pub) to learn more.
+## Want to contribute start here
+https://docs.realm.pub/dev-guide/introduction

docs/_data/toc.yml

@@ -5,13 +5,15 @@
       url: "user-guide/getting-started"
     - title: "Tavern"
       url: "user-guide/tavern"
+    - title: "Imix"
+      url: "user-guide/imix"
+    - title: "Golem"
+      url: "user-guide/golem"
     - title: "Eldritch"
       url: "user-guide/eldritch"
       children:
         - title: Standard Library
           url: "user-guide/eldritch#standard-library"
-    - title: "Golem"
-      url: "user-guide/golem"
 - title: Developer Guide
   url: dev-guide
   links:
@@ -46,15 +48,11 @@
           url: "dev-guide/eldritch#Overview"
         - title: "Creating a function"
           url: "dev-guide/eldritch#creating-a-function"
+        - title: "OS Specific functions"
+          url: "dev-guide/eldritch#os-specific-functions"
+        - title: "Notes about using dictionary type Dict"
+          url: "dev-guide/eldritch#notes-about-using-dictionary-type-dict"
         - title: "Notes about asynchronous Eldritch code"
           url: "dev-guide/eldritch#notes-about-asynchronous-eldritch-code"
-  # links:
-  #   - title: "Getting Started"
-  #     url: "user-guide/getting-started"
-  #   - title: "Eldritch"
-  #     url: "user-guide/eldritch"
-  #     children:
-  #       - title: Standard Library
-  #         url: "user-guide/eldritch#StandardLibrary"
 - title: "About"
   url: "about"

docs/_docs/dev-guide/eldritch.md

@@ -2,7 +2,7 @@
 title: Eldritch
 tags: 
  - Dev Guide
-description: Want to contribute to Eldritch? Start here!
+description: Want to implement new functionality in the agent? Start here!
 permalink: dev-guide/eldritch
 ---
 
@@ -29,46 +29,50 @@ Want to contribute to Eldritch but aren't sure what to build check our ["good fi
 ## What files should I modify to make an Eldritch function.
 ---
 #### Documentation
-`docs/_docs/user-guide/Eldritch.md`
+`docs/_docs/user-guide/eldritch.md`
 Add your function to the docs. Give your function a unique and descriptive name. Assign it to an Eldritch module.
 
 Currently Eldritch has four modules your function can fall under:
 * `file`: Is used for any on disk file processing.
 * `pivot`: Is used to migrate to identify, and migrate between systems. The pivot module is also responsible for facilitating connectivity within an environment.
 * `process`: Is used to manage running processes on a system.
 * `sys`: Is used to check system specific configurations and start new processes.
+
 If your function does not fall under a specific module reach out to the core developers about adding a new module or finding the right fit.
 
 Specify the input and output according to the [Starlark types spec.](https://docs.rs/starlark/0.6.0/starlark/values/index.html)
 If there are OS or edge case specific behaviors make sure to document them here. If there are limitations Eg. if a function doesn't use file streaming specify that it can't be used for large files.
+Please add your function in alphabetical order this makes it easy to search by key words.
 ```markdown
 ### module.function
 module.function(arg1: str, arg2: int, arg3: list) -> bool
 
 The <b>module.function</b> describe your function and edge cases.
-
 ```
+
 #### Eldritch definition
 `implants/lib/eldritch/src/module.rs`
 Add a function definition here, where `module.rs` is the name of the module you selected above. This is how the Eldritch language is made aware that your function exists.
 
 Add the import for your functions implementation at the top, try to keep these in alphabetical order for readability.
 Then add the function definition under the methods function
-```RUST
+```rust
 ...
 mod function_impl;
 ...
 #[starlark_module]
 fn methods(builder: &mut MethodsBuilder) {
 ...
-    fn function(_this: ModuleLibrary, arg1: String, arg2: u8, arg3: Vec<String>) -> bool {
+    fn function(this: ModuleLibrary, arg1: String, arg2: u8, arg3: Vec<String>) -> anyhow::Result<String> {
+        if false { println!("Ignore unused this var. _this isn't allowed by starlark. {:?}", this); }
         function_impl::function(arg1, arg2, arg3)
     }
 ```
 
 You may notice that some functions follow the pattern:
-```RUST
-    fn function(_this: ModuleLibrary, arg1: String, arg2: u8, arg3: Vec<String>) -> NoneType {
+```rust
+    fn function(this: ModuleLibrary, arg1: String, arg2: u8, arg3: Vec<String>) -> anyhow::Result<NoneType> {
+        if false { println!("Ignore unused this var. _this isn't allowed by starlark. {:?}", this); }
         function_impl::function(arg1, arg2, arg3)?;
         Ok(NoneType{})
     }
@@ -81,7 +85,7 @@ Add your function implementation here, where `/module/` is the name of the modul
 This file will contain the actual implementation, helper functions, and unit tests for your function.
 
 Here's a template of how your code should look:
-```RUST
+```rust
 use anyhow::Result;
 
 fn helper(argz: String) -> bool {
@@ -124,6 +128,31 @@ mod tests {
 }
 ```
 
+### `eldritch/lib.rs` tests
+
+Lastly you'll need to add your function to the `eldritch/lib.rs` integration test.
+
+```rust
+    #[test]
+    fn test_library_bindings() {
+        let globals = get_eldritch().unwrap();
+        let mut a = Assert::new();
+        a.globals(globals);
+        a.all_true(
+            r#"
+dir(file) == ["append", "compress", "copy", "download", "exists", "hash", "is_dir", "is_file", "mkdir", "read", "remove", "rename", "replace", "replace_all", "template", "timestomp", "write"]
+dir(process) == ["kill", "list", "name"]
+dir(sys) == ["dll_inject", "exec", "is_linux", "is_macos", "is_windows", "shell"]
+dir(pivot) == ["arp_scan", "bind_proxy", "ncat", "port_forward", "port_scan", "smb_exec", "ssh_exec", "ssh_password_spray"]
+dir(assets) == ["copy","list"]
+"#,
+        );
+    }
+```
+
+Add your function in alphabetical order to the list of the module it belongs to.
+This test is done to ensure that all functions are available to the interpreter.
+
 **Implementation tips:**
 * If working with files & network connections use streaming to avoid issues with large files.
 * If your function depends on resources outside of eldritch (Eg. files, network, etc.) implement helper function that allow the user to proactively test for errors. If your function requires a specific type of file to work consider implementing a function like `is_file` or `is_lnk`.
@@ -148,13 +177,13 @@ Any methods added to the Eldritch Standard Library should have tests collocated
 * Chunk out implementation code into discrete helper functions so each can be tested individually.
 
 ### Example PR for an Eldritch method.
-Check out [this basic example of a PR](https://github.com/KCarretto/realm/pull/69/files) to see what they should look like.
-This PR implements the `file.is_file` function into Eldritch and is a simple example of how to get started.
+Check out [this basic example of a PR](https://github.com/KCarretto/realm/pull/231) to see what they should look like.
+This PR implements the `sys.hostname` function into Eldritch and is a simple example of how to get started.
 
 # OS Specific functions
 ---
+Limit changes to the implementation file.
 
-### Limit changes to the implementation file
 OS specific restrictions should be done in the **Eldritch Implementation** you should only have to worry about it in your: `function_impl.rs`.
 This ensures that all functions are exposed in every version of the Eldritch language.
 To prevent errors and compiler warnings use the `#[cfg(target_os = "windows")]` conditional compiler flag to supress OS specific code.
@@ -165,6 +194,71 @@ For all non supported OSes return an error with a message explaining which OSes
     return Err(anyhow::anyhow!("This OS isn't supported by the dll_inject function.\nOnly windows systems are supported"));
 ```
 
+# Notes about using dictionary type `Dict`
+---
+The `Dict` type requires dynamic memory allocation in starlark. In order to achieve this we can leverage the `starlark::Heap` and push entries onto it. It's pretty simple to implement and starlark does some magic to streamline the process. To make the heap available to your function simply add it as an argument to your function.
+
+## Different function declerations
+`implants/lib/eldritch/src/module.rs`
+
+```rust
+    fn function<'v>(this: SysLibrary, starlark_heap: &'v Heap, arg1: String, arg2: u8, arg3: Vec<String>) -> anyhow::Result<Dict<'v>> {
+```
+
+`implants/lib/eldritch/src/module/function_impl.rs`
+```rust
+pub fn function(starlark_heap: &Heap) -> Result<Dict> {
+```
+
+## Split starlark boilerplate and function implementation
+One note is when working with starlark `Dict` types it preferedd that a `handle_` function be implemented which returns a real data type and that data type is translated from the rust data type to starlark `Dict` in the `function` for example:
+
+```rust
+struct OsInfo {
+    arch:           String,
+}
+
+fn handle_get_os() -> Result<OsInfo> {
+    return Ok(OsInfo { 
+        arch:           whoami::arch().to_string(),
+    });
+}
+
+pub fn get_os(starlark_heap: &Heap) -> Result<Dict> {
+
+    let cmd_res = handle_get_os()?;
+
+    let res = SmallMap::new();
+    let mut dict_res = Dict::new(res);
+    let arch_value = starlark_heap.alloc_str(&cmd_res.arch);
+    dict_res.insert_hashed(const_frozen_string!("arch").to_value().get_hashed().unwrap(), arch_value.to_value());
+    Ok(dict_res)
+}
+```
+
+Splitting the code to handle inserting data into the `Dict` helps keep the code organized and also allows others looking to eldritch as an example of how things can be implemented to more clearly delineate where the technique stops and the eldritch boilerplate begins.
+
+## Testing
+When testing you can pass a clean heap from your test function into your new function.
+```rust
+...
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_function() -> anyhow::Result<()>{
+        let test_heap = Heap::new();
+        let res = function(&test_heap)?;
+        assert!(res.contains("success"));
+    }
+}
+```
+
+## Example PR
+Example of how to return a dictionary:
+PR #[238](https://github.com/KCarretto/realm/pull/238/files) This PR implements the `sys.get_os` function which returns a dictionary of string types.
+
 # Notes about asynchronous Eldritch code
 ---
 ### Async example
@@ -179,7 +273,7 @@ We'll create the following three functions to manage concurrent tasks:
 * `async fn run_function` - Async function runner that gets spawned by the `handle_function` function.
 
 An example of how you might run multiple concurrent tasks asynchronously.
-```RUST
+```rust
 // Async handler for Eldritch function
 async fn run_function(argstr: String) -> Result<String> {
     // Do async stuff
@@ -237,7 +331,7 @@ You'll need to write tests for your synchronous and asynchronous code.
 Async tests will start 
 
 Tests for async functions may look like this:
-```RUST
+```rust
 // Command implementation code.
 // ....