Scoutr Go

A simple way to put an API in front of a DynamoDB, Firestore, or Azure CosmosDB (MongoDB) backend.

This is based off of the Python implementation of the scoutr.

Sample implementation

An sample implementation of this project is provided in the examples folder.

Requirements

At minimum, two tables are required for this to work: an auth table and a groups table. Additionally, an optional audit log table can be used to track all API calls and changes to records in the data table. The configuration of each table is detailed next.

Auth Table

The auth table must have a primary key of id. The table name does not matter, as this is passed in during instantiation.

Groups Table

The groups table must have a primary key of group_id. The table name does not matter, as this is passed in during instantiation.

Audit Log Table

The audit log table must have a primary key of time. For DynamoDB, it should also have a TTL attribute of expire_time configured. The table name does not matter, as this is passed in during instantiation. If a value is not specified, it is assumed that no audit logs should be kept.

Access Control

Scoutr provides full access control over the endpoints a set of users is permitted to call and the output that is returned. This is done using field filters, field exclusions, and permitted endpoints, which are outlined in the next section.

This access control functionality is implemented at both a user and a group level. A user can be a member of zero or more groups. The implementation of auth identifiers and groups is outlined in their respective sections.

The two types of access control supported are via API Gateway or via OIDC. Helper functions have been created for each access control type to assist with passing the correct request format into Scoutr.

API Gateway Authentication

For API Gateway authentication, the request format is generated by the InitAPIGateway function.

Example

Refer to the example serverless endpoint

OIDC Authentication

It is assumed that there is an Apache server running in front of the application that performs OIDC authentication and passes the OIDC claims as headers.

The simplest method to setup the API is to use net/http. Helper functions have been provided to make the setup as simple as possible. The InitHTTPServer function automatically generates the belows endpoints:

GET /user/ - Returns information about the authenticated user
POST /user/has-permission/ - Determine if user has permission to access an endpoint. The body of this request should contain method and path keys as JSON.
GET /<primary_list_endpoint>/ - Primary endpoint used to list data. The value of primary_list_endpoint is determined by an argument passed to InitHTTPServer()
GET /audit/ - List and search all audit logs
GET /audit/<item>/ - List audit logs for a particular resource
GET /history/<item>/ - Show history for a particular resource
POST /search/<search_key>/ - Search endpoint that allows searching by any key for one or more values. The body of this request should be a JSON list of values.

Example

Refer to the example net/http applications

Concepts

Field filters

List of field filters to apply to queries by this group. Each item in this list must be structured as:

If the type of value is a string, it will be filtered using a field = value operation. To support multiple values for a single field, tf the type of value is a list, it will be filtered using a field IN ['value1', 'value2', ..., 'valueN'] operation. When multiple field filters are specified, they are combined together using an AND operation.

Syntax

[
    {"field": "field1", "value": "filter_value"},
    {"field": "field2", "value": ["value1", "value2"]},
]

Field exclusions

Field exclusions allow for excluding one or more fields from the output of all queries. These fields are from any output during the post-processing phase of all queries. Additionally, if a user attempts to create or update an item that contains a field from this list, the operation will be denied.

Syntax

[
    "field1",
    "field2"
]

Permitted endpoints

Before taking any action, every call from API gateway is validated to ensure the user has permissions to perform the call. For convenience, regular expressions can be used within the endpoint field.

Syntax

[
    {"method": "GET|POST|PUT|DELETE", "endpoint": "/endpoint"},
    {"method": "GET|POST|PUT|DELETE", "endpoint": "^/endpoint2/.+$"}
]

Groups

A group object be made up of:

group_id - Identifier for the group
permitted_endpoints - Optional list of permitted endpoints
filter_fields - Optional list of field filters
exclude_fields - Optional list of field exclusions
update_fields_permitted - Optional list of the only fields that can be updated
update_fields_restricted - Optional list of fields to restrict updates for

The name of the group table must be passed in to the Config struct.

Example

{
    "group_id": "read-only",
    "permitted_endpoints": [
        {
            "endpoint": "^/item/.+$",
            "method": "GET"
        },
        {
            "endpoint": "^/items.*$",
            "method": "GET"
        },
        {
            "endpoint": "^/search/.+$",
            "method": "POST"
        }
    ],
    "exclude_fields": [
        "supersecret"
    ],
    "update_fields_permitted": [
        "comments"
    ],
    "update_fields_restricted": [
        "type"
    ],
    "filter_fields": [
        {
            "field": "provider",
            "value": "Provider A"
        },
        {
            "field": "product",
            "value": [
                "Product A",
                "Product B"
            ]
        }
    ]
}

Auth Identifier

Types

There are three types of accepted authentication identifiers:

USERNAME
OIDC_GROUP
API_KEY

Though not required, it is recommended for each object type to have a type key that corresponds to its authentication type (OIDC_GROUP, USERNAME, or API_KEY).

The field requirements for each object type are outlined in the following sections

USERNAME

id (primary key) - this is the user's username (i.e. johndoe)

Though not required, it is recommended to also include a name field containing the user's full name to make it easier to identify the user at a glance.

OIDC_GROUP

id (primary key) - this is expected to be the group id (i.e. group123) from the OIDC header

Though not required, it is recommended to also include a name field containing the group's display name to make it easier to identify the group at a glance.

If a user is a member of more than one OIDC group, the permissions granted by each configured group will be combined together to generate the effective permissions applied to the user.

API_KEY

id (primary key) - this is the api key id
name
username
email

Groups

Optionally, each auth object can include a groups object, which should be a list of group ids that the user is a member of:

{
    "groups": [
        "read-only",
        "product-a-only"
    ]
}

Any permissions defined in the groups are combined together to make up the user's permissions. In addition, the same permissions that a group defines (filter_fields, exclude_fields, update_fields_permitted, update_fields_restricted, permitted_endpoints) can be expressed at the user level. These permissions will be combined together with the permissions outlined in the groups the user is a member of. Permissions defined at the user level DO NOT override those specified at the group level - they are combined.

The name of the user table must be passed in to the constructor.

Audit Logs

For every authorized, successful call to the API, an entry will be logged in the audit log table. Each record will follow the below format:

{
  "action": "CREATE|UPDATE|DELETE|GET|LIST|SEARCH|{CUSTOM-ACTION}",
  "body": {
    "key": "value"
  },
  "method": "HTTP method from API gateway",
  "path": "/endpoint/path",
  "path_params": {
    "key": "value"
  },
  "query_params": {
    "key": "value"
  },
  "resource": {
    "key": "value"
  },
  "time": "2019-10-04T18:44:30.166635",
  "user": {
    "api_key_id": "ID",
    "name": "John Doe",
    "source_ip": "1.2.3.4",
    "username": "johndoe",
    "user_agent": "curl"
  }
}

The following fields may not be included or may not have values for all types of actions:

body
query_params
path_params
resource

Endpoint Structure

The helper methods within Scoutr assume that your API consists of the following endpoint types:

List all records
List all unique values for a key
Search multiple values for a single search key
Get single item by key
Update single item by key
Delete single item by key
List all audit logs
View item history

List

The list all items endpoint will return a list of all items within the backend that the user has permission to see and that meet any specified filter criteria.

List by Unique Key

The list by unique key endpoint provides a means to display all unique values for a single search key. It is implemented by specifying a value for the uniqueKey argument of the ListUniqueValues() function. This is only supported in DynamoAPI currently.

Serverless Example

# Unique listing of all values of the `status` key that the user is permitted to see
list-statuses:
  handler: listUnique
  events:
    - http:
        path: statuses
        method: get
        private: true
  environment:
    UniqueKey: status

Implementation Example

func handler(event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
	// Convert log retention to int
	logRetention, err := strconv.Atoi(os.Getenv("LogRetentionDays"))
	if err != nil {
		panic(err)
	}

	// Build config
	config := config.Config{
		DataTable:        os.Getenv("DataTable"),
		AuthTable:        os.Getenv("AuthTable"),
		AuditTable:       os.Getenv("AuditTable"),
		GroupTable:       os.Getenv("GroupTable"),
		LogRetentionDays: logRetention,
	}

	// Initialize api gateway
	api, request := helpers.InitAPIGateway(event, config)

	// List the data
	data, err := api.List(request)

	// Handle any errors
	if errorResponse := helpers.APIGatewayErrorHandler(err); errorResponse != nil {
		return *errorResponse, nil
	}

	// Send response
	return helpers.ProcessAPIGatewayResponse(data)
}

Search

Lookup information about multiple items (POST /search/{search_key})

[
    "record-a",
    "record-b"
]

Get

Retrieve a single record from the backend. The Get() function accepts two arguments:

req - the Request object containing information about the request
id - the id of the item to lookup

If this returns more than one record, it will throw a BadRequest error. If no records are returned, a NotFound error will be thrown.

Create

The Create() function accepts the req argument, with req being the Request object, an item argument, with item being a map[string]string of the data to be inserted, and a validation argument in order to perform validation on all the supplied data. Refer to the data validation section for more information.

Update

The Update() function accepts a couple of arguments:

req Request object

partitionKey Mapping of the partition key to value. For instance, if the table's partition key is id, it is expected this mapping would be:

map[string]string{
    "id": "value"
}

item map[string]string of fields to be updated

validation map[string]utils.FieldValidation of fields to perform validation against. Refer to the data validation section for more information.

**auditAction** A string value to use as the Action in the audit logs. This should be set to UPDATE in most cases.

Delete

The Delete() function accepts a couple of arguments:

req Request object

partitionKey Mapping of the partition key to value. For instance, if the table's partition key is id, it is expected this mapping would be:

map[string]string{
    "id": "value"
}

List audit logs

The ListAuditLogs() function accepts:

req Request object

pathParams Any search parameters to apply

queryParams Query parameters from API Gateway

History

req Request object

key Resource key to search on

value Resource value to search on

queryParams Query parameters from API Gateway

actions List of actions to filter on

Filtering

There are two levels of filtering that are supported:

Path-based filtering
Querystring-based filtering

The List() function accepts a single req argument as a Request object. For filtering to be applied, its PathParams and QueryParams fields should contain values. These are intended to contain the values of PathParameters and QueryStringParameters, respectively, that API Gateway passed into Lambda. In the case of net/http, these values should be set using the request.URL.Query() function and any path parameters that are set by the httprouter package. Refer to the net/http example and the InitHTTPServer function to see usage.

Dynamic path filters

The List() function also supports dynamic path filtering. When search_key and search_value are passed into the method as PathParams, it will dynamically modify the path parameters to construct a search filter where

search_key = search_value

To configure this in API Gateway, setup path parameters on the resource:

/endpoint/{search_key}/{search_value}

Or when using serverless:

events:
  - http:
      path: endpoint
      method: get
      private: true
  - http:
      path: endpoint/{search_key}/{search_value}
      method: get
      private: true

When using the dynamic path filters, there is no need to construct additional endpoints that support filtering by a specific key. However, using this method provides no limitations over what fields can be used as a filter. If that is a concern for your API, you will need to construct static path filters.

Static path filters

Static path filters can be constructed in a similar manner to the dynamic path filters, except that the search key is manually specified:

/endpoint/status/{status}

In order to properly work, the path variable must exactly match the key in the backend table that you want to perform the filter against.

Querystring Filters

In addition to path filters, querystring filtering is also supported. The List() endpoint accepts all querystrings via the QueryParams field of the request object. Each querystring should be a field_name=search_value format:

/endpoint?status=Active&type=ABC

Path parameters always take precedence over querystring parameters. The below query:

/endpoint/type/ABC?status=Active&type=Azure

Would result in this filter criteria:

type = ABC AND status = Active

Magic Operators

For more complex queries, querystring search supports the below magic operations:

in (value is in list)
notin (value is not in list)
ne (not equal)
startswith (string starts with)
contains (string contains)
notcontains (string does not contain)
exists (attribute exists)
gt (greater than)
lt (less than)
ge (greater than or equal)
le (less than or equal)
between (value is between)

Note that DynamoDBAPI does not support the in operation and FirestoreAPI only accepts the following magic operations:

in
gt
lt
ge
le
between

The MongoDBAPI supports these operations:

in
notin
ne
startswith
contains
exists
gt
ge
lt
le
between

To use a magic operator, append __operator to the key name. For example:

To search for all items with the product key containing the word "Product"

/items?product__contains=Product

To search for all items with the product key starting with the word "Product"

/items?sku__startswith=Product

Usage of all the magic operators is straightforward, with the exception of the in and between operators. The in operators checks to see if the the value is included in a list of options. It should follow the JSON list syntax:

/items?product__in=["Product A", "Product B"]

The between operator checks to see if the value is, inclusively, between a low and high value. It should also follow a JSON list syntax:

/items?num__between=[0, 3]

It also works for string values, such as two dates:

/items?date__between=["2019-01-01", "2019-12-31"]

To find items that have an attribute:

/items?name__exists=true

To search for items that do not have an attribute:

/items?name__exists=false

Data validation

For convenience, support for data validation on all create and update calls is supported. In order to implement the validation, a map[string]utils.FieldValidation should be passed to the validation map of the Create() or Update() functions. The syntax of this object is outlined below.

On Create() calls, all items specified in the validation map are assumed to be required fields. If a field is missing from the user input, an error will be thrown saying that the field is required.

Syntax

validation := map[string]utils.FieldValidation{
    "field1": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {
        if value != "hello" {
            return false, fmt.Sprintf("Invalid value '%s' for attribute 'field1'", value), nil
        }

        return true, "", nil
    },
    "field2": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {
        if value != "world" {
            return false, fmt.Sprintf("Invalid value '%s' for attribute 'field2'", value), nil
        }

        return true, "", nil
    },
}

The key of each item in the dictionary should match a field name that you want to perform validation against. The corresponding value for the key should be a callable that returns a boolean, string, and error. The boolean should be true if the field validated successfully, or false if it did not. The string should contain the error message that should be displayed to the user. The error should be nil if there were not any errors while running validation. If an error was encountered, this error value will be returned to the user.

The callable that you provide must accept three arguments:

value - Contains the input value for this field
item - Contains the entire data object that was passed from the user
existingItem - Contains the existing data object. This will only have a value on update calls. For create calls, this will be None.

Example

func validateUser(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {
    var itemType string
    if existingItem != nil {
        itemType = existingItem["type"]
    } else {
        itemType = item["type"]
    }

    if _, ok := item["type"]; !ok {
        return false, "Type field is required", nil
    }

    if itemType == "Type1" {
        re := regexp.MustCompile("^\d{10}$")
        if re.MatchString(value) {
            return true, "", nil
        } else {
            return false, "Value does not match pattern", nil
        }
    } else if itemType == "Type2" {
        re := regexp.MustCompile("^[a-z]+$")
        if re.MatchString(value) {
            return true, "", nil
        } else {
            return false, "Value does not match pattern", nil
        }
    } else {
        return false, "Validation failed", nil
    }
}

fieldValidation := map[string]utils.FieldValidation{
    "user": validateUser,
    "type": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {
        validOptions := []string{"ABC", "DEF"}

        found := false
        for _, item := range validOptions {
            if item == value {
                found = true
                break
            }
        }

        if !found {
            return false, fmt.Sprintf("Invalid value. Supported options are %s", validOptions), nil
        }
    }
}

Sentry support

Coming soon

Name		Name	Last commit message	Last commit date
Latest commit History 215 Commits
.github		.github
.vscode		.vscode
docs		docs
examples		examples
pkg		pkg
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
go.mod		go.mod
go.sum		go.sum

License

MichaelPalmer1/scoutr-go

Folders and files

Latest commit

History

Repository files navigation

Scoutr Go

Sample implementation

Requirements

Auth Table

Groups Table

Audit Log Table

Access Control

API Gateway Authentication

Example

OIDC Authentication

Example

Concepts

Field filters

Syntax

Field exclusions

Syntax

Permitted endpoints

Syntax

Groups

Example

Auth Identifier

Types

USERNAME

OIDC_GROUP

API_KEY

Groups

Audit Logs

Endpoint Structure

List

List by Unique Key

Serverless Example

Implementation Example

Search

Get

Create

Update

Delete

List audit logs

History

Filtering

Dynamic path filters

Static path filters

Querystring Filters

Magic Operators

Data validation

Syntax

Example

Sentry support

About

Topics

Resources

License

Security policy

Stars

Watchers

Forks

Releases 3

Contributors 2

Languages