Snowflake component for Edgee

This component enables seamless integration between Edgee and Snowflake, allowing you to collect and forward analytics events to your Snowflake data warehouse.

Quick Start

1. Download the latest component version from our releases page
2. Place the snowflake.wasm file in your server (e.g., /var/edgee/components)
3. Add the following configuration to your edgee.toml:

[[components.data_collection]]
id = "snowflake"
file = "/var/edgee/components/snowflake.wasm"
settings.account_identifier = "xy12345.us-east-1"
settings.access_token = "*******"
settings.warehouse = "COMPUTE_WH"
settings.database = "ANALYTICS"
settings.table = "edgee_events"

Event Handling

First, create a new table with the following schema in your Snowflake database:

CREATE TABLE edgee_events (
    uuid VARCHAR(36) NOT NULL,
    event_type VARCHAR(20) NOT NULL,
    timestamp TIMESTAMP_NTZ NOT NULL,
    consent VARCHAR(20) NOT NULL,
    context OBJECT NOT NULL,
    data OBJECT NOT NULL
);

Schema Design

This approach combines scalar columns with semi-structured data types:

Scalar Columns (uuid, event_type, timestamp, consent):

• Fast Filtering: Direct indexing and filtering on common query fields
• Efficient Sorting: Timestamp-based ordering without JSON parsing
• Better Compression: Native data type storage
• Timestamp Storage: TIMESTAMP_NTZ (No Time Zone) preserves exact "wall clock" time

OBJECT Columns (context, data):

• Structured Storage: Optimized for key-value pairs with consistent schemas
• JSON Querying: Full support for JSON path expressions like context:client.ip

Querying Event Data

Once events are stored, you can query them using Snowflake's JSON functions:

-- Get all page view events
SELECT
    uuid as event_id,
    timestamp,
    context:page.url::STRING as page_url,
    context:client.ip::STRING as client_ip,
    context:user.user_id::STRING as user_id
FROM edgee_events
WHERE event_type = 'Page'
ORDER BY timestamp DESC;

-- Get track events with specific names
SELECT
    uuid as event_id,
    timestamp,
    data:Track.name::STRING as event_name,
    data:Track.properties as properties,
    context:user.user_id::STRING as user_id
FROM edgee_events
WHERE event_type = 'Track'
AND data:Track.name::STRING = 'purchase'
ORDER BY timestamp DESC;

Event Mapping

The component maps Edgee events to Snowflake records as follows:

Edgee Event	Snowflake Record	Description
Page	Scalar columns + OBJECT context/data	Page event with data:Page.* in data OBJECT
Track	Scalar columns + OBJECT context/data	Track event with data:Track.* in data OBJECT
User	Scalar columns + OBJECT context/data	User event with data:User.* in data OBJECT

Column Mapping:

• uuid: Event UUID (VARCHAR)
• event_type: "Page", "Track", or "User" (VARCHAR)
• timestamp: Event timestamp (TIMESTAMP_NTZ)
• consent: Consent status (VARCHAR)
• context: User, client, session, campaign data (OBJECT)
• data: Event-specific payload (OBJECT)

Configuration Options

Required Configuration

[[components.data_collection]]
id = "snowflake"
file = "/var/edgee/components/snowflake.wasm"
settings.account_identifier = "xy12345.us-east-1"  # Your Snowflake account identifier
settings.access_token = "*******"                   # Any Snowflake auth token (OAuth/JWT/PAT)
settings.warehouse = "COMPUTE_WH"                   # Compute warehouse name
settings.database = "ANALYTICS"                     # Target database name
settings.table = "edgee_events"                     # Target table name

Optional Configuration

settings.schema = "PUBLIC"     # Schema name (defaults to "PUBLIC")
settings.role = "ANALYST"      # Snowflake role (default to "")
settings.auth_type = "Auto"    # Authentication method (defaults to "Auto")

Authentication Setup

The component supports all Snowflake authentication methods through the access_token setting. You can optionally specify the authentication method using the auth_type setting for explicit control:

Auto Detection (Default)

settings.access_token = "your_token_here"
settings.auth_type = "Auto"  # Let Snowflake detect token type (default)

• Use Case: Simplest setup - Snowflake automatically detects the token type
• Benefit: Works with any authentication method without additional configuration

Programmatic Access Token (PAT)

settings.access_token = "sfp_5FZvZ4M1..."  # Generated PAT token
settings.auth_type = "PAT"  # Explicitly specify PAT

• Use Case: Simple setup for production environments
• How to Generate: Follow Snowflake's PAT documentation

OAuth Token

settings.access_token = "ETMsjXXXXXXX..."  # OAuth access token
settings.auth_type = "OAuth"  # Explicitly specify OAuth

• Use Case: Integration with existing OAuth flows
• How to Generate: Use OAuth 2.0 flow with Snowflake authorization server

Key-Pair Authentication (JWT)

settings.access_token = "eyJhbGciOiJSUzI1NiJ5..."  # JWT token
settings.auth_type = "JWT"  # Explicitly specify JWT

• Use Case: Enhanced security with private key authentication
• How to Generate: Create JWT using your private key following Snowflake's key-pair docs

Required Permissions

Ensure your authentication method has permissions to:

• Use the specified warehouse
• Access the target database and schema
• Insert data into the target table

Integration Details

The component uses Snowflake's SQL API (/api/v2/statements/) for data ingestion:

• Data Format: Events stored with separate columns for scalars and OBJECT types for semi-structured data
• SQL Pattern: Parameterized INSERT INTO ... SELECT ... FROM VALUES statements with PARSE_JSON() for context and data fields

Development

Prerequisites

• Rust
• Edgee CLI

Setup

# Fetch WIT interface definitions (required before first build)
edgee components wit

Building from Source

# Build the component
make build

# Or use Edgee CLI directly
edgee components build

# Development build
make dev

# Clean build artifacts
make clean

Testing

# Run unit tests
make test

Contributing

Development Workflow

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

Code Standards

• Follow Rust naming conventions
• Add documentation for public functions
• Include unit tests for new features
• Use cargo fmt for consistent formatting

Interested in contributing? Read our contribution guidelines

Security

Report security vulnerabilities to [email protected]

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.