Skip to content

Latest commit

 

History

History
448 lines (331 loc) · 12 KB

VALIDATOR_GUIDE.md

File metadata and controls

448 lines (331 loc) · 12 KB

DRAFT - Lumera Protocol Validator Operations Manual

Introduction

This manual provides detailed instructions for operating a validator node on the Lumera Protocol. It covers hardware requirements, software setup, security practices, common operations, and troubleshooting guidelines.

1. Hardware and Software Requirements

Validator

1.1 Minimum Hardware Requirements

  • CPU: 8 cores, x86_64 architecture
  • RAM: 32 GB RAM
  • Storage: 2 TB NVMe SSD
  • Network: 1 Gbps dedicated line
  • Operating System: Ubuntu 22.04 LTS or higher

1.2 Recommended Hardware Requirements

  • CPU: 16 cores, x86_64 architecture
  • RAM: 64 GB RAM
  • Storage: 4 TB NVMe SSD
  • Network: 5 Gbps dedicated line
  • Operating System: Ubuntu 22.04 LTS or higher

Supernode

1.1 Minimum Hardware Requirements

  • CPU: 8 cores, x86_64 architecture
  • RAM: 16 GB RAM
  • Storage: 1 TB NVMe SSD
  • Network: 1 Gbps
  • Operating System: Ubuntu 22.04 LTS or higher

1.2 Recommended Hardware Requirements

  • CPU: 16 cores, x86_64 architecture
  • RAM: 64 GB RAM
  • Storage: 4 TB NVMe SSD
  • Network: 5 Gbps
  • Operating System: Ubuntu 22.04 LTS or higher

1.3 Notes

  • ARM-based processors (like Apple M1) are not recommended for production
  • Hard drives (HDD) are not suitable due to I/O requirements
  • SATA SSDs may underperform during high load

1.4 Software

  • lumerad binary: Downloaded from official releases or compiled from source.
  • Go 1.21 or later.
  • Git.
  • jq.
  • curl or wget.
  • gpg

2. Setting Up the Validator Node

2.1. Base System Setup

  1. System Updates
    sudo apt update && sudo apt upgrade -y
    sudo apt install build-essential jq curl git wget snap unzip gpg -y
  1. Security Basics
# SSH Configuration
sudo vim /etc/ssh/sshd_config
# Set:
# PermitRootLogin no
# PasswordAuthentication no
# MaxAuthTries 3

# Firewall Setup
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 26656/tcp
sudo ufw enable
  1. Go Installation
wget https://go.dev/dl/go1.21.5.linux-amd64.tar.gz
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf go1.21.5.linux-amd64.tar.gz

echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.profile
echo "export PATH=$PATH:$(go env GOPATH)/bin" >> ~/.profile
source ~/.profile

2.2. lumerad Binary Installation

  • Download the appropriate binary from the official releases page on GitHub.
  • Verify the checksum of the downloaded binary.
  • Make the binary executable: chmod +x lumerad.
  • Move the binary to an executable path (e.g., /usr/local/bin).

2.3. Node Initialization and Configuration

  1. Initialize the Node:
lumerad init <moniker> --chain-id <chain-id>
- Replace `<moniker>` with your validator's moniker.
- Replace `<chain-id>` with the chain ID of your network.
  1. Copy Genesis File:
    • Copy the downloaded genesis.json file to $HOME/.lumerad/config directory.
  2. Configure Node:
    • Adjust parameters in $HOME/.lumerad/config/config.toml:
      • Update persistent_peers with the list of initial peers provided by the core team.
    • Adjust parameters in $HOME/.lumerad/config/app.toml:
      • Example: set min-gas-prices.
  3. Start the Node:
lumerad start

3. Security Best Practices

3.1. Key Management

  • Securely store node_key.json and priv_validator_key.json.
  • Use hardware wallets or encrypted storage for private keys.
  • Never expose your private keys.
  • Use a strong password for keyring.
  • Always back up your keys offline

3.2. Sentry Node Architecture

  • Implement a sentry node architecture to mitigate DDoS attacks.
  • Validators should only connect to trusted full nodes.
  • Sentry nodes shield validator nodes from direct internet exposure.
# In sentry config.toml
pex = true
persistent_peers = "[validator-node-id]@[validator-private-ip]:26656"
private_peer_ids = "[validator-node-id]"

# In validator config.toml
pex = false
persistent_peers = "[sentry1-node-id]@[sentry1-private-ip]:26656,[sentry2-node-id]@[sentry2-private-ip]:26656"
private_peer_ids = ""

3.3. Firewall Configuration

  • Configure the firewall to allow only necessary traffic.
  • Restrict access to ports used by lumerad (e.g., port 26656).
  • Use SSH key authentication.
  • Disable root login

3.4. Regular Updates

  • Keep the OS and software up-to-date with the latest patches.
  • Regularly update the lumerad binary.

3.5. Monitoring

  • Set up tools to monitor node performance and health.
  • Set up alerting to be notified in case of issues

3.6. Access Control

  • Limit access to authorized personnel.
  • Use strong passwords and/or SSH keys.

3.7. Avoiding Publicly Accessible Nodes

  • Avoid running validators directly on the public internet; use private networks and secure tunnels.

4. Validator Operations

4.1. Basic Commands

  • Check node status: lumerad status
  • Display validator node ID: lumerad tendermint show-node-id
  • Get validator information: lumerad query staking validator <your_validator_address>

4.2. Staking and Bonding

  • Delegate tokens to validator: lumerad tx staking delegate <validator_address> <amount> --from <key_name> --chain-id <chain_id> --gas-prices <gas_price>
  • Unbond tokens from validator: lumerad tx staking unbond <validator_address> <amount> --from <key_name> --chain-id <chain_id> --gas-prices <gas_price>
  • Redelegate tokens from one validator to another: lumerad tx staking redelegate <source_validator_address> <destination_validator_address> <amount> --from <key_name> --chain-id <chain_id> --gas-prices <gas_price>

4.3. Governance

Actively participate in network governance.

  • List all governance proposals: lumerad query gov proposals
  • Query specific proposal: lumerad query gov proposal <proposal_id>
  • Check voting status on proposal: lumerad query gov votes <proposal_id>
  • Vote on proposal: lumerad tx gov vote <proposal_id> <option> --from <key_name> --chain-id <chain_id> --gas-prices <gas_price>

4.4. Distribution

Manage rewards and commissions

  • Withdraw rewards: lumerad tx distribution withdraw-rewards <validator_address> --from <key_name> --chain-id <chain_id> --gas-prices <gas_price>
  • Withdraw all rewards and commissions: lumerad tx distribution withdraw-all-rewards --from <key_name> --chain-id <chain_id> --gas-prices <gas_price>

4.5 Slashing

  • Unjail validator if slashed: lumerad tx slashing unjail --from <key_name> --chain-id <chain_id> --gas-prices <gas_price>

4.6 Validator Management

  • Edit validator data: lumerad tx staking edit-validator --new-moniker <new_name> --website <new_website> --identity <keybase_id> --details <new_details> --chain-id <chain_id> --from <key_name> --gas-prices <gas_price>
  • Show validator public key: `lumerad tendermint show-validator

4.7 Troubleshooting

  • Consult official documentation for common issues and solutions.
  • Contact the core team or other validators for assistance.

5. Security Recommendations

5.1. Hardware Security Module (HSM)

  • Recommended: YubiHSM 2.
  • Alternative: Ledger Nano S/X with Tendermint app.
  • Follow HSM-specific documentation for key generation and usage.

5.2. System Hardening

# Disable root login
sudo passwd -l root

# Set up automatic security updates
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

# Secure shared memory
echo "tmpfs     /run/shm     tmpfs     defaults,noexec,nosuid     0     0" >> /etc/fstab

# Set up fail2ban
sudo apt install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

5.3. Process Isolation

# Create dedicated user
sudo useradd -m -s /bin/bash validator
sudo usermod -aG sudo validator

# Set up process limits
sudo vim /etc/security/limits.conf
# Add:
# validator soft nofile 65535

6. Monitoring Setup

6.1 Node Monitoring

# Install Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

# Create service file
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=validator
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

6.2 Alerting Setup

  1. Prometheus Configuration
# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'validator'
    static_configs:
      - targets: ['localhost:26660']
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
  1. Alert Rules
# /etc/prometheus/alerts.yml
groups:
- name: validator
  rules:
  - alert: ValidatorDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
  - alert: BlocksMissed
    expr: validator_missed_blocks > 10
    for: 10m
    labels:
      severity: warning

7. Maintenance Procedures

7.1. Regular Maintenance

  1. Daily Tasks

    • Check validator status.
    • Monitor system resources.
    • Review logs for errors.
    • Verify synchronization status.

  2. Weekly Tasks

    • Update security patches.
    • Back up validator keys.
    • Check disk usage.
    • Review performance metrics.

  3. Monthly Tasks

    • Full system audit.
    • Review and update documentation.
    • Test backup restoration.
    • Check for software updates.

7.2. Emergency Procedures

  • Node Recovery
# Quick status check
lumerad status

# Reset node (if needed)
lumerad unsafe-reset-all

# Restore from backup
cp backup/priv_validator_key.json ~/.[$binary]/config/
cp backup/node_key.json ~/.[$binary]/config/

# Restart service
sudo systemctl restart [binary]
  • Double-Sign Prevention
    • Never run validator keys on multiple machines.
    • Always use recent snapshots for recovery.
    • Implement proper backup procedures.

8. Useful Commands

8.1. Node Management

# Check node status
lumerad status

# Check sync status
lumerad status 2>&1 | jq .sync_info

# Check validator status
lumerad query staking validator $(lumerad keys show validator --bech val -a)

# Check blocks signed
lumerad query slashing signing-info $(lumerad tendermint show-validator)

8.2. Common Operations

# Unjail validator
lumerad tx slashing unjail --from validator --chain-id [chain-id] --gas-prices [gas-price]

# Edit validator
lumerad tx staking edit-validator \
    --new-moniker [new-name] \
    --website [new-website] \
    --identity [keybase-id] \
    --details [new-details] \
    --chain-id [chain-id] \
    --from validator \
    --gas-prices [gas-price]

# Withdraw rewards
lumerad tx distribution withdraw-rewards $(lumerad keys show validator --bech val -a) \
    --from validator \
    --commission \
    --chain-id [chain-id] --gas-prices <gas_price>

9. Troubleshooting Guide

9.1. Common Issues

  1. Node Not Syncing
    • Check network connectivity.
    • Verify genesis checksum.
    • Check disk space.
    • Review peer connections.
  2. Missing Blocks
    • Check system resources.
    • Review network latency.
    • Verify time synchronization.
    • Check validator status.
  3. Slashing Events
    • Document the incident.
    • Check for double-signing evidence.
    • Prepare recovery plan.
    • Contact chain support.

9.2. Support Resources

  • Chain Documentation: [DOCS_URL]
  • Community Forum: [FORUM_URL]
  • Technical Support: [SUPPORT_URL]
  • Emergency Contact: [EMERGENCY_CONTACT]

This manual provides essential information for operating a validator node on the Lumera Protocol. Follow the security best practices, stay informed about updates, and engage with the community to ensure the health and stability of the network.

(C) 2024 Lumera Protocol