The open source high performance ELT framework powered by Apache Arrow
-
Updated
Jan 2, 2025 - Go
Google BigQuery enables companies to handle large amounts of data without having to manage infrastructure. Google’s documentation describes it as a « serverless architecture (that) lets you use SQL queries to answer your organization's biggest questions with zero infrastructure management. BigQuery's scalable, distributed analysis engine lets you query terabytes in seconds and petabytes in minutes. » Its client libraries allow the use of widely known languages such as Python, Java, JavaScript, and Go. Federated queries are also supported, making it flexible to read data from external sources.
📖 A highly rated canonical book on it is « Google BigQuery: The Definitive Guide », a comprehensive reference.
Another enriching read on the subject is the inside story told in the article by the founding product manager of BigQuery celebrating its 10th anniversary.
The open source high performance ELT framework powered by Apache Arrow
Privacy and Security focused Segment-alternative, in Golang and React
tbls is a CI-Friendly tool for document a database, written in Go.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
Scratch is a swiss army knife for big data.
BigQuery emulator server implemented in Go
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift, Databricks) in real-time.
Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
Seamlessly save and load protocol buffers to and from BigQuery using Go.
An exporter for converting BigQuery results into Prometheus metrics
RTLSDR ADS-B dump1090 to Google BigQuery
The simplest tool to manage views of BigQuery.
A serverless bot which periodically checks configured BigQuery capacity commitments, reservations and assignments against actual slot consumption of running jobs and reports findings to Slack/Google Chat.
Appengine Datastore Mapper in Go
bqloader is a simple ETL framework to load data from Cloud Storage into BigQuery.
Released May 19, 2010