Skip to content

A columnar In-Memory Database as Drop-In Replacement for MySQL supporting 10x performance in OLAP workloads and similar performance in OLTP

License

Notifications You must be signed in to change notification settings

launix-de/memcp

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

MemCP - Ultra-Fast In-Memory Database ๐Ÿš€

Ready to supercharge your applications? MemCP is a blazing-fast, MySQL-compatible database that runs entirely in-memory, delivering unprecedented performance for modern web applications and APIs.

Why Switch from MySQL? ๐Ÿ’ก

โšก 10-100x Faster Than Traditional Databases

  • Zero disk I/O latency - everything runs in RAM
  • Sub-millisecond query response times
  • Ultra-fast REST APIs with built-in HTTP server
  • No connection overhead - direct in-process access

๐Ÿ”Œ Drop-in MySQL Compatibility

-- Your existing MySQL queries work immediately
CREATE TABLE users (id INT, name VARCHAR(100), email VARCHAR(255));
INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
SELECT * FROM users WHERE id = 1;

๐ŸŒ Built-in REST API Server

# Start MemCP with REST API
./memcp --api-port=4321

# Query via HTTP instantly
curl -X POST http://localhost:4321/sql/mydb \
  -d "SELECT * FROM users" \
  -H "Authorization: Basic cm9vdDphZG1pbg=="

๐Ÿ“Š Perfect for Modern Workloads

  • Microservices - Embedded database per service
  • APIs and Web Apps - Ultra-low latency responses
  • Real-time Analytics - Process data as fast as it arrives
  • Development & Testing - Instant setup, no configuration

Architecture & Languages ๐Ÿ—๏ธ

MemCP combines the best of multiple worlds with a carefully chosen tech stack:

Go (Storage Engine & Core)

  • High-performance storage engine built in Go
  • Concurrent request handling with goroutines
  • Memory-efficient data structures
  • Cross-platform compatibility

Scheme (SQL Parser & Compiler)

  • Advanced SQL parser written in Scheme
  • Query optimization and compilation
  • Extensible language for complex transformations
  • Functional programming advantages for parsing

Flexible Scripting Support

  • Command-line argument support for automation
  • Dynamic query generation and processing
  • Easy integration with existing workflows

Key Advantages ๐ŸŽฏ

๐Ÿ”ฅ Ultra-Fast REST APIs

Traditional setup: Client โ†’ HTTP Server โ†’ Database Connection โ†’ Disk I/O MemCP: Client โ†’ HTTP Server โ†’ In-Memory Data โœจ

// Response times you'll see
MySQL (with network + disk):  10-50ms
MemCP (in-memory):           0.1-1ms  // 50x faster!

โšก Docker

docker pull carli2/memcp
docker run -it -p 4321:4321 -p 3307:3307 carli2/memcp

๐Ÿง  Smart Memory Management

  • Automatic data optimization for memory usage
  • Configurable memory limits
  • Efficient garbage collection
  • Data persistence options when needed

๐Ÿ”ง Developer-Friendly

  • Comprehensive test suite with 150+ test cases
  • YAML-based testing framework
  • Extensive error handling and validation
  • Built-in performance monitoring

Quick Start ๐Ÿš€

# 1. Build MemCP
go get
make

# 2. Start with REST API
./memcp --api-port=4321 --mysql-port=3307

# 3. Create your first database
curl -X POST http://localhost:4321/sql/system \
  -d "CREATE DATABASE myapp" \
  -u root:admin

# 4. Start building lightning-fast apps!
curl -X POST http://localhost:4321/sql/myapp \
  -d "CREATE TABLE products (id INT, name VARCHAR(100), price DECIMAL(10,2))" \
  -u root:admin

Authentication

  • Default credentials: root / admin.
  • Set the initial root password via CLI: --root-password=supersecret at the first run (on a fresh -data folder), or via Docker env ROOT_PASSWORD.
  • Docker Compose example:
services:
  memcp:
    image: yourrepo/memcp:latest
    environment:
      - ROOT_PASSWORD=supersecret
      - PARAMS=--api-port=4321
    ports:
      - "4321:4321"  # HTTP API
      - "3307:3307"  # MySQL protocol
    volumes:
      - memcp_data:/data
volumes:
  memcp_data: {}
  • Change the credentials with:
curl -X POST http://localhost:4321/sql/system \
  -d "ALTER USER root IDENTIFIED BY 'supersecret'" \
  -u root:admin

Performance Comparison ๐Ÿ“ˆ

Operation MySQL (SSD) MySQL (Memory) MemCP
Simple SELECT 5-15ms 1-3ms 0.1ms
Complex JOIN 50-200ms 10-50ms 1-5ms
INSERT/UPDATE 10-30ms 2-8ms 0.2ms
REST API Call 20-100ms 10-60ms 1-10ms

Benchmarks run on standard hardware with 1000+ concurrent requests

Use Cases ๐Ÿ’ผ

  • ๐ŸŽฎ Gaming Backends - Real-time leaderboards and player data
  • ๐Ÿ’ฐ Financial APIs - High-frequency trading and analytics
  • ๐Ÿ“ฑ Mobile Apps - Ultra-responsive user experiences
  • ๐Ÿ›’ E-commerce - Product catalogs and inventory management
  • ๐Ÿ“Š Analytics Dashboards - Real-time data visualization
  • ๐Ÿงช Development & Testing - Instant database provisioning

Contributing ๐Ÿค

We'd love your help making MemCP even better!

๐ŸŒŸ Why Contribute?

  • Work with cutting-edge database technology
  • Learn Go, Scheme, and database internals
  • Impact thousands of developers worldwide
  • Build ultra-high-performance systems

๐Ÿ› ๏ธ Easy Ways to Contribute

  • ๐Ÿ“ Add test cases - Expand our comprehensive test suite
  • ๐Ÿ› Fix bugs - Help us squash issues and improve stability
  • โšก Performance optimization - Make fast even faster
  • ๐Ÿ“š Documentation - Help other developers get started
  • ๐Ÿ”ง New features - SQL functions, operators, and capabilities

๐Ÿš€ Getting Started

# 1. Fork the repository
# 2. Clone your fork
git clone https://github.com/yourusername/memcp.git

# 3. Set up development environment
cd memcp
go build -o memcp

# 4. Run the test suite
python3 run_sql_tests.py tests/01_basic_sql.yaml 4400

# 5. Make your changes and add tests
# 6. Submit a pull request!

๐ŸŽฏ Current Contribution Opportunities

  • Vector database features - Advanced similarity search
  • Additional SQL functions - String, math, and date functions
  • Performance benchmarking - Automated performance testing
  • Driver development - Language-specific database drivers
  • Documentation examples - Real-world usage scenarios

Testing ๐Ÿงช

MemCP includes a comprehensive test framework:

# Run all tests
make test

# Or if you want to contribute, deploy this as a Pre-commit hook:
cp git-pre-commit .git/hooks/pre-commit

# Run specific test suites
python3 run_sql_tests.py tests/01_basic_sql.yaml 4400      # Basic operations
python3 run_sql_tests.py tests/02_functions.yaml 4400     # SQL functions
python3 run_sql_tests.py tests/07_error_cases.yaml 4400   # Error handling

Performance Testing ๐Ÿ“Š

MemCP includes an auto-calibrating performance test framework that adapts to your machine.

Running Performance Tests

# Run perf tests (uses calibrated baselines)
PERF_TEST=1 make test

# Calibrate for your machine (run ~10 times to reach target time range)
PERF_TEST=1 PERF_CALIBRATE=1 make test

# Freeze row counts for bisecting performance regressions
PERF_TEST=1 PERF_NORECALIBRATE=1 make test

# Show query plans for each test
PERF_TEST=1 PERF_EXPLAIN=1 make test

How Calibration Works

  1. Initial run starts with 10,000 rows per test
  2. Each calibration run scales row counts by 30% up/down
  3. Target is 10-20 seconds query time per test
  4. Baselines are stored in .perf_baseline.json
  5. After ~10 runs, row counts stabilize in the target range

Output Format

โœ… Perf: COUNT (7.9ms / 30000ms, 20,000 rows, 0.39ยตs/row, 11.4MB heap)
         โ”‚       โ”‚         โ”‚            โ”‚        โ”‚           โ””โ”€ Heap memory after insert
         โ”‚       โ”‚         โ”‚            โ”‚        โ””โ”€ Time per row
         โ”‚       โ”‚         โ”‚            โ””โ”€ Calibrated row count
         โ”‚       โ”‚         โ””โ”€ Threshold (from baseline ร— 1.3)
         โ”‚       โ””โ”€ Actual query time
         โ””โ”€ Test name

Performance Debugging Cookbook

Detecting a performance regression:

# 1. Freeze baselines to use consistent row counts
PERF_TEST=1 PERF_NORECALIBRATE=1 make test

# 2. If a test fails threshold, you have a regression

Bisecting a performance bug:

# 1. Checkout the known-good commit, run calibration
git checkout good-commit
PERF_TEST=1 PERF_CALIBRATE=1 make test  # run 10x to calibrate

# 2. Save the baseline
cp .perf_baseline.json .perf_baseline_good.json

# 3. Bisect with frozen row counts
git bisect start
git bisect bad HEAD
git bisect good good-commit
git bisect run bash -c 'PERF_TEST=1 PERF_NORECALIBRATE=1 make test'

Analyzing slow queries:

# Show query plans to understand execution
PERF_TEST=1 PERF_EXPLAIN=1 make test

Environment Variables

Variable Values Description
PERF_TEST 0/1 Enable performance tests
PERF_CALIBRATE 0/1 Update baselines with new times
PERF_NORECALIBRATE 0/1 Freeze row counts (for bisecting)
PERF_EXPLAIN 0/1 Show query plans

License ๐Ÿ“„

MemCP is open source software. See the LICENSE file for details.


Ready to experience database performance like never before? Get Started โ€ข Contribute โ€ข Join our Community

MemCP: Because your applications deserve better than "good enough" performance. โšก

About

A columnar In-Memory Database as Drop-In Replacement for MySQL supporting 10x performance in OLAP workloads and similar performance in OLTP

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Contributors 5