中文版

A high-performance MySQL binlog replication client for Rust. This is a clean, idiomatic Rust implementation of a MySQL binlog parser with comprehensive support for MySQL 5.6, 5.7, and 8.0, including advanced features like JSON/JSONB types, GTID replication, and modern MySQL 8.0 events.

gitee.com/rust_us/mysql-cdc-rs

github.com/rust-us/mysql-cdc-rs

• High Performance: Zero-copy parsing with memory optimization and object pooling
• Complete Event Support: Comprehensive support for all major MySQL binlog events
• Advanced Row Processing: Field-level change detection with incremental updates
• Thread-Safe Architecture: Lock-free design with local caching for concurrent access
• Extensible Plugin System: Custom event handlers and type decoders
• Real-time Monitoring: Built-in performance metrics and statistics
• Memory Management: Intelligent memory usage monitoring and automatic cleanup
• Error Recovery: Robust error handling with configurable recovery strategies

The library is built with a modular, extensible architecture designed for high performance and maintainability:

• Event Factory: Dynamic event parser registration and creation
• Column Parser: Extensible type system with custom decoder support
• Row Parser: Object-oriented design with zero-copy bitmap processing
• Metadata Manager: Thread-safe table mapping with LRU caching
• Memory Manager: Object pooling and intelligent resource management
• Extension Registry: Plugin system for custom handlers and processors

• Core Parser Architecture: Object-oriented design with comprehensive error handling
• Advanced Row Processing: Zero-copy parsing with field-level change detection
• Event Handler System: Sync/async event processing with custom handlers
• Performance Optimization: Memory pooling, caching, and zero-copy operations
• Monitoring & Statistics: Real-time performance metrics and analysis
• Type System: Extensible column parser with custom type decoder support
• MySQL 8.0 Support: Latest event types including TRANSACTION_PAYLOAD_EVENT

• Metadata Management: Unified metadata system with GTID state management
• Memory Management: Advanced memory monitoring and automatic cleanup
• Async Processing: Non-blocking event stream processing
• Plugin System: Dynamic extension loading and hot-swapping
• Configuration Management: Comprehensive config system with hot-reload

• SSL encryption support is limited
• Split packets (>16MB) require additional handling
• Some advanced MySQL 8.0 features are still in development

┌─────────────────────────────────────────────────────────────┐
│                    Binlog Parser Core                       │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │ Event       │  │ Column      │  │ Row                 │  │
│  │ Factory     │  │ Parser      │  │ Parser              │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │ Event       │  │ Metadata    │  │ Memory              │  │
│  │ Decoder     │  │ Manager     │  │ Manager             │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │ Stream      │  │ Error       │  │ Extension           │  │
│  │ Reader      │  │ Handler     │  │ Registry            │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

+-- binlog/              # Core binlog parsing engine
    +-- events/          # Event type definitions and parsers
    +-- column/          # Column type system and decoders
    +-- row/             # Row-level event processing
    +-- decoder/         # Event decoding and registry
    +-- metadata/        # Table metadata and GTID management
    +-- factory/         # Event factory and creation
+-- binlog_cli/          # Command-line interface
+-- common/              # Shared types and utilities
+-- connection/          # MySQL connection and replication
+-- memory/              # Memory management and allocation
+-- relay_log/           # Relay log processing
+-- web/                 # Web interface and monitoring
+-- tests/               # Comprehensive test suite

In order to reduce the probability of execution errors and improve the functional features, we unify the versions of the Rust toolchain and switch the following commands:

• How to install Rust development environment in Windows operating system

$ rustup install nightly

$ rustup toolchain list
stable-aarch64-apple-darwin (default)
nightly-aarch64-apple-darwin (override)

#$ rustup override set nightly
# Or 
$ rustup default nightly

$ rustup toolchain list
stable-x86_64-pc-windows-msvc (default)
nightly-x86_64-pc-windows-msvc (active)

You can check it out in the rustup toolchain list. If not, it will be automatically downloaded.

$ cargo test
running 2 tests
test tests::bench_add_two ... ok
test tests::it_works ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Install gnuplots

Add

[dev-dependencies]
criterion = "0.3"

[[bench]]
name = "my_benchmark"
harness = false

编写测试文件($PROJECT/benches/my_benchmark.rs)

use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn fibonacci(n: u64) -> u64 {
    match n {
        0 => 1,
        1 => 1,
        n => fibonacci(n-1) + fibonacci(n-2),
    }
}

fn criterion_benchmark(c: &mut Criterion) {
    c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20))));
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

$ cargo bench
test result: ok. 0 passed; 0 failed; 7 ignored; 0 measured; 0 filtered out; finished in 0.01s

     Running benches\aes_bencher.rs (target\release\deps\aes_bencher-f9a9ea35aca2ab2d.exe)
Gnuplot not found, using plotters backend
add_two                 time:   [604.03 ps 619.40 ps 636.72 ps]
                        change: [-2.3923% +0.4866% +3.6080%] (p = 0.76 > 0.05)
                        No change in performance detected.

# 显示详细日志
cargo bench --bench aes_benchmark -- --verbose

# 只运行特定测试
cargo bench -p cryptolib --bench sm4_benchmark
cargo bench --bench aes_benchmark -- -n "AES Parallel"

报告在目录 target\criterion 下查看

Add to your Cargo.toml:

[dependencies]
mysql-cdc-rs = { git = "https://github.com/rust-us/mysql-cdc-rs" }

use mysql_cdc_rs::binlog::BinlogParser;
use mysql_cdc_rs::binlog::ParserConfig;

// Create parser with default configuration
let config = ParserConfig::default();
let mut parser = BinlogParser::new(config)?;

// Parse binlog file
let events = parser.parse_file("mysql-bin.000001")?;
for event in events {
    match event {
        BinlogEvent::WriteRows(write_event) => {
            println!("INSERT: {} rows", write_event.rows.len());
        }
        BinlogEvent::UpdateRows(update_event) => {
            println!("UPDATE: {} rows", update_event.rows.len());
        }
        BinlogEvent::DeleteRows(delete_event) => {
            println!("DELETE: {} rows", delete_event.rows.len());
        }
        _ => {}
    }
}

use mysql_cdc_rs::binlog::row::RowEventHandler;

struct MyRowHandler;

impl RowEventHandler for MyRowHandler {
    fn on_row_insert(&self, table: &TableMapEvent, row: &RowData) -> Result<()> {
        println!("New row inserted in {}.{}", table.database_name, table.table_name);
        Ok(())
    }
    
    fn on_row_update(&self, table: &TableMapEvent, before: &RowData, after: &RowData) -> Result<()> {
        println!("Row updated in {}.{}", table.database_name, table.table_name);
        Ok(())
    }
}

// Register custom handler
let mut parser = BinlogParser::new(config)?;
parser.event_handlers_mut().register_sync_handler(Arc::new(MyRowHandler));

# Clone repository
git clone https://github.com/your-repo/mysql-cdc-rs.git
cd mysql-cdc-rs

# Build with optimizations
cargo build --release

# Run tests
cargo test

# Build CLI tool
cargo build --bin binlog_cli --release

Caused by:
  process didn't exit successfully: `mysql-cdc-rs\target\debug\build\openssl-sys-94071a3d762a0669\build-script-main` (exit code: 101)
  --- stdout
  cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_OPENSSL_NO_VENDOR
  X86_64_PC_WINDOWS_MSVC_OPENSSL_NO_VENDOR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_VENDOR
  OPENSSL_NO_VENDOR unset
  running "perl" "./Configure" "--prefix=/mysql-cdc-rs/target/debug/build/openssl-sys-5ad9f46fc53da764/out/openssl-build/install" "--openssldir=SYS$MANAGER:[OPENSSL]" "no-dso" "no-shared" "no-ssl3" "no-tests" "no-comp" "no-zlib" "no-zlib-dynamic" "--libdir=lib" "no-md2" "no-rc5" "no-weak-ssl-ciphers" "no-camellia" "no-idea" "no-seed" "no-capieng" "no-asm" "VC-WIN64A"

  Error configuring OpenSSL build:
      Command: "perl" "./Configure" "--prefix=/mysql-cdc-rs/target/debug/build/openssl-sys-5ad9f46fc53da764/out/openssl-build/install" "--openssldir=SYS$MANAGER:[OPENSSL]" "no-dso" "no-shared" "no-ssl3" "no-tests" "no-comp" "no-zlib" "no-zlib-dynamic" "--libdir=lib" "no-md2" "no-rc5" "no-weak-ssl-ciphers" "no-camellia" "no-idea" "no-seed" "no-capieng" "no-asm" "VC-WIN64A"
      Failed to execute: program not found

Instructions for compiling and configuring using Perl programs. From https://strawberryperl.com/ Download the Windows installation package. And installation, restart the command-line terminal and build again.

strawberry-perl-5.38.2.2-64bit.msi

See BinlogCLI README.md

http://s.codealy.com/rust_us/mysql_cdc_rs/2024.02%20binlog%20cli%20view.webm

See Binlog README.md for detailed event documentation.

This is a pure Rust implementation that doesn't require MySQL server libraries while supporting both file parsing and live replication from MySQL master servers. The goal is to parse every field of binlog events with maximum accuracy and performance.

• Enhanced Row Parser: Zero-copy parsing with field-level change detection
• Advanced Monitoring: Real-time performance metrics and statistics
• Memory Optimization: Object pooling and intelligent resource management
• Event Handler System: Extensible sync/async event processing
• MySQL 8.0 Support: Complete support for latest event types
• Error Recovery: Robust error handling with configurable strategies

• Zero-Copy Parsing: Minimize memory allocations and data copying
• Object Pooling: Reuse event objects to reduce GC pressure
• Concurrent Processing: Thread-safe design with local caching
• Memory Monitoring: Real-time memory usage tracking and limits
• Incremental Updates: Field-level change detection for efficient processing
• Streaming Support: Process large binlog files with constant memory usage

The library provides a comprehensive plugin system for customization:

• Event Filters: Filter events based on custom criteria
• Event Processors: Process events with custom business logic
• Type Decoders: Add support for custom MySQL data types
• Row Handlers: Handle row-level changes with custom logic
• Monitoring Extensions: Add custom metrics and monitoring

Built-in monitoring provides detailed insights:

• Parse Performance: Rows/second, bytes/second, parse times
• Memory Usage: Current usage, peak usage, allocation patterns
• Error Statistics: Error rates, recovery success, error types
• Cache Performance: Hit ratios, cache sizes, eviction rates
• Row Complexity: Column counts, data sizes, null percentages

• Object-oriented parser architecture
• Zero-copy parsing optimization
• Event handler system
• Performance monitoring
• MySQL 8.0 event support

• Unified metadata management system
• Advanced memory management
• Async/await support
• Plugin system with hot-swapping
• Configuration management

• Distributed parsing
• Advanced error recovery
• Performance analytics
• Multi-source replication
• Cloud-native deployment

We welcome contributions! Please see our Contributing Guide for details.

1. Install Rust nightly toolchain
2. Clone the repository
3. Run tests: cargo test
4. Check formatting: cargo fmt
5. Run lints: cargo clippy

This project is licensed under the MIT License - see the LICENSE file for details.

• MySQL team for the excellent binlog format documentation
• Rust community for the amazing ecosystem
• Contributors who help improve this project