hyprstream_core/
lib.rs

1/*!
2# Hyprstream: Real-time Aggregation Windows and High-Performance Cache for Apache Arrow Flight SQL
3
4Hyprstream is a next-generation application for real-time data ingestion, windowed aggregation, caching, and serving.
5Built on Apache Arrow Flight and DuckDB, and developed in Rust, Hyprstream dynamically calculates metrics like running
6sums, counts, and averages, enabling blazing-fast data workflows, intelligent caching, and seamless integration with
7ADBC-compliant datastores.
8
9## Key Features
10
11### Data Ingestion via Apache Arrow Flight
12- Streamlined ingestion using Arrow Flight for efficient columnar data transport
13- Real-time streaming support for metrics, datasets, and vectorized data
14- Seamless integration with data producers for high-throughput ingestion
15- Write-through to ADBC datastores for eventual data consistency
16
17### Intelligent Read Caching with DuckDB
18- In-memory performance using DuckDB for lightning-fast caching
19- Optimized querying for analytics workloads
20- Automatic cache management with configurable expiry policies
21- Time-based expiry with future support for LRU/LFU policies
22
23### Data Serving with Arrow Flight SQL
24- High-performance queries via Arrow Flight SQL
25- Support for vectorized data and analytical queries
26- Seamless integration with analytics and visualization tools
27
28### Real-Time Aggregation
29- Dynamic metrics with running sums, counts, and averages
30- Lightweight state management for aggregate calculations
31- Dynamic weight computation for AI/ML pipelines
32- Time window partitioning for granular analysis
33
34## Usage
35
36Basic usage example with programmatic configuration:
37
38```rust,no_run
39use hyprstream::config::{Settings, EngineConfig, CacheConfig};
40use hyprstream::service::FlightServiceImpl;
41use std::sync::Arc;
42use std::collections::HashMap;
43
44#[tokio::main]
45async fn main() -> Result<(), Box<dyn std::error::Error>> {
46    // Create configuration programmatically
47    let mut settings = Settings::default();
48
49    // Configure primary storage engine
50    settings.engine.engine = "duckdb".to_string();
51    settings.engine.connection = ":memory:".to_string();
52    settings.engine.options.insert("threads".to_string(), "4".to_string());
53
54    // Configure caching (optional)
55    settings.cache.enabled = true;
56    settings.cache.engine = "duckdb".to_string();
57    settings.cache.connection = ":memory:".to_string();
58    settings.cache.max_duration_secs = 3600;
59
60    // Create and initialize the service
61    let service = FlightServiceImpl::from_settings(&settings).await?;
62
63    // Use the service in your application...
64    Ok(())
65}
66```
67
68For detailed configuration options and examples, see:
69- [`config`](crate::config) module for configuration options
70- [`storage`](crate::storage) module for storage backend details
71- [`examples`](examples/) directory for more usage examples
72*/
73
74pub mod metrics;
75pub mod storage;
76pub mod service;
77pub mod config;
78pub mod aggregation;
79
80pub use service::FlightSqlService;
81pub use storage::StorageBackend;
82pub use metrics::MetricRecord;
83pub use aggregation::{TimeWindow, AggregateFunction, GroupBy, AggregateResult};
hyprstream_core/lib.rs

hyprstream_core/
lib.rs
