This repository contains the services for the bidding bounded context and projector services.

The bidding bounded context has been split into command and query services, following the CQRS pattern. The Projector Services tier is decomposed into two subsystems and are responsible for consuming domain events from Kafka and maintaining their respective materialized views.

• Project Structure
• Design
  • High Level Architecture
  • Service Design
    • Bid Command Service
    • Bid Query Service
    • Auction Projector
    • Bid Projector
  • Key Architecture Decisions
    • CQRS Pattern
    • Polyglot Persistence (MongoDB, PostgreSQL)
    • Projector services for Domain Events
    • Event Streaming with Apache Kafka
    • Not using 2PC or Transactional Outbox pattern
    • Observability: Prometheus, Prometheus Alarms, and Grafana
• Run Locally (Mac)
• Links

├── docker-compose.yaml
├── generated-client
├── grafana
├── openapi
│   ├── bid-command
│   └── bid-query
├── pkg
├── prometheus.yml
├── qodana.yaml
├── scripts
├── services
│   ├── auction-projector
│   │   ├── Dockerfile
│   │   ├── cmd
│   │   └── internal
│   │       ├── events
│   │       ├── projections
│   │       └── projector
│   ├── bid-command
│   │   ├── Dockerfile
│   │   ├── cmd
│   │   ├── internal
│   │   │   ├── application
│   │   │   ├── domain
│   │   │   ├── infrastructure
│   │   │   ├── presentation
│   │   │   └── server
│   │   ├── openapi
│   │   └── sqlc
│   ├── bid-projector
│   │   ├── Dockerfile
│   │   ├── cmd
│   │   └── internal
│   │       ├── events
│   │       ├── projections
│   │       └── projector
│   ├── bid-query
│   │   ├── Dockerfile
│   │   ├── cmd
│   │   ├── internal
│   │   │   ├── application
│   │   │   ├── infrastructure
│   │   │   ├── presentation
│   │   │   ├── read_model
│   │   │   └── server
│   │   └── openapi
└── sqlc.yaml

Presentation Layer: Maps HTTP requests from OpenAPI schema to Application commands, basic input validation, and translates Application result/errors to HTTP status codes and OpenAPI schema
Application Layer: Implements the Place Bid use case. Coordinates domain objects and infrastructure via ports, but doesn’t know any concrete DB or MQ details.
Domain Layer: Contains entities (Bid, Auction), domain events (Bid.Placed), and rules. Also defines the ports for the Application and Infrastructure layers.
Infrastructure Layer: Provides adapters for persistence with postgreSQL and redis and for publishing domain events with Kafka

Presentation Layer: Maps HTTP requests from OpenAPI schema to Application commands, basic input validation, and translates Application result/errors to HTTP status codes and OpenAPI schema
Application Layer: Implements the List Bids use case by invoking read-model bid repository through ports and also handles sorting/ cursor pagination logic
Read Model: Stores denormalized, query-optimized view of bids used for listing and cursor pagination
Infrastructure Layer: Implements MongoDB and Redis adapters that materialize, query, and cache the read model exposed through ports

Projector: Subscribes to Kafka topics, deserialises events and forwards them to the router
Router: Maps events to handlers, rejects unsupported events
Handlers: business logic for processing events like AuctionOpened
Projections: Currently only one projection, redis projection. Populates the bid command service’s redis for fast lookup for authoritative checks

Projector: Subscribes to Kafka topics, deserialises events and forwards them to the router
Router: Maps events to handlers, rejects unsupported events
Handlers: business logic for processing events like BidPlaced
Projections: Currently only one projection, MongoDB projection. Populates the bid query service’s MongoDB with denormalised bid documents for listing and cursor pagination

• Rationale:
  • Separation of Concerns: Different models for read and write operations
  • Independent Scaling
  • Performance Optimization: Tailored data storage and retrieval
• Trade-offs:
  • Increased Complexity: More services to manage and maintain
  • Data Consistency Challenges: Eventual consistency between read and write models

• Rationale:
  • PostgreSQL
    • ACID transactions for bid command service
    • Strong consistency required for authoritative bid data
  • MongoDB
    • Flexible schema for bid read model
    • Easy to store denormalized bid documents for listing and cursor pagination
    • Horizontal scaling to handle large volumes of bid data
• Trade-offs:
  • Operational complexity
  • More infrastructure to manage

• Rationale:
  • Decouple consumption of events from Bidding query and command services
  • Enable simple fanout from projector services to future services like Opensearch, Notification service without touching Bidding CQRS services
• Trade-offs: Complexity, maintainability
• Alternatives Considered: Consuming events directly in the bidding CQRS

Kafka is used as the central message bus for asynchronous communication between our microservices. Our services produce domain events for other services to consume. This decouples our services, allowing them to evolve independently.

• Rationale:
  • At least once delivery
  • Ordering
  • Real-time, High-Throughput Data Flow
  • Strong durability and replication
  • Zero data lost on consumer failure
    
• Trade-offs:
  • Atomic Transactions: complex to achieve need 2PC or transactional outbox pattern (future work - use GitHub - ihippik/wal-listener to mitigate this)
  • Maintainability: Kafka introduces significant operational overhead, need to monitor broker health, manage partitions, consumer groups, etc

• Rationale
  • Time constraints
  • Lower latency
  • Implementation complexity
  • Low probability of critical events dropping due to app being hosted on Fargate, which is regionally resilient
• Trade-offs
  • Low probability of critical events dropping
• Alternatives Considered: Using an open source WAL listener like GitHub - ihippik/wal-listener

Prometheus and Grafana is used for monitoring application and infrastructure metrics along with Prometheus Alarms for detecting critical issues (P99 Latency, high 5xx error rates, etc.)

• Rationale
  • Prometheus
    • powerful query language (PromQL)
    • many open source integrations (Grafana, OTel, Thanos)
  • Prometheus Alarms
    • leverages metrics from Prometheus
  • Grafana
    • seamless integration with Prometheus
    • dashboard allows us to quickly identify trends and bottlenecks
• Trade-offs
  • Operational Overhead and Costs
    • prometheus and grafana servers need to be managed
  • Long term storage
    • storing data long term in Prometheus is extremely expensive
    • careful data retention policies must be set, we might export data to a remote storage solution like Thanos

open -a Docker
docker compose up -d 

• Kafka UI
• Bid Command Swagger
• Bid Query Swagger
• Prometheus
• Grafana Bid Platform Dashboard
  • Username: admin
  • Password: admin