This project is a comprehensive backend utility service built with NestJS, implementing a robust 3-layer architecture. It provides a wide range of functionalities including API management, worker processes, blockchain interactions, AI capabilities, game services, and multi-storage data management.

• NestJS Backend Utility Service
  • Overview
  • Table of Contents
  • Architecture
    • 1. Access (Gateway) Layer
      • Restful API Implementation
      • WebSocket Gateway
      • JSON-RPC
      • GraphQL API
      • Telegram Bot Integration
    • 2. Executor (Business) Layer
      • Worker Processes
      • Blockchain Integration
      • AI Capabilities
      • Game Services
      • Telegram Bot
    • 3. Data Layer
    • 4. Others
      • Service Communication
      • Worker Thread Integration
      • Resilience Patterns
      • Crawler Module
      • Docker Support
      • Google Cloud Build
  • Technologies Used
  • Getting Started
  • Contact

The project follows a 3-layer architecture:

1. Access (Gateway) Layer
2. Business (Executor) Layer
3. Data Layer

This layer handles incoming requests and implements various middleware and utilities:

• Guards: JWT authentication and RBAC (Role-Based Access Control) for secure access
• Rate Limiter: Prevents API abuse by limiting request frequency
• Interceptors:
  • Request caching: Improves performance by caching frequently accessed data
  • Response formatting: Ensures consistent API responses
  • Context awareness: Provides request-specific information to handlers
  • Swagger integration: Automatically generates API documentation
  • Integrates structured logging into request and response cycles, allowing for better tracking and debugging of API calls.
• Exception filters: Customize error handling for consistent error responses
• Pipes: For input validation and data transformation

• Socket.IO with Redis Adapter: Enables real-time, bidirectional communication with scalability, allowing multiple server instances for efficient message distribution.
• Channel management by rooms: Organizes connections for efficient message distribution
• Message compression: Reduces bandwidth usage for real-time communications

• The JRPCModule provides a JSON-RPC interface for the application. It is configured to handle requests at the /rpc endpoint.
• Handlers: DomainHandler which acts like a controller in a RESTful API to manage requests.

• Apollo Server integration with NestJS: Provides a powerful GraphQL API endpoint at /graphql

• Socket-based listener and responder: Enables real-time interaction with Telegram
• Multiple interactive functions: Provides various bot commands and features
• Internationalization (i18n) support: Allows multi-language bot responses

This layer contains the core business logic and processing:

• Schedulers: Manage time-based tasks and recurring jobs
• BullMQ consumers: Handle background processing and job queues

• Smart contract interactions: Allows the service to interact with blockchain-based applications
• Transaction synchronization: Keeps the service in sync with blockchain state

• RAG (Retrieval-Augmented Generation) pipeline: Enhances AI responses with relevant retrieved information
• LLM (Large Language Model) integration: Enables advanced natural language processing
• Function calling with LangChain.js: Allows dynamic execution of functions based on AI input
• Performance monitoring with Langfuse: Tracks and optimizes AI model performance

• Authorization game server using Colyseus: Manages game session authentication, ensures real-time synchronization of game state with clients.
• Turn-based game lifecycle framework: Provides structure for implementing core game loop for turn-based games

• UI Components: Custom keyboards, inline buttons with callback_query handling and pattern matching
• Request Processing: Queue-based system for handling Telegram API requests
• Stateful Conversation Management:
  • User session with Redis for persistent state storage
  • Finite State Machine (FSM) implementation for complex conversation flows

This layer manages data storage and retrieval across multiple databases:

• PostgreSQL: Primary relational database for structured data
• TypeORM: An ORM for TypeScript that simplifies database interactions with PostgreSQL
• TimescaleDB: Time-series database for handling time-based data efficiently
• Redis: In-memory data structure store for caching and real-time operations
• pgvector: Enables vector similarity search in PostgreSQL
• Milvus: Distributed vector database for large-scale similarity search
• Firestore: NoSQL database for flexible, scalable data storage
• Elasticsearch: Full-text search and analytics engine
• Google Cloud Storage: Used for file uploads and storage

The project implements multiple methods for service communication:

• RabbitMQ: Message broker for reliable inter-service communication
• Redis Pub/Sub: For lightweight, real-time messaging between services
• Google Pub/Sub: Scalable, fully-managed messaging service for larger deployments

To enhance performance and prevent blocking the main event loop, the project incorporates a worker thread for thread pool (heavy computations, file I/O, CPU-bound, ...) tasks. This ensures that the main thread remains responsive to I/O and network requests. The worker thread implementation leverages Node.js's worker_threads module, enabling parallel execution and improved application throughput.

Reliable patterns for building resilient applications using nestjs-resilience: Provides strategies to ensure that application can handle failures and recover quickly, improving the reliability and fault-tolerance of services: Circuit Breaker, Retry, Timeout, Bulkhead, Fallback, Rate Limiting.

Implements scraper proxies for data collection from various sources, enhancing the ability to gather and process information from external websites: DexTools, DexScreener, RapidAPI, Twitter, CoinGecko, ...

The project includes Docker configurations for all services and monitoring tools in the docker-compose.yml file, enabling easy deployment and scaling of the entire application stack.

Includes cloudbuild.yaml file for triggering Google Cloud Build, facilitating containerization for deployment on Cloud Run and virtual machines (VMs). The configuration automates the build process, ensuring that the application is packaged and deployed efficiently.

• NestJS: A progressive Node.js framework for building efficient and scalable server-side applications
• BullMQ: Redis-based queue for handling distributed jobs and messages
• LangChain.js: Framework for developing applications powered by language models
• Colyseus: Multiplayer game framework for Node.js
• Elasticsearch: Distributed search and analytics engine
• OneSignal: Cross-platform push notification service
• Google Cloud Storage: Object storage service by Google Cloud

1. Clone the repository:

   git clone https://github.com/kiyoshitaro/app-server.git
   

2. Install dependencies:

   cd app-server
   npm install
   

3. Set up environment variables:

   cp .env.sample .env
   

   Edit the .env file with your configuration details.

4. Start the development server:

   npm run start:dev
   

5. For Docker deployment, use:

   bash start.sh
   

For any inquiries, please contact Hung Nguyen.