Skip to content

wanmail/aws-config-graph

Repository files navigation

AWS Config Resource Collector

A tool for collecting AWS Config resources and storing them in Neo4j graph database.

Features

  • Multiple collection methods:
    • S3: Collect from AWS Config snapshots stored in S3
    • Local: Collect from local JSON files
    • API: Collect directly from AWS Config API
  • Scheduled collection support
  • Docker containerized deployment
  • Support for AWS Config Aggregator
  • Relationship mapping between resources

Prerequisites

  • Python 3.9+
  • Neo4j 5.x
  • AWS credentials configured
  • Docker and Docker Compose (for containerized deployment)

Installation

Local Development

  1. Create virtual environment and install dependencies:
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
  1. Configure AWS credentials:
aws configure

Docker Deployment

  1. Create .env file with required variables:
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password
BUCKET_NAME=your-config-bucket
  1. Start services using docker-compose:
docker-compose up -d

Usage

Command Line Interface

  1. Initialize Neo4j indexes:
python index.py bolt://localhost:7687 --username neo4j --password your_password
  1. Collect resources:
  • From S3:
python main.py --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password your_password s3 your-bucket --prefix config/
  • From local files:
python main.py --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password your_password local /path/to/config/files
  • From AWS Config API:
python main.py --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password your_password api --aggregator-name your-aggregator
  1. Schedule daily collection:
python main.py --command schedule --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password your_password --schedule-time "02:00" s3 your-bucket

AWS Credentials

You can provide AWS credentials in several ways:

  1. Using AWS profile:
python main.py --profile your-profile-name ...
  1. Using access keys directly:
python main.py \
  --aws-access-key-id YOUR_ACCESS_KEY_ID \
  --aws-secret-access-key YOUR_SECRET_ACCESS_KEY \
  [--aws-session-token YOUR_SESSION_TOKEN] \
  ...
  1. Using environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)

  2. Using EC2 instance profile or container role (no additional configuration needed)

AWS Config Setup

How to collect multi accounts' resources in single account

Collect resources in a config aggregator

Creating Aggregators for AWS Config

Delivery configuration snapshots to a single bucket

Just set all config recorder delivery configuration to a single bucket.

And set the bucket permissions, refer to Permissions for the Amazon S3 Bucket for the AWS Config Delivery Channel .

You can add all accounts to the AWS:SourceAccount or use aws:SourceOrgID .

How to enable AWS Config Recorder in multiple accounts and regions

AWS SSM quick setup

Create an AWS Config configuration recorder using Quick Setup

However, Quick Setup does not support in the follow regions:

Europe (Milan)

Asia Pacific (Hong Kong)

Middle East (Bahrain)

China (Beijing)

China (Ningxia)

AWS GovCloud (US-East)

AWS GovCloud (US-West)

CloudFormation StackSet(Recommended)

my-awsconfig-stackset walkthrough

Use boto3

aws-config-enable

Project Structure

  • main.py: Main entry point and CLI interface
  • collect.py: Resource collection implementations
  • merge.py: Neo4j data merging logic
  • index.py: Neo4j index initialization
  • args.py: Command line argument parsing
  • docker-compose.yml: Docker services configuration
  • Dockerfile: Container build configuration

Contributing

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add some amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

About

A tool for collecting AWS Config resources and storing them in Neo4j graph database.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published