Skip to content

Implement watch mode for DynamoDB backend #389

@abtreece

Description

@abtreece

Problem

The DynamoDB backend currently only supports polling mode. Watch mode is not implemented, forcing users to use interval-based polling which has higher CPU overhead and delayed updates.

Current Status

func (c *Client) WatchPrefix(ctx context.Context, prefix string, keys []string, waitIndex uint64, stopChan chan bool) (uint64, error) {
    <-stopChan
    return 0, nil
}

Recommendation

Implement event-driven updates using DynamoDB Streams:

  1. Enable DynamoDB Streams on tables used by confd
  2. Subscribe to stream events (INSERT, MODIFY, REMOVE)
  3. Process stream records to detect changes matching watched prefixes
  4. Return from WatchPrefix when relevant changes detected

Expected Benefits

  • Lower CPU usage (event-driven vs. polling)
  • Faster change detection
  • Better scalability for large deployments

Implementation Considerations

  • DynamoDB Streams have 24-hour retention
  • Stream records provide old/new images for comparison
  • May need to handle stream shard management

Files

  • pkg/backends/dynamodb/client.go

Priority

Medium - Feature enhancement, not blocking

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions