Bug: Migration failures with distributed ClickHouse cluster using SENTRY_DISTRIBUTED_CLICKHOUSE_TABLES #7326
Description
Summary:
When attempting to perform Snuba migrations on a distributed ClickHouse cluster with the following configuration (in settings.py), migrations fail unexpectedly. The issue is reproducible with Snuba version 25.7.0.
Snuba version: 25.7.0
Configuration:
import os
from snuba.settings import *
env = os.environ.get
DEBUG = env("DEBUG", "0").lower() in ("1", "true")
SENTRY_DISTRIBUTED_CLICKHOUSE_TABLES = True
MIGRATIONS_LOCK_TIMEOUT = int(env("MIGRATIONS_LOCK_TIMEOUT", "600"))
MIGRATIONS_BATCH_SIZE = int(env("MIGRATIONS_BATCH_SIZE", "1"))
CLICKHOUSE_MUTATIONS_SYNC = int(env("CLICKHOUSE_MUTATIONS_SYNC", "1"))
CLICKHOUSE_ALTER_SYNC = int(env("CLICKHOUSE_ALTER_SYNC", "1"))
CLICKHOUSE_REPLICATION_ALTER_PARTITIONS_SYNC = int(env("CLICKHOUSE_REPLICATION_ALTER_PARTITIONS_SYNC", "2"))
CLUSTERS = [
{
"host": env("CLICKHOUSE_HOST", "sentry-clickhouse-headless"),
"port": int(9000),
"secure": env("CLICKHOUSE_SECURE", False),
"ca_certs": env("CLICKHOUSE_CA_CERTS", None),
"verify": env("CLICKHOUSE_VERIFY", False),
"user": env("CLICKHOUSE_USER", "default"),
"password": env("CLICKHOUSE_PASSWORD", ""),
"max_connections": int(os.environ.get("CLICKHOUSE_MAX_CONNECTIONS", 100)),
"database": env("CLICKHOUSE_DATABASE", "default"),
"http_port": 8123,
"storage_sets": {
"cdc",
"discover",
"eap_items",
"events",
"events_ro",
"metrics",
"migrations",
"outcomes",
"querylog",
"sessions",
"transactions",
"profiles",
"functions",
"replays",
"generic_metrics_sets",
"generic_metrics_distributions",
"search_issues",
"generic_metrics_counters",
"spans",
"events_analytics_platform",
"group_attributes",
"generic_metrics_gauges",
"metrics_summaries",
"profile_chunks",
},
"single_node": False,
"cluster_name": "default",
"distributed_cluster_name": "default",
},
]
REDIS_HOST = "sentry-sentry-redis-master"
REDIS_PORT = 6379
REDIS_PASSWORD = env("REDIS_PASSWORD", "")
REDIS_DB = int(env("REDIS_DB", 1))Steps to Reproduce:
- Set up a distributed ClickHouse cluster.
- Configure Snuba as above, ensuring
SENTRY_DISTRIBUTED_CLICKHOUSE_TABLES = True. - Run Snuba migrations.
Expected Behavior:
Migrations should complete successfully using distributed tables.
Actual Behavior:
Migrations fail. (Please specify error messages and ClickHouse version if available.)
Environment:
- Snuba: 25.7.0
- ClickHouse: 23.12.3