Join us on slack
Streaming reference architecture built around Kafka.
A collection of components to build a real time ingestion pipeline.
Please take a moment and read the documentation and make sure the software prerequisites are met!!
| Connector | Type | Description | Docs |
|---|---|---|---|
| AzureDocumentDb | Sink | Kafka connect Azure DocumentDb sink to subscribe to write to the cloud Azure Document Db. | Docs |
| BlockChain | Source | Kafka connect Blockchain source to subscribe to Blockchain streams and write to Kafka. | Docs |
| Bloomberg | Source | Kafka connect source to subscribe to Bloomberg streams and write to Kafka. | Docs |
| Cassandra | Source | Kafka connect Cassandra source to read Cassandra and write to Kafka. | Docs |
| Coap | Source | Kafka connect Coap source to read from IoT Coap endpoints using Californium. | Docs |
| Coap | Sink | Kafka connect Coap sink to write kafka topic payload to IoT Coap endpoints using Californium. | Docs |
| *DSE Cassandra | Sink | Certified DSE Kafka connect Cassandra sink task to write Kafka topic payloads to Cassandra. | Docs |
| Druid | Sink | Kafka connect Druid sink to write Kafka topic payloads to Druid. | Docs |
| Elastic | Sink | Kafka connect Elastic Search sink to write Kafka topic payloads to Elastic Search. | Docs |
| FTP/HTTP | Source | Kafka connect FTP and HTTP source to write file data into Kafka topics. | Docs |
| HBase | Sink | Kafka connect HBase sink to write Kafka topic payloads to HBase. | Docs |
| Hazelcast | Sink | Kafka connect Hazelcast sink to write Kafka topic payloads to Hazelcast. | Docs |
| Kudu | Sink | Kafka connect Kudu sink to write Kafka topic payloads to Kudu. | Docs |
| InfluxDb | Sink | Kafka connect InfluxDb sink to write Kafka topic payloads to InfluxDb. | Docs |
| JMS | Sink | Kafka connect JMS sink to write Kafka topic payloads to JMS. | Docs |
| MongoDB | Sink | Kafka connect MongoDB sink to write Kafka topic payloads to MongoDB. | Docs |
| MQTT | Source | Kafka connect MQTT source to write data from MQTT to Kafka. | Docs |
| Redis | Sink | Kafka connect Redis sink to write Kafka topic payloads to Redis. | Docs |
| ReThinkDB | Source | Kafka connect RethinkDb source subscribe to ReThinkDB changefeeds and write to Kafka. | Docs |
| ReThinkDB | Sink | Kafka connect RethinkDb sink to write Kafka topic payloads to RethinkDb. | Docs |
| Yahoo Finance | Source | Kafka connect Yahoo Finance source to write to Kafka. | Docs |
| VoltDB | Sink | Kafka connect Voltdb sink to write Kafka topic payloads to Voltdb. | Docs |
0.2.5 (24 March 2017)
- Adding Azure DocumentDb Sink
- Adding UPSERT to Elastic Search
- Cassandra improvements
withunwrap - Upgrade to Kudu 1.0 and CLI 1.0
- Add ingest_time to CoAP Source
- Multiple bug fixes
- Release for CP 3.2.0 and Kafka 0.10.2.0
- Release for CP 3.1.2 and Kafka 0.10.1.1
0.2.4 (26 Jan 2017)
- Added FTP and HTTP Source.
- Added InfluxDB tag support. KCQL: INSERT INTO targetdimension
SELECT * FROM influx-topic WITHTIMESTAMP sys_time() WITHTAG(field1, CONSTANT_KEY1=CONSTANT_VALUE1, field2,CONSTANT_KEY2=CONSTANT_VALUE1) - Added InfluxDb consistency level. Default is
ALL. Useconnect.influx.consistency.levelto set it to ONE/QUORUM/ALL/ANY - InfluxDb
connect.influx.sink.route.querywas renamed toconnect.influx.sink.kcql - Added support for multiple contact points in Cassandra
0.2.3 (5 Jan 2017)
- Added CoAP Source and Sink.
- Added MongoDB Sink.
- Added MQTT Source.
- Hazelcast support for ring buffers.
- Redis support for Sorted Sets.
- Added start scripts.
- Added Kafka Connect and Schema Registry CLI.
- Kafka Connect CLI now supports pause/restart/resume; checking connectors on the classpath and validating configuration of connectors.
- Support for
Struct,Schema.STRINGandJsonwith schema in the Cassandra, ReThinkDB, InfluxDB and MongoDB sinks. - Rename
export.query.routetosink.kcql. - Rename
import.query.routetosource.kcql. - Upgrade to KCQL 0.9.5 - Add support for
STOREASso specify target sink types, e.g. Redis Sorted Sets, Hazelcast map, queues, ringbuffers.
Requires gradle 3.0 to build.
To build
gradle compileTo test
gradle testTo create a fat jar
gradle shadowJarYou can also use the gradle wrapper
./gradlew shadowJar
To view dependency trees
gradle dependencies # or
gradle :kafka-connect-cassandra:dependencies