Blockchain to Kafka

A Kafka Connector to hook into the live streaming providing a real-time feed for new bitcoin blocks and transactions provided by www.blockchain.info . The connector subscribes to notification on blocks, transactions or an address and receives JSON objects describing a transaction or block when an event occurs. This JSON is then pushed via Kafka connect to a Kafka topic and therefore can be consumed either by a Sink or a live stream processor.

Since is a direct WebSocket connection the source will only ever use one connector task at any point. There is no point spawning more and then have duplicate data.

One thing to remember is the subscription API from blockchain doesn’t offer an option to start from a given timestamp. This means if the connect worker is down then you will miss some data.