Apache Kafka

Apache Kafka is similar to Facebook's Scribe engine (if you're familiar to it) and can be used to handle activity streams.

It works by allowing developers to maintain one or more activity streams (or feeds).

These feeds can be organized in topics, and then broadcast (published) to all subscribed consumers.

The parts of Kafka that manage and publish the feeds are called producers.

Kafka is basically the middle layer between the producers and consumers, a complex system that can handle any kind of traffic and requests on any type of scale.

This is because Kafka was built to handle distributed server architectures, allowing developers to scale their Pub/sub systems on cloud services.

What is new in this release:

• A new Java producer for ease of implementation and enhanced performance.
• Delete topic support.
• Per topic configuration of preference for consistency over availability.
• Scala 2.11 support and dropping support for Scala 2.8.
• LZ4 compression.

What is new in version 0.8.2.1:

• A new Java producer for ease of implementation and enhanced performance.
• Delete topic support.
• Per topic configuration of preference for consistency over availability.
• Scala 2.11 support and dropping support for Scala 2.8.
• LZ4 compression.

What is new in version 0.8.0-beta1:

• Fixed existing perf tools.
• Wire existing producer and consumer to use the new ZK data structure.
• Create/delete ZK path for a topic in an admin tool.
• Implemented an embedded controller.

What is new in version 0.7.1:

• Avoid duplicated message during consumer rebalance
• Support configurable send / receive socket buffer size in server
• javaapi ZookeeperConsumerConnectorTest duplicates many tests in the scala version
• Make # of consumer rebalance retries configurable
• SyncProducer should log host and port if can't connect.
• Reduce duplicate messages served by the kafka consumer for uncompressed topics
• Avoid logging stacktrace directly
• Make backoff time during consumer rebalance configurable
• Improve log4j appender to use kafka.producer.Producer, and support zk.connect|broker.list options
• Separate out Kafka mirroring into a stand-alone app
• Hadoop producer should use software load balancer

What is new in version 0.7.0:

• Kafka now supports block level compression. See compression wiki for details. Currently only gzip is sup ported.
• Log retention can now depend on space (log.retention.size).

What is new in version 0.6:

• Fixed Bugs:
• SimpleProducer lose messages when socket gets an io exception
• If the producer sends an invalid MessageSet the broker will append it, corrupting the log
• ByteBufferMessageSet logs error about fetch size
• Shutdown Kafka when there is any disk IO error
• Propagate server all exceptions to consumer
• close() in SimpleConsumer should be synchronized
• Consumer Code documentation
• Improve EventHandler in AsyncProducer
• The zookeeper based Producer doesn't remove a dead broker from its list while serving a produce request
• Consumer logs ERROR during close
• Improvements:
• Clean-up jUnit test
• MessageSet does not implement Java iterable
• Default Encoder for Kafka log4 appender
• Explicitly name all threads
• Refactor public API into java and Scala API
• Improve API docs for all public APIs
• Remove the java FileMessageSet API
• Unit test fail on some Linux machines
• Create SBT sub projects for perf, examples, contrib that are currently using Ant
• Add JMX for changing log4j level dynamically at Kafka server
• Upgrade to Zookeeper 3.3.3
• javaapi MessageSet doesn't need to implement writeTo socket
• New Features:
• New producer API