To allow for a gradual migration from non-secure to secure connections, the Kafka cluster must support both connection types simultaneously. The correct way to achieve this is by configuring multiple listeners on the brokers. By adding a new listener configured for SSL on a different port, the existing PLAINTEXT listener can remain active. This allows existing applications to continue connecting without interruption while new or updated applications can be configured to use the new, secure SSL listener. This approach provides the necessary transition period without causing a service outage for legacy clients.

A. Enable SSL on the current Listener, and do not enable mTLS.

Modifying the existing listener to use SSL would immediately break all current non-SSL client connections, violating the requirement for a gradual migration.

B. Modify the advertised listeners setting on brokers to use SSL.

The advertised.listeners setting tells clients how to connect. Changing this without adding a corresponding SSL listener will cause connection failures and does not enable coexistence.

D. Enable SSL on the current listener, and do not implement SSL on application side.

This is a technically non-functional scenario. If a broker listener requires SSL, a client application cannot connect to it without also being configured for SSL.

1. Confluent Documentation, "Encryption and Authentication with SSL for Apache Kafka": In the "Configuring Kafka Brokers" section, the documentation explicitly states how to configure multiple listeners. It provides an example: listeners=PLAINTEXT://kafka1:9092,SSL://kafka1:9093. The text clarifies, "This allows clients to connect to the broker using either plaintext or SSL," which is the exact requirement for the migration scenario.

Source: Confluent Documentation, Security > Encryption and Authentication with SSL for Apache Kafka > Configuring Kafka Brokers.

2. Apache Kafka Documentation, "Security": Section 3.3, "Configuring Kafka Brokers," describes the listeners configuration property. It specifies that this property is a "comma-separated list of listeners" and provides an example of configuring both PLAINTEXT and SSL listeners on different ports (listeners = PLAINTEXT://:9092,SSL://:9093), confirming this is the standard method for supporting multiple protocols.

Source: Apache Kafka Documentation, Section 3. Security > 3.3 Encryption and Authentication using SSL.

The kafka-consumer-groups.sh command is the standard administrative tool for managing and inspecting consumer groups. When used with the --describe flag, it provides detailed information about a consumer group, including the current offset for each partition, the log-end-offset (the offset of the last message), and the calculated lag. Lag is the difference between the log-end-offset and the consumer's current offset, indicating how far behind the consumer is. This makes it the correct tool for quickly checking consumer lag as requested in the scenario.

A. bin/kafka-consumer-lag.sh: This is not a standard command-line tool provided with Apache Kafka or the Confluent Platform. It does not exist in the official distribution.

C. bin/kafka-consumer-group-throughput.sh: This command does not exist in the standard Apache Kafka or Confluent Platform toolset. Throughput is a different metric from lag.

D. bin/kafka-reassign-partitions.sh: This tool is used for cluster administration, specifically for moving topic partitions between brokers. It is not used for monitoring consumer performance or lag.

1. Confluent Documentation: "kafka-consumer-groups". The documentation explicitly states that the --describe option can be used to "list the topic partitions, current-offset, log-end-offset, and lag for a consumer group."

Source: Confluent Platform Documentation, CLI Tools, kafka-consumer-groups, "Describe a consumer group" section.

2. Apache Kafka Documentation: "Kafka Operations - Consumer Groups". The official Apache Kafka documentation details the usage of kafka-consumer-groups.sh. The "Describing a Consumer Group" section shows an example output with columns for CURRENT-OFFSET, LOG-END-OFFSET, and LAG.

Source: Apache Kafka 3.6 Documentation, Section 4.4 "Consumer Groups".

3. Gwen Shapira, Todd Palino, Rajini Sivaram (2021): Kafka: The Definitive Guide, 2nd Edition. Chapter 11, "Administering Kafka," describes the kafka-consumer-groups.sh script as the primary tool for inspecting consumer group state, including offsets and lag.

Source: O'Reilly Media, ISBN: 978-1098108035, Chapter 11, "Consumer Groups" section.

Apache Kafka provides two primary mechanisms for data retention, configured via the cleanup.policy topic-level setting. The delete policy, which is the default, discards entire log segments after a configured retention period (time-based) or size limit is reached. This is suitable for transient data. The compact policy enables log compaction, which retains at least the last known value for each message key within a partition. This is ideal for use cases requiring a full snapshot of the latest state, such as restoring a database. It is also possible to set the policy to compact,delete to both compact the log and then delete records older than a certain age.

B. default is not a valid value for the cleanup.policy configuration. The policy that is used by default is delete.

D. cleanup is part of the configuration parameter's name (cleanup.policy), not a valid value for the policy itself.

1. Confluent Documentation: In the "Topic-Level Configs" section, the cleanup.policy parameter is defined. It explicitly states, "A string that is either delete, compact, or both (compact,delete). This string designates the retention policy to use on old log segments."

Source: Confluent Documentation, "Server Configuration Properties," Section: cleanup.policy.

2. Apache Kafka Documentation: The official Apache Kafka documentation corroborates this in its "Topic Configs" section. It specifies the valid values for cleanup.policy are "delete, compact".

Source: Apache Kafka Documentation, "3.7 Configuration," Section: "Topic Configs," Parameter: cleanup.policy.

3. Kleppmann, M. (2017). Designing Data-Intensive Applications. O'Reilly Media. Chapter 11 discusses log-structured storage and messaging systems. It details the concepts of log compaction (as used in Kafka's compact policy) and time-based retention (as used in the delete policy) as fundamental strategies for managing log data.

Source: Chapter 11, "Stream Processing," Section: "Messaging Systems." (Note: While a book, it is a peer-reviewed and highly respected academic-level text in the field).

The setting unclean.leader.election.enable directly addresses the trade-off between availability and data consistency. When a partition leader fails, Kafka normally elects a new leader from the set of in-sync replicas (ISRs). If no ISRs are available, the partition becomes unavailable for writes. By enabling unclean leader election, Kafka is permitted to elect a leader from out-of-sync replicas as a last resort. This keeps the partition available (high availability), but at the cost of potentially losing messages that the new leader had not yet replicated (lower message reliability).

B. An odd number of brokers is a best practice for the fault tolerance of the consensus protocol (ZooKeeper or KRaft controllers), not a direct trade-off between availability and message reliability.

C. linger.ms is a producer configuration that balances latency and throughput by batching records. It does not influence the cluster's availability during a broker failure.

D. retention.ms determines how long messages are stored. Setting it to -1 (infinite retention) enhances data durability but does not impact the system's availability during a leader failure.

1. Apache Kafka Documentation, Broker Configs: The official documentation for the unclean.leader.election.enable parameter states: "Indicates whether to enable replicas that are not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss." This explicitly describes the trade-off.

Source: Apache Kafka Documentation, Section 3.3 Broker Configs, unclean.leader.election.enable.

2. Confluent Documentation, High Availability: "If all replicas for a partition are unavailable, Kafka by default will wait for a replica to come back online... This default behavior prefers avoiding data loss over availability. This preference can be changed... by setting the broker property unclean.leader.election.enable=true... This trades durability for availability."

Source: Confluent Documentation, "High Availability" section within the Apache Kafka Technical Overview.

3. Designing Event-Driven Systems, by Ben Stopford (O'Reilly, 2018): "There is one final configuration parameter that affects the availability/durability trade-off: unclean.leader.election.enable. If this is set to true, it will allow a nonsynchronized replica to become the leader. This means you will have a leader, but you will have lost data."

Source: Chapter 10, "Building a Reliable System with Kafka," Section on "Durability vs. Availability." (Note: Ben Stopford is a technologist at Confluent).

In any Apache Kafka cluster, there is always exactly one active Controller at any given time. This Controller is one of the brokers, elected through a consensus mechanism (like ZooKeeper or KRaft). Its primary responsibility is to manage the cluster's state, including partition leadership, broker failures, and topic creation/deletion. The number of brokers, topics, partitions, or consumers in the cluster does not alter this fundamental architectural principle of having a single active Controller to ensure consistent metadata management and avoid "split-brain" scenarios.

B. two: A Kafka cluster is designed with a single active controller to prevent conflicting state changes and ensure metadata consistency.

C. three: Having multiple active controllers is not supported by Kafka's architecture as it would lead to an inconsistent and unstable cluster state.

D. Four: While any of the four brokers is eligible to become the controller, only one broker is elected to be active in that role at any time.

1. Confluent Documentation, Apache Kafka® Internals: "One of the brokers in the cluster is designated as the controller. The controller is responsible for electing a leader for each partition." This statement confirms the single-controller design.

Source: Confluent Documentation, "Apache Kafka Internals" section.

2. Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A Distributed Messaging System for Log Processing. In Proceedings of the NetDB '11: 6th International Workshop on Networking Meets Databases. "One of the brokers is elected as the 'controller'. This controller detects broker failures and is responsible for electing new leaders for all partitions on the failed broker."

Source: Section 3.2, "Distribution".

3. Narkhede, N., Shapira, G., & Palino, T. (2017). Kafka: The Definitive Guide. O'Reilly Media. "The controller is a broker that has additional responsibilities... At any given time, there is only one controller in a Kafka cluster."

Source: Chapter 6, "Data Delivery", section "Replication".