Install Confluent Platform using ZIP and TAR Archives¶

This topic provides instructions for running Confluent Platform locally or running a production-ready Confluent Platform configuration in a multi-node environment.

With the production-ready installation method, you will connect to every node manually, download the archive, and run the Confluent Platform installation commands.

Ready to build?

Sign up for Confluent Cloud with your cloud marketplace account and unlock $1000 in free credits: AWS Marketplace, Google Cloud Marketplace, or Microsoft Azure Marketplace.

Prerequisites¶

You must complete these steps for each node in your cluster.
Before installing Confluent Platform, your environment must meet the prerequisites as described in software and hardware requirements.

Get the software¶

Go to the installation page.
Select the SELF-MANAGED tab. You will be prompted to enter your name, company, email address, and country. When finished, click Download Free.

On the installation page, choose your package type (ZIP or TAR) and click Download, or download a package directly by using the curl command.

Confluent Platform

ZIP

curl -O https://packages.confluent.io/archive/8.0/confluent-8.0.0.zip

TAR

curl -O https://packages.confluent.io/archive/8.0/confluent-8.0.0.tar.gz

Confluent Platform using only Confluent Community components

ZIP

curl -O https://packages.confluent.io/archive/8.0/confluent-community-8.0.0.zip

TAR

curl -O https://packages.confluent.io/archive/8.0/confluent-community-8.0.0.tar.gz

Extract the contents of the archive. For ZIP files, run this command in a terminal.

unzip confluent-8.0.0.zip

For TAR files run this command:

tar xzf confluent-8.0.0.tar.gz

You should have these directories:

Folder	Description
/bin/	Driver scripts for starting and stopping services
/etc/	Configuration files
/lib/	Systemd services
/libexec/	Multi-platform CLI binaries
/share/	Jars and licenses
/src/	Source files that require a platform-dependent build

Configure CONFLUENT_HOME and PATH¶

To more easily use the Confluent CLI and all of the command-line tools that are provided with Confluent Platform, you can optionally configure the CONFLUENT_HOME variable and add the Confluent Platform \bin folder to your PATH. Then you can use the CLI tools without navigating to the CONFLUENT_HOME directory.

Set the environment variable for the Confluent Platform home directory. This is the directory where Confluent Platform is installed. For example:
```
export CONFLUENT_HOME=~/confluent-8.0.0
```
Add the Confluent Platform bin directory to your PATH
```
export PATH=$PATH:$CONFLUENT_HOME/bin
```
Test that you set the CONFLUENT_HOME variable correctly by running the confluent command:
```
confluent --help
```
Your output should show the available commands for managing Confluent Platform.

Tip

For a tutorial that shows you how to configure and run multiple clusters locally, for learning and development purposes, see Tutorial: Set Up a Multi-Broker Kafka Cluster.

Configure Confluent Platform for production¶

Tip

You can store passwords and other configuration data securely by using the confluent secret commands. For more information, see Manage Secrets in Confluent Platform.

Configure Confluent Platform with the individual component properties files. By default these are located in CONFLUENT_HOME/etc/. You must minimally configure the following components.

Kafka¶

For Kafka in KRaft mode, you must configure a node to be a broker or a controller. In addition, you must create a unique cluster ID and format the log directories with that ID.

Typically in a production environment, you should have a minimum of three brokers and three controllers.

Navigate to the Kafka properties file for KRaft (find example configuration files under /etc/kafka/) and customize the following:
Configure the process.roles, node.id and controller.quorum.voters for each node.
- For process.roles, set whether the node will be a broker or a controller. combined mode, meaning process.roles is set to broker,controller, is currently not supported and should only be used for experimentation.
- Set a system-wide unique ID for the node.id for each broker/controller.
- controller.quorum.voters should be a comma-separated list of controllers in the format nodeID@hostname:port
```
############################# Server Basics #############################

# The role of this server. Setting this puts us in KRaft mode
process.roles=broker

# The node id associated with this instance's roles
node.id=2

# The connect string for the controller quorum
controller.quorum.voters=1@controller1:9093,3@controller3:9093,5@controller5:9093
```
Configure how brokers and clients communicate with the broker using listeners, and where controllers listen with controller.listener.names.
- listeners: Comma-separated list of URIs and listener names to listen on in the format listener_name://host_name:port
- controller.listener.names: Comma-separated list of listener_name entries for listeners used by the controller.
For more information, see KRaft Configuration for Confluent Platform.
Configure security for your environment.
- For general security guidance, see KRaft Security in Confluent Platform.
- For role-based access control (RBAC), see Configure Metadata Service (MDS) in Confluent Platform.
- For configuring SASL/SCRAM for broker-to-broker communication, see KRaft-based Confluent Platform clusters.

Control Center¶

Navigate to the Control Center properties file (/etc/confluent-control-center/control-center-production.properties) and customize the following:
```
# host/port pairs to use for establishing the initial connection to the Kafka cluster
bootstrap.servers=<hostname1:port1,hostname2:port2,hostname3:port3,...>
# location for Control Center data
confluent.controlcenter.data.dir=/var/lib/confluent/control-center
# the Confluent license
confluent.license=<your-confluent-license>
```
This configuration is for a three node multi-node cluster. For more information, see Control Center Configuration. For information about Confluent Platform licenses, see Manage Confluent Platform Licenses Using Control Center.

Navigate to the Kafka server configuration file and enable Confluent Metrics Reporter.

##################### Confluent Metrics Reporter #######################
# Confluent Control Center and Confluent Auto Data Balancer integration
#
# Uncomment the following lines to publish monitoring data for
# Confluent Control Center and Confluent Auto Data Balancer
# If you are using a dedicated metrics cluster, also adjust the settings
# to point to your metrics Kafka cluster.
metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
confluent.metrics.reporter.bootstrap.servers=localhost:9092
#
# Uncomment the following line if the metrics cluster has a single broker
confluent.metrics.reporter.topic.replicas=1

Schema Registry¶

Navigate to the Schema Registry properties file (/etc/schema-registry/schema-registry.properties) and specify the following properties:

# Specify the address the socket server listens on, e.g. listeners = PLAINTEXT://your.host.name:9092
listeners=http://0.0.0.0:8081

# The advertised host name. This must be specified if you are running Schema Registry
# with multiple nodes.
host.name=192.168.50.1

# List of Kafka brokers to connect to, e.g. PLAINTEXT://hostname:9092,SSL://hostname2:9092
kafkastore.bootstrap.servers=PLAINTEXT://hostname:9092,SSL://hostname2:9092

This configuration is for a three node multi-node cluster. For more information, see Deploy Schema Registry in Production on Confluent Platform.

Start Confluent Platform¶

To start Confluent Platform, make sure you have configured CONFLUENT_HOME and then use the following Kafka CLI commands. In addition, you must format the log directories with a unique cluster ID before you start Confluent Platform.

You should start your KRaft controllers first, then Kafka brokers, and then Schema Registry must be started before any other components.

Before you start Kafka, you must use the kafka-storage tool with the random-uuid command to generate a cluster ID for each new cluster. You only need one cluster ID, which you will use to format each node in the cluster.
```
bin/kafka-storage random-uuid
```
This results in output like the following:
```
q1Sh-9_ISia_zwGINzRvyQ
```
Then use the cluster ID to format storage for each node in the cluster with the kafka-storage tool that is provided with Confluent Platform, and the format command like the following example, specifying the properties file for a controller.
```
bin/kafka-storage format -t q1Sh-9_ISia_zwGINzRvyQ -c etc/kafka/controller.properties
```
Previously, Kafka would format blank storage directories automatically and generate a new cluster ID automatically. One reason for the change is that auto-formatting can sometimes obscure an error condition. This is particularly important for the metadata log maintained by the controller and broker servers. If a majority of the controllers were able to start with an empty log directory, a leader might be able to be elected with missing committed data. To configure the log directory, either set metadata.log.dir or log.dirs. For more information, see Inter-broker listeners.

Start a Kafka controller. Run this command in its own terminal.

kafka-server-start $CONFLUENT_HOME/etc/kafka/controller.properties

Start a Kafka broker.

kafka-server-start $CONFLUENT_HOME/etc/kafka/broker.properties

Start Schema Registry. Run this command in its own terminal.

schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties

Start other Confluent Platform components as desired.

Control Center

$CONTROL_CENTER_HOME refers to the directory where you extracted the Control Center archive.
```
control-center-start $CONTROL_CENTER_HOME/etc/confluent-control-center/control-center.properties
```

Kafka Connect

connect-distributed $CONFLUENT_HOME/etc/schema-registry/connect-avro-distributed.properties

Confluent REST Proxy

kafka-rest-start $CONFLUENT_HOME/etc/kafka-rest/kafka-rest.properties

ksqlDB

ksql-server-start $CONFLUENT_HOME/etc/ksqldb/ksql-server.properties

Uninstall¶

Remove the Confluent directory. For example, if you have Confluent Platform 8.0.0 installed:
```
rm -rf $CONFLUENT_HOME
```

Remove the Confluent Platform data files.

rm -rf /var/lib/<confluent-platform-data-files>