Monitor Confluent Platform with Confluent for Kubernetes

Monitor your Confluent for Kubernetes (CFK) environment using the following tools and resources:

  • Confluent Health+ with Telemetry
  • JMX metrics monitoring integrations
  • Confluent Control Center (Legacy)

Confluent Health+

Confluent Health+ provides ongoing, real-time analysis of performance and configuration data for your Confluent Platform deployment. From this analysis, Health+ sends out notifications to alert users to potential environmental issues before they become critical problems.

For more information, see Confluent Health+.

Telemetry Reporter

The Confluent Telemetry Reporter is a plugin that runs inside each Confluent Platform service to push metadata about the service to Confluent. Telemetry Reporter enables product features based on the metadata, like Health+. Data is sent over HTTP using an encrypted connection.

Telemetry is disabled by default in CFK. You can enable and configure it globally at the CFK level.

When you globally enable Telemetry, you have an option to disable for specific Confluent Platform components.

Each Confluent Platform component CR provides the status condition, in status.conditions, whether Telemetry is enabled or disabled.

For more information and supported settings for Telemetry Reporter, see Confluent Telemetry Reporter.

For a list of the metrics that are collected for Health+, see Telemetry Reporter Metrics.

Globally configure Telemetry

To globally enable Telemetry Reporter for CFK and all Confluent Platform components:

  1. Set the followings in the CFK values file.

    telemetry:
  operator:
     enabled: true  --- [1]
  enabled: true     --- [2]
    • [1] Enable Telemetry Report for CFK.
    • [2] Enable Telemetry Report for all Confluent Platform components.

  2. Apply the change with the following command:

    helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values <path-to-values-file> \
  --namespace <namespace>

To globally configure the Telemetry Reporter settings for all Confluent Platform components:

  1. Set the following in the CFK values file:

    telemetry:
  secretRef:                 --- [1]
  directoryPathInContainer:  --- [2]

    • [1] [2] CFK supports the secretRef and directoryPathInContainer methods to load Telemetry configuration through Helm. Specify only one method.

      The Telemetry configuration should be specified in telemetry.txt, and the file must contain api.key and api.secret.

      If using a proxy, additional properties are required as shown below.

      api.key=<cloud_key>
api.secret=<cloud_secret>
proxy.url=<proxy_url>           # Only required if proxy is enabled
proxy.username=<proxy_username> # Only required if proxy requires credential
proxy.password=<proxy_password> # Only required if proxy requires credential

    • [1] secretRef takes the precedence over directoryPathInContainer if both are configured.

      The expected key is telemetry.txt.

      If the referenced secretRef is not read or data is not in the expected format, CFK will fail to start.

    • [2] Provide the directory path where telemetry.txt is present.

      If telemetry.txt is not in the expected format, CFK will fail to start.

    See Provide secrets for Confluent Platform application CR for providing the keys/values and required annotations when using Vault.

  2. To apply changes in Telemetry settings, in the referenced Secret, or in the telemetry.txt file, manually restart CFK and Confluent Platform:

Disable Telemetry for a Confluent Platform component

To disable Telemetry for a specific Confluent Platform component, set the following in the component CR and apply the change with the kubectl apply command:

telemetry:
  global: false

JMX Metrics

CFK deploys all Confluent components with JMX metrics enabled by default. These JMX metrics are made available on all pods at the following endpoints:

  • JMX metrics are available on port 7203 of each pod.

  • Jolokia (a REST interface for JMX metrics) is available on port 7777 of each pod.

  • JMX Prometheus exporter is available on port 7778.

    Authentication / encryption is not supported for Prometheus exporter in CFK.

Configure security on JMX metrics endpoints

By default, authentication and encryption are not enabled on JMX/Prometheus metric endpoints, but you have options to configure authentication and TLS for JMX/Prometheus metric endpoints in the component CR:

spec:
  metrics:
    authentication:
      type:                    --- [1]
    prometheus:                --- [2]
      rules:                   --- [3]
        - attrNameSnakeCase:
          cache:
          help:
          labels:
          name:
          pattern:
          type:
          value:
          valueFactor:
      blacklist:               --- [4]
      whitelist:               --- [5]
    tls:
      enabled:                 --- [6]

  • [1] Set to mtls for mTLS authentication.

    If you set this to mtls, you must set tls.enabled: true ([6]).

    Note that CFK does not currently support mTLS authentication for Prometheus even with this set to mtls.

  • [2] Specify Prometheus configurations to override the default settings.

    See Prometheus for more information about the rules, blacklist, and whitelist properties.

  • [3] A list of rules to apply.

    For example:

    spec:
  metrics:
    prometheus:
      rules:
        - pattern: "org.apache.kafka.metrics<type=(\w+), name=(\w+)><>Value: (\d+)"
          name: "kafka_$1_$2"
          value: "$3"
          valueFactor: "0.000001"
          labels:
            "$1": "$4"
            "$2": "$3"
          help: "Kafka metric $1 $2"
          cache: false
          type: "GAUGE"
          attrNameSnakeCase: false

  • [4] An array of patterns (in the string format) to identify what not to query.

    For example:

    spec:
  metrics:
    prometheus:
      blacklist:
      - "org.apache.kafka.metrics:*"

  • [5] An array of patterns (in the string format) to identify what to query.

    For example:

    spec:
  metrics:
    prometheus:
      whitelist:
      - "org.apache.kafka.metrics:type=ColumnFamily,*"

  • [6] If set to true, metrics are configured with global or component TLS as described in Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes.

    This setting is ignored for Prometheus. CFK currently does not support TLS for Prometheus.

Configure external access on JMX metrics

By default, CFK does not configure external access on JMX metric endpoints.

As an advanced configuration option, CFK supports enabling external access on JMX metrics endpoint for Kafka as an advanced configuration option. You need this setup only if the Jolokia HTTP-based solution is not enough.

Note that JMX metrics external access is currently only supported for Kafka, and not for other Confluent Platform components.

To enable external access for Kafka at the component CR level, use the annotation:

  1. Set up an Kubernetes external access service that allows incoming requests to an external FQDN to be routed to the pod’s JMX/Prometheus metric endpoint port. See Networking Service.

  2. Apply the annotation on the Kafka CR using a comma-separated list of the DNS (or IP) and the port you got from the previous step when you created a service.

    "platform.confluent.io/jmx-rmi-server-urls": "pod1.external-address:<port1>,pod2.external-address:<port2>, ... ,podn.external-address:<portn>"

    For example, using the external IP address of the node, 35.35.35.35 and nodePorts 30001 to 30003, the following CR configures a node port external access on the Kafka nodes:

    kind: Kafka
metadata:
  name: kafka
  namespace: operator
  annotations:
    "platform.confluent.io/jmx-rmi-server-urls": "35.35.35.35:30001,35.35.35.35:30002,35.35.35.35:30003"
spec:
  replica: 3

Configure Prometheus and Grafana

You can configure Prometheus to capture and aggregate JMX metrics from Confluent components. Then you configure Grafana to visualize those metrics in a dashboard.

For example configuration scenarios, see an example of monitoring with Prometheus and Grafana in jmx-monitoring-stacks and an example of monitoring with Grafana metrics dashboard.

Confluent Control Center

In Confluent for Kubernetes (CFK), you can configure and deploy Confluent Control Center to monitor Confluent Platform. Control Center is a web-based tool for managing and monitoring Confluent Platform. Control Center provides a user interface that enables developers and operators to:

  • Get a quick overview of cluster health
  • Observe and control messages, topics, and Schema Registry
  • Develop and run ksqlDB queries

For details about Control Center, see Confluent Control Center Overview.

Note

You cannot install Control Center on a host on an Internet Protocol version 6 (IPv6) network.

To deploy and use Control Center in CFK, you need to configure the following components:

The above components connect to one another as follows:

  • Kafka and KRaft connects to Control Center’s Prometheus service.
  • Control Center connects to Control Center’s Prometheus service.
  • Control Center connects to Control Center’s Alertmanager.
  • Prometheus connects to Alertmanager.

See Configure KRaft for Control Center for custom resource (CR) examples for Control Center, Kafka, and KRaft.

Configure Control Center

Control Center shares the same custom resource definition (CRD) as Confluent Control Center (Legacy).

See the Confluent Control Center (Legacy) sections, such as in Authenticaion, RBAC, TLS, Networking, Storage for additional configuration details.

For the Prometheus and Alertmanager services, Control Center supports basic and mTLS authentication methods and TLS encryption method.

Some of the TLS-related features, such as secrets in vaults, auto-generated certificates, or Java KeyStore format certificates, are not currently supported for Control Center.

Server-side configuration for Prometheus and Alertmanager

Configure the server-side configuration for Prometheus and Alertmanager:

kind: ControlCenter
spec:
  services:
    alertmanager:            --- [1]
      authentication:
        basic:               --- [2]
        type:                --- [3]
      image:                 --- [4]
      tls:                   --- [5]
    prometheus:              --- [6]
      authentication:
        basic:               --- [7]
        type:                --- [8]
      externalAccess:        --- [9]
      image:                 --- [10]
      tls:                   --- [11]
      pvc:
        dataVolumeCapacity:  --- [12]

  • [1] The server-side settings for the Alertmanager service.

  • [2] The configuration for basic authentication with the Alertmanager service. See Basic authentication for the available settings and credential configurations.

  • [3] The authentication type for the Alertmanager service. Valid options are basic and mtls.

  • [4] The image and the image tag of the Alertmanager service, confluentinc/cp-enterprise-alertmanager:<tag>.

    Use the same image tag specified in spec.image.application, which is the Control Center release number you are installing.

  • [5] TLS settings for the Alertmanager service. For the available settings, see Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes.

    Some of the TLS-related features, such as secrets in vaults, auto-generated certificates, or Java KeyStore format certificates, are not supported.

  • [6] The server-side settings for the Prometheus service.

  • [7] The configuration for basic authentication with the Prometheus service. See Basic authentication for the available settings and credential configurations.

  • [8] The authentication type for the Prometheus service. Valid options are basic and mtls.

  • [9] The external access method to use for Prometheus. See Configure Networking for Confluent Platform in Confluent for Kubernetes for the available options.

  • [10] The image and the image tag of the Prometheus service, confluentinc/cp-enterprise-prometheus:<tag>.

    Use the same image tag specified in spec.image.application, which is the Control Center release number you are installing.

  • [11] TLS settings for the Prometheus service. For the available settings, see Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes.

    Some of the TLS-related features, such as secrets in vaults, auto-generated certificates, or Java KeyStore formats certificates, are not supported.

  • [12] Required. The Prometheus data size.

Client-side configuration for Prometheus and Alertmanager

Configure the client-side configuration to access Prometheus and Alertmanager:

kind: ControlCenter
spec:
  dependencies:
    alertManagerClient:      --- [1]
      authentication:        --- [2]
      tls:                   --- [3]
      url:                   --- [4]
    prometheusClient:        --- [5]
      authentication:        --- [6]
      tls:                   --- [7]
      url:                   --- [8]
  • [1] The client-side settings to connect to the Alertmanager. The settings must match the server-side settings you configured in services.alertmanager.
  • [2] The client-side authentication settings for the Alertmanager. See Confluent Platform authentication for available settings and credential configurations.
  • [3] The client-side TLS configuration for the Alertmanager.
  • [4] Required. The URL endpoint of the Alertmanager service in the <control-center-instance-name>.<control-center-cluster-name>.<namespace>.svc.cluster.local:9093 pattern.
  • [5] The client-side settings to connect to the Prometheus service. The settings must match the server-side settings you configured in services.prometheus ([6]).
  • [6] The client-side authentication settings for the Prometheus service. See Confluent Platform authentication for available settings and credential configurations.
  • [7] The client-side TLS configuration for the Prometheus service.
  • [8] Required. The URL endpoint of the Prometheus service in the <control-center-instance-name>.<control-center-cluster-name>.<namespace>.svc.cluster.local:9090 pattern.

Configure Kafka for Control Center

Configure Kafka to send data to the Prometheus service in Control Center.

kind: Kafka
spec:
  dependencies:
    metricsClient:
      authentication:  --- [1]
      tls:             --- [2]
      url:             --- [3]
  • [1] The authentication settings for the Prometheus service. The settings must match the server-side authentication settings you configured in the ControlCenter CR in the previous section.
  • [2] The client-side TLS settings for the Prometheus service.
  • [3] Required. The URL endpoint of the Prometheus service.

The following is a sample Kafka CR for Control Center integration:

kind: Kafka
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
  image:
    application: docker.io/confluentinc/cp-server:7.9.1
    init: confluentinc/confluent-init-container:2.11.1
  replicas: 3

Configure KRaft for Control Center

Configure KRaft to send data to the Prometheus service in Control Center.

kind: KRaftController
spec:
  dependencies:
    metricsClient:
      authentication:  --- [1]
      tls:             --- [2]
      url:             --- [3]
  • [1] The authentication settings for the Prometheus service. The settings must match the server-side authentication settings you configured in the ControlCenter CR in the previous section.
  • [2] The client-side TLS settings for the Prometheus service.
  • [3] Required. The URL endpoint of the Prometheus service.
kind: KRaftController
spec:
  image:
    application: docker.io/confluentinc/cp-server:7.9.1
    init: confluentinc/confluent-init-container:2.11.1

Sample CR snippets of Control Center, Kafka, and KRaft

This section contains snippets of sample CRs to configure Control Center with supported authentication.

mTLS authentication to Prometheus

kind: KRaftController
spec:
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: Kafka
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: ControlCenter
spec:
  image:
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: alertmanager-client-tls
  services:
    prometheus:
      image: confluentinc/cp-enterprise-prometheus:2.0.0
      authentication:
        type: mtls
      tls:
        secretRef: prometheus-tls
      pvc:
        dataVolumeCapacity: 10Gi
    alertmanager:
      image: confluentinc/cp-enterprise-alertmanager:2.0.0
      authentication:
        type: mtls
      tls:
        secretRef: alertmanager-tls
---

Basic authentication and TLS to Prometheus

kind: KRaftController
spec:
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: Kafka
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
---
kind: ControlCenter
spec:
  image:
    application: confluentinc/cp-enterprise-control-center-next-gen:2.0.0
    init: confluentinc/confluent-init-container:v0.1208.3
    pullSecretRef:
      - confluent-registry
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      authentication:
        type: basic
        basic:
          secretRef: prometheus-client-creds
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      authentication:
        type: basic
        basic:
          secretRef: alertmanager-client-creds
      tls:
        enabled: true
        secretRef: alertmanager-client-tls
  services:
    prometheus:
      image: confluentinc/cp-enterprise-prometheus:2.0.0
      authentication:
        type: basic
        basic:
          secretRef: prometheus-credentials
      tls:
        secretRef: prometheus-tls
      pvc:
        dataVolumeCapacity: 10Gi
    alertmanager:
      image: confluentinc/cp-enterprise-alertmanager:2.0.0
      authentication:
        type: basic
        basic:
         secretRef: alertmanager-credentials
      tls:
        secretRef: alertmanager-tls
---

TLS-enabled Control Center

The following is a sample Control Center CR with TLS enabled and deployed with Confluent Platform 7.9.0:

kind: KRaftController
metadata:
  name: kraftcontroller
  namespace: operator
spec:
  configOverrides:
    jvm:
      - "-Djavax.net.ssl.trustStore=/mnt/jvmtruststore/truststore.jks"
      - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
  mountedVolumes:
    volumes:
      - name: customtruststore
        secret:
          secretName: mycustomtruststore
    volumeMounts:
      - name: customtruststore
        mountPath: /mnt/jvmtruststore
  dataVolumeCapacity: 10G
  image:
    application: confluentinc/cp-server:7.9.0
    init: confluentinc/confluent-init-container:2.11.0
  replicas: 3
  dependencies:
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
---
kind: Kafka
metadata:
  name: kafka
  namespace: operator
spec:
  configOverrides:
    jvm:
      - "-Djavax.net.ssl.trustStore=/mnt/jvmtruststore/truststore.jks"
      - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
  mountedVolumes:
    volumes:
      - name: customtruststore
        secret:
          secretName: mycustomtruststore
    volumeMounts:
      - name: customtruststore
        mountPath: /mnt/jvmtruststore
  dependencies:
    kRaftController:
      clusterRef:
        name: kraftcontroller
    metricsClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
  dataVolumeCapacity: 10G
  image:
    application: confluentinc/cp-server:7.9.0
    init: confluentinc/confluent-init-container:2.11.0
  replicas: 3
---
kind: ControlCenter
metadata:
  name: controlcenter-next-gen
  namespace: operator
spec:
  replicas: 1
  image:
    application: confluentinc/cp-enterprise-control-center-next-gen:2.0.0
    init: confluentinc/confluent-init-container:2.11.0
    pullSecretRef:
      - confluent-registry
  dataVolumeCapacity: 10Gi
  dependencies:
    prometheusClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9090
      tls:
        enabled: true
        secretRef: prometheus-client-tls
    pvc:
      dataVolumeCapacity: 10Gi
    alertManagerClient:
      url: https://controlcenter-next-gen.operator.svc.cluster.local:9093
      tls:
        enabled: true
        secretRef: alertmanager-client-tls

Confluent Control Center (Legacy)

Confluent Control Center (Legacy) is a web-based tool for managing and monitoring Confluent Platform. Control Center (Legacy) provides a user interface that enables developers and operators to:

  • Get a quick overview of cluster health
  • Observe and control messages, topics, and Schema Registry
  • Develop and run ksqlDB queries

For the metrics available for monitoring, see Metrics available in Control Center.

Configure Control Center (Legacy) to monitor Kafka clusters

The Confluent Metrics Reporter collects various metrics from an Apache Kafka® cluster. Control Center (Legacy) then uses those metrics to provide a detailed monitoring view of the Kafka cluster.

By default, the Confluent Metrics Reporter is enabled and configured to send Kafka metrics to a set of topics on the same Kafka cluster.

To send metrics to a different cluster, or to configure specific authentication settings, configure the Kafka custom resource (CR):

kind: Kafka
spec:
  metricReporter:
    enabled:                       --- [1]
    authentication:
      type:                        --- [2]
      jaasConfigPassThrough:
        secretRef:                 --- [3]
        directoryPathInContainer:  --- [4]
    tls:
      enabled:                     --- [5]

  • [1] Set to true or false to enable or disable the metrics reporting.

  • [2] Set to the authentication type to use for Kafka. See Configure authentication to access Kafka for details.

  • [3] Set to the Kubernetes Secret name used to authenticate to Kafka.

  • [4] Set to the directory path in the Kafka container where the Kafka authentication credentials are injected by Vault.

    See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.

  • [5] Set to true if the Kafka cluster has TLS network encryption enabled.

Once Confluent Metrics Reporter is set up for a Kafka cluster, configure Control Center (Legacy) to monitor the cluster.

By default, Control Center (Legacy) is set up to monitor the Kafka cluster it is using to store its own state. This Kafka cluster is defined using spec.dependencies.kafka in the Confluent Control Center (Legacy) CR.

If there is another Kafka cluster to monitor, you can configure that in the Control Center (Legacy) CR as below:

kind: ControlCenter
spec:
  monitoringKafkaClusters:
  - name:                         --- [1]
    bootstrapEndpoint:            --- [2]
    authentication:
      type:                       —-- [3]
      jaasConfig:                 —-- [4]
      jaasConfigPassThrough:      —-- [5]
      oauthbearer:                —-- [6]
        secretRef:                —-- [7]
        directoryPathInContainer: --- [8]
    tls:
      enabled:                    —-- [9]

  • [1] Set to Kafka cluster name.

  • [2] Set to the Kafka bootstrap endpoint.

  • [3] Set to the Kafka client authentication type.

    When RBAC is not enabled, valid options are plain and mtls.

    When RBAC is enabled, the only valid option is oauthbearer.

  • [4] [5] For authenticating to a Kafka cluster using SASL/PLAIN, see Client-side SASL/PLAIN authentication for Kafka.

  • [6] When Confluent Control Center (Legacy) authorization type is set to RBAC (spec.authorization.type: rbac) and the authentication type is set to oauthbearer in [3], use the OAuth method to authenticate to the Kafka cluster.

  • [7] The username and password are loaded through secretRef. The expected key is bearer.txt, and the value for the key is:

    username=<username>
password=<password>

    An example command to create a secret to use for this property:

    kubectl create secret generic oauth-client \
  --from-file=bearer.txt=/some/path/bearer.txt \
  --namespace confluent

  • [8] The directory in the Confluent Control Center (Legacy) container where the expected Bearer credentials are injected by vault. See above ([7]) for the expected format.

    See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.

  • [9] For authenticating to a Kafka cluster using mTLS, see Client-side mTLS authentication for Kafka.

Configure Control Center (Legacy) to monitor remote Kafka clusters

To monitor a Kafka cluster in a different Kubernetes cluster:

  1. Configure the Control Center (Legacy) CR as described in monitoring additional Kafka clusters.

    When RBAC is enabled, you must set authentication.type to oauthbearer, and provide oauthbearer credentials in the Control Center (Legacy) CR.

  2. Configure the replication listener with external access on the remote Kafka as described in Configure Kafka in MRC with external access URLs.

    If RBAC is enabled on the remote Kafka cluster, the Kafka token listener piggy backs on the replication listener.

  3. Configure DNS where Control Center (Legacy) runs to be able to resolve the Kafka replication listener endpoint.

Configure Control Center (Legacy) to monitor ksqlDB, Connect and Schema Registry clusters

You can configure Control Center (Legacy) to provide a detailed monitoring or management view of ksqlDB, Connect, and Schema Registry clusters.

The following is an example of the dependencies section in a Control Center (Legacy) CR. The example connects two Schema Registry clusters, two ksqlDB clusters, and two Connect clusters to Control Center (Legacy):

kind: ControlCenter
spec:
 dependencies:
   schemaRegistry:
     url: https://schemaregistry.confluent.svc.cluster.local:8081
     tls:
       enabled: true
     authentication:
       type: mtls
     clusters:
     - name: schemaregistry-dev
       url: https://schemaregistry-dev.confluent.svc.cluster.local:8081
       tls:
         enabled: true
       authentication:
        type: mtls
   ksqldb:
   - name: ksql-dev
     url: https://ksqldb.confluent.svc.cluster.local:8088
     tls:
       enabled: true
     authentication:
       type: mtls
   - name: ksql-dev1
     url: https://ksqldb-dev.confluent.svc.cluster.local:8088
     tls:
       enabled: true
     authentication:
       type: mtls
   connect:
   - name: connect-dev
     url: https://connect.confluent.svc.cluster.local:8083
     tls:
       enabled: true
     authentication:
       type: mtls
   - name: connect-dev2
     url: https://connect-dev.confluent.svc.cluster.local:8083
     tls:
       enabled: true
     authentication:
       type: mtls

For an example scenario to configure Confluent Control Center (Legacy) to monitor multiple ksqlDB, Connect, and Schema Registry clusters, see Connect Control Center to Multiple Connect, ksqlDB, and Schema Registry Clusters.