HELM 添加仓库与Kafka集群安装

背景简介

由于需要自搭建 kafka集群，准备采用 helm chart直接部署，因此记录如何添加 helm 仓库。

环境配置

1. 系统：Debian 6.1.66-1 (2023-12-09) x86_64 GNU/Linux
2. helm v3.14.0

详细步骤

搜索 kafka 仓库

myserver@peag-k8s-master:~$ helm search hub kafka -o json

json 返回值格式化后如下

[
   {
      "app_version": "v3.5.1",
      "description": "Helm chart for Apache Kafka.",
      "repository": {
         "name": "kafka",
         "url": "https://helm-charts.itboon.top/kafka/"
      },
      "url": "https://artifacthub.io/packages/helm/kafka/kafka",
      "version": "13.2.1"
   },
   ******
      {
      "app_version": "0.9.68",
      "description": "A multi-protocol edge/service proxy for interfacing apps and services to streaming data.",
      "repository": {
         "name": "zilla",
         "url": "oci://ghcr.io/aklivity/charts/zilla"
      },
      "url": "https://artifacthub.io/packages/helm/zilla/zilla",
      "version": "0.9.68"
   }
]

选择自己需要的仓库并添加，样例如下：

myserver@peag-k8s-master:~$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

仓库添加完成后，搜索kafka

myserver@peag-k8s-master:~$ helm search repo kafka
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                       
bitnami/kafka                   26.11.3         3.6.1           Apache Kafka is a distributed streaming platfor...
bitnami/dataplatform-bp2        12.0.5          1.0.1           DEPRECATED This Helm chart can be used for the ...
bitnami/schema-registry         16.9.3          7.5.3           Confluent Schema Registry provides a RESTful in...

也可继续查看kafka版本

myserver@peag-k8s-master:~$ helm search repo bitnami/kafka -l
NAME            CHART VERSION   APP VERSION     DESCRIPTION                                       
bitnami/kafka   26.11.3         3.6.1           Apache Kafka is a distributed streaming platfor...
bitnami/kafka   26.11.2         3.6.1           Apache Kafka is a distributed streaming platfor...
bitnami/kafka   26.11.1         3.6.1           Apache Kafka is a distributed streaming platfor...
bitnami/kafka   26.10.0         3.6.1           Apache Kafka is a distributed streaming platfor...

获取 value 文件并按需求更新

myserver@peag-k8s-master:~$ helm inspect values bitnami/kafka > kafka.yaml
myserver@peag-k8s-master:~$ ls
kafka.yaml

请按需更新，如无需更新，直接安装即可

准备数据持久化 PV

---
# pv
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pega23-install-kafka-pv
spec:
  capacity:
    storage: 1024Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /vm-server/dev/pega23/data/kafka/pv

安装 kafka集群

myserver@peag-k8s-master:~/pega23-demo/kafka$ helm install pega23-demo-kafka bitnami/kafka -n pega23-demo -f kafka.yaml 
NAME: pega23-demo-kafka
LAST DEPLOYED: Wed Feb 28 15:59:24 2024
NAMESPACE: pega23-demo
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kafka
CHART VERSION: 26.11.3
APP VERSION: 3.6.1

** Please be patient while the chart is being deployed **

Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:

    pega23-demo-kafka.pega23-demo.svc.cluster.local

Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

    pega23-demo-kafka-controller-0.pega23-demo-kafka-controller-headless.pega23-demo.svc.cluster.local:9092
    pega23-demo-kafka-controller-1.pega23-demo-kafka-controller-headless.pega23-demo.svc.cluster.local:9092
    pega23-demo-kafka-controller-2.pega23-demo-kafka-controller-headless.pega23-demo.svc.cluster.local:9092

The CLIENT listener for Kafka client connections from within your cluster have been configured with the following security settings:
    - SASL authentication

To connect a client to your Kafka, you need to create the 'client.properties' configuration files with the content below:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="user1" \
    password="$(kubectl get secret pega23-demo-kafka-user-passwords --namespace pega23-demo -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1)";

To create a pod that you can use as a Kafka client run the following commands:

    kubectl run pega23-demo-kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.6.1-debian-12-r12 --namespace pega23-demo --command -- sleep infinity
    kubectl cp --namespace pega23-demo /path/to/client.properties pega23-demo-kafka-client:/tmp/client.properties
    kubectl exec --tty -i pega23-demo-kafka-client --namespace pega23-demo -- bash

    PRODUCER:
        kafka-console-producer.sh \
            --producer.config /tmp/client.properties \
            --broker-list pega23-demo-kafka-controller-0.pega23-demo-kafka-controller-headless.pega23-demo.svc.cluster.local:9092,pega23-demo-kafka-controller-1.pega23-demo-kafka-controller-headless.pega23-demo.svc.cluster.local:9092,pega23-demo-kafka-controller-2.pega23-demo-kafka-controller-headless.pega23-demo.svc.cluster.local:9092 \
            --topic test

    CONSUMER:
        kafka-console-consumer.sh \
            --consumer.config /tmp/client.properties \
            --bootstrap-server pega23-demo-kafka.pega23-demo.svc.cluster.local:9092 \
            --topic test \
            --from-beginning

WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
  - controller.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/