k8s-7: kafka+zookeeper的单节点与集群的持久化

article/2025/10/31 7:18:13

之前在k8s环境中有这个需求，看了好多的文档，都有坑，踩了一边总结一下，需要的朋友可自取

一、单节点部署

建议开发环境使用，且此处采用动态挂载的，生产不建议

1、安装zk

cat > zk.yaml <<EOF
apiVersion: v1
kind: Service
metadata:labels:app: zookeeper-servicename: zookeeper-service
spec:ports:- name: zookeeper-portport: 2181targetPort: 2181selector:app: zookeeper
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: zookeepername: zookeeper
spec:replicas: 1selector:matchLabels:app: zookeepertemplate:metadata:labels:app: zookeeperspec:containers:- image: wurstmeister/zookeeperimagePullPolicy: IfNotPresentname: zookeeperports:- containerPort: 2181
EOF

2.安装kafka

cat > kafka.yaml <<EOF
apiVersion: v1
kind: Service
metadata:name: kafka-servicelabels:app: kafka
spec:type: NodePortports:- port: 9092name: kafka-porttargetPort: 9092nodePort: 30092protocol: TCPselector:app: kafka---apiVersion: apps/v1
kind: Deployment
metadata:name: kafkalabels:app: kafka
spec:replicas: 1selector:matchLabels:app: kafkatemplate:metadata:labels:app: kafkaspec:containers:- name: kafkaimage: wurstmeister/kafkaimagePullPolicy: IfNotPresentports:- containerPort: 9092env:- name: KAFKA_ADVERTISED_PORTvalue: "9092"- name: KAFKA_ADVERTISED_HOST_NAMEvalue: "kafka-service" #[kafka的service的clusterIP]- name: KAFKA_ZOOKEEPER_CONNECTvalue: zookeeper-service:2181- name: KAFKA_BROKER_IDvalue: "1"
EOF

3.验证

 进入容器 创建生产者消费者测试

生产者：

kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic demo

wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==

 

消费者：

kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092  --topic demo --from-beginning

 

 

二、集群部署

这里是生产使用的。直接采用pv-pvc的挂载方式

1.zook的pv-pvc.yaml

kind: PersistentVolume
apiVersion: v1
metadata:name: k8s-pv-zk0annotations:volume.beta.kubernetes.io/storage-class: "wms-zook"labels:type: local
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 192.168.XX.XXpath: "/mnt/nas/zmj_pord/nfs/wms-zookeeper/pv0"
---
kind: PersistentVolume
apiVersion: v1
metadata:name: k8s-pv-zk1annotations:volume.beta.kubernetes.io/storage-class: "wms-zook"labels:type: local
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 192.168.XX.XXpath: "/mnt/nas/zmj_pord/nfs/wms-zookeeper/pv1"
---
kind: PersistentVolume
apiVersion: v1
metadata:name: k8s-pv-zk2annotations:volume.beta.kubernetes.io/storage-class: "wms-zook"labels:type: local
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 192.168.XX.XXpath: "/mnt/nas/zmj_pord/nfs/wms-zookeeper/pv2"

2.zk的all.yaml

---
apiVersion: v1
kind: Service
metadata:name: zk-hslabels:app: zk
spec:ports:- port: 2888name: server- port: 3888name: leader-electionclusterIP: Noneselector:app: zk
---
apiVersion: v1
kind: Service
metadata:name: zk-cslabels:app: zk
spec:type: NodePortports:- port: 2181name: clientselector:app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:name: zk-pdb
spec:selector:matchLabels:app: zkmaxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: zk
spec:selector:matchLabels:app: zkserviceName: zk-hsreplicas: 3updateStrategy:type: RollingUpdatepodManagementPolicy: Paralleltemplate:metadata:labels:app: zkspec:affinity:podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:- labelSelector:matchExpressions:- key: "app"operator: Invalues:- zktopologyKey: "kubernetes.io/hostname"containers:- name: kubernetes-zookeeperimagePullPolicy: Alwaysimage: "mirrorgooglecontainers/kubernetes-zookeeper:1.0-3.4.10"resources:requests:memory: "1Gi"cpu: "0.5"ports:- containerPort: 2181name: client- containerPort: 2888name: server- containerPort: 3888name: leader-electioncommand:- sh- -c- "start-zookeeper \--servers=3 \--data_dir=/var/lib/zookeeper/data \--data_log_dir=/var/lib/zookeeper/data/log \--conf_dir=/opt/zookeeper/conf \--client_port=2181 \--election_port=3888 \--server_port=2888 \--tick_time=2000 \--init_limit=10 \--sync_limit=5 \--heap=512M \--max_client_cnxns=60 \--snap_retain_count=3 \--purge_interval=12 \--max_session_timeout=40000 \--min_session_timeout=4000 \--log_level=INFO"readinessProbe:exec:command:- sh- -c- "zookeeper-ready 2181"initialDelaySeconds: 10timeoutSeconds: 5livenessProbe:exec:command:- sh- -c- "zookeeper-ready 2181"initialDelaySeconds: 10timeoutSeconds: 5volumeMounts:- name: datadirmountPath: /var/lib/zookeepersecurityContext:runAsUser: 1000fsGroup: 1000volumeClaimTemplates:- metadata:name: datadirannotations:volume.beta.kubernetes.io/storage-class: "wms-zook"spec:accessModes: [ "ReadWriteOnce" ]resources:requests:storage: 5Gi

3.kafka的pv-pvc

apiVersion: v1
kind: PersistentVolume
metadata:name: k8s-pv-kafka0namespace: toolslabels:app: kafkaannotations:volume.beta.kubernetes.io/storage-class: "wms-kafka"
spec:capacity:storage: 5GaccessModes:- ReadWriteOncenfs:server: 192.168.XX.XXpath: "/mnt/nas/zmj_pord/nfs/wms-kafk/pv0"
---
apiVersion: v1
kind: PersistentVolume
metadata:name: k8s-pv-kafka1namespace: toolslabels:app: kafkaannotations:volume.beta.kubernetes.io/storage-class: "wms-kafka"
spec:capacity:storage: 5GaccessModes:- ReadWriteOncenfs:server: 192.168.XX.XXpath: "/mnt/nas/zmj_pord/nfs/wms-kafka/pv1"
---
apiVersion: v1
kind: PersistentVolume
metadata:name: k8s-pv-kafka2namespace: toolslabels:app: kafkaannotations:volume.beta.kubernetes.io/storage-class: "wms-kafka"
spec:capacity:storage: 5GaccessModes:- ReadWriteOncenfs:server: 192.168.xx.XXpath: "/mnt/nas/zmj_pord/nfs/wms-kafka/pv2"

4.kafka的all.yaml

（这里要把健康检查#掉）

---
apiVersion: v1
kind: Service
metadata:name: kafka-hslabels:app: kafka
spec:ports:- port: 9092name: serverclusterIP: Noneselector:app: kafka
---
apiVersion: v1
kind: Service
metadata:name: kafka-cslabels:app: kafka
spec:selector:app: kafkatype: NodePortports:- name: clientport: 9092#  nodePort: 19092
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:name: kafka-pdb
spec:selector:matchLabels:app: kafkaminAvailable: 2
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: kafka
spec:serviceName: kafka-hsreplicas: 3selector:matchLabels:app: kafkatemplate:metadata:labels:app: kafkaspec:affinity:podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:- labelSelector:matchExpressions:- key: "app"operator: Invalues:- kafkatopologyKey: "kubernetes.io/hostname"podAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 1podAffinityTerm:labelSelector:matchExpressions:- key: "app"operator: Invalues:- zktopologyKey: "kubernetes.io/hostname"terminationGracePeriodSeconds: 300containers:- name: kafka# imagePullPolicy: IfNotPresentimage: registry.cn-hangzhou.aliyuncs.com/jaxzhai/k8skafka:v1#image: wurstmeister/kafkacommand: [ "/bin/bash", "-ce", "tail -f /dev/null" ]resources:requests:memory: "200M"cpu: 500mports:- containerPort: 9092name: servercommand:- sh- -c- "exec kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=${HOSTNAME##*-} \--override listeners=PLAINTEXT://:9092 \--override zookeeper.connect=zk-0.zk-hs.prod-zmj-wms.svc.cluster.local:2181,zk-1.zk-hs.prod-zmj-wms.svc.cluster.local:2181,zk-2.zk-hs.prod-zmj-wms.svc.cluster.local:2181 \--override log.dir=/var/lib/kafka \--override auto.create.topics.enable=true \--override auto.leader.rebalance.enable=true \--override background.threads=10 \--override compression.type=producer \--override delete.topic.enable=true \--override leader.imbalance.check.interval.seconds=300 \--override leader.imbalance.per.broker.percentage=10 \--override log.flush.interval.messages=9223372036854775807 \--override log.flush.offset.checkpoint.interval.ms=60000 \--override log.flush.scheduler.interval.ms=9223372036854775807 \--override log.retention.bytes=-1 \--override log.retention.hours=168 \--override log.roll.hours=168 \--override log.roll.jitter.hours=0 \--override log.segment.bytes=1073741824 \--override log.segment.delete.delay.ms=60000 \--override message.max.bytes=1000012 \--override min.insync.replicas=1 \--override num.io.threads=8 \--override num.network.threads=3 \--override num.recovery.threads.per.data.dir=1 \--override num.replica.fetchers=1 \--override offset.metadata.max.bytes=4096 \--override offsets.commit.required.acks=-1 \--override offsets.commit.timeout.ms=5000 \--override offsets.load.buffer.size=5242880 \--override offsets.retention.check.interval.ms=600000 \--override offsets.retention.minutes=1440 \--override offsets.topic.compression.codec=0 \--override offsets.topic.num.partitions=50 \--override offsets.topic.replication.factor=3 \--override offsets.topic.segment.bytes=104857600 \--override queued.max.requests=500 \--override quota.consumer.default=9223372036854775807 \--override quota.producer.default=9223372036854775807 \--override replica.fetch.min.bytes=1 \--override replica.fetch.wait.max.ms=500 \--override replica.high.watermark.checkpoint.interval.ms=5000 \--override replica.lag.time.max.ms=10000 \--override replica.socket.receive.buffer.bytes=65536 \--override replica.socket.timeout.ms=30000 \--override request.timeout.ms=30000 \--override socket.receive.buffer.bytes=102400 \--override socket.request.max.bytes=104857600 \--override socket.send.buffer.bytes=102400 \--override unclean.leader.election.enable=true \--override zookeeper.session.timeout.ms=6000 \--override zookeeper.set.acl=false \--override broker.id.generation.enable=true \--override connections.max.idle.ms=600000 \--override controlled.shutdown.enable=true \--override controlled.shutdown.max.retries=3 \--override controlled.shutdown.retry.backoff.ms=5000 \--override controller.socket.timeout.ms=30000 \--override default.replication.factor=1 \--override fetch.purgatory.purge.interval.requests=1000 \--override group.max.session.timeout.ms=300000 \--override group.min.session.timeout.ms=6000 \--override inter.broker.protocol.version=0.10.2-IV0 \--override log.cleaner.backoff.ms=15000 \--override log.cleaner.dedupe.buffer.size=134217728 \--override log.cleaner.delete.retention.ms=86400000 \--override log.cleaner.enable=true \--override log.cleaner.io.buffer.load.factor=0.9 \--override log.cleaner.io.buffer.size=524288 \--override log.cleaner.io.max.bytes.per.second=1.7976931348623157E308 \--override log.cleaner.min.cleanable.ratio=0.5 \--override log.cleaner.min.compaction.lag.ms=0 \--override log.cleaner.threads=1 \--override log.cleanup.policy=delete \--override log.index.interval.bytes=4096 \--override log.index.size.max.bytes=10485760 \--override log.message.timestamp.difference.max.ms=9223372036854775807 \--override log.message.timestamp.type=CreateTime \--override log.preallocate=false \--override log.retention.check.interval.ms=300000 \--override max.connections.per.ip=2147483647 \--override num.partitions=1 \--override producer.purgatory.purge.interval.requests=1000 \--override replica.fetch.backoff.ms=1000 \--override replica.fetch.max.bytes=1048576 \--override replica.fetch.response.max.bytes=10485760 \--override reserved.broker.max.id=1000 "env:- name: KAFKA_HEAP_OPTSvalue : "-Xmx300M -Xms200M"- name: KAFKA_OPTSvalue: "-Dlogging.level=INFO"volumeMounts:- name: datadirmountPath: /var/lib/kafkareadinessProbe:tcpSocket:port: 9092timeoutSeconds: 5initialDelaySeconds: 20# exec:#  command:#   - sh#   - -c#   - "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server=0.0.0.0:9092"securityContext:runAsUser: 1000fsGroup: 1000volumeClaimTemplates:- metadata:name: datadirannotations:volume.beta.kubernetes.io/storage-class: "wms-kafka"spec:accessModes: [ "ReadWriteOnce" ]resources:requests:storage: 5G

5.测试

略，参考单节点


http://chatgpt.dhexx.cn/article/Wgw0BDMh.shtml

相关文章

Docker 安装Zookeeper

Docker 安装Zookeeper

第一步&#xff1a;查看本地镜像和检索拉取Zookeeper 镜像 # 查看本地镜像 docker images # 检索ZooKeeper 镜像 docker search zookeeper # 拉取ZooKeeper镜像最新版本 docker pull zookeeper:latest [rootlocalhost ~]# docker images REPOSITORY TAG …
阅读更多...
mac pro m1：搭建zookeeper集群并设置开机自启

mac pro m1：搭建zookeeper集群并设置开机自启

0. 引言 之前我们讲解过搭建zookeeper单节点&#xff0c;但在实际生产中&#xff0c;为了保证服务高可用&#xff0c;通常我们是采用集群模式。所以本次我们来实操集群模式的搭建 1. zk集群模式 zk可以作为注册中心和配置中心&#xff0c;常用在微服务各类组件的多节点服务治…
阅读更多...
一款实用的数据恢复软件—zook data recovery wizard

一款实用的数据恢复软件—zook data recovery wizard

zook data recovery wizard是RecoveryTools下的一款子品牌&#xff0c;同时也是一款功能实用的数据恢复软件&#xff0c;该软件可以从Windows中恢复已删除&#xff0c;损坏&#xff0c;格式化和丢失的数据&#xff0c;能够支持从驱动器&#xff0c;SD卡&#xff0c;硬盘&#x…
阅读更多...
zook 报错 Unable to read additional data from server sessionid 0x0

zook 报错 Unable to read additional data from server sessionid 0x0

zook报错启动报错&#xff1a; 2017-09-25 18:33:46,913 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread1183] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting r…
阅读更多...
hadoop大数据集群搭设(hadoop+zook+HBase+hive)百分百成功

hadoop大数据集群搭设(hadoop+zook+HBase+hive)百分百成功

hadoop大数据集群搭设 前言所需软件虚拟机准备工作一、Jdk安装二、安装zookeeper三、HBase安装四、mysql安装配置五、安装hive 前言 经过长时间的测试总结出在目前集群搭建最稳定的步骤是&#xff1a; 至少我按这个过程基本0失误&#xff0c;且初始化次数最少。当然也可以尝试…
阅读更多...
zookeeper客户端命令（三）

zookeeper客户端命令（三）

zookeeper客户端命令&#xff08;三&#xff09; 问题背景zookeeper分布式技术基本概念&#xff08;一&#xff09;zookeeper单机及集群部署&#xff0c;附安装包下载&#xff08;二&#xff09;zookeeper客户端命令&#xff08;三&#xff09; zook客户端指令节点创建测试集群…
阅读更多...
Zookper集群搭建

Zookper集群搭建

&#x1f345;程序员小王的博客&#xff1a;程序员小王的博客 &#x1f345; 欢迎点赞 &#x1f44d; 收藏 ⭐留言 &#x1f4dd; &#x1f345; 如有编辑错误联系作者&#xff0c;如果有比较好的文章欢迎分享给我&#xff0c;我会取其精华去其糟粕 一、搭建zookper集群前的准备…
阅读更多...
分布式系统服务框架Zookeeper介绍与原理实现

分布式系统服务框架Zookeeper介绍与原理实现

分布式数据管理之痛点 为了确保微服务之间松耦合&#xff0c;每个服务都有自己的数据库, 有的是关系型数据库&#xff08;SQL&#xff09;&#xff0c;有的是非关系型数据库&#xff08;NoSQL&#xff09;。 开发企业事务往往牵涉到多个服务&#xff0c;要想做到多个服务数据…
阅读更多...
给视频加滚动字幕，给视频加字幕制作mv 录制的视频配背景音乐

给视频加滚动字幕，给视频加字幕制作mv 录制的视频配背景音乐

给视频添加滚动字幕方法其实很简单&#xff0c;像我们下载的电影&#xff0c;歌曲&#xff0c;用手机录制的视频都可以加字幕&#xff0c;或者滚动字幕&#xff0c;也可以加背景音乐或其它声音&#xff0c;给视频开头或结尾加一张图片或多张图片等等都是可以实现的&#xff0c;…
阅读更多...
手把手叫你制作一个精美的在线音乐播放器

手把手叫你制作一个精美的在线音乐播放器

最近项目中要增加一些特殊的功能&#xff0c;实现音乐的在线播放。虽说网上源码一大把&#xff0c;demo一大堆&#xff0c;但是能用的其实寥寥无几&#xff0c;看来关键时刻还是自己动手&#xff0c;丰衣足食啊。话不多说&#xff0c;直接看效果图吧&#xff1a; 看是不是很美观…
阅读更多...
微信小程序中将图片与音乐制作成MV

微信小程序中将图片与音乐制作成MV

最近一直在开发一个类似于小年糕的微信小程序&#xff0c;在开发制作MV功能时 &#xff0c;花费了一些心思&#xff0c;其间主要遇到了以下一些问题点&#xff1a; 1. 上传图片的动画效果如何像播放视频一样实现播放与暂停&#xff1f; 2. 用户上传的图片数量不确定&#xf…
阅读更多...
FL Studio中文版21最新免费音乐编曲软件制作工具

FL Studio中文版21最新免费音乐编曲软件制作工具

FL Studio较为适合专业的音乐制作者&#xff0c;操作难度较大&#xff0c;学习门槛也较高&#xff1b;Studio One则主打一站式的音乐制作&#xff0c;从编曲到录音到后期的专辑制作都可以在其中实现&#xff0c;同时操作难度不大&#xff0c;对初学者和业余爱好者都较为友好。 …
阅读更多...
mv

mv

mv 移动文件或改名 mv 命令&#xff08;move 的缩写&#xff09;&#xff0c;既可以在不同的目录之间移动文件或目录&#xff0c;也可以对文件和目录进行重命名。 该命令的基本格式如下&#xff1a; [rootlocalhost ~]# mv 【选项】 源文件 目标文件“mv” 默认执行命令(mv -…
阅读更多...
怎样做音乐相册怎样制作？手把手教你制作

怎样做音乐相册怎样制作？手把手教你制作

大家平时出门游玩的时候&#xff0c;会拍摄一些好看的照片吗&#xff1f;那你们会将这些照片分享在社交平台上吗&#xff1f;普通的照片分享&#xff0c;有时会显得比较枯燥单调&#xff0c;其实我们可以将这些照片制作成音乐相册&#xff0c;这样就可以丰富照片的内容&#xf…
阅读更多...
Audacity（电脑音频剪辑软件）官方中文版V3.0.2 | 完全免费的音乐制作软件audacity下载

Audacity（电脑音频剪辑软件）官方中文版V3.0.2 | 完全免费的音乐制作软件audacity下载

Audacity 是一款专业易于使用且完全免费的音乐制作软件&#xff0c;高度可定制的界面能实时显示预览&#xff0c;具备多轨音频剪辑和音频录制功能&#xff0c;提供了数十种可修改或自己创建的插件以及无限的撤消和重做能力&#xff0c;能将磁带和唱片转换成数字唱片或 CD&#…
阅读更多...
手把手教你批量制作MV连播视频

手把手教你批量制作MV连播视频

一、点击下载CR MVMixer解压到D盘 找到图标双击打开CR MVMixer 本软件授权注册码可以登陆酷软街获取 右下角点击注册&#xff0c;填入注册码激活。 软件操作页面&#xff0c;左侧是加载音乐MP3跟LRC歌词的 右侧可以添加视频素材&#xff0c;设置封面。 二、加载素材&#xf…
阅读更多...
MV制作器UI版来噜

MV制作器UI版来噜

一、项目介绍 MV制作器UI版来噜 在之前的制作MV的项目中&#xff0c;我们使用代码来调用文心大模型API&#xff0c;批量的获取图片&#xff0c;然后我们需要使用专业德视频剪辑工具譬如pr或剪映等完成后续的步骤&#xff0c;但并不是所有人都会使用剪辑软件… 那有没有什么可…
阅读更多...
轻音——基于Swing与JavaFx的音乐播放器

轻音——基于Swing与JavaFx的音乐播放器

简介 轻音是一款我个人开发的支持本地和在线音乐的播放器&#xff0c;我制作这款软件的初衷也是为了消除不同平台之间版权的差异&#xff0c;让听歌不再“跨平台化”&#xff0c;实现听歌自由使用的技术&#xff1a;100% Java 编程语言&#xff0c;98% Swing 开发GUI&#xff…
阅读更多...
LaTeX会议论文添加版权信息

LaTeX会议论文添加版权信息

1. 在导言区增加 \usepackage{fancyhdr} \renewcommand{\headrulewidth}{0pt} \renewcommand{\footrulewidth}{0pt} 2. maketitle后增加 \thispagestyle{fancy} \fancyhead{} \lhead{} \lfoot{\copyright~2014 IEEE} \cfoot{} \rfoot{}&#xff0c; 效果如下
阅读更多...
CCF论文会议 IEEE 如何查询某个会议期刊的所有文章

CCF论文会议 IEEE 如何查询某个会议期刊的所有文章

1. 下载CCF英文列表 CCF英文会议期刊 列表目录 下载 2. 找到要查询的会议&#xff0c;点击后面的网址 3. 就可以看到该会议期刊列年的整理好的记录&#xff0c;随便点击一个进入 4. 可以看到该年收录的所有文章 5. 可以点击这里查看文章详情 微程序学堂
阅读更多...
推荐文章

Copyright @ 2022~2023