Kafka数据缓存

步骤

1.创建Kafka主题

2.创建Kafka消费者

3.创建定时任务

4.通过Flume数据采集到Kafka

操作步骤

启动zookeeper和kafka

master,slave1,slave2

参考：Kafka——集群安裝部署(自带Zookeeper)

创建Kafka主题

master节点：在此插入非格式文本 [root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh -create --zookeeper 192.168.128.130:2181 -replication-factor 1 --partitions 1 --topic shop

Created topic shop.

[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh --list --zookeeper 192.168.128.130:2181

__consumer_offsets
shop
test

创建消费者

slave1 节点：

[root@slave1 ~]# cd /usr/local/kafka_2.11-2.3.1/

[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.128.131:9092 --topic shop --from-beginning

创建定时任务

创建定时任务，定时生成数据

crontab -e

* * * * * python /opt/data.py /opt/flinkproject/$(date +"\%Y-\%m-\%d-\%H-\%M-\%S").log 1000

开启Flume数据采集到Kafka

master节点：

/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/spool-memory-kafka.conf -n client -Dflume.root.logger=INFO,console

Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
Info: Including Hadoop libraries found via (/usr/local/hadoop-2.6.5/bin/hadoop) for HDFS access
Info: Including HBASE libraries found via (/usr/local/hbase-1.3.6/bin/hbase) for HBASE access
Info: Including Hive libraries found via (/usr/local/apache-hive-1.2.1-bin) for Hive access
...

在Kafka消费者端查看