“Kafka数据缓存”的版本间的差异
第15行: | 第15行: | ||
===创建Kafka主题=== | ===创建Kafka主题=== | ||
master节点: | master节点: | ||
− | + | <nowiki>在此插入非格式文本</nowiki> | |
[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh -create --zookeeper 192.168.128.130:2181 -replication-factor 1 --partitions 1 --topic shop | [root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh -create --zookeeper 192.168.128.130:2181 -replication-factor 1 --partitions 1 --topic shop | ||
第32行: | 第32行: | ||
[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.128.131:9092 --topic shop --from-beginning | [root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.128.131:9092 --topic shop --from-beginning | ||
+ | |||
+ | ===开启Flume数据采集到Kafka=== | ||
+ | master节点: | ||
+ | |||
+ | /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/spool-memory-kafka.conf -n client -Dflume.root.logger=INFO,console | ||
+ | |||
+ | <nowiki>Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh | ||
+ | Info: Including Hadoop libraries found via (/usr/local/hadoop-2.6.5/bin/hadoop) for HDFS access | ||
+ | Info: Including HBASE libraries found via (/usr/local/hbase-1.3.6/bin/hbase) for HBASE access | ||
+ | Info: Including Hive libraries found via (/usr/local/apache-hive-1.2.1-bin) for Hive access | ||
+ | ...</nowiki> | ||
+ | |||
+ | ===在Kafka消费者端查看=== | ||
+ | slave1 节点: | ||
+ | |||
+ | [root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.128.131:9092 --topic shop --from-beginning | ||
+ | |||
+ | <nowiki>951595538181,Lrjbe,18,man,650012,218.58,313015,scan,13938126759,mOdhYpNBjS@126.com,2019-08-01 | ||
+ | 041595538181,Htnlr,48,woman,430031,191.20,313015,pv,15019460573,FwQiachIBs@huawei.com,2019-08-01 | ||
+ | 541595538181,Ytsow,54,woman,430031,617.12,313013,buy,18648305921,NdJwnsbZfc@gmail.com,2019-08-03 | ||
+ | 261595538181,Mefuv,26,man,152121,675.57,313017,scan,14798316570,jeGETmJozh@gmail.com,2019-08-04 | ||
+ | 291595538181,Ukosa,25,woman,220902,618.96,313013,fav,13467840315,rlbwhmkAsT@163.com,2019-08-04 | ||
+ | 831595538181,Etsxc,28,man,230121,994.51,313015,fav,15078310469,VMZhFmAWBq@huawei.com,2019-08-06 | ||
+ | 941595538181,Swmrx,30,woman,230121,659.97,313017,scan,15256218374,cgTIRywUDG@huawei.com,2019-08-03 | ||
+ | 551595538181,Vgyrs,51,woman,250983,121.45,313016,cart,15980167923,IeobPAmzsQ@126.com,2019-08-03 | ||
+ | 841595538181,Ksoab,30,woman,430031,274.40,313015,cart,18873845162,xhJOXmneKv@gmail.com,2019-08-01</nowiki> |
2020年7月26日 (日) 06:45的版本
步骤
1.创建Kafka主题
2.创建Kafka消费者
3.通过Flume数据采集到Kafka
操作步骤
启动zookeeper和kafka
master,slave1,slave2
创建Kafka主题
master节点: 在此插入非格式文本 [root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh -create --zookeeper 192.168.128.130:2181 -replication-factor 1 --partitions 1 --topic shop
Created topic shop.
[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh --list --zookeeper 192.168.128.130:2181
__consumer_offsets shop test
创建消费者
slave1 节点:
[root@slave1 ~]# cd /usr/local/kafka_2.11-2.3.1/
[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.128.131:9092 --topic shop --from-beginning
开启Flume数据采集到Kafka
master节点:
/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/spool-memory-kafka.conf -n client -Dflume.root.logger=INFO,console
Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh Info: Including Hadoop libraries found via (/usr/local/hadoop-2.6.5/bin/hadoop) for HDFS access Info: Including HBASE libraries found via (/usr/local/hbase-1.3.6/bin/hbase) for HBASE access Info: Including Hive libraries found via (/usr/local/apache-hive-1.2.1-bin) for Hive access ...
在Kafka消费者端查看
slave1 节点:
[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.128.131:9092 --topic shop --from-beginning
951595538181,Lrjbe,18,man,650012,218.58,313015,scan,13938126759,mOdhYpNBjS@126.com,2019-08-01 041595538181,Htnlr,48,woman,430031,191.20,313015,pv,15019460573,FwQiachIBs@huawei.com,2019-08-01 541595538181,Ytsow,54,woman,430031,617.12,313013,buy,18648305921,NdJwnsbZfc@gmail.com,2019-08-03 261595538181,Mefuv,26,man,152121,675.57,313017,scan,14798316570,jeGETmJozh@gmail.com,2019-08-04 291595538181,Ukosa,25,woman,220902,618.96,313013,fav,13467840315,rlbwhmkAsT@163.com,2019-08-04 831595538181,Etsxc,28,man,230121,994.51,313015,fav,15078310469,VMZhFmAWBq@huawei.com,2019-08-06 941595538181,Swmrx,30,woman,230121,659.97,313017,scan,15256218374,cgTIRywUDG@huawei.com,2019-08-03 551595538181,Vgyrs,51,woman,250983,121.45,313016,cart,15980167923,IeobPAmzsQ@126.com,2019-08-03 841595538181,Ksoab,30,woman,430031,274.40,313015,cart,18873845162,xhJOXmneKv@gmail.com,2019-08-01