“丝路通:Kafka数据缓存”的版本间的差异

来自CloudWiki
跳转至： 导航， 搜索
（→‎在Kafka消费者端查看）
 
（未显示同一用户的6个中间版本）
第10行： 第10行：
 
==操作步骤==
 
==操作步骤==
 
===启动zookeeper和kafka===
 
===启动zookeeper和kafka===
−
 
−
master,slave1,slave2
 
  
 
参考：[[丝路通:Kafka单机版安装]]
 
参考：[[丝路通:Kafka单机版安装]]
  
 
===创建Kafka主题===
 
===创建Kafka主题===
−
master节点：
+
[root@master kafka_2.11-2.3.1]cd kafka_2.11-2.3.1/
−
<nowiki>在此插入非格式文本</nowiki>
+
 
−
[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh -create --zookeeper 192.168.128.130:2181 -replication-factor 1 --partitions 1 --topic shop
+
[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh -create --zookeeper 10.0.0.30:2181 -replication-factor 1 --partitions 1 --topic goods
  
−
  Created topic shop.
+
  Created topic goods.
  
−
[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh --list --zookeeper 192.168.128.130:2181  
+
[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh --list --zookeeper 10.0.0.30:2181  
  
 
  <nowiki>__consumer_offsets
 
  <nowiki>__consumer_offsets
−
shop
+
goods</nowiki>
−
test</nowiki>
 
  
 
===创建消费者===
 
===创建消费者===
−
slave1 节点：
 
−
 
−
[root@slave1 ~]# cd /usr/local/kafka_2.11-2.3.1/
 
  
−
[root@slave1 kafka_2.11-2.3.1]#  bin/kafka-console-consumer.sh --bootstrap-server 192.168.128.131:9092 --topic shop --from-beginning
+
[root@slave1 kafka_2.11-2.3.1]#  bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.30:9092 --topic goods --from-beginning
  
−
===创建定时任务===
+
===复制文件到采集区===
−
创建定时任务，定时生成数据
+
将清洗后的数据文件 0820_ali_xiaolong.csv 复制到采集区 /opt/flinkproject
−
 
 
−
crontab -e
 
−
 
 
−
<nowiki>* * * * * python /opt/data.py /opt/flinkproject/$(date +"\%Y-\%m-\%d-\%H-\%M-\%S").log 1000</nowiki>
 
  
 
===开启Flume数据采集到Kafka===
 
===开启Flume数据采集到Kafka===
−
master节点：
 
−
 
 
/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/spool-memory-kafka.conf -n client -Dflume.root.logger=INFO,console
 
/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/spool-memory-kafka.conf -n client -Dflume.root.logger=INFO,console
  
第54行： 第42行：
  
 
===在Kafka消费者端查看===
 
===在Kafka消费者端查看===
−
slave1 节点：
+
[root@slave1 kafka_2.11-2.3.1]#  bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.30:9092 --topic goods --from-beginning
−
 
 
−
[root@slave1 kafka_2.11-2.3.1]#  bin/kafka-console-consumer.sh --bootstrap-server 192.168.128.131:9092 --topic shop --from-beginning
 
  
−
  <nowiki>951595538181,Lrjbe,18,man,650012,218.58,313015,scan,13938126759,mOdhYpNBjS@126.com,2019-08-01
+
  <nowiki>134,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/H5627a52b6e3c46dab3d0a608dd9158feP.jpg_300x300.jpg,Summer Garment Custom Latest Design Casual Style T Shirts_OEM Service Premium Short Sleeve T Shirts For Men,9,Piece,Sunshine Outdoor Products Co._ Ltd.,//024sunshine.en.alibaba.com/company_profile.html#top-nav-bar,China,1,Apparel Design Apparel_Textiles Accessories Services
−
041595538181,Htnlr,48,woman,430031,191.20,313015,pv,15019460573,FwQiachIBs@huawei.com,2019-08-01
+
135,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB1q0jljAKWBuNjy1zjq6AOypXaL.jpg_300x300.jpg,Beads link machine Wear beads machine,2800,Set,Hangzhou Xiaoshan Yuefeng Machinery Factory,//yuefeng.en.alibaba.com/company_profile.html#top-nav-bar,China,6,Apparel Design Apparel_Textiles Accessories Services
−
541595538181,Ytsow,54,woman,430031,617.12,313013,buy,18648305921,NdJwnsbZfc@gmail.com,2019-08-03
+
136,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB1Jfe2g46I8KJjy0Fgq6xXzVXaS.jpg_300x300.jpg,Custom slim men's trousers sweat jogger pants men,5,Piece,Shenzhen Cuichang Garment Co._ Ltd.,//szcuichang.en.alibaba.com/company_profile.html#top-nav-bar,China,8,Apparel Design Apparel_Textiles Accessories Services
−
261595538181,Mefuv,26,man,152121,675.57,313017,scan,14798316570,jeGETmJozh@gmail.com,2019-08-04
+
137,14,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB132IagSfD8KJjSszhq6zIJFXal.jpg_300x300.jpg,2018 OEM ManufactureSlim Fit Joggers Gym Jogger Pants Tapered Wholesale SweatPa,5,Piece,Shenzhen Cuichang Garment Co._ Ltd.,//szcuichang.en.alibaba.com/company_profile.html#top-nav-bar,China,8,Apparel Design Apparel_Textiles Accessories Services
−
291595538181,Ukosa,25,woman,220902,618.96,313013,fav,13467840315,rlbwhmkAsT@163.com,2019-08-04
+
</nowiki>
−
831595538181,Etsxc,28,man,230121,994.51,313015,fav,15078310469,VMZhFmAWBq@huawei.com,2019-08-06
 
−
941595538181,Swmrx,30,woman,230121,659.97,313017,scan,15256218374,cgTIRywUDG@huawei.com,2019-08-03
 
−
551595538181,Vgyrs,51,woman,250983,121.45,313016,cart,15980167923,IeobPAmzsQ@126.com,2019-08-03
 
−
841595538181,Ksoab,30,woman,430031,274.40,313015,cart,18873845162,xhJOXmneKv@gmail.com,2019-08-01</nowiki>
 

2020年9月11日 (五) 15:04的最新版本

步骤

1.创建Kafka主题

2.创建Kafka消费者

3.创建定时任务

4.通过Flume数据采集到Kafka

操作步骤

启动zookeeper和kafka

参考：丝路通:Kafka单机版安装

创建Kafka主题

[root@master kafka_2.11-2.3.1]cd kafka_2.11-2.3.1/

[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh -create --zookeeper 10.0.0.30:2181 -replication-factor 1 --partitions 1 --topic goods 

Created topic goods.

[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh --list --zookeeper 10.0.0.30:2181 

__consumer_offsets
goods

创建消费者

[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.30:9092 --topic goods --from-beginning

复制文件到采集区

将清洗后的数据文件 0820_ali_xiaolong.csv 复制到采集区 /opt/flinkproject

开启Flume数据采集到Kafka

/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/spool-memory-kafka.conf -n client -Dflume.root.logger=INFO,console 

Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
Info: Including Hadoop libraries found via (/usr/local/hadoop-2.6.5/bin/hadoop) for HDFS access
Info: Including HBASE libraries found via (/usr/local/hbase-1.3.6/bin/hbase) for HBASE access
Info: Including Hive libraries found via (/usr/local/apache-hive-1.2.1-bin) for Hive access
...

在Kafka消费者端查看

[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.30:9092 --topic goods --from-beginning 

134,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/H5627a52b6e3c46dab3d0a608dd9158feP.jpg_300x300.jpg,Summer Garment Custom Latest Design Casual Style T Shirts_OEM Service Premium Short Sleeve T Shirts For Men,9,Piece,Sunshine Outdoor Products Co._ Ltd.,//024sunshine.en.alibaba.com/company_profile.html#top-nav-bar,China,1,Apparel Design Apparel_Textiles Accessories Services
135,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB1q0jljAKWBuNjy1zjq6AOypXaL.jpg_300x300.jpg,Beads link machine Wear beads machine,2800,Set,Hangzhou Xiaoshan Yuefeng Machinery Factory,//yuefeng.en.alibaba.com/company_profile.html#top-nav-bar,China,6,Apparel Design Apparel_Textiles Accessories Services
136,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB1Jfe2g46I8KJjy0Fgq6xXzVXaS.jpg_300x300.jpg,Custom slim men's trousers sweat jogger pants men,5,Piece,Shenzhen Cuichang Garment Co._ Ltd.,//szcuichang.en.alibaba.com/company_profile.html#top-nav-bar,China,8,Apparel Design Apparel_Textiles Accessories Services
137,14,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB132IagSfD8KJjSszhq6zIJFXal.jpg_300x300.jpg,2018 OEM ManufactureSlim Fit Joggers Gym Jogger Pants Tapered Wholesale SweatPa,5,Piece,Shenzhen Cuichang Garment Co._ Ltd.,//szcuichang.en.alibaba.com/company_profile.html#top-nav-bar,China,8,Apparel Design Apparel_Textiles Accessories Services
取自“http://www.openbrains.net/mediawiki/index.php?title=丝路通:Kafka数据缓存&oldid=20975”