“丝路通:Kafka数据缓存”的版本间的差异
(→在Kafka消费者端查看) |
(→在Kafka消费者端查看) |
||
第44行: | 第44行: | ||
[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.30:9092 --topic goods --from-beginning | [root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.30:9092 --topic goods --from-beginning | ||
− | <nowiki> | + | <nowiki>134,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/H5627a52b6e3c46dab3d0a608dd9158feP.jpg_300x300.jpg,Summer Garment Custom Latest Design Casual Style T Shirts_OEM Service Premium Short Sleeve T Shirts For Men,9,Piece,Sunshine Outdoor Products Co._ Ltd.,//024sunshine.en.alibaba.com/company_profile.html#top-nav-bar,China,1,Apparel Design Apparel_Textiles Accessories Services |
− | + | 135,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB1q0jljAKWBuNjy1zjq6AOypXaL.jpg_300x300.jpg,Beads link machine Wear beads machine,2800,Set,Hangzhou Xiaoshan Yuefeng Machinery Factory,//yuefeng.en.alibaba.com/company_profile.html#top-nav-bar,China,6,Apparel Design Apparel_Textiles Accessories Services | |
− | + | 136,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB1Jfe2g46I8KJjy0Fgq6xXzVXaS.jpg_300x300.jpg,Custom slim men's trousers sweat jogger pants men,5,Piece,Shenzhen Cuichang Garment Co._ Ltd.,//szcuichang.en.alibaba.com/company_profile.html#top-nav-bar,China,8,Apparel Design Apparel_Textiles Accessories Services | |
− | + | 137,14,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB132IagSfD8KJjSszhq6zIJFXal.jpg_300x300.jpg,2018 OEM ManufactureSlim Fit Joggers Gym Jogger Pants Tapered Wholesale SweatPa,5,Piece,Shenzhen Cuichang Garment Co._ Ltd.,//szcuichang.en.alibaba.com/company_profile.html#top-nav-bar,China,8,Apparel Design Apparel_Textiles Accessories Services | |
− | + | </nowiki> | |
− | |||
− | |||
− | |||
− |
2020年9月11日 (五) 15:04的最新版本
目录
步骤
1.创建Kafka主题
2.创建Kafka消费者
3.创建定时任务
4.通过Flume数据采集到Kafka
操作步骤
启动zookeeper和kafka
创建Kafka主题
[root@master kafka_2.11-2.3.1]cd kafka_2.11-2.3.1/
[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh -create --zookeeper 10.0.0.30:2181 -replication-factor 1 --partitions 1 --topic goods
Created topic goods.
[root@master kafka_2.11-2.3.1]# bin/kafka-topics.sh --list --zookeeper 10.0.0.30:2181
__consumer_offsets goods
创建消费者
[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.30:9092 --topic goods --from-beginning
复制文件到采集区
将清洗后的数据文件 0820_ali_xiaolong.csv 复制到采集区 /opt/flinkproject
开启Flume数据采集到Kafka
/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/spool-memory-kafka.conf -n client -Dflume.root.logger=INFO,console
Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh Info: Including Hadoop libraries found via (/usr/local/hadoop-2.6.5/bin/hadoop) for HDFS access Info: Including HBASE libraries found via (/usr/local/hbase-1.3.6/bin/hbase) for HBASE access Info: Including Hive libraries found via (/usr/local/apache-hive-1.2.1-bin) for Hive access ...
在Kafka消费者端查看
[root@slave1 kafka_2.11-2.3.1]# bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.30:9092 --topic goods --from-beginning
134,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/H5627a52b6e3c46dab3d0a608dd9158feP.jpg_300x300.jpg,Summer Garment Custom Latest Design Casual Style T Shirts_OEM Service Premium Short Sleeve T Shirts For Men,9,Piece,Sunshine Outdoor Products Co._ Ltd.,//024sunshine.en.alibaba.com/company_profile.html#top-nav-bar,China,1,Apparel Design Apparel_Textiles Accessories Services 135,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB1q0jljAKWBuNjy1zjq6AOypXaL.jpg_300x300.jpg,Beads link machine Wear beads machine,2800,Set,Hangzhou Xiaoshan Yuefeng Machinery Factory,//yuefeng.en.alibaba.com/company_profile.html#top-nav-bar,China,6,Apparel Design Apparel_Textiles Accessories Services 136,8,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB1Jfe2g46I8KJjy0Fgq6xXzVXaS.jpg_300x300.jpg,Custom slim men's trousers sweat jogger pants men,5,Piece,Shenzhen Cuichang Garment Co._ Ltd.,//szcuichang.en.alibaba.com/company_profile.html#top-nav-bar,China,8,Apparel Design Apparel_Textiles Accessories Services 137,14,Apparel_Textiles & Accessories,Apparel,Apparel Design Services,//s.alicdn.com/@sc01/kf/HTB132IagSfD8KJjSszhq6zIJFXal.jpg_300x300.jpg,2018 OEM ManufactureSlim Fit Joggers Gym Jogger Pants Tapered Wholesale SweatPa,5,Piece,Shenzhen Cuichang Garment Co._ Ltd.,//szcuichang.en.alibaba.com/company_profile.html#top-nav-bar,China,8,Apparel Design Apparel_Textiles Accessories Services