“Hadoop 搭建”的版本间的差异
(创建页面,内容为“修改主机名 hostnamectl set-hostname master hostnamectl set-hostname slave1 hostnamectl set-hostname slave2 配置hosts映射vi /etc/hosts 172.16.1.3 master 172…”) |
|||
第1行: | 第1行: | ||
− | 修改主机名 | + | '''修改主机名'''<br> |
− | hostnamectl set-hostname master | + | hostnamectl set-hostname master<br> |
− | hostnamectl set-hostname slave1 | + | hostnamectl set-hostname slave1<br> |
− | hostnamectl set-hostname slave2 | + | hostnamectl set-hostname slave2<br> |
− | + | 配置hosts映射<br> | |
− | 172.16.1.3 master | + | vi /etc/hosts<br> |
− | 172.16.1.4 slave1 | + | 172.16.1.3 master<br> |
− | 172.16.1.5 slave2 | + | 172.16.1.4 slave1<br> |
+ | 172.16.1.5 slave2<br> | ||
+ | <br> | ||
+ | 关闭防火墙(省略)<br> | ||
+ | systemctl stop firewalld<br> | ||
+ | systemctl disable firewalld<br> | ||
− | + | 时间同步<br> | |
− | + | tzselect 5 9 1 1<br> | |
− | + | 写入到export<br> | |
+ | vi /etc/ntp.conf<br> | ||
+ | server 127.127.1.0<br> | ||
+ | fudge 127.127.1.0 stratum 10<br> | ||
+ | 重启ntp<br> | ||
− | + | 从节点同步(不要启动ntp)<br> | |
− | + | ntpdate master<br> | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | SSH免密<br> | |
− | + | ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa<br> | |
+ | cat id_dsa.pub >> authorized_keys<br> | ||
+ | scp /root/.ssh/authorized_keys root@slave1:/root/.ssh/authorized_keys<br> | ||
+ | scp /root/.ssh/authorized_keys root@slave2:/root/.ssh/authorized_keys<br> | ||
+ | <br> | ||
+ | ssh localhost 被动生成.ssh文件(由于权限问题自动生成)<br> | ||
+ | ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa<br> | ||
+ | cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys<br> | ||
+ | 如果A想 免密登录到B:<br> | ||
+ | A: ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa<br> | ||
+ | B: cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys<br> | ||
+ | 结论:B包含了A的公钥,A就可以免密的登录了<br> | ||
+ | 排错方法:公钥信息清除: ssh-keygen -R IP<br> | ||
+ | JDK配置<br> | ||
+ | vi /etc/profile<br> | ||
+ | export JAVA_HOME=/usr/java<br> | ||
+ | export PATH=$PATH:$JAVA_HOME/bin<br> | ||
+ | 测试java -version<br> | ||
− | + | Zookeeper配置<br> | |
− | + | tar –zxvf /opt/soft/zookeeper-3.4.10.tar.gz –C /usr/zookeeper 解压zookeeper<br> | |
− | + | 将zoo_sample.cfg文件拷贝一份,命名为为“zoo.cfg”<br> | |
− | + | cd /usr/zookeeper/conf/zoo.cfg | |
− | + | zoo.cfg文件配置 | |
− | + | tickTime=2000 | |
− | + | initLimit=10 | |
− | + | syncLimit=5 | |
− | + | dataDir=/usr/zookeeper/zookeeper-3.4.10/zkdata | |
− | + | clientPort=2181 | |
− | + | dataLogDir=/usr/zookeeper/zookeeper-3.4.10/zkdatalog | |
− | + | server.1=master:2888:3888 | |
− | + | server.2=slave1:2888:3888 | |
− | + | server.3=slave2:2888:3888 | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | tar –zxvf /opt/soft/zookeeper-3.4.10.tar.gz –C /usr/zookeeper 解压zookeeper | ||
− | 将zoo_sample.cfg文件拷贝一份,命名为为“zoo.cfg” | ||
− | cd /usr/zookeeper/conf/zoo.cfg | ||
− | zoo.cfg文件配置 | ||
− | tickTime=2000 | ||
− | initLimit=10 | ||
− | syncLimit=5 | ||
− | dataDir=/usr/zookeeper/zookeeper-3.4.10/zkdata | ||
− | clientPort=2181 | ||
− | dataLogDir=/usr/zookeeper/zookeeper-3.4.10/zkdatalog | ||
− | server.1=master:2888:3888 | ||
− | server.2=slave1:2888:3888 | ||
− | server.3=slave2:2888:3888 | ||
在zookeeper的目录中,创建zkdata和zkdatalog两个文件夹。Zkdatalog文件夹,是为了指定zookeeper产生日志指定相应的路径 | 在zookeeper的目录中,创建zkdata和zkdatalog两个文件夹。Zkdatalog文件夹,是为了指定zookeeper产生日志指定相应的路径 | ||
− | mkdir zkdata | + | mkdir zkdata |
− | mkdir zkdatalog | + | mkdir zkdatalog |
− | + | 进入zkdata文件夹,创建文件myid | |
− | echo “1” > myid master | + | echo “1” > myid master |
− | echo “2” > myid slave1 | + | echo “2” > myid slave1 |
− | echo “2” > myid slave2 | + | echo “2” > myid slave2 |
环境变量zookeeper | 环境变量zookeeper | ||
− | export ZOOKEEPER_HOME=/// | + | export ZOOKEEPER_HOME=/// |
− | export PATH=$PATH:$ZOOKEEPER_HOME/bin | + | export PATH=$PATH:$ZOOKEEPER_HOME/bin |
启动:zkServer.sh start | 启动:zkServer.sh start | ||
状态:zkServer.sh status | 状态:zkServer.sh status | ||
第79行: | 第80行: | ||
hadoop-env.sh | hadoop-env.sh | ||
配置JDK环境变量 | 配置JDK环境变量 | ||
− | export JAVA_HOME=/绝对路径 | + | export JAVA_HOME=/绝对路径 |
− | export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin | + | export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin |
core-site.xml 核心配置 | core-site.xml 核心配置 |
2020年11月23日 (一) 10:10的版本
修改主机名
hostnamectl set-hostname master
hostnamectl set-hostname slave1
hostnamectl set-hostname slave2
配置hosts映射
vi /etc/hosts
172.16.1.3 master
172.16.1.4 slave1
172.16.1.5 slave2
关闭防火墙(省略)
systemctl stop firewalld
systemctl disable firewalld
时间同步
tzselect 5 9 1 1
写入到export
vi /etc/ntp.conf
server 127.127.1.0
fudge 127.127.1.0 stratum 10
重启ntp
从节点同步(不要启动ntp)
ntpdate master
SSH免密
ssh-keygen -t dsa -P -f ~/.ssh/id_dsa
cat id_dsa.pub >> authorized_keys
scp /root/.ssh/authorized_keys root@slave1:/root/.ssh/authorized_keys
scp /root/.ssh/authorized_keys root@slave2:/root/.ssh/authorized_keys
ssh localhost 被动生成.ssh文件(由于权限问题自动生成)
ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
如果A想 免密登录到B:
A: ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
B: cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
结论:B包含了A的公钥,A就可以免密的登录了
排错方法:公钥信息清除: ssh-keygen -R IP
JDK配置
vi /etc/profile
export JAVA_HOME=/usr/java
export PATH=$PATH:$JAVA_HOME/bin
测试java -version
Zookeeper配置
tar –zxvf /opt/soft/zookeeper-3.4.10.tar.gz –C /usr/zookeeper 解压zookeeper
将zoo_sample.cfg文件拷贝一份,命名为为“zoo.cfg”
cd /usr/zookeeper/conf/zoo.cfg zoo.cfg文件配置 tickTime=2000 initLimit=10 syncLimit=5 dataDir=/usr/zookeeper/zookeeper-3.4.10/zkdata clientPort=2181 dataLogDir=/usr/zookeeper/zookeeper-3.4.10/zkdatalog server.1=master:2888:3888 server.2=slave1:2888:3888 server.3=slave2:2888:3888
在zookeeper的目录中,创建zkdata和zkdatalog两个文件夹。Zkdatalog文件夹,是为了指定zookeeper产生日志指定相应的路径
mkdir zkdata mkdir zkdatalog
进入zkdata文件夹,创建文件myid
echo “1” > myid master echo “2” > myid slave1 echo “2” > myid slave2
环境变量zookeeper
export ZOOKEEPER_HOME=/// export PATH=$PATH:$ZOOKEEPER_HOME/bin
启动:zkServer.sh start 状态:zkServer.sh status Zookeeper排错 关闭服务,杀掉进程///////已经生成的文件(version-2等),进行删除 重新开启服务 (zoo.cfg myid ---------对应关系修改) Hadoop搭建
hadoop-env.sh 配置JDK环境变量
export JAVA_HOME=/绝对路径 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
core-site.xml 核心配置
//默认文件系统是什么 namenode通信 //定义什么端口在哪里启动namenode //定义namenode在哪里 //master表示主机名称 <property> <name>fs.default.name</name> <value>hdfs://master:9000</value> </property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/hadoop/hadoop-2.7.3/hdfs/tmp</value>
</property>
hdfs-site.xml配置
//副本的数量是多少
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
//配置 namenode元数据存放位置 <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/hadoop/hadoop-2.7.3/hdfs/name</value> </property>
//配置datanode块数据存放位置 <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/hadoop/hadoop-2.7.3/hdfs/data</value> </property>
mapred-site.xml配置
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
yarn-site.xml配置
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>localhost</value> </property>
初始化:hdfs namenode -format
创建目录
并初始化一个空的fsimage VERSION CID
vim core-site.xml
<configuration>
<property>
<name>fs.default.name</name> <value>hdfs://master:9000</value>
</property> <property>
<name>hadoop.tmp.dir</name> <value>/usr/hadoop/hadoop-2.7.3/hdfs/tmp</value>
<description>A base for other temporary directories.</description> </property> <property>
<name>io.file.buffer.size</name> <value>131072</value>
</property> <property>
<name>fs.checkpoint.period</name> <value>60</value>
</property> <property>
<name>fs.checkpoint.size</name> <value>67108864</value>
</property> </configuration>
5.使用vim命令编辑yarn-site.xml文件,并在配置文件中加入以下内容: <configuration> <property>
<name>yarn.resourcemanager.address</name> <value>master:18040</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:18030</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>master:18088</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:18025</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>master:18141</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property>
</configuration>
7. 使用vim命令编辑hdfs-site.xml文件,并在配置文件中加入以下内容:
<configuration>
<property>
<name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/hadoop/hadoop-2.7.3/hdfs/name</value> <final>true</final>
</property>
<property> <name>dfs.datanode.data.dir</name> <value>file:/usr/hadoop/hadoop-2.7.3/hdfs/data</value> <final>true</final> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:9001</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property>
</configuration> 8. 使用vim命令编辑mapred-site.xmll文件,并在配置文件中加入以下内容: cp mapred-site.xml.template mapred-site.xml 然后对其进行编辑,并在配置文件中加入以下内容: <property>
<name>mapreduce.framework.name</name> <value>yarn</value> </property>
格式化Hadoop并开启集群 master中格式化hadoop
输入hadoop namenode -format 进行hadoop的格式化操作