“2021高训真源：Hadoop伪分布式”的版本间的差异

来自CloudWiki
跳转至： 导航， 搜索
（→‎新增路径至环境变量）
 
（未显示同一用户的7个中间版本）
第1行： 第1行：
 +
 +
 
==复制和移动文件==
 
==复制和移动文件==
 
复制hadoop-2.7.6 和jdk-8u221-linux-x64的压缩包至ubuntu主目录
 
复制hadoop-2.7.6 和jdk-8u221-linux-x64的压缩包至ubuntu主目录
第26行： 第28行：
 
export JRE_HOME=${JAVA_HOME}/jre   
 
export JRE_HOME=${JAVA_HOME}/jre   
 
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib   
 
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib   
−
export PATH=$PATH:$JAVA_HOME/bin</nowiki>
+
export PATH=$PATH:$JAVA_HOME/bin
 +
export HADOOP_HOME=/home/maxin/hadoop-2.7.6 
 +
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin</nowiki>
 +
 
 +
检测java
 +
 
 +
java -version
 +
 
 +
<nowiki>java version "1.8.0_221"
 +
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
 +
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
 +
</nowiki>
 +
 
 +
==配置hadoop-env.sh==
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ pwd
 +
 
 +
/home/maxin/hadoop-2.7.6
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ mkdir -p data/tmp
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls
 +
 
 +
<nowiki>bin  etc      lib      LICENSE.txt  README.txt  share
 +
data  include  libexec  NOTICE.txt  sbin</nowiki>
 +
 
 +
maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6/etc/hadoop/
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hadoop-env.sh
 +
 
 +
export JAVA_HOME=/bigData/jdk1.8.0_121
 +
export HADOOP_HOME=/home/maxin/hadoop-2.7.6
 +
 
 +
==配置hdfs==
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ pwd
 +
 
 +
/home/maxin/hadoop-2.7.6/etc/hadoop
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hdfs-site.xml
 +
 
 +
  <nowiki><configuration>
 +
    <property>
 +
        <name>dfs.replication</name>
 +
        <value>1</value>
 +
  </property>
 +
</configuration></nowiki>
 +
 
 +
==安装并配置ssh登录==
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
 +
 
 +
<nowiki>Generating public/private rsa key pair.
 +
Created directory '/home/maxin/.ssh'.
 +
Your identification has been saved in /home/maxin/.ssh/id_rsa
 +
Your public key has been saved in /home/maxin/.ssh/id_rsa.pub
 +
The key fingerprint is:
 +
SHA256:oJvwuALPDCs2eQK8VmAk8GYL1APNhGwKz2zT9Xho2jw maxin@maxin-virtual-machine
 +
The key's randomart image is:
 +
+---[RSA 3072]----+
 +
|+oB.            |
 +
|+* =  .          |
 +
|B++....+        |
 +
|.**...+.o        |
 +
|o.+..= .S        |
 +
|+. =.oE          |
 +
|o*= +  .        |
 +
|+B+o            |
 +
|+.=              |
 +
+----[SHA256]-----+</nowiki>
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ chmod 0600 ~/.ssh/authorized_keys
 +
 
 +
检验ssh:
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh localhost
 +
 
 +
<nowiki>The authenticity of host 'localhost (127.0.0.1)' can't be established.
 +
ECDSA key fingerprint is SHA256:5pnkOaodDfEffo7VJnMJl1DuTDM3rhbsy7lTDHKr2xk.
 +
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
 +
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
 +
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.8.0-38-generic x86_64)
 +
 
 +
* Documentation:  https://help.ubuntu.com
 +
* Management:    https://landscape.canonical.com
 +
* Support:        https://ubuntu.com/advantage
 +
 
 +
0 updates can be installed immediately.
 +
0 of these updates are security updates.
 +
 
 +
Your Hardware Enablement Stack (HWE) is supported until April 2025.
 +
 
 +
The programs included with the Ubuntu system are free software;
 +
the exact distribution terms for each program are described in the
 +
individual files in /usr/share/doc/*/copyright.
 +
 
 +
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
 +
applicable law.</nowiki>
 +
 
 +
maxin@maxin-virtual-machine:~$ ls
 +
 
 +
<nowiki>公共的  视频  文档  音乐  hadoop-2.7.6        jdk-8u221-linux-x64.tar.gz
 +
模板    图片  下载  桌面  hadoop-2.7.6.tar.gz</nowiki>
 +
 
 +
maxin@maxin-virtual-machine:~$ exit
 +
 
 +
<nowiki>注销
 +
Connection to localhost closed.</nowiki>
 +
 
 +
==格式化hdfs节点==
 +
Namenode 相当于主节点
 +
 
 +
bin/hdfs namenode -format
 +
 
 +
<nowiki>21/01/20 09:57:10 INFO namenode.FSImage: Allocated new BlockPoolId: BP-924457617-127.0.1.1-1611107830921
 +
21/01/20 09:57:10 INFO common.Storage: Storage directory /home/maxin/hadoop-2.7.6/data/tmp/dfs/name has been successfully formatted.
 +
</nowiki>
 +
 
 +
==启动hdfs,并检验==
 +
maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls
 +
 
 +
<nowiki>bin  etc      lib      LICENSE.txt  README.txt  share
 +
data  include  libexec  NOTICE.txt  sbin</nowiki>
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ cd sbin
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ start-dfs.sh
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ jps
 +
 
 +
<nowiki>6672 DataNode
 +
6516 NameNode
 +
7081 Jps
 +
6861 SecondaryNameNode</nowiki>
 +
 
 +
在浏览器打开：
 +
 
 +
http://10.0.0.128:50070
 +
 
 +
==配置mapreduce==
 +
配置yarn-site.xml，mapred-site.xml，yarn-evn.sh
 +
 
 +
将已配置好的文件（网盘） 上传至
 +
 
 +
使用start-all.sh启动整个集群
 +
 
 +
maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ jps
 +
 
 +
<nowiki>8353 DataNode
 +
8549 SecondaryNameNode
 +
8812 NodeManager
 +
8684 ResourceManager
 +
8189 NameNode
 +
9102 Jps
 +
</nowiki>

2021年1月20日 (三) 02:46的最新版本


复制和移动文件

复制hadoop-2.7.6 和jdk-8u221-linux-x64的压缩包至ubuntu主目录

分别解压他们

然后移动jdk的目录： 

maxin@maxin-virtual-machine:~/下载$ sudo mkdir /usr/lib/jvm
[sudo] maxin 的密码： 
maxin@maxin-virtual-machine:~/下载$ ls
'新建 DOCX 文档.docx'   VMwareTools-10.3.10-13959562.tar.gz
 npm-debug.log          vmware-tools-distrib
maxin@maxin-virtual-machine:~/下载$ cd ..
maxin@maxin-virtual-machine:~$ ls
公共的  视频  文档  音乐  hadoop-2.7.6         jdk-8u221-linux-x64
模板    图片  下载  桌面  hadoop-2.7.6.tar.gz  jdk-8u221-linux-x64.tar.gz
maxin@maxin-virtual-machine:~$ sudo mv jdk-8u221-linux-x64 /usr/lib/jvm/java-8-oracle
maxin@maxin-virtual-machine:~$ ls /usr/lib/jvm/java-8-oracle
jdk1.8.0_221
maxin@maxin-virtual-machine:~$

新增路径至环境变量

gedit ~/.bashrc 

export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jdk1.8.0_221
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/home/maxin/hadoop-2.7.6  
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

检测java

java -version 

java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

配置hadoop-env.sh

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ pwd 

/home/maxin/hadoop-2.7.6

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ mkdir -p data/tmp

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls 

bin   etc      lib      LICENSE.txt  README.txt  share
data  include  libexec  NOTICE.txt   sbin

maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6/etc/hadoop/

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hadoop-env.sh 

export JAVA_HOME=/bigData/jdk1.8.0_121
export HADOOP_HOME=/home/maxin/hadoop-2.7.6

配置hdfs

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ pwd 

/home/maxin/hadoop-2.7.6/etc/hadoop

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hdfs-site.xml 

 <configuration>
    <property>
         <name>dfs.replication</name>
         <value>1</value>
   </property>
</configuration>

安装并配置ssh登录

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh-keygen -t rsa -P -f ~/.ssh/id_rsa 

Generating public/private rsa key pair.
Created directory '/home/maxin/.ssh'.
Your identification has been saved in /home/maxin/.ssh/id_rsa
Your public key has been saved in /home/maxin/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:oJvwuALPDCs2eQK8VmAk8GYL1APNhGwKz2zT9Xho2jw maxin@maxin-virtual-machine
The key's randomart image is:
+---[RSA 3072]----+
|+oB.             |
|+* =  .          |
|B++....+         |
|.**...+.o        |
|o.+..= .S        |
|+. =.oE          |
|o*= +  .         |
|+B+o             |
|+.=              |
+----[SHA256]-----+

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ chmod 0600 ~/.ssh/authorized_keys

检验ssh:

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh localhost 

The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:5pnkOaodDfEffo7VJnMJl1DuTDM3rhbsy7lTDHKr2xk.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.8.0-38-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

0 updates can be installed immediately.
0 of these updates are security updates.

Your Hardware Enablement Stack (HWE) is supported until April 2025.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

maxin@maxin-virtual-machine:~$ ls 

公共的  视频  文档  音乐  hadoop-2.7.6         jdk-8u221-linux-x64.tar.gz
模板    图片  下载  桌面  hadoop-2.7.6.tar.gz

maxin@maxin-virtual-machine:~$ exit 

注销
Connection to localhost closed.

格式化hdfs节点

Namenode 相当于主节点

bin/hdfs namenode -format 

21/01/20 09:57:10 INFO namenode.FSImage: Allocated new BlockPoolId: BP-924457617-127.0.1.1-1611107830921
21/01/20 09:57:10 INFO common.Storage: Storage directory /home/maxin/hadoop-2.7.6/data/tmp/dfs/name has been successfully formatted.

启动hdfs,并检验

maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls 

bin   etc      lib      LICENSE.txt  README.txt  share
data  include  libexec  NOTICE.txt   sbin

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ cd sbin

maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ start-dfs.sh

maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ jps 

6672 DataNode
6516 NameNode
7081 Jps
6861 SecondaryNameNode

在浏览器打开：

http://10.0.0.128:50070

配置mapreduce

配置yarn-site.xml，mapred-site.xml，yarn-evn.sh

将已配置好的文件（网盘） 上传至

使用start-all.sh启动整个集群

maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ jps 

8353 DataNode
8549 SecondaryNameNode
8812 NodeManager
8684 ResourceManager
8189 NameNode
9102 Jps
取自“http://www.openbrains.net/mediawiki/index.php?title=2021高训真源：Hadoop伪分布式&oldid=24664”