查看“2021高训真源：Hadoop伪分布式”的源代码



==复制和移动文件==
复制hadoop-2.7.6 和jdk-8u221-linux-x64的压缩包至ubuntu主目录

分别解压他们

然后移动jdk的目录：

 <nowiki>maxin@maxin-virtual-machine:~/下载$ sudo mkdir /usr/lib/jvm
[sudo] maxin 的密码： 
maxin@maxin-virtual-machine:~/下载$ ls
'新建 DOCX 文档.docx'   VMwareTools-10.3.10-13959562.tar.gz
 npm-debug.log          vmware-tools-distrib
maxin@maxin-virtual-machine:~/下载$ cd ..
maxin@maxin-virtual-machine:~$ ls
公共的  视频  文档  音乐  hadoop-2.7.6         jdk-8u221-linux-x64
模板    图片  下载  桌面  hadoop-2.7.6.tar.gz  jdk-8u221-linux-x64.tar.gz
maxin@maxin-virtual-machine:~$ sudo mv jdk-8u221-linux-x64 /usr/lib/jvm/java-8-oracle
maxin@maxin-virtual-machine:~$ ls /usr/lib/jvm/java-8-oracle
jdk1.8.0_221
maxin@maxin-virtual-machine:~$</nowiki> 

==新增路径至环境变量==
gedit ~/.bashrc

 <nowiki>export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jdk1.8.0_221
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/home/maxin/hadoop-2.7.6  
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin</nowiki>

检测java

java -version

 <nowiki>java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
</nowiki>

==配置hadoop-env.sh==
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ pwd

 /home/maxin/hadoop-2.7.6

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ mkdir -p data/tmp

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls

 <nowiki>bin   etc      lib      LICENSE.txt  README.txt  share
data  include  libexec  NOTICE.txt   sbin</nowiki>

maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6/etc/hadoop/

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hadoop-env.sh

 export JAVA_HOME=/bigData/jdk1.8.0_121
 export HADOOP_HOME=/home/maxin/hadoop-2.7.6

==配置hdfs==
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ pwd

 /home/maxin/hadoop-2.7.6/etc/hadoop

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hdfs-site.xml

  <nowiki><configuration>
    <property>
         <name>dfs.replication</name>
         <value>1</value>
   </property>
</configuration></nowiki>

==安装并配置ssh登录==
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

 <nowiki>Generating public/private rsa key pair.
Created directory '/home/maxin/.ssh'.
Your identification has been saved in /home/maxin/.ssh/id_rsa
Your public key has been saved in /home/maxin/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:oJvwuALPDCs2eQK8VmAk8GYL1APNhGwKz2zT9Xho2jw maxin@maxin-virtual-machine
The key's randomart image is:
+---[RSA 3072]----+
|+oB.             |
|+* =  .          |
|B++....+         |
|.**...+.o        |
|o.+..= .S        |
|+. =.oE          |
|o*= +  .         |
|+B+o             |
|+.=              |
+----[SHA256]-----+</nowiki>
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ chmod 0600 ~/.ssh/authorized_keys

检验ssh:

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh localhost

 <nowiki>The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:5pnkOaodDfEffo7VJnMJl1DuTDM3rhbsy7lTDHKr2xk.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.8.0-38-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

0 updates can be installed immediately.
0 of these updates are security updates.

Your Hardware Enablement Stack (HWE) is supported until April 2025.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.</nowiki>

maxin@maxin-virtual-machine:~$ ls

 <nowiki>公共的  视频  文档  音乐  hadoop-2.7.6         jdk-8u221-linux-x64.tar.gz
模板    图片  下载  桌面  hadoop-2.7.6.tar.gz</nowiki>

maxin@maxin-virtual-machine:~$ exit

 <nowiki>注销
Connection to localhost closed.</nowiki>

==格式化hdfs节点==
Namenode 相当于主节点

bin/hdfs namenode -format

 <nowiki>21/01/20 09:57:10 INFO namenode.FSImage: Allocated new BlockPoolId: BP-924457617-127.0.1.1-1611107830921
21/01/20 09:57:10 INFO common.Storage: Storage directory /home/maxin/hadoop-2.7.6/data/tmp/dfs/name has been successfully formatted.
</nowiki>

==启动hdfs,并检验==
maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls

 <nowiki>bin   etc      lib      LICENSE.txt  README.txt  share
data  include  libexec  NOTICE.txt   sbin</nowiki>

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ cd sbin

maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ start-dfs.sh

maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ jps

 <nowiki>6672 DataNode
6516 NameNode
7081 Jps
6861 SecondaryNameNode</nowiki>

在浏览器打开：

http://10.0.0.128:50070

==配置mapreduce==
配置yarn-site.xml，mapred-site.xml，yarn-evn.sh

将已配置好的文件（网盘） 上传至

使用start-all.sh启动整个集群

maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ jps

 <nowiki>8353 DataNode
8549 SecondaryNameNode
8812 NodeManager
8684 ResourceManager
8189 NameNode
9102 Jps
</nowiki>