2021高训真源：Hadoop伪分布式

复制和移动文件

复制hadoop-2.7.6 和jdk-8u221-linux-x64的压缩包至ubuntu主目录

分别解压他们

然后移动jdk的目录：

maxin@maxin-virtual-machine:~/下载$ sudo mkdir /usr/lib/jvm
[sudo] maxin 的密码： 
maxin@maxin-virtual-machine:~/下载$ ls
'新建 DOCX 文档.docx'   VMwareTools-10.3.10-13959562.tar.gz
 npm-debug.log          vmware-tools-distrib
maxin@maxin-virtual-machine:~/下载$ cd ..
maxin@maxin-virtual-machine:~$ ls
公共的  视频  文档  音乐  hadoop-2.7.6         jdk-8u221-linux-x64
模板    图片  下载  桌面  hadoop-2.7.6.tar.gz  jdk-8u221-linux-x64.tar.gz
maxin@maxin-virtual-machine:~$ sudo mv jdk-8u221-linux-x64 /usr/lib/jvm/java-8-oracle
maxin@maxin-virtual-machine:~$ ls /usr/lib/jvm/java-8-oracle
jdk1.8.0_221
maxin@maxin-virtual-machine:~$

新增路径至环境变量

gedit ~/.bashrc

export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jdk1.8.0_221
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/home/maxin/hadoop-2.7.6  
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

检测java

java -version

java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

配置hadoop-env.sh

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ pwd

/home/maxin/hadoop-2.7.6

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ mkdir -p data/tmp

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls

bin   etc      lib      LICENSE.txt  README.txt  share
data  include  libexec  NOTICE.txt   sbin

maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6/etc/hadoop/

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hadoop-env.sh

export JAVA_HOME=/bigData/jdk1.8.0_121
export HADOOP_HOME=/home/maxin/hadoop-2.7.6

配置hdfs

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ pwd

/home/maxin/hadoop-2.7.6/etc/hadoop

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hdfs-site.xml

 <configuration>
    <property>
         <name>dfs.replication</name>
         <value>1</value>
   </property>
</configuration>

安装并配置ssh登录

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh-keygen -t rsa -P -f ~/.ssh/id_rsa

Generating public/private rsa key pair.
Created directory '/home/maxin/.ssh'.
Your identification has been saved in /home/maxin/.ssh/id_rsa
Your public key has been saved in /home/maxin/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:oJvwuALPDCs2eQK8VmAk8GYL1APNhGwKz2zT9Xho2jw maxin@maxin-virtual-machine
The key's randomart image is:
+---[RSA 3072]----+
|+oB.             |
|+* =  .          |
|B++....+         |
|.**...+.o        |
|o.+..= .S        |
|+. =.oE          |
|o*= +  .         |
|+B+o             |
|+.=              |
+----[SHA256]-----+

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ chmod 0600 ~/.ssh/authorized_keys

检验ssh:

maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh localhost

The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:5pnkOaodDfEffo7VJnMJl1DuTDM3rhbsy7lTDHKr2xk.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.8.0-38-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

0 updates can be installed immediately.
0 of these updates are security updates.

Your Hardware Enablement Stack (HWE) is supported until April 2025.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

maxin@maxin-virtual-machine:~$ ls

公共的  视频  文档  音乐  hadoop-2.7.6         jdk-8u221-linux-x64.tar.gz
模板    图片  下载  桌面  hadoop-2.7.6.tar.gz

maxin@maxin-virtual-machine:~$ exit

注销
Connection to localhost closed.

格式化hdfs节点

Namenode 相当于主节点

bin/hdfs namenode -format

21/01/20 09:57:10 INFO namenode.FSImage: Allocated new BlockPoolId: BP-924457617-127.0.1.1-1611107830921
21/01/20 09:57:10 INFO common.Storage: Storage directory /home/maxin/hadoop-2.7.6/data/tmp/dfs/name has been successfully formatted.

启动hdfs,并检验

maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls

bin   etc      lib      LICENSE.txt  README.txt  share
data  include  libexec  NOTICE.txt   sbin

maxin@maxin-virtual-machine:~/hadoop-2.7.6$ cd sbin

maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ start-dfs.sh

maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ jps

6672 DataNode
6516 NameNode
7081 Jps
6861 SecondaryNameNode

在浏览器打开：

http://10.0.0.128:50070

配置mapreduce

配置yarn-site.xml，mapred-site.xml，yarn-evn.sh

将已配置好的文件（网盘）上传至

使用start-all.sh启动整个集群