2021高训真源：Hadoop伪分布式
复制和移动文件
复制hadoop-2.7.6 和jdk-8u221-linux-x64的压缩包至ubuntu主目录
分别解压他们
然后移动jdk的目录：
maxin@maxin-virtual-machine:~/下载$ sudo mkdir /usr/lib/jvm [sudo] maxin 的密码： maxin@maxin-virtual-machine:~/下载$ ls '新建 DOCX 文档.docx' VMwareTools-10.3.10-13959562.tar.gz npm-debug.log vmware-tools-distrib maxin@maxin-virtual-machine:~/下载$ cd .. maxin@maxin-virtual-machine:~$ ls 公共的 视频 文档 音乐 hadoop-2.7.6 jdk-8u221-linux-x64 模板 图片 下载 桌面 hadoop-2.7.6.tar.gz jdk-8u221-linux-x64.tar.gz maxin@maxin-virtual-machine:~$ sudo mv jdk-8u221-linux-x64 /usr/lib/jvm/java-8-oracle maxin@maxin-virtual-machine:~$ ls /usr/lib/jvm/java-8-oracle jdk1.8.0_221 maxin@maxin-virtual-machine:~$
新增路径至环境变量
gedit ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jdk1.8.0_221
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/home/maxin/hadoop-2.7.6
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
检测java
java -version
java version "1.8.0_221" Java(TM) SE Runtime Environment (build 1.8.0_221-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
配置hadoop-env.sh
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ pwd
/home/maxin/hadoop-2.7.6
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ mkdir -p data/tmp
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls
bin etc lib LICENSE.txt README.txt share data include libexec NOTICE.txt sbin
maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6/etc/hadoop/
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hadoop-env.sh
export JAVA_HOME=/bigData/jdk1.8.0_121 export HADOOP_HOME=/home/maxin/hadoop-2.7.6
配置hdfs
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ pwd
/home/maxin/hadoop-2.7.6/etc/hadoop
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ gedit hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
安装并配置ssh登录
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh-keygen -t rsa -P -f ~/.ssh/id_rsa
Generating public/private rsa key pair. Created directory '/home/maxin/.ssh'. Your identification has been saved in /home/maxin/.ssh/id_rsa Your public key has been saved in /home/maxin/.ssh/id_rsa.pub The key fingerprint is: SHA256:oJvwuALPDCs2eQK8VmAk8GYL1APNhGwKz2zT9Xho2jw maxin@maxin-virtual-machine The key's randomart image is: +---[RSA 3072]----+ |+oB. | |+* = . | |B++....+ | |.**...+.o | |o.+..= .S | |+. =.oE | |o*= + . | |+B+o | |+.= | +----[SHA256]-----+
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ chmod 0600 ~/.ssh/authorized_keys
检验ssh:
maxin@maxin-virtual-machine:~/hadoop-2.7.6/etc/hadoop$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established. ECDSA key fingerprint is SHA256:5pnkOaodDfEffo7VJnMJl1DuTDM3rhbsy7lTDHKr2xk. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.8.0-38-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage 0 updates can be installed immediately. 0 of these updates are security updates. Your Hardware Enablement Stack (HWE) is supported until April 2025. The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law.
maxin@maxin-virtual-machine:~$ ls
公共的 视频 文档 音乐 hadoop-2.7.6 jdk-8u221-linux-x64.tar.gz 模板 图片 下载 桌面 hadoop-2.7.6.tar.gz
maxin@maxin-virtual-machine:~$ exit
注销 Connection to localhost closed.
格式化hdfs节点
Namenode 相当于主节点
bin/hdfs namenode -format
21/01/20 09:57:10 INFO namenode.FSImage: Allocated new BlockPoolId: BP-924457617-127.0.1.1-1611107830921 21/01/20 09:57:10 INFO common.Storage: Storage directory /home/maxin/hadoop-2.7.6/data/tmp/dfs/name has been successfully formatted.
启动hdfs,并检验
maxin@maxin-virtual-machine:~$ cd hadoop-2.7.6
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ ls
bin etc lib LICENSE.txt README.txt share data include libexec NOTICE.txt sbin
maxin@maxin-virtual-machine:~/hadoop-2.7.6$ cd sbin
maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ start-dfs.sh
maxin@maxin-virtual-machine:~/hadoop-2.7.6/sbin$ jps
6672 DataNode 6516 NameNode 7081 Jps 6861 SecondaryNameNode
在浏览器打开：
