HDFS的管理

HDFS文件系统

块

HDFS文件被分成块进行存储

HDFS的默认大小是128MB

两类节点：Namenode datanode

namenode是管理节点，存放文件元数据

1. 文件与数据块的映射表

2.数据块与数据节点的映射表

datanode是HDFS的工作节点，存放文件内容

每个数据块都有三个副本，防止某个节点或某个机架出现问题导致的数据丢失,分布在两个机架内的三个节点

文件:Hd2-3.png

secondnamenode : 存储映射文件和定期同步数据

HDFS读写文件的流程

流程

HDFS 访问

1.命令行访问

[root@master ~]# hadoop fs -ls hdfs://master.hadoop:8020 Found 1 items drwx------ - root hdfs 0 2017-12-20 01:56 hdfs://master.hadoop:8020/user/root/.staging

2.Web界面访问

3.JAVA访问

[root@master ~]# hadoop fs -ls hdfs://master.hadoop:8020 Found 1 items drwx------ - root hdfs 0 2017-12-20 01:56 hdfs://master.hadoop:8020/user/root/.staging

[root@master ~]# hadoop fs -help Usage: hadoop fs [generic options]

       [-appendToFile <localsrc> ... <dst>]
       [-cat [-ignoreCrc] <src> ...]
       [-checksum <src> ...]
       [-chgrp [-R] GROUP PATH...]
       [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
       [-chown [-R] [OWNER][:[GROUP]] PATH...]
       [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
       [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
       [-count [-q] [-h] [-v] [-t [<storage type>]] <path> ...]
       [-cp [-f] [-p | -p[topax]] <src> ... <dst>]
       [-createSnapshot <snapshotDir> [<snapshotName>]]
       [-deleteSnapshot <snapshotDir> <snapshotName>]
       [-df [-h] [<path> ...]]
       [-du [-s] [-h] <path> ...]
       [-expunge]
       [-find <path> ... <expression> ...]
       [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
       [-getfacl [-R] <path>]
       [-getfattr [-R] {-n name | -d} [-e en] <path>]
       [-getmerge [-nl] <src> <localdst>]
       [-help [cmd ...]]
       [-ls [-d] [-h] [-R] [<path> ...]]
       [-mkdir [-p] <path> ...]
       [-moveFromLocal <localsrc> ... <dst>]
       [-moveToLocal <src> <localdst>]
       [-mv <src> ... <dst>]
       [-put [-f] [-p] [-l] <localsrc> ... <dst>]
       [-renameSnapshot <snapshotDir> <oldName> <newName>]
       [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
       [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
       [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
       [-setfattr {-n name [-v value] | -x name} <path>]
       [-setrep [-R] [-w] <rep> <path> ...]
       [-stat [format] <path> ...]
       [-tail [-f] <file>]
       [-test -[defsz] <path>]
       [-text [-ignoreCrc] <src> ...]
       [-touchz <path> ...]
       [-truncate [-w] <length> <path> ...]
       [-usage [cmd ...]]

创建目录

 hadoop fs -mkdir /1daoyun
 hadoop fs -ls /

 服务账户：
   每一个服务都对应着一个用户

   Smoke User	ambari-qa

Hadoop Group hadoop HDFS User hdfs Proxy User Group users HCat User hcat Hive User hive WebHCat User hcat Mapreduce User mapred Sqoop User sqoop Tez User tez Yarn User yarn ZooKeeper User zookeeper

  如果报错peermission deny,可以通过切换用户来解决： su hdfs

 [root@master ~]# su hdfs

[hdfs@master root]$ hadoop fs -chmod -R 777 / [hdfs@master root]$ su Password:

Found 9 items drwxrwxrwx - root hdfs 0 2018-07-24 01:34 /1daoyun drwxrwxrwx - yarn hadoop 0 2017-12-20 01:56 /app-logs drwxrwxrwx - hdfs hdfs 0 2017-12-19 08:51 /apps drwxrwxrwx - yarn hadoop 0 2017-12-19 08:48 /ats drwxrwxrwx - hdfs hdfs 0 2017-12-19 08:48 /hdp drwxrwxrwx - mapred hdfs 0 2017-12-19 08:48 /mapred drwxrwxrwx - mapred hadoop 0 2017-12-19 08:48 /mr-history

上传文件

[root@master ~]# cat > test.txt hello weihai [root@master ~]# hadoop fs -put test.txt /1daoyun [root@master ~]# hadoop fs -ls /1daoyun/ Found 1 items -rw-r--r-- 3 root hdfs 13 2018-07-24 02:18 /1daoyun/test.txt [root@master ~]# hadoop fs -cat /1daoyun/test.txt hello weihai