JAVA API 连接 HA(High Available) Hadoop集群

使用JAVA API连接HDFS时我们需要使用NameNode的地址,开启HA后,两个NameNode可能会主备切换,如果连接的那台主机NameNode挂掉了,连接就会失败. HDFS提供了nameservices的方式进行访问,这样只要有一个NameNode活着,都可以正常访问.

HDFS NameNode HA

在没有HA的环境中,通常使用NameNode hostname访问HDFS的URL.

hdfs://ochadoop111.jcloud.local:8020

为了保证HDFS服务的高可用,生产环境是必须要开启NameNode HA的,此时应该用nameservices作为统一的logical name连接HDFS.

使用Ambari Enable NameNode HA之后,我的集群在ochadoop112.jcloud.local上增加了一个NameNode, HA相关的配置会自动产生.

使用nameservices访问HDFS的URL:

hdfs://mycluster:8020

JAVA API连接HDFS HA代码如下:

1Configuration conf=new Configuration(false);
2String nameservices = "mycluster";
3String[] namenodesAddr = {"ochadoop111.jcloud.local:8020","ochadoop112.jcloud.local:8020"};
4String[] namenodes = {"nn1","nn2"};
5conf.set("fs.defaultFS", "hdfs://" + nameservices);
6conf.set("dfs.nameservices",nameservices);
7conf.set("dfs.ha.namenodes." + nameservices, namenodes[0]+","+namenodes[1]);
8conf.set("dfs.namenode.rpc-address." + nameservices + "." + namenodes[0], namenodesAddr[0]);
9conf.set("dfs.namenode.rpc-address." + nameservices + "." + namenodes[1], namenodesAddr[1]);
10conf.set("dfs.client.failover.proxy.provider." + nameservices,"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
11String hdfsRPCUrl = "hdfs://" + nameservices + ":" + 8020;
12DistributedFileSystem dfs = new DistributedFileSystem();
13try {
14    dfs.initialize(URI.create(hdfsRPCUrl),conf);
15    Path tmpPath2 = new Path("/tmp2");
16    dfs.mkdir(tmpPath2,new FsPermission("777"));
17    FileStatus[] list = dfs.listStatus(new Path("/"));
18    for (FileStatus file : list) {
19        System.out.println(file.getPath());
20    }
21    dfs.setQuota(tmpPath2,100,1000);
22} catch (IOException e) {
23    e.printStackTrace();
24} finally{
25    try {
26        dfs.close();
27    } catch (IOException e) {
28        e.printStackTrace();
29    }
30}