Hadoop Balancer fails with - IOException: Couldn't set up IO streams (LeaseRenewer Warning)
NickName:Yash Sharma Ask DateTime:2014-05-09T14:22:29

Hadoop Balancer fails with - IOException: Couldn't set up IO streams (LeaseRenewer Warning)

I am stumbling across this error while running the Hadoop Balancer via Namenode. Anytips on cracking this. The process is also blocking the current user and giving an Out of Memory error on issuing any other command.

   14/05/09 11:30:05 WARN hdfs.LeaseRenewer: Failed to renew lease for [DFSClient_NONMAPREDUCE_-77290934_1] for 936 seconds.  Will retry shortly ...

 java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams; Host Details : local host is: "hadoop01.xx.xx.xx.xx.com/30.0.1.176"; destination host is: "hadoop01.xx.xx.xx.xx.com":8022; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
        at org.apache.hadoop.ipc.Client.call(Client.java:1242)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at $Proxy10.renewLease(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at $Proxy10.renewLease(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:458)
        at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:649)
        at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
        at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
        at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
        at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
        at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.IOException: Couldn't set up IO streams
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:671)
        at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1291)
        at org.apache.hadoop.ipc.Client.call(Client.java:1209)
        ... 15 more
    Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:664)
        ... 18 more

Copyright Notice:Content Author:「Yash Sharma」,Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/23557879/hadoop-balancer-fails-with-ioexception-couldnt-set-up-io-streams-leaserenew

More about “Hadoop Balancer fails with - IOException: Couldn't set up IO streams (LeaseRenewer Warning)” related questions

Hadoop Balancer fails with - IOException: Couldn't set up IO streams (LeaseRenewer Warning)

I am stumbling across this error while running the Hadoop Balancer via Namenode. Anytips on cracking this. The process is also blocking the current user and giving an Out of Memory error on issuing...

Show Detail

Balancer not working in hdfs HA

When I try to run balancer: HADOOP_USER_NAME=hdfs hdfs --config /etc/hadoop/conf.balancer/conf/ balancer -threshold 5 or HADOOP_USER_NAME=hdfs hdfs balancer -fs hdfs://hdp I get the following ...

Show Detail

Hadoop3 balancer vs disk balancer

I read Hadoop ver 3 document about disk balancer and it said "Diskbalancer is a command line tool that distributes data evenly on all disks of a datanode. This tool is different from Balancer ...

Show Detail

How to run hadoop balancer from client node?

I want to ask how can I run the hadoop balancer? I've tried before on the namenode to run hadoop balancer command, but it has no effect at all (my new datanode still empty). I also read that hadoop

Show Detail

Hdfs balancer not working

I'm sure nobody started the balancing, what could be wrong? Balancing run thus: # hdfs balancer -threshold 1 16/02/19 15:10:02 INFO balancer.Balancer: Using a threshold of 1.0 16/02/...

Show Detail

How to run hadoop cluster balancer from gateway machine?

I have written a script which checks Hadoop block report and if needed it runs hadoop balancer. I read this article about hdfs balancer design and it looks like we need to run HDFS balancer on sep...

Show Detail

How to Use HDFS Intra-DataNode Disk Balancer in Apache Hadoop 2?

We use Hadoop 2.7.3. HDFS Disk Balancer is a new feature for Hadoop3. What is the best alternative way to do HDFS Disk balancer in Hadoop2?

Show Detail

Hadoop namenode format fails on mac

I am trying to install Hadoop to try it out on my MacBook Pro. I am following the steps in this article, http://dennyglee.com/2012/05/08/installing-hadoop-on-osx-lion-10-7/. However, I am failing...

Show Detail

CDH5 hdfs balancer error

running 'hdfs balancer', i do see it showing the rack topology installed, over-utilized and underutilized nodes, it says "Need to move 5 TB to make the cluster balanced" and "Decided to move 10 GB ...

Show Detail

Hadoop DistributedCache causes IOException

I have written some code to do a selfjoin task using Hadoop. For this purpose I use DistributedCache class.When I run the code locally in Netbeans the job is done correctly but when I try to run it...

Show Detail