Skip to content

network problems

by on December 25, 2012

HBase logo

A co-worker who checks the HBase  on one of our cluster say that “things are going slow”

trying tocopy data into hadoop takes ages instead of seconds “hadoop dfs -copyfromlocal data0.trv /tmp/data.trv

So – we’ve started to check the network using the iperf tool

We have a 10GB network, but when checking we saw that some connections give us less than 1GB.

I’ve requested an IT network expert to have a look in the switch (IBM BNT 68264R), but he could not find anything.
Note to myself – have a list of all MAC and to which physical port in the switch each server is connected.

I’ve rebooted the switch (unplugged the power) to see if the issue is with the switch.

I’ve changed cables

and the weirdest thing: after switching cables and ports and going back to the old cable, things got suddenly better!

it seems re-plugging the cables does some work.
Is it because they are SFP+ with some electrical signature?

Waiting anxiously for the RJ-45 10GB network!

Anyways – I’m going to play some more with the cables to see if it does any help.

meanwhile some scripts (as I have 15 servers in this cluster, i need to check connection between all of them):

start_iperf_server.sh

#!/bin/bash
nohup iperf -s -D > /tmp/iperf_server_`hostname`_`date +%Y%m%d_%H%M%S`.log 2>&1 &

check_iperf.sh
#!/bin/bash
pgrep -fl iperf
ls -lrt /tmp/iperf*

stop_iperf.sh
#!/bin/bash
pgrep -fl iperf
pkill -9 iperf
pgrep -fl iperf

run_iperf_loop.sh
#!/bin/bash
for i in `seq 11 25`
do
 iperf -c host$i
done

Advertisements

From → Uncategorized

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: