Hadoop

Define and install a rack topology script

In network terminology, all the physical servers in the network should’ve been present in a rack in the data center. In hadoop, the racks assignment is significant as it plays a vital role in terms of data locality, bandwidth etc.

We can assign the rack of the hosts in the cluster in two ways.

One is to manually assign the rack in hosts section – assign racks.

The second one is to provide a rack topology script and list of servers – rack details as an separate input file.

Below is the rack topology script which takes topology.data file as an input. CM server will cross check the hosts reporting to them with the hosts information given in the topology.data file. If the host is not present or no rack info given in the file, CM will assign default rack to the host.

/hadoop/topology.script

-----
#!/bin/bash 

while [ $# -gt 0 ] ; do
  nodeArg=$1
  exec< /hadoop/topology.data 
  result="" 
  while read line ; do
    ar=( $line ) 
    if [ "${ar[0]}" = "$nodeArg" ] ; then
      result="${ar[1]}"
    fi
  done 
  shift 
  if [ -z "$result" ] ; then
    echo -n "/default/rack "
  else
    echo -n "$result "
  fi
done 

# cat /hadoop/topology.data 
master     /rack1
master2    /rack2
worker1    /rack3
worker2    /rack3
worker3    /rack2

Specify the script location in the below field.

HDFS – Configuration –

 

7 thoughts on “Define and install a rack topology script

  1. Hi,Kannan
    Thanks for the posts you wrote!
    Could you share some real CCA131 exam questions that you encountered?
    I thought write a rack topology script is too hard for CCA131 certificate,and will not be in the CCA131 certificate.
    Please tell me some questions you remember in your exam!

    1. Not required. You can assign manually in the hosts section. This is just to know an alternate approach as the blueprint topic implies.

  2. Hi Kannan

    net.topology.script.file.name #parameter is missing in the HDFS -> configuration -> search for the parameter

    Thanks
    krish

Leave a Reply

Your email address will not be published. Required fields are marked *