Hadoop

How your resume shouldn’t be – observations on screening profiles for Hadoop Admin role

I've screened around 50 profiles/resume in recent months for the technical round of Hadoop Admin/Engineer position and it's shocking to see how terrible the candidates resumes are. All the resumes came through Consultancies and they do have major share for the poor quality of those resumes. I'm listing the below points I observed during screening… Continue reading How your resume shouldn’t be – observations on screening profiles for Hadoop Admin role

Hadoop

Hadoop Cluster – Pre Maintenance procedure

In IT, it's inevitable that all the servers will go for monthly security, vulnerability patching and hadoop servers are no exception. There'd be a separate Systems team to perform OS related patching, security updates, etc and your role is to bring down/up the cluster, ensure the application is good post patching. You've have to schedule… Continue reading Hadoop Cluster – Pre Maintenance procedure

Hadoop

Configure the Fair Scheduler to resolve application delays

Fair scheduling is a method of assigning resources to applications such that all apps get, on average, an equal share of resources over time. Using fair scheduler we can separate pools(queues) for each team and configure the resources for the pool which will help in overcoming application delays. In the exam you may be asked… Continue reading Configure the Fair Scheduler to resolve application delays

Hadoop

Determine reason for application failure

There are many possible causes for a job/application failure varying from code error, environments, files availability, permissions, mapreduce/yarn configuration, resources allocation and even due to server i/o, network issue etc., So the first thing you’ve to do when a job fails is, to look at the error message and correlate with your job. If an… Continue reading Determine reason for application failure

Hadoop

Benchmark the cluster (I/O, CPU, network)

Benchmarking is the process of stress testing the resources of the cluster. It’s very useful in understanding the performance of your cluster and to check whether it’s performing as expected before taking it live. Here we are going to test speed in which files are being read/write in HDFS, time taken for mappers/reducers to process… Continue reading Benchmark the cluster (I/O, CPU, network)

Hadoop

Resolve performance problems/errors in cluster operation

This is again another scenario based topic. Some of the common performance problems are jobs running slowly, services crash due to out of memory, etc., Example: When I’m adding Yarn roles, one of the node managers failed to start. In this case, we have to identify the cause for the failure. Select Log files dropdown… Continue reading Resolve performance problems/errors in cluster operation

Hadoop

Resolve errors/warnings in Cloudera Manager

This is a typical scenario based question and the solution is solely depend upon the errors/warnings appears in the Cloudera Manager. Some examples: The warnings could be space issue, service health status, low resources allocations, etc., The errors could be log directories are full, services down and other critical events.     In these scenarios,… Continue reading Resolve errors/warnings in Cloudera Manager