Commission/decommission a node

When you want to remove the node from the cluster, you shouldn’t just delete the cloudera agents, services installed as it will impact the whole cluster. You should go for decommission first. Decommissioning a host decommissions and stops all roles on the host without requiring you to individually decommission the roles on each service. After […]

Continue Reading

Rebalance the cluster

In HDFS, the blocks of the files are distributed among the datanodes as per the replication factor. Whenever you add a new datanode, the node will start receiving,storing the blocks of the new files. Though this sounds alright, the cluster is not balanced when you look at administrative point view. HDFS provides a balancer utility […]

Continue Reading

Configure proxy for Hiveserver2/Impala

A proxy server is a server or application that acts as an intermediary for requests from clients seeking resources from other servers/applications. A client connects to the proxy server, requesting some service or resource available from a different server and the proxy server evaluates the request and sends it to the intended server/service/application. The server’s […]

Continue Reading

Configure ResourceManager HA

The YARN ResourceManager is responsible for tracking the resources in a cluster and scheduling applications. Before CDH 5, the ResourceManager was a single point of failure in a YARN cluster. If Resource manager is down, then none of the jobs will run in the cluster. The ResourceManager high availability (HA) feature adds redundancy in the […]

Continue Reading

Configure NameNode HA

Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a secondary namenode. The secondary namenode […]

Continue Reading

Enable/configure log and query redaction

Data redaction is the suppression of sensitive data, such as any personally identifiable information (PII) such as credit card number, email address, social security number. Cloudera has a data redaction feature, which will mask the credit card, email address with random or custom strings(we specify), so that in queries, log files those random strings will […]

Continue Reading

Perform OS-level configuration for Hadoop installation

Before installing CDH in our server, we’ve to make the below configuration changes in OS level for successful installation. Disable SELINUX  “Security-Enhanced Linux (SELinux) is a Linux kernel security module that provides a mechanism for supporting access control security policies” If SElinux is enabled, then cloudera server installation will fail in the server. To disable […]

Continue Reading

Define and install a rack topology script

In network terminology, all the physical servers in the network should’ve been present in a rack in the data center. In hadoop, the racks assignment is significant as it plays a vital role in terms of data locality, bandwidth etc. We can assign the rack of the hosts in the cluster in two ways. One […]

Continue Reading

Create an HDFS user’s home directory

Let’s assume that you’re asked to create an HDFS user home directory for a user named “goldberg“. First of all, verify that the user is exists in the server. If you’re creating the user directory for goldberg and set the ownership as goldberg:goldberg with restricted permissions, then none can access the dir except user ‘goldberg’. […]

Continue Reading