When you want to remove the node from the cluster, you shouldn’t just delete the cloudera agents, services installed as it will impact the whole cluster. You should go for decommission first. Decommissioning a host decommissions and stops all roles on the host without requiring you to individually decommission the roles on each service. After […]
Continue ReadingCategory: Hadoop
Rebalance the cluster
In HDFS, the blocks of the files are distributed among the datanodes as per the replication factor. Whenever you add a new datanode, the node will start receiving,storing the blocks of the new files. Though this sounds alright, the cluster is not balanced when you look at administrative point view. HDFS provides a balancer utility […]
Continue ReadingConfigure proxy for Hiveserver2/Impala
A proxy server is a server or application that acts as an intermediary for requests from clients seeking resources from other servers/applications. A client connects to the proxy server, requesting some service or resource available from a different server and the proxy server evaluates the request and sends it to the intended server/service/application. The server’s […]
Continue ReadingConfigure ResourceManager HA
The YARN ResourceManager is responsible for tracking the resources in a cluster and scheduling applications. Before CDH 5, the ResourceManager was a single point of failure in a YARN cluster. If Resource manager is down, then none of the jobs will run in the cluster. The ResourceManager high availability (HA) feature adds redundancy in the […]
Continue ReadingConfigure NameNode HA
Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a secondary namenode. The secondary namenode […]
Continue ReadingEnable/configure log and query redaction
Data redaction is the suppression of sensitive data, such as any personally identifiable information (PII) such as credit card number, email address, social security number. Cloudera has a data redaction feature, which will mask the credit card, email address with random or custom strings(we specify), so that in queries, log files those random strings will […]
Continue ReadingEfficiently copy data within a cluster/between clusters
DistCp (distributed copy) is a tool used for large inter/intra-cluster copying of HDFS data. It uses MapReduce to effect its distribution, error handling and recovery, and reporting. It expands a list of files and directories into input to map tasks, each of which will copy a partition of the files specified in the source list. […]
Continue ReadingPerform OS-level configuration for Hadoop installation
Before installing CDH in our server, we’ve to make the below configuration changes in OS level for successful installation. Disable SELINUX “Security-Enhanced Linux (SELinux) is a Linux kernel security module that provides a mechanism for supporting access control security policies” If SElinux is enabled, then cloudera server installation will fail in the server. To disable […]
Continue ReadingDefine and install a rack topology script
In network terminology, all the physical servers in the network should’ve been present in a rack in the data center. In hadoop, the racks assignment is significant as it plays a vital role in terms of data locality, bandwidth etc. We can assign the rack of the hosts in the cluster in two ways. One […]
Continue ReadingCreate an HDFS user’s home directory
Let’s assume that you’re asked to create an HDFS user home directory for a user named “goldberg“. First of all, verify that the user is exists in the server. If you’re creating the user directory for goldberg and set the ownership as goldberg:goldberg with restricted permissions, then none can access the dir except user ‘goldberg’. […]
Continue Reading