Execute file system commands via HTTPFS

HttpFS is a service that provides HTTP access to HDFS. i.e we can access the HDFS from other filesystems from browsers, and using programming languages.

HttpFS has a REST HTTP API supporting all HDFS filesystem operations (both read and write).

Using HttpFS, we can Read and write data in HDFS using HTTP utilities (such as curl or wget) and HTTP libraries from languages other than Java.

Cloudera offers HttpFS role as part of the HDFS service and you can assign the HttpFS role to hosts during initial setup or at any time .

If not assigned, go to HDFS – instances – Add role instances

Select the host on which you want to assign HttpFS and finish (Client Deploy).

Once added, you can see it in the HDFS – instances page. (Assigned for standby host)

Now to execute filesystem commands, login to any nodes in the cluster.

I logged into the master host and listing a file in the hdfs, /hadoop/conf directory.

To execute the same command via HttpFS

# curl ‘http://httpfshost:port/webhdfs/v1/hdfspath?op=OPERATION&user.name=user’

Here our httpfs host is standby and the default port is 14000, operation for ls is LISTSTATUS and username is root.

For more info on httpfs and webhdfs api commands, check

https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-hdfs-httpfs/index.html

https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

 

Problem Scenarios:

· Assign a httpfs role to the given host and create a file in hdfs.

· Using httpfs, create a directory in hdfs, etc.,

Thus we covered how to Execute file system commands via HTTPFS

Use the comments section below to post your doubts, questions and feedback.

Please follow my blog to get notified of more certification related posts, exam tips, etc.

 


 

  1. How to create a file in hdfs using httpfs, what is the command
    and how much internet speed is required during the exam
    it it ok if i have around 1mbps speed
    And what is the process for documents means i have to upload scan proof and they will take my photo

Leave a Reply

%d bloggers like this: