Configure the Fair Scheduler to resolve application delays

Fair scheduling is a method of assigning resources to applications such that all apps get, on average, an equal share of resources over time.

Using fair scheduler we can separate pools(queues) for each team and configure the resources for the pool which will help in overcoming application delays.

In the exam you may be asked to create a pool with min and max resources so that jobs submitted in the pool won’t be delayed.

Before we begin, ensure that the Fair Scheduler is chosen as the Yarn’s default scheduler.

Yarn – configuration – search ‘scheduler’



Now go to Yarn – Resource Pools section

Select configuration and you’ll be redirected to Dynamic Resource Pool Configuration.

Create Resource Pool

Here provide resource limits such as the weight for this pool, min/max cores and memory for this pool and other details.

Set scheduling policy, preemption etc., if specified.

Once the pool is configured, you can submit the jobs to this pool using the below parameter in the syntax.

# hadoop jar mapreduce.jar –D name

Problem Scenario:

· Create a resource pool with given min and max resources.

· Create a pool with given resources and set scheduling policy as DRF.

Thus we covered how to use fair scheduler to create resource pools to resolve application delays.

Use the comments section below to post your doubts, questions and feedback.

Please follow my blog to get notified of more certification related posts, exam tips, etc.



7 thoughts on “Configure the Fair Scheduler to resolve application delays

  1. Great work- the whole blog is extremely useful for aspiring cloudera admins!
    Would you know of any example scenario to test your configuration? For instance can you submit a couple of the example map reduce jobs and then see the result of your configuration in action? It would be good for the overall understanding if somewhere a scenario exists where you can submit and then follow the jobs that are running.

  2. Hi Kannan,
    didn’t you post the topic “Revise YARN resource assignment based on user feedback” ? I have found it.

  3. Hi Kannan,
    didn’t you post the topic “Revise YARN resource assignment based on user feedback” ? I have never found it.

Leave a Reply

Your email address will not be published. Required fields are marked *