Fair Scheduler The scheduler synchronizes the heartbeat to assign tasks. In short, it goes through the following steps :

1、 Yes map/reduce Whether the resource limit has been reached

2、 Yes pool Queue by Fair Algorithmic sorting

3、 Then the cycle pool queue , stay pool Medium job Queue by Fair Algorithmic sorting , loop job, choice task

4、 If you choose one task, Jump out of pool The cycle of , And then I'll do it again pool Sort , Repeat step 2

Fair Scheduler Scheduling strategy is very important for pool and job All the scheduling algorithms are consistent , First explain several important variables in the scheduling algorithm :

minshare : Minimum share slots assigned to the schedulable The minimum number of shares ,pool The minimum number of shares of a pool is the minimum number of shares that each resource pool needs to allocate map or reducer slots Count , That is, in the configuration file minMaps or minReduces,Job Of minShare Defined as 0

demand :  This is defined as number of currently running tasks + number of unlaunched tasks (tasks that are either not yet launched or need to be speculated). In a nutshell JOB Of slot Requirement , The result is through ( Running Tasks Number + Not running Tasks Number ) calculated ,pool Pooled demand It's all in the pool Job Of demand The sum of the .FairScheduler Will start a UpdateThread Thread to update regularly Demand value , The update interval can be determined by mapred.fairscheduler.update.interval To configure , Not configured or by default 2500 millisecond

runningTasks: Running Tasks Number ,pool Pooled runningTasks The value is equal to all of them job Of runningTasks The sum of the .

weight: The weight .Pool The weight is in the configuration file weight Configuration item ,job The weight of is related to priority , for example normal=1.0,high=2.0,very high=4.0 etc. , however job The weight of will be recalculated during operation , If it's on mapred.fairscheduler.sizebasedweight Configuration item , that weight Will recalculate :

weight = Math.log1p(demand) / Math.log(2);
weight *= getPriorityFactor(job.getPriority());

in other words job The weight will increase with job Of slot Change with the change of demand . If it's on mapred.fairscheduler.weightadjuster Configuration item , that weight The calculation will be recalculated ,Weightadjuster We need to cooperate with mapred.newjobweightbooster.factor and mapred.newjobweightbooster.duration Two configuration items to use ,factor Is the weighting factor ,duration For the weighting period , If meet :【 current time (currentTime)-jobStartTime < duration】 That is to say, it is still in the period of weight adjustment , that weight = weight*factor.

Fair Scheduler The core idea is to ensure that all assignments can get the same share of resources as possible , First, we will consider the resource deficit of the job to select the job , Then it's about priorities , The so-called resource deficit can be roughly understood as the ratio of the resources obtained and the resources needed at present , Let's talk about it in detail Fair Scheduler Core scheduling algorithm , In essence, it's right pool or job Carry out certain regular sorting operations , The process is as follows :

1、 To calculate job Of minShare,minShare be equal to minShare and demand The value is the smaller of the two

2、 Judge runningTasks Is less than minShare, If so , High priority

3、 step 2 If you're not satisfied , So judge runningTasks/max(mindshare,1.0) Value , The small value takes precedence

4、 If step 3 Medium runningTasks/max(mindshare,1.0) Have the same value , Judge runningTasks/weight Value , As mentioned earlier, if it is turned on sizebasedweight Configuration item , Then the job will change with the operation of the job weight Weight value , For large jobs, the change curve of weight value will be similar to a parabola .

5、 If the weight values are consistent , The last comparison startTime value .

Welcome to join Hadoop Technology group :147681830

Hadoop learning --Fair Scheduler More related articles on job scheduling analysis

  1. Hadoop learning --Capaycity Scheduler Source code analysis

    Capacity Scheduler Scheduling strategy when a new job Whether it is allowed to be added to the queue for initialization , Judge whether the current queue and user have reached the upper limit of initialization number , The following is a detailed introduction of the whole judgment logic from the code level .Capaycity ...

  2. Hadoop Learning notes —20. Website log analysis project case

    1.1 Project source The data log of this practice comes from a technology learning forum in China , The forum is sponsored by a training institution , It brings together a lot of technology learners , People post every day . Reply , Pictured 1 Shown . chart 1 Project source website - Technology Learning Forum The purpose of this practice is ...

  3. Hadoop Learning notes —20. Website log analysis project case ( One ) Project introduction

    Website log analysis project case ( One ) Project introduction : Current page Website log analysis project case ( Two ) Data cleaning :http://www.cnblogs.com/edisonchou/p/4458219.html Website log analysis project case ...

  4. Hadoop Learning notes —20. Website log analysis project case ( Two ) Data cleaning

    Website log analysis project case ( One ) Project introduction :http://www.cnblogs.com/edisonchou/p/4449082.html Website log analysis project case ( Two ) Data cleaning : Current page Website log analysis project case ...

  5. Hadoop Learning notes —20. Website log analysis project case ( 3、 ... and ) Statistical analysis

    Website log analysis project case ( One ) Project introduction :http://www.cnblogs.com/edisonchou/p/4449082.html Website log analysis project case ( Two ) Data cleaning :http://www.cnbl ...

  6. Hadoop learning --Capaycity Scheduler Configuration parameter description

    Here's a list of capacity About queue and user Description of parameters related to resource usage : mapred.capacity-scheduler.queue.xxx.capacity: The resource capacity percentage of the queue , the ...

  7. Linux 2.6 Completely fair scheduling algorithm CFS(Completely Fair Scheduler) analysis

    transfer http://www.ibm.com/developerworks/cn/linux/l-completely-fair-scheduler/index.html? ca=drs-cn-0125 ...

  8. Fair Scheduler Medium Delay Schedule analysis

    The main purpose of delay scheduling is to improve data locality (data locality), Reduce the transmission of data in the network . For those whose input data is not local MapTask, The scheduler will delay scheduling them , But the slot To those who are local MapTa ...

  9. Study Hadoop+Spark Big data analysis and machine learning integrated development -windows Virtual machine is used to simulate the construction of multi node cluster

    Record learning <Hadoop+Spark Big data analysis and machine learning integrated development > This book . The fifth chapter Hadoop Multi Node Cluster windows Virtual machine is used to simulate the construction of multi node cluster 5 ...

Random recommendation

  1. Be tasted —— stay C++ Call in C# Callback function for ——COM The way

    This is the easy way , Let me share with you , At the same time SafeArray The method of defining the structure inside 1. Requirements describe This is the demand ,C++ Code and C# Code communicates with each other (C++ Generally do service ,C# Do the client ),C++ Get the data ready on one side , however ...

  2. How to be in EPLAN P8 Create the electrical component symbol you want in

    1. open Eplan P8 , Create a new one called " The new project " Project , Then choose the menu " Tools "----" Master data "-----" Symbol library "-- ...

  3. php String in the common function ( Four ) ord() Get the character's ascii code chr() obtain ascii The character corresponding to the code

    ord('a');//=>97 Return to lowercase a Of ascii Code value 97 chr(97);//=>a return ascii On the code table 97 Corresponding A lowercase letter a

  4. Oracle in REGEXP_SUBSTR function ( turn )

    Oracle in REGEXP_SUBSTR function Oracle in REGEXP_SUBSTR Function instructions : The title is as follows : stay oracle in , Use a statement to implement '17,20,23' Split into '17','20','2 ...

  5. utilize for Circular search 1-100 Between odd and and 0-100 Even sum of

    For the convenience of your own calculation , The following code is only for 1-10 Odd sum of and 0-10 Even sum of 1-10 The odd number of from 1 The beginning is respectively The code is as follows /* Name: Loop statements give an odd number . Even numbers are added to sum Copyright ...

  6. DOM Method introduction - The second lesson

    1.console.log() Right? The elements are displayed on the console 2.appendChild() It's appending elements to the end 3.innerHTML What we get is all the contents of the element, including the sub elements 4.innerText Getting elements ...

  7. .net Read the rich text editor text as it is word file

    key word : Rich text editor   Generate word  style To solve the problem raised in the title , I need to know , stay .net Environment to read the content of the database, dynamic generation word There are at least 2 Ways of planting ,[ Mode one ] One way is to add references to the project , For example, in “ add ...

  8. Grunt Build tool plug-ins —— And less Tools 3 and watch With automatic compilation

    grunt less convert to css Slow speed And the page will all refresh ? I've had a problem recently ,grunt In the use less, When the modification is finished .less The pattern inside , Corresponding Of .css The file will take seconds to modify , And then it's going to take a couple of seconds to display on the browser ...

  9. HDU-2298 Toxophily ( Introduction to the trichotomy series )

    The question : The speed of the Italian gun is v, Seek for (0,0) hit (x,y) My goal , The angle of the projectile . Answer key : set up f(α) The angle is α when , f(α) = vsin(α) * t - 4.9 * t * t   ① ...

  10. SVN Server side privilege management

    Create user Click... On the menu Users You can see the created user in the window area on the right Create group You can also modify groups :   Assign permissions Example 1 : Developers have read and write access ( Group permission ) Enter the permission assignment interface : Add groups or users : On the add page, you can ...