1. Balance disk utilization

hadoop balancer -Threshold 20

perhaps

sh $HADOOP_HOME/bin/start-balancer.sh –t 20%

Parameters 20 It's a scale parameter , Express 20%, That's to say, even each other DataNode The deviation of direct disk usage is 20% within .

threshold default setting :10, Parameter value range :0-100, Parameter meaning : The target parameter to determine whether the cluster is balanced , every last datanode
The difference between the storage utilization and the total storage utilization of the cluster should be less than this threshold , Theoretically , The smaller the parameter is set , The more balanced the whole cluster is , But in an online environment ,hadoop The cluster is in progress balance when , Also write and delete data concurrently , So it may not reach the set balance parameter value .

2. kill hadoop Running job

$hadoop job -list

$hadoop job -kill job_201212111628_11166

hadoop More articles on performance tuning

  1. hadoop Performance tuning and operation and maintenance

    hadoop Performance tuning and operation and maintenance . Hardware options . Operating system tuning and jvm tuning . hadoop Operation and maintenance Hardware options 1) hadoop Running environment 2)   Principle one : The reliability of master node is better than that of slave node Principle two : Multichannel, multicore , high frequency ...

  2. [ Daniel translation series ]Hadoop(16)MapReduce performance tuning : Optimize data serialization

    6.4.6  Optimize data serialization How to store and transfer data has a great impact on performance . In this section, we will introduce the best practices of data serialization , from Hadoop Squeeze out the maximum performance in the process . Compression is Hadoop An important part of optimization . Compression can reduce the number of job outputs ...

  3. [ Daniel translation series ]Hadoop(8)MapReduce performance tuning : Performance measurement (Measuring)

    6.1  measurement MapReduce And environmental performance indicators The basis of performance tuning is the performance index and experimental data of the system . Based on these indicators and data , To find the performance bottleneck of the system . Performance indicators and experimental data can only be obtained through a series of tools and processes . In this part , General introduction ...

  4. Hadoop Examples of job performance indicators and parameter tuning ( Two )Hadoop Job performance tuning 7 A suggestion

    author :Shu, Alison Hadoop Two scenarios of job performance tuning : One . Users observe poor job performance , Ask for help . ( One )eBayEagle Job performance analyzer 1. Hadoop Abnormal index of operation performance 2. Hado ...

  5. hbase Compression test for performance tuning

    An overview of the article : 1. Sequential writing 2. Sequential reading 3. Write at random 4. random block read 5.SCAN data 0 Performance testing tools hbase org.apache.hadoop.hbase.PerformanceEvaluation ...

  6. [Spark performance tuning ] Chapter two : Decipher it completely Spark Of HashShuffle

    Topic of this lesson Shuffle It's the natural enemy of distributed systems Spark HashShuffle Introduce Spark Consolidated HashShuffle Introduce Shuffle How to be Spark Performance killer ...

  7. [Spark performance tuning ] The third chapter : Spark 2.1.0 in Sort-Based Shuffle Inside the story

    Topic of this lesson Sorted-Based Shuffle  The birth and introduction of Shuffle Six puzzling questions in Chinese Sorted-Based Shuffle Sorting and source appreciation Shuffle Memory management at run time ...

  8. [Spark performance tuning ] Chapter four : Spark Shuffle in JVM Memory usage and configuration details

    The topic of this lesson is JVM Analysis of memory usage architecture Spark 1.6.x and Spark 2.x Of JVM analyse Spark 1.6.x before on Yarn Computing memory usage cases Spark Unified Mem ...

  9. Spark Resource allocation for performance tuning

    Spark Resource allocation for performance tuning     The king of performance optimization is to give more resources ! There are more machines ,CPU More , More memory , Performance and speed improvements , It's obvious . Basically , Within a certain range , Increase resources and improve performance , It's proportional : Finished writing ...

Random recommendation

  1. IIS Set the default home page static page , No static pages , Take the route

    stay Global.asax Add... To the file protected void Application_BeginRequest(Object sender, EventArgs e)         {      ...

  2. Oracle Parameters

    1. ARCHIVE_LAG_TARGET forces a log switch after the specified amount of time elapses. Valid values are 0(disabled) ...

  3. Elasticsearch-2.3.x The way to fill a hole

    Use version notes :2.3.2 Force cannot be used root User start ? Because in 2.x The version emphasizes security , prevent attracker invasion root user , So it is recommended that users create other users to start . Of course , It can be done by configuration root User start . ...

  4. yii Master slave database separation - Reprint http://www.yiichina.com/doc/guide/2.0/db-dao

    Database replication and read-write separation Many databases support database replication database replication To improve availability and responsiveness . In database replication , Data always comes from the master server To From the server . All write operations such as insert and update are performed on the master server ...

  5. C# Custom cursor WaitCursor

    A kind of : Put the image file in the project folder 1 If the image file is .cur Format : Cursor cur=new Cursor( file name ); this.cursor=cur; Two words It's over 2 If the image file is other ...

  6. JS+CSS Create a three fold menu , Auto shrink other stages js

    <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="C ...

  7. [ note ] machine learning (Machine Learning) - 00. Catalog / The outline / Written in the book before

    The catalogue will be updated according to my learning progress , Give yourself an outline to look at the whole learning process systematically . Source of learning materials What we learn is Coursera Wu Enda (Andrew Ng) Teacher's machine learning video ( Course portal , Recently " Most brain ...

  8. Lightscape

    Lightscape It is an advanced lighting simulation and visual design system , It is used for accurate lighting simulation and flexible and convenient visual design of 3D model . Lightscape It's a lighting rendering software , Its unique calculation method of light energy transfer and the unique effect of material properties ...

  9. JS in [object object] How to take the value

    error message : It was meant to show JSON Object's   As a result, the console printed [object object] There needs to be a simple transformation , as follows : var jsonData = JSON.stringify(data);// Turn into ...

  10. 【oracle introduction 】 Database system paradigm

    To standardize the relational data model , The design of relational database system must abide by certain rules , This rule becomes the relational database paradigm . 1. First normal form 1NF If the value in the field is no longer divisible , It is in line with the first paradigm , namely 1NF. 2. Second normal form 2NF ...