1、Zookeeper brief introduction

Zookeeper It's a distributed service framework , It is mainly used to solve some data management problems often encountered in distributed applications , Such as : Unified naming service 、 State synchronization service 、 Cluster management 、 Management of distributed application configuration items, etc .

2、zookeeper Basic concepts

2.1 role

Zookeeper There are three main types of roles in , As shown in the following table :

The system model is shown in the figure :

2.2 Design purpose

1. Final consistency :client No matter which one is connected to Server, Show it all in the same view , This is a zookeeper The most important performance .

2. reliability : Simple 、 robust 、 Good performance , If the news m Accepted by a server , Then it will be accepted by all servers .

3. The real time :Zookeeper Ensure that the client will get updates from the server within an interval , Or server failure information . But because of network delay and so on ,Zookeeper There is no guarantee that two clients can get the newly updated data at the same time , If you need the latest data , It should be called before reading data. sync() Interface .

4. It's nothing to wait for (wait-free): Slow or ineffective client Don't interfere with fast client Request , Make each client Can effectively wait .

5. Atomicity : Updates can only succeed or fail , There is no intermediate state .

6. Sequence : It includes global order and partial order : Globally ordered means if the message is on a server a In the message b Before release , At all Server On the news a All will be in the news b Before being released ; Partial order is if a message b In the message a Published by the same sender ,a It's bound to be in b front .

3. zookeeper working principle

Zookeeper The core of this is broadcasting , This mechanism ensures that Server Synchronization between . The protocol that implements this mechanism is called Zab agreement .Zab There are two modes of protocol , They are recovery modes ( Elector ) And broadcast Pattern ( Sync ). When the service starts or after the leader crashes ,Zab It's in recovery mode , When leaders are elected , And most of them Server Finished and leader After status synchronization of , Recovery mode is over . State synchronization ensures leader and Server Have the same system state . In order to ensure the order consistency of transactions ,zookeeper Incremental transactions adopted id Number (zxid) To identify transactions . All proposals (proposal) It was added when it was proposed zxid. In the implementation zxid It's a 64 Digit number , It's high 32 Is it epoch use To mark leader Does the relationship change , One at a time leader Be chosen , It will have a new one epoch, The identity currently belongs to that leader The reign of . low 32 Bits for increasing count .

Every Server There are three states in the working process :

LOOKING: At present Server I do not know! leader Who is it? , Searching for .

LEADING: At present Server It's an elected leader.

FOLLOWING:leader It has been elected , At present Server Keep up with it .

3.1 Select the main process

When leader Collapse or leader Lose most of follower, Now zk Enter recovery mode , The recovery model needs to re elect a new leader, Let all Server All return to the right state .Zk There are two kinds of election algorithms : One is based on basic paxos Realized , The other is based on fast paxos algorithmic . The default election algorithm of the system is fast paxos.

basic paxos technological process :

1 . Election thread by current Server The thread that initiated the election served as , Its main function is to count the voting results , And choose the recommended Server;

2 . Election thread first to all Server Initiate an inquiry ( Including myself );

3 . After the election thread receives the reply , Verify if the inquiry is self initiated ( verification zxid Is it consistent ), And then get each other's id(myid), And stored in the current query object list , Finally, get the proposal of the other party leader Related information (id,zxid), And store the information in the voting record of the current election ;

4. Received all Server After reply , Just figure it out zxid The biggest one Server, And will the Server The information is set to the next time you vote Server;

5. The thread will now zxid maximal Server Set to current Server To recommend Leader, If the winner at this time Server get n/2 + 1 Of Server Number of votes , Set the currently recommended leader For the winner Server, Will be based on the winning Server Information about setting your own state , otherwise , Continue the process , until leader To be elected .

Through process analysis, we can get : To make Leader Get the majority Server Support for , be Server The total must be an odd number 2n+1, And live Server No less than n+1. Every Server The above process will be repeated after startup . In recovery mode , If it's just recovered from a crash or just started server Data and session messages are also recovered from disk snapshots Rest ,zk Transaction logs are recorded and snapshots are taken regularly , It is convenient for state recovery during recovery .

The specific flow chart of the selection is as follows :

fast paxos The process is in the election process , some Server First of all, to all Server Propose to be leader, When others Server After receiving the proposal , solve epoch and zxid The conflict of , And accept the offer , Then send a message to the other party to accept the proposal , Repeat the process , In the end, we will be able to elect Leader. The flow chart is as follows :

3.2 Synchronization process

Finish leader in the future ,zk Enter the state synchronization process .

1. leader wait for server Connect ;

2. Follower Connect leader, Will be the biggest zxid Send to leader;

3. Leader according to follower Of zxid Determine the synchronization point ;

4. Notify... When synchronization is complete follower Has become a uptodate state ;

5. Follower received uptodate After the news , Can accept again client Of the requests for services .

The flow chart is shown below :

3.3 Workflow
3.3.1 Leader Workflow

Leader There are three main functions :

1. Restore data ;

2. Maintenance and Learner The heart of , receive Learner Ask and judge Learner The type of request message for ;

3. Learner The main message types are PING news 、REQUEST news 、ACK news 、REVALIDATE news , Depending on the message type , Different treatment .

PING News means Learner Heartbeat information of ;REQUEST The news is Follower Proposal information sent , Including write request and synchronization request ;ACK The news is Follower I'm right Reply to , More than half Follower adopt , be commit The proposal ;REVALIDATE Messages are used to extend SESSION Valid time .

Leader The work flow diagram of is as follows :

3.3.2 Follower Workflow

Follower There are four main functions :

1. towards Leader Send a request (PING news 、REQUEST news 、ACK news 、REVALIDATE news );

2. receive Leader Message and process ;

3. receive Client Request , If it's a write request , Send to Leader Vote ;

4. return Client result .

Follower The message loop processing of comes from Leader The news of :

1. PING news : Heartbeat message ;

2. PROPOSAL news :Leader The proposal launched , requirement Follower vote ;

3. COMMIT news : Information about the latest proposal on the server side ;

4. UPTODATE news : Indicates that synchronization is complete ;

5. REVALIDATE news : according to Leader Of REVALIDATE result , Close to revalidate Of session Or is it allowed to receive messages ;

6. SYNC news : return SYNC Result to client , This message was originally initiated by the client , Used to force the latest updates .

Follower The work flow diagram of is as follows :

4、zookeeper Installation configuration

ZooKeeper There are three installation modes of , Respectively : standalone mode (stand-alone)、 Cluster mode and cluster pseudo distribution mode .

4.1 standalone mode

download zookeeper After the installation package of , Unzip to the appropriate directory . Get into zookeeper In the catalog conf subdirectories , establish zoo.cfg:

tickTime=2000
dataDir=/Users/apple/zookeeper/data
dataLogDir=/Users/apple/zookeeper/logs
clientPort=4180

Parameter description :
tickTime: This time is for Zookeeper Time interval between servers or between clients and servers to maintain heartbeat , That is, every one of them tickTime Time will send a heartbeat .zookeeper The basic unit of time used in , Millisecond value .

dataDir: Data directory , It can be any directory .

dataLogDir:log Catalog , It can also be any directory . If the parameter is not set , Will use and dataDir Same settings .

clientPort: monitor client The port number of the connection

4.2 Pseudo cluster mode

The so-called pseudo cluster , It means to start multiple... In a single machine zookeeper process , And form a cluster . Start up 3 individual zookeeper Process, for example .

take zookeeper Directory copy of 2 Share :

zookeeper0/conf/zoo.cfg File for :

tickTime=2000
initLimit=5
syncLimit=2
dataDir=/Users/apple/zookeeper0/data
dataLogDir=/Users/apple/zookeeper0/logs
clientPort=4180
server.0=127.0.0.1:8880:7770
server.1=127.0.0.1:8881:7771
server.2=127.0.0.1:8882:7772

Several new parameters have been added , Its meaning is as follows :

1 initLimit: zookeeper The cluster contains multiple server, One of them is leader, The rest of the cluster server by follower.initLimit Parameter configuration initializes the connection ,follower and leader The longest heartbeat time between . This parameter is set to 5, Explain that the time limit is 5 times tickTime, namely 5*2000=10000ms=10s.

2 syncLimit: This parameter configures leader and follower Sending messages between , The maximum length of time for requests and responses . This parameter is set to 2, Explain that the time limit is 2 times tickTime, namely 4000ms.

3 server.X=A:B:C among X It's a number , What number is this server.A Is that the server Where IP Address .B Configure the server And in the cluster leader The port used to exchange messages .C Configuration election leader The port used when . Because the configuration is pseudo cluster mode , So each of them server Of B,C Parameters must be different .
reference zookeeper0/conf/zoo.cfg, To configure zookeeper1/conf/zoo.cfg, and zookeeper2/conf/zoo.cfg file . Just change dataDir, dataLogDir, clientPort Parameters can be .

Set before dataDir New China myid file , Write a number , The number indicates the number server. The number must be in line with zoo.cfg In the document server.X Medium X One-to-one correspondence .

/Users/apple/zookeeper0/data/myid Write in file 0,/Users/apple/zookeeper1/data/myid Write in file 1,/Users/apple/zookeeper2/data/myid Write in file 2.

Respectively into the /Users/apple/zookeeper0/bin,/Users/apple/zookeeper1/bin,/Users/apple/zookeeper2/bin Three directories , start-up server.

4.3 Cluster pattern

The configuration of cluster mode is basically the same as that of pseudo cluster .

Because in cluster mode , various server Deployed on different machines , So each of them server Of conf/zoo.cfg Files can be exactly the same .

Here's an example :

tickTime=2000
initLimit=5
syncLimit=2
dataDir=/home/zookeeper/data
dataLogDir=/home/zookeeper/logs
clientPort=4180
server.43=10.1.39.43:2888:3888
server.47=10.1.39.47:2888:3888
server.48=10.1.39.48:2888:3888

In the example 3 platform zookeeper server, They are deployed in 10.1.39.43,10.1.39.47,10.1.39.48 On . It should be noted that , various server Of dataDir In the catalog myid The numbers in the file must be different ,10.1.39.43 server Of myid by 43,10.1.39.47 server Of myid by 47,10.1.39.48 server Of myid by 48.

5、zookeeper Common commands

ZooKeeper The service command :

1. start-up ZK service : ./zkServer.sh start

2. see ZK Service status : ./zkServer.sh status

3. stop it ZK service : ./zkServer.sh stop

4. restart ZK service : ./zkServer.sh restart

zk Client commands :

ZooKeeper Command line tools are similar to Linux Of shell Environmental Science , Use it to ZooKeeper Visit , Data creation , Data modification and other operations . Use zkCli.sh -server 127.0.0.1:2181 Connect to ZooKeeper service , After successful connection , The system will output ZooKeeper Related environment and configuration information .

Some simple operations of the command line tool are as follows :

1. Show root 、 file :ls / Use ls Command to view the current ZooKeeper What's included in

2. Show root 、 file : ls2 / View the current node data and see the update times and other data

3. create a file , And set the initial content :create /zk“test” Create a new znode node “ zk ” And the string associated with it

4. Get file content : get /zk confirm znode Whether to include the string we created

5. Modify file content : set /zk "zkbak" Yes zk The associated string is set

6. Delete file : delete /zk Will just create the znode Delete

7. Exit client : quit

8. Help order : help

ZooKeeper Four word commands are often used :

ZooKeeper Support the interaction between certain four word command letters . Most of them are query commands , Used to obtain ZooKeeper The current status of the service and related information . Users on the client side can go through telnet or nc towards ZooKeeper Submit the corresponding order

1. By command :echo stat|nc 127.0.0.1 2181 To see which node is selected as follower perhaps leader

2. Use echo ruok|nc 127.0.0.1 2181, Test if the Server, If reply imok Indicates that the .

3. echo dump| nc 127.0.0.1 2181, Lists unprocessed sessions and temporary nodes .

4. echo kill | nc 127.0.0.1 2181, Turn off the server

5. echo conf | nc 127.0.0.1 2181, Output details of related service configuration .

6. echo cons | nc 127.0.0.1 2181, List all full connections of clients connected to the server / Details of the conversation .

7. echo envi |nc 127.0.0.1 2181, Output details about the service environment ( The difference in conf command ).

8. echo reqs | nc 127.0.0.1 2181, List unprocessed requests .

9. echo wchs | nc 127.0.0.1 2181, List servers watch Details of .

10. echo wchc | nc 127.0.0.1 2181, adopt session List servers watch Details of , Its output is a connection with watch List of related sessions .

11. echo wchp | nc 127.0.0.1 2181, List servers by path watch Details of . It outputs a and session The relevant path .

In this paper, from :http://www.cnblogs.com/kunpengit/p/4045334.html

Reference resources :http://www.ibm.com/developerworks/cn/opensource/os-cn-zookeeper/

zookeeper working principle 、 Installation configuration 、 More related articles about tool commands

  1. [ Reprint ] zookeeper working principle 、 Installation configuration 、 Introduction to tool commands

    Reprinted from http://www.cnblogs.com/kunpengit/p/4045334.html 1 Zookeeper brief introduction Zookeeper It's a distributed service framework , It is mainly used to solve the problems often encountered in distributed applications ...

  2. [ Re posting ]keepalived Working principle and configuration 、 Use

    keepalived Working principle and configuration . Use https://www.iteye.com/blog/aoyouzi-2288124 keepalived What is it? keepalived It is the guarantee of high availability in cluster management ...

  3. original :centos7.1 Next ZooKeeper Cluster installation configuration +Python Examples of actual combat

    centos7.1 Next ZooKeeper Cluster installation configuration +Python Examples of actual combat download :http://apache.fayea.com/zookeeper/zookeeper-3.4.9/zookeepe ...

  4. Zookeeper 3、Zookeeper working principle ( detailed )

    1.Zookeeper Role » The leader (leader), Responsible for the initiation and resolution of the vote , Update system status » Learners' (learner), Including followers (follower) And the observer (observer),follow ...

  5. Zookeeper 3、Zookeeper working principle ( turn )

    1.Zookeeper Role » The leader (leader), Responsible for the initiation and resolution of the vote , Update system status » Learners' (learner), Including followers (follower) And the observer (observer),follow ...

  6. hbase and ZooKeeper Cluster installation configuration

    One :ZooKeeper Cluster installation configuration 1: decompression zookeeper-3.3.2.tar.gz And rename to zookeeper. 2: Get into ~/zookeeper/conf Catalog : Copy zoo_sample.cfg ...

  7. Distributed coordination services ZooKeeper working principle

    Distributed coordination services ZooKeeper working principle original 2016-02-19 Du Yishu Performance and architecture Performance and architecture Performance and architecture WeChat ID yogoup Function is introduced Website performance improvement and architecture design Big data processing framework Hadoop.R ...

  8. java Review points ( One )------- java Characteristics of language 、java How it works 、 Configure environment variables 、java Use of commands

    One .java Characteristics of language : (1) Simple and object-oriented (2) Robust and safe : java Language in compiling and running programs , They have to be strictly checked , Prevent mismatches . If you reference an illegal type , Or perform an illegal type operation ,java To lose weight ...

  9. Oracle NET working principle 、 Configuration and connection troubleshooting

    One .Oracle NET The configuration file Oracle NET It's a software layer , Support the conversion between different network protocols . Different physical machines can communicate with each other through this software layer , To be specific, it is to achieve the goal of oracle Remote access to . oracl ...

Random recommendation

  1. jQuery Constructor analysis

    In my last article   Elaborated jQuery The general framework of , I know that all the code is written in a self calling anonymous function , And passed in window object , The source code is like this : (function( window, undefined ...

  2. IOS Several simple ways to achieve animation

    1. Use  NSTimer To achieve [NSTimer scheduledTimerWithTimeInterval:0.01 target:self selector:@selector(setNeed ...

  3. jQuery : eq() vs get()

    .get(index) and .eq(index) both return a single "element" from a jQuery object array, but ...

  4. UILabel Using skills

    UILabel The use of various properties and methods of ( turn ) #import "LabelTestViewController.h" @implementation LabelTestViewCon ...

  5. Customize View-- A simple circle Progress effect

    Let's look at the renderings first We want to achieve a custom and then draw an arc in a circle View, This is the idea : First, create a class ProgressView, Inherited from View class , And then rewrite two of them , One is a parameter , One ...

  6. [wordpress] Sort by custom field and query by custom field

    Wordpress in , Sorting and querying according to custom fields is done through WP_Query() Method As per A custom sort Sort the number fields from small to large $args = array( 'post_type' => ...

  7. C/C++ And lua Achieve intermodulation

    1, stay lua Call in script C/C++ Functions in code stay C++ When you define a function in, you must use lua_State Is the parameter , With int Only the return value can be Lua The call . /* typedef int (*lua_CFunction) ...

  8. java Communication between threads and notify And notifyAll difference .

    jvm Communication between multiple threads is through Thread lock . Conditional statements . as well as wait().notify()/notifyAll form . The following is to implement an output of two different statements that enable multiple threads to loop . package com.app. ...

  9. POI Generate word Complete document, case and explanation

    One , Online API Explain Actually POI Generation Word The rule of a document is to convert the acquired data into xml Formatted data , And then through xpath Analyze the application value of form type , Judgment, etc , Then put the value in word In the document , At the end of the day . ...

  10. [ original ]K8PackWebShell ASPX Whole station packing tools

    [ original ]K8PackWebShell ASPX Whole station packing tools [K.8]( Yes no Rar Execution permission is OK ) 2011-06-11 01:49:21|   classification : Original tools Name: K8PackWebShell ...