Object pool technology is widely used in server development . In the implementation of various object pools , Especially the connection pool of database , It can be said that each server must implement the part .
 
apache common pool Official documents can be referred to : https://commons.apache.org/proper/commons-pool/.
 
combination JedisPool see Commons Pool Object pool technology
 
combination JedisPool, Let's get to know commons pool The overall design of :
 


 
 
What's often for users is ObjectPool, What users see is a pool of objects , For the use of Redis For connected users , Namely JedisPool. Object pool ObjectPool Provides borrowing objects , Return object , Verification objects, etc API, Need specific configuration GenericObjectPoolConfig To determine the size of the pool , And create factory interfaces for concrete pooled objects PooledObjectFactory To create as needed , The destruction , Activate , Passivate each object .
 
PooledObjectFactory Interface , Used to create pool objects (makeObject), Passivate unused pool objects (passivateObject), Activate the pool object to be used (activateObject), Validate pool objects (valiateObject), Destroy the pool object in question (destroyObject).
 
If needed commons-pool, Then you need to provide a PooledObjectFactory The concrete implementation of the interface , A simpler way is to use BasePooledObjectFactory This abstract class , Only two methods need to be implemented :create() and wrap(T obj).JedisFactory It's used to create every Jedis Connected object factory class , It directly realizes PooledObjectFactory,makeObject In the process of , Directly created PooledObject<Redis>.
 
When we use JedisPool.getResource(), For return jedis When the connection , The actual call is one of GenericObjectPool Of borrowObject Method , stay Jedis Borrow an object from the connection pool .
 
When borrowing objects , Go first idleObjects(LinkedBlockingDeque<Pooled<Jedis>>) List to see if there are free objects , If it exists, use ; If it doesn't exist , We need to consider that the maximum number of connection pools is not exceeded , Use PooledObjectFactory To initialize , What we use here is JedisFactory.makeObject To create a connection , And activate it .
 
 
about Jedis object , You can't always reuse the same object , It will fail after a period of use , Connection exception . You need to use JedisPool To get resources , Pay attention to recycling resources at the end , It's actually returnObject, Take the following code as an example :
 

Jedis jedis = jedisPool.getResource();
try {
while (true) {
String productCountString = jedis.get("product");
if (Integer.parseInt(productCountString) > 0) {
if (acquireLock(jedis, "abc")) {
int productCount = Integer.parseInt(jedis.get("product"));
System.out.println(String.format("%tT --- Get product: %s", new Date(), productCount));
// System.out.println(productCount);
jedis.decr("product");
releaseLock(jedis, "abc");
return "Success";
}
Thread.sleep(1000L);
} else {
return "Over";
}
}
} finally {
jedis.close();
}
 
 
 
JedisCluster The connection of / Perform source code research
 
 
What we use JedisCluster(Redis Cluster pattern ) Need to initialize and use JedisCluster object , Through this object Redis Related operations of , Here's a list of JedisCluster The basic class diagram structure of :
 


 
 
On a mission BinaryJedisCluster Related commands set/get/exist etc. redis On command , They all use callbacks :
 

@Override
public String set(final byte[] key, final byte[] value) {
return new JedisClusterCommand<String>(connectionHandler, maxRedirections) {
@Override
public String execute(Jedis connection) {
return connection.set(key, value);
}
}.runBinary(key);
}
 
 
Initialize a JedisClusterCommand object , perform runBinary Method , Conduct execute(Jedis connection) Callback , In fact, we can see that the function before callback is used Jedis Connections are managed internally .
 
It can be guessed that JedisSlotBasedConnectionHandler Implementation of the definition of the parent class getConnection() obtain Redis How to connect :
 
 
@Override
public Jedis getConnection() {
// In antirez's redis-rb-cluster implementation,
// getRandomConnection always return valid connection (able to
// ping-pong)
// or exception if all connections are invalid List<JedisPool> pools = getShuffledNodesPool(); for (JedisPool pool : pools) {
Jedis jedis = null;
try {
jedis = pool.getResource(); if (jedis == null) {
continue;
} String result = jedis.ping(); if (result.equalsIgnoreCase("pong")) return jedis; pool.returnBrokenResource(jedis);
} catch (JedisConnectionException ex) {
if (jedis != null) {
pool.returnBrokenResource(jedis);
}
}
} throw new JedisConnectionException("no reachable node in cluster");
}
 
 
 
The method of calling getShuffledNodesPool(), It's from JedisClusterInfoCache All of the JedisPool, perform shuffle operation , Randomly get the corresponding JedisPool, Get rid of it getResource Get the connection .
 
It's random to get connection, But in fact, it's not like this , We can go through slot To get the corresponding Connection, stay JedisClusterCommand.run In the last line of the method , The third parameter is whether it is tryRandomMode, The call mode is shown as not random Mode.
 
return runWithRetries(SafeEncoder.encode(keys[0]), this.redirections, false, false);
 
According to slot To locate the specific JedisPool,getResource Get the corresponding Jedis Connection, But this method also indicates that there is no guarantee that the available connection will be available .
 
@Override
public Jedis getConnectionFromSlot(int slot) {
JedisPool connectionPool = cache.getSlotPool(slot);
if (connectionPool != null) {
// It can't guaranteed to get valid connection because of node
// assignment
return connectionPool.getResource();
} else {
return getConnection();
}
}
 
 
stay JedisClusterInfoCache Cached Map<String,JedisPool>(host:port->JedisPool) and Map<Integer, JedisPool>(slot->JedisPool), Used to query connections , So how are these two caches found , And that's where it comes in Jedis.clusterNodes, It can be done through the Redis Connect to find the configuration of other connections , For example, you can discover the configuration of the entire cluster , Three of them master, Three slave, And be able to identify your own connections , Refer to the documentation : http://redis.io/commands/cluster-nodes
 
 
5974ed7dd81c112d9a2354a0a985995913b4702c 192.168.1.137:6389 master - 0 1468809898374 26 connected 0-5640
d08dc883ee4fcb90c4bb47992ee03e6474398324 192.168.1.137:6390 master - 0 1468809898875 25 connected 5641-11040
ffb4db4e1ced0f91ea66cd2335f7e4eadc29fd56 192.168.1.138:6390 slave 5974ed7dd81c112d9a2354a0a985995913b4702c 0 1468809899376 26 connected
c69b521a30336caf8bce078047cf9bb5f37363ee 192.168.1.137:6388 master - 0 1468809897873 28 connected 11041-16383
532e58842d001f8097fadc325bdb5541b788a360 192.168.1.138:6389 slave c69b521a30336caf8bce078047cf9bb5f37363ee 0 1468809899876 28 connected
aa52c7810e499d042e94e0aa4bc28c57a1da74e3 192.168.1.138:6388 myself,slave d08dc883ee4fcb90c4bb47992ee03e6474398324 0 0 19 connected
 
 
Distribute slot It can only be in master On node , Not in slave On node , It means Redis The cluster does not take the form of read-write separation . When Redis Clustered slot When there is a change , Will reinitialize the Cache, Reset slot.
 
And execute every get/set etc. Redis In operation , The real core entrance , It's actually JedisClusterCommand.runWithRetries Method :
 
 
private T runWithRetries(byte[] key, int redirections, boolean tryRandomNode, boolean asking) {
if (redirections <= 0) {
throw new JedisClusterMaxRedirectionsException("Too many Cluster redirections?");
} Jedis connection = null;
try { if (asking) {
// TODO: Pipeline asking with the original command to make it
// faster....
connection = askConnection.get();
connection.asking(); // if asking success, reset asking flag
asking = false;
} else {
if (tryRandomNode) {
connection = connectionHandler.getConnection();
} else {
connection = connectionHandler.getConnectionFromSlot(JedisClusterCRC16.getSlot(key));
}
} return execute(connection);
} catch (JedisConnectionException jce) {
if (tryRandomNode) {
// maybe all connection is down
throw jce;
} // release current connection before recursion
releaseConnection(connection);
connection = null; // retry with random connection
return runWithRetries(key, redirections - 1, true, asking);
} catch (JedisRedirectionException jre) {
// if MOVED redirection occurred,
if (jre instanceof JedisMovedDataException) {
// it rebuilds cluster's slot cache
// recommended by Redis cluster specification
this.connectionHandler.renewSlotCache(connection);
} // release current connection before recursion or renewing
releaseConnection(connection);
connection = null; if (jre instanceof JedisAskDataException) {
asking = true;
askConnection.set(this.connectionHandler.getConnectionFromNode(jre.getTargetNode()));
} else if (jre instanceof JedisMovedDataException) {
} else {
throw new JedisClusterException(jre);
} return runWithRetries(key, redirections - 1, false, asking);
} finally {
releaseConnection(connection);
}
}
 
 
The emergence of Redis Retries problem
 
You can refer to : http://carlosfu.iteye.com/blog/2251034, Very well . Again , Our exception stack of occurrences :
 
- 2016-06-04 00:02:51,911 [// - - ] ERROR xxx - Too many Cluster redirections?
redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections?
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:97)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:131)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:152)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:131)
 
There are too many redirections abnormal , There have been JedisConnectionException, Full stack content :
 
 
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:198)
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
at redis.clients.jedis.Protocol.process(Protocol.java:141)
at redis.clients.jedis.Protocol.read(Protocol.java:205)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:216)
at redis.clients.jedis.Connection.getBulkReply(Connection.java:205)
at redis.clients.jedis.Jedis.get(Jedis.java:101)
at redis.clients.jedis.JedisCluster$3.execute(JedisCluster.java:79)
at redis.clients.jedis.JedisCluster$3.execute(JedisCluster.java:76)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:119)
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:30)
at redis.clients.jedis.JedisCluster.get(JedisCluster.java:81)
at redis.RedisClusterTest.main(RedisClusterTest.java:30)
 
 
 
Exception information in debug state :
 
jce = {redis.clients.jedis.exceptions.JedisConnectionException@1014} "redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream."
detailMessage = "Unexpected end of stream."
cause = {redis.clients.jedis.exceptions.JedisConnectionException@1014} "redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream."
stackTrace = {java.lang.StackTraceElement[0]@1017}
suppressedExceptions = {java.util.Collections$UnmodifiableRandomAccessList@1018} size = 0
 
 
On this question , You can refer to :http://blog.csdn.net/jiangguilong2000/article/details/45025355
 
client buffer control . On the client side and server In the interaction , Each connection is associated with a buffer relation , this buffer Used to queue waiting to be client Received response information . If client Can't timely consumer response information , that buffer It will be overstocked and given to server Bring memory pressure . If buffer The data in the backlog reaches the threshold value , Will cause the connection to be closed ,buffer Removed .
 
  Execute the command to query the parameter on the development environment :config get client-output-buffer-limit
 

1) "client-output-buffer-limit"
2) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
 
About Redis All the parameters on the , You can refer to :http://shift-alt-ctrl.iteye.com/blog/1882850
 
JedisMovedDataException
 
jre = {redis.clients.jedis.exceptions.JedisMovedDataException@2008} "redis.clients.jedis.exceptions.JedisMovedDataException: MOVED 8855 192.168.1.137:6390"
targetNode = {redis.clients.jedis.HostAndPort@2015} "192.168.1.137:6390"
slot = 8855
detailMessage = "MOVED 8855 192.168.1.137:6390"
cause = {redis.clients.jedis.exceptions.JedisMovedDataException@2008} "redis.clients.jedis.exceptions.JedisMovedDataException: MOVED 8855 192.168.1.137:6390"
stackTrace = {java.lang.StackTraceElement[0]@1978}
suppressedExceptions = {java.util.Collections$UnmodifiableRandomAccessList@1979} size = 0
 
 
Timeout exception in log :
 
4851:S 18 Jul 11:05:38.005 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
You can refer to github on redis The discussion of the : https://github.com/antirez/redis/issues/641, close AOF, It can solve the problem temporarily .JedisCluster The application of Apache Commons Pool Object pool technology  
 

JedisCluster The application of Apache Commons Pool More articles on object pooling Technology

  1. Object pooling Technology org.apache.commons.pool

    Using object pooling properly , It can effectively reduce the consumption of object generation and initialization , Improve the efficiency of the system .Jakarta Commons Pool Component provides a complete framework for object pooling , And a number of object pool implementations with their own characteristics , Sure ...

  2. Apache Commons Pool A story

    Apache Commons Pool A story Recently I came across a problem in my work due to commons-pool Problems caused by improper use of , After learning the correct posture , Write this simple story , Help you understand Apache Commons ...

  3. Pooling - Apache Commons Pool

    For those that take a long time to create , Or objects with more resources , For example, network connection , Resources like threads , Pooling is often used to manage these objects , In order to achieve the purpose of improving performance . For example, database connection pool (c3p0, dbcp), java Thread pool of Execu ...

  4. apache commons pool

    apache commons Under the pool Among them borrowObject Function source code shows the process of generating available objects : If stack There are free objects in , be pop object , Activate the object (activate function ), Verify the object (v ...

  5. Apache Commons Pool A story project

    Apache Commons Pool A story Recently I came across a problem in my work due to commons-pool Problems caused by improper use of , After learning the correct posture , Write this simple story , Help you understand Apache Commons ...

  6. Tomcat Development web Project Report Illegal access: this web application instance has been stopped already. Could not load [org.apache.commons.pool.impl.CursorableLinkedList$Cursor]. error

    Development Java web project , stay tomcat The following error will be reported after operation : Illegal access: this web application instance has been stopped already ...

  7. NoClassDefFoundError: org/apache/commons/pool/impl/GenericObjectPool

    error :Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/pool/impl ...

  8. Spring + Tomcat Initiate error reporting java.lang.ClassNotFoundException: org.apache.commons.pool.impl.GenericObjectPool

    Error is as follows : -- ::,-[TS] INFO http-- org.springframework.beans.factory.support.DefaultListableBeanFactory - ...

  9. org/apache/commons/pool/impl/GenericObjectPool Solutions to exceptions

    org/apache/commons/pool/impl/GenericObjectPool Solutions to exceptions webwork+spring+hibernate Integration of framework , One start Tomcat The server is out ...

Random recommendation

  1. With Amazon S3 and RapidMiner Applying machine learning to text mining

    This mining typically uses machine learning technology , Such as the clustering , classification , Association rules , And predictive modeling . These technologies reveal meaning and relationships in potential content . Text mining is applied to such fields as competitive intelligence , Life science , Customer voice , Media and Publishing , Laws and taxes , Law enforcement , Emotional analysis and trends ...

  2. 【POJ】3133 Manhattan Wiring

    http://poj.org/problem?id=3133 The question :n×m The grid of , Yes 2 individual 2,2 individual 3, They don't overlap . There are obstacles 1. Now please 2 To 2 The path and 3 To 3 The shortest length of disjoint paths of -2.(2<=n, ...

  3. alter database open resetlogs

    Use resetlogs Options , The current log serial number will be (log sequence number) Reset to 1, And discard all log information . You need to use resetlogs Options : In incomplete recovery ( Media recovery ): Using backup ...

  4. Unity 3D Call camera to capture photos videotape

    1, To call a camera, first turn on the camera driver , If the user allows it, you can use . 2, Definition WebCamTexture The variable used to capture a single photo . 3, Continuous capture must be thread enabled . Implementation code : using UnityEngine; ...

  5. python3, Interprocess communication

    This article comes from python 3.5 Version of the official document multiprocessing Module provides two methods for interprocess communication : 1. Process queue queue The Queue class is a near clone o ...

  6. IT Wheel series ( Two )——mvc API Automatic generation of instruction documents ——Swagger Use ( One )

    This article mainly introduces how to use Swashbuckle Plug in VS 2013 Automatically generate MVC API Documentation of the project . To better illustrate swagger Generate , Let's start with a new empty one API Project start . First step . newly build mvc api ...

  7. Navicat Premium 12.1.12.0 Installation and activation

    Reprinted from :Navicat Premium 12.1.12.0 Installation and activation author : My memory will never die link :https://www.jianshu.com/p/5f693b4c9468 navicat12112 ...

  8. Elasticsearch&#160;Search&#160;APIs

    Elasticsearch Search APIs By: Give the guest QQ:1033553122 1. Search for 1 Search for... In all types of a single index 1 Searches for... In the specified type of a single index 1 Search multiple specified indexes for 1 ...

  9. BizTalk Server How to deal with big news

    What's big news ? Unfortunately , The answer to this question is not directly related to a specific message size , binding , It's up to you  Microsoft  The specific bottleneck of  BizTalk Server  System .  The problems associated with big messages can be divided into the following categories : Out of memory error ...

  10. e1084. Catch errors and exceptions

    All errors and exceptions extend from Throwable. By catching Throwable, it is possible to handle all ...