One 、 Background
In the early days of system development , It's easy to happen ： Different business line developers , Because of the impact of technology stack and version time , When selecting models, you will give priority to those you are familiar with , for example MQ Middleware is commonly used ：Kafka、Rocket、Rabbit etc. , This makes it easy to ignore component differences between projects ;
In the middle and late stage of system development , After the business is relatively stable , Generally, modules with high resource consumption will be reconstructed step by step , Integrated management of public services , So as to make the system more integrated , In the process , Solving the middleware differences of different projects usually bears the brunt , For example, common cache centers ,MQ Message management, etc ;
This situation is generally difficult to avoid , At the beginning of the system, in order to quickly support the business , Bury a lot of pits , Once the business can develop stably , And sustainability is verified , You will start to give due consideration to module refactoring step by step , cost reduction .
Two 、 Reconstruction ideas
2.1 Initial problems
In the early stage of research and development of a start-up company , There are five projects developed in parallel on the business line , At that time, for MQ The usage of is as follows ：
Rocket： The core business 3 A project , The version is different ;
Kafka： The data weight is too high ,1 Projects adopt ;
Redis： be based on Python Connect , Queue message mode ;
At first, because I didn't use much , The whole is still under control , As the business continues to iterate , When communication is required between projects , It began to be chaotic and difficult to maintain , Then you're forced to start refactoring , Unified messaging component .
2.2 Secondary selection
Based on the comprehensive consideration of business , Several existing projects are reviewed MQ The redesign , The overall framework is as follows ：
MQ Component selection ： use RocketMQ;
replace Redis Queue mode of component ;
Will be based on Python System modification Java Language ;
Provide two services: message production and consumption ;
MQ The functions of are uniformly maintained by the above services ;
There is no change in component selection on the core business line , replace kafka One reason is that it involves a large number of settlement businesses ,Redis Queue mode deprecated , be based on Python Our management system has few functions , It's just easy to change , Unified line of business programming language . After this design , From the perspective of overall thinking, it will be much more reasonable .
3、 ... and 、 Transformation process
3.1 The whole idea
Description of core roles involved , From left to right ：
Production client ： The node that needs to request the server to communicate , Call the message sending interface encapsulated by the production server ;
Production server side ： Encapsulate message sending API, And maintain routing management , Permission identification, etc , Message landing storage, etc ;
Message storage layer ： It is mainly stored based on message oriented middleware , The database level is used to deal with the secondary scheduling in specific cases ;
Consumer services ： Encapsulate message reception API, And according to the route identification , Request the specified consumer interface , Complete communication ;
Consumer client ： Respond to the request of the consumer server , Encapsulate the specific business logic during consumption ;
There is not much difference in the overall technical difficulty , But introducing two services 【 Production and consumption 】, Used to manage MQ Communication process , Adapt to specific business logic , Introduce the database for one-time floor storage , It is more flexible in handling exception flow , In this way, the whole message module has strong scalability .
There are many choices of message oriented middleware , But given the familiarity of business line developers , And refer to the test comparison report provided by many parties , Finally determine the selection RocketMQ Components , meanwhile RocketMQ Related features ： High performance 、 high reliability , And support for distributed transactions , It is also a core consideration .
Based on the current microservice architecture pattern , hold MQ The functionality itself is integrated into two core services , Unified management and iteration , And component version control , For all production messages , Perform global routing control , And under certain circumstances , Through application service level function design , Realize message delay consumption , And the secondary scheduling execution of the failure message , And manual triggering of some single messages .
Secondary storage of message entities , It mainly adapts to some specific function points , Some messages can be delayed , For example, when MQ When the queue piles up , Or when the monitoring warning line is reached , It can be configured , Intervene that some messages are only stored in the warehouse , Don't push MQ, Wait until the service is relatively idle .
Message oriented middleware serves as a stable support for decoupling between systems , When managing at the service level , Need to have a clear design route , And the monitoring and recording of key process nodes , Ensure the stability and fault tolerance of the whole link .
Same series ： Distributed concept | Distributed transactions | Kafka colony | RocketMQ Components | Redis colony
Four 、 Source code address
A smile from cicada
Accumulation is a lonely and boring process