Transformation of message oriented middleware under distributed services

The cicada smiled 2021-10-14 06:47:03

One 、 Background

In the early days of system development , It's easy to happen : Different business line developers , Because of the impact of technology stack and version time , When selecting models, you will give priority to those you are familiar with , for example MQ Middleware is commonly used :Kafka、Rocket、Rabbit etc. , This makes it easy to ignore component differences between projects ;

In the middle and late stage of system development , After the business is relatively stable , Generally, modules with high resource consumption will be reconstructed step by step , Integrated management of public services , So as to make the system more integrated , In the process , Solving the middleware differences of different projects usually bears the brunt , For example, common cache centers ,MQ Message management, etc ;

This situation is generally difficult to avoid , At the beginning of the system, in order to quickly support the business , Bury a lot of pits , Once the business can develop stably , And sustainability is verified , You will start to give due consideration to module refactoring step by step , cost reduction .

Two 、 Reconstruction ideas

2.1 Initial problems

In the early stage of research and development of a start-up company , There are five projects developed in parallel on the business line , At that time, for MQ The usage of is as follows :


  • Rocket: The core business 3 A project , The version is different ;
  • Kafka: The data weight is too high ,1 Projects adopt ;
  • Redis: be based on Python Connect , Queue message mode ;

At first, because I didn't use much , The whole is still under control , As the business continues to iterate , When communication is required between projects , It began to be chaotic and difficult to maintain , Then you're forced to start refactoring , Unified messaging component .

2.2 Secondary selection

Based on the comprehensive consideration of business , Several existing projects are reviewed MQ The redesign , The overall framework is as follows :


  • MQ Component selection : use RocketMQ;
  • replace Redis Queue mode of component ;
  • Will be based on Python System modification Java Language ;
  • Provide two services: message production and consumption ;
  • MQ The functions of are uniformly maintained by the above services ;

There is no change in component selection on the core business line , replace kafka One reason is that it involves a large number of settlement businesses ,Redis Queue mode deprecated , be based on Python Our management system has few functions , It's just easy to change , Unified line of business programming language . After this design , From the perspective of overall thinking, it will be much more reasonable .

3、 ... and 、 Transformation process

3.1 The whole idea


Description of core roles involved , From left to right :

  • Production client : The node that needs to request the server to communicate , Call the message sending interface encapsulated by the production server ;
  • Production server side : Encapsulate message sending API, And maintain routing management , Permission identification, etc , Message landing storage, etc ;
  • Message storage layer : It is mainly stored based on message oriented middleware , The database level is used to deal with the secondary scheduling in specific cases ;
  • Consumer services : Encapsulate message reception API, And according to the route identification , Request the specified consumer interface , Complete communication ;
  • Consumer client : Respond to the request of the consumer server , Encapsulate the specific business logic during consumption ;

There is not much difference in the overall technical difficulty , But introducing two services 【 Production and consumption 】, Used to manage MQ Communication process , Adapt to specific business logic , Introduce the database for one-time floor storage , It is more flexible in handling exception flow , In this way, the whole message module has strong scalability .

3.2 Details

  • Component selection

There are many choices of message oriented middleware , But given the familiarity of business line developers , And refer to the test comparison report provided by many parties , Finally determine the selection RocketMQ Components , meanwhile RocketMQ Related features : High performance 、 high reliability , And support for distributed transactions , It is also a core consideration .

  • Microservice architecture

Based on the current microservice architecture pattern , hold MQ The functionality itself is integrated into two core services , Unified management and iteration , And component version control , For all production messages , Perform global routing control , And under certain circumstances , Through application service level function design , Realize message delay consumption , And the secondary scheduling execution of the failure message , And manual triggering of some single messages .

  • data storage

Secondary storage of message entities , It mainly adapts to some specific function points , Some messages can be delayed , For example, when MQ When the queue piles up , Or when the monitoring warning line is reached , It can be configured , Intervene that some messages are only stored in the warehouse , Don't push MQ, Wait until the service is relatively idle .

Message oriented middleware serves as a stable support for decoupling between systems , When managing at the service level , Need to have a clear design route , And the monitoring and recording of key process nodes , Ensure the stability and fault tolerance of the whole link .

Same series Distributed concept  |  Distributed transactions  | Kafka colony  | RocketMQ Components  | Redis colony

Four 、 Source code address

GitEE· Address
Wiki· Address
 A smile from cicada
A smile from cicada
Accumulation is a lonely and boring process
230 Original content
official account
Please bring the original link to reprint ,thank
Similar articles