How to deal with serious message accumulation caused by message queue?
Performance and architecture 2021-06-04 09:48:48


1. Why is there a message heap ?

Mostly because of Consumer Something is wrong. , Not discovered in time , Or it takes a long time to recover , This leads to a large backlog of messages in MQ in .

2. What's the consequence of message piling up ?

2.1 The message is discarded

for example RabbitMQ There is a message expiration time TTL, Overdue messages are thrown away , So there's no news at all .

2.2 The disk is full

If the accumulation is too large , May result in insufficient disk space , Then new news won't come in .

2.3 Huge amount of messages to be processed

If the message doesn't expire , And there's enough disk space , That is to generate a large number of messages waiting to be consumed ,Consumer The nightmare of .

3. How to deal with it ?

3.1 A situation in which a message is discarded

First , To prevent message expiration , Expiration time should not be set .

If the expiration time is set , Caused the loss of messages , How to remedy it ?

It can only be in the dead of night , While the traffic is at its lowest , Write a temporary program to fill in the message .

For example, an order message is lost , Then we need to find out which order messages are missing , And then send it back to the queue .

3.2 Insufficient disk

Systems are usually monitored , When the space threshold is reached, an alert is issued , At this time, we have to deal with it immediately .

for example , Create temporary message queues on other machines , Write another temporary Consumer, As a transfer of information , Take out the messages in the message backlog queue , Put it in a temporary queue .

Quickly evacuate the backlog of information , Get disk space back to normal .

3.3 Fast processing of massive backlog of messages

Consumer After returning to normal , In the face of a mountain of news , How to deal with it ?

How to deal with it according to the normal situation before , It takes a long time to spend , In the process, new news keeps coming in .

for example , There's a backlog 100 Ten thousand , Yes 3 individual Consumer, Every second can handle 200 strip ,3 individual Consumer It can handle a total of 600 strip .

It will take more than an hour to finish .

How many new news will be piled up in more than an hour ?

So it's not going to work properly , Need to speed up .

for example Kafka, This message is overstocked Topic Yes 3 individual Partition, That can be used at most 3 individual Consumer, So add Consumer of no avail .

Still usable Temporary queues The way .

Create a new one Topic, Set to 20 individual Partition

Consumer No more business logic , Handling only , Put the message on hold Topic in

this 20 individual Partition There can be 20 individual Consumer 了 , They deal with the original business logic .

this 20 individual Consumer It can handle a total of 4000 The article , In this way, we can finish the backlog in a few minutes 100 Ten thousand .

There won't be too much new news in these few minutes , So it will be back to normal soon , after , Then restore the whole structure to its original form .

To sum up , The message backlog is more troublesome , It's better to guard against it in advance , Do a good job in health monitoring of hardware and message system . If a message is lost , We need to find the missing message manually , Then fill in . At a time when consumption doesn't come , Consider using a temporary queue as a transit , Improve processing power .

Recommended reading

OAuth2 The illustration

Easy to understand Kubernetes Core concept of

Architecture technology trends that developers have to understand :Service Mesh

Please bring the original link to reprint ,thank
Similar articles

2021-08-09

2021-08-09