1. Why is there a message heap ？
Mostly because of Consumer Something is wrong. , Not discovered in time , Or it takes a long time to recover , This leads to a large backlog of messages in MQ in .
2. What's the consequence of message piling up ？
2.1 The message is discarded
for example RabbitMQ There is a message expiration time TTL, Overdue messages are thrown away , So there's no news at all .
2.2 The disk is full
If the accumulation is too large , May result in insufficient disk space , Then new news won't come in .
2.3 Huge amount of messages to be processed
If the message doesn't expire , And there's enough disk space , That is to generate a large number of messages waiting to be consumed ,Consumer The nightmare of .
3. How to deal with it ？
3.1 A situation in which a message is discarded
First , To prevent message expiration , Expiration time should not be set .
If the expiration time is set , Caused the loss of messages , How to remedy it ？
It can only be in the dead of night , While the traffic is at its lowest , Write a temporary program to fill in the message .
For example, an order message is lost , Then we need to find out which order messages are missing , And then send it back to the queue .
3.2 Insufficient disk
Systems are usually monitored , When the space threshold is reached, an alert is issued , At this time, we have to deal with it immediately .
for example , Create temporary message queues on other machines , Write another temporary Consumer, As a transfer of information , Take out the messages in the message backlog queue , Put it in a temporary queue .
Quickly evacuate the backlog of information , Get disk space back to normal .
3.3 Fast processing of massive backlog of messages
Consumer After returning to normal , In the face of a mountain of news , How to deal with it ？
How to deal with it according to the normal situation before , It takes a long time to spend , In the process, new news keeps coming in .
for example , There's a backlog 100 Ten thousand , Yes 3 individual Consumer, Every second can handle 200 strip ,3 individual Consumer It can handle a total of 600 strip .
It will take more than an hour to finish .
How many new news will be piled up in more than an hour ？
So it's not going to work properly , Need to speed up .
for example Kafka, This message is overstocked Topic Yes 3 individual Partition, That can be used at most 3 individual Consumer, So add Consumer of no avail .
Still usable Temporary queues The way .
Create a new one Topic, Set to 20 individual Partition
Consumer No more business logic , Handling only , Put the message on hold Topic in
this 20 individual Partition There can be 20 individual Consumer 了 , They deal with the original business logic .
this 20 individual Consumer It can handle a total of 4000 The article , In this way, we can finish the backlog in a few minutes 100 Ten thousand .
There won't be too much new news in these few minutes , So it will be back to normal soon , after , Then restore the whole structure to its original form .
To sum up , The message backlog is more troublesome , It's better to guard against it in advance , Do a good job in health monitoring of hardware and message system . If a message is lost , We need to find the missing message manually , Then fill in . At a time when consumption doesn't come , Consider using a temporary queue as a transit , Improve processing power .