How do you solve the transaction problem after the database is divided into database and table?
Gu Liji 2021-07-20 04:50:38

One 、 summary

With the development of time and business , The amount of data in the database table will be larger and larger , Accordingly , Data manipulation , The cost of adding, deleting, modifying and checking will be more and more . therefore , Split some of the large tables into multiple tables in multiple databases .

This article is based on the asynchronous assurance of non transaction message to complete the transaction problem in sub database and sub table .

Two 、 Need to solve the problem

2.1 Original business

Because after sub database and sub table , The new table is in another database , How to ensure the transaction of the main database and sub database is a problem that must be solved .

terms of settlement : By creating a flow table in the main library , Map the logic of operating the database to a pipeline record . When the whole big business is done ( The pipeline is inserted into the pipeline table ), And then execute this flow in other ways , Ensure ultimate consistency .

2.2 Running water

The so-called running water , It can be understood as a transaction message

Above by creating a flow table in the database , Use a pipeline record to represent a business processing logic , therefore , A running water must be able to execute correctly in the end . therefore , When extracting a piece of business code from the pipeline, we must consider :

  • ** Pipeline delay processing .** Pipelining is not real-time , It's implemented asynchronously with an over pipelined actuator . therefore , If in the original logic , Special attention should be paid to whether the subsequent process has real-time dependence on the pipeline ( For example, pipeline results will be used in subsequent business logic to do some calculations, etc ).

  • ** Pipelining disorder .** Ensure that even if the later generated pipeline is executed first , There can't be a problem .

  • ** The final success of running water .** For each inserted pipeline , This flow of water must ensure that it can be executed successfully

therefore , When we extract water :

  • The simpler the pipelining, the better

  • The less we rely on, the better

  • The extracted pipeline has no real-time dependency in the business logic

2.3 Pipelined processors

Pipelining processor is to ensure that pipelining process as fast as possible , It can also ensure that the running water can be implemented successfully .

Imagine a scenario : When a pipeline fails to process , If the drain actuator has to wait for the current pipeline to execute successfully before continuing to execute later , Then it will affect the implementation of the follow-up flow , What's more, it's stuck as a record all the time , Cause problems in the whole system

therefore , Set... In the pipeline actuator 2 A mission :

  • The first task , Pipelining tasks , The pipeline has been executed at the fastest speed , If pipelining fails , It doesn't affect the following flow process

  • The second task , Pipeline verification task , This task is to check the flow records in sequence , Make sure that all flows are executed successfully , If you fail , retry , After repeated retries fail, an alarm is issued to allow manual intervention .

2.4 The flow process is complete

Because the flow table is placed in the original database , After the completion of pipeline processing, it is the operation branch library , If the sub database operation is completed, update the old table pipeline message , So it's the quaku business again , How to ensure the update and sub database of pipeline state is also in the process of a transaction ?

The solution is : Create a flow table in the branch library , When the wastage treatment is completed , Not to update the old watch status , Instead, it is inserted into the branch flow table 、

The benefits of doing this :

  • In general, the pipeline is uniquely indexed , So if the pipeline is repeated many times , When inserting the branch library flow table, it must be because the unique index detection fails , The entire transaction will be rolled back ( Of course, we can also make idempotent judgment before dealing with the flow of water )

  • In this way, by judging whether the main library pipeline is in the branch library, we can judge whether a pipeline has been executed

3、 ... and 、 The basic framework of pipelined processor

Pipelined processors do not contain any business-related processing logic , The core function is :

  • Inform the service access party when to handle what kind of flow

  • Test the success of flow execution

notes : The pipelining executor does not know what logic the pipelining represents , Specifically, the business system needs to identify and execute the corresponding business logic .

3.1 Running the task

Pipeline processing scheduling task is to scan the pipeline to be processed , Then inform the business system which flow to execute .

The schematic diagram is as follows :

3.2 Pipeline verification task

The task of pipeline verification is to compare the pipeline records in the main database and the branch database , Reprocess the flow notification business system that fails to execute , If multiple retries fail, an alarm will be issued .

Flow diagram :

Four 、 Why not use transaction messages

Because it is an existing project to carry out transformation ( I am engaged in Internet finance , Therefore, it is absolutely intolerable that any message is lost or the message processing fails ), Without using transaction messages, there are 1 One reason

  • Additional message queues need to be introduced , Increases the complexity of the system , It also needs additional logic guarantee and handling when message queue communication fails

  • Actually 1 Not the main reason , It's because transaction messages need to be manually commit and rollback( Using a database doesn't require ), So here comes the question ,spring Transactions are transitive , It's a big problem when we submit the transaction information , for example A.a() It's a business , But another thing B.b() Call... Again A.a() The transaction message commit is placed in A.a() still B.b() What about China? ?

At the end

Welcome to my official account 【 Calm as a code 】, Massive Java Related articles , Learning materials will be updated in it , The sorted data will also be put in it .

Like what you write , Just pay attention ! Focus , Neverlost , Continuous updating !!!

Please bring the original link to reprint ,thank
Similar articles