Detailed introduction to canal (use summary)
CZQ_ Darren 2021-06-04 09:02:32

Canal Introduce

Canal brief introduction

image

canal [kə'næl], Waterways / The Conduit / Ditch , The main use is based on MySQL database Incremental log analysis , Provide Incremental data subscription and consumption

In the early days, Alibaba was deployed due to the dual computer rooms in Hangzhou and the United States , There is a business demand for synchronization across computer rooms , The implementation is mainly based on the business trigger Get incremental changes . from 2010 Year begins , The business gradually tries to get incremental changes from database log parsing for synchronization , Thus, a large number of incremental database subscription and consumption businesses are derived .

Incremental subscription and consumption based on logs include

  • database mirroring
  • Database real-time backup
  • Index building and real-time maintenance ( Split heterogeneous indexes 、 Inverted index, etc )
  • Business cache Refresh
  • Incremental data processing with business logic

Current canal Support source side MySQL The version includes 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x

MySQL working principle

image

  • MySQL master Write data changes to binary log ( binary log, The records are called binary log events binary log events, Can pass show binlog events To view the )
  • MySQL slave take master Of binary log events Copy to its trunk log (relay log)
  • MySQL slave replay relay log Middle event , Change the data to reflect its own data

Canal working principle

image

  • canal simulation MySQL slave Interaction protocol , Pretend to be MySQL slave , towards MySQL master send out dump agreement
  • MySQL master received dump request , Start pushing binary log to slave ( namely canal )
  • canal analysis binary log object ( Originally byte flow )


Canal Preparation for use

MySQL Get ready

1、 For self built MySQL , It needs to be turned on first Binlog Write function , To configure binlog-format by ROW Pattern ,my.cnf The configuration in is as follows

[mysqld]
log-bin=mysql-bin # Turn on binlog
binlog-format=ROW # choice ROW Pattern
server_id=1 # To configure MySQL replaction Need to define , Do not mix canal Of slaveId repeat

2、 to grant authorization canal link MySQL The account has the function of MySQL slave Authority , If you have an account, you can directly grant

-- Log in with the command :mysql -u root -p
-- establish mysql user user name :canal password :canal
CREATE USER canal IDENTIFIED BY 'canal';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;

3、 Restart the database , Check whether the configuration is effective

mysql> show variables like 'binlog_format';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW |
+---------------+-------+
mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin | ON |
+---------------+-------+

Canal Preparation for use

Canal download , Use here 1.1.5 Version as an example

github link :https://github.com/alibaba/canal/releases

Need to download ( Others can be downloaded according to your own needs ):

  • canal.deployer-1.1.5.tar.gz
  • canal.adapter-1.1.5.tar.gz

Decompress after downloading

1.1.5 After decompression, the directory structure is as follows

bin :canal start-up 、 restart 、 Stop the file
conf :canal The configuration file
lib :canal Required for operation jar package , Note the database driver package version , It can be replaced manually
logs :canal Run log
plugin : Some expansion packs , Such as message queuing

Create databases and tables

Record on pit : Do not use underscores for database names , At the beginning with canal_test_01 How to name , It was found that the database could not be scanned

-- Source database :canal01
CREATE DATABASE `canal01` CHARACTER SET 'utf8mb4';
-- establish user01 surface
CREATE TABLE `user01` (
`id` int(64) NOT NULL AUTO_INCREMENT,
`username` varchar(64) DEFAULT NULL,
`password` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;
-- Target database :canal02
CREATE DATABASE `canal02` CHARACTER SET 'utf8mb4';
-- establish user02 surface
CREATE TABLE `user02` (
`id` int(64) NOT NULL AUTO_INCREMENT,
`username` varchar(64) DEFAULT NULL,
`password` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;


Canal-deployer Use

1、conf\canal.properties The configuration file ( Here's just the configuration I use )

# canal-deployer Of ip Address , No configuration, default is local
canal.ip =
# canal-deployer Port number
canal.port = 11111
# Pattern
canal.serverMode = tcp
# Load multiple profiles , Need to be in canal.deployer-1.1.5\conf Create the corresponding folder under , And plus instance.properties The configuration file
canal.destinations = example
# Configure multiple, separated by English commas
# canal.destinations = example,example2

2、conf\example\instance.properties The configuration file ( Here's just the configuration I use )

# Cannot be associated with my.ini Under the server_id=1 identical
canal.instance.mysql.slaveId=1234
# mysql Address and port
canal.instance.master.address=127.0.0.1:3306
# Database account password , have access to root Account number or just created canal account number
canal.instance.dbUsername=root
canal.instance.dbPassword=root
canal.instance.connectionCharset=UTF-8
# Configured to monitor , regular expression
canal.instance.filter.regex=canal01\\..*
# Configuration does not monitor , regular expression
canal.instance.filter.black.regex=mysql\\.slave_.*

3、 Check lib In the catalog jar package , There must be a corresponding database driver package , And the version should be right

4、 stay bin Directory canal-deployer, Start successfully, output as follows

# Post run output
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m;
support was removed in 8.0 Listening for transport dt_socket at address: 9099
# stay logs\canal Under the table of contents , Will record canal.log
# The following record shows that canal-deployer The successful running
2021-04-28 09:26:57.194 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2021-04-28 09:26:57.244 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2021-04-28 09:26:57.257 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2021-04-28 09:26:57.399 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.3.51(192.168.3.51):11111]
2021-04-28 09:26:59.440 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......
# stay logs\example Under the table of contents , Will record example.log
# The following record shows that canal-deployer The connection to the database is successful
2021-04-28 09:30:37.159 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2021-04-28 09:30:37.175 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^fooddb\..*$
2021-04-28 09:30:37.175 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$
2021-04-28 09:30:37.270 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2021-04-28 09:30:37.288 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2021-04-28 09:30:37.305 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position
{"identity":{"slaveId":-1,"sourceAddress":{"address":"ieonline.microsoft.com","port":3306}},"postion":{"gtid":"","included":false,"journalName":"binlog.000068","position":14636,"serverId":1,"timestamp":1619408047000}}
2021-04-28 09:30:37.744 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=binlog.000068,position=14636,serverId=1,gtid=,timestamp=1619408047000] cost : 466ms , the next step is binlog dump

5、 establish springboot Engineering monitoring canal-deployer( Skippable )

Import maven coordinate (pom.xml)

<!--Canal-->
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.client</artifactId>
<version>1.1.4</version>
</dependency>

establish CanalClient class , And to spring management

import java.net.InetSocketAddress;
import java.util.List;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.CanalEntry.Column;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalEntry.EntryType;
import com.alibaba.otter.canal.protocol.CanalEntry.EventType;
import com.alibaba.otter.canal.protocol.CanalEntry.RowChange;
import com.alibaba.otter.canal.protocol.CanalEntry.RowData;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Component;
@Component
public class CanalClient implements InitializingBean {
private final static int BATCH_SIZE = 1000;
@Override
public void afterPropertiesSet() throws Exception {
// Create links
CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress("127.0.0.1", 11111), "example", "", "");
try {
// Open the connection
connector.connect();
// Subscribe to database tables , All tables
connector.subscribe(".*\\..*");
// Roll back to not in progress ack The place of , The next time fetch When , Can be from the last one without ack Where you start to take
connector.rollback();
while (true) {
// Get a specified amount of data
Message message = connector.getWithoutAck(BATCH_SIZE);
// Get batch ID
long batchId = message.getId();
// Get the number of batches
int size = message.getEntries().size();
// If there is no data
if (batchId == -1 || size == 0) {
try {
// Thread to sleep 2 second
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
} else {
// If there's data , Processing data
printEntry(message.getEntries());
}
// Conduct batch id The confirmation of . After confirmation , Less than or equal to this batchId Of Message Will be confirmed .
connector.ack(batchId);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
connector.disconnect();
}
}
/**
* Print canal server analysis binlog Get the entity class information
*/
private static void printEntry(List<Entry> entrys) {
for (Entry entry : entrys) {
if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {
// Turn on / Close the entity type of the transaction , skip
continue;
}
//RowChange object , Contains all the characteristics of a row of data changes
// such as isDdl Whether it is ddl Change operation sql Concrete ddl sql beforeColumns afterColumns Data fields before and after the change, etc
RowChange rowChage;
try {
rowChage = RowChange.parseFrom(entry.getStoreValue());
} catch (Exception e) {
throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(), e);
}
// Get operation type :insert/update/delete type
EventType eventType = rowChage.getEventType();
// Print Header Information
System.out.println(String.format("================》; binlog[%s:%s] , name[%s,%s] , eventType : %s",
entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),
entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),
eventType));
// Judge whether it is DDL sentence
if (rowChage.getIsDdl()) {
System.out.println("================》;isDdl: true,sql:" + rowChage.getSql());
}
// obtain RowChange Every row of data in the object , Print out
for (RowData rowData : rowChage.getRowDatasList()) {
// If it's a delete statement
if (eventType == EventType.DELETE) {
printColumn(rowData.getBeforeColumnsList());
// If it is a new statement
} else if (eventType == EventType.INSERT) {
printColumn(rowData.getAfterColumnsList());
// If it's an updated statement
} else {
// Data before change
System.out.println("------->; before");
printColumn(rowData.getBeforeColumnsList());
// Changed data
System.out.println("------->; after");
printColumn(rowData.getAfterColumnsList());
}
}
}
}
private static void printColumn(List<Column> columns) {
for (Column column : columns) {
System.out.println(column.getName() + " : " + column.getValue() + " update=" + column.getUpdated());
}
}
}

start-up springboot project , stay user01 Add data to table , The console outputs as follows , Monitor successful

================》; binlog[binlog.000070:5154] , name[canal01,user01] , eventType : INSERT
id : 1 update=true
username : a update=true
password : a update=true


Canal-adapter Use

1、conf\application.yml The configuration file ( Here's just the configuration I use )

server:
port: 8081 # canal-adapter Run port number
canal.conf:
mode: tcp # canal client The pattern of : tcp kafka rocketMQ rabbitMQ
consumerProperties:
# canal tcp consumer
canal.tcp.server.host: 127.0.0.1:11111
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
srcDataSources: # Source database
canal01: # Custom name
# What we use here is mysql8.0, Need to add serverTimezone=UTC
url: jdbc:mysql://127.0.0.1:3306/canal01?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
username: root
password: root
# Configure multiple source databases
# canal02:
# url: jdbc:mysql://127.0.0.1:3306/canal02?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
# username: root
# password: root
canalAdapters: # Adapter list
- instance: example # canal Instance name or MQ topic name
groups: # Group list
- groupId: # grouping id, If it is MQ The pattern will use this value
outerAdapters: # List of adapters in the group
# - name: logger # Log print adapter
- name: rdb # Designated as rdb Type synchronization
key: mysql1 # Appoint adapter The only key, And table mapping configuration outerAdapterKey Corresponding
# Target database configuration
properties:
jdbc.driverClassName: com.mysql.jdbc.Driver
jdbc.url: jdbc:mysql://127.0.0.1:3306/canal02?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
jdbc.username: root
jdbc.password: root
threads: 5 # Number of threads executing in parallel , It can be left unchanged , The default is 1
# Configure multiple target databases
# - instance: example2 # canal instance Name or mq topic name
# groups:
# - groupId: g2
# outerAdapters:
# - name: rdb
# key: mysql2
# properties:
# jdbc.driverClassName: com.mysql.jdbc.Driver
# jdbc.url: jdbc:mysql://127.0.0.1:3306/canal01?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
# jdbc.username: root
# jdbc.password: root

2、 To configure RDB Table mapping file (conf/rdb/mytest_user.yml file )

dataSourceKey: canal01 # Of the source data source key, Corresponding to the srcDataSources The value in
destination: example # cannal Of instance perhaps MQ Of topic
groupId: g1 # Corresponding MQ Mode of groupId, It only synchronizes the correspondence groupId The data of
outerAdapterKey: mysql1 # adapter key, Corresponding to the above configuration outAdapters Medium key
concurrent: true # By primary key hash Parallel synchronization , The parallel synchronized table must ensure that the primary key will not be changed and that the primary key cannot be a foreign key of other synchronized tables !!
dbMapping:
database: canal01 # Of the source data source database/shcema
table: user01 # Source data source table name
targetTable: user02 # Write the name of the watch directly
targetPk: # Primary key mapping
id: id # If it is a composite primary key, you can wrap and map multiple
mapAll: true # Whether to map the whole table , As like as two peas ( If targetColumns Mapping is also configured , with targetColumns Configuration is subject to )
# targetColumns: # Field mapping , Format : Target table fields : Source table fields , If the field name is the same, the field name of the source table may not be filled in
# id:
# name:
# role_id:
# c_time:
# test1:
etlCondition: "where c_time>={}"
commitBatch: 3000 # The size of the batch submission

3、 Check lib In the catalog jar package , There must be a corresponding database driver package , And the version should be right

4、 stay bin Directory canal-adapter, Start successfully, output as follows

# Post run output
2021-04-28 11:12:38.281 [main] INFO o.s.b.w.s.c.AnnotationConfigServletWebServe
rApplicationContext - Refreshing org.springframework.boot.web.servlet.context.An
notationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr
28 11:12:38 CST 2021]; parent: org.springframework.context.annotation.Annotation
ConfigApplicationContext@1a677343
2021-04-28 11:12:39.012 [main] INFO org.springframework.cloud.context.scope.Gen
ericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039
2021-04-28 11:12:39.108 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$B
eanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.Configur
ationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.aut
oconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGL
IB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (f
or example: not eligible for auto-proxying)
2021-04-28 11:12:39.610 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServe
r - Tomcat initialized with port(s): 8081 (http)
2021-04-28 11:12:39.632 [main] INFO org.apache.coyote.http11.Http11NioProtocol
- Initializing ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:39.646 [main] INFO org.apache.catalina.core.StandardService -
Starting service [Tomcat]
2021-04-28 11:12:39.647 [main] INFO org.apache.catalina.core.StandardEngine - S
tarting Servlet Engine: Apache Tomcat/8.5.29
2021-04-28 11:12:39.660 [localhost-startStop-1] INFO org.apache.catalina.core.A
prLifecycleListener - The APR based Apache Tomcat Native library which allows op
timal performance in production environments was not found on the java.library.p
ath: [C:\ProgramData\Oracle\Java\javapath;C:\Windows\Sun\Java\bin;C:\Windows\sys
tem32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Wind
ows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Java
\jdk1.8.0_131\bin;C:\Java\jdk1.8.0_131\jre\bin;D:\Tools\JAVA\apache-maven-3.6.3\
bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Com
mon;C:\go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;;C:\User
s\Administrator\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\Je
tBrains\WebStorm 2019.3.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4
\bin;;C:\Users\Administrator\go\bin;C:\Users\Administrator\AppData\Roaming\npm;C
:\Users\Administrator\AppData\Local\Yarn\bin;D:\Tools\zookeeper-3.4.13\bin;;.]
2021-04-28 11:12:39.830 [localhost-startStop-1] INFO o.a.catalina.core.Containe
rBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationCont
ext
2021-04-28 11:12:39.831 [localhost-startStop-1] INFO org.springframework.web.co
ntext.ContextLoader - Root WebApplicationContext: initialization completed in 15
50 ms
2021-04-28 11:12:39.990 [localhost-startStop-1] INFO o.s.boot.web.servlet.Servl
etRegistrationBean - Servlet dispatcherServlet mapped to [/]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2021-04-28 11:12:40.001 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2021-04-28 11:12:40.002 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class
is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SP
I and manual loading of the driver class is generally unnecessary.
2021-04-28 11:12:40.597 [main] INFO com.alibaba.druid.pool.DruidDataSource - {d
ataSource-1} inited
2021-04-28 11:12:40.849 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springf
ramework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.143 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.serv
let.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup
date [Wed Apr 28 11:12:38 GMT+08:00 2021]; parent: org.springframework.context.a
nnotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:41.245 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto publi
c java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adap
ter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.S
tring)
2021-04-28 11:12:41.247 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java
.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.la
uncher.rest.CommonRest.count(java.lang.String,java.lang.String)
2021-04-28 11:12:41.248 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" on
to public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otte
r.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public
java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapt
er.launcher.rest.CommonRest.etl(java.lang.String)
2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public
com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.cana
l.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.la
ng.String,java.lang.String)
2021-04-28 11:12:41.250 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.a
libaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adap
ter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.Str
ing)
2021-04-28 11:12:41.251 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.L
ist<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.a
dapter.launcher.rest.CommonRest.destinations()
2021-04-28 11:12:41.256 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/error]}" onto public org.springframework.http.Respons
eEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.b
oot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.htt
p.HttpServletRequest)
2021-04-28 11:12:41.258 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springf
ramework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.ser
vlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse)
2021-04-28 11:12:41.298 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/webjars/**] onto handler of type [class org.springframe
work.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.299 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/**] onto handler of type [class org.springframework.web
.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.604 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Registering beans for JMX exposure on startup
2021-04-28 11:12:41.618 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2021-04-28 11:12:41.620 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'configurationPropertiesRebinder' has been autodetected
for JMX exposure
2021-04-28 11:12:41.622 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'environmentManager' has been autodetected for JMX expos
ure
2021-04-28 11:12:41.625 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'environmentManager': registering with JMX server
as MBean [org.springframework.cloud.context.environment:name=environmentManager,
type=EnvironmentManager]
2021-04-28 11:12:41.644 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'refreshScope': registering with JMX server as MBe
an [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=Refre
shScope]
2021-04-28 11:12:41.659 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'configurationPropertiesRebinder': registering wit
h JMX server as MBean [org.springframework.cloud.context.properties:name=configu
rationPropertiesRebinder,context=2ef14fe,type=ConfigurationPropertiesRebinder]
2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## syncSwitch refreshed.
2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## start the canal client adapters.
2021-04-28 11:12:41.678 [main] INFO c.a.otter.canal.client.adapter.support.Exte
nsionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools
\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:41.777 [main] INFO c.a.otter.canal.client.adapter.rdb.config.C
onfigLoader - ## Start loading rdb mapping config ...
2021-04-28 11:12:41.837 [main] INFO c.a.otter.canal.client.adapter.rdb.config.C
onfigLoader - ## Rdb mapping config loaded
2021-04-28 11:12:42.224 [main] INFO com.alibaba.druid.pool.DruidDataSource - {d
ataSource-2} inited
2021-04-28 11:12:42.234 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterLoader - Load canal adapter: rdb succeed
2021-04-28 11:12:42.246 [main] INFO c.alibaba.otter.canal.connector.core.spi.Ex
tensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Too
ls\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:42.275 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2021-04-28 11:12:42.276 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader
.AdapterProcessor - =============> Start to connect destination: example <======
=======
2021-04-28 11:12:42.276 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## the canal client adapters are running now ......
2021-04-28 11:12:42.286 [main] INFO org.apache.coyote.http11.Http11NioProtocol
- Starting ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:42.296 [main] INFO org.apache.tomcat.util.net.NioSelectorPool
- Using a shared selector for servlet write/read
2021-04-28 11:12:42.316 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServe
r - Tomcat started on port(s): 8081 (http) with context path ''
2021-04-28 11:12:42.320 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapt
erApplication - Started CanalAdapterApplication in 5.287 seconds (JVM running fo
r 6.186)
2021-04-28 11:12:42.354 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader
.AdapterProcessor - =============> Subscribe destination: example succeed <=====
========
# logs\adapter Under the table of contents , Will record adapter.log
# The following record shows that canal-adapter Successfully run
2021-04-28 11:12:37.612 [main] INFO o.s.c.annotation.AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343: startup date [Wed Apr 28 11:12:37 CST 2021]; root of context hierarchy
2021-04-28 11:12:38.027 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-28 11:12:38.254 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - No active profile set, falling back to default profiles: default
2021-04-28 11:12:38.281 [main] INFO o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 CST 2021]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:39.012 [main] INFO org.springframework.cloud.context.scope.GenericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039
2021-04-28 11:12:39.108 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-28 11:12:39.610 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8081 (http)
2021-04-28 11:12:39.632 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:39.646 [main] INFO org.apache.catalina.core.StandardService - Starting service [Tomcat]
2021-04-28 11:12:39.647 [main] INFO org.apache.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.29
2021-04-28 11:12:39.660 [localhost-startStop-1] INFO org.apache.catalina.core.AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\ProgramData\Oracle\Java\javapath;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Java\jdk1.8.0_131\bin;C:\Java\jdk1.8.0_131\jre\bin;D:\Tools\JAVA\apache-maven-3.6.3\bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;;C:\Users\Administrator\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\JetBrains\WebStorm 2019.3.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4\bin;;C:\Users\Administrator\go\bin;C:\Users\Administrator\AppData\Roaming\npm;C:\Users\Administrator\AppData\Local\Yarn\bin;D:\Tools\zookeeper-3.4.13\bin;;.]
2021-04-28 11:12:39.830 [localhost-startStop-1] INFO o.a.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2021-04-28 11:12:39.831 [localhost-startStop-1] INFO org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 1550 ms
2021-04-28 11:12:39.990 [localhost-startStop-1] INFO o.s.boot.web.servlet.ServletRegistrationBean - Servlet dispatcherServlet mapped to [/]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2021-04-28 11:12:40.001 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2021-04-28 11:12:40.002 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
2021-04-28 11:12:40.597 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
2021-04-28 11:12:40.849 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.143 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 GMT+08:00 2021]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:41.245 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.247 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String)
2021-04-28 11:12:41.248 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" onto public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String)
2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.250 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.251 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.List<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.destinations()
2021-04-28 11:12:41.256 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2021-04-28 11:12:41.258 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2021-04-28 11:12:41.298 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.299 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.604 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2021-04-28 11:12:41.618 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2021-04-28 11:12:41.620 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2021-04-28 11:12:41.622 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'environmentManager' has been autodetected for JMX exposure
2021-04-28 11:12:41.625 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2021-04-28 11:12:41.644 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2021-04-28 11:12:41.659 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=2ef14fe,type=ConfigurationPropertiesRebinder]
2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## syncSwitch refreshed.
2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2021-04-28 11:12:41.678 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:41.777 [main] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Start loading rdb mapping config ...
2021-04-28 11:12:41.837 [main] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Rdb mapping config loaded
2021-04-28 11:12:42.224 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-2} inited
2021-04-28 11:12:42.234 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: rdb succeed
2021-04-28 11:12:42.246 [main] INFO c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:42.275 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2021-04-28 11:12:42.276 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: example <=============
2021-04-28 11:12:42.276 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2021-04-28 11:12:42.286 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:42.296 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2021-04-28 11:12:42.316 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2021-04-28 11:12:42.320 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 5.287 seconds (JVM running for 6.186)
2021-04-28 11:12:42.354 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: example succeed <=============

5、 towards user01 Add data to table ,user02 The table also has corresponding data , Synchronous success

2021-04-28 11:27:59.619 [pool-2-thread-1] DEBUG c.a.o.canal.client.adapter.rdb.s
ervice.RdbSyncService - DML: {"data":{"id":1,"username":"a","password":"a"},"dat
abase":"canal01","destination":"example","old":null,"table":"user01","type":"INS
ERT"}

If you need user02 Synchronize to user01:

​ (1) Add corresponding canal-deployer Add the corresponding example

​ (2)canal-adapter Of application.yml Add the corresponding source database and target database to the configuration file ( See configuration above for details )

​ (3) newly build RDB Table mapping file , stay rdb Under the table of contents ,canal-adapter Will scan by default rdb All the yml file

Synchronize multiple tables :

​ (1) Add tables to both databases ( Here we use person For example )

​ (2) To configure RDB Table mapping file , take mytest_user.yml A copy of , Rename it to mytest_person.yml

​ (3) modify mytest_person.yml Medium table( Source table )、targetTable( Target table )

6、 If needed Java engineering , You can be on your own github Of canal Download canal-adapter The engineering of , The configuration is exactly the same as above

Please bring the original link to reprint ,thank
Similar articles

2021-08-09

2021-08-09