Deploy and run Canal locally

  • Updated on 8th Aug 2023

Background

The version used in this article is the latest released package of Canal as of the time of this article’s publication, canal-1.1.8-alpha-2. The latest package can be downloaded from canal release page。 The local deployment and running environment are Windows 10, with MySQL version 8.0 or above.

Prerequisites

Canal release package

Download the Canal release package canal.admin-1.1.8-SNAPSHOT.tar.gz and canal.deployer-1.1.8-SNAPSHOT.tar.gz using the address provided in the previous text. Then, extract the downloaded package through the command line as follows:

## After executing the following command, 
## it is necessary to transfer the relevant files to the canal-admin directory
> tar -zxvf .\canal.admin-1.1.8-SNAPSHOT.tar.gz

## After executing the following command, 
## it is necessary to transfer the relevant files to the canal-deployer directory
> tar -zxvf .\canal.deployer-1.1.8-SNAPSHOT.tar.gz

Confirm MySQL connector version

Since we are using MySQL 8.0, it is necessary to confirm whether the default drivers provided in canal-admin and canal-deployer are mysql8.0+. If the driver is for mysql8.0+, then no action is needed. If not, you will need to replace the corresponding package in the lib folder under the extracted directory. (Under Linux, you may need to check if authorization is required.)

MySQL connector official download address:https://downloads.mysql.com/archives/c-j/

On the download page, you need to select Product Version (choose a version above 8.0) and Operating System (choose Platform Independent for the operating system).

Deploying and running Canal admin

If not specifically stated, the root directory for this section of the document is canal-admin.

Creating database

The Canal Admin requires support for the MySQL database. Its database and corresponding tables are created through a script located at: conf/canal_manager.sql.

Change configuration file

The location of the configuration file is:conf/application. yml. You need to modify the corresponding configuration in this file according to the actual situation.

Start running

Enter the bin directory and run the command line:

bin> .\startup.bat

After successful startup, access through the browser:http://127.0.0.1:8089/,Enter the default account and password admin/123456,Then enter the Canal background management.

Backend Management UI for Canal

Cluster management

In cluster management, click New Cluster to set up a new cluster configuration. The “ZK Address” is the Zookeeper address of the server cluster[1]. Here, use the default address: 127.0.0.1:2181. The Canal cluster consists of multiple Canal Deployer nodes grouped together to enhance system availability and scalability.

After the cluster is established, it is also necessary to add a configuration template to the cluster. Otherwise, when running Canal Deployer later, an error “requestGet for canal config error: canal.properties config is empty”will occur.

In the cluster management list, select a cluster, and in the “操作” dropdown menu, click “主配置” as shown in the figure below:

Adding Cluster Configuration Template for Canal

Then, on the Server Configuration page, click “载入模板” to load the default canal.properties configuration template as shown in the figure below:

Cluster Configuration Template for Canal

Adjust the relevant configurations in this configuration template according to the actual situation. Here, the Canal service mode I am using is “RocketMQ”, with the corresponding configuration set as: canal.serverMode = rocketMQ, and the MQ Properties and RocketMQ sections in the configuration template are properly set.

Since I am using Alibaba Cloud’s RocketMQ, the MQ Properties section in the configuration needs to be adjusted as follows:

canal.aliyun.accessKey =[Obtain from Alibaba Cloud]
canal.aliyun.secretKey =[Obtain from Alibaba Cloud]

# Set this value to "cloud", if you want open message trace feature in aliyun.
canal.mq.accessChannel = cloud

If accessKey and secretKey are not set, the Canal Deployer will not start up properly. Similarly, the RocketMQ section also needs to be adjusted as shown in the following example:

rocketmq.producer.group = GID_xxx
rocketmq.customized.trace.topic = canal_xxx
rocketmq.namespace =
rocketmq.namesrv.addr = [Aliyun RocketMQ VPC TCP Endpoint, Canal Default Tcp]
rocketmq.retry.times.when.send.failed = 1

TableMetaTSDB

The timing table structure is designed to address the issue of DDL synchronization. For details, refer to the 时序表结构方案设计. If you are not concerned with changes in the table structure, you can disable this feature by setting canal.instance.tsdb.enable = false in the cluster server’s canal.properties configuration.

Deploying and running Canal deployer

If not specifically stated, the root directory of this section is canal-deployer.

Change configuration file

In the conf directory, first back up canal.properties, and then rename canal_local.properties to canal.properties.

The revised content is as follows:

# Current node register ip
canal.register.ip = 127.0.0.1

# canal admin config
canal.admin.manager = 127.0.0.1:8089
canal.admin.port = 11110
canal.admin.user = admin

# Admin password, the password encrypted with MySQL's password,
# corresponds to the password set in the admin's conf/application.yml.
canal.admin.passwd = 6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9

# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster = cluster-dev
canal.admin.register.name = server-01

To adjust the admin password, you can set it using the following mysql command:

# mysql5.0
select password('xxx')

# mysql8.0
select upper(sha1(unhex(sha1('xxx'))))

Copy RocketMQ connector dependency package

Due to the use of the RocketMQ service model, you need to copy the file connector.rocketmq-xxx-SNAPSHOT-jar-with-dependencies.jar from the plugin directory to the lib directory. Otherwise, subsequent startup will fail, and the following error will appear in the canal.log:

interface com.alibaba.otter.canal.connector.core.spi.CanalMQProducer) could not be instantiated: class could not be found.

Start running deployer

Enter the bin directory and run the command line:

bin> .\startup.bat

After successful startup, the server-01 will automatically register with the cluster (if it does not successfully register automatically, it can be added manually through “新建Server”). The result is as shown in the following figure:

Canal Server List

Then, you can view the log status of server-01 in [操作->日志].

Canal Server Log

However, there is still a problem here, in the log file rocketmq_client.log under the logs directory, there are a large number of the following exceptions:

WARN RocketmqClient - get Topic [TBW102] RouteInfoFromNameServer is not exist value WARN RocketmqClient - updateTopicRouteInfoFromNameServer Exception org.apache.rocketmq.client.exception.MQClientException: CODE: 17 DESC: No topic route info in name server for the topic: TBW102

After investigation and analysis, the abnormality is due to RocketMQ’s own reasons[2], and there is no good solution. It’s either to mask the log or to adjust the source code.

About logs

It has been found that the log files canal.log and rocketmq_client.log are saved on a daily basis, and logs from previous days are archived in the directory.

Some of the content in this post refers to Canal Introduction.


  1. ZooKeeper server needs to be deployed locally separately, and the post will be supplemented later

  2. Due to the disabled Broker automatically creating the Topic, see details:https://github.com/apache/rocketmq/issues/3179