Dynamic data sources

1. background

There are many requirements for dynamic data sources in real business scenarios , And if you want to communicate with multiple databases, you really need to encapsulate this tool , Aim at bi Tools may involve getting data from different business libraries or data warehouses , Dynamic data sources make more sense .

2. rely on

<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>4.3.7.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
<version>4.3.7.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>4.3.7.RELEASE</version>
</dependency> <dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>1.0.9</version>
</dependency>
<dependency>
<groupId>com.viewhigh.bi.common</groupId>
<artifactId>common</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.8.10</version>
</dependency>

3. Principle analysis of multiple data sources

① Initialize the default data source when the application starts , And register the default data source to spring In the context of , In the process, we need to achieve EnvironmentAware Interface setEnvironment Method , We know setEnvironment Method is called when the context is initialized , Then we can use this opportunity to initialize the default data source according to the configuration file , Of course, you can initialize 1 One or more .

/**
* Created by zzq on 2017/6/14.
* Responsible for initializing data source configuration
*/
public class DataSourceRegister<T> implements EnvironmentAware, ImportBeanDefinitionRegistrar {
private javax.sql.DataSource defaultTargetDataSource;
static final String MAINDATASOURCE = "mainDataSource"; public final void setEnvironment(Environment environment) {
DruidEntity druidEntity = FileUtil.readYmlByClassPath("db_info", DruidEntity.class); defaultTargetDataSource = DataSourceUtil.createMainDataSource(druidEntity);
} public final void registerBeanDefinitions(AnnotationMetadata annotationMetadata, BeanDefinitionRegistry beanDefinitionRegistry) {
// 0. Add the master data source to the data source collection
DataSourceSet.putTargetDataSourcesMap(MAINDATASOURCE, defaultTargetDataSource);
//1. establish DataSourceBean
GenericBeanDefinition beanDefinition = new GenericBeanDefinition();
beanDefinition.setBeanClass(DataSource.class);
beanDefinition.setSynthetic(true);
MutablePropertyValues mpv = beanDefinition.getPropertyValues();
//spring The name of the contract is defaultTargetDataSource and targetDataSources
mpv.addPropertyValue("defaultTargetDataSource", defaultTargetDataSource);
mpv.addPropertyValue("targetDataSources", DataSourceSet.getTargetDataSourcesMap());
beanDefinitionRegistry.registerBeanDefinition("dataSource", beanDefinition);
}
}

Register in the above code Data SourceBean You can specify a default data source , This data source is used by default and stored in defaultTargetDataSource, Other data sources exist targetDataSources.

② So if you want to use other data sources, you need to targetDataSources Through the specified key Just switch . Before that, we need to rewrite Spring in AbstractRoutingDataSource Type of determineCurrentLookupKey Method , The return value is corresponding to the data source to be started key, In this way, the purpose of switching multiple data sources is achieved .

/**
* Created by zzq on 2017/6/13.
*/
public class DataSource extends AbstractRoutingDataSource {
@Override
protected Object determineCurrentLookupKey() {
String keyDataSource = DataSourceSet.getCurrDataSource();
LogUtil.info("*** The current data source is [{}]", keyDataSource == null ? " Default data source " : keyDataSource);
return keyDataSource;
}
}

4. design scheme

  • Usage mode :

1)  Find an initialization time when the application starts , And use import Just import the data source registration class

@Import({DataSourceRegister.class})
@SpringBootApplication//(exclude={DataSourceAutoConfiguration.class,HibernateJpaAutoConfiguration.class})
@ComponentScan("com.XXX.bi")
//@EnableCaching
public class BiApplication {
public static void main(String[] args) {
LogUtil.setEnabled(true);// Turn on log output SpringApplication sa = new SpringApplication(BiApplication.class);
sa.setBannerMode(Banner.Mode.LOG);
sa.run(args);
}
}

2)  Use... By annotation

Annotations are easier to understand , But it's not flexible enough compared to code ; Examples are as follows :

@ActivateDataSource("001")
public List findAll() {
String sql = "select * from td_bi_datasourcetype where is_remove=0 and organization_id=? "; Map map = new HashMap();
map.put("id", String.class);
map.put("code", String.class);
map.put("remark", String.class);
map.put("name", String.class);
map.put("is_remove", Integer.class); DynamicBean dynamicBean = new DynamicBean(map);
String orgId = Identity.getOrganizationId();
return jdbcTemplateExtend.query(sql, new Object[]{orgId}, dynamicBean.getObject().getClass());
}

Provides marking annotations at the beginning of a method , And specify the data source key, For annotation parameters , The current key The corresponding data source .

I believe you have guessed , We maintain a DataSourceSet aggregate , Data sources are stored in this collection key And corresponding to the actual DataSource object . And in contextHolder The data source currently set is stored in key value , In this way, the system is called directly when the query method is triggered determineCurrentLookupKey Method , In this method contextHolder Of key value ;

/**
* Created by zzq on 2017/6/13.
*/
public class DataSourceSet {
private static final ThreadLocal<String> contextHolder = new ThreadLocal(); private static List<String> dataSourceKeyList = new CopyOnWriteArrayList<String>(); private static Map targetDataSourcesMap = new ConcurrentHashMap(); public static Object putTargetDataSourcesMap(Object key, Object dataSource) {
dataSourceKeyList.add(key.toString());
return targetDataSourcesMap.put(key, dataSource);
} public static Object removeTargetDataSourcesMap(Object key) {
try {
dataSourceKeyList.remove(key);
return targetDataSourcesMap.remove(key);
} catch (Exception e) {
e.printStackTrace();
throw new CustomException(00000, " remove DataSourceSet Exception in data source information , Probably because of dataSourceKeyList or targetDataSourcesMap There is no the item term ");
}
} public static Map getTargetDataSourcesMap() {
return targetDataSourcesMap;
} public static void setCurrDataSource(String ds) {
contextHolder.set(ds);
} public static String getCurrDataSource() {
return contextHolder.get();
} public static void clearCurrDataSource() {
contextHolder.remove();
} public static boolean containsDataSource(String dataSourceKey) {
return dataSourceKeyList.contains(dataSourceKey);
}
}

So you can do it aspectJ Of aop In surround mode , Method starts by calling DataSourceSet Set the data source for key To achieve the purpose of switching data sources , Call the reset after the end of the method call key To switch back to the original data source ;

public class DataSourceAspect {
@Before("@annotation(ads)")
public void activateDataSource(JoinPoint point, ActivateDataSource ads) throws Throwable {
String keyDataSource = ads.value();
if (!process(keyDataSource, point))
return;
LogUtil.info("method:{} ", point.getSignature().getName());
DataSourceUtil.activateDataSource(keyDataSource, null);
} @After("@annotation(ads)")
public void resetDataSource(JoinPoint point, ActivateDataSource ads) {
String keyDataSource = ads.value();
if (!process(keyDataSource, point))
return;
LogUtil.info("method:{} ", point.getSignature().getName());
DataSourceUtil.resetDataSource(keyDataSource);
} private boolean process(String keyDataSource, JoinPoint point) {
if (keyDataSource == null) {
LogUtil.info(" The data source annotation has been identified , but value by null[{}]", point.getSignature().getName());
return false;
}
if (keyDataSource.equals(DataSourceRegister.MAINDATASOURCE)) return false;
return true;
}
}

And in the DataSourceUtil A series of actions during data source creation are encapsulated in ; Then you are likely to ask at this time , The application creates a data source at startup , What if you create a data source dynamically during the run time of the program , Now we can uncover this problem :

/**
* from bean Acquisition data source
*
* @param keyDataSource
* @return
*/
private static DataSource loadDataSource(String keyDataSource) {
if (dataSourceGetStrategy == null) {
synchronized (DataSourceUtil.class) {
if (dataSourceGetStrategy == null) {
if (!App.getContext().containsBeanDefinition(DATASOURCEGETSTRATEGY))
throw new CustomException(ResType.OverrideGetDataSourceInfo);
dataSourceGetStrategy = (DataSourceGetStrategy) App.getContext().getBean(DATASOURCEGETSTRATEGY);
}
}
}
return dataSourceGetStrategy.getDataSource(keyDataSource);
}

Then an abstract class is provided in the data source help class :

/**
* The abstract class must have subclasses that implement its abstract methods , Used for dynamic data source information acquisition
* <p>
* Created by zzq on 2017/6/19.
*/
public abstract class DataSourceGetStrategy {
public abstract javax.sql.DataSource getDataSource(String keyDataSource); @Bean(name = DataSourceUtil.DATASOURCEGETSTRATEGY)
public DataSourceGetStrategy getDataSourceReadStrategy() {
return this;
}
}

If you want to use the dynamic data source framework, you have to implement it getDataSource Method , So in this method, you can get the previously passed in datasourcekey, You can create your own data source , The following example creates an Ali druid data source :

/**
* Created by zzq on 2017/6/19.
*/
@Configuration
public class GetDataSource extends DataSourceGetStrategy {
@Autowired
private JdbcTemplateExtend jdbcTemplateExtend; @Override
public DataSource getDataSource(String keyDataSource) {
String sql = "select t1.url,t1.userName,t1.`password`,t2.driverClassName from " +
"td_bi_datasource t1 inner join td_bi_datasourcetype t2 on " +
"t1.dataSourceType_id=t2.id where " +
"t1.`id`=? and t1.is_remove=0 AND t2.is_remove=0 and t1.organization_id=? and t2.organization_id=?"; String orgId = Identity.getOrganizationId(); List<DataSourceAndType> dataSourceInfoList = jdbcTemplateExtend.query(sql, new Object[]{keyDataSource, orgId, orgId}, DataSourceAndType.class);
DataSourceAndType dataSourceInfoEntity = null;
if (dataSourceInfoList.size() > 0)
dataSourceInfoEntity = dataSourceInfoList.get(0); if (dataSourceInfoEntity == null)
return null;
DruidDataSource datasource = new DruidDataSource(); datasource.setUrl(dataSourceInfoEntity.getUrl()); dbType.put(keyDataSource, dataSourceInfoEntity.getUrl()); datasource.setUsername(dataSourceInfoEntity.getUserName());
datasource.setPassword(dataSourceInfoEntity.getPassword());
datasource.setDriverClassName(dataSourceInfoEntity.getDriverClassName());
datasource.setMaxWait(13000);
return datasource;
}
}

3)  Code calls use

I believe that the way code is called will make more people feel more comfortable !

and aspect The same thing , The following code :

try {
DataSourceUtil.activateDataSource(dataSourceKey, dataSource);
// Do your own thing
} finally {
DataSourceUtil.resetDataSource(dataSourceKey);
}

The conventional way can be used try finally Handle , If you have a better way to use it ! The idea is to activate the data source in front of your code , Release the data source at the end of your code call .

PS:

① In the end, I want to stress that , Don't worry about the performance problems after creating data sources frequently , Because after a creation , When used many times DataSourceSet There will be records , Switch data sources directly , There's no performance cost ;

② If there is a temporary data source that does not want to be cached, use DataSourceUtil.activateDataSource(dataSourceKey, dataSource); Two parameter method overload , The second parameter can directly create its own data source object , After use , The framework also releases resources without reservation ;

③ SpringBoot In dynamic data sources Bean The name is :dataSource

      

 GenericBeanDefinition beanDefinition = new GenericBeanDefinition();
beanDefinition.setBeanClass(DataSource.class);
beanDefinition.setSynthetic(true);
MutablePropertyValues mpv = beanDefinition.getPropertyValues();
//spring The name of the contract is defaultTargetDataSource and targetDataSources
mpv.addPropertyValue("defaultTargetDataSource", defaultTargetDataSource);
mpv.addPropertyValue("targetDataSources", DataSourceSet.getTargetDataSourcesMap());
beanDefinitionRegistry.registerBeanDefinition("dataSource", beanDefinition);

Project address :https://github.com/qq472708969/dynamicDataSource  !

spring boot More articles on dynamic data source solutions

  1. (43). Spring Boot Dynamic data sources ( Automatic switching of multiple data sources )【 Learn from scratch Spring Boot】

    In the last article, we introduced multiple data sources , But we will find that in practice, we rarely directly obtain the data source object for operation , What we often use is jdbcTemplate Or is it jpa To operate the database . In this section, we will introduce how to dynamically switch multiple data sources . add ...

  2. 43. Spring Boot Dynamic data sources ( Automatic switching of multiple data sources )【 Learn from scratch Spring Boot】

    [ video & Communication platform ] àSpringBoot video http://study.163.com/course/introduction.htm?courseId=1004329008&utm ...

  3. SaaS System architecture ,Spring Boot Dynamic data source implementation !

    At this time, I'm preparing to do a set from scratch SaaS System , Previous experience is to develop a single database system, and have not been exposed to SaaS System , So I had a headache when I received this task , But the way is more difficult than the way , A rare opportunity . I found a lot on the Internet about SaaS Information ...

  4. Spring Boot Dynamic data sources ( Multiple data sources can switch by themselves )

    This paper implements a case scenario : A system needs to read and manage data from its main database . Another part of the business involves many other databases , It is required to be able to flexibly specify the database to be operated in any method . In order to use it in the simplest way in development , this paper ...

  5. Spring Boot Dynamic data sources (Spring Annotate data sources )

    This paper implements a case scenario : A system needs to read and manage data from its main database , There is also a part of the business involved in a number of other databases , It is required that the specific database to be operated can be flexibly specified in any method . In order to use it in the simplest way in development , This article is based on ...

  6. Spring Boot Dynamic data sources ( Automatic switching of multiple data sources )

    This paper implements a case scenario : A system needs to read and manage data from its main database , There is also a part of the business involved in a number of other databases , It is required that the specific database to be operated can be flexibly specified in any method . In order to use it in the simplest way in development , The basis of this paper ...

  7. 22. Spring Boot Dynamic data sources ( Automatic switching of multiple data sources )

    from :https://blog.csdn.net/catoop/article/details/50575038

  8. Spring Implement dynamic data sources , Support dynamic join 、 Delete and set weights and read write separation

    As the project grows bigger , When the number of visits is gradually increasing . It is inevitable to use multiple data sources and set the read-write separation . Let's explain before we start , Because most projects use Spring, So here are some operations that may depend on Spring. In projects I've been through ...

  9. Spring Boot Multi data source configuration ( Two )MongoDB

    stay Spring Boot Multi data source configuration ( One )durid.mysql.jpa It's been said in the integration Spring Boot How to configure mysql Multiple data sources . This article talks about Spring Boot How to configure mongoDB many ...

Random recommendation

  1. Binary Tree Level Order Traversal

    Given a binary tree, return the level order traversal of its nodes' values. (ie, from left to right, ...

  2. 271. Encode and Decode Strings

    subject : Design an algorithm to encode a list of strings to a string. The encoded string is then sent ove ...

  3. use KNN Algorithm classification CIFAR-10 Picture data

    KNN classification CIFAR-10, And do Cross Validation,CIDAR-10 The database data is as follows : knn.py : The main test process from cs231n.data_utils import lo ...

  4. Android WindowManager Use

    often , We see floating windows that can be moved on the desktop , There are a lot of such scenes , Like traffic statistics , Desktop lyrics, etc , Security software cleaning widget This kind of small part is mainly through WindowManager ; WindowManager.Layout ...

  5. Thinkphp excel Import and export

    It's very useful to save it 1. Go to PHPexcel Download the latest program from the official website *   The plane is here I use it 1.78 Put it in vender Inside stay   function.php Write two ways The path, of course, is this *Commo ...

  6. ti8168 eth0 start-up

    ti8168 There's no network in the original file system eth0 Interface , In order to have this interface, you need to configure /etc/network/interfaces file Detailed configuration, such as the following ( Red to configure ) # /etc/network/inter ...

  7. [ turn ]C# call C++dll

    This article is reprinted to http://www.cnblogs.com/ysharp/archive/2012/05/25/2517803.html In cooperative development ,C# You often need to call C++DLL, When passing parameters, we often encounter the question ...

  8. thymeleaf It is said that js Of onclick Incident

    html: <img th:onclick="'javascript:imgClick(\''+${card.id}+'\',\''+${card.name}+'\');'" ...

  9. ( turn )spring Planning tasks ,springMvc Planning tasks ,Spring@Scheduled,spring Timing task

    One . Plan task implementation class 1. use @Component The annotation identifies the scheduled task class , such spring It can scan automatically 2. Use annotations in methods to identify the method to be executed :@Scheduled(cron="*/30 * * * ...

  10. JVM Tuning some related content

    JVM Tuning Tools Jconsole,jProfile,VisualVM Jconsole : jdk Bring their own , Simple function , But it can be used when the system has a certain load . There is a very detailed tracking of garbage collection algorithm . See here for details ...