We talked about three directions of data development technology ：1） Stream batch integration has become the mainstream development mode ,2） Code automation technology is maturing ,3）OLAP Cubes Will eventually decline . For data development practitioners , In the development of Technology , How to maintain personal competitiveness , I think the most important are the following three abilities .
Only a deep understanding of the business , To really know what stage the current business is in , What's the problem , What are the key objectives . Corresponding to enterprise data construction , Be sure to solve “ Why? ” The problem of , What is the current business status of data warehouse service , To solve business problems , What goals are expected to be achieved , These can't be solved by technology automation . Then there is the model design 、 Implementation landing .
The data will be used to build analysis reports , Service data products , It seems that after the data is generated , It has nothing to do with data development practitioners , So that many practitioners laugh at themselves “ Human flesh SQL machine ”, yes “ Data movers ”, It is also often called by partners “ETL The engineer ”. The ability to go deep into data refers to the ability to go beyond production data , Can continue to think about what can be obtained from these data , Whether through mathematical statistics or machine learning , Explore whether insights can be tapped to drive business growth , And operational guidelines , Is to do “ Data diggers ”.
The overall view of the data link is not only to know what the whole data architecture looks like , Be familiar with how data flows , It can also do global optimization of data links . Such as the stability guarantee of the whole data link , Design of organization and management mechanism of data assets , Full link value evaluation of data 、 Cost management , Data quality management and testing 、 Construction of monitoring mechanism, etc .