Presented by DotData
The notion of using data to predict future outcomes is far from new. Even highly technical products that performed “predictive analytics” analysis have already been available to enterprise organizations for many years. The notion of developing and deploying custom-built predictive solutions, however, have, for the most part, been the exclusive domain of Fortune 500 companies.
The rarity of predictive analytics in the enterprise is mostly due to the technical complexity needed to create, train, and deploy the complex AI and Machine Learning (ML) models required to successfully develop predictive solutions. Over the past few years, the world of AI and ML development has seen rapid change. One of the most critical areas of progress has been the automation of the training of ML models.
The advent of “AutoML” platforms has allowed data science teams to accelerate the testing and training of ML algorithms and accelerate the development of predictive algorithms. However, the problem is that all AutoML platforms to date have still focused on data scientists as the key constituency.
While this has allowed large enterprises to accelerate the development of predictive solutions, it has remained outside of the reach of midsized organizations with a deep BI infrastructure — but without the data science resources of their larger brethren. In the past year, however, the advancement of AutoML has moved past the selection and optimization of ML algorithms with the advent of new platforms designed to automate 100% of the AI/ML development lifecycle.
“AutoML 2.0” platforms are now available with the specific goal of making AI/ML development easier for BI teams without data science expertise. The core functionality of these platforms is in three critical areas: First, making it easier for BI teams to leverage available data sets for AI/ML. Second, to make the process of “prepping” data for AI/ML algorithms less manual and finally, to automate the hardest part of the AI/ML development lifecycle — Feature Engineering.
The first challenge for any BI team in developing AI/ML models rests on the nature of the data itself. While BI data has (typically) already undergone a significant amount of cleansing, transformation, and normalization, it still requires transformation necessary to make it ready for AI/ML development — aka data preparation. AutoML 2.0 platforms automate the steps necessary to prepare the data itself to make it “AI friendly.”
The second challenge is in how data must be structured to develop AI/ML models. While the vast majority of BI data resides in relational data repositories, AI/ML models require data to be in a flat-file format. Traditionally, one of the most challenging and time-consuming parts of the AI/ML development process has been building these flat-file tables. AutoML 2.0 platforms eliminate this problem by automatically connecting to relational data sets and creating the necessary flat-files “on the fly.”
Finally, the process of “feature engineering” is the last — and often most complex — task for BI teams. FE is the process of applying domain knowledge to extract analytical representations from raw data, making it ready for machine learning. It involves the application of business knowledge, mathematics, and statistics to transform data into a format that can be directly consumed by machine learning models. In a traditional data science process, the development of “features” is a very iterative and time-consuming “trial and error” process that requires data scientists and subject-matter experts.
The development of features involves creating a hypothesis — for each feature — developing the feature and then validating it. This repetitive and iterative process is time-consuming, resource-intensive, and expensive. Once again, AutoML 2.0 platforms are often able to automate this process giving BI teams the ability to build AI/ML models without having to become data scientists.
The world of BI and AI are converging. The rapid advancement of AutoML 2.0 platforms is enabling an entirely new set of platforms and tools that take the complexity and manual steps out of AI/ML development. Armed with AutoML 2.0, BI teams in even small and mid-sized organizations will be able to build use-case driven AI/ML models to transform how businesses respond to changing economic conditions, market dynamics, and business stressors.
Ryohei Fujimaki is CEO at dotData.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. Content produced by our editorial team is never influenced by advertisers or sponsors in any way. For more information, contact firstname.lastname@example.org.