How Intelligent is your Machine Learning Application?

February 1, 2018

How Intelligent is your Machine Learning Application?

There is a tremendous amount of excitement and hype around machine learning. Machine Learning (ML) and Artificial Intelligence (AI) are often trumpeted as cure-alls that can help businesses anticipate and prepare for the future. In reality, we are just beginning to see these technologies evolve into commercially viable offerings. The real value of machine learning and other advance analytic technologies is how they are embedded within business applications to make them more powerful. By embedding ML and AI within applications, the complexities of the technology are abstracted and users do not need to understand the underlying models and algorithms.

Throughout the last year, I have had briefings and conversations with dozens of vendors claiming to have solutions based on machine learning technology. The truth is, I’ve seen this act before — vendors are eager to claim that their products include the latest hot technology — even if they don’t.  This isn’t a new phenomenon. A few years ago almost every data management company I met with claimed to offer predictive analytics. As I watched these demos it became clear that there was nothing more than a collection of bar charts and reports.

Even if an application uses machine learning algorithms it’s probably not what you think

There are two problems with many of the tools and solutions on the market. Offerings are either based on programmatic rules-based models or use offline learning models.

Rules can be extremely powerful in helping organizations codify consistent and predictable business policy.  However, implementing rules based systems are not the same as machine learning tools. Many so-called machine learning applications that I have seen are actually rules-based, pre-programmed tools that cannot adapt without manually changing the rules.  There is a place for rules-based but it is too easy to conflate rules with machine learning. These rules-based applications can be quite complex; however, they rely on pre-programmed if-then rules. For example, you could have a rules-based medical diagnostic application where users enter symptoms and the system suggests several diagnoses.

In addition to rules-based applications, the majority of ML-based applications use offline machine learning models. An offline learning model is derived from a machine learning algorithm but the underlying model never adapts once it is trained and deployed. These offline machine learning models can be extremely useful when the data changes slowly. However, this approach has limitations. To be a true machine learning based system I believe that the underlying models must adapt as new data is fed into the system. But in reality it is not a simple process. In production, it is not easy to analyze data and automatically refine models in real time while the application is in use.

Take the example of a medical imaging device that includes embedded machine learning algorithms and models. You may think that the results will improve as it collects more image data. Instead, the underlying model included a pre-trained dataset.  The system was not developed to iterate the model based on the new data.  Training had been done using an offline ML algorithm to train a model that was then embedded in the software. In order to be effective, the system would have to be designed to continually train based on new data.

Machine learning that will drive success – applications that can adapt

One of the reasons that I see problems with many of the tools I have evaluated is that the market is immature. But the market is changing. I am beginning to see emerging solutions that are based on online learning models. These applications are pre-trained, but unlike offline models they adapt and improve as they process new data. The models go through an iterative process as data is ingested – new associations between data points are made. Due to the complexity and volume of the data, new patterns and associations could have easily been overlooked by human observation.

Let’s look at a retailer’s e-commerce recommendation application. The application provides customers with suggestions on items they might like. The suggestions are based on data about the particular customer as well as aggregate customer data, product data, and independent reviews. However, as we all know, customer trends can change quickly, especially in the retail space. Often times these changes begin with subtle behavioral differences. If the recommendation application’s underlying models remain static, it is likely that the suggestions will become less and less relevant. On the other hand, if the application can “learn” as it is exposed to new data, the suggestions will actually become more accurate and the company is likely to see an increase to its cross and up-selling. By designing an application based on online learning models, the application will truly become strategic. In addition, because the models are iteratively improved, business users do not need to understand the underlying ML algorithms and models.


Applications that use machine learning have the potential to change how companies compete. Supply chain management, IT security, strategic planning, sales, and marketing are just a few of the areas where new, machine-learning applications will change the way companies do business. However, businesses must ask the right questions when vendors approach them saying they have and cool new machine learning based application. When you are sitting down with a vendor or meeting with one at an industry conference, below are some of the questions you’ll want to ask:

1. Does the application use online or offline ML models? Depending on the answer, you want to understand what problems this approach solves.

2. What parts of the application use machine learning versus traditional programmatic and rules-based decision making? Again, if the problem is related to codifying consistent and predictable rules, a programmatic model may be the best answer.

3. How will the application adapt when it goes into production? Will my results improve as more data is ingested? How does training take place and how long does it take?

Dan Kirsch , , , , , , ,
About Dan Kirsch

Dan’s research focus is on how compliance, governance, security and privacy are impacting the software industry and customer requirements. Additionally Dan is looking at mobile market.

Leave a Reply

Your email address will not be published.