Hype cycles are interesting to me. These are the cycles as described by Gartner where new innovation undergoes predictable phases of inflated expectations, disillusionment, enlightenment and finally adoption. Big Data and machine learning have been undergoing a hype cycle for the past decade and organizations are finally figuring it out. Tools have matured and data science teams are learning how to perfect building predictive and classification models to inform better decisions. You can recognize when we’re nearing the end of a hype cycle by the explosion of vendors competing to deliver largely similar solutions designed to accelerate adoption of the technology.
Machine learning addresses predicting outcomes, but making decisions has been oddly under-addressed by industry innovators.
As with all hype cycles, when one matures, another is ready to take its place. There is a reason for this. The previous cycle made a lot of promises, but over time adopters understand the technology’s limitations and a new wave of innovation emerges to fill the gap. In the world of analytics, that transition is happening now. Machine learning has helped move us past the shortcomings of Business Intelligence. Organizations need to do more with data than understand trends and create dashboards, they need to accurately predict future outcomes and make informed decisions based on those insights. Machine learning addresses the first half of that statement, but the second half, making decisions, has been oddly under-addressed by industry innovators. This gap is beginning to be filled by a category of technology called Intelligent Applications, which I believe represents the next era of innovation in analytics.
We wouldn’t let a self-driving car crash randomly so its machine learning algorithms can learn how to drive.
Understandably, there is confusion between machine learning with AI. There is a surge of new companies who use machine learning and claim artificial intelligence. Machine learning is not AI. In fact, machine learning represents only a small (albeit important) part of AI. To illustrate, let’s look at a technology that is close to fulfilling the promise of AI: self-driving cars. Think about the technology that has to come together to drive a car and consider how much of that is machine learning vs. other forms of engineering. It’s true that machine learning is used to help classify images from cameras so algorithms can tell the difference between a car, a person or a leaf. Machine learning is likely also used to optimize control loops such as how much pressure on a brake is needed to stop a car in different weather conditions.
These are important, but they are such a small part of the self-driving solution. Logic such as understanding the rules of the road and the ability to follow GPS directions. Logic that knows to apply the brake when cameras see slowing traffic. This logic comes out of the brains of engineers and is coded as inference rules, decision trees and plain old software. Imagine creating a self-driving car using only machine learning algorithms. Would we let the car loose randomly, allowing it to crash so that the machine learning algorithms perform better next time? It wouldn’t work of course. Rules and logic are just as important, or perhaps more important, than machine learning in creating AI.
There is a difference between predicting the probability of default and applying logic needed to approve a loan in real-time.
So how does this translate to the business world? Our analogy illustrates how far we are from delivering AI in business applications. Teams are spending millions creating big-data architectures and machine learning capabilities, but they are addressing a small portion of the capabilities needed to achieve AI. For example, there is a tremendous difference between a machine learning algorithm that predicts the probability that a loan will default and an AI application that approves a loan and sets a price that maximizes value for the lender while delivering the best loan for the borrower in real-time. The second requires knowledge about cost of funds, potential recovery options, an understanding of the products a lender is offering and regulatory restrictions that may influence a credit offer. All of this together requires much more than machine learning, it requires an intelligent application.
What does this next generation of intelligent applications look like? For one thing, they will be better at capturing human intelligence in the form of data modeling, business rules and logic. Those of us who have been in the analytics industry for a while may think ‘business rules, that’s so 90’s’. Well perhaps, but that doesn’t make it wrong. A more modern concept is ‘ontologies’, defined as a representation of entities, ideas and events with interdependent properties and relationships. More simply, an ontology captures how data should be connected and how a system should react to new conditions. Ontologies form the basis for intelligent applications and our example of self-driving cars use them extensively: for example, how gas pedal pressure relates to vehicle acceleration relates to steering relates to car travel.
The concept of ontologies may seem abstract, but there is a familiar example of an ontological system that many of us use every day – the spreadsheet.
The concept of ontologies may seem abstract, but there is a familiar example of an ontological system that many of us use every day – the spreadsheet. Think about the information a financial analyst enters into a spreadsheet. Formulas for how revenue relates to cashflow relates to income and company performance. How conversion rates impact portfolio profitability and how new sales transactions impact inventory and tax obligations. These are ontologies. Those of you who follow my articles are not surprised that I would bring the conversation back to spreadsheets. I am fascinated that a significant portion of institutional knowledge at most companies still live in these spreadsheets and I feel they represent a tremendous opportunity for innovation.
Intelligent applications will be order of magnitude faster to develop than today’s software, resulting in an explosion of innovation.
We are starting to paint a picture for the technology that powers intelligent applications. The tools will be able to capture human intelligence in the form of ontologies and easily incorporate machine learning. They will be accessible to business experts, similar to a spreadsheet, but they will be much more powerful. They will run in the cloud so authors can collaborate as they create intelligent applications. They will support powerful data relationships to more easily model realistic data structures without requiring transformation. The language for creating ontologies will be declarative, similar to spreadsheet expressions, so that authors will not require coding skills. Most importantly, intelligent application platforms will break down the walls between subject matter experts, data scientists and developers so that all of these teams can work on the same ontology, the same data, the same business logic and the same analytics, at the same time. The pendulum will swing from a focus on technology to a focus on business expertise. As a result, creating intelligent applications will require an order of magnitude less time than today’s software development cycles, resulting in an explosion of innovation.
The less skeptical believe AI will free society from less fulfilling work and allow us to explore creative pursuits considered indulgent by prior generations.
Intelligent application platforms are just beginning to move from early innovation to wider adoption. Teams are realizing that they can build predictors and classifiers using machine learning, but they have not figured out how to automate the massive amount of expert knowledge needed to make smarter decisions. This understanding will finally break the decades old reliance on spreadsheets as business experts are forced to move their business logic into platforms that can be automated. When the business, scientists and technologists are all using the same tools, innovation will begin to accelerate exponentially and we will begin what will likely be a rapid journey towards true AI in business. While some are concerned that AI will ‘take our jobs’, the less skeptical believe AI will free society from less fulfilling work and allow us to explore innovative or creative pursuits considered indulgent by prior generations.