2021: A Data Odyssey

Is your company ready for the new era of AI and ML-empowered data management? It’s time to strap in, says TIBCO’s Bob Eve

Putting the advantages of AI and ML into the data-driven engine room of any business enables us to create a high-octane fuel mix, one capable of breaching the gravitational pull of the past and steering an organisation toward a new universe of machine-enabled efficiency and intelligence.

We humans love our devices, machines and technology tools in all their forms. But machines on their own are of little use without information in the form of data. So today, we are all gaining a deeper appreciation for the need to feed our applications with the right data through the most efficient and intelligent channels, if we actually want to get any purposeful performance out of the process.

Looking to the heavens, if we’re not quite on a mission to Mars, then we’re on a mission to better, more timely and more accurate data. We’re also on a mission to more business-friendly data that is more usable. These are the critical jet thrusters on our path to gaining greater business value from data all round, right across the galaxy.

HAL-logic: who’s steering this thing?

But given the complexity of the data navigation task in front of us and the importance of working with these data streams in an accurate and manageable way, we must also remember that steering this ship is too important and mission-critical a task to leave to humans alone.

In a universe of data complexity, the warp speed boosts we can gain from Artificial Intelligence (AI) and Machine Learning (ML) now represent the only prudent, intelligent and manageable route to being able to maximise data’s value. In 2021, data-driven leaders will move AI/ML to the core of their data management strategy, automating thousands of activities daily, while using human control and intervention only when they need to.

This is a bold new frontier in many ways. It is the coming together of traditional approaches to Data Management (DM) combined with a fused bond to some of the more esoteric techniques being applied to Content Management (CM). The resultant combination is a unified Information Management (IM) layer that sees organisations consolidate data and leave the gravitational pull that was previously holding them down.

As Forrester’s Michele Goetz and Noel Yuhanna have noted in their brief, Research Overview: Machine Learning For Information Platforms, “Machine learning is the mechanism that converts documented information management and governance logic to an executable service or job… Machine learning automates the manual administrative task to configure and optimise workloads across the ecosystem, based on understanding and interpreting demands from queries, analytic models, data pipelines, and API calls.”

Mission control: pre-flight checks

So there’s a complex journey ahead of us, but it can be made simpler if we inject AI and ML into our strategic planning operations from day zero. Our early stage processes will be focused on data discovery as we find out what data resources exist throughout the business and work to understand how they will impact our mission ahead.

We will need to corral, classify, clean and collate information into its various ontologies (state and form), taxonomies (labels and names) and hierarchies (interrelationships). It is from this point that we can start to see the advantages of shifting left away from the manual development of data models, because machine learning can perform this task much more quickly and at a far deeper granular level.

When we have a solid handle on the datasets that we know will be running in live production environments on the journey ahead, we can perform ever-deeper levels of granular data inspection using AI and ML driven metadata analysis. This is a continuous process to ensure we stay on the stable trajectory that we set out for the days (or Martian sols) to come.

Once we’re in orbit, past a couple of separation stages and thinking about getting through any troubling asteroid belts (no really, this analogy has plenty of legs yet! – stay with me), then we can think about engaging autopilot and allowing AI and ML to manage our data for a defined period of time. But even on autopilot, there’s an 80:20 Pareto principle split here that will still require human intervention to deal with exceptions and anomalies that our AI and ML systems have yet to learn.

Okay if it’s not 80:20, it could be 75:25, but in fact it’s more likely to be 90:10 or better, we don’t need to split hairs or atoms at this stage, we just need to make sure that there’s enough oxygen for the crew to keep functioning.

Deep space: avoiding black holes

Deeper into the mechanics of our engine room, we need to know what fuel mix the business is running on if we’re going to achieve high-octane performance. This means we need to know which data is delivered to which applications at runtime in which order and in what quantity and capacity. Our AI and ML layer should have enough intelligence to be able to make those decisions for us, but there is always scope for additional learning, some of which can be made on-the-fly.

System maintenance along the way for a smooth flight and the clean execution of business mean that we will take a continuously exacting approach to data preparation and cleansing, data ingestion, data integration and wider levels of data analytics management, tuning and optimisation. The process of data exploration and pattern detection is almost as limitless as the infinity of space itself.

The human users also need to think about the fact that they need to know when they should be ‘hands off’ and stand back to let the computer systems do what they need to do. After all, during space shuttle lift off, you don’t put your hands on the steering wheel or lean out of the window with a sextant to see where you’re going.

The crux of the matter here are the little pockets of opportunity to automate on the journey. We may not necessarily know the size, shape and location of all of those pockets at the start of any one journey, it is (to an extent) always a journey into the unknown.

Infinity & beyond (back down to Earth)

After we complete the first leg of our journey to the new era of AI and ML-empowered data management, we can come back down to Earth for a moment and realise what we have achieved. We can now use AI and ML to perform better data management from end-to-end. This will result in us being able to get our hands on better data and therefore achieve better data insights.

Being able to wrangle, analyse and manage our data through machine-driven intelligence allows us to create a network of data templates and structures that define best practice in whichever chosen industry vertical we find ourselves operating in.

In space travel terms, this means that all future missions should be easier. In data-driven business terms, this means that enterprises reach the point where they can drive new revenue streams by monetising data as they also optimise supply chains and customer experiences by sharing data with suppliers and channels.

If this is ‘just’ 2021: a data odyssey, then this is the immediate future. When we look to 2030 and beyond, then we can truly start to think about what happens in the next galaxy. The important point to remember is that we need AI and ML in the engine room, on the rocket launchers and spread throughout mission control from the start. Once we achieve these goals, then we’ll have clearance for takeoff.

After all, this is not rocket science, well… maybe just a bit.