While producers of data platforms and services are in a period of incredible growth and change, the stakes are enormous, says Raghu Ramakrishnan, CTO for Data at Microsoft
Data analytics is an integral part of how organisations plan and execute, and it is rapidly becoming ever more entrenched. Organisations are becoming more data-centric – both in gathering data to base decisions on, and in becoming more data-driven in how they make decisions and execute on them. Despite the new opportunities opened up by this new data dawn, businesses need the right tools to navigate its landscape.
An increasingly complex landscape
Our ability to instrument virtually all important technical touchpoints and organisational interactions has grown dramatically in recent years. Whether it’s telemetry from devices and complex software, or logs from customer or user touch-points, we now have an unprecedented ability to observe, learn and improve how we operate.
This diversity of observational data, ranging from simple readings to video streams to documents and logs that capture complex interactions, has led to much of the data complexity we see today, and the volumes are enormous.
Along with the complexity of data, there is the challenge of mastering the plethora of tools available for managing and extracting value from this wide range of data. Finally, an organisation’s data estate is now widely dispersed across on-premise data stores, public and private clouds, and increasingly, edge devices.
Data governance
As organisations begin to recognise the value in their data, there’s also a growing realisation that data is a core asset that must be secured and that access must be governed appropriately.
In many domains, data is highly sensitive (e.g., health-care, finance, customer profiles) and subject to stringent regulations (e.g., GDPR) that must be complied with. This requires comprehensive approaches to identity, security, access control, governed data movement and sharing, and right-use policies, including tools to enforce and audit compliance.
Managing your data real-estate
This presents a challenge that data owners everywhere are grappling with, namely, what tools can we use to manage data in this increasingly complex environment? Frankly, we are in the early stages of data-driven decision making, let alone data-driven closed-loop actions.
The tools available today are either traditional relational data management and reporting tools or new tools that aim to make some kind of analysis or inference better. We are also beginning to see more tools that allow for a good handle on an organisation’s data real-estate, including smart catalogues, tracking data use, and policy-based governance – but we need to see these mature and cover the full gamut of today’s data complexity.
Businesses have a difficult transition in front of them. But the impact of a confluence of dramatic changes in the data environment will make it easier to bring organisational focus to the hard problem of revamping a firm’s approach to data.
The dramatic growth in data complexity, the emergence of public clouds with their elastic resource models, and the growing importance of responsible data use and governance are all huge shifts taken alone. Taken together, they represent a tectonic shift in the data landscape and have forced the issue in a positive way for society.
Dawn of new data
We live in interesting times for data. For producers of data platforms and services, this is a period of incredible growth and also change, but the stakes are enormous.
For owners of data, this is similarly a period of great opportunity and risk. For us as individuals, everything has changed. Our lives will for evermore be visible through the lens of data about us, and this will greatly improve many facets of our life, yet requires us to think anew about every aspect of what we hold private and what we consider to be fair grounds for decisions.