Automation – the bedrock of creating value

Businesses will find it hard to fully leverage the potential of data and its related technologies without robust data automation, writes Rob Mellor, VP and GM of WhereScape EMEA

According to the 2019 Gartner fourth annual Chief Data Officer (CDO) Survey, the implementation of a data and analytics strategy was ranked the third most-critical success factor when it comes to a CDO’s organisation.

When it comes to data, we’re all aware of the four ‘Vs’ – variety, velocity, veracity and volume – yet for many organisations, their data warehousing infrastructure is no longer equipped to handle them. Additionally, value, the fifth ‘V’, is even more elusive. So, taking into account the scale of data that many modern companies have means that meeting these challenges requires a new approach – with automation being the foundation.

45 percent of a CDO’s time is spent looking at methods of using data for value creation and revenue generation. This means being able to harness data in a way that is realistic, practical, and actually beneficial. The data warehouse can help meet these expectations, providing enterprise data with a centralised space that business users, the CDO included, can use to develop insights.

For the CDO to succeed in monetising data within the organisation, then, creating a successful data warehouse is crucial.

The traditional waterfall approach to data warehousing that was first introduced in the 1970s, however, only delivers a mere fraction of the value it potentially has to offer.

The approach needs to evolve to address new data sources and adapt to business demands – essentially becoming more responsive as organisational needs change. Using automation software to design, develop, deploy, and operate data warehouses provides a wide-ranging value to business leaders. This change gives flexibility when business needs demand it, and incorporates new data sources and technologies more easily.

What can the CDO do?

The data warehouse is invaluable for providing business users with the information they need, being the central storage point for enterprise data. Yet the gap between user expectations and the data warehouse’s ability to provide up-to-date, consumable data in a timely manner has grown, motivated by users becoming more aware of the potential benefits of data-driven decision making.

Businesses both want and need insights from data a lot faster than before. Additionally, the ever-rising growth of new forms of data intensifies this business need, particularly when it comes to semi- or un-structured information such as client communications, real-time messages, sensor data, social media, and audio/video files. 

Customarily, data warehouse development and evolution meant long-cycle IT projects, which contrasted heavily with the needs of more agile project design and build environments. To support digital transformation efforts, CDOs should take the lead in the re-architecting of data warehouses, from creativity to acceleration and automation, in order to increase the business’ time-to-value ratio.

CDOs can help their organisation achieve this by following these five steps

Know what the desired outcome should be

CDOs need to understand the specific challenges business teams face that data could help out with, before making any type of decision as to the future of the data warehouse infrastructure. Fundamentally, a data warehouse automation and modernisation program needs to be built around assisting decision-making, leading to differentiation in the market place.

A recent TDWI survey suggested that the top reason for data warehouse modernisation is the realignment to business objects. The CDO can help chart the course for how business goals and technology meet by enabling collaboration between business and IT teams. This will, in turn, lead to overall business transformation, enhanced by the new data warehouse’s approach to data driven decisions.

Know what you already have

Sophisticated data management tools are already deployed as part of most organisations’ data infrastructure, however these may not be working to the best of their abilities. Companies that are already using Oracle, SQL Server or Teradata have a range of data management and movement tools already within their IT real estate. These can be automated and leveraged more effectively as part of a data warehouse automation push.

Nevertheless, throughout the inventorying process, CDOs should be ensuring they have thought about the capacity requirements of their data warehouse. It’s no secret that data is continually growing at an exponential pace, so even if the data warehouse is fit for purpose today, it’s important to ensure the automation processes, general infrastructure and storage requirements are of a speed and standard that is capable of handling this in the future too.

Additionally, it’s important that data warehouse automation integrates with the business as it currently is, and will authentically be in the future, rather than as the business believes it may be in an ideal world. CDOs need to encourage their teams to understand the value of the data available, along with the automated analytics and evaluation processes which can be used to meet precise business priorities. In order to support this, it’s essential to design the data warehouse automation strategy for not just an ideal set up of data, but for the realistic unpredictability of the business data landscape. Data modeling approaches such as Data Vault 2.0 can be automated to provide even more flexibility to organizations to easily address change.

Ensure automation is efficient

As with any other large-scale transformation project, data warehouse automation requires resources. However, these are often scarce due to strict budgets and competing priorities. So, CDOs need to think about what should actually be automated in order to free up future man hours. Hand-coding SQL, writing scripts or manually managing metadata are all examples of how automating tasks can be more cost-effective. All of these systematic processes can either eliminate the need for human involvement (and, indeed, human error) or dramatically speed up the process.

Always be open to change

Data warehouse automation and modernisation should be seen as an avenue of constant, on-going development. CDOs should be able to re-strategize different parts of the business’ infrastructure to match business needs and any new data sources that may emerge. CDOs should also look to take a staged approach to the initial automation and modernisation process to minimize disruption and ease the transition for business users, setting out a schedule of when each different requirement should be met. Additionally, due to ever-changing business needs and new technologies being used, post-production change is inevitable and has to be planned for.

CDOs should also make sure to prepare for the human change that automation will create. In business teams, users can be re-deployed to increase efforts on analysing business intelligence and in turn converge the insights into business value. Elsewhere, in IT teams, automation gives a new scope to plan for the future by looking at new analytics tools, or planning for better and smarter ways to deliver on business priorities in the future.

Enabling a data warehouse automation mindset

Data warehouse automation is more than just software you purchase, it’s also a culture you implement into the business. Yes, tools and technologies form the basis of the process, but a good data warehouse strategy needs a transparent process, strong leadership, and an unwavering focus on the business’ end goals in order to thrive.

Businesses will find it hard to fully leverage the potential of data and its related technologies without robust data automation. It’s important that the CDO takes responsibility when it comes to data-driven transformations, seeking out the best ways in which large-scale data usage can guide future business decisions, and ensuring that the fifth ‘V’ – value – is always taken into consideration.