Big data need not mean high costs and lengthy training cycles

These days it means more to understand business processes and to transform them into data-driven opportunities – using cloud technology when needed

For a lot, but certainly not all enterprises, a typical big data software is Apache Hadoop. Hadoop is the new legacy standard in enterprises when it comes to on-premise tools. Cloud-based big data is still fairly new for enterprises and cloud providers, as the technology has only risen up to become cloud-ready in the last two years or so. Popular tools here are Apache Kafka, Apache Spark or Apache Flink – these are all run best on bare metal servers.

Different strokes for different folks

The pro for Hadoop is the nearly unlimited scalability with an affordable price for hardware. A negative is, however, is high operational cost. Although this can be negated by using commercial distributions for an additional price.

The same counts for Kafka, Spark or Flink. That’s why modern tech giants always operate the largest installations in their own data centres, but enterprises often struggle to attract talent to get that scale into stable productions. In short − it’s great to have the tools on premise, but to operate them can be challenging.

E.ON were confronted with exactly this problem. When I joined, a large Hadoop cluster was running, but on IaaS infrastructures in a public cloud. That had massive implications on stability and performance, and it was extremely costly to let the system breathe. Don’t get me wrong, when I started at E.ON, more than two years ago, the team had already two years of operational experience with Hadoop. That shows how fast an enterprise needs to adopt new big data technologies to get an advantage against competitors.

Given that E.ON’s business units have a growing appetite, we decided to use public cloud big data rather than operate big data tools ourselves. That was the initial departure from traditional infrastructure into cloud technology, using native tools instead of trying to reinvent the wheel for the 10th time. That point in time was an enormous boost to our transformation. We are now able to bring products to customers quicker than ever before.

From new terrain to ROI

Our key ambition is to oversee a fully digitized grid. Having information about the energy infrastructure at any time is imperative to creating transition in the energy industry. To achieve this, we must decarbonize our environment and make it possible to be informed about the production, transport and use of energy over its entire lifecycle.

Our digital strategy and transformation are an important part of our overall product development and lifecycle as you can see from a yearly sustainability report. To name a few our digital solutions are: future energy home management; energy monitoring for municipalities; building energy management; eMobility, and quartermaster Solutions, which are all based on cutting edge technology that we’re using and developing.

Data-driven opportunities

Big data does not mean heavy operation costs and long-running training cycles. Nowadays it means more to understand business processes and to transform them into data-driven opportunities, using cloud technology when needed to develop your own solutions.

But using data has strong ethical consequences. I’m not a fan of storing every bit of personalised data just to have it. Clearly divide between machine and robotics data from assets, circuits, transformers, windmills, solar farms, engines, cars and so on and personal data from customer solutions to build different approaches for data analysis. Personal data belongs to the individual, and it needs to get it processed there − edge computing brings a lot of advantages to achieve this. Always stay upfront, improve and drive the digital change.