To tackle climate change, decarbonization of energy production is believed to be a key initiative to be pushed in the coming decades. However, decarbonizing energy will add millions of distributed ‘micro-assets’ such as solar panels and batteries into grids that were originally designed to support a few centrally located power plants.
With the increased effort to predict supply from renewable energy sources and a sharp increase in the number of data-producing assets across the grid, ensuring stable operation will become significantly more complex. That’s because these assets are controlled, owned and operated by multiple entities, limiting grid providers’ operational control.
Grid operators and other stakeholders will need to capture, standardize and share data to make informed decisions about everything from demand planning to intraday energy trading.
Data sharing is nothing new to the energy sector; the interdependencies of cross-border grid networks, such as Europe’s ENTSO-E grid, already make sharing data between organizations an operational necessity. But data sharing will become even more important as energy generation and distribution evolve from centralized, plannable fossil fuel sources to decentralized and flexible renewable sources. The additional complexity creates a need for significantly increased sharing up and down the whole grid, from the transmission system operator (TSO) via the distribution system operator (DSO) down to individual assets.
To achieve their sustainability goals, ensure supply stability and operate profitably, organizations will need to share far larger volumes of high-frequency data with near real-time freshness. But to do that, organizations throughout the energy sector must overcome some fundamental challenges.
Data sharing challenges in the energy sector
Securing critical national infrastructure
The systems that support energy supply need the highest levels of reliability, security, resilience and stability — if the lights go out, lives could be at risk. While organizations responsible for these systems need to modernize their data infrastructures to meet new data-sharing demands, modernization initiatives cannot compromise security, supply stability or service quality.
Modern, cloud-based systems must also account for the data sovereignty needs of critical national infrastructure. In many public cloud providers, the ‘control plane’ (the part that controls the deployment and operations behind the local data centers) resides in a separate territory from the data centers providing the cloud service.
Trust
Companies across the sector must share fine-grained, high-frequency data to meet the complex demands of the net zero grid of the future. They also need to remain competitive and maintain rigorous security, so data-sharing platforms and mechanisms must be trustworthy, protecting commercially sensitive information and data that carries a potential national security risk.
Interoperability
Data exists in numerous formats and means different things to different organizations. Organizations often spend valuable resources on cleansing and standardizing shared data.
There are established ways to manage this for the current levels of regular operational data exchange, such as using the paradigm that sharing data means copying a file from A to B. However, as the volume and frequency of data sharing grows, the sector will need to ensure all parties can derive insight and value from the exchange of information. This means the sector will need to align on protocols, contracts and schemas for data exchange where they’re not already in place.
Adaptability
As the grid evolves to offer the flexibility required to generate, store and consume energy from distributed renewables, the IT and OT systems that support it must keep pace. Existing monolithic architectures don’t offer the flexibility to adapt quickly to changing conditions and evolve at speed to meet new demands — and replacing them comes at an immense cost of change.
Four ways energy companies can meet new data sharing demands
These are significant challenges, but our experience helping organizations modernize their capabilities and ways of working has shown that they can be overcome. Although the energy sector has unique needs for balancing flexibility and adaptability with stability and reliability, it’s still possible to apply modern infrastructure and engineering principles to solve the data sharing conundrum.
1. Architect for adaptability and stability
A key principle for creating a modern, evolutionary architecture is pragmatic simplicity. All architectures are a trade-off, but the lower the complexity, the easier it is to deliver stable operations and resilient systems.
Architectural fitness function tests are a useful way to assess and prioritize areas for improvement to meet new data-sharing needs. As organizations break down monolithic architectures into microservices and expand CI/CD automation, contract testing can help establish that systems are talking to each other effectively.
To address sovereignty, one option to explore is to challenge the paradigm of a preference to move to the cloud and instead build cloud-like capabilities into on-premises data centers. In this way, organizations can retain control over the digital infrastructure and data that support grid operations and decision-making — while delivering the flexibility to quickly adapt and continuously evolve systems to meet changing demands.
In a recent webinar, we heard how Elia Group, a transmission system operator in Belgium and Germany, is modernizing its IT infrastructure to bring cloud capabilities to grid operations. Andreas Reisenauer, Elia Group’s Head of Infrastructure, explained how crucial this initiative is for the group’s sustainability goals — to dive into all the details of the project, you can now watch the webinar on demand.
2. Manage data as a product to increase its quality, utility and trust
Product thinking, a proven approach that brings together elements of design thinking, agile development, lean methodologies and more, enables organizations to treat data as a product — and drive continuous improvement of data quality.
Managing data as a product is one of the key tenets of data mesh. This approach supports interoperability by enabling multiple teams or organizations to easily utilize data and create data products that can be shared securely through role-based or organization-based exit nodes.
3. Create data spaces to enable trusted sharing
Many industries are beginning to create data spaces to enable the trusted sharing of live, standardized data, providing an appropriate balance of data privacy and utility. There are several options for building such data spaces, including:
Point-to-multipoint sharing, where each individual entity shares with others
Sharing through third-party data trustees who serve as the data exchange platform
Purpose-built platforms that serve as exchanges for data, overseen by government-appointed referees
There’s already some encouraging movement in this area across the energy sector, with the EU’s creation of the Common European Energy Data Space and the UK developing an MVP for a ‘digital spine’ for energy data sharing.
4. Remember that data is only one part of the story
Effective data sharing is about more than just data. The legacy business applications that support data exchange must be refactored and re-engineered to meet new requirements. The underlying computational platform must evolve to become more flexible and adaptable, and provide platform services for applications and data products. And communications networks and control platforms must also be modernized.
It’s a complex challenge that needs to be tackled at every layer. That’s not easy when the teams responsible for different layers operate in silos, so there’s also work to be done to evolve organizational cultures and ways of working.