Digital Energy Journal- Data mesh at Equinor

Join | Existing Member Login

You are Home » News » View Article

Data mesh at Equinor

Monday, June 20, 2022

Equinor is finding the 'data mesh' concept very helpful in its digital transformation. Sun Maria Lehmann and Jorn Olmein explained how they use it

Equinor has been following the ideas of the 'mesh' architecture in how it develops its use of digital technologies in the organisation, said Sun Maria Lehmann, Leading Engineer, Enterprise Data Management and Architecture in Equinor. She was speaking at a webinar organised by the Society of Professional Data Managers (SPDM) in November.

This idea was originally conceived by Zhamak Dehghani of consultancy Thoughtworks in a 2019 paper (which you can download at https://martinfowler.com/articles/data-monolith-to-mesh.html )

The model can be described as 'distributed accountability,' with different departments and offices responsible for their digital systems, but with corporate wide oversight to make sure it all works together, she said.

Each department becomes a node on the mesh, and treats its data outputs to other departments as a 'product', where it takes responsibility for delivery and customises it to what its 'customer', whoever is consuming the data, needs.

It can be described as a 'federated model' of digital technology. Just like a federation of states, it means a union of partially self-governing entities, with some overriding authority.

The data products provided by any mesh node can be further split into 'portfolios', which makes them more manageable. Examples of data domains can be seismic, well, production volumes, environmental, renewables.

Data product principles

Equinor has established the basic principle that data is an asset and should be safeguarded, and also made available to others within the company, and sometimes externally, she said.

This principle has led to data sharing initiatives, data standardisation efforts, and Equinor's contribution to initiatives like OSDU, she said.

In 2020 Equinor's corporate executive committee approved the idea that the company should have a 'data product focus', where individual departments provide data in a suitable format for others to use as a core 'product' of their work.

This also means working together with the data 'consumers' in other departments, to find out exactly what data they need and what their priorities are.

Ms Maria Lehmann uses the analogy of a pancake restaurant, to show people making data how they should think about it.

Just as the management of a pancake restaurant would think about what kind of pancakes to make, who they are for, what suppliers they should use, people who provide data as a product should think about what data they provide, who it is for, and how they supply it, she said.

For example, a pancake restaurant may decide to serve large families. In the same way, a 'data product' could be orientated around meeting the needs of many individuals in the company, rather than what one person needs.

Equinor aims to be 'domain driven', which in this context means recognising that people who use the data have the best idea about how they need it.

To try to develop good data products quickly, Equinor encourages the use of techniques like personas [imagining specific people who would use it], use cases [thinking through how someone uses it] and value stream mapping [working out how the data is useful]. Rapid iterations and quick deployments are encouraged.

Continuing the pancake shop analogy, this is like imagining what sort of people would want to eat your pancakes, when and why, and why they would prefer your pancake shop to another one.

Sometimes you are reliant on others to make your system working, such as to provide good input data. The pancake shop analogy is being reliant on certain suppliers, she said.

Key principles about data products are that they should be managed, owned, and valuable, she said, with known quality and rights to use them. To continue the pancake analogy, if you make pancakes, you should have managed processes, and re-usable recipes, owned by your pancake restaurant.

Consumption patterns

It is important to try to understand how data is 'consumed', added Jorn Olmheim, Enterprise Data Architect, Equinor. For example, data can be consumed by individuals in the business functions of the company, or by the enterprise software. It can be consumed by new digital projects, including analytics, machine learning experiments or other jobs.

'We try to gather as much as possible, in order to help us, or guide the business, to where they need focus when they develop their data products,' he said. 'We try to use these consumption patterns both to find the right data products and to decide on the priorities of which data products to start with.'

Then you can work out what kind of integration method would work - for example can it be done just with a simple API interface.

Working with mesh

There are four 'pillars' to how Equinor implements the mesh idea, he said. Working with data in the right way, a 'federated' system made up of individual units with some internal autonomy, which are 'domain driven', so working around their own business needs. Data is stored on Equinor's own 'Omnia' cloud platform, and there is 'product thinking' throughout, in how data is provided to others.

'It is important to look at this as much more than data,' he said.

The data product also includes consideration of how data is ingested into each mesh 'node', how it is transformed or updated, and any infrastructure which is needed, such as data pipelines, storage and databases.

According to the Thoughtworks concept of mesh, there is an overall governance layer for the whole company, ensuring the various nodes can interoperate, such as by setting standards, definitions, or master data management.

There is also a platform for the whole company, upon which people can build services.

Between the platform and the governance layer, is a layer where domains are responsible for building their data products, using the tools provided by the platform, and according to the standards set by the governance layer.

'It is important the business [units] take responsibility for their data, it really drives this development forward.'

Departments within Equinor do not have the same levels of complexity in their data. For example, renewables have a lot less requirements than oil and gas subsurface, he said.

The 'mesh' evolves as the data products from the various departments work together, and build on each other's data, he said.

The whole idea is very abstract. 'It is important to have some concrete examples. We're starting to get those now. The best way to explain it to people is to show them,' he said.

'We have to look at delivering data to the end user as a software development product. Once you think about it that way, it's easier to think about what this about.'

Perhaps surprisingly, the hardest people to convince to work with data in this way are old school data managers. 'They have already worked in their ways for a long time,' he said.

Equinor and OSDU

Equinor has been involved in OSDU since it started, he said.

'The problem is, its quite limited in what it can do and what it can store,' he said. 'Once OSDU gets more mature and ready for more data types we will consider that.'

'We will never come to a place where we can get all our Equinor data in OSDU. For example, financial data will never be part of OSDU. Also co-called 'industry 4.0 data' from sensors in facilities.'

'It's going to be interesting how we fit data products in OSDU with data products not in OSDU.'

Associated Companies
» Equinor

CREATE A FREE MEMBERSHIP

To attend our free events, receive our newsletter, and receive the free colour Digital Energy Journal.