Big Data in Oil & Gas: Implications for the E&P organization

Share this on your networks

By Emmanuel Udeh, Subsurface Data Analyst at Shell

There is an alarm ringing in the Oil & Gas industry. We need to pay immediate attention to the glaring need to implement a robust and value-based data management process for our information and data assets.

IDC Diagram

Source: IDC

The Exploration & Production (E&P) organization requires lots of information and data in order to make the right and accurate decision. Today’s technology has made it possible to obtain high quality information in real time, but this has also increased the associated problems required to properly manage these information and data assets.

Some common issues faced by E&P companies with regards to information and data management include:

  • Enterprise Data Management (DM) strategy – Many E&P companies lack a clear enterprise DM strategy. A common scenario in most cases is that DM strategies are driven by specific corporate needs or proposed by external consultants, thereby creating a situation whereby different sub-optimal solutions are implemented in different parts of the organization.
  •  Resources – There is a scarcity of competent E&P data managers who have a good understanding of DM challenges faced by the industry.
  • Data magnitude – The large and growing volume of expensive data continuously generated during E&P data acquisitions makes clear the business need for implementing robust DM strategies across board.
  •  Lack of standard data formats- E&P data most often are generated and stored in proprietary formats applicable to product vendors. This creates a big challenge with data sharing and integration.
  •  Lack of clear ownership – E&P engineers are usually more focused on their deliverables and less on data preservation and management. This means data issues fall through the gaps most of the time. E&P engineers need to analyze and own their data and the implementation of a clear enterprise DM strategy will make every individual accountable for their technical data, ensuring that storage and archival processes are not left in the hands of a few data management practitioners.

As they struggle to manage the issues mentioned above, most E&P organizations end up with too much information and little or no management processes to ensure proper harvesting of their data assets. Even where some data management practices are adopted, they may not be implemented fully, leading to security breaches or loss of the potential value that could be harnessed from the various datasets.

A number of large organizations have managed to mitigate some of these problems by implementing and embedding within their organizational framework processes, standards and principles which describe the governance, roles and responsibilities for the planning and implementation of DM activities. The underlying premise of these guiding principles seeks to highlight that accurate, secure, good quality information and data management is a key business driver.

Data control procedures outline the processes to enable effective exchange of data between different arms of the E&P organization, from Subsurface Development to Asset Management and Production. These procedures help to ensure that data flows smoothly from organization to organization within the business efficiently, and most importantly, securely.

All these processes, including the correct application of data governance policies and information risk management practices, will help to ensure that every organization’s information assets are protected against existing risks and threats. Data which forms the factual basis upon which all interpretations, conclusions and decisions are made, whether strategic or operational, should be guarded with the same emphasis placed on People, Processes and Technology.

Information risk management is the identification, assessment, and prioritization of risks followed by co-ordinated and economical application of resources to minimize, monitor, and control the probability and/or impact of unfortunate events or incidents. Efficient information risk management is only as strong as our weakest link. Therefore it is important that everyone in the organization understands the value of the organization’s information assets, along with the need to have quality and accurate data management systems in place.

Molten Group Data

Source: Molten Group

That is a major part of the problem, because very few people appreciate the value of data assets. The cost of acquisition is often not fully understood across the whole organization. According to a study by the Molten Group, upstream divisions of super majors spend in the range $1 billion – $3 billion per annum on the acquisition of data (excluding management and technology costs) and 70% of that data is in the technical data domain. In that same study, it was also discovered that on the average less than 1% of the value of an E&P organization’s data asset is invested on an operating cost basis per year to maintain it. This is not only inadequate but also creates an environment where data isn’t managed properly and, most importantly, securely.

A critical success factor for managing information and data assets is to have the right organizational structure and people in place with clear accountabilities. The technical data management discipline should be visibly and strongly embedded, with every DM staff member clearly aware of their roles and responsibilities

Managing E&P Information and Data (I&D) assets should not be left to the DM practitioners alone; this is supposed to be everybody’s responsibility from top executives to the lowest tiers within the organizational structure. Compliance should be actively pursued and rigorously enforced with regular audits to measure compliance. A CompuServe study showed that negligent insiders are the highest cause of data breaches within an organization.

The organization’s commitment to continuous data management improvement can only be secured if DM practitioners proactively ensure that the tangible benefits of properly securing and managing I&D assets is clearly visible to all important stakeholders. These benefits should be measured against associated expenses.

E&P organizations need to implement a secure and efficient I&D management system because it can drive business performance and bring such benefits as:

  • Revenue increase – improved finding efficiency and better operational results from better informed decision makers.
  • Risk reductions – sustaining licenses to operate, maintaining the value of data assets, avoiding unintended data loss, disclosure or damage to reputation.
  • Acquisition cost reduction – reducing the cost of data acquisition, e.g. by properly and securely archiving expensive data sources such as seismic, can prevent costs associated with reacquiring such data and can yield tens of millions of dollars per year from such optimization
  • OPEX optimization – reducing the current cost of data management by standardizing technologies and formalizing the organizational capabilities.

References (Generic Project Data Control Procedure; Molten Group Big Data DM Whitepaper 2013; CompuServe Whitepaper: Data Breach Management 2008)

Further reading

    Did you like this article? Get our new articles in your inbox with our occasional email newsletter.

    We will never share your details with anyone else, except that your data will be processed securely by EmailOctopus ( in a third country, and you can unsubscribe with one click at any time. Our privacy policy:

    data migration planning guide