Data Management in Oil and Gas Industry

In this lecture, following Best Practices for Effective Data Management in oil and gas industry is covered:

• Outline of oil and gas business goals. ...

• Priorities of data protection and security. ...

• Focus on data quality. ...

• Reduce data issues. ...

• Accessibility of data by the expert team. ...

• Data recovery strategy. ...

• Quality data management

Learning Tip: Developing a data management is the backbone of a successful oil and gas company. Having clean, quality, reliable data that gives strong insight about hydrocarbon asset and dynamic behavior patterns is essential to creating development plans.

To ensure your company’s data quality, you must follow these data management best practices. Otherwise, you could end up with dirty data and consequently: engineering chaos.

For oil and gas companies in particular, data means beginning, continuity of business and exit. Entire value creation is based on :

      – predictive asset optimization

      –optimizing upstream and midstream operations

      –implementing modern technologies for environmental and human monitoring and logistics

The power of AI and IoT in oil and gas industry has extensively large by information/data-based operational optimizations in the upstream, midstream and downstream. The first three detail upstream operations and asset optimization.

Therefore,  oil industry has long history on data collection and storage including certain habits in data management.

Here is the list of reason for companies to have a very strong backup systems:

• Systems failures

• Security compromise

• Inadequate security

• Malware

• Cyber threats and risks

• Accidental or malicious data deletion

• Internal malpractices

• IT Failure

• Different data sources

Here is the life-cycle of a data management

• Develop Data Management Strategy

• Identify Framework for Data Governance

• Implement Data Governance

• Get feedback from users

• Fix or Improve Governance

• If needed, fix or improve strategy

· Benefits of good Data Management.

· Business case aspects and barriers.

· Data governance: strategy, organization, policies and standards, projects and issues.

· Architecture: modeling, technology and tools.

· Framework, governance, architecture, security.

· Difference between reference and Master Data Management.

· Data quality management: definition and dimensions of data quality (accuracy, currency, coverage, relevance, accessibility and comparability).

· Data quality tools and capabilities.

Forces acting on E&P Companies here & now

• Asset Mix/Optimization

• Shareholder Return

• Oil & Gas Price

• Technology

• Workforce Demographics

• Globalization

• Environment

Without a good data management, oil and gas companies can not operate and be successful.

Therefore, assessment is needed:

ASSESS YOUR STRATEGY

• Do you filter, validate and cleans?

• Do you detect junk in received data?

• Is data in repository reliable?

• Do you backup your data?

• Is your data secured?

In this lecture, both quality of data and quality of data management is covered.

Having poor-quality data can make it difficult to operate daily operations. Poor quality data can cause serious consequences such as:

• Wastes resources

• Costs effort, time and money

• Damages the quality of their analytics

• Negatively impacts the operations

• Results delays in operations

• Hinders compliance with government and industry regulations

  
  

Data Quality Metrics

1. Accuracy measures the number and types of errors in a data set. Different types of inaccuracies may be present in a data set, including:

• Anomalous Value

• Anomalous Set

• Anomalous Units

• Anomalous  Format

2. Completeness is important that all critical fields in a record be fully populated. The completeness metric measures the number of records with incomplete data. It is tracked by identifying records with empty fields and typically expressed as a percentage of the total number of records.

3. Consistency measures how individual data points pulled from two or more sets of data synchronize with one another. If two data points are in conflict, it indicates that one of both of the records are inaccurate. Inconsistency can have a number of causes, including:

• Data entered incorrectly in one or more sources

• Data entered differently in one or more sources

• Data sourced at different times (indicating newer data may reflect changes from older data)

• Different structure/schema between the data sources not fully matching

4. Vintage means the age of data in a database. Modern data are likely to be more accurate and relevant, as value and measurement techniques can change over time. In addition, there is a significant risk of multiplying errors when older data is moved through the pipeline, as all intermediate data repositories get populated with results from the outdated data.

5. Uniqueness tracks duplicate data. Any data recorded twice can unduly weight any results. It’s important to identify duplicates and either merge them or delete the duplicates.

6. Validity measures how well data conforms to standards. That is, data entered into a given field should be of the proper type; if the wrong data format is entered, that data may be unusable. It’s important to check that each piece of data conforms to the correct type and format of data required.

The objective of this lecture is to learn how to define an architecture based on a complete set of requirements as follow:

• the business problem in architectural terms

• The business value of solving the problem is clear

• The relevance of potential solutions is clear

Architectural Approach

•Boundaryless Data Flow is critical in today’s business environment

•Good professional architecture is a key enabler of Boundaryless data Flow

•Architectural framework is an enabler of good professional architecture and is free for own use

•Business Scenarios give a complete picture of the requirements

•The Architecture Development Method provides a rigorous process and can be used with other frameworks

Master Data

• Data about the business entity

• Business rules typically dictate the format and permitted ranges of master data values.

• Common organizational master data

• individuals,

• organizations, and their roles,

• customers,

• citizens,

• vendors, suppliers, business partners, competitors,

• employees, etc.

• Products, internal and external, inventory, and related concepts.

• Financial structures, including general ledger accounts, cost centers, profit centers, etc.

• Location concepts, for the organizations

• Individuals and other entities that concern the enterprise.

Although, as we outlined, limiting your data to only the necessary information (fit-for purpose) your company needs to meet its goals is a great way to improve data quality, there are so many more steps to take to ensure the data your company is collecting remains clean and reliable.

1. Data should be regularly checked for accuracy as old data can become outdated and irrelevant to your asset.

2. Your data management software has to maintain your database to keep it from negatively impacting your analytics, and other processes.

3. Train all team members who have access to the data about the proper ways to collect and input data.

4. Ensure the data is checked and cleaned before it is used in any analytics or reporting

5. Implement accuracy metrics to keep all aspects of your company’s data use clean and reliable.

Developing and implementing a strong and productive data management is key issue for oil and gas companies.

In this lecture, some of the tools are reviewed. An oil and gas company has to have following Data Management Tools :

• Hydrocarbon Asset Data Management

  • Seismic/Geology/Well Databases

  • Production/Field Data

• Commercial/Procurement Databases Management

• Inventory/Machinery Database

• Administrative and Financial Resources Management

• Multimedia Sources Management

Oil and Gas Software Proficiency

• Industry-Specific Software: Proficiency in specialized software, such as Schlumberger’s Petrel, Halliburton’s Landmark, and Honeywell’s UniSim, used for geological modeling, reservoir simulation, and process automation.

• Enterprise Resource Planning (ERP) Systems: Knowledge of ERP systems like SAP and Oracle used in oil and gas for inventory, financial management, supply chain, and asset management.

• Supervisory Control and Data Acquisition (SCADA) Systems: Skills in SCADA systems for monitoring and controlling field equipment, particularly in production and drilling operations.

2. Data Management and Database Administration

• Database Management Systems (DBMS): Proficiency in DBMS software such as SQL Server, Oracle Database, and NoSQL databases for handling large datasets, including geological, production, and historical data.

• Data Warehousing and Big Data Tools: Competency in data warehousing and big data tools (e.g., Hadoop, Spark) for managing and analyzing large volumes of data generated in oil and gas operations.

• Data Quality and Integrity Management: Skills in ensuring data accuracy, completeness, and reliability, critical for decision-making and regulatory compliance.

3. Data Analytics and Visualization

• Data Analytics Tools: Knowledge of tools like Power BI, Tableau, and Spotfire to analyze and visualize data, enabling insights into production performance, operational efficiency, and cost analysis.

• Predictive Analytics and Machine Learning: Familiarity with predictive analytics and machine learning applications in the industry, such as predictive maintenance, production forecasting, and anomaly detection.

• Geospatial Analysis: Skills in using Geographic Information Systems (GIS) and spatial data analysis tools to map and interpret geographic data related to oil and gas exploration.

4. Network and Communication Systems

• Industrial Network Management: Proficiency in managing and troubleshooting industrial networks (e.g., Modbus, OPC, Ethernet/IP) that connect SCADA systems and field devices.

• Cybersecurity Protocols for Industrial Control Systems (ICS): Knowledge of cybersecurity protocols, such as ISA/IEC 62443, NIST standards, and firewall configurations to secure ICS and OT (Operational Technology) networks.

• Remote Communication Technologies: Skills in configuring and managing remote communication systems, such as satellite, microwave, and fiber-optic networks, essential for offshore and remote site connectivity.

5. Operational Technology (OT) Integration

• Real-Time Data Monitoring and Control: Competency in implementing and supporting real-time data acquisition systems for monitoring drilling, production, and pipeline operations.

• Distributed Control Systems (DCS): Knowledge of DCS for automated control and monitoring of processes, especially in refining, petrochemical, and midstream operations.

• Internet of Things (IoT) and Edge Computing: Familiarity with IoT devices, edge computing platforms, and telemetry systems to gather real-time data from field operations.

6. Cybersecurity and Risk Management

• Cybersecurity Best Practices: Knowledge of cybersecurity best practices specific to the oil and gas sector, including network segmentation, intrusion detection, and regular system updates.

• Incident Response and Recovery Planning: Competency in developing incident response plans, conducting risk assessments, and ensuring business continuity in case of cyberattacks or system failures.

• Compliance with Industry Regulations: Understanding of regulations like NERC-CIP (for North American operators) and GDPR (for personal data protection) to ensure compliance and data protection.

7. Cloud Computing and Data Storage

• Cloud Platforms (AWS, Azure, Google Cloud): Proficiency in cloud platforms commonly used for scalable data storage, backup, and collaboration in oil and gas.

• Hybrid and Multi-Cloud Strategies: Skills in deploying and managing hybrid or multi-cloud environments that combine on-premise and cloud solutions for data flexibility and security.

• Virtualization and Containerization: Knowledge of virtualization (e.g., VMware) and containerization (e.g., Docker, Kubernetes) for efficient resource allocation, system isolation, and rapid deployment of applications.

8. Automation and Digital Transformation

• Robotic Process Automation (RPA): Competency in RPA tools like UiPath or Automation Anywhere to streamline repetitive processes, such as reporting and data entry, in back-office operations.

• Artificial Intelligence (AI) Applications: Familiarity with AI-driven applications in exploration, production optimization, and equipment maintenance to improve decision-making and efficiency.

• Digital Twin Technology: Understanding of digital twin technology for creating virtual models of physical assets, which allows for predictive maintenance, testing, and scenario analysis.

9. Project Management and Coordination

• Project Planning and Scheduling Software: Knowledge of project management tools like Primavera, MS Project, and Agile methodologies for managing complex IT and OT projects in oil and gas.

• Cross-Functional Collaboration: Skills in working with geoscientists, engineers, field personnel, and management teams to implement IT solutions that support business goals.

• Documentation and Reporting: Competency in creating detailed documentation, project reports, and user guides to support system maintenance and future upgrades.

10. Health, Safety, and Environment (HSE) Awareness

• HSE Software and Compliance Systems: Familiarity with HSE software solutions to monitor safety incidents, manage compliance, and maintain records for regulatory reporting.

• Risk Assessment and Hazard Analysis: Understanding of hazard analysis and risk management processes to support safe IT implementations and system integrations.

• Emergency Preparedness for IT Systems: Skills in planning and preparing for emergencies (e.g., power outages, data breaches) to minimize downtime and ensure continuity of critical operations.

11. Problem-Solving and Technical Support

• Technical Troubleshooting and Support: Strong troubleshooting skills to diagnose and resolve hardware, software, and network issues in real-time, ensuring minimal impact on operations.

• Asset Management and Inventory Control: Knowledge of IT asset management practices to keep track of hardware, software, licenses, and ensure efficient resource allocation.

• End-User Training and Support: Ability to train staff on new software or IT tools, create user guides, and provide technical support to ensure successful adoption and usage.

Seismic data are datasets resulting from these seismic waves in the form of wriggly lines (spaghetti-like) detected by receivers known as geophones (on land) or hydrophones (in a marine environment), depicting information about the Earth‘s subsurface.

Now, these are processed seismic data generated in SEGY format. SEGY is an abbreviation of Society of Exploration Geophysics format Y.  The raw, unprocessed, or field data are in SEGD format in the custody of the processing centers to be processed, and their output is in SEGY format.

Normally, the clients are given copies of the SEGD datasets for safekeeping, in the beginning, on 7 or9-track tape, now normally on Linear – Tape – Open (LTO) media. As we speak, processing companies are probably finding ways to store these field data which in total size can be from the 50s to 100s depending on the extent of the ―projects.

Please note that now even other seismic attributes and well attributes are stored in SEGY format.

It is only after processing that these seismic curvatures have geological and geophysical meaning to them.

Currently with the latest in storage technology, terabyte, zettabytes, yottabytes2 of data are being accessed, used, and archived in a matter of minutes.

This is where the basic understanding of seismic data management is needed to ensure that nothing is lost in the geophysical sense, right from seismic acquisition, seismic processing, data gathering, seismic interpretation, data back up, and right up to data archiving.

As the seismic acquisition, seismic processing, and seismic interpretation undergo advancement with new technology, the same goes with seismic data management.

We have to adapt in our approach to these advancements but the concept still remains.

To recollect, two standard formats seismic can be stored:

(1) SEGD: field data or unprocessed dataset

(2) SEGY: processed seismic dataset,

G&G Data means geological, geophysical and geochemical data and other similar information acquired through various techniques.

• G&G, Seismic data

• Geodesy, GPS, GIS

• Borehole data (drilling report, logs and cores).

Beside G&G data, from an oil and gas field there are many other data collected for right management and operations. Most of these data are collected from wellbore as listed:

• Well logs

• Well testing records

• Pressure measurements

• Well coring analysis

• Image logs

• Core descriptions

• Well markers

• Well trajectory (gyro) data.

• Lithology

• Mud logging data …etc

Drilling a single well or multiple wells simultaneously, collecting data real time and running successful drilling operations for field development is the backbone of an oil and gas company. The availability of real-time drilling data communication is highly critical to the exploration and production business of any petroleum company. Incomplete, inaccurate or inconsistent data handling could results in severe damages, commercial lost, injury, environmental pollution, or even loss of lives.

Current drilling programming and drilling data handling techniques are reviewed in this lecture. Furthermore, we evaluate the different data types, data loads, and different networks including limited bandwidth networks.

Historical data in addition to streaming data flow to oil and gas companies has turned the data management of manual form almost in disuse. Implementing higher capacity computer data management solutions guarantee the coherence, integrity and traceability of the data and consequently increase the quality and reliability of data based operations/management.

The use of standards to achieve effective data management in the Oil & Gas industry is a growing need. Today technologies such as Big Data, Machine Learning or Deep Learning, allow to achieve significant improvements in the operations of the oil & gas industry. However, part of the success of its implementation, it depends on how organized each company has data.

Field development planning is extremely important part of the oil and gas business that also depends  heavily on data. Adequate and consistent data and data gathering plan helps to operate business in a right way for success.

Due to complexity of sub-surface hydrocarbon structures and dynamic reservoir behaviour, accurate modeling of field development scenarios before making significant capital investments can optimize hydrocarbon production and yield tremendous savings of time, effort and money.

Exploration activities cost tremendous amount of expenditures for gravity/magnetic surveys, seismic acquisition and interpretation, geological studies and finally drilling. After spending millions of dollars to these activities and collect data, generally companies undervalue the data. During the life of a HC field, in later stages many arbitrary decisions are made without proper data analysis/integration and studies.

Field assessment provides strong technical basis for increasing the value through identification of proper operation and development options of the oilfield. With a greater speed and depth of analysis, the target is to gain a comprehensive understanding of reservoir mechanics to define proper reservoir management practices ensures the achievement of the goals.

Critical diagnostic and holistic approach provides insight metrics for optimal well design, placement, construction and operation for long-term excellence in achieving ultimate recovery and production sustainability while meeting short-term goals.

3D sub-surface modeling and simulation has to start with the decision of first well. Earth model of a field helps to properly design and optimize both field operations and data acquisition programs.

Maximizing THE VALUE of A Hydrocarbon assets

Storing and Retrieval of data during the reservoir lifecycle constitutes a major challenge due to the inability of differently structured databases to communicate, along with software incompatibility issues. Often, this is given insufficient attention, destroying much or all of the value of the data gathered. In this lecture, we aim to share best practices as well as latest developments in this area.

• Data requirements for green and brown-field operations, secondary and tertiary recoveries.

• Cost-effective data gathering.

• Industry Standards for applications and data systems.

• Data management challenges and best practices.

Higher oil price, declining oil production due to natural depletion and political instability has increased the importance of the world marginal oil inventory and new technologies to increase the recovery from the existing reservoirs. New technologies in unconventional gas development have drastically impacted gas price and the availability. Similar sort of breakthrough and optimization are also expected in oil recovery.

There are risks involved in all investments. The key is understanding those risks and making decisions accordingly. Oil and gas investments in particular carry a complicated and variable set of factors that affect overall risk, factors that may create significant swings in a well’s production potential that ultimately affect overall investment performance.

1. The main objectives of the Field Scale Study are

                 a. Construction of a reliable representative static and simulation model

                       i. to incorporating all available data

                       ii. to manage, monitor and optimize operations

                       iii. to use it as a tool to optimize field development plan

b. Incorporate all lessons learned from different studies, geological, geophysical, petrophysical and dynamic data into 3D Earth model.

c. Transfer Earth model into 3D reservoir model for further reservoir dynamic simulation.

d. Uncertainty mitigation due to comprehensive understanding of the reservoir static and dynamic characteristics.

A complete and consistent database from a reservoir or field of interest is often missing in oil and gas company network. Gathering a complete database and updating consistently is extremely vital for successful operations. 

The most common problems faced by engineers and professionals during data analysis is the issue of missing data. Given the nature of the analysis, the data needs to be presented in complete matrices. In an environment where the data is scares, this can cause serious problems and make the dataset unusable.

Consistency of database can be assured through a set of analysis and calculation. Upon importing the data one of the first tasks that needs to be performed is to visualize any apparent relationship between different parameters in a data set. This can reveal both any outlier data records and the main characteristics of parameters in the data set.

Integrated operations and team work are methods of utilizing new processes, technology, and information to facilitate efficient exploration and production within the petroleum industry by using various discipline expertise simultaneously. It provides members of a team with the ability to work together, even when far apart, to combine different disciplines, increase production, and save money on costs.

Members of an integrated operations team can include engineers, scientists, oil executives, and environmentalists. As technology changes, integrated operations change as well to include the newest research techniques and methods within the industry.

Much of integrated operations is based upon computer and telecommunications systems that enable people to work together from separate disciplines. As efficiency is increased through the better use of resources, the costs automatically go down. It is similar to production and chain management techniques that are found in other parts of industry and production. Better supervision of certain tasks, such as oil well drilling, decreases the amount of mistakes that are made, which also helps the companies save money in the long-term.

Another part of integrated operations involves streamlining the data and methods for ongoing improvement. Once these systems are put in place, they increase the amount of oil that is extracted while reducing the cost.