You are Home   »   News   »   View Article

How data managers can do scripting

Friday, November 1, 2019

Alvin Alexander, geo technician from JX Nippon, taught himself to do scripting with Python to automate common data management tasks. He explained how he does it at our KL conference.

Alvin Alexander, geo technician from JX Nippon, taught himself to do scripting with Python to automate common data management tasks, and believes other data managers can do the same.

He shared his experiences and advice with a talk at the Digital Energy Journal forum in Kuala Lumpur in October.

It can be a lot more fun working by making scripts, rather than doing lots of manual work with a keyboard, such as making and copying folders, or copying data from one spreadsheet to another.

Mr Alexander also has a bad wrist pain which emerges when he uses the mouse too much. So he writes scripts to automate as much as possible.

Mr Alexander has only been programming for 7 years, learning mainly from 'a lot of online courses'. But he says the fastest way to learn is to encounter real problems and have to solve them.

And when you automate tasks, there are usually a lot less mistakes than from in manual work, such as from pressing the wrong key.

But the most important motivator for scripting can be that it is more fun finding a better way to do something.

'If you're not having fun you're not doing something right for you,' he said. But when you have fun, it increases your motivation, you are more productive, so it is better for the company too.

A starting point is to recognise that there is a very big difference between scripting and programming, Mr Alexander said. Programming is complicated work of building software. Scripts are something designed for a specific problem, such as 'generate folders from a list', or doing machine learning on log data. Data managers should focus on scripting, not programming.

When considering building an automated solution, a first question is how often you spend doing a task. If it is a five second task, it probably doesn't need to be automated. 'If it takes three days, it probably needs to be automated.' If you have done something twice, maybe you will need to do it again, and it makes sense to automate it.

And if you have good skills in a programming language, particularly Python, maybe you will want to automate everything. Or there might be scripts available free online you can just copy.

Examples of tasks

Mr Alexander gave some examples of common oil and gas data management tasks which could be done faster with scripts.

A 'plain boring' task data managers might be asked to do is generate 200 folders from a list of folder names you want created. 'It takes forever', he said. 'Copy one item, create a folder, rename, paste. In Python you can do it in 5 lines of code.'

A second example is to consolidate 5 Excel spreadsheets into one. Instead of doing this, you can convert the spreadsheet into code. 'It doesn't matter whether you have 5 or 5000 excel sheets, it is the same amount of code,' he said.

A third example is extracting data from a well log (LAS) file. For example, you want vertical permeability data at a certain depth. Usually this is done by opening the LAS file with a text editor, and copying the columns into Excel. You can buy software to do this, but it is probably not free.

Mr Alexander wrote his own 'parser', a software tool to break up well log data into useful elements.

You can make an automated solution with free code which will look for key words in the text. So it can copy pressure data only for a certain range of depths. A more involved question is to show how the permeability is varying by depth.

A fourth example is drawing curves. You can get Python code to draw perfect Sine waves, squares, perfect angle spiral, much better than drawing it by hand

In one project, Mr Alexander needed to trace a typical gamma ray curve. He tried doing it by hand using a mouse, or by looking for software which could do it digitally, but neither worked well. So he wrote a script to trace the curve image.

A fifth example was a tool which can create a list of all of the data files on a CD, no matter what the folder structure is. This is very useful to a data manager who has been handed a CD with unknown contents and wants to understand what is on there.

Using Python

'Python, among all programming language that I learned, is the most friendly, almost like talking to you, almost human language,' he said. 'Not like Java or JavaScript.'

It can be amazing how little code you need to write with Python, because most of the code you might need is freely available. 'The community is so nice, big, friendly, they provide almost everything for us. We almost never need to write any code from scratch.'

Mr Alexander recommends the Anaconda Distribution software for installing Python, with all its modules, and a store of tools to run on it, 'like Google Play store for Android'.

You don't need to use an IDE (Integrated Development Environment), as you do in computer programming.

There is an open source web application called Jupyter Notebook which can be used to run specific lines of code. 'It's very easy to use,' he said. 'Since I learned this, I learned Python a million times faster. IDE is so confusing. There's so many unnecessary things. In this is it is only code. A lot of the scientific community really favour this.'

Original work

Mr Alexander concluded with the Gustave Flaubert quote, 'Be regular and orderly in your life, so that you may be violent and original in your work.' - this sums up the approach to data management, if you are orderly in how you go about managing data, it is possible to do much more with it.

In oil and gas, the people doing the 'violent and original work' are the people who work with the data, such as the geologists, geophysicists, petrophysicist, reservoir engineers, other engineers - and many data managers are themselves also geologists so users of the data.

Associated Companies
» JX Nippon Oil & Gas Exploration
comments powered by Disqus


To attend our free events, receive our newsletter, and receive the free colour Digital Energy Journal.


Latest Edition May June 2022
Jun 2022

Download latest and back issues


Learn more about supporting Digital Energy Journal