Standalone python script to read gdb to numpy array

NickEbner

Hi, a simple processing task I'd like to share with my work colleagues requires reading a gdb to a numpy array.

I need this simple script to standalone without the need for a licence key. I also understand that there is an external API with low-level functions for just this task.

Can anyone give me some instructions on how to do this?

Sincerely,

Nick.

JosephCaluag1

Hi Nick,

You can use the db.read_channel as a function that can retrieve the data and store it as numpy data.

Here is a link to a standalone script that writes to a database https://github.com/GeosoftInc/gxpy/blob/master/examples/stand-alone/chanadd.py.

For example you can replace the db.write_channel call to db.read_channel(l, channel_name) to read the data.

Regards,
Joseph

NickEbner

Thanks Joseph. Unfortunately I get an error loading Geodist dll. The critical part of this task is being able to read the gdb without a licence installed. Is the lack of a licence likely the problem? In which case is there an alternative way I can simply read a gdb to a numpy array on a computer without a licence installed?

Much appreciated,

Nick.

JacquesBeaurain

Hi Nick,

We can probably improve the documentation and process a bit to make it easier to use the free APIs without a license. I will make sure this happens.

In the meantime here are two ways to do it. I recommend the first as it will ensure a system has all the necessary dependencies installed.

1. Install Oasis montaj viewer. Nothing more needed, this should just work.
2. Copy all the files (including the GeosoftFiles directory) from C:\Program Files\Geosoft\GX Developer\redist\bin into the site-packages\geosoft directory in your Python modules directory. And delete the geosoft.key file there. You might need to figure out which third partie dlls are needed in this case.

Note that we require an active sign-in via Geosoft Connect on each system where the API is used. This would automatically be enforced when the context is initialized.

Cheers,
Jacques

NickEbner

Thanks Jacques

It seems I have the code working... to a point. I get the below error when I try to open any (i've tried a few!) gdb as follows:


with gxpy.gx.GXpy() as gxp: # get the current gx context
gdb = gxpy.gdb.Geosoft_gdb.open(database)
lines = gdb.lines()
print(lines)
for line in lines:
npd,ch,fid = gdb.read_line(line)


runfile('H:/Python/scripts/gdb_read.py', wdir='H:/Python/scripts')
GX_GEOSOFT_BIN_PATH: None
Working directory: H:\Python\scripts
{'L62800': 500, 'L62900': 501, 'L63000': 502}
Traceback (most recent call last):

File "", line 1, in
runfile('H:/Python/scripts/gdb_read.py', wdir='H:/Python/scripts')

File "C:\Miniconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 688, in runfile
execfile(filename, namespace)

File "C:\Miniconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "H:/Python/scripts/gdb_read.py", line 85, in
npd,ch,fid = gdb.read_line(line) #, channels=['Synctime'])

File "C:\Miniconda3\lib\site-packages\geosoft\gxpy\gdb.py", line 1313, in read_line
data = self.read_line_vv(line, channels, dtype, fid, common_fid=True)

File "C:\Miniconda3\lib\site-packages\geosoft\gxpy\gdb.py", line 1263, in read_line_vv
vv[1].refid(fid, nvd)

File "C:\Miniconda3\lib\site-packages\geosoft\gxpy\vv.py", line 293, in refid
self._vv.re_fid(fid[0], fid[1], length)

GXError: Attempt to access Unknown Symbol [-1] failed because this is an invalid symbol.

IanMacLeod

This looks like a bug related to empty lines or channels in the database. Our current test databases have data in all channels on all lines. Can you send me one test database that triggers this exception? (send to ian.macleod@geosoft.com).

Thanks

IanMacLeod

@NickEbner

We have tested this with 9.2.1 as much as we can and we cannot duplicate your problem. Your coding pattern looks correct and we suspect something is special about the databases you have. It would really be helpful if you can send us a small database that fails.

It may also be just one peculiar line, in which case use a try: except: to simply skip and report problematic lines:

for line in gdb.list_lines():
   try:
      npd, ch, fid = gdb.read_line(line)
      # do your work with npd...

   except geosoft.gxapi.GXError:
      print('Unable to process line {}'.format(line))

If this fails on all your lines then we must see your database to figure out what is going on.

IanMacLeod

Nick, have not heard from you so I expect you have moved on, but in case not, and for the benefit of others who may be reading this thread:

We have now experienced a similar error message (Unknown Symbol [-1]), which is in fact hiding the actual problem message. In our case it was an attempt to write too much data to a channel, and it will have been something different for you as you are reading a database. This message-hiding behaviour is fixed for v9.3, which we expect to be available mid-November.

Again, if you can provide a sample database we can determine the actual message in your case.

NickEbner

Hi Ian, I made a few attempts at sending the data through from my nick 'at' newexco.com.au address. Did you receive any? One has a drop box link to the data.

IanMacLeod

Thanks @NickEbner. Unfortunately it did not make it through. I have sent you a private link to upload the file. If you can respond first thing Oz time I will look at it this evening so you don't lose another day.
Cheers...

IanMacLeod

@NickEbner, database received, thanks so much.

Yes, the bug is exposed by reading all channels by default where the database does not have a defined Z channel. This is has been fixed for 9.3, which is expected to be released by mid-November together with our 9.3 platform release.

One workaround for 9.2 is to ensure that there is a defined 'Z' channel as follows:

gdb.xyz_channels = ('x_rx_gda94', 'y_rx_gda94', 'Rx_Elev')
for line in gdb.list_lines():
   try:
      npd, ch, fid = gdb.read_line(line)
      # do your work with npd...

   except geosoft.gxapi.GXError:
      print('Unable to process line {}'.format(line))

Alternately, specify only the channels you want to read as the bug is triggered only when a default channel list is constructed to read all channels. For example:

for line in gdb.list_lines():
   try:
      npd, ch, fid = gdb.read_line(line, ('Synctime', 'MagTF', 'TxCurrent'))
      # do your work with npd...

   except geosoft.gxapi.GXError:
      print('Unable to process line {}'.format(line))

NickEbner

Thank you Ian. Unfortunately I now get this from your code above:

{'L62800': 500, 'L62900': 501, 'L63000': 502}
Traceback (most recent call last):

File "", line 1, in
runfile('H:/Python/scripts/gdb_read.py', wdir='H:/Python/scripts')

File "C:\Miniconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 688, in runfile
execfile(filename, namespace)

File "C:\Miniconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "H:/Python/scripts/gdb_read.py", line 91, in
npd, ch, fid = gdb.read_line(line, ('Synctime', 'MagTF', 'TxCurrent'))

File "C:\Miniconda3\lib\site-packages\geosoft\gxpy\gdb.py", line 1313, in read_line
data = self.read_line_vv(line, channels, dtype, fid, common_fid=True)

File "C:\Miniconda3\lib\site-packages\geosoft\gxpy\gdb.py", line 1263, in read_line_vv
vv[1].refid(fid, nvd)

File "C:\Miniconda3\lib\site-packages\geosoft\gxpy\vv.py", line 293, in refid
self._vv.re_fid(fid[0], fid[1], length)

GXAPIError: Unable to bind the "?ReFid_VV@@YAJPEAUh_gxx@@PEBJPEBN21@Z" wrapper function. This might be due to a missing DLL or license, or you are running an application outside of Oasis montaj.

IanMacLeod

@NickEbner,
This is because the geosoft.gxpy module uses core 9.2 function ReFid_VV that requires a subscribed/licensed Geosoft ID. You stated unlicensed use was a requirement when you opened this thread, but I missed that.

For an unlicensed user with 9.2 you will need to read the channels individually, though if the fiducials do not line up it will be up to you to deal with that. In the example you sent all the data on each line used the same fiducial base and I have verified that the following code works for an unlicensed user on your data with OM 9.2, gx 9.2.1:

import geosoft.gxpy.gx as gx
import geosoft.gxpy.gdb as gxdb

gxc = gx.GXpy()
filename = 'YOUR_FILE_NAME'
with gxdb.Geosoft_gdb.open(filename) as gdb:
    for line in gdb.list_lines():

        # data, channels, fid = gdb.read_line(line, ('Synctime', 'MagTF', 'TxCurrent'))  # requires a license!

        # read individual channels, no license required
        syncdata, syncfid = gdb.read_channel(line, 'Synctime')
        magdata, magfid = gdb.read_channel(line, 'MagTF')
        txdata, txfid = gdb.read_channel(line, 'TxCurrent')

        # work with numpy arrays...

For the 9.3 release we have been working through the libraries to remove license restrictions that make it difficult to read or write data (see https://github.com/GeosoftInc/gxpy/issues/26 ). I have reopened the issue to make ensure we verify your specific case against the 9.3 build.