Forum connection issue and exporting magnetic survey
Options
RhondaStewart2
Posts: 1 Staff
1) When I go to the forum (after successfully signing in with my Seequent ID), the only thing I see is this message: geosoft.vanillacommunity.com refused to connect. Perhaps this is just a temporary issue related to current world issues?
2) I'm trying to export a magnetic survey from Geometrics into a GDB file. It's about 4.5MM measurements. Some way in, the process fails with this message:
"Process is terminated due to StackOverflowException. An unhandled exception of type 'System.StackOverflowException' occurred in geoengine.core.gxnet.dll"
Another message indicates an out-of-memory error in the GX code.
The out-of-memory is happening somewhere in the GX code; I have other exports that run the same Geometrics code, except with a different plug-in to write to a different format (instead of GDB). I know that Montaj can handle much larger datasets than 4.5MM, so I'm sure that I need to change my use of the API.
So far, I've just been allocating big CVVs, filling them with data, and at the end saving them to the CDB. I do no explicit management of FIDs. Some of the CVVs are sparse - e.g. 1 GPS/1000 mag records. But up to a couple million records this has been no problem. I figured that I might run into this issue eventually.
What I need to understand now is how to write a larger database, with some columns completely filled and others not. It seems to me that I need to understand a few particular issues:
1) How to write data in blocks rather than by filling a giant array for the full survey. CDBWrite seems like a useful class, but I don't find any examples. Or maybe I can write to CVVs and commit them in blocks, clear them, fill again, etc., rather than commit only at the end of 4.5MM blocks.
2) I need to understand fiducials better to handle sparse columns. As far as I can tell if I write record 0 into a CVV and then record 1000, the CVV gets expanded to hold all the records in between. It seems as if the CVV.SetFid* methods could be useful, but the API docs aren't very helpful.
Is there a page somewhere that explains fiducials and how to manipulate them?
Is there an example program for writing larger files? Is there for example sample code derived from your own ASCII import utilities?
Bob Cram
Geometrics
2) I'm trying to export a magnetic survey from Geometrics into a GDB file. It's about 4.5MM measurements. Some way in, the process fails with this message:
"Process is terminated due to StackOverflowException. An unhandled exception of type 'System.StackOverflowException' occurred in geoengine.core.gxnet.dll"
Another message indicates an out-of-memory error in the GX code.
The out-of-memory is happening somewhere in the GX code; I have other exports that run the same Geometrics code, except with a different plug-in to write to a different format (instead of GDB). I know that Montaj can handle much larger datasets than 4.5MM, so I'm sure that I need to change my use of the API.
So far, I've just been allocating big CVVs, filling them with data, and at the end saving them to the CDB. I do no explicit management of FIDs. Some of the CVVs are sparse - e.g. 1 GPS/1000 mag records. But up to a couple million records this has been no problem. I figured that I might run into this issue eventually.
What I need to understand now is how to write a larger database, with some columns completely filled and others not. It seems to me that I need to understand a few particular issues:
1) How to write data in blocks rather than by filling a giant array for the full survey. CDBWrite seems like a useful class, but I don't find any examples. Or maybe I can write to CVVs and commit them in blocks, clear them, fill again, etc., rather than commit only at the end of 4.5MM blocks.
2) I need to understand fiducials better to handle sparse columns. As far as I can tell if I write record 0 into a CVV and then record 1000, the CVV gets expanded to hold all the records in between. It seems as if the CVV.SetFid* methods could be useful, but the API docs aren't very helpful.
Is there a page somewhere that explains fiducials and how to manipulate them?
Is there an example program for writing larger files? Is there for example sample code derived from your own ASCII import utilities?
Bob Cram
Geometrics
0
Comments
-
For #2. You should never allocate a VV beyond what you need for that actual number of items, so if you have data at Fid 0, 1000 and 2000, set the length to 3, insert the items at indices 0, 1 and 2, and use the VV functions SetFidStart = 0 and SetFidIncr = 1000. When the data is added to a channel, the database only stores three values for that channel, but when it is displayed for a line where the Fid increment is 1 (the normal situation), then values between 0 and 1000 and between 1000 and 2000 are interpolated and displayed; they don't really exist anywhere except in the view of the database. Note that selecting data in the database and doing "statistics" works with the displayed values, not the actual data stored in the channel. To see a report of the actual data you need to run the channel report GX.0
-
For #1, the VV class is the way to go. There are definite limits on the amount of data you can store in a channel in a line, which you can adjust somewhat by modifying the block size when creating a database, but splitting data into lines is one way to reduce the amount of memory required for temporary storage, and the number of lines can very large. Getting a stack overflow is certainly a bug - the warning about being out of memory is more what should be expected. To see information about the maximum size for channels and databases, go to the Help for "New Database". The default max channel data is 64MB, or about 16 million float values, but it can be increased up to 1GB.0
-
Thank you for the information. The export is a C# program, and I used the pagesize argument to CDB.CreateEx to increase the max channel size. I will also look into breaking the channels into lines.
-- Bob Cram
Geometrics
0
