How/whether to use fids and interpolation?
Options
RobertCram
Posts: 15 
I have an export, using C#, from an application into a GDB file. Let's say that a file includes 1,000,000 measurements. The system gets a GPS location once per 1,000 measurements. The current behavior is to export 1,000,000 measurements, with 1,000,000 lat/lon measurements interpolated from the 1,000 actual GPS measurements. Currently, the export includes no explicit management of fiducials. Questions and comments.
1) Would there be advantages to exporting only the original 1,000 GPS measurements, and letting Montaj do the interpolation? This question is about advantages inside Montaj only; the existing export function will scale to much larger sizes without problems. If the DB and the Montaj memory model will include X/Y (lat/lon) spaces for the entire interpolated locations, and any interpolation functions simply fill in those spaces, then it might be simpler to continue with the existing scheme, which fills in those spaces in the first place.
2) An advantage of the current implementation is simplicity: once the DB has been exported, the Montaj user simply opens the database and grids it or otherwise manipulates it. If a sparse GPS export would require additional work on the part of the user, to set up interpolations, etc., that sparse export would not be as valuable. If on the other hand the sparse export included some information inside the database that the GPS channel was to be interpolated, and this happened automatically in Montaj, that would be more valuable.
3) I see some references to re-fid and interpolation in the documentation. It isn't clear from the documentation that I have seen, whether these functions change values in the database (or in the case of interpolation add values to the database), or whether they add metadata to the database for Montaj to use, or whether they are effective only during the current session. What is the case?
4) The exporting application includes a fiducial (which currently is not exported). It is (something like) the number of milliseconds since the start of the Unix epoch, and therefore may include gaps in any particular dataset. Can this sequence (including gaps) be used as the fiducial, or must the fiducial include no gaps. It wouldn't be difficult for the exporting application to provide a continuous FID, but it would be nice to know if this is necessary.
5) An occasional once-per-second (once-per-thousand instrument measurements) GPS measurement fails to arrive. If interpolation is used, is this a problem, or will values still be interpolated correctly if the interpolation gap between GPS measurements (in FID increments) is uneven?
6) Do there exist any examples illustrating various solutions to these questions? The language (C,C#, python) is unimportant to me.
7) Am I overthinking this?. If I simply set X and Y values for only the indexes into the data that have actual GPS values (something like "_xVV.SetReal(currentMeasurementIndex, dCurrentLongitude);" will this automatically cause the X and Ys to be interpolated? Maybe the X and Y channels are magical this way? If I do this, will I save space in the database and in the memory model, or (referring back to question 3 above, am I simply moving the work and space from one point to another?
Thanks,
-- Bob Cram
1) Would there be advantages to exporting only the original 1,000 GPS measurements, and letting Montaj do the interpolation? This question is about advantages inside Montaj only; the existing export function will scale to much larger sizes without problems. If the DB and the Montaj memory model will include X/Y (lat/lon) spaces for the entire interpolated locations, and any interpolation functions simply fill in those spaces, then it might be simpler to continue with the existing scheme, which fills in those spaces in the first place.
2) An advantage of the current implementation is simplicity: once the DB has been exported, the Montaj user simply opens the database and grids it or otherwise manipulates it. If a sparse GPS export would require additional work on the part of the user, to set up interpolations, etc., that sparse export would not be as valuable. If on the other hand the sparse export included some information inside the database that the GPS channel was to be interpolated, and this happened automatically in Montaj, that would be more valuable.
3) I see some references to re-fid and interpolation in the documentation. It isn't clear from the documentation that I have seen, whether these functions change values in the database (or in the case of interpolation add values to the database), or whether they add metadata to the database for Montaj to use, or whether they are effective only during the current session. What is the case?
4) The exporting application includes a fiducial (which currently is not exported). It is (something like) the number of milliseconds since the start of the Unix epoch, and therefore may include gaps in any particular dataset. Can this sequence (including gaps) be used as the fiducial, or must the fiducial include no gaps. It wouldn't be difficult for the exporting application to provide a continuous FID, but it would be nice to know if this is necessary.
5) An occasional once-per-second (once-per-thousand instrument measurements) GPS measurement fails to arrive. If interpolation is used, is this a problem, or will values still be interpolated correctly if the interpolation gap between GPS measurements (in FID increments) is uneven?
6) Do there exist any examples illustrating various solutions to these questions? The language (C,C#, python) is unimportant to me.
7) Am I overthinking this?. If I simply set X and Y values for only the indexes into the data that have actual GPS values (something like "_xVV.SetReal(currentMeasurementIndex, dCurrentLongitude);" will this automatically cause the X and Ys to be interpolated? Maybe the X and Y channels are magical this way? If I do this, will I save space in the database and in the memory model, or (referring back to question 3 above, am I simply moving the work and space from one point to another?
Thanks,
-- Bob Cram
0
Comments
-
I tried number 7) above, in the hope that Montaj would automagically interpolate. This doesn't seem to happen, so I tried the helpfully-named menu choice Coordinates/Interpolate X/Y.
That requires growing the database, suggesting that (as speculated in number 1) above), that interpolation will fill those values into the database and into the loaded memory model, just as if I had originally saved all the interpolated values in the DB in the first place.
And, when trying the interpolate function, Montaj says that the database is full, so I need to grow it (using Database/Maintenance). Unfortunately, "grow" provides for new channels or lines, but doesn't provide an option for this current case (to add interpolated values into the current line and channels).
It seems, then, that the best course is to continue to include all of the interpolated values in original export.
Is this true?
-- Bob0