Issues with ICESat-2 data.

Description

From kenneth.l.cockerill@nasa.gov:

I wanted to pass along some errors that we are seeing when attempting to Open/View/Convert some ICESat-2 mission data using Hyrax 1.13.5. The original products are in H5 format and I ran a number of them through the Hyrax Interface, trying various format conversions.

Some of the types of messages I received are as follows:

  • libdap exception building response: error_code = 1002: An internal error was encountered in h5get.cc at line 938: Unsupported HDF5 64-bit Integer type:/quality_assessment/packet_counts/alt_sci/qa_pce1_dnf Please report this to support@opendap.org

  • File out netcdf, unable to end the define mode: /workingdata/opendap/nckdzg04: NetCDF: HDF error

  • fileout.netcdf - Failed to add dimension dim235: NetCDF: NC_UNLIMITED size already in use

  • fileout.netcdf - Failed to define chunking for variable _quality_assessment_cal_error: NetCDF: Invalid argument

My comments to Ken:

  • The first issue "... will require DAP4 or that the data provider set an option in the HDF5 handler that offers a work around for types not supported by DAP2"

  • The remaining three: "These could be more complex issues… But I suspect that they are related to the DAP4 data type issue. Internally the server is using DAP2 for most of its work, we hope to change over to DAP4 during the next year (and have the DAP2 responses built from DAP4 objects, since the latter subsumes the former)."

Ken has uploaded data on www.opendap.org that we can access using FTP in the ftpuser directory: "James – I uploaded a file called ATLAS.tar.gz that contains 6 ICESat-2 data files:"

Environment

None

Activity

Show:
James Gallagher
January 8, 2019, 9:13 PM
Edited

Conclusion: from kenneth.l.cockerill@nasa.gov: Implement a proposed solution where the data are encoded using Float64 variables. Mark this as resolved and add a new story.

Kent Yang
October 18, 2018, 4:23 PM

Guys,

More information to share with you on my side before I moved.
1) To get the correct DMR output of ATL 01 for the default option.

You need to build the hdf5 handler from the master of github.: https://github.com/OPENDAP/hdf5_handle
change h5.conf as follows:

H5.EnableCF=true to H5.EnableCF=false or comment out

H5.DefaultHandleDimension=true to H5.DefaultHandleDimension=false or comment out

2) I received more ATL files from kenneth.l.cockerill@nasa.gov via Slav. Among them are
ATL03,02,08,12. the Default DMR output can be generated and include 64-bit integer.

James Gallagher
October 18, 2018, 4:00 PM

OK. We’ll do what we can.

Thanks for looking into this.

James


James Gallagher
jgallagher@opendap.org

Kent Yang
October 15, 2018, 9:40 PM

Hi James,

Nathan probably doesn’t know that my time is limited again to work on Hyrax. Sorry if I cannot contribute as before.

Kent

From: Kent Yang
Sent: Monday, October 15, 2018 4:36 PM
To: 'Nathan Potter (Jira)'
Subject: RE: [JIRA] () Issues with ICESat-2 data.

Hi Nathan,

All I can say is that with BES command line, I can generate the DMR output as shown in the attachment. With the OLFS added, I got the error in the screenshot.
I briefed James and other folks in your developer meeting. My time working on this support is limited. I simply don’t have time to dig in more. This is the most I can do.

From: Nathan Potter (Jira) jira@opendap.atlassian.net
Sent: Monday, October 15, 2018 4:26 PM
To: Kent Yang
Subject: [JIRA] () Issues with ICESat-2 data.

https://avatar-cdn.atlassian.com/982957953e66bcab03fb675856e1df4e?s=48&d=https%3A%2F%2Fopendap.atlassian.net%2Fsecure%2Fuseravatar%3FownerId%3Dndp%26avatarId%3D10401%26noRedirect%3Dtrue

Nathan Potter<https://opendap.atlassian.net/secure/ViewProfile.jspa?name=ndp> commented on [Bug] HK-96<https://opendap.atlassian.net/browse/HK-96?atlOrigin=eyJpIjoiZmY2Y2ExNmRjODU1NDM5ZGE1NGRhNDc5NGYwMTYwYTciLCJwIjoiaiJ9>

Re: Issues with ICESat-2 data.<https://opendap.atlassian.net/browse/HK-96?atlOrigin=eyJpIjoiZmY2Y2ExNmRjODU1NDM5ZGE1NGRhNDc5NGYwMTYwYTciLCJwIjoiaiJ9>

The attached PDF file does not contain the error response returned by the BES. It contains an screenshot of a browser that shows an OLFS internal error caused by the BES response. This indicates the BES is returning garbage, and the contents of this garbage really matters here.

If you would please take the time to track down the BES response document I would really appreciate it. If you don't know how to do that I can help you.

[Add Comment]<https://opendap.atlassian.net/browse/HK-96#add-comment?atlOrigin=eyJpIjoiZmY2Y2ExNmRjODU1NDM5ZGE1NGRhNDc5NGYwMTYwYTciLCJwIjoiaiJ9>

Add Comment<https://opendap.atlassian.net/browse/HK-96#add-comment?atlOrigin=eyJpIjoiZmY2Y2ExNmRjODU1NDM5ZGE1NGRhNDc5NGYwMTYwYTciLCJwIjoiaiJ9>

Get Jira notifications on your phone! Download the Jira Cloud app for Android<https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail> or iOS<https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8>

Kent Yang
October 15, 2018, 9:36 PM

Hi Nathan,

All I can say is that with BES command line, I can generate the DMR output as shown in the attachment. With the OLFS added, I got the error in the screenshot.
I briefed James and other folks in your developer meeting. My time working on this support is limited. I simply don’t have time to dig in more. This is the most I can do.

From: Nathan Potter (Jira) jira@opendap.atlassian.net
Sent: Monday, October 15, 2018 4:26 PM
To: Kent Yang
Subject: [JIRA] () Issues with ICESat-2 data.

https://avatar-cdn.atlassian.com/982957953e66bcab03fb675856e1df4e?s=48&d=https%3A%2F%2Fopendap.atlassian.net%2Fsecure%2Fuseravatar%3FownerId%3Dndp%26avatarId%3D10401%26noRedirect%3Dtrue

Nathan Potter<https://opendap.atlassian.net/secure/ViewProfile.jspa?name=ndp> commented on [Bug] HK-96<https://opendap.atlassian.net/browse/HK-96?atlOrigin=eyJpIjoiZmY2Y2ExNmRjODU1NDM5ZGE1NGRhNDc5NGYwMTYwYTciLCJwIjoiaiJ9>

Re: Issues with ICESat-2 data.<https://opendap.atlassian.net/browse/HK-96?atlOrigin=eyJpIjoiZmY2Y2ExNmRjODU1NDM5ZGE1NGRhNDc5NGYwMTYwYTciLCJwIjoiaiJ9>

The attached PDF file does not contain the error response returned by the BES. It contains an screenshot of a browser that shows an OLFS internal error caused by the BES response. This indicates the BES is returning garbage, and the contents of this garbage really matters here.

If you would please take the time to track down the BES response document I would really appreciate it. If you don't know how to do that I can help you.

[Add Comment]<https://opendap.atlassian.net/browse/HK-96#add-comment?atlOrigin=eyJpIjoiZmY2Y2ExNmRjODU1NDM5ZGE1NGRhNDc5NGYwMTYwYTciLCJwIjoiaiJ9>

Add Comment<https://opendap.atlassian.net/browse/HK-96#add-comment?atlOrigin=eyJpIjoiZmY2Y2ExNmRjODU1NDM5ZGE1NGRhNDc5NGYwMTYwYTciLCJwIjoiaiJ9>

Get Jira notifications on your phone! Download the Jira Cloud app for Android<https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail> or iOS<https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8>

Done

Assignee

Kent Yang

Reporter

James Gallagher

Labels

Fix versions

Time remaining

0m

Story Points

None

Epic Link

Priority

High