FreeForm: As a data provider, I would like to specify a regex for matching format file to data files because the one-to-one relation does not fit our data well.

Description

Hi James, thanks much for looking at this. I was hoping this might work,
but in the interim did as you suggest and made a parallel directory
structure of links to the individual files in our dataset directory.
Then we place .fmt files either in the parent directory of each dataset,
or next to each file if a directory contains files with different
formats. This should be workable for us, but not ideal.

What would be ideal in the FreeForm handler is a configuration option
which would allow us to specify a filename pattern (regex) and matching
format filename. E.g., the script I wrote that populates this parallel
directory tree does this; here is a sample set of entries for one of our
datasets:

tb_.*n(19|22|37)(h|v).bin: n_low.fmt
tb_.*n(85|91)(h|v).bin: n_high.fmt
tb_.*s(19|22|37)(h|v).bin: s_low.fmt
tb_.*s(85|91)(h|v).bin: s_high.fmt

These match various combinations of hemisphere, frequency, and
polarization with a corresponding format file.

In any case, it would be cool if the FreeForm handler config allowed
this kind of matching between filename and format file. Then you could
have a directory of format files located independent of the actual
datasets and configure the relationship between them using something
like this key/value representation above. I don't think this is possible
in the current configuration , but it would be nice to find out I was
wrong!

Thanks for your help,
Kevin Beam
NSIDC

Activity

Show:
James Gallagher
November 8, 2017, 11:49 PM

I think this might be one way to address the issue of just how we support S3 as a data store for stuff accessed using freeform.

Slav Korolev
April 17, 2019, 9:10 PM
Edited

Data from https://nsidc.org/data/nsidc-0080 is used for testing.

According to description in https://daacdata.apps.nsidc.org/pub/DATASETS/nsidc0080_nrt_polar_stereo_tbs/ :"Data are stored as scaled 2-byte integers representing brightness temperature values (in tenths of a kelvin), ranging from 50 K to 350 K. The brightness temperatures are multiplied by a factor of ten so that the precision of the data and the units in the data file are tenths of a kelvin. For example, a stored integer value of 2358 represents a brightness temperature value of 235.8 K. A value of 0 represents missing data."

Format file (n_low.fmt etc.) which support such data could be like this:

// 2-byte integers representing brightness temperature values (in tenths of a kelvin)

binary_input_data "two_byte_int_data"temperature 1 2 ARRAY["stuff" 1 to 1000] of int16 0

ASCII_output_data "brightness in K * 10"temperature 1 4 short 0

Because file format is the same for all nsidc-0080 data the format files in this test will be identical for any regular expression.Here are regular expressions and data files which are matching them:

tb_.*n(19|22|37)(h|v).bin: n_low.fmt // North verticalhttps://daacdata.apps.nsidc.org/pub/DATASETS/nsidc0080_nrt_polar_stereo_tbs/tb_f18_20190306_nrt_n19v.bin

tb_.*n(85|91)(h|v).bin: n_high.fmt // North verticalhttps://daacdata.apps.nsidc.org/pub/DATASETS/nsidc0080_nrt_polar_stereo_tbs/tb_f18_20190306_nrt_n91v.bin

tb_.*s(19|22|37)(h|v).bin: s_low.fmt //South verticalhttps://daacdata.apps.nsidc.org/pub/DATASETS/nsidc0080_nrt_polar_stereo_tbs/tb_f18_20190306_nrt_s19v.bin

tb_.*s(85|91)(h|v).bin: s_high.fmt //South horizontalhttps://daacdata.apps.nsidc.org/pub/DATASETS/nsidc0080_nrt_polar_stereo_tbs/tb_f17_20130101_nrt_s91h.bin

To get working FreeForm data in OPeNDAP Hyrax the following steps should be done:

Add the following lines into <full-pass-to>/hyrax/build/etc/bes/site.conf:

In the file <full-pass-to>/hyrax/build/etc/bes/modules/ff.conf edit line:

Put binary and format files into <full-pass-to>/hyrax/build/share/hyrax/data/ff folder.

Goto http://localhost:8080/opendap/data/ff/contents.html

Done

Assignee

Slav Korolev

Reporter

James Gallagher

Priority

Medium

Labels

Story Points

None

Fix versions

Epic Link

None

Time remaining

0m
Configure