Ingest datasets metadata into MDS, upload datasets to S3.


Precondition: There is a collection of HDF5 files in a POSIX file system that Hyrax can access.

Postcondition: Those files are moved to S3 and users can access the data in those files using Hyrax in exactly the same way as when those files were in the POSIX file system.

The software must:

  1. Move data to S3

  2. Put DMR++ responses in the MDS using pathname that match POSIX filesystem.

    1. Put all the metadata responses in the MDS - this should be done in the 'Streamline MDS population' story

Implement this by adding functionality to the scripts/utility(ies) that are developed to 'populate the MDS.'

  1. Use the site map command to find the files to operate on

    1. For each file, populate MDS and move to S3

  2. Write the path of each file successfully operated on into a file

OLD text:
Given that some dataset foo.h5 has DDS, ..., DMR++ responses in the MDS, and is now only stored in S3, how does a user know about foo.h5?

A design for this involves three things:

  1. We need access to files to build the DMR++ response.

  2. Users need to reference these datasets using some path, and we have to be able to show that path to the users, even though the data files are no longer on the POSIX file system where the server initially read them to build the DMR++ responses.

  3. Users will provide the pathnames they are shown when they make requests and the MDS will use those paths (really strings, which can be arbitrary) to look for different responses.

The structure we show users for the data must mirror the structure used in the MDS.

The software that does this should also be responsible for moving the data files to S3. This means that it will use the S3 access URL when it writes the DMR++ response that is added to the MDS.

Your pinned fields
Click on the next to a field label to start pinning.


Nathan Potter


James Gallagher