Add a BES Catalog that uses the CMR web api to build a virtual directory for NASA data in S3
We can build on the BES Catalog system to make a virtual directory of NASA data stored in S3. This child of BESCatalog will use the CMR web api developed by the CMR team during PI 18.1 to find out about Items and Nodes in a (virtual) hierarchy of data. This will enable users to view data stored in S3, which is a flat data store, as if it is organized in a hierarchy.
Virtual Directory Maker
Amazon's S3 is an object store, meaning data are represented using a simple key-value organization schemes.
A client can request the contents of a directory which will return a list of sub-directories of a list of files.
The root-most directory is for a single collection or a single provider.
95% of all directory listing responses should be returned in less than a second.
A file listing should contain at a minimum:
Producer granule ID/original filename
URL to download the file
Size of the file
Last modified time
File type (optional - if present and reliable)
The initial directory hierarchy needed for OPeNDAP within a collection is year, month, day based on the temporal date of the data.