Add a BES Catalog that uses the CMR web api to build a virtual directory for NASA data in S3

Description

We can build on the BES Catalog system to make a virtual directory of NASA data stored in S3. This child of BESCatalog will use the CMR web api developed by the CMR team during PI 18.1 to find out about Items and Nodes in a (virtual) hierarchy of data. This will enable users to view data stored in S3, which is a flat data store, as if it is organized in a hierarchy.

Objective
Summary
Virtual Directory Maker

Description
Amazon's S3 is an object store, meaning data are represented using a simple key-value organization schemes.

Acceptance Criteria

  • A client can request the contents of a directory which will return a list of sub-directories of a list of files.

  • The root-most directory is for a single collection or a single provider.

  • 95% of all directory listing responses should be returned in less than a second.

  • A file listing should contain at a minimum:

    • Producer granule ID/original filename

    • URL to download the file

    • Size of the file

    • Last modified time

    • File type (optional - if present and reliable)

  • The initial directory hierarchy needed for OPeNDAP within a collection is year, month, day based on the temporal date of the data.

Assignee

Nathan Potter

Reporter

James Gallagher

Priority

Medium

Labels

Fix versions

None

Epic Name

CMR Virtual Directories