Add a BES Catalog that uses the CMR web api to build a virtual directory for NASA data in S3


We can build on the BES Catalog system to make a virtual directory of NASA data stored in S3. This child of BESCatalog will use the CMR web api developed by the CMR team during PI 18.1 to find out about Items and Nodes in a (virtual) hierarchy of data. This will enable users to view data stored in S3, which is a flat data store, as if it is organized in a hierarchy.

Virtual Directory Maker

Amazon's S3 is an object store, meaning data are represented using a simple key-value organization schemes.

Acceptance Criteria

  • A client can request the contents of a directory which will return a list of sub-directories of a list of files.

  • The root-most directory is for a single collection or a single provider.

  • 95% of all directory listing responses should be returned in less than a second.

  • A file listing should contain at a minimum:

    • Producer granule ID/original filename

    • URL to download the file

    • Size of the file

    • Last modified time

    • File type (optional - if present and reliable)

  • The initial directory hierarchy needed for OPeNDAP within a collection is year, month, day based on the temporal date of the data.


Nathan Potter


James Gallagher




Fix versions


Epic Name

CMR Virtual Directories