MDS bug - LMT of data not used

Description

Check LMT info for a dataset. The MDS does not look at the underlying data (file) to check for changes.

Modify the handlers that work with the local file system to return whether or not the item is stale. (Stale == LMT test)

Modify handlers that work with remote data to return whether or not the item is stale. (Stale == HTTP/1.1 caching tests)

However, some items in the MDS are put there by an external process and should not be tested for stale-ness because the MDS is a store for some things (and a cache in other cases).

In the current implementation, the MDS is either and cache or a store, not both. We should take this into consideration and probably make a story for this.

Files with 'add_method(...)' (I removed the handlers that don't read from data sources - jg) :

  • CSVRequestHandler.cc ------------- (comma separated values)

  • FitsRequestHandler.cc -------------- (Flexible Image Transport System)

  • FFRequestHandler.cc --------------- (FreeForm - ascii and binary data from various sources, described by a 'format' file)

  • GDALRequestHandler.cc ---------- (geospatial data, generally GeoTiff, but GDAL can be configured to read from lots of different sources)

  • DmrppRequestHandler.cc -------- (Read data from Amazon's S3 web object store - it actually is much more general and can read from any web object store)

  • NCRequestHandler.cc -------------- (netCDF files)

Files with 'add_handler(...)' :

  • HDF4RequestHandler.cc ---------- HDF4 data files

  • HDF5RequestHandler.cc ---------- HDF5 data files

Handlers I'm not sure about:

  • SQLHandler (not part of the regular build)

  • NCMLHandler (I have to think about just what using cached objects means for this code)

Environment

None

Activity

Show:
James Gallagher
June 3, 2019, 11:27 PM

Another take on the short plan:

Modify the class BESRequestHandler so that it holds a simple function that will return a LMT using a time_t value. Include an implementation for this function in the BESRequestHandler class that will work for any file on the BES's local file system.

For some of the handlers listed above, data are read from places other than a local file system so the generic method won't work. Provide a specialization for it in those handler's specialization of BESRequestHandler. For now, lets ignore those handlers. We can add methods that will always return a time older than the cached response to force the cache to accept its current entry as up-to-date, or conversely always return one younger to force a refresh always.

The best way to get the information about the handler to use is to use the BESContainer object - Every file the BES read from is modeled as a 'container'. The class BESContainer implements the basic interface for this and it contains code that returns the type of the thing in the container (generally a file). This type information can be used along with the find_handler() method of the BESRequestHandlerList singleton to get a pointer to the Specialization of the BESRequestHandler for the particular data set (e.g., file).

We can put this together in a few ways:

  1. Modify the GlobalMetadataStore::is_dds_available(...) (and similar methods for the das, dmr, ...) so that it performs the test and returns false if the cache entry is too old

  2. Put the modification into the code that calls the GlobalMetadataStore::is_dds_available(...) methods (see void BESDDSResponseHandler::execute(BESDataHandlerInterface &dhi))

If we chose the first option, we will need to expand on the arguments that GlobalMetadataStore::is_dds_available(...) takes. Maybe pass in the BESContainer? Maybe extract some more info from it?

If we go with option #2, we could get the LMT time in void BESDDSResponseHandler::execute(BESDataHandlerInterface &dhi) and then pass that into GlobalMetadataStore::is_dds_available(...) (se we'd still need to add to the method's signature, but only to pass in something pretty simple. The dependency on classes defined in bes/dispatch would be avoided.

Note that I am just using the is_dds_avaialable() and BESDDSResponseHandler::execute(...) as examples. To implement this, the matching methods for the DAS and DMR will need to be modified.

And, there is a second implementation of the MDS in the bes/modules/ dmrpp_handler that will need to be fixed up too.

Assignee

Sam Lloyd

Reporter

James Gallagher

Labels

Fix versions

Time remaining

0m

Story Points

2

Affects versions

Epic Link

Priority

Medium
Configure