Search the message-bes file for these names to find four memory fault errors.
Likely source of the error based on correlating the system log and bes log:
Jul 28 07:18:34 ip-172-31-47-159 kernel: beslistener general protection ip:7f76b35d90d4 sp:7ffd6adca180 error:0 in libbes_dispatch.so.15.3.2[7f76b35af000+c2000]
2019-07-28T07:18:34GMT|&|25094|&|ERROR: BES Internal Error: Unable to add container '__DDSLoader_Container_ID_5__agg/disc2.gesdisc.eosdis.nasa.gov/TRMM_L3/TRMM_3B42_daily/1997/365/3B42_daily.1998.01.01.7.bin', type of data must be specified (BESContainerStorageVolatile.cc:128)
Other strings that show up in the /var/log/messages system log associated with various memory errors (mostly segmentation faults or general protection):
How to reproduce this error.
On the master branch (or HK 474), in modules/ncml_module/tests, run
besstandalone -c bes_unknown_type_aggregation_error.conf -i TRMM_3A11_Aggregation.ncml.dds-bescmd.xml
after building with --enable-asan (make sure to disable container overflow detection using: ASAN_OPTIONS=detect_container_overflow=0). This should show up in a debugger without ASAN on, as well.
And you will see:
I think this is fixed. The DDSLoader code was generating an exception in-between the time when the DHI (including its container object) was hijacked and when the new container was added. This meant the original container was ‘released’ twice. Bad. Fixed DDSLoader so that the hijack process is robust if an exception is thrown by the code that makes the hijacking container.