Search the message-bes file for these names to find four memory fault errors.
libbes_dispatch
Likely source of the error based on correlating the system log and bes log:
Jul 28 07:18:34 ip-172-31-47-159 kernel: beslistener[25094] general protection ip:7f76b35d90d4 sp:7ffd6adca180 error:0 in libbes_dispatch.so.15.3.2[7f76b35af000+c2000]
2019-07-28T07:18:34GMT|&|25094|&|get.ddx,dap2,/usr/share/hyrax/agg/TRMM_3B42_daily_Aggregation_1998.ncml
2019-07-28T07:18:34GMT|&|25094|&|ERROR: BES Internal Error: Unable to add container '__DDSLoader_Container_ID_5__agg/disc2.gesdisc.eosdis.nasa.gov/TRMM_L3/TRMM_3B42_daily/1997/365/3B42_daily.1998.01.01.7.bin', type of data must be specified (BESContainerStorageVolatile.cc:128)
Â
Other strings that show up in the /var/log/messages system log associated with various memory errors (mostly segmentation faults or general protection):
in beslistener
libnss
libdap
How to reproduce this error.
On the master branch (or HK 474), in modules/ncml_module/tests, run
besstandalone -c bes_unknown_type_aggregation_error.conf -i TRMM_3A11_Aggregation.ncml.dds-bescmd.xml
after building with --enable-asan (make sure to disable container overflow detection using: ASAN_OPTIONS=detect_container_overflow=0). This should show up in a debugger without ASAN on, as well.
And you will see:
Â
I think this is fixed. The DDSLoader code was generating an exception in-between the time when the DHI (including its container object) was hijacked and when the new container was added. This meant the original container was ‘released’ twice. Bad. Fixed DDSLoader so that the hijack process is robust if an exception is thrown by the code that makes the hijacking container.