Persistent leaks in the libxml2-based NCML parser.

Description

Calling xmlFreeParserCtxt(_context); works for the DMR and DMR++ parsers, but not for the NcML parser. In fact, we might be getting lucky with the first two. This post suggests that making a parser context might not be the best overall plan.

Thanks for reporting this problem.

In fact the good solution is probably to use xmlSAXUserParseFile and
xmlSAXUserParseMemory to create the contexts, and not
xmlCreateMemoryParserCtxt and xmlCreateFileParserCtxt which are to be used
only for DOM.

So a bit more research is needed. If the above is true, fix the leak in
void SaxParserWrapper::setupParser(const string& filename) that comes fromxmlFreeParserCtxt(_context);

 

Here’s info about the leak:

{code}

256 bytes in 1 blocks are definitely lost in loss record 507 of 651
==5734== at 0x4C29BC3: malloc (vg_replace_malloc.c:299)
==5734== by 0x66A096C: xmlInitParserCtxt (in /usr/lib64/libxml2.so.2.9.1)
==5734== by 0x66A0DAB: xmlNewParserCtxt (in /usr/lib64/libxml2.so.2.9.1)
==5734== by 0x66B47BD: xmlCreateURLParserCtxt (in /usr/lib64/libxml2.so.2.9.1)
==5734== by 0xE6775A5: ncml_module::SaxParserWrapper::setupParser(std::string const&) (SaxParserWrapper.cc:480)
==5734== by 0xE676D5D: ncml_module::SaxParserWrapper::parse(std::string const&) (SaxParserWrapper.cc:328)
==5734== by 0xE62E68C: ncml_module::NCMLParser::parseInto(std::string const&, agg_util::DDSLoader::ResponseType, BESDapResponse*) (NCMLParser.cc:185)
==5734== by 0xE646E92: ncml_module::NCMLRequestHandler::ncml_build_data(BESDataHandlerInterface&) (NCMLRequestHandler.cc:297)
==5734== by 0x4F16B71: BESRequestHandlerList::execute_current(BESDataHandlerInterface&) (BESRequestHandlerList.cc:273)
==5734== by 0x4F16691: BESRequestHandlerList::execute_each(BESDataHandlerInterface&) (BESRequestHandlerList.cc:167)
==5734== by 0xC30B1F3: BESDataResponseHandler::execute(BESDataHandlerInterface&) (BESDataResponseHandler.cc:148)
==5734== by 0x61FAB49: BESXMLInterface::execute_data_request_plan() (BESXMLInterface.cc:285)

{code}

Environment

None

Activity

Show:
James Gallagher
June 21, 2019, 5:36 PM

I bet to completely clean up libxml2, given that the NcML parsers now use xmlSAXUserParseFile, we need to arrange a call to void xmlCleanupParser (void) using at_exit().

James Gallagher
June 21, 2019, 7:52 PM

Using xmlSAXUserParseFile() removes the above leak rooted in xmlSAXUserParseFile, but this remains, even though when I made other fixes in Dmrpp parser and D4Parser, the equivalent leaks went away.

{code}

==16822== HEAP SUMMARY:

==16822== in use at exit: 738,792 bytes in 2,871 blocks

==16822== total heap usage: 4,082,359 allocs, 4,079,488 frees, 163,582,150 bytes allocated

==16822==

==16822== 72 bytes in 1 blocks are definitely lost in loss record 264 of 668

==16822== at 0x4C2B955: calloc (vg_replace_malloc.c:711)

==16822== by 0x87CA443: ??? (in /usr/lib64/libnss3.so)

==16822== by 0x87CA4E8: ??? (in /usr/lib64/libnss3.so)

==16822== by 0x87CA0A8: ??? (in /usr/lib64/libnss3.so)

==16822== by 0x87BEFAA: ??? (in /usr/lib64/libnss3.so)

==16822== by 0x87C3755: ??? (in /usr/lib64/libnss3.so)

==16822== by 0x876DDF4: ??? (in /usr/lib64/libnss3.so)

==16822== by 0x876E630: NSS_InitContext (in /usr/lib64/libnss3.so)

==16822== by 0x6A14551: ??? (in /usr/lib64/libcurl.so.4.3.0)

==16822== by 0x6A14A2F: ??? (in /usr/lib64/libcurl.so.4.3.0)

==16822== by 0x6A0BAED: ??? (in /usr/lib64/libcurl.so.4.3.0)

==16822== by 0x69E2DDC: ??? (in /usr/lib64/libcurl.so.4.3.0)

==16822==

==16822== 72 bytes in 1 blocks are definitely lost in loss record 265 of 668

==16822== at 0x4C2B955: calloc (vg_replace_malloc.c:711)

==16822== by 0x1C90AB03: ??? (in /usr/lib64/libnsspem.so)

==16822== by 0x1C90ABA8: ??? (in /usr/lib64/libnsspem.so)

==16822== by 0x1C90A748: ??? (in /usr/lib64/libnsspem.so)

==16822== by 0x1C8FD1AE: ??? (in /usr/lib64/libnsspem.so)

==16822== by 0x1C9026D1: ??? (in /usr/lib64/libnsspem.so)

==16822== by 0x8793D2E: ??? (in /usr/lib64/libnss3.so)

==16822== by 0x8794359: ??? (in /usr/lib64/libnss3.so)

==16822== by 0x87A0E07: SECMOD_LoadModule (in /usr/lib64/libnss3.so)

==16822== by 0x87A0FC8: SECMOD_LoadUserModule (in /usr/lib64/libnss3.so)

==16822== by 0x6A150DA: ??? (in /usr/lib64/libcurl.so.4.3.0)

==16822== by 0x6A0BAED: ??? (in /usr/lib64/libcurl.so.4.3.0)

==16822==

==16822== LEAK SUMMARY:

==16822== definitely lost: 144 bytes in 2 blocks

==16822== indirectly lost: 0 bytes in 0 blocks

==16822== possibly lost: 176,623 bytes in 428 blocks

==16822== still reachable: 562,025 bytes in 2,441 blocks

==16822== of which reachable via heuristic:

==16822== stdstring : 1,600 bytes in 46 blocks

==16822== length64 : 166,608 bytes in 434 blocks

==16822== suppressed: 0 bytes in 0 blocks

==16822== Reachable blocks (those to which a pointer was found) are not shown.

==16822== To see them, rerun with: --leak-check=full --show-leak-kinds=all

{code}

James Gallagher
October 10, 2019, 12:13 AM

This code fails tests on CentOS7.

Also, the function SaxParserWrapper::getCurrentParseLine() const only ever returns -1 given the fix that we stopped using the 'context' calls from the libxml2 library and not use xmlSAXUserParseFile().

James Gallagher
October 24, 2019, 4:53 PM
Edited

Look at/for leaks, I see this:

 

How I got this:

  1. Running on a new Centos7 host with a fresh source build from the master branch of hyrax-dependencies, libdap4 and bes.

  2. valgrind --leak-check=yes besstandalone -c bes.conf -i fnoc1_improved.ncml.bescmd

  3. I built the _fnoc1_improved.ncml.bescmd_ file using: ./generate_bescmd.sh fnoc1_improved.ncml das fnoc1_improved.ncml.bescmd

Clearly, there’s quite a bit going on here… The next step would be to checkout the branch and see how it fairs. I’ll shoot for that, but might not get to it soon.

 

Done

Assignee

Sam Lloyd

Reporter

James Gallagher

Labels

Fix versions

Time remaining

0m

Story Points

1

Epic Link

Priority

Medium
Configure