Final combination of DMR++ Array retrieval optimizations (pthreads, parallel transmission)

Description

Combine the most effective set of optimizations from the dmrpp_profile branch.

The current branch seems to be the fastest.

Take the work done for read_chunk_unconstrained() and insert_chunk_unconstrained() and apply them to:

How can `DmrppArray::read_contiguous()` be made to read data in parallel? This amounts to replacing the Chunk::read_chunk() with something else (or using multi_curl in read_chunk()). The chunk 'inflate' operation is inherently serial. The chunk insertion could be parallelized, but that is not the most important part to optimize.

Activity

Show:
James Gallagher
August 22, 2018, 2:45 AM

Memory error using the curl handle pool with pthreads:

edamame:tests jimg$ time -p besstandalone -c bes.conf -i s3/airs_whole_local.dap > /dev/null

=================================================================

==62959==ERROR: AddressSanitizer: attempting double-free on 0x60d000197d70 in thread T256:

    #0 0x1081b9fdd in wrap_free (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x56fdd)

    #1 0x7fff582f23fe in Curl_setopt (libcurl.4.dylib:x86_64+0x153fe)

    #2 0x7fff583015fd in curl_easy_setopt (libcurl.4.dylib:x86_64+0x245fd)

    #3 0x10c5d3540 in dmrpp::CurlHandlePool::get_easy_handle(dmrpp::Chunk*) CurlHandlePool.cc:247

    #4 0x10c5c1943 in dmrpp::Chunk::read_chunk() Chunk.cc:487

    #5 0x10c64144a in dmrpp:\:process_one_chunk_unconstrained(dmrpp::Chunk*, dmrpp:\:DmrppArray*, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&) DmrppArray.cc:862

    #6 0x10c641dc3 in dmrpp::one_chunk_unconstrained_thread(void*) DmrppArray.cc:881

    #7 0x7fff59deb660 in _pthread_body (libsystem_pthread.dylib:x86_64+0x3660)

    #8 0x7fff59deb50c in _pthread_start (libsystem_pthread.dylib:x86_64+0x350c)

    #9 0x7fff59deabf8 in thread_start (libsystem_pthread.dylib:x86_64+0x2bf8)

0x60d000197d70 is located 0 bytes inside of 140-byte region [0x60d000197d70,0x60d000197dfc)

freed by thread T257 here:

    #0 0x1081b9fdd in wrap_free (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x56fdd)

    #1 0x7fff582f23fe in Curl_setopt (libcurl.4.dylib:x86_64+0x153fe)

    #2 0x7fff583015fd in curl_easy_setopt (libcurl.4.dylib:x86_64+0x245fd)

    #3 0x10c5d3540 in dmrpp::CurlHandlePool::get_easy_handle(dmrpp::Chunk*) CurlHandlePool.cc:247

    #4 0x10c5c1943 in dmrpp::Chunk::read_chunk() Chunk.cc:487

    #5 0x10c64144a in dmrpp:\:process_one_chunk_unconstrained(dmrpp::Chunk*, dmrpp:\:DmrppArray*, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&) DmrppArray.cc:862

    #6 0x10c641dc3 in dmrpp::one_chunk_unconstrained_thread(void*) DmrppArray.cc:881

    #7 0x7fff59deb660 in _pthread_body (libsystem_pthread.dylib:x86_64+0x3660)

    #8 0x7fff59deb50c in _pthread_start (libsystem_pthread.dylib:x86_64+0x350c)

    #9 0x7fff59deabf8 in thread_start (libsystem_pthread.dylib:x86_64+0x2bf8)

previously allocated by thread T241 here:

    #0 0x1081b4952 in wrap_strdup (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x51952)

    #1 0x7fff582f241d in Curl_setopt (libcurl.4.dylib:x86_64+0x1541d)

    #2 0x7fff583015fd in curl_easy_setopt (libcurl.4.dylib:x86_64+0x245fd)

    #3 0x10c5d3540 in dmrpp::CurlHandlePool::get_easy_handle(dmrpp::Chunk*) CurlHandlePool.cc:247

    #4 0x10c5c1943 in dmrpp::Chunk::read_chunk() Chunk.cc:487

    #5 0x10c64144a in dmrpp:\:process_one_chunk_unconstrained(dmrpp::Chunk*, dmrpp:\:DmrppArray*, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int> > const&) DmrppArray.cc:862

    #6 0x10c641dc3 in dmrpp::one_chunk_unconstrained_thread(void*) DmrppArray.cc:881

    #7 0x7fff59deb660 in _pthread_body (libsystem_pthread.dylib:x86_64+0x3660)

    #8 0x7fff59deb50c in _pthread_start (libsystem_pthread.dylib:x86_64+0x350c)

    #9 0x7fff59deabf8 in thread_start (libsystem_pthread.dylib:x86_64+0x2bf8)

Thread T256 created by T0 here:

    #0 0x1081b1e1d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4ee1d)

    #1 0x10c6450d1 in dmrpp:\:DmrppArray::read_chunks_unconstrained() DmrppArray.cc:927

    #2 0x10c64802f in dmrpp:\:DmrppArray::read() DmrppArray.cc:1081

Provided dSYM: [/Users/jimg/src/opendap/hyrax_git/build/lib/libdap.25.dylib.dSYM/Contents/Resources/DWARF/libdap.25.dylib] does not match symbol owner 0x7faa17512160

    #3 0x10bcd7711 in libdap::Vector::serialize(libdap:\:D4StreamMarshaller&, libdap:\:DMR&, bool) (libdap.25.dylib:x86_64+0x163711)

    #4 0x10c03fad9 in libdap:\:D4Group::serialize(libdap:\:D4StreamMarshaller&, libdap:\:DMR&, bool) (libdap.25.dylib:x86_64+0x4cbad9)

    #5 0x10b79154e in BESDapResponseBuilder::serialize_dap4_data(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1352

    #6 0x10b78f4d7 in BESDapResponseBuilder::send_dap4_data_using_ce(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1293

    #7 0x10b79024e in BESDapResponseBuilder::send_dap4_data(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1323

    #8 0x10b7705f3 in SendDap4Data::send_internal(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:302

    #9 0x10b7653a2 in Sender::send(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:80

    #10 0x10b763e84 in BESDapTransmit::send_basic_dap4data(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:365

Provided dSYM: [/Users/jimg/src/opendap/hyrax_git/build/lib/libbes_dispatch.15.dylib.dSYM/Contents/Resources/DWARF/libbes_dispatch.15.dylib] does not match symbol owner 0x7faa1750c960

    #11 0x107d53753 in BESTransmitter::send_response(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, BESResponseObject*, BESDataHandlerInterface&) (libbes_dispatch.15.dylib:x86_64+0x1aa753)

    #12 0x10b739d30 in BESDap4ResponseHandler::transmit(BESTransmitter*, BESDataHandlerInterface&) BESDap4ResponseHandler.cc:118

Provided dSYM: [/Users/jimg/src/opendap/hyrax_git/build/lib/libbes_xml_command.1.dylib.dSYM/Contents/Resources/DWARF/libbes_xml_command.1.dylib] does not match symbol owner 0x7faa1750cd60

    #13 0x107908d5d in BESXMLInterface::transmit_data() (libbes_xml_command.1.dylib:x86_64+0x17d5d)

    #14 0x1079041ed in BESXMLInterface::execute_data_request_plan() (libbes_xml_command.1.dylib:x86_64+0x131ed)

    #15 0x107bb35bd in BESInterface::execute_request(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (libbes_dispatch.15.dylib:x86_64+0xa5bd)

Provided dSYM: [/Users/jimg/src/opendap/hyrax_git/build/bin/besstandalone.dSYM/Contents/Resources/DWARF/besstandalone] does not match symbol owner 0x7faa1750c710

    #16 0x1078648d8 in StandAloneClient::executeCommand(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (besstandalone:x86_64+0x1000298d8)

    #17 0x1078623aa in StandAloneClient::executeCommands(std::__1::basic_ifstream<char, std::__1::char_traits<char> >&, int) (besstandalone:x86_64+0x1000273aa)

    #18 0x107847e6d in StandAloneApp::run() (besstandalone:x86_64+0x10000ce6d)

    #19 0x107e24a04 in BESApp::main(int, char**) BESApp.cc:58

    #20 0x10784b12e in main (besstandalone:x86_64+0x10001012e)

    #21 0x7fff59ad3014 in start (libdyld.dylib:x86_64+0x1014)

Thread T257 created by T0 here:

    #0 0x1081b1e1d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4ee1d)

    #1 0x10c6450d1 in dmrpp:\:DmrppArray::read_chunks_unconstrained() DmrppArray.cc:927

    #2 0x10c64802f in dmrpp:\:DmrppArray::read() DmrppArray.cc:1081

    #3 0x10bcd7711 in libdap::Vector::serialize(libdap:\:D4StreamMarshaller&, libdap:\:DMR&, bool) (libdap.25.dylib:x86_64+0x163711)

    #4 0x10c03fad9 in libdap:\:D4Group::serialize(libdap:\:D4StreamMarshaller&, libdap:\:DMR&, bool) (libdap.25.dylib:x86_64+0x4cbad9)

    #5 0x10b79154e in BESDapResponseBuilder::serialize_dap4_data(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1352

    #6 0x10b78f4d7 in BESDapResponseBuilder::send_dap4_data_using_ce(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1293

    #7 0x10b79024e in BESDapResponseBuilder::send_dap4_data(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1323

    #8 0x10b7705f3 in SendDap4Data::send_internal(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:302

    #9 0x10b7653a2 in Sender::send(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:80

    #10 0x10b763e84 in BESDapTransmit::send_basic_dap4data(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:365

    #11 0x107d53753 in BESTransmitter::send_response(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, BESResponseObject*, BESDataHandlerInterface&) (libbes_dispatch.15.dylib:x86_64+0x1aa753)

    #12 0x10b739d30 in BESDap4ResponseHandler::transmit(BESTransmitter*, BESDataHandlerInterface&) BESDap4ResponseHandler.cc:118

    #13 0x107908d5d in BESXMLInterface::transmit_data() (libbes_xml_command.1.dylib:x86_64+0x17d5d)

    #14 0x1079041ed in BESXMLInterface::execute_data_request_plan() (libbes_xml_command.1.dylib:x86_64+0x131ed)

    #15 0x107bb35bd in BESInterface::execute_request(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (libbes_dispatch.15.dylib:x86_64+0xa5bd)

    #16 0x1078648d8 in StandAloneClient::executeCommand(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (besstandalone:x86_64+0x1000298d8)

    #17 0x1078623aa in StandAloneClient::executeCommands(std::__1::basic_ifstream<char, std::__1::char_traits<char> >&, int) (besstandalone:x86_64+0x1000273aa)

    #18 0x107847e6d in StandAloneApp::run() (besstandalone:x86_64+0x10000ce6d)

    #19 0x107e24a04 in BESApp::main(int, char**) BESApp.cc:58

    #20 0x10784b12e in main (besstandalone:x86_64+0x10001012e)

    #21 0x7fff59ad3014 in start (libdyld.dylib:x86_64+0x1014)

Thread T241 created by T0 here:

    #0 0x1081b1e1d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4ee1d)

    #1 0x10c6450d1 in dmrpp:\:DmrppArray::read_chunks_unconstrained() DmrppArray.cc:927

    #2 0x10c64802f in dmrpp:\:DmrppArray::read() DmrppArray.cc:1081

    #3 0x10bcd7711 in libdap::Vector::serialize(libdap:\:D4StreamMarshaller&, libdap:\:DMR&, bool) (libdap.25.dylib:x86_64+0x163711)

    #4 0x10c03fad9 in libdap:\:D4Group::serialize(libdap:\:D4StreamMarshaller&, libdap:\:DMR&, bool) (libdap.25.dylib:x86_64+0x4cbad9)

    #5 0x10b79154e in BESDapResponseBuilder::serialize_dap4_data(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1352

    #6 0x10b78f4d7 in BESDapResponseBuilder::send_dap4_data_using_ce(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1293

    #7 0x10b79024e in BESDapResponseBuilder::send_dap4_data(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, libdap:\:DMR&, bool) BESDapResponseBuilder.cc:1323

    #8 0x10b7705f3 in SendDap4Data::send_internal(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:302

    #9 0x10b7653a2 in Sender::send(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:80

    #10 0x10b763e84 in BESDapTransmit::send_basic_dap4data(BESResponseObject*, BESDataHandlerInterface&) BESDapTransmit.cc:365

    #11 0x107d53753 in BESTransmitter::send_response(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, BESResponseObject*, BESDataHandlerInterface&) (libbes_dispatch.15.dylib:x86_64+0x1aa753)

    #12 0x10b739d30 in BESDap4ResponseHandler::transmit(BESTransmitter*, BESDataHandlerInterface&) BESDap4ResponseHandler.cc:118

    #13 0x107908d5d in BESXMLInterface::transmit_data() (libbes_xml_command.1.dylib:x86_64+0x17d5d)

    #14 0x1079041ed in BESXMLInterface::execute_data_request_plan() (libbes_xml_command.1.dylib:x86_64+0x131ed)

    #15 0x107bb35bd in BESInterface::execute_request(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (libbes_dispatch.15.dylib:x86_64+0xa5bd)

    #16 0x1078648d8 in StandAloneClient::executeCommand(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (besstandalone:x86_64+0x1000298d8)

    #17 0x1078623aa in StandAloneClient::executeCommands(std::__1::basic_ifstream<char, std::__1::char_traits<char> >&, int) (besstandalone:x86_64+0x1000273aa)

    #18 0x107847e6d in StandAloneApp::run() (besstandalone:x86_64+0x10000ce6d)

    #19 0x107e24a04 in BESApp::main(int, char**) BESApp.cc:58

    #20 0x10784b12e in main (besstandalone:x86_64+0x10001012e)

    #21 0x7fff59ad3014 in start (libdyld.dylib:x86_64+0x1014)

SUMMARY: AddressSanitizer: double-free (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x56fdd) in wrap_free

==62959==ABORTING

James Gallagher
August 22, 2018, 3:01 AM

Must protect void CurlHandlePool::release_handle(dmrpp_easy_handle *handle) and dmrpp_easy_handle *
CurlHandlePool::get_easy_handle(Chunk *chunk) using a mutex and a condition variable.

Done

Assignee

Unassigned

Reporter

James Gallagher

Labels

None

Epic Link

Components

Sprint

None

Fix versions

Priority

Medium