There are a number of segmentation faults in the DMR++ software for large runs

Description

Here are the places where the handler running on C7 fails:

Likely destructor fail:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 /lib64/libc.so.6(+0x81499)[0x7fe32dadc499] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5ChunkD2Ev+0x3c)[0x7fe323f741fa] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZSt8_DestroyIN5dmrpp5ChunkEEvPT_+0x1f)[0x7fe323f769ed] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZNSt12_Destroy_auxILb0EE9__destroyIPN5dmrpp5ChunkEEEvT_S5_+0x26)[0x7fe323f765f\ b] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZSt8_DestroyIPN5dmrpp5ChunkEEvT_S3_+0x23)[0x7fe323f75dd7] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZSt8_DestroyIPN5dmrpp5ChunkES1_EvT_S3_RSaIT0_E+0x27)[0x7fe323f75139] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZNSt6vectorIN5dmrpp5ChunkESaIS1_EED2Ev+0x36)[0x7fe323f747a6] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp11DmrppCommonD1Ev+0x2f)[0x7fe323f7431d] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp10DmrppArrayD2Ev+0x4a)[0x7fe323f831da] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp10DmrppArrayD0Ev+0x18)[0x7fe323f83242] /home/centos/hyrax/build/lib/libdap.so.25(_ZN6libdap3DDSD2Ev+0x3e)[0x7fe3291bc67e] /home/centos/hyrax/build/lib/libdap.so.25(_ZN6libdap3DDSD0Ev+0x9)[0x7fe3291bc939] /home/centos/hyrax/build/lib/bes/libdap_module.so(_ZN18BESDataDDSResponseD1Ev+0x4b)[0x7fe3299a7ccf] /home/centos/hyrax/build/lib/bes/libdap_module.so(_ZN18BESDataDDSResponseD0Ev+0x18)[0x7fe3299a7d50] /home/centos/hyrax/build/lib/libbes_dispatch.so.15(_ZN18BESResponseHandlerD2Ev+0x4b)[0x7fe32f9631ed] /home/centos/hyrax/build/lib/bes/libdap_module.so(_ZN22BESDataResponseHandlerD1Ev+0x3b)[0x7fe32999934d] /home/centos/hyrax/build/lib/bes/libdap_module.so(_ZN22BESDataResponseHandlerD0Ev+0x18)[0x7fe32999939e] /home/centos/hyrax/build/lib/libbes_dispatch.so.15(_ZN23BESDataHandlerInterface5cleanEv+0x45)[0x7fe32f98a0b9] /home/centos/hyrax/build/lib/libbes_xml_command.so.1(_ZN15BESXMLInterface5cleanEv+0x1b5)[0x7fe32f68248f] /home/centos/hyrax/build/lib/libbes_xml_command.so.1(_ZN15BESXMLInterfaceD1Ev+0x2b)[0x7fe32f67f2bf]

Two dmrpp_easy_handle::read_data() calls:

1 2 3 4 5 /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp17dmrpp_easy_handle9read_dataEv+0x27)[0x7fe323f7973b] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk10read_chunkEv+0x230)[0x7fe323f78032] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp31process_one_chunk_unconstrainedEPNS_5ChunkEPNS_10DmrppArrayERKSt6vec\ torIjSaIjEES8_+0x36)[0x7fe323f804fa] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp30one_chunk_unconstrained_threadEPv+0x3f)[0x7fe323f8072a]
1 2 3 4 5 6 7 /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp17dmrpp_easy_handle9read_dataEv+0x27)[0x7fe323f7973b] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk10read_chunkEv+0x230)[0x7fe323f78032] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp31process_one_chunk_unconstrainedEPNS_5ChunkEPNS_10DmrppArrayERKSt6vec\ torIjSaIjEES8_+0x36)[0x7fe323f804fa] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp30one_chunk_unconstrained_threadEPv+0x3f)[0x7fe323f8072a] /lib64/libpthread.so.0(+0x7e25)[0x7fe32ee26e25] /lib64/libc.so.6(clone+0x6d)[0x7fe32db59bad]

The rest are calls to Chunk::set_rbuf()

1 2 3 4 7fe31f204000-7fe31f208000 r--p 000c2000 ca:01 /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk8set_rbufEPcj+0x30\ )[0x7fe323f78a0a] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk13inflate_chunkEbbjj+0xaf)[0x7fe323f77ce1] *** Error in `/home/centos/hyrax/build/bin/beslistener': realloc(): invalid old size: 0x00007fe3080dd170 ***
1 2 3 4 5 /lib64/libc.so.6(+0x7f5e4)[0x7fe32dada5e4] /lib64/libc.so.6(+0x816db)[0x7fe32dadc6db] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk8set_rbufEPcj+0x30)[0x7fe323f78a0a] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk13inflate_chunkEbbjj+0xaf)[0x7fe323f77ce1] *** Error in `/home/centos/hyrax/build/bin/beslistener': double free or corruption (out): 0x00007fe3083829d0 ***
1 2 3 4 5 6 7 8 /lib64/libc.so.6(+0x81499)[0x7fe32dadc499] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk8set_rbufEPcj+0x30)[0x7fe323f78a0a] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk13inflate_chunkEbbjj+0x159)[0x7fe323f77d8b] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp31process_one_chunk_unconstrainedEPNS_5ChunkEPNS_10DmrppArrayERKSt6vec\ torIjSaIjEES8_+0x18d)[0x7fe323f80651] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp30one_chunk_unconstrained_threadEPv+0x3f)[0x7fe323f8072a] /lib64/libpthread.so.0(+0x7e25)[0x7fe32ee26e25] /lib64/libc.so.6(clone+0x6d)[0x7fe32db59bad]
1 2 3 4 5 6 7 8 /lib64/libc.so.6(+0x81499)[0x7fe32dadc499] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk8set_rbufEPcj+0x30)[0x7fe323f78a0a] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk13inflate_chunkEbbjj+0xaf)[0x7fe323f77ce1] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp31process_one_chunk_unconstrainedEPNS_5ChunkEPNS_10DmrppArrayERKSt6vec\ torIjSaIjEES8_+0x18d)[0x7fe323f80651] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp30one_chunk_unconstrained_threadEPv+0x3f)[0x7fe323f8072a] /lib64/libpthread.so.0(+0x7e25)[0x7fe32ee26e25] /lib64/libc.so.6(clone+0x6d)[0x7fe32db59bad]
1 2 3 4 5 6 7 8 /lib64/libc.so.6(+0x81499)[0x7f844fc23499] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk8set_rbufEPcj+0x30)[0x7f84460bfa0a] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp5Chunk13inflate_chunkEbbjj+0xaf)[0x7f84460bece1] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp31process_one_chunk_unconstrainedEPNS_5ChunkEPNS_10DmrppArrayERKSt6vec\ torIjSaIjEES8_+0x18d)[0x7f84460c7651] /home/centos/hyrax/build/lib/bes/libdmrpp_module.so(_ZN5dmrpp30one_chunk_unconstrained_threadEPv+0x3f)[0x7f84460c772a] /lib64/libpthread.so.0(+0x7e25)[0x7f8450f6de25] /lib64/libc.so.6(clone+0x6d)[0x7f844fca0bad]

Environment

None

Status

Assignee

James Gallagher

Reporter

James Gallagher

Labels

None

Fix versions

Story Points

None

Affects versions

Hyrax 1.15.1

Epic Link

Priority

High
Configure