manticore server crash on query - not consistent, unable to reproduce yet

[popalot2]

Bug Description:

Manticore 9.3.2 428075261@25050208 (columnar 4.2.1 0fe99f4@25050206) (secondary 4.2.1 0fe99f4@25050206) (knn 4.2.1 0fe99f4@25050206)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)```

OS: almalinux 9
Server crashed on query and got stuck - didn't terminate without kill -9
Plain distributed index
Will keep monitoring and will try to provide a reproducible case, crash is not consistent, 99.999% of queries work and rerunning the queries from the log didn't reproduce the crash. 
Perhaps a race condition or memory corruption.
Don't have a core dump currently.

```------- FATAL: CRASH DUMP -------
[Sun May 18 03:51:46.572 2025] [489688]

--- crashed SphinxAPI request dump ---
AAABJgAAufgAAAAYAAAABAAABFgAAAAAAAATugAAAAYAAAAAAAAABAAAAAdpZCBkZXNjAAAAFUBzdWJ0aXRsZXMg
ImhhIG1vcnNvIgAAAAAAACqzaWR4X3N1YnRpdGxlc19jb250ZW50X3Byb2RfYmlnX2VuXzRfMCxpZHhf
c3VidGl0bGVzX2NvbnRlbnRfcHJvZF9iaWdfZW5fNF8xLGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9k
X2JpZ19lbl80XzIsaWR4X3N1YnRpdGxlc19jb250ZW50X3Byb2RfYmlnX2VuXzRfMyxpZHhfc3VidGl0
bGVzX2NvbnRlbnRfcHJvZF9iaWdfZW5fNF80LGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2Jp
Z19lbl80XzUsaWR4X3N1YnRpdGxlc19jb250ZW50X3Byb2RfYmlnX2VuXzRfNixpZHhfc3VidGl0bGVz
X2NvbnRlbnRfcHJvZF9iaWdfZW5fNF83LGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2JpZ19lbl81
XzAsaWR4X3N1YnRpdGxlc19jb250ZW50X3Byb2RfYmlnX2VuXzVfMSxpZHhfc3VidGl0bGVzX2NvbnRl
bnRfcHJvZF9iaWdfZW5fNV8yLGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2JpZ19lbl81XzMs
aWR4X3N1YnRpdGxlc19jb250ZW50X3Byb2RfYmlnX2VuXzVfNCxpZHhfc3VidGl0bGVzX2NvbnRlbnRf
cHJvZF9iaWdfZW5fNV81LGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2JpZ19lbl81XzYsaWR4X3N1
YnRpdGxlc19jb250ZW50X3Byb2RfYmlnX2VuXzVfNyxpZHhfc3VidGl0bGVzX2NvbnRlbnRfcHJvZF9i
aWdfZW5fNl8wLGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2JpZ19lbl82XzEsaWR4X3N1YnRp
dGxlc19jb250ZW50X3Byb2RfYmlnX2VuXzZfMixpZHhfc3VidGl0bGVzX2NvbnRlbnRfcHJvZF9iaWdf
ZW5fNl8zLGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2JpZ19lbl82XzQsaWR4X3N1YnRpdGxlc19j
b250ZW50X3Byb2RfYmlnX2VuXzZfNSxpZHhfc3VidGl0bGVzX2NvbnRlbnRfcHJvZF9iaWdfZW5fNl82
LGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2JpZ19lbl82XzcsaWR4X3N1YnRpdGxlc19jb250
ZW50X3Byb2RfYmlnX2VuXzdfMCxpZHhfc3VidGl0bGVzX2NvbnRlbnRfcHJvZF9iaWdfZW5fN18xLGlk
eF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2JpZ19lbl83XzIsaWR4X3N1YnRpdGxlc19jb250ZW50X3By
b2RfYmlnX2VuXzdfMyxpZHhfc3VidGl0bGVzX2NvbnRlbnRfcHJvZF9iaWdfZW5fN180LGlkeF9zdWJ0
aXRsZXNfY29udGVudF9wcm9kX2JpZ19lbl83XzUsaWR4X3N1YnRpdGxlc19jb250ZW50X3Byb2Rf
YmlnX2VuXzdfNixpZHhfc3VidGl0bGVzX2NvbnRlbnRfcHJvZF9iaWdfZW5fN183LGlkeF9zdWJ0aXRs
ZXNfY29udGVudF9wcm9kX2JpZ19lc19wdF9mcl9pdF80XzAsaWR4X3N1YnRpdGxlc19jb250ZW50X3By
b2RfYmlnX2VzX3B0X2ZyX2l0XzRfMSxpZHhfc3VidGl0bGVzX2NvbnRlbnRfcHJvZF9iaWdfZXNfcHRf
ZnJfaXRfNF8yLGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9kX2JpZ19lc19wdF9mcl9pdF80XzMs
aWR4X3N1YnRpdGxlc19jb250ZW50X3Byb2RfYmlnX2VzX3B0X2ZyX2l0XzRfNCxpZHhfc3VidGl0bGVz
X2NvbnRlbnRfcHJvZF9iaWdfZXNfcHRfZnJfaXRfNF81LGlkeF9zdWJ0aXRsZXNfY29udGVudF9wcm9k
X2JpZ19lc19wdF9mcl9pdF80XzYsaWR4X3N1YnRpdGxlc19jb250ZW50X3Byb2RfYmlnX2VzX3B0X2Zy
....
--- request dump end ---
--- local index: nMntent_prod_big_india_7_1
Manticore 9.3.2 428075261@25050208 (columnar 4.2.1 0fe99f4@25050206) (secondary 4.2.1 0fe99f4@25050206) (knn 4.2.1 0fe99f4@25050206)
Handling signal 6
-------------- backtrace begins here ---------------
Program compiled with Clang 16.0.6
Configured with flags: Configured with these definitions: -DDISTR_BUILD=rhel9 -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_JIEBA=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadb.so.3 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (rhel9) (cross-compiled)
Stack bottom = 0x7ef1bc0527d0, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7ef1bc050000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x227)[0x55f07f011867]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x364)[0x55f07ee7ea34]
/lib64/libc.so.6(+0x3e730)[0x7f201b03e730]
/lib64/libc.so.6(+0x8ba6c)[0x7f201b08ba6c]
/lib64/libc.so.6(raise+0x16)[0x7f201b03e686]
/lib64/libc.so.6(abort+0xd3)[0x7f201b028833]
/lib64/libc.so.6(+0x29170)[0x7f201b029170]
/lib64/libc.so.6(+0x95b17)[0x7f201b095b17]
/lib64/libc.so.6(+0x9666c)[0x7f201b09666c]
/lib64/libc.so.6(+0x98fb1)[0x7f201b098fb1]
/lib64/libc.so.6(malloc+0x99)[0x7f201b099929]
/usr/bin/searchd(_Znwm+0x9)[0x55f07ef0fb59]
/usr/bin/searchd(_ZN21CSphTokenizerSettingsaSERKS_+0x25b)[0x55f08003155b]
/usr/bin/searchd(_ZN17CSphTokenizerBase9CloneBaseEPKS_18ESphTokenizerClone+0xf4)[0x55f08003aad4]
/usr/bin/searchd(_ZNK18CSphTokenizer_UTF8ILb1EE5CloneE18ESphTokenizerClone+0x6d)[0x55f08003729d]
/usr/bin/searchd(_ZNK13CSphIndex_VLN13DoGetKeywordsI16DiskIndexQword_cILb1ELb0EEEEbRN3sph8Vector_TI15CSphKeywordInfoNS3_13DefaultCopy_TIS5_EENS3_14DefaultRelimitENS3_16DefaultStorage_TIS5_EEEEPKcRK21GetKeywordsSettings_tbP10CSphString+0x5b5)[0x55f07efd1675]
/usr/bin/searchd(_ZNK13CSphIndex_VLN11IsQueryFastERK9CSphQueryRKN3sph8Vector_TI20SecondaryIndexInfo_tNS3_13DefaultCopy_TIS5_EENS3_14DefaultRelimitENS3_16DefaultStorage_TIS5_EEEEf+0x23b)[0x55f07ef2092b]
/usr/bin/searchd(_ZNK13CSphIndex_VLN23GetPseudoShardingMetricERK11VecTraits_TIK9CSphQueryERKS0_IlEiRb+0x348)[0x55f07ef20ea8]
/usr/bin/searchd(_ZN15SearchHandler_c19CalcThreadsPerIndexEi+0x2ee)[0x55f07ee9a2be]
/usr/bin/searchd(_ZN15SearchHandler_c16RunLocalSearchesEv+0x6c2)[0x55f07ee95492]
/usr/bin/searchd(_ZN15SearchHandler_c9RunSubsetEii+0x532)[0x55f07ee96e02]
/usr/bin/searchd(_ZN15SearchHandler_c10RunQueriesEv+0xa8)[0x55f07ee93918]
/usr/bin/searchd(_Z19HandleCommandSearchR16ISphOutputBuffertR13InputBuffer_c+0x334)[0x55f07eea0004]
/usr/bin/searchd(_Z17ExecuteApiCommand16SearchdCommand_etiR13InputBuffer_cR21GenericOutputBuffer_c+0xd6)[0x55f07eea9726]
/usr/bin/searchd(_Z8ApiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE+0x7b5)[0x55f07edd1a15]
/usr/bin/searchd(_Z10MultiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EESt4pairIitE7Proto_e+0x106)[0x55f07edcf9b6]
/usr/bin/searchd(+0x100c4a2)[0x55f07edd04a2]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_+0x1c)[0x55f08014528c]
/usr/bin/searchd(make_fcontext+0x2f)[0x55f0804c2fcf]
Trying boost backtrace:

Manticore Search Version:

Manticore 9.3.2 428075261@25050208 (columnar 4.2.1 0fe99f4@25050206) (secondary 4.2.1 0fe99f4@25050206) (knn 4.2.1 0fe99f4@25050206)

Operating System Version:

almalinux 9

Have you tried the latest development version?

No

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed

[sanikolaev]

@popalot2 thanks for the issue. Can you please provide your full searchd log?

The query which caused the crash looks smth like this:

���&�������������X�����������������������id desc����@subtitles "ha morso"������*idx_subtitles_content_prod_big_en_4_0,idx_subtitles_content_prod_big_en_4_1,idx_subtitles_content_prod_big_en_4_2,idx_subtitles_content_prod_big_en_4_3,idx_subtitles_content_prod_big_en_4_4,idx_subtitles_content_prod_big_en_4_5,idx_subtitles_content_prod_big_en_4_6,idx_subtitles_content_prod_big_en_4_7,idx_subtitles_content_prod_big_en_5_0,idx_subtitles_content_prod_big_en_5_1,idx_subtitles_content_prod_big_en_5_2,idx_subtitles_content_prod_big_en_5_3,idx_subtitles_content_prod_big_en_5_4,idx_subtitles_content_prod_big_en_5_5,idx_subtitles_content_prod_big_en_5_6,idx_subtitles_content_prod_big_en_5_7,idx_subtitles_content_prod_big_en_6_0,idx_subtitles_content_prod_big_en_6_1,idx_subtitles_content_prod_big_en_6_2,idx_subtitles_content_prod_big_en_6_3,idx_subtitles_content_prod_big_en_6_4,idx_subtitles_content_prod_big_en_6_5,idx_subtitles_content_prod_big_en_6_6,idx_subtitles_content_prod_big_en_6_7,idx_subtitles_content_prod_big_en_7_0,idx_subtitles_content_prod_big_en_7_1,idx_subtitles_content_prod_big_en_7_2,idx_subtitles_content_prod_big_en_7_3,idx_subtitles_content_prod_big_en_7_4,idx_subtitles_content_prod_big_en_7_5,idx_subtitles_content_prod_big_en_7_6,idx_subtitles_content_prod_big_en_7_7,idx_subtitles_content_prod_big_es_pt_fr_it_4_0,idx_subtitles_content_prod_big_es_pt_fr_it_4_1,idx_subtitles_content_prod_big_es_pt_fr_it_4_2,idx_subtitles_content_prod_big_es_pt_fr_it_4_3,idx_subtitles_content_prod_big_es_pt_fr_it_4_4,idx_subtitles_content_prod_big_es_pt_fr_it_4_5,idx_subtitles_content_prod_big_es_pt_fr_it_4_6,idx_subtitles_content_prod_big_es_pt_fr

Is this familiar to you? If so, can you try it a few times in a row?

Chances are the cause is RAM shortage. Can you ensure it's not the case?

[popalot2]

Can you please provide your full searchd log?
The query which caused the crash looks smth like this:

The full base64 encoded section is too large for single github-proxy.voidd.workers.devment, I put it here: https://pastebin.pl/view/435913c8
The query is bellow, the original query also had facets but it's split in the query log into multiple queries.
I tried rerunning the query several times, with and without facets but the crash did not happen.
The machine has 128GB RAM and 128GB swap, I don't think RAM is the problem unless there is some sort of memory leak (I will keep an eye out for ongoing memory usage)
The index is about 4TB on disk but manticore is configured with mmap and file so it shouldn't be an issue as far as I understand.

access_plain_attrs=mmap
  access_blob_attrs=mmap
  access_doclists=file
  access_hitlists=file
  access_dict=mmap

>free
               total        used        free      shared  buff/cache   available
Mem:       131285200    17671148     7080128        3168   119136024   113614052
Swap:      134217720     7086376   127131344

query:

SELECT id,vid,allowembed,title,viewcount,likecount,dislikecount,sublang,fullcaptions,captions,uploaderid,uploaddate,videolang,categoryid FROM idx_subtitles_content_prod_big_en_4_0,idx_subtitles_content_prod_big_en_4_1,idx_subtitles_content_prod_big_en_4_2,idx_subtitles_content_prod_big_en_4_3,idx_subtitles_content_prod_big_en_4_4,idx_subtitles_content_prod_big_en_4_5,idx_subtitles_content_prod_big_en_4_6,idx_subtitles_content_prod_big_en_4_7,idx_subtitles_content_prod_big_en_5_0,idx_subtitles_content_prod_big_en_5_1,idx_subtitles_content_prod_big_en_5_2,idx_subtitles_content_prod_big_en_5_3,idx_subtitles_content_prod_big_en_5_4,idx_subtitles_content_prod_big_en_5_5,idx_subtitles_content_prod_big_en_5_6,idx_subtitles_content_prod_big_en_5_7,idx_subtitles_content_prod_big_en_6_0,idx_subtitles_content_prod_big_en_6_1,idx_subtitles_content_prod_big_en_6_2,idx_subtitles_content_prod_big_en_6_3,idx_subtitles_content_prod_big_en_6_4,idx_subtitles_content_prod_big_en_6_5,idx_subtitles_content_prod_big_en_6_6,idx_subtitles_content_prod_big_en_6_7,idx_subtitles_content_prod_big_en_7_0,idx_subtitles_content_prod_big_en_7_1,idx_subtitles_content_prod_big_en_7_2,idx_subtitles_content_prod_big_en_7_3,idx_subtitles_content_prod_big_en_7_4,idx_subtitles_content_prod_big_en_7_5,idx_subtitles_content_prod_big_en_7_6,idx_subtitles_content_prod_big_en_7_7,idx_subtitles_content_prod_big_es_pt_fr_it_4_0,idx_subtitles_content_prod_big_es_pt_fr_it_4_1,idx_subtitles_content_prod_big_es_pt_fr_it_4_2,idx_subtitles_content_prod_big_es_pt_fr_it_4_3,idx_subtitles_content_prod_big_es_pt_fr_it_4_4,idx_subtitles_content_prod_big_es_pt_fr_it_4_5,idx_subtitles_content_prod_big_es_pt_fr_it_4_6,idx_subtitles_content_prod_big_es_pt_fr_it_4_7,idx_subtitles_content_prod_big_es_pt_fr_it_5_0,idx_subtitles_content_prod_big_es_pt_fr_it_5_1,idx_subtitles_content_prod_big_es_pt_fr_it_5_2,idx_subtitles_content_prod_big_es_pt_fr_it_5_3,idx_subtitles_content_prod_big_es_pt_fr_it_5_4,idx_subtitles_content_prod_big_es_pt_fr_it_5_5,idx_subtitles_content_prod_big_es_pt_fr_it_5_6,idx_subtitles_content_prod_big_es_pt_fr_it_5_7,idx_subtitles_content_prod_big_es_pt_fr_it_6_0,idx_subtitles_content_prod_big_es_pt_fr_it_6_1,idx_subtitles_content_prod_big_es_pt_fr_it_6_2,idx_subtitles_content_prod_big_es_pt_fr_it_6_3,idx_subtitles_content_prod_big_es_pt_fr_it_6_4,idx_subtitles_content_prod_big_es_pt_fr_it_6_5,idx_subtitles_content_prod_big_es_pt_fr_it_6_6,idx_subtitles_content_prod_big_es_pt_fr_it_6_7,idx_subtitles_content_prod_big_es_pt_fr_it_7_0,idx_subtitles_content_prod_big_es_pt_fr_it_7_1,idx_subtitles_content_prod_big_es_pt_fr_it_7_2,idx_subtitles_content_prod_big_es_pt_fr_it_7_3,idx_subtitles_content_prod_big_es_pt_fr_it_7_4,idx_subtitles_content_prod_big_es_pt_fr_it_7_5,idx_subtitles_content_prod_big_es_pt_fr_it_7_6,idx_subtitles_content_prod_big_es_pt_fr_it_7_7,idx_subtitles_content_prod_big_india_4_0,idx_subtitles_content_prod_big_india_4_1,idx_subtitles_content_prod_big_india_4_2,idx_subtitles_content_prod_big_india_4_3,idx_subtitles_content_prod_big_india_4_4,idx_subtitles_content_prod_big_india_4_5,idx_subtitles_content_prod_big_india_4_6,idx_subtitles_content_prod_big_india_4_7,idx_subtitles_content_prod_big_india_5_0,idx_subtitles_content_prod_big_india_5_1,idx_subtitles_content_prod_big_india_5_2,idx_subtitles_content_prod_big_india_5_3,idx_subtitles_content_prod_big_india_5_4,idx_subtitles_content_prod_big_india_5_5,idx_subtitles_content_prod_big_india_5_6,idx_subtitles_content_prod_big_india_5_7,idx_subtitles_content_prod_big_india_6_0,idx_subtitles_content_prod_big_india_6_1,idx_subtitles_content_prod_big_india_6_2,idx_subtitles_content_prod_big_india_6_3,idx_subtitles_content_prod_big_india_6_4,idx_subtitles_content_prod_big_india_6_5,idx_subtitles_content_prod_big_india_6_6,idx_subtitles_content_prod_big_india_6_7,idx_subtitles_content_prod_big_india_7_0,idx_subtitles_content_prod_big_india_7_1,idx_subtitles_content_prod_big_india_7_2,idx_subtitles_content_prod_big_india_7_3,idx_subtitles_content_prod_big_india_7_4,idx_subtitles_content_prod_big_india_7_5,idx_subtitles_content_prod_big_india_7_6,idx_subtitles_content_prod_big_india_7_7,idx_subtitles_content_prod_big_ru_uk_4_0,idx_subtitles_content_prod_big_ru_uk_4_1,idx_subtitles_content_prod_big_ru_uk_4_2,idx_subtitles_content_prod_big_ru_uk_4_3,idx_subtitles_content_prod_big_ru_uk_4_4,idx_subtitles_content_prod_big_ru_uk_4_5,idx_subtitles_content_prod_big_ru_uk_4_6,idx_subtitles_content_prod_big_ru_uk_4_7,idx_subtitles_content_prod_big_ru_uk_5_0,idx_subtitles_content_prod_big_ru_uk_5_1,idx_subtitles_content_prod_big_ru_uk_5_2,idx_subtitles_content_prod_big_ru_uk_5_3,idx_subtitles_content_prod_big_ru_uk_5_4,idx_subtitles_content_prod_big_ru_uk_5_5,idx_subtitles_content_prod_big_ru_uk_5_6,idx_subtitles_content_prod_big_ru_uk_5_7,idx_subtitles_content_prod_big_ru_uk_6_0,idx_subtitles_content_prod_big_ru_uk_6_1,idx_subtitles_content_prod_big_ru_uk_6_2,idx_subtitles_content_prod_big_ru_uk_6_3,idx_subtitles_content_prod_big_ru_uk_6_4,idx_subtitles_content_prod_big_ru_uk_6_5,idx_subtitles_content_prod_big_ru_uk_6_6,idx_subtitles_content_prod_big_ru_uk_6_7,idx_subtitles_content_prod_big_ru_uk_7_0,idx_subtitles_content_prod_big_ru_uk_7_1,idx_subtitles_content_prod_big_ru_uk_7_2,idx_subtitles_content_prod_big_ru_uk_7_3,idx_subtitles_content_prod_big_ru_uk_7_4,idx_subtitles_content_prod_big_ru_uk_7_5,idx_subtitles_content_prod_big_ru_uk_7_6,idx_subtitles_content_prod_big_ru_uk_7_7,idx_subtitles_content_prod_big_ja_ko_4_0,idx_subtitles_content_prod_big_ja_ko_4_1,idx_subtitles_content_prod_big_ja_ko_4_2,idx_subtitles_content_prod_big_ja_ko_4_3,idx_subtitles_content_prod_big_ja_ko_4_4,idx_subtitles_content_prod_big_ja_ko_4_5,idx_subtitles_content_prod_big_ja_ko_4_6,idx_subtitles_content_prod_big_ja_ko_4_7,idx_subtitles_content_prod_big_ja_ko_5_0,idx_subtitles_content_prod_big_ja_ko_5_1,idx_subtitles_content_prod_big_ja_ko_5_2,idx_subtitles_content_prod_big_ja_ko_5_3,idx_subtitles_content_prod_big_ja_ko_5_4,idx_subtitles_content_prod_big_ja_ko_5_5,idx_subtitles_content_prod_big_ja_ko_5_6,idx_subtitles_content_prod_big_ja_ko_5_7,idx_subtitles_content_prod_big_ja_ko_6_0,idx_subtitles_content_prod_big_ja_ko_6_1,idx_subtitles_content_prod_big_ja_ko_6_2,idx_subtitles_content_prod_big_ja_ko_6_3,idx_subtitles_content_prod_big_ja_ko_6_4,idx_subtitles_content_prod_big_ja_ko_6_5,idx_subtitles_content_prod_big_ja_ko_6_6,idx_subtitles_content_prod_big_ja_ko_6_7,idx_subtitles_content_prod_big_ja_ko_7_0,idx_subtitles_content_prod_big_ja_ko_7_1,idx_subtitles_content_prod_big_ja_ko_7_2,idx_subtitles_content_prod_big_ja_ko_7_3,idx_subtitles_content_prod_big_ja_ko_7_4,idx_subtitles_content_prod_big_ja_ko_7_5,idx_subtitles_content_prod_big_ja_ko_7_6,idx_subtitles_content_prod_big_ja_ko_7_7,idx_subtitles_content_prod_big_vi_id_de_ar_tr_4_0,idx_subtitles_content_prod_big_vi_id_de_ar_tr_4_1,idx_subtitles_content_prod_big_vi_id_de_ar_tr_4_2,idx_subtitles_content_prod_big_vi_id_de_ar_tr_4_3,idx_subtitles_content_prod_big_vi_id_de_ar_tr_4_4,idx_subtitles_content_prod_big_vi_id_de_ar_tr_4_5,idx_subtitles_content_prod_big_vi_id_de_ar_tr_4_6,idx_subtitles_content_prod_big_vi_id_de_ar_tr_4_7,idx_subtitles_content_prod_big_vi_id_de_ar_tr_5_0,idx_subtitles_content_prod_big_vi_id_de_ar_tr_5_1,idx_subtitles_content_prod_big_vi_id_de_ar_tr_5_2,idx_subtitles_content_prod_big_vi_id_de_ar_tr_5_3,idx_subtitles_content_prod_big_vi_id_de_ar_tr_5_4,idx_subtitles_content_prod_big_vi_id_de_ar_tr_5_5,idx_subtitles_content_prod_big_vi_id_de_ar_tr_5_6,idx_subtitles_content_prod_big_vi_id_de_ar_tr_5_7,idx_subtitles_content_prod_big_vi_id_de_ar_tr_6_0,idx_subtitles_content_prod_big_vi_id_de_ar_tr_6_1,idx_subtitles_content_prod_big_vi_id_de_ar_tr_6_2,idx_subtitles_content_prod_big_vi_id_de_ar_tr_6_3,idx_subtitles_content_prod_big_vi_id_de_ar_tr_6_4,idx_subtitles_content_prod_big_vi_id_de_ar_tr_6_5,idx_subtitles_content_prod_big_vi_id_de_ar_tr_6_6,idx_subtitles_content_prod_big_vi_id_de_ar_tr_6_7,idx_subtitles_content_prod_big_vi_id_de_ar_tr_7_0,idx_subtitles_content_prod_big_vi_id_de_ar_tr_7_1,idx_subtitles_content_prod_big_vi_id_de_ar_tr_7_2,idx_subtitles_content_prod_big_vi_id_de_ar_tr_7_3,idx_subtitles_content_prod_big_vi_id_de_ar_tr_7_4,idx_subtitles_content_prod_big_vi_id_de_ar_tr_7_5,idx_subtitles_content_prod_big_vi_id_de_ar_tr_7_6,idx_subtitles_content_prod_big_vi_id_de_ar_tr_7_7,idx_subtitles_content_prod_big_extra_4_0,idx_subtitles_content_prod_big_extra_4_1,idx_subtitles_content_prod_big_extra_4_2,idx_subtitles_content_prod_big_extra_4_3,idx_subtitles_content_prod_big_extra_4_4,idx_subtitles_content_prod_big_extra_4_5,idx_subtitles_content_prod_big_extra_4_6,idx_subtitles_content_prod_big_extra_4_7,idx_subtitles_content_prod_big_extra_5_0,idx_subtitles_content_prod_big_extra_5_1,idx_subtitles_content_prod_big_extra_5_2,idx_subtitles_content_prod_big_extra_5_3,idx_subtitles_content_prod_big_extra_5_4,idx_subtitles_content_prod_big_extra_5_5,idx_subtitles_content_prod_big_extra_5_6,idx_subtitles_content_prod_big_extra_5_7,idx_subtitles_content_prod_big_extra_6_0,idx_subtitles_content_prod_big_extra_6_1,idx_subtitles_content_prod_big_extra_6_2,idx_subtitles_content_prod_big_extra_6_3,idx_subtitles_content_prod_big_extra_6_4,idx_subtitles_content_prod_big_extra_6_5,idx_subtitles_content_prod_big_extra_6_6,idx_subtitles_content_prod_big_extra_6_7,idx_subtitles_content_prod_big_extra_7_0,idx_subtitles_content_prod_big_extra_7_1,idx_subtitles_content_prod_big_extra_7_2,idx_subtitles_content_prod_big_extra_7_3,idx_subtitles_content_prod_big_extra_7_4,idx_subtitles_content_prod_big_extra_7_5,idx_subtitles_content_prod_big_extra_7_6,idx_subtitles_content_prod_big_extra_7_7,idx_subtitles_content_prod_big_delta_en_4_0,idx_subtitles_content_prod_big_delta_en_5_0,idx_subtitles_content_prod_big_delta_en_6_0,idx_subtitles_content_prod_big_delta_en_7_0,idx_subtitles_content_prod_big_delta_es_pt_fr_it_4_0,idx_subtitles_content_prod_big_delta_es_pt_fr_it_5_0,idx_subtitles_content_prod_big_delta_es_pt_fr_it_6_0,idx_subtitles_content_prod_big_delta_es_pt_fr_it_7_0,idx_subtitles_content_prod_big_delta_india_4_0,idx_subtitles_content_prod_big_delta_india_5_0,idx_subtitles_content_prod_big_delta_india_6_0,idx_subtitles_content_prod_big_delta_india_7_0,idx_subtitles_content_prod_big_delta_ru_uk_4_0,idx_subtitles_content_prod_big_delta_ru_uk_5_0,idx_subtitles_content_prod_big_delta_ru_uk_6_0,idx_subtitles_content_prod_big_delta_ru_uk_7_0,idx_subtitles_content_prod_big_delta_ja_ko_4_0,idx_subtitles_content_prod_big_delta_ja_ko_5_0,idx_subtitles_content_prod_big_delta_ja_ko_6_0,idx_subtitles_content_prod_big_delta_ja_ko_7_0,idx_subtitles_content_prod_big_delta_vi_id_de_ar_tr_4_0,idx_subtitles_content_prod_big_delta_vi_id_de_ar_tr_5_0,idx_subtitles_content_prod_big_delta_vi_id_de_ar_tr_6_0,idx_subtitles_content_prod_big_delta_vi_id_de_ar_tr_7_0,idx_subtitles_content_prod_big_delta_extra_4_0,idx_subtitles_content_prod_big_delta_extra_5_0,idx_subtitles_content_prod_big_delta_extra_6_0,idx_subtitles_content_prod_big_delta_extra_7_0 WHERE MATCH('@subtitles "ha morso"') ORDER BY id desc LIMIT 5050 OPTION max_matches=5050, idf='plain,tfidf_normalized', max_query_time=4000, retry_count=0, retry_delay=0;

[popalot2]

Chances are the cause is RAM shortage. Can you ensure it's not the case?

Just to clarify, my main complaint is not the failure of the query or even the crash. My main complaint is that the process stopped serving requests and didn't even fully exit. Whatever the error was it seems there is a problem with signal/error handling which caused the process to hang until kill -9 was issued. This is the first thing that I think should be reproduced and fixed.

[sanikolaev]

Can you please provide your full searchd log?

[popalot2]

Can you please provide your full searchd log?

sure, I already provided a link to the crash record,
here is the full searchd log, nothing else interesting there https://pastebin.pl/view/2b3d9eee

[sanikolaev]

Thanks, but there's only one crash logged. If you have more crash logs, could you please share them? It’s important for us to:

• see if there's any pattern with what the daemon is doing when it crashes
• check whether the backtraces are the same

[popalot2]

Thanks, but there's only one crash logged. If you have more crash logs, could you please share them? It’s important for us to:

• see if there's any pattern with what the daemon is doing when it crashes
• check whether the backtraces are the same

I only had the one crash on the latest version,
here is a similar crash (also the server was hanging and needed kill -9) but it was on 6.3.8, after which I upgraded to 9.3.2

https://pastebin.pl/view/16b35d1a

[sanikolaev]

Thanks @popalot2. The crashes seem identical. Do you have a coredump for either of them?

[popalot2]

Thanks @popalot2. The crashes seem identical. Do you have a coredump for either of them?

No, I don't have the core dumps, I also didn't have any more crashes since opening the bug. Will update if that changes.