Manticore Search 9.3.2

Version 9.3.2

Released: May 2nd 2025

This release includes multiple bug fixes and stability improvements, better table usage tracking, and enhancements to memory and resource management.

❤️ Special thanks to @cho-m for fixing build compatibility with Boost 1.88.0 and @benwills for improving documentation around stored_only_fields.

• 🪲 v9.3.2 Fixed an issue where the "Show Threads" column showed CPU activity as a float instead of a string; also fixed a PyMySQL result set parsing error caused by a wrong data type.

• 🪲 v9.3.1 Issue #3343 Fixed leftover tmp.spidx files when the optimize process was interrupted.

• 🆕 v9.3.0 PR #3337 Added a per-table command counter and detailed table usage statistics.

• 🪲 v9.2.39 Issue #3236 Fix: prevent table damage by removing complex chunk updates. Using wait functions inside the serial worker broke serial processing, which could damage tables.
  Reimplemented autoflush. Removed external polling queue to avoid unnecessary table locks. Added "small table" condition: if the number of documents is below the 'small table limit' (8192) and no Secondary Index (SI) is used, flushing is skipped.

• 🪲 v9.2.38 Fix: skip creating Secondary Index (SI) for filters using ALL/ANY on string lists, without affecting JSON attributes.

• 🪲 v9.2.37 Issue #2898 Added backtick support for system tables.

• 🪲 v9.2.36 Fix: use a placeholder for cluster operations in legacy code. In the parser, we now clearly separate fields for table and cluster names.

• 🪲 v9.2.35 Fix: crash when unquoting a single '.

• 🪲 v9.2.34 Issue #3090 Fix: handling of large document IDs (could previously fail to find them).

• 🪲 v9.2.33 Fix: use unsigned integers for bit vector sizes.

• 🪲 v9.2.32 Fix: reduce peak memory usage during merge. docid-to-rowid lookups now use 12 bytes per document instead of 16 bytes. Example: 24 GB RAM for 2 billion docs instead of 36 GB.

• 🪲 v9.2.31 Issue #3238 Fix: incorrect COUNT(*) value in large real-time tables.

• 🪲 v9.2.30 Fix: undefined behavior when zeroing string attributes.

• 🪲 v9.2.29 Minor fix: improved a warning text.

• 🪲 v9.2.28 Issue #3290 Improvement: enhanced indextool --buildidf

• 🪲 v9.2.27 Issue #3032 With the Kafka integration, you can now create a source for a specific Kafka partition.

• 🪲 v9.2.26 Issue #3301 Fix: ORDER BY and WHERE on id could cause OOM (Out Of Memory) errors.

• 🪲 v9.2.25 Issue #3171 Fix: crash caused by segmentation fault when using grouper with multiple JSON attributes on an RT table that has multiple disk chunks

• 🪲 v9.2.24 Issue #3246 Fix: WHERE string ANY(...) queries failed after RAM chunk flush.

• 🪲 v9.2.23 PR #518 Minor auto-sharding syntax improvements.

• 🪲 v9.2.22 Issue #2763 Fix: global idf file was not loaded when using ALTER TABLE.

• 🪲 v9.2.21 Fix: global idf files can be large. We now release tables sooner to avoid holding unnecessary resources.

• 🪲 v9.2.20 PR #3277 Improvement: better validatation of sharding options.

• 🪲 v9.2.19 PR #3275 Fix: build compatibility with Boost 1.88.0.

• 🪲 v9.2.18 Issue #3228 Fix: crash during distributed table creation (invalid pointer issue).

• 🪲 v9.2.17 PR #3272 Fix: multiline fuzzy FACET issue.

• 🪲 v9.2.16 Issue #3063 Fix: bug in distance calculation when using the GEODIST function.

• 🪲 v9.2.15 Issue #3027 Minor improvement: support for Elastic's query_string filter format.

Manticore Search 9.2.14

Version 9.2.14

Released: March 28th 2025

Minor changes

• Commit fe94 Implemented --mockstack flag to calculate recursive operation stack requirements. The new --mockstack mode analyzes and reports the necessary stack sizes for recursive expression evaluation, pttern matching operations, filter processing. Calculated stack requirements are output to console for debugging and optimization purposes.
• Issue #3058 Enabled boolean_simplify by default.
• Issue #3172 Added a new config option: searchd.kibana_version_string, which can be useful when using Manticore with specific versions of Kibana or OpenSearch Dashboards that expect a particular Elasticsearch version.
• Issue #3211 Fixed CALL SUGGEST to work with 2-character words.
• Issue #490 Improved fuzzy search: previously, it couldn't sometimes find "defghi" when searching for "def ghi" if another matching document existed.
• ⚠️ BREAKING Issue #3165 Changed _id to id in some HTTP JSON responses for consistency. Make sure to update your app accordingly.
• ⚠️ BREAKING Issue #3186 Added a check for server_id during cluster joins to ensure each node has a unique ID. The JOIN CLUSTER operation can now fail with an error message indicating a duplicate server_id when the joining node has the same server_id as an existing node in the cluster. To resolve this issue, ensure that each node in the replication cluster has a unique server_id. You can change the default server_id in the "searchd" section of your configuration file to a unique value before attempting to join the cluster. This change updates the replication protocol. If you are running a replication cluster, you need to:
  • First, cleanly stop all your nodes
  • Then, start the node that was stopped last with --new-cluster, using the tool manticore_new_cluster in Linux.
  • Read about restarting a cluster for more details.

Bugfixes

• Commit 6fda Fixed a crash caused by losing the scheduler after a wait; now, specific schedulers like serializer are properly restored.
• Commit c333 Fixed a bug where weights from the right joined table couldn't be used in the ORDER BY clause.
• Issue #2644 gcc 14.2.0: fixed lower_bound call error with const knn::DocDist_t*&. ❤️ Thank you @Azq2 for the PR.
• Issue #3018 Fixed an issue with handling uppercase table names during auto-schema inserts.
• Issue #3119 Fixed a crash when decoding invalid base64 input.
• Issue #3121 Fixed two related KNN index issues on ALTER: float vectors now keep their original dimensions, and KNN indexes are now properly generated.
• Issue #3123 Fixed a crash when building a secondary index on an empty JSON column.
• Issue #3138 Fixed a crash caused by duplicate entries.
• Issue #3151 Fixed: fuzzy=1 option couldn't be used with ranker or field_weights.
• Issue #3163 Fixed a bug where SET GLOBAL timezone had no effect.
• Issue #3181 Fixed an issue where text field values could be lost when using IDs larger than 2^63.
• Issue #3189 Fixed: UPDATE statements now correctly respect the query_log_min_msec setting.
• Issue #3247 Fixed a race condition when saving real-time disk chunks that could cause JOIN CLUSTER to fail.

Manticore Search 7.4.6

Version 7.4.6

Released: Feb 28th 2025

Major changes

• Issue #105 - No need to set EXTRA=1 or MCL=1 when running Manticore in Docker anymore.
• Issue #832 Integration with Kibana for easier and more efficient data visualization.

Minor changes

• Issue #1727 Fixed floating-point precision differences between arm64 and x86_64.
• Issue #2995 Implemented performance optimizations for join batching.
• Issue #3039 Implemented performance optimizations for EstimateValues in histograms.
• Issue #3099 Added support for Boost 1.87.0. ❤️ Thank you, @cho-m for the PR.
• Issue #77 Optimized block data reuse when creating filters with multiple values; added min/max to attribute metadata; implemented pre-filtering of filter values based on min/max.

Bugfixes

• Commit 73ac Fixed handling of expressions in joined queries when attributes from both the left and right tables are used; fixed the index_weights option for the right table.
• Issue #2915 Using avg() in a SELECT ... JOIN query could lead to incorrect results; this is now fixed.
• Issue #2996 Fixed an incorrect result set caused by the implicit cutoff when join batching was enabled.
• Issue #3031 Fixed a crash in the daemon during shutdown when an active chunk merge was in progress.
• Issue #3037 Fixed an issue where IN(...) could produce incorrect results.
• Issue #3038 Setting max_iops / max_iosize in version 7.0.0 could degrade indexing performance; this is now fixed.
• Issue #3042 Fixed a memory leak in the join query cache.
• Issue #3052 Fixed the handling of query options in joined JSON queries.
• Issue #3054 Fixed an issue with the ATTACH TABLE command.
• Issue #3079 Fixed inconsistencies in error messages.
• Issue #3087 Setting diskchunk_flush_write_timeout=-1 per table did not disable index flushing; this is now fixed.
• Issue #3088 Resolved duplicate entries after bulk replacing large IDs.
• Issue #3126 Fixed a daemon crash caused by a full-text query with a single NOT operator and an expression ranker.
• Issue #3128 Fixed a potential vulnerability in the CJSON library. ❤️ Thank you, @tabudz for the PR.

Manticore Search 7.0.0

Version 7.0.0

Released: Jan 30th 2025

📢📢📢Announcement blog post📢📢📢

Major changes

• Issue #1497 Added new Fuzzy Search and Autocomplete functionality for easier searching.
• Issue #1500 Integration with Kafka.
• Issue #1928 Introduced secondary indexes for JSON.
• Issue #2361 Updates and searches during updates are no longer blocked by chunk merging.
• Issue #2787 Automatic disk chunk flush for RT table to improve performance; now, we automatically flush a RAM chunk to a disk chunk, preventing performance issues caused by the lack of optimizations in RAM chunks, which could sometimes lead to instability depending on chunk size.
• Issue #2811 Scroll option for easier pagination.
• Issue #931 Integration with Jieba for better Chinese tokenization.

Minor changes

• ⚠️ BREAKING Issue #1111 Fixed support for global_idf in RT tables. Requires table recreation.
• ⚠️ BREAKING Issue #2103 Removed Thai characters from internal cjk charset. Update your charset definitions accordingly: if you have cjk,non_cjk and Thai characters are important for you, change it to cjk,thai,non_cjk, or cont,non_cjk, where cont is the new designation for all continuous-script languages (i.e., cjk + thai). Modify existing tables using ALTER TABLE.
• ⚠️ BREAKING Issue #2458 Incorporated zlib support in Windows packages.
• ⚠️ BREAKING Issue #2468 CALL SUGGEST / QSUGGEST now compatible with distributed tables. This increases the master/agent protocol version. If you are running Manticore Search in a distributed environment with multiple instances, make sure to first upgrade the agents, then the masters.
• ⚠️ BREAKING Issue #2889 Changed column name from Name to Variable name for PQ SHOW META.
• ⚠️ BREAKING Issue #879 Introduced per-table binlog with new options: binlog_common, binlog for create table / alter table. You need to make a clean shutdown of the Manticore instance before upgrading to the new version.
• ⚠️ BREAKING Issue #1789 Fixed incorrect error message when a node joins a cluster with the wrong replication protocol version. This change updates the replication protocol. If you are running a replication cluster, you need to:
  • First, cleanly stop all your nodes
  • Then, start the node that was stopped last with --new-cluster, using the tool manticore_new_cluster in Linux.
  • Read about restarting a cluster for more details.
• ⚠️ BREAKING Issue #2308 Added support for multiple tables in ALTER CLUSTER ADD and DROP. This change also affects the replication protocol. Refer to the previous section for guidance on handling this update.
• Issue #2997 Fixed dlopen issue on Macos.
• Commit 4954 Changed the default cutoff for OPTIMIZE TABLE on tables with KNN indexes to improve search performance.
• Commit cfc8 Added COUNT(DISTINCT) support for ORDER BY in FACET and GROUP BY.
• Issue #1103 Improved clarity in logging chunk merging.
• Issue #1130 Added support for DBeaver.
• Issue #1546 Implemented secondary indexes for POLY2D()/GEOPOLY2D() functions.
• Issue #1630 HTTP requests now support Content-Encoding: gzip.
• Issue #1831 Added SHOW LOCKS command.
• Issue #2187 Allowed Buddy request to daemon to bypass the searchd.max_connections constraint.
• Issue #2208 Added support for joining tables via the JSON HTTP interface.
• Issue #2235 Log successfully processed queries via Buddy in their original form.
• Issue #2249 Added special mode to run mysqldump for replicated tables.
• Issue #2268 Improved external files renaming on copy for CREATE TABLE and ALTER TABLE statements.
• Issue #2402 Updated searchd.max_packet_size default to 128MB.
• Issue #2419 Added support for IDF boost modifier in JSON "match".
• Issue #2430 Enhanced binlog writing synchronization to prevent errors.
• Issue #2479 Added support for SHOW TABLE INDEXES command.
• Issue #2485 Set session metadata for Buddy replies.
• Issue #2490 Millisecond resolution for aggregations at compatibility endpoints.
• Issue #2500 Changed error messages for cluster operations when replication fails to start.
• Issue #2584 New performance metrics in SHOW STATUS: min/max/avg/95th/99th percentile per query type for the last 1, 5, and 15 minutes.
• Issue #2639 Replaced all instances of index with table in requests and responses.
• Issue #2643 Added distinct column to HTTP /sql endpoint aggregation results.
• Issue #268 Implemented autodetection of data types imported from Elasticsearch.
• Issue #2744 Added collation support for string JSON field comparison expressions.
• Issue #2752 Added support for ...

Manticore Search 6.3.8

Manticore Search 6.3.6

Released: November 22nd 2024

Version 6.3.8 continues the 6.3 series and includes only bug fixes.

Bug fixes

• PR #2777 Fixed the calculation of available threads when query concurrency is limited by the threads or max_threads_per_query settings.

Manticore Search 6.3.6

Manticore Search 6.3.6

Released: August 2nd 2024

Version 6.3.6 continues the 6.3 series and includes only bug fixes.

Bug fixes

• Issue #2477 Fixed a crash introduced in version 6.3.4, which could occur when dealing with expressions and distributed or multiple tables.
• Issue #2352 Fixed a daemon crash or internal error upon early exit caused by max_query_time when querying multiple indexes.

Manticore Search 6.3.4

Version 6.3.4

Released: July 31st 2024

Version 6.3.4 continues the 6.3 series and includes only minor improvements and bug fixes.

Minor changes

• Issue #1130 Added support for DBeaver.
• Issue #2146 Improved escaping of delimiters in word forms and exceptions.
• Issue #2268 Improved external files renaming on copy for CREATE and ALTER TABLE statements.
• Issue #2315 Added string comparison operators to SELECT list expressions.
• Issue #2363 Added support for null values in Elastic-like bulk requests.
• Issue #2374 Added support for mysqldump version 9.
• Issue #2375 Improved error handling in HTTP JSON queries with JSON path to the node where the error occurs.

Bug fixes

• Issue #2280 Fixed performance degradation in wildcard queries with many matches when disk_chunks > 1.
• Issue #2332 Fixed crash in MVA MIN or MAX SELECT list expressions for empty MVA arrays.
• Issue #2339 Fixed incorrect processing of Kibana's infinite range request.
• Issue #2342 Fixed join filter on columnar attributes from the right table when the attribute is not in the SELECT list.
• Issue #2343 Fixed duplicate 'static' specifier in Manticore 6.3.2.
• Issue #2344 Fixed LEFT JOIN returning non-matching entries when MATCH() over the right table is used.
• Issue #2350 Fixed saving of disk chunk at RT index with hitless_words.
• Issue #2364 The 'aggs_node_sort' property can now be added in any order among other properties.
• Issue #2368 Fixed error on full-text vs filter order in the JSON query.
• Issue #2376 Fixed bug related to incorrect JSON response for long UTF-8 requests.
• Issue #2684 Fixed calculation of presort/prefilter expressions that depend on joined attributes.
• Issue #301 Changed the method of calculating data size for metrics to read from the manticore.json file instead of checking the entire size of the data directory.
• Issue #302 Added handling of validation requests from Vector.dev.

Manticore Search 6.3.2

Version 6.3.2

Released: June 26th 2024

Version 6.3.2 continues the 6.3 series and includes several bug fixes, some of which were discovered after the release of 6.3.0.

Breaking changes

• ⚠️Issue #2305 Updated aggs.range values to be numeric.

Bug fixes

• Commit c51c Fixed grouping by stored check vs rset merge.
• Commit 0e85 Fixed a crash in the daemon when querying with wildcard characters in an RT index using a CRC dictionary and local_df enabled.
• Issue #2200 Fixed a crash in JOIN on count(*) without GROUP BY.
• Issue #2201 Fixed JOIN not returning a warning when attempting grouping by a full-text field.
• Issue #2230 Addressed issue where adding an attribute via ALTER TABLE did not take KNN options into account.
• Issue #2231 Fixed failure in removing manticore-tools Redhat package in version 6.3.0.
• Issue #2242 Corrected issues with JOIN and multiple FACET statements returning incorrect results.
• Issue #2250 Fixed ALTER TABLE producing an error if the table is in a cluster.
• Issue #2252 Fixed the original query being passed into buddy from the SphinxQL interface.
• Issue #2267 Improved wildcard expansion in the CALL KEYWORDS for RT index with disk chunks.
• Issue #271 Fixed hanging of incorrect /cli requests.
• Issue #274 Resolved issues where concurrent requests to Manticore could get stuck.
• Issue #275 Fixed hanging of drop table if exists t; create table t via /cli.

Replication-related

• Issue #2270 Added support for cluster:name format in the /_bulk HTTP endpoint.

Manticore Search 6.3.0

Version 6.3.0

Released: May 23rd 2024

Major changes

• Issue #839 Implemented float_vector data type; implemented vector search.
• Issue #1673 INNER/LEFT JOIN (beta stage).
• Issue #1744 Implemented autodetection of date formats for timestamp fields.
• Issue #1720 Changed Manticore Search license from GPLv2-or-later to GPLv3-or-later.
• Commit 7a55 Running Manticore in Windows now requires Docker to run Buddy.
• Issue #1541 Added a REGEX full-text operator.
• Issue #2091 Ubuntu Noble 24.04 support.
• Commit 514d Revamp of time operations for better performance and new date/time functions:
  • CURDATE() - Returns current date in local timezone
  • QUARTER() - Returns the integer quarter of the year from a timestamp argument
  • DAYNAME() - Returns the weekday name for a given timestamp argument
  • MONTHNAME() - Returns the name of the month for a given timestamp argument
  • DAYOFWEEK() - Returns the integer weekday index for a given timestamp argument
  • DAYOFYEAR() - Returns the integer day of the year for a given timestamp argument
  • YEARWEEK() - Returns the integer year and the day code of the first day of current week for a given timestamp argument
  • DATEDIFF() - Returns the number of days between two given timestamps
  • DATE() - Formats the date part from a timestamp argument
  • TIME() - Formats the time part from a timestamp argument
  • timezone - Timezone used by date/time-related functions.
• Commit 30e7 Added range, histogram, date_range, and date_histogram aggregates to the HTTP interface and similar expressions into SQL.

Minor changes

• Issue #1285 Support of Filebeat versions 8.10 - 8.11.
• Issue #1771 ALTER TABLE ... type='distributed'.
• Issue #1788 Added the ability to copy tables using the CREATE TABLE ... LIKE ... WITH DATA SQL statement.
• Issue #2072 Optimized the table compacting algorithm: Previously, both manual OPTIMIZE and automatic auto_optimize processes would first merge chunks to ensure the count did not exceed the limit, and then expunge deleted documents from all other chunks containing deleted documents. This approach was sometimes too resource-intensive and has been disabled. Now, chunk merging occurs solely according to the progressive_merge setting. However, chunks with a high number of deleted documents are more likely to be merged.
• Commit ce6c Added protection against loading a secondary index of a newer version.
• Issue #1417 Partial replace via REPLACE INTO ... SET.
• Commit 7c16 Updated default merge buffer sizes: .spa (scalar attrs): 256KB -> 8MB; .spb (blob attrs): 256KB -> 8MB; .spc (columnar attrs): 1MB, no change; .spds (docstore): 256KB -> 8MB; .spidx (secondary indexes): 256KB buffer -> 128MB memory limit; .spi (dictionary): 256KB -> 16MB; .spd (doclists): 8MB, no change; .spp (hitlists): 8MB, no change; .spe (skiplists): 256KB -> 8MB.
• Issue #1859 Added composite aggregation via JSON.
• Commit 216b Disabled PCRE.JIT due to issues with some regex patterns and no significant time benefit.
• Commit 55cd Added support for vanilla Galera v.3 (api v25) (libgalera_smm.so from MySQL 5.x).
• Commit 86f9 Changed metric suffix from _rate to _rps.
• Commit c0c1 Improved docs about balancer HA support.
• Commit d1d2 Changed index to table in error messages; fixed bison parser error message fixup.
• Commit fd26 Support manticore.tbl as table name.
• Issue #1105 Support for running indexer via systemd (docs). ❤️ Thank you, @subnix for the PR.
• Issue #1294 Secondary indexes support in GEODIST().
• Issue #1394 Simplified SHOW THREADS.
• Issue #1424 Added support for the default values (agent_connect_timeout and agent_query_timeout) for create distributed table statement.
• Issue #1442 Added expansion_limit search query option that overrides searchd.expansion_limit.
• Issue #1448 Implemented ALTER TABLE for int->bigint conversion.
• Issue #146 Meta information in MySQL response.
• Issue #1494 SHOW VERSION.
• Issue #1582 Support of deleting documents by id array via JSON.
• Issue #1589 Improve error "unsupported value type".
• Issue #1634 Added Buddy version into SHOW STATUS.
• Issue #1641 Match requests optimisation in case of zero docs for a k...

Manticore Search 6.2.12

Manticore Search 6.2.12

Released: August 23rd 2023

Version 6.2.12 continues the 6.2 series and addresses issues discovered after the release of 6.2.0.

Bugfixes

• ❗Issue #1351 "Manticore 6.2.0 doesn't start via systemctl on Centos 7": Modified TimeoutStartSec from infinity to 0 for better compatibility with Centos 7.
• ❗Issue #1364 "Crash after upgrading from 6.0.4 to 6.2.0": Added replay functionality for empty binlog files from older binlog versions.
• PR #1334 "fix typo in searchdreplication.cpp": Corrected a typo in searchdreplication.cpp: beggining -> beginning.
• Issue #1337 "Manticore 6.2.0 WARNING: conn (local)(12), sock=8088: bailing on failed MySQL header, AsyncNetInputBuffer_c::AppendData: error 11 (Resource temporarily unavailable) return -1": Lowered the verbosity level of the MySQL interface warning about the header to logdebugv.
• Issue #1355 "join cluster hangs when node_address can't be resolved": Improved replication retry when certain nodes are unreachable, and their name resolution fails. This should resolve issues in Kubernetes and Docker nodes related to replication. Enhanced the error message for replication start failures and made updates to test model 376. Additionally, provided a clear error message for name resolution failures.
• Issue #1361 "No lower case mapping for "Ø" in charset non_cjk": Adjusted the mapping for the 'Ø' character.
• Issue #1365 "searchd leaves binlog.meta and binlog.001 after clean stop": Ensured that the last empty binlog file is removed properly.
• Commit 0871: Fixed the Thd_t build issue on Windows related to atomic copy restrictions.
• Commit 1cc0: Addressed an issue with FT CBO vs ColumnarScan.
• Commit c6bf: Made corrections to test 376 and added a substitution for the AF_INET error in the test.
• Commit cbc3: Resolved a deadlock issue during replication when updating blob attributes versus replacing documents. Also removed the rlock of the index during commit because it's already locked at a more basic level.

Minor changes

• Commit 4f91 Updated info on /bulk endpoints in the manual.

MCL

• Support of Manticore Columnar Library v2.2.4