Tantivy v0.24.1

Tantivy 0.24.1

• fix: Set rust-version to 1.81

Tantivy 0.24

Tantivy 0.24 will be backwards compatible with indices created with v0.22 and v0.21. The new minimum rust version will be 1.75. Tantivy 0.23 will be skipped.

Bugfixes

• fix potential endless loop in merge #2457(@PSeitz)
• fix bug that causes out-of-order sstable key. #2445(@fulmicoton)
• fix ReferenceValue API flaw #2372(@PSeitz)
• fix OwnedBytes debug panic #2512(@b41sh)
• catch panics during merges #2582(@rdettai)
• switch from u32 to usize in bitpacker. This enables multivalued columns larger than 4GB, which crashed during merge before. #2581 #2586(@fulmicoton-dd @PSeitz)

Breaking API Changes

• remove index sorting #2434(@PSeitz)

Features/Improvements

• Aggregation

  • Support for cardinality aggregation #2337 #2446 (@raphaelcoeffic @PSeitz)
  • Support for extended stats aggregation #2247(@giovannicuccu)
  • Add Key::I64 and Key::U64 variants in aggregation to avoid f64 precision issues #2468(@PSeitz)
  • Faster term aggregation fetch terms #2447(@PSeitz)
  • Improve custom order deserialization #2451(@PSeitz)
  • Change AggregationLimits behavior #2495(@PSeitz)
  • lower contention on AggregationLimits #2394(@PSeitz)
  • fix postcard compatibility for top_hits, add postcard test #2346(@PSeitz)
  • reduce top hits memory consumption #2426(@PSeitz)
  • check unsupported parameters top_hits #2351(@PSeitz)
  • Change AggregationLimits to AggregationLimitsGuard #2495(@PSeitz)
  • add support for counting non integer in aggregation #2547(@trinity-1686a)

• Range Queries

  • Support fast field range queries on json fields #2456(@PSeitz)
  • Add support for str fast field range query #2460 #2452 #2453(@PSeitz)
  • modify fastfield range query heuristic #2375(@trinity-1686a)
  • add FastFieldRangeQuery for explicit range queries on fast field (for RangeQuery it is autodetected) #2477(@PSeitz)

• add format backwards-compatibility tests #2485(@PSeitz)

• add columnar format compatibility tests #2433(@PSeitz)

• Improved snippet ranges algorithm #2474(@gezihuzi)

• make find_field_with_default return json fields without path #2476(@trinity-1686a)

• Make BooleanQuery support minimum_number_should_match #2405(@LebranceBW)

• Make NUM_MERGE_THREADS configurable #2535(@Barre)

• RegexPhraseQuery
  RegexPhraseQuery supports phrase queries with regex. E.g. query "b.* b.* wolf" matches "big bad wolf". Slop is supported as well: "b.* wolf"~2 matches "big bad wolf" #2516(@PSeitz)

• Optional Index in Multivalue Columnar Index
  For mostly empty multivalued indices there was a large overhead during creation when iterating all docids (merge case).
  This is alleviated by placing an optional index in the multivalued index to mark documents that have values.
  This will slightly increase space and access time. #2439(@PSeitz)

• Store DateTime as nanoseconds in doc store DateTime in the doc store was truncated to microseconds previously. This removes this truncation, while still keeping backwards compatibility. #2486(@PSeitz)

• Performace/Memory

  • lift clauses in LogicalAst for optimized ast during execution #2449(@PSeitz)
  • Use Vec instead of BTreeMap to back OwnedValue object #2364(@fulmicoton)
  • Replace TantivyDocument with CompactDoc. CompactDoc is much smaller and provides similar performance. #2402(@PSeitz)
  • Recycling buffer in PrefixPhraseScorer #2443(@fulmicoton)

• Json Type

  • JSON supports now all values on the root level. Previously an object was required. This enables support for flat mixed types. allow more JSON values, fix i64 special case #2383(@PSeitz)
  • add json path constructor to term #2367(@PSeitz)

• QueryParser

  • fix de-escaping too much in query parser #2427(@trinity-1686a)
  • improve query parser #2416(@trinity-1686a)
  • Support field grouping title:(return AND "pink panther") #2333(@trinity-1686a)
  • allow term starting with wildcard #2568(@trinity-1686a)

• Exist queries match subpath fields #2558(@rdettai)

• add access benchmark for columnar #2432(@PSeitz)

• extend indexwriter proptests #2342(@PSeitz)

• add bench & test for columnar merging #2428(@PSeitz)

• Change in Executor API #2391(@fulmicoton)

• Removed usage of num_cpus #2387(@fulmicoton)

• use bingang for agg and stacker benchmark #2378#2492(@PSeitz)

• cleanup top level exports #2382(@PSeitz)

• make convert_to_fast_value_and_append_to_json_term pub #2370(@PSeitz)

• remove JsonTermWriter #2238(@PSeitz)

• validate sort by field type #2336(@PSeitz)

• Fix trait bound of StoreReader::iter #2360(@adamreichold)

• remove read_postings_no_deletes #2526(@PSeitz)

Tantivy v0.22

What's Changed

Tantivy 0.22 will be able to read indices created with Tantivy 0.21.

Bugfixes

• Fix null byte handling in JSON paths (null bytes in json keys caused panic during indexing) #2345(@PSeitz)
• Fix bug that can cause get_docids_for_value_range to panic. #2295(@fulmicoton)
• Avoid 1 document indices by increase min memory to 15MB for indexing #2176(@PSeitz)
• Fix merge panic for JSON fields #2284(@PSeitz)
• Fix bug occuring when merging JSON object indexed with positions. #2253(@fulmicoton)
• Fix empty DateHistogram gap bug #2183(@PSeitz)
• Fix range query end check (fields with less than 1 value per doc are affected) #2226(@PSeitz)
• Handle exclusive out of bounds ranges on fastfield range queries #2174(@PSeitz)

Breaking API Changes

• rename ReloadPolicy onCommit to onCommitWithDelay #2235(@giovannicuccu)
• Move exports from the root into modules #2220(@PSeitz)
• Accept field name instead of Field in FilterCollector #2196(@PSeitz)
• remove deprecated IntOptions and DateTime #2353(@PSeitz)

Features/Improvements

• Tantivy documents as a trait: Index data directly without converting to tantivy types first #2071(@ChillFish8)

• encode some part of posting list as -1 instead of direct values (smaller inverted indices) #2185(@trinity-1686a)

• Aggregation

  • Support to deserialize f64 from string #2311(@PSeitz)
  • Add a top_hits aggregator #2198(@ditsuke)
  • Support bool type in term aggregation #2318(@PSeitz)
  • Support ip adresses in term aggregation #2319(@PSeitz)
  • Support date type in term aggregation #2172(@PSeitz)
  • Support escaped dot when addressing field #2250(@PSeitz)

• Add ExistsQuery to check documents that have a value #2160(@imotov)

• Expose TopDocs::order_by_u64_field again #2282(@ditsuke)

• Memory/Performance

  • Faster TopN: replace BinaryHeap with TopNComputer #2186(@PSeitz)
  • reduce number of allocations during indexing #2257(@PSeitz)
  • Less Memory while indexing: docid deltas while indexing #2249(@PSeitz)
  • Faster indexing: use term hashmap in fastfield #2243(@PSeitz)
  • term hashmap remove copy in is_empty, unused unordered_id #2229(@PSeitz)
  • add method to fetch block of first values in columnar #2330(@PSeitz)
  • Faster aggregations: add fast path for full columns in fetch_block #2328(@PSeitz)
  • Faster sstable loading: use fst for sstable index #2268(@trinity-1686a)

• QueryParser

  • allow newline where we allow space in query parser #2302(@trinity-1686a)
  • allow some mixing of occur and bool in strict query parser #2323(@trinity-1686a)
  • handle * inside term in lenient query parser #2228(@trinity-1686a)
  • add support for exists query syntax in query parser #2170(@trinity-1686a)

• Add shared search executor #2312(@MochiXu)

• Truncate keys to u16::MAX in term hashmap #2299(@PSeitz)

• report if a term matched when warming up posting list #2309(@trinity-1686a)

• Support json fields in FuzzyTermQuery #2173(@PingXia-at)

• Read list of fields encoded in term dictionary for JSON fields #2184(@PSeitz)

• add collect_block to BoxableSegmentCollector #2331(@PSeitz)

• expose collect_block buffer size #2326(@PSeitz)

• Forward regex parser errors #2288(@adamreichold)

• Make FacetCounts defaultable and cloneable. #2322(@adamreichold)

• Derive Debug for SchemaBuilder #2254(@GodTamIt)

• add missing inlines to tantivy options #2245(@PSeitz)

New Contributors

• @imotov made their first contribution in #2160
• @PingXia-at made their first contribution in #2173
• @giovannicuccu made their first contribution in #2235
• @BlackHoleFox made their first contribution in #2265
• @ditsuke made their first contribution in #2282
• @MochiXu made their first contribution in #2312

Full Changelog: 0.21...v0.22.0

Tantivy v0.21

Bugfixes

• Fix track fast field memory consumption, which led to higher memory consumption than the budget allowed during indexing #2148#2147(@PSeitz)
• Fix a regression from 0.20 where sort index by date wasn't working anymore #2124(@PSeitz)
• Fix getting the root facet on the FacetCollector. #2086(@adamreichold)
• Align numerical type priority order of columnar and query. #2088(@fmassot)

Breaking Changes

• Remove support for Brotli and Snappy compression #2123(@adamreichold)

Features/Improvements

• Implement lenient query parser #2129(@trinity-1686a)
• order_by_u64_field and order_by_fast_field allow sorting in ascending and descending order #2111(@naveenann)
• Allow dynamic filters in text analyzer builder #2110(@fulmicoton @fmassot)
• Aggregation
  • Add missing parameter for term aggregation #2149#2103(@PSeitz)
  • Add missing parameter for percentiles #2157(@PSeitz)
  • Add missing parameter for stats,min,max,count,sum,avg #2151(@PSeitz)
  • Improve aggregation deserialization error message #2150(@PSeitz)
  • Add validation for type Bytes to term_agg #2077(@PSeitz)
  • Alternative mixed field collection #2135(@PSeitz)
• Add missing query_terms impl for TermSetQuery. #2120(@adamreichold)
• Minor improvements to OwnedBytes #2134(@adamreichold)
• Remove allocations in split compound words #2080(@PSeitz)
• Ngram tokenizer now returns an error with invalid arguments #2102(@fmassot)
• Make TextAnalyzerBuilder public #2097(@adamreichold)
• Return an error when tokenizer is not found while indexing #2093(@naveenann)
• Delayed column opening during merge #2132(@PSeitz)

New Contributors

• @naveenann made their first contribution in #2093
• @cjrh made their first contribution in #2144
• @GodTamIt made their first contribution in #2145
• @ethever made their first contribution in #2165

Full Changelog: 0.20.2...0.21

0.20.1

What's Changed

• Fix building on windows with mmap by @ChillFish8 in #2070

Tantivy v0.20

What's Changed

Bugfixes

• Fix phrase queries with slop (slop supports now transpositions, algorithm that carries slop so far for num terms > 2) #2031#2020(@PSeitz)
• Handle error for exists on MMapDirectory #1988 (@PSeitz)
• Aggregation
  • Fix min doc_count empty merge bug #2057 (@PSeitz)
  • Fix: Sort order for term aggregations (sort order on key was inverted) #1858 (@PSeitz)

Features/Improvements

• Add PhrasePrefixQuery #1842 (@trinity-1686a)
• Add coerce option for text and numbers types (convert the value instead of returning an error during indexing) #1904 (@PSeitz)
• Add regex tokenizer #1759(@mkleen)
• Move tokenizer API to seperate crate. Having a seperate crate with a stable API will allow us to use tokenizers with different tantivy versions. #1767 (@PSeitz)
• Columnar crate: New fast field handling (@fulmicoton @PSeitz) #1806#1809
  • Support for fast fields with optional values. Previously tantivy supported only single-valued and multi-value fast fields. The encoding of optional fast fields is now very compact.
  • Fast field Support for JSON (schemaless fast fields). Support multiple types on the same column. #1876 (@fulmicoton)
  • Unified access for fast fields over different cardinalities.
  • Unified storage for typed and untyped fields.
  • Move fastfield codecs into columnar. #1782 (@fulmicoton)
  • Sparse dense index for optional values #1716 (@PSeitz)
  • Switch to nanosecond precision in DateTime fastfield #2016 (@PSeitz)
• Aggregation
  • Add date_histogram aggregation (only fixed_interval for now) #1900 (@PSeitz)
  • Add percentiles aggregations #1984 (@PSeitz)
  • [breaking] Drop JSON support on intermediate agg result (we use postcard as format in quickwit to send intermediate results) #1992 (@PSeitz)
  • Set memory limit in bytes for aggregations after which they abort (Previously there was only the bucket limit) #1942#1957(@PSeitz)
  • Add support for u64,i64,f64 fields in term aggregation #1883 (@PSeitz)
  • Add count, min, max, and sum aggregations #1794 (@guilload)
  • Switch to Aggregation without serde_untagged => better deserialization errors. #2003 (@PSeitz)
  • Switch to ms in histogram for date type (ES compatibility) #2045 (@PSeitz)
  • Reduce term aggregation memory consumption #2013 (@PSeitz)
  • Reduce agg memory consumption: Replace generic aggregation collector (which has a high memory requirement per instance) in aggregation tree with optimized versions behind a trait.
  • Split term collection count and sub_agg (Faster term agg with less memory consumption for cases without sub-aggs) #1921 (@PSeitz)
  • Schemaless aggregations: In combination with stacker tantivy supports now schemaless aggregations via the JSON type.
    • Add aggregation support for JSON type #1888 (@PSeitz)
    • Mixed types support on JSON fields in aggs #1971 (@PSeitz)
  • Perf: Fetch blocks of vals in aggregation for all cardinality #1950 (@PSeitz)
• Searcher with disabled scoring via EnableScoring::Disabled #1780 (@shikhar)
• Enable tokenizer on json fields #2053 (@PSeitz)
• Enforcing "NOT" and "-" queries consistency in UserInputAst #1609 (@denis Bazhenov)
• Faster indexing
  • Refactor tokenization pipeline to use GATs #1924 (@trinity-1686a)
  • Faster term hash map #1940 (@PSeitz)
  • Refactor vint #2010 (@PSeitz)
• Faster search
  • Work in batches of docs on the SegmentCollector (Only for cases without score for now) #1937 (@PSeitz)
  • Faster fast field range queries using SIMD #1954 (@fulmicoton)
  • Improve fast field range query performance #1864 (@PSeitz)
• Make BM25 scoring more flexible #1855 (@alexcole)
• Switch fs2 to fs4 as it is now unmaintained and does not support illumos #1944 (@Toasterson)
• Made BooleanWeight and BoostWeight public #1991 (@fulmicoton)
• Make index compatible with virtual drives on Windows #1843 (@yukun Guo)
• Auto downgrade index record option, instead of vint error #1857 (@PSeitz)
• Enable range query on fast field for u64 compatible types #1762 (@PSeitz) [#1876]
• sstable
  • Isolating sstable and stacker in independant crates. #1718 (@fulmicoton)
  • New sstable format #1943#1953 (@trinity-1686a)
  • Use DeltaReader directly to implement Dictionnary::ord_to_term #1928 (@trinity-1686a)
  • Use DeltaReader directly to implement Dictionnary::term_ord #1925 (@trinity-1686a)
• Add seperate tokenizer manager for fast fields #2019 (@PSeitz)
• Make construction of LevenshteinAutomatonBuilder for FuzzyTermQuery instances lazy. #1756 (@adamreichold)
• Added support for madvise when opening an mmaped Index #2036 (@fulmicoton)
• Rename DatePrecision to DateTimePrecision #2051 (@guilload)
• Query Parser
  • Quotation mark can now be used for phrase queries. #2050 (@fulmicoton)
  • PhrasePrefixQuery is supported in the query parser via: field:"phrase ter"* #2044 (@adamreichold)
• Docs
  • Update examples for literate docs #1880 (@PSeitz)
  • Add ip field example #1775 (@PSeitz)
  • Fix doc store cache documentation #1821 (@PSeitz)
  • Fix BooleanQuery document #1999 (@RT_Enzyme)
  • Update comments in the faceted search example #1737 (@DawChihLiou)

New Contributors

• @mhlakhani made their first contribution in #1733
• @pinkforest made their first contribution in #1746
• @DawChihLiou made their first contribution in #1737
• @mkleen made their first contribution in #1759
• @lonre made their first contribution in #1803
• @gyk made their first contribution in #1843
• @alexcole made their first contribution in #1855
• @Toasterson made their first contribution in #1944
• @vsop-479 made their first contribution in #1970
• @Tony-X made their first contribution in #1985
• @RTEnzyme made their first contribution in #1999
• @tottoto made their first contribution in #2018
• @nyurik made their first contribution in #2038
• @bazhenov made their first contribution in #1609
• @lavrd made their first contribution in #1422
• @tnxbutno made their first contribu...

Tantivy v0.19.2

Fixes an issue in the skip list deserialization, which deserialized the byte start offset incorrectly as u32.
get_doc will fail for any docs that live in a block with start offset larger than u32::MAX (~4GB).
Causes index corruption, if a segment with a doc store file larger 4GB is merged. (@PSeitz)

Tantivy v0.19.1

Hotfix on handling user input for get_docid_for_value_range (@PSeitz)

Tantivy v0.19

What's Changed

Bugfixes

• Fix missing fieldnorms for u64, i64, f64, bool, bytes and date #1620 (@PSeitz)
• Fix interpolation overflow in linear interpolation fastfield codec #1480 (@PSeitz @fulmicoton)

Features/Improvements

• Add support for IN in queryparser , e.g. field: IN [val1 val2 val3] #1683 (@trinity-1686a)
• Skip score calculation, when no scoring is required #1646 (@PSeitz)
• Limit fast fields to u32 (get_val(u32)) #1644 (@PSeitz)
• The DateTime type has been updated to hold timestamps with microseconds precision.
  DateOptions and DatePrecision have been added to configure Date fields. The precision is used to hint on fast values compression. Otherwise, seconds precision is used everywhere else (i.e terms, indexing) #1396 (@evanxg852000)
• Add IP address field type #1553 (@PSeitz)
• Add boolean field type #1382 (@boraarslan)
• Remove Searcher pool and make Searcher cloneable. (@PSeitz)
• Validate settings on create #1570 (@PSeitz)
• Detect and apply gcd on fastfield codecs #1418 (@PSeitz)
• Doc store
  • use separate thread to compress block store #1389 #1510 (@PSeitz @fulmicoton)
  • Expose doc store cache size #1403 (@PSeitz)
  • Enable compression levels for doc store #1378 (@PSeitz)
  • Make block size configurable #1374 (@kryesh)
• Make tantivy::TantivyError cloneable #1402 (@PSeitz)
• Add support for phrase slop in query language #1393 (@saroh)
• Aggregation
  • Add aggregation support for date type #1693(@PSeitz)
  • Add support for keyed parameter in range and histgram aggregations #1424 (@k-yomo)
  • Add aggregation bucket limit #1363 (@PSeitz)
• Faster indexing
  • #1610 (@PSeitz)
  • #1594 (@PSeitz)
  • #1582 (@PSeitz)
  • #1611 (@PSeitz)
  • Added a pre-configured stop word filter for various language #1666 (@adamreichold)

New Contributors

• @ryanrussell made their first contribution in #1380
• @boraarslan made their first contribution in #1382
• @pier-oliviert made their first contribution in #1415
• @kianmeng made their first contribution in #1445
• @akr4 made their first contribution in #1473
• @waywardmonkeys made their first contribution in #1524
• @nigel-andrews made their first contribution in #1608
• @theduke made their first contribution in #1624

Full Changelog: 0.18...0.19

Tantivy v0.18.1

• Hotfix on positions. #1629 (@fmassot, @fulmicoton, @PSeitz)

Tantivy v0.18

• For date values chrono has been replaced with time (@uklotzde) #1304
• Add histgram aggregation (@PSeitz)
• Add support for fastfield on text fields (@PSeitz)
• Add terms aggregation (@PSeitz)
• Add support for zstd compression (@kryesh)