Import Export index and auto swapping of indexes

[geminigeek]

hi,

i would like to request for a functionality to import export individual indexes and auto swapping of index on import if a new index is available based on a naming schema.

use case:
i have a low resource machine capable of running meilisearch, but not optimal for indexing, i want to build index on another machine and rsync that to the low resource machine, then trigger an import index command , that trigger should import the index and then should run a auto swap logic for the index based on a naming schema. this will make automating updates easy and efficient

[macraig]

Hi @geminigeek , can you expand on what you mean by "auto swap logic for the index based on a naming schema"?

[geminigeek]

naming schema can be anything you guys can decide a terminology of naming convention for example "indexName_autoSwap_epochTime"

old index name someIndex_autoSwap_epochTime (unix time)
new index name someIndex_autoSwap_newEpochTime (unix time)

extract epoch time from index name , condition ( if old epochTime is time and newEpochTime is time and old epochTime < newEpochTime), on index import swap and rename old like this "someIndex_autoSwap_rollback" i would like this ideally as it will be definitive we have an older index to compare results

[macraig]

If I understand correctly the idea would be:

1. have one index called indexName_autoSwap_110
2. import an index called indexName_autoSwap_210
3. check 210 > 110, then swap both indexes

Does that mean that you would have two index names? I imagine that to address the index in the end we only want indexName. So where would the _autoSwap_110 part be stored after importing?

How would this help you share the index between two instances?

Could you explain how the indexing machine works? Is it only available for indexing purposes? Why not run the search directly on the indexing machine instead?

[dureuill]

Rather than reusing the name of the index for this purpose, could we not use an explicit metadata attached to the index?

We could either reuse the existing updatedAt or create a new version field.

Then on import the body would contain:

• an URI describing the index to import
• the desired name for the index
• relevant metadata about the index (updatedAt, version if we add it)
• what to do if an index with that name exists: "error" | "overwrite" | "swap" (we'd need a way to specify the swapped name) | "overwriteIfNewer" | "swapIfNewer"

[geminigeek]

If I understand correctly the idea would be:

1. have one index called indexName_autoSwap_110
2. import an index called indexName_autoSwap_210
3. check 210 > 110, then swap both indexes

Does that mean that you would have two index names? I imagine that to address the index in the end we only want indexName. So where would the _autoSwap_110 part be stored after importing?

How would this help you share the index between two instances?

Could you explain how the indexing machine works? Is it only available for indexing purposes? Why not run the search directly on the indexing machine instead?

yes we will have 2 index momentarily , after swap old one need to be renamed or renamed in a way to rollback, A/B Test
as we swap index we will retain the old name, original name "indexName_autoSwap_110" all new named index will be ,

adding a name tag on index is requires as meilisearch uses only index name to initiate create an index object , else we will not be able to see in code this index is auto swappable

indexName_autoSwap_210 > indexName_autoSwap_110
indexName_autoSwap_211 > indexName_autoSwap_110
indexName_autoSwap_212 > indexName_autoSwap_110

our search SDK/ software will use only this index name "indexName_autoSwap_110"

sharing index import/export is important to run meilisearch locally on the server where the front end is hosted or backend is hosted. that machine need not have much ram just enough. in my machine where i build index, i can make multiple indexes for use in multiple projects , then i just need to export them to different machine.

the index machine is in Germany, have 128 Gb ram, it can make index faster , then after the index is done i will export it to my machine serving frontend where meilisearch runs just for search purpose in singapore , canada etc, if i run index on germany the network roundtrip is more, from locations far away.

it might not be ideal for all, but for my use case this setup works

[geminigeek]

Rather than reusing the name of the index for this purpose, could we not use an explicit metadata attached to the index?

We could either reuse the existing updatedAt or create a new version field.

Then on import the body would contain:

• an URI describing the index to import
• the desired name for the index
• relevant metadata about the index (updatedAt, version if we add it)
• what to do if an index with that name exists: "error" | "overwrite" | "swap" (we'd need a way to specify the swapped name) | "overwriteIfNewer" | "swapIfNewer"

yes, anything can be done, just the functionality of export and import of individual index is required, rest can be managed by developer on their own , based on requirement and use case

ps: adding a name tag on index is requires as meilisearch uses only index name to create an index object , else we will not be able to see in code this index is auto swappable