RFC: Database sync tool

[macraig]

The purpose of this discussion is to discuss the need for a database syncing tool that syncs data from users' databases to Meilisearch.

[paradoxloop]

This is a huge painpoint for us. Currently we have numerous tables (that have heave write updates) and we a column in each called "last_sent_search" which we poll every hour and push all the ones that have an "updated_at" greater than "last_sent_search":

• This doesnt help us manage deleted which have not worked through yet and just filter on return.
• Places an uncessary burden on db and isn't even real time.

We know we could use postgres triggers (or one of the syncing tools) to achieve sometime more reliable but have not done the engineering work yet. A more native solution for meilesearch would be great.

[macraig]

Feedback from @vaddenz in meilisearch/documentation#2926 (comment)

Usability: Following the project's configuration guide, the project was unable to start and throwed some exceptions. Here is a related issue: long2ice/meilisync#94
Reliability: Seems that meilisync was unable to capture some changes made in the DB
Poor maintenance
As for the sync tool, I'd suggest that MeiliSearch consider officially integrating some common utilized projects, e.g. Canal, Debezium etc

[ja7ad]

@macraig New sync tool is available with custom features and concurrent sync .

https://github.com/ja7ad/meilibridge

[ja7ad]

I think instead of writing sync tools we can create a connector destination for existing cdc projects such as debzium, airbyte or estuary.

I'm trying to contribute to this projects to add destination connector Meilisearch.

[joebnb]

is this feasible by official? we really need this to gather document from database

[macraig]

Hi @joebnb , we'd like to do this eventually. Can you expand on your use case and needs? It will help us shape a solution in the future. Thanks in advance!

[joebnb]

Hi @joebnb , we'd like to do this eventually. Can you expand on your use case and needs? It will help us shape a solution in the future. Thanks in advance!

Thank for attention this issue, i'm using meilisearch on web develop in rust, and want get a tool that not implement by Java(Elastic Search),cause the performance. like 99% of web dev they all have a relation data base. and most of them need develop full text serach from database, and it would be three type (postgres,mysql,other sql).

Current significant problem is when developer integrate meilisearch to them project, they read documents and install and tried perform search, when they may say meilisearch is good choice let's find up how to import data from my database,but there is no solution or only very heavy soultion.

• write your sync logic by your self(i do this,but there no Change Data Capture)
  • if implement CDC, i need to learn Postgres trigger and function and sql listener with sea-orm(current i'm doing,it's not valuable for uncommon knowledge，because this kind tool most of are support database sync), and i need maintain it by myself.
• use https://www.peerdb.io/ （CDC included）
  • it's a very heavy service when i look docker-compose found

actually i need cdc, i want when insert or update database meilisearch will do a real-time sync automatically.once configured.this action is transparent for developers, and i just operate the database and all things would be done

This is related discussion: https://github.com/orgs/meilisearch/discussions/680#discussioncomment-12510702

[macraig]

Thanks for the detailed description! We'll take it into account for our future solution :)

[florian-strasser]

I totally agree and would say that some sort of database connectivity would improve the process by 100%. I guess currently everyone is developing there own tools to get there data recreated as documents and keeping this up2date is quite painful.

This is actually the reason which holds me back from actually using meilisearch in a real live project

[macraig]

Hi @florian-strasser, we know this is a common pain point, and we plan to work on solving it later in the year. In the meantime, feel free to share more about your specific needs so we can consider them when shaping the solution.