@orama/plugin-data-persistence seems to require Node.js transform streams

[lmammino]

Describe the bug

I am using the restore function to load a previously persisted index and do client-side full-text search on the browser. When I am bundling the frontend using Vite I am presented with the following error:

Uncaught TypeError: Class extends value
undefined is not a constructor or null

I haven't done a super deep investigation but my current understanding on why this is happening is the following:

• The @orama/plugin-data-persistence module depends on the dpack module to support dpack serialization/deserialization. (this happens regardless of whether dpack is being used... I am only using JSON in my code)
• dpack requires Transform streams from Node.js.
• Vite, by default, does not polyfill Node.js modules and therefore the Transform stream is missing at runtime

I found a couple of workarounds for this problem so far:

Workaround 1: polyfill Node.js streams

Install vite-plugin-node-polyfills and then, In your Vite config:

// vite.config.ts

import { defineConfig } from 'vite'
import { nodePolyfills } from 'vite-plugin-node-polyfills'

// https://vite.dev/config/
export default defineConfig({
  plugins: [nodePolyfills({ include: ['stream', 'util'] })]
})

This will make the Node.js stream available at runtime. The drawback of this approach is that your bundle size will grow significantly (~500kb!).

Workaround 2: build your own JSON restore

If don't intend to use dpack and your persisted index is in JSON, you can "manually" restore it as follows (loosely extrapolated from this file):

import {
  type Orama,
  type RawData,
  create,
  load,
  search,
} from '@orama/orama'
import type { Schema } from './types' // your index schema type

async function loadDbFromJson(data: RawData): Promise<Orama<Schema>> {
  const db = await create({
    schema: {
      __placeholder: 'string',
    },
  })
  await load(db, data)

  return db as unknown as Orama<Schema>
}

// load your index data somehow (e.g. using fetch)
const resp = await fetch('https://.../_search.json.gz', { // your URL here
  headers: {
     'accept-encoding': 'gzip',
     accept: 'application/json; charset=utf-8',
  },
})
const data = await resp.json()
const db = await loadDbFromJson(data)

// use db for search
const results = await search(db, ...)

This approach keeps your bundle even smaller because you don't need to use the @orama/plugin-data-persistence client-side! But it's more code and complexity to maintain for the developer...

To Reproduce

1. Create an Orama index
2. Populate it with some data
3. Persist it using the@orama/plugin-data-persistence persistence plugin
4. restore the index in a client application bundled with Vite (default config)

This sample repo at this commit can be used as a starting point. Most of the restore business logic can be added in /src/TypeAhead.tsx

Expected behavior

• It would be nice if the internal implementation would not require the Node.js transform stream, so the plugin would work out of the box with a default Vite configuration
• It would be even better if dpack was optional...

On the second point, this is possibly a breaking change but I am thinking that it is a bit of a waste to require dpack even when is not going to be used.

An alternative approach would be to make that an optional dependency and to force the user to pass it explicitly to the plugin when needed. Something like:

import { restore } from "@orama/plugin-data-persistence";
import { dpackDeserializer } from "@orama/plugin-data-persistence/dpack"; // this could be a sub-module or another independent module that needs to be explicitly installed

const newInstance = await restore(JSONIndex, {
  deserializer: new dpackDeserializer()  // could be 'json' by default
});

Environment Info

OS: Mac Os 
Node: v20.18.1
Orama: 3.0.5
Orama persistence plugin: 3.0.5

Affected areas

Initialization

Additional context

No response

[micheleriva]

@matijagaspar could you look at this please?

[moritzlang]

I have the same issue when using @orama/plugin-data-persistence in a browser.