Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-04-09

Channels

#datomic
<
2022-04-09
>
nottmey20:04:45

I just stumbled upon this https://forum.datomic.com/t/datomic-fulltext-search-equivalent/874 from 2019 and I’m wondering whether the statements regarding search are outdated or not. To sum it up, when I follow the setup for datomic cloud, there will be no standard way of searching string based attributes in an indexed way? (Ofc. there is scanning the index with a custom predicate, but that’s not advisable, right?) So when my frontend needs any reasonable search, the next big thing is to setup a third party search service? Are there common solutions or other options?

enn20:04:15

Honestly you are best off using some secondary projection (like ES) anyway. Among its other limitations, the (on-prem) Datomic fulltext indexing can’t be removed on an attribute after being enabled, meaning that if you do go to ES down the road you’ll still be paying the cost of that indexing essentially forever.

👍 1
nottmey21:04:09

yea, on-prem fulltext is a dead-end (it’s just nice because of the datalog predicate) I’m more like wondering what “paved paths” exist as an alternative.

tony.kay23:04:04

We’re using ElasticSearch (Opensearch). It was too hard to hook a thread up to follow the transactions (via tx-range) to keep the indexes up-to-date. But yeah, it was an in-house solution and not something that was very “paved”

👌 1
nottmey11:04:10

Idea: Before leaving the comfy space of the datomic query engine for including full-text-search, I could (temporarily) use the schema of datomic for a rudimentary (non-scanning) search itself, right? -> Tokenizing every searchable attribute and saving a relation between entity and token. -> Then, on query, tokenizing the search request, matching the tokens and scoring the related entities based on the matches. Has anyone heard of this approach before? Are there e.g. “datomic extensions” available for it? (I know it would generate a lot more data, depending on how loosely you generate tokens. But I assume you would “pay” for this overhead anyways, by running ES somewhere else.)

nottmey11:04:10

Idea: Before leaving the comfy space of the datomic query engine for including full-text-search, I could (temporarily) use the schema of datomic for a rudimentary (non-scanning) search itself, right? -> Tokenizing every searchable attribute and saving a relation between entity and token. -> Then, on query, tokenizing the search request, matching the tokens and scoring the related entities based on the matches. Has anyone heard of this approach before? Are there e.g. “datomic extensions” available for it? (I know it would generate a lot more data, depending on how loosely you generate tokens. But I assume you would “pay” for this overhead anyways, by running ES somewhere else.)