Hi, I would like some suggestions with the following, I have a data structure similar to this:
clojure
[{:db/id 10 :record/rows [20 30 40 50]}
{:db/id 60 :record/rows [70 80]}
{:db/id 20 :row/name "name1" :row/valid true :row/value "value1"}
{:db/id 30 :row/name "name2" :row/valid false :row/value "value2"}
{:db/id 40 :row/name "name3" :row/valid false :row/value "value3"}
{:db/id 70 :row/name "name4" :row/valid true :row/value "value4"}
{:db/id 80 :row/name "name5" :row/valid false :row/value "value5"}]
With several hundred thousand maps. I need to query this data structure to obtain for example:
"Get :row/name of rows that have :row/valid false grouped by :record/rows" or
"All rows that are in the same record as the row with :row/value value4"
I can do this with a datalog query:
clojure
(d/q '[:find (pull ?r [* {:record/rows [:row/name]}])
:where
[?e :row/valid false]
[?r :record/rows ?e]]
@conn)
=> [[{:db/id 60, :record/rows [#:row{:name "name4"} #:row{:name "name5"}]}]
[{:db/id 10, :record/rows [#:row{:name "name1"} #:row{:name "name2"} #:row{:name "name3"}]}]]
(d/q '[:find (pull ?r [* {:record/rows [:row/name]}])
:where
[?e :row/value "value4"]
[?r :record/rows ?e]]
@conn)
=> [[{:db/id 60, :record/rows [#:row{:name "name4"} #:row{:name "name5"}]}]]
But unfortunately the performance of datahike when inserting thousands of rows is not so good, (around 4 minutes per each 100000 rows).
The intention is to use this as an internal data structure for a migration tool (from xml files to an sql database), so the datahike data is not meant to be permanent, I'm currently using it with an in-memory datastore.
Is it worth using datahike for this purpose, or should instead use something more light weight, like pure clojure (by manipulating maps), [specter](https://github.com/redplanetlabs/specter) or [meander](https://github.com/noprompt/meander)?
With pure clojure, it might require several functions to replicate the queries. And the two other libraries have a learning curve.
I like the fact that with datahike the queries are concise. Any suggestions would be appreciated.