Fork me on GitHub
#re-frame
<
2023-05-09
>
joshcho18:05:15

My app-db is growing in size, with many nested maps. Decomposition of these nested maps is resulting in a lot of maps, which would be better handled with an actual database. Is moving to an actual db for app-db (like re-posh, subgraph, re-graph) worthwhile? I am doing a large refactor atm, so it would be a good time to consider it. What am I losing from staying with vanilla maps? I am also curious about performance metrics.

👀 2
p-himik19:05:20

It is worthwhile. Store data in EAV tuples or in normalized maps, gather whatever data you need when querying. It's much easier to work with the data and reason about things that way. Performance will depend upon your use cases. Around 5-6 years ago I tried Datascript but it wasn't worth it because the performance was more than 10x worse than querying and normalizing data from plain maps nested maps entity->id->field->value. I've heard that they have since improved the performance dramatically but I haven't checked it since.

👍 1
hifumi12323:05:43

I've heard datascript has always proritized low-cost database creation than performance, so chances are little has changed on that front

hifumi12323:05:50

With that said, Asami is something to check out if you dont mind trading off cost of creation for query performance. It includes a basic query planner IIRC, and you can easily make in-memory databases from CLJS using maps

steveb8n01:05:31

I can recommend doxa. It is designed with reframe reference checking in mind

joshcho07:05:55

Thank you for the answers. As I understand, choice of datascript/asami/doxa is for the db, and I would need to handroll subscriptions or use another library like re-posh/subgraph on top of it. Would you recommend handrolling or using some library? Using the library I imagine would restrict the choice of databases.

joshcho07:05:13

Also, regarding datascript vs. asami: the app requests large amount of data from the server, and needs to store that data in the db. Then it would create a view based on that data, with minor changes made to the db from time to time. Would datascript be a better fit for this as it's optimized for db creation? What are some other options?

steveb8n07:05:56

with doxa you can pass a pull expression to the sub as an arg i.e. a single sub allows you to dx/pull any entity. not sure if that’s true for dx/q but pull gets you pretty far

steveb8n07:05:47

here’s a sub fn I have in my apps…

steveb8n07:05:57

(defn single-entity “return an de-normalised entity pulled from the api db using a pull expression” [db [ {`:keys_ [id pull-expression] :as args}]]` (when (nil? id) (throw (ex-info “id missing” args))) (let [entity (dx/pull (`:api db) pull-expression [:id id])] (when (seq entity) entity)))`

👍 1
hifumi12307:05:10

asami may require a bit more work... but you should be able to use queries as subscriptions too

steveb8n07:05:46

@U0BBFDED7 can share more about Doxa if you decide to use it

👍 1
joshcho07:05:38

Reading the rationale.md is very promising, may be exactly what I am looking for

joshcho07:05:44

In the rationale.md: "`doxa` has been optimised to work on relatively small amounts of data and if you mostly query the database using multiple joins, a probably better choice would be to use another db like datascript or asami. test before you choose." I imagine for the re-frame use case, multiple joins rarely occur, as subscriptions typically act on small pieces of data. Is this the right evaluation? (As for my app, I believe it wouldn't need multiple joins as I have been chugging along okay with just maps.)

steveb8n07:05:15

you can control that as part of your sub design. keep them granular enough that they always return “small” amounts of data. I use it for single entities but also for lists of entities and it performs well but the lists are always limited to 10s of entities

👍 1
joshcho07:05:27

How has the performance been? How does it fair empirically compared with vanilla map?

steveb8n07:05:24

excellent. it returns exactly (i.e. identical?) the same value for the same pull or query. that means that it doesn’t re-run those db reads and then re-frame doesn’t bother re-rendering because it has an identical? value

👀 3
joshcho07:05:40

Also, maybe I am not understanding sth about re-frame, but in the single-entity subscription function above, how does re-frame know how to recompute the entities?

steveb8n07:05:04

if you use it right, you should see very few re-renders unless the result of a sub changes

joshcho07:05:54

I see. Does dx/pull take care of the recomputation? I guess I am asking about how to let re-frame know that recomputation needs to happen, if we need to at all.

steveb8n07:05:55

I don’t understand “know how to re-compute the entities”

steveb8n07:05:22

re-frame runs ALL active subs any time app-db is changed

joshcho07:05:43

That makes sense, and it's cached at the doxa level

steveb8n07:05:44

but that doesn’t matter if the subs are a single cache lookup

joshcho07:05:56

That makes way more sense

steveb8n07:05:09

yep. you can see the cache meta-data in the db along with the normalised data

👍 1
steveb8n07:05:01

re-frame 10x makes it easy to see it

steveb8n07:05:15

or your fav tool if you prefer something else

steveb8n07:05:41

I often use the “scope-capture” lib to run these queries from a REPL. feels like a super-power

joshcho07:05:54

That makes sense, I am looking at subs tab for the first time on re-frisk, things make more sense now

steveb8n07:05:23

yeah, the ALL active subs thing is not obvious up front

joshcho07:05:24

I imagine the cached meta-data is viewable on re-frame 10x?

steveb8n07:05:42

re-frisk too most likely

steveb8n07:05:08

I gotta go. any last questions?

joshcho07:05:16

Nope, thanks for the help!

steveb8n07:05:24

glad to help 🙂

joshcho07:05:26

Tremendously helpful, I will play around with doxa

zalky22:05:57

@U6D1FU8F2, shameless plug 😎 but, https://github.com/zalky/reflet is a library I put together over the years. I think it does a really good job at fulfilling reactive graph data requirements for Re-frame. It was specifically designed to make integration into existing apps that already have a non-graph data design really easy. The goal is to avoid major re-writes if they're not necessary. Your graph and non-graph data all live in the same db, so you can selectively apply the graph data model where it would bring the most benefit. It's highly performant, and comes with a debugger specifically designed to work with a graph paradigm.

👀 3
joshcho14:05:18

@U0510KXTU doxa is working well for me. I just have a minor issue; given entity, attribute, and value where value is a reference, if I use merge to replace value with value', I get [value, value'] (as in documentation). If I use put, I get entity, attribute, value' but if value' previously existed with its own set of attributes, they are overwritten. How should one handle this case? Right now the only way I can think of is manually deleting then merging, which seems inconvenient. Never mind, deletion doesn't work with references. Just using dx/update for now.

steveb8n20:05:15

There's a doxa channel. Best to ask future questions there

👍 2