This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-05-09
Channels
- # announcements (3)
- # beginners (61)
- # biff (20)
- # cider (13)
- # clerk (6)
- # clojure (58)
- # clojure-brasil (5)
- # clojure-europe (30)
- # clojure-nl (1)
- # clojure-norway (10)
- # clojure-uk (5)
- # clr (25)
- # core-async (2)
- # cursive (19)
- # datahike (5)
- # datalevin (1)
- # docker (1)
- # emacs (3)
- # fulcro (4)
- # hoplon (3)
- # hyperfiddle (91)
- # java (2)
- # juxt (5)
- # london-clojurians (1)
- # lsp (38)
- # malli (12)
- # nrepl (9)
- # off-topic (7)
- # polylith (15)
- # portal (49)
- # rdf (2)
- # re-frame (43)
- # releases (2)
- # shadow-cljs (30)
- # spacemacs (15)
- # sql (36)
- # tools-build (20)
- # xtdb (3)
My app-db is growing in size, with many nested maps. Decomposition of these nested maps is resulting in a lot of maps, which would be better handled with an actual database. Is moving to an actual db for app-db (like re-posh, subgraph, re-graph) worthwhile? I am doing a large refactor atm, so it would be a good time to consider it. What am I losing from staying with vanilla maps? I am also curious about performance metrics.
It is worthwhile.
Store data in EAV tuples or in normalized maps, gather whatever data you need when querying. It's much easier to work with the data and reason about things that way.
Performance will depend upon your use cases. Around 5-6 years ago I tried Datascript but it wasn't worth it because the performance was more than 10x worse than querying and normalizing data from plain maps nested maps entity->id->field->value
. I've heard that they have since improved the performance dramatically but I haven't checked it since.
I've heard datascript has always proritized low-cost database creation than performance, so chances are little has changed on that front
With that said, Asami is something to check out if you dont mind trading off cost of creation for query performance. It includes a basic query planner IIRC, and you can easily make in-memory databases from CLJS using maps
Thank you for the answers. As I understand, choice of datascript/asami/doxa is for the db, and I would need to handroll subscriptions or use another library like re-posh/subgraph on top of it. Would you recommend handrolling or using some library? Using the library I imagine would restrict the choice of databases.
Also, regarding datascript vs. asami: the app requests large amount of data from the server, and needs to store that data in the db. Then it would create a view based on that data, with minor changes made to the db from time to time. Would datascript be a better fit for this as it's optimized for db creation? What are some other options?
with doxa you can pass a pull expression to the sub as an arg i.e. a single sub allows you to dx/pull any entity. not sure if that’s true for dx/q but pull gets you pretty far
(defn single-entity
“return an de-normalised entity pulled from the api db using a pull expression”
[db [ {`:keys_ [id pull-expression] :as args}]]`
(when (nil? id) (throw (ex-info “id missing” args)))
(let [entity (dx/pull (`:api db) pull-expression [:id id])] (when (seq entity) entity)))`
asami may require a bit more work... but you should be able to use queries as subscriptions too
In the rationale.md:
"`doxa` has been optimised to work on relatively small amounts of data and if you mostly query the database using multiple joins, a probably better choice would be to use another db like datascript
or asami
. test before you choose."
I imagine for the re-frame use case, multiple joins rarely occur, as subscriptions typically act on small pieces of data. Is this the right evaluation? (As for my app, I believe it wouldn't need multiple joins as I have been chugging along okay with just maps.)
you can control that as part of your sub design. keep them granular enough that they always return “small” amounts of data. I use it for single entities but also for lists of entities and it performs well but the lists are always limited to 10s of entities
How has the performance been? How does it fair empirically compared with vanilla map?
excellent. it returns exactly (i.e. identical?) the same value for the same pull or query. that means that it doesn’t re-run those db reads and then re-frame doesn’t bother re-rendering because it has an identical? value
Also, maybe I am not understanding sth about re-frame, but in the single-entity
subscription function above, how does re-frame know how to recompute the entities?
if you use it right, you should see very few re-renders unless the result of a sub changes
I see. Does dx/pull
take care of the recomputation? I guess I am asking about how to let re-frame know that recomputation needs to happen, if we need to at all.
I often use the “scope-capture” lib to run these queries from a REPL. feels like a super-power
That makes sense, I am looking at subs tab for the first time on re-frisk, things make more sense now
@U6D1FU8F2, shameless plug 😎 but, https://github.com/zalky/reflet is a library I put together over the years. I think it does a really good job at fulfilling reactive graph data requirements for Re-frame. It was specifically designed to make integration into existing apps that already have a non-graph data design really easy. The goal is to avoid major re-writes if they're not necessary. Your graph and non-graph data all live in the same db, so you can selectively apply the graph data model where it would bring the most benefit. It's highly performant, and comes with a debugger specifically designed to work with a graph paradigm.
@U0510KXTU doxa is working well for me. I just have a minor issue; given entity, attribute, and value where value is a reference, if I use merge to replace value with value', I get [value, value'] (as in documentation). If I use put, I get entity, attribute, value' but if value' previously existed with its own set of attributes, they are overwritten. How should one handle this case? Right now the only way I can think of is manually deleting then merging, which seems inconvenient. Never mind, deletion doesn't work with references. Just using dx/update for now.