This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-10-22
Channels
- # announcements (9)
- # asami (52)
- # aws (1)
- # babashka (7)
- # babashka-sci-dev (12)
- # beginners (72)
- # calva (24)
- # cider (9)
- # clj-kondo (76)
- # cljs-dev (15)
- # clojure (19)
- # clojure-australia (4)
- # clojure-europe (33)
- # clojure-france (9)
- # clojure-gamedev (17)
- # clojure-nl (6)
- # clojure-portugal (5)
- # clojure-uk (5)
- # clojurescript (61)
- # clojureverse-ops (4)
- # code-reviews (23)
- # conjure (1)
- # data-science (2)
- # datalevin (6)
- # datomic (49)
- # gratitude (1)
- # helix (24)
- # holy-lambda (14)
- # jobs (3)
- # lsp (92)
- # malli (7)
- # missionary (8)
- # pathom (12)
- # proletarian (3)
- # re-frame (4)
- # remote-jobs (1)
- # shadow-cljs (4)
- # spacemacs (3)
- # sql (9)
- # tools-build (90)
- # vim (1)
- # xtdb (11)
Hey there 👋
I’m scanning through some asami code and I noticed that the d/entity
function has an arity-3 variant that includes a nested?
boolean. What does this do exactly? I’m experimenting with setting it to false
in an attempt to achieve a bit of laziness while crawling the graph, but it still seems to auto-pull nested entities.
Hmmm, I thought it wasn’t supposed to when set to false? I haven’t looked at it in a long while though!
Seems like the key lies on these lines here: https://github.com/threatgrid/asami/blob/40e8cac1eb8dd87ddf1e975455da27b9d688ab6b/src/asami/entities/reader.cljc#L82-L83
it covers the case of things like:
[{:db/ident "first"
:val 1}
{:db/ident "second"
:val 2
:inner {:db/ident "first"}}]
This has 2 “top-level” individually addressable entities. If you call:
(entity db "second")
you’ll get: {:val 2 :inner {:db/ident "first"}}
But if you use:
(entity db "second" true)
you’ll get: {:val 2 :inner {:val 1}}
If it finds a “top-level” entity, then it just includes a reference to the entity (e.g. {:db/ident "first"}
)
But if you say, “No, I really do want you to go in and retrieve this other thing” then it will
I see. So maybe this is a side effect of how I’m transacting my data. My use case is language AST analysis, so I’m directly transacting the AST straight into asami. I don’t think I have any true “top-level” entities in this case…
It’s a sequence of a single map, which is very deeply nested. I’m basically transacting one big tree.
You could always add on the attribute of {:tg/entity true}
if you want to tell Asami to treat the entity as a top-level thing
“top level” was an idea that was just created to deal with ingesting and exporting JSON
Ah nice, I was just going to ask whether I should normalize my data manually ahead of time. I should be able to walk this tree and assoc :tg/entity
easily.
Out of curiosity, are there ways to “walk” the graph other than with d/entity
? Maybe something lower level, like with d/graph
which I admittedly haven’t looked far into yet.
I’m shooting for something lazy. I saw your comment in the docs that hinted at d/entity
being lazy one day.
I’m wondering if I could contrib something here, but I think my use-case is too narrow. I know asami supports durable storage now too, so that might complicate things considerably in the lazy department…
Naively I want to slap a lazy-seq
in there and call it a day, but surely that’s overly simplistic?
Well, the entities are based on key-value pairs that are lazy. Right now, they get turned into an entity via (into {} …)
what it needs is a lazy-seq type of approach where you have an object that meets the clojure.lang.Associative interface, and holds onto a map of what has been read so far, plus the context to load up anything that’s missing
Yea, sounds a lot like Wilker’s https://github.com/wilkerlucio/pathom3/blob/master/src/main/com/wsscode/pathom3/interface/smart_map.cljc
That “lazy map” would be awesome to have as its own library even. I wonder if there is prior art in generalizing it.
To generalize it, I imagine you’d need to have the user pass in their own lazy-seq’d thunk that produces the next kv pair? Hm…that wouldn’t really work though…it’d be O(N) on key access…
Even datascript has a schema upfront for things like refs…asami’s would need to be more dynamic
Does asami support any special keywords on d/entity
maps? For example, datascript lets you walk backwards using reverse refs with an underscore: :person/_friends
Yea I think so too. As it stands I don’t see how it could happen. Are you picturing a separate API for this, like d/lazy-entity
kind of thing?
That makes sense. For some reason I thought it would change existing behavior, but if it does it means the implementation is wrong 🙂
I might take a crack at this and if I come up with something promising I will definitely share. It sounds like an interesting puzzle.
I think it should be relatively tractable. Reading an entity would return an object that is both an atom (so it can be shared) and a db (which is an immutable read-only structure). Getting any value by key is just a lookup in the map, and if it’s not there, then do a lookup in the graph associated with the db. There are 3 cases here: • If it’s a simple value, then just associate the key with the value in the cache, and return the value. • If it’s a sequential type (these are stored as linked lists), then I think it’s appropriate to get all of these eagerly. • If it’s an object type, then create a new cached entity, just like this one, with the same db, and a new root ID to build on
Cool, I think we’re on the same page. What is your reasoning behind eagerly fetching sequences? Not disagreeing at all, just curious. Does that have to do with durability?
Not at all. It just didn’t seem to be worth the hassle of laziness. But it would be so easy to wrap it in a lazy-seq
that perhaps it’s no big deal
Can do eager at first and then reach for lazy-seq
after maybe. That would keep the first draft focused on one thing.