Fork me on GitHub
#asami
<
2021-11-15
>
zeitstein12:11:21

Thinking through using entity to read a deep tree. Using entity returns a StackOverflowError:

(defn deep-level [id]
  [{:id id}
   {:id (dec id) :child {:id id}}])

(defn deep-branch [level]
  (vec (mapcat deep-level (range 1 level))))

(def deep-data (deep-branch 200)) ; upper limit on my machine

(d/transact conn deep-data)
(d/entity conn 0 true)
@quoll, have you thought about providing a 'traversal' limit for entity? I'm thinking of something like Datomic' pull https://docs.datomic.com/cloud/query/query-pull.html#recursive-specifications. I guess the recent discussion on making entity lazy is relevant.

quoll12:11:34

Oh. That's a regression. During entity retrieval it's supposed to track where it's been and cut loops

quoll12:11:57

I’ll be online soon, and will look at it

quoll13:11:32

OK, online now, and… oh! I hadn’t looked at this properly when it was on my phone! This is a different issue altogether!

quoll13:11:43

@U02E9K53C9L I made the conscious decision to parse structures (and rebuild them) recursively. That made for a nice parallel for the stack and the data structure, but also I couldn’t think of any use case for a schema where things would get so deep as to exhaust the stack. Obviously, I wasn’t very imaginative 🙂

quoll13:11:54

I suppose I could rewrite it to not recurse, but that would take a lot of work. Given the limited use of this, then I think you have a point about just imposing a limit

quoll13:11:23

I could add it to the structure that records where it’s been (which avoids loops)

zeitstein14:11:10

If it's a quick fix, it would be nice to have a band-aid 🙂 But, it's not a burning issue for me by any means. Yet, at least 🙂 I might end up having to break up reading so much data at once, anyway. As always, thank you for being so kind with your time!

quoll14:11:19

Not a problem

zeitstein14:11:28

A couple of hopefully quick questions: 1. Any suggestions on what to do when an attribute's value should naturally be structured as collection, but there is no need to use Asami's (awesome) arrays/entities? It's just data that I would want to read, but not query for or anything like that. Store as string? Or don't worry about it? :) 2. Do you have an estimate on when durable storage might be available on CLJS? Thanks!

quoll14:11:27

1. Serializing may be a good idea. There is actually serializing support internally, but the destructuring tends to get to it first. 🙂 I need to test, but maybe it would work if the structure were put directly into a :db/add statement? Something like: [:db/add eid :attribute [:v1 :v2 {:k "v"}] 2. No. I have a lot happening at the moment that is taking my weekends and evenings, and this isn’t at the top of my list

quoll14:11:42

(Cisco isn’t working on Asami after all. It’s just me)

zeitstein14:11:45

Your example works, but it's fidgety:

[:db/add eid :attribute [:v1 :v2]]
stores nil, while
[:db/add eid :attribute ["a" "b"]]
works correctly.

zeitstein14:11:54

I see now in the changelog you've mentioned this before > Arrays and maps can now be serialized as values. This will show up in the API soon.

quoll14:11:01

The problem with the keyword thing is that it interacts with the refs

quoll14:11:24

Since that’s a syntax to do lookups for objects

quoll14:11:04

What’re they called…? ID Refs, I think?

quoll14:11:54

So if I say [:db/ident "fred"] then it will replace it with the entity reference that has that value as a :db/ident

zeitstein15:11:06

Yes, that's what I thought, as well.

zeitstein15:11:52

Though, I think {:db/ident "fred"} was used in wiki. Though neither form works in a :db/add statement.

zeitstein15:11:03

[{:db/ident "hej"}
 {:db/ident "test"}
 [:db/add "test" :ref [:db/ident "hej"]]]
:ref stores nil, as above.

zeitstein15:11:25

Map version produces datom:

"test" :ref #:db{:ident "hej"}

quoll17:11:25

There are two problems with the above:

[{:db/ident "hej"}
 {:db/ident "test"}
 [:db/add "test" :ref [:db/ident "hej"]]]
• Using [:db/ident "hej"] as a ref ID means that it looks up the database for that value… and it’s not there. Because you’re in a transaction that’s creating it. I need to check, but this behavior may be consistent with Datomic. A ref lookup like this would work in a second transaction, after the {:db/ident "hej"} ha already been inserted. • If you wanted that to store an array of 2 elements instead of being a ref lookup, then it would fail, because it looks like a ref lookup. What this means is that it’s a vector of 2 elements and the first is a keyword. In general, I’d like arbitrary objects to be storable in this place, but I’m not sure what to do about the ambiguity. For now, I’ve tried saying, “Lookup :id/ident "hej" and if that fails, then it’s a 2 element array.” But maybe that’s bad. What if you WANTED a 2 element array of [:id/ident "hej"]? It’s not possible (actually, a list might work instead). Or what if you wanted the ref, but you have a typo. Instead of getting a nil because you looked up the wrong thing, you’ll get a 2 element array inserted.

quoll17:11:42

But I pushed it up anyway, because… why not? 😜

zeitstein19:11:53

• Right, sorry about that. I had been using the entity form exclusively, where it's possible to do this in a single transaction. • Before getting into it, I want to say having to serialize is not that big a deal for my use case. (The entity stack overflow is a bigger issue.) I'm getting confused, so I'm going to try to break it up. Sorry if it gets too long, and feel free to disregard. 1. First of all, from an API standpoint, it might be confusing that [:db/add "test :ref {:db/ident "hej"}] would not produce the same result as {:db/ident "test" :ref {:db/ident "hej"}} (the single transaction issue). 2. The issue you raise is how to treat [:db/ident "hej"]. One idea is to only allow the map form {:db/ident "hej"} for lookup refs. I imagine wanting to store a map (as opposed to an entity ref) like that would be a much rarer use case. 3. Another issue is storing Clojure's collections as values (nodes having only incoming edges) vs. as Asami's arrays/entities (regular nodes). Two ideas that come to mind are: having another kind of attribute annotation, or perhaps utilising the transaction schema idea you've been working on?

quoll19:11:30

on those last things: 1. I can’t see if it happens, but I thought [:db/add "test :ref {:db/ident "hej"}] might do the same lookup as the id ref form (the vector). If not, then perhaps it should. I’m getting uncomfortable with the ambiguity though. 2. I preference Datomic’s syntax every time. That means that I can’t drop the vector form, since that’s what Datomic does. 3. Annotations work, though this is messy. The schema is a better idea. I have been thinking about this, but haven’t written code yet.

zeitstein19:11:56

1. Not sure if there was a misunderstanding. To be clear, I was referencing the fact that, in a single transaction, this works as intended

[{:db/ident "hej"}
 {:db/ident "test"}
 {:db/ident "test" :ref {:db/ident "hej"}}]
but this doesn't
[{:db/ident "hej"}
 {:db/ident "test"}
 [:db/add "test" :ref {:db/ident "hej"}]]
But, to what you were saying: the map form does indeed work. 2. Makes sense! 3. Looking forward to it :)

zeitstein19:11:23

I'd really like to contribute, but being a beginner means that prospect is daunting and is going to take time. I'll start digging into the repo. Thanks for your hard work!

quoll21:11:26

1. Hmmm… OK. I can probably make work. Those idents get stored in different phases, (entities vs :db/add) but I think I can make it work either way

👍 1