Fork me on GitHub
#datomic
<
2018-12-14
>
ro620:12:46

I'm trying to write a transaction function that links two entities together via a :ref, but one or both of the entities may not already exist in the db at the time this runs. I was hoping the upsert behavior would allow me to ignore whether they exist or not, but I'm getting

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: :db.error/not-an-entity Unable to resolve entity:
I'm using a lookup ref on an identity/unique attribute in the entity and value positions.

ro620:12:04

if I can't abstract over entity existence, what's the idiomatic way to check if the target entity existed from the result of a pull expression?

tomjack21:12:40

you just can't use lookup refs

tomjack21:12:45

but you can do e.g. [[:db/add "e1" :ident1 "id1"] [:db/add "e2" :ident2 "id2"] [:db/add "e1" :ref "e2"]]

ro621:12:15

so basically those first two are just setting up temp ids?

tomjack21:12:25

or even [{:ident1 "id3" :ref {:ident2 "id4"}}]?? 🙂

ro623:12:00

is that :ref thing really valid syntax?

matthavener16:12:26

I’m just using the schema from tomjack. :ref is a :db.type/ref attribute

tomjack21:12:01

right. in the old days we would've had to pick a partition there for each entity

ro621:12:14

sorry, the :ident1 stuff is throwing me a bit

tomjack21:12:46

[{:db/ident :ident1
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/unique :db.unique/identity}
 {:db/ident :ident2
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/unique :db.unique/identity}
 {:db/ident :ref
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/ref}]

ro621:12:53

those would be the identity/unique val?

ro621:12:01

that's interesting. I wonder if there's a technical reason why lookup refs can't work in this case.

ro621:12:19

I wanted to do it that way so the tx fn could accept anything valid to identify an entity, so I guess lookup ref or entity id at this point, and pass it through to Datomic without introspecting to figure out which type it was

tomjack21:12:57

given the existence of the "default partition" (and lack of partitions in cloud) it would seem reasonable to me

ro621:12:24

maybe just a "not yet implemented" on that one

matthavener21:12:38

ro6: I’ve always wished idents would eval to temp ids as well as entity ids, would be really cool

matthavener21:12:09

there’s a case where the semantics might be confusing though (imo)

ro621:12:27

what's that? I haven't thought deeply about it

matthavener21:12:14

[[:db/retract [:ident1 "thing"] :ident1 "thing"] 
{:thing1 "thing" .. other stuff ..}
{:thing1 "other" :ref [:ident1 "thing"]}]

matthavener21:12:09

currently, that ref would always point to the “old” thing

ro621:12:06

hm, I'll have to stare at that later...

ro623:12:00

is that :ref thing really valid syntax?

mikethompson23:12:23

What do I miss out on when if I use cloud Datomic. I'm aware of: 1. No excision 2. No backup/restore 3. Lambda cold starts (> 10 seconds) What else could be a pain point? I'm not trying to be negative. Everything is a tradeoff. I just want to go in with my eyes open.

ro623:12:09

Nothing in the Peer library that isn't also in the Client lib, so the entity api and multi-database queries

ro623:12:03

The cold start thing is surmountable. In the early stages you can just hit the endpoint periodically to keep it warm (I actually have an endpoint called /keep-warm), and I think they are working on a direct link from API Gateway to the EC2 instances that would bypass Lambda, and make $ sense as you scale.

lilactown23:12:57

I really like the lambda idea, but I think a lot of the slowdown is because the lambda is inside the VPC. could be wrong

steveb8n23:12:56

I’ve tested this and you can see the ENI init is approx 8 secs, the other 8 is the lambda. I consistently see 16 sec cold starts.

steveb8n23:12:33

after the first one, other lambdas start in approx 8 secs i.e. ENI is already in place

steveb8n23:12:19

if Ions stop using Lambda we will still have ENI cold start but that will be once and, with any active app, that will be almost never

lilactown23:12:51

hmm yeah. 8s still sounds rather slow

lilactown23:12:37

presumably that’s just by how Lambda, the Ultimate is implemented and could be optimized a lot further

lilactown23:12:08

it seems like the lambda code itself would be the simplest part to optimize since it’s just a data transform layer depending on the type of event.

lilactown23:12:33

I wonder if the new “layers” stuff announced at re:Invent would help. my completely naive hypothesis is that a lot of the cold-start time is loading the dependencies necessary for speaking to the Ion code in the right format

lilactown23:12:19

I’m surprised it’s that long outside of the ENI init, though

steveb8n23:12:20

agreed. I have some cljs lambdas not in a VPC and they cold start in 3 secs

steveb8n23:12:48

I suspect the 8 secs is due to my Ion deps

lilactown23:12:11

well the Ion deps shouldn’t affect cold start time

steveb8n23:12:29

you are right, they are not in the Lambda. my mistake

steveb8n23:12:27

can’t wait for that direct access architecture. will be a big upgrade for all of us

ro623:12:07

true, I've read that

ro623:12:28

takes it from 2 sec to 15 or so maybe

lilactown23:12:46

I guess we can cross our fingers that AWS helps ameliorate the cold start time in a VPC. the security model of datomic makes it hard to do anything else