Fork me on GitHub
#datomic
<
2019-05-01
>
donaldball15:05:58

Does anyone have a recipe for deploying a datomic pro jar to a private s3 maven that works with STS credentials e.g. granted by using aws-vault?

Drew Verlee16:05:30

so db.unique/identity means no duplicate EAV's right? e.g insert 1 color blue, 1 color blue wouldn't be allowed. as where db.unique/value. means no duplicatte AV so 1 color blue, and 2 color blue wouldn't be allowed.

Drew Verlee16:05:19

where the format is Entity, attribute, value

Ben Hammond16:05:19

no if its a :db.unique/identity then that VA will be used to identiy the entity

Ben Hammond16:05:25

like a primary key

Ben Hammond16:05:28

if you have two contradictory identities pointed at the same entity then you will get an exception

Ben Hammond16:05:34

:db.unique/value will chuck an exception if you try to assert color blue on two different entities

Ben Hammond16:05:09

:db.unique/identity will attempt to conclude that 1 and 2 are the same entity

Ben Hammond16:05:27

which might chuck a contradiction exception, depending

Drew Verlee16:05:56

Thanks @benha, if you dont use db.unique/value would it mean then you can refer to an entity using multiple identities?

Ben Hammond16:05:22

it would, which can be extremely helpful

Drew Verlee16:05:38

thanks again. I'm not sure my last statement is right, nothing in the docs makes it clear to me that once you assign db.unique/value to an entity its not the only AV you can use to refer to the entity. Just confirming that this is indeed the case 🙂 secondly, assuming the above isnt true, it would seem a good thing to use db.unique/value as its rare that you want to return two entities if your using a key, so it would be nice to enforce that. Again, thats assuming db.unique/value doesn't close that entity to one AV (primary key).

Ben Hammond16:05:37

and sometimes a nasty surprise

uwo16:05:05

Is it always best practice to dereference the return value of d/transact, in order to realize and handle exceptions that only occur after .get is called on the returned Future? Or is it acceptable to call d/transact without a dereference, fire-and-forget style?

favila16:05:45

This question is really "is it acceptable not to inspect for and handle errors?" Probably not?

favila16:05:24

Without deref, you will only get a timeout exception thrown from d/transact (d/transact-async will never throw)

favila16:05:39

so you will never know about tx problems without deref-ing

uwo16:05:12

Thank you!!

yedi19:05:29

hey all, so the following will return all values of ?e where age is 42

[:find ?e 
 :where [?e :age 42]]

yedi19:05:52

what if I wanted to determine if there was any ?e in the dataset that had age set to 42? without having to check the entire unified dataset

yedi19:05:02

so basically clojure's some

yedi19:05:31

not sure if this question even makes sense from a logic programming perspective

Alex Miller (Clojure team)19:05:23

I don't know if this is the best answer, but Datomic gives you raw access to the indexes

Alex Miller (Clojure team)19:05:44

and this is basically a question you can ask of the VAET index via the datoms api https://docs.datomic.com/client-api/datomic.client.api.html#var-datoms

Alex Miller (Clojure team)19:05:29

(d/datoms db {:index :vaet, :components [42 :age]}) gives you an iterable of those entities that have :age 42, which you check for emptiness

yedi19:05:00

mmm, but unfortunately you wouldn't be able to do that within a datomic rule

yedi19:05:41

my use-case is trying to speed up a query by not having to check all ?es if the first ?e already passes the clause

Alex Miller (Clojure team)19:05:25

well, not sure I'm qualified to answer that

yedi19:05:40

no worries, thanks for attempting

yedi20:05:15

another question - can someone confirm that ors in datalog don't short-circuit? e.g: both fast-clause and slow-clause will run even if fast-clause is true

(or [(fast-clause? e)]
    [(slow-clause? e)])

Keith20:05:29

I’ve been wondering the same thing myself. I suspect that they do not short-circuit, but I would love clarification on that.

yedi20:05:44

from some testing on my end, it does not appear to do so

donaldball20:05:26

Have you tried using rules? I’d be curious to see if they have the same behavior.

yedi20:05:50

what i tested was using rules

donaldball20:05:03

well then! 🙂

Keith20:05:29

I’ve been wondering the same thing myself. I suspect that they do not short-circuit, but I would love clarification on that.

benoit20:05:30

@yedi This should return only one value:

[:find ?e .
 :where [?e :age 42]]
Note the period after ?e.

yedi20:05:09

thats true, but the clause im looking at is within a rule

grzm20:05:12

@jaret @marshall Have you considered exporting the DatomicLambdaRole name as an export from the Cloudformation templates for Datomic Cloud? It would make adding policies to the role easier instead of looking it up and supplying it as a parameter. Small thing, but I've wished for this more than once.

yedi20:05:50

so instead of [?e :age 42] that would bind all the results to ?e it'd be nice to have something like [?b (some? e? :age 42)]

yedi20:05:22

or something like that idk

benoit20:05:41

Regarding the or, you should interpret or as a union of sets. Datomic will find all entities matching the fast-clause and the slow-clause and "set union" them.

yedi20:05:01

ok that makes sense and is a good way to look at it (so basically from an procedural perspective: no short circuiting)

benoit20:05:00

IMO the best way is to not take the "procedural perspective" at all because it's going to cause more confusion.

yedi20:05:12

yea that's fair

benoit20:05:27

For your query, keep in mind that you can call clojure functions in your clauses.

benoit20:05:45

[(datomic.api/datoms ...) ?result]

yedi20:05:24

ok i think i might need to do that. basically im dealing with a performance problem when using queries with multiple deeply nested rules

yedi20:05:59

where rules with the same variables are showing up tons of times in the query beacuse theyre included within other rules

benoit20:05:34

That said, the perf issues I encountered in clauses where most often due to bad ordering.

yedi20:05:37

hm not sure if that made sense, or even if that is the source of the performance problems

yedi20:05:56

yea our team already went through and made a bunch of ordering changes that def helped

yedi20:05:25

but now weve limited the remaining perf problems to specific queries that leverage a ton of rules

benoit20:05:26

A few heuristics I found helpful: 1. Make sure to have most specific rules first (cardinality of the matching set should be pretty low) 2. Make sure you can navigate from one rule to the next. If a rule starts doing something completely different from the rule before it, it basically means you will do a cartesian product)

yedi20:05:54

ok interesting, ill have to do some thinking around those heuristics

yedi20:05:24

and if you kinda want more clarity into the case im looking at, i have a rule that looks something like this:

(visible-instances [?person ?group] ?instance)
 [?instance :instance/group ?group]
 (or-join [?person ?group ?instance]
          (group-admin? ?person ?group)
          (person-can-see-instance? ?person ?instance))

yedi20:05:54

where the rule person-can-see-instance? also has the clause (group-admin? ?person ?group)

yedi20:05:25

so if there are like 5000 instances, that group-admin rule might be getting applied 5000 times, even if its already been done once

yedi20:05:40

and im not sure if thinking about datalog in this way even makes sense

benoit20:05:59

AH AH I optimized this kind of rules for permissions last week in our system.

benoit20:05:56

Are you trying to check whether a given user has permission to view a given instance?

yedi20:05:18

well the example above returns all instances the user has the permission to see

yedi20:05:37

person-can-see-instance? does what you say, checks for a specific instance

yedi21:05:54

so in visible-instances if the user is the group-admin, i don't want it to resolve the clause (person-can-see-instance? ?person ?instance) because i already know it can see all instances in that group

yedi21:05:12

but based off your explanation above, it is gonna resolve both clauses and merge them

benoit21:05:19

[(visible-instances [?person ?group] ?instance)
 [?instance :instance/group ?group]
 (group-admin? ?person ?group)]

[(visible-instances [?person ?group] ?instance)
 (not (group-admin? ?person ?group))
 (person-can-see-instance? ?person ?instance)]

yedi21:05:19

into a set

benoit21:05:49

I don't know whether it solve your overall perf issues but it should bypass your expensive clauses..

yedi21:05:27

i think this is the correct version:

[(visible-instances [?person ?group] ?instance)
 [?instance :instance/group ?group]
 (group-admin? ?person ?group)]

[(visible-instances [?person ?group] ?instance)
 [?instance :instance/group ?group]
 (not (group-admin? ?person ?group))
 (person-can-see-instance? ?person ?instance)]

yedi21:05:34

but yea, that seems suuuuuper promising

yedi21:05:08

well i guess the group-admin? clauses should go above the ?instance clauses, so that it makes the set go to 0 before it tries to find all the instances for that group

benoit21:05:41

good point

benoit21:05:59

But I'm not sure why you need two different rules for visible-instances and person-can-see-instance. It seems those could be one rule and only take the ?person and the ?instance. If you call the rule without binding ?instance it will bind all visible instances. If you bind ?instance it will let you know whether the person can see this specific instance.

yedi22:05:58

all true, except it won't be limited to visible instances in that group, unless you added that binding prior to the rule

yedi22:05:48

which you could do, but in the effort of DRY, making a rule that already encapsulate that group binding seems preferred

wei21:05:55

is retractEntity not available in datomic cloud? I'm getting the following exception trying to remove an entity in a transaction:

ExceptionInfo Unable to resolve data function: :db.fn/retractEntity  clojure.core/ex-info (core.clj:4739)

wei21:05:40

oh nevermind it's been renamed

wei21:05:37

was looking at the on-prem docs, in cloud it's db/retractEntity

yedi22:05:21

@me1740 thanks a lot, that helped for that one query

yedi22:05:05

but it appears that this strategy may get unwieldy. like if your "or" clause is supposed to have 3 or more diff branches instead of 2

yedi22:05:41

i would have to include multiple negation clauses at the top level of the rule (similar to how we did (not (group-admin? ?person ?group))) instead of just the one