Fork me on GitHub
#datomic
<
2019-02-06
>
Twice04:02:55

Hey guys! Sorry for a stupid question: how should I omit certain :db/ids in my query? Tried with (not [?e :db/id ?currId]) but that clearly doesn't work.

favila05:02:20

[(!= ?e ?blacklisted-eid)] (or a literal)

favila05:02:55

For larger sets (not [(contains? ?bad-eid-set ?e)])

👍 5
favila05:02:04

Can be faster

favila05:02:25

Too clever: (not [(identity ?e) ?blacklisted-eid])

favila05:02:46

Also you can repeat the binding clause with a literal

favila05:02:47

(not [?bound1 ?bound2 ?blacklisted-eid]) [?bound1 ?bound2 ?e]

favila05:02:04

(This is probably going to be slower)

Twice05:02:56

Thanks! Now it seems obvious 😁

thomas11:02:24

Hi Everyone... we are currently running Datomic on-prem version 5067 and management is wondering what it would take to upgrade to the latest version? Any idea? Any Incompatibilities? (we use postgres as a DB).

chrisblom12:02:21

Normally you can just stop the transactor, update and start the updated transactor, and then update all the clients, but some versions have had special cases. 5067 is pretty old so better check https://docs.datomic.com/on-prem/changes.html

thomas11:02:47

and what would be a good way to do this? do a Datomic DB dump... install new version and import?

chrisblom12:02:27

With dynamodb stop,update,start worked fine, but i'd do a backup before just to be sure

favila13:02:14

We’ve upgraded for 5 years (without changing storage) and have rarely had to do anything other than restart the txor with the newer version

John Miller16:02:06

Ion question: Can you deploy multiple projects to a single query group? We have several workloads that peak at different times and so could efficiently share the same compute resources. But they are different projects.

marshall16:02:51

@jmiller Each query group can only be the target of a single Code Deploy application, so you could serve multiple applications from a single QG, but the code for all of them would need to be in the same Ion repository

marshall16:02:07

or use a “master” repository that had deps on all the others

John Miller18:02:02

Thanks. That was how it looked but I wanted to make sure.

tyler20:02:39

I’m running into some unexpected behavior using CAS, I was hoping someone might be able to point me in the right direction. I’m trying to transact a payload:

[[:db/add "account" :account/email (:email identity)]
 [:db/cas "account" :account/organization nil "org"]
 [:db/add "org" :organization/name (:name args)]
 [:db/add "org" :organization/id id]]

Where :account/organization is cardinality one and :account/email is unique identity. I’d expect that if I ran this transaction twice, the :db/cas would fail as the organization is no longer nil, however, I am able to transact this many times without failure. Feels like I’m missing something fundamental but I cannot see it.

Dustin Getz20:02:57

Those string tempids identify a new entity each time the transaction runs

Dustin Getz20:02:41

:account/email maybe is not set to :identity ?

tyler20:02:42

Ah, so if I break it into two steps: Upsert the account and then upsert the organization it should work?

Dustin Getz20:02:22

You shouldn’t have to do that

tyler20:02:03

If I upsert the account first and then transact:

[[:db/cas [:account/email (:email identity)] :account/organization nil "org"]
 [:db/add "org" :organization/name (:name args)]
 [:db/add "org" :organization/id id]]
It behaves as expected.

Dustin Getz20:02:21

It could be a bug

Dustin Getz20:02:26

Is the first version dangling a bunch of entities or something? Is there any evidence that an upsert didn’t resolve?

tyler20:02:59

I’m trying that right now with a fresh db. I don’t think so but going to confirm.

tyler20:02:45

The accounts behave as expected, but a new organization entity gets created each time

Dustin Getz20:02:44

And in the second form @ 3:44?

marshall20:02:29

@tyler can you try with the map form instead of list form?

marshall20:02:44

for the things that arent the call to cas

tyler20:02:18

Yeah one sec

marshall20:02:55

{:db/id "account" :account/email (:email identity)}
{:db/id "org"
 :organization/name (:name args)
 :organization/id id}
[:db/cas "account" :account/organization nil "org"]

marshall20:02:57

something like that

tyler21:02:08

It has the same behavior with the map form it looks like.

tyler21:02:11

Fwiw it seems to only be a problem when there is a tempid in the entity position of :db/cas

tyler21:02:00

I also tried hacking it with a boolean attribute to rule out weirdness with two entities and it still doesn’t work. This is on datomic cloud btw.

marshall21:02:03

interesting. i’m guessing there is an issue with resolving the tempid passed to a db fn

favila22:02:49

In my experience db fns do not get their tempids resolved (at least for strings this seems impossible, since you don't know what type the db fn expects: a real string or a tempid), and it is impossible to resolve a tempid within a transaction. So in practice you can't use tempids with tx fns unless you can expand to something where it doesn't matter

❤️ 5
favila22:02:33

A larger question is when is tempid resolution performed? It seems impossible in the most general case to know that a tempid resolves to an eid because of an indexed/upserting attribute+value. You can't see those assertions until all expansion is done, and you can't expand everything without running the tx fns.

marshall23:02:13

Im not sure if that's well documented. Ill look into improving that

marshall23:02:08

But it may be worth making it more obvious somewhere

tyler23:02:05

Thanks this explains what I was seeing.

favila03:02:56

Tx fns need to be written defensively. If you need a real eid (eg to read from the vs just expanding to something template-like) use d/entid on input to coerce it, check for nils and strings and throw if you can’t make sense of it

favila03:02:29

IMO if cas can’t work with tempids it should throw when it encounters them

favila03:02:58

(Also it would be nice if the replacement value could be nil to mean “retract”)

Dustin Getz12:02:43

Cas should be able to look at the valueType of the argument to know of a string is a ref or scalar

Dustin Getz13:02:10

“In my experience db fns do not get their tempids resolved (at least for strings this seems impossible, since you don’t know what type the db fn expects: a real string or a tempid), and it is impossible to resolve a tempid within a transaction. So in practice you can’t use tempids with tx fns unless you can expand to something where it doesn’t matter” This doesn’t entirely follow for me, can you say more

favila15:02:48

:db.fn/cas is fundamentally a function call

favila15:02:59

[:db.fn/cas "some-string" :a "b" "c"] --- is "some-string" a tempid placeholder or a literal string

favila15:02:21

without fancy type info how can datomic know? so datomic just invokes the fn as-is, with a string

favila15:02:15

(with tempid records--the old tempid method--at least it had a type so conceivably datomic could do deep walks of all tx fn input and replace tempids with eids)

favila15:02:29

second problem: tempids that resolve to existing eids need to do so by datomic detecting a [:db/add "tempid" :unique-attr "unique-value"] pattern in the tx data

favila15:02:02

however, you can only look for those patterns after all tx expansion is done (including those by tx fns)

favila15:02:05

so how can you substitute a tempid for a real id in a tx fn call if you can't know what substitutions to make until all tx fns are evaluated?

favila15:02:15

there might be hacky half-measure ways through this (double expansion for eg, or doing substitution eagerly during expansion instead of after, or implicit restrictions on tx fn behavior) but datomic doesn't do that

favila15:02:13

so bottom line, tx fns get as arguments exactly what was present as parameters (no transformation) and it's up to the tx fn to interpret it and throw if it can't do it's work

favila15:02:08

d/entid can help with coercing lookup ref and ident sugar to real eids and you should use it in tx fn bodies where you want to allow such sugar (and check for nil)

favila15:02:44

and if your tx fn needs a real eid e.g. because it wants to do a db read and check some invariant, it can't accept a tempid

favila15:02:08

it should throw

Dustin Getz19:02:27

You lost me in the first part, i think the “fancy type info” is perfectly well defined, e.g. this cas is on a :db.valueType/long which is known because :account/balance has schema [[:db.fn/cas 42 :account/balance 100 110]]

favila19:02:37

:db.fn/cas is a function invocation

favila19:02:42

it is not special or privledged

Dustin Getz19:02:01

the function knows the type of its arguments

favila19:02:22

in this case, the caller would need to know the type of its arguments

favila19:02:31

(the caller is the TX runner/applier itself)

Dustin Getz19:02:54

i dont see why

favila19:02:38

how do you know that "bar1" in [:myfoo "bar1"] is a tempid or just the string "bar1"?

Dustin Getz19:02:53

why does the caller need to know that, i would implement cas to pass the entity name to d/entity or some such

favila19:02:15

that moves to the second problem

favila19:02:30

tempid resolution can't be done until all the ops are expanded

favila19:02:04

[[:create-fancy-thing "foo"][:db.fn/cas "foo" :a nil 1]]

favila19:02:00

suppose create-fancy-thing expands to [:db/add "foo" :thing-type :fancy][:db/add "foo" :upserting-thing-id 123]

favila19:02:32

suppose :upserting-thing-id exists already, so "foo" should resolve to an existing eid

favila19:02:53

how can :db.fn/cas know that?

Dustin Getz19:02:00

i dont see why it matters, :db.fn/cas knows “foo” can be treated as a ref, and in proper Clojure form allows us to shoot foot off if malformed

favila19:02:28

:db.fn/cas can never assert the precondition

favila19:02:57

not that it can assert it true or false, it can't assert it at all

favila19:02:00

what eid is "foo"?

favila19:02:11

unknown and unknowable

Dustin Getz19:02:30

so the effect has to be delayed, which means txfns need an expansion phase and an effect phase

favila19:02:41

not just that

favila19:02:53

suppose you expand, find the tempids

favila19:02:09

what if the tx fn is not pure?

favila19:02:32

you'll have to expand again, this time with the tempid mapping available (e.g. smuggled in through d/entid)

favila19:02:55

but if txfns are not pure, the expansion will be different

Dustin Getz20:02:17

by not pure you mean (launch-missles!) or something different

favila20:02:26

d/squuid is sufficient

favila20:02:39

just that calling with same arguments+db does not produce exactly the same value every single time

favila20:02:48

yeah that's stronger than "pure"

favila20:02:51

idempotent?

favila20:02:22

there are perfectly legit reasons to have that, eg issuing new ids

favila20:02:45

now to ensure retriability you need blessed randomness sources that are reseedable

Dustin Getz20:02:57

This is all onprem anyway, Cloud doesn’t even have this, right?

favila20:02:58

there's no bottom to this problem

favila20:02:51

another approach: with a fancy enough type system you could discover all the places in the un-expanded tx that had tempids, then you could make a flow graph

favila20:02:02

if you expand them in the right order maybe it will all work out

favila20:02:11

ah, but what if a tx fn itself makes a new tempid

favila20:02:00

cloud doesn't have cas?

favila20:02:09

it certainly has tx fns doesn't it?

Dustin Getz20:02:14

i think cloud has a closed set of txfns? NOt sure

favila20:02:15

you can write cas yourself

favila20:02:37

in fact cas has two annoying problems which caused me to write one myself anyway

favila20:02:41

one is not throwing on tempids

favila20:02:51

the other is not doing retractions

favila20:02:13

[:db.fn/cas e :attr "x" nil] should work IMO

favila20:02:51

nope, cloud can do anything

Dustin Getz20:02:20

“classpath transaction functions” you are right

Dustin Getz20:02:28

“Transaction functions must be pure functions, i.e. free of side effects.”

favila20:02:47

true for onprem too

favila20:02:08

I think you can't guarantee how many times they will run

Dustin Getz20:02:08

so squuid is disallowed (and not a thing anymore anyway)

favila20:02:29

does "pure" mean "same input same output"?

favila20:02:37

or just "no side effects"

Dustin Getz20:02:56

i think they mean the former

favila20:02:25

perhaps, but I doubt they hold to it

favila20:02:42

if they were super strict about it they could do double-expansion like I talked about

favila20:02:51

but I doubt they would ever consider that

Dustin Getz20:02:19

Thanks for the deep dive

favila20:02:45

np, it's a really subtle gotcha

favila20:02:42

nm double expansion won't work

favila20:02:15

the tx fn can't produce the same value for the same input (or at least can't without severely limiting its usefulness)

favila20:02:31

the first time it's called (d/entid "foo") will be nil

favila20:02:36

the second time it won't be

favila20:02:46

how can a fn produce the same output both cases?

favila20:02:53

even cas can't do that

favila20:02:04

so double expansion is also a no-go

kenny21:02:44

Datomic free uses io.netty/netty-all 4.0.39.Final. We are using other dependencies that require Netty 4.1. If I try excluding io.netty/netty-all from Datomic, I get an exception:

Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
io.netty.handler.codec.http.HttpRequest
Is there a way to make Datomic work with a newer version of Netty?

kenny21:02:47

Seems like this has occurred before: https://clojurians-log.clojureverse.org/datomic/2016-07-04. Any solutions?

kenny21:02:26

I updated to org.apache.activemq/artemis-core-client {:mvn/version "1.5.0"} and it appears to work.

spieden22:02:04

do ions have any story for application initialization? e.g. ion-starter just does a lazy initialization of the db in get-connection — presumably there’s no lifecycle to hook into for application initialization like this?

favila22:02:49

In my experience db fns do not get their tempids resolved (at least for strings this seems impossible, since you don't know what type the db fn expects: a real string or a tempid), and it is impossible to resolve a tempid within a transaction. So in practice you can't use tempids with tx fns unless you can expand to something where it doesn't matter

❤️ 5
devn23:02:29

https://github.com/Datomic/ion-event-example -- would anyone in here mind cloning and running clojure deps to verify it's not just me this is broken for?

lilactown23:02:56

@devn clojure deps what are you expecting to happen when you run that command, and what are you getting?

devn23:02:21

org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for com.datomic:ion:jar:0.9.26
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.datomic:ion:pom:0.9.26 from/to datomic-cloud (): Unable to load AWS credentials from any provider in the chain
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain

marshall23:02:04

@devn you need aws creds in your env

devn23:02:27

i believe i have creds, i will double check

marshall23:02:50

Doesnt matter what they are, as that repo is publicly readable, but they need to be available

devn23:02:13

still no dice-- :thinking_face:

marshall23:02:19

Either envars or profile set up with aws-configure

devn23:02:26

i have creds in ~/.aws/config -- two profiles, with no default. i will try to s3 ls that bucket.

devn23:02:37

err creds in ~/.aws/credentials

marshall23:02:00

If no default then you need to set your profile name

marshall23:02:16

Via AWS_PROFILE envar

devn23:02:39

ah, yeah, that's the ticket

devn23:02:11

actually, turned out that it wasn't a default missing, but instead that i hadn't updated my session credentials. my bad! thanks again.

Dustin Getz13:02:10

“In my experience db fns do not get their tempids resolved (at least for strings this seems impossible, since you don’t know what type the db fn expects: a real string or a tempid), and it is impossible to resolve a tempid within a transaction. So in practice you can’t use tempids with tx fns unless you can expand to something where it doesn’t matter” This doesn’t entirely follow for me, can you say more