Fork me on GitHub
#datomic
<
2018-05-04
>
davidw07:05:00

I'm trying to upsert an entity and use its temp id with :db.fn/cas, like so

[{:db/id "my-foo-id"
  :foo/id "my-foo-id"
  ... other foo attributes ...}
 [:db.fn/cas "my-foo-id" :foo/is-processed? nil true]]
 
but I get an exception
:db.error/not-a-keyword Cannot interpret as a keyword: my-foo-id, no leading :
is there a way to do this?

val_waeselynck07:05:52

@davidw I doubt it, because transactions functions happen prior to tempid resolution (for good reasons) - for such cases, you may want to write your own transaction function that is aware of identity attributes

davidw07:05:58

the intention is to use this with datomic cloud so a custom transaction function isn't an option

davidw07:05:39

do you have a reference for the transaction function, tempid resolution order? I'm interested in the good reasons.

val_waeselynck07:05:18

I don't have any doc ereference about that, but we can reason about it

val_waeselynck07:05:05

a transaction function is local - it cannot see the whole transaction it participates in. But tempid resolution has to consider the entire transaction

val_waeselynck07:05:06

What's more, if tempid resolution happened before transaction functions, what would happen with tempids emitted by transactions functions ?

val_waeselynck07:05:04

And yeah, about Cloud, that's exactly the sort of limitation causing me to not be too enthusiastic about it yet, so I don't know what to tell you really 😕

davidw07:05:00

that makes sense. meaning I can see why it works that way. it's disappointing because it's seems like a valid think to want to do.

davidw07:05:45

do you know off the top of your head if you can use a look up ref with cas?

[:db.fn/cas [:foo/id "my-foo-id"] :foo/is-processed? nil true]

val_waeselynck07:05:11

I don't sorry 😕 I don't use cas much

davidw07:05:39

no problem, I'll test it.

davidw07:05:52

the only other way I can think to achieve what I want is to insert the empty entity in one transaction and then use the lookup ref with cas in another. It's not as nice as I was hoping for but I think it should work. I'll check it out. Thanks for your help.

val_waeselynck07:05:20

Datomic Cloud is definitely lacking in transactional expressive power as of today IMHO

alexmiller12:05:11

That will change in time

lopalghost14:05:52

So, how do you approach user-defined fields in Datomic? In say Postgresql I would roll user-defined fields into a map and store that as json. Not perfect, but it works well enough. Storing maps doesn't seem practical with Datomic and I'm not sure I want to let users arbitrarily modify the schema. Anyone have a good solution?

eraserhd14:05:24

Storing JSON or EDN in strings works decently. I have arbitrary, user-supplied queries and pull expressions in string fields.

eraserhd14:05:51

However, I'm also curious what kind of application is it in which users want to modify the schema.

lopalghost15:05:12

In Postgresql you can index by json fields, which is nice. Storing serialized objects in Datomic wouldn't offer the same advantage. User-defined data fields are a pretty common requirement for enterprise software. In Datomic it could be as easy as just modifying the schema whenever a user wants to add a field, but that doesn't seem like a good practice.

tony.kay15:05:25

Curious if anyone knows: when you set “no history” on a Datomic attribute, does this allow Datomic to do update-in-place at the storage level? Wondering how much of a performance boost that is likely to give if you don’t need history (and update-in-place would imply “a lot”).

favila17:05:49

no it does not imply that

favila17:05:08

the storage layer is used as a key-value blob store

favila17:05:22

you still will dirty the "blocks" of data with value updates

favila17:05:36

you just won't keep the old values

favila17:05:58

also once a key+value is written it is never mutated (there are only a few exceptions: well-known key names whose values hold the root pointers)

favila17:05:50

allowing them mutate would give up a lot

tony.kay18:05:35

Yeah, that makes sense. I guess I’ll just have to micro-benchmark and get a sense of how much it can help.

favila18:05:37

it's only purpose is to save storage

favila18:05:53

it may cut down on index time, but that's not it's primary purpose

favila18:05:29

also you have no guarantees there will never be any history at all--before the index is written, you will see "old" values in the log

tony.kay18:05:03

https://docs.datomic.com/on-prem/best-practices.html#nohistory-for-high-churn ” cost of storing history is frequently not worth the impact on database size or indexing performance.” Thinking about it now, I’m sure the “indexing performance” comment is just due to there being less data.

tony.kay15:05:55

The docs say that it reduces indexing overhead, which implies update-in-place in my mind

tony.kay16:05:43

@lopalghost “seem like a good practice” feels like something, from my perspective, that is coming from a certain SQL security mindset. I have that leaning as well, but as I’ve thought about it schema in Datomic: 1. Gives a clear name for something, with a namespace. Allowing schema to “flex” to include user-namespaced things seems natural to me. 2. Gives a clear type to it, that because of (1) can also be given a data specification (e.g. clojure.spec). Personally, I think opening Datomic schema up to extension through a user UI is pretty powerful

tony.kay16:05:36

of course, new challenges as well…rules like “only grow schema” need to be followed 🙂

lopalghost17:05:38

@tony.kay I'm starting to come around on that line of thinking. I'm definitely coming from a mindset of sql security that might not be relevant to Datomic. Has anyone else tried opening the schema to modification by users?

tony.kay17:05:01

I’m working with a client (consulting) that is doing so

tony.kay17:05:12

product is young and not yet released, though

rnagpal18:05:45

Trying to find all entities where :alarm/cleared_at is nil

rnagpal18:05:50

[:find ?e
          :in $
          :where
           [?e :alarm/cleared_at ?cleared]
           [(nil? ?cleared)]]

rnagpal18:05:59

but it return empty list

favila18:05:39

@rnagpal Datomic does not store nil and datalog cannot refer to nil

favila18:05:04

There is no assertion of ?e :alarm/cleared_at at all

favila18:05:13

so that clause never matches

favila18:05:49

you maybe want [(missing? $ ?e :alarm/cleared_at)], but that will match every entity in the entire system that lacks a cleared_at

favila18:05:04

(including schema, transaction entities, etc)

favila18:05:33

your data model may need refinement if you need to say "I assert it was not cleared"

favila18:05:20

some possibilities: ?e needs another indexed attribute to indicate it is "alarm-like"

favila18:05:43

(i.e. :alarm/cleared_at could be expected on it)

favila18:05:19

or there is a sentinel value of cleared_at to indicate "not-cleared"

favila18:05:34

or there is another attribute :alarm/not-cleared

favila18:05:54

(or :alarm/cleared)

favila18:05:09

and you have to set or retract both attributes together all the time in your application code

eraserhd19:05:03

1. Is it possible that d/tx-report-queue slows down the processing of queries?

favila19:05:55

Seems very unlikely by itself; although obviously cycles devoted to reading it are consumed that wouldn't be

favila19:05:30

I think the peers get all TXs all the time anyway; tx-report-queue is just adding them to a user-readable queue

eraserhd19:05:40

2. Does Datomic cache query results, such that the same query on the same db doesn't hit indexes and such?

favila19:05:07

No, but a repeat run will typically be very hot: query plan is cached, indexes were loaded, etc

rnagpal19:05:17

Thanks @favila This worked for my case

'[:find ?e
                                 :in $
                                 :where
                                 [(missing? $ ?e :alarm/cleared_at)]
                                 [?e :alarm/alarm_id]]

favila19:05:33

@rnagpal Reverse the order of those clauses

favila19:05:56

@rnagpal first clause visits every entity in the db; second clause filters

favila19:05:08

the second one is more selective, so use it first

rnagpal19:05:19

cool. Got you. Thanks @favila

eraserhd20:05:23

Huh, I just found out that nesting expressions started working. When did that happen? e.g. [(string/starts-with? ?foo (str ?bar "-"))]