Fork me on GitHub
#datomic
<
2018-12-17
>
val_waeselynck08:12:50

Are reversed attributes (e.g :order/_customer) not supported in Datomic Cloud transactions?

val_waeselynck08:12:07

More precisely, it seems they're not supported in a to-many form, e.g:

[{:customer/id "cust-id-1"
  :order/_customer 
  [{:order/id "order-id-1"}
   {:order/id "order-id-2"}]}]

souenzzo17:12:09

is it supported in datomic peer? I think that I also had this "problem" in datomic peer.

val_waeselynck18:12:39

I haven't checked recently, but I believe it is. It could be that I'm getting this impression from DataScript.

Oleh K.14:12:40

Please help when I try this on some data (not all data produces error) I get StackOverflow error despite of the fact that there are only 11 entities in total in cardinality many field:

(d/q '[:find [(pull ?e [:db/id
                                          :user/name
                                          :user/description
                                          :user/account
                                          :user/creator
                                          :user/images  ;;cardinality many entity
                                          :user/subscriptions
                                          ]) ...]
                         :in $ ?creator
                         :where [?e :user/creator ?creator]]
                       (d/db conn) (Long. (:creator args)))

val_waeselynck15:12:56

@U5JRN2PU0 could it be that there's a cycle in your data graph?

val_waeselynck15:12:42

ah wait that doesn't make sense, there's no recursion here

Oleh K.15:12:38

:user/images entities also have :image/account and :image/creator and these are the same entities as :user/creator and :user/account

Oleh K.15:12:13

accordingly

val_waeselynck15:12:48

it's weird that your ref-typed attributes are not in maps, I didn't know that was allowed

Oleh K.14:12:20

if I do like this all is ok:

(let [in (d/entity (d/db conn) (:db/id x))]
            {:id            (:db/id in)
             :name          (:user/name in)
             :account       (:user/account in)
             :creator       (:user/creator in)
             :images        (:user/images in)
             :description   (:user/description in)
             :subscriptions (:user/subscriptions in)})

Oleh K.14:12:16

The error occurs only if the field :user/images is present in the pull vector, (cardinality many field)

Oleh K.14:12:57

And there are only 7 main entities in my database and only 11 :user/images

eprozium14:12:23

the wikipedia page of Datomic seems quite empty. Any chance having an update?

Joe Lane17:12:39

For only $1 a day! 🚀

eraserhd17:12:59

right?? That's actually super cheap. Last time I looked (quite a while back), you couldn't get an EC2 instance that runs Java for that.

ro617:12:18

has anyone come up with a clever way to test tx fns locally without having to rewrite the syntax of the :tx-data back to a local fn invoke (and explicitly pass the db again)? I'm using Cloud and Ions.

ro618:12:43

When I throw an error from inside a transaction fn with eg ex-info, is there a way for me to get the ex-data back as data from the Datomic Cloud Client? All I'm getting right now is:

{....
 "cognitect.anomalies/category": "cognitect.anomalies/fault",
    "cognitect.anomalies/message": "My exception message",
 ....}
none of the data I associated with the exception from the transactor.

ro618:12:46

Do I need to serialize all the error data as a string and pass through the message?

Joe Lane19:12:02

@robert.mather.rmm Looks like you’ve got 2 questions. The first is around testing ion tx functions and the second is around exceptions. Have you attempted returning your own anomolies map? I have no idea if returning your own anomolies map is best practice/a good idea/even possible. Can you provide a sketch of what you’re looking for with problem 1?

ro616:12:54

1) When developing locally, I write something like:

(d/transact conn {:tx-data (my.tx/fn db {:some "args"})})
and when I want to switch to running it remotely on the Cloud transactor, I change that syntax to:
(d/transact conn {:tx-data [['my.tx/fn #_db {:some "args"}]]})
That's not a big deal, but it seems like something pretty natural to abstract over. Just wondering if anyone else had approached that.

Joe Lane16:12:56

Looking on https://docs.datomic.com/cloud/transactions/transaction-functions.html it looks very different from what you showed above.

Joe Lane16:12:23

Do you need to transact the data when testing? Are these integration tests?

mgrbyte16:12:56

to answer the original question: 1) is executing locally, not within the transaction, where as 2) is within the transaction. Think you should use 2) when developing locally as well (shouldn't be any difference between local and remote). my.fx/fn just needs to be on the transactor classpath.

ro616:12:50

You mean the last section on that page? It looks different because all their args are literals and sometimes I have bound values or fns where the semantic I want is "evaluate this stuff to values locally, then send to the transactor" When you use the '(my.tx/fn locally-bound-name) syntax, locally-bound-name is sent as a literally rather than getting evaled right?

ro616:12:38

@mgrbyte I develop locally using Datomic Cloud Client against an in-memory db using https://github.com/ComputeSoftware/datomic-client-memdb, so when developing locally I get :db.error/not-a-data-function Unable to resolve data function when using the syntax in (2).

Joe Lane16:12:39

untested, but does

`(my.tx/fn ~locallybound-name)
work?

ro616:12:47

It does, just tested. I think that and the way I do it evaluate similarly before being sent. I do think your syntax is a bit more intuitive/explicit about what's going on, but semantically they both work.

Joe Lane16:12:32

Cool, well I hope this was helpful Robert

ro616:12:40

I appreciate the thoughts for sure. The thread of this whole thing for me is about seamlessly switching between local and transactor while developing at the REPL, and for that, my second question is actually the one that bugs me more. Throwing data-rich errors, pulling contextual info from the site where they occur (as data) is common practice for me. If all I'm able to get from the transactor when throwing an error is a single message string, I'm going to end up 1) Using EDN or Transit to put everything I need in that string 2) Just returning an error code and the db tx info, so the caller can construct a good error to propagate by running it's own queries. Either approach requires me to treat error handling differently when running tx fns locally vs on the transactor, which seems to me like pointless mental overhead. I was surprised with all the focus on data, data, data that when I threw an ex-info with data from the transactor, it dropped my data and returned a string.

ro617:12:19

I realize that not pretending a remote call is a local call is also a thing in Clojureville, but I'm not sure if the specific reasons for that apply in this case.

Joe Lane17:12:20

So, to be clear, you’re intentionally throwing the exception in your tx function, right?

ro617:12:16

It's an explicit statement like

(if some-condition
  (throw (ex-info "Can't complete tx because...."
                            {:type ::an-error-code
                              :context {:some "relevant"
                                              :info "here"}})))

ro617:12:47

ha, layout failed there. Guess I should use the code thing in Slack

ro617:12:29

oh, it doesn't seem to be available in a thread like this

ro617:12:40

@U0CJ19XAM Does your Ions usage not involve stuff like this? Maybe you end up using CAS more, or just don't have as many operations that need atomicity?

Joe Lane17:12:19

I’ve been questioning how much I should or shouldn’t be using tx-fn’s for atomicity. I don’t have many hard requirements for atomic tx-fn’s.

ro617:12:37

That makes sense. My app has many critical operations that need transactions for correctness. Are you questioning due to write throughput concerns or just generally?

Joe Lane17:12:58

Haha, no I am more questioning if I should use them more 😉

Joe Lane17:12:12

A test project i’m going to spin up over the holidays will be designed where the primary group only has an application of exclusively tx-fns, then all reads will come off a query group. I want to see how building an application like that feels.

Joe Lane17:12:32

(last example of the “Planning your system” page on the operations seconds of the docs)

ro617:12:25

Ah! Minus the stuff we've been talking about here, I'd say an enthusiastic "yes" to tx fns. It's a killer capability relative to any other system I'm aware of.

ro617:12:48

"My database is an atom, and I can call swap! on it" is such a clean mental model.

Joe Lane17:12:28

Are you using namespaced keywords in your exception?

ro617:12:04

not exclusively, and not for the keys

Joe Lane17:12:54

From https://github.com/cognitect-labs/anomalies “Extensible: As maps are open, applications can add their own context via namespaced keywords. That said, try to do as much as possible by dispatching only via ::anom/category.” (emphasis mine)

ro617:12:09

interesting, good thought. Maybe I need to cozy up to cognitect.anomolies to get my data through. I'll play with that

Joe Lane17:12:51

Are you using the Synchronous api? and Do you have a centralized call to transact or are they dispersed through your application? If they are dispersed then the calling code KNOWS the context of the possible errors right? The generic anomolies errors could be enough and then the calling code could enrich / interpret the error and provide meaning.

ro617:12:52

Synchronous and dispersed. What you're saying is what I meant by option (2) above. I think the default error includes db ref info so I could grab whatever I need from the same snapshot the tx fn saw and combine that with local info to deduce whatever I need, but the throw from within the tx fn still feels like the best place to do it from a coding perspective.

ro617:12:29

and again, it's different when developing local vs on-transactor

ro616:12:54

1) When developing locally, I write something like:

(d/transact conn {:tx-data (my.tx/fn db {:some "args"})})
and when I want to switch to running it remotely on the Cloud transactor, I change that syntax to:
(d/transact conn {:tx-data [['my.tx/fn #_db {:some "args"}]]})
That's not a big deal, but it seems like something pretty natural to abstract over. Just wondering if anyone else had approached that.

ro616:12:40

I appreciate the thoughts for sure. The thread of this whole thing for me is about seamlessly switching between local and transactor while developing at the REPL, and for that, my second question is actually the one that bugs me more. Throwing data-rich errors, pulling contextual info from the site where they occur (as data) is common practice for me. If all I'm able to get from the transactor when throwing an error is a single message string, I'm going to end up 1) Using EDN or Transit to put everything I need in that string 2) Just returning an error code and the db tx info, so the caller can construct a good error to propagate by running it's own queries. Either approach requires me to treat error handling differently when running tx fns locally vs on the transactor, which seems to me like pointless mental overhead. I was surprised with all the focus on data, data, data that when I threw an ex-info with data from the transactor, it dropped my data and returned a string.