Fork me on GitHub
#datomic
<
2022-09-01
>
Brett Rowberry03:09:28

I’m on an application team using PostgreSQL. We’re considering Datomic Cloud. There is an analytics team that wants us to stream all our data to them via https://aws.amazon.com/kinesis/data-firehose/. I’m imagining there’s a way to hook up the https://blog.datomic.com/2013/10/the-transaction-report-queue.html to it somehow. Maybe I could write an Ion subscribing with the Datomic client that knows how to publish to Firehose?

Daniel Jomphe11:09:32

Datomic Cloud's Client API doesn't support the Peer API's txReportQueue. You might want to install a transaction function that pushes each transaction to a queue as a side-effect. C.f. https://forum.datomic.com/t/tx-report-queue/316/4

thanks2 1
Brett Rowberry13:09:18

Let's pretend I know nothing about Datomic, except for marketing and some YouTube videos (which is true 😆). What does it mean to install a transaction function ?

Daniel Jomphe13:09:20

Yes, so it's app-level code that you deploy for the transactor instead of for the user-level-ion-app. Ion can deploy code to: • Query groups • Primary group (incl. transactor)

Brett Rowberry13:09:40

Makes sense. Thanks!

Daniel Jomphe13:09:55

You're welcome. Honestly, when discussing these things, we're in the darker corners of Datomic experience. It's much easier to get answers for Datomic On-Prem than for Cloud. OTOH, given its constraints, it seems it might be harder to shoot ourselves in the foot with Cloud than it is with On-Prem's more flexible design.

👍 1
jumar11:09:07

I'm struggling to make a query to list "recipes" in my datomic db which are either public or private. I tried to use or as in

'[:find (pull ?e pattern)
              :in $ pattern account-id
              ;; Or clause: 
              :where (or [?e :recipe/public? true]
                         (and [?owner :account/account-id account-id]
                              [?e :recipe/owner ?owner]
                              [?e :recipe/public? false]))]
.. but that's failing with:
Caused by: java.lang.AssertionError: Assert failed: All clauses in 'or' must use same set of vars, had [#{?e} #{?e} #{?owner ?e}]
(apply = uvs)
Then I tried or-join:
'[:find (pull ?e pattern)
              :in $ pattern account-id
              ;; Or clause: 
              :where (or-join [?e]
                              [?e :recipe/public? true]
                              (and [?owner :account/account-id account-id]
                                   [?e :recipe/owner ?owner]
                                   [?e :recipe/public? false]))]
but that doesn't return what I want - it seems to only return recipes that are public. Before, I had those as two separate queries and it worked well:
;; public recipes
[:find (pull ?e pattern)
                              :in $ pattern
                              :where [?e :recipe/public? true]]

;; private recipes
'[:find (pull ?e pattern)
                                        :in $ ?account-id pattern
                                        :where
                                        [?owner :account/account-id ?account-id]
                                        [?e :recipe/owner ?owner]
                                        [?e :recipe/public? false]]

jumar12:09:15

Ok, silly mistake - it looks like my version is actually working and it's just a higher-level logic that transforms this into something that makes a test fail.

thumbnail12:09:52

Glad you figured it out, i take it that the or-join one is correct 😁?

jumar12:09:01

Well, it actually is a bit weird. pull requires me to use an argument name without question mark like pattern. But or-join requires me to use one with question mark it seems like ?account-id. So the correct version of my query is:

`[:find (pull ?e [*])
              :in $ pattern account-id
              ;; Or clauses: 
              ;; `or-join` is needed because the second clause (`and`) uses a different set of variables
              :where (or-join [?e ?account-id]
                              [?e :recipe/public? true]
                              (and [?e :recipe/public? false]
                                   [?owner :account/account-id ?account-id]
                                   [?e :recipe/owner ?owner]))]
I also had to specify both ?e and ?account-id in the list that comes as the first arg of or-join.

jumar12:09:37

For plain account-id I get this error:

Caused by: java.lang.RuntimeException: Unable to resolve symbol: account-id in this context

jumar12:09:16

Ah, no, it simply doesn't work. I forgot to replace one occurence. With ?account-id I get this error:

... 81 more
Caused by: java.lang.Exception: Unable to find data source: $__in__3 in: ($ pattern $__in__3)
using this query:
[:find (pull ?e pattern)
              :in $ pattern ?account-id
              :where (or-join [?e ?account-id] ; it's the same error whether I specify ?account-id here or not
                              [?e :recipe/public? true]
                              (and [?e :recipe/public? false]
                                   [?owner :account/account-id ?account-id]
                                   [?e :recipe/owner ?owner]))]

jumar12:09:27

Ok, I found the real problem - sometimes, this ?account-id input arg can be nil. It's this condition that makes it fail with such a weird error. So I can include this clause only when ?account-id is not nil:

[?owner :account/account-id ?account-id]

pppaul12:09:33

i'm not really sure why you have to make a complicated query for this

jumar12:09:52

What's the simpler version?

pppaul12:09:53

why can't you just focus on the public/private query alone?

pppaul12:09:12

no or-join, no and

jumar12:09:39

I need a single query for reasons. So I'm looking for the way to get both results from a single query, instead of having two queries (which actually are longer than a single one)

jumar12:09:48

For now, it's seems it's the ?account-id parameter that makes it fail when it's nil. If there's a simpler way, I'm happy to learn it 🙂.

pppaul12:09:14

i feel like doing or-join is probably making things way too complicated

jumar12:09:47

"or" condition doesn't sound to me like something unusual. But I'm Datomic noob and mostly familiar with SQL databases.

pppaul12:09:30

or is done via rules

pppaul12:09:39

but there is also the or statement

pppaul12:09:16

or-join is very special case, i'm not sure you really need it here, but i've never run into a situation where i have used it

jumar12:09:12

Again, I tried or and it doesn't work:

Caused by: java.lang.AssertionError: Assert failed: All clauses in 'or' must use same set of vars, had [#{?e} #{?e} #{?owner ?e}]
(apply = uvs)
It's because I need ?owner in the second set of rules.

pppaul12:09:53

(and [?e :recipe/public? false]
[?owner :account/account-id ?account-id]
[?e :recipe/owner ?owner]))
i think you may make a rule for this, and i'm not sure the ?owner is doing anything

pppaul12:09:55

but i guess you want all recipes, and all owner recipes

jumar12:09:59

It should return only recipes whose owner is the account-id specified as the input argument

pppaul12:09:12

if the owner has a relation to their recipes, you can get that without a query, just a pull

jumar12:09:33

yes, the result should be: all public recipes + all recipes owned by given account.

pppaul12:09:45

i feel like you are better off with a pull

jumar12:09:10

I'm using pull, so I'm not sure I understand. I would appreciate a full query if you can provide that.

pppaul12:09:41

you can pull the related recipes from the owner

pppaul12:09:06

so, query all public, (1 line query), and pull the owner recipes

pppaul12:09:49

(d/pull db '[{:recipe/_owner [*]}] owner-id)

jumar12:09:46

But I want a single query. I'm trying to pass my query to an interceptor which executes it and that doesn't know anything about the query. Interceptor uses datomic.api/query function to get the results and pass it further.

pppaul12:09:55

i do almost everything in datomic via pulls, very rarely do i make queries, and they are usually doing stuff that is crazy (history stuff)

pppaul12:09:12

you can have a pull as part of a query

jumar12:09:46

Again, I use it already and this is what I have now:

(da/query
      {:query '[:find (pull ?e pattern)
                :in $ pattern ?account-id
                :where (or-join [?e]
                                [?e :recipe/public? true]
                                (and [?e :recipe/public? false]
                                     [?owner :account/account-id ?account-id]
                                     [?e :recipe/owner ?owner]))]
       :args [db recipe-pattern account-id]})

pppaul12:09:48

you gotta use pulls when dealing with direct relations like this, it's just much simpler

jumar12:09:27

I have complicated recipe-pattern already: https://github.com/jacekschae/learn-pedestal-course-files/blob/main/increments/43-list-conversations/src/main/cheffy/recipes.clj#L9-L28 Not sure if that can easily be combined with what I need.

pppaul12:09:35

you don't need all this complicated stuff in the where, just query all public recipies, and pull related ones on the owner

pppaul12:09:09

that's not a complicated pull

pppaul12:09:48

you can add recipe/_owner in that pattern

pppaul12:09:01

i would recommend aliasing names too

pppaul12:09:32

'(:recipe/_owner :as :owner) something like that

pppaul13:09:10

the pull has to operate on the account-id, though

pppaul13:09:36

i don't think you can do 2 pulls in a query

jumar13:09:52

Sorry, I simply don't understand how to do this filtering with pull syntax. I need to get the union of two sets: • All the public recipes - returning all the information as specified by my recipe-pattern • All the private recipes owned by given account-id (if any)

pppaul13:09:07

step 2 = (d/pull db '[{:recipe/_owner [*]}] owner-id)

pppaul13:09:50

step 1 = [?e :recipe/public? true] where clause (only 1 clause)

jumar13:09:53

That's a separate action that I do not want to perform because the actor executing the query doesn't know anything about it. In my case, the client would pass this:

{:query '[:find (pull ?e pattern)
                :in $ pattern ?account-id
                :where (or-join [?e]
                                [?e :recipe/public? true]
                                (and [?e :recipe/public? false]
                                     [?owner :account/account-id ?account-id]
                                     [?e :recipe/owner ?owner]))]
       :args [db recipe-pattern account-id]})
and the executor would take it and pass to datomic.api/query - it has no clue about some additional pull that needs to be executed.

pppaul13:09:24

can you add a callback?

jumar13:09:18

Maybe. But I feel like I'm making it more complicated just because of the desire to use pull 🙂

pppaul13:09:24

can you do this in 2 steps, where one is the query and another is just a pull (which can be done in a query as well)

jumar13:09:01

Maybe I should think about allowing a sequence of queries, regardless whether this is the best approach here or not. Thanks for the ideas. I'll see what I can do.

pppaul13:09:12

pull is really good, though. you can get away with almost never using queries if you use pull

pppaul13:09:13

the client can ask for all public recipes, and all owner recipes, you get 2 very tiny queries, 1 line each.

pppaul13:09:48

i have a feeling it'll be faster too, cus who knows what datomic is trying to do with that or-join stuff

pppaul13:09:31

also, you probably want to do something special with the all recipes query (filter, sort, limit) and adding that stuff to an already complicated query is probably going to cause issues

jumar15:09:06

Does Datomic have any support for java.time package? It seems that I have to use an instance of java.util.Date as a value of instant attribute. I didn't find much about this topic, just an older discussion: https://forum.datomic.com/t/java-time/1406

magnars16:09:36

Unfortunately Datomic doesn't let you extend the types it supports, but I have been using "Datomic Type Extensions" via https://github.com/magnars/datomic-type-extensions with this java.time-package https://github.com/magnars/java-time-dte for several years in different projects now. It is slightly leaky, but certainly useable.

👍 1
pppaul15:09:33

could a db/function handle something like auto-coercion?