Fork me on GitHub
#datomic
<
2017-05-04
>
slpssm00:05:51

Simplest example: :entity/state is a keyword

[:find ?e :where [?e :entity/state :pending]]
works
[:find ?e :where [?e :entity/state ":pending"]]
also works.
[:find ?e :where [?e :entity/state "pending"]]
used to work in the old version but no longer works in the new version.

jeff.terrell01:05:18

I have a question. Say I want to express the equivalent of this (pseudo?) SQL query in Datomic/datalog:

SELECT author.name, count(*) AS num_books
  FROM author
  JOIN book
    ON author.id = book.author_id
  GROUP BY author.id
  HAVING num_books >= 5
I think that in the peer library, you could just fetch all the author/book combinations then group (`GROUP BY` above) and filter (`HAVING` above) in good ol' Clojure without any waste of bandwidth or anything, since you essentially have all the data locally anyway. (If this is not correct, let me know!) If instead you were using the client library, and you took the same approach of fetching all the author/book combinations, I'm guessing you'd transfer data (potentially a lot of data) that you didn't end up using. Is that right? How would you avoid that in the client library?

favila03:05:07

@slpssm third one if it ever worked was a bug. Second one is I think a concession to Java users of the API

favila05:05:03

[:find (pull ?aid [:author/name]) (count ?bid) :where [?aid :author/books ?bid]] @jeff.terrell

isaac09:05:20

There is any way to specify :db/txInstant while import data pipelining? Datomic tx must ensure (:db/txInstant early-tx) <= (:db/txInstant later-tx), howerver, pipelining can’t ensure the ordering of imports.

augustl11:05:42

@jeff.terrell you're correct about the peer library. All query engines need their working set in memory 🙂

augustl11:05:59

I somehow managed to miss the existence of the client library

marshall12:05:41

@jeff.terrell The approach would be essentially the same with the client - the difference is that the work would happen on the peer server instead of in your process

marshall12:05:13

You’d still do the “limit by > 5” in your app

marshall12:05:27

using something like the query @favila posted aboe

marshall12:05:18

Incidentally, since the client will return results in a channel, I’d do the filtering with a transducer

jeff.terrell13:05:35

@marshall - OK. I think that's about what I expected. Thanks for confirming. Now, if there was a long tail, with lots of results < 5, I'd be wasting a lot of bandwidth, right? Is there a simple way to avoid that? Maybe ship the filter up to the server side, somehow?

jeff.terrell13:05:57

If this were SQL and I was trying to limit results somehow in a way that wasn't supported by SQL, I'd reach for a database function.

marshall13:05:19

TBD 🙂 Sort in the peer server + limit/offset - which I believe is a request in our customer feedback portal

marshall13:05:30

you should vote for it 🙂 or if it’s not there, you should add it

favila13:05:09

@isaac pipelining does not necessitate out of order

jeff.terrell13:05:55

@marshall - I think the feature I want to suggest is "custom server-side database functions to limit or transform results server-side when using the client library". Does that sound like a reasonable request to you, or would that be a bad idea for any reason that you can see?

jeff.terrell13:05:37

I went ahead and added this.

kirill.salykin13:05:09

When I am trying to suggest features - it stays on the account page and just adds “#”

kirill.salykin13:05:38

http://api.eu-west-1.receptive.io/widget/ping Failed to load resource: the server responded with a status of 400 (Bad Request) Also I have this error, is this connected?

kirill.salykin13:05:30

but doesnt work for me

marshall13:05:54

@kirill.salykin Are you running any ad blockers or ghostery or anything like that?

kirill.salykin13:05:06

I disabled AdBlock

marshall13:05:14

depending on config they can prevent external redirects

marshall13:05:22

hrm. what browser?

kirill.salykin13:05:29

chrome and safari

kirill.salykin13:05:34

will try Firefox

marshall13:05:39

weird. i wonder if there’s a system firewall issue

marshall13:05:51

sounds like you’re not getting content from receptive

kirill.salykin13:05:12

receptive responses with “{”message”:“invalid user”}”

marshall13:05:16

yeah, i just verified it works ok for me in chrome

marshall13:05:28

let’s go to a private chat

marshall13:05:29

heh. for those of you awaiting updates with baited breath about Kirill’s plight in accessing the feature portal - software is hard, integrating multiple softwares is harder 🙂

isaac14:05:56

@favila It’s just keep ordering of returning, not guarantee order of executing.

karol.adamiec14:05:27

how to modify this query: ‘[:find ?name ?filter :where [?e :part/name ?name] [?e :part/filter ?filter]] so it returns […] for ?filter? :part/filter is a many keywords type, but above query returns only first keyword from collection …

karol.adamiec14:05:31

got it somehow working like that: (d/q ‘[:find ?name (distinct ?filter) :where [?e :part/name ?name] [?e :part/filter ?filter]] (d/db (d/connect database-uri)))

karol.adamiec14:05:41

but i feel i am missing something… ;/

favila14:05:00

@isaac using the impl of pipelining in the docs, yes. But within a process you are guaranteed that order of d/transact call is order of execution

favila14:05:08

So with an implementation of pipeline that calls d/transact in correct order you know they will run in correct order

favila14:05:33

I also wrote one (less elegant) that didn't use core async, used reduction, kept inflights in an accumulating vector, and did "gc" of inflight futures when the vector reached capacity

favila14:05:12

the advantage of it was it didn't run any additional threads

isaac15:05:14

You mean, the d/transact-async is keep ordering of executions same with ordering of invokates?

favila15:05:13

the problem with the pipelining code in the docs is core.async/pipeline does not preserve order of invocation because it uses a threadpool

favila15:05:32

(it's not immediately obvious that it does this)

isaac15:05:49

datomic.api is not open source, and the docs has no instructions about this

isaac15:05:15

Anyway, it’s a good feature, thanks, @favila

favila15:05:33

core.async/pipeline-blocking doesn't run input "tasks" in-order because it runs them in parallel on a threadpool

favila15:05:46

so which runs first is up to chance

favila15:05:54

(this code is open)

favila15:05:21

but d/transact-async etc send transactions to the transactor in the order they are invoked

favila15:05:01

so you could just (doseq [tx txs] @(d/transact-async conn tx))

isaac15:05:03

“but d/transact-async etc send transactions to the transactor in the order they are invoked” it is key point

favila15:05:22

except that the flood of transactions would kill the transactor

favila15:05:37

well, not kill it, but the transactors would be unresponsive

favila15:05:59

it would eventually curn through them all

favila15:05:32

the pipelining code is just to ensure queue depth

favila15:05:36

is reasonable

isaac15:05:02

well, thanks for you hints, let my try your code. 🙂

spieden21:05:00

is there any work on client api libraries for non jvm languages taking place yet?

jdkealy23:05:23

can a transactor support multiple databases? If not, how would you test using multiple databases on a localhost environment?