Can anyone point me to an example that explains how to obtain all of the results from a Datomic client api query that returns a significant number of results?

I'm running a query and can get the first chunk of results but I'm not sure how to get subsequent chunks until I have the whole result set.

celldee: looks like you just pass the :offset param

matthavener: do you have an example that you could show?

celldee: sorry, I don’t. I haven’t used the client API but I was curious so I read through the docs.

matthavener: no worries. This is my first attempt to use it for querying. The client/q functions seems to return a channel that I can take from but it chunks results. I read the :offset parameter to mean that you want to skip a number of results.

My query looks like this - (def first-query {:query '[:find ?e ?id :where [?e :active-chance/id] [?e :active-chance/lastName] [?e :active-chance/id ?id]] :args [db] :limit -1 :chunk 10000})

If I leave out the :limit argument then I only get 1000 results returned and with the :chunk argument I get 10000 results back. I'm expecting over 90000 items in the result set, which is what I get when I run the query in the Datomic console.

Also, if I leave out the :chunk argument then I get 1000 results

I'm imagining a mechanism where I keep taking from the channel until the result set is exhausted. Just not sure how to do that. I suppose if I was more conversant with core.async this would be obvious, however, I'm a bit of a Clojure novice.

While running an importer I’ll intermittently get :db.error/transactor-unavailable. We are pipelining our transactions, as in http://docs.datomic.com/best-practices.html#pipeline-transactions, with some logic to attempt a few retries with a 2 second timeout, but that doesn’t appear to be sufficient. What’s the right way to handle back pressure, or is this another issue?

also, any advice on what to do if this occurs during an import? Critical failure, cannot continue: Heartbeat failed

(this is with a dev: db)

when the reference material says “be willing to wait for several minutes for a transaction to complete”, should I just use an exponential backoff retry when I get transactor unavailable?

@celldee you can just repeatedly 'take' from the channel with the <!! Operator

Once it's empty it will return nil

@uwo heartbeat failure means the transactor is unable to write to storage. It will self destruct as a result.

@marshall Thanks. What would that look like please? I'm still trying to learn Clojure.

http://stackoverflow.com/questions/33497426/idiomatic-clojure-taking-multiple-items-off-a-channel

That might help ^

@marshall Thanks very much!

Also helpful : https://cognitect.com/events/training-sessions/core-async-clojurewest-2015

Luke does a good job of covering the concepts of core async in those videos

Quick question — is it possible to query Datomic backed by DynamoDB without a transactor running? It doesn't look like it.

(datomic.api/connect "datomic:ddb://...")

org.apache.activemq.artemis.api.core.ActiveMQNotConnectedException: AMQ119007: Cannot connect to server(s). Tried with all available servers.
    type: #object[org.apache.activemq.artemis.api.core.ActiveMQExceptionType$3 0x2512bf7b "NOT_CONNECTED"]
                                        clojure.lang.ExceptionInfo: Error communicating with HOST localhost on PORT 4334

I know transactions aren't possible without one of course but I figured that queries might be, given that in the case of a transactor failure, queries are still possible

@alexandergunnarson no, a transactor is required for queries. The the fact that it continues to work (for some amount of time) after a transactor fails is an implementation detail, not something that should be depended upon.

Ah okay, makes sense. Thanks!

Datomic’s architecture could in theory support read-only models but it doesn’t right now

Does that mean that in the time between the primary transactor failing and the standby transactor coming into play, you can’t run queries, even against the peers? So you (temporarily) lose reads as well as writes?

@seancorfield I’ve never seen a read fail in practice. I’d have to check if that’s a guarantee or just the way it always works (eventually, a peer will “notice” that it’s transactor is dead and stop working, but that is longer than the failover window)

Good question for @jaret, he probably knows the answer.

actually @seancorfield I should have read the docs before answering: Datomic does indeed guarantee that reads are available during a failover: http://docs.datomic.com/ha.html

OK, glad to hear that. I would have been (unpleasantly) surprised if that wasn’t the case 🙂