the datomic transactor’s properties file needs a host= or alt-host= that has a name that other docker containers can resolve to the storage container

(in the dev storage case, the storage and transactor happen to be the same process, but that this is the general principle)

so connecting to “localhost” connects to the peer container localhost, which is not correct

datomic connection works like: 1) transactor writes its hostname into storage 2) d/connect on a peer connects to storage, retrieves transactor hostname 3) peer connects to transactor hostname

you appear to be failing at step 3 in your first error, step 2 in your second error

@favila not sure if I understand correctly… I changed host=localhost to host=datomic-transactor and log now says:

Launching with Java options -server -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=50
Starting datomic:<DB-NAME>, storing data in: data ...
System started datomic:<DB-NAME>, storing data in: data

from where you peer is, can you resolve “datomic-transactor” ?

Since I’m connecting from the docker host, I altered /etc/hosts to map datomic-transactor for 127.0.01 (localhost) … same problem when connecting to `

datomic:

I will try from my docker peer server but thought that i hat to create a database first (before launching the peer)

try nc -zv datomic-transactor 4334 from a terminal running in the same context as your peer

$ nc -zv datomic-transactor 4334
found 0 associations
found 1 connections:
     1:	flags=82<CONNECTED,PREFERRED>
	outif lo0
	src 127.0.0.1 port 52204
	dst 127.0.0.1 port 4334
	rank info not available
	TCP aux info available

Connection to datomic-transactor port 4334 [tcp/*] succeeded!

Just to see if I understand peer-servers corretly… can I start a peer-server without (d/create-database <URI>) first? Because I get:

Execution error at datomic.peer/get-connection$fn (peer.clj:661).
Could not find my-db in catalog

Full report at:
/tmp/clojure-3528411252793798518.edn

no, you need the db first

now try nc -zv datomic-transactor 4335

(4335 is storage)

$ nc -zv datomic-transactor 4335
found 0 associations
found 1 connections:
     1:	flags=82<CONNECTED,PREFERRED>
	outif lo0
	src 127.0.0.1 port 52844
	dst 127.0.0.1 port 4335
	rank info not available
	TCP aux info available

Connection to datomic-transactor port 4335 [tcp/*] succeeded!

if both of these work, your bin/repl-jline should succeed if you run it from the same terminal

(specifically the create-database you were trying before)

Now it does… just wondering why it didn’t before. But I tried with a new repl…

even works with localhost in the uri…

what was in your host= before?

host=localhost…

so that means the transactor bound to the docker container’s localhost, 127.0.0.1; probably not the same as the peer’s?

(i’m fuzzy on docker networking)

Not sure but it does work now. Thank you very much @favila for your quick response and fruitful help!

I’m fairly new to docker and datomic but your explanations do make perfect sense!

I usually see and use host=0.0.0.0 alt-host=something-resolveable so I don’t have to worry about how the host= resolves on both transactor and peer

Okay, will try this as well. Thanks again!

the transactor will use host= for binding, and advertise both for connecting

and the peers will end up using alt-host

use a since-db

https://github.com/cognitect-labs/day-of-datomic-cloud/blob/master/tutorial/filters.repl#L102

you could use a since and an as-of db to get an ‘in-between’

Oooo, ok! I'll try that.

I'm struggling figuring out how I'm supposed to join across these dbs. I'm trying:

'[:find #_?date ?tx ?w ?a ?v ?op
  :keys #_date tx db/id attr v op
  :in $as-of $since ?workspaces-group ?start ?stop
  :where
  [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w]
  [$as-of ?w _ ]
  [$since ?w _]
  [?w ?a ?v ?op ?tx]
  #_[?tx :db/txInstant ?date]
  #_[?a :db/ident ?attr]]

Nil or missing data source. Did you forget to pass a database argument?

you need to pass the “regular” db as well as the others (i believe

so :in $ $asof $since

I think it’s just that this clause doesn’t specify a db

[?w ?a ?v ?op ?tx]

oh, right

yes that’s definitely why

thx @favila

But what is supposed to go there?

which db value do you want that information from

I think both?

then you’d need 2 clauses

one for each db

and you’ll only get datoms that are the same in both

This?

'[:find ?tx ?w ?a ?v ?tx ?op
  :in $as-of $since ?workspaces-group
  :where
  [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w]
  [$since ?w ?a ?v ?tx ?op]
  [$as-of ?w ?a ?v ?tx ?op]]

Doesn't that only return datoms where ?a ?v ?tx ?op in both since and as-of are the same?

I'm pretty sure this is what I want:

'[:find ?tx ?w ?a ?v ?tx ?op
  :in $as-of $since ?workspaces-group
  :where
  [$since ?w ?a ?v ?tx ?op]
  [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w]]

Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
processing clause: (?w ?a ?v ?tx ?op), message: java.lang.ArrayIndexOutOfBoundsException

Not really sure what that exception means. Here's a larger stacktrace:

clojure.lang.ExceptionInfo: processing clause: (?w ?a ?v ?tx ?op), message: java.lang.ArrayIndexOutOfBoundsException {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "processing clause: (?w ?a ?v ?tx ?op), message: java.lang.ArrayIndexOutOfBoundsException", :dbs [{:database-id "f3253b1f-f5d1-4abd-8c8e-91f50033f6d9", :t 105925, :next-t 105926, :history false}]}
	at datomic.client.api.async$ares.invokeStatic(async.clj:58)
	at datomic.client.api.async$ares.invoke(async.clj:54)
	at datomic.client.api.sync$unchunk.invokeStatic(sync.clj:47)
	at datomic.client.api.sync$unchunk.invoke(sync.clj:45)
	at datomic.client.api.sync$eval50206$fn__50227.invoke(sync.clj:101)
	at datomic.client.api.impl$fn__11664$G__11659__11671.invoke(impl.clj:33)
	at datomic.client.api$q.invokeStatic(api.clj:350)
	at datomic.client.api$q.invoke(api.clj:321)
	at datomic.client.api$q.invokeStatic(api.clj:353)
	at datomic.client.api$q.doInvoke(api.clj:321)

Got it. See the duplicate :find here:

'[:find ?tx ?w ?a ?v ?tx ?op
  :in $as-of $since ?workspaces-group
  :where
  [$since ?w ?a ?v ?tx ?op]
  [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w]]

what is the set of ?w you are interested in?

those currently monitored only? or ones that were ever monitored?

I want all ?w added or retracted between 2 dates that were on the :aws-workspaces-group/monitored-workspaces card many ref attr.

the confusing here is there are two different entity histories to consider

This query gives me some results

[:find ?w ?a ?v ?tx ?op
 :in $as-of $since ?workspaces-group
 :where
 [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w]
 [$since ?w ?a ?v ?tx ?op]]

are either of those history dbs?

No. Called like this:

(d/q
  '[:find ?w ?a ?v ?tx ?op
    :in $as-of $since ?workspaces-group
    :where
    [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w]
    [$since ?w ?a ?v ?tx ?op]]
  (d/as-of db stop-date)
  (d/since db start-date)
  [:application-spec/id workspaces-group-id])

so this gives you ?w that were monitored at the moment of stop-date, then looks for datoms on those ?w entities since start-date (if you make that $since a history-db)

in particular, if there’s a ?w that used to be monitored between start and stop, you won’t see it

is that what you want?

No. I want ?w the used to be monitored between start and stop included as well.

you want ones that started to be monitored after start, or those that were monitored at start or any time between start and stop?

Correct

…so both?

Yes

Then I think you need something like this:

(d/q '[:find ?w ?a ?v ?tx ?op
       :in $as-of $since ?workspaces-group
       :where
       [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w]
       [$since ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w _ true]
       [$since ?w ?a ?v ?tx ?op]]
     (d/as-of db start)
     (-> db (d/history) (d/as-of end) (d/since start))
     workspaces-group)

as-of-start gets you whatever workspaces were already being monitored at start moment

then you look for groups again in $since for any that began to be monitored between start and end

?w is now the set-union of both

then you look for any datoms added to ?w between start (not-inclusive) and end (inclusive)

it’s possible you want to include ?start there too, in which case you need to decrement start-t of $since by one

Won't [$since ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w _ true] not work if ?workspaces-group was not transacted within start and stop?

doh you are right, this is unification not union

doing this efficiently might need two queries

you can’t use two different data sources in an or

Why would this not work?

(d/q
  '[:find ?w ?a ?v ?tx ?op
    :in $as-of $since ?workspaces-group
    :where
    [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w]
    [$since ?w ?a ?v ?tx ?op]]
  (d/as-of db stop-date)
  (-> (d/history db) (d/as-of stop-date) (d/since start-date))
  [:application-spec/id workspaces-group-id])

It would miss ?w that were removed from workspaces-group between start and stop

it’s the first choice I offered you earlier

Oh, right. That query also hangs for 10+ seconds. Didn't let it finish.

this is only ?w that were part of the group at the very moment of end-date

using $since instead would miss ?w that were in the group at the moment of start-date

So perhaps query for all ?w at start-date and any added up to end-date. Pass that to a second query that uses (-> (d/history db) (d/as-of stop-date) (d/since start-date)) to get all datoms

yes, so 3 queries

The first one needs to be 2 queries, huh?

maybe you can unify later, let me think

(d/q '[:find ?w ?a ?v ?tx ?op
       :in $as-of $since ?workspaces-group
       :where
       [$as-of ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w-at]
       [$since ?workspaces-group :aws-workspaces-group/monitored-workspaces ?w-since _ true]
       (or-join [?w-at ?w-since ?w]
                [(identity ?w-at) ?w]
                [(identity ?w-since) ?w])
       [$since ?w ?a ?v ?tx ?op]]
     (d/as-of db start)
     (-> db (d/history) (d/as-of end) (d/since start))
     workspaces-group)

Same problem as the other query, I think. ?workspaces-group isn't in $since

yeah, and if I solved that, there would be the same problem with as-of

Right

ugh, maybe a sentinel value, like -1

if you know that set will be small across all time, you could filter by ?tx like you were doing before

Using an unfiltered history db?

yeah

just to get ?w

you still want the $since to get changes to ?w entities themselves

What would you consider small? < 1,000,00?

I would not consider that small…

but really “small” is just “this query is fast enough”

I wonder if we can step back

[:application-spec/id workspaces-group-id]

I think it will often be in the 10-50 thousand range.

is that an immutable identifier?

Yes

i.e., a lookup ref?

so once asserted on an entity, it is never retracted and never asserted on a different entity

Right

> it is never retracted Unless the entity itself is retracted

With 3 queries I'd do: 1. Query for all ?w that are monitored in as-of. 2. Query for all ?w added to monitored in since. 3. Pass the union of ?w in 1 and 2 to a history db and get all the datoms

correct; the db in 3 is either the same as 2 or just with a since adjusted 1 tx backward

(depending on what you want)

Should 2 be querying a (-> db (d/as-of stop-date) (d/since start-date))?

yes. the since sets an exclusive outer range

txs that occur exactly at start-date are excluded

If none are added then that query will throw. Guess I just catch that and return an empty set.

> the db in 3 is either the same as 2 or just with a since adjusted 1 tx backward Oh, right it would be the same. Since now we know all the ?w it's easy to search for the matching datoms.

that will omit changes to ?w that occurred exactly at start-date.

this difference should only happenmatter if you ever change ?w and group membership in the same tx

Haha, right. Adjusting 1 tx back is easy though

Wait the db for 2 needs to include retracts. If a workspace was retracted between start and end, it would not be included in query 3.

I think that just means changing the passed in db to be (-> (d/history db) (d/as-of stop-date) (d/since start-date))

I also don't think the lookup ref for :application-spec/id will be present in that db so I'll need to have the db/id for ?workspace-group

yes, sorry I misread your earlier db constructor. it needs d/history

you can look up the id in the query

In query 2?

both?

I could do it in query 1. Since query 2 is filtered by as-of and since, I don't think the :application-spec/id attribute will be included since it would have been transacted before the since filter.

Unless there is some special condition for lookup refs

unsure how lookup refs are resolved with history or filtered dbs

i.e., this query would never return any results given :application-spec/id was transacted before start-date

(d/q '[:find ?w
       :in $ ?workspaces-group-id
       :where
       [?workspace-group :application-spec/id ?workspaces-group-id]
       [?workspace-group :aws-workspaces-group/monitored-workspaces ?w]]
     (-> (d/history db) (d/as-of stop-date) (d/since start-date))
     workspaces-group-id)

And this throws:

(d/q '[:find ?w
       :in $ ?workspaces-group
       :where
       [?workspace-group :aws-workspaces-group/monitored-workspaces ?w]]
     (-> (d/history db) (d/as-of stop-date) (d/since start-date))
     [:application-spec/id workspaces-group-id])

So I think that means I need the ?workspace-group db/id before I do query 2.

but it may not exist at that time, right?

Which time?

as-of. the time for query 1

a group can be created and destroyed in between start and end time

Ah. If ?workspace-group doesn't exist at time 1, we would never need to run this query

Landed here:

(defn get-workspaces-over-time2
  [db workspaces-group-id start-date stop-date]
  (let [group-db-id (:db/id (d/pull db [:db/id] [:application-spec/id workspaces-group-id]))
        cur-ws (->> (d/q '[:find ?w
                           :in $ ?workspace-group
                           :where
                           [?workspace-group :aws-workspaces-group/monitored-workspaces ?w]]
                         (d/as-of db start-date) [:application-spec/id workspaces-group-id])
                    (map first))
        added-ws (->> (d/q '[:find ?w
                             :in $ ?workspaces-group
                             :where
                             [?workspace-group :aws-workspaces-group/monitored-workspaces ?w]]
                           (-> (d/history db) (d/as-of stop-date) (d/since start-date))
                           group-db-id)
                      (map first))
        all-ws (set (concat cur-ws added-ws))
        datoms (d/q '[:find ?w ?a ?v ?tx ?op
                      :in $ [?w ...]
                      :where
                      [?w ?a ?v ?tx ?op]]
                    (d/history db) all-ws)]
    datoms))

processing clause: [?w ?a ?v ?tx ?op], message: java.util.concurrent.TimeoutException: Query canceled: timeout elapsed

so you have a set of ?w at this point?

Right

how large is it?

874

your history db is unfiltered?

Yes

Using (-> (d/history db) (d/as-of stop-date) (d/since start-date)) hangs "forever". I've been letting it run since I sent the 874 message

It also caused the datomic solo instance to spike to 2000% cpu 🙂

well, last resort you can use d/datoms :eavt for each ?w

with your filtered db

Yeah... That results in ?w number of DB queries, right?

you can run them in parallel, but yes

they are lazily tailed though

queries are eager, datom-seeking is lazy

it could be the problem is result-set size

(mapcat #(d/datoms filtered-history-db :eavt %) (sort all-ws))

Hmm, ok. That is a potential solution. Thank you for working with me on this. It's been incredibly insightful. Any idea why that last query is so expensive?

Why'd you sort all-ws?

it probably won’t make a difference, but it increases the chance the next segment (in between datom calls) is already loaded

(the entire index is sorted, so fetching 1 2 3 4 5 is better than 5 2 1 4 3)

> Any idea why that last query is so expensive?

my suspicion is the result set size is large

Interesting. A bit surprised by that. Would really like to know what's in there that would cause it to be so big 🙂 In this case it shouldn't be that big.

well if your instance ever calms down that mapcat will tell you for sure

I’m not saying it will be fast, but it will use almost no memory

(just make sure you don’t hold the head on your client…)

Doing a count on it... Also hung. Must be huge.

748650

Oh wow, there is definitely an attribute in there that gets updated all the time that is useless here.

That one should probably even be :db/noHistory

I wonder if restricting the query to the attrs I'm interested would increase the perf.

After filtering out those high churn attrs, I get a coll of 576 datoms

Would need to pull the db-ids of all the attrs to filter since those are also transacted outside the between-db.

with a whitelist (or even blacklist) of attrs, you may be able to retry your query

i.e. not use datoms

Weird error doing that:

processing clause: {:argvars nil, :fn #object[datomic.core.datalog$expr_clause$fn__23535 0x11f3ef5d "datomic.core.datalog$expr_clause$fn__23535@11f3ef5d"], :clause [(ground $__in__3) [?a ...]], :binds [?a], :bind-type :list, :needs-source true}, message: java.util.concurrent.TimeoutException: Query canceled: timeout elapsed

[?e ?card-many-a ?v] [?e ?card-many-a ?v2] [(!= ?v ?v2)]

I just get [] when trying this, so maybe I'm misunderstanding something. I just tried with the mbrainz database (to use a public dataset) to do something like find tracks with multiple artists (:track/artists is cardmany ref).

(d/q '[:find ?e
       :where
       [?e :track/artists ?a]
       [?e :track/artists ?a2]
       [(!= ?a ?a2)]]
     db)

are you sure db is what you think it is? are you sure any track actually has multiple artists?

Here’s a minimal example:

(d/q '[:find ?e
       :where
       [?e :artist ?v]
       [?e :artist ?v2]
       [(!= ?v ?v2)]]
     [[1 :artist "foo"]
      [2 :artist "bar"]
      [2 :artist "baz"]])

I'm sure there are multiple artists on some tracks, and I know of a few tracks specifically.

the official cloud docs even have an example showing multiple artists on a track

however, your example returns []

?

(d/q ’[:find ?e :where [?e :artist ?v] [?e :artist ?v2] [(!= ?v ?v2)]] [[1 :artist “foo”] [2 :artist “bar”] [2 :artist “baz”]]) => #{[2]}

(from my repl)

This query doesn't work for me at all. Is this an on-prem thing?

I don’t think so? what happens?

oh, I bet it needs some kind of db somewhere in the data sources to know where to send the query

hm, not sure how I feel about that

try this:

(d/q ’[:find ?e        :in $ $db :where        [?e :artist ?v]        [?e :artist ?v2]        [(!= ?v ?v2)]]      [[1 :artist “foo”]       [2 :artist “bar”]       [2 :artist “baz”]] some-db)

it shouldn’t matter what db you provide since it’s not read

I was just trying to demonstrate in a low-effort, db-agnostic way that the self-join should work

Unable to resolve symbol: "foo" in this context

that sounds like copy-paste error?

log-in-query is not in the client API You can use tx-range, however: https://github.com/cognitect-labs/day-of-datomic-cloud/blob/master/tutorial/log.clj