Fork me on GitHub
#datomic
<
2020-04-23
>
Sam DeSota00:04:31

I don't see it documented, but it appears that you can only use the d/tx-range to map over 1000 datoms at a time? Is this correct? Requesting on a datomic cloud database with about 4 million datoms, where 13194139533312 is my first txid:

(count (into [] (d/tx-range (conn) {:start 13194139533312 :end nil}))) ;; => 1000

Sam DeSota00:04:31

Also, this behavior applies to ts

(count (into [] (d/tx-range (conn) {:start 0 :end 4000}))) ;; => 1000

Sam DeSota00:04:39

I fixed via

(defn infinite-tx-range [conn {:keys [start end]}]
  (let [current-end (+ start 1000)]
    (if (and (some? end) (>= current-end end))
      (d/tx-range conn {:start start :end end})
      (lazy-cat (d/tx-range conn {:start start :end current-end})
                (infinite-tx-range conn {:start (+ start 1000) :end end})))))

Sam DeSota00:04:00

Definitely feels like a bug.

Joe Lane00:04:37

Look at the namespace docstring https://docs.datomic.com/client-api/datomic.client.api.html you need to specify :limit -1 along with :start and :end . Example:

(count (into [] (d/tx-range (conn) {:start 0 :end 4000 :limit -1}))) ;; => 4000

Sam DeSota00:04:11

Ah, got it. Thank you very much.

đź‘Ť 4
Joe Lane00:04:37

Look at the namespace docstring https://docs.datomic.com/client-api/datomic.client.api.html you need to specify :limit -1 along with :start and :end . Example:

(count (into [] (d/tx-range (conn) {:start 0 :end 4000 :limit -1}))) ;; => 4000

Sam DeSota14:04:30

I noticed that my datomic tx count was growing faster than I expected, after inspecting the tx log, there appears to be random no-op transactions a few times per second:

[#datom[13194144633312 50 #inst "2020-04-22T23:19:36.303-00:00" 13194144633312 true]]
[#datom[13194144633313 50 #inst "2020-04-22T23:19:36.549-00:00" 13194144633313 true]]
[#datom[13194144633314 50 #inst "2020-04-22T23:19:36.771-00:00" 13194144633314 true]]
[#datom[13194144633315 50 #inst "2020-04-22T23:19:37.336-00:00" 13194144633315 true]]
[#datom[13194144633316 50 #inst "2020-04-22T23:19:38.186-00:00" 13194144633316 true]]
[#datom[13194144633317 50 #inst "2020-04-22T23:19:38.919-00:00" 13194144633317 true]]
[#datom[13194144633318 50 #inst "2020-04-22T23:19:39.696-00:00" 13194144633318 true]]
[#datom[13194144633319 50 #inst "2020-04-22T23:19:40.024-00:00" 13194144633319 true]]

Sam DeSota14:04:13

Just double checking, is this normal behavior?

favila14:04:57

It’s normal behavior…for an application that is transacting a few times times per second 🙂

Sam DeSota14:04:49

Right, but these txs have no datoms besides txInstant, and I'm probably not transacting that often. So probably a bug on my end?

Sam DeSota14:04:05

Got it, thank you

favila14:04:16

you should check before you submit the tx whether your tx-data is empty

đź‘Ť 4
favila14:04:41

datomic won’t drop a transaction--every time you call d/transact it will transact at least tx-instant, or fail

favila14:04:14

it’s also possible to submit non-empty tx that ends up not changing anything. that would also look like an empty tx

đź’Ż 4
favila14:04:34

e.g. if you reassert a datom that is already there

Sam DeSota14:04:13

Ah interesting, that's helpful

Sam DeSota15:04:58

Working adding some monitoring for the issue above ^ but all the (cast/dev) calls break with an error like this, was only able to find one older slack message in the archive, but there was no resolution for the issue. Any hints?

> (cast/dev {:msg "Test"})
No implementation of method: :-dev of protocol: #'datomic.ion.cast.impl/Cast found for class: nil

marshall15:04:50

cast/dev does not cast in production https://docs.datomic.com/cloud/ions/ions-monitoring.html#dev You’d need to redirect cast/dev before calling it

marshall15:04:19

if you’re running in an active ion, use cast/event instead

Sam DeSota15:04:21

Got it. Will cast/dev break in a REPL? Both cast/dev + cast/event break with a similar error locally in a REPL. Want to make sure it won't break my production ion.

> (cast/event {:msg "CodeDeployEvent"})
No implementation of method: :-event of protocol: #'datomic.ion.cast.impl/Cast found for class: nil

marshall15:04:43

something else is going on there do you have datomic.ion.cast in your require and also check versions you’re using

Sam DeSota15:04:49

This is my setup, checking out latest versions

;; versions
com.datomic/ion {:mvn/version "0.9.35"}
com.datomic/client-cloud {:mvn/version "0.8.81"} 

;; ns 
(ns my.util
  (:require [datomic.client.api :as d]
            [datomic.ion :as ion]
            [datomic.ion.cast :as cast]))

(defn transact [& args]
  (cast/event {:msg "CodeDeployEvent"})

  (apply d/transact args))

Sam DeSota16:04:04

These appear to be latest versions in ion starter

Sam DeSota16:04:22

Weird, isolated deps and loaded just this namespace and still having the same problem

Sam DeSota16:04:13

Okay, so I guess event/cast just doesn't work locally at all? I just cloned ion starter and same error.

Sam DeSota16:04:48

I guess I have to throw up a test endpoint to see if it works in ions

Sam DeSota16:04:43

In case anyone else runs into this, cast/event does not appear to work locally, though perhaps that can be fixed with https://docs.datomic.com/cloud/ions/ions-monitoring.html#local-workflow. When deploying to ions, it is able report to cloud watch correctly.

Sam DeSota16:04:28

Yes, calling (cast/initialize-redirect :stdout) fixes the issue locally.

onetom16:04:38

Why does the groupping happens differently in my :2 & :3 examples?

(let [names [[1 "Jane"] [2 "JaNe"] [3 "JANE"]
               [4 "paul"] [5 "Paul"]
               [6 "EVE"]
               [7 "bo"]]
        q #(->> (d/q % names)
                (sort-by (comp count second)))]

    (pp/pprint
      {:1
       (->> names
            (group-by (comp str/upper-case second))
            (vals)
            (map set)
            (sort-by count)
            #_(filter (comp pos? dec count)))

       :2
       (q '[:find ?upcase-name (distinct ?id+name)
            :in [?id+name ...]
            :where
            [(untuple ?id+name) [?id ?name]]
            [(clojure.string/upper-case ?name) ?upcase-name]])

       :3
       (q '[:find (distinct ?id+name)
            :with ?upcase-name
            :in [?id+name ...]
            :where
            [(untuple ?id+name) [?id ?name]]
            [(clojure.string/upper-case ?name) ?upcase-name]])}))
output is:
{:1
 (#{[6 "EVE"]}
  #{[7 "bo"]}
  #{[5 "Paul"] [4 "paul"]}
  #{[1 "Jane"] [2 "JaNe"] [3 "JANE"]}),
 :2
 (["BO" #{[7 "bo"]}]
  ["EVE" #{[6 "EVE"]}]
  ["PAUL" #{[5 "Paul"] [4 "paul"]}]
  ["JANE" #{[1 "Jane"] [2 "JaNe"] [3 "JANE"]}]),
 :3
 ([#{[6 "EVE"] [1 "Jane"] [5 "Paul"] [2 "JaNe"] [3 "JANE"] [4 "paul"]
     [7 "bo"]}])}
i would expect
(q '[:find (distinct ?id+name) :with ?upcase-name ...
and
(q '[:find (distinct ?id+name) ?upcase-name ...
form groups the same way

favila17:04:10

I encountered this recently too. feels like a bug?

onetom17:04:20

im pondering over this for more than an hour. read the related docs in https://docs.datomic.com/on-prem/query.html#with a few times, but i don't see any mistakes i'm making, so yes, feels like a bug to me too. where and how can i report it?

favila17:04:18

hopefully it gets visibility here, but opening a support ticket is a guaranteed way to get attention

onetom17:04:34

i've also seen situations where using the set function as an aggregate behaved differently than using distinct. it feels like a related issue maybe. have u seen anything like that? should they not be the same from a functional perspective?

onetom17:04:43

here is a more minimal example for other who might also want to play with it:

(let [names ["a" "A" "b"]]
    [(-> '[:find (distinct ?name) ?upcase-name :in [?name ...]
           :where [(clojure.string/upper-case ?name) ?upcase-name]]
         (d/q names))

     (-> '[:find (distinct ?name) :with ?upcase-name :in [?name ...]
           :where [(clojure.string/upper-case ?name) ?upcase-name]]
         (d/q names))])

=> [[[#{"a" "A"} "A"] [#{"b"} "B"]] [[#{"a" "b" "A"}]]]

donyorm22:04:51

So I'm trying to automate deployments with Amazon Codebuild, and having a working deploy script (it runs fine on my local machine). However, when it runs on the codebuild server I get the following error: Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.35 in central () . I can download the exact zip used by codebuild and run the script fine on my local machine. Why would clojure-cli not know to look for the ion jar in datomic's repo?

Alex Miller (Clojure team)22:04:10

It probably is - the error just reports the last place it looked

Alex Miller (Clojure team)22:04:06

I think this is actually maybe a known issue with code build though

Alex Miller (Clojure team)22:04:47

Where code build can’t see stuff in a different region or different vpn or something

donyorm22:04:22

Huh any chance you know a workaround? I suppose this isn't strictly necessary, but it would be nice

Alex Miller (Clojure team)22:04:10

They’ve run into this on the Datomic team iirc

Alex Miller (Clojure team)22:04:54

I’m not remembering the details

Alex Miller (Clojure team)22:04:12

Don’t think they’re available rn

donyorm22:04:17

I think I found the issue (https://stackoverflow.com/questions/48984763/aws-codebuild-cant-access-maven-repository-on-github), thanks for the hint that it was codebuild's fault

donyorm22:04:51

@U05120CBV unfortunately, I'm running this codebuild in us-east-1, so I guess it's a different issue?

tatut05:04:39

I just had this issue and ended up packaging my own ~/.m2 repo (with just com/datomic included) in a private s3 bucket, downloading and extracting that in the codebuild

tatut05:04:03

it is really unfortunate workaround but I couldn't get access to the datomic releases, even in the same region

stijn08:04:54

Are your permissions for Codebuild setup correctly? You need either Administrator access or add this to an IAM policy that is attached to the codebuild instance profile:

{
            "Sid": "DatomicReleasesAccess",
            "Effect": "Allow",
            "Action": "*",
            "Resource": [
                "arn:aws:s3:::datomic-releases-1fc2183a/*",
                "arn:aws:s3:::datomic-releases-1fc2183a"
            ]
        }

donyorm15:04:28

@U0539NJF7 That's probably it. I'll look into it.

donyorm15:04:29

Yep that seemed to do it. Thanks, stijn