This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-04-23
Channels
- # announcements (3)
- # babashka (68)
- # beginners (297)
- # calva (13)
- # cider (4)
- # clj-kondo (8)
- # cljs-dev (10)
- # cljsrn (26)
- # clojure (100)
- # clojure-europe (4)
- # clojure-germany (1)
- # clojure-italy (9)
- # clojure-nl (5)
- # clojure-spec (9)
- # clojure-uk (41)
- # clojurescript (69)
- # conjure (70)
- # cursive (44)
- # data-science (20)
- # datascript (2)
- # datomic (55)
- # emacs (1)
- # exercism (3)
- # graalvm (2)
- # kaocha (11)
- # leiningen (6)
- # meander (9)
- # mental-health (1)
- # off-topic (73)
- # pathom (6)
- # pedestal (1)
- # re-frame (3)
- # reagent (52)
- # reitit (8)
- # rum (39)
- # shadow-cljs (152)
- # spacemacs (10)
- # tools-deps (28)
- # xtdb (5)
I don't see it documented, but it appears that you can only use the d/tx-range
to map over 1000 datoms at a time? Is this correct? Requesting on a datomic cloud database with about 4 million datoms, where 13194139533312 is my first txid:
(count (into [] (d/tx-range (conn) {:start 13194139533312 :end nil}))) ;; => 1000
Also, this behavior applies to t
s
(count (into [] (d/tx-range (conn) {:start 0 :end 4000}))) ;; => 1000
I fixed via
(defn infinite-tx-range [conn {:keys [start end]}]
(let [current-end (+ start 1000)]
(if (and (some? end) (>= current-end end))
(d/tx-range conn {:start start :end end})
(lazy-cat (d/tx-range conn {:start start :end current-end})
(infinite-tx-range conn {:start (+ start 1000) :end end})))))
Definitely feels like a bug.
Look at the namespace docstring https://docs.datomic.com/client-api/datomic.client.api.html you need to specify :limit -1
along with :start
and :end
. Example:
(count (into [] (d/tx-range (conn) {:start 0 :end 4000 :limit -1}))) ;; => 4000
Look at the namespace docstring https://docs.datomic.com/client-api/datomic.client.api.html you need to specify :limit -1
along with :start
and :end
. Example:
(count (into [] (d/tx-range (conn) {:start 0 :end 4000 :limit -1}))) ;; => 4000
I noticed that my datomic tx count was growing faster than I expected, after inspecting the tx log, there appears to be random no-op transactions a few times per second:
[#datom[13194144633312 50 #inst "2020-04-22T23:19:36.303-00:00" 13194144633312 true]]
[#datom[13194144633313 50 #inst "2020-04-22T23:19:36.549-00:00" 13194144633313 true]]
[#datom[13194144633314 50 #inst "2020-04-22T23:19:36.771-00:00" 13194144633314 true]]
[#datom[13194144633315 50 #inst "2020-04-22T23:19:37.336-00:00" 13194144633315 true]]
[#datom[13194144633316 50 #inst "2020-04-22T23:19:38.186-00:00" 13194144633316 true]]
[#datom[13194144633317 50 #inst "2020-04-22T23:19:38.919-00:00" 13194144633317 true]]
[#datom[13194144633318 50 #inst "2020-04-22T23:19:39.696-00:00" 13194144633318 true]]
[#datom[13194144633319 50 #inst "2020-04-22T23:19:40.024-00:00" 13194144633319 true]]
Just double checking, is this normal behavior?
It’s normal behavior…for an application that is transacting a few times times per second 🙂
Right, but these txs have no datoms besides txInstant, and I'm probably not transacting that often. So probably a bug on my end?
Got it, thank you
datomic won’t drop a transaction--every time you call d/transact it will transact at least tx-instant, or fail
it’s also possible to submit non-empty tx that ends up not changing anything. that would also look like an empty tx
Ah interesting, that's helpful
Working adding some monitoring for the issue above ^ but all the (cast/dev)
calls break with an error like this, was only able to find one older slack message in the archive, but there was no resolution for the issue. Any hints?
> (cast/dev {:msg "Test"})
No implementation of method: :-dev of protocol: #'datomic.ion.cast.impl/Cast found for class: nil
cast/dev does not cast in production https://docs.datomic.com/cloud/ions/ions-monitoring.html#dev You’d need to redirect cast/dev before calling it
Got it. Will cast/dev
break in a REPL? Both cast/dev
+ cast/event
break with a similar error locally in a REPL. Want to make sure it won't break my production ion.
> (cast/event {:msg "CodeDeployEvent"})
No implementation of method: :-event of protocol: #'datomic.ion.cast.impl/Cast found for class: nil
something else is going on there do you have datomic.ion.cast in your require and also check versions you’re using
This is my setup, checking out latest versions
;; versions
com.datomic/ion {:mvn/version "0.9.35"}
com.datomic/client-cloud {:mvn/version "0.8.81"}
;; ns
(ns my.util
(:require [datomic.client.api :as d]
[datomic.ion :as ion]
[datomic.ion.cast :as cast]))
(defn transact [& args]
(cast/event {:msg "CodeDeployEvent"})
(apply d/transact args))
These appear to be latest versions in ion starter
Weird, isolated deps and loaded just this namespace and still having the same problem
Okay, so I guess event/cast
just doesn't work locally at all? I just cloned ion starter and same error.
I guess I have to throw up a test endpoint to see if it works in ions
In case anyone else runs into this, cast/event
does not appear to work locally, though perhaps that can be fixed with https://docs.datomic.com/cloud/ions/ions-monitoring.html#local-workflow. When deploying to ions, it is able report to cloud watch correctly.
Yes, calling (cast/initialize-redirect :stdout)
fixes the issue locally.
Why does the groupping happens differently in my :2
& :3
examples?
(let [names [[1 "Jane"] [2 "JaNe"] [3 "JANE"]
[4 "paul"] [5 "Paul"]
[6 "EVE"]
[7 "bo"]]
q #(->> (d/q % names)
(sort-by (comp count second)))]
(pp/pprint
{:1
(->> names
(group-by (comp str/upper-case second))
(vals)
(map set)
(sort-by count)
#_(filter (comp pos? dec count)))
:2
(q '[:find ?upcase-name (distinct ?id+name)
:in [?id+name ...]
:where
[(untuple ?id+name) [?id ?name]]
[(clojure.string/upper-case ?name) ?upcase-name]])
:3
(q '[:find (distinct ?id+name)
:with ?upcase-name
:in [?id+name ...]
:where
[(untuple ?id+name) [?id ?name]]
[(clojure.string/upper-case ?name) ?upcase-name]])}))
output is:
{:1
(#{[6 "EVE"]}
#{[7 "bo"]}
#{[5 "Paul"] [4 "paul"]}
#{[1 "Jane"] [2 "JaNe"] [3 "JANE"]}),
:2
(["BO" #{[7 "bo"]}]
["EVE" #{[6 "EVE"]}]
["PAUL" #{[5 "Paul"] [4 "paul"]}]
["JANE" #{[1 "Jane"] [2 "JaNe"] [3 "JANE"]}]),
:3
([#{[6 "EVE"] [1 "Jane"] [5 "Paul"] [2 "JaNe"] [3 "JANE"] [4 "paul"]
[7 "bo"]}])}
i would expect
(q '[:find (distinct ?id+name) :with ?upcase-name ...
and
(q '[:find (distinct ?id+name) ?upcase-name ...
form groups the same wayim pondering over this for more than an hour. read the related docs in https://docs.datomic.com/on-prem/query.html#with a few times, but i don't see any mistakes i'm making, so yes, feels like a bug to me too. where and how can i report it?
hopefully it gets visibility here, but opening a support ticket is a guaranteed way to get attention
i've also seen situations where using the set
function as an aggregate behaved differently than using distinct
.
it feels like a related issue maybe.
have u seen anything like that?
should they not be the same from a functional perspective?
here is a more minimal example for other who might also want to play with it:
(let [names ["a" "A" "b"]]
[(-> '[:find (distinct ?name) ?upcase-name :in [?name ...]
:where [(clojure.string/upper-case ?name) ?upcase-name]]
(d/q names))
(-> '[:find (distinct ?name) :with ?upcase-name :in [?name ...]
:where [(clojure.string/upper-case ?name) ?upcase-name]]
(d/q names))])
=> [[[#{"a" "A"} "A"] [#{"b"} "B"]] [[#{"a" "b" "A"}]]]
Submitted the issue as https://support.cognitect.com/hc/en-us/requests/2668
So I'm trying to automate deployments with Amazon Codebuild, and having a working deploy script (it runs fine on my local machine). However, when it runs on the codebuild server I get the following error: Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.35 in central (
. I can download the exact zip used by codebuild and run the script fine on my local machine. Why would clojure-cli not know to look for the ion jar in datomic's repo?
It probably is - the error just reports the last place it looked
I think this is actually maybe a known issue with code build though
Where code build can’t see stuff in a different region or different vpn or something
Huh any chance you know a workaround? I suppose this isn't strictly necessary, but it would be nice
They’ve run into this on the Datomic team iirc
I’m not remembering the details
Don’t think they’re available rn
I think I found the issue (https://stackoverflow.com/questions/48984763/aws-codebuild-cant-access-maven-repository-on-github), thanks for the hint that it was codebuild's fault
@U05120CBV unfortunately, I'm running this codebuild in us-east-1, so I guess it's a different issue?
I just had this issue and ended up packaging my own ~/.m2 repo (with just com/datomic included) in a private s3 bucket, downloading and extracting that in the codebuild
it is really unfortunate workaround but I couldn't get access to the datomic releases, even in the same region
Are your permissions for Codebuild setup correctly? You need either Administrator access or add this to an IAM policy that is attached to the codebuild instance profile:
{
"Sid": "DatomicReleasesAccess",
"Effect": "Allow",
"Action": "*",
"Resource": [
"arn:aws:s3:::datomic-releases-1fc2183a/*",
"arn:aws:s3:::datomic-releases-1fc2183a"
]
}
@U0539NJF7 That's probably it. I'll look into it.