This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-04-27
Channels
- # architecture (6)
- # beginners (36)
- # boot (4)
- # cider (74)
- # cljsrn (5)
- # clojure (87)
- # clojure-denver (2)
- # clojure-finland (2)
- # clojure-gamedev (5)
- # clojure-italy (10)
- # clojure-nl (1)
- # clojure-uk (45)
- # clojurescript (33)
- # code-reviews (5)
- # core-async (12)
- # cursive (17)
- # datascript (2)
- # datomic (71)
- # duct (4)
- # emacs (19)
- # figwheel (1)
- # fulcro (4)
- # garden (1)
- # hoplon (18)
- # jobs (5)
- # leiningen (2)
- # off-topic (73)
- # onyx (14)
- # overtone (2)
- # portkey (32)
- # re-frame (62)
- # reagent (46)
- # shadow-cljs (76)
- # spacemacs (2)
- # sql (14)
- # tools-deps (5)
- # yada (3)
if I get the following exception when using the client lib, is this due to the networking not being OK or the S3 permissions?
Caused by: clojure.lang.ExceptionInfo: Unable to connect to system: {:cognitect.anomalies/category :cognitect.anomalies/unavailable, :cognitect.anomalies/message "Connection refused"} {:config {:server-type :cloud, :region "eu-central-1", :system "datomic", :query-group "datomic", :endpoint " ", :proxy-port 8182, :endpoint-map {:headers {"host" ""}, :scheme "http", :server-name "", :server-port 8182}}}
I'm able to curl
, so I guess the networking is OK, but the error doesn't mention anything about specific database access...
So today, I refactored a dozen small queries, executed sequentially, into one query that's an or-join
. I was surprised to find that it runs an order of magnitude slower. Is there any information on why this could be, or any information on how planning works?
Here's the big form: https://gist.github.com/eraserhd/23ab2c6d65c26638b0b5ab342521bfa7
The and
clauses are sorted so that they should be consistently be in the same order. This reduces the query from 1.2s to 0.8s, presumably because of compile caching? Although, I tried inlining a the new-result rule and it went back to 1.2s. I don't know why. The individual queries take a total of ~57ms.
That should be 2g, which should be waaay more than enough. The dev database is seeded from a file with at most 300 tuples in it.
the only thing I can think to try is make one named rule and put each "and" as a separate implementation
the only thing I can think of that could possibly do that is if memory pressure causes evictions for the various parts running in parallel
alternatively, there's something wrong with the query compiler; but since these are all big scans anyway I don't know what it could do that would be worse!
this is a minor point, but !=
should be faster than not=
in cases where you don't need clojure type coersion
When you create a database via datomic.api/create-database (e.g. datomic:<sql://test-db?jdbc:...>), where does the created database (e.g. test-db) get created? Does it create an actual new table or db in the backing sql instance or is it all in the datomic.datomic_kvs table?
so "create-database" with an sql storage is an "INSERT" statement, not "CREATE TABLE" or "CREATE SCHEMA"
cool, thanks
FYI transactor only needs SELECT INSERT UPDATE DELETE permissions for its own datomic_kvs table, and peers only need SELECT
ah, cool
didn't know that
Are there any good rules of thumb regarding how many actual datomic_kvs tables/transactors to have? I realize you only use one transactor per datomic_kvs table, but at what point would you want multiple transactors? For example, if an organization had a few projects would it make sense to have a single transactor and each project have it's own virtual databases or would each want their own transactor?
in general fewer transactors is easier; you split only if your write volume exceeds what a transactor can handle (remember, single writer for all dbs) or the io exceeds what the underlying storage can handle; or if you want fewer points of failure; or if you really just want to be super-sure different orgs have their data completely siloed
Thanks. That make sense.
Yeah, just confirmed. If I extract the query and run it (without rebuilding it), the first time is about 300ms, and every subsequent time is about 20ms. If I wrap the captured query with (read-string (pr-str ...)), it goes back to 300ms every time.
Although, if I try minor perturbations of the query, it still seems to work. So there's something in the query form with a bad hash or equal?
uh, yeah.... whoa
dev=> (hash foo/test-q)
-1500982317
dev=> (hash (read-string (pr-str foo/test-q)))
1049440012
dev=> (hash (read-string (pr-str (read-string (pr-str foo/test-q)))))
-604086260