This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-03-12
Channels
- # beginners (4)
- # boot (58)
- # braid-chat (9)
- # cider (19)
- # clojure (26)
- # clojure-austin (8)
- # clojure-berlin (1)
- # clojure-poland (2)
- # clojure-russia (238)
- # clojurescript (22)
- # core-async (2)
- # cursive (6)
- # datomic (32)
- # emacs (5)
- # hoplon (1)
- # jobs (10)
- # keechma (1)
- # ldnclj (2)
- # off-topic (5)
- # om (7)
- # onyx (4)
- # proton (1)
- # re-frame (10)
- # reagent (5)
- # ring-swagger (10)
just experienced something odd with datomic in production, there was a stream of "transactor unavailable" exceptions in the log, and when i manually repled into that peer, and tried something like (d/sync conn) , i got the same error. we've seen this before ,and it never recovers
similar to this: http://datomic.narkive.com/ho4GcCOS/recover-from-db-error-transactor-unavailable-exception
@greywolve: do you have metrics/monitoring (or logs you can grep)? One case where this can happen is with extremely large transaction sizes (1MB+).
it's happened a couple of times now, next time we'll have some flight recorder metrics too
is there anything i can check the transactor for?, that's the only thing we have in the peer logs
this is our onyx cluster, we have other peers up on our regular servers, and they seem fine
function metric-grep () {
cat *.log | perl -n -e 'print "$1 $2\n" if /^(.*) INFO .* '"$1"' {.*?'"$2"' ([0-9]+).*?}/' | less
}
metric-grep :TransactionBytes :hi
or metrics (max over one minute), just to double check, what’s the largest transaction size?
Ok, transaction size unlikely to be the issue then. Hmm, I’m not familiar enough with what Onyx is doing to reason about it much further difference wise yet. Have you done the basics lein deps :tree
check for any dependency conflicts, etc.?
bkamphaus: onyx isn't really doing any more than reading from the log api (polling it), and using datomic's transact, that's about it, nothing fancy. i'll check the deps though to be safe
If there's a final tx from the transactor logs, it will be logged with a uuid - you can use that against the log API with tx-range to figure out which final transaction the peer made before failing. It's a key in the nested data structure, not something you can look up directly, and you need a reasonable t/tx/inst bound for the tx-range.
On phone now, I can pull up a code example when I get back to a keyboard :)
@greywolve https://gist.github.com/benkamphaus/7eaa6484a254a14f8f1f just pulled this out of another project and slightly refactored without testing in isolation (will test it and fix any typos if I get a chance later), so you may have to make a minor correction or two.