This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-07-01
Channels
- # atom-editor (11)
- # babashka (25)
- # beginners (142)
- # boot (9)
- # calva (3)
- # cider (19)
- # clara (15)
- # clj-kondo (6)
- # cljs-dev (20)
- # clojars (11)
- # clojure (164)
- # clojure-dev (9)
- # clojure-europe (6)
- # clojure-italy (17)
- # clojure-nl (3)
- # clojure-spec (19)
- # clojure-sweden (10)
- # clojure-uk (23)
- # clojurescript (34)
- # code-reviews (31)
- # conjure (20)
- # cursive (14)
- # datomic (54)
- # emacs (1)
- # fulcro (51)
- # graalvm (24)
- # graphql (6)
- # helix (3)
- # jobs (3)
- # kaocha (1)
- # malli (2)
- # meander (15)
- # off-topic (81)
- # pathom (2)
- # re-frame (43)
- # reagent (26)
- # reitit (1)
- # releases (1)
- # sci (12)
- # shadow-cljs (29)
- # sql (22)
- # timbre (3)
- # tools-deps (15)
Hello! I’m wondering if I can get some help with my Datomic Cloud instance that seems to have gone sound — in fact, I’m on a call with @plexus trying to puzzle this out. 1. I’m getting “channel 2: open failed: connect failed: Connection refused” errors on the proxy, when a Datomic Client tries to access the Datomic Cloud instance. 2. In AWS CloudWatch, I see the following alarm, which occurred very close to when we started seeing Datomic connection errors occurring. Can anyone propose any recommendations? @plexus, any other data worth sharing? (Sorry, gotta pop off for 30m. Thank you, all!)
Error I get from a REPL connection:
Execution error (ExceptionInfo) at datomic.client.impl.cloud/get-s3-auth-path (cloud.clj:178).
Unable to connect to localhost:8182
reading some more AWS docs it seems we have exceeded the allocated write throughput, which is supposed to only cause throttling, but instead the datomic instance has gone under or become unreachable...
if you have any Alerts (not just "Events") in that dashboard, look at those too by navigating to cloudwatch logs
Thank you @U050ECB92 — is this the dashboard? (Sorry, on a call…. 🙂
(Empty dashboard was the reason I was asking Datomic team at Conj 2019 about getting help upgrading last year, which I never got around to.)
your instance and/or JVM got wedged and b/c solo is not an HA system there is nothing to fail-over to
Posting this datomic log event, before I destroy the solo instance:
2020-06-25T22:54:28.953-07:00
{
"Msg": "RestartingDaemonException",
"Ex": {
"Via": [
{
"Type": "clojure.lang.ExceptionInfo",
"Message": "Unable to load index root ref bd9b3c36-2912-437d-8fc7-6953ab60a1b2",
"Data": {
"Ret": {},
"DbId": "bd9b3c36-2912-437d-8fc7-6953ab60a1b2"
},
"At": [
"datomic.index$require_ref_map",
"invokeStatic",
"index.clj",
843
]
}
],
"Trace": [
[
"datomic.index$require_ref_map",
"invokeStatic",
"index.clj",
843
],
[
"datomic.index$require_ref_map",
"invoke",
"index.clj",
836
],
[
"datomic.index$require_root_id",
"invokeStatic",
"index.clj",
849
],
[
"datomic.index$require_root_id",
"invoke",
"index.clj",
846
],
[
"datomic.adopter$start_adopter_thread$fn__21647",
"invoke",
"adopter.clj",
67
],
[
"datomic.async$restarting_daemon$fn__10442$fn__10443",
"invoke",
"async.clj",
162
],
[
"datomic.async$restarting_daemon$fn__10442",
"invoke",
"async.clj",
161
],
[
"clojure.core$binding_conveyor_fn$fn__5739",
"invoke",
"core.clj",
2030
],
[
"datomic.async$daemon$fn__10439",
"invoke",
"async.clj",
146
],
[
"clojure.lang.AFn",
"run",
"AFn.java",
22
],
[
"java.lang.Thread",
"run",
"Thread.java",
748
]
],
"Cause": "Unable to load index root ref bd9b3c36-2912-437d-8fc7-6953ab60a1b2",
"Data": {
"Ret": {},
"DbId": "bd9b3c36-2912-437d-8fc7-6953ab60a1b2"
}
},
"Type": "Alert",
"Tid": 6306,
"Timestamp": 1593150867958
Okay, terminated the datomic instance, which didn’t work… terminated the bastion-host instance, which didn’t work… terminated the datomic proxy script, and restarted… forced some sort of reauthentication, which did work! Thank you, all!
you’d definitely need to restart the proxy script after restarting the bastion instance
Is it possible to implement a correct task queue in Datomic? Mostly importantly, ensure that multiple task retrievers won't get the same task from the top of the queue. (In PostgreSQL for example I needed to use LOCK FOR UPDATE
)
It's certainly possible to make a queue out of datomic, but why not just use an actual queue? I also don't necessarily think it's a good idea to use datomic as a queue, depending on the throughput, failure semantics, and data retention you need.
@U0CJ19XAM Good point. I am looking into https://github.com/Factual/durable-queue as well.
Actually, I just realized a queue might not satisfy what I need. There isn't a static queue. Tasks have priorities and they might be changed dynamically. Every task retriever grabs the top-priority job from the database at the moment it accesses the database. Is there a established solution or pattern for something like that?
Depends on the domain, if this is something for humans (like a Jira / Trello clone) then this is easy. If this is for machines, it depends on your throughput, scale, and failure modes.
That being said, you may be interested in https://github.com/clojure/data.priority-map
The job retrievers are machines. I don't think an in-memory solution would work well for my particular case, plus, the tasks and their attributes (from which to compute the priority) are already stored in a datomic database so I that's why I was wondering if there's some sort of locking mechanism between querying and updating...
The performance of the priority sorting isn't that much of a problem, at the moment an index on the priority attribute should work well enough
In other words, is there a way to say "change the first item satisfying my query to have attribute [:task/taken true]
" -- all within an atomic transaction
Yes, via a transaction function, but I don't think it's going to work out well in the end. What happens once a task is taken but then the task retriever dies? What are your retry policies? How do you distinguish between a slow job and a failed job?
Do you have different levels of prioritization like low, medium and high, or is everything prioritized globally? If you can do the former, I think SQS with a queue per level is likely a better approach.
Thanks. That makes sense. What if I'm not using a standard cloud service? Can Kafka serve a similar purpose?
(It could do this as well, but may be more difficult to operate. Again, I know nothing of your problem domain, scale, other constraints, etc. so it's hard to make a good recommendation)
I don't suppose there is any way to force a peer to use an alternative address than the one provided by the transactor (retrieved from storage), is there?
I’m not sure about ports. I dimly recall that you can specify port in the connection string, but that might only be for dev storage
or alternative port, at least?
i.e., transactor running on transactorUrl:4334
but is being reverse proxied at
with appropriate firewall rules VPN etc etc
okay looks like we're able to change the port on the LB. still curious if this is possible, tho