This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-07-06
Channels
- # admin-announcements (2)
- # boot (51)
- # bristol-clojurians (1)
- # capetown (14)
- # cider (4)
- # cljs-dev (3)
- # cljsrn (23)
- # clojure (76)
- # clojure-gamedev (2)
- # clojure-germany (2)
- # clojure-greece (2)
- # clojure-hk (5)
- # clojure-poland (1)
- # clojure-quebec (3)
- # clojure-russia (19)
- # clojure-spec (7)
- # clojure-sweden (10)
- # clojure-uk (77)
- # clojurescript (42)
- # clojurex (5)
- # core-async (40)
- # cursive (12)
- # datomic (58)
- # editors (1)
- # events (1)
- # heroku (1)
- # hoplon (4)
- # jobs (6)
- # jobs-discuss (1)
- # ldnclj (2)
- # lein-figwheel (1)
- # leiningen (5)
- # om (66)
- # onyx (51)
- # other-languages (80)
- # proton (20)
- # protorepl (12)
- # quil (3)
- # re-frame (3)
- # reagent (18)
- # spacemacs (14)
- # untangled (78)
- # yada (16)
My transactor keeps getting stuck in a state where it can't start and just hangs on start for 50 min and dies with:
Jun 30 21:51:49 ip-172-16-211-135.ec2.internal bash[2005]: Launching with Java options -server -Xms1g -Xmx24g -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -Xloggc:/var
Jun 30 21:51:53 ip-172-16-211-135.ec2.internal bash[2005]: Starting datomic: ...
Jun 30 21:51:53 ip-172-16-211-135.ec2.internal bash[2005]: System started datomic:
Jun 30 22:55:36 ip-172-16-211-135.ec2.internal bash[2005]: Critical failure, cannot continue: Indexing retry limit exceeded.
Jun 30 22:56:07 ip-172-16-211-135.ec2.internal systemd[1]: datomic.service: Main process exited, code=exited, status=255/n/a
I'm at something of a loss, I can't make it get into this state intentionally and once it gets into that state it stays that way to up five days
Is there such a thing in datomic as a "rebuild all the indexes from scratch" command or is this a silly question
@arthur: doesn’t look like that’s necessarily the solution to your problem. Your system may be underprovisioned (i.e. not enough dynamo throughput provisioned), or you might have catastrophic GC happening with a heap that large (really no benefit to going more than ~12 max heap on a transactor). It’s hard to say what is, though logs/metrics will show bottlenecks for write throughput and things like laggy heartbeat that would point to catastrophic GC. If you can’t find an obvious culprit in storage throughput, you may want to reach out to someone at Cognitect support in case it’s a bug or something their diagnostic tools will help you locate.
Seeing some funky behavior where the tx metadata associated with a particular transaction becomes stale after the first transaction. That is, the db/txInstant
and db/id
of the transaction associated with the latest entity do not change after the first transaction, even when the actual entity attributes returned reflect more recent transactions.
It can’t be Redundancy Elimination (http://docs.datomic.com/transactions.html#redundancy-elimination) because of the :db/txInstant
@pheuter not sure I follow what you mean. An entity can span multiple transactions, so you need some logic defining which transaction concerning the entity you want information about.
I see, I was assuming that given a single d/db
value, a (pull ?tx-id [*])
will return the latest transaction attributes, no?
how does your query relate the entity in question to the transaction?
how are ?a
and ?v
bound? if not at all, should replace with _
— and that version of the query should return multiple relations of the pulled e
to the pulled ?tx-id
.
in our particular case we’re dynamically generating these queries from a json request, they’re being filled in with actual values, so it looks more like:
[:find (pull ?e [*]) (pull ?tx-id [*])
:where [?e :user/firstName “Bob" ?tx-id]]
so in that case you’ll get the ?tx-id
for when that specific fact — the user first name — was specified.
let’s say the query is this:
[:find (pull ?e [*]) (pull ?tx-id [*])
:where [?e :user/firstName _ ?tx-id]]
any ?tx-id
in the database (note if you’re using a present database this does not include history) in which a fact about :user/firstName
for that ?e
was asserted.
i see, so we can potentially be getting back an older transaction as long as the entity has a first name, even if it has changed since then
out of order reply
No that’s not what’s going on, I don’t think.
transaction ids are for facts about entities
there is no idea of an entity id -> transaction id mapping that transcends facts
entities are projected from facts
transaction ids are for facts
if you want to get the latest transaction with an entity, you get the max ?tx-id for [?e ?tx-id]
yeah, i guess i was hoping to be able to order all transaction facts associated with an entity by txInstant
and get the most recent one
not necessary to order by txInstant
and less reliable to do so (transaction granularity means transactions can occur on the same instant) — transaction ids are guaranteed to ascend monotonically, just max that.
of course if you want readable time it might make sense. Here’s an SO answer where I don’t follow the advice I just gave you 🙂 http://stackoverflow.com/a/24655980
hm, seeing the following error now: clojure.lang.PersistentList cannot be cast to java.lang.Number
query looks like this:
[:find (pull ?e pattern) (pull ?tx-id [*]):where [?e :account/email _] [?e _ _ (max ?tx-id)]]
definitely can’t nest function calls inside single where clauses in general and aggregates can only legally be called in the :find
clause as @marshall points out. I think you’ll run into issues using the pull
combined.
You could probably get max txInstant and still pull the ?tx-id? Or try and subquery it, e.g. http://stackoverflow.com/a/30771949
not the first thing I’d reach for, depends on what your need to do things in one query is. In general, you’re better off breaking up pull/entity (project analogues) of query from the where clause/select portions, they’re more composable and general purpose that way.
Also keep in mind since query occurs in process on the peer and a lot of this data will be cached there isn't the same cost for multiple queries as there would be in a traditional rdbms
if you’re building queries (as your earlier discussion implies) via REST or a client layer where you don’t get the same benefit of caching and have to round trip to the peer it’s a different story.
yeah, in this case i’m building a query dynamically from a client request and the desired behavior is to return the latest tx metadata associated with the entities returned
it’s a fair point that queries are heavily cached on the peer at it may not be the worst thing to make two queries to fulfill each request
hm, is it possible to pass a particular entity into a subquery as a query arg?
[:find (pull ?e [*]) (pull ?latest-tx [*])
:in $
:where
[?e :account/email _]
[(datomic.api/q [:find (max ?tx-id) :where [?e _ _ ?tx-id]]
$ ?e) [[?latest-tx]]]
[?e _ _ ?latest-tx]]