This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-11-05
Channels
- # 100-days-of-code (1)
- # announcements (7)
- # beginners (84)
- # boot (1)
- # cider (22)
- # cljdoc (14)
- # cljs-dev (45)
- # cljsrn (6)
- # clojure (65)
- # clojure-conj (7)
- # clojure-finland (1)
- # clojure-italy (7)
- # clojure-nl (2)
- # clojure-serbia (1)
- # clojure-uk (111)
- # clojurescript (58)
- # cursive (8)
- # datomic (68)
- # duct (1)
- # emacs (33)
- # figwheel (3)
- # figwheel-main (9)
- # fulcro (33)
- # graphql (1)
- # juxt (30)
- # kaocha (4)
- # off-topic (22)
- # pathom (47)
- # pedestal (4)
- # planck (6)
- # re-frame (1)
- # reagent (1)
- # reitit (13)
- # shadow-cljs (49)
- # spacemacs (7)
- # sql (6)
- # tools-deps (60)
this weekend i started a mini project to explore datomic cloud (solo topo). i accidentally used the wrong key-pair when setting up cloudformation. after the stack was created I deleted the parent stack (which also removed storage, compute stacks). i then followed https://docs.datomic.com/cloud/operation/deleting.html to remove all durable storage stuff, too. i used aws’ tag search feature to find anything residual created by cloudformation to make sure it was removed, too (the only thing still around were the old terminated ec2 instances). when i try to follow the marketplace template again (this time picking the correct key pair) i keep getting a failure creating the new Storage stack:
The following resource(s) failed to create: [StorageF7F305E7]. . Rollback requested by user.
inside of that storage failure, the error resources are:
The following resource(s) failed to create: [LogTableReadScalingPolicy, MountTarget1, LogTableWriteScalingPolicy, MountTarget2, MountTarget0, AdminPolicy].
i’m not clear on why these steps failed, other than maybe they weren’t 100% cleaned up properly following the delete steps?
should i even be able to re-create a stack with the exact same stack name (and app name, for that matter)?
in general yes, although it could be that some resources did not get deleted, which would cause the kind of issue you’re seeing
i’ve had issues with CF in the past where a stack gets stuck in some interminable state, and i’ve had to contact aws support directly to clear things up
I received this exception during the ValidateService
while deploying my Ions:
{
"Type": "clojure.lang.Compiler$CompilerException",
"Message": "java.lang.StackOverflowError, compiling:(potemkin/walk.clj:10:13)",
"At": [
"clojure.lang.Compiler",
"analyze",
"Compiler.java",
6792
]
}
This is the first time I have received this exception. It appears to be coming from clj-http. Upon redeploying the Ions, deploy succeeded. It smells like some sort of race condition. Any idea how I can avoid this problem in the future?@kenny if you are using solo, it's likely a memory limit problem we all experience. The location in the exception is irrelevant. There is a workaround if you upgrade the datomic stack and edit the CF template to increase JVM memory Params. I haven't done this yet, I just rerun deploys and ignore it
@kenny i’m working on a doc improvement that should help suss that out and reproduce locally I’ll ping you when i’ve got it put together
@kenny Take a look here: https://docs.datomic.com/cloud/ions/ions-reference.html#jvm-settings Can you try running locally with the same JVM settings used in your stack and let us know if you don’t see the same behavior locally?
the idea is to load the ion code in a JVM with the same memory settings used by Datomic Cloud
Oh. I test my Ions exactly as you described and have never hit the exception that was hit in production.
Probably with far less memory. I haven't configured any JVM properties on my system and I'm guessing the defaults are really low.
Our Ion tests run via CI where the machine only has 4gb. The tests have been run thousands of times without hitting that exception.
And we are running production topology with i3.large. Unless you think having extra memory is the issue, I don't think that exception was due to RAM.
Gotcha. Our CI runs on 4gb and has not hit that exception. Datomic Ions run on i3.large and according to that chart that means they have 10.52gb available.
No, it is a CircleCI container. It runs the Ions by executing clojure
, as you suggested, and has not hit the exception.
It is using the default clojure
uses which I'd guess is the Java defaults. I was assuming, perhaps incorrectly, that the production topology configuration would configure that higher than the defaults. I do now see the GC flags which are not being used right now. I'll try a few runs with those flags.
No failures out of 10 runs. I don't know what the failure rate in production is either because I have only hit this one time.
I suspect something that library was doing was using a lot of stack space (deeply recursive structure of some sort maybe). Your production instance happened to be doing something else at the time you tried to deploy and the combined load exceeded the -Xss
@U0539NJF7 They are the various instance types used by Datomic Cloud
ok. so, if that still fails with the StackOverflowError, what are the resolutions for that?
I would first want to see whether that failure occurs locally with the same stack settings
but, in general, the resolution is to alter the use, compiling, and loading of libraries that are causing it
(because we tried updating the jvm settings on the template, but it made the nodes not come up after termination - probably made a mistake)
ok, I think that makes sense. we had the problem earlier with using the aleph http client and loading the netty namespaces was failing. replacing it with clj-http worked in this case
@U0539NJF7 interestingly for us, the lib that caused this problem in the first place was clj-http 🙃