Fork me on GitHub
#datomic
<
2019-08-14
>
johnjelinek04:08:03

tryna set up ions

johnjelinek04:08:30

looks like something error'd:

clojure -A:dev -m datomic.ion.dev '{:op :deploy-status :execution-arn arn:aws:states:us-east-2:101416954809:execution:datomic-dev-Compute-784GREJAJTLX:dev-Compute-784GREJAJTLX-bd6deb15afeee59dd2dd16943cf3c0313f534c34-1565755664290}'

{:deploy-status "FAILED", :code-deploy-status "FAILED"}

johnjelinek04:08:09

{...
      "status": "Failed",
      "errorInformation": {
        "code": "HEALTH_CONSTRAINTS",
        "message": "The overall deployment failed because too many individual instances failed deployment, too few healthy instances are available for deployment, or some instances in your deployment group are experiencing problems."
      }
}

johnjelinek04:08:04

all my CloudWatch Logs look like:

START RequestId: a012efad-3c01-4a75-8b08-615978c5f177 Version: $LATEST
2019-08-14T04:11:53.124Z	a012efad-3c01-4a75-8b08-615978c5f177	{ event: 
{ codeDeploy: { deployment: [Object] },
lambda: { cI: 4, c: [Array], uI: -1, u: [], dI: -1, d: [], common: [Object] } } }
END RequestId: a012efad-3c01-4a75-8b08-615978c5f177

marshall13:08:32

you need to determine why the instances are not starting up - usually caused by an error in the ion code that is preventing it from loading

johnjelinek23:08:53

I was using the starter ion code

marshall23:08:35

Search your cloudwatch logs for the system you deployed to

marshall23:08:54

See if any errors or exceptions show up in there

johnjelinek01:08:40

I posted the messages from cloudwatch logs above

marshall01:08:40

The logs from your datomic stack. Not codedeploy

marshall01:08:54

Take a look at the link to the docs i posted. It includes details about finding the datomic stack logs

johnjelinek04:08:26
replied to a thread:tryna set up `ion`s

any idea how to troubleshoot?

mkvlr08:08:14

we’ve been running with a shared valcache for about a week in production now. When deploying, valcache is briefly accessed by two instances which we heard from @jaret is not offically supported but should work. We’re now seeing EOFException pop up originating in datomic.index/reify every now and then. Is this a setup you plan to support or should we stop running like this?

mkvlr08:08:03

this is the stacktrace if it helps. Also happy to report this elsewhere if that’s better.

mkvlr08:08:13

Another issue we’ve seen is this stackoverflow error. This occured only once and due to recursion we don’t have the full stacktrace but there’s datomic.query/fn in the stacktrace. We’re thinking to increase the number of frames printed in our bug tracker. Any other advise on how to track this down? Thanks!

jaret13:08:33

@mkvlr we do not currently have plans to specifically support valcache being accessed by two instances. I theorized that it should work based on seeing multiple separate services share valcache, but it appears to affect indexing with that EOFException. Re: your other error. I’d be happy to look at your Datomic logs to see the error in query. If you’d like to open a case with support (<mailto:[email protected]|[email protected]>) we can use that to share files and we won’t lose our communication to slack archiving. In general, I think it would be useful to look at the entire datomic log for both errors.

mkvlr14:08:16

@jaret thanks. Talking to my colleagues we believe the EOFException did occur before we were running with two nodes. I guess we’ll reconfigure our nodes to use different valcaches and let you know if it does happen again. And will get in touch with support for the query error, thanks again!

jaret14:08:18

Oh interesting. It might be worth it to have us investigate the EOFException as well via the support portal.

jaret14:08:39

Especially if you’ve kept logs from before and after the switch to sharing valcache.