Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2018-09-05

Channels

#datomic
<
2018-09-05
>
steveb8n09:09:00

I’m seeing this Ion-specific error :db.error/invalid-rule-fn The following forms do not name predicates or fns: (clojure.string/starts-with?) when using rules in a query. the same clause works fine when not in a rule. This query runs fine when invoked from laptop to cloud but fails with the above when deployed as an Ion. It previously worked fine as a non-rule clause in an Ion. Any suggestions?

stuarthalloway13:09:09

@steveb8n that seems not-right 🙂. We will try to repro.

steveb8n13:09:24

thanks. let me know if I can provide more info

stuarthalloway13:09:29

that was a good problem description already!

jaret12:09:55

@steveb8n I am working on reproducing this error for the team to review. Would you be able to put your rule and query in a gist? I was able to invoke a rule without running into an error and I want to look at the specifics of your case and incorporate them into my testing.

jdkealy14:09:29

I'm having scaling issues that are coming to a head. I believe it's Datomic related. I have queries that are intermittently taking a long time to execute. Like 3-5 seconds for a query that used to take a few milliseconds. Any advice on where to look and try to figure this out ?

stuarthalloway14:09:41

can you reliably reproduce?

jdkealy14:09:51

i've taken some endpoints away from datomic and used elasticsearch and that sees to have relieved some of the pressure

jdkealy14:09:35

i can reproduce a few slow queries, but can't reproduce the ones that are supposed to be fast but are acting slow

jaret14:09:51

Hey @U1DBQAAMB this is getting a little hard to track. We’d like to ask that you log a support case to <mailto:[email protected]|[email protected]>. With all of the points you’ve run into so far (memcached, query performance etc). I also reached out so we can setup a call, but it will be great to have all of this information in a case we can track and update.

jdkealy14:09:48

ok thanks Jaret, i'll summarize in an email, much appreciated

jdkealy15:09:40

I sent an email. The gist is datomic queries are taking too long, my site keeps going down. My clients are ok with adding more servers and such especially for this week (this is their highest volume week of the year). I'm replacing datomic endpoints with elasticsearch endpoints, it's relieving some of the pressure, but some dead simple endpoints that sometimes take 5MS to load are intermittently taking over 5seconds, sometimes as long as a minute.

jdkealy15:09:19

When they hit a minute, then containers get taken offf the load balancer, and i start having lots of issues. I've already added another server. I'm using Elastic Container services. There are 3 containers per server and 4 servers in total.

jdkealy15:09:00

I'm seeing that there are only 4 peers connected (i guess peers work by IP? which would maybe explain why i'm not getting a performance bump by adding more containers ? )

jdkealy15:09:48

and just had one 314922 MS query

jdkealy14:09:35

i do have one slow endpoint i can reproduce, but that's because of a slow query...

lilactown16:09:32

how do I figure out why an ion deploy failed?

lilactown18:09:52

I have no alerts to speak of

lilactown16:09:24

it seems like my ion deploys fail randomly, and changing an inconsequential thing and push/deploying again fixes it. Trying to figure out why

lilactown16:09:47

okay, I just tried running a deploy from the codedeploy dashboard, and I’m seeing that the deploy-validate script is failing

lilactown16:09:15

the rollback succeeds

lilactown16:09:52

the log tail for the deploy-validate is just 

[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
...

lilactown17:09:40

and now this time, pushing a new revision and deploying again did not fix it

steveb8n18:09:13

You might be experiencing out of memory errors like me and others. Causes intermittent deploy fails on solo. You would see this in the logs as a stackoverflow

steveb8n18:09:51

If so, there is a work around by editing the CF template but you should verify it first

lilactown18:09:54

@steveb8n do you know offhand what exactly I should be searching for in my cloudwatch logs?

lilactown18:09:45

offhand I don’t see any exceptions, errors or alerts related to stack overflows

steveb8n18:09:36

You should be able to see stackoverflow in the logs for http invocations. If not, then maybe that's not it

lilactown18:09:30

> in the logs for http invocations sorry, I’m very new to both AWS and datomic. I’m not sure what you mean by “http invocations.” All I’ve been doing is running the deploy commands, and they fail. The rollback succeeds and my app works as expected on the last successfully deployed revision I am on the solo topology, so this is the best lead I have so far.

steveb8n18:09:06

Ah that's a different behaviour to what we saw if I'm recalling correctly

steveb8n18:09:19

But good to rule it out since it's another intermittent one

lilactown18:09:19

okay, I am seeing stack overflows

lilactown18:09:26

:datomic.cluster-node/-main failed: java.lang.StackOverflowError, compiling:(riddley/compiler.clj:12:3)

lilactown18:09:25

searching for StackOverflow (capitalization!)

lilactown18:09:06

@steveb8n you said there was an underlying cause & fix?

steveb8n18:09:39

Yep, the causes is lack of memory so you can ignore the line number

steveb8n18:09:13

In my case I fixed by overriding the libs listed in the push warning

steveb8n18:09:19

Another team found a more reliable solution by editing the cloud formation template and increasing JVM mem params

steveb8n18:09:02

But to do that, you have to upgrade the initial setup so you have access to the CF templates

steveb8n18:09:35

I haven't done this but the instructions seem pretty straight forward

lilactown18:09:58

:thinking_face: I’m trying to understand why overriding libs would fix it. do you mean forcing datomic to load your specified versions?

lilactown18:09:18

or did you add the versions specified by the push warning to your deps.edn?

steveb8n19:09:14

Yep, that's exactly what I did. I still occasionally see stackoverflows but then I just redeploy

steveb8n19:09:30

The mem fix will be more reliable

lilactown19:09:59

which one?? 😂

steveb8n19:09:03

I don't understand why the deps versions would affect memory either

steveb8n19:09:53

The JVM fix was discussed by Stu about a week ago so you might find it by scrolling back slack retains enough history. Otherwise ping Stu for specifics

lilactown19:09:21

@steveb8n I’m going to assume you meant you overrode them locally. I asked two questions 😛

steveb8n19:09:02

Yes overrode locally in deps.edn

lilactown19:09:47

thanks I’m trying it now

lilactown19:09:25

doesn’t seem to have affected anything

lilactown19:09:03

i still can’t deploy

stuarthalloway19:09:59

@U4YGF4NGM I answered on the ticket. You need to increase your stack size from -Xss256k to -Xss512k

lilactown23:09:30

that fixed it. I also had some code errors, not sure if the two were related at all. but at least I was able to see logs of my errors now

👍 4
lilactown23:09:27

I’m trying to access query params in my api-gateway ion. I see that there is a :query-string key passed in via the input map, although it’s not documented in the reference

lilactown23:09:12

I also see there’s a :datomic.ion.edn.api-gateway/data entry with a map containing :queryStringParameters and key-value pairs of the params, which is ultimately what I’d like

lilactown23:09:20

I’m guessing I should just use :query-string for now? the :datomic.../data key looks not for public use