Fork me on GitHub
#datomic
<
2018-10-24
>
ro603:10:56

Is anyone taking steps to recover the Java logging that Ions throws away by default (eg below WARN level)? (see: https://docs.datomic.com/cloud/ions/ions-monitoring.html#java-logging)

steveb8n04:10:50

@robert.mather.rmm not yet but I’m super interested in whatever solutions anyone is using for operations/monitoring of Ions apps.

joshkh10:10:16

quick ions question. are code deploys supposed to have zero downtime? i'm seeing the following behaviour: 1. deploy a revision, 2. test the lambda and it fails, 3. wait a little bit, try again, and it works:

; A few seconds after an Ions deploy

$ aws lambda invoke --function-name my-compute-group-testfn /dev/stdout
{
"isBase64Encoded" : false,
"statusCode" : 500,
"headers" : {},
"body" : "java.io.IOException: Premature EOS, presumed disconnect"
}{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

; And then a few more second later
$ aws lambda invoke --function-name my-compute-group-testfn /dev/stdout
{"statusCode":200,"headers":{"Content-Type":"application\/edn"},"body":"(some working result)","isBase64Encoded":true}{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

ro620:10:18

Solo topology or production?

stijn07:10:28

I'm seeing this too, both solo and production topology

stijn08:10:06

(although we have only tested with production topology without HA). I can report about full production topology later. But it seems like this only happens on the first request.

joshkh10:10:03

also (unrelated) - how might one go about applying ring middleware to functions that have been ionized with datomic.ion.lambda.api-gateway/ionize? for example, i'd like to use ring.middleware.format.

joshkh10:10:45

figured out that i have to wrap each function individually before ionizing it. 👍

grzm10:10:09

@steveb8n this is a stackoverflow error in Cider, not on a deployed machine. datomic.ions.cast/initialize-redirect sends cast/event, cast/dev and cast/alert to somewhere other than Cloudwatch for local development. (https://docs.datomic.com/cloud/ions/ions-monitoring.html#local-workflow)

steveb8n10:10:30

Ah ok, my mistake

joshkh13:10:15

i think there might be a bug in the ion/get-params fn. it's dropping the first letter of keys that are defined at the root level:

$ aws ssm put-parameter --name rootlevelparam --type String --value "somevalue"
{
    "Version": 1
}
(ion/get-params {:path "/"})
=> {"ootlevelparam" "somevalue"}

joshkh14:10:29

speaking of which, should lambdas created via ions have access to SSM by default, presumably as part of their generated role? i'm getting the following error User: arn:aws:sts::accid:assumed-role/my-compute-and-region/someid is not authorized to perform: ssm:GetParametersByPath on resource: arn:aws:ssm:my-region:accid:parameter/location/here/ (Service: AWSSimpleSystemsManagement; Status Code: 400; Error Code: AccessDeniedException; Request ID: some-uuid

amar15:10:46

Hi. Has anyone come across this error before? Seems to happen if I am transacting using a transaction function.

java.lang.RuntimeException: Reader tag must be a symbol
  File "NativeConstructorAccessorImpl.java", in sun.reflect/newInstance0
  File "NativeConstructorAccessorImpl.java", line 62, in sun.reflect/newInstance
  File "DelegatingConstructorAccessorImpl.java", line 45, in sun.reflect/newInstance
  File "Constructor.java", line 423, in java.lang.reflect/newInstance
  File "Reflector.java", line 180, in clojure.lang/invokeConstructor
  File "form-init513985284846454305.clj", line 1, in user/[fn]
  File "error.clj", line 135, in datomic.error/deserialize-exception
  File "error.clj", line 117, in datomic.error/deserialize-exception
  File "peer.clj", line 399, in datomic.peer.Connection/datomic.peer.Connection
  File "connector.clj", line 169, in datomic.connector/[fn]
  File "connector.clj", line 167, in datomic.connector/[fn]
  File "MultiFn.java", line 233, in clojure.lang/invoke
  File "connector.clj", line 194, in datomic.connector/[fn]
  File "connector.clj", line 189, in datomic.connector/[fn]
  File "connector.clj", line 187, in datomic.connector/[fn]
  File "core.clj", line 2022, in clojure.core/[fn]
    (f))
  File "AFn.java", line 18, in clojure.lang/call
  File "FutureTask.java", line 266, in java.util.concurrent/run
  File "ThreadPoolExecutor.java", line 1149, in java.util.concurrent/runWorker
  File "ThreadPoolExecutor.java", line 624, in java.util.concurrent/run
  File "Thread.java", line 748, in java.lang/run

amar21:10:52

For posterity, it seems the issue was related to using destructuring with namespaced keys. The transaction function had something like

(let [{:person/keys [age name]} data] ,,,)
which gets stored in datomic as
(let[#:person{:keys[age name]} data] ,,,)
changing to
(let [age (:person/age data) name (:person/name data)] ,,,)
was one fix. The root cause was an old version of tools.reader and/or another dependency. Upgrading dependencies independent of the code change resolves the issue.

kenny18:10:24

I see the latest version of client-cloud is 0.8.66 on Maven (https://search.maven.org/search?q=g:com.datomic%20AND%20a:client-cloud&amp;core=gav), but the Datomic releases page (https://docs.datomic.com/cloud/releases.html#current) says the current release of client-cloud is 0.8.63. Which is correct?

jaret18:10:26

@U083D6HK9 the release page is correct. I’ll confirm with the team if we need to update to .66

4
csm19:10:48

So I created my first ion to hook up to API gateway, and everything works, except my HTTP response bodies are turning into base-64. Is there an obvious thing I messed up that would cause that?

ro620:10:18

Nope. As long as you set the binary encoding option in API Gateway to */* (under Deployment in the tutorial), you should get a normal response as you'd expect from the outside (eg using Curl). I guess the Gateway does the translation for you, but not from within the testing UI. Everyone seems to run into this! (such as: https://forum.datomic.com/t/base-64-encoded-response-body-in-api-gateway-method-tester/560)

csm21:10:37

ok, I think my problem was I deployed the API before setting the binary content type. A re-deploy seemed to fix that

pvillegas1220:10:45

Is there any good example of updating a to-many relationship as one adds objects over time? Suppose I have a datom model that has a to-many ref rules. I’ll be adding rules over time, is the solution to this transact a new datom as

(d/transact conn {:tx-data [
  {:model/name "name" :model/rules (conj past-rules {:rule/name "new-rule"})
]})

favila00:10:53

The map syntax is sugar for [:db/add ...]

favila00:10:10

It’s not a “merge” or “reset”

favila00:10:50

The conj is therefore unnecessary

pvillegas1201:10:15

How would it be with :db/add?

pvillegas1201:10:38

@U09R86PA4 would it be something like [db/add model-id :model/rules rule-id]?

favila01:10:56

You can still use the map syntax just understand what it is doing

favila01:10:20

{:db/id a :many-ref [b]} expands to a single db/add, never ever any db/retract

favila01:10:56

There is no map syntax for retraction

pvillegas1201:10:50

Yeah, thanks! My conceptual problem was the ability to add a datom to a many-ref with a single datom on the many side

csm21:10:25

can I have multiple ion “projects” per compute stack, or am I limited to one? That is, only one resources/datomic/ion-config.edn? I understand I can split things up into deps, I just want to understand the deployment strategy

eoliphant00:10:46

AFAIK query groups are the only 'unit' of separation in a given system

csm22:10:09

also, I frequently get 502 errors on what looks like lambda cold starts (https://forum.datomic.com/t/api-gateway-internal-server-error/678)

eoliphant00:10:48

we've had some success by just pointing CW scheduled events at them

kenny22:10:59

@csm I have also ran into that. I don't have a solution.

luchini23:10:46

We are facing a fascinating problem with Datomic Ions. As our code base grew during the last couple of weeks, deploying to Datomic Ions started failing (CodeDeploy itself gives up and rolls back a previous version to the instance). We initially thought it could be a problem in our setup (we were still on 441) so we updated to 441-8505 but the same behavior kept plaguing us. After spending a considerable amount of time investigating, we found two ways to “solve” the issue but neither seems reasonable enough: 1) Hack Datomic’s EC2 instances and bump the stack size up of the JVM process 2) Keep the code base much simpler than we would need to 🙂

luchini23:10:58

Now that I come to think of it, it’s more like one solution 😄

luchini23:10:41

The indication that increasing the stack would work was in the exception that was thrown by starting up datomic.cluster-node was a a stack overflow one.

luchini23:10:51

I wonder the reasoning behind keeping the stack at 256K. I’m pretty sure @marshall has a superb reason for it.

marshall23:10:10

i have no such thing 😉

marshall23:10:17

We’re aware of the stack size issue

marshall23:10:40

it was configured the way it was to make solo run well on a very small instance

marshall23:10:08

we’re working on general options all around, but editing the CFT to set the stack size higher is a reasonable option for now

luchini23:10:13

We’ve got production topology and dedicated query groups… 🙂

marshall23:10:33

yep, so you’re not going to hit the same memory issues as if you bumped the stack size on a small solo node

luchini23:10:26

I’m not an expert in CF so, what’s the implication of changing it on the CFT? Do we lose this setting when we upgrade the stack in the future?

marshall11:10:49

Yes, it may be that you’ll have to re-set it on upgrade. It depends a bit on what changes in the upgrade

luchini15:10:18

Thank you!

stijn09:10:10

is it this entry that you are talking about? "JvmFlags": "-Dclojure.spec.skip-macros=true -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -XX:MaxDirectMemorySize=256m"

luchini19:10:02

I believe so… it’s missing the -Xss though

luchini23:10:43

(btw… we’ve currently changed there, but I felt… dirty 😄 )