Fork me on GitHub
#datomic
<
2018-07-31
>
henrik09:07:47

What’s the proper way of setting environment variables for Ions?

steveb8n09:07:31

@henrik we had this discussion late last week. Stu implied that this question will be answered soon so, for now, we wait

henrik09:07:24

Excellent!

henrik09:07:49

In the meantime I’ll proceed with a dirty and/or unsecure hack.

steveb8n10:07:52

same here. just go in an manually add the env vars to the lambdas

eoliphant18:07:08

I add a ENV=XXX tag to the config, then use that to ‘root’ grabbing stuff out of the AWS param store. hacky but works, looking forward to thenative solution

👍 4
joshkh10:07:42

has anyone experienced this when creating a datomic cloud client?

(d/client (get-in env [:datomic :client]))
CompilerException java.lang.IllegalArgumentException: Can't define method not in interfaces: recent_db, compiling:(datomic/client/impl/shared.clj:304:1) 

joshkh10:07:44

i don't get the error with 0.8.46, but i do when i upgrade to 0.8.54+

henrik13:07:50

This is a persistent problem for me:

java.lang.Exception: namespace 'cheshire.factory' not found, compiling:(cheshire/core.clj:1:1)
Cheshire works fine locally, but pushed to AWS (Datomic Ions), it generates an error. Why might this be?

Joe Lane13:07:41

@henrik jackson conflict possibly, look at the deps tree

Joe Lane13:07:43

clojure -Stree

Joe Lane13:07:12

Compare that list with any deps conflicts that datomic ions barks about when you push your code up.

Joe Lane13:07:44

I’ve found that I dont actually need cheshire, and that clojure.data.json works well enough, however I understand thats not a great answer.

henrik15:07:59

@joe.lane I don’t quite get it, these are the deps listed for cheshire:

cheshire/cheshire 5.8.0
  com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.9.0
  com.fasterxml.jackson.core/jackson-core 2.9.0
  tigris/tigris 0.1.1
  com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.9.0
They only appear for cheshire, nowhere else, including jackson.

Joe Lane15:07:10

I dont see com.datomic/client-cloud included

henrik15:07:42

Whops! I might have moved that into dev deps.

Joe Lane15:07:21

Well thats a bit of a problem haha

😄 4
Joe Lane15:07:22

Also, for the dev deps, what version of com.datomic/ion-dev do you have?

Joe Lane15:07:19

Update to "0.9.175", it includes as output (when you push) a collection of dependency conflicts.

henrik15:07:05

Where can I see what the most recent versions of the libs are?

Joe Lane15:07:09

Which I believe your issue is rooted in.

👍 8
henrik15:07:55

Right, it’s telling me explicitly about the conflicts now, that’s nice.

henrik15:07:52

It’s kind of weird that Cognitect is using outdated versions of their own libraries

jaret15:07:22

@henrik we should be on 0.9.175 throughout docs and repos. May I ask where you got 0.9.160 from? so I can make sure we update that. https://docs.datomic.com/cloud/releases.html

henrik15:07:14

@jaret Oh sorry, I meant the deps. It’s telling me that in Datomic Cloud, com.cognitect/transit-clj #:mvn{:version "0.8.285" is used, while I’m relying on com.cognitect/transit-clj {:mvn/version "0.8.309"}. Or can I control which version Datomic Cloud uses?

Joe Lane16:07:12

Yeah, I ran into some issues with things like the priority-map library and the aws java sdk.

henrik16:07:38

Well, good news is I got a different error now. java.net.ConnectException: Connection refused

jaret16:07:54

Such great news 😉. Are you sourcing your AWS creds?

henrik16:07:54

@jaret Thank you! 🙂 Possibly, I have no idea what that means. What would I be doing in this case?

jaret16:07:28

Let me back up. When are you getting this error?

henrik16:07:41

API Gateway is reporting this as I’m trying to access the web server endpoint. It was working fine as of 0.9.160 and no cheshire. The house of cards started collapsing with the introduction of cheshire, alas.

jaret16:07:28

And you pushed/re-deployed after removing cheshire?

henrik16:07:21

Not yet, I’m trying that just now.

jaret16:07:18

@joe.lane btw I wanted to ask if you had a link to your ion tutorial/talk.

jaret16:07:42

I was going to add it to the documentation as a community example, but was only able to find your talk notes on your github.

Joe Lane16:07:18

I’ve got several (unpublished…) small projects with ions if you’re in going to start adding community examples. Do you mean a blogpost or video? Because the source code should be present.

jaret16:07:26

I know we’d love to see more community examples in whatever format you have them in. Github links work as well, but I know you mentioned on twitter after your websocket IOT Ion talk that a tutorial/talk would follow.

jaret16:07:53

I don’t want to put any pressure on you 🙂 Just want to make people can find the great stuff everyone is making.

Joe Lane16:07:10

haha yeah, thats on my todo list. Need a platform to host it first.

Joe Lane16:07:23

(read: yak shave)

jaret16:07:36

Understood. ✂️ 🙂

jaret16:07:49

Need a yak shave emoji

henrik16:07:45

No dice, I’m afraid. I’ll downgrade to 0.9.160, see if that gets it up and running again.

Joe Lane16:07:27

What is giving the connection refused error?

henrik16:07:09

I’m not sure. This is as reported by the “test” tool in API Gateway.

henrik16:07:29

Though the List of Reprimands is smaller:

:dependency-conflicts
{:deps
  {commons-codec/commons-codec #:mvn{:version "1.10"},
   org.slf4j/jcl-over-slf4j #:mvn{:version "1.7.14"},
   com.fasterxml.jackson.core/jackson-core #:mvn{:version "2.9.5"},
   com.cognitect/http-client #:mvn{:version "0.1.80"},
   com.google.guava/guava #:mvn{:version "18.0"},
   com.cognitect/s3-creds #:mvn{:version "0.1.18"},
   org.slf4j/slf4j-api #:mvn{:version "1.7.14"},
   com.amazonaws/aws-java-sdk-kms #:mvn{:version "1.11.349"},
   com.amazonaws/aws-java-sdk-s3 #:mvn{:version "1.11.349"}},}

rhansen16:07:39

I can't get ions to work 😕 I'm pretty certain the problem is related to my deps, but I'm unable to re-create the StackOverflow error localy after applying the deps clojure.ions warns about. 😕 (details in thread)

rhansen16:07:44

My deps.edn:

{:paths ["backend" "resources"]
 :deps
 {org.clojure/clojure {:mvn/version "1.9.0"}
  com.datomic/ion {:mvn/version "0.9.16"}
  ring/ring-core {:mvn/version "1.7.0-RC1"}
  ring/ring-json {:mvn/version "0.4.0"}
  email-validator {:mvn/version "0.1"}
  clj-mailgun {:mvn/version "0.2.0"}
  crypto-password {:mvn/version "0.2.0"}
  danlentz/clj-uuid {:mvn/version "0.1.7"}
  clojure.java-time {:mvn/version "0.3.2"}}

 :mvn/repos {"datomic-cloud" {:url ""}}

 :aliases
 {:dev {:extra-deps {com.datomic/client-cloud {:mvn/version "0.8.54"}
                     com.datomic/ion-dev {:mvn/version "0.9.175"}
                     ring/ring-jetty-adapter {:mvn/version "1.7.0-RC1"}
                     org.eclipse.jetty/jetty-server {:mvn/version "9.4.9.v20180320"}
                     org.eclipse.jetty/jetty-client {:mvn/version "9.4.9.v20180320"}}}
  :test {:extra-paths ["tests"]
         :extra-deps {expectations {:mvn/version "2.2.0-rc3"}}}}}

rhansen16:07:44

the error message:

"Msg": "IonLambdaDispatcherFailedtoStart",
    "Ex": {
        "Cause": null,
        "Via": [
            {
                "Type": "clojure.lang.Compiler$CompilerException",
                "Message": "java.lang.StackOverflowError, compiling:(potemkin/namespaces.clj:88:34)",
                "At": [
                    "clojure.lang.Compiler",
                    "analyzeSeq",
                    "Compiler.java",
                    7010
                ]
            },
            {
                "Type": "java.lang.StackOverflowError",
                "Message": null,
                "At": [
                    "clojure.lang.RestFn",
                    "applyTo",
                    "RestFn.java",
                    130
                ]
            }
        ],

Joe Lane16:07:24

Look in clojure -Stree for a dependency that requires potemkin and then remove it. See if the issue persists. That lib does some crazy stuff and I wouldn’t be surprised if its causing issues.

👍 4
henrik16:07:59

I see clj-http depends on potemkin 😕

henrik16:07:59

@joe.lane Do you track ion-starter to keep up to date with the releases? Or is there a release page somewhere?

henrik16:07:13

Oh man, what’s going on?

java.lang.Exception: namespace 'clj-http.headers' not found, compiling:(clj_http/core.clj:1:1)

jaret16:07:43

@henrik can you try the base tutorial? That looks like you may have some basic auth in your app?

jaret16:07:17

Or are you getting that error with the base tutorial and ion-starter git repo?

henrik16:07:58

@jaret No, this is after moving some stuff over from another project. It includes things that talk to Google Cloud, hence cheshire and clj-http.

henrik16:07:28

Are some libraries unsupported in Ions? I’m not sure I understand the patterns behind what works and not.

henrik17:07:11

I ripped out the deps I added from my other project, and now it works fine. Some or all of these are the culprit(s):

buddy/buddy-sign {:mvn/version "3.0.0"}
environ {:mvn/version "1.1.0"}
org.clojure/data.json {:mvn/version "0.2.6"}
clj-http {:mvn/version "3.9.0"}
clj-time {:mvn/version "0.14.4"}

kenny17:07:59

How do you guys approach testing with Datomic Cloud? I see Stu mentioned here (https://forum.datomic.com/t/integration-testing/465) that he has some "tricks" to unit test with Datomic, but I have not seen any posts about that. Our approach right now is to create a new db with a unique suffix on each test run, deleting the db after each test completes. This ensures a clean DB for each test. Is there a better approach to this? With the peer library you could simply create a new in-memory connection for each test, allowing you to use your normal DB names without a suffix. It'd be great if there was something similar for Datomic Cloud. Is there a more reasonable solution to this problem?

👍 8
eoliphant18:07:15

we do pretty much the same thing. Moved a clj/datomic microservice into ion code, moved to client api, then just updated my test fixture that would dynamically create and tear down an in-mem db to do what you described. Pretty much just worked

steveb8n22:07:29

it’s not blessed by Cognitect (yet) but works well for me. I can reproduce all cloud behaviours in local/mem db (at least those that I need)

kenny22:07:48

I just wrote an (almost) exact version of that haha

ghadi17:07:17

Favorite datomic-ish ER tool besides Omnigraffle?

henrik18:07:19

So, buddy-sign, unfortunately has this dep tree:

buddy/buddy-sign 3.0.0
  buddy/buddy-core 1.5.0
    org.bouncycastle/bcprov-jdk15on 1.59
    commons-codec/commons-codec 1.11
    org.bouncycastle/bcpkix-jdk15on 1.59
    cheshire/cheshire 5.8.0
      com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.9.0
      com.fasterxml.jackson.core/jackson-core 2.9.0
      tigris/tigris 0.1.1
      com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.9.0
    net.i2p.crypto/eddsa 0.3.0
Meaning it contains cheshire, which is Forbidden.
java.lang.Exception: namespace 'cheshire.factory' not found, compiling:(cheshire/core.clj:1:1)

henrik18:07:08

Is it just a fact of life that anything referencing cheshire is going to blow up when it reaches the Cloud?

chrisblom18:07:19

can you exclude it?

henrik18:07:44

I do need to sign jwt tokens. Maybe there’s another lib I can use.

eoliphant18:07:08

i’m using cheshire as a top level dep

eoliphant18:07:40

no probs so far

henrik18:07:22

But how! Would you mind sharing your deps.edn?

eoliphant02:08:51

sorry just saw. did the upgrade to 402 fix your problem?

henrik04:08:20

Yes, it did, thank you!

Joe Lane18:07:16

The issue is due to that particular version of cheshire having a jackson dep conflict with the version of jackson-core used in datomic-cloud.

jaret18:07:50

Are you on that CFT?

jaret18:07:23

We updated the jackson libs in 402-8396.

jaret19:07:46

You’ll want to upgrade your Datomic Cloud System to that and I think you should be fine to include Cheshire. https://docs.datomic.com/cloud/operation/upgrading.html

henrik19:07:59

Ah, no, 397. That might be it!

johnj19:07:38

@jaret observation: datomic-free hasn't been updated to the latest version in clojars https://clojars.org/com.datomic/datomic-free