Fork me on GitHub
#datomic
<
2018-07-09
>
Oliver George00:07:26

I'm thinking through the right place to do schema migrations. The key issue being that a deployment is unstable until schema migrations are run. I've seen examples of doing the schema migration as part of a memoized get-connection helper. Are there other approaches I'm missing? Perhaps there should be a hook to run a schema migrations as part of the code deploy deploy step. Ref: https://github.com/Datomic/ion-starter/blob/master/src/datomic/ion/starter.clj#L69

eoliphant00:07:14

yeah some sort of :init-method in the config would be nice for that kind of stuff. But i’m just doing the memoization for now was well

Oliver George00:07:07

@eoliphant the downside I see regarding memoization is the risk that the schema fails at "run time". I'd prefer to the deploy to fail. Could argue it's unlikely given schemas should grow, not break, but human error is a thing. e.g. added a unique constraint but values in db aren't unique.

miridius01:07:31

You could write your own deploy code that transacts the schema before running the ions deploy op, perhaps

Oliver George01:07:25

Hi @U0DHVSBHA, yes that would work. For a commit based deploy there is little chance of the local environment (running the migration) being inconsistent. That's the risk I see. Deployment becomes coupled to local dev env and the code pushed. More moving parts. More complexity.

eoliphant00:07:09

yeah I totally agree, but I think that’s going to have to be something they build in. If say the :init-function returns falsy, etc yeah then kill the deploy. Though even that gets interesting, since you could possibly say successfully transact in some schema, but then have something else in the code fail, such that it returns false. So you’d roll back the deployment, but still have made changes.

Oliver George00:07:02

True enough but happily we're talking about an unlikely case given "grow don't break" schema approach. Does require some coding practices reduce the risk of surprises. Ref: https://docs.datomic.com/cloud/best.html#plan-for-accretion

eoliphant00:07:02

yep, that’s what I do with all my datomic stuff. And similarly one would just have to be disciplined about not doing anything too hinky in this proposed init-method.

Oliver George00:07:32

Crazy idea would be a schema migration transaction function which runs a suite of tests before committing (if that's even possible).

eoliphant00:07:01

ok i’m pulling my hair out right now.. My ionized gw function is base64 encoding my responses lol

orlandow00:07:55

That happened to me too, maybe this helps:

eoliphant00:07:14

ah that would be interesting

eoliphant00:07:05

i use conformity for my on-prem stuff. it’s like flyway/liquibase lite. But provides a little structure around the process

Oliver George00:07:59

Thanks, I'll check it out.

eoliphant01:07:37

well it doesn’t support the client api yet though 😞

eoliphant01:07:58

i was planning to fork it and see if I could add that support

cjsauer17:07:39

I took a stab at this using a small gist. Haven't discussed a PR or anything though. https://gist.github.com/cjsauer/4dc258cb812024b49fb7f18ebd1fa6b5

oscar01:07:30

What I do for migrations is test that all of the queued migrations work using d/with. If I don't throw any exceptions, then I commit the transactions.

eoliphant01:07:05

anyone had this problem? I had this function go a bit wonky on me. I’ve stripped it down to this

(defn handle-request*
  "Handle API Requests"
  [{:keys [headers body]}]
  (log/debug "here's something")
  {:body  "body here" #_(json/generate-string {:its "ok"})
   :headers {"Content-Type" "application/json"}
   :status 200}
  #_{:status 200
but in the gateway log, I’m seeing the following (and the encoded value is returned to my client)
(c562e562-8313-11e8-8b30-f3eb1fd30d3f) Endpoint response body before transformations:
{
    "body": "Ym9keSBoZXJl",
    "headers": {
        "Content-Type": "application/json"
    },
    "statusCode": 200,
    "isBase64Encoded": true
}
`

oscar01:07:50

@eoliphant Have you added */* as a Binary Media Type?

eoliphant01:07:26

i was having another issue and recreated the gateway

eoliphant01:07:34

forgot to do that, again.. thanks

eoliphant01:07:30

been pulling my hair out all day trying to get logging working via a logback appender for loggly. And just realized that most of the typical java ecosystem stuff will probably never work. Since most of it depends on classpath scanning, etc etc, so when your ions deploy all that’s already taken place.. ugh.. Maybe a good use case for modules 🙂

pradyumna08:07:38

hi, is there a preferred strategy to manage data locally in a standalone application, which ordinarily online to access several other datomic databases. When offline the app should be able to still perform with whatever information it has cached. It should be able to store locally some of the work and then try to update the remote databases as applicable. Of course, the issue of conflict needs to be addressed in a sane way. I was thinking may be have a local datomic instance to serve as a cache for multiple remote datomic instances. or is there something better and simpler.

steveb8n08:07:49

@pradyumna take a look at the AWS AppSync javascript lib. It handles all of these requirements for you, including conflict resolution. I’m helping out on a project which hopes to expose Ions using AppSync so it should fit pretty well

pradyumna08:07:52

thanks @steveb8n. i checked this. unfortunately its not exactly fitting in. its clojure (jvm, not javascript)

eoliphant12:07:11

You’d probably have to implement this yourself @pradyumna like most db’s there’s no explicit support for that use case AFAIK. You could potentially use something like onyx for moving the updates between databases, but you’d be on the hook for conflict resolution ,etc

eoliphant13:07:23

Hi, I’m still trying to get some form of logging working. In the course of this I’ve run into another issue. Given what I mentioned previously about commons/sl4j/etc stuff probably never working, I tried creating a custom logger with timbre, that just fires entries via rest in to loggly. I’m using cljs-ajax for this, and it works fine in local dev, but when I call in now, i’m getting a classnotfoundexception for org.apache.http.HttpResponse, so there are presumably some classloader conflicts there. I noticed that the ion-event-example uses some cognitect http-client lib, but I can’t seem to find it in any of the repos

stuarthalloway14:07:53

hi @eoliphant I would stand down, the next release will solve this.

👍 4
eoliphant14:07:07

ah awesome 🙂

eoliphant14:07:18

ions are pretty friggin cool. This was the only real nitnoid so far

stuarthalloway14:07:01

that's great to hear, thanks!

oscar17:07:50

When I instantiate a Client, the following is printed:

Reflection warning, cognitect/hmac_authn.clj:80:12 - call to static method encodeHex on org.apache.commons.codec.binary.Hex can't be resolved (argument types: unknown, java.lang.Boolean).
Reflection warning, cognitect/hmac_authn.clj:80:3 - call to java.lang.String ctor can't be resolved.
It returns the Client, though, and it seems like there aren't any issues. I just want to know if this will be a problem.

stuarthalloway19:07:57

hi @U0LSQU69Z what version of Clojure and Java are you running?

oscar19:07:27

openjdk version "1.8.0_172" clojure "1.9.0"

stuarthalloway19:07:58

that won't harm anything, but I will squelch it in a future build

oscar19:07:58

Cool. Thanks!

timgilbert20:07:35

Hi everybody, I could have sworn I saw a project on here that you could point to a datomic database and get a GraphViz diagram of the schema, but now I can't seem to find it. Anyone remember it?

eoliphant20:07:02

Hey @stuarthalloway, I think there may be an issue ionized lambdas handling of OPTIONS requests. For this ‘echo’ ion

(defn api-request*
  "lambda entry point"
  [{:keys [headers body request-method]}]
  (try
    {:status 200
     :body   (json/generate-string request-method)}
....
I get “post” “get”, etc just fine but a "message": "Internal server error" for an OPTIONS request, need that since API gateway expects the lambda to respond to the CORS preflight stuff

oscar21:07:53

Do you have an OPTIONS method next to your ANY method in API Gateway?

oscar21:07:47

Something similar was happening to me because I hit "Enable API Gateway CORS". If you did the same, delete the OPTIONS method and handle it in your Ion. This is happening because AWS matches your OPTIONS request content-type to */* and base64 encodes it. The "Mock Integration" that the preconfigured CORS handler generates expects JSON and throws when it can't parse.

eoliphant21:07:01

yeah that’s exactly what I did

eoliphant21:07:12

that did it. thanks!

oscar21:07:21

No problem!