Fork me on GitHub
#datomic
<
2018-12-11
>
joshkh12:12:27

wondering if someone from cognitect is around to help? yesterday we upgraded from solo to production after our ec2 instance tipped over. the upgrade went well and we can connect to our cloud instance, but we can't deploy our existing ion functions.

Alex Miller (Clojure team)12:12:48

I’m from Cognitect, but probably not qualified to help. But if I were, I would ask what “can’t deploy” means

joshkh13:12:19

whoops. i solved the problem shortly after asking. that's how it works right? 😉 the upgrade path to production failed so we had to delete the existing compute stack and install as-new (using the previous app-name). our bastion cloud connections came back, however the already-deployed ions were throwing an internal server error, and re-deploying them resulted in {:deploy-status "FAILED", :code-deploy-status "FAILED"}. i tested the lambdas via the AWS console which worked as expected, and a third code push finally succeeded. i still had to edit our API gateway, reselect the proxy resource and authorizer functions, then deploy the API.

joshkh13:12:53

just a wild guess, but i'm assuming that the new stack with the old app name crossed some wires. problem solved.

ro614:12:13

Is there a common practice for purposefully triggering rollback of the CodeDeploy from within the app? Eg if a Datomic schema migration fails in production or another startup condition isn't met. What condition is CodeDeploy polling to determine that "the service is up"?

marshall15:12:10

@robert.mather.rmm The Datomic process not starting is the most common cause of rollback usually caused by a bug or deps conflict in an ion that throws when the ns is loaded

ro615:12:18

@marshall Can my app explicitly signal an issue and cause CodeDeploy to roll back?

Joe Lane15:12:44

Throw an exception when trying to load a namespce.

Joe Lane15:12:45

@robert.mather.rmm ^^ Do you think this covers your usecase?

ro615:12:07

Probably. I'll have to try it out. I generally set an uncaught exception handler (ala https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions), but maybe I can do that last in my startup process.

stijn16:12:54

@robert.mather.rmm so, you are loading a bunch of stuff at compile time instead of at first request time?

ro617:12:14

Initially yes. The Code Deploy was timing out and rolling back because it only gave 2 minutes for startup. I switched to first request and doing everything lazy, I was hoping to switch back. Is the time to establish the Datomic connection proportional to data size or something?

ro617:12:30

To me it's quite desirable to be able to do schema transaction/migration and check everything worked before exposing the new instances to the world.

stijn15:12:44

yes, I think it would benefit our use case too, just checking if it is possible. i'll try it out

stijn16:12:24

how does that work out with the 'database loading'?

val_waeselynck17:12:22

Failing to deploy on Ion because of a mysterious error thrown when calling d/client: "Assert failed: cfg". Does anyone know what this could mean? The error is thrown by:

(d/client {:server-type :ion
                 :region "eu-central-1"
                 :system "linnaeus"
                 :query-group "linnaeus"
                 :endpoint ""})

val_waeselynck17:12:48

Here's what the error looks like in Cloudwatch (reported via cast/alert):

val_waeselynck17:12:21

Running on com.datomic/ion {:mvn/version "0.9.26"} and org.clojure/clojure {:mvn/version "1.9.0"} on a freshly-updated stack.

ro617:12:14

Initially yes. The Code Deploy was timing out and rolling back because it only gave 2 minutes for startup. I switched to first request and doing everything lazy, I was hoping to switch back. Is the time to establish the Datomic connection proportional to data size or something?

ro617:12:30

To me it's quite desirable to be able to do schema transaction/migration and check everything worked before exposing the new instances to the world.

marshall17:12:06

@val_waeselynck Can you go to the latest ion? (0.9.28) and also, what version of ion-dev are you using?

val_waeselynck17:12:53

@marshall running on com.datomic/ion-dev {:mvn/version "0.9.176"}

val_waeselynck18:12:03

Let me try updating ion

marshall18:12:17

ion-dev also

marshall18:12:29

is now com.datomic/ion-dev “0.9.186”

marshall18:12:44

working on fixing the duplicate rows in that table now

marshall18:12:07

also, what version of client?

val_waeselynck18:12:50

com.datomic/client-cloud {:mvn/version "0.8.71"}, but that's a :dev dep

marshall18:12:13

yeah, that’s fine; should be overridden on the instance anyway

val_waeselynck18:12:53

deploying, stay tuned...

marshall18:12:16

you re-pushed yes?

val_waeselynck18:12:57

Nope, still same error 😕

marshall18:12:59

This happens on deploy or invoke>

val_waeselynck18:12:39

Note that I'm making this call at ns-load time

val_waeselynck18:12:52

Don't know if that's OK

marshall18:12:12

the call to client?

marshall18:12:25

like in a def ?

val_waeselynck18:12:26

as in (def client (d/client ...))

marshall18:12:48

well, i wouldnt expect it to be an issue; could you put it in a memoized fn instead?

marshall18:12:48

if that fixes the issue i can explore the reasons further and then either fix or at least document

val_waeselynck18:12:54

I could, but it could take some time for me to be able to reproduce the issue then. I'll try that tomorrow. Is there a particular rationale for this memoization anyway?

val_waeselynck18:12:05

OK you just answered

marshall18:12:24

I’ll also try doing a similar thing in a test without the memoized client call

marshall18:12:37

to see if i can reproduce the behavior you see

val_waeselynck18:12:40

I think (and this is totally not rigorous reporting) that it used to work on a previous deploy - I'm suspecting our stack is in a pathological state, will re-create it tomorrow

marshall18:12:13

ok. let me know if you’d like some help investigating at all

val_waeselynck18:12:56

By the time our timezones meet, I will either have succeeded or be in need for more help 🙂

marshall18:12:07

fair enough

marshall18:12:25

Val, what version of Cloud are you running?

marshall18:12:33

This should be something that is fixed in recent releases

marshall18:12:32

@val_waeselynck I’ve confirmed that a change in the latest version loads user namespaces earlier in the process to reduce instance cycle time; one consequence is that you can’t connect to a database as a side effect of ns loading I would recommend following the pattern shown in the example I provided (delay connection until first invoke). I will also look into making this more evident in documentation

val_waeselynck06:12:10

@marshall thanks, initializing the connection lazily did work. I suggest adding a comment with that link in the tutorial's repo as well.

timeyyy19:12:15

the link for datomic-cloud repository is broken -> https://docs.datomic.com/cloud/releases.html

marshall19:12:39

@timeyyy_da_man thanks, i’ll fix it

timeyyy21:12:24

Thanks marshall.

timeyyy21:12:17

Just a suggestion, the free tier of amazon is for t2.micro, when creating the solo topology`t2.micro` is not in the dropdown, i had to go figure out how to update the CF stack. Would be nice to have it accessible from the dropdown.

marshall22:12:48

you can’t run solo nodes on t2.micro instances

timeyyy22:12:25

hmm, you are correct, the template doesn't create successfully.