Fork me on GitHub
#untangled
<
2016-07-12
>
cjmurphy06:07:34

I am trying to run todomvc against a :dev datomic database rather than how it ships (:free or :mem), but getting this error:

user=> 16-07-12 06:34:36 chris-XPS-L521X ERROR [untangled.server.impl.components.handler:72] - Parser error:
 {:status 500,
 :body
 {:type "class java.lang.Exception",
  :message
  "processing rule: (q__26671 ?e), message: Unable to find data source: $__in__2 in: ($ $__in__2)"}}

cjmurphy06:07:58

The transactor is running, and "datomic:" is under /usr/local/etc/todomvc.edn's :url key.

ethangracer16:07:36

@tony.kay @currentoor @therabidbanana pushed the change we were talking about yesterday to untangled-datomic. see https://github.com/untangled-web/untangled-datomic/commit/d6aac56485e09559a556e15b2c57964801bfe297. basically a wrapper for a transaction vector where you can specify a function to call after the schema migration completes successfully. that function takes the db connection so you can migrate the data

ethangracer16:07:09

didn’t want to put the connection directly in the transaction function since it would be bad to call d/transact during the migration, which itself would be calling d/transact...

therabidbanana16:07:23

Should we expect these functions to be executed multiple times? Seems like it's possible they could be?

currentoor16:07:24

@therabidbanana: they have side effects right? So they shouldn't be called multiple times right?

ethangracer16:07:53

no, in production the function should be called exactly once

ethangracer16:07:56

when the migration is performed

ethangracer16:07:17

if you’re in a dev environment on an in-memory database it’ll be called every time, of course

ethangracer16:07:40

but the idea is when a migration happens you might want to port data just that one time, to match the new migration

ethangracer16:07:44

e.g. renaming an attribute on all existing entities

currentoor16:07:03

@ethangracer: if I have only data to migrate, so no schema changes will an empty transaction vector work?

currentoor16:07:18

seems like it should but just making sure

ethangracer16:07:27

wouldn’t that just be a standard transaction?

therabidbanana16:07:41

Well, I guess my point is that I was trying to read through this code before and I couldn't work out how they're prevented from running multiple times, other than conformity just being idempotent.

therabidbanana16:07:02

We don't use in-memory in dev mode much and I still see the logs for every single migration conforming

therabidbanana16:07:08

Every time I boot

therabidbanana17:07:15

Maybe that's just a misconfiguration on our side?

ethangracer17:07:19

interesting. that would be a question for @adambros or @tony.kay — I think the system keeps track of which migrations have applied, but I’m not sure of the internals

ethangracer17:07:33

it may log even though it does nothing

ethangracer17:07:58

what I do know is that we have an app on production servers now and when we apply the migrations it’s not botching the data

tony.kay17:07:56

Each migration is checked. We're using conformity, and it stores the applied migrations in the db so it knows what to re-apply.

tony.kay17:07:13

I think our code does output which ones are being looked at so you can debug ordering issues and such

tony.kay17:07:30

and it always looks at all of them...conformity decides if they get applied

tony.kay17:07:40

we could have better output, I guess (e.g. ALREADY APPLIED)

therabidbanana17:07:21

I guess I don't see the part where conformity gets asked if it should run the after function for this new feature, if it is always checking all migrations, unless maybe ensure-conforms raises an error? https://github.com/untangled-web/untangled-datomic/commit/d6aac56485e09559a556e15b2c57964801bfe297#diff-e89eff854d20e40f1cd602547aafc92fR252

therabidbanana17:07:45

But if it was raising an error it'd be showing migration failures in our logs, I'd think

ethangracer17:07:58

ah, good question

ethangracer18:07:02

so, I think there needs to be a check if the database already conforms before running the migration

ethangracer18:07:27

right now it’s happening after

tony.kay18:07:54

we want the check after as a simple sanity check

tony.kay18:07:05

but checking before so we know when to run data migration is needed

ethangracer18:07:11

right, so both

ethangracer18:07:16

ok, working on a patch

tony.kay18:07:41

for that matter, you can then skip the ensure-conforms, since it probably starts with a short-circuit using conforms-to 🙂

ethangracer18:07:11

and run the transactions ourselves? ensure-conforms is currently executing the migrations

tony.kay18:07:36

right, but no need to call it if we're already calling conforms-to? and that returns false, correct?

ethangracer18:07:02

yes, no need to call ensure-conforms, but we do have to call d/transact on each tx in the migration

tony.kay18:07:37

oh right. thx

tony.kay18:07:42

I trust you've got it 🙂

ethangracer20:07:23

@therabidbanana great catch on the conformity, thanks for the code review!

ethangracer20:07:25

patch is pushed

tony.kay20:07:41

confidence? Tested well?

ethangracer20:07:55

hard to test well because it’s such an integration test with datomic

tony.kay20:07:09

oh, you guys are prob using snapshots...work with it for a few days and let me know. I'll push a release once we know it is solid

tony.kay20:07:19

you want me to push a snapshot to clojars?

ethangracer20:07:58

let me play with it for the rest of the day, and try applying a migration to QA first

ethangracer20:07:07

I do think it’s solid though

tony.kay20:07:27

oh, well, I pushed a snapshot to clojars anyway...no guarantees on snapshots 🙂

therabidbanana20:07:46

@ethangracer: glad I could help - especially since I know it won't be very long before we need this ourselves.

juno22:07:09

Hello. I’m trying out untangled-websocket and I hit a little bit of an issue. In https://github.com/untangled-web/untangled-websockets/blob/master/src/untangled/websockets/networking.cljs at line 98. The callback function utilizes the deref global-error-callback but this function is passed in at client creation. Here is an error message when server returns an exception to the client. `No protocol method IDeref.-deref defined for type function: function (status,error){ return null; }`. For the client initialization. :networking (wn/make-channel-client "/chsk" :global-error-callback (fn [status error] nil)) Thank you.

jasonjckn23:07:04

how do I run untangled-client tests after checking out the repo?

jasonjckn23:07:43

I did

npm install; lein doo chrome test
I get a window that pops up that just says "chrome ready"