Clojurians Log v2

For those running Datomic in production, what are your operational approaches for performing schema updates and data migrations? Let's say I've got a form:

<form>
  <input name="birthday">
</form>

{:db/ident       :person/birthday
 :db/valueType   :db.type/inst}

<form>
  <input name="birthday">
  <input name="date-format-preference">
</form>

{:db/ident       :person/date-format-preference
 :db/valueType   :db.type/keyword} ;; [:yyyy-mm-dd :mm-dd-yyyy]

(let [ents-w-birthdays (d/query {:query '{:find [?e ?c]
                                          :in [$]
                                          :where [[?e :person/birthday _]
                                                  [?e :person/continent ?c]]}
                                 :args [(d/db conn)]})
      migrated-entities (->> ents-w-birthdays
                             vec
                             (map (fn [entity continent]
                                    (if (= continent "North America")
                                      (merge entity {:person/date-format-preference :mm-dd-yyyy})
                                      (merge entity {:person/date-format-preference :yyyy-mm-dd})))))]

  migrated-entities)

We use an approach very similar to what you have described leveraging https://github.com/recbus/caribou to track and apply migrations to keep things "up-to-date". Caribou has the concept of a data "correction" as a migration (using Clojure's iteration behind the scenes to manage long-running corrections). The drawbacks we see are exactly the ones you have noted. But, in three and a half years we have had only one failure to start -and that was indeed running a data correction where the developer was new to the correction process. The benefits of this approach are that application code does not need to live in two worlds at once -which can be very tricky to manage.

The general idea of tracking all migrations in caribou and leaving it to ensure the schema and data (seeds, corrections) are "current" is pretty liberating.

What we usually do is running all the migrations first before deploying the new code. The approach is that we only add new attributes to the schema, so, these new attributes only are going to be used by the new code until it is deployed. The old code doesn't know/care about these new attributes and hence nothing breaks. We don't keep track of which migrations were run, we execute all of them every time we deploy. For old entities, with the new code: - When reading we use default values or calculate them if the attribute is not present. If this approach is not possible then we would need to run a script to set the correct value for each one. - When updating or inserting we assign the corresponding value.

On application startup, before opening my web server up to traffic, I re-transact my schema into the DB so the new attribute is present in Datomic. Doing this on startup so I don't get into any funky situations where my app code has a new form field for creating a new attribute, but my DB schema doesn't know about it yetAre you using conformity for this too? You should. If you are merely retransacting static schema in code you could indeed get into "funky states", e.g. an old app version deploys and changes the schema to an older revision, or you delete :db/index true from your code and expect the index to go away on next deploy. (It won't)

I recommend treating all schema changes as if it were a separate library that your application code depends on, and not mixing any of that into your application lifecycle

So: use migration frameworks for schema and data migrations (as you are), apply those separately (eg in CI or a one-off job), and only then turn on the application code that depends on it (whether via deployment, feature flag, or whatever), If you are adding an index, include a final blocking d/sync-index to make sure the index is in place before you deploy. (Conformity may do this already, I don't remember.)

a useful thing to have to reinforce this is an "at least this high" marker in your database (conformity already keeps a queryable set of applied norms). An application can check this at startup, and refuse to start if it doesn't find what it needs. You should strive to make backward-compatible changes (so old code can use newer database states), but if you find yourself unable to do this, you can also use a marker like this to make the application refuse to start if it encounters a value it doesn't understand.