Currently I keep all bits of the current schema in separate https://github.com/fpischedda/unrefined/blob/main/src/clj/fpsd/unrefined/persistence/datahike/schema.clj and concat them all in a vector for the https://github.com/fpischedda/unrefined/blob/main/src/clj/fpsd/unrefined/persistence/datahike.clj#L18, which works fine when starting from scratch. Now that I have to do incremental changes to an existing db this is clearly not possible. For future changes I was thinking about keeping track of the changes applied so far (both in db and code), at startup the application will verify if the db is up to date with what the code expects and, if some changes are missing then apply them. Or maybe something more manual, run with bb. Or maybe I am just complicating everything and there is easy solution to this problem 🙂

Hi @U0165ADKUDU, I am right now working on a migration tool, @konrad.kuehne almost finished it and I am testing it right now. It needs documentation but if you are up to trying things out I can throw it over the fence for you... maybe you have good ideas on what needs to be better.

Hi @timok! This is great to ear, looking forward to try it! 🙂 But at the same time yesterday evening I've started working on my own (naive) version of a migration tool for my little project; probably I will continue with it just as an exercise 🙂 The basic idea is • somewhere I maintain a list of changes I want to apply to a db; changes are written manually, no automatic tool to generate them • I feed the list to the "migrator" • it compares to what have been already applied to the db ◦ in the happy path it applies the changes and record the migration ◦ in case of errors (applied migrations history in not aligned with the app "view" of migrations), report them and stop This process can happen both at the application startup time (current implementation) or outside the main app process (like bb scripts). I am curious to see how you have approached the problem!

Hi @U0165ADKUDU, I need a day or two to open the PR but happy to talk about it then 👍

Very nice approach. If you like, I could add some comments there.

> Very nice approach. Thanks! 🙂 > If you like, I could add some comments there. Yes, please 🙏

@U0165ADKUDU here you are with our solution: https://github.com/replikativ/datahike/pull/598 Feel free to comment 👍

I feel I am missing a bit of nomenclature here and experience with datomic 🙂 • what does norm stand for? • the discussion linked in the PR mentions datomic specific stuff in stork and conformity, can someone briefly describe what should be ported to datahike? On the plus side: • transaction data as edn is clearly a good choice • code of the PR looks simple and direct enough I feel I need a bit more context, I'll try to collect the missing info 🙂

questions: • why is it needed to have an :norm key in the transaction edn? • why isn't it unique? • isn't the filename enough? • maybe more helpful to have a :description key instead

> isn't the filename enough? ah I've missed https://github.com/replikativ/datahike/pull/598/files#diff-2f747a48a15846b2f6af676194a5da9552437e28558e46ec718d5916abdc7089R38 earlier yeah it makes sense to use the filename if the :norm key is not specified. 👍 What if the filename contains spaces? keyword does not complain but the result looks weird

user> (keyword "with spaces")
:with spaces

Adding a timestamp is redundant because you always have a timestamp with your transactions anyway or am I missing something?

you are totally right! I am going to remove it from my implementation as well 🙂

I don't quite get what you mean with 'in line' in your last question. If you mean something like serialization, then yes that totally makes sense. In case someone creates the db from scratch the migrations need to be applied in a certain order. I need to think about it but we will have to compile the norms at first and transact them in order... :thinking_face:

Yes I am thinking about that use case; another concern I have is that multiple people may be working on the same project and adding migrations in two different branches, there is a chance of conflicting transactions. Another source of problems could be that from your machine you could run some migrations againts a staging or production db and then you forget to commit the migration itself; this happened at $work before we introduced automatic migrations during deployments and it is probably an edge case of our "peculiar" setup at that time 🙂

(using a relational db and the rambler migration tool, nothing related to datahike or other datalog dbs)

Some mistakes can not be remedied by a migration tool. But as a linux-user I am not used to filenames with spaces, so that is a good point. Thanks for your help 👍