Clojurians Log v2

a table is pile of data on the disk, and index is a structure that makes lookups of data in that pile fast

ah, sorry, I see by index you don't mean a database index, you mean ragtimes index

From the docs: > An index simply maps IDs to migrations. So it seems to me that the index protocol is an abstraction to handle that mapping, and it can be backed by an SQL table. As a counter-example, Migratus is an SQL-only migration library with a hardcoded way to index migrations, so there's no need for a separate protocol.

But if you look at things that use ragtime, like the duct module, or basically every example I can find, they just use a map. Sure, you can extend the protocol to be backed by a table, but how/when/why? It's all very handwavy

"it maps IDs to migrations" Why do I care about this to the extent that it's a required argument to some of the API methods? Why isn't the table of applied migrations sufficient for mapping IDs to migrations?

So much of this seems to assume I actually know what's going on, when I'm basically on planet ELI5 😅

> But if you look at things that use ragtime, like the duct module [...] they just use a map Can you share a link where Duct uses Ragtime with maps? > Why isn't the table of applied migrations sufficient for mapping IDs to migrations? Assuming I'm correct in my previous message, that's just a design decision of Ragtime. If you care only about SQL and don't want to learned deeper, maybe Migratus would be a better fit?

not all migrations have been applied yet

the canonical set of migrations is the whatever, say sql files you have on disk, and the index maps ids to those, the table in the database just tracks which migrations have been applied to a given database

@p-himik https://github.com/duct-framework/migrator.ragtime/blob/758cce7bbbad8acee62e89270e5c61eda9008da9/src/duct/migrator/ragtime.clj#L79

migrate all builds an index over all the migrations, then uses that index to find migrations by id https://github.com/weavejester/ragtime/blob/master/core/src/ragtime/core.clj#L64 https://github.com/weavejester/ragtime/blob/master/core/src/ragtime/core.clj#L71

the first hashmap passed there is what is used as the index, so if you want to do something different, you can pass in something else that supports the index protocol there

> If you care only about SQL and don't want to learned deeper, maybe Migratus would be a better fit? > I do want to learn deeper, that's why I'm asking. Given that everything I find just uses an in-memory map, I don't understand how an index that's blown away on restart helps me efficiently manage migrations. And since duct is supposed to be production ready, I'm assuming the approach it takes is a good example to follow

> the first hashmap passed there is what is used as the index, so if you want to do something different, you can pass in something else that supports the index protocol there > I would have to override the integrant init hook to change the index, which doesn't really seem like intended usage 🤷🏻‍♂️

it is because ragtime is trying to be super generic, so how ever your migrations are stored might not have an efficient way to find them by id, so it builds an in memory index over the ids so it can find the migration for a given id quickly

it is a protocol so if however you are storing your migrations does give you fast lookup by id, then you can just use that if you want

> I'm assuming the approach it takes is a good example to follow FWIW, I myself am absolutely not a fan of migrations having anything to do with the run time of an app. IMO they should be relegated to a separate preparation step that's run before your main app.

Maybe the thing I'm missing is understanding what happens on restart. So let's say I have 5 migrations. 4 have been applied while running the application, and so they all appear in the in-memory index. I write a fifth migration and shut down the app. The index is lost. I restart the app and... Now what? Do I apply all migrations, which adds the previous 4 to the index and treats them as a noop against the db because they have already been applied before (according to the migrations table in the db), and then adds the fifth to the index and also applies it? Do I just apply the fifth migration? Does that mean the index has no idea about the previous 4 now? Does that cause problems?

unclear what you mean

in general you just migrate up or whatever and it looks at what migrations exist (what sql files or however you are defining migrations) and what migrations have been applied, and applies the ones that exist and haven't been applied

Hmmm... Okay, put it this way: Am I right in understanding that ragtime's index is just a working cache that helps it do its job of applying/rolling back migrations? So it doesn't much matter if it's persistent or not, it'll get built as necessary and used to perform the required action?

correct

And it being explicitly passed in is just to offer the ability to tune the impl, if necessary

Though duct implies that it's really not necessary (in the majority case)

It's one of those "if you need it, you'll know" type situations