Clojurians Log v2

I’m honestly at a loss, sorry

It looks like it’s been passed a closed file

Is there any chance you’ve used it with a with-open?

To explain what you’re seeing, you’re trying to access the “transaction file”. This is a file that contains a sequence of transaction records, and it’s instantiated in asami.durable.store (line 273).

Each transaction record contains the information about a commit point: • roots of the 3 triple-index trees • root of the data pool index (this is the structure that maps from strings/data/etc to ID numbers, and back) • The number of nodes, blocks, and pool nodes that have been used (this is because all of the index files are grown past what is needed, and these numbers allow the database to truncate to just what has been used). • The largest node ID allocated. • Timestamp This data makes a fixed size record, and the TxFile is just a sequence of these records.

All up, they are 9 long values. Meaning 9*8=72 bytes.

so the first transaction commit data is at position 0 in the file, the second is at offset 72, the third is at 144, etc

Every time a transaction is committed, then it goes to the end of this file and writes 72 bytes there.

tx-file-size is a function that gives you the size of the file. This is needed when it starts, so it knows where to append to. It’s also used when finding a transaction (say with the since function), because this gives the upper bound of the array to search in when looking for the transaction you’ve requested.

It appears that you’ve asked for the latest version of the database, so it’s called latest in https://github.com/quoll/asami/blob/0434e590dba60c5c561e5e62cbe6ea0b5e051f4b/src/asami/durable/flat_file.clj#L216 This has asked, “How big is the file?” so it can divide by the transaction size, getting the transaction number (starting at 0). This is going to be passed to get-tx to read in the data found at that transaction offset. e.g. if 3 transactions exist, then there will be 216 bytes. Divide by 72, and it now knows that there are 3 transactions, with the final one being transaction #2. That’s at offset 144 (2*72). So it will read that record.

BUT, it hasn’t got that far. Instead, it’s tried to call tx-file-size and failed. This failed on https://github.com/quoll/asami/blob/0434e590dba60c5c561e5e62cbe6ea0b5e051f4b/src/asami/durable/flat_file.clj#L193 when it attempted a .getFilePointer on the file

Oh… one other trick… the file pointer is ALWAYS at the end of the file. There is never a seek to somewhere in the middle of the file.

That’s because write operations always append, and the file position is used for this. But read operations are always done via a memory mapping of the file. So the file position never changes

I hope that didn’t overwhelm you with unnecessary background information.

Anyway… it comes back to where I started: latest has been called on this transaction file, but the file is already shut. If the TxFile object was closed by any other code, then something that thinks it owns the TxFile will think everything is fine, but the RandomAccessFile object underlying it will have been closed, and it won’t know

TxFiles belong to a asami.durable.store/DurableConnection so, if that was closed or released then it would have closed the underlying files, even though the object is still around

both delete-database and release (both are protocol functions for a Connection) will close the file

Thanks a lot, I really appreciate understanding how things work. It turns out the problem was caused by incorrect integration with Mount, where :stop closed the files but subsequent :start has nat re-opened them. I have yet to find out what exactly I do wrong / how to do it right.

So on :start I do asami.core/connect and on :stop I do asami.core/shutdown . I guess that when I call 1. connect, 2. shutdown, 3. connect then the 2nd connect does not really do everything I would expect it, presumably some state survives from the previous connect call and thus it mistakenly concludes that there is no need to do all the initialisation and does not re-open the tx file. Any tips how to correct the situation? Looking into it I see that connect does first check if-let [conn (@connections uri)] and reading shutdown I see it only releases the connections but does not clear the connections atom so the subsequent connect won’t do anything.

The only global state values are: • file resources (the files that the process has open) • The global connection atom. This maps URIs to open connections. Maybe a shutdown isn’t removing the latter?

Oh… that’s exactly what you just said!

Exactly ,see the updated ☝️

Hey, you caught me getting out of the shower! 🤣 About to go downstairs and wish my kids a merry Christmas

I guess delete-database and release either need handles back to the global connections map, or they shouldn’t be called directly

Sorry for that 🙂 Please ignore me and go enjoy your kids, and merry Christmas to you all!

Merry Christmas

I guess delete-database actually deletes the files from the disk, no? I do not want to do that, I want the data to persist 🙂

Yes, but the connection map has to be updated!

Even though that’s not a function you need

Yes. so there should be likely a function that both does shutdown AND resets the connections map.

I realized I can solve my problem simply by not calling shutdown from Mount, and instead rely on the jvm shutdown hook that calls it upon JVM exit.

I will try to get a Boxing Day release done 🙂

> Looking into it I see that connect does first check if-let [conn (@connections uri)] and reading shutdown I see it only releases the connections but does not clear the connections atom so the subsequent connect won’t do anything. This meeting mostly doesn’t need me, so I checked in on this. I’m now removing the connections during shutdown, but I will explain: the shutdown operation was specifically to be called during shutdown. Nothing should attempt to access anything at all after shutdown is called. In fact, I didn’t expect anyone to call shutdown. You’ll see that it’s only used as a JVM shutdown hook.

> I guess delete-database actually deletes the files from the disk, no? I do not want to do that, I want the data to persist 🙂 I was focused the connection map when this came in, so I forgot to say… If you want to close everything but not delete the files, then the operation you were looking for was release. delete-database does what the name says.

Thank you! My use case is using clojure.tools.namespace.repl/refresh to reload my code and thus I need to cleanly release stuff before it and re-initialize it afterwards. I have changed my Mount use so that it only starts Asami and does nothing on shutdown, and is marked with ^{:on-reload :noop} so that tools-ns doesn’t mess with it. This seems to be working fine.

Well, it’s safer to do it the what I’ve changed it, but yeah… I specifically intended shutdown for JVM shutdown. I never really expected anyone to try to use it to clear things. release was different though, and that needed to be fixed

I did not manage this. (Sorry!) I am hoping for something soon, but between not turning on the computer (which was necessary for my health!) and lingering health issues I just didn’t get to it

Soon though. I need to incorporate Raphael into Asami, and I’ll release when I do