Clojurians Log v2
datahike
I have been developing with datahike on Linux without issue, but when I run on Windows 10 I get the error below. I have full read/write permissions on the directory as well. I have run the process with User and Admin privileges and get the same error. Has anyone else come across this error when running on Windows 10?
Caused by: java.nio.file.AccessDeniedException: C:\Users\ahuggins\Downloads\arena-db-win-release-2024.06.05.2\windows-jre\bin\.arena_db_data\1643fed1-9f2d-514e-b007-53bbb8682214.ksv.new -> C:\Users\ahuggins\Downloads\arena-db-win-release-2024.06.05.2\windows-jre\bin\.arena_db_data\1643fed1-9f2d-514e-b007-53bbb8682214.ksv
at java.base/sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at java.base/sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at java.base/sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.base/java.nio.file.Files.move(Unknown Source)
at konserve.filestore.BackingFilestore._atomic_move(filestore.clj:150)
at konserve.impl.defaults$update_blob.invokeStatic(defaults.cljc:56)
at konserve.impl.defaults$update_blob.invoke(defaults.cljc:50)
at konserve.impl.defaults$io_operation.invokeStatic(defaults.cljc:237)
at konserve.impl.defaults$io_operation.invoke(defaults.cljc:233)
at konserve.impl.defaults.DefaultStore._assoc_in(defaults.cljc:356)
at konserve.core$assoc_in.invokeStatic(core.cljc:146)
at konserve.core$assoc_in.invoke(core.cljc:139)
at konserve.core$assoc.invokeStatic(core.cljc:159)
at konserve.core$assoc.invoke(core.cljc:152)
at datahike.writing$fn__33694.invokeStatic(writing.cljc:236)
at datahike.writing$fn__33694.doInvoke(writing.cljc:177)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at datahike.writing$fn__33662$G__33642__33669.invoke(writing.cljc:172)
at datahike.writing$create_database.invokeStatic(writing.cljc:260)
at datahike.writing$create_database.doInvoke(writing.cljc:256)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$apply.invoke(core.clj:662)
at datahike.writer$fn__34618.invokeStatic(writer.cljc:160)
at datahike.writer$fn__34618.doInvoke(writer.cljc:158)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.MultiFn.invoke(MultiFn.java:229)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$apply.invoke(core.clj:662)
at datahike.api.impl$create_database.invokeStatic(impl.cljc:42)
at datahike.api.impl$create_database.doInvoke(impl.cljc:41)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at arena.db$initialize_db.invokeStatic(db.clj:14)
at arena.db$initialize_db.invoke(db.clj:11)
at arena.core$fn__55857.invokeStatic(core.clj:44)
at arena.core$fn__55857.invoke(core.clj:44)
at donut.system$apply_stage_fn.invokeStatic(system.cljc:612)
at donut.system$apply_stage_fn.invoke(system.cljc:609)
at donut.system$handler_stage_fn$fn__55644.invoke(system.cljc:693)
at donut.system$apply_signal_stage$fn__55657.invoke(system.cljc:746)
... 14 moreI think you are right. I will try a test with the konserve library next.
That is very helpful, thanks! It would be cool if we can make sure it runs smoothly on Windows, there is no reason why it shouldn't.
Thanks for the reply! The small test I made works just fine. Here's the code for the small test. Also a screen grab of the error when initializing datahike and the directory contents.
Weird. I assume you ran your script in the same folder the db lives in.
Is this what you mean?
yes
It was able to run inside there and replace the file contents without error.
That is super weird.
What happens if you use the same absolute paths as in the Exception?
I changed the function to use the absolute path as shown in the snippet. It ran without error.
Ok, thanks! The only other thing I can think of is different file access rights for the files created by datahike/konserve. Could you compare those against the files you created manually?
These are screenshots of permissions for the test, ksv, and ksv.new files respectively.
They look the same, right?
Sorry, yes the are the same.
This is the konserve code used https://github.com/replikativ/konserve/blob/main/src/konserve/filestore.clj#L152, it looks the same as what you have.
The only thing I could imagine being a problem would be something weird with the path construction, but it looks ok. I don't have access to a Windows machine right now, can you open an issue to keep track of this? Also if you have ideas feel free to bring them up as well.
I agree. It has baffled me for the past few days. I will open an issue so that there is record of what I am seeing. Thank you for your help!
It would be reasonable to git clone a local konserve copy and put it into your deps and then instrument these lines to see a bit more clearly what is exactly invoked and how it compares to your code snippet.
Sorry, I failed to give my conclusion. The goal of the example code was to reproduce the same symptoms that I am experiencing with konserve and datahike on Windows. It suggests that, in konserve, a file might still be locked for some reason while being moved to/from resulting in an AcessDenied error. I have not read enough of the konserve code base to have confirmed that hypothesis yet. Also note that the example code runs just fine on Linux. As for possible workarounds, maybe a slight delay added between closing the file and calling move. I plan to read more into the konserve code for assoc-in soon and test some ideas.
Yes, that makes sense.
Did you get any further insights?
I have. I was able to reproduce the error with a minimal example that held a target path open for sometime and then try to move to it. Here's the code and a screenshot of the error.
The error also occurs with konserve when calling the assoc-in fn. From what I have read, it seems that the move function doesn't have a good history with windows.
oh no 🤨
Any suggestions for workarounds?
In your example the file is still open for writing when it is moved. Maybe we can change that. We could also have a special implementation for Windows.
Strange, we use this call https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#move-java.nio.file.Path-java.nio.file.Path-java.nio.file.CopyOption...- with ATOMIC_MOVE option. Can you try that manually independently of Datahike?