datomic 2024-03-13 | Slack Archive

Has anyone run into this?

2024-03-12 12:45:16.622 WARN  default    datomic.index - {:message "merge-db failed", :pid 2733131, :tid 1486}
java.io.IOException: No such file or directory
        at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.base/java.io.File.createTempFile(File.java:2170)
        at datomic.common$create_temp_directory.invokeStatic(common.clj:563)
        at datomic.common$create_temp_directory.invoke(common.clj:559)
        at datomic.fulltext$create_indexing_job.invokeStatic(fulltext.clj:176)
        at datomic.fulltext$create_indexing_job.invoke(fulltext.clj:170)
        at datomic.fulltext$do_indexing_job.invokeStatic(fulltext.clj:241)
        at datomic.fulltext$do_indexing_job.invoke(fulltext.clj:237)
        at datomic.fulltext$build_index$fn__4484$fn__4510.invoke(fulltext.clj:351)
        at datomic.fulltext$build_index$fn__4484.invoke(fulltext.clj:345)
        at datomic.fulltext$build_index.invokeStatic(fulltext.clj:334)
        at datomic.fulltext$build_index.invoke(fulltext.clj:324)
        at datomic.index$merge_db_STAR_.invokeStatic(index.clj:1846)
        at datomic.index$merge_db_STAR_.invoke(index.clj:1713)
        at datomic.index$merge_db$fn__5630.invoke(index.clj:1912)
        at datomic.index$merge_db.invokeStatic(index.clj:1903)
        at datomic.index$merge_db.invoke(index.clj:1893)
        at datomic.update$process_request_index$fn__13127$fn__13128$fn__13132.invoke(update.clj:216)
        at datomic.update$process_request_index$fn__13127$fn__13128.invoke(update.clj:211)
        at datomic.update$process_request_index$fn__13127.invoke(update.clj:207)
        at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2047)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

Ivar Refsdal08:03:31

Is the data-dir property in the transactor set to an existent and writable folder?

emil0r08:03:04

The data storage is SQL. Does it still need the data dir?

Ivar Refsdal08:03:13

Yes

Ivar Refsdal08:03:40

As far as I know.. I believe we've had similar problems. This makes the transactor crash when it tries to index, right?

emil0r08:03:16

Yes. Hmm… but we’ve had it without the data dir for 2 years now

Ivar Refsdal08:03:36

So perhaps now it finally needs some disk space for re-indexing?

emil0r08:03:10

It’s based off on the sql-transactor sample that comes with the tarball

emil0r08:03:19

No, we’ve had re-indexing before

emil0r08:03:34

Especially as we run excisions every now and then

emil0r08:03:32

Did your transactor crash consistently Ivar?

emil0r08:03:57

Hmm… so the docs say data-dir should be specified, but the sample config has it commented out 😐. Even so… why would the reindexing work before then

Ivar Refsdal08:03:31

I believe it crashed when indexing was done/requested

Ivar Refsdal08:03:57

I agree that the documentation is bad (and that the transactor should check on startup that data-dir is writable and exit if not)

emil0r08:03:13

Right… so this on-prem has a lot of memory given to it. Might be that if it can fit everything into memory it doesn’t touch the HD. We’ve just not managed to trigger it yet

Ivar Refsdal08:03:45

IIRC it also took quite some time before this crash started happening for us, I don't recall the details exactly though

emil0r08:03:32

Sound like a build up over time then

Ivar Refsdal08:03:57

Probably.. I would just issue https://docs.datomic.com/pro/clojure/index.html#datomic.api/request-index after setting the data-dir property (to an existent and writable folder) and see if that solves the problem.

emil0r08:03:06

Ran that without data-dir set and everything was indexed correctly according to the logs

emil0r08:03:13

Will try with a much lower memory config

Ivar Refsdal08:03:37

Here it is in the docs: > Transactors use the data-dir specified in the transactor properties file as the storage location for the dev and free storage protocols as well as for temporary storage with all storage protocols. The data-dir path must be writable by the Datomic process and the appropriate filesystem permissions must be granted to allow Datomic to create additional temporary directories within the data-dir. https://docs.datomic.com/pro/operation/deployment.html#misconfigured-data-dir

emil0r08:03:52

Yeah. I’m trying to trigger the error. Have something in mind

emil0r09:03:14

Will add the data-dir either way 🙂

👍 2

☕ 2

emil0r08:03:48

This is for on-prem

2024-03-13

Channels