Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2023-05-11

Channels

#xtdb
<
2023-05-11
>
msuess09:05:55

Hello! I am using the checkpointing feature for both the kv and the lucene indices. I set it up locally to create a checkpoint every 30S for quick testing. I am passing the same S3 configurator in both configs. I am seeing this error: 

2023-05-11T09:21:46.577Z INFO [xtdb.checkpoint:326] - Uploading checkpoint at '{:xtdb.api/tx-id 510208}'
2023-05-11T09:21:51.368Z INFO [xtdb.checkpoint:326] - Uploading checkpoint at '{:xtdb.api/tx-time #inst "2023-05-11T09:21:34.516-00:00", :xtdb.api/tx-id 510208}'
2023-05-11T09:22:16.194Z INFO [xtdb.checkpoint:326] - Uploaded checkpoint: {:xtdb.checkpoint/cp-format "lucene-8", :tx {:xtdb.api/tx-id 510208}, :xtdb.s3.checkpoint/s3-dir "checkpoint-510208-2023-05-11T09:21:46.577-00:00/", :xtdb.checkpoint/checkpoint-at #inst "2023-05-11T09:21:46.577-00:00"}
2023-05-11T09:22:24.357Z WARN [xtdb.checkpoint:326] - Checkpointing failed
                                                    java.lang.Thread.run                       Thread.java: 1589
                      java.util.concurrent.ThreadPoolExecutor$Worker.run           ThreadPoolExecutor.java:  642
                       java.util.concurrent.ThreadPoolExecutor.runWorker           ThreadPoolExecutor.java: 1144
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run  ScheduledThreadPoolExecutor.java:  304
                                     java.util.concurrent.FutureTask.run                   FutureTask.java:  317
                     java.util.concurrent.Executors$RunnableAdapter.call                    Executors.java:  577
                                                                     ...
                       xtdb.checkpoint.ScheduledCheckpointer/schedule/fn                    checkpoint.clj:  104
                               xtdb.checkpoint.ScheduledCheckpointer/run                    checkpoint.clj:   82
                            xtdb.checkpoint.ScheduledCheckpointer/run/fn                    checkpoint.clj:   83
                                              xtdb.checkpoint/checkpoint                    checkpoint.clj:   44
                        xtdb.lucene/checkpoint-src/reify/save-checkpoint                        lucene.clj:  289
                                     xtdb.lucene/checkpoint-src/reify/fn                        lucene.clj:  291
                              org.apache.lucene.store.Directory.copyFrom                    Directory.java:  183
                        org.apache.lucene.store.FSDirectory.createOutput                  FSDirectory.java:  254
                org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>                  FSDirectory.java:  406
                org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>                  FSDirectory.java:  410
                                     java.nio.file.Files.newOutputStream                        Files.java:  228
                    java.nio.file.spi.FileSystemProvider.newOutputStream           FileSystemProvider.java:  484
                        sun.nio.fs.UnixFileSystemProvider.newByteChannel       UnixFileSystemProvider.java:  218
                           sun.nio.fs.UnixException.rethrowAsIOException                UnixException.java:  111
                           sun.nio.fs.UnixException.rethrowAsIOException                UnixException.java:  106
                         sun.nio.fs.UnixException.translateToIOException                UnixException.java:   94
java.nio.file.FileAlreadyExistsException: /private/var/folders/8g/_h2hhtt93dvbd7c56tb4zdhm0000gn/T/checkpointing9186776802355545322/_ee.nvd
    file: "/private/var/folders/8g/_h2hhtt93dvbd7c56tb4zdhm0000gn/T/checkpointing9186776802355545322/_ee.nvd"

2023-05-11T09:22:46.380Z Marcos-MacBook-Pro-Work.fritz.box WARN [xtdb.checkpoint:326] - Checkpointing failed
                                                    java.lang.Thread.run                       Thread.java: 1589
                      java.util.concurrent.ThreadPoolExecutor$Worker.run           ThreadPoolExecutor.java:  642
                       java.util.concurrent.ThreadPoolExecutor.runWorker           ThreadPoolExecutor.java: 1144
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run  ScheduledThreadPoolExecutor.java:  304
                                     java.util.concurrent.FutureTask.run                   FutureTask.java:  317
                     java.util.concurrent.Executors$RunnableAdapter.call                    Executors.java:  577
                                                                     ...
                       xtdb.checkpoint.ScheduledCheckpointer/schedule/fn                    checkpoint.clj:  104
                               xtdb.checkpoint.ScheduledCheckpointer/run                    checkpoint.clj:   82
                            xtdb.checkpoint.ScheduledCheckpointer/run/fn                    checkpoint.clj:   83
                                              xtdb.checkpoint/checkpoint                    checkpoint.clj:   44
                        xtdb.lucene/checkpoint-src/reify/save-checkpoint                        lucene.clj:  289
                                     xtdb.lucene/checkpoint-src/reify/fn                        lucene.clj:  291
                              org.apache.lucene.store.Directory.copyFrom                    Directory.java:  183
                        org.apache.lucene.store.FSDirectory.createOutput                  FSDirectory.java:  254
                org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>                  FSDirectory.java:  406
                org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>                  FSDirectory.java:  410
                                     java.nio.file.Files.newOutputStream                        Files.java:  228
                    java.nio.file.spi.FileSystemProvider.newOutputStream           FileSystemProvider.java:  484
                        sun.nio.fs.UnixFileSystemProvider.newByteChannel       UnixFileSystemProvider.java:  218
                           sun.nio.fs.UnixException.rethrowAsIOException                UnixException.java:  111
                           sun.nio.fs.UnixException.rethrowAsIOException                UnixException.java:  106
                         sun.nio.fs.UnixException.translateToIOException                UnixException.java:   94
java.nio.file.FileAlreadyExistsException: /private/var/folders/8g/_h2hhtt93dvbd7c56tb4zdhm0000gn/T/checkpointing9186776802355545322/_6q_Lucene84_0.tim
    file: "/private/var/folders/8g/_h2hhtt93dvbd7c56tb4zdhm0000gn/T/checkpointing9186776802355545322/_6q_Lucene84_0.tim"

tatut09:05:22

I’ve had this exact same problem

tatut09:05:55

mitigate with 

:keep-dir-between-checkpoints? false

🙏 3
msuess09:05:51

Using only kv checkpoints fixes it as well as increasing the frequency to five minutes.

msuess09:05:39

correction: increasing the frequency does not fix the problem