Hey @U0F7M1KA7 - if you're starting Crux with crux/start-node, that call shouldn't block - after all, it needs to return the node to you. Could you share the app.core/init-db function, or the relevant parts of it? Happy to go via DM if that's preferable.

Also, are you trying to run it using Crux's http-server module? It's been a while since I've used Tomcat et al, but I suspect you don't want us to be starting up an HTTP server as well

Hi @U050V1N74, many thanks Here is the code :init use start-node and :destroy stop-node

(defonce node (atom nil)))

(defn start-node [storage-dir]
  (reset! node
          (crux/start-node {:crux.node/topology '[crux.standalone/topology
                                                  crux.kv.rocksdb/kv-store]
                            :crux.standalone/event-log-dir "opt/crux-store/eventlog-1"
                            :crux.kv/db-dir "opt/crux-store/db")})))

(defn stop-node
  []
  (.close @node))

I don't use a http-server for this

thanks - probably unrelated, but you'll want to include :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv in the topology map too (we've got a docs change coming to clarify this)

I don't use the crux http-server here

if you call start-node at the REPL, does it block?

no

using lein ring server it's all fine

hmm :thinking_face: I wonder what's different between that and what lein ring does...

after that, I'd probably try lein ring uberjar, see whether that starts correctly (taking Tomcat out of the loop)

ok

i'll do now

i also tried to setup a reitit route to init the db, but this blocks too

No problem with the Jar

it works perfectly

ok, so seems related specifically to the war/Tomcat. :thinking_face:

yes i think too but i not very confident with it 🙂 I will search what's going on

I don't know of anything in Crux that'd conflict with Tomcat I'm afraid. Will have a think...

many thanks

You're welcome 🙂

Hi @U050V1N74 I had this logs in catalina.out may be it tell you something ? otherwise i continue digging 🙂 Many thanks

28-Jul-2020 17:50:36.434 WARNING [https-jsse-nio-8443-exec-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ehdsicts2] appears to have started a thread named [crux.io.cleaner-thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
 java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
 $cleanup_loop.invokeStatic(io.clj:29)
 $cleanup_loop.invoke(io.clj:27)
 clojure.lang.AFn.run(AFn.java:22)
 java.lang.Thread.run(Thread.java:748)
28-Jul-2020 17:50:36.435 SEVERE [https-jsse-nio-8443-exec-3] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ehdsicts2] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@7977fff8]) and a value of type [org.agrona.ExpandableDirectByteBuffer] (value [ExpandableDirectByteBuffer{address=140448946589376, capacity=128, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=128 cap=128]}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Thanks 🙂 I'd be surprised if fixing that fixed your problem, but it is interesting - we evidently assume that some of our namespaces aren't ever going to be unloaded, which doesn't apply if Crux is being run in an application server.

It's probably no more than a handful of references - are you able to create a shutdown function that we could then temporarily populate with manual cleanup, see if that makes a difference? (or, at least, removes those errors 🙂)

I did try to start crux node insert some docs in it and then close the node but still the same problem. Is that what you mean ?

Can it be the allocated memory of tomcat ?

sorry - could what be the allocated memory of tomcat?

ah - it looks like you have control over a :destroy function - I wonder whether explicitly closing the resources that Tomcat says we're leaking might fix those logged errors

first one to try would be something like (doto @#' .interrupt .join) - that should clear the .cleaner-thread

but I doubt this'll fix the wider issue, tbh - if it's that the Crux node isn't starting at all

Increasing the allocated memory of tomcat but that is not woking too. The crux node is starting and the insert are working correctly (all the files in the db and eventlog-1 folder are generated but tomcat freeze after that; Each time tomcat reload files, it create more and more folder in the temp folder

mm, it's not a lot of memory that it'll be leaking each time, but will add up over time and many re-deploys

any idea what it's doing when it freezes? cpu to 100, say, or blocked waiting at a specific line?

might be worth trying to get hold of a thread dump

It seems that there is a problem with threadlocals Memeory protection from tomcat but don't know what do with this 😞 https://github.com/redisson/redisson/issues/599

I did try

(defonce node (atom nil)); (start-node "crux-store"))

(defn start-node [storage-dir]
  (reset! node
          (crux/start-node {:crux.node/topology '[crux.standalone/topology
                                                  crux.kv.rocksdb/kv-store]
                            :crux.standalone/event-log-dir "/Users/ouvanous/ehdsi-cts/crux-store/eventlog-1"
                            :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv
                            :crux.kv/db-dir (str "/Users/ouvanous/ehdsi-cts/crux-store/db")}))
  nil)

(defn stop-node
  []
  (.close @node)
  (reset! node nil)
  (doto @#' .interrupt .join))

but no success

did it at least prevent the cleaner-thread error message?

sorry i don't know. I am a bit lost here

i'll continue searching

I wonder would it be useful to use jconsole to see where the threads are blocked?

Here is the response from weavejester on lein-ring: "You may want to check whether or not Crux is blocking the current thread, and whether or not Tomcat allows background threads, or what its permission model is like for servlets." Don't know exactly what to do but i'll search 🙂

Hi @U7KD4HQ1W Here are jconsole screenshots

Don't know what to do with that ... 😞

according to SO if you kill -3 <tomcat-pid> it'll print all that to catalina.out, save you screenshotting them 🙂

https://stackoverflow.com/questions/18573411/tomcat-thread-dump

there's one JDK utility that'll do it, too, but I never remember which one

ah, jstack <pid> > <out-file>

here is the output of catalina.out

https://gist.github.com/ouvanous/72aabd4e15e9d882b5ea9c6f3f510792

and jstack one

https://gist.github.com/ouvanous/889a5c2b122c6b3ac6a2557b9ad68211

thanks - I can't see any Crux in the jstack output, was it running at that point?

arf ... sorry i did this with some yesterday tests ...

i'll recompile and send you a gist

I don't find how to use jstack when deployed as tomcat change process everytime it try to restart

Can this be a problem ? 20-07-30 09:08:50 ouvasam INFO [crux.hash.jnr:1] - dlopen(libgcrypt.dylib, 9): image not found

That should be fine - Crux has a number of hashing strategies, this is saying that it couldn't load one of them. If it can't find any of them, it falls back to java.security

ok many thanks

i'm using mount to manage my crux node:

(defn ^ICruxAPI start-jdbc-sqlite-node []
  (crux/start-node
    {:crux.node/topology '[crux.jdbc/topology]
     :crux.jdbc/dbtype   "sqlite"
     :crux.jdbc/dbname   "glam_db.sqlite"}))

(defstate crux-node
  :start (start-jdbc-sqlite-node)
  :stop (.close crux-node))

probably a good time to introduce Github 'Discussions' - we were invited into the beta this morning 🙂 https://github.com/juxt/crux/discussions

oh cool, github's needed a feature like that! should i make a post then?

yes please - seems like a good candidate for our first post 🙂

will see what notifications we all get, too

posted! https://github.com/juxt/crux/discussions/1008

yep, got notifications for that, cool 🙂

now to actually find the issue...

another strange thing is that this started happening halfway into my development session yesterday, when I feel like I hadn't touched any code that could have changed crux's behavior in the meantime... made me wonder if the root cause had to do with files somewhere that are somehow invalid

thanks for the help @U050V1N74 🙂