Fork me on GitHub
#xtdb
<
2020-07-28
>
ouvasam09:07:24

Crosspost #ring Hi, i am trying to compile a ring app as war (lein ring uberwar), to run on tomcat. It is may be a stupid question, but i can't find how to start crux node on launch. Once i start the DB it blocks. I did try to use the

{:ring {:init app.core/init-db ... }
If someone has a tip ? Many thanks and have a nice day

jarohen09:07:11

Hey @U0F7M1KA7 - if you're starting Crux with crux/start-node, that call shouldn't block - after all, it needs to return the node to you. Could you share the app.core/init-db function, or the relevant parts of it? Happy to go via DM if that's preferable.

jarohen09:07:37

Also, are you trying to run it using Crux's http-server module? It's been a while since I've used Tomcat et al, but I suspect you don't want us to be starting up an HTTP server as well

ouvasam09:07:12

Hi @U050V1N74, many thanks Here is the code :init use start-node and :destroy stop-node

(defonce node (atom nil)))

(defn start-node [storage-dir]
  (reset! node
          (crux/start-node {:crux.node/topology '[crux.standalone/topology
                                                  crux.kv.rocksdb/kv-store]
                            :crux.standalone/event-log-dir "opt/crux-store/eventlog-1"
                            :crux.kv/db-dir "opt/crux-store/db")})))

(defn stop-node
  []
  (.close @node))

ouvasam09:07:06

I don't use a http-server for this

jarohen09:07:29

thanks - probably unrelated, but you'll want to include :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv in the topology map too (we've got a docs change coming to clarify this)

ouvasam09:07:32

I don't use the crux http-server here

👍 3
jarohen09:07:52

if you call start-node at the REPL, does it block?

ouvasam09:07:01

using lein ring server it's all fine

jarohen09:07:07

hmm :thinking_face: I wonder what's different between that and what lein ring does...

jarohen09:07:19

after that, I'd probably try lein ring uberjar, see whether that starts correctly (taking Tomcat out of the loop)

ouvasam09:07:33

i'll do now

ouvasam09:07:55

i also tried to setup a reitit route to init the db, but this blocks too

ouvasam09:07:48

No problem with the Jar

ouvasam09:07:00

it works perfectly

jarohen09:07:35

ok, so seems related specifically to the war/Tomcat. :thinking_face:

ouvasam09:07:59

yes i think too but i not very confident with it 🙂 I will search what's going on

jarohen09:07:01

I don't know of anything in Crux that'd conflict with Tomcat I'm afraid. Will have a think...

ouvasam09:07:02

many thanks

jarohen09:07:11

You're welcome 🙂

ouvasam08:07:29

Hi @U050V1N74 I had this logs in catalina.out may be it tell you something ? otherwise i continue digging 🙂 Many thanks

28-Jul-2020 17:50:36.434 WARNING [https-jsse-nio-8443-exec-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ehdsicts2] appears to have started a thread named [crux.io.cleaner-thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
 java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
 $cleanup_loop.invokeStatic(io.clj:29)
 $cleanup_loop.invoke(io.clj:27)
 clojure.lang.AFn.run(AFn.java:22)
 java.lang.Thread.run(Thread.java:748)
28-Jul-2020 17:50:36.435 SEVERE [https-jsse-nio-8443-exec-3] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ehdsicts2] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@7977fff8]) and a value of type [org.agrona.ExpandableDirectByteBuffer] (value [ExpandableDirectByteBuffer{address=140448946589376, capacity=128, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=128 cap=128]}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

jarohen08:07:10

Thanks 🙂 I'd be surprised if fixing that fixed your problem, but it is interesting - we evidently assume that some of our namespaces aren't ever going to be unloaded, which doesn't apply if Crux is being run in an application server.

jarohen08:07:07

It's probably no more than a handful of references - are you able to create a shutdown function that we could then temporarily populate with manual cleanup, see if that makes a difference? (or, at least, removes those errors 🙂)

ouvasam08:07:30

I did try to start crux node insert some docs in it and then close the node but still the same problem. Is that what you mean ?

ouvasam08:07:48

Can it be the allocated memory of tomcat ?

jarohen08:07:17

sorry - could what be the allocated memory of tomcat?

jarohen08:07:13

ah - it looks like you have control over a :destroy function - I wonder whether explicitly closing the resources that Tomcat says we're leaking might fix those logged errors

jarohen08:07:47

first one to try would be something like (doto @#' .interrupt .join) - that should clear the .cleaner-thread

jarohen08:07:03

but I doubt this'll fix the wider issue, tbh - if it's that the Crux node isn't starting at all

ouvasam09:07:27

Increasing the allocated memory of tomcat but that is not woking too. The crux node is starting and the insert are working correctly (all the files in the db and eventlog-1 folder are generated but tomcat freeze after that; Each time tomcat reload files, it create more and more folder in the temp folder

jarohen09:07:47

mm, it's not a lot of memory that it'll be leaking each time, but will add up over time and many re-deploys

jarohen09:07:44

any idea what it's doing when it freezes? cpu to 100, say, or blocked waiting at a specific line?

jarohen09:07:27

might be worth trying to get hold of a thread dump

ouvasam11:07:01

It seems that there is a problem with threadlocals Memeory protection from tomcat but don't know what do with this 😞 https://github.com/redisson/redisson/issues/599

ouvasam11:07:45

I did try

(defonce node (atom nil)); (start-node "crux-store"))

(defn start-node [storage-dir]
  (reset! node
          (crux/start-node {:crux.node/topology '[crux.standalone/topology
                                                  crux.kv.rocksdb/kv-store]
                            :crux.standalone/event-log-dir "/Users/ouvanous/ehdsi-cts/crux-store/eventlog-1"
                            :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv
                            :crux.kv/db-dir (str "/Users/ouvanous/ehdsi-cts/crux-store/db")}))
  nil)

(defn stop-node
  []
  (.close @node)
  (reset! node nil)
  (doto @#' .interrupt .join))

ouvasam11:07:50

but no success

jarohen12:07:31

did it at least prevent the cleaner-thread error message?

ouvasam12:07:20

sorry i don't know. I am a bit lost here

ouvasam12:07:29

i'll continue searching

peter hull07:07:18

I wonder would it be useful to use jconsole to see where the threads are blocked?

ouvasam07:07:32

Here is the response from weavejester on lein-ring: "You may want to check whether or not Crux is blocking the current thread, and whether or not Tomcat allows background threads, or what its permission model is like for servlets." Don't know exactly what to do but i'll search 🙂

ouvasam08:07:35

Hi @U7KD4HQ1W Here are jconsole screenshots

ouvasam08:07:55

Don't know what to do with that ... 😞

jarohen08:07:40

according to SO if you kill -3 <tomcat-pid> it'll print all that to catalina.out, save you screenshotting them 🙂

jarohen08:07:14

there's one JDK utility that'll do it, too, but I never remember which one

jarohen08:07:34

ah, jstack <pid> > <out-file>

ouvasam08:07:10

here is the output of catalina.out

ouvasam08:07:47

and jstack one

jarohen08:07:03

thanks - I can't see any Crux in the jstack output, was it running at that point?

ouvasam08:07:18

arf ... sorry i did this with some yesterday tests ...

ouvasam08:07:30

i'll recompile and send you a gist

ouvasam09:07:20

I don't find how to use jstack when deployed as tomcat change process everytime it try to restart

ouvasam11:07:01

Can this be a problem ? 20-07-30 09:08:50 ouvasam INFO [crux.hash.jnr:1] - dlopen(libgcrypt.dylib, 9): image not found

jarohen11:07:52

That should be fine - Crux has a number of hashing strategies, this is saying that it couldn't load one of them. If it can't find any of them, it falls back to java.security

ouvasam11:07:57

ok many thanks

lgessler17:07:02

hi i'm having an issue on crux-jdbc and crux-core 20.07-1.9.2-beta where i get Execution error at crux.topology/resolve-id (topology.clj:15). No such var: lru/wrap-lru-cache . nesting details (also should i be posting this here or somewhere else like as a github issue?)

👀 3
lgessler17:07:34

i'm using mount to manage my crux node:

(defn ^ICruxAPI start-jdbc-sqlite-node []
  (crux/start-node
    {:crux.node/topology '[crux.jdbc/topology]
     :crux.jdbc/dbtype   "sqlite"
     :crux.jdbc/dbname   "glam_db.sqlite"}))

(defstate crux-node
  :start (start-jdbc-sqlite-node)
  :stop (.close crux-node))
when I call start-jdbc-sqlite-node on its own without calling mount/start it works OK, the issue only occurs when I call mount/start to begin my application. this error doesn't occur when i downgrade to 20.06-1.8.5-alpha, so i'm not positive what's going on but it seems maybe crux is involved somehow

jarohen17:07:02

probably a good time to introduce Github 'Discussions' - we were invited into the beta this morning 🙂 https://github.com/juxt/crux/discussions

lgessler17:07:22

oh cool, github's needed a feature like that! should i make a post then?

jarohen17:07:40

yes please - seems like a good candidate for our first post 🙂

jarohen17:07:10

will see what notifications we all get, too

jarohen17:07:46

yep, got notifications for that, cool 🙂

jarohen17:07:53

now to actually find the issue...

lgessler17:07:40

another strange thing is that this started happening halfway into my development session yesterday, when I feel like I hadn't touched any code that could have changed crux's behavior in the meantime... made me wonder if the root cause had to do with files somewhere that are somehow invalid

lgessler17:07:47

thanks for the help @U050V1N74 🙂

🙏 3
jarohen18:07:48

Evening all - 1.10.0 is out 🚀 Full release notes at https://github.com/juxt/crux/releases/tag/20.07-1.10.0, but here are the highlights: * SQL query support https://github.com/juxt/crux/tree/20.07-1.10.0/crux-sql * Speculative transactions - 'what if this transaction were applied?' https://github.com/juxt/crux/tree/20.07-1.10.0/docs/speculative.adoc * More types of entity IDs - particularly, strings and longs. This one needs an index version bump - details in the release notes. * EQL Projections (aka 'pull syntax') https://github.com/juxt/crux/tree/20.07-1.10.0/docs/projections.adoc Enjoy 🙂

parrot 42
🎉 6