hi… Clojure perma-noob here, it seems… I want to get going with Crux and Postgres (because I’m familiar with Postgres and it’s on my system… could go another route)…

is there a tutorial on that, step by step, or a finished simple project, to study?

I looked at RocksDB and was baffled at how to get it installed and configured, so dunno about that route.

Truly awesome intro interplanetary tutorial by the way!

And very effective.

you don't need to think about rocksdb at all, really. it's included in the dependency .jar, all you have to do is set up the crux config as per the docs

ah gotcha… I now realize it’s in the jar that gets pulled in…

not one you download and run in the cli.

in that case, how does it persist your data? I mean, if you restart your project… or whatever.

all the rocksdb data will just be a folder on your filesystem somewhere

oh sweet, this is getting better by the second.

I have lots of experimental projects that sometimes need persistence and this is ideal.

yeah, crux+rocksdb is super easy to set up. easier than even SQLite

I’m so damn excited about this… you have no idea. Well, maybe you do.

The old pattern of firing up Postgres etc etc for a toy project got so annoying.

voila… there it is in the docs :db-dir (io/file "/tmp/rocksdb"

maybe not /tmp though, for me, lol.

God this is cool.

so, I had

(def node
    (crux/start-node
     {:crux.node/topology :crux.standalone/topology
      :crux.node/kv-store "crux.kv.memdb/kv"
      :crux.standalone/event-log-dir "data/eventlog-1"
      :crux.kv/db-dir "data/db-dir-1"
      :crux.standalone/event-log-kv-store "crux.kv.memdb/kv"}))

that was a dumb question, sorry.

I’ll give it a go and report back.

yup, not a dumb question. the docs probably mention it but you should repeat the rocksdb config 3 times, for the index store, tx log, and document store. probably best to put each in their own folder for performance

the 3 different stores are pretty much just an implementation detail you don't need to worry about if you don't care about scaling beyond a single machine

I don’t for quite a while. Should that happen, I’ll put on a 3 piece suit and be right back here.

so… with

[juxt/crux-core "19.12-1.6.0-alpha"]
[juxt/crux-rocksdb "21.04-1.16.0-beta"]

(def node
    (crux/start-node
     {:crux.node/topology :crux.standalone/topology
      :crux.node/kv-store "crux.rocksdb/->kv-store"
      :crux.standalone/event-log-dir "crux.rocksdb/->kv-store"
      ;; I'll change this when I want the data to survive reboots
      :crux.kv/db-dir "/tmp/rocksdb"
      :crux.standalone/event-log-kv-store "crux.rocksdb/->kv-store"})
    )

Syntax error (FileNotFoundException) compiling at (crux/rocksdb.clj:1:1).
Could not locate crux/system__init.class, crux/system.clj or crux/system.cljc on classpath.
ERROR: Unhandled REPL handler exception processing message {:nrepl.middleware.print/stream? 1, :nrepl.middleware.print/print c\
ider.nrepl.pprint/pprint, :nrepl.middleware.print/quota 1048576, :nrepl.middleware.print/options {:right-margin 70}, :op stack\
trace, :session 64e6c494-4d3d-4be6-b7c3-4c6a96357a85, :id 21}
java.lang.ClassNotFoundException: com.sun.tools.javac.util.List

@macrobartfast That’s an nREPL (or CIDER?) bug that I saw folks talk about just the other day… let me see if I can pull that up…

I think you might be looking at an old version of the config

https://opencrux.com/reference/rocksdb.html

oh, also.. those crux versions have to match

https://github.com/clojure-emacs/orchard/issues/105 and https://github.com/clojure-emacs/orchard/issues/103

1.6.0-alpha with 1.16.0-beta isn't going to go well :P

haha ok… something skittered across my subconscious when I looked at those versions. But I ahead and eval’ed it anyway.

So, I’ll start with fixing my versions.

did you get the 1.6.0 from a tutorial somewhere? link it here and I'll bet the juxt people will fix it

The CIDER/orchard issue is about REPL middleware and dynamic classpath stuff but if you’re working with Shadow-cljs you may not have much leeway in terms of simplifying/trimming your nREPL/CIDER config…?

I did get it from a tutorial…. https://opencrux.com/reference/rocksdb.html

so, can I assume there’s a crux-rocksdb 19.12-1.6.0-alpha ? staring at https://clojars.org/juxt/crux-rocksdb and all confused now.

yeah, it's an ancient version

you want to bump crux-core to [juxt/crux-core "21.04-1.16.0-beta"]

@seancorfield going through those issues you linked now… thank you for remembering this being discussed and sharing it.

side note, possibly related, all cider-connect-clj&cljs invocations produce this before my prompt appears; it hasn’t caused any practical problems so I have ignored it:

WARNING: CIDER 0.24.0 requires cider-nrepl 0.24.0, but you're currently using cider-nrepl 0.25.9. The version mismatch might b\
reak some functionality!
More information.WARNING: No Clojure project was detected. The
refactor-nrepl middleware was not enabled. (You can mute this
                                                warning by customizing `cljr-suppress-no-project-warning'.)

I seem to have a lil’ learning to do about version compatibility.

trying to upgrade Cider now.

while I’m going down this rabbit hole… well, it’s becoming a hippo hole… I’m actually just, atm, trying to store my reagent state and get it back later for the front-end… could I be taking another route to sidestep things for now? I have been having fun with the in memory Crux database but I have to persist it somehow.

is the up-to-date crux version not working for you? btw, still wondering about which tutorial you were looking at. think you meant to link something else?

it’s throwing the same error… I think @seancorfield has found an issue that may be tough for me to resolve now as I’m using shadow-cljs and he says ‘but if you’re working with Shadow-cljs you may not have much leeway’…

up-to-date crux version with the up-to-date docs, I mean :) https://opencrux.com/reference/rocksdb.html should point you to the 1.16 docs but here's the permalink https://opencrux.com/reference/21.04-1.16.0/rocksdb.html

the link I sent… if you go to the deps.edn tab in the ‘Project Dependency’ section it shows the dep version I used, I think…

ah, I mean the config format. it should look something like

(crux/start-node {:crux/index-store         {:kv-store                                       {:crux/module crux.rocksdb/->kv-store                                     :db-dir      "/tmp/crux/rocksdb-index"                                      ...}})

the config format changed considerably between 1.6 and 1.16

Ah! I understand now. Sorry for the confusion!http://confusion.NoNo, I didn’t get that from a tutorial… I was just trying to figure it out. I didn’t see a full example so was cobbling it together.

I’ll look at your example and try to tweak things.

oh, yeah sorry for the terminology confusion. I think of tutorials and documentation as different categories

it seems like a full example of what I’m creating would be handy in the crux rocksdb page, however.

yeah, I think the docs are missing a "quickstart" section as your use case I'm sure is pretty common. going off the docs I surmise the way you do this in clojure code is

(crux/start-node {:crux/index-store    {:kv-store {:crux/module 'crux.rocksdb/->kv-store                                                 :db-dir      "/tmp/indexes"}}                   :crux/document-store {:kv-store {:crux/module 'crux.rocksdb/->kv-store                                                 :db-dir      "/tmp/docs"}}                   :crux/tx-log         {:kv-store {:crux/module 'crux.rocksdb/->kv-store                                                 :db-dir      "/tmp/txlog"}}})

oh, actually there is a complete example here https://opencrux.com/reference/21.04-1.16.0/configuration.html#_sharing_modules_references```(crux/start-node {:my-rocksdb {:crux/module 'crux.rocksdb/->kv-store :db-dir (io/file "/tmp/rocksdb")} :crux/tx-log {:kv-store :my-rocksdb} :crux/document-store {:kv-store :my-rocksdb}})```

oh, yay!

I had glossed over that because I thought it was for a special case (no excuses though!).

you're right, the section is about a special case (where you want to share the same rocksdb folder). not sure why you'd ever want to do it tbh. I did just try the code I pasted above (2 messages up) and it works, starts a CruxNode and makes those rocksdb directories.

Oh fantastic. Yeah, your version has the index-store, as well, which theirs doesn’t, I don’t think.

oh, right, just noticed that.

so it isn't a complete example after all :)

yeah, I think the proposed quick start would walk a user through an in memory version, then right into the RocksDB version.

it would also be good to clarify for the uninformed like me that RocksDB is pulled right in as a dep… the jar mentioned made me think you would download a jar and run it.

but these things would be fairly obvious for most Clojure users without explanation.

so, um, do I have to create the directories in /tmp, or does it do that?

it should do it automatically

I could try it, but I have the other issue to work through first (the orchard issue @seancorfield mentioned).

I am highly motivated to figure it all out, that’s for sure. Crux is so cool.

it is! I'm pretty sure all you have to do is change the crux-core dependency version and you're good to go

it’s a huuuge deal for me, because I’ve been toiling through clj/cljs for ever (understatement alert) and finally have a grip on the basics… the editor and cider… and can make the front and back end and the persistence part is really the last thing.

@macrobartfast These are all little reasons that add up to why I don’t use nREPL/CIDER/Orchard at all 🙂

gotcha! if I recall you work at the repl directly?

are you one of… those?

Plain Socket REPL. VS Code + Clover for evaluation (+ Calva for everything else, with nREPL disabled). Previously it was Atom + Chlorine.

aha… ok, not a only-at-the-repl person.

We run Socket REPLs in many of our staging/production processes so I can connect VS Code to those processes and have the exact same UX.

I never type into the REPL 🙂

that is a cool approach, because VS Code is actually pretty awesome.

and VS Code jacked into running processes is also cool.

honestly, Emacs (in my hands) is a bit of a hazard for connecting to a running process.

I fixed a bug in production today by just eval’ing a new function defn 🙂

haha, that is so great! just be sure to make a note!

or maybe you have a way of committing it at the same time.

We find production REPLs really useful for debugging issues where we need to dig into the database state in production (not Crux, but still applies).

which is a huge time saver. I’m trying to imagine selling that approach to some of the managers at the places I’ve been around, haha…

you know, the places where it takes three weeks to get a VM provisioned for you.

We deploy to production multiple times a day quite often.

drum roll, please…..

WE HAVE A NODE!

it works.

you all rock so much. so grateful.

I have to say, my explorations with Crux so far have all been really great. (better than with Datomic) I wish we were in a position to introduce it at work and migrate some of our stuff to it.

I didn’t have Clojure figured out enough to jump on Datomic, so my timing is lucky.

this is my start with it all.

congrats! I think crux solves some really tricky problems for you, not least of which is keeping programming fun

oh man, I can’t tell you what the last 5 weeks has meant for me… I have spent a long time struggling through a lot of issues in Clojure, often with the help of Sean, actually, and just in the last five weeks it finally all clicked into place.

I actually got close to walking away from programming, but stuck with it.

and all of a sudden the front end fell into place, having worked through the back end stuff, and now Crux.

Programming is hard, and Clojure doubly so. Congrats @macrobartfast!

nice. not sure how much programming experience you have but clojure is imo a very difficult language to get started with. A lot of powerful, modular tooling, having to juggle 2-3 languages in one, weird concepts like lazy seqs. It's very fun once you get going, though, hopefully you find it to be the same

Thank you Sean. You and @dpsutton, with your help, are actually are the key players in my staying with it all.

yeah, @kevin842, this was in some ways my first language, because I although I started with Java I couldn’t really get into it because of the verbosity and lack of aesthetics… and JS was similar (though better)…

so I put my hopes on Clojure.

but, yeah, a bit of a steep wall to climb, haha.

I’ve been programming professionally for about 40 years now and Clojure is the most fun I’ve ever had — and seeing stuff like Crux now… well, I just wish I’d had Clojure + Crux decades ago!

heck yeah.

There’s that line from Bruce Hauman in his Flappy Birds Figwheel demo, where he says something like, “this is how programming should be, right?“.

Also, I saw a survey… Clojure programmers are the happiest.

I’m pretty damn happy right now.

“Clojure developers are the happiest - When Hermann looked at the use of words indicating happiness (e.g., “awesome,” “cool,” “fun”), comments made to posts in the Clojure subreddit came out on top. Lisp developers were a close second (not surprising, since Clojure is dialect of that language) followed by Scala, Ruby and JavaScript.”

When we introduced Clojure at work — to a team of CFML devs — they were fairly bewildered but after a while they said Clojure made them “happy” 🙂

Hey, New York Times has been using Kafka successfully in their publishing workflow: https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ I believe that adding bitemporality and immutability to the mix could be of great help. Simple caching, bitemporal queries for an article at a point in time. View the changelog for an article. Are there known uses of Crux for publishing media? Any writeups? Anything similar to that confluent article that gives insight into what value Crux could provide? Teodor

Just want to say thank you to whoever wrote the Crux-in-a-Box guide (https://opencrux.com/howto/in-a-box.html). This is helpful. A few weeks go I was trying to figure out what was the sort of "minimal viable production setup" and this looks like it.

Just caught up with all the new activity. Sorry for the confusion, Marco. It does sound like we should take "Crux-in-a-Box" and make a full-blown Quickstart. One of the challenges with that is the configurability of Crux. There are a lot of possible permutations (in memory, kafka + postgres, kafka + rocks, rocks for everything) which means the documentation either needs to be verbose, contain a lot of pointers (which isn't great in a Quickstart), or actually be dynamic, which would be neat but probably take too much time to build. If anyone has any strong opinions about what they'd like to see in the Quickstart, we're all ears. For the moment we can surface "Crux-in-a-Box" to that position but it would be great if we can service everybody somehow.

I'm still a bit worried about where Marco managed to find a configuration doc that references 19.12-1.6.0. :thinking_face:

On the far end (more or less the opposite of "quickstart"), we've published a bibliography if anyone is interested: https://twitter.com/juxtpro/status/1388173046586875913

Please excuse the lack of proper BibTeX for the time being. 😉

how hard would it be to get the valid time of docs indexed in lucene? I see the indexer fn only gets passed docs right now https://github.com/juxt/crux/blob/master/crux-lucene/src/crux/lucene/multi_field.clj#L20 use case is to score by vt recency

seems like ev https://github.com/juxt/crux/blob/master/crux-lucene/src/crux/lucene.clj#L245 may include a :tx key that I can just pass through

I would benefit from having a combined doc/tx structure though.. I have a hacky entity+tx function that returns a map with both, but iirc there was some temporality+ work going on that might subsume that?