Fork me on GitHub
#xtdb
<
2021-04-28
>
dumrat07:04:57

In crux tutorial, in the evict step (https://juxt.pro/blog/crux-tutorial-evict#continue), this code does not yield the result as suggested:

(crux/entity-history
 (crux/db node)
 :person/kaarlang
 :desc
 {:with-docs? true})
#_#_
=> [{:crux.tx/tx-time #inst "2020-06-18T16:11:03.410-00:00",
     :crux.tx/tx-id 2,
     :crux.db/valid-time #inst "2020-06-18T16:11:03.410-00:00",
     :crux.db/content-hash #crux/id "c3ad3191fff06083fedf3640b625566c02033a6b",
     :crux.db/doc
     #:crux.db{:id #crux/id "efe634523d6867a3c6e4089074adf29b07b45f43",
               :evicted? true}}]
Instead I get an empty vector. Is this due to some change after tutorial was written?

refset08:04:16

Hey @UC1DTFY1G I'll look into it this morning - we try to keep the tutorial consistent with API changes, and in fact we have some basic test coverage for this specific step here: https://github.com/juxt/crux/blob/master/crux-test/test/crux/space_tutorial_test.clj#L552 ...so perhaps that test needs improving 🙂 I'll report back!

Tomas Brejla17:04:53

Hi, I'm a crux noob who only just recently tried running parts of the crux tutorials on in-memory node. Everything seems to work fine on that in-memory node. But I wanted to continue using dedicated cruxdb node. So I used crux-builder and started a docker container, which uses crux.rocksdb for all its storage needs. I tried to perform the same basics (transactions, queries), this time using the "remote client API node" created via (crux/new-api-client ""). And very soon I got stuck in the point that I'm unable to retrieve my stored data when I use :in in my query. The same code works just fine when using in-memory node. Any hint what's going on here? Have I made some stupid mistake, or am I missing something obvious? Thank you for any ideas.

refset17:04:23

Hi - welcome! Thanks for writing up such a complete example 🙂 I will dig into this again in an hour or two...but looking at the code I think it's possible that some :in arguments to q don't currently get formatted for HTTP correctly: https://github.com/juxt/crux/blob/68550ac488b70a41f254c39eadafcbac68284612/crux-http-client/src/crux/remote_api_client.clj#L145-L146 (e.g. if (pr-str query) works then perhaps that next line should similarly be (pr-str (vec args)))

refset17:04:17

And sorry you hit a snag so early on in your Crux adventure, I hope you are able to work around it in the meantime

Tomas Brejla18:04:56

No problem and no hurry 😉 I just wonder.. is going trough the (crux/new-api-client path uncommon? If not, I'd expect many of the people must have hit similar issue before me.

Steven Deobald19:04:58

@U01LFP3LA6P If this is a real bug, that would tell us a lot about who is using new-api-client (which is to say, probably no one). In general, if you're already writing Clojure, most people tend to use Crux directly. I'm also quite happy with RocksDB as the underlying store for the project I'm working on. For that simple use case, the Rocks crux-in-a-box config works quite well: https://opencrux.com/howto/in-a-box.html

Tomas Brejla19:04:22

Yeah, that's why I'm asking whether it's common or uncommon path. I have never used any system where any "serious" database itself would be included directly inside the same "space" (process/jvm/...) as your actual running application. But it such paradigm shift is preferred in crux, I'll be more than happy to try to walk this path 🙂

Tomas Brejla19:04:15

It's just that after years and years of using databases which were "sitting in their own space", it's a bit hard to imagine how one would deploy (and scale) any non-trivial (when it comes to size) database directly "inside your app".

Tomas Brejla19:04:57

perhaps I should read more of the docs first and try to understand the general/preferred pattern of deployment

Steven Deobald19:04:00

Using the Crux library directly is certainly the more common route for Clojure programs, but Crux itself shouldn't dictate your process topology. The HTTP client/server is one option for separate processes. For the time-being, if you don't have a specific concern about in-process Crux, that's definitely the most hardened way of talking to Crux — and you can always change later. In the case of non-Clojure consumers, it can't dictate process topology even if we wanted it to, but we're still not 100% confident about the best route for the Go and JavaScript crowd. HTTP is, again, an option... but personally I've always felt weird about shoehorning HTTP into the role of database protocol.

Steven Deobald19:04:26

@U01LFP3LA6P Took me a minute to set it up, but I am seeing the same behaviour as you. I'm guessing the line @U899JBRPF pointed to is wonky in the client. Sorry about that! Hopefully in-process is alright for the time-being til it's patched.

👍 2
refset20:04:12

> that would tell us a lot about who is using `new-api-client` (which is to say, probably no one) a counterpoint here is that the ability to supply arguments via :in has only existed since the 1.13 release (23 Dec 2020), and if it is the root cause I've speculated about then I don't think it would show up when passing in simpler values like strings or numbers

Tomas Brejla20:04:18

> . For the time-being, if you don't have a specific concern about in-process Crux, that's definitely the most hardened way of talking to Crux No problem, I'm really just starting to look around, so in-memory, in-process db is more than fine for learning. It's just the curious second part of my brain, which is constantly asking questions and the constant feel of "local, in-memory = for learning/testing only", which might be totally not true in this case. > and you can always change later. Interesting.. If such moment comes, what options are there? I somehow thought that the direct alternative of local in-memory node is a remote node and using it via http client..

Steven Deobald20:04:06

@U01LFP3LA6P Well, whether your db is in-memory and whether it's in-process are separate concerns. I've never used an in-memory Crux node for anything, in-process or otherwise. The crux-in-a-box link I sent configures RocksDB for each of the 3 stores, in-process.

Tomas Brejla21:04:38

Oh, my bad.. I totally missed that link facepalm . Thank you so much, I'll look at it 🙇 .

Steven Deobald21:04:58

No worries! We're in the process of cleaning up the http://opencrux.com layout, which will hopefully make it easier to find things like this. 🙂

Tomas Brejla21:04:54

If I had more clojure apps (uberjars, each has its own jvm) and I needed them to access the same database (ie. they should all operate on the same set of data), how would I set up crux node(s)? Should each clojure jvm has its own crux node and they all use some central, shared store (for example some central sql db accesed via jdbc)?

👍 6
craftybones03:04:19

That’s a good question. I’d like to know that too

Tomas Brejla17:04:16

Defn podcast 64 with with James Henderson actually tuches this topic a bit starting at 1:05:30.

🙂 4
nivekuil20:04:24

yeah, the idea of crux is to have the indexes and query engine colocated with your application, so reads are really fast and scalable, while leveraging other external storage systems (kafka, jdbc etc.) as the golden stores. Hence the "unbundled" marketing

✔️ 4
nivekuil20:04:14

it's interesting because you can get the durability/avaliability/other operational characteristics of a wide variety of different systems, with a common, powerful query layer on top of all that

💯 8
refset10:04:28

^ I second all that, though I'd add that in the extreme there are downsides to blindly coupling your database entirely to the rest of your application monolith. Like it's probably still a good idea to tease apart the pieces of your application that can comfortably operate statelessly, otherwise your primary release cycle risks being slowed down by having to do db-level things, like rolling upgrades, rebuilding indexes etc.

refset10:04:44

In relation to the original issue, I've opened a PR which you'd be welcome to build and test - https://github.com/juxt/crux/pull/1507 The root cause was definitely more time-consuming to figure out and solve than I'd initially assumed 😅 But thanks for the report!

👏 4