Fork me on GitHub
#xtdb
<
2021-02-16
>
nivekuil00:02:24

I thought I saw this mentioned at some point, but could maps used as :crux.db/id have their keys indexed as if they were in the document body itself? that would cut down on data duplication, where you want to use some keys in the document as essentially primary keys

👍 3
refset11:02:05

It's a fairly reasonable suggestion, I think, given the current data model. Would you like to open an issue to start a longer-term reflection? I am happy to do so also. The main caveat I can think of is how you would then handle attributes that appear in both the ID and at the top level but where the values don't match. (Is it an error / invalid doc? Should they be merged? etc.).

nivekuil14:02:05

sure I can make an issue. I think you would prefer the attribute declared at the top level, for 1. backwards compatibility and 2. declaring an attribute at the top level has only one meaning (indexing), while it has two in :crux.db/id (indexing and uniqueness)

👍 3
nivekuil00:02:05

maybe that would lead to overindexing in some cases though. but you could maintain the current behavior by nesting one deeper in :crux.db/id?

👍 3
Timofey Sitnikov21:02:25

Hi all, I am from Postgres world and really considering Crux, but it looks complicated to build. What would be the simplest stack to build local crux based db? It looks to me like you can do the SQLite standalone JDBC datastore, would there be an example repo of such setup?

refset22:02:35

Hi @timofey.sitnikov there are certainly many possible combinations to explore with Crux, but I think experimenting is quite straightforward once you get comfortable with how the configuration map works (supplied to crux.api/start-node). Asides from the config map you just need to make sure you have the necessary modules available on the classpath (in your deps.edn / project.clj), and of course connectivity to whatever external services like Kafka/JDBC/S3 you might want. If you'd rather consume Crux in the same fashion as Postgres, as a remote database via HTTP/JDBC, you can build and run your Crux node as described here: https://opencrux.com/blog/crux-build.html For SQLite + RocksDB, your basic config map could be:

{:crux.jdbc/connection-pool {:dialect 'crux.jdbc.h2/->dialect, :db-spec {:dbname "sqlite-dir"}}
 :crux/tx-log {:crux/module `crux.jdbc/->tx-log, :connection-pool :crux.jdbc/connection-pool}
 :crux/document-store {:crux/module `crux.jdbc/->document-store, :connection-pool :crux.jdbc/connection-pool}
 :crux/index-store {:kv-store {:crux/module crux.rocksdb/->kv-store :db-dir "rocksdb-indexes-dir"}}}

refset23:02:57

> To build the stack and learn how to use it, if I have 2 hours per day, how many days will I need? I'll respond here to avoid creating another thread, but assuming you're already comfortable with Clojure I'd imagine you would feel comfortable enough with Crux within 3-4 good hours at the REPL to rule it in or out. In reality the much bigger learning hurdle is the shift from SQL to Documents+Datalog for modelling & querying, and I would like to be as helpful as I can be on that front 🙂

😀 3
refset10:02:15

Sorry to generate another notification, @timofey.sitnikov - but I just want to confirm you've seen this thread...in case it helps.

Timofey Sitnikov11:02:24

@U899JBRPF, Thank you. I did, it looks to me like a week, to include refreshing myself with datalog, which I only learned in college years ago.

🙏 3
🤞 3
Steven Deobald22:02:07

@timofey.sitnikov If you are working in Clojure, you can have a look at this service to see an example which uses RocksDB for all three underlying stores (index, doc, & tx): https://github.com/pariyatti/kosa/blob/master/src/kutis/record.clj#L23 This is just a regular Crux consumer, not any sort of official documentation. Hopefully we'll have a document for the "smallest case" on-disk storage out soon. 🙂

6
Timofey Sitnikov22:02:54

Thank you @steven427. Struggling to make a decision, use Postgres that I know or learn something new that has a natural fit into Clojure world. To build the stack and learn how to use it, if I have 2 hours per day, how many days will I need?

Brian Abbott22:02:09

Hey guys - my meetup group is looking for someone to present on Crux - either this upcoming meetup (2/23) or, for March (3/23) -- would anyone be interested?

Steven Deobald22:02:26

I'll relay the question but I'm guessing one of us would be quite happy to take it up.

6
refset23:02:41

Hey @UFVRVT0L8 I've yet to join a Provo meetup live because they usually seem to run in the middle of the night (most JUXTers are UK-based) 🙂 is there flexibility on the time slot? Also, thanks for the offer! 🙏

Brian Abbott23:02:54

We could try to adjust since you guys are UK based. The consensus was to have someone close to the active codebase so, I think if we had to adjust the time that would be fine!

refset18:02:01

Hello again! I am keen to take you up on the offer for the 23rd March slot. Would starting at 2300 UTC be possible?

Brian Abbott22:02:21

(note - this is for Clojure Provo - next to London Clojurians, we are the most regular and well attended meetup focusing 100% on Clojure)

Steven Deobald22:02:49

@timofey.sitnikov I think you can run through some short tutorials to get a sense of the space, which will help you make a more informed decision: https://opencrux.com/blog/crux-command-line.html — 10 minutes to get a Crux node running without any code; run some queries with curl. https://opencrux.com/blog/crux-confluent-cloud.html — 5 minutes to get a Crux node running with Kafka on Confluent Cloud https://opencrux.com/tutorials/space-adventure.html — a longer (and more thorough) tutorial of Crux concepts ... still quite quick, though.

6
Steven Deobald22:02:40

There isn't a great deal to learning Crux. Datalog may feel unfamiliar at first, but especially EDN Datalog variants (like Crux has) are quite easy to understand if you're already familiar with Clojure. The act of actually getting Crux up and running only takes a few minutes.