Fork me on GitHub
#clojure
<
2021-09-13
>
West11:09:37

Crux/xtdb vs datomic? I'm thinking about learning a database. Of course I could learn SQL because it's everywhere, but I want to learn something I'll enjoy working with. I know they both use datalog for queries.

p-himik11:09:29

Just out of interest - why do you assume you won't enjoy working with SQL?

West11:09:37

Isn't there a lot of crufty stuff that are carried over from the past? Like, SQL commands are passed as strings so before you even start you need to sanitize all the inputs somehow. Or has this problem been largely solved? Also how easy is it to blow the entire database away?

West11:09:40

Just the fact that for Clojure you need a glue layer like HoneySQL or HugSQL is a little suspect (though I suppose you could do everything with str). Maybe I'm just psyching myself out? I might just be making databases bigger in my head than they actually are.

delaguardo12:09:03

You “can” work with relational databases using SQL without any extra layers. I think jdbc-next is a good abstract library to do that. HoneySQL and HugSQL are providing some tooling create queries from normal clojure data for you but they are completely optional (imho) And I wonder if you want to learn “databases” in general or how to work with some of them from clojure?

p-himik12:09:39

> Or has this problem been largely solved? Yes, by using the right tools. You can still shoot yourself in the foot if you compose SQL queries by concatenating strings of course, but virtually nobody does that. > Just the fact that for Clojure you need a glue layer like HoneySQL or HugSQL is a little suspect But why? This glue is exactly what makes it great. Sure, we could use e.g. React directly with ClojureScript. But it's much nicer to use e.g. Reagent.

Maravedis12:09:44

Also, learning SQL is never a waste of time, from a professional point of view. It's still omni-present almost everywhere. Knowing how to write it, how not to write it and what's the rationale behind relational databases are solid foundations to build upon, imho.

10
manutter5112:09:23

We use datomic at work, and while there’s things that datomic can do that SQL couldn’t equal, I still think that for day-to-day work I would much prefer to use HugSQL. Partly that’s because I have years of experience in SQL and have only been doing datomic for a few years, but mostly because there’s a lot of bread-and-butter stuff that SQL is fine for, and arguably better.

manutter5112:09:32

Mind you, datalog is kinda fun to learn and is definitely worth knowing. And don’t forget NoSQL db’s like Mongo (which we also use).

manutter5112:09:53

Each “flavor” has its uses.

p-himik12:09:15

To add a bit to the above - if you don't have good reasons to use something other than SQL, then you should use SQL. :) So many stories of projects deciding to switch to SQL even though initially they thought something else could potentially be better.

💯 2
West12:09:13

I guess I'll put it this way. If I wanted to learn something ubiquitous and well-established, I would've learned LAMP and Wordpress. Some crazy number of businesses still use it and there's a lot of work I could feasibly do as a freelancer. The reason I settled on Clojure was because of the experience working with it. I initially started learning Python (and little shell scripts before that), but the expressiveness, the REPL, and functional paradigm really made it a pleasure to use Clojure. I strive to create a super comfy workflow for myself in all my projects. It's worked alright so far. Maybe I'll just try all the major databases and see which one I like using the most.

West12:09:42

Of course I don't get that choice as a professional, but since I'm still a hobbyist, I think I'll stick with this approach.

p-himik12:09:11

> Maybe I'll just try all the major databases and see which one I like using the most That's a reasonable approach. It's hard to compare SQL RDMS to LAMP. :)

West12:09:39

Oh yeah, side-note. I was gonna mention Kafka, but it looks like crux is based on it anyway.

schmee12:09:46

whatever you end up doing, I can really recommend trying out Datascript. It lets you play around with Datalog and get a feel for it without all the hassle of setting up a DB.

West12:09:35

> whatever you end up doing, I can really recommend trying out Datascript. It lets you play around with Datalog and get a feel for it without all the hassle of setting up a DB. > I may decide to do that. I wonder why anyone would use datascript for any other reason though. The whole point of a DB is persistence right? Wouldn't an in-memory database kinda defeat the purpose? Why not just use an atom with a map as the db?

manutter5112:09:29

I don’t think any of us actually answered the original question--If I were new to datalog-based db’s, I think I’d try crux/xtdb just because it’s free and the bitemporal index idea is kinda cool.

👍 6
manutter5112:09:06

There’s a free version of datomic, but then you get into the slight differences between free/on-prem/cloud, etc.

p-himik12:09:32

> Why not just use an atom with a map as the db? Because you can't query a plain atom in the same way you can query the persistent DB. And with Datascript you can. Helps when you want to have the same queries on the frontend and on the backend.

schmee13:09:20

Datascript started out as a Clojurescript thing for managing state on the front-end, hence the “no persistence”. In-memory databases are useful for many reasons thought: you may want the query power of a DB, but you can fit the entire data set in memory for instance. or for dev environments where you don’t care about persistence etc

tatut13:09:24

I recommend learning all three: SQL, XTDB and Datomic... plus others as well

emccue13:09:25

@U2FRKM4TW > but virtually nobody does that. Maybe no one in this room, but i actually laughed at that one. I've seen way too much code like that

p-himik13:09:52

Yeah, I might've grown a bit biased - finally, something to blame this community for! :D

seancorfield18:09:01

Chiming in from the Pacific Coast (hence, a bit late): I would definitely say "learn SQL" even if you also learn XTDB and datalog queries -- SQL really is a fundamental IT skill that it is hard to avoid in the real world, and the pure relational model is interesting, powerful, and ubiquitous. Many non-SQL databases even offer a SQL-based API of sort some (XTDB does, and there's an entire reporting engine built on top of MongoDB to provide SQL-based access, for example).

👍 7
seancorfield18:09:58

At work, we had a fairly long love affair with MongoDB but ultimately migrated all that data back to MySQL for the ubiquitous convenience of SQL-based tooling.

seancorfield18:09:23

That said, if we were starting from scratch today -- instead of a decade ago -- we'd probably be pretty tempted to try to build our system with either Datomic Cloud or to use XTDB. I suspect we'd end up needing to support some sort of ad hoc SQL-like query functionality for the business team tho'...

emccue20:09:21

XTDB has the ability to do that sql query stuff - fwiw

seancorfield20:09:53

@U3JH98J4R As I said an hour ago "Many non-SQL databases even offer a SQL-based API of sort some (XTDB does" 🙂

emccue20:09:43

reading comprehension was never my suit

seancorfield20:09:09

It's a good message that warrants repetition 🙂

West01:09:04

I get the sense that databases are a bit like version control, in the sense that maybe there are better alternatives than git, maybe mercurial or svn does certain things better, but it’s everywhere and works great, and has great tooling. Not something I’m used to experiencing in the dev world, especially not the linux ricing world.

seancorfield01:09:50

Yeah, for some core subset of queries, inserts, and updates, most DBs are somewhat interchangeable. Where they tend to differ most is in the data types they support in tables.

xlfe10:09:08

If you've never done SQL, that's a good place to start I think. Then when you pickup a datalog variant (datomic, datascript, crux, datahike, etc - checkout https://clojurelog.github.io/) - you'll appreciate the difference 😄

indy09:09:19

I think what was missed in this thread is learning the basics like indexing, joins, how to think in relations, modelling many to one, many to many, one to one relationships, normalization and denormalization, different cons and benefits of using different kinds of databases. This knowledge is useful when working with most kind of databases. Indexing is by far the most important set of concepts in the database world. SQL, Datalog are query interfaces which is sort of orthogonal to the underlying database engines.

Michael Gardner18:09:40

How do y'all feel about returning non-booleans from predicates? I've always been fine with it, relying on Clojure's consistent & pervasive concept of truthiness. But I'm curious how others feel

Russell Mull18:09:43

It's idiomatic and powerful, if you do it tastefully. I find that embracing nil-punning means you don't have as many predicates in general, instead writing functions that return data to the best of their ability.

seancorfield18:09:40

If a predicate ends in ?, there is an expectation that it will return strictly true or false -- and the core Clojure team has made changes to newly-introduced functions over the years to preserve that idiom. If your "predicate" does not end in ?, it can return anything that can be treated as truthy/falsey. See this Clojure FAQ https://clojure.org/guides/faq#qmark_bang

2
Michael Gardner20:09:14

I've gone back & forth. Being strict about it often requires writing things in an awkward way, or at least wrapping the result in boolean. And I feel like Clojure isn't generally strict about concrete types— many functions don't specify what kind of collection they return, for example

Michael Gardner20:09:23

and you can specify in the docstring that you're only returning something truthy/falsey, but callers can still end up relying on the specific value just like relying on a fn returning, say, a vector instead of a list

Michael Gardner20:09:26

it's also tricky sometimes to come up with a good name without the extra contextual hint from ?

seancorfield20:09:51

In addition to the Clojure FAQ, the Clojure Style Guide also ties the ? to (strictly) boolean return values: https://guide.clojure.style/#naming-predicates -- and when core functions were added that ended in ? but did not return (strictly) boolean values, that was considered a bug and fixed. So, I think there's very strong precedent that if a function name ends in ? it should return only true or false.

seancorfield20:09:28

I would certainly reject code in a PR that had a ?-named predicate that returned anything but true or false at this point.

Michael Gardner21:09:57

I think the naming is a secondary concern, though. Even if you don't use ?, there's still the question of how strict to be about return types. The robustness principle is in tension with flexibility of implementation here, IMO: either you commit to a specific type preemptively, possibly over-specifying your "API", or else you risk callers relying on implementation details

Michael Gardner21:09:40

same with returning a specific collection type versus just "a collection" of unspecified type

luizmineo18:09:41

What would be the recommended way to handle a https://github.com/ptaoussanis/carmine connection pool with https://github.com/stuartsierra/component (or other DI library)? As far as I understood from the documentation, you just pass the server connection info to each call of car/wcar, and the pool is handled internally by carmine, so we can't explicitly create and close it, like we do with a jdbc datasource.

lukasz18:09:38

There's a GH issue for this - not sure if it got resolved, BUT I also looked at Carmine's code and looks like (not 100% sure) like there's an option to pass your own pool implementation

lukasz18:09:30

Incidentally - we have created a component for Redis, https://github.com/nomnom-insights/nomnom.omega-red - used in production, works 🆗

🙌 2
lukasz18:09:24

it's on my (long) todo list to investigate this closer

luizmineo18:09:30

Awesome, thank you! I'll take a look on omega-red

vemv18:09:48

This leaves me wondering if it would be much of a rabbit hole to use vanilla Java interop. A similar recipe is sometimes shared for Kafka. I'm using https://github.com/lerouxrgd/celtuce at work currently, it's fine but IMO it needs an extra push to be fully polished (see the one PR that I opened; I should get back to it as time allows)

lukasz19:09:39

I used Jedis via interop - it gets messy quickly because of varargs in java methods, not pleasant

schmee20:09:59

I saw this a few days ago, a new Clojure wrapper for Jedis: https://github.com/tolitius/obiwan

👀 6
🙌 4
schmee20:09:31

can’t say anything about this particular project but tolitius makes solid stuff 👌