xtdb 2020-05-18 | Slack Archive

Akshay C. Gollapalli08:05:57

I built something with crux, that I think is pretty cool (early stage): https://github.com/acgollapalli/dataworks

👏 28

🙂 20

😲 8

malcolmsparks08:05:27

I really like your business case section. Very well written and some great ideas and points.

👍 8

🙂 4

Akshay C. Gollapalli16:05:20

Thanks!

Jorin10:05:13

The problem addressed with dataworks reminds me a lot of the original premise of the unbundled architecture, as described in https://www.confluent.io/blog/data-dichotomy-rethinking-the-way-we-treat-data-and-services/

👍 12

Akshay C. Gollapalli16:05:13

Thanks, this was a worhwhile read. Clarified what the unbundled architecture is for me.

Jorin16:05:09

Nice, glad to hear it was useful 🙂

ordnungswidrig13:05:55

Bring the code to the data is also the idea behing datomic ions.

👍 8

refset15:05:16

Yep, definitely, and it seems like a rapidly accelerating trend ahead of us. Interesting background on the broader context: 1. http://www.martingeddes.com/think-tank/five-reasons-moores-law-networks/ - why moving data around will always be the bottleneck 2. https://www.ericsson.com/en/blog/2020/2/distributed-compute-and-storage-technology-trend - how various levels of the hardware+network+software stack are likely to evolve > Compute units will be embedded either inside the memory or storage fabrics, opening up for near-memory computing

👍 12

Akshay C. Gollapalli16:05:49

Didn’t know datomic ions was a thing until after I wrote it. They seem pretty similar. Thanks for taking a look.

Akshay C. Gollapalli16:05:34

I can definitely see a trend, though those links might require another read through before I really get them. Thanks for the recs.

ordnungswidrig10:05:18

@US65YEL1M I think one important difference is that datomic leverages the whole AWS code packaging and deployment mashinery. I found it quite hard to get used to.

gusxavier19:05:46

Hey, folks! First of all, thanks for crux! I'm trying to do a query to select documents that have a field OR another. If I do something like:

'{:find  [u]
  :where [[u :user/email]
          [u :user/first-name]]}

I have users that have email AND first-name, right? How can I achieve the same but with OR? I saw in crux query specs that a ::or term exists but couldn't make it work.

Akshay C. Gollapalli19:05:28

I think this should work

'{:find  [u]
  :where [(or [u :user/email]
              [u :user/first-name]])}

gusxavier19:05:01

this was my first try. But then i get the following error:

Or requires same logic variables: [[:term [:triple {:e e, :a
   :user/email}]] [:term [:triple {:e e, :a :user/first-name}]]]

gusxavier19:05:05

Maybe it's only possible to use or with terms [e a v] and not [e a] but I'm not sure

Akshay C. Gollapalli19:05:05

Ah. You might try rules then. A rule can have multiple definitions. So you can have one definition that looks for the name and one that looks for the email, and just use the rule in the where clause.

gusxavier20:05:03

Cool. My next try would indeed work with rules but I asked to see if there was a simpler way

gusxavier20:05:12

Thank you so much for your help! 🙂

Jacob O'Bryant20:05:24

@UD6HD2C7N It works if you use or-join:

'{:find  [u]
  :where [(or-join [u]
            [u :user/email]
            [u :user/first-name])]}

❤️ 4

gusxavier20:05:29

Indeed, it does!

gusxavier20:05:34

Thanks for the help! 🙂

🙂 4

dvingo19:05:42

hi! yep if a logic variable (`u` in this case) is in a query then the query engine will return all documents that satisfy all the clauses. There's an overview here: https://opencrux.com/docs#queries_basic_query you can use or for this case. The docs mention this test file which has lots of query examples you can peruse. here are some or examples: https://github.com/juxt/crux/blob/master/crux-test/test/crux/query_test.clj#L451

👍 4

gusxavier19:05:22

Didn't know about this test files. Really good stuff in here. I'll check it! Thanks!

dvingo19:05:52

this site also really helped me get more comfortable with datalog: http://www.learndatalogtoday.org/

👍 8

gusxavier19:05:24

It's a great site, indeed! I used when learned datalog to work with datomic. Thanks for the tip!

dvingo22:05:02

I have a sole rocksdb node and I just ran a backup/restore successfully on my dev machine. My next question is how would you go about triggering the backup to run on a live instance? It's running alongside a webserver, so one thought is to expose an endpoint that triggers the backup with proper auth involved (just feels wrong to me, would rather not expose that functionality at all over the network). The other idea is to have a repl server run on app start and connect locally via a script. I'm wondering if there are other options.

refset10:05:46

Hey, another option is to initiate and push the backups periodically from within the app code, as demonstrated in this old k8s example: https://github.com/juxt/crux/blob/5991a94545d088690e7fc244b6a8be5587140ffc/docs/example/standalone_webservice/src/example_standalone_webservice/main.clj#L740

refset10:05:12

The parameters could even be fetched from a local or remote location each time and adjusted dynamically (to change the frequency, push target etc.). I can't really recommend a particular strategy, but I expect there are well-trodden path for apps that equivalently embed SQLite

dvingo14:05:54

Thanks Jeremy! seems like a great option to me

🙂 4

refset15:05:10

Excellent - I will be interested to hear how it works out! Just please don't take my suggestion here as strong advice or some kind of guarantee that it's definitely the best strategy 😅

refset15:05:38

And to reiterate a point I made a while ago: using crux-sql or crux-kafka is still likely to be safer and simpler in the long run, as you can rely on the well-established best-practices for those underlying systems (or the included backup services for hosted options)

dvingo19:05:14

hah no worries about that - I'm just looking for ideas! I may go with opening a socket repl just to learn how to do it. yep that makes sense to go with a proper stack eventually. I was just thinking through how to trigger a backup from within the running node and was curious of some options. thanks again

🙂 4

2020-05-18

Channels