Fork me on GitHub
#xtdb
<
2020-10-15
>
victorb10:10:57

hello juxt folks, is the crux-site repository not public or am I missing something else? Trying to build the documentation site locally, but the GitHub repository http://github.com/juxt/crux-site seems to either be private, or the submodule in .gitmodules is pointing to the wrong repository

refset10:10:28

Hi, yep it's private, so trying to build the docs locally won't be possible. Are you trying to add a new page or just update an existing adoc?

victorb10:10:35

no, wanted to add in some unsolicited styling changes, namely implement the media query prefers-color-scheme: dark so I can read the crux docs at night as well as during the day 😄

victorb10:10:23

(the media query would look at the users prefered setting for dark/light mode and serve a darker page if the user prefers it, otherwise it'll be just as the site is currently)

refset10:10:42

ah - good idea, and a generous thought! I'm a big fan of dark modes too 🙂 I'll have a chat with the team and see if there's some way to make such a collaboration possible

🙏 3
refset12:10:22

@UEJ5FMR6K hey, sorry to keep you waiting. If you still want to hack on a dark mode I just pushed this: https://github.com/crux-labs/crux-site-mock (hopefully it's enough! Also we can give you access to the crux-site repo as/when you want to create a PR)

victorb13:10:25

Cheers, I'll take a look once I have some spare cycles again :thumbsup:

parrot 3
Bryan14:10:37

Hey everyone, what’s the general feeling about backend datastores? We’re running on GCE, managed Kafka and not expecting (stupid-large) scale. What would be your “default” choice for backend store in terms of index, doc store, and log?

refset15:10:36

Hey 🙂 there's definitely a lot of options available if you're prepared to look at the entire GCP landscape. For instance, I think Pub/Sub Lite could make pretty decent low-cost Kafka alternative for Crux. Firestore might also be a good choice for a tx-log if you can tolerate slightly higher transaction latencies. I expect Firestore or Cloud Storage (objects) would work well for a doc-store. For the index-store I think Rocks on Persistent Disk is a reasonable combination. As it happens I was reading last night about how good the latency for Persistent Disk is: https://medium.com/google-cloud/persistent-disks-and-replication-9b9412fd9565

refset15:10:45

Cloud SQL is probably the easiest thing to get going with though, to avoid writing new doc-store or tx-log modules in the first instance (which isn't hard, but could take a few days)

adamfeldman16:10:17

TIL Pub/Sub Lite retention is unlimited, unlike Pub/Sub. But it’s a zonal service, no replication across zones.

✔️ 3
refset16:10:25

It's rather confusing to call it "Lite" when the (implicitly) full-fat version doesn't support unlimited retention

👍 3
Bryan19:10:46

all super good info, really appreciate the perspective! @U01CSB0GMK6 ☝️

🙏 3
adamfeldman21:10:02

A few notes from when I last thought about Crux on GCP (these are thought experiments, I’ve not attempted to build anything yet): • Firestore has 2 modes – “Datastore” mode is the one intended for server-side access (and has much higher write throughput) • “native” firestore is for if you want to sync it down to a browser or mobile client. has write-throughput limitations. https://cloud.google.com/datastore/docs/firestore-or-datastore. • Firestore is available with multi-region replication – I thought that was cool (but you do pay for the increased cost, and latency) https://cloud.google.com/datastore/docs/locations. It’s my understanding that’s strongly consistent replication. • Cloud Spanner of course also does multi-region replication, but still has a decently-high entry-level cost (I think of it as a price floor) (and maybe has other complexities related to how writes work, but those may have since been resolved – there’s now some sort of google-created JDBC driver!). • Cloud Bigtable entry-level cost dropped earlier this year (IIRC). but it’s an eventually-consistent datastore…

adamfeldman21:10:55

(final note: my thought experiment started with the assumption that using GCP’s native, managed datastore services is a good idea!)

adamfeldman21:10:19

(okay, one more: Unlike S3, GCS is globally strongly consistent for many operations https://cloud.google.com/storage/docs/consistency)

ordnungswidrig09:10:51

One nice property of firestore is the support for realtime subscriptions.

ordnungswidrig09:10:58

I wonder if one could use a shared redis instance for the kv store. :thinking_face:

refset09:10:28

> I wonder if one could use a shared redis instance for the kv store Yep it can definitely work: https://github.com/crux-labs/crux-redis 🙂 My measurements showed it as only ~2-3x slower than RocksDB, despite the network hop, but it's a pretty expensive way to store data and there's a hard upper limit of 2^32 keys per zset (which equates to <1B triples) I discussed Redis at some length in the recent showcase https://twitter.com/juxtpro/status/1314612039390842880

❤️ 3
ordnungswidrig09:10:26

I’m still exploring how crux can be uses in a serverless case. So far my understanding is that you basically need nodes running continously to take advantage of locally stored KV.

refset23:10:22

I think running a shared Rocks on EFS with single writer node and many read-only secondary nodes all running in Lambda could work well. Pure speculation for the moment but we'll give it a shot soon

❤️ 3