Fork me on GitHub
#xtdb
<
2020-10-15
>
victorb10:10:57

hello juxt folks, is the crux-site repository not public or am I missing something else? Trying to build the documentation site locally, but the GitHub repository http://github.com/juxt/crux-site seems to either be private, or the submodule in .gitmodules is pointing to the wrong repository

refset10:10:28

Hi, yep it's private, so trying to build the docs locally won't be possible. Are you trying to add a new page or just update an existing adoc?

victorb10:10:35

no, wanted to add in some unsolicited styling changes, namely implement the media query prefers-color-scheme: dark so I can read the crux docs at night as well as during the day šŸ˜„

victorb10:10:23

(the media query would look at the users prefered setting for dark/light mode and serve a darker page if the user prefers it, otherwise it'll be just as the site is currently)

refset10:10:42

ah - good idea, and a generous thought! I'm a big fan of dark modes too šŸ™‚ I'll have a chat with the team and see if there's some way to make such a collaboration possible

šŸ™ 3
refset12:10:22

@UEJ5FMR6K hey, sorry to keep you waiting. If you still want to hack on a dark mode I just pushed this: https://github.com/crux-labs/crux-site-mock (hopefully it's enough! Also we can give you access to the crux-site repo as/when you want to create a PR)

victorb13:10:25

Cheers, I'll take a look once I have some spare cycles again :thumbsup:

parrot 3
Bryan14:10:37

Hey everyone, whatā€™s the general feeling about backend datastores? Weā€™re running on GCE, managed Kafka and not expecting (stupid-large) scale. What would be your ā€œdefaultā€ choice for backend store in terms of index, doc store, and log?

refset15:10:36

Hey šŸ™‚ there's definitely a lot of options available if you're prepared to look at the entire GCP landscape. For instance, I think Pub/Sub Lite could make pretty decent low-cost Kafka alternative for Crux. Firestore might also be a good choice for a tx-log if you can tolerate slightly higher transaction latencies. I expect Firestore or Cloud Storage (objects) would work well for a doc-store. For the index-store I think Rocks on Persistent Disk is a reasonable combination. As it happens I was reading last night about how good the latency for Persistent Disk is: https://medium.com/google-cloud/persistent-disks-and-replication-9b9412fd9565

refset15:10:45

Cloud SQL is probably the easiest thing to get going with though, to avoid writing new doc-store or tx-log modules in the first instance (which isn't hard, but could take a few days)

adamfeldman16:10:17

TIL Pub/Sub Lite retention is unlimited, unlike Pub/Sub. But itā€™s a zonal service, no replication across zones.

āœ”ļø 3
refset16:10:25

It's rather confusing to call it "Lite" when the (implicitly) full-fat version doesn't support unlimited retention

šŸ‘ 3
Bryan19:10:46

all super good info, really appreciate the perspective! @U01CSB0GMK6 ā˜ļø

šŸ™ 3
adamfeldman21:10:02

A few notes from when I last thought about Crux on GCP (these are thought experiments, Iā€™ve not attempted to build anything yet): ā€¢ Firestore has 2 modes ā€“ ā€œDatastoreā€ mode is the one intended for server-side access (and has much higher write throughput) ā€¢ ā€œnativeā€ firestore is for if you want to sync it down to a browser or mobile client. has write-throughput limitations. https://cloud.google.com/datastore/docs/firestore-or-datastore. ā€¢ Firestore is available with multi-region replication ā€“ I thought that was cool (but you do pay for the increased cost, and latency) https://cloud.google.com/datastore/docs/locations. Itā€™s my understanding thatā€™s strongly consistent replication. ā€¢ Cloud Spanner of course also does multi-region replication, but still has a decently-high entry-level cost (I think of it as a price floor) (and maybe has other complexities related to how writes work, but those may have since been resolved ā€“ thereā€™s now some sort of google-created JDBC driver!). ā€¢ Cloud Bigtable entry-level cost dropped earlier this year (IIRC). but itā€™s an eventually-consistent datastoreā€¦

adamfeldman21:10:55

(final note: my thought experiment started with the assumption that using GCPā€™s native, managed datastore services is a good idea!)

adamfeldman21:10:19

(okay, one more: Unlike S3, GCS is globally strongly consistent for many operations https://cloud.google.com/storage/docs/consistency)

ordnungswidrig09:10:51

One nice property of firestore is the support for realtime subscriptions.

ordnungswidrig09:10:58

I wonder if one could use a shared redis instance for the kv store. :thinking_face:

refset09:10:28

> I wonder if one could use a shared redis instance for the kv store Yep it can definitely work: https://github.com/crux-labs/crux-redis šŸ™‚ My measurements showed it as only ~2-3x slower than RocksDB, despite the network hop, but it's a pretty expensive way to store data and there's a hard upper limit of 2^32 keys per zset (which equates to <1B triples) I discussed Redis at some length in the recent showcase https://twitter.com/juxtpro/status/1314612039390842880

ā¤ļø 3
ordnungswidrig09:10:26

Iā€™m still exploring how crux can be uses in a serverless case. So far my understanding is that you basically need nodes running continously to take advantage of locally stored KV.

refset23:10:22

I think running a shared Rocks on EFS with single writer node and many read-only secondary nodes all running in Lambda could work well. Pure speculation for the moment but we'll give it a shot soon

ā¤ļø 3