xtdb 2020-12-23 | Slack Archive

jarohen17:12:17

Hey all - it's been a while since our last release, we've got loads to tell you about in 20.12-1.13.0 🚀 In the headlines: * A much more consistent HTTP API, including JSON support (breaking changes for users hitting HTTP directly, users using the Crux HTTP client are unaffected) * A new Lucene module for full-text search 🔍 * Checkpointing for Crux nodes, so that new nodes added to a cluster become available to query much faster * Changes to the index structure to natively support more value types (BigDecimal/BigInteger/byte arrays/java.time), and improve ingest speed by ~10-15% - re-index required * And many more - in fact, we've closed ~80 issues/PRs in this release. Thanks to all involved for raising issues and testing release candidates 🙏 Full details in the release notes at https://github.com/juxt/crux/releases/tag/20.12-1.13.0. As always, if you've any issues, give us a shout on here. We hope you have a great Christmas, and all the best for 2021! James

🚀 42

dominicm22:12:55

#Also sent to the channel

The checkpointing is huge, that's such a great achievement. This makes using Crux for long-term applications far more viable. The introduction of lucene is a really nice boost too. Crux is really shaping up, nice :)

🙏 6

nivekuil21:12:22

I see a bunch of Query timed out. errors moving from dev-SNAPSHOT to 1.13, using all in-memory stuff (crux/start-node {:crux.http-server/server {:port 3000}}). was there a breaking change between these versions?

refset23:12:47

Hmm, nothing obviously relevant has changed that I'm aware of since the last RC release. How much memory does the JVM have available to it? And roughly how many documents are you talking about? You'll probably have a happier node if you switch to using crux-rocksdb, in any case.

nivekuil23:12:45

"-Xmx2G" with "-XX:+UseZGC" with ￱~￱5k docs being ingested on init. The in-memory stuff is handy since I can clear everything with just an integrant reload, though I could try bumping the heap a bit

nivekuil23:12:49

no luck on 4g, trying rocks for index/doc/tx

nivekuil23:12:17

same behavior with rocks. wonder what's going on

nivekuil23:12:38

I think it may have something to with eql/project

nivekuil23:12:16

(q '{:find [(eql/project ?e [*])]      :in   [?e]}    "this id does not exist")

nivekuil23:12:21

^ that seems to hang

nivekuil23:12:00

while it returns #{[{}]} on dev-SNAPSHOT

jarohen04:12:39

#Also sent to the channel

hey @kevin842, thanks for raising this. it affects projections where the projected var isn't bound by any other clause, and is caused by a change in how we treat missing docs from the doc-store. I've got a proposed fix, will see it through our CI and deploy a snapshot

jarohen04:12:53

all done 🙂

nivekuil06:12:08

it works!

dominicm22:12:55

replied to a thread:Hey all - it's been a while since our last release, we've got loads to tell you about in `20.12-1.13.0` :rocket: In the headlines: * A much more consistent HTTP API, including JSON support (breaking changes for users hitting HTTP directly, users using the Crux HTTP client are unaffected) * A new Lucene module for full-text search :mag: * Checkpointing for Crux nodes, so that new nodes added to a cluster become available to query much faster * Changes to the index structure to natively support more value types (BigDecimal/BigInteger/byte arrays/java.time), and improve ingest speed by ~10-15% - re-index required * And many more - in fact, we've closed ~80 issues/PRs in this release. Thanks to all involved for raising issues and testing release candidates :pray: Full details in the release notes at https://github.com/juxt/crux/releases/tag/20.12-1.13.0. As always, if you've any issues, give us a shout on here. We hope you have a great Christmas, and all the best for 2021! James

The checkpointing is huge, that's such a great achievement. This makes using Crux for long-term applications far more viable. The introduction of lucene is a really nice boost too. Crux is really shaping up, nice :)

🙏 6

jarohen04:12:39

replied to a thread:I see a bunch of `Query timed out.` errors moving from `dev-SNAPSHOT` to 1.13, using all in-memory stuff `(crux/start-node {:crux.http-server/server {:port 3000}})`. was there a breaking change between these versions?

hey @kevin842, thanks for raising this. it affects projections where the projected var isn't bound by any other clause, and is caused by a change in how we treat missing docs from the doc-store. I've got a proposed fix, will see it through our CI and deploy a snapshot

2020-12-23

Channels