Fork me on GitHub

what could one do to minimize read-after-write latency on the same node? I think lowering :poll-sleep-duration would do a lot -- is there a minimum/recommendation? I'm also curious what's in the way of a crux node writing to its own index, other than design elegance, as you have the doc and the tx from submit-tx already and that's all the indexer is polling for I think?


We haven't established a baseline minimum but I would be happy to help you figure it out for your setup. I expect Kafka should be able to cope fine with 5ms polling, possibly lower. The cost will be increased idle CPU usage and network traffic


I turned jdbc (postgres) down to 10ms polls and it looks like the idle cpu usage of my container went up ￱~￱3x (from very little). Not sure how that scales with db size but I'd think a shorter interval is probably ok.. 1s is quite long as a default


For JDBC, which is what this question relates to, the costs will definitely be higher given all the things an RDBMS has to do. We could definitely implement a non-polling solution, or help you do so. Since most (all?) JDBC backends offer LISTEN/NOTIFY functionality for "pushing" updates around. It would probably completely eliminate all the polling overheads. We just haven't made it a priority yet 🙂


thanks for all the offers of help :p don't think this is a priority as of yet, but I'm very excited to try out the lucene stuff in the works! I'm still figuring out what a good production stack would look like, probably giving redpanda (the new c++ kafka) a go next


Cool, sounds interesting! Which JDBC database are you working with? Postgres? If you want to play with Lucene now we already pushed a set of Release Candidate snapshots up to Clojars: And the docs:


yup, postgres. I'm sure a lot of special optimization can be done there, maybe making use of hstore for k/v and whatever timescaledb uses for the log

👌 3

oh, there's docs. very nice, will take a look

🙂 3

also, I think the docs and the code disagree on the default poll-sleep-duration; docs say 1 second, code says 100ms?


(reply deleted - I was looking at Kafka!)


oh 😄 I'll have another think in that case


Yes I think I agree the docs are out of sync with what the code is showing. I'll make sure we discuss & update it soon, thanks 🙂


I'm not sure you want to write it locally, as you'd be missing some documents but not others. Some of the log readers are real-time, they steam as soon as it hits the log.


ah yeah that would break crux's guarantee of tx ordering within a cluster. which tx log is real-time? I see both jdbc and kafka have the poll wait option (in kafka it's called poll-wait-duration instead of poll-sleep-duration?)


I think the rocks one might be. The redis one I wrote is streaming-with-timeout.


I dunno actually, the doc store can be eventually consistent, so maybe it would be in line with crux's semantics to have the doc for the local tx but not the docs for earlier ones?


this is the one right? redis seems like a much better fit than s3 stores for docs given the sizes involved but I wonder about the durability


That's doc store and also a log


Redis can be configured for good durability.


if it were durable enough then I was thinking redis could be used as a write-through cache in front of s3, and the indexers could pull from redis


trying out lucene, but I'm not sure how this is supposed to work. I see content in the lucene directory, but can't seem to get a query to return anything

;; returns stuff
(q {:find  '[?e]
    :where '[[?e :entry/title "Wednesday Poem"]]})

;; returns #{}
(q {:find  '[?e ?v ?s]
    :where '[[(text-search :entry/title "W*") [[?e ?v ?s]]]
             [?e :crux.db/id]]})


Weird. That looks to me like it should work. Did you update the Clojars coordinate for crux-core also?


yup, everything's on 20.11


maybe it was improperly indexed, actually. text-search does work when I add a new doc manually. although there is stuff that looks like my domain data in the index dir


Well I've managed to reproduce the issue locally. I think we may not have accounted for namespaced attributes properly and I can't see that we have a test to check them... Do you also see the problem for non-namespaced attributes?


ah no, my successful test case was just :foo :)


:foo/foo appears to be broken, yup


great, okay I will write a failing test and open an issue. Thanks for testing the module out - I'm very sorry you hit a stumbling block so soon. We will definitely get it fixed before the main release 🙂


no worries! all part of the fun. glad I could help

🙏 3