Fork me on GitHub
#xtdb
<
2021-01-16
>
gabo01:01:29

disregard this message Hi, there 👋 I was following along https://juxt.pro/blog/crux-tutorial-delete Crux tutorial (about delete), but when I try to show the entity history with the :with-docs? option the evaluation doesn't finish. This part:

(crux/entity-history
 (crux/db node #inst "2116-01-01T09")
 :kaarlang/clients
 :desc
 {:with-docs? true})
I'm going to attach a repo case to this message. The crux version I'm using is "20.12-1.13.0-beta" If it's relevant this is my java version:
$ java --version
openjdk 11.0.9.1 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2)
And my Clojure CLI version:
clojure -Sdescribe
{:version "1.10.1.763"
 :config-files ["/usr/local/lib/clojure/deps.edn" "/home/gabo/.clojure/deps.edn" "deps.edn" ]
 :config-user "/home/gabo/.clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/usr/local/lib/clojure"
 :config-dir "/home/gabo/.clojure"
 :cache-dir ".cpcache"
 :force false
 :repro false
 :main-aliases ""
 :repl-aliases ""}

gabo01:01:00

Yeah, read the message above after posting facepalm Thanks for letting me know tho

gabo01:01:28

Sorry for the noise 😓 I read James's message just a few comments above a run it again with "dev-SNAPSHOT" and it works

refset11:01:58

It's not a problem at all, I'm more sorry you hit the issue without better warning - thanks for flagging up that it was causing you a headache! It's good motivation to get the next release out soon, or at least put a note on that section of the tutorial in the meantime 🙂

levitanong09:01:40

Hi all, some of my txs aren't getting committed (i'm using custom tx functions but this happens to regular puts, if memory serves) sometimes. What could possibly the cause for a transaction not getting committed?

jonpither09:01:20

Hi @U0BR5D7A6 txes should be committed, do you have some example code or a test case, or anything in the logs?

levitanong09:01:51

I'm still a bit new, so I don't know if you mean there's a crux-specific log. As for example code, it's a bit hard to nail down. It's nondeterministic as far as I can tell. One moment txes are working, the next moment they're not. In general, restarting a server is enough to make it work, but there are times when it just fixes itself after some time.

levitanong09:01:53

In all instances that this has happened though, queries are fine. It's just transactions that go wonky

jonpither09:01:28

By default txes are processed asynchronously, you need to call await-tx if you want to ensure a tx has been processed.

levitanong09:01:43

Alright, I'll try that! thanks!

levitanong09:01:47

What i've been doing up until now was a combination of submit-tx and sync. Isn't this supposed to be roughly the same as await-tx though?

jonpither09:01:19

sync will ensure a node is caught up at the time you call it, there's still a race condition. Await-tx for a specific tx is what you need.

jonpither09:01:37

Typing with thumbs!

levitanong09:01:22

Ah, I see! Thanks so much for the help! ❤️

👍 3
levitanong11:01:50

Hmm, so i'm still getting the issue even with await-tx, and it's also nondeterministic. @_@

👎 3
levitanong11:01:05

However, once i'm in a situation where a transaction doesn't get committed, no transaction will be committed

Steven Deobald11:01:26

@U0BR5D7A6 A sample test case that reproduces the issues (even inconsistently) and the version number of the Crux you're using would really help. This sounds pretty peculiar, so it might take more than a description to nail it down.

levitanong11:01:45

Alright, now that I know that there's no known reason that could easily explain this behavior, then I'll invest in a day or two of narrowing it down, then file an issue. 🙂

refset11:01:38

Hi 🙂 if you can share the Crux version and the config options you're passing to start-node in the meantime that may well help to triangulate the issue

levitanong12:01:36

Hallo! I'm on the dev-SNAPSHOT, and the config options are:

{:crux.jdbc/connection-pool {:dialect {:crux/module 'crux.jdbc.psql/->dialect}
                                       :db-spec {:jdbcUrl
                                                 (:jdbc-database-url config)}
                                       #_#_:pool-opts {}}
           :crux/tx-log {:crux/module 'crux.jdbc/->tx-log
                         :connection-pool :crux.jdbc/connection-pool}
           :crux/document-store {:crux/module 'crux.jdbc/->document-store
                                 :connection-pool :crux.jdbc/connection-pool}}

🙏 3
jonpither20:01:59

@U0BR5D7A6 thanks, do you have the code used for submitting txes?

levitanong03:01:23

I figured out the problem, and while I can't conclusively say this explains all the behaviors, this should pretty much be it. • A race condition was causing a custom-defined function to be sometimes present or absent. • (crux/submit-tx node [[:crux.tx/fn :any-custom-fn-that-doesnt-exist]]) will fail silently • The tx will of course not be committed. If the silent failure is a heretofore unknown issue, then i'll file an issue in the github with a repro repo. Details • I'm deploying on heroku, where I wanted to make sure I could save as much on database rows as possible. • I don't need historical versions of my custom functions, so I was evicting them before putting them in. • evict was taking almost 30 seconds even if it should only realistically be deleting one record per custom function • I was using await-tx elsewhere, but had neglected to use it in this situation. await-tx on the evict submission before the puts on the custom functions seems to have prevented the issue altogether.

Toyam Cox04:01:14

Why do you temporarily create functions? Seems like a strange use of the feature

levitanong05:01:09

It's a workaround for replace, rather than append. I've placed the puts of the custom functions in my application's initialization code. So if I don't put any cleanup code, the database will constantly be accumulating the history of the custom function.

levitanong05:01:05

And to my knowledge, there's no way to mark a specific entity as "do not store history"

Toyam Cox05:01:47

How about you do it at application bootstrap and check for the function being equal to itself before putting in a new one?

levitanong06:01:47

Do you mean the function not being equal to itself? By my understanding, match cannot do this, and so a custom function must be used, and we’re back to square one. :rolling_on_the_floor_laughing:

Toyam Cox06:01:22

Are you sure that a match can not do this? I would be surprised.

Toyam Cox06:01:33

Since after all, the function is just a list in the database.

Toyam Cox06:01:12

Alternately, you don't need to be atomic, do you? Just crux/entity your key and (=) the result

Toyam Cox06:01:36

That skips the transaction, and does your submission logic outside the app

levitanong08:01:42

My understanding of match is, it will make a transaction proceed IF it matches a certain value. I cannot make a transaction proceed if it doesn't match a certain value. In a way, it's a (when (= db-value match-value)) but what I need is (when (not= db-value match-value)). Yes, using crux/entity would serve the not= functionality that I need, but the broader problem is that I don't need historical version of a custom function at all. And even with the not=, over time this would accumulate history that I don't need, presumably as I change implementations. Having said that, is there any sort of ideological reason one shouldn't use evict for the purpose of replacing a document?

nivekuil11:01:21

I would think that evict, being a tx, still takes up space on the tx log, so you're better off looking before you leap if you want to save space. I think pruning history would better be done as some kind of periodic compaction job by some external process

😧 3
Toyam Cox19:01:23

Who cares if there is a history maintained of the past versions of the function? You are right match doesn't negative-match, but you can call entity and confirm the not= yourself.

levitanong06:01:34

I care when my project is hosted on heroku, and I have to pay extra when my database rows go over budget 😛

Toyam Cox23:01:06

Which it does anyway if you evict stuff.

levitanong03:01:45

Yup! Which is why I went with evicting the custom functions and putting them in at application bootstrap 😆