@U6VPZS1EK this sounds quite similar to https://github.com/xtdb/xtdb/issues/1603, for which we have a PR going in to the upcoming 1.21.0 release - if it is, it should be fixed in beta3. Could you confirm your XT version?

Is your definition of put!/`save!` approximately just something like (defn put! [node doc] (xt/submit-tx node [[::xt/put doc]])) ? Or are you calling await-tx / sync in there also?

@U050V1N74 That’s fantastic! (Well, you know, fantastic that this could be a known issue around race condition and already fixed. Not fantastic, as we all know that this class of problem are some of the most difficult. 🙂 @taylor.jeremydavid Correct, I merely submitted :tx, but did not await/sync — in fact, when I did do an await or sync, it just hung, despite :tx-time being after the current time! I will try upgrading to beta3 in the next hour or so, and give it a try, and will keep you posted.

Thank you, much!!!! FWIW, this level of help/support is an utter delight — I so much appreciate it, and IMHO, bodes well for any commercial offering you choose to make around xtdb!

The table lock in https://github.com/xtdb/xtdb/pull/1707/files#diff-3666bdb5606704c7f8cfedda843dc0895c7f0f353e4716abac0a3619242c7072 seems like a pretty big tradeoff. But I suppose if you need more scalability, there's kafka then

Err… Now only 198 of the 245 records got written. Next run, only 187 records got written. This does seem like a multithreading issue… 🙂 Also, getting this strange error:

(doseq [m (take 15 meetings)]
    (zm/save! jdbc m))
Execution error (AbstractMethodError) at xtdb.node.XtdbNode/submit_tx_async (node.clj:221).
Receiver class xtdb.jdbc.JdbcTxLog does not define or inherit an implementation of the resolved method 'abstract java.lang.Object submit_tx(java.lang.Object, java.lang.Object)' of interface xtdb.db.TxLog.

(I originally was using 1.20.0, and the latest was using 1.21.0-beta3.)

>

Execution error (AbstractMethodError) at xtdb.node.XtdbNode/submit_tx_async (node.clj:221)

(@taylor.jeremydavid: I was just writing this up as an issue in GitHub… I pulled down 1.21.0-beta3 via deps.edn — are you suggesting I can resolve the issue somehow, or is this a build/release issue on your side?) 🙏

Hey @U6VPZS1EK - just making sure you see this: https://github.com/xtdb/xtdb/issues/1603#issuecomment-1097092135 - does that look/sound about right? I did this from memory (not by watching the recording), so maybe I'm missing something.

Really!!! I had no idea that you were required to realize/deref the futures?? I'll check this out. It would be a great answer if deref’ing is required — I'll definitely search around to find this requirement. (And if this is the case, I suddenly have fear of all the other places in my code where I didn't do this, either!!! 😂😱😱

Huh…. Not an authoritative source, but definitely making me think I have a fundamental misunderstanding of futures. :man-facepalming:

Thank you for ask this help!! I’ll keep you posted — I will attempt to reproduce tonight

Ah oops, I didn't mean to imply that it's required to deref futures in general, just that in order for your use-case to work I don't really see a way around it :thinking_face: that said, I'm definitely a future/threading-novice so I may be missing some obvious tricks 😅 I could imagine creating some mechanism (with core.async or whatever) with a decrementing atom counter, where the xt/sync only gets called (or some other code gets 'notified') after the last future succeeds and the count reaches 0 ...but I don't really know if that would actually be useful to you.

deref'ing the future in this case turns it into a no-op, though?

as in, you might as well not bother with the future

Well, if you deref it in the loop. But the idea would be to first create a seq of futures, and after that deref them all to make sure they are realized. But that is getting pretty hacky for just doing sideeffects in parallel. In general I recommend using https://github.com/clj-commons/claypoole for all parallel stuff in clojure, instead of the core functions. In these case using pdoseq would make sure that everything inside happens as soon as possible, no memory is used for the results, and the top level call will return only after all the tasks are done

Hey @U8ZQ1J1RR thanks for chiming in! There were a couple of comments on the issue that dug into the underlying problem/confusion some more...I guess Claypoole has some nice answers for that too :thinking_face: https://github.com/xtdb/xtdb/issues/1603#issuecomment-1097563892

Let's see

Ok, so a bit guessing here, but mysql throws when the parallel submissions cannot get a table lock at the same time, and they give up trying?

And that can in principle happen without any parallelism in the application, just having two nodes can collide? That's… ugh 😞. Sounds like a retry helpers are a critical, not optional feature in any code that is meant to scale.

And indeed with the new table locks, there's probably no benefit in local parallelism anyway since the SQL db is going to make everything sequential anyway

Hmh, actually I guess that even with just node, having a bog standard jetty might hit this problem since it does multithread the request handling

Perhaps worth mentioning in https://docs.xtdb.com/clients/clojure/#_submit_tx that submit-tx will not handle exceptions from underlying implementations, and that retrying is the caller's responsibility?

I haven't really checked the implementations, but can MySQL really throw when trying to get a single table locked, and it just reacts to some internal retries running out? Or does that only happen when really two different transactions get a lock on two (or more) tables in different order, this nobody ever getting all the locks to proceed?

I'm way over my head here since I don't use MySQL, but does "SELECT * FROM %s ORDER BY event_offset LIMIT 1 FOR UPDATE" lock every row in the table, since there is no where, or just the one since there's a limit? And if it does lock every row, does it do it in the same order? Can simply running two of those in parallel deadlock since the transactions start locking in different order and nobody gets every row?

https://dev.mysql.com/doc/refman/8.0/en/innodb-locking-reads.html > For index records the search encounters, locks the rows and any associated index entries sounds like it could be every index it hits, not just what it returns

@U6VPZS1EK Have you tested the same code with postgres backend? That uses an explicit table lock, and I don't see any other locking operations in that module, so I wonder if that will never deadlock

> Have you tested the same code with postgres backend? I've not got this all booted up enough to respond properly to your other excellent comments/questions just yet (i.e. I'd need to spend a few hours digging in!), but having spoken to Gene earlier in the week I know the answer to this question is 'no' 🙂 I also know he's not working on big or hugely important system that justifies running a separate non-MySQL db at this point

You can use * in the pull syntax https://docs.xtdb.com/language-reference/1.20.0/datalog-queries/#pull

But it seems not in a q(uery)?

Pull is used in the find binding in a q

Ah. Thanks. I didn't understand that the first time through.