Fork me on GitHub
#aleph
<
2022-07-07
>
Matthew Davidson (kingmob)07:07:59

Hi @pereira.cj. You have a few issues going on at once, but the main one is buffering pauses because it’s not needed. First, d/loop. This behavior seems expected to me. The only effect buffering should have here is to allow the websocket stream to accumulate some data before you start take!ing. If you open the ws and subscribe, but don’t do any take!s, the buffer will fill up. When you start the d/loop, it will print out all the buffered values very fast, and then slow down to the actual rate of incoming messages. The buffer is empty because it doesn’t need to buffer; messages are processed faster than they come in. Check the v values, they’re sequential, so no trades are being missed (Though, if you wait too long to start taking, the buffer may fill up, and messages will be missed then, but that’s not a bug). Otherwise d/loop seems to work for me. It goes into an infinite loop of printing trades as they come. I’m not seeing any disconnections. There might be something rate- or load-sensitive that I’m not seeing from my timezone. Another possibility to consider is network flakiness. The Bybits docs say that to prevent network timeouts, you should https://bybit-exchange.github.io/docs/spot/#t-heartbeat. See the heartbeats config option in Aleph for more on that (or send them in your own separate thread if you like). Second, consume-async. Your code has a bug. The anonymous fn literal #() has a few limitations, one of which is that it has to be a single valid form. You have to wrap in a do like you do inside the d/loop. Better still, switch to fn, #() is meant to be for short one-liners. Once I wrapped it in a do, consumer-async also works as expected, an inifinte loop of messages. Given the message sizes, I don’t think increasing the max frame parameters should have an effect here. I checked this with both 0.4.6 and the latest 0.5.0-rc2, same behavior in both cases.

Javier Pereira18:07:53

Thanks @U10EC98F5, your buffering explanation makes sense however once the initial buffer is cleared the d/loop stops printing messages eg: message! "{\"topic\":\"trade\",\"event\":\"sub\",\"params\":{\"symbol\":\"BTCUSDT\",\"binary\":\"false\",\"symbolName\":\"BTCUSDT\"},\"code\":\"0\",\"msg\":\"Success\"}" message! "{\"topic\":\"trade\",\"params\":{\"symbol\":\"BTCUSDT\",\"binary\":\"false\",\"symbolName\":\"BTCUSDT\"},\"data\":{\"v\":\"2290000000006808487\",\"t\":1657216191559,\"p\":\"20906.82\",\"q\":\"0.001099\",\"m\":true}}" #<Deferred@76fbe620: :not-delivered> I am aware of the heartbeat, the connection stays open long enough for now, I'll add it later. Yes I am new to clojure! thanks for the #() tip now the consume-async has the same behaviour as the d/loop i.e. no messages are ever printed after the buffer is cleared. Yeah i didn't think the max frame parameters would affect it but was trying everything! You got me thinking though, I am running in a container and using graalvm. Tried switching to the clojure image, it didn't help. The connection stays open unless you wait too long which is expected. Not really sure what to do next, I can sit at the repl and eval takes! forever and everything is fine until I use a d/loop or consume-async at which point the fun stops and nothing happens. Unless i s/close! the connection which prints the close message as expected.

Matthew Davidson (kingmob)02:07:04

Hmmm. First, I would try removing Graal and Docker from the situation entirely, just to simplify the problem. I'm not certain Graal, in particular, is 100% compatible with Aleph yet. My next guess is it's related to logging config somehow. It sounds like logging on other threads goes somewhere else; is there another file to check? If takes are available immediately, they can be processed on the same thread, but if not, a callback is scheduled and that's almost always run on a background thread. This might be one difference between the buffered up mesgs, and new mesgs. One way to test that is not to hunt around for logs, but to append mesgs to an atom as they come in. That won't be affected by logging config. If the number of messages keeps growing after the initial buffered messages, then you know it's just a logging problem.

Matthew Davidson (kingmob)02:07:24

One of the Graal issues I recall was related to statically-initialized threads being a problem for Graal. There may be some threads missing, which may or may not be fixed with a Graal command line param. Check the GitHub issues. But try removing Graal first 😄

Javier Pereira16:07:37

I tried your logging to an atom suggestion first and it works. Which means it's been working all along (with graal in a container) only the println was lost to the abyss. Just like the atom I would have thought stdout would be the same on any thread but clearly that is not the case. Thanks so much for your suggestions! I'll keep them in mind in case I encounter more resistance. Thanks again!👍

Matthew Davidson (kingmob)06:07:16

Logging in Java (and by extension, Clojure) is way more complicated than it ought to be. If you don’t want to learn the intricacies of logback.xml just yet, give the Timbre logging lib a try. One major issue with bare println is it’s not multi-thread friendly (one line might print in the middle of another line). Still, I can’t recall ever seeing println fail to display anything from other threads before. Exceptions getting swallowed by the default exception handler on other threads, yes, but never println. Not sure if that’s a Graal thing or not… Anyway, glad to hear it’s all working

😎 1
Matthew Davidson (kingmob)07:07:23

(Minor note, since you seem to be newish to Clojure: fn docstrings go before the param vector. The reverse still works, since it’s an unused string literal evaluation, but your docstring won’t be picked up)

Matthew Davidson (kingmob)08:07:54

https://clojars.org/aleph/versions/0.5.0-rc3 is out. This includes some minor changes to better handle callbacks running on non-Clojure threads, potentially without an app classloader in the chain.

❤️ 2
🎉 1
Matthew Davidson (kingmob)13:07:27

Dirigiste 1.0.2 is out. I know nobody uses it directly, but it would be very helpful if people could test out the new version with Aleph https://clojars.org/org.clj-commons/dirigiste/versions/1.0.2 It includes several changes from @rborer

🎉 5
Matthew Davidson (kingmob)13:07:19

Also, 1.0.1 includes changes from @vemv