Fork me on GitHub
#clojure
<
2022-12-13
>
siva01:12:13

will there be loom support in clojure?

hiredman01:12:39

What support do you think clojure would need?

seancorfield01:12:44

It depends what you mean by "loom support". You can use virtual threads with Clojure today via Java interop. Several people have posted examples in various threads.

🔥 1
seancorfield01:12:19

There have also been some examples posted of structured concurrency (I think that's the term they used), using the incubator features of JDK 19.

seancorfield01:12:54

If you search this Slack for virtual thread you should find quite a few examples from around mid-September onward @U7Z3PDA5A

Alex Miller (Clojure team)02:12:57

I think we will eventually do something with loom for core.async, not sure what else would be useful in core

🔥 1
didibus02:12:00

I'm thinking agent, ref and future could have their thread pool switched to virtual threads. I can't remember if you can already do that or not.

Alex Miller (Clojure team)02:12:15

I think generally you would not want to do that for many cases of agent and future (not sure how ref applies)

Alex Miller (Clojure team)02:12:03

for example, a future that is a background I/O probably needs to consume a real thread, but I think that's the kind of thing that will need to be explored

upvote 1
💯 1
Joshua Suskalo15:12:25

future for background IO is the exact kind of thing that a virtual thread would be good for since all core-java IO primitives will not pin the OS thread when they block.

💯 1
Alex Miller (Clojure team)16:12:45

yeah wasn't sure about that, haven't looked at the loom stuff in detail yet

👍 1
didibus21:12:35

It's the other way around, when you're doing heavy CPU compute you'd want a real thread, for all IO and very small CPU compute you'd benefit from a virtual thread.

Joshua Suskalo21:12:59

That ones a little more dubious. Using a real OS thread for CPU intensive stuff doesn't really gain you much? The OS scheduler already has long eviction times on all the commonly used OSes, and using a virtual thread means you're less likely to have to evict the entire core's cache for the data you're interacting with.

Joshua Suskalo21:12:07

At the moment the only things I can think of where you really genuinely want an OS thread is when you're using native code calling out to heavy computations or blocking IO in a way that will pin the underlying OS thread.

hiredman21:12:23

it sounds like virtual threads only yield the real thread on a "blocking" operation, so potentially spinning up a bunch of cpu bound work with blocking io on some other threadpool could lead to thread starvation

hiredman21:12:36

virtual thread starvation

hiredman21:12:56

similar to what you can do with go loops if they don't do any channel ops

hiredman22:12:34

loom is going to be a big sea change, lots of stubbed toes and mistakes figuring how best to use it, exciting possibilities, but I am not looking forward to all the mistakes

hiredman22:12:55

it does sound more like virtual thread scheduling is round locks, and blocking io happens to involve locks (otherwise it would be a nightmare passing stuff between threads), but it makes me wonder about alternative lock implementations (spin loops around an atomiclong or atomicreference) and if those will end up being a big no-no for mixing with virtual threads

Joshua Suskalo22:12:46

Not sure where you got the impression they only yield on blocking operations, my understanding after reading the JEP is that they can be stopped at any safepoint, which is any function return or sample point for a profiler.

Joshua Suskalo22:12:17

Maybe I misread though.

hiredman22:12:02

the email linked above leans pretty hard into saying it is blocking io (but then later talks about locks around blocking io)

hiredman22:12:07

apparently taking the monitor of an object will cause the virtual thread to be pinned to the platform thread

Joshua Suskalo22:12:25

monitors have been updated to work with it I believe

hiredman22:12:11

could be, the jep doc says it, and it says update 2022/12/07 11:20, so 🤷

Joshua Suskalo22:12:29

oh, hmm. I guess that's right then.

Joshua Suskalo22:12:44

I just remember seeing issues around monitors that got worked out, I guess "worked out" meant pinning the os thread

hiredman22:12:34

it'll be the wild west for a bit

ghadi23:12:11

IO, most devices in java.util.concurrent (e.g. locks)

ghadi23:12:04

eventually monitorenter/exit will participate, but not yet in the preview

ghadi23:12:31

that becomes relevant for LazySeqs ^, which do a monitorenter/exit around thunk realization

ghadi23:12:09

if your lazy seq calls a loom-friendly thunk, it will still pin the carrier thread

seancorfield23:12:18

That's good to know @U050ECB92 thank you!

hiredman23:12:21

I mean, they want to get rid of object monitors too

ghadi23:12:41

get rid of that limitation, yeah

ghadi23:12:05

the plan, last I heard, was to reimplement monitors in Java

seancorfield23:12:09

(I'm invested in this because we're about to update to JDK 19 at work and want to start using virtual threads -- don't we @U0NCTKEV8 🙂 )

hiredman23:12:22

they in the broad sense, my take reading jeps is if they(some people working on some jeps, not just loom) could undo the decision years ago for every object to have a monitor they would

hiredman23:12:13

so will loom support monitors first or will they go away (seems unlikely they would ever go away)

hiredman23:12:02

@U04V70XH6 I think I feel a little more apprehensive about the move, it is one thing to speculate about usage patterns based on reading the jep, another to use it

ghadi23:12:06

they won't go away for all objects, but there are already warning in the JDK on many classes saying: > This is a https://docs.oracle.com/en/java/javase/19/docs/api/java.base/java/lang/doc-files/ValueBased.html class; programmers should treat instances that are https://docs.oracle.com/en/java/javase/19/docs/api/java.base/java/time/LocalDate.html#equals(java.lang.Object) as interchangeable and should not use instances for synchronization, or unpredictable behavior may occur. For example, in a future release, synchronization may fail. The equals method should be used for comparisons.

ghadi23:12:43

whatever value classes end up being, they won't have identity so you can't synchronize on them

ghadi23:12:08

but lazyseqs are not value classes, nor will ever be

hiredman23:12:42

a lot of our concurrency at work is actually kind of implicit in things like jetty and netty's request handling

hiredman23:12:40

moving that to virtualthreads, and what changes that makes (do netty pipelines make the most sense for virtual threads?), dunno where that will end up

ghadi23:12:11

the lazyseqs are the thing I'm concerned about (but the concern should dissipate if monitors get loom support)

hiredman23:12:23

having you looked at moving them to a reentrantlock?

Alex Miller (Clojure team)23:12:44

we have a story to look at that for 1.12

Alex Miller (Clojure team)23:12:53

but I haven't yet, not sure if Ghadi did

Alex Miller (Clojure team)23:12:29

conceptually, does not seem hard to me (and should help to use the read-only side in the common case now)

ghadi23:12:26

not going to want to allocate a lock for this

ghadi23:12:51

but it's an option... there are ways to do this that only introduce a lock when contention is detected

ghadi00:12:29

doing nothing is always an option, because it's allegedly a temporary limitation

didibus01:12:36

Doesn't a heavy compute will consume the underlying threads? Loom doesn't have any preemption right now, so you would cause your other vthreads to starve as I understand. Go and Erlang do preemption based on some heuristics, Go will preempt after X amount of time, and Erlang after X amount of recursion and function call.

Joshua Suskalo01:12:05

yeah that's the conclusion we came to

didibus01:12:00

I saw they mentioned maybe adding preemption eventually. But currently it's still somewhat cooperative, in that it runs until a fixed yield point, which is IO basically.

didibus01:12:50

Once they add full preemption, then I think it could be argued you can just use vthread for everything.

didibus01:12:48

Oh, slack hadn't updated it seems haha, so I missed all the in-between convo

didibus01:12:38

Go followed a similar evolution. They started yielding on IO only. Then they added yields at all function call. Finally they added yielding anywhere and doing so after some fixed amount of time. I suspect we might see something similar in Java.

hiredman16:12:20

It is a bummer how slavish a reproduction of core.async that is, up to including core.asyncs resource management issues

hiredman18:12:14

I haven't seen much discussion about virtual threads as gc roots. It seems likely that they are considered roots just like real threads, which is a pretty significant difference from core.async go blocks

Joshua Suskalo18:12:32

yeah, they are gc roots

Joshua Suskalo18:12:39

they're supposed to be API-identical to OS threads

hiredman19:12:19

our chat backend at work (lots of core.async) ends up with kind of three big gc roots 1. the code that takes http data and routes it to a core.async channel 2. the code that routes redis pubsub data to a core.async channel 3. timeouts and I've done some work to eliminate 2 & 3 as gc roots by having them indirectly go through the reference that 1 keeps (sort of a weak reference channel). naively switching to virtual threads means new gc roots everywhere a go block is launched.

M J07:12:08

Where do I exactly use transactions for HugSQL? I would appreciate an example of a clojure.java.jdbc/with-db-transaction

p-himik07:12:58

You use with-db-transaction instead of with-db-connection when you need a transaction.

M J08:12:07

I still don't understan how to implement in in my function. This is what we have for example, currently. How do I use ir

(defn delete-products! [database-connection {:keys [workspace-id deleted-at product-ids-array]}]
  (let [products (map #(Integer/parseInt %) product-ids-array)]
    (dosync
       (first
        (sql-soft-delete-products!
          database-connection
          {:product-ids  products
           :workspace-id workspace-id
           :deleted-at   deleted-at}))
      (first
        (sql-update-workspace-components-products!
          database-connection
          {:product-ids products
           :deleted-at  deleted-at}))
      )))

p-himik08:12:10

Your function accepts the connection. So seems like you gotta establish the transaction outside of it.

M J08:12:18

This is the first call to the database-cnnection I see

(defn delete-multiple! [{{{:keys [workspace-id folder-id]} :path {:keys [asset-ids]} :body} :parameters
                         :keys                                                              [database-connection user]}]
  (if (= (:status (service/delete-multiple! database-connection
                                            {:workspace-id    workspace-id
                                             :asset-ids-array asset-ids}))
         :failure)
    (bad-request "")
    (do
      (.trigger pusher/my-pusher (str "selected-folder-" folder-id)
                "delete-multiple" {"folder-id" folder-id})
      (response ""))))

M J08:12:59

So where do I put it? Its not my code Im just being onboarded and new to hugsql in clojure

M J08:12:57

Does dosync alone handle different calls to database, where it exits if one fails, so I dont even need "with-db-transaction"? If so, it would make it easier

iperdomo08:12:10

if you need transaction at database level, I think you don't need the dosync call. dosync is for changing ref values - https://clojure.org/reference/refs

M J08:12:15

Okay, then how do I use transaction at database level? Say with the EXAMPLE ABOVE

iperdomo08:12:18

those (db.collection/,,,) functions are husql generated functions

iperdomo08:12:07

basically you wrap the body of your functions call, using jdbc/with-db-transaction where tx is used as connection parameter to your hugsql function calls

M J09:12:04

Its still not working, im obviously missing something. How do I wrap it here exactly please?

(defn delete-products! [database-connection {:keys [workspace-id deleted-at product-ids-array]}]
  (let [products (map #(Integer/parseInt %) product-ids-array)]
    (dosync
       (first
        (sql-soft-delete-products!
          database-connection
          {:product-ids  products
           :workspace-id workspace-id
           :deleted-at   deleted-at}))
      (first
        (sql-update-workspace-components-products!
          database-connection
          {:product-ids products
           :deleted-at  deleted-at}))
      )))

iperdomo09:12:33

your function gets a database-connection as parameter, you need to: 1) remove the dosync you're not modifying any ref 2) you need to use (clojure.java.jdbc/with-db-transaction [tx database-connection] ,,,)

iperdomo09:12:04

in your (sql-*) function calls, you change database-connection with tx

M J09:12:51

So I d that

(defn delete-products! [database-connection {:keys [workspace-id deleted-at product-ids-array]}]
  (clojure.java.jdbc/with-db-transaction [tx database-connection]
   (let [products (map #(Integer/parseInt %) product-ids-array)]

       (first
        (sql-soft-delete-products!
          tx
          {:product-ids  products
           :workspace-id workspace-id
           :deleted-at   deleted-at}))
      (first
        (sql-update-workspace-components-products!
          tx
          {:product-ids products
           :deleted-at  deleted-at}))
      ))
  )
No errors jump, but this is the API response
"No implementation of method: :get-level of protocol: #'clojure.java.jdbc/Connectable found for class: com.mchange.v2.c3p0.ComboPooledDataSource"
message
: 
"Unauthorized"

M J10:12:40

Any ideas?

iperdomo10:12:56

See https://clojure-doc.org/articles/ecosystem/java_jdbc/reusing_connections/#using-connection-pooling - it seems that your connection parameter is a c3p0 connection pool ... your db-spec should be {:datasource database-connection}

pavlosmelissinos13:12:53

Hello all, how do you handle JSON logs in the REPL during development? Do you have a dev-time-specific log appender/formatter that emits plain-text logs for human consumption or is there a way to leverage structured logs directly (e.g. something that directs logs to something like portal)? Assume a conventional logging implementation, such as timbre/logback/etc.

eskos13:12:17

Yep, I have logback-test.xml for local dev. And/or CI as well.

eskos13:12:00

That’s doubly helpful since I can locally run all log levels at debug or even lower without causing the actual envs to be super noisy.

pavlosmelissinos13:12:33

Hmm, you lose information if you do that though

pavlosmelissinos13:12:09

e.g. you can't query the logs if they're plain text, like you would with a JSON log aggregator in the cloud

eskos13:12:03

My local environment isn’t the cloud 🙂 I see your point though, but that’d be very project specific then - would need to run equivalent if not the same tools in full local dev as well.

pavlosmelissinos13:12:56

Thanks for sharing your experience, that's what I'll probably do in the short-term, I just feel it's wasted potential 🙂 What do I mean: if the local destination of the logs can handle structured payloads (like taps & portal does), I don't see why it can't be configured to show just the message by defaut and you'd have the option to expand it (portal already supports both of these if I'm not mistaken). I mean it's technically possible. Maybe I should ask there but I didn't want to make the question tool-specific (I don't use portal - yet), I'm definitely open to other suggestions!

eskos14:12:21

I have no idea, maybe they do, maybe they don’t. In theory the appender ecosystem of logback should work perfectly for that, but besides reading the project’s readme, I’ve never looked into portal either 🙂

🙂 1
practicalli-johnny17:12:43

I use mulog to capture events and it has many options for log publishing. Mostly I use the pretty console output. If there are a lot f logs I'll run Potemkin (usually in a docker container) and public to that, which provides a nice UI for logs. I also run the Repl process in a terminal and then connect to it from a editor, watching the logs in the terminal when relevant

👍 1
seancorfield17:12:00

I have code in my dev REPL startup that wires tools.logging into Portal -- so log messages show up there (in VS Code). We don't use structured logging (yet) but I would imagine something similar would work for however you are logging JSON data?

pavlosmelissinos17:12:57

I'm aware of mulog and it sounds very promising but it's not used in my team so it's hard to sell replacing the existing stack (I'd definitely consider it for personal projects though, it seems to be a more natural fit for structural logging in general) 🙂 potemkin and your script @U04V70XH6 look very relevant, I'll check those out, thanks a lot. 🙂

seancorfield17:12:21

I love having the logs flowing into Portal while I'm working -- saves a lot of context-switching -- but it can generate a lot of output pretty quickly at times so we tend to run in dev with more logging filters in place than we use in QA/production.

👍 1
pavlosmelissinos17:12:03

Makes sense, that's more or less what I had in mind. 🙂 I might have to take a look at Portal as well while I'm at it!

seancorfield18:12:47

Portal is awesome-sauce 💥

😄 1
pavlosmelissinos18:12:34

Wait, @U05254DQM did you mean to mention another library? https://github.com/clj-commons/potemkin (or maybe I misunderstood)

Annie Carlson15:12:32

Does anyone have profiling (I know this a confusing word to use) tools they like for leinigan to get a breakdown of what is taking time for starting the REPL, etc? I'm trying to figure out why starting the REPL is taking so long on a coworker's computer.

Ben Sless15:12:11

you can use visualvm for that

🙌 1
Ben Sless16:12:11

Is there a chance they're loading the application code from user.clj?

Annie Carlson16:12:00

Thanks, I'll look into it. The coworker is running the same repo with the same command in the same nix-shell setup, so I don't think it's related to what is being pulled into user.clj, but that's a good thought!

Ben Sless16:12:09

with lein there are global user.clj and profiles, too, without knowing exactly what's loaded and executed it's hard to opine

Annie Carlson16:12:54

It's true, worth a check! Its a fresh install on a new computer for someone with no Clojure experience, so I doubt he added anything to his global profiles, but always bette rto rule something out instead of assume.

Ben Sless16:12:36

you can also record the execution using JFR

jumar16:12:02

There’s also :verbose option for require

phill00:12:46

A broken NFS directory on the path, a dysfunctional IPv6 configuration, a spinning disk if you're accustomed to solid-state, lack of DNS caching, or a virus checker can make things slow. Crazy-high CPU utilization can suggest JVM thrashing with too low (express or implied) Xmx.

Ben Sless04:12:34

Visual VM can give you flags to add to the jvm so it will know how to connect to it once it starts

jumar06:12:22

A good list: https://clojurians.slack.com/archives/C03S1KBA2/p1670977426595109?thread_ts=1670946932.710529&amp;cid=C03S1KBA2 I would also add this https://ask.clojure.org/index.php/9357/how-do-you-diagnose-the-cause-of-slow-25min-app-startup-time?show=9361#a9361 > Buddy is "doing work" generating keys when the namespace is loaded, and that was draining the entropy on the virtual server, so a workaround was to call it like so: > time clojure -A:server -J-Djava.security.egd=file:/dev/urandom -e "(require 'riverdb.server :verbose)"

Ty18:12:26

How might one go about defining integration in clojure(script)? Mostly curious from an academic standpoint, not looking to find a library that does it for me or anything. I've seen the numeric sampling approach of actually calling the function. I'm more interested in something that would generate a function that itself is the integral of the original function, and itself is a reified function. Is this possible? How about the same for derivation?

Joshua Suskalo18:12:32

So you're more or less looking at a way to do an algebraic method? Or are you looking for some other mechanism? I've heard in the Julia community they've figured out how do do automatic differentiation with dual numbers using extensible numerics, but unfortunately that requires more legwork in Clojure because we don't have extensible numerics.

Joshua Suskalo18:12:10

If you want to do it algebraically you'd do so by wrapping the function in a macro which understands numeric code.

Joshua Suskalo18:12:52

If you want to do it with dual numbers you can either make a new set of numeric operations which support extensible numeric types, or to use a macro which does this automatically for a certain scope.

Joshua Suskalo18:12:13

Unfortunately regardless of the tack you take there's little you can do to have this work in an automated fashion for libraries you didn't write.

hiredman22:12:02

the email linked above leans pretty hard into saying it is blocking io (but then later talks about locks around blocking io)