Fork me on GitHub
#clojure
<
2020-12-01
>
lilactown01:12:49

When would I use unsynchronized-mutable over volatile-mutable in a deftype?

noisesmith01:12:00

the only reason I can imagine is if you know the code is single threaded in all cases and need the performance boost of not propagating the value

noisesmith01:12:26

I can't imagine that coming up much in normal clojure code

noisesmith01:12:19

see "Java Concurrency In Practice" - best source I know of for what these terms imply design wise with the jvm

noisesmith01:12:21

the doc string says basically "if you don't know what these mean don't use them", for good reason

... Fields can be qualified
  with the metadata :volatile-mutable true or :unsynchronized-mutable
  true, at which point (set! afield aval) will be supported in method
  bodies. Note well that mutable fields are extremely difficult to use
  correctly, and are present only to facilitate the building of higher
  level constructs, such as Clojure's reference types, in Clojure
  itself. They are for experts only - if the semantics and
  implications of :volatile-mutable or :unsynchronized-mutable are not
  immediately apparent to you, you should not be using them.

lilactown01:12:30

Yeah I find that doc insulting tbh. I’m plenty experienced enough to burn myself and recover if I have a good explanation to fall back on

hiredman01:12:47

"I have a marvelous explanation of these features but this docstring is too narrow to contain them"

😆 9
lilactown01:12:59

I’ve been writing Clojure professionally for years but only touched Java concurrency primitives a handful of times. Understanding how they work in the context of Clojure’s higher level abstractions would be lovely

lilactown01:12:10

Anyway I’ll just litter my code with volatile-mutable to replace my CLJS mutable notations and hopefully someone will open a PR and explain why I shouldn’t of it’s wrong

hiredman01:12:09

Explaining volatile, even metaphorically, is tricky, it pulls in the java memory model, and now you need a metaphor for how you determine the order of reads and writes to memory in a concurrent program

hiredman02:12:32

Maybe, like, imagine you have a set of all the reads and writes a concurrent program does to some field. For the reads and writes that happen on the same thread you can establish an order easily, just the order the reads and writes in the order they occur in your program

hiredman02:12:34

For reads and writes across threads, you can't establish an order, any interleaving, any concurrent execution could happen

hiredman02:12:51

Volatile fields (and I encourage corrections) can be thought of as saying there is a total order of those reads and writes. It doesn't say what the order is, but there is some order

💯 3
hiredman02:12:20

So one place they are used is inside transducers that hold mutable state, so if you use that transducer in a go channel, where the state is passed the between threads, it ensures that when one thread manipulates the state it sees all the changes that the previous thread made

3
kenj02:12:21

I’m probably wrong, but I was under the impression you could explain volatile in terms of informing the low level bits that two threads might be reading/writing to a variable in memory, and as such, avoiding generating code to keep said variable in a CPU cache, where another thread running on another core, would not be able to read the latest value.

kenj02:12:59

In other words, always read/write from RAM.

jumar03:12:47

It doesn’t have to read from RAM. It just have to reflect the value wrby the other thread. The cache coherence protocol might ensure that without forcing cpu caches to be written to the main memory

hiredman02:12:33

That is an explanation of the observed behavior, but I don't believe that is the actual mechanics

hiredman02:12:44

The mechanics of the generated code depends on the memory model of the cpu architecture (I believe x86's is pretty loose say compared to arm's)

hiredman02:12:52

Maybe I have that reversed

hiredman02:12:18

http://gee.cs.oswego.edu/dl/jmm/cookbook.html is a decent place to start digging deeper

Alex Miller (Clojure team)02:12:20

with volatiles all threads will see writes (non volatile don't without some other synchronization that creates a happens-before constraint)

Alex Miller (Clojure team)02:12:44

with volatiles, reads and writes can't be atomic though - you can't read and then write assuming that the value hasn't changed

hiredman02:12:22

Which is problematic when people start sprinkling vswap! around thinking it's like swap! But better

Alex Miller (Clojure team)02:12:56

I would not recommend using :unsynchronized-mutable unless you have an external synchronization mechanism (like you are using locking around all calls to the field)

3
hiredman02:12:22

(vswap! Is not technically something you can use with a volatile mutable field, but clojure now exposes volatile two ways, and that is the other)

Alex Miller (Clojure team)02:12:04

there really is too much to say in that docstring other than "here be dragons, you're outside normal Clojure semantics". the real answer is to read JCIP and/or the JMM

cfleming04:12:58

I think a slightly edited version of what you wrote above: > with volatiles all threads will see writes (non volatile don’t without some other synchronization that creates a happens-before constraint) > with volatiles, reads and writes can’t be atomic though - you can’t read and then write assuming that the value hasn’t changed Would be a pretty good summary.

cfleming04:12:44

I read both JCIP and the JMM a long time ago, and just remembering basically that has served me for pretty much all uses of volatile since then - I’ve forgotten all the details about the mechanics.

lilactown03:12:58

I appreciate the explanation from both of you 🙂

manutter5113:12:54

We’re having an issue with our production web app that has me stumped. It’s a clojure app built with http-kit as the web server and connections to external datomic and mongo db’s. Occasionally the whole server will just hang for a minute or two, and the logs are showing the last operation before the hang was to dispatch a potentially long-running Mongo query. I can understand how that might lock up the thread that was dispatching the query, but how can it be locking up the whole server so that it doesn’t even respond to the heartbeat query from our monitoring system? It’s not even making more than a ripple on the CPU usage, according to monitoring on the host.

Alex Miller (Clojure team)13:12:02

From the description, could be either GC pause or that the thread blocked on the query is holding a lock that others are waiting on. For the former, I’d turn on the jvm gc logging - its designed to be run under load and you can shunt them off to their own log. If the latter or something else, getting a thread dump (kill -3 on the pid or use jstack) will tell you what threads are doing when it locks up and what monitors are held

manutter5113:12:35

Ok, thanks much. I hadn’t thought much about the possibility of a shared lock, so this plus the GC logging gives us something to pursue. This issue has happened 3 times that I know of since July, so data gathering is slow, but now we know where to spread the nets.

dominicm14:12:28

http-kit is built on core.async, so if you're locking up the core.async thread pool (e.g. doing I/O inside of a go), that could cause problems too.

dominicm14:12:39

I'm not sure how http-kit feels about I/O inside of handler functions.

Alex Miller (Clojure team)14:12:15

core.async now has a system property you can set to catch use of blocking async ops in go blocks. Might be worth turning that on in dev (not in prod!) to see if that’s happening

manutter5114:12:31

We’re not using core.async in the app itself, not sure if that matters.

manutter5114:12:41

Thanks for the link, I’ll definitely check that out

Alex Miller (Clojure team)14:12:39

Easy thing to check out at least

manutter5114:12:46

Oo, yes, this is going to come in very handy

borkdude14:12:20

@U09LZR36F What makes you think http-kit is based on core.async?

dominicm14:12:52

The haziness of memory 😁

dominicm14:12:25

That's got to be a lie, right 😁

borkdude14:12:43

No, http-kit is a light weight web server in Clojure + some Java without any deps

Alex Miller (Clojure team)14:12:59

admittedly that sounded weird to me too :)

borkdude14:12:19

@U06CM8C3V are you using the async part of httpkit?

dominicm14:12:33

http://http-kit.github.io/http-kit/org.httpkit.server.html#var-as-channel Hmm, I'd always thought the channels in http-kit were core.async channels. I guess I'm just crazy :)

borkdude14:12:43

@U06CM8C3V Maybe this issue will have some relevant info. Not sure: https://github.com/http-kit/http-kit/issues/345

borkdude14:12:32

Is the request or response body very large as in, would it be problematic to hold it in memory at once? This can also be a problem with http-kit

manutter5114:12:38

We’re not doing anything fancy with http-kit, just

(http/start
     (-> env
         (assoc :handler #'handler/app)
         (assoc :max-body max-body)
         (assoc :threads server-threads)
         (update :io-threads #(or % (* 2 (.availableProcessors (Runtime/getRuntime)))))
         (update :port #(or port %))))

manutter5114:12:21

Hmm, that’s actually calling start from luminus.http-server

manutter5114:12:58

which is basically calling http-kit/run-server, wrapped inside a try/catch

manutter5114:12:31

Yeah, looks like the response could be pretty large sometimes. :thinking_face:

borkdude14:12:28

http-kit doesn't have an optimal scenario for this. I have made notes about that here: https://github.com/borkdude/babashka/wiki/HTTP-client-and-server-considerations#http-kit-holds-entire-requestresponses-in-memory I might switch to jetty in the future for babashka because of this. Although for babashka scripts it may not be a dealbreaker, I don't feel comfortable about it.

borkdude15:12:10

Also see: https://github.com/http-kit/http-kit/issues/90 Again, not sure if this is your issue, but it could be.

manutter5115:12:07

This is really helpful, thanks much

manutter5115:12:32

Wait, I take back what I said about a large response — the code I’m looking at pulls potentially a lot of data out of Mongo, but then it batches it out to offline processing (via future) for a later download. The http response itself is basically empty.

borkdude15:12:12

ok well, that's not it then.

borkdude15:12:28

what version of http-kit are you using?

borkdude15:12:58

ok. not sure what the issue is and if it's related to http-kit then, sorry

manutter5115:12:55

np, I appreciate the input and I learned some stuff, so all good

dominicm15:12:34

What mongo client do you use?

dominicm15:12:20

It's probably worth getting metrics in place for JVM memory stats (heap, memory use, etc)

manutter5116:12:08

we’re at monger 3.5.0

manutter5117:12:43

That doesn’t seem to be our issue, it’s happening for us when we’re querying for data.

seancorfield18:12:52

@U06CM8C3V You won't get a lot of insight with New Relic on http-kit I'm afraid. We were using http-kit in production and we ended up switching (back) to Jetty because that has official support in New Relic and you get a lot more information.

seancorfield18:12:15

We worked with New Relic quite a bit to try to get http-kit supported but in the end they just considered it too niche to expend any effort. We tried configuring transaction recognition and wrote some middleware to help with transaction boundary recognition, but in the end we just gave up.

seancorfield18:12:37

All our apps are on Jetty now, except one built on Netty and we don't get a lot of the core web transaction metrics from New Relic on that either -- but we have a lot of custom metrics that we added ourselves, via New Relic's (now obsolete) metrics plugin library.

borkdude18:12:52

@U04V70XH6 out of curiosity, do you use ring-jetty or pedestal or ...?

seancorfield18:12:32

ring-jetty -- we try to keep things as "stock" as possible so we get the best out of New Relic.

manutter5118:12:34

@U04V70XH6 Wow, that's good to know, thanks for sharing that. I'll look into switching our app back to ring-jetty

rutledgepaulv18:12:15

+1 newrelic and ring jetty also worked well for us, never saw a reason to pursue alternative http servers. even if you're using websockets there's good support for that in the sunng87 jetty adapter

seancorfield18:12:22

We went with Netty for our websocket app because our frontend is JS and uses http://Socket.IO so we needed to support that on the backend and it's "easy" with netty-socketio (and we use that with Netty directly via Java interop).

manutter5113:12:55

I’m talking with the IT folks about installing some additional JMX based monitoring (New Relic) on the server, but I’m not sure what I should be looking for in the output.

Renzo Tomlinson18:12:22

anyone worked through Eric Normand's http://PurelyFunctional.tv series and can maybe shed a little light on a problem I'm working on? I'm on video LispCast - Intermediate Property-Based Testing with test.check - 3 Strategies for properties: generate the output and towards the end there are two functions lines and words that both fail on the input ["" ""] because the naive implementation of these functions use the built-in str/join and split functions which lose the empty string values. Eric gives some hints that we might want to slightly modify the generator in the defspec and that the solutions for the functions themselves would be recursive. I suppose I could just not allow the generator to produce empty strings, but I don't think that is the intention. My current line of thinking is to replace the empty strings with some kind of placeholder, but I don't really have any way to verify that this is the correct approach as I'm still trying to wrap my head around what it means to write a proper property-based test. Any guidance or insights would be greatly appreciated!

seancorfield19:12:11

I've been through that course but I don't recall the details of that example. When I get time later today I'll pull that lesson back up and see if I can provide guidance based on that.

Renzo Tomlinson19:12:16

@U04V70XH6 sounds great and looking forward to it. I'm really digging property-based testing so I'm hoping to learn as much as I can.

seancorfield19:12:33

The three courses are really good. Some of the Advanced PBT stuff is pretty mind-blowing.

Renzo Tomlinson19:12:16

I have Advanced PBT queued up next.

seancorfield21:12:56

OK, just re-watched that... The "recursive" part is there already: the generators for lines (and for words) are built on top of lower-level generators (not strictly recursive except in the sense that the generators call other generators). The main issue he leaves open to the reader is the question of whether the generated output is what you would expect from processing the input: and the empty lines (or words) case is quite a tricky one to think about. Consider this behavior of clojure.string/split-lines:

user=> (str/split-lines "\n\n")
[]
user=> (str/split-lines "\n \n")
["" " "]
user=> (str/split-lines "\n \n\n")
["" " "]
user=> (str/split-lines "\n \n\n ")
["" " " "" " "]
Is that really what you want from a lines function? If so, how do you formalize that such that you can correctly generate that type of output? And then, how would you turn that output into the appropriate input? I think the real problem here is that splitting lines is "lossy" by which I mean that there are multiple inputs that can generate the same output (see the middle two items).

seancorfield21:12:37

(a less charitable way to couch that is that he picked a really hard problem, ran into difficulty trying to specify the behavior, and punted on it 🙂 )

seancorfield21:12:31

But the good part about this -- and he says this in the lesson -- is that trying to generatively test the function immediately uncovered several bizarre corner cases that are very hard to reason about! @US0NPHU9E

Renzo Tomlinson22:12:52

I think the part I'm getting caught up on is determining what I actually want the lines function to do. Presumably I don't want it to be lossy since I lose the inverse property and this would also make it equivalent to the existing str/split-lines function. If lines is not lossy, then I suppose I'm checking before \n to determine if I should add an empty string?

Renzo Tomlinson22:12:06

hmm, okay. I think I'll try that out and see where I get. Thanks for running through that, @U04V70XH6. Hopefully I'll have a satisfactory answer soon.

seancorfield22:12:44

Yup, and that is exactly the hard part: should lines (or words) be lossy? Since words will throw away whitespace between items, it is definitely lossy: "foo bar" and "foo bar" must both yield ["foo" "bar"]

seancorfield22:12:55

And then the question is how you convert from the generated output to an appropriate input, since one output should really generate a random set of inputs that should all map back to that one output.

seancorfield22:12:51

And at that point, you'll be generating inputs from a seed output values (and now you're layering another set of generators on top of the two-or-more you already have).

Renzo Tomlinson04:12:18

i got lines solved. at least to a satisfactory answer 😄

Renzo Tomlinson04:12:13

i still need to do words but the process is making a little more sense and the flow of using the shrunken test input of a failing test to develop the code is coming together

Renzo Tomlinson04:12:54

i think i'm running into some memory issues with lein test-refresh or maybe with using quick-check in the repl but that's a different issue.

3
Renzo Tomlinson05:12:10

ah, i think it's because of recursion and i'm not spending any time on optimizing. because of the way clojure.test.check does its expansion and shrinkage, it probably isn't too friendly with recursive functions

hjrnunes18:12:42

Hi. I have a record that implements this interface:

public interface MyAsyncService extends Function<Object, Future<Result>>
[...]
(defrecord MyService
   MyAsyncService
   ...)
but when I try to pass it to a Java method that expects Function I get a java.lang.ClassCastException: class MyService cannot be cast to class java.util.function.Function Is this not something that can be done? Do I have to implement Function separatly?

hiredman18:12:58

are you sure Function in the interface refers to java.util.function.Function ?

hiredman18:12:30

and are you sure the source for the interface definition you are looking at matches the bytecode you are running (was the bytecode compiled from the same version, was the source changed and not recompiled, etc)

hjrnunes18:12:08

@hiredman Yep, just confirmed on both sides. But your second point is a good clue, I'll double check that. Thanks

hjrnunes18:12:17

@hiredman rebuilt everything cleanly and it's all good. Many thanks!

Michael19:12:27

I haven't seen any updates on spec.alpha2 / alpha.spec in a while. Is it still under active development? Trying to decide whether to embrace spec1, wait for spec2's release, or start using spec2 now despite its unreleased status

Alex Miller (Clojure team)19:12:04

spec 2 is still under active design. I would recommend using spec 1 for now.

Michael19:12:04

thanks. It might be too soon to ask, but are there plans for some kind of compatibility shim or tool to ease the spec1 -> spec2 transition?

Alex Miller (Clojure team)19:12:57

too early, but we will presumably have a guide, a tool, compatibility layer, something

Michael19:12:19

that's what I figured. Looking forward to getting my hands on spec 2 whenever it's ready!

seancorfield19:12:24

Spec 2 is still being worked on in design mode. Rich is thinking about function specs and how those should work (differently to what's currently in Spec 1 / Spec 2 probably).

seancorfield19:12:41

Spec 2 is definitely not ready to be used: it is quite buggy.

Michael19:12:30

gotcha, good to know.

vncz19:12:43

What is the best way to filter a vector based on the properties of multiple events?

vncz19:12:08

For example, take all the a and b whose f(a,b) holds true

dpsutton19:12:34

You’d need to filter the Cartesian product

vncz19:12:11

Yeah, compute the Cartesian product first I guess

dpsutton19:12:21

If you want to see an example there’s a lambda island screencast of the solution to the first advent of code problem which can benefit from this approach :)

vncz19:12:02

troll Ahah that's where the question originated

vncz19:12:16

This does not look cool, but it does the job.

pez22:12:58

This is not perfect, but anyway:

(for [x (butlast report)
      y (rest report)
      :when (= 2020 (+ x y))]
  (* x y))

vncz19:12:36

Right now I'm doing some sort of weird reduce

jaide23:12:37

Is there a function or macro to get the line\number file when called?

jjttjj23:12:11

(defmacro info [x]
  (let [f  *file*
        md (meta &form)]
    (assoc md :file f)))

(info "hi")
=> {:line 10, :column 8, :file "/path/to/x.clj"}

jaide00:12:29

Ooooh thanks! In a cross-lang discord someone mentioned how nice the dbg! macro\fn is in rust. I figured if we can get the file\line meta then the rest is a really easy macro 😄

dpsutton00:12:25

Rust will always have a file source location. This is not necessarily true when you’ve got a reader

jaide00:12:03

I think I understand. This may not supply expected info when eval’d from a REPL right?

dpsutton00:12:37

That, if you eval it from the source buffer, or just call read-string or read ever. Basically only things that use the line numbering pushback reader when clojure requires and compiles things will have this information

jaide00:12:09

I see, well sometimes is better than nothing.

dpsutton00:12:30

yeah totally

jaide00:12:18

Oh seems that *file* is not available in cljs?

jaide00:12:06

Found it. Gotta reference cljs.analyzer/*cljs-file*

jaide23:12:23

(defn my-func
  []
  (let [{:keys [file line]} (file-name-line-number)]
    (println file line)
    (do-something important)))

jaide00:12:18

Oh seems that *file* is not available in cljs?