go on

In production Clojure you will almost never see ref but you will see some atom instances.

For refs to work in transactions they need try locking and ordered ids

Which is more complicated then just an atomic compare and exchange

I see

Part of the reason refs are less common is that they target a use case somewhere in between "embarrassingly parallel" and amdahl's law. Even for the use cases where refs might apply, I think the fact that they're less commonly used makes it harder to apply them since there aren't many examples or references.

so basically just technical details with implementation of STM and atoms

I don't have the view personally that what you see in a typical production app is what you should aspire to. Sean is correct that it's common to not see refs or agents but of course that really doesn't say whether they're suitable to your problem space. When I read your question, I took it as you asking why there isn't some universal abstraction for concurrency primitives. Is that what you're asking?

it just seems that atoms and refs could be merged, but it seems just not the way that Clojure implements them all on the back-end

not that they can’t

just that they don’t

I just want to make sure I’m making the right call

with regards to merging them in jo_lisp

and I believe from this conversation that it is indeed the right call

What are the trade offs in making them distinct versus a shared abstraction?

none as far as I can see

thats the root question though

I feel like hiredman described a very real trade off.

yeah, the trade-off it seems is more of an implementation detail inside clojure than language design

I don't view it that way, personally. It is not an implementation detail but rather a very specific set of guarantees.

If you want an STM that does it all with a single concurrency primitive, you're going to need to describe why it's not great for X workloads, how to avoid contention, etc.

Atoms and stm have different use cases and make different trade-offs. For a similar example, while you could use lists as your only datastructure and forgo having maps and vectors, that's probably not a great idea in practice.

I think they are fundamentally diametric, in that if your program uses STM, you really don’t want Atoms anywhere, as any use of Atoms inside STM transactions will happen multiple times, so it seems that they are, at least for the most part, and either-or scenario

However,

if Atoms worked properly inside a STM transaction, then they could be intermixed freely without concern for using atoms inside a transaction - as thats OK and not a side effect

thus, I would conclude that if they could be worked together, they should be - to make a more coherent language design

however, if you have atoms working properly inside a STM transaction - then whats the point of refs?

and now we are back to my question 🙂

I feel like your supposition is that when you can unify two disparate guarantees you should, and that this is the hallmark of “good” language design. I don't agree.

You could make the same argument for unifying all data structures to be backed by a list implementation, but then you just end up with really slow maps and vectors.

your presuming you cannot, with good performance, merge atoms and refs

I am

cools 🙂 then we are on the same page then

if it can be done, with good perf, it should be

even if you could, the different types inform quite a bit for the reader of the code about how to use them. With an atom, there's no question whether you should make changes within a transaction or how transactions should be grouped. With refs, that's an important part of the design

there's no presumption of impossible, only highly improbable, speaking for myself. I hope that gives you motivation!

well I think its worth a shot

I may not be able to get there, but I think its worth trying

I'm all for an experiment.

One fundamental difference between atoms and stm is that stm keeps some history for readers and atoms do not. That will fundamentally change your memory and computational characteristics.

This thread was mostly just double checking my logic and making sure I wasn’t missing something dumb here

@U7RJTCH6J could you explain what you mean by history?

atoms inside a STM transaction would function exactly the same, from the transaction’s perspective

just that their changes get reverted if the TX needs to retry

(and shoved in the TX history buffer if I get your meaning to give consistent views of the atom from the outside)

and the inside

https://github.com/cgrand/megaref

This isn't exactly what you're talking about but it might be interesting given this conversation.

Though just to be clear, that description is awfully specific, and imo for good reason.

All the reference types have functions for set and update (eg. reset! and swap!). It's common to come to clojure and try to provide a reference agnostic interface. I'm sure there's a discussion somewhere that does a better job explaining why that hasn't been done.

@U036UA9LZSQ Just to clarify, you're creating an alternative "Clojure-like" Lisp but it won't have compatible semantics?

Ah, this is it, right: https://github.com/Zelex/jo_lisp/blob/main/README.md

Correct, initially intended to just rewrite clojure totally

I made some changes to IO and Atoms/STM

IO was lacking imo

I don’t know how people get by without more io

Because you don't like Java's I/O ecosystem?

I guess that’s what people do, yeah, just not use clojure itself to do IO and instead interact directly with Java classes

seems the only sane way anyway

since this isn’t hosted on Java, and instead on C/C++, I needed something more self contained

Clojure is designed as a hosted language -- on the JVM, on the JS engines, and on the CLR, and now on Dart.

I think there have been attempts to write Clojure-to-C in the past -- have you looked at prior art?

yeah

large work-in-progress, don’t use for any real purpose yet pls

too late, deploying to prod

doh!

I won't use it for anything if it doesn't have compatible semantics to Clojure on other platforms, don't worry 🙂

deal

I’m specifically trying to innovate in the concurrency/parallelism department (eventually)

the tradeoff between atoms and refs is perf. atoms are half implemented in hardware and super fast particularly if uncontended. refs give you coordination, but are way more complicated.

an alex miller appears

I can sort of imagine a Clojure implementation not implementing ref at all, but implementing atom using as close to the metal features as possible.

that's what atoms are now

@U064X3EF3 what do you mean half-implemented in hardware?

I can't imagine the opposite approach -- trying to make atom work in the larger, coordinated, more complex (and slower) context of ref...

compare-and-swap under the hood right? So machine code level for the JVM.

clojure atoms rely on java Atomic objects, which rely on spin locks implemented in Java intrinsics that are really reaching hardware stuff almost directly

ah, where-as the C++ hosted language just like does the assembly commands to do it

I knew about the spin loop but didn't know it reached so deep.

we're just stealing existing the 1000 person years of effort Java has already done here

x86 uses

lock  cmpxchgq

so can leverage that directly

so does clojure :)

cools!

🙂

but refs are not that

no, they aren’t

the way I do it is something like

if(in_transaction) { do thing but store to TX buffer} else do normal CAS

roughly

and that if is way slower than the cas

its more complicated than that

not really

the if(in_transaction) should pretty much always be correctly predicted by the CPU, so speculative execution should take care of that

in fact, the cost might be almost entirely hidden

if there is any lesson to learn from Clojure's design, it's about taking things apart, this is just yet another case of it

except, in this case, I don’t think thats a benefit

it means the programmer has to choose between STM or Atoms, and not intermix, and god help you if you use libraries made by others

thus, I believe this is in this case a language flaw

and that is your design prerogative :)

I prefer that choice, personally.

I believe I could correctly argue that its a design flaw (a small one, but a design flaw none-the-less)

I'm not a big fan of floor wax + dessert topping scenarios.

Clojure in generally is a really great language

if this is the only thing I’m complaining about, thats really really good

you should hear me complain about C++ design choices

Just think about it a bit and keep an open mind 🙂 The purpose of STM in a language and how it interacts with everything in a consistent way is whats important to make programming simpler. (using a rich term)

"you should hear me complain about C++ design choices" -- would love to hear... after my eight years on the ANSI C++ Standards Committee (and three years as its secretary) 🙂

I look forward to seeing how it turns out. Making new choices is how we get new (and occasionally) better things.

@U036UA9LZSQ Do you have any docs describing the I/O approach you're taking/what functions you're providing as an alternative to Clojure's Java-based stuff? (and of course file I/O is different in cljs on Node.js and on the CLR etc because the underlying libraries are different)

I created #jo_lisp in case anyone is interested in following up on this -- seems worth a channel for folks who might want to "kick the tires" on jo

I think the main distinction is performance of atoms over refs. The other thing I can see is the programmer ergonomics, refs are just a whole lot more complicated. When do you use ensure, commute, alter? Atoms are simpler to understand, it'll retry the change until it wins the commit, it's eventually consistent, you use it with swap! to be able to atomically read something and update it. Or you use it with reset if you can get away with overwriting whatever is currently there. Also, they are more distant, are you sure your function isn't called in a transaction? Better not accidentally add IO or non-idempotent behavior since your function could be retried.

And in practice, I feel STMs are even slower than a lock, so you might as well just use locking around your two or three atoms when you care for that.

Having said that, if you could have atoms work as refs inside of transactions, so you'd use atoms most of the time, they'd be just as fast, and only if you want to atomically make changes to multiple atoms instead of locking you could put that in a dosync, and if arguably that's faster than locking, ya it be cool.

Question to you though @U036UA9LZSQ what happens if I'm calling reset! or swap! outside a transaction at the same time I'm also making use of the same atom inside a transaction?

Maybe a better Q for #jo_lisp at this point @U0K064KQV?

(this thread has gone way off-topic for #beginners at this point!)

I wonder if atoms could have an extra function like swap! that also takes a rollback function, and if the transaction could use that to rollback if it failed.and is retrying.

Oh sorry, ya, can you move a thread? Or should I repost, though my last comment was directed at Clojure actually, but ya this whole thread seems too advanced for beginners

No, we can't move threads here. Feel free to have that discussion in another channel, at this point. Or not. 🙂

(.getDeclaredMethod (.getClass e) "getFaultInfo" (into-array Class [])) from a bit of googling I have come up with that code as the translation.

thanks for the response @U04V70XH6, but the keys will be dynamic here , requirement is if the value of that key is collection, then ignore that

Ah, misunderstood your question. So, for a given hash map, you can do:

(reduce-kv (fn [m k v] (if (coll? v) m (assoc m k v))) {} my-map)

Then you can mapv that across your data:

(mapv #(reduce-kv (fn ...) %) data)

Instead of reduce-kv, you could do

(into {} (remove (comp coll? val)) my-map)

(def data  [{:person-name "john"
    :person-id 1234
    :person-address ["Holand"]}
    {:person-name "hari"
    :person-id 3456
    :person-address ["NY"]}])
#'user/data
user=> (mapv #(into {} (remove (comp coll? val)) %) data)
[{:person-name "john", :person-id 1234} {:person-name "hari", :person-id 3456}]

yeah that worked! but (remove (comp coll? val)) wanted to understand how keys are also getting removed here?

Thanks for the help @U04V70XH6, Will check your answer tomorrow and respond, Going to the bed now!

When you apply a sequence function to a hash map, you get a sequence of MapEntrys that look like 2-element vectors. key and val are the functions that pull the key and value respectively from a MapEntry. (comp coll? val) is an anonymous function equivalent to #(coll? (val %)) so it tests if the value in the MapEntry is a collection. So remove leaves just the MapEntrys where the value is not a collection, and then into {} "pours" that sequence of MapEntry back into a hash map.

i see, Thank you for your detailed explanation, This will help to explore more on the hashmap