@joshjones @noisesmith : I failed to clarify this in my original question, but noisesmith is right: I don't care about arbitrary modifications The sitaution is: I have some state (say a DB). I want to modify the state (say get next available counter). So I want a way to simultaneously modify the state + return some aux data. When other people are modifying the state/DB, I don't care.

@tbaldridge: again, I failed to mention this in orignal question, but that looks like it may be expensive if my data is an eav store with indexes 🙂

but swap! is implemented in terms of compare and swap

so it saves you the expense of a second atom or volatile

so I think I'm going to add an extra field:

(defn eav-new []
  (atom
    {:eav {}
    :ave {}
    :next-id 1
    :ret nil}))

https://clojuredocs.org/search?q=swap! <-- whoa, was not aware until now, that f may be called multiple times and need to be side effect free; unaware of this optimistic concurrency

Ick, don't do that please. Just use compare-and-set! Swap is just a wrapper for CAS. So write your own variant of swap that does what you need. Don't conflate the needs of a different return value with your data model.

@tbaldridge : if I do CAS wont I have to loop and/or decide how long to wait on contention ?

that seems highly non trivial to get the right performance on

Um...that's literally what swap! is: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Atom.java#L75-L87 And no, it's actually quite fast, CAS on a x86 machine compiles to a single CPU instruction. CAS is the underpinning of every single synchronization construct, locks, atoms, refs, vars, namespaces, etc. they all use CAS.

I've have a half baked idea relating to error messages. It's one of those "surely someone's done this" things but i'm finding it difficult to google.

Aim: see/map the range of errors a programmer may encounter as a way to measure how specific/friendly/nasty they are. This might provide data suitable to score for how good errors are. Something objective and which we can improve on.

Method: take a working program, randomly change one thing. compile and record if error occurs. run tests and see what errors occur. (vaguely inspired by the way spell checking happens but backwards)

"Change one thing" might mean a single character change (add/remove/replace/swap) or sexpr based (raise/barf/change brackets/swap). "Error score" might relate to specificity (NPE would score low) and proximity (did it report near the file/line/col we changed).

Thus ends my plea to the #lazyweb

Heh, I guess you could score your own program code on how error friendly it is with that technique. Adding specs and tests would improve your score (and thus developer happiness)

Hmm- I'm interested in generative testing, but...

I'm not sure how we'd measure that..

So for instance if an error message was all like "Your Momma so big when she sits around the house she sits around the house" we'd mark that as unfriendly? Or friendly?

(I have been tempted to return that as an error message, in cases of great stupidity, but have so far manfully resisted doing so.)

@tbaldridge : this part I don't understand: how can CAS be fast if there is a comparison involved?

@tbaldridge: if I give you two maps, how can you check whether they're equal or not with just one x86 instr ?

this seems like it should take worst case O(n) time

@qqq are you under the impression that your proccesor knows about maps?

@tagore: no; I'm saying clojure equality has to be expensive

I'm familiar with machine arch / have written x86 assembly before.

@qqq It's not at all easy to follow things through to the machine, but...

In the case of clojure, since data structures are immutable a simple pointer comparison is an equality check, right?

@qqq there are usually cpu ops for registers (x86 is CMPXCHG), but obviously when comparing something like two maps, this will not be a single op

@tagore @joshjones: I should clarify my question.

Does CAS just do pointer comparison or does it check for values?

i.e. as far as CAS is concerned, is [1 2 3] and (let [x 1 y 2 z 3] [z y z]) the same or different ?

because if they're the same, it seems like CAS equality checks can be very very expensive

especially if it's a large map and I'm only updating one key

I don't know, and that's why the stuff I write that absolutely has to be as fast as possible is written in C.

I'm not saying it's not a good question.

ne more thing, by "CAS" I mean clojure-CAS< not x86-cas

I am saying that I have written some software that must be as fast as possible, and I didn't use Clojure for it.

If you are concerned with perfomance...

First of all, how are going to measure and fix cache misses?

And how are you going top measure and fix branch misprediction?

Actually, I' just curious whether it's wort ase O(1) orworstcase O(n)

O(something) tends to obscure more than it reveals in many scenarios.

@qqq it's an (identical? ...) check

it's literally a pointer comparison so it's the same cost as doing an equality check on a 64bit int

@tbaldridge : that clarifies it; thanks!

I',m essentially unwilling to talk low-level performance without examples, because real performance questions are very specific, and have to do with looking at how cache is used, etc.

@qqq to flesh it out, the javadoc for AtomicReference.compareAndSet says:

Atomically sets the value to the given updated value
     * if the current value {@code ==} the expected value

If you're not down to the level of measuring branch mis-prediction and cache performance, you should likely be working on your algorithms instead.

@tagore read the conversation history and it will make a bit more sense

this all started because @qqq wanted different return semantics than swap! provided. My assertion was that writing your own loop and using compare-and-set! was not significantly slower than swap!

Hmm...

Have you measured?

I don't need to, I understand how the JVM works.

Swap calls CAS in a loop, I'm suggesting that @qqq call CAS in a loop. If anything his code might be faster due to the removal of the .apply in the middle of his loop, perhaps leading to better inlining. But in the end, any work being done inside the loop is going to dominate the execution time.

But you say "not significantly slower"

Hmm- I'l defer to your knowledge of the JVM, as I know little about it, but...

Especially, if any work is being done with PHM. Persistent collections are "fast", but way more expensive than any overhead of doing a CAS loop in Clojure.

That sort of ordering certainly matters when writing C, especially when cache is a consideration.

I can't say I've ever written anything for the JVM that had to be all that performant though.

Well one of the biggest problems with C is it doesn't have a memory model. So there isn't much to compare between the two.

Indeed- on the other hand that is one of C's greatest strengths 😉

heh, some would disagree. Lock free structures are really hard when the compiler is allowed to shoot you in the foot.

C is very good at being what it is. It's too bad so many people treat it as something else.

Hmm- where I want to lay things out very precisely in memory, in order to do lots of vector math and not incur any overhead, I'm hard-pressed to think of a reasonable alternative to C.

Especially if I want to measure lots of things about cache-misses, branch-mis-prediction, etc.

Well if you're interested in all of that, have you listened to talks by Martin Thompson?

Nope.

I've just spent a lot of time optimizing things to the bone.

https://www.youtube.com/watch?v=MC1EKLQ2Wmg

He talks a lot about this stuff and the vast majority of his work is on the JVM.

Ah, I see...

so anyway, good link for anyone here interested in high performance code.

but I think we may be a bit #off-topic now 🙂

I guess what I'd say is that I spent something like 6 months optimizing one particular algorithm for Intel CPUs.

I think that is sometimes worthwhile- but only rarely.

I like your link, but I'm familiar with what he's talking about- modern pipelined architectures are complicated, but if you want to get every cycle from them...

OTOH, I might be missing something here, but I don't see how you can do so if the JVM is in the way

That said, I think it is very rare that people need to be as fast as possible

And it's likely that, unless you are willing to put months into measuring things, a smart com;ier will produce faster code than you can.

But, if your code must run as fast as possible, humans are still able to contribute something, I think.

I can't write assembly anywhere near as efficient as a C compiler does, but if I'm willing to put months into optimizing things I can use my knowledge of the problem, the compiler, and the machine to produce something that is very efficient, and that relies on my understanding to be so.

This has to do, mainly with how things are laid out in cache, etc.

And how I avoid branching.

One amusing consequence of modern architectures is that caching calculated values is often a loss, where it would have been a win 20 years ago.

I'd look into some of Martins work, he delves into stuff like CPU pinning, sending messages between sockets in the patter best supported by the motherboard, RAM pinning, all on the JVM.

You can do most of what you describe on the JVM, it just takes some understanding of the machine

Perhaps- it seems easier to do it in C though, if you really care about cache misses and branch mis-prediction.

And it has been my experience that you either do or you don't.

I think it is really very rare that you ought to care,

And in most cases things that are slow are slow for entirely different reasons (bad string handling is a common suspect, and that's a long way from bare-metal)

Bad algorithms too...

Hi, I have an issue with an small application I am creating using the reloaded workflow, component and the reloaded.repl package. Basically, when I issue a (reset) function, which refreshes namespaces and re-starts the component again, old function definitions are still in place. The component includes an immutant web server whose ring-handler is created by a function. What I am seeing is that the function that creates this handler is not being refreshed despite of (reset) being issued. I am running all this from emacs (cider + scratch buffer). Any ideas where I might look at? Thanks in advance!

anyone used the debug interceptor in re-frame?

the output https://www.refheap.com/134063 doesn't look all too helpful to my eyes

"The output produced by clojure.data/diff can take some getting used to, but you should stick with it -- your effort will be rewarded."

Do you have https://github.com/binaryage/cljs-devtools installed ?

nope, I've been using firefox as my main browser

I'll set it up on chromium

The comment about taking time to get used to the output is related to the use of https://clojuredocs.org/clojure.data/diff

when showing you the difference between state before the event, and state after.

Yeah, having cljs-devtools installed is pretty essential for all clojurescript work

is anyone using weavejester’s new integrant library already ? i’m a bit unsure where to put my entire system config — since it’s now just data, should this perhaps be in resources/config.edn ?

@lmergen You can integrate it with aero if you're inclined 😉

@lmergen https://github.com/weavejester/integrant/issues/12 is of note to you 🙂

@dominicm that is exactly what i did!

or rather, i did this:

(defmethod aero/reader 'ref
  [{:keys [profile] :as opts} tag value]
  (ig/ref value))

i was just wondering whether that approach made sense, and/or whether it could be improved

@lmergen If you're calling it 'ref you're overriding aero's built-in ref!

yeah, you’re right, i should namespace it

I liked the name ig/ref 🙂. Short, but namespaced.

makes sense

How did I miss the Integrant lib? Randomly checking #clojure proves beneficial.

@yonatanel the trick is to channel everything straight to your brain instead of trying to filter 😁

What's a good blob store db to use with clojure? I'm lookin for something that would serve a similar role (need not be API compatible) as S3 or GCP/CloudStore

you can use s3 from clojure

I need to clarify: I need something that I can run locally. I want a blob store. It'll serve as similar role (need not be API compatible) with S3 / GCP/Cloudstore.

This is a high volume local setup. I don't want to deal with S3 bandwidth costs.

I would go for https://github.com/replikativ/konserve with the file store

@qqq so there's this thing called a file system... 😀

question about testing using lein -- i have this working, but am guessing there is a much better way: I have some setup/teardown code that must run before/after ALL tests, not just tests within a namespace. the goal is to run lein test and have it run the setup and teardown only once. so, use-fixture won't help. Using a simplified version of what's at https://stuartsierra.com/2016/05/19/fixtures-as-caches , I created a new namespace called my.app.setup-tests, and in there put a single (deftest ^:special-tag setup-runtests-teardown ... and in there I call my setup code, call run-all-tests, then call teardown code. I get the results of run-all-tests which will tell me if there were failures or errors, and then call (is true) or (is false) to propagate the errors/failures to originally calling lein test. In my project.clj, I add :test-selectors {:default (fn [m] (:special-tag m))} to have lein test only call my special deftest, and it works. Now, I'm sure there's an easier way. Thoughts?

there's some simplifications possible -for example, if a function invokes is, it becomes a success/failure if a deftest invokes it (even indirectly)

so you could have a "single test namespace" that actually loads test definitions from all the others, and has the apropriate fixtures

cool - how would I "load test definitions from all the others" ?

joshjones they are namespaces, require them, and call the functions

if the function invokes is, it shows up as a test case

excellent, this was what i was looking for -- i suppose i can get all functions in the namespace, and then see if they have :test as metadata on them, in order to know they're tests and call them. or is there an easier way you can think of? @noisesmith

I mean - that's what clojure.test itself is doing...

there might be something even simpler

for example, what about a leiningen plugin that loads are your test files (like clojure.test does) runs your setup, runs clojure.test/run-all-tests, then runs your tear-down?

then you don't need to do anything unusual in your tests - finding all files under test/ and then running clojure.test/run-all-tests is not all that complicated

well, i'm already calling run-all-tests in my current iteration

right, so why not just do the file-seq / load-file of your test directory too? seems simpler than the other options

sorry, i don't follow

you don't even need lein test at this point - call (file-seq "test/") and for everything that comes back with a name ending in ".clj" call load-file on it; then run your pre-test-hook, then call (clojure.test/run-all-tests) then call your post-test-hook

then your test namespaces don't have to be weird - they can be normal test namespaces

you avoid complex higher order code that tries to figure out how to construct tests

well, we have a travis script that calls lein test

then write a lein plugin that does this instead, it's like 15 lines of code max

and it's a lot simpler to make that than it is to make N indirect test namespaces with automatically generated code

correction: it's (file-seq ( "test")) - that gives all the eligible files, just select the ones that have names ending in clj

or you can do your other plan, I just think making an alternate test runner is simpler

alright, will give both a shot and see which is less dirty 🙂 thanks for your help

one more thing though -- if I do the lein plugin, can i hook into lein test and have it run?

it would be more like lein mytest

you would have replaced lein test altogether (in order to have a wrapper around it)

well, i need lein test to work

you can't ask CI to run lein mytest ? - lein test isn't magic, it just runs a file-seq, selects the clj files and loads them, then uses clojure.test

maybe so, but we're open-sourcing this at some point and really would rather have the standard commands work as people are familiar with them

make a PR so lein test accepts a global test wrapper 😄

(as an optional config of course)

it just seems like it's asking for trouble to do logic that should surround test running, from underneath inside the test runner

So, I find myself literally never using letfn, and instead always using let. Is that just me? I know about mutual recursion; I just can’t remember the last time I did mutual recursion that it wasn’t an explicit exercise in attempting to do mutual recursion...

@lvh I'm in the same place as you on letfn. I don't think I've ever needed to use it.

I guess maybe a recursive descent parser counts as mutually recursive, but I’m guessing I’d use toplevel production rules instead of a letfn. (Not to mention, for anything nontrivial that needs parsing: a parser generator.)

you never need to use it, it's just a convenience op, no? it does nothing you could not do in other ways, but it could in principle make your code more expressive? tons of stuff like this in clojure.

it specifically does something you can't do in other ways: it allows direct mutually recursive calls of anonymous functions

anonymous fns? you have to name your fns in letfn, no?

functions can be anonymous and named - anonymous as in not vars interned to a namespace

😄 maybe we need a different word to reflect what these functions really are

huh? if you can refer to then by name, who cares what the mech is? my pt is that the fnspec in letfn requires a name. it's not let*.

OK "it specifically does something you can't do other ways: it allows mutually recursive calls of first class functions"

it's the only thing in clojure that allows doing this directly without using deref or mutation of some sort

ok. but i guess that would count as other ways, obviously.

sure, if you allow indirect methods of doing something, nothing in cs is unique - it's the only thing in clojure that implements this functionality without hacks or the necessity of namespace level bindings

this has a historical importance, because common lispers complained about clojure not having generalized tail call optimization

letfn and trampoline both attempt to address this

sorry wait that's totally wrong

well my point was just that it is a convenience op. i don't see why thstvis controversial. can you give an example where use of letfn could not be replaced? iow it is not primitive.

beenthwredonethat

mobileink can you show an example of locally bound functions that mutually call one another without making a namespace level binding or using deref?

i asked you first! 😉 lemme thunk about it.

ok i thought about it. you can do that without using letfn?

that's something you can't do without letfn or def or mutables

and sometimes def or mutables aren't good solutions

ok, but i did not say "letfn or x or y or z". i said letfn. you've just acknowleged my point.

(ok, self nitpick, you could do it with a promise, which is not as bad as a mutable, but at least as ugly)

mobileink you could do everything with asm.java, you don't need anything else that is in clojure.jar

the thing is, it would suck

not being able to do an algorithm without def, sucks

I want to ask for some directional advice: I began working on a function that would convert the lwjgl javadocs into a data structure that describes the library with the intention of using that to generate a thin clojure wrapper over the library (with a few changes, like stripping out extraneous 'gl-' and 'm-' and other prefixes) but i've got a non-program problem i'm being faced with-- how should i package it? I had planned to make each class its own namespace; java doesn't have namespaces so it abused classes to be namespaces in lwjgl. C doesn't have namespaces, so it prepends junk to the front of each function name. Clojure has them, so I figured it would be wise to just use namespaces where lwjgl is using classes. However, if I do that then it naturally seems like each package (class container) would be a namespaces container (library). But there are over 30 packages in lwjgl. Should i just make 30ish libraries?

it might suck, but that's not televant. i mean relevant, but i'm kinda liking "televant" ;)

mobileink - so you're agreed that clojure could just ditch everything but asm.java and we'd be fine?

get a grip, dude! where the hell did that come from? all i said is that letfn is a convenience op.