This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-03-26
Channels
- # aws (4)
- # beginners (21)
- # boot (12)
- # cider (3)
- # cljs-dev (1)
- # cljsrn (10)
- # clojure (190)
- # clojure-nl (1)
- # clojure-russia (7)
- # clojure-spec (1)
- # clojure-sweden (9)
- # clojure-uk (30)
- # clojurebridge (1)
- # clojurescript (105)
- # cursive (4)
- # emacs (8)
- # jobs (1)
- # jobs-rus (4)
- # klipse (1)
- # luminus (5)
- # om (3)
- # onyx (2)
- # pedestal (5)
- # powderkeg (15)
- # re-frame (13)
- # reagent (1)
- # ring-swagger (5)
- # rum (5)
- # vim (8)
@joshjones @noisesmith : I failed to clarify this in my original question, but noisesmith is right: I don't care about arbitrary modifications The sitaution is: I have some state (say a DB). I want to modify the state (say get next available counter). So I want a way to simultaneously modify the state + return some aux data. When other people are modifying the state/DB, I don't care.
@tbaldridge: again, I failed to mention this in orignal question, but that looks like it may be expensive if my data is an eav store with indexes 🙂
but swap! is implemented in terms of compare and swap
so it saves you the expense of a second atom or volatile
so I think I'm going to add an extra field:
(defn eav-new []
(atom
{:eav {}
:ave {}
:next-id 1
:ret nil}))
then, when I use swap!, I put the ret value in :ret , and since swap! returns the value swapped in, I can just :ret it to get the return valuehttps://clojuredocs.org/search?q=swap! <-- whoa, was not aware until now, that f may be called multiple times and need to be side effect free; unaware of this optimistic concurrency
Ick, don't do that please. Just use compare-and-set! Swap is just a wrapper for CAS. So write your own variant of swap that does what you need. Don't conflate the needs of a different return value with your data model.
@tbaldridge : if I do CAS wont I have to loop and/or decide how long to wait on contention ?
Um...that's literally what swap! is: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Atom.java#L75-L87 And no, it's actually quite fast, CAS on a x86 machine compiles to a single CPU instruction. CAS is the underpinning of every single synchronization construct, locks, atoms, refs, vars, namespaces, etc. they all use CAS.
I've have a half baked idea relating to error messages. It's one of those "surely someone's done this" things but i'm finding it difficult to google.
Aim: see/map the range of errors a programmer may encounter as a way to measure how specific/friendly/nasty they are. This might provide data suitable to score for how good errors are. Something objective and which we can improve on.
Method: take a working program, randomly change one thing. compile and record if error occurs. run tests and see what errors occur. (vaguely inspired by the way spell checking happens but backwards)
"Change one thing" might mean a single character change (add/remove/replace/swap) or sexpr based (raise/barf/change brackets/swap). "Error score" might relate to specificity (NPE would score low) and proximity (did it report near the file/line/col we changed).
Sounds a bit like Mutation Testing https://en.wikipedia.org/wiki/Mutation_testing @U055DUUFS
Thanks Sean.
"Test suites are measured by the percentage of mutants that they kill." sounds like the plot to an X-men movie.
And even has a quote from The Watchmen "Who will guard the guards?"
Thus ends my plea to the #lazyweb
Heh, I guess you could score your own program code on how error friendly it is with that technique. Adding specs and tests would improve your score (and thus developer happiness)
So for instance if an error message was all like "Your Momma so big when she sits around the house she sits around the house" we'd mark that as unfriendly? Or friendly?
I think that falls foul of the Code of Conduct… you might want to delete it.
(I have been tempted to return that as an error message, in cases of great stupidity, but have so far manfully resisted doing so.)
@tbaldridge : this part I don't understand: how can CAS be fast if there is a comparison involved?
@tbaldridge: if I give you two maps, how can you check whether they're equal or not with just one x86 instr ?
In the case of clojure, since data structures are immutable a simple pointer comparison is an equality check, right?
@qqq there are usually cpu ops for registers (x86 is CMPXCHG), but obviously when comparing something like two maps, this will not be a single op
@tagore @joshjones: I should clarify my question.
i.e. as far as CAS is concerned, is [1 2 3] and (let [x 1 y 2 z 3] [z y z]) the same or different ?
because if they're the same, it seems like CAS equality checks can be very very expensive
I don't know, and that's why the stuff I write that absolutely has to be as fast as possible is written in C.
I am saying that I have written some software that must be as fast as possible, and I didn't use Clojure for it.
@qqq it's an (identical? ...) check
it's literally a pointer comparison so it's the same cost as doing an equality check on a 64bit int
@tbaldridge : that clarifies it; thanks!
I',m essentially unwilling to talk low-level performance without examples, because real performance questions are very specific, and have to do with looking at how cache is used, etc.
@qqq to flesh it out, the javadoc for AtomicReference.compareAndSet
says:
Atomically sets the value to the given updated value
* if the current value {@code ==} the expected value
... just confirming what tb said, yes, it is a pointer comparison (`==`), and if you follow that to the Unsafe
java class, it is a native
method, which would lend credence that it would use a cpu op made specifically for thatIf you're not down to the level of measuring branch mis-prediction and cache performance, you should likely be working on your algorithms instead.
@tagore read the conversation history and it will make a bit more sense
this all started because @qqq wanted different return semantics than swap! provided. My assertion was that writing your own loop and using compare-and-set!
was not significantly slower than swap!
I don't need to, I understand how the JVM works.
Swap calls CAS in a loop, I'm suggesting that @qqq call CAS in a loop. If anything his code might be faster due to the removal of the .apply
in the middle of his loop, perhaps leading to better inlining. But in the end, any work being done inside the loop is going to dominate the execution time.
Especially, if any work is being done with PHM. Persistent collections are "fast", but way more expensive than any overhead of doing a CAS loop in Clojure.
That sort of ordering certainly matters when writing C, especially when cache is a consideration.
I can't say I've ever written anything for the JVM that had to be all that performant though.
Well one of the biggest problems with C is it doesn't have a memory model. So there isn't much to compare between the two.
heh, some would disagree. Lock free structures are really hard when the compiler is allowed to shoot you in the foot.
C is very good at being what it is. It's too bad so many people treat it as something else.
Hmm- where I want to lay things out very precisely in memory, in order to do lots of vector math and not incur any overhead, I'm hard-pressed to think of a reasonable alternative to C.
Especially if I want to measure lots of things about cache-misses, branch-mis-prediction, etc.
Well if you're interested in all of that, have you listened to talks by Martin Thompson?
He talks a lot about this stuff and the vast majority of his work is on the JVM.
so anyway, good link for anyone here interested in high performance code.
but I think we may be a bit #off-topic now 🙂
I guess what I'd say is that I spent something like 6 months optimizing one particular algorithm for Intel CPUs.
I like your link, but I'm familiar with what he's talking about- modern pipelined architectures are complicated, but if you want to get every cycle from them...
OTOH, I might be missing something here, but I don't see how you can do so if the JVM is in the way
And it's likely that, unless you are willing to put months into measuring things, a smart com;ier will produce faster code than you can.
But, if your code must run as fast as possible, humans are still able to contribute something, I think.
I can't write assembly anywhere near as efficient as a C compiler does, but if I'm willing to put months into optimizing things I can use my knowledge of the problem, the compiler, and the machine to produce something that is very efficient, and that relies on my understanding to be so.
One amusing consequence of modern architectures is that caching calculated values is often a loss, where it would have been a win 20 years ago.
I'd look into some of Martins work, he delves into stuff like CPU pinning, sending messages between sockets in the patter best supported by the motherboard, RAM pinning, all on the JVM.
You can do most of what you describe on the JVM, it just takes some understanding of the machine
Perhaps- it seems easier to do it in C though, if you really care about cache misses and branch mis-prediction.
And in most cases things that are slow are slow for entirely different reasons (bad string handling is a common suspect, and that's a long way from bare-metal)
Hi,
I have an issue with an small application I am creating using the reloaded workflow, component and the reloaded.repl package. Basically, when I issue a (reset)
function, which refreshes namespaces and re-starts the component again, old function definitions are still in place. The component includes an immutant web server whose ring-handler is created by a function. What I am seeing is that the function that creates this handler is not being refreshed despite of (reset)
being issued.
I am running all this from emacs (cider + scratch buffer). Any ideas where I might look at?
Thanks in advance!
The idea with reloaded is that when you shut down your component, the web server should be stopped. The old definitions are literally destroyed - as in the namespace is deleted and a new one created. The problem here is that if the old code is still running, it's still using the old definitions, and as the old namespace was deleted and recreated, the new definition is something the running process will never find.
the output https://www.refheap.com/134063 doesn't look all too helpful to my eyes
"The output produced by clojure.data/diff can take some getting used to, but you should stick with it -- your effort will be rewarded."
Do you have https://github.com/binaryage/cljs-devtools installed ?
The comment about taking time to get used to the output is related to the use of https://clojuredocs.org/clojure.data/diff
when showing you the difference between state before the event, and state after.
Yeah, having cljs-devtools
installed is pretty essential for all clojurescript work
is anyone using weavejester’s new integrant
library already ? i’m a bit unsure where to put my entire system config — since it’s now just data, should this perhaps be in resources/config.edn
?
@lmergen https://github.com/weavejester/integrant/issues/12 is of note to you 🙂
or rather, i did this:
(defmethod aero/reader 'ref
[{:keys [profile] :as opts} tag value]
(ig/ref value))
i was just wondering whether that approach made sense, and/or whether it could be improved
@yonatanel the trick is to channel everything straight to your brain instead of trying to filter 😁
What's a good blob store db to use with clojure? I'm lookin for something that would serve a similar role (need not be API compatible) as S3 or GCP/CloudStore
you can use s3 from clojure
I need to clarify: I need something that I can run locally. I want a blob store. It'll serve as similar role (need not be API compatible) with S3 / GCP/Cloudstore.
I would go for https://github.com/replikativ/konserve with the file store
@qqq so there's this thing called a file system... 😀
question about testing using lein -- i have this working, but am guessing there is a much better way:
I have some setup/teardown code that must run before/after ALL tests, not just tests within a namespace. the goal is to run lein test
and have it run the setup and teardown only once.
so, use-fixture
won't help. Using a simplified version of what's at https://stuartsierra.com/2016/05/19/fixtures-as-caches , I created a new namespace called my.app.setup-tests
, and in there put a single (deftest ^:special-tag setup-runtests-teardown ...
and in there I call my setup code, call run-all-tests
, then call teardown code. I get the results of run-all-tests
which will tell me if there were failures or errors, and then call (is true)
or (is false)
to propagate the errors/failures to originally calling lein test
. In my project.clj, I add :test-selectors {:default (fn [m] (:special-tag m))}
to have lein test
only call my special deftest
, and it works.
Now, I'm sure there's an easier way. Thoughts?
there's some simplifications possible -for example, if a function invokes is, it becomes a success/failure if a deftest invokes it (even indirectly)
so you could have a "single test namespace" that actually loads test definitions from all the others, and has the apropriate fixtures
joshjones they are namespaces, require them, and call the functions
if the function invokes is, it shows up as a test case
excellent, this was what i was looking for -- i suppose i can get all functions in the namespace, and then see if they have :test
as metadata on them, in order to know they're tests and call them. or is there an easier way you can think of? @noisesmith
I mean - that's what clojure.test itself is doing...
there might be something even simpler
for example, what about a leiningen plugin that loads are your test files (like clojure.test does) runs your setup, runs clojure.test/run-all-tests, then runs your tear-down?
then you don't need to do anything unusual in your tests - finding all files under test/ and then running clojure.test/run-all-tests is not all that complicated
right, so why not just do the file-seq / load-file of your test directory too? seems simpler than the other options
you don't even need lein test at this point - call (file-seq "test/") and for everything that comes back with a name ending in ".clj" call load-file on it; then run your pre-test-hook, then call (clojure.test/run-all-tests) then call your post-test-hook
then your test namespaces don't have to be weird - they can be normal test namespaces
you avoid complex higher order code that tries to figure out how to construct tests
then write a lein plugin that does this instead, it's like 15 lines of code max
and it's a lot simpler to make that than it is to make N indirect test namespaces with automatically generated code
correction: it's (file-seq (
- that gives all the eligible files, just select the ones that have names ending in clj
or you can do your other plan, I just think making an alternate test runner is simpler
one more thing though -- if I do the lein plugin, can i hook into lein test
and have it run?
it would be more like lein mytest
you would have replaced lein test altogether (in order to have a wrapper around it)
you can't ask CI to run lein mytest
? - lein test isn't magic, it just runs a file-seq, selects the clj files and loads them, then uses clojure.test
maybe so, but we're open-sourcing this at some point and really would rather have the standard commands work as people are familiar with them
make a PR so lein test accepts a global test wrapper 😄
(as an optional config of course)
it just seems like it's asking for trouble to do logic that should surround test running, from underneath inside the test runner
So, I find myself literally never using letfn, and instead always using let. Is that just me? I know about mutual recursion; I just can’t remember the last time I did mutual recursion that it wasn’t an explicit exercise in attempting to do mutual recursion...
@lvh I'm in the same place as you on letfn
. I don't think I've ever needed to use it.
I guess maybe a recursive descent parser counts as mutually recursive, but I’m guessing I’d use toplevel production rules instead of a letfn. (Not to mention, for anything nontrivial that needs parsing: a parser generator.)
you never need to use it, it's just a convenience op, no? it does nothing you could not do in other ways, but it could in principle make your code more expressive? tons of stuff like this in clojure.
it specifically does something you can't do in other ways: it allows direct mutually recursive calls of anonymous functions
functions can be anonymous and named - anonymous as in not vars interned to a namespace
😄 maybe we need a different word to reflect what these functions really are
huh? if you can refer to then by name, who cares what the mech is? my pt is that the fnspec in letfn requires a name. it's not let*.
OK "it specifically does something you can't do other ways: it allows mutually recursive calls of first class functions"
it's the only thing in clojure that allows doing this directly without using deref or mutation of some sort
sure, if you allow indirect methods of doing something, nothing in cs is unique - it's the only thing in clojure that implements this functionality without hacks or the necessity of namespace level bindings
this has a historical importance, because common lispers complained about clojure not having generalized tail call optimization
letfn and trampoline both attempt to address this
sorry wait that's totally wrong
well my point was just that it is a convenience op. i don't see why thstvis controversial. can you give an example where use of letfn could not be replaced? iow it is not primitive.
mobileink can you show an example of locally bound functions that mutually call one another without making a namespace level binding or using deref?
that's something you can't do without letfn or def or mutables
and sometimes def or mutables aren't good solutions
ok, but i did not say "letfn or x or y or z". i said letfn. you've just acknowleged my point.
(ok, self nitpick, you could do it with a promise, which is not as bad as a mutable, but at least as ugly)
mobileink you could do everything with asm.java, you don't need anything else that is in clojure.jar
the thing is, it would suck
not being able to do an algorithm without def, sucks
I want to ask for some directional advice: I began working on a function that would convert the lwjgl javadocs into a data structure that describes the library with the intention of using that to generate a thin clojure wrapper over the library (with a few changes, like stripping out extraneous 'gl-' and 'm-' and other prefixes) but i've got a non-program problem i'm being faced with-- how should i package it? I had planned to make each class its own namespace; java doesn't have namespaces so it abused classes to be namespaces in lwjgl. C doesn't have namespaces, so it prepends junk to the front of each function name. Clojure has them, so I figured it would be wise to just use namespaces where lwjgl is using classes. However, if I do that then it naturally seems like each package (class container) would be a namespaces container (library). But there are over 30 packages in lwjgl. Should i just make 30ish libraries?
it might suck, but that's not televant. i mean relevant, but i'm kinda liking "televant" ;)
mobileink - so you're agreed that clojure could just ditch everything but asm.java and we'd be fine?