Fork me on GitHub
#clojure
<
2016-01-04
>
bradford11:01:11

Quick Q — I’ve got f(g n k v) and g n [k v], how do I “lift” x y out of the vector for use in f ? Feels like I gotta do something with partial and apply

larhat11:01:39

@bradford:

user=> (defn x [a b c d] 42)
#'user/x
user=> (apply x (list* "a" "b" ["c" "d"]))
42

bradford12:01:54

cool! thanks

meow13:01:23

I've never used list* so far. Interesting. Wonder if I've been missing out. 😉

Lambda/Sierra13:01:53

@bradford: apply accepts singular arguments before the final list

meow13:01:52

Yeah, I would favor apply over list*.

meow13:01:12

@stuartsierra: Do you see list* used in the wild much? I can't recall ever coming across it but I've only been doing clojure about 8 months now so your experience outweighs mine quite a bit. simple_smile

meow13:01:33

And what is the * meant to signify?

Lambda/Sierra14:01:07

I've used list* on rare occasions. The * is an old convention from Common Lisp used to denote an "alternate" version of a core function.

meow14:01:25

Ah, I see. Thanks.

meow14:01:43

Speaking of naming conventions, I've got one that I wrestle with. Sometimes I will have a situation where I've got several variations of some function and then I pass a particular version off to some other function where it gets called. No problem there. So they all get named somewhat similarly. Then I'll have one odd duck where rather than just define a function I need to define a function that returns a function. And I always wonder if I should somehow name that one differently. Does that make sense?

akiva14:01:24

I’ve seen people use [something]-factory if it’s a function that can return a number of different functions. But if it’s a function the return value of which just happens to be a function, I wouldn’t sweat a special name. It reminds me in a way of Hungarian notation from a billion years ago.

drusellers14:01:25

Speaking of naming, is there any community consensus around naming of database methods - thinking CRUD stuff mostly

Lambda/Sierra14:01:26

I like to name functions for the thing they return. So a function returning a function would be named "…-fn"

akiva14:01:51

So basically the opposite of what I just said. [laughs]

akiva14:01:12

So you’d have stuff like …-int?

Lambda/Sierra14:01:32

No, I don't suffix everything.

Lambda/Sierra14:01:30

Say a function returns a user's age, I would call it age. If it returns a function to calculate a user's age, I'd call it age-fn.

akiva14:01:00

Ah. Right. I do the same because age is conceptual. But, to me, a function is just another type so adding -fn describes the type returned rather than the concept. Although I guess age-fn does both at the same time.

Lambda/Sierra14:01:04

I don't hold to this pattern everywhere, but I find it helpful to prevent confusion in the unusual case of a function-returning-function.

akiva14:01:50

Makes sense.

agile_geek14:01:34

I like that as a default naming convention. Haven't come across it in most of the code I've read so far.

meow14:01:25

Sometimes I do use the -fn approach but it doesn't always feel right.

meow14:01:15

I have a similar issue when it comes to parameter names that expect the argument to be a function. Sometimes I use names like age-fn or age-f or sometimes get-age-f depending. I haven't settled on a convention I'm 100% happy with.

meow14:01:43

Sometimes it's enough to point out that the argument needs to be a function. Other times I feel like it is more important to emphasize what that function is meant to do or what it should return as a value.

bronsa14:01:22

@meow: list* is mostly there to help bootstrapping some clojure features like apply and syntax-quote

meow14:01:44

@bronsa: Cool. Thanks for the info. It's good to know the history behind a lot of these language decisions. Clojure (and its community) is great that way. Honor the past, just don't get stuck in it.

meow15:01:30

@stuartsierra: Just read your recent blog posts. I'm a big fan of your writing. Hope you never stop. simple_smile

wotbrew15:01:02

Hello all, bit of a general question, imagine I have a fn like swap!. It applies a function atomically, using something like CAS to transition state. If I do (my-swap! x f) and (f x) is identity of x is it valid for the mechanism under the hood to ignore the CAS update? Does this invalidate the concurrency semantics of swap!? I don't think it does, as the resultant history (if operation is dropped) is possible anyway... I am thinking this is a potential optimization to my faraday-atom library...

wotbrew15:01:03

I notice this optimization isn't performed in clojure.lang.Atom (at a glance)

wotbrew15:01:20

Though the AtomicReference swap in that case is very cheap

Lambda/Sierra15:01:44

@danstone: it depends on what other behavior might be attached to the CAS. For example, updating an AtomicReference has ordering guarantees related to the Java memory model.

wotbrew15:01:42

@stuartsierra In this case its persisting state to an external store (dynamodb) using what is called a conditional-put.

wotbrew15:01:51

I just imagine dropping the operation is similar to it simply completing immediately in this context

Lambda/Sierra16:01:21

@danstone: In that case, Dynamo is in control of the state. How do you know the CAS is going to succeed without going to Dynamo?

wotbrew16:01:35

I don't, thats why I am unsure here. The fact is I am saying compareAndSet(x, x) here... Which means, I think it is possible in any history of concurrent operations for this to be a noop.

wotbrew16:01:52

It is possible of course if I contend with other threads/machines for x to become y in the spin loop, in which case I will arrive at a different answer. Both are possible

wotbrew16:01:45

My idea is that it doesn't actually matter which one is picked, it may make an outcome more unlikely, but it is still correct (I think?)

wotbrew16:01:54

I think if this was local state with ordering guarantees for threads seeing the new value of the reference in respect to wall clock time, then it would be different

wotbrew16:01:34

But wall clock time is irrelevant in this case, as the network can slow down acknowledgement of each concurrent swap for however long it wants

wotbrew16:01:34

To put it another way. I have two functions `(defn f [n] (case n 0 0 (inc n)) (defn g [n] (inc n))` Lets start the ref at 0. Here are the possible histories if I apply the 2 functions concurrently (assuming the state transition semantics of a clojure atom) f -> g == 1 g -> f == 2 If there is no contention and f is called first, 0 is set to 0(noop) If there is contention between f and g, we may arrive at a history in which f does not set the ref to 0 if it doesn't complete before g. However it is just as likely that it does complete first. There is no way to distinguish between the valid history (f -> g) and one in which I didn't call f at all. Perhaps there is some subtlety here that I am missing?

wotbrew16:01:35

I think it is important to be clear I am not surfacing a compare and set operation, Just a function like swap! that promises to apply your function atomically and not invalidate or discard your operation due to a race condition.

jonahbenton16:01:56

@danstone yes, if atomic semantics are implied by the surface api, but underneath there is a codepath where the atomic read is not performed, then as @stuartsierra was suggesting, the resulting behavior may be surprising, per https://en.wikipedia.org/wiki/Memory_barrier

jonahbenton16:01:52

it sounds though like the use case for your swap! may be more limited than a general purpose swap!

wotbrew16:01:53

@jonahbenton: The CAS itself is remote, the only thing I do locally is apply the function. That said, I don't see why this wouldn't be valid for local atoms as well.

wotbrew16:01:44

Obviously I am applying f to a value that is read in a way that reflects all writes so far up until the point the read was acknowledged.

jonahbenton16:01:23

@danstone so you are applying f to a local copy of a value maintained at dynamo, and if that application provides a new value, you do conditional put at dynamo?

wotbrew16:01:54

That sounds about right

jonahbenton16:01:58

i see- it's a potential optimization to avoid the network round trip

wotbrew16:01:02

I avoid the conditional-put part of the deal

wotbrew16:01:40

(f x) == x is not going to happen very often for most users, but it turns out a lot of our operations do something like (if (pred x) (do-stuff x) x)

jonahbenton17:01:54

right. so is it fair to say: x represents a deref'd dynamo value, current as of the time of the last deref. when (pred x) is true, (do-stuff x) may either result in a change to x, or not. Without this optimization, both outcomes will hit dynamo; with this optimization, only the path where x has changed will hit dynamo with a conditional put.

wotbrew17:01:17

@jonahbenton: I gotta go catch a train, but will be more than willing to continue this further if you (or others) PM me?

jonahbenton17:01:30

sure, quick response- with this optimization, it sounds like you miss a signal from (putitem key x x) that the value at dynamo for key has changed to something other than x, so anything following (do-stuff x) may be incorrect

sgerguri17:01:03

This may or may not be a problem depending on what happens in (do-stuff x), and in the outside world that triggered the action. Given how tricky the context might be I propose that the optimisation is simply not worth the potential consistency issues in general.

wotbrew19:01:02

@sgerguri: It could save a lot of money in provisioning costs simple_smile

wotbrew19:01:27

In all my examples I am of course assuming pure functions

wotbrew19:01:44

The core proposition is this. The semantics of an atom are such that there is no guarantee in the ordering of operations applied with swap! from seperate threads. They can be re-ordered due to retries, imagine a long running task and a short running one, the short running one can be dispatched after the long running one and still be executed first. There is no dispatch ordering or global ordering like there is in say, an agent (where each operation is applied in order according to receipt time by its internal queue). If this is true, then we have to accept multiple potential histories (outcomes) for any set of concurrent operations against the atom. Where (f x) is a noop we can safely say that this operation is irrelevant, a valid history is one where this operation doesnt happen.The only caveat I can see is one has to be willing to accept the change in likelyhood from one potential history to another. However outcome likelyhood is already skewed torwards short-running tasks for example, I don't think random outcome distribution from a set of ops is a property atoms have as it stands.

wotbrew19:01:49

I'm convinced that this optimisation is a good idea for cases where the CAS operation is remote or incurs significant cost.

wotbrew19:01:42

^^ I hope I'm not just being stubborn but I remain unconvinced that this optimisation will actually break the atomic transition model or any consistency properties of it

wotbrew19:01:58

And I really thought there would be some kind of subtle gotcha here

kephale20:01:45

is it correct that the ddos on clojars can lead to weird JVM errors?

Lambda/Sierra20:01:17

@kephale: I don't see how it could affect the JVM. It could certainly cause weird errors in build tools like Leiningen.

kephale20:01:12

yeah, i dont really mean the JVM itself, but i’m running some code that was unchanged since it last worked, and getting what seem like random errors with segfaults much deeper than my code

kephale20:01:30

just sanity checking before i pull out anymore hair attempting to debug these

Lambda/Sierra20:01:21

I doubt that was caused by a Clojars outage.

kephale20:01:04

hum… things like … # j clojure.lang.Compiler$LetExpr.doEmit(Lclojure/lang/Compiler$C;Lclojure/lang/Compiler$ObjExpr;Lclojure/asm/commons/GeneratorAdapter;Z)V+30 … … # V [libjvm.dylib+0x4c3067] PSScavengeKlassClosure::do_klass(Klass*)+0x9 … … # C [libsystem_platform.dylib+0x1331] platformmemmove$VARIANT$Ivybridge+0x31 …

kephale20:01:27

went away after i emptied my .m2/ and refetched

kephale20:01:19

anywho, thanks for the reply @stuartsierra

jcomplex21:01:47

hiya quick question. I have this (defonce accounts (atom (array-map))) and it gets populated as time goes on. I am writing tests and want to clear this out before and after a test. Any ideas? Do I use empty or some other clojure logic?

jr21:01:52

seems like accounts should be a function parameter

ghadi21:01:20

kephale: i've seen when actively open jars are touched, or if some deployment tool messes with /usr/lib/jvm that shit happens

kephale21:01:32

@ghadi aha awesome, that sounds right then, i have 1000 lein processes open on the same NFS, some are ending/restarting as that goes

kephale21:01:40

normally that works just fine, but it has had issues over the course of today

ghadi21:01:39

so what are you going to do

ghadi21:01:48

uberjar? containerize?

kephale21:01:31

first pass just try using the clojars mirror and hope for the best, otherwise it will be uberjar and rewrite launch scripts

kephale21:01:22

or possibly hard coding the offline flag into the lein run call

ghadi21:01:36

strace should tell you for sure if a process if being naughty, right?

tcrawley21:01:07

kephale: the jars in ~/.m2/repository shouldn't be changing at all once downloaded - lein/mvn/whatever won't change the files, only download them if they are missing

kephale21:01:29

@tcrawley: that is what i’d expected, but i’m almost entirely certain when saying that i hadn’t changed any of the code, then when launching this morning suddenly started getting segfaults from weird places, replacing .m2/repository seeeeeemed to resolve it on my dev machine, and now swapping over to your mirror seems to have resolved the issues on the cluster

kephale21:01:22

FWIW, a decent number of the dependencies encapsulate natives as well

ghadi21:01:41

oh... that's a convenient fact to just drop as an afterthought 😃

tcrawley21:01:52

it is possible that you had an incomplete jar in the repo (where the download was attempted while http://clojars.org wasn't fully available), but the tooling should have caught that when trying to verify the checksums

tcrawley21:01:24

and the mirror vs. http://clojars.org should make no difference at this point, since the mirror is an rsync of the master repo

tcrawley21:01:28

so should be identical

kephale21:01:29

the suspicion was something like incomplete jars, which is why i swapped to the mirror, and i do spin up a bunch of processes at once, so i’m just wondering if there is a really timing sensitive thing with incomplete jars

kephale21:01:57

anywho, i’ll report back when it becomes more clear… the combo of natives + potential incomplete transfers was just something i wanted to see if anyone had run into

tcrawley21:01:04

it seems like incomplete jars would also just throw some form of ZipException instead of causing a segfault

tcrawley21:01:33

but, computers are terrible, so who knows?

kephale21:01:13

hehe, anyway thank you @tcrawley for getting the mirror up and everything else!

tcrawley21:01:21

my pleasure!

Alex Miller (Clojure team)22:01:50

my mental model is that maven/lein will download a jar, check it against the md5, then replace it only if successful, so an incomplete jar shouldn't happen

Alex Miller (Clojure team)22:01:19

however, I have had the pretty weird experience of having a jar that was actually an html error page more than once

jonahbenton23:01:51

@danstone agree with the proposition that this optimization does not run the risk of making the history of changes in values seen by dynamo incorrect in the context of concurrency guarantees. That said, two potential gotchas come initially to mind: clojure atom + swap! have additional semantics- atoms have watchers and swap! returns newval. The equivalent to watchers on dynamo look to be streams. I have not used them but guess this optimization may produce surprising results for stream users, if e.g. the application expects to simply deliver a heartbeat value to dynamo via the atom, a stream user might not see the heartbeat. Similarly, the staleness of the newval returned by swap! on an atom with this optimization may be different than it would have been without it- the slower f is, the more stale it will be with this optimization.