Fork me on GitHub
#clojure
<
2016-02-23
>
george.w.singer00:02:51

Just posted a question about transducers and their reducing functions: http://stackoverflow.com/questions/35566494/how-many-reducing-functions-do-collection-values-pass-through-in-a-call-to-trans Been confused about this for 2 days now 😢

Alex Miller (Clojure team)00:02:28

It's the latter case and there are no intermediate collections.

hiredman01:02:41

the way transduce works is you have a reducing function f like (fn [accum value] ...) and you have a transform (called xform in the doctsring) which is sort of the transducer

hiredman01:02:29

transduce applies the transform (xform) to the reducing function f, which then returns a new reducing g, and then the collection is reduced with g

hiredman01:02:41

so the xform isn't a transformation of data, but a transformation of the function used to reduce the data

hiredman01:02:57

so the map transducer is something like (defn map [f] (fn [r] (fn [accum value] (r accum (f value))))) if I recall

hiredman01:02:32

so 'f' is the function being mapped, 'r' is a reducing function, and it returns a new reducing function

hiredman01:02:38

so when you call (map inc) you get back a function that takes and returns a reducing function

hiredman01:02:52

so you can stack a bunch of those together in (comp ...)

hiredman01:02:21

that is a lot of "so"s

bradford03:02:40

Heyas. Is there a macro/lib for a conditional map syntax? kinda like.. . {:a true :c false} -> {:c (binding)}

bradford03:02:39

I'm doing tons of data munging and this would make my code a lot cleaner than the scattering of assocs/dissocs

jonahbenton03:02:21

hey @bradford what would the intended semantics be?

bradford04:02:48

@jonahbenton: basically, it only binds the symbol on the left to the form on the right if right evaluates to true

bradford04:02:27

then I can just have a big {:map} in my code of optional expressions, instead of lots of nested ifs, assoc’s, and dissocs

khepin05:02:49

Hi everyone, not sure what's best practice to post a longer question here so I wrote it up in a gist: https://gist.github.com/khepin/d995f8ebe455b103fac1 I'm having some perf issues and would love any pointer to what I'm missing / doing wrong here

arrdem07:02:05

bradford: clojure.core.match is probably what you want.

arrdem08:02:41

user> (defn g [{:keys [a b c]}]
        (match [a b]
          [true true]
          ,,(println "C was " c)
    
          :else
          ,,(println "Didn't match!")))
#'user/g
user> (g nil)
Didn't match!
nil
user> (g {:a true})
Didn't match!
nil
user> (g {:a true :b true})
C was  nil
nil
user> 
or something

arrdem08:02:29

the tricky bit is that core.match doesn't really have a notion of free variables to be bound within patters (at least that I know of) so I had to separate the destructure from the conditional.

arrdem08:02:08

To do better than this you'd have to have some way to encode a "free variable" in a pattern match structure which does not conflict with referencing a variable bound value. For instance [{:a true :b true :c c}] doesn't work because even if c isn't a closed over symbol it is interpreted to be a symbol constant rather than a binding to be established.

arrdem08:02:02

Reading the overview [1] it looks like you should be able to say [{... :c (_ :as c)}] but that seems to be interpreted to mean that the kv pair [:c Anything] exists within the matched/destructured map which isn't true.

bigkahuna08:02:24

Apologies for incoherent message. Stuck on cattle train into London :-)). Not a dig at OOP

bigkahuna08:02:00

Trying to read Slack on crowded chain not a good idea =D>

pesterhazy08:02:01

Using stuart sierra's component lib, is it possible to selectively restart an individual component (not the whole system)?

malcolmsparks09:02:12

@alexisgallagher: In response to your question about doing a side-effect in an atom, if you want to do a side-effect, instead use a ref and send that side-effecting function to an agent inside the ref. Only when the transaction is successful (doesn't have to be retried) does the send actual get sent, otherwise it's queued in an 'outbox' which gets purged if the transaction has to retry. This way you can ensure that your side-effects are performed once and only once

malcolmsparks09:02:35

I used to think this was true for atoms, but my recent experiments have show it's only true for refs.

malcolmsparks09:02:15

@alexisgallagher: reading a bit more of the thread... I think you have to be aware that functions applied to atoms and refs are potentially retried, many times. If you're OK with that, go ahead and call your side-effect synchronously. If it's a side-effect like (launch-missiles!) you may be ok, if it's (send-email!) please ensure I'm not on your mailing list simple_smile

mpenet09:02:47

I guess you could use add-watch! to mitigate the retry issue (and get acces to old val), but that's might be adding hair on a already very hairy solution

mpenet09:02:43

(didn't read the whole thread)

pesterhazy09:02:34

that would be kind of useful, for example if one component takes a long time to load (think a cache, remote db cx)

boycott09:02:53

re: yesterdays discussion about ssl, has anyone got letsencrypt working for their jetty/whatever server? I’m about 50% there (I think) using jetty 9 https://letsencrypt.org

pesterhazy09:02:16

That seems like a useful pattern for interactive dev:

(defn handle* [req] ...) (defn handle [req] (@#'handle* req))
to make sure a function is reloaded on each run (rather than remembered in ring's route list)

pesterhazy09:02:44

Does anyone use something like this? Maybe extracted the pattern as a macro?

martinklepsch09:02:00

@pesterhazy: @boycott I only was looking into certificate generation, not actual usage with something like jetty

boycott09:02:43

I have got letsencrypt running on an nginx server, much more straightforward than the whole key, pem, use these two files for this server, these two catted together for this one craziness.

pesterhazy09:02:30

@boycott: if you do manage it, write it up as a blog post

boycott09:02:25

hopefully in the next couple of weeks

mpenet10:02:25

the error should be pretty explicit

dev-hartmann11:02:55

hey clojurians, i could need some help with the following code snippet

dev-hartmann11:02:56

(defn auth-user [credentials] (let [user (store/find-user-by :name (:name credentials)) unauthed [false {:message "Invalid username or password"}]] (if user (do (println user) (println (str "user name:" (:name user)) ) (if (hs/check (:password credentials) (:password user)) ;;[true {:user (dissoc user :password)}] unauthed) unauthed))))

dev-hartmann11:02:56

when i print the user, everything is fine

dev-hartmann11:02:06

=> ({:name ben2, :email <mailto:[email protected]|[email protected]>, :password bcrypt+sha512$1b5c6807d704b5dc0d59f9e5$12$2432612431322472386731364c726252323956596264736f7556673965364551674c6a44624f537a69305a58544f6f7236495a3536576f3350497753, :user_roles ()})

dev-hartmann11:02:34

but (:password user) return nil

dev-hartmann11:02:19

as you might have noticed, I am a newb

michael_at_sosupper11:02:25

I'm a newb too, but does (:name user) => "ben2" as you would expect?

michael_at_sosupper11:02:49

that's probably because you have a map inside a seq

michael_at_sosupper11:02:03

try (:password (first blah))

dev-hartmann11:02:10

sh**t, see it now

dev-hartmann11:02:17

ther are parens around the map

michael_at_sosupper11:02:25

yeah, db gave you a list of stuff simple_smile

michael_at_sosupper11:02:39

I make that same mistake sometimes

michael_at_sosupper11:02:57

Any Expectations test people around? I see odd behavior, where it appears an (expect) is being done out of order. I have an (expect ...) line before a line that changes the DB, but the expect fails because it appears the expect is actually happening after the db change

dev-hartmann11:02:16

now it's working 😉 thanks a lot.. problem blindness 😉

karl11:02:51

@dev-hartmann: looks like your map is wrapped in a list.

karl11:02:10

Whoops, Slack is sloow 😞

urbanslug11:02:47

Hello, so what does clojure have that's similar to haskell typeclasses?

urbanslug11:02:56

or OOP objects?

mostr11:02:58

protocols

dev-hartmann11:02:11

@karl: no biggie, thanks anyway 😉

mostr11:02:49

I guess type classes and oop objects aren’t quite the same or even similar things simple_smile

mostr12:02:12

oop objects -> records, type classes -> protocols, AFAIK

jonahbenton12:02:18

hey @raphael recur is to be used from the "tail position" in its containing form- tail essentially meaning last. the idea is that the containing form gets turned into a loop, and recur's location determines where the loop ends. so a line of code that appears inside the containing form but after recur presents confusing semantics. e.g. in (fn [] (recur) "xxxx") the containing form is the fn, and "xxxx" is in the tail position. recur wants to loop back up to fn, but the "xxxx" is still sitting out there- inside the form but outside the end of the loop. so the compiler will complain. does that help?

raphael12:02:53

in fact I don't undertand when the compiler take care of what there is after the recur

jonahbenton12:02:37

hey @pesterhazy you actually can restart individual components in a system- but in doing so you get a different instance of the component than exists in the system map or in dependent components.

pesterhazy12:02:11

@jonahbenton: can I patch that back in into the system afterwards?

jonahbenton12:02:41

sure, the system map is just a map

jonahbenton12:02:11

but in general the semantics aren't intended to be the granular sort where you may have a backoff retry or other machinery. at the component level it's more coarse grained.

jonahbenton12:02:25

@raphael can you say more? the reason recur exists is that there are 2 things that can happen within the containing form at "tail position"- either you loop back up, or you don't, and the value of the last expression- in tail position- becomes the value of the containing form. so you need to tell the compiler which one to do- (recur) means loop, and the absence of recur means fall through and exit the loop

raphael12:02:13

ok so recur not working like something like f(){return f();} ?

jonahbenton12:02:45

sorry- misread

raphael12:02:10

yes I think too

jonahbenton12:02:05

recur is a compromise solution for problems that in other languages may be solved with recursion

jonahbenton12:02:18

the jvm does not support tail recursion

jonahbenton12:02:44

recur is like goto

raphael12:02:06

yes I think I understand thank a lot @jonahbenton simple_smile

jonahbenton12:02:18

sure, good luck

edvorg12:02:11

Hi guys. I'm trying to understand how can one tweak clojure.core/*compiler-options* in leiningen project. What exactly am I trying to do is to allow direct-linking but no luck so far.

mpenet12:02:26

I guess :jvm-opts ["-Dclojure.compiler.direct-linking=true"]

edvorg12:02:50

Thank you, @mpenet I tried this on simple app

(ns zippy.core

  (:gen-class))

(defn f1 []
  (println "f1 impl 1"))

(defn f1 []
  (println "f1 impl 2"))

(defn -main
  "Start a production system."
  [& args]
  (f1)
  )
Correct me if I wrong. This code should print f1 impl 1 if direct-linking is enabled, but it prints f impl 2

mpenet12:02:14

I dont' really know how it works internally, but my guess is that only runtime redefs would cause a first f1 call

mpenet12:02:38

but then again, I didn't check how it works

edvorg12:02:10

All I know from clojure docs and changes.md is that runtime function overriding won't work. So I wonder what is the best way to check that direct-linking is really enabled. This code is the only idea I came up to..

edvorg12:02:59

I'm trying to benchmark my application and see if this allows any performance improvement, but no difference so far

mpenet12:02:08

try to override a fn at runtime

mpenet12:02:24

well perf difference is barely noticeable in practice

edvorg12:02:41

Not sure what do you mean by runtime. The code above isn't runtime overriding?

mpenet12:02:21

I guess not since it's aot compiled

mpenet12:02:43

you try to eval some code in your main that try to overwrite one of your fn

edvorg12:02:58

Yeah, I see, ok, let me check again

edvorg12:02:54

> well perf difference is barely noticeable in practice I understand that, but I need numbers simple_smile

mpenet12:02:10

there were some reports in the mailing list

edvorg12:02:46

clojure dev or clojure ?

joost-diepenmaat13:02:44

as far as I understand direct linking, printing “f impl 2” is expected

edvorg13:02:49

Got it working, btw, thanks fo help

edvorg13:02:25

yes, thank you, @joost-diepenmaat: it's my bad, I misunderstood what exactly runtime overriding means

mpenet13:02:50

edvorg: clojure

joost-diepenmaat13:02:59

good it’s working as expected simple_smile

edvorg13:02:15

@mpenet: thank you very much

mpenet13:02:52

I personally don't know of any benchmark that shows a noticeable benefit. If you find any please let me know

edvorg13:02:18

yes, of course. any meaningful report i get, i'll post there

mpenet13:02:44

but there are other benefits, startup time, and class file size

edvorg13:02:45

we are only concerned about performance at the moment, we optimized a lot of stuff already, so we are looking for some new ideas

jonahbenton13:02:50

hey @edvorg what does a profiler say about where your bottlenecks are?

edvorg13:02:06

@jonahbenton: We optimized mostly everything we could see in profiler. Now we have some external api cals and some heavy data transformations we can't get rid of. We refactored code with transducers where it's possible and got small improvement from this. So what I'm looking for are some unusual optimization paths that can't be seen in profiler

edvorg13:02:49

Cause too dispersed over the code

edvorg13:02:56

Like direct linking in theory should make jit work better

edvorg13:02:02

So.. need to try

alexisgallagher13:02:20

@malcolmsparks: thanks. Yup, I was aware that no side-effects were allowed inside the atom's update func, and that that function might be retried. What's puzzled me mainly is how to trigger a side-effect only when the update function actually returns a changed value. I had this solution using dosync and refs:

alexisgallagher13:02:51

defn transform-item [x] ...)
(defn do-side-effect-on-change [] nil)

(def my-ref (ref ...))
(when (dosync (let [old-value @my-ref
                    _ (alter! my-ref transform-item)
                    new-value @my-ref]
                (not= old-value new-value)))
  (do-side-effect-on-change))

alexisgallagher13:02:45

But what made me think I should be using an atom instead of refs is the catechism to the effect that refs are for coordinated state changes and I'm really only updating one thing here.

mpenet13:02:57

alexisgallagher: you could then just use an atom + add-watch to trigger your fn

alexisgallagher13:02:33

But in the end it seems like refs are the best way to go, whether I return a value from the dosync to report the change or use an agent if I want that fact asynchronously.

alexisgallagher13:02:20

@mpenet: I don't want the watch triggered on every change, only by changes due to this snippet. Couldn't figure out how to scope a watch within a transaction.

mpenet13:02:14

then you'd have to do this manually with an atom: using compare and swap, check it's return value and act when the swap occured.

alexisgallagher13:02:07

But I think that fails because then I don't have access to the old and new value from within the transaction, do I? I may be missing something obvious here...

urbanslug13:02:49

Where can I ask beginner clojure questions?

urbanslug13:02:11

nvm found #C053AK3F9

jonahbenton14:02:14

@alexisgallagher: you can see one of the issues through the snippet above. the value of my-ref can change in between in the let assignment to old-value and the alter!

hiredman14:02:47

if I recall correctly, because the transaction is altering the value of the ref, it can't

jonahbenton14:02:57

right- not within the transaction, but in the context of the wider world

hiredman14:02:42

I am not sure what you mean then

jonahbenton14:02:54

and the value of my-ref after the alter may not be the final value for the ref after the transaction

jonahbenton14:02:08

clojure's stm is an mvcc

hiredman14:02:24

dosync defines the bounds of the transaction

hiredman14:02:36

both reads are in the dosync

jonahbenton14:02:59

correct, but other threads may be executing their own transactions

jonahbenton14:02:17

dosync does not serialize across threads

hiredman14:02:26

an issue would be, if it also read another ref that it did not also write to, then if I recall you need to use ensure on that ref

hiredman14:02:42

jonahbenton: what?

hiredman14:02:10

I just don't know what point you are trying to make

hiredman14:02:42

dosync is a transaction, its body is executed atomiclly

hiredman14:02:10

you won't see operations from other threads interleaved in there

jonahbenton14:02:13

right- not within a single transaction- the view of the world is fixed. but another thread may complete a transaction against the same ref

jonahbenton14:02:28

when it comes time to commit, at the end of dosync

hiredman14:02:48

right, and dosync will retry the transaction

hiredman14:02:33

once the transaction completes, that code returns true or false indicating if the transaction actually changed the state of the ref

jonahbenton14:02:51

right- so my point is- within a transaction, you may detect a successful change, but that's different from reporting change across transactions

hiredman14:02:02

sure, same as with an atom etc

hiredman14:02:35

basically you are talking about expanding the critical section

hiredman14:02:02

but, that is, as far as I know not what is the ask here, this is "hey, I am updating this state conditionally, and want to run action A if it updates"

hiredman14:02:34

which, well, roll your own update loop using compare-and-set!

hiredman14:02:40

the twisted and complicated ways people end up doing that is crazy, there is a built in function that lets you write your own swap! with whatever behavior you require

jonahbenton14:02:04

yeah- i was suggesting agents- e.g. enforce explicit ordering of the incoming message stream- because reporting on change across threads is not what atoms and refs solve for, and reinventing the retry logic is probably undesirable.

jonahbenton14:02:02

iow, just use a queue

hiredman14:02:13

sure, but you are definitely serializing everything then, and you have to sort of invert the control, the agent has to execute the side effecting stuff, so you aren't reading values from identities at that point, you may have to restructure your code to be more actor like

jonahbenton14:02:17

sure- the code snippet above can be handed directly to an agent- receive a message, transform agent state, if a change occurred fire off the notification- but if you have other logic to also execute following the change, or if you have to use await- then it's not a good pattern.

jonahbenton14:02:46

if the system is actually request-response, not change-notify, the pattern doesn't fit

hiredman14:02:18

depends on what version of clojure

ska14:02:38

1.7.0 in this case

hiredman14:02:49

oh, I would have thought that had it

hiredman14:02:16

I thought there was a nicer error message for missing the parameter vector these days

ghadi14:02:29

macro parse problems are getting much nicer in $version.next

ska14:02:41

well, it came back with a java.lang.IllegalArgumentException: Parameter declaration "when-let" should be a vector

hiredman14:02:42

IllegalArgumentException Parameter declaration "when-let" should be a vector clojure.core/assert-valid-fdecl (core.clj:7111)

hiredman14:02:55

that is the good error message

hiredman14:02:02

what is wrong with it?

ska14:02:04

indeed it is simple_smile

ska14:02:11

Just the wrong here

ska14:02:59

the when-let for is just fine, it is the defn, as you already mentioned earlier

ska14:02:11

s/for/form/

ghadi14:02:13

(tbh I'd expect Param declaration (when-let ....) should be a vector)

ghadi14:02:42

show the whole erroneously-placed form

ska14:02:16

I thought it was pointing me to the parameters of when-let.

hiredman14:02:18

ska: well when defn sees a list there it assumes you are using the multi arity version of fn, like (defn f ([] 1) ([x] x)), in which case the when-let there should be a vector

ska14:02:24

Just imagine that tiny snippet to be a longer function with a real body in the when-let and a longer function name and all... It was just when I cut everything else away, that I realized my stupidity.

ska14:02:59

Anyway, thanks for listening. Feeling better now. Even better than M-x doctor. 😉 Back to work again.

alexisgallagher15:02:07

Now I'm confused again. Are you two concluding my dosync approach is incorrect? @hiredman @jonahbenton

hiredman15:02:42

alexisgallagher: not exactly incorrect, but I think you would be much better off using an atom and compare-and-set!

alexisgallagher15:02:29

Hmmm. Not clear to me how to get the same result with that, given that I'm actually starting with an update function. Wouldn't that require doing locking around the compare-and-set! ?

hiredman15:02:04

you compare and set until it returns true

alexisgallagher15:02:34

So I build my own retry loop?

hiredman15:02:41

in a loop, which is what swap! does

hiredman15:02:57

yeah, and you can return whatever you want from your swap!

hiredman15:02:34

e.g. you can make it return true/false based on if your business logic actually changed the state of the atom

facundo17:02:21

Hi everyone, I'm just starting to use clojure and doing an exercise I came across an error I just don't understand, maybe someone can help me

nkraft17:02:48

Anyone here used ClojureWerkz CassaForte library?

facundo17:02:12

Is that :b#something keyword illegal for some reason? It doesn't seem to be outside of a let

hiredman17:02:40

it isn't in let either

hiredman17:02:56

whatever repl you are using just has problems with it

hiredman17:02:02

a bare clojure repl is fine with it

facundo17:02:42

I'm using lein, so I guess it's the lein repl?

hiredman17:02:08

lein's repl piles a few things on top of the bare clojure repl

hiredman17:02:39

there are a few different libraries involved, so one of those is being dumb

mnespor17:02:13

I'm a little surprised that works? From http://clojure.org/reference/reader : "Symbols begin with a non-numeric character and can contain alphanumeric characters and *, +, !, -, _, ', and ? (other characters may be allowed eventually)." ... Keywords - Keywords are like symbols, except: They can and must begin with a colon, e.g. :fred. They cannot contain '.' or name classes. Like symbols, they can contain a namespace, :person/name A keyword that begins with two colons is resolved in the current namespace: In the user namespace, ::rect is read as :user/rect"

hiredman17:02:15

I guess it could be related to the version of clojure used too, sometimes what the reader allows in symbols changes, but I don't recall that being one of the things that has changed

hiredman17:02:42

mnespor: the reader is in fact much more lax than what the docs specify

hiredman17:02:12

and clojure breaks those rules too, '/' isn't listed there, but '/' is division

hiredman17:02:20

some libraries that used to be popular actually used '#' in keywords as part of their dsls, and if the reader suddenly clamped down on that stuff it would break things

hiredman17:02:44

there is at least one jira issue that is sort of like "figure out what to do with this mess"

facundo17:02:32

I came across this while using enlive

mnespor17:02:37

Even more confusing, trailing # is the syntax for defining gensyms

facundo17:02:45

needed # to pick stuff by id

mnespor17:02:30

interesting

hiredman17:02:15

'>' is actually used in the names of the functions clojure generates for defrecords (a more recent example of clojure using characters outside of the ones listed there)

mpenet18:02:20

@nkraft nope but I'd recommend https://github.com/mpenet/alia is a bit more up to date and used by some pretty large projects now (disclaimer: I authored it)

nkraft18:02:32

Thanks. I actually gave up on Cassaforte and switch to Alia, which looks pretty good with Hayt.

mpenet18:02:49

Yeah the dsl is identical with hayt, cassaforte actually used hayt internally until recently

nkraft19:02:35

@mpenet: While I have your attention (more or less), how does one get pooling options to work? I've been trying things like :pooling-options {:core-connections-per-host [:local 16 :remote 16]} but I get unpacking and nth keyword errors.

nkraft19:02:12

Actually, that's a very incomplete bit of code, but I hope you get the idea.

juhoteperi19:02:58

I hope I would need to use Cassandra because those library names are great @mpenet!

nkraft19:02:40

It's a nice library, with some pretty impressive performance.

mpenet19:02:07

@nkraft I am on my phone right now, I ll check quickly

mpenet19:02:42

@nkraft I think core-connections-per-host should be a map or a vector or vectors

nkraft19:02:36

@mpenet: Yup, that was it. Problem with looking at old examples, I guess. Got the pooling working, Thanks!

mpenet19:02:53

If you need to push for perf and dont care about lazyness you can also play with :result-set-fn opt in execute, you can hit IReduce path and have a bit more control over what you get in return and avoid seqs

mpenet19:02:49

Ex: using #(into [] %), possibly with an xform

nkraft19:02:51

I'll put that on the list of things to play with. I need to push several billion entities to a Cassandra datastore. Fun, fun, fun.

futuro22:02:19

What webservers do people like to use? I'm seeing immutant, aleph, http-kit, etc., but I'm not really sure what the pros/cons of each are

nkraft22:02:12

I pretty much stick to http-kit. For us it's been the best performer of the group.

futuro22:02:15

@nkraft: cool, good to know

gerred23:02:16

Is YourKit pretty useful for Clojure?

danielcompton23:02:28

They had a great deal at the start of the year for a personal license for $100-$200 but I think it was a temporary thing

gerred23:02:35

aw, dang, missed it.