Fork me on GitHub
#clojure
<
2018-01-30
>
alexstokes00:01:07

> Likewise, bindings created with binding can be assigned to, which provides a means for a nested context to communicate with code before it on the call stack.

alexstokes00:01:22

can anyone produce an example where you would want to do this?

greglook00:01:00

side channels for extra information

alexstokes00:01:19

so you wouldn’t have any data escape the top-level binding

greglook00:01:36

it’s a thread-bound stack, so your changes wouldn’t escape upwards

alexstokes00:01:44

i guess i don’t see why you wouldn’t just return the “extra information” with your regular information

alexstokes00:01:52

but perhaps that is just a lisp idiom?

greglook00:01:57

one example where I used it was a multiplexing codec implementation

greglook00:01:12

it reads and writes headers to the byte data to label the codec to use

noisesmith00:01:35

@alexstokes that's the clean way to do it, but it also requires changing every function in between to carry that contextual data

greglook00:01:36

but the encode/decode functions are value-focused - you get back the encoded bytes or the decoded value

greglook00:01:57

instead, I had the codec implementations record the headers written or read to a dynamic var on the side

noisesmith00:01:57

and sometimes you are not at liberty to change that data

alexstokes00:01:00

@greg316 link to this code?

greglook00:01:10

so the consumer could bind the *headers* var and check at the end of the operation

greglook00:01:13

sure, one sec

greglook00:01:27

I have since rewritten it and gotten rid of that bit, but it did the job for a while 🙂

noisesmith00:01:55

@alexstokes especially consider the situation where you might have multiple threads doing the operation, so it isn't safe to use a top level var either - and code that has to be "in the middle" can't pass the value in or out

greglook00:01:17

it is definitely not the simplest way to do things, but sometimes it’s the only option you have

noisesmith00:01:10

I think in every option where you do that you could also use an instance of an object that is contextually shared by multiple higher order functions - but that's much messier

noisesmith00:01:56

eg. you'd have a codec-context-factory that creates a codec-context, and it would give you a decoder function and a driver function that you'd use together - I'm sure you can imagine how clumsy that would be to use

noisesmith00:01:36

if that architecture makes sense to you you're probably happily using spring with no use for clojure 😄

jjttjj02:01:08

Anyone know what might be causing this? Basically, there's a nested dependency in this codax library for an old version of encore. Using lein's :managed-dependencies for a newer version of encore or using :exlcusions on the codax dependency and manually a newer version of nippy doesn't work. The only luck I have is by cloning the codax repo and changing it to use a newer version of nippy which no longer depends on an old version of encore. Shouldn't that be the same thing as doing [com.taoensso/timbre :exclusions [com.taoensso/nippy]]?

pfeodrippe12:01:17

I'm testing some kafka streams code, but the relevant part is about the reference to a function (a var). I have code like below and the problem is that when used directly at the pipeline (at .filter), this pipeline sees the any change at the function, but when used like the commented line below (at kf/filter), it doesn't

(->
 (.stream builder new-electric-data-topic)
 (kf/map-values avro->clj)
 (kf/to-stream)
 (.filter (reify Predicate (test [_ k v] (voltage-event? k v)))) ;;; THIS WORKS when changing the function at REPL
#_ (kf/filter voltage-event?) ;; BUT WHEN I USE THIS, I don't get the change, have to restart the stream
 (kf/peek peek-test))

pfeodrippe12:01:44

The function that I'm changing is voltage-event? It's just one-line with true or false for this test

pfeodrippe13:01:02

And kf/filter is below

(defn filter
  [stream f]
  (.filter stream (reify Predicate (test [_ k v] (f k v)))))

pfeodrippe13:01:24

The problem could be that I return a different stream every time I call this function (because of immutability, maybe I should use a macro?)

pfeodrippe13:01:55

Or a function calling your defined function doesn't retain the reference to the original var, but I doubt it

qqq13:01:57

is it too insane to change all (fn [...]) to (fn some-name [...]) for the sole purpose that it'll be easier to read stack frames ?

pfeodrippe13:01:47

good idea uhaeuaehueh I'll start to use this

noisesmith13:01:26

I do that, it's great - it also acts as a brief code documentation

bronsa13:01:14

i do that often

qqq13:01:39

I need a macro which does this:

(indent-println-by-2
...)
then for any printlns executed in that region, it gets indented by 2 spaces ... this would drastically simplify prnitln debugging in making things easier to read

noisesmith13:01:45

@pfeodrippe that happens because vars used inside a function are looked up when used, but when you pass a function by name to another function or method, it's looked up once on that invocation and reused - which is an issue for long running things

noisesmith13:01:27

@pfeodrippe you can work around this by passing a var in instead of the function itself eg. #'voltage-event?

moxaj13:01:10

@qqq I think with-out-str could help you

noisesmith13:01:34

well, pprint already takes a writer, that could be a string-writer

noisesmith13:01:01

peregrine.circle=> (let [sw (java.io.StringWriter.)] (pprint {:a 0 :b 1 :c 2 :d [1 2 3] :e (range 20)} sw) sw)
#object[java.io.StringWriter 0x232b391e "{:a 0,\n :b 1,\n :c 2,\n :d [1 2 3],\n :e (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19)}\n"]
peregrine.circle=> (doseq [line (clojure.string/split-lines (str *1))] (println "  " line))
   {:a 0,
    :b 1,
    :c 2,
    :d [1 2 3],
    :e (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19)}
nil

pfeodrippe13:01:34

@noisesmith Understood, thanks! I've changed it to be a defmacro instead of a defn (just a few changes at the body), so the user doesn't need to pass a var every time, but now I know if I need it again 😃

qqq14:01:14

@moxaj: hmm, so grab out-str, parse the \n, and manuall insert the indendataion

moxaj14:01:31

yes, that's the idea

noisesmith14:01:52

with-out-str does more than you need though, since pprint already takes a writer arg

moxaj14:01:13

@noisesmith but I'd guess he wouldn't want to modify the code, just sprinkle in some indent-println-by-2

noisesmith14:01:00

oh you wanted println not pprint? never mind

qqq14:01:03

@noisesmith, @moxaj: right, what moxaj said, I dont want to modify underlying code

noisesmith14:01:09

then you need to mess with *out*

qqq14:01:17

more than just println, anything that dumps out things to the console

qqq14:01:30

basically, I want to specify an int, and after every \n or \r, have it insert 'n' spaces

qqq14:01:00

i can't believe no one else has run into this issue / solved it already

noisesmith14:01:05

OK, then you need with-out-str I guess - be careful, this will totally break with lazy processing

noisesmith14:01:51

you need to force all laziness that prints before exiting that block, or else the indentation won't apply

noisesmith14:01:04

but you already signed up for pain if you use IO in laziness

noisesmith14:01:49

and I think that's why nobody has "resolved this issue" - laziness makes it very messy and we use lazy things a lot

qqq14:01:45

you're saying, it's possible that (foo) runs fine, but it returns a thunk, which when executed, throws an exception / does println

noisesmith14:01:19

I'm saying that if the lazy tail of foo prints, it won't use the indentation you wrapped around it's construction

noisesmith14:01:26

unless forced inside that scope

noisesmith14:01:45

and, if some other lazy tail is forced in that scope, it uses that scope's indentation

noisesmith14:01:39

which might be fine for your usage, but would get a lot of complaints and broken results if provided as a general indented printer facility

bellis17:01:59

I want to drive an interactive program from Clojure; anyone familiar with code for doing that, or should I be just generally looking at transducers to interact asynchronously with InputStreams and OutputStreams?

ghadi17:01:17

I'm not sure that transducers make sense, but what you're describing is essentially REPL (just IS/OS not Reader/Writer)

bellis17:01:59

Nope, I'm talking about something that will start an instance of bash, ssh into another machine, and run commands there 🙂

bellis17:01:32

I wanna have something vaguely like https://en.wikipedia.org/wiki/Expect but no tcl/tk

ghadi17:01:42

hugo duncan did a whole lot of work around this 4-5 years ago

ghadi17:01:53

had a clojure-lite -> bash compiler

ghadi17:01:00

and used it for server management

ghadi17:01:19

might be some useful libs in there

bellis17:01:59

nifty! thank you!

ghadi17:01:54

no prob. there are other very useful things in that org

Al Baker17:01:59

yeah, pallet works through ssh'ing in and running a series of shell commands to move the system through its init states

noisesmith17:01:29

@benjaminster I've had best luck with using ProcessBuilder and Process to do streaming IO to shells

bellis17:01:17

yeah, it looks like that plus some way to turn an InputStream into something more easily digested by pure functional code would be one way to go

noisesmith17:01:01

be very careful about that transformation - for example don't use laziness it just makes things more complicated

noisesmith17:01:39

that's what I was saying in the other message about conch - an abstraction that forces non-io code to care about io concerns is worse than no abstraction at all

bellis17:01:18

absolutely

noisesmith17:01:31

I tried using conch but found that it added complexity without actually simplifying my task which is matched pretty well by the built in classes (fundamentally at that layer I want to do imperative IO, lazy-seqs only add bugs at that level)

noisesmith17:01:35

come to think of it you could make something that converts an IO channel to an IReduceInit though...

bellis17:01:24

conch has some interesting ideas but is (as you pointed out) aimed at a different target...

noisesmith17:01:51

I think I have a gist of doing this - digging

bellis18:01:18

I want to have timeouts and such so... I think something that gobbles up an inputstream and bleeds it over a core.async channel might be a good funnel to get the pipeline going

sparkofreason18:01:57

We have a requirement to allow users to upload a JSON file, parse the records, and stream to kafka. I'd like to harden the process so it doesn't easily cause out-of-memory exceptions when the file is somehow goofy. Currently planning to use parsed-seq from cheshire, not clear how/if it protects against this scenario. Or am I worrying too much?

noisesmith18:01:37

that gist is super simplistic, but I think it has most of the moving parts you need to do something interesting with streaming input / output from a task. The annoying thing is the JVM gives you no straightforward way to get the PID of your task - you get an abstraction that lets you shut it down or check if it's still running but no PID 😦

bellis18:01:39

thanks! oh gosh, that's a classic Java API with misnamed methods... .getOutputStream gives you what most would call the process' stdin and .getInputStream gives you stdout

noisesmith18:01:15

haha yeah - as far as Java is concerned it's the OutputStream because that's where java can put output haha

noisesmith18:01:22

that's the only way it makes sense at least

noisesmith18:01:31

it helps my mental model if I don't think of a Process object as the OS process, but rather the ambassador for one inside the jvm, so it's outputstream is the stream that sends your output to the process input - that's a better fit for what's going on on a low level anyway

noisesmith18:01:02

also don't forget to join the error stream with the input stream, or set up handlers for both

noisesmith18:01:07

but any pure JVM solution is going to have that problem (unless it was fixed in 1.9 ?)

hiredman18:01:55

@dave.dixon I would put some kind of cap on the size of the json file, and setup cheshire to not turn keys in to keywords, I doubt parsed-seq will do anything there

sparkofreason18:01:21

@hiredman Thanks. Theoretically the amount of data we need to accept could be large. I suppose I could cap it and make the user upload multiple files, though it complicates the application a bit.

hiredman18:01:14

the problem with json is it is a context free language, ideally you would want some kind of regular language

markmarkmark18:01:43

it is possible to get an actual pid from Java with 1.9

noisesmith18:01:21

awsome! great to hear

hiredman18:01:28

fun fact: years ago, the way the jvm started new processes on linux was implemented in such a way as to require (at least for an initial instant) the same amount of memory for the child process as the jvm required, which meant if you had a jvm with a 7 gig heap on a machine with 8 gigs of memory (and no swap) it would fail if you tried to System/exec

noisesmith18:01:13

wow - yeah the naiive way to spawn a process under linux would do that if you didn't have overcommit enabled

johnnyillinois19:01:57

Hi all, Does anyone know how to run a function after all tests are complete? Sort of like a fixture around all tests?

noisesmith19:01:17

the simplest thing would be to make your own function invoking clojure.test (that wouldn't help if you are using eg. lein test though)

johnnyillinois19:01:49

I would like to be able to use lein test

johnnyillinois19:01:03

Would creating my own test runner be an option?

noisesmith19:01:25

I'm not sure - I guess worst case you fork lein test as your own plugin

johnnyillinois19:01:15

Yeah, I can look into that. The actual use case is that: my tests start a selenium browser and the tests go much slower if I have to restart the browser between namespaces. I'll look into it thanks

greglook19:01:25

@johnnyillinois you could use an alias in your project, something like this (using lein-shell):

:aliases {"test" ["do" ["shell" "start-browser-selenium"] ["test" ":selenium"] ["shell" "stop-browser-selenium"]]}

noisesmith19:01:04

oh yeah - that's a good call

johnnyillinois19:01:08

hey, greg can you help me out

noisesmith19:01:19

there's also :prep-tasks (or is that only for :uberjar ?)

johnnyillinois19:01:25

the same process that runs the tests, needs to be able to access the selenium browser

greglook19:01:25

that’s meant for builds

grzm19:01:58

With the clj and clojure CLI tools, what’s the practical purpose for having both -R and -C? When would I want them to be different?

johnnyillinois19:01:29

Doesn't that command have the selenium browser running if in a different process?

greglook19:01:00

Why do you need it to run in the same process? Wouldn’t the browser always be in a different process anyway?

grzm19:01:01

I understand the usefulness of having separate resolve-deps and make-classpath functions in the code, but I’m not clear on why they should be exposed to the user.

bellis19:01:49

When playing around with clj and deps.edn I had the same feeling. https://clojurians.slack.com/archives/C03S1KBA2/p1517340478000678

grzm19:01:39

boot-tools-deps (https://github.com/seancorfield/boot-tools-deps/blob/master/src/boot_tools_deps/core.clj#L135) provides -A to specify a list of aliases to provide to both.

grzm19:01:21

@moxaj thanks for the link! Voted!

johnnyillinois21:01:31

That's a good point @grzm. I guess I'm using: https://github.com/semperos/clj-webdriver/tree/v0.7.2 so I can look into how to configure it to use an already running driver

grzm21:01:27

@johnnyillinois I’m not sure I follow. I kinda jumped into the middle of your conversation with a completely unrelated topic. Apologies if that caused some confusion. Looks like you’re discussing Leiningen config?

johnnyillinois22:01:45

Yeah, don't worry about thanks grzm

joshmiller23:01:23

I’m looking to sample from a probability distribution, i.e. either take n from a vector of probabilities like [0.2 0.4 0.3 0.1] (where I’d get 1 40% of the time) or do the same for a map like {:a 0.2 :b 04. :c 0.3 :d 0.1} — is this already implemented somewhere?

noisesmith23:01:15

there's a random-sample function which iirc was derived for a goofy thing I did on IRC once that was accidentally useful

noisesmith23:01:07

[email protected]:~/clojure/peregrine$ rlwrap bench
Clojure 1.9.0
+user=> (doc random-sample)
-------------------------
clojure.core/random-sample
([prob] [prob coll])
  Returns items from coll with random probability of prob (0.0 -
  1.0).  Returns a transducer when no collection is provided.
nil
+user=> (random-sample 0.3 (range 100))
(2 4 5 7 9 18 28 30 32 34 39 58 59 63 64 66 69 71 73 74 75 76 80 89 98)

joshmiller23:01:31

Yeah, I came across that, but I have a pre-existing probability distribution, rather than the sample probability for each element and it didn’t look like I could specify that there.

noisesmith23:01:42

check the code for random-sample - it's very simple and you can make something new using map (map takes multiple collection args, so the first could be probabilities and the second elements)

noisesmith23:01:59

s/map/mapcat/ (since we are arbitrarily dropping items use mapcat)

noisesmith23:01:43

something like (mapcat (fn [p e] (when (< (rand) p) [e])) probs elts)

joshmiller23:01:51

Aha, cool, will do, thanks.

noisesmith23:01:08

actually I think that code might just be it!

noisesmith23:01:57

+user=> (defn psample [probs elts] (mapcat (fn [p e] (when (< (rand) p) [e])) probs elts))
#'user/psample
+user=> (psample [0.0 1.0 0.0 1.0] (range))
(1 3)
looks good to me