This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-01-30
Channels
- # arachne (23)
- # bangalore-clj (2)
- # beginners (64)
- # boot (20)
- # cider (3)
- # clara (11)
- # cljs-dev (29)
- # cljsrn (10)
- # clojure (143)
- # clojure-brasil (4)
- # clojure-dev (22)
- # clojure-dusseldorf (3)
- # clojure-italy (26)
- # clojure-sanfrancisco (13)
- # clojure-seattle-old (2)
- # clojure-spec (15)
- # clojure-uk (27)
- # clojured (1)
- # clojurescript (52)
- # core-async (13)
- # cursive (2)
- # datomic (106)
- # fulcro (45)
- # garden (1)
- # graphql (11)
- # hoplon (98)
- # jobs (11)
- # juxt (7)
- # keechma (2)
- # leiningen (36)
- # off-topic (39)
- # parinfer (13)
- # re-frame (34)
- # reagent (5)
- # ring (1)
- # rum (4)
- # shadow-cljs (83)
- # sql (1)
- # timbre (1)
- # unrepl (49)
- # vim (1)
- # yada (42)
> Likewise, bindings created with binding can be assigned to, which provides a means for a nested context to communicate with code before it on the call stack.
can anyone produce an example where you would want to do this?
so you wouldn’t have any data escape the top-level binding
right?
i guess i don’t see why you wouldn’t just return the “extra information” with your regular information
but perhaps that is just a lisp idiom?
@alexstokes that's the clean way to do it, but it also requires changing every function in between to carry that contextual data
but the encode/decode functions are value-focused - you get back the encoded bytes or the decoded value
instead, I had the codec implementations record the headers written or read to a dynamic var on the side
and sometimes you are not at liberty to change that data
@greg316 link to this code?
https://github.com/multiformats/clj-multistream/blob/0.5.1/src/multicodec/header.clj#L18
I have since rewritten it and gotten rid of that bit, but it did the job for a while 🙂
@alexstokes especially consider the situation where you might have multiple threads doing the operation, so it isn't safe to use a top level var either - and code that has to be "in the middle" can't pass the value in or out
https://github.com/multiformats/clj-multistream/blob/0.5.1/src/multicodec/header.clj#L18
@noisesmith yeah makes sense
thanks @greg316
it is definitely not the simplest way to do things, but sometimes it’s the only option you have
I think in every option where you do that you could also use an instance of an object that is contextually shared by multiple higher order functions - but that's much messier
eg. you'd have a codec-context-factory that creates a codec-context, and it would give you a decoder function and a driver function that you'd use together - I'm sure you can imagine how clumsy that would be to use
if that architecture makes sense to you you're probably happily using spring with no use for clojure 😄
fair enough
Anyone know what might be causing this? Basically, there's a nested dependency in this codax library for an old version of encore. Using lein's :managed-dependencies
for a newer version of encore or using :exlcusions
on the codax dependency and manually a newer version of nippy doesn't work. The only luck I have is by cloning the codax repo and changing it to use a newer version of nippy which no longer depends on an old version of encore. Shouldn't that be the same thing as doing [com.taoensso/timbre :exclusions [com.taoensso/nippy]]
?
I'm testing some kafka streams code, but the relevant part is about the reference to a function (a var). I have code like below and the problem is that when used directly at the pipeline (at .filter
), this pipeline sees the any change at the function, but when used like the commented line below (at kf/filter
), it doesn't
(->
(.stream builder new-electric-data-topic)
(kf/map-values avro->clj)
(kf/to-stream)
(.filter (reify Predicate (test [_ k v] (voltage-event? k v)))) ;;; THIS WORKS when changing the function at REPL
#_ (kf/filter voltage-event?) ;; BUT WHEN I USE THIS, I don't get the change, have to restart the stream
(kf/peek peek-test))
The function that I'm changing is voltage-event?
It's just one-line with true
or false
for this test
And kf/filter
is below
(defn filter
[stream f]
(.filter stream (reify Predicate (test [_ k v] (f k v)))))
The problem could be that I return a different stream every time I call this function (because of immutability, maybe I should use a macro?)
Or a function calling your defined function doesn't retain the reference to the original var, but I doubt it
is it too insane to change all (fn [...])
to (fn some-name [...])
for the sole purpose that it'll be easier to read stack frames ?
good idea uhaeuaehueh I'll start to use this
I do that, it's great - it also acts as a brief code documentation
I need a macro which does this:
(indent-println-by-2
...)
then for any printlns executed in that region, it gets indented by 2 spaces ... this would drastically simplify prnitln debugging in making things easier to read@pfeodrippe that happens because vars used inside a function are looked up when used, but when you pass a function by name to another function or method, it's looked up once on that invocation and reused - which is an issue for long running things
@pfeodrippe you can work around this by passing a var in instead of the function itself eg. #'voltage-event?
well, pprint already takes a writer, that could be a string-writer
peregrine.circle=> (let [sw (java.io.StringWriter.)] (pprint {:a 0 :b 1 :c 2 :d [1 2 3] :e (range 20)} sw) sw)
#object[java.io.StringWriter 0x232b391e "{:a 0,\n :b 1,\n :c 2,\n :d [1 2 3],\n :e (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19)}\n"]
peregrine.circle=> (doseq [line (clojure.string/split-lines (str *1))] (println " " line))
{:a 0,
:b 1,
:c 2,
:d [1 2 3],
:e (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19)}
nil
@noisesmith Understood, thanks! I've changed it to be a defmacro instead of a defn (just a few changes at the body), so the user doesn't need to pass a var every time, but now I know if I need it again 😃
with-out-str does more than you need though, since pprint already takes a writer arg
@noisesmith but I'd guess he wouldn't want to modify the code, just sprinkle in some indent-println-by-2
oh you wanted println not pprint? never mind
@noisesmith, @moxaj: right, what moxaj said, I dont want to modify underlying code
then you need to mess with *out*
OK, then you need with-out-str I guess - be careful, this will totally break with lazy processing
you need to force all laziness that prints before exiting that block, or else the indentation won't apply
but you already signed up for pain if you use IO in laziness
and I think that's why nobody has "resolved this issue" - laziness makes it very messy and we use lazy things a lot
you're saying, it's possible that (foo) runs fine, but it returns a thunk, which when executed, throws an exception / does println
I'm saying that if the lazy tail of foo prints, it won't use the indentation you wrapped around it's construction
unless forced inside that scope
and, if some other lazy tail is forced in that scope, it uses that scope's indentation
which might be fine for your usage, but would get a lot of complaints and broken results if provided as a general indented printer facility
I want to drive an interactive program from Clojure; anyone familiar with code for doing that, or should I be just generally looking at transducers to interact asynchronously with InputStream
s and OutputStream
s?
I'm not sure that transducers make sense, but what you're describing is essentially REPL (just IS/OS not Reader/Writer)
Nope, I'm talking about something that will start an instance of bash
, ssh
into another machine, and run commands there 🙂
I wanna have something vaguely like https://en.wikipedia.org/wiki/Expect but no tcl/tk
yeah, pallet works through ssh'ing in and running a series of shell commands to move the system through its init states
@benjaminster I've had best luck with using ProcessBuilder and Process to do streaming IO to shells
yeah, it looks like that plus some way to turn an InputStream into something more easily digested by pure functional code would be one way to go
be very careful about that transformation - for example don't use laziness it just makes things more complicated
that's what I was saying in the other message about conch - an abstraction that forces non-io code to care about io concerns is worse than no abstraction at all
I tried using conch but found that it added complexity without actually simplifying my task which is matched pretty well by the built in classes (fundamentally at that layer I want to do imperative IO, lazy-seqs only add bugs at that level)
come to think of it you could make something that converts an IO channel to an IReduceInit though...
conch has some interesting ideas but is (as you pointed out) aimed at a different target...
I think I have a gist of doing this - digging
I want to have timeouts and such so... I think something that gobbles up an inputstream and bleeds it over a core.async
channel might be a good funnel to get the pipeline going
We have a requirement to allow users to upload a JSON file, parse the records, and stream to kafka. I'd like to harden the process so it doesn't easily cause out-of-memory exceptions when the file is somehow goofy. Currently planning to use parsed-seq
from cheshire, not clear how/if it protects against this scenario. Or am I worrying too much?
that gist is super simplistic, but I think it has most of the moving parts you need to do something interesting with streaming input / output from a task. The annoying thing is the JVM gives you no straightforward way to get the PID of your task - you get an abstraction that lets you shut it down or check if it's still running but no PID 😦
thanks! oh gosh, that's a classic Java API with misnamed methods... .getOutputStream
gives you what most would call the process' stdin and .getInputStream
gives you stdout
haha yeah - as far as Java is concerned it's the OutputStream because that's where java can put output haha
that's the only way it makes sense at least
it helps my mental model if I don't think of a Process object as the OS process, but rather the ambassador for one inside the jvm, so it's outputstream is the stream that sends your output to the process input - that's a better fit for what's going on on a low level anyway
also don't forget to join the error stream with the input stream, or set up handlers for both
but any pure JVM solution is going to have that problem (unless it was fixed in 1.9 ?)
@dave.dixon I would put some kind of cap on the size of the json file, and setup cheshire to not turn keys in to keywords, I doubt parsed-seq will do anything there
@hiredman Thanks. Theoretically the amount of data we need to accept could be large. I suppose I could cap it and make the user upload multiple files, though it complicates the application a bit.
the problem with json is it is a context free language, ideally you would want some kind of regular language
it is possible to get an actual pid from Java with 1.9
awsome! great to hear
fun fact: years ago, the way the jvm started new processes on linux was implemented in such a way as to require (at least for an initial instant) the same amount of memory for the child process as the jvm required, which meant if you had a jvm with a 7 gig heap on a machine with 8 gigs of memory (and no swap) it would fail if you tried to System/exec
wow - yeah the naiive way to spawn a process under linux would do that if you didn't have overcommit enabled
Hi all, Does anyone know how to run a function after all tests are complete? Sort of like a fixture around all tests?
the simplest thing would be to make your own function invoking clojure.test (that wouldn't help if you are using eg. lein test
though)
hmm... yup
I would like to be able to use lein test
Would creating my own test runner be an option?
I'm not sure - I guess worst case you fork lein test as your own plugin
Yeah, I can look into that. The actual use case is that: my tests start a selenium browser and the tests go much slower if I have to restart the browser between namespaces. I'll look into it thanks
@johnnyillinois you could use an alias in your project, something like this (using lein-shell
):
:aliases {"test" ["do" ["shell" "start-browser-selenium"] ["test" ":selenium"] ["shell" "stop-browser-selenium"]]}
oh yeah - that's a good call
hey, greg can you help me out
there's also :prep-tasks (or is that only for :uberjar ?)
the same process that runs the tests, needs to be able to access the selenium browser
With the clj
and clojure
CLI tools, what’s the practical purpose for having both -R
and -C
? When would I want them to be different?
Doesn't that command have the selenium browser running if in a different process?
Why do you need it to run in the same process? Wouldn’t the browser always be in a different process anyway?
I understand the usefulness of having separate resolve-deps
and make-classpath
functions in the code, but I’m not clear on why they should be exposed to the user.
When playing around with clj
and deps.edn
I had the same feeling. https://clojurians.slack.com/archives/C03S1KBA2/p1517340478000678
boot-tools-deps (https://github.com/seancorfield/boot-tools-deps/blob/master/src/boot_tools_deps/core.clj#L135) provides -A
to specify a list of aliases to provide to both.
@grzm https://dev.clojure.org/jira/browse/TDEPS-15. I think I recall @alexmiller saying it's coming
That's a good point @grzm. I guess I'm using: https://github.com/semperos/clj-webdriver/tree/v0.7.2 so I can look into how to configure it to use an already running driver
@johnnyillinois I’m not sure I follow. I kinda jumped into the middle of your conversation with a completely unrelated topic. Apologies if that caused some confusion. Looks like you’re discussing Leiningen config?
Yeah, don't worry about thanks grzm
I’m looking to sample from a probability distribution, i.e. either take n
from a vector of probabilities like [0.2 0.4 0.3 0.1]
(where I’d get 1
40% of the time) or do the same for a map like {:a 0.2 :b 04. :c 0.3 :d 0.1}
— is this already implemented somewhere?
there's a random-sample function which iirc was derived for a goofy thing I did on IRC once that was accidentally useful
+justin@fuckmonkey:~/clojure/peregrine$ rlwrap bench
Clojure 1.9.0
+user=> (doc random-sample)
-------------------------
clojure.core/random-sample
([prob] [prob coll])
Returns items from coll with random probability of prob (0.0 -
1.0). Returns a transducer when no collection is provided.
nil
+user=> (random-sample 0.3 (range 100))
(2 4 5 7 9 18 28 30 32 34 39 58 59 63 64 66 69 71 73 74 75 76 80 89 98)
Yeah, I came across that, but I have a pre-existing probability distribution, rather than the sample probability for each element and it didn’t look like I could specify that there.
check the code for random-sample - it's very simple and you can make something new using map (map takes multiple collection args, so the first could be probabilities and the second elements)
s/map/mapcat/ (since we are arbitrarily dropping items use mapcat)
something like (mapcat (fn [p e] (when (< (rand) p) [e])) probs elts)
Aha, cool, will do, thanks.
actually I think that code might just be it!
+user=> (defn psample [probs elts] (mapcat (fn [p e] (when (< (rand) p) [e])) probs elts))
#'user/psample
+user=> (psample [0.0 1.0 0.0 1.0] (range))
(1 3)
looks good to me