Fork me on GitHub
#clojure
<
2017-08-10
>
novel09:08:00

@qqq have you already tried lisping (http://slidetocode.com/lisping)?

novel09:08:22

even tough it has no REPL support for Clojure (only Scheme), I like it for code editing and navigation.

lambder11:08:24

which channel is best to ask for s3-wagon-private help?

sundarj11:08:06

#aws perhaps?

ido11:08:11

Hey clojurists. I am using a library and need to bind a var a few times. Is there a way to do a 'one off' binding for a whole namespace? push-thread-bindings which is used internally by binding has a creepy "WARNING: This is a low-level function" in the docs.

misha12:08:56

@ido you cannot guarantee, that nothing in ns will send some closures to another thread, and/or that if so – sending those ns-wide bindings will be actually ok, can you? Or is it ok for your use case?

ido12:08:55

I am ok with this @misha

misha12:08:29

that's all I had opieop

misha13:08:40

is there any difference in having this top level form at the beginning of the specs ns, and having it declared and called as a macro?

(let [pairs [['foo 'x.y.z.foo]
             ['bar 'x.y.z.bar] ...]]  ;; there would be like 10+ namespaces listed here
  (doseq [[a n] pairs]
    (create-ns n)
    (alias a n)))
Are there any obvious problems with this, or any better alternatives? (intentions is just to get short aliased keywords for spec names, like ::foo/id and ::bar/id

misha13:08:15

apparently, cursive's rename keyword feature does not see those create-ns/alias inside let, and renaming ::bar/id will drop its namespace alias. cc @cfleming

misha15:08:17

swap! is synchronous, right? It does not return until fn is successfully applied, does it?

jeremys15:08:56

@misha that’s right

misha15:08:31

Is there an equivalent of compare-and-set!, but only for a sub-value in an atom (like single kv in a map)?

misha15:08:01

how cheap are set-validator! calls? are those atomic?

lvh16:08:14

@misha no, if you want that you may want multiple atoms, or refs

lvh16:08:45

(it’s likely you don’t want that)

misha16:08:42

@lvh, I did this so far:

(defn lock! [!machine]
  (let [m @!machine]
    (if (:locked? m)
      false
      (compare-and-set! !machine m (assoc m :locked? true)))))

(defn transition! [!machine]
  (when (lock! !machine)
    (swap! !machine f)

lvh16:08:28

I don’t understand what you’re trying to do, but I especially don’t get why you’d make that :locked explicit

lvh16:08:49

I’d just use swap! and if you really care about concurrent updates, 2 atoms if you don’t need coordination/transactionality between them, and 2 refs if you do

misha16:08:55

basically, transition!should be no-op if machine is locked

misha16:08:13

I need to ensure only 1 concurrent worker advancing machine. But many incoming events.

misha16:08:38

I want single atom, which accepts new events in a queue at any point in time. But has only 1 worker processing events at a time.

hiredman16:08:24

are you sure you don't want an agent?

misha16:08:38

I am not sure at all : )

hiredman16:08:57

agents queue up actions to execute and a single action is executed at a time

misha16:08:54

Another requirement: event handlers might contain side effects, so I can't afford to retry those. Probably this is where "single system-wide(?) worker at a time" comes from

hiredman16:08:31

because a single action is executed at a time from a queue agents never need to re-run actions

misha16:08:34

but can I enqueue new events at the same time?

misha16:08:54

or if I'll have an agent, I'd need to have separate ref for events q?

misha16:08:08

yeah, reading this right now

misha16:08:48

I might be holding too tight to an idea of a single thing (atom/ref/agent)

misha16:08:05

https://clojurescript.org/about/differences

Agents are currently not implemented

misha16:08:30

so I'd need to make an (alternative) atom implementation for cljs

leonoel16:08:14

an agent is, basically, a persistent queue inside an atom. it's not very difficult to implement

leonoel16:08:43

but it would be better if they were implemented in cljs, that's for sure

misha16:08:25

since cljs is a single thread env (right?), very naive implementation will suffice, so it should be just an inconvenience to have 2 implementations, rather than 1

leonoel16:08:59

yes, and you probably don't need all features of clojure agents, anyway

hiredman16:08:42

clojurescript is going to be an entirely different animal there

hiredman16:08:41

you will effectively never see a swap! retried in clojurescript unless the function that you pass to swap! also calls swap! on the same atom

misha16:08:21

if I use agent in a lib, how should I clean those up? shutdown-agents is system wide, and is not appropriate for a lib's teardown function, I think. Should I just put a note in README?

hiredman16:08:17

if you don't send them actions to run they just get gc'ed

misha16:08:10

I wonder if I should put any safety flag in an agent value, in case some user's event handler will emit events in an infinite (go?) loop. So that agent can reject any new jobs. Or am I overthinking it?

hiredman16:08:54

in general, I get the impression that people who work on clojurescript have never used clojure for a significant amount of time, so clojurescript often does what they think it does, and not what clojure actually does

misha16:08:07

is this a general observation, or for something specific above? (single threaded env does indeed atrophy "concurrency thinking" to some degree)

hiredman17:08:46

just me whining

langmartin17:08:46

I need some type advice; I've used spec to describe a really big map of related data that is passed around between a bunch of functions. when there's a spec violation, the error message contains all the arguments, and emacs goes to a dark place

langmartin17:08:58

my initial idea is to create a new type with a different toString but leave it enough of a map to be a valid s/keys container. is that reasonable?

hiredman17:08:02

or just don't dump it all out?

langmartin17:08:05

second idea is to remove specs from those functions and only put them farther down in the details where the world object has been broken up already

langmartin17:08:53

I don't know how to avoid dumping it all out, the cider error buffer gets the spec error, and the spec error ignores print-len

hiredman17:08:32

there is some binding spec uses

hiredman17:08:06

explain-out

hiredman17:08:11

*explain-out*

hiredman17:08:50

it is bound to the function used to print explain data, so fiddle with that

fominok17:08:23

hey! Is there any newbie corner for questions?

sundarj17:08:46

@fominok #beginners 🙂

fominok17:08:19

very thanks!

sundarj17:08:29

no problem!

yedi18:08:35

whats the most clojuric way of doing (for [item items] (do stuff)) but also binding the index as well as item

noisesmith18:08:28

(for [[index item] (map-indexed list items)] ...)

pesterhazy19:08:55

(map-indexed identity items), no?

misha19:08:19

@pesterhazy map-indexed expects arity-2 function, identity is arity-1, it'll throw in clojure, might work somehow in clojurescript

pesterhazy19:08:53

I think I usually use (map vector (range) [:a :b :c])

rauh19:08:18

@yedi In addition, if you can guarantee that items is a vector, you can just use reduce-kv

noisesmith19:08:06

@pesterhazy I prefer list over vector when the collection only gets created in order to pull it apart again - the difference between a couple conses and a 16 slot array

noisesmith19:08:18

but that’s a personal preference of course, perf likely doesn’t matter here

tbaldridge19:08:54

@noisesmith the 16 slot array (and isn't it 32) doesn't get created at first. Vectors have a tail that grows by 1 until it reaches 32 elements, then it's pushed into the tree.

tbaldridge19:08:56

And clojure has some "borrow" semantics around arrays, so that [1 2] only allocates an array (needed for the vararg call) and then that array is wrapped with a vector object.

javazquez19:08:56

what is the recommended way to consume streaming APIs with Clojure?

tbaldridge19:08:32

@javazquez define "streaming API". Streaming XML, audio, video? Is it streaming over HTTP SSE, long polling, websockets, etc.

noisesmith19:08:48

@pesterhazy @tbaldridge thanks for the info, vector on two args benchmarks as 3x faster than list so my only remaining argument is character count 😄

noisesmith19:08:28

criterium results for anyone who cares

peregrine.circle=> (crit/bench (vector 1 2))
WARNING: Final GC required 10.2356120766764 % of runtime
WARNING: Final GC required 1.0077331583723619 % of runtime
Evaluation count : 4943172240 in 60 samples of 82386204 calls.
             Execution time mean : 10.204242 ns
    Execution time std-deviation : 0.301496 ns
   Execution time lower quantile : 9.784033 ns ( 2.5%)
   Execution time upper quantile : 10.777083 ns (97.5%)
                   Overhead used : 2.257806 ns

Found 1 outliers in 60 samples (1.6667 %)
        low-severe       1 (1.6667 %)
 Variance from outliers : 15.8380 % Variance is moderately inflated by outliers
nil
peregrine.circle=> (crit/bench (list 1 2))
Evaluation count : 1730757660 in 60 samples of 28845961 calls.
             Execution time mean : 33.978224 ns
    Execution time std-deviation : 1.770012 ns
   Execution time lower quantile : 32.142992 ns ( 2.5%)
   Execution time upper quantile : 38.101712 ns (97.5%)
                   Overhead used : 2.257806 ns

Found 7 outliers in 60 samples (11.6667 %)
        low-severe       4 (6.6667 %)
        low-mild         3 (5.0000 %)
 Variance from outliers : 38.4645 % Variance is moderately inflated by outliers
nil

noisesmith20:08:44

@pesterhazy BUT - map-indexed with list is twice as fast as map with vector and range - of course the (map-indexed vector coll) version is fastest though - just a little bit faster than the version using list

pesterhazy20:08:33

interesting to know, thanks for running the numbers

hlolli21:08:54

Pipeing question, I want to do a basic pipe operation from clojure string into an application, the application in question can receive from stdin line this cat file.txt | app -, insetead of the cat, would I use byte-buffer writer and send it to an exec of getRuntime. Maybe some code example would get me going..

bfabry21:08:34

I would use the ProcessBuilder api

bfabry21:08:17

here's an example I have handy

(defn run-command-stream-output
  "Runs a shell command and streams the output.
  An exception is thrown if the exist status is non-zero."
  [command]
  (log/info "headphones.common.shell/run-command-stream-output:: " command)
  (let [process (.start (ProcessBuilder. (into-array String command)))]
    (future (io/copy (.getInputStream process) (System/out)))
    (future (io/copy (.getErrorStream process) (System/err)))
    (if (not= (.waitFor process) 0)
      (throw (Exception. (str "Failed command '" (clojure.string/join " " command) "'"))))))

bfabry21:08:15

you'd be wanting to getOutputStream instead of getInputStream but other than that pretty similar

hlolli21:08:25

ah ok, it's part of Java. ok nice.

hlolli21:08:47

no just send text into the app, the output is a side effect (no return value or stdout stuff happening)

bfabry21:08:26

yeah but what I mean is, the function to call on the process to get stream to write to it, is getOutputStream 🙂

hlolli21:08:49

ah ok I see

bfabry21:08:14

it's kind of the opposite of what you'd expect. getOutputStream returns an OutputStream and anything you write to it will go to stdin of the proc. vice versa for getInputStream. it makes sense if you think about it for a bit but it's definitely surprising first up

hlolli21:08:26

it's defenitely confuseing in terms of what is going where. Of course the binary has its own stream mechanism, reading what you wrote, the binary would implement something on the lines of getInputStream.

hlolli21:08:58

But how does the clojure string become a stream, with the into-array and process builder above?

bfabry21:08:45

the into-array is just building the command

bfabry21:08:45

to move the clojure string into the stream of the process's stdin you'd do something like (.write (.getOutputStream process) (.getBytes string))

hlolli21:08:24

ok nice, I was looking at spit, your bit looks more legit.

hlolli22:08:47

my app just hangs with this, maybe .getBytes needs to be some sort a buffer?

(ly-compile-from-string ["lilypond" "-"] basic-string)

(defn ly-compile-from-string
  [command ly-str]
  (let [process (.start (ProcessBuilder. (into-array String command)))]
    (future (.write (.getOutputStream process) (.getBytes ly-str)))
    (future (io/copy (.getInputStream process) (System/out)))
    (future (io/copy (.getErrorStream process) (System/err)))
    (if (not= (.waitFor process) 0)
      (throw (Exception. (str "Failed command '" (clojure.string/join " " command) "'")))
      (println "finished"))))

bfabry22:08:30

you probably want to wrap the output stream in a PrintWriter or something

bfabry22:08:56

the lilypond process actually ends though?

hlolli22:08:17

no, it just hangs, I se the print from stderr showing it starts just as expected, but it's waiting/hanging

bfabry22:08:06

sounds like your executable, lilypond, is never finishing

hlolli22:08:37

well, it finished the same string if it's either echoed and piped or in a file cat piped.

bfabry22:08:59

the general pattern appears to work

user=> (def process (.start (ProcessBuilder. (into-array String ["read"]))))
#'user/process
user=> (future (.waitFor process) (println "finished"))
#<Future@ef0ece1 pending>
user=> (def writer (java.io.PrintWriter. (.getOutputStream process)))
#'user/writer
user=> (.println writer "bash")
nil
user=> (.close writer)
nil
finished
user=>

hlolli22:08:47

wait, try it...

bfabry22:08:52

user=> (def process (.start (ProcessBuilder. (into-array String ["read"]))))
#'user/process
user=> (future (.waitFor process) (println "finished"))
#<Future@6d354b41 pending>
user=> (def writer (java.io.PrintWriter. (.getOutputStream process)))
#'user/writer
user=> (.println writer "bash\n")
nil
user=> (.flush writer)
nilfinished

bfabry22:08:05

read command exits on newline

hlolli22:08:17

this is what Im running, not finishing

(def basic
  "  \\version \"2.19.59\"

  {
   c' e' g' e'
   }
  ")

(ly-compile-from-string ["lilypond" "-"] basic)

(defn ly-compile-from-string
  [command ly-str]
  (let [process (.start (ProcessBuilder. (into-array String command)))
        writer (java.io.PrintWriter. (.getOutputStream process))]
    (.println writer ly-str)
    (.flush writer)
    (future (.waitFor process) (println "finished"))))

the command line that succeeds
cat  | lilypond -

bfabry22:08:08

does the lilypond process wait for its input stream to be closed?

bfabry22:08:14

you may need to call .close on writer

bfabry22:08:27

it's really dependent on the command you're executing what its stop conditions are

bfabry22:08:01

cat http://test.ly | lilypond - will close the input stream once it's exhausted http://test.ly

hlolli22:08:07

good question

bfabry22:08:21

what happens if you just do $ lilypond - and type the data in, do you need to hit ^D to make it stop?

hlolli22:08:59

yup I need to hit ^D to make it stop

bfabry22:08:13

oh ok, so you just need to call (.close writer) once you're done writing to it

bfabry22:08:25

that will tell the lilypond process that you're done sending it data, and it will end

hlolli22:08:39

so it would be a good move to write line by line and do close when the last line is written?

bfabry22:08:57

shrug I don't think it would make a difference. so long as you call .close when you're done

hlolli22:08:27

hehe ok I try to add close into the sequence...

hlolli22:08:21

got it, thanks a million for patience

hlolli22:08:41

it sends finished and the .pdf file I expected was there. Nice, thanks a lot!

bfabry22:08:43

an example with cat, which has similar behaviour to lilypond by the sounds

user=> (def process (.start (ProcessBuilder. (into-array String ["cat" "-" ">" "foo.txt"]))))
#'user/process
user=> (future (.waitFor process) (println "finished"))
#<Future@204d0de5 pending>
user=> (def writer (java.io.PrintWriter. (.getOutputStream process)))
#'user/writer
user=> (.println writer "blah")
nil
user=> (.close writer)
nilfinished

bfabry22:08:54

no worries!