Fork me on GitHub
#clojure
<
2018-03-01
>
currentoor01:03:52

if i have a defmacro inside a defmacro what’s the correct way to access the inner macro’s &env?

currentoor01:03:29

is it ~'&env?

noisesmith01:03:39

yes, it should shadow the other

noisesmith01:03:51

but you might find things much nicer if you can avoid nested defmacro

noisesmith01:03:51

currentoor - wait, to access the inner &env, use enough ~ to get out of the inner macros but not enough to get out of the surrounding macro's

noisesmith01:03:26

you can write a function that takes the macro form and returns the processed / fixed form, it might help restore some sanity

currentoor01:03:48

@noisesmith looks like ~'&env is working

noisesmith01:03:34

sounds right - I had to be more general because I can't assume how many layers of are in play - I write macros that have no , I write macros that have multiple levels of `...

currentoor01:03:49

i agree nested macros should be avoided but i have a particular use case where the performance improvements and connivence justify it, IMHO

noisesmith01:03:22

sure, but even then you can just make a defn that takes a list of symbols as an arg, and returns a new list of symbols that happens to contain defmacro

noisesmith01:03:47

I'm using the weechat slack interface, i just realized most of what I wrote above looks weird because of how slack uses `

currentoor01:03:58

lol no worries

currentoor01:03:23

not sure what you mean by > returns a new list of symbols that happens to contain defmacro

noisesmith01:03:59

a macro is a function (handled specially by the compiler) that happens to return a list of symbols for the compiler to eval

noisesmith01:03:34

which means that you can often fix the problem of a macro being too complex by writing a function that takes a list of symbols (the args to the macro) and returns a list of symbols (the expansion)

currentoor01:03:03

thanks but i want to turn (div {:foo "bar"}) into (js/React.createElement #js {:foo "bar"}) in cljs and (div-element {:foo "bar"}) in clj

currentoor01:03:10

for all the html elements

currentoor01:03:17

i’m not sure that would help here

matan01:03:52

just curious, I have some code writing a huge collection to a file, the collection being generated through vanilla code of mine involving some recursion. Running the code, I observe all of my cores close to 100% utilized. It's always nice to see hardware fully utilized, but how can this actually be? Nothing special in how I write to the file either

(with-open [file (  "output.txt"  :append false)]
  (doall (map #(.write file (str %)) result)))

hiredman01:03:47

your core is pegged rrying to run the gc to recover memory would be my guess

hiredman01:03:14

don't use map+doall use doseq

matan01:03:22

mmmm good guess, I should profile it

matan01:03:09

Just a thought ― would GC really utilize 12 cores though?

mfikes01:03:15

A stretch at a theory: If result is lazily computed, perhaps using pmap, it could crank up all the cores

noisesmith01:03:49

@matan the jvm gc is concurrent, it can use all the cores you let it use

matan01:03:51

yes, result is lazily computed, as lazy is the clojure default, but I'm not using any pmap in my own code

noisesmith01:03:25

and regardless of whether the gc is the issue here, use run! because nobody needs those objects that it is putting in your heap otherwise

noisesmith01:03:44

and for other lazy things that you don't need to keep in memory check out dorun

hiredman01:03:56

I dunno, the use of doall+map like that would lead to me to believe there could be all kinds issues elsewhere

matan01:03:59

okay, re-reading about run! and dorun

ghadi01:03:26

either of those will work. doall retains the head of the sequence

ghadi01:03:54

Think of doall as an accordion stretching out a seq, and think of run/dorun as lighting a fuse on one end of the seq and burning through to the end

mfikes01:03:23

With run! really doing well on directly-reducible collections

matan02:03:12

Indeed using run! my cpu utilization is now dramatically reduced (pun not intended :))

matan02:03:49

Though I still have it occupying ~50% of the total cores' time

matan02:03:47

What would be your advice? or is it essentially inevitable having so much GC when looping in a tight loop generating data in clojure? I guess If I were to also incorporate something like pmap for generating the data, it would be at rough competition with the GC activity.

hiredman02:03:08

your loop isn't tight

hiredman02:03:34

because results is lazy, your loop body is effectively computing each part of it

matan02:03:20

which is wrong because? what am I missing?

matan02:03:36

the data has to be computed I mean

hiredman02:03:26

sure, you want to compute something, but calling that loop a tight loop when it is doing some arbitrary computation in the middle of it which you haven't described is odd

matan02:03:15

I'm not sure how you mean that, but I tend to think it is just a typical artefact of clojure's immutability paradigm ― lots of objects being allocated when using an idiomatic clojure coding style centered around immutable data. https://purelyfunctional.tv/article/java-generational-garbage-collection/

hiredman02:03:23

I mean, because results is lazy, whatever computation (who knows what, you haven't said what it is) that goes in to building each step of it, is now happening in the middle of your loop, so it cannot really be considered "tight" (a tight loop meaning a loop that does very little each iteration and executes many times)

hiredman02:03:14

because of the laziness, there is not enough context in the loop to determine what is going on when the loop loops

hiredman02:03:37

so it is difficult for a person you just share the loop with to tell you what is happening

matan02:03:33

Yes of course, drop the "tight loop" term then; just a loop. There's a lot of CPU intensive code involved in generating the data, and I assume it is the general clojure case of creating many objects when deriving data from data in very idiomatic clojure code.

hiredman02:03:05

it could be, but I would be hesitant to assume that

matan02:03:22

I think sharing the specific code would just be tedious for anyone out here

hiredman02:03:40

my assumption, given the doall+map code is you are not very familiar with clojure and there could be any number of problematic idioms lurking in there

hagmonk02:03:12

@matan reflection generates garbage too

gonewest81803:03:00

Anyone using Circle CI 2.0 for their clojure projects… are you building your own Docker images with Oracle JDK? Testing only with the available OpenJDK images? Using VM executors? ?

jgh03:03:49

do you mean for the actual build/test process or for the result?

jgh03:03:40

for building

jgh03:03:53

for the result i use openjdk:9-jre

jgh03:03:16

and copy the uberjars into it via Dockerfile

gonewest81803:03:23

I meant for the testing. For instance, projects that build and test against Oracle JDK 7,8, and 9 and OpenJDK 7,8, and 9 in order to confirm no JDK specific issues. The official clojure images on dockerhub are OpenJDK based, only.

jgh03:03:42

sounds like you might need to make your own images in that case

devn04:03:44

I was messing around with storing deps as the first form of the clojure source file itself.

'{:deps {cheshire {:mvn/version "5.8.0"}}}
(ns foo.bar (:require [cheshire.core :as json]))
(println (json/parse-string "{\"a\": 1}"))
~/bin/clj2:
#!/bin/sh
FILE=$!
clj -Sdeps "$(clj --eval '(fnext (read-string (slurp "'"$FILE"'")))')" -i $FILE
clj2 foo.clj => {a 1}

dmarjenburgh08:03:16

@yogidevbear Sounds like you want something like a flood-fill algorithm?

(defn flood-fill [matrix starting-pos replacement]
  (let [target (get-in matrix starting-pos)
        paint (fn [m pos]
                (cond-> m
                  (= (get-in m pos) target) (assoc-in pos replacement)))
        neighbours (fn [m pos]
                     (when (= (get-in m pos) target)
                       (for [dir [[0 1] [0 -1] [1 0] [-1 0]]]
                         (mapv + pos dir))))]
    (loop [matrix matrix stack [starting-pos]]
      (if (and (seq stack))
        (let [pos (peek stack)]
          (recur (paint matrix pos) (into (pop stack) (neighbours matrix pos))))
        matrix))))

yogidevbear09:03:25

Wow, thanks for the reply @U05469DKJ 👍 I was very surprised to get feedback on the issue so long after posting about it. I found a solution that worked, but I'll take a closer look at what you sent over and see if I can learn something from it 🙂

delaguardo10:03:50

@foxlog clj -Sdeps "{:deps {hello {:local/root \"/tmp/hello-world\"}}}" -m hello something like that

Alex Miller (Clojure team)12:03:18

@foxlog clj and deps.edn are designed to primarily work in a “project” context where you have the deps.edn in the current directory. The example from @delaguardo treats it like a local dependency instead.

Alex Miller (Clojure team)12:03:38

If your question is really packaging and deployment, that’s a bigger topic

borkdude13:03:03

it would be nice if you could link to subsections in a guide on http://clojure.org.

sundarj13:03:13

click the links in the table of contents

borkdude13:03:42

oh, didn’t notice that, thanks

Alex Miller (Clojure team)14:03:53

All headers get an anchor by default. Ideally they would have something that showed up in the gutter on mouse over to easily grab those

sundarj14:03:39

could show an anchor link on hovering the headings like i've seen some sites do

sundarj14:03:46

oh sorry, i just realised that's what you said

sundarj14:03:42

need more caffeine

val_waeselynck10:03:47

@alexmiller I would happily contribute that but I thought you did not accept any contributions to the style of http://clojure.org

Alex Miller (Clojure team)13:03:14

If you wanted to tell me how to do it, I’d listen :)

val_waeselynck17:03:01

@alexmiller you can set the :sectanchors: AsciiDoctor document attribute, which will generate a a.anchor element inside each section title. You can then style that element with rules like

h2 > a.anchor {
  position: absolute;
  width: 1.5ex;
  margin-left: -1.5ex;
  display: block;
  text-decoration: none;
}

h2:hover > a.anchor:before, h3:hover > a.anchor:before, h4:hover > a.anchor:before, h5:hover > a.anchor:before, h6:hover > a.anchor:before {
  content: "\00A7";
}
I can open a Github issue with that information if that makes your life easier

danm15:03:41

Gah, I am getting a weird error

gtrak15:03:59

zip-visit is convenient for pre/postwalking https://github.com/akhudek/zip-visit

gtrak15:03:38

or specter might do it for you

joelv15:03:56

I'm building an webservice with compojure-api 1.1.12 and I'm having trouble getting the swagger integration to show my responses. This is what I have

(PUT "/replay/:message-id" []
      :header-params [authorization :- String]
      :path-params [message-id :- String]
      :summary "Replay stuff"
      :swagger {:data {:description "Replays stuff"
                       :responses {200 {:description "reply stuff" :schema s/Str}}}}
      (ok "true"))

joelv15:03:23

nevermind i figured it out

borkdude16:03:27

what is the channel to ask/learn about clojure deps/cli stuff?

mbjarland16:03:54

I have a question on conch (github Raynes/conch)…couldn’t find a channel for it so I’ll throw it out here and see if anybody has an idea. I’d like to spawn a process and while the process is running, print out the process output line-by-line. It seems to me (at least from my testing at the repl) that conch always completes execution before you get a chance to mess with the process out/err. If anybody has another clojure lib for dealing with shell execution and stream handling like this that would also be great. I know I can hand roll this myself by going directly against java process etc, but was hoping not to. Oh and from what I understand clojure.java.shell seems to have the same limitation

ghadi16:03:49

I'd hand roll it tbh

ghadi16:03:31

Grab the stdout inputstream and read it on a background thread while pumping into a channel or wherever

ghadi16:03:41

Usually these libs give access to the streams

mbjarland16:03:29

right, yeah, I’ve done this in groovy more times than I’d like to admit so not a blocker, was just hoping for an easy/ier out

noisesmith16:03:26

@mbjarland I found ProcessBuilder to generat a Process was much easier than using conch

mbjarland16:03:04

ok yeah I’ve seen that guy hanging around. Seems conch has a low level api as well with some helpers to deal with exactly this. I’ll give that a shot first and escape to java land once I fail there

noisesmith17:03:49

the thing is that once what you want is to consume and feed streams on a byte by byte basis, that’s exactly what ProcessBuilder gives you, the abstractions given by conch in best case give you exactly what the low level api would, in the realistic case make it more complicated to do the same task

noisesmith17:03:05

don’t give up simplicity for clojure syntax :P

mbjarland21:03:40

yeah, ended up going with processbuilder in the end. The reason I hesitated was I’ve built this kind of live-per-line handling a bunch of times in the past and it’s not difficult in theory but getting all the edge cases perfect is a bit of a pain

qqq16:03:37

Can someone point me at relevant documentation on whether I can assume that

(= Float/POSITIVE_INFINITY 
         (- (*
             Float/POSITIVE_INFINITY
             Float/POSITIVE_INFINITY)
            23))
always returns true (returns true for me right now)

noisesmith16:03:15

@mbjarland from the Process you can get InputStream (the stdout) ErrorStream (the stderr) and OutputStream (the stdin)

noisesmith16:03:34

@qqq the floating point spec

eraserhd17:03:11

Is there any way to filter exceptions during a test run? I'm using pedestal, which includes the config for my entire interceptor chain, and I'm getting 8000-line stack traces.

noisesmith17:03:22

@eraserhd I'm sure there's a way - with ring you can use a middleware that swallows all exceptions inside it, maybe someone in #pedastel can reveal the interceptor version of that

eraserhd17:03:11

Oh, that didn't occur to me, I'll go ask there.

bajrachar17:03:28

I've a question re: Using Transients inside reducers/fold - particularly inside combinef and reducef - I am trying to do a apply merge-with over transient coll inside combinef - Is there a way to do merge on transients? Thank you.

noisesmith18:03:39

@bajracher calling persistent! on a transient is cheap - it's literally flipping a bit, and if you are merging you are likely done with the transient

noisesmith18:03:56

which makes me wonder, is calling transient also cheap? I forget

schmee18:03:10

yes, it’s constant time

noisesmith18:03:25

awesome - hoping it's a low constant :P

noisesmith18:03:18

`public TransientHashMap asTransient() { return new TransientHashMap(this); } ... ok looks relatively cheap https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentHashMap.java#L291

bajrachar18:03:37

@noisesmith - mainly my motivation for using transient was also to be space efficient - not sure if that makes sense - I will be mutating the intermediate results quite a bit - its my impression that the persistent version will actually end up occupying more space as a result of immutability

noisesmith18:03:30

@bajrachar all persistent! does is flip a bit, they use the same internal storage, in fact they are the same object except for some extra data storage the transient uses

noisesmith18:03:43

@bajrachar the usage of transient isn't to reduce memory consumption - all hash-maps and vectors use the same internal tree structure following the same hashing and expansion rules when transient as when persistent - it's to reduce production of garbage and allow reassignment inside the original object instead of cloning leaves

noisesmith18:03:32

the difference is that instead of making a new tree that replaces a leaf, you edit a leaf - same memory usage in resulting structure, less short term garbage creation

noisesmith18:03:49

if your priority is to use less memory, just use a mutable collection (but use it smartly)

Alex Miller (Clojure team)18:03:51

that’s not how transients work

noisesmith18:03:22

I'd appreciate your explanation

Alex Miller (Clojure team)18:03:17

or I guess I’d say that shortchanges the details

bajrachar18:03:39

@noisesmith @alexmiller - Thank you for taking some time to answer my question

bajrachar18:03:34

I am basically trying to generate a index - and keep things in memory to do the intermediate processing

noisesmith18:03:39

bajrachar for reference this is the official doc, I think it contains the important points here

schmee18:03:04

take a look at this for a very well-written explanation of how transients are implemented: http://hypirion.com/musings/understanding-clojure-transients

bajrachar18:03:30

I've used in-memory db - like lmdb - it seems a bit slow on writes -

bajrachar18:03:57

so was trying to accomplish with using just clojure data structures - it seems to take a lot of memory

noisesmith18:03:25

I may have missed the mark above, but everything I'm reading here still makes me think a transient would use the same memory in the end that the persistent version did, though I'd be interested in hearing why that's not so if it isn't

bajrachar18:03:45

@noisesmith - my understanding with immutability was that it basically retained all intermediate changes - thus taking more space than say a local transient would

noisesmith18:03:08

the intermediates go out of scope and get garbage collected

noisesmith18:03:30

the difference in a transient is that you don't generate that garbage in the first place (and don't pay for fresh allocation) - you just mutate

noisesmith18:03:37

the retained memory usage is the same

bajrachar18:03:50

ok - I see - so I guess in that case there is really nothing for me to gain by using transient - 🙂

noisesmith18:03:59

they are faster though

bajrachar18:03:44

Well - I have things working by using an in-memory db - particularly - lmdb

bajrachar18:03:22

however was looking to see if I could just use data structures instead

bajrachar18:03:34

they just seem to take up more space

noisesmith18:03:07

yeah, it's a space / speed optimization baked into the design

bajrachar18:03:39

Thanks @noisesmith - for clarifying this

noisesmith18:03:33

@bajrachar as I read more, there’s a “compress” of the transient that happens when you call persistent! but it’s O(1) and I think the compressing is limited

noisesmith18:03:53

it would be interesting to get input from someone with better understanding of the code though

bajrachar18:03:46

Also apparently you can't make a List transient?

bajrachar18:03:12

I guess that makes sense

noisesmith18:03:52

right, there'd be no benefit

noisesmith18:03:28

taking your original question more literally, (conj! t0 (persistent! t1)) is effectively merge

noisesmith18:03:32

just don't use t1 again

noisesmith18:03:02

(and just like merge, use the return value and don't use the original again, etc.)

bajrachar18:03:37

ok - I will give it a try - thanks

bajrachar18:03:34

also - found an example in the doc for assoc! - https://clojuredocs.org/clojure.core/assoc!

bajrachar18:03:47

example for merge

noisesmith18:03:21

that doesn't do anything conj! doesn't do already, and I doubt it would do it better :p

bajrachar18:03:50

and - it's not necessarily merging two transients in the example -

noisesmith18:03:23

right, but that's just a mutatis mutandis issue

emccue19:03:53

I asked this earlier but I didnt get any advice. Does anyone know how I can generate some POJOs to use with a library that calls jackson and determines how to build a response through reflection

emccue19:03:55

My thought was to use (:gen-class), but I want to have a macro for this and I dont think gen-class is usable in that way.

noisesmith19:03:00

@emccue the easiest thing is likely deftype

noisesmith19:03:25

if all you need is fields that just works, if you need gettors/settors you can use defprotocol to make those, then deftype to implement

devn19:03:14

When is metadata evaluated when constructing a def?

Alex Miller (Clojure team)19:03:15

depends which metadata you’re talking about :)

devn19:03:38

(def ^{:hi 1} a 2}

Alex Miller (Clojure team)19:03:38

var meta is evaluated, function signature metadata is not

devn19:03:21

I guess I'm curious about order. Is the var constructed first, and then the user-supplied metadata is evaluated and added to it? Or does the metadata get evaluated first, then the var is created, then it's attached?

noisesmith19:03:19

and then there's the value attached to the var - that's three things

noisesmith19:03:49

I know the var is created in the ns before the value is realized and attached to it

bhauman20:03:40

@noisesmith you're a vi guy right?

bhauman20:03:30

could use some eyes on the above issue

noisesmith20:03:36

@bhauman I'll check it out

noisesmith20:03:04

@bhauman this would be in command mode only - the way a "leader" works is that (in this example) "\" would put us into a special dispatch state, then "e" would edit, "a" would apropos etc.

noisesmith20:03:48

the thing is to implement a leader you basically need a "mode" state that detects that a previous keypress introduced a leader, and also turns it off again when the command under that leader is invoked

noisesmith20:03:05

so you might want to wait for a PR that implements a leader in rebl-readline

chris20:03:59

leader is basically just another keymap

chris20:03:07

it operates the same way Esc does in emacs

chris20:03:30

(in that it doesn't have to be toggled at the same time to act as a modifier)

noisesmith20:03:46

@chris ahh, where escape prefix can replace meta, yes

schmee20:03:29

@noisesmith I’ve already made these changes in my local rebel-readline and the bindings work as expected, at least in normal mode 🙂

noisesmith20:03:48

@schmee oh, so it's easier than I thought? cool

noisesmith20:03:48

I think leader preference is pretty particular, I've seen "," and " " as leaders for example

chris20:03:56

anyone know if leader exists in gnu-readline? I'm wondering if there's a spot for in it eg .inputrc

chris20:03:19

because yeah, I've always used space as leader

schmee20:03:04

agreeing on keybindings for a thing like this is going to be impossible, config is the only way 🙂

noisesmith20:03:31

I don't think readline even understands the concept of a leader?

noisesmith20:03:41

except the hard-coded emacs-compat of escape

bhauman20:03:15

yeah the leader is just another keystroke here

bhauman20:03:22

in the readline

Datomic Platonic21:03:22

Is #datomic accessible from here?

mbjarland21:03:22

Does anybody have experience with the google app engine standard environ and clojure after they released Java 1.8 and removed the whole whitelist debacle?