Fork me on GitHub
#clojure
<
2018-01-06
>
norman01:01:05

I’m looking forward to seeing a talk on the what/why of this. I don’t see how this does anything but confuse the clojure tooling landscape, or at least that’s how my jaded “oh cognitect is just on the maven/clojars/leiningen hate train again”. I’m quite sure this is largely my lack of ability to see the big picture

Alex Miller (Clojure team)01:01:18

Did you read the blog post?

Alex Miller (Clojure team)01:01:50

That is the what and why

norman01:01:25

Yes. I’m just not seeing it. I guess I’ll wait for the community to absorb it and explain why this makes my life better

Alex Miller (Clojure team)01:01:17

There is no hate here. Leiningen has served a critical role in making Clojure tractable for many many people. It solved problems with the tools at hand and opened up the maven ecosystem. But in the meantime we lost sight of some of the original benefits of Clojure.

Alex Miller (Clojure team)01:01:35

Many projects don’t need Leiningen or artifacts or Maven or Clojars for people to use them. clj and git deps dramatically lower the bar for creating and using those projects.

vemv01:01:32

There's a bunch of stuff (beyond dep management) routinely difficult and unnecessarily tangled in Leiningen. A couple I recall: - how do I invoke this lein plugin's code from a regular Clojure ns (my workaround: fork, remove/adapt lein ns references) - how do I create a simple script that runs with my project's dependencies loaded? (workaround: lein-exec)

norman03:01:30

Thanks. I won’t argue. I’m all for progress. This doesn’t look like progress to me, but I’m 100% willing to be shown how it will be, if it is. Thanks for you work!

the2bears06:01:17

You haven't really explained your concerns.

norman02:01:09

I wasn’t trying to raise specific concern - I was expressing my desire for a talk that goes into more depth about the motivation so I could more fairly evaluate it.

vemv01:01:58

Pretty excited about git/local/jar deps as they are such a pita to get working with Lein

vemv01:01:25

git dependencies are presumably just an occasional lifesaver, else you can end up like this https://thenextweb.com/dd/2016/03/08/cocoapods-popular-basically-took-five-github-servers/

Alex Miller (Clojure team)01:01:05

Git deps are cached locally so unlikely to be an issue

vemv01:01:28

They could be a problem on CI though. Not sure if CI providers transparently proxy/cache git requests; you need that extra layer as machines are constantly created from scratch

Alex Miller (Clojure team)01:01:03

I don’t think what we are doing here is anything like the link above

vemv01:01:32

Say my project has 10 git deps, wouldn't 10 git clones be performed with each push to CI?

Alex Miller (Clojure team)02:01:50

probably. how is this different than downloading Maven artifacts on every CI push?

vemv02:01:52

I may be mistaken, but Maven artifacts are simpler/faster to cache/download/cdn? Single files vs. directory cloning

vemv02:01:03

Presumably a git server suffers more with 10 clones than a Maven one with 10 downloads

Alex Miller (Clojure team)02:01:51

it’s not obvious to me that these are significantly different

Alex Miller (Clojure team)02:01:14

I’m not saying you’re wrong, just that these are assertions without evidence

Alex Miller (Clojure team)02:01:23

try it out and let me know :)

Alex Miller (Clojure team)02:01:44

there is nothing preventing you from making/using artifacts when that’s appropriate

bronsa02:01:49

the issue in that blog specifically was quite different

bronsa02:01:16

they were using a single repo and fetching it regularly

Alex Miller (Clojure team)02:01:43

sounds like more than regularly :)

vemv02:01:08

as if everyone used cljsjs as a git source? yeah I can see the difference

vemv02:01:29

> there is nothing preventing you from making/using artifacts when that’s appropriate I based my assertions in the fact that Maven downloads consists of the discrete transmition of 2-3 files, which maybe can be a "dumb" http request

vemv02:01:06

seems simpler than the git dance

bronsa02:01:22

a git clone is definitely more heavyweight in terms of both IO and CPU load than a maven fetch, but I think it's hardly the case that we'll ever be a problem for github

Alex Miller (Clojure team)02:01:23

git’s pretty smart :)

Alex Miller (Clojure team)02:01:58

when it becomes an issue, I’m happy to look at it :)

hiredman02:01:37

the main issue with the cocoapod case, if I recall, was it was all in single repo, so all the users of cocoapod were constantly hitting that one repo

vemv02:01:53

good, happy to hear these aren't comparable cases

bronsa02:01:17

and that single repo had a huge amount of objects

bronsa02:01:33

which a normal source repo will never even approximate

Drew Verlee03:01:35

I have a couple open source library ideas. Is there a place where i could post the ideas and get feedback on what looks like it has the most promise and usefulness. I also know there are probably lots of great projects that need help to, i feel like helping a project is somewhat harder then making your own (assuming its a small enough endeavor).

vemv03:01:04

Awesome to hear! Probably #clojure / #clojurescript are good for questions in the form "is there currently a good library for X?" #off-topic may be better for "I have X idea, wdyt?", although that could as well be asked here just my impression

Drew Verlee04:01:12

Thanks Vemv. I’ll just pester for advice here then. Probably tomorrow.

qqq06:01:41

(jcuda.cudnn.cudnnTensorDescriptor.)
  (cudnnTensorDescriptor.)
  
the second works fine; the first gives ne am error of: clojure.lang.Compiler$CompilerException , java.lang.ClassNotFoundException

pyr07:01:54

@alexmiller that's a great workaround for missing ones, thanks

dominicm09:01:28

@drewverlee clojureverse might be a good place to put it, long term eyes on it.

qqq10:01:51

what is wrong with this code:

(.-length (float-array 20)) 
and how do I fix it?

bronsa10:01:31

use alength

bronsa10:01:48

there's no length field on array

bronsa10:01:09

it's just java syntax

qqq10:01:21

ah right

andrea.crotti16:01:46

anyone knows how to generate something like <script src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&callback=initMap" async defer></script> with Hiccup?

timgilbert17:01:24

I haven’t tried it, but I would expect something like [:script {:src "..." :async "async" :defer "defer"]? Matching the html5 convention of <tag attr> being the same as <tag attr="attr">

andrea.crotti16:01:05

I tried a few things but nothing works

Lone Ranger16:01:37

hallo everyone 🙂 is there a dedicated channel for discussing the engineering and design of Clojure? For instance, I'm particularly interested in how the for macro implementation works -- but I have dozens of other questions as well 🙂

andrea.crotti16:01:40

but it gives something that doesn't make sense either

schmee16:01:30

@goomba clojure implementation gets discussed in this channel a fair bit, so I’d say just fire away 🙂

Lone Ranger16:01:59

Okay great 😄 how the hell does the for macro implementation work?? 😄 the fact that it can accept any number/order of :when:, :let, and :while keywords is pretty astounding

Lone Ranger16:01:40

when I macroexpand the code it's pretty intimidating

noisesmith16:01:51

have you looked at the source?

Lone Ranger16:01:04

I have -- it's some pretty advanced macro work

noisesmith16:01:42

it works on the bindings in terms of the current pair, then all the rest. with let / when / while it takes the current pair to construct the proper form, then nests all the rest inside the proper construct

noisesmith16:01:49

it's one of the simpler things that for is doing actually

noisesmith16:01:03

I guess it's a bit trickier than all that, but the basic structure to look for is "process this binding pair; then put the rest inside it"

Lone Ranger16:01:51

okay but let's say I have (let [x (range 10) y (range 10) :let [res (+ x y)]] res)... how on earth does it carry along the bindings in the proper order

noisesmith16:01:01

I assume the outer let should be for

Lone Ranger16:01:12

subtle but nuanced point as well is when you carry a symbol in a macro depending on how you write your macro you can lose the original value associated with the binding

noisesmith16:01:22

it processes them one binding pair at a time

noisesmith16:01:39

you can shadow, yes

Lone Ranger16:01:54

okay so you're saying that it does something like

noisesmith16:01:12

so, it starts with x, (range 10) (mod-pairs here is empty) and treats the rest as just next-groups

Lone Ranger16:01:09

no I'm still losing it

Lone Ranger16:01:23

unless they're using a special construct that I don't know about

noisesmith16:01:33

and since that pair doesn't start with :let or :while or :when it is going to emit a let form that expands the next-groups

noisesmith16:01:18

recall that this is a macro, all we are doing here is constructing the lists thatwill create the code we actually want to run

Lone Ranger16:01:00

(for [x (range 10) y (range 10 :let [res (+ x y)]] res) ;;=>

;=> pseudocode  option #1 
for x in range 10:
    for y in range 10:
        res <- (+ x y)
        yield (???) res

noisesmith16:01:25

I don't know if htat helps you, that doesn't help me at all

noisesmith16:01:37

in fact it actively confuses things because for doesn't do a for loop

noisesmith16:01:43

it isn't a loop, it's a generator

Lone Ranger16:01:07

I'm trying to understand in a way I can wrap my mind around what the generated code looks like

Lone Ranger16:01:24

if you take away the chunking and just do one at a time

noisesmith16:01:28

reading the macro-expanded code is a weird place to start, to me - why not look at the source? it uses emit-bind to process each binding pair

noisesmith16:01:01

in that context, it figures out what kind of form to generate, and then puts all the other bindings inside a recursive call to emit-bind inside that form

Lone Ranger16:01:03

I found the source pretty intimidating haha

Lone Ranger16:01:11

and I couldn't find any articles discussing it 😛

noisesmith16:01:15

the source is less intimidating than the macroexpansion

Lone Ranger16:01:27

okay let's annotate it then and see if we can't gain some better understanding

Lone Ranger16:01:06

(assert-args
     (vector? seq-exprs) "a vector for its binding"
(even? (count seq-exprs)) "an even number of forms in binding vector")
that's pretty much just error checking I think

noisesmith16:01:40

let's do this in a thread if you don't mind

Lone Ranger16:01:58

(let [to-groups (fn [seq-exprs]
    (reduce1 (fn [groups [k v]]
             (if (keyword? k)
                 (conj (pop groups) (conj (peek groups) [k v]))
                 (conj groups [k v])))
             [] (partition 2 seq-exprs)))

Lone Ranger16:01:54

okay so let's see what is that doing

Lone Ranger16:01:28

I see we're splitting up the k v pairs in the seq-exprs

noisesmith16:01:41

so it's figuring out whether the left side of the binding is a keyword

noisesmith16:01:51

the k / v is binding, value

Lone Ranger16:01:30

(conj (pop groups) (conj (peek groups) [k v]))

Lone Ranger16:01:36

what do you suppose is going on there?

noisesmith16:01:59

with a vector, pop efficiantly removes the last item, and peek efficiently fetches the last item

noisesmith16:01:10

that idiom is the efficient way to update the last item in a vector

Lone Ranger16:01:38

sure, but I mean, why are we doing that differently for keyword than a non-keyword

Lone Ranger16:01:53

I'm not sure I can visualize the underlying data structure

noisesmith16:01:57

because keywords don't introduce a new collection to iterate

noisesmith16:01:30

so instead of introducing a new iteration nesting, they introduce something which acts in the last context - thus they get added to the thing before it

Lone Ranger16:01:52

what do you suppose is the data structure of groups? a vector?

noisesmith16:01:54

this is exactly what you were asking about - how to make :let and :when etc. do the right thing

Lone Ranger16:01:03

okay it starts out as an empty vector

noisesmith16:01:09

it's a reduce, so we know for a fact it's a vector

noisesmith16:01:37

that's why I started with "ith a vector, pop efficiantly removes the last item, and peek efficiently fetches the last item"

Lone Ranger16:01:39

let's say we have [x (range 10)] as the first pair

noisesmith16:01:42

implicitly, "this is a vector"

Lone Ranger16:01:56

that will make groups <- [[x (range 10)]]

noisesmith16:01:10

no - groups is empty

Lone Ranger16:01:10

if the next is :when (< x 3)

Lone Ranger16:01:25

right and then it does

Lone Ranger16:01:38

(conj groups [k v]))

noisesmith16:01:38

right, next step OK

Lone Ranger16:01:22

now groups is [[x (range 10]], k is :when, and v is (< 3 x)

noisesmith16:01:29

so [[x (range 10)]] is groups, and k is :when and v is (< x 3)

Lone Ranger16:01:43

so it's going to make that into

Lone Ranger16:01:08

[[x (range 10) :when (< 3 x)]]?

Lone Ranger16:01:40

because it applies (conj (pop groups) (conj (peek groups) [k v]))?

Lone Ranger16:01:58

gotcha, fascinating

Lone Ranger16:01:16

okay I think I understand that part

Lone Ranger16:01:38

now this part

noisesmith16:01:39

anything starting with a keyword, we know that goes inside the thing before it

Lone Ranger16:01:45

err (fn [& msg] (throw (IllegalArgumentException. ^String (apply str msg))))... that's just an error handling function

noisesmith16:01:11

emit-bind is where things get interesting

Lone Ranger16:01:19

yeah... do you have time to look at that?

Lone Ranger16:01:32

I don't want to take up your whole day haha

noisesmith16:01:21

if you skip to the bottom, it calls to-groups on the seq-exprs to generate a list of binding contexts, then for each one it uses emit-bind to create the next nested form

noisesmith16:01:39

well, emit-bind recursively creates each nested form itself that is

Lone Ranger16:01:46

by the way the reason I'm working on this is I'm doing my level best to create Clojure hosted on Python

Lone Ranger16:01:26

it's going pretty well but the for macro is killing me

Lone Ranger16:01:27

okay let me unpack what you just said

Lone Ranger16:01:40

you're talking about this part

Lone Ranger16:01:44

`(let [iter# ~(emit-bind (to-groups seq-exprs))]
(iter# ~(second seq-exprs)))))

Lone Ranger16:01:32

heh even this part is pretty difficult to implement

Lone Ranger16:01:45

(fn emit-bind [[[bind expr & mod-pairs]
& [[_ next-expr] :as next-groups]]]

noisesmith16:01:47

right- putting the stuff in that top let block in context, how it actually gets used

noisesmith16:01:31

big picture, emit-bind is going to take each of those groups made in the to-groups reduce call, and make a sub-form out of it

Lone Ranger16:01:32

(the guys at cogintect are very very clever)

Lone Ranger16:01:40

okay, tracking

Lone Ranger16:01:03

so it will take something like

noisesmith16:01:05

and then it will self-call to do the same thing to the next - making it a nested form

Lone Ranger16:01:19

[[x (range 10) :when (< 3 x)]] and produce some kind of form out of it

Lone Ranger16:01:59

now this part

Lone Ranger16:01:14

we can probably come back to but I feel like there is a very important reason they're doing this

Lone Ranger16:01:16

giter (gensym "iter__")
gxs (gensym "s__")

Lone Ranger16:01:37

so they're kind of indirectly calling the autogensym forms instead of the forms directly

noisesmith16:01:40

we need to create new names for each iteration

noisesmith16:01:59

if it just used iter# every level of self-call woudl get the same name

Lone Ranger16:01:23

so for instance if we just used x it would get shadowed?

noisesmith16:01:44

well x is not even valid as a binding inside `

noisesmith16:01:51

so it needs to be x#

noisesmith16:01:58

but then x# would shadow, yes

noisesmith16:01:37

and x# is a shorthand for (gensym "x") - so they are just doing that same thing but in a more controlled way

Lone Ranger16:01:24

alright so now we're at do-mod

Lone Ranger16:01:36

f*ck that formatting is bad

Lone Ranger17:01:01

deleted that block we both can just look at the source

noisesmith17:01:20

that's what I've been doing this whole time, easier to do it side by side

Lone Ranger17:01:43

skipping down to the end I think now I see where the major form that gets emitted is

noisesmith17:01:10

the false branch of (if next-groups ...)

noisesmith17:01:48

oh I'm using the output of (source for) - I could open github though

Lone Ranger17:01:07

okay so the two "loops" being setup

noisesmith17:01:08

oh you aren't even looking at master :P

noisesmith17:01:39

that's alpha-14

noisesmith17:01:07

we're looking at the same thing now though

Lone Ranger17:01:08

I just dug up master

Lone Ranger17:01:12

which should we look at?

noisesmith17:01:32

master would work for me, I like my editor more than the github UI

Lone Ranger17:01:40

okay, I'm looking at master

noisesmith17:01:50

in master, line 4590 is the start of for in core.clj

Lone Ranger17:01:25

so 4632 and 4640 (in master)

noisesmith17:01:48

the actual body being emitted in a loop inside a lazy-seq on line 4655, which is inside a let in the else-branch of next-groups

Lone Ranger17:01:02

okay so a lot of this has to do with chunking results

Lone Ranger17:01:22

that's the difference between do-cmod and do-mod

Lone Ranger17:01:37

I'm sorry brb I need another coffee and my dogs are going crazy, be back in 5

Lone Ranger17:01:42

okay sorry back

Lone Ranger17:01:38

so if we're ignoring chunked sequences, we're close to being done

Lone Ranger17:01:59

which is fine, for me, conceptually

Lone Ranger17:01:54

either we have another groups or we're done with that groupses

Lone Ranger17:01:26

ohhhhhhhhhhhhhhhhhh

Lone Ranger17:01:39

wow, that's very cool

Lone Ranger17:01:12

so this is really a transformation

noisesmith17:01:20

yes - every macro is right/

Lone Ranger17:01:11

from [x (range 10) :when (< 3 x)] to

(let [x (first groups)
            xs (rest groups)]
         (if (< 3 x)
             (cons x res) then continue macro
             res then continue macro

Lone Ranger17:01:24

very very interesting

Lone Ranger17:01:00

okay, thanks sir! That gives me what I need

Lone Ranger17:01:07

REALLY appreciate your help

noisesmith17:01:47

cool - it's also helpful to me to actually go in and read in detail

Lone Ranger17:01:00

obviously my pseudocode isn't very accurate but it gives me enough to implement

Lone Ranger17:01:01

thanks @noisesmith!!! you were right looking at the source code was a much better idea

schmee18:01:23

I can’t get Ultra’s nice stack traces to work. my profiles.clj looks like this:

{:user {:plugins [[venantius/ultra "0.5.2"]]
        :ultra {:repl {:width 250}
                :stacktraces true}}}
and the pretty-printing of data structures in the REPL works fine, but stacktraces are as ugly as ever. Anything else I need to do to activate stack traces pretty-printing?

schmee18:01:07

Seems that Ultra loads correctly and overwrites pst as it should:

user=> clojure.repl/pst
#<Fn@9429385 io.aviso.repl/pretty_pst>
I solved my immediate problem by doing this:
(defn refresh []
  (let [e (clojure.tools.namespace.repl/refresh)]
    (if (instance? Throwable e)
      (clojure.repl/pst e)
      e)))
Still feel like this shouldn’t be necessary though…

noisesmith18:01:08

the problem , I bet, is that refresh deletes and replaces the things that ultra is updating

noisesmith18:01:42

so one alternative is to initialize ultra after calling refresh, another would be to exclude the things ultra updates from being refreshed