Fork me on GitHub
#clojure
<
2020-08-03
>
ksd01:08:47

Suppose a library contains function public a which depends on private function b. Is there a way to implement my own version of b and have a use my b , without having to copy a portion of the library’s source?

noisesmith01:08:04

nothing cleaner than copying the code (alter-var-root and intern exist, but they won't lead to a cleaner result, since your code now relies on modifying something that other code might want to use as is)

dpsutton01:08:48

If it makes sense that b should be hot swappable or behind some polymorphic construct consider a PR to the library as well

ksd02:08:44

that makes sense, thanks

souenzzo12:08:50

doubt about "clojure internals" Why does use a "inline value"¹ is faster then "a reference value"²? (let-benchmark in thread) ¹ (mapv #(+ % 1) coll) ² (mapv #(+ % one) coll) "faster" means that it will take less time in a HUGE coll

souenzzo12:08:26

My benchmark:

(let [coll (repeat 1e6 {})
      m {:my :meta}]
  (doseq [[id f] {:inside-fun #(mapv
                                 (fn [e]
                                   (with-meta e {:my :meta}))
                                 coll)
                  :inside-let #(mapv
                                 (fn [e]
                                   (with-meta e m))
                                 coll)}]
    (prn id)
    (dotimes [_ 3]
      (time (dotimes [_ 1e2]
              (f))))))
My results
:inside-fun
"Elapsed time: 1901.891841 msecs"
"Elapsed time: 1904.658297 msecs"
"Elapsed time: 1947.197886 msecs"
:inside-let
"Elapsed time: 6607.768999 msecs"
"Elapsed time: 6632.838103 msecs"
"Elapsed time: 6749.213546 msecs"

souenzzo12:08:47

Actually, use a "inline function" looks a bit better then "reference function"

(let [coll (doall (repeat 1e6 {}))
      add-meta (fn [e]
                 (with-meta e {:my :meta}))]
  (doseq [[id f] {:inside-fun #(mapv (fn [e]
                                       (with-meta e {:my :meta}))
                                     coll)
                  :add-meta2  #(mapv add-meta
                                     coll)}]
    (prn id)
    (dotimes [_ 3]
      (time (dotimes [_ 1e2]
              (f))))))
:inside-fun
"Elapsed time: 1999.221744 msecs"
"Elapsed time: 1996.666202 msecs"
"Elapsed time: 2033.77291 msecs"
:add-meta2
"Elapsed time: 2036.254896 msecs"
"Elapsed time: 2072.717188 msecs"
"Elapsed time: 2062.882369 msecs"
=> nil

borkdude13:08:59

The var one is dereferenced only once when the function is evaluated, so there is no significant difference:

user=> (def one 1)
#'user/one
user=> (mapv #(+ % one) [1 1 (do (alter-var-root #'one (constantly 2)) 1) 1 1 1])
[3 3 3 3 3 3]

souenzzo13:08:54

@U04V15CAJ I understand it, but did you see the first benchmark? I think that the answer is something like "when where is no external references, the JVM can do a better optimization"

borkdude13:08:09

Hmm, that's an interesting difference.

souenzzo13:08:43

It also occurs in bb @U04V15CAJ (using coll of 1e4 because 1e6 was too slow)

:inside-let
"Elapsed time: 917.150869 msecs"
"Elapsed time: 922.404064 msecs"
"Elapsed time: 949.586256 msecs"
:inside-fun
"Elapsed time: 1750.694556 msecs"
"Elapsed time: 1756.074617 msecs"
"Elapsed time: 1783.673856 msecs"
nil

potetm14:08:33

I would expect one to be dereferenced on each call. Is that not the case?

borkdude14:08:34

There are two different things at play here, I think. names referring to local bindings vs names referring to vars. In this example: https://clojurians.slack.com/archives/C03S1KBA2/p1596460259320200?thread_ts=1596457910.319500&amp;cid=C03S1KBA2 the var is only dereferenced once

borkdude14:08:20

I see now that this example is also not good, since the alter-var-root is executed before mapv is doing anything

borkdude14:08:39

it should be a lazy collection without chunking to fix the example

borkdude14:08:34

Anyway, there are a couple of different situations in this thread, they should be discussed separately

potetm14:08:21

Yeah I was fixing to say, “something’s not right about that.”

potetm14:08:32

vars are always derefd at runtime

potetm14:08:48

it’s why re-evaling defs at the repl works

potetm14:08:51

is my understanding

potetm14:08:55

yeah:

(def one 1)

(last (mapv #(+ % one)
            (repeat 1e8 1)))

(alter-var-root #'one (constantly 2))

borkdude14:08:31

user=> (def x 1)
#'user/x
user=> (defn foo [] (+ x 1))
#'user/foo
user=> (foo)
2
user=> (def x 2)
#'user/x
user=> (foo)
3
user=> (def foo (partial + x 1))
#'user/foo
user=> (foo)
3
user=> (def x 3)
#'user/x
user=> (foo)
3

potetm14:08:30

:thinking_face:

borkdude14:08:35

So I guess when used in a function body it's compiled down to a deref, but when passed as an arg, it it's only dereferenced once

potetm14:08:09

ah yeah, that makes perfect sense

potetm14:08:40

otherwise you’re passing vars, not values

borkdude14:08:54

so scratch what I said. direct linking influences this for function calls, not with other usages. ^:const influences this for all usages.

potetm14:08:16

Well, of course. I was assuming repl interaction.

potetm14:08:42

(i.e. no direct linking, no const)

borkdude14:08:27

if it would work otherwise, the REPL would be a hard place to work in it without reloading all your code all the time

potetm14:08:34

interesting that #() works more like partial than (fn [])

potetm14:08:51

didn’t think about it before, but that does make sense

borkdude14:08:54

huh, how so?

borkdude14:08:08

#() expands into (fn [])

potetm14:08:04

oh wait, misread

borkdude14:08:07

user=> '#()
(fn* [] ())

dominicm16:08:37

Is it a type hint thing? In one case it's a primitive?

ak-coram15:08:28

I have a 7-level deep list comprehension (`for`) and compilation of the ns fails with "filename too long". I assume the generated code (300+ lines) has too many nested lambdas and the classnames hit the 255 byte limit. Does anyone know of another for implementation which is maybe not as optimized as the one in core, but avoids this problem?

ghadi15:08:04

that's doseq, but you could do something similar with for

ak-coram15:08:10

@ghadi thanks, I'll look into it

ghadi15:08:07

you could probably separate your 7 nested fors into 7 functions that each used for once

ak-coram15:08:19

yeah, but it'd be pretty verbose (especially since I use data from every nesting level in the for body)

ak-coram15:08:24

the real annoying thing about this is that it works fine when developing with the repl, but blows up when compiling

ak-coram15:08:13

jsn: thanks, but I just have a deeply nested structure (with fixed levels). so the code looks more like (for [[p1 ys] xs, [p2 zs] ys, ...] ...)

ak-coram15:08:00

guess I could also just quote the whole for and eval it to avoid the compilation issue, but that just feels wrong 🙂

dpsutton15:08:39

do you mean compiling to disk?

ak-coram15:08:47

@dpsutton I'm using AOT, that's when the error comes up

dpsutton15:08:22

can you move this function into a non-AOT'd namespace?

dpsutton15:08:54

(not sure if AOT is transitive)

ak-coram15:08:23

@dpsutton: everything is AOT'd (except for library deps), I can't ship source code with the application

ghadi15:08:02

how about splitting up your code?

ghadi15:08:08

seems like the biggest win for you

ak-coram15:08:53

@ghadi: I think that's the way to go as well; I was just toying with different ideas

Ben Sless17:08:31

Question regarding transducers and concatenation: I have 2+ collections which are returned from different functions. Ideally, they wouldn't even be collections, but eductions or something similar. I want to take these two collections, concat them, and pass them through a transducer without generating an intermediary lazy sequence. The partial solution I came up with is a new type which implements CollReduce:

(require '[clojure.core.protocols :as p])

(deftype Concatenation [coll1 coll2]
  p/CollReduce
  (coll-reduce
    [this f]
    (p/coll-reduce
     coll2
     f
     (p/coll-reduce coll1 f)))

  (coll-reduce
    [this f init]
    (p/coll-reduce
     coll2
     f
     (p/coll-reduce coll1 f init))))
Really not sure this is the right way, and my first arity is wrong. Ideas and suggestions welcome This is what already works:
(def c (Concatenation. [1 2 3] [4 5 6]))

(transduce (comp (map inc) (filter even?) (map inc)) conj c)

(def e (->Eduction (comp (map inc) (filter even?) (map inc)) c))

(reduce conj [] e)

phronmophobic17:08:53

you should be able to simply use cat:

> (let [a [1 2 3]
        b [4 5 6]]
    (into [] (comp cat  
                   (map inc)
                   (filter even?)
                   (map inc))
          [a b]))
;; [3 5 7]

👍 3
Ben Sless17:08:14

That works if I pack them in a vector first, wondering if there's something more lightweight

Ben Sless17:08:40

But I guess it's simpler than my hare brained solution 🙃

phronmophobic17:08:43

I don't think there's any way to get around grouping/packing them into something (even creating a Concatenation is packing them). not sure what the most lightweight option is.

Ben Sless17:08:09

Technically a Concatenation would be lighter, it's an object with just one method and two members. A vector is an array + lots of stuff and methods. But with the price being confusing my coworkers I'm not sure it's worth it

ghadi17:08:07

that is an assertion without evidence

ghadi17:08:22

agree that cat is what you want

Ben Sless17:08:23

How is it without evidence?

ghadi17:08:03

1) reduce impls need to handle early termination (this impl is incorrect -- try passing a (take 5) transducer) 2) cat doesn't care about the source being in a vector, cat just unpacks items no matter where they came from (could be from an eduction source) 3) Concatenation only works on pairs of collections, so you'd have to make cons cells to do more (you said you have 2+ collections) 4) coll-reduce is not as fast as implementing clojure.lang.IReduceInit

ghadi17:08:33

"vector is an array + lots of stuff and methods isn't" doesn't make it slow

Ben Sless17:08:50

Absolutely. I was talking about the allocation. And I started by saying my implementation was incomplete and asking for suggestions on how I can flesh it out =\

ghadi17:08:02

sorry to be negative 🙂

Ben Sless17:08:09

It's okay, I tend to take things unnecessarily to heart

Ben Sless17:08:46

Won't coll-reduce be approximately as fast for IReduceInit since it's extended to it?

ghadi17:08:11

the dispatch for IReduceInit is special cased

ghadi17:08:22

it will be faster to dispatch but still dominated by the reduction time

ghadi17:08:50

but IReduceInit is "better" in that I think Rich regrets having the arity of reduce that doesn't take the explicit "init" arg

ghadi17:08:28

I think eduction+cat is the way to go here, all the benefits you're looking for + in the stdlib

Ben Sless17:08:21

I think I'll go with that one, thanks 🙂

Ben Sless17:08:07

btw, I tested allocation overhead of a vector vs. a Concatenation. It's measurable, but all of it is in the ns range

Ben Sless17:08:43

So I'll let it rest as premature optimization

Alex Whitt17:08:31

Anyone know of a way to execute a shell command, but redirect its output immediately to stdout? I don't mean printing :out after it's done executing. I've got a long-running command and I'd like to watch the output as it goes, before the process exits.

phronmophobic17:08:21

something like:

(require '[clojure.java.shell :as sh])
(require '[ :as io])
(let [{:keys [out]} (sh/sh "cat" "README.md")]
   (io/copy out System/out))

Alex Whitt17:08:59

But sh doesn't return until the process exits, AFAICT

phronmophobic17:08:41

oh, right. you might have to call (.exec (Runtime/getRuntime) ...) directly

Alex Whitt18:08:08

Ah, that's worth looking into. Thank you!

noisesmith18:08:59

I think the best option is ProcessBuilder

noisesmith18:08:05

snippet:

(ins)user=> (-> ["ls" "-l"] (ProcessBuilder.) (.inheritIO) (.start) (.waitFor))
total 72
drwxrwxr-x 2 justin justin 4096 May 29 23:07 2-literate-9-2019
drwxrwxr-x 2 justin justin 4096 Apr 17 17:14 3-simple-9-2019
drwxrwxr-x 2 justin justin 4096 Apr 17 17:14 4-macros-9-2019
drwxrwxr-x 2 justin justin 4096 Apr 17 17:14 5-fennel-4-2020
drwxrwxr-x 4 justin justin 4096 Jul  2 19:53 6-scheme-4-2020
drwxrwxr-x 2 justin justin 4096 Jul 18 07:09 7-raw
drwxrwxr-x 5 justin justin 4096 Aug  1 13:24 8-arm-assembly
drwxrwxr-x 2 justin justin 4096 Jul 27 15:06 9-learner
-rw-rw-r-- 1 justin justin 1594 Aug 23  2019 composition.csd
drwxrwxr-x 2 justin justin 4096 Dec 13  2019 fennel-dist
drwxr-xr-x 2 justin justin 4096 Jul 14  2019 fennel-libs
-rw-r--r-- 1 justin justin  382 Jul 14  2019 Makefile
-rw-r--r-- 1 justin justin  779 Jul 24  2019 osc-in.csd
drwxrwxr-x 2 justin justin 4096 Aug  1  2019 simple-a
drwxrwxr-x 2 justin justin 4096 Aug 15  2019 simple-b
-rw-rw-r-- 1 justin justin 4826 Sep  7  2019 synth-compiler.fnl
-rw-rw-r-- 1 justin justin 1188 Jul 26 08:16 todo.otl
0

Alex Whitt18:08:37

Also worth looking into, thank you as well

noisesmith18:08:40

if you don't include .waitFor the shell command will run in the background, but that can mess with the repls usage of the same IO ports

noisesmith18:08:16

instead of .inheritIO you can map each of stdin, stdout, stderr to an arbitrary stream (eg. if you wanted to write a state machine that ineracts with a python command line)

noisesmith18:08:09

oh, and this even works for programs that use terminal drawing (eg. replace ["ls" "-l"] with ["htop"] or ["vim"] and the programs work like normal, and your repl resumes when they exit)

devn19:08:09

Did I just miss the Clojure talk where someone quotes Charles Kettering?

devn19:08:12

> A problem well stated is a problem half-solved. — Charles Kettering

Kevin20:08:02

Hello. I wrote a commandline tool (compiled with GraalVM) that prints to the console using println. When running this executable I see the output in my terminal. Now I'm trying to use my executable in a separate program (specifically Godot's OS.execute function). This program should then output anything from my executable. The problem is that it's not printing anything, as if my program is not writing to STDOUT. If I use another command (e.g. ls) it does work. Am I supposed to do something special with Clojure to "properly" write to STDOUT for other programs to catch that? e.g. an EOF or something?

Alex Miller (Clojure team)20:08:23

println should be writing and flushing to stdout

Kevin20:08:37

All right, then I'm just doing something wrong

Kevin20:08:11

My program does block the process though, so maybe Godot is waiting for it to finish

Alex Miller (Clojure team)20:08:12

any chance you're doing something lazy and it's not being realized?

Alex Miller (Clojure team)20:08:30

that's a common issue - top-level is a map or something

Kevin20:08:55

afaik I'm using mapv everywhere

Kevin20:08:05

Ok, apparently Godot is waiting for the process to finish 😕

Kevin20:08:17

Which is really annoying because then I can't implement a watcher executable

Alex Miller (Clojure team)20:08:21

ah, if you did anything with future or agent you might need to (shutdown-agents)

noisesmith20:08:23

clojure is poorly suited for that kind of IPC - as a workaround you could leave a socket server running and then execute code by sending / receiving TCP via the socket

noisesmith20:08:10

you might have to start the clojure daemon process from outside Godot - I don't know what facilities Godot has for running helpers

noisesmith20:08:23

I also don't know if Godot has socket support, but seeing that's a pre-requisite for http etc. one would hope it is htere

noisesmith20:08:32

and worst case you could shell out to nc

Kevin20:08:14

Yeah I'll have to investigate a bit more what my options are regarding Godot. It's unfortunate that godot doesn't seem to support async logging

ghadi20:08:19

Godot is waiting? (Sorry, couldn't resist)

😄 9
noisesmith20:08:38

in one window:

$ clj -J-Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}"
Clojure 1.10.1
user=>
in another
$ echo '*clojure-version*' | nc localhost 5555
user=> {:major 1, :minor 10, :incremental 1, :qualifier nil}

noisesmith20:08:13

I wonder if there's an easy way to get that without the prompt

Alex Miller (Clojure team)20:08:41

clj -e '*clojure-version*'

Alex Miller (Clojure team)20:08:57

or I guess I don't understand what you're asking

noisesmith20:08:05

@alexmiller sorry, I meant as a way for @kevin.van.rooijen to implement his goal (a long running watcher that Godot can consume individual results from)

noisesmith20:08:20

the socket repl works, but you end up needing to manually remove a prompt

Alex Miller (Clojure team)20:08:30

if you start the socket server programmatically, I think there is a config attr for whether to prompt or not

noisesmith20:08:55

oh right, there's a dynamic var for the prompt function, or a higher order fn for the prompt right?

Alex Miller (Clojure team)20:08:18

well, I guess there's configurable support in clojure.main/repl

Alex Miller (Clojure team)20:08:48

but that's not specially exposed out through the socket server config - you'd need to make your own accept function with whatever customization you need

noisesmith20:08:41

yeah - clojure.core.server/repl just passes a config to clojure.main/repl, I see that now, doesn't look like it would be super hard to set up, but also not turnkey

noisesmith20:08:36

@kevin.van.rooijen @alexmiller aha this works:

$ clj
Clojure 1.10.1
(cmd)user=> (ns server.quiet (:require [clojure.main :as m] [clojure.core.server :as s]))
nil
(cmd)server.quiet=> (defn repl [] (m/repl :init s/repl-init :read s/repl-read :prompt (constantly "")))
#'server.quiet/repl
(cmd)server.quiet=> (s/start-server {:port 6666 :name 'repl :accept 'server.quiet/repl})
#object[java.net.ServerSocket 0x736d6a5c "ServerSocket[addr=localhost/127.0.0.1,localport=6666]"]
server.quiet=>
$ echo '*clojure-version*' | nc -q0 localhost 6666
{:major 1, :minor 10, :incremental 1, :qualifier nil}
$

noisesmith20:08:33

of course there are tools like nrepl with proper protocols that differentiate one off results from longer running repl sessions etc.

noisesmith20:08:31

interesting note, I wasn't able to do this without making a new ns, if you try to define an :accept function inside ns user, the repl process blows up with

user=> Exception in thread "Clojure Connection repl 1" java.io.FileNotFoundException: Could not locate use
r__init.class, user.clj or user.cljc on classpath.
        at clojure.lang.RT.load(RT.java:462)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6839.invoke(core.clj:6126)
        at clojure.core$load.invokeStatic(core.clj:6125)
        at clojure.core$load.doInvoke(core.clj:6109)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5908)
        at clojure.core$load_one.invoke(core.clj:5903)
        at clojure.core$load_lib$fn__6780.invoke(core.clj:5948)
        at clojure.core$load_lib.invokeStatic(core.clj:5947)
        at clojure.core$load_lib.doInvoke(core.clj:5928)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.core$load_libs.invokeStatic(core.clj:5985)
        at clojure.core$load_libs.doInvoke(core.clj:5969)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.core$require.invokeStatic(core.clj:6007)
        at clojure.core.server$accept_connection.invokeStatic(server.clj:73)
        at clojure.core.server$start_server$fn__8879$fn__8880$fn__8882.invoke(server.clj:117)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.lang.Thread.run(Thread.java:748)

Kevin20:08:39

It's an interesting idea. Godot does have some TCP functionality. That would probably be a good option

noisesmith20:08:14

at the very least, it's likely the lowest friction option for getting ad-hoc results from a long running clojure process

marciol21:08:19

Hey, I’m trying to convince Robert Virding to put some Clojurisms inside LFE and he asked: > One thing I don’t get with how clojure uses `{ }` and `[ ]`. Do they return literal values like LFEs `#( )` and `#m( )` or are they evaluated like LFE `(tuple …)` and `#m( … )`? I tried to look in some tutorials online but they describe them as literal values in the reader which are then evaluated. So are the literals ar are they evaluated and if so when? What is the best answer?

noisesmith21:08:47

literals are evaluated while reading

noisesmith21:08:01

eg. :foo is turned into a keyword by the reader

noisesmith21:08:17

[:foo] is turned into a vector containing a keyword by the reader

noisesmith21:08:22

this happens before eval

marciol21:08:55

at the reading time

noisesmith21:08:00

they are not like #() because #() only expands to a form that gets further evaluating

noisesmith21:08:12

right, this is done by the reader

marciol21:08:21

ok, without expansion. I got it

marciol21:08:28

I’ll tell him

noisesmith21:08:42

server.quiet=> '#(read but not evaluated yet)
(fn* [] (read but not evaluated yet))

noisesmith21:08:05

you can use ' to see what is provided by the reader before evaluation

noisesmith22:08:42

@marciol btw what is LFE in this context? I think I glossed it enough to answer your question but I am realizing I have no idea what that acronym is

marciol22:08:00

Lisp Flavoured Erlang @noisesmith

marciol22:08:14

He is in doubt about how literals are processed in Clojure, if they are expansioned so [] turns out (vector …) or not.

noisesmith22:08:20

I'm not sure what "literal" vs. "evaluated" is supposed to mean there, but it might be significant that the data types returned by [] and {} are values rather than objects for equality - they aren't a place that is changed and equal based on pointer location, but a value that is equal based on nested contents

noisesmith22:08:44

the reader does the same thing vector does, but AFAIK doesn't actually call the vector function

noisesmith22:08:41

the easy and literal test of what you are asking:

server.quiet=> '[]
[]
reading but not evaluating still gives us [], and not (vector)

noisesmith22:08:04

but I wonder when that difference would matter - if you wanted to locally override vector and leverage the existing literals?

marciol22:08:56

yes, so the reader will handle accordingly

noisesmith22:08:16

(cmd)server.quiet=> (with-redefs [clojure.core/vector +] (vector 1 2))
3
(cmd)server.quiet=> (with-redefs [clojure.core/vector +] [1 2])
[1 2]
(ins)server.quiet=> (with-redefs [clojure.core/vector +] (read-string "[1 2]"))
[1 2]

marciol22:08:33

I think that he is trying to emulate the same behaviour in LFE to handle literals on the reader.

marciol22:08:10

Or, better saying, I caused the discussion about use the same approach

noisesmith22:08:21

@marciol you might be interested in the tagged literal feature, if you haven't looked into it yet https://clojure.org/reference/reader#tagged_literals

noisesmith22:08:40

it allows custom tags (that use vector or map literals or even eg. strings, but create whatever result data you like)

noisesmith22:08:32

but I think a tagged reader with side effects would be considered pathological

marciol22:08:22

Yes, I’m a fan of Aero for this reason @noisesmith