Fork me on GitHub
#clojure
<
2018-04-28
>
denik01:04:38

I’m using rewrite-clj and looking to insert a [::a/str ::b/of ::c/edn] into a zipper. the problem is that I quoting the edn throws a reader exception because I do not have (and should not have to) require the namespace. Therefore I’m trying to insert a string as edn. How do I do that in rewrite-clj?

noisesmith01:04:36

Use the full name of the namespace instead of a shorthand

noisesmith01:04:56

Asking for a shorthand without defining it makes no sense

noisesmith01:04:21

To be clear: ::foo/bar is a shorthand where it wants to expand the current alias for foo

noisesmith01:04:01

Use :foo/bar if foo is the full name

jeff.terrell01:04:08

Question for y'all: I currently have some futures, each doing work. I know in advance how many operations these futures will do before finishing. I'd like to implement a progress bar in the main thread, but I'm not sure how to do it. Should I pass a core.async channel to each future? (Is that a good use case for core.async?) Should I use something other than futures, like agents?

denik01:04:11

@noisesmith I’m aware of that. But I’m actually also modifying the ns form to require appropriate aliases. So the code in the file ends up being correct. I just can’t read it at compile time.

noisesmith01:04:56

Your wasting your own time

noisesmith01:04:16

That kind of thing is never worth the effort in my experience

denik01:04:18

@noisesmith I don’t think you can say that without being aware of the use case

noisesmith01:04:15

You are using a convenience that lets you refer to a shorthand for another namespace you are already using, but without the meaningful namespace to refer to, thus the so called convenience increases code complexity because you have to manipulate the way the complier uses your code

noisesmith01:04:34

I've never seen good results from that sort of plan

noisesmith01:04:05

But you do you, of course

denik01:04:18

just to be clear, of the file that’s being changed, I’m adding both requires and aliases namespaces and code that uses them below

noisesmith01:04:18

Oh, so the ns is something you actually use after the rewrite? If so I misunderstood your goal

denik01:04:27

some editor plugins do this when they guess a namespace for you, e.g. str/join -> ns form insert [clojure.string :as str]

noisesmith01:04:37

Right, my objection was only to metaprogramming in order to access ::a/b shorthand, if you aren't doing that you can disregard

denik01:04:11

so do you know who I could make this work?

denik01:04:44

I essentially want to insert a string as edn

lloydshark02:04:26

@jeff.terrell One option could be to pass a "progress" atom into your futures and update that. Then you can add a watch to it to update your progress bar.

jeff.terrell03:04:09

Ah, I forgot that you could add watches to atoms. That could work. Still curious whether this is a good fit for core.async, but at least now I have a clear path forward. Thanks!

lloydshark04:04:26

I like core async - but given what you had described if you weren't already using core async it might be overkill.

👍 8
eraserhd02:04:41

@denik I'll bet you could use (symbol "::foo/bar"), since Clojure doesn't validate it and pr-str passes it through.

denik02:04:19

that works! thanks!!!

emccue05:04:41

I am trying to write a custom kafka deserializer

emccue05:04:44

It is unable to find the EdnDeserializer class

emccue05:04:52

and I am not sure how to resolve that

arrdem05:04:07

@emccue so this one's a bit tricky - you have to AOT your Serdes so Kafka can find them.

arrdem05:04:17

At work we've got some code which deals with this >.>

emccue05:04:32

well, I guess for now I will abide the string deserializer

emccue05:04:00

I am only on the first page of the tutorial

emccue05:04:43

and this seems like a pretty involved thing, so whatever

emccue05:04:19

edn/read-string solves the problem for now

emccue05:04:33

todays badly named and made macro

emccue05:04:44

real proud of myself

qqq07:04:03

Is it possible to have a 'private' field in deftype or defrecord? Q: Why do you want it to be private? It's a ref to a C resource. Improper usage of it / use after calling free would result in crash of the JVM. THis resource should only be useable via the well defined protocols Q: Why not just leave it public and be very careful? I prefer to make it impossible to crash the JVM instead of "just don't copy this field of this object."

leonoel07:04:45

you can tag a deftype field with :volatile-mutable or :unsynchronized-mutable which will effectively make it visible only inside the type, but it's a kind of hack

leonoel07:04:36

the correct way to restrict visibility in lisps is to close over, but deftype and defrecord don't support that

leonoel07:04:37

that's why I almost always use reify when I need to close over mutable state to make pseudo-objects

leonoel07:04:49

or just a function if the protocol is simple enough

qqq08:04:58

I used to use:

(defn make-... [ ... args ]
  (let [internal-state ..]
    (fn dispatch [kw & args ] ... )))
but this feels very "objects are just closures" and not very protocol-ish

leonoel08:04:42

with reify, you have closures and protocols

leonoel08:04:55

and the output bytecode is roughly the same as with deftype

qqq19:04:49

defonce does not seem to survive remove-ns in Clojure, is there anyway to define a global that survives the reloading of all namespaces ?

the2bears19:04:48

I'm not sure, but I wonder how that could work? What if you need to re-define your 'new-defonce'? I suppose one way is to use an external lock, like a file as a marker, that something has already been defined (and resources allocated such as a connection) so don't do it again. Key is you need to know when to re-define and when not to.

noisesmith19:04:07

why would reloading all namespaces affect defonce in a way that reloading its namespace doesn't? if you are talking about using clojure.tools.namespace/refresh that explicitly destroys vars so that the reload will work, but you can give it a list of namespaces not to touch

noisesmith19:04:55

or are you talking about a file based lock because it shouldn't even get a new definition after a full vm restart?

noisesmith19:04:18

the simple fix with resources is not to define them at the top level

qqq19:04:24

I was thinking a Java class with a mutable static field.

qqq19:04:46

I can store a global in the mutable static field of some Java class -- reloading the clojure namesapces won't effect it, and then I can read it back out.

qqq19:04:52

However, I'd prefer a pure clojure solution.

qqq19:04:42

I have a amp of references to external C resources -- I don't want to lose these handles een when I refresh all the clojure namespaces. These refs do NOT have to survive JVM restart. They only need to survive Clojure reloading all namespaces.

noisesmith19:04:01

reify inside let is the straightforward way to do that I think

noisesmith19:04:13

don't put it inside a def and it's immune to redefinition :P

qqq19:04:35

Where do you store the reference of the object reify creates ?

noisesmith19:04:47

that's the idea, you don't store it - you pass it to consumers

noisesmith19:04:04

since it represents a bond with some running mutable state, it shouldn't have a top level handle

qqq19:04:38

THat's not what I was thinking. The API I want is something like: (global-dict-get! k) (global-dict-set! k v) and have this survive across all refreshes

noisesmith19:04:13

then you have all the problems that come with global mutables - it's possible, but it's a pain in the ass and won't ever stop being so

noisesmith19:04:39

it's not thread safe, it makes reloads of code and program state problematic, etc. etc.

qqq19:04:47

In this case there is no other choice. I need to keep a map of refs to external C resources.

qqq19:04:55

I can't just lose these handles when I reload namespaces.

qqq19:04:03

Not keeing this around would be a memory leak.

noisesmith19:04:11

you don't need it to exist at global scope - it can belong to the running function that captures thescope

noisesmith19:04:25

with a try/catch/finally for cleanup

qqq19:04:44

I need ot also play with these resources at the repl.

qqq19:04:04

If they're not global state, I don't see how I can write functions and test them in the repl hat gets access to these resources.

noisesmith19:04:25

in that case the need for global scope is a dev time problem

noisesmith19:04:53

and there's all kinds of dev-only hacks, you can even start a repl inside the let block

noisesmith19:04:03

(not to say that's the most sensible solution at all)

noisesmith19:04:20

@qqq of course it's possible to have a global mutable value, but clojure doesn't have facilities to do so sanely (and arguably no other language does either)

qqq20:04:10

Let me take a step back and avoid the XY problem. I'm doing CUDA in Clojure. Yes, I know about JCuda / ClojureCuda. It's important for me to retain refs to all my Clojure contexts / memories -- because if I lose ref to them, I can't free them, and it becomes a memory leak on the GPU, which forces me to restart the Clojure repl to recover the memory. During dev time, I need eomse way to keep track of a bunch of CUDA resources -- and not lose references ot them. My current dev setup (all in Emacs + Cider + eval last sexp, eval buffer, ...) involves: allocating some CUDA resources initializing their value write some code, run it, iterate thus results in the "I need a global map to stay alive across reloading namesapces" problem

qqq20:04:26

This is also 100% dev time problem; during runtime, I am not reloading namespaces / dynamically modifhing funtions. 🙂

tbaldridge23:04:02

I suggest here reading up on Java Finalizers and using them to clean up. Worst case situation is that you have to do (System/gc) every so often if you're afraid you've left resources open

noisesmith20:04:13

if it needs to be in a var, use defonce

noisesmith20:04:41

if you use something like tools.namspace/refresh that wipes things out regardless, blacklist that ns as something tools.namespace doesn't touch

cddr20:04:40

When using clj, should you exclude “.cpcache” from source control or allow it to be checked in?

pesterhazy20:04:57

definitely gitignore it

pesterhazy21:04:32

supporting evidence, the tools.deps.alpha project ignores it: https://github.com/clojure/tools.deps.alpha/blob/master/.gitignore

👍 4
mfikes21:04:37

I'd also consider putting it in your global git ignore file with other things you never want to check in

joelsanchez22:04:45

@qqq why can't you store your state into namespaces you won't change? they won't be reloaded if you leave them alone

hmaurer22:04:43

Hello! I am trying to AOT-compile a basic hello owrld http-kit app but I get the following error when attempting to run it:

Caused by: java.lang.ClassNotFoundException: org.httpkit.server.AsyncChannel

hmaurer22:04:43

could anybody help me out?

noisesmith22:04:24

@hmaurer are you trying to run the jar you created?

hmaurer22:04:36

(basically yes, I am trying to run the class with java hello (where hello is my class name))

noisesmith22:04:19

why are you copying the clojure standard lib, and if that's needed, why aren't you also copying the httpkit lib and the other dependencies you are using?

hmaurer22:04:38

well, boot should generate class files for the httpkit lib

hmaurer22:04:47

it does, but it seems to be missing some

hmaurer22:04:53

(AsyncChannel being one of the missing ones)

noisesmith22:04:53

if it generates files for httpkit, it will also generate them for clojure

hmaurer22:04:54

not sure why

hmaurer22:04:01

nah it doesn’t add clojure

hmaurer22:04:04

(from what I saw)

hmaurer22:04:20

oh, maybe that’s because I didn’t add clojure as a dependency

noisesmith22:04:22

OK, this looks like a boot user error, but I know nothing about boot, sorry

hmaurer22:04:23

but either way

hmaurer22:04:28

alright 😞

noisesmith22:04:35

there's a #boot channel

hmaurer22:04:45

so AOT should work?

noisesmith22:04:55

it's something clojure can do

noisesmith22:04:02

and yes, it works

noisesmith22:04:26

@hmaurer your approach seems odd to me though - the usual thing is either to make a simple jar that also declares deps, then run it with the help of a dependency mananager, or to make a fat jar (aka "uberjar") containing all of your dependencies

hmaurer22:04:17

I tried to AOT all namespaces (boot aot -a) but I get this:

java.lang.ClassNotFoundException: jsr166y.forkjoin.ParallelArray

hmaurer22:04:21

I’ll ask on the boot channel I guess

hmaurer22:04:33

@noisesmith I want to use GraalVM to generate a native image

hmaurer22:04:38

and for that I need working class files

noisesmith22:04:09

sure, aot is contagious so aot compiling your top level lib will aot compile everything it accesses

hmaurer22:04:11

it worked well for a simple clojure app without dependencies (e..g without http-kit)

noisesmith22:04:25

but if you aren't finding a java class, then your classpath isn't set up correctly

noisesmith22:04:44

and org.httpkit.server.AsyncChannel is going to be a java class

noisesmith22:04:59

you don't need aot to get a class for this, it comes with the httpkit jar, if you don't find it your classpath is broken, this is why people use dependency managers https://github.com/http-kit/http-kit/blob/master/src/java/org/httpkit/server/AsyncChannel.java

hmaurer22:04:58

@noisesmith I mean, I can run this in the clojure repl just fine

hmaurer22:04:05

the issue only arises when AOT-compiling

noisesmith22:04:14

sure, boot set your classpath properly when you started the repl

noisesmith22:04:30

it's not aot-compiling that's causing the problem, it's running java without the correct classpath

hmaurer22:04:32

but it does copy over part of it

noisesmith22:04:03

you need all the classes that were in the httpkit jar, not just the classes you make from the clj files

noisesmith22:04:15

the aot output will not include those classfiles - they are already compiled

noisesmith22:04:24

you can't just use the compilation output as your classpath

hmaurer22:04:26

why does it include some of them then?

noisesmith22:04:39

it includes things it compiled, in my experience

hmaurer22:04:41

target/org
└── httpkit
    ├── server
    │   └── Channel.class
    ├── server$fn__187.class
    ├── server$fn__212.class
    ├── server$fn__215$G__206__228.class
    ├── server$fn__215$G__207__221.class
    ├── server$fn__215.class
    ├── server$fn__236$G__202__241.class
    ├── server$fn__236$G__203__238.class
    ├── server$fn__236.class
    ├── server$fn__247$G__204__252.class
    ├── server$fn__247$G__205__249.class
    ├── server$fn__247.class
    ├── server$fn__258$G__208__265.class
    ├── server$fn__258$G__209__261.class
    ├── server$fn__258.class
    ├── server$fn__271$G__210__278.class
    ├── server$fn__271$G__211__274.class
    ├── server$fn__271.class
    ├── server$fn__284$G__200__289.class
    ├── server$fn__284$G__201__286.class
    ├── server$fn__284.class
    ├── server$fn__295.class
    ├── server$fn__297.class
    ├── server$fn__299.class
    ├── server$fn__301.class
    ├── server$fn__303.class
    ├── server$fn__305.class
    ├── server$loading__6434__auto____185.class
    ├── server$run_server$stop_server__196.class
    ├── server$run_server.class
    ├── server$sec_websocket_accept.class
    ├── server$send_websocket_handshake_BANG_$fn__308.class
    ├── server$send_websocket_handshake_BANG_.class
    ├── server$with_channel.class
    └── server__init.class

hmaurer22:04:51

it generates this in target/ for httpkit

noisesmith22:04:54

all of those things are generated by aot compiling clojure namespaces

noisesmith22:04:01

none of them are class files in the httpkit jar

noisesmith22:04:10

you can tell boot to create a jar containing the aot compiled code plus the code for all your deps

noisesmith22:04:48

also, you can create a jar for graalvm that doesn't include the compiled form of your code, as long as you pass a command line that invokes clojure.main on your namespace and the namespace is a resource in that jar

noisesmith22:04:17

(it might be that the point here is to precompile, just making clear these are not intrinsically the same thing)

hmaurer22:04:31

yep you are right

hmaurer22:04:42

thank you!

hmaurer22:04:54

I wasn’t looking to create a jar though, but to create a native-image

hmaurer22:04:02

what you said made pointed me to the source of the problem

noisesmith22:04:25

OK - java is as happy to use a jar as a class file on disk in my experience, why not hand it a working jar rather than a directory?

noisesmith22:04:40

especially if there's a standard way to create that jar and it's known to contain everything you need

hmaurer22:04:04

ah, I’ll try that then

hmaurer22:04:19

right now I was trying to manually copy over stuff from my maven repository

hmaurer22:04:29

using java xf path-to-jar

hmaurer22:04:33

but that’s probably a stupid idea

noisesmith22:04:38

(or you can recreate a dependency manager by hand from scratch by mixing all the contents of all the jars you use, but this seems like it's not the best use of one's time)

hmaurer22:04:07

the uber jar still seems to be missing javax.xml.bind.DatatypeConverter

noisesmith22:04:26

that's a class that comes with the jvm - if graalvm doesn't have it you'll have to find a replacement? https://docs.oracle.com/javase/7/docs/api/javax/xml/bind/DatatypeConverter.html

noisesmith22:04:53

unless there's a config or command line that is needed to make it use that class

hmaurer22:04:09

(I was trying to run it with the JVM at this point, not Graal)

hmaurer22:04:15

but apparently it still requires a flag

hmaurer22:04:22

java --add-modules java.xml.bind hello works

noisesmith22:04:30

oh, yeah, java 9 makes everything a bit harder

hmaurer22:04:43

well, I’ll sleep less stupid tonight

hmaurer22:04:47

thanks for your help 🙂

👍 4
hmaurer22:04:20

it works with Graal without a flag

hmaurer22:04:25

let’s see if I can gen a native-image

hmaurer22:04:29

I’m sure everything is going to blow up

hmaurer22:04:09

😱 it works

hmaurer22:04:47

7.5MB, startup time of < 20ms

hmaurer22:04:23

@noisesmith the memory footprint is also 3.4mb instead of 80mb on the JVM

hmaurer22:04:26

that’s pretty interesting

mfikes23:04:30

I'm trying native-image -cp cljs.jar -J-Xmx64G cljs.main to see if there is a chance it can AOT compile the ClojureScript compiler

👍 4
hmaurer12:05:25

Oh slick; how is it going?

qqq23:04:26

I don't get the logic here -- how does Graal make Clojure better?

mfikes23:04:43

This article illustrates some ridiculously small startup latencies: https://www.innoq.com/en/blog/native-clojure-and-graalvm/

qqq23:04:46

I saw this article on news.yc this morning, but it went over my head. What is the fundamental inefficiency in the JVM that Graal eliminates that makes this happen?

mfikes23:04:20

But even if you just use the Graal Java VM (without trying to produce native binaries), there is a talk by an engineer at Twitter showing that it uses 12% less CPU, and for the enterprise version he said 20-something % less CPU. So across Twitter's server farm that is probably more than a million USD per year in savings

mfikes23:04:12

The gist I got from the Twitter engineer talk is that the Graal compiler is cleaner and thus compiler engineers are able to make further advances in optimizing things, whereas the existing C2 Java optimizer has reached the point where it is difficult for humans to improve it without breaking it. The Graal compiler evidently does better at escape analysis.

arrdem23:04:32

@mfikes got a link for that talk?

tbaldridge23:04:53

native-image is really just tree shaking + compile-to-native. That alone is pretty cool. So @qqq to answer your question, what you're getting rid of is hot code loading that the JVM supports. native-image currently doesn't support the creation or loading of new Java classes. Combine that with static analysis and it's fairly easy to see how you could get some really impressive performace.

tbaldridge23:04:57

PyPy's toolchain has been doing this with a restricted set of Python for about 10 years. Some systems like this can even go a step further and take the runtime state needed for your program and lay it out in such a way that "starting" your app is little more than mmaping some data into memory and making the data copy-on-write.

tbaldridge23:04:37

Get that, and the startup time of your app is the time it takes the kernel to map some pages into memory + the opening of any IO required by your app.

mfikes23:04:02

I wonder how close Lumo got to that with the V8 snapshot stuff

qqq23:04:48

@tbaldridge: Does that mean 'native-image 'doesn't support def-type, def-record, and reify ?

tbaldridge23:04:59

no, it means that native-image doesn't support eval, and therefore doesn't support REPLs

tbaldridge23:04:22

So the compilation path is: clojure app -> AOT -> java .class files -> native-image

tbaldridge23:04:35

the resulting native image cannot use reflection, dynamic code loading, or eval

tbaldridge23:04:43

because at the native-image level it's no longer java. There's no java classes, no bytecode, no JIT, just native code and a GC.

tbaldridge23:04:33

@mfikes yeah, I think V8 snapshot still probably includes a JIT?

mfikes23:04:50

Since the ClojureScript compiler ostensibly only produces JavaScript, it might be possible to get it to run via native-image, but I bet somewhere the ClojureScript compiler will do something verboten.

mfikes23:04:34

I don't know the details of the V8 snapshot feature, other than it sounds a lot like the "save lisp and die" idea

tbaldridge23:04:36

Well, native-image only supports Java code as input

mfikes23:04:55

Yeah, I'm trying to compile the ClojureScript compiler with native-image

mfikes23:04:30

The result might be a native binary that behaves like the ClojureScript comipler

mfikes23:04:26

Does it need to be Java source, or can it work on bytecode generated by Clojure?

tbaldridge23:04:00

No, it works on bytecode

mfikes23:04:07

AFAICT it is chewing on the classfiles in the AOT-compiled cljs.jar

tbaldridge23:04:08

but the Clojure macro system will probably give you fits

tbaldridge23:04:12

since macros have to use eval

mfikes23:04:14

That article seemed to successfully run some fairly non-trivial Clojure programs via native-image

tbaldridge23:04:43

sure, the average Clojure program doesn't use eval or reflection (since they properly type hint)

mfikes23:04:25

I'm fairly certain the ClojureScript compiler will do something along those lines somewhere 🙂

mfikes23:04:49

It has been using 18 cores for the last hour trying to compile the JAR 🙂

🏎️ 4
tbaldridge23:04:25

Yeah, it will, sadly to get CLJS up and running you need macros, macros require eval, and eval requires reflection, so it'll probably bomb out sometime in the next day or so 😄

tbaldridge23:04:41

(and I thought Pixie compiling in 24min was bad)

mfikes23:04:22

Yeah, it has already used 14 hours of CPU time in compiling.

lol 4