This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-04-28
Channels
- # beginners (35)
- # boot (5)
- # cider (16)
- # cljs-dev (22)
- # cljsrn (11)
- # clojars (1)
- # clojure (205)
- # clojure-uk (19)
- # clojurescript (76)
- # cursive (22)
- # duct (4)
- # editors (1)
- # emacs (4)
- # fulcro (1)
- # hoplon (3)
- # leiningen (5)
- # off-topic (25)
- # onyx (29)
- # other-languages (5)
- # parinfer (1)
- # re-frame (13)
- # reagent (8)
- # reitit (2)
- # shadow-cljs (3)
- # tools-deps (11)
- # vim (6)
I’m using rewrite-clj
and looking to insert a [::a/str ::b/of ::c/edn]
into a zipper. the problem is that I quoting the edn throws a reader exception because I do not have (and should not have to) require the namespace. Therefore I’m trying to insert a string as edn. How do I do that in rewrite-clj
?
Use the full name of the namespace instead of a shorthand
Asking for a shorthand without defining it makes no sense
To be clear: ::foo/bar
is a shorthand where it wants to expand the current alias for foo
Use :foo/bar if foo is the full name
Question for y'all: I currently have some futures, each doing work. I know in advance how many operations these futures will do before finishing. I'd like to implement a progress bar in the main thread, but I'm not sure how to do it. Should I pass a core.async
channel to each future? (Is that a good use case for core.async
?) Should I use something other than futures, like agents?
@noisesmith I’m aware of that. But I’m actually also modifying the ns form to require appropriate aliases. So the code in the file ends up being correct. I just can’t read it at compile time.
Your wasting your own time
That kind of thing is never worth the effort in my experience
@noisesmith I don’t think you can say that without being aware of the use case
You are using a convenience that lets you refer to a shorthand for another namespace you are already using, but without the meaningful namespace to refer to, thus the so called convenience increases code complexity because you have to manipulate the way the complier uses your code
I've never seen good results from that sort of plan
But you do you, of course
just to be clear, of the file that’s being changed, I’m adding both requires and aliases namespaces and code that uses them below
Oh, so the ns is something you actually use after the rewrite? If so I misunderstood your goal
some editor plugins do this when they guess a namespace for you, e.g. str/join
-> ns form insert [clojure.string :as str]
Right, my objection was only to metaprogramming in order to access ::a/b shorthand, if you aren't doing that you can disregard
@jeff.terrell One option could be to pass a "progress" atom into your futures and update that. Then you can add a watch to it to update your progress bar.
Ah, I forgot that you could add watches to atoms. That could work. Still curious whether this is a good fit for core.async
, but at least now I have a clear path forward. Thanks!
I like core async - but given what you had described if you weren't already using core async it might be overkill.
@denik I'll bet you could use (symbol "::foo/bar"), since Clojure doesn't validate it and pr-str passes it through.
@emccue so this one's a bit tricky - you have to AOT your Serdes so Kafka can find them.
Is it possible to have a 'private' field in deftype or defrecord? Q: Why do you want it to be private? It's a ref to a C resource. Improper usage of it / use after calling free would result in crash of the JVM. THis resource should only be useable via the well defined protocols Q: Why not just leave it public and be very careful? I prefer to make it impossible to crash the JVM instead of "just don't copy this field of this object."
you can tag a deftype
field with :volatile-mutable
or :unsynchronized-mutable
which will effectively make it visible only inside the type, but it's a kind of hack
the correct way to restrict visibility in lisps is to close over, but deftype and defrecord don't support that
that's why I almost always use reify when I need to close over mutable state to make pseudo-objects
I used to use:
(defn make-... [ ... args ]
(let [internal-state ..]
(fn dispatch [kw & args ] ... )))
but this feels very "objects are just closures" and not very protocol-ishdefonce does not seem to survive remove-ns in Clojure, is there anyway to define a global that survives the reloading of all namespaces ?
I'm not sure, but I wonder how that could work? What if you need to re-define your 'new-defonce'? I suppose one way is to use an external lock, like a file as a marker, that something has already been defined (and resources allocated such as a connection) so don't do it again. Key is you need to know when to re-define and when not to.
why would reloading all namespaces affect defonce in a way that reloading its namespace doesn't? if you are talking about using clojure.tools.namespace/refresh that explicitly destroys vars so that the reload will work, but you can give it a list of namespaces not to touch
or are you talking about a file based lock because it shouldn't even get a new definition after a full vm restart?
the simple fix with resources is not to define them at the top level
I can store a global in the mutable static field of some Java class -- reloading the clojure namesapces won't effect it, and then I can read it back out.
I have a amp of references to external C resources -- I don't want to lose these handles een when I refresh all the clojure namespaces. These refs do NOT have to survive JVM restart. They only need to survive Clojure reloading all namespaces.
reify inside let is the straightforward way to do that I think
don't put it inside a def and it's immune to redefinition :P
that's the idea, you don't store it - you pass it to consumers
since it represents a bond with some running mutable state, it shouldn't have a top level handle
THat's not what I was thinking. The API I want is something like: (global-dict-get! k) (global-dict-set! k v) and have this survive across all refreshes
then you have all the problems that come with global mutables - it's possible, but it's a pain in the ass and won't ever stop being so
it's not thread safe, it makes reloads of code and program state problematic, etc. etc.
In this case there is no other choice. I need to keep a map of refs to external C resources.
you don't need it to exist at global scope - it can belong to the running function that captures thescope
with a try/catch/finally for cleanup
If they're not global state, I don't see how I can write functions and test them in the repl hat gets access to these resources.
in that case the need for global scope is a dev time problem
and there's all kinds of dev-only hacks, you can even start a repl inside the let block
(not to say that's the most sensible solution at all)
@qqq of course it's possible to have a global mutable value, but clojure doesn't have facilities to do so sanely (and arguably no other language does either)
Let me take a step back and avoid the XY problem. I'm doing CUDA in Clojure. Yes, I know about JCuda / ClojureCuda. It's important for me to retain refs to all my Clojure contexts / memories -- because if I lose ref to them, I can't free them, and it becomes a memory leak on the GPU, which forces me to restart the Clojure repl to recover the memory. During dev time, I need eomse way to keep track of a bunch of CUDA resources -- and not lose references ot them. My current dev setup (all in Emacs + Cider + eval last sexp, eval buffer, ...) involves: allocating some CUDA resources initializing their value write some code, run it, iterate thus results in the "I need a global map to stay alive across reloading namesapces" problem
This is also 100% dev time problem; during runtime, I am not reloading namespaces / dynamically modifhing funtions. 🙂
I suggest here reading up on Java Finalizers and using them to clean up. Worst case situation is that you have to do (System/gc)
every so often if you're afraid you've left resources open
if it needs to be in a var, use defonce
if you use something like tools.namspace/refresh that wipes things out regardless, blacklist that ns as something tools.namespace doesn't touch
When using clj
, should you exclude “.cpcache” from source control or allow it to be checked in?
definitely gitignore it
supporting evidence, the tools.deps.alpha project ignores it: https://github.com/clojure/tools.deps.alpha/blob/master/.gitignore
I'd also consider putting it in your global git ignore file with other things you never want to check in
@qqq why can't you store your state into namespaces you won't change? they won't be reloaded if you leave them alone
Hello! I am trying to AOT-compile a basic hello owrld http-kit app but I get the following error when attempting to run it:
Caused by: java.lang.ClassNotFoundException: org.httpkit.server.AsyncChannel
@hmaurer are you trying to run the jar you created?
(basically yes, I am trying to run the class with java hello
(where hello
is my class name))
why are you copying the clojure standard lib, and if that's needed, why aren't you also copying the httpkit lib and the other dependencies you are using?
if it generates files for httpkit, it will also generate them for clojure
OK, this looks like a boot user error, but I know nothing about boot, sorry
there's a #boot channel
it's something clojure can do
and yes, it works
@hmaurer your approach seems odd to me though - the usual thing is either to make a simple jar that also declares deps, then run it with the help of a dependency mananager, or to make a fat jar (aka "uberjar") containing all of your dependencies
I tried to AOT all namespaces (boot aot -a
) but I get this:
java.lang.ClassNotFoundException: jsr166y.forkjoin.ParallelArray
@noisesmith I want to use GraalVM to generate a native image
sure, aot is contagious so aot compiling your top level lib will aot compile everything it accesses
but if you aren't finding a java class, then your classpath isn't set up correctly
and org.httpkit.server.AsyncChannel is going to be a java class
you don't need aot to get a class for this, it comes with the httpkit jar, if you don't find it your classpath is broken, this is why people use dependency managers https://github.com/http-kit/http-kit/blob/master/src/java/org/httpkit/server/AsyncChannel.java
@noisesmith I mean, I can run this in the clojure repl just fine
sure, boot set your classpath properly when you started the repl
it's not aot-compiling that's causing the problem, it's running java without the correct classpath
you need all the classes that were in the httpkit jar, not just the classes you make from the clj files
the aot output will not include those classfiles - they are already compiled
you can't just use the compilation output as your classpath
it includes things it compiled, in my experience
target/org
└── httpkit
├── server
│ └── Channel.class
├── server$fn__187.class
├── server$fn__212.class
├── server$fn__215$G__206__228.class
├── server$fn__215$G__207__221.class
├── server$fn__215.class
├── server$fn__236$G__202__241.class
├── server$fn__236$G__203__238.class
├── server$fn__236.class
├── server$fn__247$G__204__252.class
├── server$fn__247$G__205__249.class
├── server$fn__247.class
├── server$fn__258$G__208__265.class
├── server$fn__258$G__209__261.class
├── server$fn__258.class
├── server$fn__271$G__210__278.class
├── server$fn__271$G__211__274.class
├── server$fn__271.class
├── server$fn__284$G__200__289.class
├── server$fn__284$G__201__286.class
├── server$fn__284.class
├── server$fn__295.class
├── server$fn__297.class
├── server$fn__299.class
├── server$fn__301.class
├── server$fn__303.class
├── server$fn__305.class
├── server$loading__6434__auto____185.class
├── server$run_server$stop_server__196.class
├── server$run_server.class
├── server$sec_websocket_accept.class
├── server$send_websocket_handshake_BANG_$fn__308.class
├── server$send_websocket_handshake_BANG_.class
├── server$with_channel.class
└── server__init.class
all of those things are generated by aot compiling clojure namespaces
none of them are class files in the httpkit jar
you can tell boot to create a jar containing the aot compiled code plus the code for all your deps
also, you can create a jar for graalvm that doesn't include the compiled form of your code, as long as you pass a command line that invokes clojure.main on your namespace and the namespace is a resource in that jar
(it might be that the point here is to precompile, just making clear these are not intrinsically the same thing)
OK - java is as happy to use a jar as a class file on disk in my experience, why not hand it a working jar rather than a directory?
especially if there's a standard way to create that jar and it's known to contain everything you need
(or you can recreate a dependency manager by hand from scratch by mixing all the contents of all the jars you use, but this seems like it's not the best use of one's time)
that's a class that comes with the jvm - if graalvm doesn't have it you'll have to find a replacement? https://docs.oracle.com/javase/7/docs/api/javax/xml/bind/DatatypeConverter.html
unless there's a config or command line that is needed to make it use that class
oh, yeah, java 9 makes everything a bit harder
wow, nice
@noisesmith the memory footprint is also 3.4mb instead of 80mb on the JVM
I'm trying native-image -cp cljs.jar -J-Xmx64G cljs.main
to see if there is a chance it can AOT compile the ClojureScript compiler
This article illustrates some ridiculously small startup latencies: https://www.innoq.com/en/blog/native-clojure-and-graalvm/
I saw this article on news.yc this morning, but it went over my head. What is the fundamental inefficiency in the JVM that Graal eliminates that makes this happen?
But even if you just use the Graal Java VM (without trying to produce native binaries), there is a talk by an engineer at Twitter showing that it uses 12% less CPU, and for the enterprise version he said 20-something % less CPU. So across Twitter's server farm that is probably more than a million USD per year in savings
The gist I got from the Twitter engineer talk is that the Graal compiler is cleaner and thus compiler engineers are able to make further advances in optimizing things, whereas the existing C2 Java optimizer has reached the point where it is difficult for humans to improve it without breaking it. The Graal compiler evidently does better at escape analysis.
native-image
is really just tree shaking + compile-to-native. That alone is pretty cool. So @qqq to answer your question, what you're getting rid of is hot code loading that the JVM supports. native-image
currently doesn't support the creation or loading of new Java classes. Combine that with static analysis and it's fairly easy to see how you could get some really impressive performace.
PyPy's toolchain has been doing this with a restricted set of Python for about 10 years. Some systems like this can even go a step further and take the runtime state needed for your program and lay it out in such a way that "starting" your app is little more than mmaping some data into memory and making the data copy-on-write.
Get that, and the startup time of your app is the time it takes the kernel to map some pages into memory + the opening of any IO required by your app.
@tbaldridge: Does that mean 'native-image 'doesn't support def-type, def-record, and reify ?
no, it means that native-image doesn't support eval
, and therefore doesn't support REPLs
So the compilation path is: clojure app -> AOT -> java .class files -> native-image
the resulting native image cannot use reflection, dynamic code loading, or eval
because at the native-image
level it's no longer java. There's no java classes, no bytecode, no JIT, just native code and a GC.
@mfikes yeah, I think V8 snapshot still probably includes a JIT?
Since the ClojureScript compiler ostensibly only produces JavaScript, it might be possible to get it to run via native-image
, but I bet somewhere the ClojureScript compiler will do something verboten.
I don't know the details of the V8 snapshot feature, other than it sounds a lot like the "save lisp and die" idea
Well, native-image only supports Java code as input
No, it works on bytecode
but the Clojure macro system will probably give you fits
since macros have to use eval
That article seemed to successfully run some fairly non-trivial Clojure programs via native-image
sure, the average Clojure program doesn't use eval or reflection (since they properly type hint)
I'm fairly certain the ClojureScript compiler will do something along those lines somewhere 🙂
Yeah, it will, sadly to get CLJS up and running you need macros, macros require eval, and eval requires reflection, so it'll probably bomb out sometime in the next day or so 😄
(and I thought Pixie compiling in 24min was bad)