This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # beginners (35)
- # boot (5)
- # cider (16)
- # cljs-dev (22)
- # cljsrn (11)
- # clojars (1)
- # clojure (205)
- # clojure-uk (19)
- # clojurescript (76)
- # cursive (22)
- # duct (4)
- # editors (1)
- # emacs (4)
- # fulcro (1)
- # hoplon (3)
- # leiningen (5)
- # off-topic (25)
- # onyx (29)
- # other-languages (5)
- # parinfer (1)
- # re-frame (13)
- # reagent (8)
- # reitit (2)
- # shadow-cljs (3)
- # tools-deps (11)
- # vim (6)
rewrite-clj and looking to insert a
[::a/str ::b/of ::c/edn] into a zipper. the problem is that I quoting the edn throws a reader exception because I do not have (and should not have to) require the namespace. Therefore I’m trying to insert a string as edn. How do I do that in
To be clear:
::foo/bar is a shorthand where it wants to expand the current alias for foo
Question for y'all: I currently have some futures, each doing work. I know in advance how many operations these futures will do before finishing. I'd like to implement a progress bar in the main thread, but I'm not sure how to do it. Should I pass a
core.async channel to each future? (Is that a good use case for
core.async?) Should I use something other than futures, like agents?
@noisesmith I’m aware of that. But I’m actually also modifying the ns form to require appropriate aliases. So the code in the file ends up being correct. I just can’t read it at compile time.
You are using a convenience that lets you refer to a shorthand for another namespace you are already using, but without the meaningful namespace to refer to, thus the so called convenience increases code complexity because you have to manipulate the way the complier uses your code
just to be clear, of the file that’s being changed, I’m adding both requires and aliases namespaces and code that uses them below
Oh, so the ns is something you actually use after the rewrite? If so I misunderstood your goal
some editor plugins do this when they guess a namespace for you, e.g.
str/join -> ns form insert
[clojure.string :as str]
Right, my objection was only to metaprogramming in order to access ::a/b shorthand, if you aren't doing that you can disregard
@jeff.terrell One option could be to pass a "progress" atom into your futures and update that. Then you can add a watch to it to update your progress bar.
Ah, I forgot that you could add watches to atoms. That could work. Still curious whether this is a good fit for
core.async, but at least now I have a clear path forward. Thanks!
I like core async - but given what you had described if you weren't already using core async it might be overkill.
@denik I'll bet you could use (symbol "::foo/bar"), since Clojure doesn't validate it and pr-str passes it through.
@emccue so this one's a bit tricky - you have to AOT your Serdes so Kafka can find them.
Is it possible to have a 'private' field in deftype or defrecord? Q: Why do you want it to be private? It's a ref to a C resource. Improper usage of it / use after calling free would result in crash of the JVM. THis resource should only be useable via the well defined protocols Q: Why not just leave it public and be very careful? I prefer to make it impossible to crash the JVM instead of "just don't copy this field of this object."
you can tag a
deftype field with
:unsynchronized-mutable which will effectively make it visible only inside the type, but it's a kind of hack
the correct way to restrict visibility in lisps is to close over, but deftype and defrecord don't support that
that's why I almost always use reify when I need to close over mutable state to make pseudo-objects
I used to use:
but this feels very "objects are just closures" and not very protocol-ish
(defn make-... [ ... args ] (let [internal-state ..] (fn dispatch [kw & args ] ... )))
defonce does not seem to survive remove-ns in Clojure, is there anyway to define a global that survives the reloading of all namespaces ?
I'm not sure, but I wonder how that could work? What if you need to re-define your 'new-defonce'? I suppose one way is to use an external lock, like a file as a marker, that something has already been defined (and resources allocated such as a connection) so don't do it again. Key is you need to know when to re-define and when not to.
why would reloading all namespaces affect defonce in a way that reloading its namespace doesn't? if you are talking about using clojure.tools.namespace/refresh that explicitly destroys vars so that the reload will work, but you can give it a list of namespaces not to touch
or are you talking about a file based lock because it shouldn't even get a new definition after a full vm restart?
I can store a global in the mutable static field of some Java class -- reloading the clojure namesapces won't effect it, and then I can read it back out.
I have a amp of references to external C resources -- I don't want to lose these handles een when I refresh all the clojure namespaces. These refs do NOT have to survive JVM restart. They only need to survive Clojure reloading all namespaces.
since it represents a bond with some running mutable state, it shouldn't have a top level handle
THat's not what I was thinking. The API I want is something like: (global-dict-get! k) (global-dict-set! k v) and have this survive across all refreshes
then you have all the problems that come with global mutables - it's possible, but it's a pain in the ass and won't ever stop being so
it's not thread safe, it makes reloads of code and program state problematic, etc. etc.
In this case there is no other choice. I need to keep a map of refs to external C resources.
you don't need it to exist at global scope - it can belong to the running function that captures thescope
If they're not global state, I don't see how I can write functions and test them in the repl hat gets access to these resources.
and there's all kinds of dev-only hacks, you can even start a repl inside the let block
@qqq of course it's possible to have a global mutable value, but clojure doesn't have facilities to do so sanely (and arguably no other language does either)
Let me take a step back and avoid the XY problem. I'm doing CUDA in Clojure. Yes, I know about JCuda / ClojureCuda. It's important for me to retain refs to all my Clojure contexts / memories -- because if I lose ref to them, I can't free them, and it becomes a memory leak on the GPU, which forces me to restart the Clojure repl to recover the memory. During dev time, I need eomse way to keep track of a bunch of CUDA resources -- and not lose references ot them. My current dev setup (all in Emacs + Cider + eval last sexp, eval buffer, ...) involves: allocating some CUDA resources initializing their value write some code, run it, iterate thus results in the "I need a global map to stay alive across reloading namesapces" problem
This is also 100% dev time problem; during runtime, I am not reloading namespaces / dynamically modifhing funtions. 🙂
I suggest here reading up on Java Finalizers and using them to clean up. Worst case situation is that you have to do
(System/gc) every so often if you're afraid you've left resources open
if you use something like tools.namspace/refresh that wipes things out regardless, blacklist that ns as something tools.namespace doesn't touch
clj, should you exclude “.cpcache” from source control or allow it to be checked in?
supporting evidence, the tools.deps.alpha project ignores it: https://github.com/clojure/tools.deps.alpha/blob/master/.gitignore
I'd also consider putting it in your global git ignore file with other things you never want to check in
@qqq why can't you store your state into namespaces you won't change? they won't be reloaded if you leave them alone
Hello! I am trying to AOT-compile a basic hello owrld http-kit app but I get the following error when attempting to run it:
Caused by: java.lang.ClassNotFoundException: org.httpkit.server.AsyncChannel
(basically yes, I am trying to run the class with
java hello (where
hello is my class name))
why are you copying the clojure standard lib, and if that's needed, why aren't you also copying the httpkit lib and the other dependencies you are using?
@hmaurer your approach seems odd to me though - the usual thing is either to make a simple jar that also declares deps, then run it with the help of a dependency mananager, or to make a fat jar (aka "uberjar") containing all of your dependencies
I tried to AOT all namespaces (boot
aot -a) but I get this:
sure, aot is contagious so aot compiling your top level lib will aot compile everything it accesses
but if you aren't finding a java class, then your classpath isn't set up correctly
you don't need aot to get a class for this, it comes with the httpkit jar, if you don't find it your classpath is broken, this is why people use dependency managers https://github.com/http-kit/http-kit/blob/master/src/java/org/httpkit/server/AsyncChannel.java
it's not aot-compiling that's causing the problem, it's running java without the correct classpath
you need all the classes that were in the httpkit jar, not just the classes you make from the clj files
target/org └── httpkit ├── server │ └── Channel.class ├── server$fn__187.class ├── server$fn__212.class ├── server$fn__215$G__206__228.class ├── server$fn__215$G__207__221.class ├── server$fn__215.class ├── server$fn__236$G__202__241.class ├── server$fn__236$G__203__238.class ├── server$fn__236.class ├── server$fn__247$G__204__252.class ├── server$fn__247$G__205__249.class ├── server$fn__247.class ├── server$fn__258$G__208__265.class ├── server$fn__258$G__209__261.class ├── server$fn__258.class ├── server$fn__271$G__210__278.class ├── server$fn__271$G__211__274.class ├── server$fn__271.class ├── server$fn__284$G__200__289.class ├── server$fn__284$G__201__286.class ├── server$fn__284.class ├── server$fn__295.class ├── server$fn__297.class ├── server$fn__299.class ├── server$fn__301.class ├── server$fn__303.class ├── server$fn__305.class ├── server$loading__6434__auto____185.class ├── server$run_server$stop_server__196.class ├── server$run_server.class ├── server$sec_websocket_accept.class ├── server$send_websocket_handshake_BANG_$fn__308.class ├── server$send_websocket_handshake_BANG_.class ├── server$with_channel.class └── server__init.class
you can tell boot to create a jar containing the aot compiled code plus the code for all your deps
also, you can create a jar for graalvm that doesn't include the compiled form of your code, as long as you pass a command line that invokes clojure.main on your namespace and the namespace is a resource in that jar
(it might be that the point here is to precompile, just making clear these are not intrinsically the same thing)
OK - java is as happy to use a jar as a class file on disk in my experience, why not hand it a working jar rather than a directory?
especially if there's a standard way to create that jar and it's known to contain everything you need
(or you can recreate a dependency manager by hand from scratch by mixing all the contents of all the jars you use, but this seems like it's not the best use of one's time)
that's a class that comes with the jvm - if graalvm doesn't have it you'll have to find a replacement? https://docs.oracle.com/javase/7/docs/api/javax/xml/bind/DatatypeConverter.html
native-image -cp cljs.jar -J-Xmx64G cljs.main to see if there is a chance it can AOT compile the ClojureScript compiler
This article illustrates some ridiculously small startup latencies: https://www.innoq.com/en/blog/native-clojure-and-graalvm/
I saw this article on news.yc this morning, but it went over my head. What is the fundamental inefficiency in the JVM that Graal eliminates that makes this happen?
But even if you just use the Graal Java VM (without trying to produce native binaries), there is a talk by an engineer at Twitter showing that it uses 12% less CPU, and for the enterprise version he said 20-something % less CPU. So across Twitter's server farm that is probably more than a million USD per year in savings
The gist I got from the Twitter engineer talk is that the Graal compiler is cleaner and thus compiler engineers are able to make further advances in optimizing things, whereas the existing C2 Java optimizer has reached the point where it is difficult for humans to improve it without breaking it. The Graal compiler evidently does better at escape analysis.
native-image is really just tree shaking + compile-to-native. That alone is pretty cool. So @qqq to answer your question, what you're getting rid of is hot code loading that the JVM supports.
native-image currently doesn't support the creation or loading of new Java classes. Combine that with static analysis and it's fairly easy to see how you could get some really impressive performace.
PyPy's toolchain has been doing this with a restricted set of Python for about 10 years. Some systems like this can even go a step further and take the runtime state needed for your program and lay it out in such a way that "starting" your app is little more than mmaping some data into memory and making the data copy-on-write.
Get that, and the startup time of your app is the time it takes the kernel to map some pages into memory + the opening of any IO required by your app.
@tbaldridge: Does that mean 'native-image 'doesn't support def-type, def-record, and reify ?
no, it means that native-image doesn't support
eval, and therefore doesn't support REPLs
So the compilation path is: clojure app -> AOT -> java .class files -> native-image
because at the
native-image level it's no longer java. There's no java classes, no bytecode, no JIT, just native code and a GC.
native-image, but I bet somewhere the ClojureScript compiler will do something verboten.
I don't know the details of the V8 snapshot feature, other than it sounds a lot like the "save lisp and die" idea
That article seemed to successfully run some fairly non-trivial Clojure programs via
sure, the average Clojure program doesn't use eval or reflection (since they properly type hint)
I'm fairly certain the ClojureScript compiler will do something along those lines somewhere 🙂
Yeah, it will, sadly to get CLJS up and running you need macros, macros require eval, and eval requires reflection, so it'll probably bomb out sometime in the next day or so 😄