This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-03-02
Channels
- # aleph (6)
- # beginners (57)
- # boot (1)
- # cider (27)
- # clara (23)
- # cljs-dev (166)
- # clojure (287)
- # clojure-dev (23)
- # clojure-greece (1)
- # clojure-italy (2)
- # clojure-russia (13)
- # clojure-spec (34)
- # clojure-uk (36)
- # clojurescript (68)
- # core-async (63)
- # core-logic (1)
- # cursive (1)
- # data-science (1)
- # datomic (26)
- # duct (1)
- # emacs (10)
- # figwheel (8)
- # fulcro (2)
- # garden (16)
- # graphql (8)
- # hoplon (20)
- # jobs (2)
- # leiningen (10)
- # off-topic (16)
- # onyx (2)
- # portkey (5)
- # quil (1)
- # re-frame (63)
- # reagent (95)
- # reitit (6)
- # remote-jobs (1)
- # ring (6)
- # rum (1)
- # shadow-cljs (76)
- # spacemacs (26)
- # specter (11)
- # sql (7)
- # unrepl (68)
- # vim (2)
- # yada (2)
Hi, (type (int 10)) returns java.lang.Integer. How do i create a primitive int in clojure?
int
will actually create a primitive int, but those can only exist in local scope, ie let
when you call int
in the REPL, the value will be boxed before it is returned and printed
take a look here for more info: https://clojure.org/reference/java_interop#primitives
thanks @schmee
let's say I parsed some json, i.e. {"First Name": "bar"}
and I keywordized the keys print like: {:First Name "bar"}
. Would be a safe way of printing this to EDN such that it could be read safely?
A nice discussion here: http://grokbase.com/t/gg/clojure/1236xbzeq4/how-to-escape-a-space-in-a-keyword
Although it’s possible to create keywords with spaces, they’re not something you can use as literals.
FWIW, ClojureScript itself does what @hiredman is suggesting when it needs to print a JavaScript object like that
#js {"First Name" "bar"}
specifically so you can copy and paste it in the REPL, etc., and it can be read.they are legal keywords
they are not print-read round-trippable
they are not valid literal keywords
aye, my use-case is it's coming from user input, and i wanted to try round-trippin' the data using EDN instead of JSON
definitely can, was more so wondering if folks know something that i might've missed
so let's say i'm leaning towards a serialization format that gives me all of clojure's flexibility, would that be to lean towards transit?
On the Clojure side, something like: (clojure.edn/read-string (slurp (:body request))
or variants, depending on your web server.
I know about using reader macros in cljc:
(try
(dangerous!)
(catch #?(:clj Exception
:cljs :default)
e
(solve-it)))
Now, I need to define a macro which expands to a try/catch. How do I get this macro to expand to Exception in clj and :default in cljs ?@qqq https://groups.google.com/forum/#!msg/clojurescript/iBY5HaQda4A/w1lAQi9_AwsJ https://github.com/plumatic/schema/blob/master/src/clj/schema/macros.clj#L10-L31
is there a way to tell clojure not to concern itself with whether or not a pulled in function is private from a separate namespace?
It errors out over a method that was set private, however, it doesn't error out when I pull it into clojurescript
@benzap you can use the #'
prefix on the name to override the private access.
I'm looking to build a very high performance web server. nginx-clojure looks like a good choice. Am I right?
@danielstockton I had good experience using it, it still in progress, so no benchmarks, but it's nice to have both your static resources and some handler in the save instance. I use this to be able to upload images, which then gets used in the html, and served by nginx. Combined with cashing, pages should load very close to just static files.
@gklijs In my case, i'm not planning on serving any static files. It's an API service which will receive a lot of traffic.
Ok, nginx uses multiple JVM do you probably want to use redis to keep start and/or only call other services in the handler, both should be fine.
@danielstockton just wondering, what are your constraints/numbers?
I'd like to use Clojure, that's the only constraint really. It will have to scale horizontally, so i'm not trying to get a certain number of requests from a single instance. I just want to get the most bang for my buck and minimize response times.
@gklijs Sorry, what do you mean by 'use redis to keep start'?
Meant state, not start. But really depends on what kind of API it is. The second call to the API might gets handled by a different JVM instance then the first one, that's all.
Oh right, ok. Yes, I will be making calls to other data stores within the handler (most likely redis is one of them).
I won't assume that the Clojure process has access to state shared between requests, that's for sure.
@danielstockton any reason that Undertow (through Immutant) couldn’t provide enough bang for the bucks?
I was going off these benchmarks: https://github.com/ptaoussanis/clojure-web-server-benchmarks/tree/master/results/60k-non-keepalive#60k-non-keepalive
is it same to assume that jvm exceptions are highly optimized so that if there is no exception thrown, the cost of negilible ?
i think exceptions are just longjumps and are "cheap". But I'm not a jvm guy just what i've heard
ah, so in theory, we can imagine
"global pointer handler; // points to the catch block to call on exception"
then, the cost of a "try" is:
1. save old value of handler
2. set handler
to new catch block
?
That sounds both computationally inexpensive and sufficient for implementing try/catch blocks.
A quick google suggests that try/catch
is relatively quick, it’s the throwing of the exception that’s expensive
AFAIK what’s expensive is to construct the stack trace, so to get maximum performance you should throw a pre-allocated exception, in that case it’s basically just a jump instruction
that’s what core.match
does: https://github.com/clojure/core.match/blob/master/src/main/clojure/clojure/core/match.clj#L75-L76
this blog post is excellent and tells you everything you need to know 🙂 https://shipilev.net/blog/2014/exceptional-performance/
@schmee: a quick search of the blog post seems ot suggest it doesn't covedr this 'pre allocation' trick; can you point me at where that trick is explained ?
he calls them “static” exceptions, if you really want to know what’s going on just read the whole thing
I’ve used that trick in Java :)
@alexmiller since you’re around I’ll ask a completely different question I’ve been thinking about: how long does Clojure intend to support Java 1.6? has there been any discussion around moving the oldest version up to, say, Java 8?
I would like to move to Java 8 as the min bar. We haven’t had a discussion about it for Clojure 1.10 yet
it sounds like you’re trying to begin a semantic argument that I’m totally uninterested in :)
realistically, very few people are using 1.6 (or even 1.7 now) and it’s getting difficult to even support in our build infrastructure
also, invokedynamic, which will totally solve all our problems and make clojure 1000x faster with almost zero effort
jokes aside, I’m really excited about that stuff and I’m looking at ways to get it into the Clojure compiler, hence my question about the version 🙂
@alexmiller I am genuinely confused by the the notion that java version compatibility isn't considered part of the API of Clojure. I accept the difficulties of course, I personally would love to see the minimum bar raised for my own selfish needs, because seeing better java 8 time integration would be awesome. I sometimes hear something which questions my understand of something Rich has said, so this is really an attempt to gain insight into that.
it is of course considered
generally upgrading java is an additive experience - you get access to new stuff and existing stuff continues to work, same as Clojure
so the upgrade path is still maintaining backwards compatibility
> Upgrade path There's a nuance here about the "API" of upgrading Clojure here that's interesting, and completely pivots your ability to shift the requirement. This was useful, thank you 🙂
My understanding of exceptions was that it was supposed to be fairly cheap to take the normal flow brancch, but not the exception branch. Also, I believe that try-catch blocks may have some affect on JVM optimizations
I forget now, maybe it is in that post that I need to read above. I was thinking it may have some affect on certain memory barriers and/or inlining
Also, If I remember correctly, I thought that Clojure seemed to generate heavier byte-code for try-catch than from what I saw in Java… Maybe I’m crazy though. Will have to check again
try/catch is not actually bytecode per se - it’s really a handler table attached to the bytecode
Clojure’s output should not really be any different Java’s afaik
there are effects on inlining, but generally I find there is far more mythology about this stuff than reality and it is constantly improving
Shipilev is exactly the guy to tell you the real answer and not the mythology
so I’d believe him :)
@alexmiller I’m reading through that post above, so far already good. So will continue. I don’t tend to get worked up about the effect it has on perf either. I just have wondered before, especially in Java code, when I saw try-catch control flow being used for whatever reason and it happened to be potentially a method called in a hot path. Something like loading classes having to use ClassNotFoundException errors for flow. I want to relook at my experience with what Clojure generated. I thought it had more bytecode operations involved than I expected and saw for Java
I wasn’t implying that the try/catch was just a bytecode or something like that. I just thought the branching that clj output costs more time
one possible difference is that because we are still using java 5 compatible bytecode, I don’t think we’re using the “new” exception style handling they moved to in java 6 bytecode (I think those are the right numbers, haven’t looked at that stuff in a while), but that’s pretty deep in the technical weeds. we go through asm and generate standard bytecode in the same way as all the other jvm languages in this regard.
hmm, I get a null pointer exception on (pos? ...)
in this code:
processed (reduce + (map second (get grouped :processed)))
processed (when (pos? processed)
(str "/ Processed: " processed))
When can (reduce + ...)
return nil
?user=> (reduce + nil)
0
user=> (reduce + nil nil)
nil
user=> (reduce + [nil nil])
NullPointerException clojure.lang.Numbers.ops (Numbers.java:1018)
user=> (reduce + [nil])
nil
user=> (reduce + 0 nil)
0
user=> (reduce + 0 [nil nil])
NullPointerException clojure.lang.Numbers.ops (Numbers.java:1018)
user=> (reduce + 0 [nil])
NullPointerException clojure.lang.Numbers.ops (Numbers.java:1018)
I’m curious now. This is something I looked at pretty long back. I can check it out and see if I can rediscover whatever I thought I did before
Testing a function that might get stuck in an infinite loop, so wrapping it in a future:
(defn happy-test-case [[input expected]]
(is
(let [f (future (function-under-test input))
result (deref f 1000 :timeout)]
(future-cancel f)
(= expected result))))
After running the tests I am left with high CPU usage, so I suspected some futures where left orphaned, and added future-cancel
. However it does not fix the issue. Ideas?@polymeris future-cancel only interrupts certain specific methods
if those methods never get called, the cancellation does nothing
nothing that is sane to use without shutting down the entire vm
the smart thing to do is write the code in the future so it periodically does something cancellable (or explicitly checks cancellation status of its thread)
(.isInterrupted (Thread/currentThread)) - returns true if someone called future-cancel on the future you are in
one thing you can do is to return a tuple of [cancel-fn future]
where the cancel-fn flips a condition in the loop to false to it stops
@schmee you alread get that for free via isInterrupted and future-cancel
cancellation from the outside is not safe in the jvm, period - if you need to be cancellable from the outside that needs to be done in a cooperative way
if this just happens in testing, note the still-running threads (a status you can check on a future), print a message, shut down the vm
it's basically iterate
+ a transducer, with an optional count limit. But under some circumstances the transducer filters out all values produced by iterate
, and count is never reached
I could add an until-fn
parameter, and check that fn every iteration. Then pass (.isInterrupted ...
from the test suite
does it really hurt to check for interruption? it's a cheap check, and even if you don't plan on doing it, it's a sane thing to put in a loop that's in a future
I'm tempted to write a version of loop that implicitly stops on isInterrupted actually... - would be an easy macro to write
while we are thinking outside the box, if the test hangs, kill the process, and call it failed :P
then you don't even need a future
Yeah, I think that makes the most sense. Just restart the REPL when running tests from it during development
@noisesmith @alexmiller - Just discovered that you can't reduce over a transient map - wondering why this is not supported on transient?
I think transients are not seqable?
what's the use case where you would want to reduce over a transient without making it persistent?
I’m pretty sure that’s intentional but I don’t know that I have a detailed reasoning for why
my impression is they aren't meant to be a replacement for the persistent data, but an optimization specifically for filling them while creating less gc churn
(and therefore better perf)
in that sense, seq would be outside the intended scope maybe?
well - I was trying to mostly work with transient map inside a fold reduce method - I have a nested map and I am basically making it transient on the onset - and trying to work with it before making it persistent
certainly that
yeah, I have never heard of nested transients working out nicely - I bet if you lifted the nested data to a denormalized form where the data was lifted to the top level you'd also get better space usage(?)
I don’t know that it would make sense to build an immutable seq out of a mutating collection
I would do it without transients, and if it is too slow, come up with a faster way to do it, vs trying to make the way you are doing it faster with transients
@hiredman it isn't too slow - it is just taking up too much memory for the amount of data I have
if all you are doing is building a single map concurrently, maybe use a concurrenthashmap
got that from this gist on github - https://gist.github.com/devn/b195dbc67a320021057c
no, I mean, it is hard to see where parens balance in an image, but it looks like you have (do (if ... bang-on-transient) (assoc! ...))
so you are not keeping the return value of the if, which means the mutating of the transient there violates the contract of transients
@bajrachar that will reduce memory usage by dropping your data on the floor :)
you have to use something like loop/recur and pass around the return value the call to assoc!
just like you would have to pass around the return value from assoc
the thing is that assoc! does incidentally update the input in place sometimes - just often enough to lead people to make the wrong assumptions from repl experiments
as soon as an assoc! requires a new leaf on the internal tree of vectors, you simply drop the reshuffled data if you don't use the return value
err - not leaf, I can't say that specifically - I can say there are certain operations that cause data to be shuffled then lost
do you think I am better off using something like a java hashmap in this case instead of clojure data structure?
yeah, I think I mentioned that the other day as the thing that would actually reduce your memory usage
and you can also bash it in place
hard to say, you were in hear the other day saying you are trying to come with an alternative to some in memory database thing, but everything was using too much memory right?
I would be super surprised if any data structure would beat a "database" for memory usage
hey guys, does anybody know a multi-key map implementation? I mean a map where multiple keys map to the same value?
a datastructure like a map will just have a pointer to your object, where as a database is more likely to copy it to some kind of storage optimized format
thanks @hiredman. That does the job, though I was looking something more like data.priority-map
, i.e. with automatic book-keeping in case a value is updated. But your solution gets me 90% there 🙂
yes - however I guess the put and get on the db does take up a bit of time - maybe due to the encoding/de-coding involved - and at this time I don't have enough knowledge to tune that stuff.
well - @noisesmith @hiredman- thank you for your suggestions
@qqq - no, functions don't know or store their names
the actual object name will be based on the name, but the solutions there are hacky
@noisesmith: why is it that stack frames can pull the name out
that’s the class name
you can call demunge to go from the class name to the function name but the other direction is lossy, so you won’t always get the original thing back
@qqq you get something like peregrine.circle$eval47604$foo__47605 which has the name in it, but no part of it is the actual name
yeah, you can process the string
by demunge, I specifically mean clojure.lang.Compiler/demunge
this is potentially a dumb question so apologies in advance: I’m trying to understand the Clojure compiler, and each expr has two methods: eval
and emit
. As far as I understand, eval
, well, evals the code, and emit
emits bytecode. what I don’t understand is: in what context are these functions called? why do both exist? does emit
emit bytecode for what eval
does? if not, what’s the difference?
things need to get emitted as classes to be eval'd and loaded by the JVM, except for like the 'do' block, literals, etc. You should be able to trace it.
yeah, I’m trying to wrap my head around it, but I’m confused by the difference between compile, analyze, eval, emit… a short high-level explanation about how these relate would make it easier to explore on my own 🙂
basically any code in a function (perhaps all code) needs to be code-gen'd (emitted) before it can be eval'd, and not-everything typed at the REPL needs to be emitted to run.
@bronsa I’m still waiting for you to write the comprehensive book explaining the Clojure compiler
okay, so if I’m running a Clojure program from a JAR, eval
will not be involved at all, it’s all emitted?
you can run uncompiled source from a jar - that's normal when using libraries in a repl for example
@alexmiller a book for a dozen interested people doesn't sound like a hit :)
yeah it is fun to explore on your own, I just feel like if I could sit down with someone who knows the compiler and talk with them for 30 minutes I could spare myself soooo much confusion 😛
@tbaldridge does some video series. if you just sat down and did some screencasts and put them behind a paywall of you explaining things, how you develop patches, etc, i'd subscribe and pay for sure
how does that compare to a MethodHandle? Am I correct that a MethodHandle is basically a “native” JVM way to express the same thing?
using modern jvm bytecode we could avoid creating so function classes in some cases, but for AOT it'd be necessary anyway in most cases
okay, this is the stuff I’ve been curious about lately, does this even make any sense at all? 😄 Clojure -> method resolution: emit class as bytecode + dynamic classloader, method invokation: static method call Hypothetical Java 1.8+ Clojure -> method resolution: invokedynamic + bootstrap method, method invokation: MethodHandle combinators?
john rose did a very early experiment with method handles where he implement scheme as a single classfile. where, if I recall, function values where methodhandles
@ghadi what do you gain by using a MethodHandle to call invoke instead of calling invoke directly?
ahh, right, because now all fn calls is a virtual method that does some loads and the calls a static method?
right now for an object to be callable as a function in clojure it must implement IFn
I suspect you are getting confused because of the newish direct linking bits in the compiler
in the general case calling a function invokes an instance method on the function object
the compiler has some optimizations for cases where a function doesn't close over any values
but you still need to be able to create and instance of the class and invoke it the general way
that might be it, when I disassemble (fn [a b] (+ a b))
there is both public java.lang.Object invoke(java.lang.Object arg0, java.lang.Object arg1);
and public static java.lang.Object invokeStatic(java.lang.Object a, java.lang.Object b);
, is that what you’re talking about?
IFn basically solves the problem (to some degree) that methodhandles would solve, so if you really wanted a reason to use methodhandles, you would rip out IFn and instead of using instances of IFn as functions you would use methodhandles
Hello all. Has anyone used Schema to define a schema for validation that a File is being returned from a GET? For like a file that has been uploaded and then someone is using a GEat api to retrieve it. I’m trying to use the :responses
yeah, I’m trying to fit all these pieces together somehow, basically what I’m trying to figure out is “how can I use modern JVM features to make Clojure faster?” 🙂
ghadi I saw the experiment you did with invokedynamic btw, I’m def going to take a closer look at that! 🙂
@schmee if you want the answer to that I'd dig into Graal and Truffle. With the stock JVM you aren't going to make it much faster
But with Truffle/Graal/SubstrateVM I think there's some cool situations there.
Most of that is because as it stands today Clojure is very sympathetic to the JVM. What Truffle allows is to make the JVM more sympathetic to the language.
clojure is surprisingly static for a dynamic language(I think at one point a jruby developer said it was cheating because it is so static) so the jvm does pretty well with it
@hiredman +1 to that
Stuff like transducers and other uses of HOF are weak places for this rule though. Run a transducer over primitives in CLJS vs CLJ and the JS JIT is way faster.
I wonder if they ever get the value object stuff working if that would be a way around that
there are large pockets of static calling behavior but let's not underestimate how dynamic it all is
@tbaldridge I’m actually looking at Graal/Truffle right now! thing i,s, rewriting the whole compiler in Truffle is waaaay out of my league, and even if it wasn’t, I don’t see a Truffle compiler replacing the official one any time soon. so to get these improvements to users it needs to happen in the existing compiler 🙂
direct linking came in with a tradeoff on dynamicity. indy would allow you to have static calls without trading off dynamicity
I think that’s a good way to put it
Truffle doesn't replace anything. It's a library that hooks into the JVM's JIT. Those hooks exist in Graal and JVM9.
https://gist.github.com/hiredman/d6ea3e30be757fef6da6 someone just needs to write a library of invokePrim transducers
@hiredman I don't think that keeps the comp callsites from going megamorphic though
@ghadi do you think the indy stuff in your branch a good starting point for future work in that direction?
@ghadi does the indy stuff support deoptimization?
just invokedynamic in general, is it possible to setup a condition that causes the callsite to be re-initialized?
Could I perform batching/timeouts in a transducer? For example, if nothing new comes in for a while, it should add it's state?
@dominicm no, since transducers are single-threaded and push oriented
I wasn't certain if they were single threaded. A bit of a shame, the idea of a multi-purpose batch system appealed to me. Especially as I wouldn't need to break my transducer pipelines up with an intermediate step. Thanks @tbaldridge!