This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-08-31
Channels
- # announcements (3)
- # aws-lambda (1)
- # babashka (122)
- # beginners (241)
- # calva (28)
- # cider (7)
- # clara (7)
- # clj-kondo (43)
- # clojars (5)
- # clojure (326)
- # clojure-europe (60)
- # clojure-italy (2)
- # clojure-nl (4)
- # clojure-spec (21)
- # clojure-uk (4)
- # clojurescript (162)
- # cursive (30)
- # datomic (3)
- # editors (5)
- # emacs (4)
- # figwheel-main (1)
- # fulcro (24)
- # gratitude (3)
- # helix (7)
- # honeysql (20)
- # improve-getting-started (1)
- # introduce-yourself (11)
- # jobs (3)
- # joker (2)
- # kaocha (15)
- # lsp (21)
- # lumo (2)
- # meander (3)
- # off-topic (34)
- # re-frame (6)
- # reagent (1)
- # releases (4)
- # rum (2)
- # shadow-cljs (37)
- # spacemacs (16)
- # tools-deps (16)
- # vim (23)
- # xtdb (32)
Are there any reliable ways to go from a var representing a function the the source code for that function? I noticed that clojure.repl/source-fn doesn’t work when forms aren’t compiled though a repl. I think they may be missing metadata, or are not loadable through RT/baseLoader
. I think I can write a defn-like macro that attaches the literal body of the function’s code as metadata on the var… is this insane?
It should work outside of a REPL as well, just need to have the source available as a resource.on the classpath
Which should be the case for most dependencies, since AOT should only be used for the final app packaging
the code: https://github.com/escherize/memory-whole/blob/main/src/memory_whole/trace.clj#L34 the test: https://github.com/escherize/memory-whole/blob/main/test/memory_whole/trace_test.clj#L19
in fact, the single gotcha with source-fn is that it doesn't work for code that was entered into the repl
and depending on how your load code (eg. via an extension in your editor), it might effectively be entered in the repl
> clojure.repl/source-fn doesn’t work when forms aren’t compiled though a repl I don't understand this part: source-fn should work as long as the source was loaded via require
my google fu is failing me, but IIRC there was a lib (from technomancy?) that added function source to the metadata of the var
It's very likely you're not creating your classpath properly. Are you sure you include the test directory?
The doc-string says it all: > This requires that the symbol resolve to a Var defined in a namespace for which the .clj is in the classpath > All you need for source-fn to work, is that there must be a source file for the namespace included on the classpath, that's where it looks for the source.
That might be it actually @U0K064KQV - I’ll check later tonight. I’ll look at that library too @U051SS2EU Thanks to both of you!!
Might help you to look at the source-fn code as well: https://github.com/clojure/clojure/blob/master/src/clj/clojure/repl.clj#L147
It needs the var to resolve and the :file and :line meta to exist on the var. Then it looks up the file on the classpath, opens a reader to the :line meta, and uses the clojure reader to read the next form into a string.
Maybe I need to take a break. something is not adding up:
(defn f1
([] 2)
([x] 3)
([x y] 4))
f1
;;=> #function[memory-whole.core-test/eval15605/f1--15608]
(meta #'f1)
;; => {:arglists ([] [x] [x y]), :line 29, :column 1, :file "/Users/bryanmaass/dv/memory-whole/test/memory_whole/core_test.clj", :name f1, :ns #namespace[memory-whole.core-test]}
(symbol #'f1)
;;=> memory-whole.core-test/f1
(repl/source-fn (symbol #'f1))
;; => nil
(repl/source-fn #'f1) ;; of course this shouldn't work
;; =exception>
;; 1. Unhandled java.lang.ClassCastException
;; clojure.lang.Var cannot be cast to clojure.lang.Symbol
(repl/source-fn 'f1)
should wok, if you loaded the file via require
(oh, the meta already shows that yeah)
Ya, source-fn takes a symbol, if the symbol is unqualified, it will try to resolve it to the Var in ns, otherwise it will resolve to the Var in the namespace it is qualified for.
that's the first requirement, that the Symbol passed to it must successfully resolve to a Var.
The next requirement is for the Var to have :file and :line metadata defined on them.
I think though.... since your test was testing a defn on the same line as another, I don't think that will work with source-fn. It will only go to the :line, and then it will start reading form there, so it will never see the function after the first one on that line I think, because it won't skip :column
Anyways, then the next requirement is that there exist on the classpath a source file that corresponds to the namespace of the Var it found.
If I were you, I would try running the when-let of source-fn to test which one fails:
;; try:
(resolve x)
;; try:
(:file (meta (resolve x)))
;; finally try:
(.getResourceAsStream (RT/baseLoader) (:file (meta (resolve x))))
;; Where x is the symbol you pass to source-fn
My guess is the last one fails, which means it doesn't find the source file on the classpath.
When I run over a repl, it works - but when I run it via lein test
it fails. I was printing the classpath from the test and saw the test directory on there. I’ll try these later tonight. thanks youse guys. I think this library is gonna be really neat
idk if you know what I’m working on, but basically a version of trace that can version your code by its source, and generate specs/malli schemas from the passed in arguments and return values. next I want some way to use that info at dev time to answer the age old question “What kind of data does this darn function {take/return}?”
ns + function-name is not good enough for “call-time identity” which is why its soooo important I grab the sources
Hum, ya it's possible that lein loads the test classpaths differently in a way that makes it not find the file.
one thing to check: contents of (System/getProperty "java.class.path")
or better yet
(re-seq #"[^:]+" (System/getProperty "java.class.path"))
ok, I ran that snippet, and from the repl:
"/Users/me/dv/memory-whole/test" "/Users/me/dv/memory-whole/src" "/Users/me/dv/memory-whole/test" "/Users/me/dv/memory-whole/dev-resources" "/Users/me/dv/memory-whole/resources" "/Users/me/dv/memory-whole/target/classes" ... (the rest is in .m2)
from the test:
"/Users/me/dv/memory-whole/test" "/Users/me/dv/memory-whole/src" "/Users/me/dv/memory-whole/test" "/Users/me/dv/memory-whole/dev-resources" "/Users/me/dv/memory-whole/resources" "/Users/me/dv/memory-whole/target/classes"
and they are the same.I did find a clue. the :file metadata attached to the vars defined via defn in the test namespace, during the test are "memory_whole/core_test.clj"
However when I check that from the repl I get their absolute paths.
I can match up the (broken?) file metadata I am getting in lein test
with the classpath data to find an actual, likely, file. I’ll try this today!!
that's the normal metadata, and io/resource can use it to get the file
@U051SS2EU i’ll add that to my todos
It could be a classloader problem, if the test sets up its own classloader, then the relative paths to it might not work. You'll notice the code uses the (RT/baseLoader), which is a dynamic lookup:
static public ClassLoader baseLoader(){
if(Compiler.LOADER.isBound())
return (ClassLoader) Compiler.LOADER.deref();
else if(booleanCast(USE_CONTEXT_CLASSLOADER.deref()))
return Thread.currentThread().getContextClassLoader();
return Compiler.class.getClassLoader();
}
So if something sets a classloader to Compiler/LOADED, or if the current thread is used and the test changes the current thread context classloaderMaybe you should try running your tests with: https://github.com/cognitect-labs/test-runner instead to see if lein does something weird with the classloaders
kind of a tall order. But then again this is gonna probably a bad idea…
(defmacro defm [sym & body]
`(do (defn ~sym ~@body)
(alter-meta! (resolve '~sym) assoc
:fn-body (concat (list 'defn '~sym) '~body))))
(defm i-am-now-traced
"heres a docstring >:]"
([] 1)
([a b] 2)
([{:keys [a]} {:keys [b]} {:keys [x y z]}] (str (* a b x y z))))
(meta #'i-am-now-traced)
;;=>
{...
:name i-am-now-traced,
:fn-body
(clojure.core/defn
i-am-now-traced
"heres a docstring >:]"
([] 1)
([a b] 2)
([{:keys [a]} {:keys [b]} {:keys [x y z]}] (str (* a b x y z))))}
Random note, I just realised that one can make defrecords implement protocols via metadata by using with-meta
after constructing the defrecord.
Seems a sweet middle ground for when you don't want to renounce to a familiar defrecord
structure (which might be a hard refactoring anyway for a given app) but metadata extension is handy (for me it aides a repl-driven workflow where I constantly re-eval forms using various tools... not always in the right order)
It's also great to have more defns and fewer methods (which are harder to spec and test) in your codebase
has anybody experienced java-time library having cannot find declaration to go to
problem in Intellij + cursive?
Yes, it is a known problem - that library uses a tool to make its symbols available through one common namespace, the tool that Cursive doesn't know about.
Two solutions:
• Stop using java-time
because nowadays Java time API is good and more explicit
• Require the specific namespaces instead of the whole java-time
.
E.g. (jt/zoned-date-time)
becomes (ZonedDateTime/now)
with the right import from java.time
. Unless you're using java-time.clock/with-clock
.
If you’re using Java Time, you should get a suggestion to create function stubs like this: https://cursive-ide.com/userguide/macros.html#stub-generation
If I use a clojure.core.async go-loop, and it accesses an unsynchronized map like java HashMap, will it do concurrent access? Seems like the answer is no, it behaves like a single thread, but I am not entirely sure…
I guess it depends on the go-loop details, you can always println the name of the current thread
It's always wise to treat mutable data carefully anyway.
Trying to answer your question, if nothing other than the go-loop
reads/mutates the hashmap, then there's no concurrency
Mutable data still is tricky: a write from thread T1 may not be ever visible to thread T2. Which is why the volatile
Java keyword exists (besides from other patterns, like synchronized
).
So if you have the chance, use a vanilla clojure map instead
if you have to use the java map, prefer a ConcurrentHashMap
or Collections.synchronizedMap()
instead. ideally, use a clojure map.
Go loops will get multiplexed over the go thread pool so what’s above does not seem right (there is concurrency, just not parallelism). You must supply some kind of synchronization or there is no guarantee in the jvm that other threads will see changes to an unsynchronized data structure
Aha, so I’ll still be better off with an atom and normal clojure map
my bad, TIL (or TIGot a refresher). Funny to think of go-loops as code that can race against itself :)
not a race, issue of safe publication
@U66G3SGP5 best would be to track it in a loop/recur
if possible.
It’s hard for me to imagine that you’re using core.async and you cannot track state in loop/recur.
volatile!
isn't thread safe
From the https://github.com/clojure/clojure/blob/master/changes.md#21-transducers: > volatiles - there are a new set of functions (volatile!, vswap!, vreset!, volatile?) to create and use volatile "boxes" to hold state in stateful transducers. Volatiles are faster than atoms but give up atomicity guarantees so should only be used with thread isolation. Don't we have thread isolation here?
I also see this https://clojuredocs.org/clojure.core/volatile! on clojuredocs: > Changes to references made `volatile` are always "written through" CPU caches all the way to main memory (which is somewhat expensive), this means changes are guaranteed to propagate to other threads (nescessary in stateful transducers).
if you do, then java HashMap as they wanted in the first place would be fine
But since only 1 thread is operating on the data at a time, I think that is the problem volatile solves - the safe publication issue
OK - I misunderstood the premise here, my bad - it's about the fact that a change to HashMap might not write through to main memory, so when a go block is picked up by a new thread it might not see that change
got a question with web dev (luminus). i went through the book Web Development with Clojure, somehow got the impression that “controller code” is also in the routes file. is that really the case?
do you have an example?
#luminus might also be a good place to ask luminus-related questions
it's common to add certain features (which traditionally are in the controller domain) via middleware, and it's not uncommon to attach them via the router (eg. when a specific subtree of routes use a specific middleware feature). in strict MVC this could be seen as putting controller implementation into the router.
it's possible to do MVC in a ring app but there's nothing really enforcing that structure
@UEENNMX0T @U051SS2EU it’s this code, i guess it’s the handlers that look like controller code
@UEENNMX0T oh and also didn’t know there’s a luminus channel. i’ll look it up thx
Forewarning: i don’t know luminus at all. to me, those should be in their own functions, maybe even in their own namespaces. I am a big fan of creating stand-alone functions instead of inlining anonymous functions, so I would at least pull that logic out so I can play aorund with it and potentially even reuse it somewhere else
@UEENNMX0T yeah same sentiment. Ok so looks like im extracting these into their own modules/namespaces. thanks!
ahh, yeah, I forgot that the sample project just puts the handler function inline in the router, does it even claim to be doing MVC?
because of course if this was mvc, the handler would be the controller
@U051SS2EU well yeah it does not say it was doing MVC. i think i was just assuming it cause i got so used to doing MVC all the time.
i guess my question is how do people structure their code in clojure webapps? not sure if mvc is practiced in clojure webapps
beyond a bare bones demo like above, you'd put request handling functions in their own namespace(s)
with a separate namespace for building the main app (including stateful resources like db access etc.) and sometimes a dedicated namespace for routing
I've never worked on a large project that was strictly model / view / controller architected
ok so up to us how to segregate them responsibilities. got it @U051SS2EU
yeah - I think there's a general "pattern" if you pardon the term, where in clojure we have ad-hoc combinations of higher level tools where in other languages there would be a reified / named / idiomatic design pattern
So, can anyone provide a steer in grouping a sequence into a map, but where the source data has multiple keys. So
[{:id 1
:title "Parens ftw"
:tags ["clojure" "coding" "fp"]}
{:id 2
:title "Embrace SQL"
:tags ["data" "coding"]}]
becomes
{"clojure" [{:id 1
:title "Parens ftw"
:tags ["clojure" "coding" "fp"]}]
"coding" [{:id 1
:title "Parens ftw"
:tags ["clojure" "coding" "fp"]}]}
If tags were singular, a simple group-by
would suffice, but they’re not.just as an alternative to a reduce with a nested reduce, you could use the xforms lib to do the group by @U064B4L0K
(require '[net.cgrand.xforms :as x])
(->> [{:id 1
:title "Parens ftw"
:tags ["clojure" "coding" "fp"]}
{:id 2
:title "Embrace SQL"
:tags ["data" "coding"]}]
(into {} (comp (mapcat #(mapv vector (:tags %) (repeat %)))
(x/by-key first (comp (map second) (x/into []))))))
;; =>
{"clojure" [{:id 1, :title "Parens ftw", :tags ["clojure" "coding" "fp"]}],
"coding" [{:id 1, :title "Parens ftw", :tags ["clojure" "coding" "fp"]}
{:id 2, :title "Embrace SQL", :tags ["data" "coding"]}],
"fp" [{:id 1, :title "Parens ftw", :tags ["clojure" "coding" "fp"]}],
"data" [{:id 2, :title "Embrace SQL", :tags ["data" "coding"]}]}
or, without the xforms lib
(->> [{:id 1
:title "Parens ftw"
:tags ["clojure" "coding" "fp"]}
{:id 2
:title "Embrace SQL"
:tags ["data" "coding"]}]
(transduce (mapcat #(mapv vector (:tags %) (repeat %)))
(completing #(update %1 (first %2) (fnil conj []) (second %2)))
{}))
oh - it wasn't clear at first that the strings under "tags" were becoming keywords
Oh, sorry, that isn’t important, well I don’t think.
Or with Not sure about this given that you need to accumulate separate items under the same key.(into {} (mapcat ...) ...)
.
I’ll explore more with mapcat. I was playing earlier with nested reduce
and I struggled to get it working…
(reduce (fn [m entry]
(reduce (fn [m tag]
(update m tag (fnil conj []) entry))
m
(:tags entry)))
{}
input)
{"clojure"
[{:id 1, :title "Parens ftw", :tags ["clojure" "coding" "fp"]}],
"coding"
[{:id 1, :title "Parens ftw", :tags ["clojure" "coding" "fp"]}
{:id 2, :title "Embrace SQL", :tags ["data" "coding"]}],
"fp" [{:id 1, :title "Parens ftw", :tags ["clojure" "coding" "fp"]}],
"data" [{:id 2, :title "Embrace SQL", :tags ["data" "coding"]}]}
So the outer reduce would create the keys, and for each key, I’d reduce the original sequence again?
the outer iterates the entries, the inner adds / updates the tags
Oh, right, let me digest that. It was close to one of my earlier attempts… Thank you - so simple.
of course keywordizing it is trivial, but not every domain has the guarantee that its tags are sane keywords so my first instinct is always to not keyword
Yeah, that’s why I was leaving it. I don’t think my actual domain allows for symbols.
wait! who mentioned symbols
(minor nitpicks, keywords are not symbols)
Sorrry, keywords 😆
let me guess, you know ruby
Yeah, indeed 😆
Clojars "forgot password" instantly gives 429 Too Many Requests
without ANY login attempts. Had the same yesterday but figured it'd go away after not trying for a while.
oh wow I just used timbre for logging errors and wow that cuts through the nonsense in the stack traces
https://stackoverflow.com/questions/35076096/how-to-filter-stacktrace-frames-in-logback seems logback could be configured to do the redundant removals.
I'm reluctant to use timbre because slf4j won on the JVM, and you still have to set it up for, e.g. jetty, etc.
Sure, it's an approach. But then you lack the maturity of logback's configuration around, e.g. file rotation & compression at size thresholds.
https://dzone.com/articles/filtering-stack-trace-hell heh, apparently java people are hitting this too.
We went from tools.logging
(and, I think, slf4j) to Timbre and it was great in the honeymoon period and then we just started to get frustrated by how non-standard it was and all the JVM side work we had to do to get everything working with Timbre and so we went back to tools.logging
and this time with log4j2.
There's nothing wrong with full stacktraces when they're in log files that you can go read over later. And you don't need/want color stuff in those log files -- that stuff's just "pretty" for interactive dev.
Timbre papers over the mess of JVM stuff but as soon as you work with (Java) libraries that do their own logging it really all falls apart, IMO.
And, yes, setting up and configuring all the Java logging stuff is awful -- the docs are terrible and every different Java logging library is subtly (or not so subtly) different.
If you can stay in the Clojure world and avoid any libs that do their own Java logging, Timbre works. But that's not very practical in the long run.
I think it's worth the pain of just embracing stacktraces in all their plain text, voluminous glory. At some point you're just going to have to do it anyway 😐
@U0K064KQV Stu Halloway tweeted that, yes.
I feel with logs people sometimes have a bit of the same simple VS easy. Like sure most logs hurt your eyes and seem full of noise, but when you actually need to debug something... Is the relevant info somewhere in there?
I'm not convinced. Humans can feel something is "hard", but that doesn't mean it makes it truly harder to debug. It means they are annoyed/have to concentrate more then they would want.
I wouldn't be against better tools for viewing logs. But your primary focus I think when logging should be: am I logging the information that would allow me to debug this? And secondarily you could ask, will it be easy for a human to read/understand
beware trivial inconveniences! humans have a tendency to avoid all pain and even when they’re well-intentioned, such pain-points/places of friction build up over time. we’re far too late now to fix java’s horrible stack traces, but i am firmly in the camp that clojure is not java and is not beholden to java’s faults
I'm sure you could have both. I've just often seen people sacrifice the first in favor of the second. So I'm always a little on edge. Because it seems a lot of "easier for human" leads to dumb down logging that will have less information
https://github.com/AvisoNovate/pretty/blob/c38997e779483bf1ff3d8d13c9a9b9e25c24b8be/src/io/aviso/exception.clj#L123-L127 the rules in pretty are simple enough to replicate in logback. I reckon we could hook pretty up to logback still though too.
I think, with background logging from long-lived processes, it's important to capture as much information as possible in the log file (or database or wherever you store that information) because you can't go back after the fact and get any missing information.
Well "harder" in that sense isn't a useful metric in my opinion. I'd much rather we measured: Was able to find the bug? and Time to debug?
If you haven't seen https://github.com/stuartsierra/log.dev in this space it's a good morsel 😉
in my opinion the cost of potentially losing important information is not worth it for the benefit of requiring less time to read a stack trace
I don't think there's any conflict between "log every possible piece of information" and "provide a human-friendly way to read that information".
I have a grudge about aviso in particular, because I've had the bad experience of random libraries "injecting" its pretty printer into my system just because I required some namespace
We've certainly had situations at work where what we logged just wasn't enough to debug an intermittent problem that was hard to repro -- and we now have an adjunct to our logging that actually forces a stacktrace to provide more context to what is logged. We don't always need that extra stacktrace information but when we do we are very glad of it.
I think you're right that human play an important role, and if they find things hard, won't be motivated, or need.too much focus they can easily miss things and do mistakes. But I think it's a case by case. Logs are a last resort. A junior on-call might be like... OMG my eyes only see gibberish I have no clue how to debug this. But if you call in a more senior person they can find the problem. But if the junior dev had decided to not log the stacktrace cause they don't even know what it is or how to parse it mentally, or they didn't include key details like the timestamp, request id, and all that. Well even the senior won't be able to debug it. Now.youre stuck, logs were your last resort.
And, yeah, like @U051SS2EU I've also been "bitten" by Aviso trying to be "helpful" 😕 so I tend to have a bit of a knee-jerk reaction to it -- especially since it seems to creep into a lot of beginners' Leiningen profiles setup as I recall? That and ultra
...
(maybe it's ultra that drags it in?)
Here's my take, do all the prettifying at read time. You don't want weird ASCII char code for color highlights in your actual log file
that's a big argument for structured logging - if your logs are reliably machine readable you can do that kind of pretty formatting on the display side
This reminds me of kaocha and its lovely diffs when expected does not equal actual. I’m hooked on them!
Structured logging is nice, but it shifts the "hard" from reading to writing the logs.
@U8QBZBHGD about Ruby, I think that it’s one or two things that Clojure community can learn from it. I remember the text of @U07FP7QJ0 about it: https://lambdaisland.com/blog/25-05-2017-simple-and-happy-is-clojure-dying-and-what-has-ruby-got-to-do-with-it
@UE21H2HHD I think that Kaocha has all this ergonomics builtin because of Plexus used to be a Rubyist
Maybe it depends how people go about debugging. But I do: grep logs for keywords to find where the logs are for where the issue happened. Then just inspect the content to find any clues. It means I need a way to search the log to find the data about the event I'm debugging. And then I just need to read through to find clues. So having the data is most important, it'll let me search on keywords like time, request id, customer id, or any other identifying piece of info. Once I found that, I just need to look through to see ok so this data came.here, then this happened, then it went there, and here it looks like something was off, etc.
I’m too come from Ruby, in a alternative world all Ruby people joined Clojure to bring all this wonderful tooling to this awesome language
Hum... Explain that hellish scenario? Why would there be multiple stacktraces? Is it because I've failed to capture the correct dimensions to hone down on the exact stack trace that failed?
@U0K064KQV maybe one hidden factor here is expertise / familiarity with generic text processing tooling - it's easier to deal with 100 stack traces in vim than 1 stack trace in a browser window
@U8QBZBHGD oh right not just expertise / familiarity, but also context where that tooling can be applied...
I mean, a stacktrace would show up because an error was thrown, I'm not as extreme as logging stacktraces for happy executions haha. So I can easily grep and group by an exception message, count how many unique exceptions there are, etc.
@U0K064KQV there are classes of error that don't reliably throw where the actual cause occurred, so you end up needing to see surrounding lines of context - eg. when something is stalling core.async, the error could show up in any random core.async using code, and is particularly unlikely to be reported where the culprit is
(which is why I hammer so hard on "never do blocking work in a go block" - getting this wrong is extremely painful)
I guess I'm asking, a little bit in the vein of what Clojure core often does on issues, describe an actual concrete problem, explain the issue and challenges. I think in the abstract... "Stacktraces are hard" doesn't really let you find true solutions. Why is color highlight.the solution? And it fixes what problem? Why would eliding be better? Or displaying a stack trace in a different order then the stack itself?
@U0K064KQV one interesting think is that when you have lines interleaving colors, its more easy on eyes to scan data, the same with tables where rows have alternate colors.
True true, so let me re-frame. Problem: Finding where the log data is for a reported issue Solution : Make sure to include dimensional data that will let you search over your logs and join between log entries across the entire log files In practice: Can be done with Unix tools like grep, if log files can be used as files on a single Unix filesystem. As long as you made.sure to include the correct dimensions and partitions info in your logs.
Yes, the lack of this kind of mindset is what make thinks harder than needed. So imagine the people of compiler driven development languages watching us using our REPLs. They can say: Nah, why you need it objectively, just use the raw tools, it’s pretty enough
The problem with "hard" is that it's personal. Stacktraces are hard for you, I find them super easy. People come to Clojure and say it's hard, it's only hard to them, because they are not used to s-expressions and they see it all as whitenoise. But it's easy for me.
having worked with both, having no access to complete logs and only seeing them through a limited API is a much bigger problem than ugly stack traces and much harder to work around prettier stack traces make the API access to logs a little nicer but doesn't solve what I consider the root problem
I think it's the conflict of interest between the two that I'm arguing about 😝. It's like, prioritize simple and then work on making it easier. There will be trade offs you need to make.sometimes, and i'm saying, don't make them based on hard/easy, that will not end up with a better outcome.
I have not used java in production beyond small stub classes when I hit perf problems in clojure
to be clear I'm not against being able to see prettier logs / stacks, but I've been fucked over by over zealous tooling injection that tried to put that into my app without my consent
I'm not sure, I feel easy is often trying to dumb down things. And that result in the thing being dumber, not you being able to now build more and more complex things.
But this is where I think we need to be concrete. Like what would you change to a stacktrace?
If the dumber easy thing can be desconstructed, why not? I think that there are other communities that are doing very well on this respect
You know I had to multiple times tell beginners to use e*, they had wasted considerable amount of time debugging, but they were not seeing the full exception. Is that really easier?
I feel with beginners you have to be careful. Because you can make things easier for them, but that does not make them better. Once your "easy" facade reaches it's limits, they are screwed and will realize they have no idea how anything actually works.
(ins)user=> e*
Syntax error compiling at (REPL:0:0).
Unable to resolve symbol: e* in this context
(ins)user=> *e
#error {
:cause "Unable to resolve symbol: e* in this context"
:via
...
😄@U8QBZBHGD it's a feature of clojure's compiler
I guess I mean that a stacktrace is not hard in reality. It is only jarring at first, until someone explains it to you, and you learn what it is. Then it's a super simple and very useful debug tool.
in fact, the reason I stopped using CIDER was the fact that it put all exceptions from background threads in a dedicated hidden buffer with no UI to indicate they happened, the bare repl experience of seeing it all in one window was an improvement
I think that the problem is related to mindset, some people accept that we don’t have better tooling, but because we don’t have resources to make it better, so ces’t la vie but other people assume that it hast to be hard, as a feature
A thing that makes it less clear that more information is necessarily better is that there is no cross-platform log-viewer that lets you ignore information that is rarely needed to see where your problem is. So even if the complete information helps some of the time, it doesn't always, and there is no good way to only temporarily hide it (AFAIK)
A little story I have. We used to have our team run book spread across multiple pages of wiki. It was "categorized", but the wiki search was terrible, and you couldn't restrict it to only our own team's pages since each page were not all owned or always under the same hierarchy. One day, I decided to just dump it all into one giant single wiki page. My team complained really hard, they said it's so much worse, nothing is organized anymore, it's just one big dump, they can't make sense of it, there's too much in one page. I insisted we tried it out for longer (advantage of being the team lead haha). A few weeks later, people changed their mind. Ok, this is so much better, I just need one bookmark, and can simply Ctrl+f and find super quickly what I need. In my head it was like, yup, one page is simpler than 50. A flat structure is simpler than a nested hierarchy. A single page search is simpler then a wiki-wide search.
@U0K064KQV > I guess I mean that a stacktrace is not hard in reality. It is only jarring at first, until someone explains it to you my experience was having them explained, continuing to have my eyes glaze over and not being able to extract meaning when I see them, and only painfully being able to use them. After many iterations of this they became second nature and not especially difficult. it seems like they are a variety of literacy (like math literacy), with a niche relevance, but I think it's insulting to imply they were never hard - unless most beginners and I were part of some subclass of disabled individuals (some stack trace specific relative of dyslexia???)
perhaps I'm a rare success story of overcoming a congenital inability to read stacks, but I find it more likely that experienced programmers forget how hard they were
it’d be nice if thinks are not only simple, but easy as well. I was thinking about the astounding tooling that was once provided by Xerox Parc, the Interlisp Machine, simple and easy? Yes, but hard to achieve, not easy to do volunteraly
especially with a language like clojure where various forms of indirection lead to multiple unfamiliar stack entries per function call (not to mention what multimethods, partial, core.async, etc. do to scramble stacks and make them less meaningful ...)
people new to fp (and new to programming in general) have that experience as well (this is hard and makes no sense and I just want a for loop)
and we say "well, a functional programming style has benefits, so try lets try and work through your issues"
Sorry, I didn't mean to say that they were always easy for me, or that people who find them hard are slow learners, etc. I think I mean that something being hard at first, does not mean it isn't worth having. I'm all for making things easier to learn. But I'm against designing things to be easier. That is not a good metric in my opinion. Some things are hard to learn, but once you do, become easy and invaluable tools that really help you with your job 😝
And why other functinal programming languages have better error messages and stacktraces? I know, because they have more resources, and not because they think that it is good to do it while Clojure community thinks otherwise
What % of the time has the stuff in the stack trace that people want to hide been instrumental in letting you track down a bug?
The issue is when you experience something to be hard, your first reaction is: "can we change the way this works so it's not so darn hard!" But that misses the point, the thing was not designed to be easy for beginners, it was designed to let you debug an important issue quickly and efficiently at 4am, when on a company wide call with the VP listening. And you don't want to say... Sorry, but we will need to add more logs and wait for the issue to happen again to capture the data that could let us know why it failed.
@U8QBZBHGD > why even use a higher-level language then? why try to make anything easier? unlike language abstractions, stacks are a shallow problem, they can be learned once and you don't need to expand or inflect that knowledge to move forward (beyond mapping the skill to a new vm / language stack)
the whole line rich is trying to draw with simple vs. easy is subjective vs. objective measures
where simple is an attempt (I am not sure it isn't flawed) to find some kind of objective middle
I don't remember whether I thought Clojure's stacktraces were unduly bad -- but I came from Scala which had some of the worst compiler error messages I'd ever experienced so perhaps I was prepared for "bad"?
C++ is for experts as well, I wonder if I must go back, given that it has to be hard…
I find you can overcome subjectivity as a human though. That's how you reach expertise in my opinion.
you claim to be arguing for subjectivity, so everyone has their own experience, but when others give you their experience that is counter to your goal, you reject it
And I want to re-emphasize, when objectively useful and also easy, we have the best of both worlds. But often these will conflict. That's where as a designer you'll need to make trade offs. Will you choose easy or will you choose simplicity and utility.
I think that's why we need to go concrete. Because now we're not sure what you mean. What would you change to stacktraces.so that they are easier, yet retain all their current utility?
I think the core team's change to the default of how errors were reported in 1.10 was a big step forward -- showing as clear an error message with filename/line number as possible and shunting the stacktrace off to a file for review if needed -- but a lot of tooling still just dumps the whole stacktrace into the repl/editor setup and hasn't taken advantage of that. Nine times out of ten (he said, waving his hands) that one error message is all you need.
Again, that's a case of "recording all the information" but presenting "something easier to consume". I don't think anyone's really arguing against that?
@https://app.slack.com/team/U04V70XH6 and I want to believe that they did not improved it more yet because of lack of resources and not because they don’t care about it.
Well, there's the thing: lots of Java that Clojure interop do capture exceptions and do a .printStackTrace
on it
In this case, there's nothing Clojure can do - the JVM do not accept ->
, and !
, and ?
in the names so we get those weird accept___GT_now_BANG_
on our stacktraces
improved how? there are pretty printers, lots of people use them, lots of other people hate them, should that be a core part of the language then?
@U0NCTKEV8 they improved things on this area from 1.9 fo 1.10 as @U04V70XH6 said no?
There have been some wording changes in exceptions, and some additional (useful) information added to some exceptions -- from Clojure itself -- and those have been specific, actionable things the core team can do. But there have to be specifics for core to change.
they did a lot of things, none of them was what any of those stracktrace prettiers do
In Chlorine, sometimes I also struggle with parsing stacktraces because sometimes it's not really that clear where's the function/place/file that had the problem. This is inherently a JVM problem IMHO, where the .getFilename
of a stacktrace shows a relative path from the classpath 😱
Tools to take raw stacktrace data and present it differently are pretty much all going to be subjective and there's no "one size fits all" approach for the core language in that situation.
this thread started from a discussion of stracktrace prettiers, and I don't think any other improvements have been suggested, so my assumption is that is what those who want changes want
But again, on Javascript, sometimes the stacktraces show file numbers incorrectly on pure JS files, so yeah, not a new problem 😄
@U28A9C90Q Just to be clear, we were discussing logs, not error messages. My impression is they do care to improve error messages, but the challenge is doing so without hurting performance and without making the compiler much harder to maintain. I think that's more the challenge then resourcing.
I am not wild about the stacktrace going in the temp file, but it is kind of a minor annoyance
> This is inherently a JVM problem IMHO, where the .getFilename of a stacktrace shows a relative path from the classpath as an aside, you can find the real source with io/resource
user=> ( "clojure/core.clj")
#object[java.net.URL 0x7cf162bc "jar:file:/home/justin/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar!/clojure/core.clj"]
like making a typo with git, and git suggesting what you might want instead of doing it
here is an error, if you want to know why you program crashed look in this temp file, also your /tmp is now full
@U0K064KQV yes, the way Clojure present error messages are somewhat related with log and stacktraces, but as others said, it is impossible to hide JVM, so it is a fact of life, something that we need to deal with.
Ya, I feel sometimes to hide things that are distracting for beginners, you make it harder to find for those who need the information. So again, it's that trade off between simple vs easy. A temp file is more complex in every way then just printing it all
My humble point of view is you never want to lose any of the data but may want to view it in different ways. So that makes all of you right, yeah?
@U0K064KQV I disagree. The stacktrace is not needed for solving many errors when working at the command-line/via a REPL. But it's important to have it available when it is still needed.
it is tough because it is subjective, so there are lots of opinions, and the most common way for someone to introduce their opinion is "this way seems good to me, so it must be best for everyone, so if you don't introduce it that means you don't care about new users and are a jerk"
I think that's a harsh and unfair characterization of anyone's position.
But sometimes we can provide sensible defaults that can easily be deconstructed at will, but this is not related to the duties of the core team, its something that we, the community must step in, and I won’t, so I rest my case 😅
For me it isn't really about 'easy' vs 'hard', just like viewing my email inbox without a spam filter is not 'hard'.
Part of the problem when discussing making Clojure more approachable is that there are a lot of people who are like "well, it's just a hard language and y'all need to learn all this stuff" which is basically gatekeeping and we really should try to avoid taking that position.
Exactly @U04V70XH6
I would hope that we all are continuing to learn, no matter how much experience we already have...
like, I am not going to trying to drum anyone out of whatever space for not reading stacktraces, you refuse to? fine?
Well, this is where the trade offs applies. What if someone is in a situation where they more often need the stacktrace, that has become harder to get too. What if they are in a position where they more often need a simpler error reporting, then they'd prefer it be hidden away. I think the core team was hoping to keep things simple in their implementation, and have tooling deal with this. But turns out that people find learning to setup and use the Clojure tooling hard as well. So some of that is moving a little bit upstream into core. I don't know that it's necessarily bad. I guess it's easier for me to figure out how to find some hidden debug info, then it is a beginner to parse a lot of debug data. So maybe it's a good trade off. My point being though, there's clearly a tension at play between simple vs easy vs useful and all that. Often it depends on the edge case and what not. You can only debate those trade offs in the concrete, which is where I was hoping we'd head into. Like what do you want to see improved for stacktraces? Colorizing them is fine, but do you want ASCII color sequence code in your log file? Eliding them seems terrible to me, in my experience, logs should always have the data needed to debug things at minimum. Pretty formatting them is totally fine, but I'm not sure how else you format a stack then like each entry be on one line. But I'm a bad UI designer. Log files still need to be compact for storage space though, so I'd still argue for formatting when reading them. Structured logs are great, but who comes up with the structure and the schema? Is that now creating a new problem for the engineers when writing logs? Could it push some to be lazy and skip on logging? Etc.
fwiw the hard part of structured logging in my experience is picking the impl, after that it's just a question of calling a different log macro and sometimes you realize you want to pack some more raw data into the message since it will be machine parsed and all
I am aware, we used some early versions of riak at a previous job and it those stracktraces were impossible
I have seen a lot of structured logs that become something like: {:error-message "Here's a big string which is the same that would have been without structured logs"}
. It is really useful to setup your logs to have automatic dimensions captured and logged using some structured text that you can rely on for search and join though. Like auto-capture request-id, customer-id, timestamp, hostname, environment, etc.
@U0K064KQV sure but even in that trivial case, you can search the message separately from the stack trace, and the rest is just bonus
@U28A9C90Q I'm honestly curious what you mean by "worse" stack-traces. I feel maybe we don't use the same definition. A stack trace is the trace info of the callstack. I'm finding it really hard to understand how it can be "better" or "worse" to be honest. For me, a bad stacktrace is one that failed to capture the full trace of the stack. The more it failed to trace the stack accurately, the worse the stacktrace.
@U051SS2EU Ya true, I'm into semi-structured logs personally, so won't disagree. Sometimes it can make it even harder to read though... Unless you have a tool to parse it for you, so I've seen people complain about them as well in terms of readability 😛
Exactly what @U0NCTKEV8 said, it’s really hard to read
sort of like pr on an exception, except if every frame contained the previous frame instead of as a linear kind of list
I know Erlang and Elixir improved really really all things realated to logs in general, but are different schools of thought
In this aspect, Erlang is close to Clojure mindset in several ways, as a tool for experts, simplicity, etc etc
Elixir, on the contrary, tries to be a tool for experts but doing a lot to make the life easier for experts and no experts, but it’s a target that maintainers set for themselves
At least now it seems simple enough: > Stack is the stack of function calls being evaluated when the error occurred, given as a list of tuples {Module,Name,Arity,ExtraInfo} with the most recent function call first. The most recent function call tuple can in some cases be {Module,Name,[Arg],ExtraInfo}
Though I guess if they don't print each element on its own line, it could be hard to read. That said, I also prefer the whole stack-trace and log entry to be on one line personally, That way it is easier tp grep over, since you can extract the full log as one line and then inspect just that. As opposed to needing to see lines above and below in the log file. That's also a bonus to structured logs, since say you have single line JSON entries, you can then just jq on it and get it all nicely formatted.
I used to like the logfmt format[1] and used cambium[2] to do so [1]: https://brandur.org/logfmt [2]: https://cambium-clojure.github.io/
The best thing would be for Clojure to become a non-hosted language. Then there would not be a stack beyond the Clojure core functions. I think this would be the dream for people. Make interop a very rare occurrence. Enhance the runtime so that the implementation of Core never leaks, not even into the stack. It would also mean that Clojure needs to start everything from scratch and re-invent the wheel, which is kind of against its hosted nature. For example, when I use Chechsire, it leaks jackon, because it's really just a wrapper over jackson. Alternatively, Chechsire could be implemented from scratch in Clojure, using only core functions. And assuming that core functions would not leak their java implementation as well, the stack would be easier as it would be shorter and contain only things familiar to a Clojure dev that doesn't know Java. That would probably kill Clojure though, since it really survives by piggy-backing on all the work of Java and the JVM. Otherwise it would always lag behind, and you couldn't use it for real production use cases. That said, I wonder if there'd be a compromise. Can you elide the stack that belongs to Java? And can you do that for interop as well? And can you replace that top layer with an explanation of what happened in it, without performance issues? And without making some things harder to debug? I think they are looking into that, but it's not a trivial problem.
in my experience hardly any of the stack belongs to java, but if you are confident enough about how core compiles things you can definitely simplify the call stack around clojure's internal workings (eg. doInvoke / invoke / invokeStatic / applyTo / applyToHelper / call)
The hard part is: and can you replace that top layer with an explanation of what happened in it Like say a nullpointer is thrown in the implementation, you don't want to just elide that, you'd want to say something like: "Input foo was nil and cannot be in the case of calling some-function"
it's not a top layer, it's multiple middle layers between every item we'd conceptually recognize as a function call
Java 14 introduced experimental "better exceptions" which I have enabled for development -XX:+ShowCodeDetailsInExceptionMessages
but it doesn't always help 🙂
Oh, it looks like that's become the default somewhere along the line to Java 16...?
another unavoidable factor is that we write code as if delegation were effectively free, but those refactors to delegate to helper functions all make stack traces deeper and harder to read
Java 11:
user=> (def x nil)
#'user/x
user=> (+ x x)
Execution error (NullPointerException) at user/eval1 (REPL:1).
null
Java 14 with that option (also Java 16 without it):
user=> (def x nil)
#'user/x
user=> (+ x x)
Execution error (NullPointerException) at user/eval1 (REPL:1).
Cannot invoke "Object.getClass()" because "x" is null
and thanks to very helpful things like comp
/ partial
/ #()
etc. most of those helpers become absolute gibberish on the call stack
Seems there's a ticket for it already: https://ask.clojure.org/index.php/10470/stacktraces-clojure-functions-contain-irrelevant-information
Ya, Java now reports the exact code that threw the NullPointer, but in Clojure it seems to not be super useful, since that's still inside the implementation. Well, I take it back, it shows the variable name, so that's an improvement
I always try to keep my call-stack shallow. That's why I like the top-level orchestrator pattern:
A -> B
A -> C
instead of:
A -> B -> C
Clojure can get really dense, so this in some sense ends influencing the stacktrace verbosity
I guess I never really experienced any much easier stack traces... But I've not used Ruby. Alex mentions: > I believe JRuby has spent a lot of time on this same problem (showing you a ruby stack trace that came from a jruby jvm stack trace), that might be an interesting thing to learn more about (I have not studied it). So maybe there is a way that could be brought over if more effort was put into it.
but the jruby issue is entirely different, jruby is trying to emulate ruby on top of the jvm, so it is trying to translate the jvm stacktrace into a ruby stacktrace
the goal of jruby's stacktrace stuff isn't to make stacktraces better or easier to read, or whatever, it is to make them look like what you would get from ruby
I think that's what a lot of people would want, is to have a Clojure stacktrace that hides all implementation details. So:
(a (b (c 10)))
Would have:
c
b
a
and nothing elseand other people have so much trouble with libraries that do that that they write other libraries that undo it
Well, interestingly, I don't know if that's true. I'm not familiar with all of them, but most seem to focus on formatting and coloring. Its also possible none of them are good enough.
you can see an example of what it does here https://ioavisopretty.readthedocs.io/en/latest/exceptions.html
hilariously, their "printed normally" example isn't even the normal exception printing, it is what the clojure.stacktrace library prints
A Clojure stacktrace would trace the full path conceptually from the point of view of Clojure code. It wouldn't elide Clojure core functions. At least that's how I'd imagine it. Though, I've also wondered if all the "hard" comes from having all these "fake stacktraces" everywhere and all be different. Like if Clojure just outputted raw stacktrace all the time.
No worries, for what its worth, I thought you brought great counter points that I'm still thinking about.
It's a touchy topic too, it's not like this debate doesn't come often in Clojure, so clearly there's more to learn on all sides.
Since the thread seems to have drifted away from stacktraces I'll drop this here: if you are using some kind of prettier https://github.com/stuartsierra/stacktrace.raw is useful to have in your back pocket
Reading the readme puts a lot of context as well: https://github.com/stuartsierra/stacktrace.raw#why-this-exists