This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-12-15
Channels
- # adventofcode (10)
- # babashka (28)
- # beginners (27)
- # calva (5)
- # cljs-dev (7)
- # clojure (134)
- # clojuredesign-podcast (3)
- # clojurescript (7)
- # cryogen (3)
- # cursive (12)
- # datomic (8)
- # devops (2)
- # figwheel (1)
- # fulcro (1)
- # graalvm (2)
- # jobs (6)
- # malli (2)
- # off-topic (43)
- # pathom (1)
- # reagent (4)
- # shadow-cljs (11)
What do you use for tracing in Clojure? I am trying OpenCensus with stackdriver and I am trying to figure out how to trace all fn by automate. Well ok maybe have list of exluded functions. I mean I don’t want to wrap each function with my custom span name. I want it to be automated, but not sure how to achieve it in clever way in Clojure.
Never needed to trace code much... What's the problem you're trying to solve?
(with-open [span (.startScopedSpan (.spanBuilder tracer "foo"))]
(println "foo")
(dorun 5 (repeatedly #(println "hi")))
(doall (map println [1 2 3]))
(for [x (range 10)]
(inc x)))(with-open [span (.startScopedSpan (.spanBuilder tracer "foo"))]No, take a step back. That's an implementation issue. What overall problem are you trying to solve?
On the end if I could use trace / logs / metric and read this data for 1 query and part of the trace it will make it super nice for debugging
Seems a complicated way to debug code...
whatever, at that moment I just want to learn how to do it in Clojure in a simple way.
Databases can log slow queries -- that's the right way to solve that problem.
We use a combination of New Relic and DBA-specific tools for that.
At work we use New Relic and trace everything. It is easy to find long responses and see why it is long.
I don't. But you were the one that said you have a query that takes 5 minutes instead of one
Tracing changes the performance of the code tho'. It changes what JIT can do. So in tracing the code, you inherently change what you are observing. Something to bear in mind.
New Relic should be able to trace your HTTP calls BTW -- it does for us.
I know, but I am learning stackdriver for my own purpose. New Relic is unecessary expensive for my needs.
you can also consider running probes to http endpoints from other providers, statuscake, copperegg, datadog
and for local tracing of "stuff" people often employ structured logging + filebeat (or logstash) , ship it to a elasticsearch machine(s) and analyze the detailed info from there via tools like grafana or kibana
i usually trace where i lost milliseconds, so it's interesting to see people chasing other units 😄
there's also metricbeat which ships machine data to elasticsearch , so you can track other possible things that can make your applications slow
e.g. your linux providers' automatic update cycles (ubuntu for example has decided that all the planet has to do maintenance and other daily cron stuff at 6:30 utc roughly ... which is a bit unpleasant if your hundreds of machines figure they have to do it all at almost once 😄)
My purpose is to use something which I don’t have to maintain and invest my time into it. So all prometheus and similar stuff is not what I want.
And I want to have logs / metric/ trace in one tool so I can easy select period of time to see what happen
i don't need to trace each function, i only trace things that do i/o and for places that may handle more than a reasonable amount of records i have statistics counters for both time and recourd counts processed.
it's impossible to trace every function call if you run your code on several hundreds of machines 🙂
and "i trace" -> i drop numbers into elastic and have the visualization + alerting tools hooked to it. i only work 8 hours a day 5 days a week, but the services run 365/24/7
our usecases may be very different though 🙂
but one thing is sure, if you start to trace every method call you will start to do different stuff than the jit does with the code when your traces are not there. so you are tracing "something else" than the trace-less code would do.
> it’s impossible to trace every function call if you run your code on several hundreds of machines 🙂 We are doing this in work. It is possible 😉
but what I am doing now is : I want to learn how to do it in right way for myself. I want to develop my skills in this topic.
i dont think you understand , we are logging terabyte already at the i/o + stats logging level
would you imagine how much we would log per month if we would trace every function call ? 🙂
i couldn't possible afford the storage amount and the storage speed required to write so fast
as said, it sounds like we have very different usecases 🙂
I don’t know how much we store in work, but we have at least 400 servers. I don’t know more info at that moment.
I was reading it is not recommended to store 100% of samples. But my intuition says me it is the right way. Otherwise it doesn’t make sense.
I would like to hear any tips / best practices from your experience how to do it right on production
i process payments. i have to keep all the logs. 🙂
but yes, in other fields sampling is a very healthy choice 🙂
i definitely recommend stats logging ... min max avg & percentiles
for the places which concern you
percentiles give you a really good idea how good-bad your service really is for the end user
if your logging is reasonable enough it provides a soft trace. it's not method-by-method trace yes.
as said before, i can not see the real idea behind stack tracing method by method (as it will hinder jit and show you a different result than what would take place without tracing)
and it's only affordable if your one request has considerable money behind it to pay for all the tracing 🙂
I was reading about tracking. I will start from selected function by my own and always sample. Probably I will make my own macro for this purpose. Something like defn-trace to replace defn and use name of the function for span name. How did you solve it?
replacing defn would only replace it in your current namespace ... and if you actually want to track what's going down in the machinery you should check the jvm level rather (if your aim is to improve performance)
clojutre had a talk which partially covers the topic
So what you are trying to say is… what is exactly? When do you like to use tracing? And when not?
you really need to have the pedal to the metal for new relic's tracing to make a difference though
I have a question that maybe very stupid, clojure.spec encourage using qualified keyword, so there are
#:x{:a 1 :2}#:x[:a :b]s/keys.well there is (s/keys :req [:x/a :x/b])
@lennart.buit I mean I can't write #:x[:a :b].
I think its mostly how useful it would be. Most useful was for maps, so it was added to that
yeah, I just don’t encounter that situation often, as in ‘Oh what would it be nice if I could ns-qualify this vector of keywords’. But I also find {:x/a 1 :x/b 2} more readable than #:x{:a 1 :b 2} :’).
personal preference I guess
I recall one of the objections was that it would have to extend to sets by analogy: #:x#{:a} which looks pretty bad
I've found that I rarely use that notation for maps as well. I tend to almost never have homogeneous maps.
The original design consideration was indeed, where do you stop. We decided to stop at maps.
Is there a way to know which java modules are required to run a uberjared clojure program? I tried with jdeps --list-deps my-uberjar.jar but I get "java.lang.module.InvalidModuleDescriptorException". 😞
How can I add missing functions to ClojureDocs? Currently I don't see thrown? and not-thrown? from clojure.test.
Only ways I know would be to create your own modified web site, or create an issue on this Github repo requesting addition of things you would like to see. Click on the "open a ticket" link near the bottom left of the http://ClojureDocs.org home page.
Oh, yeah, or what @U45T93RA6 said about those particular things. They are custom only-inside-of-`is` macro invocation things.
Oh, that makes sense. I really wish it was more discoverable though. Like, when searching for thrown?, I wish the UI would tell me to look at is.
I've been struggling with this a bit in expectations.clojure.test because it has several "special" forms that aren't actually functions or macros -- they're part of the DSL (`more`, more->, more-of, in, from-each). clojure.test is the "same" in terms of providing a DSL with elements that aren't functions/macros. One option I've considered is adding dummy versions of the "functions" with docstrings that explain what they are, but which throw exceptions if they're used incorrectly...
The equivalent for clojure.test would be adding macros for thrown? etc -- but changing part of Clojure itself like that would need a lot of justification.
The Clojure core has a special case for (doc catch) that has its own custom doc string, IIRC, since it is a similar only-inside-of-`try` thing.
But such things are not very common, so there is no general way to add new things like that for doc to find that I know about, other than adding dummy doc-only Vars with the desired names.
(monitor-exit try do if new recur set! . var quote throw monitor-enter def)From clojure.repl/special-doc-map
I haven't tried, but maybe one could consider a hack like alter-var-root! on that?
Trying to find where catch is handling...
Oh, duh, it's in the first line of doc itself!
(if-let [special-name ('{& fn catch try finally try} name)]Yeah, that one is pretty baked into the code.
I have created dummy defs for such things in my own code, but it was mainly so they wouldn't show up as unresolved symbols in the IDE.
It’s honestly kind of hard for me to understand the suggestion. Map = “This is a set of names with values.” List = “These are unnamed, positional, possibly duplicate values.”
It makes sense to allow you to document based on position. But integrating that into the reader is a little more dubious.
I've been struggling with this a bit in expectations.clojure.test because it has several "special" forms that aren't actually functions or macros -- they're part of the DSL (`more`, more->, more-of, in, from-each). clojure.test is the "same" in terms of providing a DSL with elements that aren't functions/macros. One option I've considered is adding dummy versions of the "functions" with docstrings that explain what they are, but which throw exceptions if they're used incorrectly...
A little bit surprised by below result:
(def f <)
(if (= f <)
1
2)
(case f
< 1
2)
; if yields 1, case yields 2Compare to this for some insight, perhaps:
user=> (case < < 1 2)
2
user=> (case '< < 1 2)
1I believe that the < as a case tag is the symbol < , not the value of the function named clojure.core/<
For the hell of it I went and instrumented case and went through its evaluation and while I can't assume 100% without looking through the java part of it, that appears to be the case 😄
In (case < < 1 2) the first < evaluates to the function named clojure.core/<
In the doc string for case , these sentences are relevant: "The test-constants are not evaluated. They must be compile-time
literals, and need not be quoted."
That's also why you can write expressions like (f x) in any test-constant which, confusingly enough, looks like a function call but is in fact just a list of two symbols.
case is a macro, and has complete control over which parts of its arguments are evaluated, and which are not.
Question: If I have a namespace map (ie. #user{:id 1 :name "John"}), is there a way to treat like a normal map so I can do (get some-namespaced-user :id) ?
You can remove the namespaces from the keys of the maps:
(into {} (map (fn [[k v]] [(keyword (name k)) v]) m))(^If you do, watch out for ambiguity: {:user/id 1 :person/id 2} )
only time it makes sense to remove the ns (or to rename keys at all) is when the data is leaving your program and your encoding doesn’t support namespacing
(Yes, this is oversimplifying in the case of an existing codebase. But it would be wise to start there and carefully work your way elsewhere by listing out what you gain.)
(No, I don’t know whether you’ve done that already. Perhaps I’m just preaching to the choir 😄 )
Just get used to namespace-qualified keywords. Don't fight them.