Fork me on GitHub
#clojure
<
2021-01-13
>
beders05:01:31

hello there, fellow clojurians. I’m trying to find a tracing library where I can provide a fn to customize the trace output. More specifically I'd like to use tap>

beders05:01:29

I looked at tools.trace but there doesn't seem to be a facility to get ahold of the output

dpsutton05:01:06

i was looking at that the other day. seems like there should be a *trace-out* that is by default *out* but able to be rerouted. i think it directly uses *out*. Thus if there's any io in your code you get all or none. wouldn't necessarily solve your issue with tap but would give another way to redirect at least

seancorfield05:01:07

Interestingly, it says of tracer "may be rebound to do anything you like" but it isn't marked dynamic so I guess... alter-var-root?

beders05:01:54

thanks for the suggestions

dpsutton05:01:06

i must be thinking of something else. its a different library. i thought it was tools.trace

seancorfield05:01:22

@beders If you don't mind using alter-var-root to change clojure.tools.trace/tracer you can make it use tap>

seanc@DESKTOP-30ICA76:~/clojure$ clj -Sdeps '{:deps {org.clojure/tools.trace {:mvn/version "RELEASE"}}}'
Clojure 1.10.1
user=> (require '[clojure.tools.trace :refer :all])
nil
user=> (alter-var-root #'clojure.tools.trace/tracer (constantly (fn [name value] (tap> [:trace name value]))))
#object[user$eval409$fn__410 0x6411d3c8 "user$eval409$fn__410@6411d3c8"]
user=> (add-tap prn)
nil
user=> (trace (* 2 3))
6
user=> [:trace nil "6"]
(add-tap (fn [[k n v]] (when (= :trace k) (println "MY TRACE" n v))))
nil
user=> (trace (* 2 3))
6
[:trace nil "6"]
user=> MY TRACE nil 6

👍 6
beders05:01:48

just fired up my repl to mess with it. Thanks a lot @seancorfield

seancorfield05:01:32

Now I know this, I might use tools.trace more since I use tap> very heavily 🙂

dpsutton06:01:34

anyone know how to use the dotrace macro from tools.trace?

dpsutton06:01:15

i'd expect

(defn foo [x] (inc x))
(dotrace [foo] (foo 3))
to work but it gives an error about binding non-dynamic vars

dpsutton06:01:48

is this a vestige of pre 1.3? > ;; As of Clojure 1.3, vars need to be explicitly marked as ^:dynamic in order for > ;; them to be dynamically rebindable:

seancorfield06:01:19

I think that's the issue with tracer as well.

seancorfield06:01:37

Time to open a JIRA or http://ask.clojure.org issue I think!

beders06:01:15

looks like alter-varying trace-fn-call gives me more control over the output

seancorfield06:01:45

@beders That won't help if you just want to trace an expression I think.

beders07:01:59

true dat. I'm instrumenting whole namespaces, so I should be good. Thanks for the hint. trace unfortunately also assumes I want the to pr-str the value

didibus07:01:19

What would people call the namespace prefix? Like if I have namespace com.foo.bar.core what would you call com.foo.bar, in Java it be called the package, but in Clojure?

seancorfield07:01:42

I would call it the "stem" but I don't know whether that's common vernacular.

seancorfield07:01:04

"parent" might be a reasonable designation?

seancorfield07:01:37

(although namespaces are not really hierarchical)

didibus07:01:14

I thought about calling it the ns-group?

didibus07:01:46

meh, I guess it doesn't matter, I'll need a doc-string to explain anyways, don't think there's an intuitive name

hiredman07:01:55

it is tricky because in clojure they have no relationship, com.foo.bar.core is not "in" or a sub part of com.foo.bar

hiredman07:01:33

but because of the way namespaces are mapped to files and directories, on the filesystem there is a relationship

hiredman07:01:45

ns-group is definitely no good, because it implies a relationship that doesn't exist, it really isn't analogous to classes and packages in java

didibus17:01:24

In my case I'm building a scoping rule based on it. Something required in foo is accessible from foo.bar and foo.baz, etc. That's why I'm trying to come up with a name for it. But ya, like you said, it's not really a thing in Clojure, even though it kinda secretly is. I guess there's just no name. I went with ns-prefix since that felt most self-explanatory

Ramon Rios09:01:38

Hi all. Does anyone here have some experience with chesire lib? Does someone have issues withUTF-8 while using parse-string ?

Ramon Rios09:01:57

Something really bizarre is happening 😅 . I got the json into a string with the right utf-8 encode and after parse using parse-string, the strings at the map are not utf-8

borkdude09:01:28

Maybe you can make a repro

Ramon Rios09:01:03

Yes, working on it. Thanks

Ramon Rios10:01:56

We found the issue. Actually it was a Yada misconfiguration 😅

Ramon Rios10:01:00

I tried to repo in a separate project but it turns out it was working

Ramon Rios10:01:09

I thought that was chesire on the beginning because we checked the issue right after the data pass througt the parse-string

Ramon Rios10:01:43

So i went check with Yada and it was missing configuration to charset UTF-8

wasser14:01:04

“Semantic qualifier” or just “qualifier” maybe? I know it sort of overloads “fully qualified” keywords, but I think they serve a similar semantic purpose.

kwladyka15:01:45

There was a trick with var (`#'`) and ring to not reload ns with handler each time after change code anywhere. Something like this

(def handler
  (-> (bidi/make-handler route/route #'resources)
      (logs/ring-debug)
      (wrap-keyword-params {:parse-namespaces? true})
      (wrap-json-body {:keywords? true :bigdecimals? true})
      (wrap-params)
      (wrap-restful-format)
      (add-headers)
      (logs/wrap-exceptions)))
But for some reason it doesn’t work for me now. Do I use it correctly? How to fix this? Manually remember and reload this ns each time when I change anything is very irritating. PS I am not talking about alter-var-root. It should work as it is.

kwladyka15:01:33

ok never mind, I had to add #' deeper in the code

Paul15:01:21

hi, anyone still using overtone (or any clojure based music live coding)? it's been years... i wonder if someone still manage to run it on windows 10... i tried setting it up with supercollider the other day but no luck...

Dr. Moltu17:01:58

Hi, i have a question, why i get clojure.lang.APersistentVector$SubVector type instead of PersistentVector in the following code? when I run the java code I get a PersistentVector :

IPersistentVector subvec = RT.subvec(
                PersistentVector.create(1, 2, 3, 4, 5),
                0,
                3
        );
        IPersistentVector vector = LazilyPersistentVector.create(subvec);
user=> (type (vec (subvec (vector 1 2 3 4 5) 0 3)))

andy.fingerhut17:01:47

Because vec doesn't coerce things that are already vectors into a different type

andy.fingerhut17:01:25

Probably as a performance optimization, since something that returns true for vector?, which a SubVector instance does, already is a vector.

andy.fingerhut17:01:59

Look at the (short) source code for the Clojure function vec to see the conditions it tests there, and the fast path where it returns the existing vector if it already is one.

Dr. Moltu17:01:30

I'm looking at this

(defn vec
  "Creates a new vector containing the contents of coll. Java arrays
  will be aliased and should not be modified."
  {:added  "1.0"
   :static true}
  ([coll]
   (if (vector? coll)
     (if (instance? clojure.lang.IObj coll)
       (with-meta coll nil)
       (clojure.lang.LazilyPersistentVector/create coll))
     (clojure.lang.LazilyPersistentVector/create coll))))

andy.fingerhut17:01:18

In your Clojure expression (type (vec (subvec (vector 1 2 3 4 5) 0 3))), I believe that the code path executed within vec is the one that leads to returning (with-meta coll nil)

👍 3
💯 3
bronsa17:01:55

you can force into a new instance by replacing vec with into []

👍 3
Dr. Moltu17:01:23

ok I get it now thank you @andy.fingerhut @bronsa: )

Alex Miller (Clojure team)17:01:29

vec guarantees that you get a vector in the return. it makes no assertions about whether that is the same or different vector from the input value.

🙌 3
afry19:01:18

Hey all, do you have any recommendations for something that can quickly bootstrap a *nix server for Clojure/Script apps? I'm looking to host a bunch of different apps, each of which should have its own server, but I don't want to go through the pain of setting one up from scratch each time. I'm thinking something like Docker? Or even a script? Any recommendations will be much appreciated 🙂

beders04:01:09

Heroku is ideal for that. Even has some clojure examples

Alexis Vincent19:01:14

Docker would be good

Alexis Vincent19:01:27

makes it platform agnostic

Alexis Vincent19:01:28

Can run on a bare linux server, managed kubernetes, Google Cloud Run, anywhere really. And gives your the isolation you want

hiredman19:01:31

sure, and then hire 8-10 guys to run all that

hiredman19:01:51

and maybe a manager or two for them

hiredman19:01:44

if you build uberjars, java -jar whatever.jar is a great way to deploy and run stuff

hiredman19:01:44

I would even start off with a single repo and building a single uber jar and just run java -cp uberjar.jar clojure.main -m some.clojure.namespace to launch the different servers

Alexis Vincent19:01:20

@hiredman Respectfully, I think perhaps you haven’t seen what modern managed container offerings can do for you.

Alexis Vincent19:01:15

I’m not recommending @andyfry01 stands up and manages a kubernetes cluster.

hiredman19:01:53

it is also trivially simple to go from an uberjar to a docker container should you choose to

hiredman19:01:43

like, saying "use docker" doesn't actually answer questions about how to build and run apps, it just moves those questions into a container

Alexis Vincent19:01:27

OP’s question was “any recommendations for something that can quickly bootstrap a *nix server for Clojure/Script apps? I’m looking to host a bunch of different apps, each of which should have its own server, but I don’t want to go through the pain of setting one up from scratch each time.”

hiredman19:01:42

I am aware of managed container offerings, and have spent time (I won't call it wasted) getting them working. they do a lot of stuff, but to pretend they don't drag along a lot of complexity you can likely avoid if you are just starting is to be willfully blind

hiredman19:01:07

perhaps I misread

hiredman19:01:15

I took "server" to mean "service"

hiredman19:01:30

not literally devoting a machine vm or otherwise to each

Alexis Vincent19:01:09

:man-shrugging:

afry19:01:15

I guess what I'm looking for is something where I can define a single "standard" server configuration, and then use that to configure multiple separate servers

Alexis Vincent19:01:38

For simple use cases I would say, throw the jar into a container and deploy to google cloud run with minimum instance count of 1.

afry19:01:52

All of these servers will be running different apps, but they'll share a lot of stuff in common (like mongodb for instance), so I just want something to get the basics configured right away

Alexis Vincent19:01:58

gcloud run deploy --image http://gcr.io/cloudrun/hello --min-instances=1 helloservice

hiredman19:01:10

I think most people use something like ansible or chef for that kind of thing, which isn't really on topic for #clojure

Alexis Vincent19:01:38

replace http://gcr.io/cloudrun/hello with your container image and your have what you need.

Alexis Vincent20:01:00

without the 8-10 + 2 team @hiredman suggests.

afry20:01:16

Haha, not yet anyway 😛

afry20:01:19

Thanks guys, appreciate it!

Alexis Vincent20:01:03

🙂 Although @hiredman is right, this is off topic for #clojure

marrs20:01:42

Hi all, I'm trying to find a way to rerun a unit test from within my repl using [clojure.tools.namespace.repl/refresh. Running something like (do (refresh) (test-var (var my-ns/my-test))) yields a correct result every time I update and rerun the test, but if I try to put that code behind a function (e.g. (defn r [] (refresh) (test-var (var my-ns/my-test)))), it only binds the test the first time it is run. Subsequent calls to (r) yield the original result. Anyone know what's going on there?

hiredman20:01:41

Use resolve instead of the var special form

hiredman20:01:41

Your function is holding on to an interned var that has been removed by refresh, you need to re-reolve it everytime

marrs21:01:32

thx @hiredman. Just saw this message. That worked a treat

seancorfield20:01:04

(yet another reason not to use "refresh"-based workflows perhaps?)

marrs20:01:15

If the alternative is :reload-all then I'll stick with refresh. I'd actually like to get to the bottom of what's going on, because I don't see what would cause the two examples to behave differently

caumond20:01:07

@seancorfield how do you do in case of namespace refactoring so. I dont feel confortable with refresh ( lot of special cases) but dont know to do in another way.

seancorfield20:01:24

@caumond Can you elaborate on what you mean by "namespace refactoring"?

caumond20:01:11

I have cases when I want to re organize name space. So copy paste function and recompile function is easy. But I always have doubts, did I update all functions using the older ones.

didibus00:01:14

restart REPL 😛

didibus00:01:05

I do use refresh, but I also don't always trust it, so when I really want to be sure I eventually restart my REPL

didibus00:01:19

Or I run an AOT compile

seancorfield21:01:18

@caumond Using consistent aliases for namespaces helps a lot there (and linters like clj-kondo can help "enforce" that) so a function reference is always foo/bar everywhere (except uses in its own ns). You can also unmap the old function from the old ns by way of clean up. Generally, if I'm moving a public function from one ns to another, it tends to have a bunch of small private functions that it is implemented by, so as a "shortcut" I will sometimes remove-ns and then load-file on the old ns to clean things up a bit more -- for me that's ctrl-; r ctrl-; f -- and I always eval each top-level form as I change it anyway, and if I'm pasting multiple functions into a new ns I'll save and reload that ns: ctrl-s ctrl-; f. Moving functions around is one of the few cases where I will save/load rather than just eval'ing forms (and leaving files "dirty" -- unsaved).

metal 3
caumond21:01:58

I ll try this. Seems reasonably simple and efficient.

seancorfield21:01:29

The main thing is to try to get to the place where you have absolutely the minimum possible compilable change between each "eval" -- or turned around, that you always "eval" as often as you possibly can while making changes. Tightening that loop is what's key for a good REPL-based workflow.

caumond22:01:06

Yes. I succeed to do that for small changes. More challenging for big ones. But im on the way. Thx

didibus00:01:08

I mean, you just described (refresh)

didibus00:01:52

All it does is unmap namespaces and their vars, then reload them from source

seancorfield00:01:12

It's a sledgehammer. And it tries to rebuild "everything". Which is why it can break stuff.

seancorfield00:01:52

I occasionally (and surgically) remove and then load a single ns in the very rare situations where I need just that.

didibus00:01:01

I don't know, its pretty clear in what it does. Remove all namespaces that were modified since the last time you run refresh and then load them again in dependency order. Or, remove all namespaces and reload them in dependency order.

seancorfield00:01:26

I am very specifically advocating tight/narrow manual control over your REPL and eschewing "magic" that tries to do clever stuff / too much stuff.

seancorfield00:01:06

And I usually don't have tools.namespace as a dependency 🙂

didibus00:01:47

I agree, but for this particular use case, I think refresh-all is the best tool. You've moved things around, you want to know, wait... could any of my code be using something that only exists in the REPL but not in my source files? Well, to know that, you need to remove all loaded namespaces, and reload your source files.

didibus00:01:46

If I can agree on one thing, is I wish refresh was more stupid actually. The whole "trying to keep your state and clean up things" is where maybe it starts to mess up.

seancorfield00:01:02

I disagree. Automated refresh is lazy and "magic" and breaks unexpectedly for a lot of people -- witness all the beginners here on Slack who trip over problems with it. We should teach better hygiene instead of just calling in a mob of cleanup crew folks.

didibus00:01:23

I'd need to review that, I've never had it break unexpectedly on me, I suspect the only unexpected thing is that their new code is broken, but their old code worked

didibus00:01:48

Like as soon as you want to check that your REPL state doesn't contain things that's missing from your source files, you've entered a difficult place. Its pretty easy to not remember what you changed where, and lose track of what you should remove-ns and load again to be sure. That's when I'll use refresh, as a faster way than restarting the REPL.

seancorfield01:01:55

Like I said: good hygiene, evaluate every change, think carefully about your refactoring. If you're moving functions between namespaces, you're changing the "public API" of those namespaces, so that should already give you pause. I'm not saying "don't refactor" -- I'm saying be methodical, be careful, take the smallest steps possible, and keep your REPL in sync at every step (if possible). Needing to "refresh" a namespace should be rare and it is easy to do manually -- no library needed. Needing to "refresh everything" means you screwed up somewhere in your process and need a better process. Having bad work practices and just papering over them with a "nuclear" library is the wrong approach (IMO).

seancorfield01:01:06

(and also like I have said several times: my REPL runs for days, or weeks without restarts so... 🙂 )

didibus01:01:10

I think I agree with you mostly, except that, once you've decided to rename something, or move a function from one namespace to another. Keeping that hygiene is pretty hard. Should you try to avoid renaming and moving things around for fun, sure. But when you decide to do it, you do have an issue. What if you forgot to update a reference? Ok, clj-kondo does help a lot, but that's new stuff 😛 We didn't always have it before. Even then, clj-kondo won't work across namespaces. So it can't guarantee you didn't forget to update a reference.

seancorfield01:01:41

clj-kondo does work across namespaces because it reparses and caches information about each ns as you make changes.

didibus01:01:10

I kinda of wasn't sure, but I just tried it, and it did not

seancorfield01:01:30

When I've updated a function arity at the definition and I switch to another ns, it highlights the now-incorrect arity in calls.

didibus01:01:54

Arity maybe, let me try that. It didn't for rename

seancorfield01:01:50

Well, this has been fascinating in parts but I have cats to attend to so I'm off (and I am going to unsub from this thread as it seems to have run its course at this point).

didibus01:01:50

Also in general, it seems if I do:

(ns foo
  (:require [bar :as b]))

(b/this-does-not-exist-in-bar)
It doesn't highlight it

didibus01:01:12

Fair enough, have a nice evening

seancorfield21:01:26

I do have require .. :reload-all bound to a hot key (`ctrl-; R`) but almost never need to use that. And I don't use any sort of ns "refresh" ever in my workflow.

kwladyka21:01:25

(let [root-logger (.getLogger (LogManager/getLogManager) "")]
    (.setLevel root-logger Level/ALL)
    (doseq [handler (.getHandlers root-logger)]
      (.setLevel root-logger Level/ALL)))
(l/debug "foo")
I don’t see debug logs. Why? .setLevel works for all levels expect ALL / FINEST / FINER / FINE, but if I will set thing to Level/WARNING it works correctly and not show info. I will set this to Level/INFO it shows info. But it doesn’t work for lower level. What I miss?

kwladyka21:01:01

As I understand this it is actually JUL issue, not Clojure but still I don’t know why this is happening.

kwladyka21:01:17

Because

(let [root-logger (.getLogger (LogManager/getLogManager) "")]
    (.setLevel root-logger Level/ALL)
    (doseq [handler (.getHandlers root-logger)]
      (.setLevel root-logger Level/ALL))

    (.fine root-logger "fine"))
doesn’t work too

hiredman21:01:18

I would try calling .getLoggerNames on the log manager, and then iterating over each logger name and getting the logger for that name, then each handler for that logger, then setting the level on each handler

paul.legato21:01:32

I realize this doesn’t actually answer the question, but if I might suggest a lateral solution: I gave up on Java’s baroque logging system long ago. Now I use https://github.com/ptaoussanis/timbre

hiredman21:01:08

I just went through something like this with log4j2 which is not going to match jul, but it took a lot of iterating over a lot of things to set the level everywhere, and what you have to do is fairly sensitive to the logging config file if you have one

kwladyka21:01:05

Thanks @U0AFX5QNN but I did timbre -> clojure.tools.logging. Long story why but this is right thing 🙂

kwladyka21:01:43

@hiredman what is strange it works for INFO and up, but not below this level

kwladyka21:01:59

and I do iteration on handlers, all examples show only this

kwladyka21:01:06

but for sure I miss something

hiredman22:01:22

what does l/debug do?

kwladyka22:01:01

“nothing”

(l/debug "foo")
=> nil

hiredman22:01:16

ok, I mean, what is the definition of l/debug

kwladyka22:01:18

(let [root-logger (.getLogger (LogManager/getLogManager) "")]
    (.setLevel root-logger Level/ALL)
    (doseq [handler (.getHandlers root-logger)]
      (.setLevel root-logger Level/ALL))

    (.fine root-logger "fine"))
even this doesn’t work, but it does for .info

kwladyka22:01:50

so for sure it is the issue about JUL itself, not clojure.tools.logging

hiredman22:01:23

(do
  (doseq [ln (enumeration-seq (.getLoggerNames (java.util.logging.LogManager/getLogManager)))
          :let [logger (.getLogger (java.util.logging.LogManager/getLogManager) ln)]
          handler (.getHandlers logger)]
    (.setLevel logger java.util.logging.Level/ALL)
    (.setLevel handler java.util.logging.Level/ALL))
  (.fine
   (.getLogger (java.util.logging.LogManager/getLogManager) "")
   "fine"))
should work

hiredman22:01:46

what I see locally is there is a logger called "" and a logger called "global"

hiredman22:01:11

don't ask me why some levels and not other get logged or whatever

kwladyka22:01:14

thank you. I am still not sure why it works, but I will figure out this soon. I mean I am trying very similar things now and it doesn’t work. Interesting.

hiredman22:01:58

another thing is logger creation is stateful (I am sort of conflating log4j2 with jul here), so when you create a new logger (which tools.logging does whenever you execute a logging statement in a namespace that has had a logger created for it yet) it is initialized in someway (I don't entirely understand) to match the setting of the root logger, but then later if you change the settings of the root logger, it won't change the settings of that namespace logger

hiredman22:01:20

so in order to correctly configure everything you have to be careful of the order things are run in

kwladyka22:01:28

sh** I had a big “typo”

(let [root-logger (.getLogger (LogManager/getLogManager) "")]
    (.setLevel root-logger Level/ALL)

    (doseq [handler (.getHandlers root-logger)]
      (.setLevel handler Level/ALL))

    (.fine root-logger "fine"))
Before I was doing (.setLevel root-logger Level/ALL) twice (also for handlers…)

kwladyka22:01:37

You save my day

jjttjj23:01:08

is there a way to hack an already existing protocol to set :extend-via-metadata to true ?

ghadi23:01:56

record scratch hold up. what are you really trying to achieve?

ghadi23:01:43

(The hidden assumption here is that you don't own this protocol, otherwise you would simply edit your source file.)

jjttjj23:01:52

Nothing that's a good idea, just had a thought to use maps as core.async channels (the maps have inner channels and metadata). I suppose I could flip this and just extend IMeta to channels (which I know also isn't a good idea, this is more of a thought experiment)

ghadi23:01:54

when I've needed to associate channels with extra information, I have either used channels as a lookup key in a map {ch {...info about ch}} (when the chans are all known to a particular controller) or flow some info that includes the channel {:some :event :in ch} (which you can send... over a channel)

hiredman23:01:43

the best way to associate a channel with extra data is to just enrich the data coming out of it with that data: e.g. wrap the channel (defn wrap [ch data] (async/go [data (async/<! ch)]))

hiredman23:01:53

channels are not values so you'll need to extend the identity metadata stuff, which will cause you call kinds of problems if you are using timeouts (similar with sticking channels in maps or sets)

ghadi23:01:44

I would have a hard time reading code that used map+channel hybrids, though it is technically possible to make non-channels participate in channel protocols

hiredman23:01:20

wrapping timeouts is very nice because you can get a nice bit of data back (this is a timeout for this connection or whatever) instead of just nil when the timeout expires

ghadi00:01:17

the CML join function could be a protocol (that allowed you extend it to Process or CompletableFuture), and it would return you a channel.

ghadi00:01:34

returns you a readport, technically

ghadi00:01:32

CML is the Simpsons of concurrency, they did it all

💯 3
hiredman00:01:47

I dunno if you've seen https://clojure.atlassian.net/browse/ASYNC-234 (which relates to ASYNC-225) but it turns out in some circumstances alts! can leak memory when used with something like a timeout channel, because there is a global reference to the timeout channel, and a callback that closes over data may be waiting on that channel even after an alts! selected a different channel

hiredman00:01:07

so you actually need nacks to not leak memory

ghadi00:01:22

nice writeup

hiredman00:01:01

it was a fun little puzzle. I was chatting with someone about ASYNC-225 which led me to the conclusion the leak must exist, and then a lot of digging trying to figure out if I didn't see a leak in a simple test case because I was wrong or because there was something else confounding things

jjttjj23:01:46

gotcha yeah all that makes sense, thanks! I think the wrapping will work well