Fork me on GitHub
#clojure
<
2021-06-04
>
Arjaz08:06:13

Hi, does anyone know if and how I can extract a list of all required namespaces from a given namespace programmatically in Clojure?

Arjaz08:06:29

Something like the following:

(ns-requires *ns*) ;; => [clojure.set clojure.string]

borkdude08:06:57

@U0220KPQ5CH a namespace is either already loaded or it is not, but it doesn't matter which namespace did this, so this information isn't saved anywhere I believe. you can inspect the aliases that are valid within a namespace

borkdude08:06:10

This can give you some info:

user=> (binding [clojure.core/*loading-verbosely* true] (require '[clojure.set] :reload))
(clojure.core/load "/clojure/set")
nil

Arjaz08:06:56

Thanks for the help. I'm trying to solve a problem where I want to test if the given namespace loads the proper list, so I can have lein test signal if something weird is going on. We're using a bit of dynamic namespace parsing, so it really matters for us which namespace loads what

borkdude08:06:55

for testing you could use with-redefs on require

borkdude08:06:51

user=> (with-redefs [require (fn [& args] (prn :req args))] (require 'clojure.set))
:req (clojure.set)
nil

thheller08:06:07

(ns-aliases (find-ns 'foo.bar))

borkdude08:06:49

@U05224H0W that's not sufficient here:

user=> (require '[clojure.set])
nil
user=> (ns-aliases *ns*)
{}

thheller08:06:38

true, it is not 100%. most of the time things will have :as aliases though so should be good enough

Arjaz08:06:17

Not for my case, unfortunately

borkdude08:06:35

What you can also do as a test is, require a namespace and test if after doing that, the other namespaces have been loaded?

thheller08:06:56

hmm yeah then it gets tricky, especially if there are some namespaces that are AOT compiled since those won't show up in regular require logs

borkdude08:06:41

unless you temporarily re-define require using with-redefs

thheller08:06:25

once it enters AOT loading it won't use require anymore so redef does nothing

Arjaz08:06:08

I think I can parse out the require list or go with the with-redefs solution

caumond08:06:29

Hi, following my previous post. I re-did evaluations with criterium. 16ns for the java method, vs 64ns for the native clojure one. Is there any reason why clojure does not implement rand-intthat way ? https://gist.github.com/a30660391c68255e65ff60970e706c22

nbardiuk08:06:34

Try to benchmark java version when you create seed each time. Notice that clojure version probably allocates it since it is not an input argument

caumond08:06:13

Seed is expected to be done once ! If I re-init seed each time, I'll have always the same result

Alex Miller (Clojure team)12:06:31

I don’t know why. Why does it matter?

caumond13:06:14

Sorry @U064X3EF3 I don't understand. Why the performance is important? the seed?

Alex Miller (Clojure team)13:06:13

What's the reason behind the whole thread?

caumond13:06:54

1. I feel like a newbie so I may have done something fundamentally wrong 2. If that implementation is quicker and equivalent (what I think), it may be an improvement, something like :

(def ^java.util.Random seed (java.util.Random.))

(defn rand-int-alternative
  "Returns a random integer between 0 (inclusive) and n (exclusive)."
  {:added "1.0"
   :static true}
  [n] (.nextInt seed n))

caumond13:06:13

If you need more context I am trying to port an operations research tool I built years ago in C++. That kind of algorithms are combinatorial optimization, with a huge number of iterations on int-array. I start with some benchmarking between java and clojure and find out the global execution time gap was due to that rand-intfunction. That kind of use case are demanding from a time execution prospective (billion of iterations are expected).

Alex Miller (Clojure team)13:06:38

thanks, that was the context I was looking for. I don't think anything is preventing you from using your alternative method. I don't know why it's implemented the way it is. I'll make a ticket out of it when I get a chance.

caumond14:06:58

yes, I agree this is not a critical issue. Thanks for your confirmation to use the 'alternative method'. It's all what I need.

borkdude08:06:46

@caumond consider using a gist for long code fragments

borkdude08:06:52

I think this would be a good question for http://ask.clojure.org as well

caumond08:06:06

ok, I'm just trying to learn what a gist is first, (;p

borkdude08:06:49

or pastebin :)

caumond08:06:12

I did it !

caumond09:06:53

And I followed the question to http://ask.clojure.org following your recommandation. Thx.

hanDerPeder09:06:50

Procrastination alert! I can implement this operation

[{:key :foo :value 1} {:key :bar :value 2}] => {:foo 1, :bar 2}
like this
(reduce #(assoc %1 (:key %2) (:value %2)) {} coll)
or maybe
(into {} (map (juxt :key :value) coll))
but not really happy with either. Any ideas?

delaguardo10:06:02

last one is good except it could be in transducer form:

(into {} (map (juxt :key :value)) coll)

☝️ 2
delaguardo10:06:14

what do you think is wrong with it?

hanDerPeder10:06:09

not sure. i think it has something todo with the fact this the expression is saying β€˜add this value under this key’, but in my head i’m thinking β€˜add as this key’. i suppose (juxt :key identity) would express that. thanks!

noisesmith17:06:17

(juxt :key idenity) creates a different output though

Michael Gardner17:06:54

whenever I find myself writing (juxt ... identity) I reach for Medley's index-by

roklenarcic10:06:43

I have a data reader #object defined. This works:

(def x #object[org.joda.time.DateTime 0x3ff80a66 "2021-07-04T09:54:54.937Z"])
but this doesn’t
(identity #object[org.joda.time.DateTime 0x3ff80a66 "2021-07-04T09:54:54.937Z"])
Why is that?

Alex Miller (Clojure team)12:06:58

What does #object read into ?

roklenarcic14:06:51

(defn joda-time-data-reader [[class-symbol _objid date-time-string :as value]]
  (if (= "org.joda.time.DateTime" (name class-symbol))
    (tf/parse date-time-string)
    (throw (ex-info "Unsupported #object instance" {:value value}))))
So this should be a jodatime instance

roklenarcic14:06:02

org.joda.time.Datetime

Alex Miller (Clojure team)14:06:14

in what way does it "not work" ?

roklenarcic14:06:54

When I put

(def x #object[org.joda.time.DateTime 0x3ff80a66 "2021-07-04T09:54:54.937Z"])
in namespace code and load the namespace in a REPL, the value of X is as expected. But if I do something like:
(defn y [] #object[org.joda.time.DateTime 0x3ff80a66 "2021-07-04T09:54:54.937Z"])
and load that namespace I get:
Syntax error compiling fn* at (src/.../plans/presentation.clj:324:1).
Can't embed object in code, maybe print-dup not defined: 2021-07-04T09:54:54.937Z

roklenarcic14:06:48

Similarly if I just put like (identity #object….) in namespace top-level and try to load it, same error

roklenarcic14:06:58

seems odd that def parses differently than other expressions

Alex Miller (Clojure team)14:06:39

right, so the data reader is reading into an object. that object can't be represented in bytecode as a value, so you can't compile the function like that. def is different case

Alex Miller (Clojure team)14:06:58

you can probably add a printer for this type of object though and get it to work

Alex Miller (Clojure team)14:06:13

you do that by extending the print-method and print-dup multimethods

Alex Miller (Clojure team)14:06:43

you then have roundtripping of read/print which are complementary operations

roklenarcic14:06:25

I see, thank you

Joshua Suskalo14:06:05

Also if you define a print operation for these values, then it would be preferable for the reader to not read #object, but rather read something like #joda/datetime

Joshua Suskalo14:06:28

This will prevent issues with other objects being read as date times.

rickmoynihan10:06:54

I’ve just type hinted the return value of a protocol function with a class. It’s a common pattern in our code to call this function inside a with-open. However it seems this is strictly speaking a breaking change, because consuming namespaces now need to import the class that the type hint relates to, is that right?!

rickmoynihan11:06:50

I know get compile errors at call sites that call (with-open [(myprotocol/open-the-thing ,,,)] ,,,)

Ian Fernandez13:06:24

I know about this article that advocates for passing system as a first argument for a protocol impl: https://puredanger.github.io/tech.puredanger.com/2014/01/03/clojure-dependency-injection/

Alex Miller (Clojure team)13:06:53

as the author, I think "advocates" is a strong description - this is an idea, it may be useful, or it may not :)

4
πŸ‘ 4
Ian Fernandez13:06:48

There's some other article that says this for functions that does IO?

Ian Fernandez13:06:48

To make your IO execution to not depend on global variables and to pass the dependency in a explicit way (a.k.a. as a first argument of a function)

Ian Fernandez13:06:12

I've been programming a lot in this way since I began on Clojure and "it just felt right", how can I explain that it gets your code better more than simply saying that It's easier to debug / It's easier to mock?

Andrew Lai14:06:47

To your question about how to explain the value proposition of passing arguments vs. depending on globals: It sounds like you're trying to convince colleagues about the benefits of pure functions. Unfortunately, you'll probably have to tailor your arguments to your audience so your pitch will resonate with their specific experiences. Some arguments you could you could be: parallelization of tests - without global state you can safely parallelize your tests. transparency - global state is an additional, silent, side-channel input. It doesn't come through the front door like other function arguments. So the developer who picks up the test/code needs to know all of the relevant context to figure out how the function will behave. If you use pure functions and you know that the behavior only depends on the arguments supplied to the function, I've found it's much easier to understand and easier for other developers to understand.

clojure-spin 2
borkdude13:06:13

@d.ian.b do you mean passing the component system as the first arg?

Ian Fernandez13:06:23

Yeah, some form as component system, can be a map of system stuff (as right now I'm using https://github.com/piotr-yuxuan/closeable-map to make the system component)

Ian Fernandez13:06:02

the idea is that instead of my-insertion-db receiving system it receives the datasource itself?

borkdude13:06:33

only what the function needs I guess

πŸ‘ 2
borkdude13:06:40

but this is just an opinion, do what you want

sheepy 2
marciol13:06:28

Just the public APIs generally needs the system-map, other private functions downstream generaly will get only what it needs

Ian Fernandez13:06:06

yeah, I generally agree on pass only what the function needs, but destructuring helps a lot with this on most part of the time πŸ˜„

borkdude13:06:04

(although I have to admit that this is quite a common pattern in the codebase I'm working on πŸ˜… )

marciol13:06:12

Ahhh, I guess that it happens because destructuring a map is so easy πŸ˜„

Ian Fernandez13:06:47

yeah, but right now I'm not using component itself, if I have to make IO to a database I can do something like

(defn my-insertion-db! [{::config/keys [datasource]} data]
  (log "some data!" (pr-str data))
  (->> data
       some-transformation
       (jdbc/execute! datasource))

Ian Fernandez13:06:02

and for testing I can do something like

(with-open [system (-> config make-dev-system!)]
  (my-insertion-db! system data))

borkdude13:06:06

it's basically the same thing, just a data structure with components

Ian Fernandez13:06:14

I think is more simple / clean to have just a map πŸ˜„

Ian Fernandez13:06:25

No function should take the entire system as an argument => I know that sometimes you would prefer to pass part of the system (the part that your IO needs to use to handle it) as the argument of your function

Ian Fernandez13:06:47

but destructuring seems to fit very well for these stuff

Ian Fernandez13:06:20

but the thing I don't like is to have the system as a global variable (people are arguing that we need to make functions that has global vars embedded and returns a function that does the IO, because is simple to mock using with-redefs)

βœ… 2
borkdude13:06:42

I agree, pass arguments in favor of globals

Ian Fernandez13:06:50

but there's an article talking about this on clojure?

Ian Fernandez13:06:05

I'm struggling to find this on clojure 😞, it has for Elixir though https://dashbit.co/blog/mocks-and-explicit-contracts

Ian Fernandez13:06:25

because polutes functions to pass around system for every stuff (I've said not every stuff, just IO stuff haha)

Ian Fernandez13:06:50

but there's an article talking about this on clojure?

marciol13:06:18

> but destructuring seems to fit very well for these stuff Exactly!

jmckitrick14:06:06

I'm trying to use Apache Tika to detect a file type. I'm getting some reflection errors I can't seem to figure out, and I haven't done much Java interop prior to this. Here's what I have:

jmckitrick14:06:08

(with-open [xin ( (get-in params [:uploaded_file :bytes]))
                            tika (Tika.)
                            ftype (org.apache.tika.Tika/.detect tika)]
                  (prn "tika ---->" ftype))

jmckitrick14:06:21

So the first complaint is 'reference to field close can't be resolved' and then 'reference to field detect on org.apache.tika.Tika can't be resolved'

jmckitrick14:06:47

I assume this is either an import issue or a namespace issue

borkdude14:06:10

(.detect tika)

jmckitrick14:06:13

That was my first attempt, and I got: 'No matching field found: close for class org.apache.tika.Tika'

jmckitrick14:06:54

I'm conflating 2 issues, probably because I tried a plain 'let' in place of 'with-open' to see if I could get past the 'close' error

borkdude14:06:57

this is because you have this object in your with-open

borkdude14:06:27

bindings in with-open are called with close at the end

jmckitrick14:06:23

Oh, silly me. A separate 'let'

jmckitrick14:06:22

And now it cannot resolve the 'detect'...

jmckitrick14:06:52

Which is why I tried the full namespace... let's try that again...

jmckitrick14:06:34

Hmm. So the 'close' is resolved, but not 'detect'

borkdude14:06:46

just try it one expression at a time

jmckitrick14:06:20

(.detect tika xin)

jmckitrick14:06:05

Thanks, @borkdude

πŸ‘ 2
dominicm15:06:10

Does anyone know a file-system backed TTL cache? I'm communicating with an API with limits with a process that will stop/start on my machine. I've not found a java thing surprisingly!

rickmoynihan15:06:26

@dominicm Do you mean for http caching, etags etc?

dominicm15:06:33

@rickmoynihan I can hit an api 20 times a day, and I only want to do that every hour to make sure I don't deplete that. I want to make sure while I'm developing I don't deplete it either by restarting my jvm a few times.

rickmoynihan15:06:56

There are some other similar projects β€œvcr” will probably help you mind them.

dominicm15:06:52

@rickmoynihan vcr is more about testing. I am looking for something that expires after 1 hour, and to fetch a new one when that happens.

dominicm15:06:15

A filesystem would be great, this project will have no other dbs or anything.

borkdude15:06:43

postgres with a timestamp column?

borkdude15:06:05

or sqlite for that matter

rickmoynihan15:06:17

could you use varnish as a proxy?

dominicm15:06:25

@borkdude I mean, if I'm gonna roll my own I'll probably just use duragent and the local-file thing πŸ˜›

borkdude15:06:45

yeah, it doesn't seem that much work to be honest

dominicm15:06:31

Feels like one of those obvious things though, I thought it would already exist.

Adam Helins15:06:20

I am not sure I understand why it works: ns A has a macro that outputs a call to another macro from ns B ; ns C calls that macro but does not require ns B ; all is cool

(defmacro macro-a [x] `(B/macro-b ~x))

delaguardo15:06:34

when macro-a expands it will make the symbol B/macro-b fully qualified

delaguardo15:06:11

and if the namespace with that symbol is loaded by any other namespace it will be resolved correctly

Adam Helins15:06:14

Right, I might have been confused on that last point in my CLJC project. Indeed, ns B was being loaded in Clojure but not in CLJS, so it was misbehaving in CLJS. Make sense!

borkdude15:06:26

just use macroexpand to see what happens?

jjttjj16:06:44

Just being lazy instead of translating this but anyone know of a clojure bigdec natural log implementation in the wild, like this java one https://stackoverflow.com/questions/739532/logarithm-of-a-bigdecimal

dpsutton16:06:52

this relation seems really handy: log(AB) = log(A) + log(B). This might get into things like numerical stability and errors related to that

quoll17:06:44

That relation is how slide rules work

Nom Nom Mousse18:06:54

Is there a way to have the std{out,err} from shell/sh sent somewhere in real-time? Now I only get :out and :err info after the job is done πŸ™‚

andy.fingerhut18:06:05

I believe Java ProcessBuilder class can enable you to do that. I don't think that the built-in-to-Clojure shell/sh can do that.

noisesmith18:06:51

yeah I have a snippet doing that with ProcessBuilder:

(let [pb (ProcessBuilder. ["lua"])
      p (.start pb)
      out (.getOutputStream p)
      in (.getInputStream p)]
  (spit out "print \"hello from lua\"\n")
  (slurp in))
very straightforward, compared to most java apis

noisesmith18:06:29

of course for real interactivity you have to use something other than slurp / spit (and that means more complexity) - I'm sure there's some good libs that simplify that streaming IO stuff out there

noisesmith18:06:48

sounds like you want the stdout / stderr for monitoring? you should be able to .read from the output stream and error stream in a loop and trigger a callback for each line / chunk / whatever

noisesmith18:06:12

you can also use processbuilder to stream to a file or pipe instead of a stream, and to join the streams if that simplifies your use case as well

Nom Nom Mousse18:06:11

Thanks to both of you πŸ™‚

Nom Nom Mousse18:06:27

I am using my software to run multiple (shell) jobs concurrently and want output from them ASAP and continually so I can see if anything goes wrong. So yes, I guess monitoring is the word.

borkdude18:06:58

@endrebak Example with babashka.process (which you can use as a regular Clojure library):

(require '[babashka.process :refer [process]])
(process ["foo"] {:inherit true})
(process ["bar"] {:inherit true})
.. wait .. e.g. @(promise)

πŸ‘ 2
noisesmith18:06:29

they mentioned they don't want the output to go to stdio of the repl

borkdude18:06:52

> and want output from them ASAP and continually so I can see if anything goes wrong oh, I thought he/she/they did say that here

Nom Nom Mousse18:06:37

But if multiple jobs send to stdout concurrently they might overlap. Also, I want to send some to the front-end and so on so it would be best to send it to a tap or something πŸ™‚

borkdude18:06:07

just make a loop that reads the output line by line, prints it and does whatever else with it

πŸ™ 2
Nom Nom Mousse18:06:33

This is all new to me. Sounds like it is not that hard then. Thanks!

borkdude18:06:10

I have a similar example to that here: https://book.babashka.org/#child_processes

borkdude18:06:35

Here is a simple example that reads the output from yes:

(require '[babashka.process :as p :refer [process]]
         '[ :as io])

(def yes (process ["yes"] {:err :inherit
                           :shutdown p/destroy}))

(with-open [rdr (io/reader (:out yes))]
  (binding [*in* rdr]
    (loop []
      (let [line (read-line)]
        (println :line line))
      (recur))))

Endre Bakken Stovner14:06:18

@borkdude I guess there should be a (when (not (nil? line)) (recur)) above?

Endre Bakken Stovner14:06:01

I tried it and it just keeps on printing :line nil forever πŸ™‚

borkdude15:06:12

@UT770EY2K you are correct, but the output from yes never ends :)

borkdude18:06:37

@endrebak With babashka.process (https://github.com/babashka/process) this is very easy:

@(babashka.process/process ["ls"] {:inherit true})

noisesmith18:06:46

and vanilla clojure / interop version of the same:

(doto (ProcessBuilder. ["ls"]) (.inheritIO) (.start))

πŸ™ 2
Nom Nom Mousse18:06:50

Cool! I'll have to look more into babashka. But let's say I have multiple processes running at the same time in futures. Can each of them send data to a tap or something with inherit? So I get messages like:

jobid1 :stdout "message from job1"
jobid3 :stderr "this job failed"
... 
I will have to play around with :inherit to understand how it works.

noisesmith18:06:22

inherit just sends to *out* and *err*- see my reply in the other thread for using the streams with a callback

borkdude18:06:42

inherit doesn't send to *out*, it sends to System/out

noisesmith18:06:05

oh, good catch

Felipe Cortez22:06:33

why does evaluating an empty anonymous function using the #() syntax yield an empty list? I expected... maybe nil?

user=> (#())
()
user=> ((fn []))
nil

dpsutton22:06:41

See what the reader returns. β€˜#()

Felipe22:06:33

I see! nice!

dpsutton22:06:49

Learned that one from noisesmith.

Joel22:06:27

is reducing a list of maps a bad idea? I'm trying to write a reducer function, but it seem problematic. I want string values to turn into a list of strings, list values to be a flattened list, but picking the right incantation (concat cons conj, etc.) seems challenging.

emccue22:06:52

@joel380 when in doubt, loop it out

emccue23:06:33

skip all them helpers and just write a big ol' loop and recur

dpsutton23:06:40

A transducer with a mapcat would work a treat on that