Fork me on GitHub
#clojure
<
2020-06-29
>
roklenarcic08:06:04

I’m looking at clojure.core.memoize and I am wondering, if multiple threads call memoized function with the same argument, does it guarantee that the underlying function is called once. It’s kinda hard to read what the thread safety guarantees are.

p-himik08:06:06

There are no such guarantees.

tianshu09:06:13

I'm trying to improve our log configuration, we are using timbre for logging. How can I route log to different appenders, should I write a custom middleware for dispatching or is there a built-in design for this case?

kwladyka09:06:40

We had a deep conversation about logging and to make it short we got to conclusion clojure.tools.logging it today better, than timbre. Unfortunately this thread expire in slack and I can’t show this conversation.

kwladyka09:06:09

another appender example for timbre

(timbre/merge-config!
    {:appenders
     {:sentry
      {:enabled? true
       :async? true
       :min-level :info
       :rate-limit nil
       :output-fn :inherit
       :fn (fn [{:keys [level ?err msg_ ?ns-str context]}]
             (let [error-message (some-> ?err (.getLocalizedMessage))
                   causes (some-> ?err (ex-data) :causes)]
               ;(println "error-message" error-message)
               ;(println "causes" causes)
               ;(println "context" context)
               ;(println "msg_" (pr-str (not-empty (force msg_))))
               (sentry/send-event (merge sentry-base
                                         {:level (get timbre->sentry-levels level)
                                          :fingerprint (when (or error-message (force msg_))
                                                         (->> (set [?ns-str error-message (force msg_)])
                                                              (clojure.set/union causes)
                                                              (remove #(= "" %))
                                                              (remove nil?)))
                                          :logger ?ns-str
                                          :extra context
                                          :message (not-empty (force msg_))
                                          :throwable ?err}))))}}})

tianshu09:06:16

hi, I want to configure to support specify one particular appender when logging, like specify a logger-name in logback, and that logger-name would dispatch to a few appenders. looks like timbre only have the concept of appenders, so I should filtering in :fn ?

kwladyka09:06:42

not sure what you mean

tianshu09:06:34

For example, I have three appenders, a, b, c , I want to able to specify which appender to use when logging

tianshu09:06:14

like (log/info :a "some log for a")

tianshu11:06:30

(defn filter-appender [{:keys [vargs config] :as data}]
  (if (keyword? (first vargs))
    (let [appender-id (first vargs)
          appender (get-in config [:appenders appender-id])]
      (if appender
        (assoc data :appenders {appender-id appender} :vargs (subvec vargs 1))
        (throw (ex-info "Appender is not available" {:appender-id appender-id}))))
    data))
At last, I wrote a middleware like this.

kwladyka11:06:17

it seems to be much different logic to what timbre offer

kwladyka11:06:22

But what is your intention?

kwladyka11:06:48

I think you can do all this things using appenders

kwladyka11:06:33

just make some logic inside them. For example throw with ex-info and check if there is :foo keys with value X.

kwladyka11:06:56

or maybe you need :ns-whitelist and :ns-blacklist

kwladyka11:06:09

or :min-level

kwladyka11:06:57

I guess you can always read all appenders configured in timbre to throw an exception if there is no appender loaded

tianshu12:06:04

To be short, I want to split my logs into different log files.

tianshu12:06:36

But the splitting can't be simply done via :ns-whitelist and :ns-blacklist. I'm working on a legacy project, and it use timbre everywhere. So I decide to stick with timbre instead of switching to logback + tools.logging. I know how to achieve this with logback.

Cas Shun10:06:10

in a deps.edn project with a git dependency, how can I access the readme of that project from code? (or can I?)

borkdude10:06:55

@auroraminor typically the README is not part of the source code (which is described in :paths of the lib's deps.edn and defaults to src), so you cannot access it from the JVM's classpath

borkdude10:06:23

However, you can probably find the lib in ~/.gitlibs including the README

kwladyka11:06:46

What tools / libraries do you use for API ci/cd tests which are run outside of the system so simulate real chain of clients requests to API endpoints. So real HTTP requests like on production. The job which QA do manually.

vemv13:06:11

https://www.pingdom.com/ maybe? Haven't used it against APIs, should be possible Pingdom itself has an api, so I can re-create setups as the bits rot

Cas Shun11:06:05

@borkdude what about in the scenario where it is in a code directory? (out of curiosity)

borkdude11:06:46

@auroraminor Then you can get it using (slurp (io/resource "README.md"))

👍 3
Alex Miller (Clojure team)13:06:45

presuming only that some other lib does not define that resource, in which case you may or may not get the one you want (depends on classpath order)

borkdude13:06:00

yes, if a lib wants to expose that it's probably better to do it under an org+lib-named directory to avoid conflicts

Alex Miller (Clojure team)13:06:31

yup, people sometimes don't think about resource location enough

borkdude11:06:36

@kwladyka clojure.test + clj-http or similar

kwladyka11:06:54

So you make separate project which you run after deploy?

kwladyka11:06:00

ok, this is the way which I want to do, but I am curious if current IT world offer some tools which satisfy needs which I even don’t know about haha 😉

borkdude11:06:16

keep it simple 😉

kwladyka11:06:43

sure, it is more research at that moment to see how others do that

jumar13:06:53

I think Apiary offered something in this area but it’s been a long time since I looked at it

jumar13:06:19

I’d probably write my own tests using clj-http but if your system has first class API with potentially many external consumers it might be useful to look at alternatives

gleisonsilva13:06:43

Hello, guys! I'm using Depstar to package an app. This app has a resource file under resource folder. I've configured a alias with 'extra-path' to tell Depstar to include that folder. When depstar builds a uberjar, the resource/ content has been included on root of the .jar file instead of preserving the folder structure. There is some way to change that?

ghadi13:06:02

if you include resource as a path, and resource/foo/a exists, you should have foo/a in your zip

Sean Poulter15:06:12

Hi folks, is there a way to read the forms from source code that handles aliased qualified keywords? I’ve tried with clojure.edn/read and clojure.code/read and get the same error. Thanks!

Execution error at: tool.core/-main ...
Invalid token: ::lib/key
src/example/input.clj
(ns example.core
  (:require
   [example.lib :as lib]))

(defn fn
  [m]
  (assoc m ::lib/key 'value))
src/example/core
(ns example.core
  (:import
   [ PushbackReader])
  (:require
   [clojure.edn :as edn]
   [ :as io]
   [clojure.pprint]))

(defn -main
  []
  (let [source "./src/example/inut.clj"]
    (with-open [r (-> source io/reader PushbackReader.)]
      (clojure.pprint/pprint
       (loop [forms []]
         (let [form (edn/read r)]
           (if form
             (recur (conj forms form))
             forms)))))))

delaguardo15:06:29

edn/read will read the source that is following edn specification. ::lib/key is not accepted https://github.com/edn-format/edn#keywords

noisesmith15:06:01

aliases inside clojure.edn is a strange combo - if it worked would you expect it to use the aliases of your current ns? a parameterizable one?

noisesmith15:06:11

and as others mentioned, it doesn't work currently

👍 3
delaguardo15:06:12

https://github.com/borkdude/edamame there is configurable edn/clojure parser that should handle ::lib/key

👍 3
Sean Poulter15:06:36

Thanks @U04V4KLKC. I’ll give that a look.

Alex Miller (Clojure team)15:06:28

auto-aliased keywords (the :: part) are not a feature of edn so edn readers won't handle it

Alex Miller (Clojure team)15:06:04

the :: requires resolution in the "current" namespace, but edn does not have this concept

Alex Miller (Clojure team)15:06:36

the Clojure reader, however, does support this, and even supports a pluggable namespace resolver

Sean Poulter15:06:45

Thanks Alex. Yes, I found a few issues and one of your posts on Google Groups about that. I pivoted and tried using clojure.core/read with read-eval off too.

noisesmith15:06:51

of course the simple solution is to fully type out the namespace instead of using :: - there's little downside

Sean Poulter15:06:23

If it were a trivial example, I would. It’d involved a lot of code changes. 😕

Alex Miller (Clojure team)15:06:01

you can specify a custom resolver using reader-resolver

👍 3
Sean Poulter15:06:45

How’d you find that Alex?

Sean Poulter15:06:58

Living in Clojure? 😁

Alex Miller (Clojure team)15:06:57

I have used it once or twice

Sean Poulter15:06:42

Great memory! Thank you heaps, I’ll check that out.

Sean Poulter15:06:18

Also, as a fanboy moment, thanks for Clojure Applied too. That was well structured and helped hit the ground running. 🤓

borkdude15:06:41

user=> (doc *reader-resolver*)
-------------------------
clojure.core/*reader-resolver*
nil

Sean Poulter15:06:36

Are you spying on my REPL? I'm flattered @borkdude. 😁

borkdude15:06:03

Didn't you know clj-kondo sends all the code you lint to my servers?

borkdude15:06:21

including REPL sessions

😳 3
😆 3
borkdude15:06:35

(let [resolver (reify clojure.lang.LispReader$Resolver
                 (resolveAlias [_ sym]
                   (case sym foo 'foobar)))]
  (binding [*reader-resolver* resolver]
    (read-string "::foo/bar"))) ;;=> :foobar/bar

👍 6
borkdude16:06:05

In a Clojure parser I made as a lib, this can be done like this:

user=> (edamame/parse-string "::foo/foo" {:auto-resolve {'foo 'foobar}})
:foobar/foo

Sean Poulter16:06:33

Nice, that’s a lovely API. 👏

Sean Poulter15:06:17

I’ve been trying to parse the AST on a ClojureScript project to track down how/when/where we use some third-party dependencies.

Sean Poulter15:06:13

The macro expansion has been giving me some grief. Coming from JavaScript, it’s an interesting twist. The AST is expanded. 😆

Sean Poulter15:06:29

I’ve naively assumed it’d be quick to read the forms and find out where we’ve used our def-ui-component macro. That’s got those darned ::lib/key keywords.

dpsutton15:06:53

clj-kondo can be used for analysis and will give you a list of var references which might be easier to use than a tree

noisesmith15:06:13

but ::lib/key isn't a var

noisesmith15:06:27

or does it also do token tracking?

borkdude15:06:55

no, it doesn't track keyword usage, but it does track namespace usage

borkdude15:06:09

not sure if keywords are included for that though, maybe not

noisesmith15:06:37

the root issue is :: resolution, so knowing about namespace aliases could be part of a solution

👍 3
borkdude15:06:00

It does report this:

:namespace-usages [{:filename "<stdin>", :row 1, :col 21, :from dude, :to foobar, :alias foo}]
for (ns dude (:require [foobar :as foo]))

💯 3
👍 3
borkdude15:06:35

(let [resolver (reify clojure.lang.LispReader$Resolver
                 (resolveAlias [_ sym]
                   (case sym foo 'foobar)))]
  (binding [*reader-resolver* resolver]
    (read-string "::foo/bar"))) ;;=> :foobar/bar

👍 6
Sean Poulter16:06:18

That works! Neat.

noisesmith16:06:30

for most projects, you could even hard-code ::foo to always be foobar

noisesmith16:06:44

(and if not, the change that makes that work is a positive one and easy to do)

borkdude16:06:00

not sure if relevant, but: clj-kondo has a consistent-alias linter that will complain if you use different aliases for different libraries (you need to configure it for each lib that you want to have this check for)

👍 6
vemv21:06:50

It also could be at least partially automatic, right? i.e. one can query all aliases in a given project, and verify that there's a 1:1 relationship between libs and aliases (and not N:1, nor 1:N)

vemv21:06:51

just throwing an idea btw, this also can be implemented quite trivially without clj-kondo (i.e. tools.reader)

borkdude21:06:26

yes, you can also use clj-kondo's own analysis output for this

👍 3
fabrao22:06:21

is there any lib to generate random number with 20 digits?

lukasz22:06:39

rand-int ?

lukasz22:06:57

ah, sorry - 20 digits. JDK has secure random generators

noisesmith22:06:02

rand-int can only go up to Integer/MAX_VALUE

fabrao22:06:56

SecureRandom 
?

noisesmith22:06:00

yeah, you could use that, or call rand-int twice and put one call into a bigdecimal and multiply by Long/MAX_VALUE, taking the sum

noisesmith22:06:12

or - more precisely

ins)user=> (defn random-128bit [] (let [input-bytes (byte-array (repeatedly 16 #(rand-int Byte/MAX_VALUE)))] (java.math.BigInteger. input-bytes)))
#'user/random-128bit
(cmd)user=> (random-128bit)
86592240965117731739361063402086081396
(cmd)user=> (random-128bit)
89598420793514743141017230696508229384
(cmd)user=> (random-128bit)
129440819857997330997587888642583312139
(cmd)user=> (random-128bit)
149170186707650034165883028019004466810
(cmd)user=> (random-128bit)
116190502875785180875553570442774846270
you could easily parameterize this for how many bytes you want the value range to be • edit to fix a bug

phronmophobic22:06:12

wouldn't this be missing values since the range of byte is from Byte/MIN_VALUE to Byte/MAX_VALUE?

noisesmith22:06:02

oh! you are right, I forgot that rand-int was positive only

phronmophobic22:06:34

right. so I think it would never generate a number like 511 (java.math.BigInteger. (byte-array [1 -1]))

noisesmith22:06:17

yeah, lemme see if I can do this via an unchecked bit shift and bitwise and...

noisesmith22:06:35

oh, just a shift needed

phronmophobic22:06:36

unsigned numbers are a pain in java

noisesmith23:06:37

thanks for catching that issue, I think I have a better solution coming

(cmd)user=> (def frq (frequencies (repeatedly 10000 random-byte)))
#'user/frq
(cmd)user=> (into #{} (map val) frq)
#{59 20 27 39 46 54 48 50 31 32 40 33 22 36 41 43 29 44 28 51 25 34 23 47 35 45 53 26 38 30 52 42 37 49}
(cmd)user=> (apply min (keys frq))
-128
(ins)user=> (apply max (keys frq))
127
user=>

noisesmith23:06:57

(import (java.math BigInteger))

(defn random-byte
  []
  (-> Byte/MIN_VALUE
      (* -2)
      (rand-int)
      (+ Byte/MIN_VALUE)))

(defn random-bigint
  [byte-count]
  (let [byte-list (repeatedly byte-count random-byte)]
    (BigInteger. (byte-array byte-list))))

phronmophobic23:06:16

you may need to mask the very first bit to make sure the result isn't negative

noisesmith23:06:30

I want a negative result

noisesmith23:06:43

if only positive was OK, that would be much easier

phronmophobic23:06:54

I thought random-bigint was supposed to have a positive result?

phronmophobic23:06:16

I mean the very first bit of the very first byte of the byte-list

noisesmith23:06:17

oh - that part

noisesmith23:06:25

cool, good idea

phronmophobic23:06:03

or you could just call .abs on the big integer

noisesmith23:06:51

I think I like this version better, easy enough to add a .abs call if the domain requires it

(import (java.math BigInteger))

(defn signed-rand
  [x]
  (let [X (- (Math/abs (long x)))]
    (-> X
        (* -2)
        (rand-int)
        (+ X))))

(defn random-bigint
  [byte-count]
  (let [byte-list (repeatedly byte-count
                              #(signed-rand Byte/MIN_VALUE))]
    (BigInteger. (byte-array byte-list))))

phronmophobic23:06:33

:thumbsup: . as long as it's clear that the range includes the negative side too

noisesmith22:06:26

just divide the number of bits required by 8 of course

darwin22:06:15

never roll your own crypto, use a reputable library call which gives you what you want in one call

3
noisesmith22:06:15

oh yeah, of course, rand-int isn't a security feature and please don't pretend it is

dpsutton22:06:21

agree. but not clear this is crypto related

Cameron Kingsbury23:06:06

trying to use a java library and running into the following:

(let [client (new OkHttpClient)
      request (doto (new Request$Builder)
                (.url "")
                (.get)
                (.addHeader "x-rapidapi-host" "")
                (.addHeader "x-rapidapi-key" "")
                (.build))]
  (doto (.newCall client request)
    (.execute)))
gives
class com.squareup.okhttp.Request$Builder cannot be cast to class
   com.squareup.okhttp.Request (com.squareup.okhttp.Request$Builder and
   com.squareup.okhttp.Request are in unnamed module of loader 'app')

hiredman23:06:32

doto returns its first argument

hiredman23:06:00

so you are throwing away the request built by the call to build and returning the builder

hiredman23:06:19

you likely want -> not doto

Cameron Kingsbury23:06:58

thanks seems right!

noisesmith23:06:10

.. might work here too since every step is a method call

noisesmith23:06:04

something like (.. (Request$Builder.) (url ...) (get) ...)

Cameron Kingsbury23:06:35

now that I've done this... I question why I used Java at all xD thanks anyway, will be useful later