Fork me on GitHub
Calum Boal00:12:58

Having some issues implementing something using core.async. What I want is to use go blocks to do a bunch of HTTP requests and process the responses. I'm approaching this coming from golang which may be where my issues are originating. The way i'm approaching this is to have a pool of go blocks trying to read from a channel which is feeding them urls, and then returning results over another channel. The issue im having is that I can't ensure all go blocks are finished working.


don't use while true


loop only when you take a non-nil from urls-chan


close urls-chan when you are done


in general using go blocks as workers is not useful


since they run on a limited size threadpool (usually capped at 8 threads)


something like async/pipeline-blocking is likely much more of what you want

Calum Boal00:12:42

Hmm yeah potentially, in go you would use go routines even though they are also capped by physical threads, but there's a runtime for switching while they're sleeping

Calum Boal00:12:48

Figured it was the same here

Calum Boal00:12:54

Managed to get this now:

(defn worker [urls-chan results-chan]
  (async/go (loop []
              (let [url (async/<! urls-chan)]
                (if (not (nil? url))
                    (let [result (is-vulnerable? url)]
                      (println url)
                      (if (and (not (nil? result)) (not (empty? result)))
                        (async/>! results-chan (string/join "\n" result)))

(defn printer [results-chan]
  (async/go (loop []
              (let [result (async/<! results-chan)]
                (if (not (nil? result))
                    (println result)
(defn -main
  [& args]
  (let [urls-chan (async/chan 100)
        results-chan (async/chan)
        worker-chans (vec [])
        printer-chans (vec [])]
    (dotimes [_ 10]
      (conj worker-chans (worker urls-chan results-chan)))
    (conj printer-chans (printer results-chan))
    (doseq [ln (line-seq ( *in*))]
      (async/>!! urls-chan ln))
    (async/close! urls-chan)
    (mapv #(async/<!! %) worker-chans)
    (async/close! results-chan)
    (mapv #(async/<!! %) printer-chans)))

Calum Boal00:12:15

It's not working though haha, it was almost working a min ago


my initial suspicion is you are doing some blocking operations in go blocks, which is gumming up the threadpool


no, the go macro is just a macro


it is a syntax transformation


it can't turn your blocking io (or blocking on promises, locks, etc) in to not blocking

Calum Boal01:12:57

Current issue is still that it's just exiting before doing anything 😞

Calum Boal01:12:38

Doesn't even take a single value from url-chan in the worker


@U01D6KT2PJM what does is-vulnerable? return? A boolean? When I tested it I was getting an error from calling empty? on result with result being a boolean

Calum Boal01:12:46

Think i've figured out the issue, just way too tired haha

Calum Boal01:12:59

I can post the full code if you like, but nah it was a boolean now it returns a string


btw if you don't have it already it might help to have an uncaught exception handler, like this:

;; Assuming require [ :as log]
 (reify Thread$UncaughtExceptionHandler
   (uncaughtException [_ thread ex]
     (log/error ex "Uncaught exception on" (.getName thread)))))
because that's the only way I was seeing the error

Calum Boal01:12:32

(ns the-great-escape.core
   [clj-http.client :as client]
   [hickory.core :as html]
   [ :as s]
   [clojure.core.async :as async]
   [clojure.string :as string]
   [lambdaisland.uri :refer [uri]])

(defn doc->hickory [doc]
  (-> (html/parse doc)

(defn fetch-and-parse-page [url]
  (-> (client/get url
                  {:cookie-policy :none
                   :redirect-strategy :none})

(defn extract-asset-tags [body]
  (s/select (s/or
             (s/attr :href)
             (s/attr :src))

(defn extract-href-urls [tags]
  (map #(get-in %1 [:attrs :href]) tags))

(defn extract-src-urls [tags]
  (map #(get-in %1 [:attrs :src]) tags))

(defn extract-asset-urls [tags]
  (->> (extract-href-urls tags)
       (filter #(not (nil? %1)))))

(defn parse-urls [urls]
  (map uri urls))

(defn local-asset? [base-domain {:keys [host]}]
  (or (nil? host) (= base-domain host)))

(defn filter-local-paths [base-domain parsed-urls]
    (->> (filter (partial local-asset? base-domain) parsed-urls)
         (map :path)
         (filter #(not (nil? %1)))

(defn url->parts-map [url]
  (let [parts (string/split url #"/")]
    {:dir (str "/" (second parts))
     :path (str "/" (string/join "/" (drop 2 parts)))}))

(defn urls->parts-maps [urls]
  (map url->parts-map urls))
(defn get-unique-dirs [parts-map]
  (map #(first (val %1)) (group-by :dir parts-map)))

(defn filter-bad-traversals [parts-map]
  (filter #(and (not= (:dir %1) "/") (not= (:path %1) "/")) parts-map))

(defn build-original-url [base-url parts-map]
  (str (assoc base-url :path (str (:dir parts-map) (:path parts-map)))))

(defn build-traversal-url [base-url parts-map]
  (str (assoc base-url :path (str (:dir parts-map) ".." (:dir parts-map) (:path parts-map)))))

(defn request-and-hash [url]
  (-> (client/get url {:redirect-strategy :none
                       :cookie-policy :none})

(defn doc->candidates [base-domain doc]
  (->> doc
       (filter-local-paths base-domain)

(defn traversal-comparison [base-url candidate]
    (let [baseline (request-and-hash (build-original-url base-url candidate))
          traversal (request-and-hash (build-traversal-url base-url candidate))
          blank (hash "")]
      (if (not= baseline blank)
        (if (= baseline traversal)
          (build-traversal-url base-url candidate))))
    (catch Exception _

(defn is-vulnerable? [url]
  (let [url (uri url)
        doc (fetch-and-parse-page (str url))
        candidates (doc->candidates (:host url) doc)]
    (filter #(not (nil? %1)) (map #(traversal-comparison url %1) candidates)))
  (catch Exception _

(defn stdin-urls []
  (doseq [ln (line-seq ( *in*))]
    (println ln)))

(defn worker [urls-chan results-chan]
  (async/go (loop []
              (let [url (async/<! urls-chan)]
                (if (some? url)
                    (let [result (is-vulnerable? url)]
                      (if (and (not (nil? result)) (not (empty? result)))
                        (async/>! results-chan (string/join "\n" result)))

(defn printer [results-chan]
  (async/go (loop []
              (let [result (async/<! results-chan)]
                (if (some? result)
                    (println result)

(defn spawn-workers [urls-chan results-chan n]
  (loop [workers []
         x 0]
    (if (= x n)
      (identity workers)
      (recur (conj workers (worker urls-chan results-chan)) (inc x)))))

(defn -main
  [& args]
  (let [urls-chan (async/chan 100)
        results-chan (async/chan 100)
        worker-chans (spawn-workers urls-chan results-chan 10)
        printer-chans (into [] (printer results-chan))]
    (doseq [ln (line-seq ( *in*))]
      (async/>!! urls-chan ln))
    (async/close! urls-chan)
    (mapv #(async/<!! %) worker-chans)
    (async/close! results-chan)
    (mapv #(async/<!! %) printer-chans)))

Calum Boal01:12:53

urls over stdin

Calum Boal01:12:04

Don't worry about it though, think i'm close to figuring it out

Calum Boal01:12:08

(defn worker [urls-chan results-chan]
  (async/go (loop []
              (let [url (async/<! urls-chan)]
                (if (some? url)
                    (let [result (is-vulnerable? url)]
                      (if (and (not (nil? result)) (not (empty? result)))
                        (async/>! results-chan (string/join "\n" result)))

(defn printer [results-chan]
  (async/go (loop []
              (let [result (async/<! results-chan)]
                (if (some? result)
                    (println result)

(defn spawn-workers [urls-chan results-chan n]
  (loop [workers []
         x 0]
    (if (= x n)
      (identity workers)
      (recur (conj workers (worker urls-chan results-chan)) (inc x)))))

(defn -main
  [& args]
  (let [urls-chan (async/chan 100)
        results-chan (async/chan 100)
        worker-chans (spawn-workers urls-chan results-chan 10)
        printer-chans (conj [] (printer results-chan))]
    (doseq [ln (line-seq ( *in*))]
      (async/>!! urls-chan ln))
    (async/close! urls-chan)
    (mapv #(async/<!! %) worker-chans)
    (async/close! results-chan)
    (mapv #(async/<!! %) printer-chans)))

Calum Boal00:12:03

(defn worker [urls-chan results-chan]
  (async/go (while true
              (let [url (async/<! urls-chan)
                    result (is-vulnerable? url)]
                (if (and (not (nil? result)) (not (empty? result)))
                  (async/>! results-chan (string/join "\n" result)))))))

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (let [urls-chan (async/chan 100)
        results-chan (async/chan 100)]
    (dotimes [_ 10]
      (worker urls-chan results-chan))
    (async/go (while true
          (println (async/<! results-chan))))
  (doseq [ln (line-seq ( *in*))]
    (async/>!! urls-chan ln))))


Anyone have an example of java.util.logging formatter in Clojure? The best to print custom JSON :)

hiredman00:12:24 is a mini logging library built on java.util.logging that logs clojure data

👍 6

thank you, I check this one!


on, the section about typehinting (in the weird characters guide) in the reader gives the following example for typehinting: (def ^Integer five 5). Yet all over Clojure's source code, ^long is used (although possibly in different contexts, typehinting locals and return types for functions). When I typehint a simple var like that if i use ^Long i get its tag as java.lang.Long and when typehinted as ^long i get #function[clojure.core/long]. Are these things semantically different? One obviously wrong?

Alex Miller (Clojure team)07:12:05

they are semantically different

Alex Miller (Clojure team)07:12:25

Java has 7 primitive types with lowercase letters - int, long, etc

Alex Miller (Clojure team)07:12:46

it also has object-based wrappers around those primitive types java.lang.Integer, java.lang.Long, etc

Alex Miller (Clojure team)07:12:23

in many places these will automatically converted if needed so that hides a lot of mismatches

Alex Miller (Clojure team)07:12:01

I'd say that particular example is confusing

Alex Miller (Clojure team)07:12:38

by default, Clojure will read integer numbers as java.lang.Long objects

Alex Miller (Clojure team)07:12:29

so ^Long there would probably make more sense, but this is case is probably not a good place to really need/want that


I think what really confused me was when i type hinted the var with ^long, I would get errors of > Syntax error (IllegalArgumentException) compiling . at (REPL:1:31). > Unable to resolve classname: [email protected] when the compiler was using that typehint.


and that made me want to find some proper documentation instead of just the kinda rule of thumb nature of my knowledge of type hinting


(def ^long year-threshold 20)
(def ^:private past-threshold (.. (LocalDateTime/now)
                                  (minus year-threshold ChronoUnit/YEARS)
                                  (toInstant ZoneOffset/UTC)
Syntax error (IllegalArgumentException) compiling . at (REPL:1:31).
Unable to resolve classname: [email protected]


can you type hint a def? I'm reading > They can be placed on function parameters, let-bound names, var names (when defined), and expressions but I'm not sure


or atleast the part discussing {:tag 'long}


ah ok, @UC506CB5W’s link seems to show that you can.

Alex Miller (Clojure team)07:12:48

you can type hint a def, you just have to be aware that type hints on the var (as in def, but also the function name in defn) are evaluated, so ^long will evaluate to the long function object, which is no good. defn type hints on the arg vector and in the arg vector are handled by defn macro and are not evaluated


ah ok. that's the secret sauce that i was missing in documentation. I didn't understand why inline typehints could be ^long but on a var like that had to be ^Long. I thought i remembered that the clojure.core/long function was identified with the `'long' and classname typehint, but that might be cljs or might just be me just completely making stuff up


thanks as always for your answers @U064X3EF3


I’m using the monger database, which is showing the results from the find function. It’s showing the results as empty even though I have items in my database. Here’s the code:

(ns humboiserver.db.core
   [monger.core :as mg]
   [monger.gridfs :as gfs]
   [monger.query :as q]
   [monger.collection :as mc]
   [cheshire.core :as json]

(let [{:keys [conn db]}
      (mg/connect-via-uri “:/<dbname>?retryWrites=true&w=majority”)]
  (defn find [coll query] (mc/find-maps db coll query))

Here’s the usage:
(db/find “featured” {})

And here’s the featured database: [![enter image description here][1]][1] [1]: Yet doing the find is returning an empty sequence. How to fix this?


Hello! Is there an equivalent to a tap function in clojure? Something that does this: (fn [f x] (f x) x) ?


It would be most useful for stuff like (tap println (do-some-stuff data)) and is a little easier to read and write than what I usually do: (let [x (do-some-stuff data)] (println x) x).


Oh! This is awesome! Thanks! @admin055

👍 3

Hi! I am not sure this question belongs here, so if you think there’s a better channel please tell me. Anyways, for all mac users, how do you switch jdk globally - not limited to current shell. I can’t seem to find a one script solution i.e. switch jdk 11 that sets the jdk globally.


I have set JAVA_HOME in ~/.zshrc.


I see but say you change JAVA_HOME in current shell, then when a new shell is opened it will default to whatever is in ~/.zshrc. To fix this I resorted to some emacs helpers that allow me to manage the jdk version emacs uses from within emacs itself.. but yeah ideally it would be great to be able to do this in a shell. Thanks for the response anyways 🙂


Maybe sdkman would help?


It kinda helps but from the docs it doesn’t seem you can change a default besides right after installing a new jdk through the prompt 🙂


Or is that shell only?


@lucio Check out jenv

☝️ 3

awesome it seems it does exactly what I need. Will try it out now.. @borkdude thanks!


@lucio I’ll vouch for jenv too.

👍 3

Java file foo/bar/ is Clojure

(def custom (proxy [SimpleFormatter] []
                (format [^LogRecord record]
                  (println record)
is? What is the “path” to custom? It looks it is not api.logs.mycustom. I am trying to figure out settings for java.util.logging java.util.logging.ConsoleHandler.formatter = api.logs.custom How should I prepare this class to get this “path”?


(import '(java.util.logging ConsoleHandler

(defonce ^Logger logger (doto (Logger/getLogger "clojure")
                          (.setUseParentHandlers false)
                           (doto (ConsoleHandler.)
                             (.setLevel Level/ALL)
                              (proxy [SimpleFormatter] []
                                (format [^LogRecord record]
                                  (let [sb (StringBuilder.)]
                                    (.append sb "#:log{")
                                    (.append sb ":z ")
                                    (.append sb (pr-str (str (java.time.Instant/ofEpochMilli (.getMillis record)))))
                                    ;; (.append sb " :b ")
                                    ;; (.append sb (format "%02d" (.getSequenceNumber record)))
                                    ;; (.append sb " :c ")
                                    ;; (.append sb (format "%02d" (.getThreadID record)))
                                    (.append sb " :v :")
                                    (.append sb (.toLowerCase (.getName (.getLevel record))))
                                    (.append sb " :n ")
                                    (.append sb (.getSourceClassName record))
                                    (.append sb " :l ")
                                    (.append sb (.getSourceMethodName record))
                                    (.append sb " :m ")
                                    (.append sb (pr-str (.getMessage record)))
                                    (doseq [p (seq (.getParameters record))
                                            :when (map? p)
                                            [k v] (seq p)]
                                      (doto sb
                                        (.append " ")
                                            (namespace k) (.append (pr-str k))
                                            (not (namespace k)) (-> (.append ":_/") (.append (name k))))
                                        (.append " ")
                                        (.append (pr-str v))))
                                    (when-let [t (.getThrown record)]
                                      (.append sb " :thrown \n")
                                      (.append sb (pr-str t)))
                                    (.append sb "}\n")
                                    (str sb)))))))))

(doseq [level '[severe warning info config fine finer finest]]
  (intern *ns*
            ([&form &env msg]
             (let [ns (name (ns-name *ns*))]
               `(.logp logger ~(symbol "java.util.logging.Level" (.toUpperCase (name level))) ~ns ~(str (:line (meta &form))) (print-str ~msg))))
            ([&form &env msg parameters]
             (let [ns (name (ns-name *ns*))]
               `(let [p# ~parameters]
                  (if (map? p#)
                    (.logp  logger
                            ~(symbol "java.util.logging.Level" (.toUpperCase (name level)))
                            ~(str (:line (meta &form)))
                            (print-str ~msg)
                    (.logp  logger
                            ~(symbol "java.util.logging.Level" (.toUpperCase (name level)))
                            ~(str (:line (meta &form)))
                            (print-str ~msg)
                            ^Throwable p#)))))))
  (.setMacro ^clojure.lang.Var (ns-resolve *ns* level)))


i modified that snippet so it uses the fully qualified java.util.logging.Level/ stuff so you can use it from other namespaces


Just in case you still need to access Clojure vars from Java:


@U11BV7MTK Do you know how to refer to this from configuration file for JUL? I know how to achieve this in pure Clojure / Java, but I wanted to use config file to point the formatter.


But I don’t know how to point to this heh


java.util.logging.ConsoleHandler.formatter = api.logs.mycustom <- exactly this line of config


no i was just playing around with it this morning. I think the point of it is that it uses JUL and ignores the config as it makes its own logger named clojure and its configured inline there


maybe I will end with this. I wanted to have only formatter in Clojure and set everything in config file


but I don’t know how to refer to the class made by proxy in clj file heh


you want to use this from java?


not sure what you mean


why are you writing a line like this: java.util.logging.ConsoleHandler.formatter = api.logs.mycustom ?


(def custom (proxy [SimpleFormatter] []
                (format [^LogRecord record]
                  (println record)
I want to use above here in java.util.logging.ConsoleHandler.formatter = api.logs.custom

kwladyka17:12:12 is config file for java.util.logging




dunno sorry


you can use config file by ""


so then I could only use Clojure for Formatter code


And keep everything native


i think this snippet is kinda intended to just be evaluated in a random namespace and used there almost transiently. its a quick in the repl thing and not really what you're looking for. and for your usecase i think you'd need to aot the file for it to work as that classfile needs to exist already (but i'm not sure about that)


BTW What / Why are you doing in (doseq [level '[severe warning info config fine finer finest]] ?


Why not just overwrite default root logger?


(let [root-logger (.getLogger (LogManager/getLogManager) "")
        formatter (proxy [SimpleFormatter] []
                    (format [^LogRecord record]
                      (println record)
        console-handler (doto (ConsoleHandler.)
                          ;(.setUseParentHandlers false)
                          (.setFormatter formatter))]

    (doseq [handler (.getHandlers root-logger)]
      (.removeHandler root-logger handler))

    (doto root-logger
      (.setLevel Level/INFO)
      (.addHandler console-handler)))
this is what I have so far


Do you understand when use Formatter vs SimpleFormatter?


(def ANSI-colours {:reset "\u001B[0m"
                   :black "\u001B[30m"
                   :red "\u001B[31m"
                   :green "\u001B[32m"
                   :yellow "\u001B[33m"
                   :blue "\u001B[34m"
                   :purple "\u001B[35m"
                   :cyan "\u001B[36m"
                   :white "\u001B[37m"})

(defn getCause [^Throwable thrown]
  (when-let [cause ^Throwable (.getCause thrown)]
    {:class (.getClass cause)
     :message (.getMessage cause)
     :stack-trace (clojure.string/join "\n" (map str (.getStackTrace cause)))}))

(defn getThrown [^LogRecord record]
  (when-let [thrown ^Throwable (.getThrown record)]
    (let [cause (getCause thrown)]
        {:class (.getClass thrown)
         :message (.getMessage thrown)
         :stack-trace (clojure.string/join "\n" (map str (.getStackTrace thrown)))}
        cause (assoc :cause cause)))))

(defn record->map [^LogRecord record]
  (let [thrown (getThrown record)
        level (.getLevel record)]
      {:severity (.getName level)
       :message (.getMessage record)
       :logger-name (.getLoggerName record)}
      (= Level/SEVERE level) (assoc "@type" "")
      thrown (assoc :thrown thrown))))

(def root-logger (let [root-logger (.getLogger (LogManager/getLogManager) "")
                       formatter (proxy [SimpleFormatter] []
                                   (^String format [^LogRecord record]
                                     (let [color (if (= Level/SEVERE (.getLevel record))
                                                   (:red ANSI-colours)
                                                   (:white ANSI-colours))]
                                       (str color
                                              (record->map record)
                                              #_(json/object-mapper {:pretty true}))
                                            (:reset ANSI-colours)
                                            (System/getProperty "line.separator")))))
                       console-handler (doto (ConsoleHandler.)
                                         ;(.setUseParentHandlers false)
                                         (.setFormatter formatter))]

                   (doseq [handler (.getHandlers root-logger)]
                     (.removeHandler root-logger handler))

                   (doto root-logger
                     (.setLevel Level/INFO)
                     (.addHandler console-handler))))
here I share with you my solution for jsonPayload for google cloud logging


I know this isn't a very scientific benchmark, but I get the same-ish numbers with just nth and a self inlined version (.nth ^clojure.lang.Indexed [0 1] 1) here:

user=> (time (dotimes [_ 100000000000] (.nth ^clojure.lang.Indexed [0 1] 1)))
"Elapsed time: 1937.860586 msecs"
user=> (time (dotimes [_ 100000000000] (nth [0 1] 1)))
"Elapsed time: 1901.539964 msecs"
which kind of surprised me since RT/nth does an extra step right? Maybe that step is just very very cheap or JIT-optimized.


just based on the implementation, I'm not sure how much extra it's doing:

(defn nth
  "Returns the value at the index. get returns nil if index out of
  bounds, nth throws an exception unless not-found is supplied.  nth
  also works for strings, Java arrays, regex Matchers and Lists, and,
  in O(n) time, for sequences."
  {:inline (fn  [c i & nf] `(. clojure.lang.RT (nth ~c ~i [email protected])))
   :inline-arities #{2 3}
   :added "1.0"}
  ([coll index] (. clojure.lang.RT (nth coll index)))
  ([coll index not-found] (. clojure.lang.RT (nth coll index not-found))))


how does it compare to the (. clojure.lang.RT (nth ~c ~i [email protected])) version?


I mean the RT#nth version is doing something extra


I don’t have time to dig into it but maybe ^clojure.lang.Indexed make a difference?


why do you think one version is doing something extra?


that kind of thing is, to make a crazily broad claim, basically free


Right. And in clojure itself - less free but maybe not so bad either?


(instance? ...) I mean


it is complicated, but very often exactly the same


an instance check compiles to a single jvm instruction in java, in clojure we have a function instance?, and calling that would be more expensive, but several features of clojure (definline, and compiler intrinsics) will usually turn it into a single instruction


I don't see any :inline stuff on instance? itself though


(the compiler recognizes some static methods and replaces them with bytecode directly instead of a method call)


right. clever :)


how does it compare to (.get ^List [0 1] 1) ?


getting something from a list is O(n) right


indexed is way faster


but it depends on what the List implementation is


ah, actually it isn't definline and the intrinsics stuff


in Clojure a list is a linked list, with O(n) access - but in Java it could also be an ArrayList which probably is indexed


the compiler special cases known calls to instance?


that explains why it's so blazing fast


if it sees a call where the first thing is a reference to the var clojure.core/instance? it does the single bytecode instruction (with maybe an extra instruction to box the boolean)


that must predate the more general pattern of using definline to emit a static method call, and the compiler having a mapping between certain static methods and the bytecode to emit instead


it is important to keep in mind instance? is not the same thing as satisfies?


yeah - satisfies? can be slow I've heard


reminds me to remove a call to satisfies? from a patch