Fork me on GitHub

Is there an API for tracking the dependencies of a namespace when loaded? That would include anything newly required and already required. I know of tns dependency tracking stuff, but that works on the source level.


on the namespace level, you have the refers and the aliases, which are created inside the call to require (usually), but they only reify the translation (of prefix to larger prefix, or symbol to prefixed symbol), not the requirement of the other ns


but, in normal code, there's usually a refer or alias created, and those can be found inside ns-refers and ns-aliases


oh, that's an interesting idea. Inspecting the ns afterwards for what it actually loaded...


Yeah, I guess without an alias though (which is common enough for what I am building) that's a problem.


I'd love to partiticpate in the loading system with hooks… yuk :D


Can someone explain to me why r/fold does not run in parallel across all available cores in this example:

(ns csv2summap.core
  (:require [ :as csv]
            [ :as io]
            [clojure.core.reducers :as r]))

(with-open [writer (io/writer "numbers.csv")]
   (take 1000000 (repeatedly #(vector (char (+ 65 (rand-int 26))) (rand-int 1000))))))

(defn sum-vals
([] {})
([m [k v]]
 (update m k (fnil + 0) (Integer/parseInt v))))

(defn merge-sums
([] {})
([& m] (apply merge-with + m)))

;; r/fold version, should run in parallel on all cores, but does not
(with-open [reader (io/reader "numbers.csv")]
  (/ 1000000 12)
  (csv/read-csv reader)))) 


did you figure out your problem? i imagine its because you are reading from a stream. r/fold partitions your sequence into contiguous chunks, you cant start processing your second "chunk" until you have completely consumed the first


is there any problem with maven today? I cannot start my project and yesterday it was working.

error in process sentinel: nrepl-server-sentinel: Could not start nREPL server: Error building classpath. Could not find artifact org.clojure:spec.alpha:jar:0.2.44 in central ()

error in process sentinel: Could not start nREPL server: Error building classpath. Could not find artifact org.clojure:spec.alpha:jar:0.2.44 in central ()


i update the version and now it worked. weird. seems related to some glitch in my internet network


Can someone throw me a hint as to what's wrong with this namespace declaration? According to the error message, spec doesn't like this form, but I see nothing wrong.


Here's the ns form: (ns cql-parser.core (:require [clj-antlr.core :as antlr]))

Jimmy Miller14:07:02

I don't see anything wrong right off the bat. In general pasting the error message here will also help people figure out what your problem is.

Jimmy Miller14:07:21

Sorry you literally did that as I wrote this.


Here's the error and traceback:


#error {
 :cause Call to clojure.core/ns did not conform to spec.
 :data #:clojure.spec.alpha{:problems [{:path [], :reason Extra input, :pred (clojure.spec.alpha/cat :docstring (clojure.spec.alpha/? clojure.core/string?) :attr-map (clojure.spec.alpha/? clojure.core/map?) :ns-clauses :clojure.core.specs.alpha/ns-clauses), :val ([:require [(quote clj-antlr.core) :as (quote antlr)]]), :via [:clojure.core.specs.alpha/ns-form], :in [1]}], :spec #object[clojure.spec.alpha$regex_spec_impl$reify__2509 0x676cf48 clojure.spec.alpha$regex_spec_impl$reify__2509@676cf48], :value (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]]), :args (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]])}
 [{:type clojure.lang.Compiler$CompilerException
   :message Syntax error macroexpanding clojure.core/ns at (cql_parser/core.clj:1:1).
   :data #:clojure.error{:phase :macro-syntax-check, :line 1, :column 1, :source cql_parser/core.clj, :symbol clojure.core/ns}
   :at [clojure.lang.Compiler checkSpecs 6972]}
  {:type clojure.lang.ExceptionInfo
   :message Call to clojure.core/ns did not conform to spec.
   :data #:clojure.spec.alpha{:problems [{:path [], :reason Extra input, :pred (clojure.spec.alpha/cat :docstring (clojure.spec.alpha/? clojure.core/string?) :attr-map (clojure.spec.alpha/? clojure.core/map?) :ns-clauses :clojure.core.specs.alpha/ns-clauses), :val ([:require [(quote clj-antlr.core) :as (quote antlr)]]), :via [:clojure.core.specs.alpha/ns-form], :in [1]}], :spec #object[clojure.spec.alpha$regex_spec_impl$reify__2509 0x676cf48 clojure.spec.alpha$regex_spec_impl$reify__2509@676cf48], :value (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]]), :args (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]])}
   :at [clojure.spec.alpha$macroexpand_check invokeStatic alpha.clj 705]}]
 [[clojure.spec.alpha$macroexpand_check invokeStatic alpha.clj 705]
  [clojure.spec.alpha$macroexpand_check invoke alpha.clj 697]
  [clojure.lang.AFn applyToHelper 156]
  [clojure.lang.AFn applyTo 144]
  [clojure.lang.Var applyTo 705]
  [clojure.lang.Compiler checkSpecs 6970]
  [clojure.lang.Compiler macroexpand1 6988]
  [clojure.lang.Compiler macroexpand 7075]
  [clojure.lang.Compiler eval 7161]
  [clojure.lang.Compiler load 7636]
  [clojure.lang.RT loadResourceScript 381]
  [clojure.lang.RT loadResourceScript 372]
  [clojure.lang.RT load 459]
  [clojure.lang.RT load 424]
  [clojure.core$load$fn__6839 invoke core.clj 6126]
  [clojure.core$load invokeStatic core.clj 6125]
  [clojure.core$load doInvoke core.clj 6109]
  [clojure.lang.RestFn invoke 408]
  [clojure.core$load_one invokeStatic core.clj 5908]
  [clojure.core$load_one invoke core.clj 5903]
  [clojure.core$load_lib$fn__6780 invoke core.clj 5948]
  [clojure.core$load_lib invokeStatic core.clj 5947]
  [clojure.core$load_lib doInvoke core.clj 5928]
  [clojure.lang.RestFn applyTo 142]
  [clojure.core$apply invokeStatic core.clj 667]
  [clojure.core$load_libs invokeStatic core.clj 5985]
  [clojure.core$load_libs doInvoke core.clj 5969]
  [clojure.lang.RestFn applyTo 137]
  [clojure.core$apply invokeStatic core.clj 667]
  [clojure.core$require invokeStatic core.clj 6007]
  [clojure.core$require doInvoke core.clj 6007]
  [clojure.lang.RestFn invoke 408]
  [user$eval150 invokeStatic form-init897609585960091026.clj 1]
  [user$eval150 invoke form-init897609585960091026.clj 1]
  [clojure.lang.Compiler eval 7177]
  [clojure.lang.Compiler eval 7166]
  [clojure.lang.Compiler eval 7166]
  [clojure.lang.Compiler load 7636]
  [clojure.lang.Compiler loadFile 7574]
  [clojure.main$load_script invokeStatic main.clj 475]
  [clojure.main$init_opt invokeStatic main.clj 477]
  [clojure.main$init_opt invoke main.clj 477]
  [clojure.main$initialize invokeStatic main.clj 508]
  [clojure.main$null_opt invokeStatic main.clj 542]
  [clojure.main$null_opt invoke main.clj 539]
  [clojure.main$main invokeStatic main.clj 664]
  [clojure.main$main doInvoke main.clj 616]
  [clojure.lang.RestFn applyTo 137]
  [clojure.lang.Var applyTo 705]
  [clojure.main main 40]]}
Error loading lighttable.nrepl.handler: Syntax error macroexpanding clojure.core/ns at (cljs/source_map/base64_vlq.clj:1:1).
Syntax error compiling var at (C:\Users\Frank Adrian\AppData\Local\Temp\form-init897609585960091026.clj:1:4762).
Unable to resolve var: lighttable.nrepl.handler/lighttable-ops in this context

Jimmy Miller14:07:04

From the looks of the error message at the end there, it seems like it could be a lighttable issue. Can you load up a clj or lein repl to try about the code and see if that is the problem?


It starts a repl, but throws an error before it does. Here's what that error says:


I can't get the error copied from my DOS prompt, but it shows the same error information as above.

Alex Miller (Clojure team)16:07:07

the actual error here is

Error loading lighttable.nrepl.handler: Syntax error macroexpanding clojure.core/ns at (cljs/source_map/base64_vlq.clj:1:1).

Alex Miller (Clojure team)16:07:35

this is a now quite old cljs error - using a newer clojurescript should fix it

Alex Miller (Clojure team)16:07:25

but not sure if that's in your app or your tool or what's bringing it in

Jimmy Miller16:07:25

It is lighttable using an old cljs version.


I finally switched to Atom+Chlorine. That worked OK.


"We can't fully display this message" above


anyway, this makes me think you had ' in your require > (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]]), :args (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]])


I'm making a program to help me learn low level programming, and prototype low level algorithms. Is this a reasonable way to provide a lazy-seq view onto the successive states of an atom? Is there something safer? I'm afraid of what happens if a watch is set and nobody accesses the lazy-seq, wondering how best to handle this. It almost looks like a core.async backpressure problem.

(defn ->region
  "Uses an atom to help implement fake mutation on top of immutable data.
  See also change-list."
   (apply vector-of :byte
          (repeat N 0))))

(defprotocol ChangeWatcher
  (register [this]
    "Should return something that offers access to the changes.")
  (unregister [this]
    "Should turn off the supply of data."))

(defn change-list
  "Given a pile of bytes allocated in an atom by ->region, creates a lazy-seq
  of the values that pile takes on over time."
  (let [marker (UUID/randomUUID)
        data-funnel (ArrayBlockingQueue. 4096) ; puts will block after this
        watcher (fn watcher
                  [_k _r old-state new-state]
                  (.put data-funnel new-state))]
    (reify ChangeWatcher
      (register [this]
        (add-watch region marker watcher)
        (repeatedly #(.take data-funnel)))
      (unregister [this]
        (remove-watch region marker)))))


it seems like you're trying to combine 3 things? 1. A history of the changes 2. A way to receive changes as they're happening 3. An easy way to clean up history you don't care about by no longer referencing the head


I think (byte-array N) is the same as (apply vector-of :byte (repeat N 0))


the core idea is to have an immutable view over a mutable API


an alternate approach would be changing the API so it always returns a reference, and then require usage of that reference


(eg. the way transients are almost mutable)


the API has two existing implementations: checked raw regions via java.nio.ByteBuffer and non-gc non-bound-checked memory regions via sun.misc.Unsafe I'd like to ensure the API is not clumsy to use from those, while also having an implementation that uses immutable data for higher level debugging / analysis


I guess the project is young enough to change the API so all operations return a handle that must be used on next access, but that makes the target code unwieldy I think


personally, I would probably go that route where each operation returns a handle and the operations result and just write a macro that makes it easier to work with


> I think `(byte-array N)` is the same as  `(apply vector-of :byte (repeat N 0))` I want a byte vector, not a byte array, very different classes


I think the api of core.cache is inherently racy. still working through it but it seems like it can't work on atoms


why wouldn't core.cache work on atoms? you might have to use compare-and-set! and write your own loop directly though.


the api of has?, hit, and miss are distinct. there seems to be no single function that does it all. so there are multiple dereferences


i can make a ttl cache that responds true to has? but then doesn't contain the value. Looking into it now but i think i can make an lu-cache null pointer


you can either use compare-and-swap! or do everything inside of one swap! which I think is the same thing right, you could either do the update inside a swap! or use


I think the api is footguns


the correct way to use core.cache is to put the immutable cache object inside your atom


right and i can get an LU-cache to null pointer


(let [lu-cache (atom (lu-cache-factory {} :threshold 4))
        thread-count 20
        loop-count 20000
        latch (java.util.concurrent.CountDownLatch. thread-count)]
    (doseq [i (range thread-count)]
      (.start (Thread. (fn []
                         (loop [c 0]
                           (if (< c loop-count)
                               (if-let [v (lookup @lu-cache i)]
                                 (do (swap! lu-cache hit i)
                                 (swap! lu-cache miss i i))
                               (recur (inc c)))
                             (.countDown latch)))))))
    (.await latch)
    (println "has but not lookup misses: " @races))


that should be standard usage. if let lookup, then hit otherwise miss


but hit for an lu cache assumes the value is there and increments the usage counter. but another thread may have cleared that value in the meantime


i'm using to search for usages in the wild and the patterns are all different and each has a race i think


if you do two operations not inside a single swap i think its broken. and the api seems to invite you to do this


you should have lookup, hit, and miss all within one swap! function or use compare-and-set! to make sure you're not introducing race conditions


> if you do two operations not inside a single swap i think its broken I think this is true of all usages of swap!


and the namespace core.cached.wrapped is totally wrong then


(defn hit
  "Is meant to be called if the cache is determined to contain a value
  associated with `e`.

  Returns the updated cache from the atom. Provided for completeness."
  [cache-atom e]
  (swap! cache-atom c/hit e))


^ that function should never ever be called then


it invites you to race


hit is not safe at all


unless its called inside a single swap that uses lookup as well.


all the usages in the wild are wrong and racy as far as i can tell


I think the cache api is fine. it does seem like its usage within in an atom isn't* super helpful


most of my experience with core.cache is via core.memoize, which I think does things the right way


i think core.memoize hits the only valid api usage.

cs   (swap! cache-atom through* f args ckey)
           val  (clojure.core.cache/lookup cs ckey ::not-found)
you have to through it and then use lookup. Using your own usages of has?, hit, and miss are racy


and even memoize points out it has to watch for a race

;; If `lookup` returns `(delay ::not-found)`, it's likely that
       ;; we ran into a timing issue where eviction and access
       ;; are happening at about the same time. Therefore, we retry
       ;; the `swap!` (potentially several times).


concurrent code is tricky


but i'm not even sure how through can work if it doesn't return a value and a cache. a lru or lu cache could have through happen and then afterwards other threads have already evicted the value you wanted in there


and it seems to work


Except for source code, it isn't clear to me that core.cache was ever very clearly documented.


It strikes me as a library that for those that one who want to use it fully, or extend it, it could use more documentation than it has source code on how to use/extend it correctly.


i agree. i would love a section on "how to use it". as it seems memoize goes through extreme hoops of delaying retries to handle it


i also have a suspicion that its swallowing lots of errors not from the memoized function but from the caching library


ah, wrapped has lookup-or-miss which seems like it should be the used function


Does compile expand macros? I'm trying to configure routes for a web app based on the files available at the start time. The problem is that app should be able to work both if run from src and if run from an uberjar. And listing files in jars is not a simple task.


Came up with this:

(defn list-files [path]
  (let [abs-path (.getPath (io/resource path))]
    (if (str/starts-with? abs-path "file:")
      (let [jar-path (-> abs-path (str/split #"!" 2) first (subs 5))
            jar-file (JarFile. jar-path)]
        (->> (enumeration-seq (.entries jar-file))
             (filter #(and (str/starts-with? % path)
                           (not (str/ends-with? % "/"))))))
      (->> (file-seq (io/file abs-path))
           (filter #(.isFile (io/file %)))))))

❤️ 2

Well isn't it "beautiful".


My advice would be to do something like data_readers.clj


Your approach is bundled as a library in resauce fwiw


I'll check resauce out, thanks! Can you elaborate on data_readers.clj? Not sure what you mean.


If you have multiple of the same path on the classpath, then java can give you a list of those. You can read them each independently and they'll tell you which parts of the web application to activate.


This is how java11 service loaders work for example. I would consider it the idiomatic java way to do things, as it handles non jar/dir classloaders.


Oh, I think you're talking about something else entirely. I still have no idea how data readers are related to what you describe, but either way - I just needed to get a list of static resources (not source files) in runtime.


Hey all, I have a bit of a weird issue at the moment. I have a Ring webserver which serves HTML. When I run it in development on my laptop, every now and then after rendering a page, the java instance will start spinning at 100% CPU and never stop. The application doesn't stop either, you can still call ring handlers. The weird thing is that I can only reproduce it on my laptop in development (lein repl). I can't reproduce it on my desktop, I can't reproduce it on my laptop with an uberjar. Both laptop and desktop use the same java / clojure version. I've tried removing things from my handler, but I can't pinpoint where the issue is exactly. Does anyone have any tips on debugging a problem like this?


@kevin.van.rooijen You should be able to generate stack traces from the process (maybe with kill -3 <pid> or jstack if you have it) and see what they are doing when you see 100% CPU.


Thanks for the suggestion, I'll try that out. jstack sounds like what I need


I would probably also try starting the app from a plain clj REPL, to rule out any lein weirdness, or any other dev/editor stuff.


Yeah I'll try plain clj. The weird thing is that I can only reproduce this issue specifically on my laptop with lein run. Which is very frustrating


Do you have any Ring middleware specific to dev? ring-reload etc? Could be something in your plugins/middleware...


Not really anything specific to dev, and I'm not using auto reloading. I did add gzip middleware recently, but I tried disabling that and it didn't change anything


Ok well. Can't seem to reproduce it anymore :man-shrugging: Been bumping my head against this for at least 2 hours


Maybe there was just something wrong with my local setup. If it happens again I'll be sure to try jstack, thanks for the tip!


Heisenbug 🐛


How do you count optimum CPU / memory for clojure app in docker container? Do you use some tools for stress test like for example 80 request at the same time? How to measure the right CPU / memory limits for app in cloud? Do you do desktop stress tests or jump right to cloud tests, because tests on localhosts gives different results (this is what I expect). Do you use tools or set some random values and modify them manually in time? The topic seems to be very hard and without only one answer. Summary up: 1. Do we have good tools in Clojure for stress tests like 80 requests at the same time? (I would like to use Clojure to generate requests) 2. How to predict CPU / memory limitation for clojure app?


I don't think there are any simple answers to this @U0WL6FA77. So much about tuning is dependent on exactly what your app does and how much traffic it's going to get. Performance tuning is very much a "black art".


As for tooling, you can use all the same stuff that can be used on Java apps.


I have no specific advice because I don't deploy things on Docker -- we have fairly large, traditional production servers. I suggest you ask in #docker as a starter.


thanks, it is good to hear not only I feel a little lost about how to do it :)


There are lots of stress test tools out there to generate interesting traffic patterns. Don't restrict your search to Java-based tools or, particularly, Clojure-based tools.


Apache Bench, jMeter, there are lots of stress test tools.


I have an idea to generate requests using spec gen but maybe it is bad idea. I don’t know.


otherwise it will be hard to achieve random reuests


Read up on various stress test tools. Don't try to write your own. This is a solved problem with a lot of solutions.


thanks, I will try to find something.


I recently used Jmeter to load test a Python API. It's very flexible and you can configure it by drag/dropping components. There's a learning curve though, but you can instrument/setup the load test script as much as you like. For a single endpoint, probably wrk/wrk2 is easier to get started. Wrk uses lua as scripting language whereas jmeter supports a combination of many like java, groovy, JavaScript...


Do I need to provide further information regarding my question above or is everybody just as mystified as me? 🙂


whether or not a fold is done in parallel depends on whether or not the collection provides a CollFold protocol implementation that is done in parallel.


vectors are an example of a collection that does have a parallel implementation


I'm going to bet that (csv/read-csv reader) produces a lazy sequence -- so that isn't foldable.


not in parallel, anyway


Right, because parallel foldability is inherent in the collection type rather than r/fold.


@U09UV3WP6 I suspect by the time most folks came online, your question had scrolled out of recent history -- I didn't see it until your follow-up prompted me to scroll all the way back to many hours before I woke up on this Sunday 🙂


I had to do a search for from:@mac


Not many people use r/fold I suspect. I implemented it for next.jdbc/plan recently but there had been no prior requests throughout the years I'd maintained


@U04V70XH6 Yeah, I think you are right. I hope my nudge was polite enough 🙂


I hope my answer was helpful? 🙂

👍 3

@U7RJTCH6J And thanks, I did the csv parsing manually and cast to a line-seq, now it appears to work.

👍 3
Joshua Suskalo23:07:45

Are there any gotchyas to be aware of when writing a call to prepl? I'm playing with it at the moment, passing it a line numbering pushback reader which wraps a piped reader connected to a piped writer which I'm writing strings to, it seems to read off the reader, but the output method (which just logs the value it gets) is never being called. I'm flushing the writer, but do I have to do something special, like write an EOF when I want to push it? At the moment I'm writing newlines whenever I want to finish a segment to send.