Fork me on GitHub
#clojure
<
2020-07-19
>
dominicm10:07:48

Is there an API for tracking the dependencies of a namespace when loaded? That would include anything newly required and already required. I know of tns dependency tracking stuff, but that works on the source level.

noisesmith14:07:42

on the namespace level, you have the refers and the aliases, which are created inside the call to require (usually), but they only reify the translation (of prefix to larger prefix, or symbol to prefixed symbol), not the requirement of the other ns

noisesmith14:07:31

but, in normal code, there's usually a refer or alias created, and those can be found inside ns-refers and ns-aliases

dominicm16:07:15

oh, that's an interesting idea. Inspecting the ns afterwards for what it actually loaded...

dominicm16:07:38

Yeah, I guess without an alias though (which is common enough for what I am building) that's a problem.

dominicm16:07:02

I'd love to partiticpate in the loading system with hooks… yuk :D

mac13:07:40

Can someone explain to me why r/fold does not run in parallel across all available cores in this example: https://gist.github.com/maacl/953b092af98ee71aa58fbe48f80d6f64

(ns csv2summap.core
  (:require [clojure.data.csv :as csv]
            [ :as io]
            [clojure.core.reducers :as r]))

(with-open [writer (io/writer "numbers.csv")]
  (csv/write-csv
   writer
   (take 1000000 (repeatedly #(vector (char (+ 65 (rand-int 26))) (rand-int 1000))))))

(defn sum-vals
([] {})
([m [k v]]
 (update m k (fnil + 0) (Integer/parseInt v))))

(defn merge-sums
([] {})
([& m] (apply merge-with + m)))

;; r/fold version, should run in parallel on all cores, but does not
(with-open [reader (io/reader "numbers.csv")]
(doall
 (r/fold
  (/ 1000000 12)
  merge-sums
  sum-vals
  (csv/read-csv reader)))) 

peterc20:07:11

did you figure out your problem? i imagine its because you are reading from a stream. r/fold partitions your sequence into contiguous chunks, you cant start processing your second "chunk" until you have completely consumed the first

papachan14:07:02

is there any problem with maven today? I cannot start my project and yesterday it was working.

error in process sentinel: nrepl-server-sentinel: Could not start nREPL server: Error building classpath. Could not find artifact org.clojure:spec.alpha:jar:0.2.44 in central ()

error in process sentinel: Could not start nREPL server: Error building classpath. Could not find artifact org.clojure:spec.alpha:jar:0.2.44 in central ()

papachan14:07:22

i update the version and now it worked. weird. seems related to some glitch in my internet network

fadrian14:07:33

Can someone throw me a hint as to what's wrong with this namespace declaration? According to the error message, spec doesn't like this form, but I see nothing wrong.

fadrian14:07:18

Here's the ns form: (ns cql-parser.core (:require [clj-antlr.core :as antlr]))

Jimmy Miller14:07:02

I don't see anything wrong right off the bat. In general pasting the error message here will also help people figure out what your problem is.

Jimmy Miller14:07:21

Sorry you literally did that as I wrote this.

fadrian14:07:58

Here's the error and traceback:

fadrian14:07:58

#error {
 :cause Call to clojure.core/ns did not conform to spec.
 :data #:clojure.spec.alpha{:problems [{:path [], :reason Extra input, :pred (clojure.spec.alpha/cat :docstring (clojure.spec.alpha/? clojure.core/string?) :attr-map (clojure.spec.alpha/? clojure.core/map?) :ns-clauses :clojure.core.specs.alpha/ns-clauses), :val ([:require [(quote clj-antlr.core) :as (quote antlr)]]), :via [:clojure.core.specs.alpha/ns-form], :in [1]}], :spec #object[clojure.spec.alpha$regex_spec_impl$reify__2509 0x676cf48 clojure.spec.alpha$regex_spec_impl$reify__2509@676cf48], :value (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]]), :args (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]])}
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message Syntax error macroexpanding clojure.core/ns at (cql_parser/core.clj:1:1).
   :data #:clojure.error{:phase :macro-syntax-check, :line 1, :column 1, :source cql_parser/core.clj, :symbol clojure.core/ns}
   :at [clojure.lang.Compiler checkSpecs Compiler.java 6972]}
  {:type clojure.lang.ExceptionInfo
   :message Call to clojure.core/ns did not conform to spec.
   :data #:clojure.spec.alpha{:problems [{:path [], :reason Extra input, :pred (clojure.spec.alpha/cat :docstring (clojure.spec.alpha/? clojure.core/string?) :attr-map (clojure.spec.alpha/? clojure.core/map?) :ns-clauses :clojure.core.specs.alpha/ns-clauses), :val ([:require [(quote clj-antlr.core) :as (quote antlr)]]), :via [:clojure.core.specs.alpha/ns-form], :in [1]}], :spec #object[clojure.spec.alpha$regex_spec_impl$reify__2509 0x676cf48 clojure.spec.alpha$regex_spec_impl$reify__2509@676cf48], :value (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]]), :args (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]])}
   :at [clojure.spec.alpha$macroexpand_check invokeStatic alpha.clj 705]}]
 :trace
 [[clojure.spec.alpha$macroexpand_check invokeStatic alpha.clj 705]
  [clojure.spec.alpha$macroexpand_check invoke alpha.clj 697]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.lang.Var applyTo Var.java 705]
  [clojure.lang.Compiler checkSpecs Compiler.java 6970]
  [clojure.lang.Compiler macroexpand1 Compiler.java 6988]
  [clojure.lang.Compiler macroexpand Compiler.java 7075]
  [clojure.lang.Compiler eval Compiler.java 7161]
  [clojure.lang.Compiler load Compiler.java 7636]
  [clojure.lang.RT loadResourceScript RT.java 381]
  [clojure.lang.RT loadResourceScript RT.java 372]
  [clojure.lang.RT load RT.java 459]
  [clojure.lang.RT load RT.java 424]
  [clojure.core$load$fn__6839 invoke core.clj 6126]
  [clojure.core$load invokeStatic core.clj 6125]
  [clojure.core$load doInvoke core.clj 6109]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.core$load_one invokeStatic core.clj 5908]
  [clojure.core$load_one invoke core.clj 5903]
  [clojure.core$load_lib$fn__6780 invoke core.clj 5948]
  [clojure.core$load_lib invokeStatic core.clj 5947]
  [clojure.core$load_lib doInvoke core.clj 5928]
  [clojure.lang.RestFn applyTo RestFn.java 142]
  [clojure.core$apply invokeStatic core.clj 667]
  [clojure.core$load_libs invokeStatic core.clj 5985]
  [clojure.core$load_libs doInvoke core.clj 5969]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.core$apply invokeStatic core.clj 667]
  [clojure.core$require invokeStatic core.clj 6007]
  [clojure.core$require doInvoke core.clj 6007]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [user$eval150 invokeStatic form-init897609585960091026.clj 1]
  [user$eval150 invoke form-init897609585960091026.clj 1]
  [clojure.lang.Compiler eval Compiler.java 7177]
  [clojure.lang.Compiler eval Compiler.java 7166]
  [clojure.lang.Compiler eval Compiler.java 7166]
  [clojure.lang.Compiler load Compiler.java 7636]
  [clojure.lang.Compiler loadFile Compiler.java 7574]
  [clojure.main$load_script invokeStatic main.clj 475]
  [clojure.main$init_opt invokeStatic main.clj 477]
  [clojure.main$init_opt invoke main.clj 477]
  [clojure.main$initialize invokeStatic main.clj 508]
  [clojure.main$null_opt invokeStatic main.clj 542]
  [clojure.main$null_opt invoke main.clj 539]
  [clojure.main$main invokeStatic main.clj 664]
  [clojure.main$main doInvoke main.clj 616]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 705]
  [clojure.main main main.java 40]]}
Error loading lighttable.nrepl.handler: Syntax error macroexpanding clojure.core/ns at (cljs/source_map/base64_vlq.clj:1:1).
Syntax error compiling var at (C:\Users\Frank Adrian\AppData\Local\Temp\form-init897609585960091026.clj:1:4762).
Unable to resolve var: lighttable.nrepl.handler/lighttable-ops in this context

Jimmy Miller14:07:04

From the looks of the error message at the end there, it seems like it could be a lighttable issue. Can you load up a clj or lein repl to try about the code and see if that is the problem?

fadrian14:07:28

It starts a repl, but throws an error before it does. Here's what that error says:

fadrian14:07:28

I can't get the error copied from my DOS prompt, but it shows the same error information as above.

Alex Miller (Clojure team)16:07:07

the actual error here is

Error loading lighttable.nrepl.handler: Syntax error macroexpanding clojure.core/ns at (cljs/source_map/base64_vlq.clj:1:1).

Alex Miller (Clojure team)16:07:35

this is a now quite old cljs error - using a newer clojurescript should fix it

Alex Miller (Clojure team)16:07:25

but not sure if that's in your app or your tool or what's bringing it in

Jimmy Miller16:07:25

It is lighttable using an old cljs version.

fadrian14:07:05

I finally switched to Atom+Chlorine. That worked OK.

noisesmith14:07:37

"We can't fully display this message" above

noisesmith14:07:19

anyway, this makes me think you had ' in your require > (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]]), :args (cql-parser.core [:require [(quote clj-antlr.core) :as (quote antlr)]])

noisesmith16:07:24

I'm making a program to help me learn low level programming, and prototype low level algorithms. Is this a reasonable way to provide a lazy-seq view onto the successive states of an atom? Is there something safer? I'm afraid of what happens if a watch is set and nobody accesses the lazy-seq, wondering how best to handle this. It almost looks like a core.async backpressure problem.

(defn ->region
  "Uses an atom to help implement fake mutation on top of immutable data.
  See also change-list."
  [N]
  (atom
   (apply vector-of :byte
          (repeat N 0))))

(defprotocol ChangeWatcher
  (register [this]
    "Should return something that offers access to the changes.")
  (unregister [this]
    "Should turn off the supply of data."))

(defn change-list
  "Given a pile of bytes allocated in an atom by ->region, creates a lazy-seq
  of the values that pile takes on over time."
  [region]
  (let [marker (UUID/randomUUID)
        data-funnel (ArrayBlockingQueue. 4096) ; puts will block after this
        watcher (fn watcher
                  [_k _r old-state new-state]
                  (.put data-funnel new-state))]
    (reify ChangeWatcher
      (register [this]
        (add-watch region marker watcher)
        (repeatedly #(.take data-funnel)))
      (unregister [this]
        (remove-watch region marker)))))

phronmophobic16:07:39

it seems like you're trying to combine 3 things? 1. A history of the changes 2. A way to receive changes as they're happening 3. An easy way to clean up history you don't care about by no longer referencing the head

phronmophobic16:07:38

I think (byte-array N) is the same as (apply vector-of :byte (repeat N 0))

noisesmith16:07:36

the core idea is to have an immutable view over a mutable API

noisesmith16:07:02

an alternate approach would be changing the API so it always returns a reference, and then require usage of that reference

noisesmith16:07:19

(eg. the way transients are almost mutable)

noisesmith16:07:39

the API has two existing implementations: checked raw regions via java.nio.ByteBuffer and non-gc non-bound-checked memory regions via sun.misc.Unsafe I'd like to ensure the API is not clumsy to use from those, while also having an implementation that uses immutable data for higher level debugging / analysis

noisesmith16:07:18

I guess the project is young enough to change the API so all operations return a handle that must be used on next access, but that makes the target code unwieldy I think

phronmophobic16:07:36

personally, I would probably go that route where each operation returns a handle and the operations result and just write a macro that makes it easier to work with

noisesmith18:07:46

> I think `(byte-array N)` is the same as  `(apply vector-of :byte (repeat N 0))` I want a byte vector, not a byte array, very different classes

dpsutton16:07:20

I think the api of core.cache is inherently racy. still working through it but it seems like it can't work on atoms

phronmophobic16:07:10

why wouldn't core.cache work on atoms? you might have to use compare-and-set! and write your own loop directly though.

dpsutton16:07:06

the api of has?, hit, and miss are distinct. there seems to be no single function that does it all. so there are multiple dereferences

dpsutton16:07:40

i can make a ttl cache that responds true to has? but then doesn't contain the value. Looking into it now but i think i can make an lu-cache null pointer

phronmophobic16:07:27

you can either use compare-and-swap! or do everything inside of one swap! which I think is the same thing right, you could either do the update inside a swap! or use

dpsutton16:07:34

I think the api is footguns

noisesmith16:07:47

the correct way to use core.cache is to put the immutable cache object inside your atom

dpsutton16:07:18

right and i can get an LU-cache to null pointer

dpsutton16:07:36

(let [lu-cache (atom (lu-cache-factory {} :threshold 4))
        thread-count 20
        loop-count 20000
        latch (java.util.concurrent.CountDownLatch. thread-count)]
    (doseq [i (range thread-count)]
      (.start (Thread. (fn []
                         (loop [c 0]
                           (if (< c loop-count)
                             (do
                               (if-let [v (lookup @lu-cache i)]
                                 (do (swap! lu-cache hit i)
                                     v)
                                 (swap! lu-cache miss i i))
                               (recur (inc c)))
                             (.countDown latch)))))))
    (.await latch)
    (println "has but not lookup misses: " @races))

dpsutton16:07:51

that should be standard usage. if let lookup, then hit otherwise miss

dpsutton16:07:12

but hit for an lu cache assumes the value is there and increments the usage counter. but another thread may have cleared that value in the meantime

dpsutton16:07:35

i'm using http://grep.app to search for usages in the wild and the patterns are all different and each has a race i think

dpsutton16:07:59

if you do two operations not inside a single swap i think its broken. and the api seems to invite you to do this

phronmophobic16:07:04

you should have lookup, hit, and miss all within one swap! function or use compare-and-set! to make sure you're not introducing race conditions

3
noisesmith16:07:58

> if you do two operations not inside a single swap i think its broken I think this is true of all usages of swap!

dpsutton16:07:34

and the namespace core.cached.wrapped is totally wrong then

dpsutton16:07:36

(defn hit
  "Is meant to be called if the cache is determined to contain a value
  associated with `e`.

  Returns the updated cache from the atom. Provided for completeness."
  [cache-atom e]
  (swap! cache-atom c/hit e))

dpsutton16:07:43

^ that function should never ever be called then

dpsutton16:07:46

it invites you to race

dpsutton16:07:53

hit is not safe at all

dpsutton16:07:06

unless its called inside a single swap that uses lookup as well.

dpsutton16:07:54

all the usages in the wild are wrong and racy as far as i can tell

phronmophobic16:07:15

I think the cache api is fine. it does seem like its usage within in an atom isn't* super helpful

phronmophobic16:07:45

most of my experience with core.cache is via core.memoize, which I think does things the right way

dpsutton16:07:35

i think core.memoize hits the only valid api usage.

cs   (swap! cache-atom through* f args ckey)
           val  (clojure.core.cache/lookup cs ckey ::not-found)
you have to through it and then use lookup. Using your own usages of has?, hit, and miss are racy

dpsutton16:07:58

and even memoize points out it has to watch for a race

;; If `lookup` returns `(delay ::not-found)`, it's likely that
       ;; we ran into a timing issue where eviction and access
       ;; are happening at about the same time. Therefore, we retry
       ;; the `swap!` (potentially several times).

phronmophobic17:07:22

concurrent code is tricky

dpsutton17:07:03

but i'm not even sure how through can work if it doesn't return a value and a cache. a lru or lu cache could have through happen and then afterwards other threads have already evicted the value you wanted in there

phronmophobic17:07:30

and it seems to work

andy.fingerhut17:07:11

Except for source code, it isn't clear to me that core.cache was ever very clearly documented.

andy.fingerhut17:07:24

It strikes me as a library that for those that one who want to use it fully, or extend it, it could use more documentation than it has source code on how to use/extend it correctly.

dpsutton17:07:25

i agree. i would love a section on "how to use it". as it seems memoize goes through extreme hoops of delaying retries to handle it

dpsutton17:07:46

i also have a suspicion that its swallowing lots of errors not from the memoized function but from the caching library

dpsutton18:07:05

ah, wrapped has lookup-or-miss which seems like it should be the used function

p-himik17:07:18

Does compile expand macros? I'm trying to configure routes for a web app based on the files available at the start time. The problem is that app should be able to work both if run from src and if run from an uberjar. And listing files in jars is not a simple task.

p-himik17:07:53

Came up with this:

(defn list-files [path]
  (let [abs-path (.getPath (io/resource path))]
    (if (str/starts-with? abs-path "file:")
      (let [jar-path (-> abs-path (str/split #"!" 2) first (subs 5))
            jar-file (JarFile. jar-path)]
        (->> (enumeration-seq (.entries jar-file))
             (filter #(and (str/starts-with? % path)
                           (not (str/ends-with? % "/"))))))
      (->> (file-seq (io/file abs-path))
           (filter #(.isFile (io/file %)))))))

❤️ 2
p-himik17:07:00

Well isn't it "beautiful".

dominicm20:07:06

My advice would be to do something like data_readers.clj

dominicm20:07:21

Your approach is bundled as a library in resauce fwiw

p-himik07:07:29

I'll check resauce out, thanks! Can you elaborate on data_readers.clj? Not sure what you mean.

dominicm08:07:57

If you have multiple of the same path on the classpath, then java can give you a list of those. You can read them each independently and they'll tell you which parts of the web application to activate.

dominicm08:07:07

This is how java11 service loaders work for example. I would consider it the idiomatic java way to do things, as it handles non jar/dir classloaders.

p-himik08:07:31

Oh, I think you're talking about something else entirely. I still have no idea how data readers are related to what you describe, but either way - I just needed to get a list of static resources (not source files) in runtime.

Kevin20:07:54

Hey all, I have a bit of a weird issue at the moment. I have a Ring webserver which serves HTML. When I run it in development on my laptop, every now and then after rendering a page, the java instance will start spinning at 100% CPU and never stop. The application doesn't stop either, you can still call ring handlers. The weird thing is that I can only reproduce it on my laptop in development (lein repl). I can't reproduce it on my desktop, I can't reproduce it on my laptop with an uberjar. Both laptop and desktop use the same java / clojure version. I've tried removing things from my handler, but I can't pinpoint where the issue is exactly. Does anyone have any tips on debugging a problem like this?

seancorfield20:07:03

@kevin.van.rooijen You should be able to generate stack traces from the process (maybe with kill -3 <pid> or jstack if you have it) and see what they are doing when you see 100% CPU.

Kevin20:07:25

Thanks for the suggestion, I'll try that out. jstack sounds like what I need

seancorfield20:07:59

I would probably also try starting the app from a plain clj REPL, to rule out any lein weirdness, or any other dev/editor stuff.

Kevin21:07:13

Yeah I'll try plain clj. The weird thing is that I can only reproduce this issue specifically on my laptop with lein run. Which is very frustrating

seancorfield21:07:24

Do you have any Ring middleware specific to dev? ring-reload etc? Could be something in your plugins/middleware...

Kevin21:07:20

Not really anything specific to dev, and I'm not using auto reloading. I did add gzip middleware recently, but I tried disabling that and it didn't change anything

Kevin21:07:11

Ok well. Can't seem to reproduce it anymore :man-shrugging: Been bumping my head against this for at least 2 hours

Kevin21:07:05

Maybe there was just something wrong with my local setup. If it happens again I'll be sure to try jstack, thanks for the tip!

Kevin21:07:46

Heisenbug 🐛

kwladyka21:07:26

How do you count optimum CPU / memory for clojure app in docker container? Do you use some tools for stress test like for example 80 request at the same time? How to measure the right CPU / memory limits for app in cloud? Do you do desktop stress tests or jump right to cloud tests, because tests on localhosts gives different results (this is what I expect). Do you use tools or set some random values and modify them manually in time? The topic seems to be very hard and without only one answer. Summary up: 1. Do we have good tools in Clojure for stress tests like 80 requests at the same time? (I would like to use Clojure to generate requests) 2. How to predict CPU / memory limitation for clojure app?

seancorfield22:07:48

I don't think there are any simple answers to this @U0WL6FA77. So much about tuning is dependent on exactly what your app does and how much traffic it's going to get. Performance tuning is very much a "black art".

seancorfield22:07:34

As for tooling, you can use all the same stuff that can be used on Java apps.

seancorfield22:07:46

I have no specific advice because I don't deploy things on Docker -- we have fairly large, traditional production servers. I suggest you ask in #docker as a starter.

kwladyka22:07:51

thanks, it is good to hear not only I feel a little lost about how to do it :)

seancorfield22:07:24

There are lots of stress test tools out there to generate interesting traffic patterns. Don't restrict your search to Java-based tools or, particularly, Clojure-based tools.

seancorfield22:07:47

Apache Bench, jMeter, there are lots of stress test tools.

kwladyka22:07:04

I have an idea to generate requests using spec gen but maybe it is bad idea. I don’t know.

kwladyka22:07:33

otherwise it will be hard to achieve random reuests

seancorfield22:07:59

Read up on various stress test tools. Don't try to write your own. This is a solved problem with a lot of solutions.

kwladyka22:07:09

thanks, I will try to find something.

joaohgomes13:07:51

I recently used Jmeter to load test a Python API. It's very flexible and you can configure it by drag/dropping components. There's a learning curve though, but you can instrument/setup the load test script as much as you like. For a single endpoint, probably wrk/wrk2 is easier to get started. Wrk uses lua as scripting language whereas jmeter supports a combination of many like java, groovy, JavaScript...

mac21:07:04

Do I need to provide further information regarding my question above or is everybody just as mystified as me? 🙂

phronmophobic21:07:07

whether or not a fold is done in parallel depends on whether or not the collection provides a CollFold protocol implementation that is done in parallel.

phronmophobic21:07:36

vectors are an example of a collection that does have a parallel implementation

seancorfield21:07:36

I'm going to bet that (csv/read-csv reader) produces a lazy sequence -- so that isn't foldable.

phronmophobic21:07:54

not in parallel, anyway

seancorfield21:07:05

Right, because parallel foldability is inherent in the collection type rather than r/fold.

seancorfield21:07:57

@U09UV3WP6 I suspect by the time most folks came online, your question had scrolled out of recent history -- I didn't see it until your follow-up prompted me to scroll all the way back to many hours before I woke up on this Sunday 🙂

phronmophobic21:07:23

I had to do a search for from:@mac

seancorfield22:07:10

Not many people use r/fold I suspect. I implemented it for next.jdbc/plan recently but there had been no prior requests throughout the years I'd maintained clojure.java.jdbc.

mac22:07:31

@U04V70XH6 Yeah, I think you are right. I hope my nudge was polite enough 🙂

seancorfield22:07:13

I hope my answer was helpful? 🙂

👍 3
mac22:07:37

@U7RJTCH6J And thanks, I did the csv parsing manually and cast to a line-seq, now it appears to work.

👍 3
Joshua Suskalo23:07:45

Are there any gotchyas to be aware of when writing a call to prepl? I'm playing with it at the moment, passing it a line numbering pushback reader which wraps a piped reader connected to a piped writer which I'm writing strings to, it seems to read off the reader, but the output method (which just logs the value it gets) is never being called. I'm flushing the writer, but do I have to do something special, like write an EOF when I want to push it? At the moment I'm writing newlines whenever I want to finish a segment to send.