Fork me on GitHub
#clojure
<
2017-11-09
>
jstew00:11:04

Anyone know of a good HTML extracting library? I want something that can use CSS or xpath selectors that can be represented in text so that I can store them in a config somewhere. Tried hickory and it works but it’s not geared toward text based parsing instructions.

noisesmith00:11:45

or - at least - I have done this with enlive before

jstew00:11:04

neat. I always thought it was a templating library. thanks!

noisesmith00:11:28

yeah, it templates too, but it can also break down html input and search it via selectors

noisesmith00:11:51

I've used it alot but have never templated with it

jstew00:11:25

looks like I could maybe store the config in edn or something. cool.

attentive07:11:04

A question about selecting Clojure libraries: Anyone know, offhand, of a site that will let you search a Clojure project by the other projects (with public dependency information eg on github) that have it as a dependency? I'd find this useful when understanding if and how certain libraries are being used.

attentive07:11:28

I feel sure this thing / project does or did exist, but I've forgotten where to find it.

pavelb08:11:09

Hi Guys, I've tried to test pmap functionality and came across a weird (to me) behaviour. It would be nice if someone could tell me where am I wrong, or how does it make sence: For my understanding pmap goal is to run a function in parallel over a collection, limiting parallelism. From pmap code I underatand that the desired parallelism level is 2+num-of-cpus In practice i see a different behaviour:

(defn- f1 [delay-num]
  (info "delay #" delay-num "started, sleeping")
  (Thread/sleep (* 1000 5))
  (info "delay #" delay-num "finished")
  delay-num)

(defn run []
  (let [delays (range 21)
        results (pmap f1 delays)]
    (info "num of cpus = " (.. Runtime getRuntime availableProcessors))
    (info "total is" (reduce + results))))
This shows all 20 delays running in parallel (I have 4 CPUs and running with clojure 1.8) Trying to understand it I came up that applying the seq on fs is to "blame". What do u say?

jumar09:11:02

@U7XBWABQT I guess this has something to do with chunked seqs. map et al process sequences in chunks of 32 elements. And this is indeed what I observe

jumar09:11:20

(defn- f1 [delay-num]
(Thread/sleep (+ 100 (rand-int 1000)))
(println (java.util.Date.) "delay #" delay-num "started")
(Thread/sleep (* 1000 5))
(println (java.util.Date.) "delay #" delay-num "finished")
delay-num)

(defn run []
(let [delays (range 41)
results (pmap f1 delays)]
(println "num of cpus = " (.. Runtime getRuntime availableProcessors))
(println "total is" (reduce + results))))

(run)


#inst "2017-11-09T09:06:30.493-00:00" delay # 1 started
#inst "2017-11-09T09:06:30.498-00:00" delay # 2 started
#inst "2017-11-09T09:06:30.559-00:00" delay # 13 started
.....

pavelb09:11:07

cool - thanx! wasn't aware of that at all. so it seems that num-of-cpus calculation is redundant right? unless you have more than 30 cpus.

bronsa09:11:50

correct, if you're pmapping over a chunked seq

bronsa09:11:14

OTOH you can dechunk it and get back the expected n+2 behaviour

bronsa09:11:26

or if your sequence isn't chunked, that will be the case too

pavelb10:11:37

understood. thanx a lot!

kurt-o-sys08:11:44

Not sure if it has been asked before - I can't find it googling for it - but with Java9 and the modularization of the JDK, I'm wondering if there's a kind of 'minimal JDK for clojure'? The reason for asking is: on small devices, RAM is still limited, and starting a few full blown JVM's takes quite some unnecessary baggage (CORBA, Swing, ...). It would be nice to have a JDK which contains the necessary modules, but not more, that clojure needs to run. I guess it could save up to maybe 30-40MB per JVM. Running 4-5 clojure apps on a 1-2 GB RAM device, well, it certainly makes a difference.

brainfreeze08:11:37

I've never actually used it: but have you looked into things like Wildfly? It's supposed to let you multiple applications in one JVM process. It might be able to help you now instead of waiting for the devs to do it.

attentive08:11:39

Thanks! That's the one.

noisesmith11:11:00

@kurt-o-sys the JVM already runtime loads classes into memory as you use them. If they end up in RAM that's because something accessed them at runtime. The main RAM killer with Clojure in particular is the assumption that RAM is cheap and throughput is more important, which informs the design of the collections and the way the core functions use those collections.

kurt-o-sys12:11:01

@cody_slack Yeah... I'm not particularly fond of application servers. I can't talk for all of them, but they still have a hard time if one of the applications is stopped unexpectedly (even in clojure, this happens 🙂 ). Many of them seem to have quite some issues with memory leaks as well. They're not real multi-tenant... I'm still hoping some day, there will be a real multi-tenant JDK. @noisesmith allright... I will check it. In my understanding - and seeing how much RAM 'hallo world' seems to use - the JDK loads the classes of the applications dynamically, but not the JDK itself. I may be completely wrong, though 🙂.

noisesmith12:11:01

Why even use a JDK if not compiling Java code? Clojure doesn't need one. you can just use a JVM.

kurt-o-sys12:11:24

right... misphrased my question 🙂

kurt-o-sys12:11:25

Being able to modularize and customize your own JVM, is there minimal clojure-JVM? - oh, I seem to answer my own question: clojure doesn't need any special JVM, so just JDK9 base (and maybe some other modules), be be ok. Could work. However, without aot, would an an optimized JVM (which can compile clojure on the fly) make sense?

noisesmith12:11:50

Clojure can't run without the compiler anyway can it?

noisesmith12:11:23

my hello.java has 21 megs resident (I threw a Thread.sleep so I would have enough time to find it in htop)

noisesmith12:11:47

that's a lot more than I would see in C, but it's a lot less than clojure.core takes up

noisesmith12:11:42

for reference

class Main {
  public static void main (String[] argv)
    throws InterruptedException {
    Thread.sleep(100000);
  }
}

kurt-o-sys12:11:29

Right... so, I'll check how much it would take for clojure - later.

noisesmith12:11:07

my clojure 1.9.0-RC1 with no other deps is 83 megs resident before I do any require calls

noisesmith12:11:42

pulling in core.async brings it up to 298 megs

noisesmith12:11:04

this is without lein, boot, or any other tooling

kurt-o-sys12:11:43

OK, so it's pretty clear what the memory is necessary for

kurt-o-sys12:11:08

it's not the JVM 🙂. Meaning, no need to start fiddling with it 🙂

noisesmith12:11:26

this is on

$ java -version
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)

noisesmith12:11:26

yeah - java isn't the primary thing driving memory usage here - I'm sure java's design and optimization criteria play a part, but it's dwarfed by the memory used by clojure itself to define data structures and functions

rauh12:11:04

FYI: you can get some info about memory usage with (bean (ManagementFactory/getMemoryMXBean))

souenzzo13:11:30

IllegalAccessException class clojure.core$bean$fn__5975$fn__5976 cannot access a member of class sun.management.MemoryImpl (in module java.management) with modifiers "public" jdk.internal.reflect.Reflection.newIllegalAccessException (Reflection.java:361) (i'm on java9)

kurt-o-sys12:11:49

ok, nice, thx a lot. This will do for now 🙂. I guess I'll just have to see if I can work it out to use clojure on small devices.

noisesmith12:11:58

Clojure's design assumes RAM is cheap (Rich Hickey has talked about this when talking about Clojure's design assumptions)

kurt-o-sys12:11:50

yeah, I know... but I just seem to have a hard time to not using clojure these days.

noisesmith12:11:04

there are other functional languages (some of them even in the lisp family) that don't make that assumption

noisesmith12:11:41

in my experience OCaml is really good about runtime memory usage if you stick to the core language as much as possible

kurt-o-sys12:11:51

Or I'll may have to see for another lisp... or maybe Erlang or OCaml.

kurt-o-sys12:11:13

nice... looks pretty cool as well.

noisesmith12:11:06

I use OCaml for a project on the raspberry pi, I could compile OCaml to machine code ~100 times then run it, in the time it takes Clojure to start up on the pi

noisesmith12:11:19

(rough rough estimate)

sundarj12:11:46

@kurt-o-sys you could also look into clojerl or LFE

kurt-o-sys12:11:46

@sundbp you have experience with them (they seem to be experimental, whatever that really means)

sundarj12:11:16

no experience, sorry

kurt-o-sys12:11:40

np... again too many things to try out.

len12:11:22

Is it possible to save maps with namespaced keys to mongo ?

jstew12:11:35

Nobody has mentioned elixir yet?

jstew12:11:40

I bet lumo would run on the pi as well.

mpenet12:11:22

@kurt-o-sys lfe is quite solid/stable

kurt-o-sys12:11:03

ok... so I may give that a try.

mpenet12:11:03

but even tho clojerl is younger it looks more appealing, lfe inherits from a lot of common lisp idioms

kurt-o-sys12:11:42

yeah... well, I can bet it'll be stable and try clojerl. No harm done in the first months anyway 🙂

mpenet12:11:02

it's prolly super alpha but it's moving fast it looks like

noisesmith12:11:19

@len I prefer to only have a few keys in my mongo document (basically whatever needs indexing) plus an embedded transit string

noisesmith12:11:45

because this allows using all the core clojure data types (plus whatever else I implement readers / writers for as needed)

noisesmith12:11:02

monger will auto-convert all map keys to strings going in, and by default will turn those into keywords coming out, which leads to terrible things like {1 :a} (key is a number) becoming {:1 "a"} (key is a terrible, illegal keyword)

len12:11:41

Thanks @noisesmith we are converting the keys to strings befor insertion and it seems to be working

noisesmith13:11:51

right, my point directly to your question is that "namespaced keys" can't exist in mongo without you encoding

len13:11:06

yeah I see that now

noisesmith13:11:47

whether it's a simple assumption like "if there's a / that's namespaced and it should be a keyword" or a more advanced usage like embedding a full encoded clojure data payload (my choice has been the latter - it simplifies a lot of things)

qqq15:11:07

1. In namespace foo.bar , I created a keyword ::apple 2. In namespace snip, I did (:require [foo.bar :as fb]) 3. I'm trying to use fb/:apple ... but am getting an 'no such var error 4. Is there a wa yot refer to a keyword in a namespace via the alias ?

seancorfield15:11:28

The :: auto-resolves the alias.

danm15:11:33

You'll see that a lot if you do XML manipulation, with the XML namespaces 😉

seancorfield15:11:53

Or use spec...

qqq15:11:36

This is from spec use. For the past few days, I've been thinking "getting errors with deep stacks is really annoying -- wouldn't it be great if, for each function, I had two functions: one to check if the input is correct format one to actually evaluate and the check-correct-format function is run before hand" -- then I realize I just rediscovered spec

bronsa15:11:32

i think you just rediscovered pre/post conditions rather than spec :)

qqq15:11:43

@bronsa: good point, those all the 'pre-conditions' are checks of the form 'this var has the following shape' -- and the shapes are composable

grav18:11:23

My project depends on two libraries that both depend on respectively some-lib 1.1.1 and some-lib 1.1.2. According to lein deps :tree, some-lib 1.1.1 is pulled in. I thought newest version would win in this case?

grav18:11:32

Oh, btw, I get a warning when running lein deps :tree that may explain it in the concrete case:

[bidi "2.1.2"] -> [prismatic/schema "1.1.3"]
 overrides
[io.nervous/eulalie "0.6.11-SNAPSHOT"] -> [prismatic/plumbing "0.5.5" :exclusions [org.clojure/clojure]] -> [prismatic/schema "1.1.7"]
So how come bidi overrides in this case?

noisesmith19:11:46

there is no “newest wins” logic - you can use :exclusions, or ensure that the the dep walking finds the version you want before the other one (including explicitly adding the one you want earlier in the list)

noisesmith19:11:11

it uses the first one it finds, and it searches from the beginning of your deps vector to the end

grav19:11:15

@noisesmith I tried putting eulalie on top without any luck. Maybe the dependency path length is taken into account?

noisesmith19:11:53

hmm - that could be

grav19:11:40

Okay. Anyway, thanks for the pointer. I ended up adding an explicit dependency to prismatic/schema "1.1.7", and I’ll ask the bidi project to bump the version

not-raspberry20:11:34

Hi. I have a problem with HugSQL used with Postgres (`[com.layerware/hugsql "0.4.7"]`, [org.postgresql/postgresql "42.1.3"]). This code:

(try
  (db/persist-job-invocation data)
  (catch java.sql.SQLException e
    (println "=================== caught java.sql.SQLException")
    (println (type e))
    (println (type (.getCause e)))
    (println "==================="))
  (catch Exception e
    (println "=================== caught Exception")
    (println (type e))
    (println (type (.getCause e)))
    (println "===================")))

throws a useful SQL exception wrapped in a useless java.lang.Exception.
=================== caught Exception
java.lang.Exception
java.sql.BatchUpdateException
===================
How to catch the real wrapped exception? Do I have to unwrap and raise the exception each time I call a HugSQL-generated DB function?

seancorfield20:11:49

@not-raspberry Take a look at the stacktrace and see what code is wrapping the exception. I thought that clojure.java.jdbc made sure to throw bare java.sql.SQLExceptions wherever it could, so maybe HugSQL is catching and rewrapping it?

not-raspberry20:11:30

HugSQL does this in the generated DB functions:

(catch Exception e
   (adapter/on-exception a e))))))))
The default adapter method:
(on-exception [this exception]
  (throw exception)))
Could this cause it? (I'll reproduce it and get an exception in a sec)

not-raspberry20:11:29

> Caused by: java.lang.Exception: Exception in :persist-job-invocation So it looks like that's hugsql.

not-raspberry20:11:17

Downgrading conman helped.

shooodooken21:11:26

does anyone know if Juxt's Clojure radar is geting an update? [ 2016 versoin: https://juxt.pro/radar.html]

cfleming21:11:02

Is there any way to adjust the :tag metadata on a local binding?

cfleming21:11:24

(defn len [x]
  (let [x (vary-meta x assoc :tag String)]
    (.length x)))
still produces a reflection warning.

ghadi21:11:59

do you mean the fn arg or the let binding?

ghadi21:11:50

for the latter I write (let [^String x .......] ........)

cfleming21:11:50

Either, really - in this case I’m trying to use the let binding to adjust the meta on the fn arg.

cfleming21:11:06

Yeah - thinking about it, what I’m trying to do doesn’t make sense.

ghadi21:11:36

it needs to go on the left-hand-side...the way you have it written it is tagged at runtime, not at compile time

cfleming21:11:41

I was hoping to be able to calculate the tag value at runtime, but of course that affects compilation.

ghadi21:11:01

yeah you can't dynamically modify at runtime without reflection.

cfleming21:11:02

I was up both late and early, is my excuse.

ghadi21:11:08

lol. welcome to NZ

cfleming21:11:52

Lein 2.8.0 switched Aether to a new version which has moved from Sonatype to Eclipse

cfleming21:11:04

So all the package names of the classes have changed.

cfleming21:11:16

Even though the code is basically the same.

cfleming21:11:30

Just trying to figure out the best way to handle that.

abdullahibra21:11:50

which is better? (-> ll rest butlast butlast) or (subvec ll 1 (-> ll count dec dec))

abdullahibra21:11:10

or if there is something better than both please refer it?

vemv21:11:37

@abdullahibra subvec should perform quite better, assuming II is a vector

vemv21:11:17

subvec surely leverages structural sharing, my guess is it's an instant operation

ghadi21:11:35

(-> v pop pop rest)

ghadi21:11:45

(or subvec)

ghadi21:11:56

butlast is linear

ghadi21:11:00

like last

abdullahibra22:11:01

another question

abdullahibra22:11:50

what is the best way to for ["hello" "world"] to swap first two characters in this list to get ["ehllo" "owrld"] ?

abdullahibra22:11:28

i can do it in ugly way

abdullahibra22:11:34

(mapv (fn [w] (let [l (vec w) tmp (get l 0)] (-> l (assoc 0 (get l 1)) (assoc 1 tmp) )) ["hello" "world"])

abdullahibra22:11:42

i feel it's ugly way to do this

abdullahibra22:11:50

is there elegant way to this?

abdullahibra22:11:01

that's exactly i have

abdullahibra22:11:03

(->> ["hello" "world"] (mapv (fn [w] (let [l (vec w) tmp (get l 0)] (-> l (assoc 0 (get l 1)) (assoc 1 tmp))))) (mapv #(apply str %)))

phronmophobic22:11:48

(->> ["hello" "world" ""]
     (map (fn [w]
            (let [split-index (min (count w)
                                   2)]
              (str
               (apply str (reverse (subs w 0 split-index)))
               (subs w split-index))))))

abdullahibra22:11:59

@smith.adriane what makes your version more elegant ?

phronmophobic22:11:15

it depends on who you ask

abdullahibra22:11:32

i want your prespective

phronmophobic22:11:46

I prefer this way of writing it because I think it emphasizes what you’re trying to do

abdullahibra22:11:10

what makes my version is less elegant in your prespective ?

phronmophobic22:11:11

the fact that you’re swapping letters is a little harder to see

phronmophobic22:11:47

and the fact that you’re using assoc and manipulating vectors makes it harder to tell you’re doing string manipulation

phronmophobic22:11:04

another try is

(->> ["hello" "world" ""]
     (map (fn [[first-letter second-letter & other-letters]]
            (apply str second-letter first-letter other-letters))))

abdullahibra22:11:33

yeah last version is better for me

phronmophobic22:11:38

which matches your problem description a little more closely

phronmophobic22:11:13

the reverse in my first version is a little closer to the “swap” part of “swap the first two characters”

vemv22:11:20

> what is the best way to for ["hello" "world"] to swap first two characters in this list to get ["ehllo" "owrld"] ? also worth nothing that what you really care about is the string function. dealing with vectors is trivial and a separate concern by reducing problems to their essence, you get to think more clearly 🙂

phronmophobic22:11:35

the other thing I would consider is that in @abdullahibra’s example, the “swapping the first two letters” gets split up into two pieces

phronmophobic22:11:44

first the swap and then turning it back into a string

phronmophobic22:11:30

so even if you were going to do it that way, I would combine both pieces into one function since I think of swapping the first two letters of a string as one operation

phronmophobic22:11:14

like

(->> ["hello" "world"]
     (mapv (fn [w]
             (let [l (vec w)
                   tmp (get l 0)]
               (-> l
                   (assoc 0 (get l 1))
                   (assoc 1 tmp)
                   (->> (apply str)))))))

phronmophobic22:11:48

the other point I would make is that tmp isn’t very descriptive

phronmophobic22:11:58

just using good names can help readability a lot imo

(->> ["hello" "world"]
     (mapv (fn [w]
             (let [l (vec w)
                   first-letter (first l)
                   second-letter (second l)]
               (-> l
                   (assoc 0 second-letter)
                   (assoc 1 first-letter)
                   (->> (apply str)))))))

noisesmith22:11:30

@smith.adriane as a suggestion, you can replace that let block with (let [[first-letter second-letter] w] ...)

noisesmith22:11:08

oh, you need it to be a vector never mind

noisesmith22:11:07

(let [[first-letter second-letter & trailing] w] (apply str second-letter first-letter trailing))

phronmophobic22:11:25

one of my snippets above is very similar to that one

noisesmith22:11:40

oh, I see that now

phronmophobic22:11:50

I was trying to show some other examples that were incremental improvements upon his original

noisesmith22:11:26

another suggestion: (assoc 0 second-letter 1 first-letter) - assoc is vararg

phronmophobic22:11:16

clojure is fun because you can often come up with a really concise readable version

seancorfield22:11:16

(let [[first-letter second-letter :as l] (vec w)] ... ) would do what you need I think? (edited to use vec instead of seq so l is a vector)

noisesmith22:11:29

you don't need seq there

noisesmith22:11:48

but as he said, he showed that above

seancorfield22:11:17

Oh, I didn't scroll back to see -- it's a very long thread 😉

phronmophobic22:11:05

I think the main take aways are 1) name intermediate variables in a way that helps convey what you’re trying to do if possible 2) if possible try to write a solution that matches a declarative description of the problem. I think the other changes are less important.

seancorfield22:11:43

(->> ["hello" "world"]
     (mapv #(let [[first-letter second-letter :as word] (vec %)]
              (clojure.string/join (assoc word 0 second-letter 1 first-letter)))))

seancorfield22:11:04

(TIL recently: str/join uses "" if you omit the separator)

leontalbot23:11:41

@seancorfield first time I see this « let in map destructuring » coding style... pretty cool

seancorfield22:11:58

associng into a vector kinda bothers me... I think I'd do (into [second-letter first-letter] (nnext word)) (and then you don't need the call to vec).