Fork me on GitHub
#beginners
<
2020-02-14
>
vxe05:02:58

I'm using clj-http, is logging a bunch of nonsense about invalid cookie headers to STDOUT. I would like this to be written to a file instead I've put the below log4j.properties in every dir in my project and its still writing to STDOUT, how can I address this?

log4j.rootLogger=INFO, A1
log4j.appender.A1=org.apache.log4j.RollingFileAppender
log4j.appender.A1.File=var/run/fml.log
log4j.appender.A1.MaxFileSize=500MB
log4j.appender.A1.MaxBackupIndex=2
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p%c - %m%n

Tzafrir Ben Ami06:02:16

if you're using Lein, try adding your log4j.properties to the project resources folder and update :dev profile in project.clj file `:profiles {:dev {:resource-paths ["resources"]} :uberjar {:aot :all}}`

vxe09:02:14

I used this to hijack the slf4j logs https://github.com/fzakaria/slf4j-timbre . Not going to use lein

hindol05:02:22

When is vector-of useful? Any benchmarks against *-array family?

andy.fingerhut06:02:00

It can save significant amounts of memory for large vectors. For example, a vector of long values takes about 1/3 the memory using vector-of vs. vector

👍 4
andy.fingerhut06:02:17

I don't have any benchmarks to show

andy.fingerhut06:02:47

vector-of should only use slightly more memory than corresponding Java primitive array, e.g. probably 5 to 10% more.

hindol06:02:39

So, the primary purpose is space efficiency?

didibus06:02:17

Ya, it turned out vector-of doesn't allow primitive math over it. But it has faster inserts

didibus06:02:25

If I remember correctly

didibus06:02:30

So basically, internally, it stores the elements unboxed, but everytime you put or take things from it, they'll get boxed

hindol06:02:56

Got it. These are great points!

andy.fingerhut12:02:00

I suspect that because it is more space efficient, there may also be some operations that it is faster at than vector, but that is a guess not checked by any benchmark measurements I have done.

hindol06:02:22

How do you manage state when you do a calculate at one go and query repetitively pattern? For example, with a prime sieve, we calculate all primes up to n at one go and then want to provide a query interface (prime? x) over the pre-calculated state. Is this when you reach out for types (`defrecord`/`deftype`)? (Sorry if I'm asking too many questions.)

em07:02:03

For the prime example, a really easy way would be to define the sieve recursively, and simply memoize To get the function var bound to the memoized version I think you could either do a def instead of defn on the anonymous form e.g. (memoize (fn [] ...)) , or potentially just rebind the var again with def outside

hindol07:02:01

I also want to retain the performance benefits of a sieve. Wouldn't this be slow?

penryu08:02:53

If you bind the initial "function" var to the memoized helper function, you'll get the benefit of memoization (constant-time return of previously calculated results) and automatic calculation AND storage of new results. defrecord and deftype aren't even required.

penryu08:02:01

Considering the nature of a prime sieve is entirely numeric, there's no need for any types, complex or otherwise, beyond those available in core Clojure.

hindol09:02:46

I doubt memoize will be equally fast though. Here is an exploration of the fastest possible sieve implementation in Clojure which relies on a boolean-array. I reckon memoize is associative? That's quite an overhead for hot loops like this. Thanks for chiming in though, good to hear your thoughts.

penryu09:02:22

Ah, so that code emits a lazyseq, which would elide unnecessary calculation for for anything not accessed. So if you want to know if the sieve contains x, you can either do a some (linear) or contains? (constant or log) on the returned seq. True, because this sieve impl returns the entire set as a lazyseq at once, you won't gain much from memoize. But unless you are searching for a low prime, you pay a very similar memory overhead for what is essentially the same constant-time lookup as an indexed memoize algorithm.

hindol09:02:54

Not quite getting your point. Maybe I am missing something. Are you talking about the concat? Sure that's a lazy seq, but it is backed by the boolean array. So the performance gain of not realizing the seq should be negligible?

penryu09:02:19

True, the data behind the lazy seq already seems to be realized, so... it's computationally and memory-wise comparable to a memoized indexed solution.

penryu10:02:19

... as long as any x you have is (< 3 x n). For any x higher than a previously calculated n, an indexed/memoized solution can be engineered to pick up from the previous n, where this solution would regenerate an entirely new seq.

didibus16:02:29

Fastest performance and immutability don't go hand in hand

didibus16:02:54

But you almost never need a deftype

didibus16:02:18

Generally, you'd use an array, or a Java mutable collection

didibus16:02:38

Transients can also help sometimes

didibus16:02:51

As a compromise between the two

didibus16:02:34

In sieves example, if you wanted to reuse the generated prime->boolean cache between invocations, I would just put it in a def. That's all

(def cache ...)
(defn sieve [n]
  ...
  cache 
  ...)

didibus16:02:29

If you are OCD about encapsulation. You can have your function close over it instead:

(def sieve
  (let [cache ...]
    (fn [n] ... cache ...)))

👍 4
didibus16:02:43

But, once you're back to the mutable world, multi-threaded use of shared variables become a problem again. So keep that in mind. If you fill the cache when its initialized. And make the sieve fn only read from it its fine. But if say two sieve fn are called concurrently, with the same prime, and they both try to grow the cache for it at the same time, you might get weird race conditions.

didibus16:02:04

Another way to handle this is to have the sieve fn take and return the cache.

didibus16:02:06

(defn sieve 
  ;; returns a cache up to n 
  ([n] ...)
  ;; returns true/false based on cache
  ([cache n] ...))

👍 4
hindol16:02:57

Yes, I have OCD about encapsulation. def with let seems nice. So is the idea of taking and returning a cache. Will try these out and bench them as well. Thanks.

hindol16:02:40

In fact the with and without cache idea is wonderful! You have shown me a new way of thinking about things, and this seems more functional as well.

hindol17:02:30

The problem with keeping a namespace global cache is that now you can have only one cache. For sieve, I guess that's okay, but not okay generally IMO. Also, the cache should not be the implementer's headache.

lepistane10:02:48

Hey yall i need to send activation mail for signup on a web app should i use https://github.com/nilenso/mailgun or https://github.com/drewr/postal

penryu10:02:51

Looks like mailgun uses a third-party service for the actual sending. If you have a mailgun account already, this would be a strong choice. postal seems to send the message via SMTP(+TLS) directly. It runs a higher risk of mails being classified as spam unless your domain DNS is configured correctly. It also requires java 1.8; none more recent.

lepistane10:02:31

@UGPM7GTKJ https://github.com/drewr/postal#encryption-gmail-example Does this remove the concern with spam classification? I'd be delegating email sending to google?

penryu10:02:11

No, the DNS measures I meant are unrelated to the encryption in transit. This might help: https://en.wikipedia.org/wiki/Sender_Policy_Framework

jumar11:02:08

You can use postal with any email provider that supports smtp so it might be a better choice if you want to have some flexibility in switching between them.

lepistane12:02:34

@UGPM7GTKJ thanks for the link and info @U06BE1L6T not being classified as spam is more important. And i understood correctly i need mail server to run postal? https://github.com/drewr/postal#dependencies

penryu12:02:48

@U45SLGVHV iiuc, postal speaks SMTP directly to the recipients' MX, so no, you shouldn't need to run a mail server yourself. But if the originating ip/host of the email doesn't match the SPF record, you stand a slightly higher risk of your mail being categorized as spam by the recipient.

penryu12:02:48

if there's a designated domain server you can use as a relay for outgoing emails, you can put that concern to rest.

jumar12:02:51

Right, you can use something like AWS SES or tons of other email services as well as your own email server (if you want) if your app uses plain SMTP. You can basically defer that decision and change it later too.

jumar13:02:05

E.g. at work, we have a standalone application which customers install on their own servers (on-prem) and the app uses postal to send scheduled reports. One company uses their own email server to send those reports while I use SES for testing.

👍 4
Piotr Brzeziński14:02:34

Hey! I have a rather simple issue that I’m not sure how to solve. I have this piece of code

(defn draw-stairs [n]
  (loop [x n]
    (when (> x 0)
      (println (str (clojure.string/join (repeat x " ")) "I"))
      (recur (- x 1)))))
which for (draw-stairs 3) outputs
I
 I
I
but I’d like to start the loop from 0 to N so I can produce
I
 I
  I
how should I reason about it?

jumpnbrownweasel14:02:46

So you want x to start at zero, used a (< x n) when condition, and increment x instead of decrementing it, is that right?

Piotr Brzeziński14:02:26

I should just do [x 0] and (< x n) lol

Piotr Brzeziński14:02:28

Ok, that worked. Thanks 😄

timcreasy14:02:49

There is also https://clojuredocs.org/clojure.core/dotimes just as an aside for cases like this where you don’t need to manage iteration cases.

timcreasy14:02:21

Which is just a macro which expands into just what you are doing 🙂 https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L3326-L3329

Piotr Brzeziński14:02:31

👍 nice, thank you 🙂

Piotr Brzeziński14:02:56

Turns out I have to use something else anyway because the testcase expects a string and not printed lines.

Piotr Brzeziński14:02:05

Probably a for loop then if I recall correctly since it can collect result?

timcreasy14:02:10

That sounds very reasonable to me!

Piotr Brzeziński14:02:20

Ok, managed to figure it out

(defn draw-stairs [n]
  (clojure.string/join "\n"
    (for [x (range 0 n)]
      (str (clojure.string/join (repeat x " ")) "I"))))

🎉 12
mloughlin14:02:00

your recursive base case (termination criteria) is when (> x 0) and your recursive call decrements by 1 each time (recur (- x 1)) The loop binding [x n] makes the first value of x equal to n from the fn parameter So you can go in the opposite direction by setting [x 0] , when (< x n) and (recur (+ x 1))

👍 4
tzzh14:02:41

is there something built in to prettify a map with one element per line and proper indentation eg

{:a 1
 :b {:c "asdsada"
     :d "zzzz"}
 :z 12312}
as far as I know pprint ’s behaviour is quite different

noisesmith17:02:57

there are dynamic vars that pprint uses that help control that behavior (though even that can be awkward)

noisesmith17:02:39

user=> (apropos #"^\*.*print.*\*$")
(clojure.core/*print-dup* clojure.core/*print-length* clojure.core/*print-level* clojure.core/*print-meta* clojure.core/*print-namespace-maps* clojure.core/*print-readably* clojure.pprint/*print-base* clojure.pprint/*print-miser-width* clojure.pprint/*print-pprint-dispatch* clojure.pprint/*print-pretty* clojure.pprint/*print-radix* clojure.pprint/*print-right-margin* clojure.pprint/*print-suppress-namespaces*)

noisesmith17:02:12

it's ... a start

user=> (binding [clojure.pprint/*print-miser-width* 2 clojure.pprint/*print-right-margin* 4] (pprint {:a 1 :b {:c "asdsada" :d "zzzz"} :z 12312}))
{:a
 1,
 :b
 {:c
  "asdsada",
  :d
  "zzzz"},
 :z
 12312}
nil

noisesmith17:02:26

this isn't perfect, but it does work precisely almost as you want for your input

ser=> (binding [clojure.pprint/*print-miser-width* 15 clojure.pprint/*print-right-margin* 18] (pprint {:a 1 :b {:c "asdsada" :d "zzzz"} :z 12312}))
{:a 1,
 :b
 {:c "asdsada",
  :d "zzzz"},
 :z 12312}
nil

donyorm16:02:36

Is there an equivalent to lein's :main key in project.clj for deps.edn? Basically I want my repls to start in a specific namespace

Alex Miller (Clojure team)16:02:16

no, although you could make your own custom repl ala https://insideclojure.org/2020/02/11/custom-repl/ - just need to modify the :init hook

Lukas17:02:01

Hey there just a quick question: Is it possible to to pass a Clojurescript function to a js library as callback parameter?

Lukas17:02:53

And if so does someone provide an example? :face_with_rolling_eyes:

dpsutton17:02:22

i think this example proves the general case: (js/setTimeout (fn [] (js/alert "I do arbitrary things")) 3000)

dpsutton17:02:46

there's nothing special about js/setTimeout. It could be any arbitrary js function

Lukas17:02:10

ty but not quite what i was looking for, since this won't work with a external js library

Lukas17:02:51

at least not out of the box i figured

dpsutton17:02:29

what issue are you running into? js/setTimeout is an external js function and here takes a cljs callback just fine

dpsutton17:02:18

here's an example with jquery: (.click (js/$ "body") (fn [event] (js/console.log "you clicked an event")))

Frederik17:02:54

iterate question: The iterate function specifically states it does not allow side effects in the function it's called on. Is there an alternative where this isn't a problem? It doesn't have to be lazy, I'd just like to get a (fixed-length sequence) like:

[x , f(x), f(f(x)), ...]
The function f mutates x (it's an ML model that gets trained), but it also returns the object (for clarity and to be able to use the code when the model is immutable). I guess I have to use loop-recur for this? Or are there other solutions?

delaguardo17:02:44

sounds like reduce is what you are looking for

(reduce 
  (fn [acc f] (f acc))
  x
  (repeate n f))

Frederik17:02:38

That's what I felt too, but couldn't figure out how! Repeating the function rather than the object is the key here. Thanks! 🙂

delaguardo17:02:52

first class functions are always your friend)

Frederik17:02:58

In python and R too, but the amount of problems you can solve with reduce alone is quite new to me 😄

noisesmith18:02:56

what about repeatedly? (repeatedly #(frob x))

noisesmith18:02:04

where repeatedly also takes an N arg (repeatedly N #(frob x))

user=> (repeatedly 10 rand)
(0.8080800356339141 0.8469572918844156 0.6770624642868489 0.9461635939606167 0.5870982429331622 0.28680625632485723 0.1834445338833387 0.24913157800173336 0.5788381193091194 0.08541291908430448)

Frederik18:02:24

Wouldn't work in case x is immutable, as x in #(frob x) wouldn't get updated

noisesmith18:02:40

you said in the start that f mutated x

noisesmith18:02:54

but yes, if you need to use return values and x is immutable, use reduce

noisesmith18:02:03

or iterate / take n

noisesmith18:02:41

user=> (take 10 (iterate inc 0))
(0 1 2 3 4 5 6 7 8 9)

Frederik18:02:26

Sorry if it wasn't clear, but the problem is indeed that the two approaches I thought of first either didn't work for immutable objects (repeatedly) or not for mutable ones (iterate). The reduce option seems to be good for both 🙂 (except that in case of immutable, my sequence will be filled with identical objects ofc)

noisesmith18:02:53

which sequence? the reduce version just gives you the last return value

Frederik18:02:46

Ah yes, ofc, sorry, was confused. Got used to using reduce in combination with adding an element to a sequence 🙂

hiredman17:02:40

https://clojure.atlassian.net/browse/CLJ-1906 is a thing which seems unlikely to ever get merged, but there is now https://clojure.atlassian.net/browse/CLJ-2555 which who knows

👀 8
Alex Miller (Clojure team)18:02:12

CLJ-2555 is based on Rich's ideas coming out of CLJ-1906 and is headed towards 1.11

Alex Miller (Clojure team)18:02:50

there's an async variant too for core.async, not sure if Ghadi has posted that yet

Lukas17:02:51

Okay guys sry i just messed up 🙈 i got it working (still learning tho). Thx for the help really appreciate it 🙂

dpsutton17:02:16

that's the point of the channel 🙂 don't feel bad. its literally why we're here to help

❤️ 4
Lukas17:02:40

awesome have a nice evening 😍

jakubl18:02:33

Hi, which one is more commonly used? (Float/parseFloat "4.5") or (edn/read-string "4.5" ) or something else?

dpsutton18:02:25

use the Float version

didibus18:02:36

But just know that they don't do the same thing

dpsutton18:02:43

not necessarily

didibus18:02:46

So it depends what you're trying to do

dpsutton18:02:01

(edn/read-string "7") will be different from the Float/parseFloat version

didibus18:02:20

Edn read-string will parse following the Clojure literal parse rules

didibus18:02:49

While the Java Number/parseNum methods have their own rules

delaguardo18:02:15

(edn/read-string …) is better because it automatically convert your string into prefered format. E.g. (type (edn/read-string "1.2")) => java.lang.Double

noisesmith18:02:07

also use Double/parseDouble unless you know you specifically need a 32 bit value

dpsutton18:02:08

i think i would use the phrase "its preferred type" rather than "your".

👍 4
delaguardo18:02:13

I mean “prefered” in platform specific way)

noisesmith18:02:14

I would only use edn/read-string where I'm expected to consume arbitrary data, if I only accept numbers, I'm going to use something more specific

dpsutton18:02:49

agreed. if you want a double, do so. (= (/ (Double/parseDouble "7") 2) (/ (edn/read-string "7") 2)) is false

jakubl18:02:00

interesting - thanks for tips everyone - this was more involved topic than i expected - since i do not like exceptions - i think i stick to edn-read-string and check the type is what i expect int?

hiredman18:02:42

read-string will of course also throw exceptions

Alex Miller (Clojure team)18:02:25

If anything, use Double/parseDouble

didibus18:02:03

You have to work backward from your use case. What are you parsing and what are you using the parsed number for afterwards?

didibus18:02:20

Depending on that you'll be able to know which parsing strategy to use

didibus18:02:19

The choice of parsing strategy is not a matter of style, but of use case. So there is no idiomatic way, all ways are appropriate, but each have their respective use case

jakubl18:02:36

really all i'm doing is parsing a port number that someone might send via (System/getenv "PORT") i felt integer should be enough

jakubl19:02:24

i expected a parse-int function along the line of this

312 user> (defn parse-int [x]
 313         (try
 314           (Integer/parseInt x)
 315           (catch Exception e
 316             nil)))
 317 #'user/parse-int
 318 user> (parse-int "d")
 319 nil
 320 user> (parse-int "4.5")
 321 nil
 322 user> (parse-int "4")
 323 4
 324 user> (parse-int "-54")
 325 -54
 326 user> 
and I was surprised not to find one - but i figured - it's my own cultural bias - and i was wondering what experienced clojurians do

seancorfield19:02:19

Again, use Long/parseLong rather than Integer/parseInt (Clojure's "native" types are Double and Long rather than Float and Integer).

seancorfield19:02:58

Clojure is a hosted language so it expects you to use the host features 🙂

jakubl20:02:13

ha good to know - regarding clojure native types - is this not what i think it is?

60 user> (class (int 5))
61 java.lang.Integer
62 user> 

dpsutton20:02:25

it is. but the defaults are not Integers but longs: (class 5) #_ -> java.lang.Long

jakubl20:02:50

got it - but for parsing port number - there is nothing inherently wrong with using Integer is there?

jakubl20:02:22

does that create problems elsewhere?

seancorfield20:02:25

It will be promoted to Long as soon as you use it so there's no real point in trying to "force" it to be Integer.

seancorfield20:02:44

(Well, depending on what you do with it)

seancorfield20:02:44

The only time I ever use Integer (or Float) in Clojure is when I'm dealing with Java interop that actually requires that type specifically.

seancorfield20:02:17

user=> (type (inc (int 5)))
java.lang.Long

jakubl20:02:08

makes perfect sense - in this case i just pass it to ring-jetty adapter - but i see - in case i was doing any math at all - no point to bother with Integer

jakubl20:02:20

love this channel - thanks all 🙂

seancorfield20:02:37

(when I first got started with Clojure, I had a tendency to use Integer/parseInt by the way -- because I hadn't gotten used to Long as a default 🙂 )

didibus01:02:55

For that use case, seems edn/read-string would do fine as well

didibus01:02:34

You can probably choose between that and Long/parseLong.

Wilson Velez20:02:13

hi, I have an ordered vector ([int double]) [[11 0.2] [23 0.5] [45 0.1]…[101 0.4]….[1000 0.8]], and I want to cut the vector from the first element when the first of its values is greater than a given value, if the value is 44 the result should be [[45 0.1]…[101 0.4]….[1000 0.8]], It works with filter but i think it is ineficient, is there a better way to do it?

(defn cut-vec [vec val] 
  (filterv #(> (get % 0) val) vec))

bfabry20:02:31

split-with

hiredman20:02:42

The most efficient way is going to be not using a vector

bfabry20:02:21

a sorted vector is a bit of an oddity

hiredman20:02:32

You might be able to get something pretty good if instead of getting rid of entries you are working willing to replace them with something else (nil for example)

hiredman20:02:42

It isn't sorted, I initially was tripped up by the wording as well, but the example function is filtering out everything

Wilson Velez20:02:54

> (defn cut-vec [vec val] (filterv #(> (get % 0) val) vec)) > => #’fund-performance.core/cut-vec > (cut-vec a 3) > => [[4 6] [7 9]] > (def a[[1 1] [2 4] [4 6] [7 9]]) > => #’fund-performance.core/a > (cut-vec a 3) > => [[4 6] [7 9]]

bfabry20:02:49

the example function is filtering out anything, but he does say it's "ordered" by the first element of the tuples. so if that's true split-at would be more "efficient" than filterv. but like you and noisesmith point out there's probably way bigger gains to be had by choosing appropriate data structures

Wilson Velez20:02:31

according to my example what do you recommend?

bfabry20:02:46

I would recommend (drop-while #(> (get % 0) val) vec)

Wilson Velez20:02:16

thanks @U050MP39D, and about the structure, what do you recommend?

bfabry20:02:23

I'd need to know a lot more about the problem. if this vector is huge then you could use a binary search to find the split point and then use subvec for instance. but if it's less than gigantic that would be a lot of work for not much reward

Wilson Velez20:02:20

It could have around 4000 elements

Wilson Velez20:02:56

the tuples are a timestamp and a percentage value

bfabry20:02:19

is the timestamp unique?

Wilson Velez20:02:28

the timestamp can also be used as a key it that helps

Wilson Velez20:02:36

yes, it is unique

Wilson Velez20:02:48

also this is for cljs

bfabry20:02:02

then I would use a sorted-map

bfabry20:02:19

and subseq for your function

Wilson Velez20:02:50

thanks, I will do that instead

bfabry20:02:01

actually. and this is all academic because 4000 elements is really small. but a sorted vector and then using subseq would actually make sense if you know that the elements will be added in sorted order

bfabry20:02:56

actually no ignore that

Wilson Velez20:02:37

sorry @U050MP39D, do you recommend me to change from vector to sorted-map or not?

bfabry20:02:04

I would definitely use a sorted-map yes

Wilson Velez20:02:42

now that I’ve read subseq I understand why vector is very inneficient in this case, but that was the spec they gave me

noisesmith20:02:52

vectors are only efficient for adding / removing elements at the end, you can use something like a finger-tree if you want fast mid-collection removal

eval-on-point23:02:00

I have a java object that would be great to use with with-open, but it has a method named release instead of close. Is there an efficient way to extend the object to give it a close method? I am looking for something like extend-type that works for java objects/interfaces rather than records/protocols. Are they interchangeable?

hiredman23:02:35

the easiest thing to do is to write a function like (defn with [x f] (try (f x) (finally (.release x))))

hiredman23:02:13

which would let you write (with (make-whatever) (fn [whatever] whatever))

hiredman23:02:35

which, if you are comfortable with, you can implement your own kind of with-open style macro on top of

eval-on-point23:02:04

Yeah that is the solution I've come up with

eval-on-point23:02:39

just wanted a second opinion, thank you very much!