Fork me on GitHub
#beginners
<
2020-10-13
>
Jim Newton07:10:31

I always forget where the documentation for the parameter list of functions, (fn, defn, defn-, etc) is. Its not in the https://clojuredocs.org/clojure.core/fn of fn 😞

delaguardo07:10:41

it is in documentation

(fn name? [params*] exprs*) (fn name? ([params*] exprs*) +)

Jim Newton07:10:07

This doesn't explain keyword arguments.

delaguardo07:10:57

keyword arguments?

Jim Newton07:10:34

yes some functions can be called like (f a :x 100 :y 200)

Jim Newton07:10:57

there's a way to define these giving default values of x and y

delaguardo07:10:01

it is a sequence of args

Jim Newton07:10:06

the descructuring rules for let are different from those of fn . Is this really not explained anywhere?

delaguardo07:10:17

they are the same

Jim Newton07:10:00

no, I cannot define a function like this, for example. (fn [a & others :as all] ...)

Jim Newton07:10:24

nor can I define a function like this (fn [[[x]]] x)

Jim Newton07:10:56

they are very similar but not the same.

Jim Newton07:10:29

wow. I didn't know that one. cool.

Jim Newton07:10:37

but :as doesn't work as far as I know

delaguardo07:10:19

(fn [& [a & other :as all]])

Jim Newton07:10:26

but not (fn [a & other :as all] (list a other all))

clojure-rte.rte-core> ((fn [a & other :as all] (list a other all)) 3)
Syntax error macroexpanding clojure.core/fn at (clojure-rte:localhost:59787(clj)*:17647:24).
(:as all) - failed: Extra input at: [:fn-tail :arity-1 :params] spec: :clojure.core.specs.alpha/param-list
a - failed: vector? at: [:fn-tail :arity-n :params] spec: :clojure.core.specs.alpha/param-list
clojure-rte.rte-core>  

delaguardo07:10:14

because this form is violating a spec for arglist

Jim Newton07:10:14

exactly, the format for arglist is different than the format for destructuring. But I don't find an explanation of this difference in the documentation anywhere

delaguardo07:10:18

note that [ and ] are not a part of arglist, it is a form that is inside of brackets

Jim Newton07:10:58

BTW, thanks for the link. I found this useful example>

Jim Newton07:10:26

exactly, and what is inside the brackets is different than destructuring rules.

delaguardo07:10:52

(fn name? [params*] exprs*)
here params* is a list of destructuring forms representing arglist [ and ] are not included into destructuring forms

Jim Newton07:10:58

Let me try again. I can destructure with [a], [a & others] , and [a & others :as all]

Jim Newton07:10:10

but I cannot define functions those three ways.

Jim Newton07:10:49

I see what you're saying though.

Jim Newton07:10:11

you're claiming that the arglist is not destructuring, but each variable in the argument list can be replaced by a destructuring form.

Jim Newton07:10:06

and you cannot use :as to get the entire value of the argument list, only the entire value of any element of the argument list.

Jim Newton07:10:51

I suspect this is really a bug in clojure core which has simply become a feature. Imagine writing a function which allows :as in destructuring forms, but explicitly disallows it in argument lists. My vague suspicion is that this is really just an overzealous syntax checker on fn.

delaguardo08:10:51

bold assumption

Jim Newton10:10:54

otherwise what is the motivation for intentionally making a function's argument list processor, simply a destructuror?

Jim Newton10:10:19

I agree, it is a very bold assumption on my part

delaguardo10:10:01

> otherwise what is the motivation for intentionally making a function’s argument list processor, simply a destructuror? I guess to simplify reading and compiling processes. As an example: (fn ([x]) ([x :as y]) …) not valid because those forms of the same arity but to throw there compiler need to know about destructuring

delaguardo10:10:51

compare with [param*] where each param can be a simple symbol or destructuring form

Jim Newton07:10:06

question about destructuring:

(defmethod rte-match :sequential
  [pattern items & {:keys [promise-disjoint]}]
  (rte-match (rte-compile pattern) items))
what is the correct way to make the recursive call to rte-match passing the given keys? I.e., I want to pass :promise-disjoint nil if :promise-disjoint nil was specifically given at the call site, but not if the value is nil simply because it's missing from the map. given that I cannot use :as all in the arglist of rte-match

Jim Newton08:10:53

I think this works.

(defmethod rte-match :sequential
  [pattern items & {:keys [promise-disjoint] :as keys}]
  (apply rte-match (rte-compile pattern) items keys))

delaguardo08:10:24

(defmethod rte-match :sequential
  [pattern items & {:keys [promise-disjoint] :as opts}]
  (apply (partial rte-match (rte-compile pattern) items) opts))

Jim Newton08:10:44

I didn't use partial , is it necessary?

introom10:10:38

i want to test if the length of a lazy-seq is larger than some value, without exhausting the seq by using count

introom10:10:46

what’s the proper approach ?

introom10:10:20

(take n) and (count) ?

delaguardo10:10:44

(seq (drop n l-coll))

yuhan10:10:36

maybe (bounded-count n coll)

introom11:10:18

bounded-count fits here. but the function itself looks twisted to reason

yuhan12:10:51

What do you mean by that? the implementation of bounded-count is twisted?

yuhan12:10:16

(defn count< [n coll]
  (< n (bounded-count n coll)))

introom12:10:59

I mean the meaning of the return value of bounded-count.

yuhan12:10:36

ah, I think of it as a sort of (min n actual-count)

Jim Newton10:10:59

I'm looking to name a parameter to one of my functions, and the semantics are potentially difficult for the caller to understand, but could nevertheless have a big impact on the performance of the function. The function has two modes, 1) it can internally lazily-compile the user's data structure into a form which is very fast. Unfortunately the compilation is slow. 2) it can use a semantically equivalent interpreter which is slow. #2 is better if the user intents to only do it once. But if the user intends to call the function multiple times with different arguments, it may be better to use option #1. What could I call the parameter to make this more apparent to the caller?

Jim Newton12:10:10

can someone explain what destructure does? Its https://clojuredocs.org/clojure.core/destructure is not so illuminating. neither does there seem to be a useful comment in the code where it's defined.

Lennart Buit12:10:16

Destructuring is the process of extracting values from collections. Say you have a vector of two elements, [1 2], you can use a let to destructure that in the first and second value:

(let [[a b] my-vector] ...)
Similar things can be done with maps, say you have {:a 1 :b 2} , you can destructure it like this:
(let [{:keys [a b]} my-map] ...)

Lennart Buit12:10:14

Ah nevermind, I guess you are specifically asking about the function

Lennart Buit12:10:24

I’m pretty sure there is a better page about it

Jim Newton12:10:42

I didn't see a reference to restructure in that link. Did I miss it?

delaguardo12:10:31

I misunderstood what you asked

Jim Newton13:10:58

I was hoping it might be a programmatic interface to destructuring. In the coming weeks, I think I'm going to have to write a lambda-list parser which mimics the semantics of function lambda-lists. I was hoping the destructure function might be of help.

delaguardo13:10:39

can’t say much why there is no documentation for destructure in clojure.core this is more like internal helper that accidentally become public (this is my opinion, don’t have any proof that this is true)

Jim Newton13:10:01

yes I think my mac spell corrected destructure -> destructuring in my OP

delaguardo13:10:13

it is programmatic interface but in the form of helper for macro writers

Flo13:10:20

https://clojure.org/guides/destructuring#_macros at the very end it's talking about the destructure fn: > However, in rare cases you might want to instead resolve the destructuring yourself in a macro. In this case, use the (undocumented) clojure.core/destructure function[...]

delaguardo13:10:06

clojure.core/destructure is a function, not macro

Jim Newton13:10:37

looks like destructure produces the first operand of let. But it also gives me a sequence whose even elements are the names of the variables which the destructing will bind. That could be very useful.

delaguardo13:10:51

you can think about it as a function to return bindings (destructure '[[x y] z]) => [gen_0 z, x (nth gen_0 0 nil), y (nth gen_0 1 nil)] ^ this is simplified form

delaguardo13:10:24

so in one sentence it transform destructure syntax into clojure syntax

1
j16:10:42

Is there a idiomatic/preferred binary tree in Clojure?

ghadi16:10:39

@watchtheblur sorted-map is a persistent red-black tree

j17:10:59

Thanks @ghadi. I was looking for something that I could traverse/search children with. I realize I can implement a binary tree using a list and its indices, but I wondered if there was a better way (maybe objects)?

hiredman17:10:03

(use a vector and indices, lists are not for looking up by position)

👍 1
noisesmith17:10:22

@watchtheblur on a sorted collection (eg sorted-map) the subseqfunction function can be used for efficient splitting, sadly it doesn't return a pair of new sorted collections though

j17:10:52

@noisesmith seems like the censuses is to use a map instead of a vector for a binary tree? So if I wanted to grab the 2 children of a node, I would need to give indicies to the map to get the child, or is the map already structured as a tree (i.e. nested maps)?

noisesmith18:10:43

there's also sorted-set, but neither directly exposes its tree structure https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentTreeSet.java

noisesmith18:10:38

it requires interop, but at least it's public

alexmiller18:10:51

I can't imagine that's what OP actually wants, but that raises the question - what do you want? :) are you looking for a "by the book here's how to implement a binary tree" or are you looking to store some data in a tree for a reason, and if so there are probably better default answers

j18:10:11

@U064X3EF3 I'm looking to practice leetcode-type questions with clojure. Since leetcode doesn't support clojure, I wanted a simple binary tree data structure to run algorithms on. I wanted to check with the community on a idiomatic approach

alexmiller18:10:16

in general, the best way to work with Clojure is to lean on the built-in data structures, so I'd probably make each node a map with slots for :left and :right (or whatever). Or a vector of 2 items. not sure which way works out better.

j19:10:13

Got it, thanks @U064X3EF3!

Mario C.18:10:04

We have an app running in wildfly at work and it started to get unresponsive with users reporting that they are getting "BadGateway" issues. Looking at the logs I noticed quite a few errors. Main ones that stick out were "OutOfMemoryError: Metaspace", "NoClassDefFound", "I/O Exception Broken Pipe (Write failed)" and "Postgres connection attempt failed". Looking at NewRelic, I noticed that the UnloadedClass count started to increase along with a huge spike in GC (ConcurrentMarkSweep), where at the highest point, is where the application went down and users started reporting the errors. Operations restarted the servers and the app was working again. But this had happened last week as well. Currently I am researching but this is honestly a little over my head/experience level. I am thinking there is perhaps a memory leak going on but I am not so sure. Does anyone have any idea what could be causing this or could point me in the right direction?

Mario C.18:10:26

Oh, another thing, which could be coincidence, but the servers both went down on Monday/afternoonish

andy.fingerhut18:10:06

I don't have experience doing this myself, but there are several attach-them-to-the-running-JVM-process memory profiling tools that can aid in looking for what classes of objects are using the most memory.

Mario C.18:10:54

Yea I was trying to hook up VisualVM but was getting issues with it freezing

noisesmith18:10:22

hprof is much lighter weight, and visualvm can load the dump file it creates

hiredman18:10:45

sounds like you are calling eval a lot

Mario C.18:10:53

Hmm, what makes you say that?

Mario C.18:10:14

That could be it actually, since some new code that went out about two weeks, uses the eval function

hiredman18:10:26

eval compiles clojure code to jvm classes, classes are allocated in metaspace

hiredman18:10:58

the correct amount to be calling eval is usually 0

💯 1
Clark Urzo18:10:51

If you really need to have some form of eval in your code, there’s always sci

Mario C.18:10:56

Actually, the code does not use eval. I thought it did but it does not after second look

seancorfield18:10:28

@mario.cordova.862 are you trying to limit the metaspace? That seems to be a common mistake for folks coming from earlier JDKs where limiting the non-heap part of memory sometimes made sense (it's been a while so I've forgotten exactly what setting I'm thinking of on earlier JDKs)

seancorfield18:10:35

(it used to be a good idea to limit PermGen -- it is not a good idea to limit Metaspace, as I recall...)

noisesmith18:10:43

yeah, in my experience setting limits on metaspace is cargo-cult and/or mindless copying of old config from ancient jvms

noisesmith18:10:01

it's not a useful setting

Mario C.18:10:17

I am just trying to figure out why this issue occurred. If its a problem with new code changes or something else.

noisesmith18:10:44

something is allocating memory off heap, that's done by some libraries, it's done when you create new classes

noisesmith18:10:29

the biggest culprits I know of are using eval to make new functions at runtime, and badly written interop with platform level stuff (eg. a memory leak in initialization of some adaptor to C code)

noisesmith18:10:59

I guess "at runtime" is when all functions are created, but I mean in a deployed app after startup

Mario C.18:10:16

I think it might be the eval but it's odd that it just started to happen. Going to check if there is a limit on metaspace

noisesmith18:10:59

IIRC that's the only way to get that specific error

Mario C.18:10:51

The metaspace one?

noisesmith18:10:18

right, without the metaspace limit you get a different flavor of "vm out of memory"

noisesmith18:10:13

and you'd need to create a lot of functions before that came up (I guess that's one benefit of a metaspace limit, catching metaspace leaks early...)

noisesmith18:10:11

and now I wonder if there's any such thing as garbage collection of Classes (probably not...)

hiredman18:10:41

there is, but it is relatively new (I forget how new, or maybe if it is just a jep that hasn't landed yet)

hiredman18:10:24

I guess it isn't that new, java 8 or so

noisesmith18:10:57

so something must be not only creating new functions via eval but also holding onto them I guess

Mario C.18:10:36

I am seeing this in the standalone.conf -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=2G

Mario C.18:10:00

This is setting a maxsize on metaspace, which should not have a limit?

noisesmith18:10:38

I've never seen a drawback to just leaving it unlimited, but you probably also want to find out what's filling your metaspace and not getting cleared up

noisesmith18:10:55

visualvm / yourkit / hprof can show you what's filling metaspace

Mario C.19:10:03

Going to investigate it then, thanks for the help guys! :thumbsup:

noisesmith19:10:51

luckily anonymous functions / partial / reify / proxy etc. create class names that will lead you to the code creating them

hiredman19:10:48

Partial's is just something like clojure.core$partial$fn

Mario C.19:10:11

So eval is okay to use when "defining" something but during run-time it is an issue?

hiredman19:10:04

It is important to understand, and an important design feature, that clojure only creates classes during compilation, running code doesn't create new classes unless you call the compiler (calling eval)

hiredman19:10:32

So using any of the features @noisesmith mentioned will only generate a single class (and partial doesn't even do that, because partial doesn't really belong in that set) at compile time, and at runtime you will just get multiple instances of that class

hiredman19:10:26

The only way to generate an unbounded number of classes is by calling eval over and over

noisesmith19:10:11

right, I shouldn't have included partial, but was thinking that instead of creating fns via eval, they might be creating reify or proxy

hiredman19:10:38

Reify and proxy will only generate new classes when compiled

noisesmith19:10:45

right, inside eval

noisesmith19:10:53

it would be weird but it's possible

Mario C.19:10:57

So if I do (eval '(+ 1 1)) this creates a class?

hiredman19:10:25

I would not try to define cases when eval is ok to use and when it isn't. I would say because clojure provides it neatly wrapped in a single function you don't realize the complexity of it, and whatever you are doing almost certainly doesn't require it

hiredman19:10:57

To execute byte code on the jvm it has to live in a class file

hiredman19:10:56

Live in a class, the class may just exist in memory

hiredman19:10:52

Eval has some special cases for very trivial expressions where it just acts as an interpreter and doesn't generate byte code, but in general for anything but the most trivial expressions, you will generate at least one class when calling it

Mario C.19:10:31

Im not so much trying to define cases when is it okay to use but I am trying to reason about some code. I am trying to verify that a certain portion of the code is the culprit. Since this part of the code will be calling eval during runtime where as the other portion is using eval but not during runtime

hiredman19:10:11

Remove the calls to eval

hiredman19:10:03

When there is a foot gun in your code you remove it or debate where it is pointing, and for #beginners the response has to be removing it. Debating about where it is pointed belongs somewhere else

👏 2
hiredman19:10:09

Hah, I know that guy

Mario C.19:10:11

I dont feel comfortable just yet removing it, since it is legacy code and its essentially the heart of the application so a little risky

andy.fingerhut19:10:32

If the application is written such that it can be shut down in an orderly fashion, and/or has multiple instances running in parallel that can take over for each other dynamically, e.g. like many highly available web services, you could potentially consider the workaround of monitoring the PermGen space usage, and when it reaches a threshold, force that instance to restart. That would be a workaround, mind you, with its own introduced operational difficulties.

Sergio23:10:30

hello, I'm writing a little tool to automate some of my daily task on Clojure and Conch (https://github.com/Raynes/conch), however I have a function that I can't make pass a test. Here the code for both:

(defn exec-cmd [expression]
  (let [bin (:bin expression)
        args (:args expression)
        config (:config expression)]
    (sh/with-programs [echo mvn docker-compose]
                      (case bin
                        "echo" (echo args config)
                        "mvn" (mvn args config)
                        "docker-compose" (docker-compose args config)))))
(deftest test-echo-cmd-exec
  (testing "Execution of echo command OK"
    (is (= (exec-cmd {:bin "echo", :args '("hello" "world!"), :config {:seq true}})
           '("hello world!")))))
Any idea how can I make the test pass?, I can make the function to use echo by using apply but I need to pass extra configuration for the programs binded by Conch (that's why in the test case I'm passing a {:seq true})

Sergio23:10:07

by the way, the test fails with:

FAIL in (test-echo-cmd-exec) (core_test.clj:7)
Execution of echo command OK
expected: (= (exec-cmd {:bin "echo", :args (quote ("hello" "world!")), :config {:seq true}}) (quote ("hello world!")))
  actual: (not (= ("") ("hello world!")))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
Tests failed.

noisesmith23:10:21

@sdmoralesma ymmv but I find conch to be over engineered and I prefer to use ProcessBuilder / Process directly:

user=> (let [b (ProcessBuilder. ["echo" "hello"])
             p (.start b)
             res (.getInputStream p)]
         (.waitFor p)
         (slurp res))
"hello\n"
*reformatted to be more readalbe

Sergio23:10:01

thank you for the answer, I'll give it a try, also I noticed that is possible to pass a folder for execution which helps in my case. 👍

noisesmith23:10:33

yeah, the API is pretty nice actually, especially if you already know the posix exec api

noisesmith23:10:50

except for the part where it doesn't let you know the PID :/

noisesmith23:10:48

using the inputStream and outputStream together you can even run an interactive program, which is much harder to do in most languages

Sergio23:10:20

with the PID at least is not an issue for me... yet. About the streams, that might improve the workflow... after all the idea is to simplify my day, thanks for the hint

alexmiller23:10:58

In Java 9 they added a .pid() method

❤️ 1
👍 1