Fork me on GitHub
#clojure
<
2021-05-02
>
Jim Newton09:05:16

what is the best way to understand the different kinds of function definition argument lists and the corresponding call-site argument lists? Simple function definitions with a fixed number of named arguments are easy. But in Clojure, function application also allows exotic argument lists involving optional arguments and key arguments.

Jim Newton09:05:05

basically if I want to implement a macro which extends the semantics of fn what are ALL the cases I need to cover as far as argument binding and syntax of call-sites ?

borkdude09:05:17

@jimka.issy There is really one "special" thing in the arg list, which is & which separates the required (fixed number) args from the variadic args.

borkdude09:05:59

These variadic args are always received as a sequence. But as a convenience you can use destructuring in the arglist, same as in other places like let.

Jim Newton09:05:14

I’m referring to the recent @didibus comment concerning an argument list like [{:keys [^Int bar]}]

borkdude09:05:17

So when you have x y & {:keys [a b]} and you pass 1 2 :a 1 :b 2: x = 1, y = 2, rest = (:a 1 :b 2) which is destructured

Jim Newton09:05:25

great! so are those rules stated anywhere, or does the programmer (me) have to understand lots and lots of interrelated concepts to understand it?

borkdude09:05:59

Clojure has functions like destructure and maybe-destructured (https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/clj/clojure/core.clj#L4504) which are very helpful for re-implementing fn

borkdude09:05:32

There are only two interrelated concepts here: variadic arguments and destructuring

Jim Newton09:05:24

what is wrong with this function invocation?

((fn [& {:keys bar}]
   (list bar)) :bar 3)

borkdude09:05:07

{:keys [bar]}

Jim Newton09:05:13

OIC, the missing [] yes indeed

Jim Newton09:05:25

so apparently this syntax [& {:keys [bar]}] allows any number of :bar value pairs at the call site, as well as any number of other unmentioned keys. The right-most value associated with an :bar is bound to the bar variable within the function.

Jim Newton09:05:52

what if I want to disallow repeated keys and disallow unmentioned keys?

Jim Newton09:05:09

in Common Lisp I can specify &allow-other-keys but I cannot preclude repeated keys

borkdude09:05:35

Clojure has an open world philosophy where additional other keys are usually not warned against

borkdude09:05:28

This is where libraries like spec come in. spec2 will have an option to specify this

borkdude09:05:42

Although at this point it's unclear if and when spec2 will see the light of day

borkdude09:05:20

Libraries like malli or prismatic/schema also support this

Jim Newton09:05:54

Not sure whether you’ve seen the post by @didibus, but which of the following would be applicable ?

(dsfn foo
 ([& {:keys [^Int bar]}]
  (list 'int 'bar bar))
 ([& {:keys [^Int baz]}]
  (list 'int 'baz baz))
 ([& {:keys [^String bar]}]
  (list 'string 'bar bar)))
to a call site such as (foo :baz 42) ? But a literal interpretation, the first would be applicable because unmentioned keys are silently ignored, but that’s not what the caller would expect.

didibus10:05:47

In standard Clojure JVM you're not allowed more than one variadic overload, so normally that scenario would throw:

CompilerException java.lang.RuntimeException: Can't have more than 1 variadic overload

didibus10:05:34

But that's similar to how you're also not allowed two overloads of the same arity. With your macro though, since it can dispatch on type, for this example I do think the expectations would be it calls the second one.

didibus10:05:10

And if someone called it like so (foo :other 10) then I'd expect the first one to apply. Since your macro does already prioritize the first match left to right.

didibus10:05:20

And I'd expect the third one if someone typed (foo :bar "hello")

didibus10:05:42

And finally if someone typed: (foo :other "hello") I'd expect the first one again.

yuhan11:05:52

I'm nowhere of an expert on language/library design but I think this is what Rich Hickey meant in Simple Made Easy when talking about pattern matching and switch statements being sources of complexity - you can have powerful rules for dispatching different behavior, but having separate semantics of arity, type, key inclusion all mixed together in a single spec leads to these sort of puzzles and edge cases

yuhan11:05:20

Where the person reading the code has to basically draw an inheritance chart or run a type inferencer in their heads to figure out what their code does, and the notion of "what the caller expects" may not always be clear

3
Jim Newton14:05:18

why shouldn’t the second apply if you typed (foo :bar "hello") This seems to be implied by your suggesting that the first should apply on (foo :other 10) , what’s the difference between the two cases?

Jim Newton15:05:08

Imagine that the two ladder cases were not there:

Jim Newton15:05:34

(dsfn foo
 ([& {:keys [^Int bar]}]
  (list 'int 'bar bar)))
In this case the clause would match if given (foo :other 10), right. The idea is that the first possible match is taken. Adding clauses later on won’t make something match a later clause if it already matches an earlier one. That’s how pattern matching works.

Jim Newton15:05:10

The way I’ve implemented this is I’ve added an additional keyword :allow-other-keys which can be used as follows

(dsfn foo
 ([& {:keys [^Int bar] :allow-other-keys true}]
  (list 'int 'bar bar)))
This means that the clause is allowed to match even if there are other keys at the call site which are not listed in :keys […]

Jim Newton15:05:59

:allow-other-keys defaults to false, so the clause won’t match if there are keys at the callsite not mentioned in :keys [ … ].

didibus00:05:41

(dsfn foo
  ([& {:keys [^Int bar]}] 100)
  ([& {:keys [^String bar]}] "100"))
If I call (foo :bar "hi") will it pick the first or second?
(dsfn foo
  ([^Int bar] 100)
  ([^String bar] "100"))
Now if I call (foo "hi") will it pick the first or second? In my opinion both should behave the same, which would be that they match the second in both case.

didibus00:05:24

I also think most of the time in Clojure allowing other keys might be best, that's the default everywhere and it's quite common to call something with a map that has more keys than what the function needs. Especially with Clojure 1.11 where both (foo :bar "hi") and (foo {:bar "hi"}) is a valid call to the function.

borkdude09:05:31

It depends on the rules of your dsfn macro, I guess you are the master of your macro

Jim Newton09:05:10

Yes, I think that the caller/programmer WOULD NOT EXPECT normal clojure evaluation rules to apply.

borkdude09:05:55

In practice, I think the signature of that function would be unusual: one branch is interested in one key, but another branch is interested in only another key with no overlap. Usually you have a common set of required keys + some optional ones.

Jim Newton09:05:55

on the other hand, I think there are other specifiers other than :keys which can be used in this syntax. right?

borkdude09:05:33

(dsfn foo ([& {:required [^String bar]}]))

borkdude09:05:48

you can make up whatever you want in macros, although this requires custom destructuring logic

borkdude09:05:18

I guess you could merge the commonalities into one arity, destructure and then check

✔️ 3
Jim Newton09:05:28

Somewhere I say some extra syntax for inside these braces. syntax that provides default values for example. Is there a section about destructuring in https://clojure.org/reference

Jim Newton13:05:13

@didibus I have a question for you about a user’s intuition w.r.t. destructuring keyword argument lists

Jim Newton14:05:17

You commented that you would like dsfn to be able to handle something like the following.

(dsfn 
 ([& {:keys [^Int bar]}]
  (list 'int 'bar bar))
 ([& {:keys [^Int baz]}]
  (list 'int 'baz baz))
 ([& {:keys [^String bar]}]
  (list 'string 'bar bar)))
The question is whether the pattern matcher should allow other keys or not. For example if the call-site argument list is (:bar 1 :xyzzy 2) should the first clause match or be rejected. The matcher can be written to have either behavior. In Common Lisp, there is a syntax for determining which behavior you want. I considered using a syntax like the following:
(dsfn 
 ([& {:keys [^Int bar] :allow-other-keys true}]
  (list 'int 'bar bar))
 ([& {:keys [^Int baz] :allow-other-keys false}]
  (list 'int 'baz baz))
 ([& {:keys [^String bar]}]
  (list 'string 'bar bar)))

didibus02:05:15

In Clojure I think being open by default is a better default.

Jim Newton13:05:20

As I understand Clojure is in general permissive about ignoring information it doesn’t care about. Why does the following form trigger an error? Shouldn’t it simply ignore the unrecognized key/value pair?

((fn [& {:keys [bar] :ignore-other-keys true}]
                        (list bar)) :bar 3)

Jim Newton13:05:52

why am I disallowed from putting extra keys in this map?

ghadi13:05:05

When something triggers an error, as a rule you must post the error :)

andy.fingerhut13:05:19

I wouldn't say Clojure is always permissive about ignoring extra things in syntax. If you are getting a bunch of error messages when you try to do that beginning with "Syntax error macroexpanding clojure.core/fn at ..." then that is from the spec syntax checking of some macro invocations, in this case fn, that was added in Clojure 1.9.0

andy.fingerhut13:05:02

Another example in a different area:

user=> (if true 5 7 8)
Syntax error compiling if at (REPL:1:1).
Too many arguments to if

Jim Newton13:05:30

why should fn care if there’s an entry in the map which it doesn’t care about?

Jim Newton13:05:52

Of course I can filter it out before I expand to the call to (fn …)

andy.fingerhut13:05:17

Probably because Clojure's destructuring facilities would never use that for anything, and it is likely a sign of a bug in the user's program that they are passing syntax that the Clojure compiler would ignore completely.

ghadi13:05:31

Post. The. Error

andy.fingerhut13:05:31

Clojure 1.10.1
user=>       ((fn [& {:keys [bar] :ignore-other-keys true}]
                        (list bar)) :bar 3)
Syntax error macroexpanding clojure.core/fn at (REPL:1:2).
:ignore-other-keys - failed: simple-symbol? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :map-binding 0 :local-symbol] spec: :clojure.core.specs.alpha/local-name
:ignore-other-keys - failed: vector? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :map-binding 0 :seq-destructure] spec: :clojure.core.specs.alpha/seq-binding-form
:ignore-other-keys - failed: map? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :map-binding 0 :map-destructure] spec: :clojure.core.specs.alpha/map-bindings
:ignore-other-keys - failed: map? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :map-binding 0 :map-destructure] spec: :clojure.core.specs.alpha/map-special-binding
:ignore-other-keys - failed: qualified-keyword? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :qualified-keys-or-syms 0] spec: :clojure.core.specs.alpha/ns-keys
true - failed: vector? at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :qualified-keys-or-syms 1] spec: :clojure.core.specs.alpha/ns-keys
:ignore-other-keys - failed: #{:as :or :syms :keys :strs} at: [:fn-tail :arity-1 :params :var-params :var-form :map-destructure :special-binding 0] spec: :clojure.core.specs.alpha/map-bindings
{:keys [bar], :ignore-other-keys true} - failed: simple-symbol? at: [:fn-tail :arity-1 :params :var-params :var-form :local-symbol] spec: :clojure.core.specs.alpha/local-name
{:keys [bar], :ignore-other-keys true} - failed: vector? at: [:fn-tail :arity-1 :params :var-params :var-form :seq-destructure] spec: :clojure.core.specs.alpha/seq-binding-form
& - failed: vector? at: [:fn-tail :arity-n :params] spec: :clojure.core.specs.alpha/param-list

andy.fingerhut13:05:53

Sorry, that was long enough I probably should have put it in a thread comment

ghadi13:05:09

1. this error represents clojure.spec checking the valid grammar of params and destructuring forms

ghadi13:05:07

2. these specs get checked at macroexpansion time for certain macros (defn/fn/let)

ghadi13:05:26

3. it returns a mouthful because of ambiguous parsing

ghadi13:05:45

and it shows the potential ways something could be short/long/incorrect

Jim Newton13:05:47

yes, @ghadi, the question is why doesnt the syntax check allow me to put a key/value in the hash that it doesn’t care about, and that hurts nothing?

ghadi13:05:56

because it does hurt something

ghadi13:05:07

it's an invalid map destructuring pattern

ghadi13:05:36

{binding key} <- is valid map destructuring

Jim Newton13:05:38

:allow-other-keys` means nothing to fn, so it would be nice if the syntax checker simply ignored it

ghadi13:05:45

binding = symbol, and you've provided a keyword

borkdude13:05:56

ad 2. for every macro I think, it's all or nothing, can be disabled using a system property (or dynamic var?)

ghadi13:05:43

let's say the invalid syntax was accepted, who/what would act on that flag?

andy.fingerhut13:05:04

I believe it would be possible to write your own macro, similar to but not the same as fn or defn that are built in, that do different syntax checking than they do.

Jim Newton13:05:21

@ghade, if nothing would act on it then it would be harmless. The idea is that I have a macro which expands to fn, my macro understands :allow-other-keys and I’d like to simply pass along my map to fn without having to rebuild it.

Jim Newton13:05:43

of course I can code walk the expression and remove :allow-other-keys, it just seems unnecessary

andy.fingerhut13:05:01

In this case, the Clojure compiler / macro-expander is trying to be helpful to developers who misunderstand the correct kinds of destructuring forms that are allowed, and letting them know that they did something useless, which is likely a bug in their code.

borkdude13:05:14

I think you should code walk as it will trip up people's programs otherwise. In these very fundamental building blocks it's actually nice that clojure has some sanity checks (although the output can be overwhelming to some people)

andy.fingerhut13:05:25

Jim, you started this discussion with the sentence "As I understand Clojure is in general permissive about ignoring information it doesn’t care about." I tried to reply that this is not a general rule that applies to everything in Clojure. Yes, there are some things it will ignore, but there are others where it checks and gives errors/warnings.

✔️ 3
Jim Newton14:05:37

yes, I’ll just filter it out before passing the map on to the clojure primitive.

borkdude13:05:57

It applies to data in/out but not so much to fundamental syntax like defn or ns

andy.fingerhut13:05:55

I'm sure there are several open JIRAs with requests to tighten up checking even more than it is checked now. The kinds of checks done aren't quite static, but they do not change rapidly across Clojure versions, either. Clojure 1.9 introduced quite a bit of additional syntax checks like this.

yuhan13:05:45

I'm glad that Clojure throws a compile-time error for that particular case - 99% of the time I destructure using :keys, and the other 1% I'll often accidentally reverse the order of the bindings just because having the keyword on the left is more familiar:

(let [{:a a} {:a 1}]
  (inc a))
so it makes sense to prevent this sort of user mistake up front by throwing an error

zendevil.eth21:05:06

I want to send an http request to my localhost:3000 server from a custom ip address. And apparently this can be done with a custom dns resolver.

(client/get "" {:dns-resolver (doto (InMemoryDnsResolver.)
                                                                              (.add "localhost" (into-array[(InetAddress/getByAddress (byte-array [127 0 0 1]))])))})  
and nothing in the byte-array but 127 0 0 1 is getting evaluated in the repl. I want to eventually run a sequence of multiple requests to simulate multiple clients making those requests.

p-himik21:05:01

A custom DNS resolver is supposed to resolve localhost as something else. You're trying to resolve it as 127.0.0.1, which should already be done by your system. I don't think you can properly simulate multiple incoming IP addresses this way, although I'm not an expert here.

Crispin01:05:41

talking TCP: after constructing the socket on the client side you need to bind the socket to the source IP you want before you connect. If your http client doesn't support that, you will need to modify it, or use one that does. https://linux.die.net/man/2/bind

Crispin01:05:16

But AFAIK you can only bind to a source IP that is bound to a local interface. You can't spoof any old address. For that you would need some kind of raw packet injection.