Fork me on GitHub
#clojure
<
2022-05-30
>
jumar03:05:55

I'm wondering why pop throws an exception when given an empty vector/list but doesn't throw on an empty queue. Why not just return an empty vector/list?

jumar04:05:19

I read about this in Joy of Clojure a long time ago and looked today at clojuredocs and Clojure Standard Library book (section 12.3) but I think none of them explains why that's the case.

potetm04:05:15

public PersistentQueue pop(){
	if(f == null)  //hmmm... pop of empty queue -> empty queue?
		return this;
	//throw new IllegalStateException("popping empty queue");

Eugen07:05:49

how should I use tools.cli with :multi AND :default ["a" "b" "c"] ? Currently it adds the supplied values to the default ones instead of replacing them. I have this issue here https://github.com/l3nz/cli-matic/issues/153#issuecomment-1140808736

Al Z. Heymer10:05:39

isn't there something like dissoc-ns which lets you dissoc all keys of a map, that are prepended with this namespace? And if not, wouldn't that actually be kind of essential to work with namespaced keywords in a map? Our current solution to this looks like

(into {} (remove #(= "namespace" (namespace (first %))) params))
It's so heavy with the stringification and string comparison, I just highly doubt this to be a desired feature. Sure, one could just dissoc the hell out of params, by determining the keys, but that still leaves a string cast of the namespace.

Ben Sless10:05:00

Why remove and not select?

Al Z. Heymer12:05:17

That's not an answer to my question, but if you need to know, the intention is different: remove retains, select filters.

reefersleep12:05:43

I don’t think there is. Your solution seems OK to me, except maybe I’d use val rather than first. What’s wrong with string comparison?

reefersleep12:05:53

Medley has a bunch of good stuff. You could use something like remove-keys, so

(medley/remove-keys 
  (comp #{"namespace"} namespace) 
  params)

Al Z. Heymer12:05:42

Thanks for the alternatives. I guess namespaced keywords really do lack a lot of support yet 😕 I mean clojure.lang.Named already provIdes .getNamespace()namespace. Why not utilize that with some tooling?

Ben Sless13:05:13

I tried to redirect the question because a positive is always easier than a negative; why do you care a map contains those keys to begin with?

seancorfield17:05:19

@U02PS2BNULE My response would be "Why are you trying to remove that group of keys?" The idiomatic approach in Clojure is for code to simply ignore keys it doesn't care about so, in general, additional keys are just left in data structures and carried around in maps.

seancorfield17:05:20

Two use cases where additional keys matter: writing a hash map to a DB (e.g., with next.jdbc) and you know the possible keys so select-keys makes sense there (because you don't want anything except the allowable subset of columns); producing JSON to interact with something external -- and again, in general, you know what the allowable subset is so select-keys makes sense there too. So I'm curious what your use case is here?

Al Z. Heymer08:05:42

@U04V70XH6 our use case is to convert that map to transit/json, then put it on the wire to another service. So, yes, in principal, we could ignore the keys here and give it to another service, however the other service should not have knowledge about these params. In our case, they provide some kind of C&C in our microservice framework. params and payload come in on the one side, get consumed by the Clojure microservice and transform the payload. Cleaned params and transformed payload are pushed to the next service. So yes, the service selects its params by resolving the namespaced keywords, and eventually merging default values. So theoretically this could be refactored to use plain unnamespaced keys assoced to several sub-maps (to dissoc :service-key), but for reasons unknown to me, we went this way. And I have to admit, I am kind of puzzled as to why namespace http://e.ga whole map, but not namespace (which is set-separation by namespace) several sub-maps.

Al Z. Heymer08:05:04

So I guess this may be a design decision to encourage to "ignore" namespaces?

Al Z. Heymer09:05:10

I wonder what your thoughts on our solution are, and am curious about how you would solve such a problem 👍

rolt09:05:49

what do you mean by string cast ? namespace is just an attribute of a keyword no ? do you really have performances problem on this part ? I would assume the transit and the transport performances would be the bottleneck anyway

Al Z. Heymer09:05:14

@U02F0C62TC1 yeah, I looked that up in Java and figured I was mistaken 😅 nvtl I feel there is some missing tooling

seancorfield16:05:18

JSON libraries will throw away the qualifier for you (either by default or under an option, depending on which library you use). I don't know about Transit libraries. You're right that you specifically need to isolate just the params that will go into the JSON so structuring the hash map internally so that the payload remains under a single key, allowing other keys alongside it, would have made your life easier. Right now, you're going to be filtering that out yourself. Personally, I'd use reduce-kv because I think that's a bit cleaner:

(reduce-kv (fn [m k v] (if (= prefix (namespace k)) m (assoc m k v))) {} params)
In general, when working with external systems, there would be some sort "adapter" code mapping from your internal domain model to the external system's data model, and for some of those systems this might be simpler -- closer to identity -- and for other systems it could potentially be quite elaborate. I don't think a single reduce-kv call that filters based on namespace qualifier is a big deal, personally. At work we have code that systematically creates entire new data structures in some cases because the output format is some nested object with weirdly-named fields and some transformed values -- but at least we get to keep our nice, clean domain model internally.

Al Z. Heymer08:06:43

I see. Actually we use spec for domain modeling, but only in a light sense unfortunately. And you're right, that this is no big deal, actually. I do catch myself sometimes trying to think about optimization where it's not that important. I guess that's one of these cases. But I see what you mean by adapter code. Our domain model only specifies the data types, not a mapping. So, yeah. That's definitely something missing here. I will keep that in mind for the next round of design discussion. This also would solve my next concern: Removing C&C keys inside the application is the one deal here. There is some missing encapsulation of application and domain model here. Preparing application data inside the application is one thing, but i feel the control-structure shouldn't be modified from within.

Al Z. Heymer08:06:16

May I ask what you use internally for the inter-process communication? If I read that correctly, you use JSON as the data carrier. What kind of Messaging carries the JSON?

seancorfield17:06:52

@U02PS2BNULE We have two specific inter-process approaches: one is simple REST with JSON payloads, the other is pub/sub via Redis (and I believe we use Nippy for encoding data structures there -- but I'm pretty sure we have wrapped it up in core.async so we can just treat it as input/output channels with regular Clojure data... I haven't had to deal with the nuts and bolts of that piece much!).

emccue14:05:21

I found this form submit on sonatype where you can upload a package as one large jar. This is now the way. Gonna script it

emccue17:05:01

import java.net.CookieManager;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;

public final class Build {
    static final String VERSION = "0.0.1";

    static void run(List<String> cmd) throws Exception {
        System.out.println(cmd);
        int exit = new ProcessBuilder(cmd)
            .inheritIO()
            .start()
            .waitFor();
        if (exit != 0) {
            System.exit(exit);
        }
    }

    static List<String> glob(String pattern) throws Exception {
        var matcher = FileSystems.getDefault().getPathMatcher("glob:" + pattern);
        try (var f = Files.walk(Path.of("."))) {
            return f.filter(matcher::matches).map(Path::toString).toList();
        }
    }

    static void clean() throws Exception {
        run(List.of("rm", "-rf", "target"));
    }

    static void compile() throws Exception {
        var javacCmd = new ArrayList<>(List.of("javac", "-g",  "--release", "17", "-d", "target/classes"));
        javacCmd.addAll(glob("./src/*.java"));
        javacCmd.addAll(glob("./src/**/*.java"));
        run(javacCmd);
    }

    static void doc() throws Exception {
        var javadocCmd = new ArrayList<>(List.of("javadoc", "-d", "target/doc"));
        javadocCmd.addAll(glob("./src/*.java"));
        javadocCmd.addAll(glob("./src/**/*.java"));
        run(javadocCmd);
    }

    static void jar() throws Exception {
        compile();
        doc();
        run(List.of("mkdir", "target/deploy"));
        run(List.of("jar", "cf", "target/deploy/async-%s.jar".formatted(VERSION), "-C", "target/classes", "."));
        run(List.of("jar", "cf", "target/deploy/async-%s-javadoc.jar".formatted(VERSION), "-C", "target/doc", "."));
        run(List.of("jar", "cf", "target/deploy/async-%s-sources.jar".formatted(VERSION), "-C", "src", "."));
    }

    static void publish() throws Exception {
        jar();
        Files.writeString(
            Path.of("target", "deploy", "async-%s.pom".formatted(VERSION)),
            """
            <?xml version="1.0" encoding="UTF-8"?>
            <project xmlns=""
                     xmlns:xsi=""
                     xsi:schemaLocation=" ">
                <modelVersion>4.0.0</modelVersion>
                
                <groupId>dev.mccue</groupId>
                <artifactId>async</artifactId>
                <version>%s</version>
                <packaging>jar</packaging>
                
                <name>Async Utilities</name>
                <description>A few helpers for async code.</description>
                <url></url>
                
                <licenses>
                  <license>
                    <name>The Apache Software License, Version 2.0</name>
                    <url></url>
                  </license>
                </licenses>
                
                <developers>
                  <developer>
                    <name>Ethan McCue</name>
                    <email>[email protected]</email>
                    <organization>McCue Software Solutions</organization>
                    <organizationUrl></organizationUrl>
                  </developer>
                </developers>
                
                <scm>
                  <connection>scm:git:</connection>
                  <developerConnection>scm:git::bowbahdoe/java-async-utils.git</developerConnection>
                  <url></url>
                </scm>    
            </project>
            """.formatted(VERSION)
        );

        run(List.of("gpg", "-ab", "target/deploy/async-%s.pom".formatted(VERSION)));
        run(List.of("gpg", "-ab", "target/deploy/async-%s.jar".formatted(VERSION)));
        run(List.of("gpg", "-ab", "target/deploy/async-%s-javadoc.jar".formatted(VERSION)));
        run(List.of("gpg", "-ab", "target/deploy/async-%s-sources.jar".formatted(VERSION)));
        run(List.of("jar", "-cvf", "target/bundle.jar", "-C", "target/deploy", "."));

        var httpClient = HttpClient.newBuilder()
                .cookieHandler(new CookieManager())
                .build();

        var scanner = new Scanner();
        System.out.print("Sonatype Username: ");
        var username = scanner.next();
        System.out.print("Sonatype Password: ");
        var password = scanner.next();

        var loginResponse = httpClient.send(
                HttpRequest.newBuilder()
                        .GET()
                        .uri(URI.create(""))
                        .header("Authorization", "Basic " + Base64.getEncoder()
                                .encodeToString((username + ":" + password).getBytes(StandardCharsets.UTF_8))
                        )
                        .build(),
                HttpResponse.BodyHandlers.ofString()
        );

        if (loginResponse.statusCode() / 10 != 20) {
            System.err.println("Invalid login");
            System.exit(1);
        }

        // {"repositoryUris":[""]}
        var boundary = "-------" + UUID.randomUUID();
        var uploadBody = """
                --%s
                Content-Disposition: form-data; name="file"; filename="bundle.jar"\r
                \r
                Content-Type: application/java-archive\r
                \r
                %s\r
                --%s--""".formatted(
                        boundary,
                Files.readString(Path.of("target/bundle.jar"), StandardCharsets.ISO_8859_1),
                boundary
        );
        var uploadResponse = httpClient.send(
                HttpRequest.newBuilder()
                        .POST(HttpRequest.BodyPublishers.ofByteArray(uploadBody.getBytes(StandardCharsets.ISO_8859_1)))
                        .header("Content-Type", "multipart/form-data;boundary=%s".formatted(boundary))
                        .uri(URI.create(""))
                        .build(),
                HttpResponse.BodyHandlers.ofString()
        );
        System.out.println(uploadResponse);

        System.out.println(uploadResponse.body());
    }

    public static void main(String[] args) throws Exception {
        var options = "Options: clean, compile, doc, jar, publish";
        if (args.length == 0) {
            System.out.println(options);
            System.exit(1);
        }
        else {
            var cmd = args[0];
            switch (cmd) {
                case "clean" -> clean();
                case "compile" -> {
                    clean();
                    compile();
                }

                case "doc" -> {
                    clean();
                    doc();
                }
                case "jar" -> {
                    clean();
                    jar();
                }
                case "publish" -> {
                    clean();
                    publish();
                }
                default -> {
                    System.out.println(options);
                    System.exit(1);
                }
            };
        }
    }
}

emccue17:05:26

This actually works! - I guess it wouldn't be that hard to make a thing for deploying tools deps projects after all

javahippie19:05:11

I want to run a command with sh in a dir, but I am building the command dynamically. I want to do something like (sh "ln" "-l" :dir "/tmp"), but dynamically, e.g. (apply sh '("ln" "-l")), but right now I just don’t get how I can pass the options this way

p-himik19:05:33

Same way, no?

(apply sh ["ln" "-l" :dir "/tmp"])

javahippie19:05:51

Thanks, that’s it 👍 Not really sure where I went wrong :thinking_face:

p-himik20:05:49

If your "/tmp" is actually some x, then '() will turn it into a symbol - it won't be resolved.

javahippie20:05:19

That makes sense, thanks for clarifying

👍 1
borkdude20:05:54

@U0N9SJHCH Also check out https://github.com/babashka/process/blob/master/API.md#sh

(sh ["ln" "-l"] {:dir "/tmp"})
or:
(sh "ln -l" {:dir "/tmp"})

👍 1
p-himik20:05:01

That actually always bothered me. The sh signature, I mean. I guess the Clojure implementation is just to make it... easier to type? At the cost of everything else.

eighttrigrams21:05:13

Is it possible to find out during macroexpansion if a function returns another function? Basically if have a macro (defmacro abc [fn-symbol] «here i want to look into the function passed in as symbol») and then I have either (defn a [] (fn [] ...)) or (defn b [] ...) and I want to find out if something of the form a or of the form b has been passed in (call example (abc a) or (abc b) ).

p-himik21:05:39

Not sure if I understand what you want exactly, but: • You can determine whether the symbol you pass into a macro resolves to a function • You cannot determine whether that function itself returns a function

eighttrigrams21:05:14

Hm, I can understand that it would be difficult to determine if a function returns a function in general. In the case I have in mind it would be that the function returned is exactly at the beginning of the outer functions definition. I thought, in the sense of code is data, that I might be able to "peek into" the function's code somehow

p-himik21:05:05

Code is data only when you have that code. If you pass a full function definition to a macro and you have a convention on where that inner function is - sure, you can determine if there is such an inner function.

eighttrigrams21:05:28

👀 oh, ok. So I am still not sure, but maybe it would work then. I have two constraints. 1. I pass the function in as a symbol. The function is there, but I know that I pass in a symbol, not the body itself. 2. The structure of the "referenced" function I know, it either has the (fn part directly after the param list, or it hasn't. You think this could be made to work?

p-himik21:05:41

> The function is there It can be there as just the bytecode, without the function definition. But if the source is also there, you can check out clojure.repl/source. All in all, given that you aren't sure and you're trying to use macros, I'd say it's worth it to look for different ways to achieve whatever you want to achieve. :)

👍 1
eighttrigrams21:05:03

Yeah, I am sure what I want to achieve in the end, but you're right, I am not totally sure about the proper way to achieve it and maybe look at one way to hard. Thank in any case!

👍 1
eighttrigrams21:05:00

What I already can see is that it has nothing to do with macros per se. It's more about the bytecode and source issue

seancorfield21:05:36

In the general case, you're not going to be able to tell that a given symbol resolves to something that, when used like a function, returns something that behaves like a function. Remember that hash maps and sets can be used like functions, as can keywords and symbols. "Function-like" things in Clojure aren't just actual functions.

seancorfield21:05:52

And you can only check the type of an expression (e.g., by calling ifn? or fn?) at runtime when you have a value. Macroexpansion happens separately before runtime, so value-based (or type-based) stuff can't be done inside macros.

eighttrigrams21:05:07

Ok, hm, so what already works is

(defn a [] (prn "a") (fn [] (prn "b")))

(defmacro abc [name]
  ((ns-resolve *ns* name))
  ""); for demo purpose just return something

(abc a)
"a" ; output from (prn "a") at time of expansion
During macroexpansion my function that I pass in gets evaluated. So theoretically I could just do this and check if I the result is a function. But I want to avoid that call. Instead I thought it would be possible to inspect that code somehow. Its bytecode if necessary

seancorfield21:05:05

I'd take a step back at this point and ask: what problem are you really trying to solve?

😀 1
seancorfield21:05:42

In the general case, you cannot predict what type of value an arbitrary function is going to return -- because it might depend on the values of the arguments passed in.

seancorfield21:05:14

If you have a very constrained problem, with a known set of possible functions as input, and you know ex facto which ones return functions and which don't, then maybe encode that knowledge in pure data, along with some tests or assertions to ensure someone else doesn't violate those constraints...

eighttrigrams21:05:02

Ok. So: I want to build an api, basically just one macro. The consumer of that macro should call it like this

(defdispatch handler some-fn)
Now, some-fn should be allowed to come in in two flavors. 1. It is a function that returns a function. 2. It is a simple function. What the macro then would do is, depending if there is an "additional" param list (case 1) add in extra information, in the other case just ignore it.

seancorfield21:05:57

Two thoughts: 1) macros make terrible APIs because they don't compose 2) force the caller to tell you which type of function it is -- either by passing some additional information or using a different def-something.

👍 1
seancorfield21:05:49

Write the API using regular functions first. Only add macros if the API is hard to use without adding a DSL on top of it.

👍 1
seancorfield21:05:07

By which I mean something like

(defn register-dispatch! [..] ..)

(register-dispatch! handler some-fn) ; or (register-dispatch! 'handler some-fn) if handler must be a name/symbol

seancorfield21:05:50

Once you have that working, then defdispatch can be added as a wrapper macro to avoid quoting the handler name if appropriate.

eighttrigrams21:05:33

Ok, I understand. And just telling register-dispatch! if some-fn is of type a or b would definitely be an option

eighttrigrams22:05:00

I wanted to have this

(defdispatch handler some-fn-1 some-fn-2 ...)

eighttrigrams22:05:19

that it can take more of those functions, and figure out on its own which one is of which type

eighttrigrams22:05:22

I could do of course

(defdispatch handler [some-fn-1 :type-a] [some-fn-2 :type-b] ...)
or something similar.

eighttrigrams22:05:03

But maybe as stated above, maybe right now I am trying a bit to hard 🙂