Fork me on GitHub

ah, i think its :integratedSecurity


@qmstuart You can pass any properties that the JDBC driver understands -- so you won't find them in the docs specifically, but there is an explanation of passing additional properties to the driver as part of the db-spec hash map...

Claudio Ferreira01:01:51

Clojurians, what are we really doing here when we " { :keys [ minimum maximum ] } " as a parameter inside that function? I became lost here


That is destructuring a map into bindings by specifying the names of the keys in the map. So you function expects a parameter that is a map that contains the keys minimum and maximum After the destructuring you can just use the names minimum and maximum rather than having to do something like (:minimum your-map) e.g. Say you have a map

{:foo 5 :bar 6}
And a function
(defn foo [{:keys [foo bar]}]
   (prn foo bar))
ANd calling it with our map
(foo {:foo 5 :bar 6})
5 6 
=> nil
If foo didn't destructure it's map argument, I would have to do something like this
(defn foo [my-map]
   (prn (my-map :foo) (my-map :bar)))
See here,, you want the section on associative destructuring.

Claudio Ferreira01:01:10

HMMM, thank you @qmstuart! And when we use ":as" after it? Are we binding the keys to next element? e. g. Acc in the next image?

Matthew Curry02:01:02

:as allows you to also bind the entire map as well, in addition to the key names specified


That is so you can still reference the entire map. So in that method your map argument is called acc, and its keys are current-bag and bags. The method also has a second parameter called stream. Im assuming this is a method to pass to reduce.


Also worth noting: if you have :or to set a default value, it binds the symbol to that value but the :as symbol is bound to the original data.


(let [{:keys [a b] :as data :or {a 1 b 2}} {:a 42 :c "Sea"}] (prn a b data))
42 2 {:a 42, :c "Sea"}


So b is bound to 2 -- the default -- but data still doesn't contain b.

Claudio Ferreira15:01:08

Thanks all. Finally i've understood that!


Quick (?) question: what idioms should I be using instead if I'm planning to use map inside another map function


@zackteo For some problems, that might well be the right solution. Can you provide some context?


Give me a moment 🙂


So I current have a list of sheets like ( "Sheet1" "Sheet2" ) and a list of lists of java objects (( [A1:C1] [A2:B3] [C2:C3] )( [A1:C1] )) and I want to transform it into a format like so ...

{[0 0 "Sheet1"] {:lrow 0, :lcol 2},
 [1 0 "Sheet1"] {:lrow 2, :lcol 1},
 [1 2 "Sheet1"] {:lrow 2, :lcol 2},
 [0 0 "Sheet2"] {:lrow 2, :lcol 2}}
where I can grab the numbers using .getFirstRow .getFirstColumn .getLastRow .getLastColumn on the java objects.


And currently I have

#(hash-map (vector (.getFirstRow %)
                   (.getFirstColumn %))
                   :lrow (.getLastRow %)
                   :lcol (.getLastColumn %)))
which works on a list of java objects (not a list of list) to produce
{[0 0] {:lrow 0, :lcol 2},
 [1 0] {:lrow 2, :lcol 1},
 [1 2] {:lrow 2, :lcol 2}}


Sorry about the mess. I'm trying to figure out how to best approach the problem


Can you explain what the actual problem is that you're trying to solve?


Am trying to add merged cells into by adding :lrow and :lcol into :coord when it is a merged-cell . Which should seem easy to do. But the only/best access point to find out about merged-cells seems to be using getMergedRegions on the sheet


I think you're struggling because you can't explain the problem very well...


So I was thinking the way to approach it is by having a list of merged-regions in the let block of throwable-read-xlsx! which I then can feed into poi-cell->fxl-cell to check if the current cell (it is mapping through) is a merged cell and adding the right keys accordingly


You're too deep in the weeds of the implementation.


Take a few steps back, away from the code, and think about the data at a higher level.


I guess I this might still be implementation level but broadly my problem seems to be a 2 step one 1. Get a hash-map which I can use to identify which cells are merged-cells. (which also contains the extra parameters I want, to make a cell a merged-cell) 2. Update the list of cells based on that hash-map


That's still implementation. You haven't actually described the problem yet.


"i have a workbook with two spreadsheets in them. i would like to ..."


Right now I have no idea what you mean by "merged-cells" and what "extra parameters" means, even in that context.


Hmmmmm :thinking_face: I wondering if this is too high level. I am adding merged-cell reading capabilities to an excel read/write library (that wraps apache poi) Merged-cells are a combination of multiple cells but only display the value of the top leftmost one


A merged cell is a range/region of cells. So a cell typically only has 1 pair of coordinates. I will need the other pair to give me the region


Like cell A1 (:row 0 :col 0) to merged-cell A1:B2 (:row 0 :col 0 :lrow 1 :lcol 1)


Maybe I was trying to jump steps by considering multiple sheets, without successfully handling a single sheet first

Yang Xu07:01:15

Hi, Who knows the best way to invoke HoneySQL from java? or some suggestions?

Yang Xu09:01:56

How to solve if I face the ’macros/usetime‘ functions or some special logics in the Clojure, like macro ... etc. Should I only resolve it using Clojure?And before I invoke something from Java I need to use the Clojure wrap it?


you can use the clojure compiler to load and run clojure library code, it will handle all of those things via its own read and eval steps


here's a small example I made, that loads up a namespace and runs it when a java daemon starts:

noisesmith17:01:07 handles all the setup, compilation etc. you just need to use the require method and the invoke method on any resulting functions you want to use

Yang Xu09:01:18

Thank you.


Hi guys, need some opinion if possible. Not 100% related to clojure. I want to do an excel-like grid at front end, but evaluate it (the formulas) at backend. I'm thinking of casting the formulas into variable assignments like so: `A.1 = 30 + 20` , concatenating them and sending them to the backend when the user clicks a button. So, the request would contain something like this (Assuming a 2x2 grid):

A.1 = nil;
A.2 = 10;
B.1 = A.2;
B.2 = B.1 + 50;
I can parse this using instaparse and have played with it a bit here (Incomplete so far and assignment is not fully handled yet) : I have two general questions: 1. Is this a reasonable way to handle this problem? 2. How should I handle Stuff like this: A.1 = A.2; A.2 = 40; Because at evaluation I need to evaluate the second statement first and then only the first. 3. How should I handle circular references? i.e. A.1 = A.2; A.2 = A.1; It is enough to identify and stop evaluation on encountering circular references.


@dumrat Can't you send a map of cells to their values instead? So something like {:A.1 nil, :A.2 10, :B.1 :A.2, :B.2 [:B.1 + 50]}?


@U013F1Q1R7G Yes I can. Still I'd have to parse the input because the users expect excel-like formulas. But I don't think this would solve #2 and #3 issues if I understand correctly.


To handle (2) and (3), I suggest building a topology graph to identify the leaves (e.g. A2 = 10) vs the internal nodes (e.g. B1 = A2) so you can start evaluating them in the correct order.


You can also detect loop while building topology graph iirc.


@U013F1Q1R7G Thanks. Do you have any suggested reading for building a topology graph? Not familiar with the concept.


Great, will check it out. Thanks!

Grigory Shepelev07:01:08

Hello there. Could you please help with understanding of the async lib ans subscriptions? Suppose I want to send https req to website every second and subscribe to it's result. And do some action each time as I get response. Suppose it's just printing the response's body. So having the following code does not produce desirable effect:

(ns async.core
  (:require [clojure.core.async
             :as async
             :refer [chan <!! <! >! >!! timeout pub sub unsub unsub-all go go-loop]]
             :as h]))

(def updates-channel (chan))

(def publishing
  (pub updates-channel :status))

(defn req [url]
  (h/get url {:async? true}
         (fn [res] 
           (go (>! updates-channel res)))
         (fn [_] nil)))

(defn listen!
  [publisher topic]
  (let [l (chan 1)]
    (sub publisher topic l)
    (go-loop []
      (when-let [{:keys [body]} (<! l)]
        (println (str "got incoming message: " "\n" body))

(listen! publishing :200)

(go-loop [seconds 1]
  (req "")
  (<! (timeout 1000))
  (recur (inc seconds)))
As far as I undestood that's because of double go-loop.

Grigory Shepelev08:01:10

I am an idiot. sorry. It should be:

(listen! publishing 200)

Yang Xu09:01:56

How to solve if I face the ’macros/usetime‘ functions or some special logics in the Clojure, like macro ... etc. Should I only resolve it using Clojure?And before I invoke something from Java I need to use the Clojure wrap it?


Hi folks, is there a fn to pop a key from a map, similar to dissoc but should also return the value of the key provided. e.g.

(pop-key {:a 1 :b 2 :c 3} :a)
;;=>[{:b 2 :c 3} 1]  

👍 3

nothing builtin


(juxt dissoc get)

🙌 3
😁 3
agata_anastazja (she/her)11:01:23

Hello, I am trying to find a way to write an acceptance test for my app. I want to verify it prints to the terminal the correct strings, but the last thing that the app needs to do is to exit. So I tried capturing it with with-out-str and leaving the program using System/exit 0, but it seems the program shuts down before any output is produced or captured. I reproduced a simpler version in repl, like so:

(defn print-many []
  (prn "here")
  (prn "there"))
=> #'user/print-many

(let [output (with-out-str (do (print-many)
                               (System/exit 0)))]
  (= output "here"))

Process finished with exit code 0
Any idea how I can test the correct output?


I think you are capturing the output of (System/exit 0) since that is the last form of your do.

agata_anastazja (she/her)13:01:33

but does it mean that if I want to exit the program there is no way for me to capture what it prints?

Eamonn Sullivan13:01:24

The way I've handled that in the past is to split out the code that puts the string together into a separate function and unittest that function. So that would mean changing print-many to return a string and not call prn directly. Just a thought. I try to keep (defn -main ...) very tiny, for just this reason.

💜 3

well, for one you're not flushing the output stream actually nevermind, prn does autoflush


invoke (flush) before exiting


and invoke System/exit after the =


you can't do any computation after the program has quit -- System/exit needs to be the last expression evaluated

👍 3
agata_anastazja (she/her)13:01:27

I guess the problem is that I use System/exit to close the program

agata_anastazja (she/her)13:01:41

and I want to run a test on a whole program


your program code should not invoke System/exit anywhere but in the top level

👍 3
agata_anastazja (she/her)13:01:19

the above is just a simplified version of how my app behaves


if you need to return from a place deep in your stack, consider using an exception


e.g in your app instead of having (if my-condition (continue-running) (System/exit 0)) consider (if my-condition (continue-running) (throw ..))

agata_anastazja (she/her)13:01:55

but the idea is that it prints something, and then when it’s done processing, quits


then at the very top level you can catch the exception and invoke System/exit


and in your tests you can capture the exception and return nil, so that you can capture the output

Alex Miller (Clojure team)14:01:47

in general, things like (System/exit) and (shutdown-agents) are world-ending kinds of calls and should be factored to the very outside of an application either by wrapping or via swappable callback so that you can test stuff without actually ending the world

👍 3

if I want to make this to a back-end of a website ( am I right I only need ring and cheshire so no framework


the bare minimum with ring is a web server backend (which only shows up directly in one line of code), ring core, and a router


the web server does the actual work, ring has an adaptor to that which makes it more clojurey, and the router decides which function to call for each request


the default back end is jetty, the default router is compojure


@roelof the ring wiki has an even simpler example, just ring and backend and no router - it's a "misbehaving" server with no routes, it just uses the same function for every request ignoring the route


I know I read that page several times and did some experiments


that is why I think things like luminus is a overkill for this back-end


looking at your code again, it looks like you are only interested in doing one thing - parsing a request URL to calculate a result - the issue you'd come up with is your function will get called with a lot of paths that won't make sense to that code


eg. browsers will ask for favicon.ico


with a router, you can have separate 404 response (or even a meaningful one), otherwise your function gets complex as it tries to handle all these unexpected strings


(or just blatantly fails for all unexpected input)


so maybe ring and computure ?


with compojure, yeah


first study those two again and try to make it work


@roelof this is a minimal compojure router, you can plug in any function that takes a ring request map and returns something ring knows how to use as a response


oke, I have not decided if the front-end will be hiccup or maybe something like reframe


with reframe your back end router will be managing requests from the cljs code for data used in rendering the page, with hiccup your back end router ends up calling code that generates the page, the router will look similar to what's above in both cases


but I will like i said study ringa nd compujure


using hiccup is easier for a first draft


before Im going to make my own code work


or even skipping hiccup and just returning some html string


for example


in the end I hope I can build a front-end like this :


but then without the movement


See how long that will costs me . maybe weeks or maybe months but that does not matter

Harley Waagmeester18:01:10

i can run lein uberjar just fine, but when do java -classpath ... target/snapshot.jar i get the error /tmp/form-init1109030110411510610.clj ( no such file or directory )

Harley Waagmeester18:01:31

i use the correct classpath


the uberjar task makes two jars


you are running the wrong one

Harley Waagmeester18:01:49

java -jar uberjar works fine


(this is because it treats making a jar for your app as a subtask for making the fat jar, and doesn't delete that pre-req)

Harley Waagmeester18:01:31

ok, i need to do lein jar ?


no, lein uberjar creates two jars, one of which is created by using lein jar


I'm just saying that when you get an error like that, the usual cause is running the regular jar instead of the uberjar


"correct classpath" with an uberjar is just one thing- the jar itself


it contains all other things that it needs

Harley Waagmeester18:01:07

i want to use the jar because it is smaller


but it's not smaller than all the separate classpath items you need to gather to make it usable

Harley Waagmeester18:01:40

the file is smaller, the system loaded into ram will be the same i imagine


the convention is to use uberjar for application deployment, and regular jars for libraries


otherwise you need to deploy and run a build tool as part of app deployment, which is not a good idea


The "skinny" jar is smaller because it only contains the files from your project. You want to use the "uber" jar that contains all the transitive dependencies as well.

Harley Waagmeester18:01:39

i usually use cider


cider is another thing you don't want to use when deploying an app


if you don't deploy, and everything is just cider usage with your editor, you don't need jar or uberjar at all

Harley Waagmeester18:01:16

but still i wonder why a trasient file in /tmp is required


that's an implementation detail of lein, it creates a bootstrap to load up your project

Harley Waagmeester18:01:44

yes, tmux, emacs, cider is my stack

Harley Waagmeester18:01:31

ah well.. i was just wondering


For development, that is fine. But for deploying, you (likely) just want an uberjar.


OK - it would be foolish to deploy an app for others to use that is run that way, but if you aren't doing that you can ignore jar / uberjar entirely

Harley Waagmeester18:01:11

this is code for my private web server, deployment from me to others would be source only


OK - running cider and emacs as a container for your app in a production environment is considered a problem. It can work but you can save yourself a lot of problems by just not doing that.

Harley Waagmeester18:01:40

the uber is 27M, that is a lot of man hours of coding

Harley Waagmeester18:01:20

lately i've been using the uberjar more often

Harley Waagmeester18:01:02

not my coding, other persons, it's really amazing

Harley Waagmeester18:01:38

sure, and now i'm looking at GraalVM, more candy to tempt me

Harley Waagmeester18:01:33

is it ok to run "lein snapshot.uberjar &", backgrounded is what i call it


one last question For a new project can I the best use leiningen or switch to the new kid clojure clj ?


For a new project I'd use clojure CLI over lein. But be aware that you may need other tools (or aliases) to build and run. See


@codeperfect If you have already created an uberjar that has a main function, then java -jar youruberjar.jar should work.


switching between the two is relatively easy, and most tutorials (especially with ring) specify lein commands for building / running, I'd recommend lein for a beginner


(unless you are following a tutorial or lesson that uses clj, then use that of course)


Hey all, I'm having a tough time wrapping my head about how to deploy a Clojure app for production. I'm hoping to explain what I'm trying to do and get pointed in the right direction! So I've got a simple ring-based HTTP server which connects to a database and serves up an index.html file for requests to "/". When I'm working on it locally, I run it with the clj tool, but now the time is coming to put it online. Coming from a Javascript background, the way I'd do that is: - install NGINX on a web server - start up a Node process running my backend server, this runs on port 3000. - the NGINX server reverse proxies requests on port 80 to port 3000, and bam, you're on the internet. This is a Clojure app though, and Clojure compiles to Java, so I need something that can build a JAR, which when invoked will start the server on port 3000. It looks like Boot and/or Leinigen will do this for you out of the box. I'm open to refactoring to use one of those tools, but I'd rather stay away from them if I can so I can stay current with language tooling. When it comes to clj though, it's unclear how you go about this. I've tried: - , and - But it isn't working out great so far. Do I even need to build a JAR in the first place? Can I just run the app on the production server the same way I do locally?


> Clojure compiles to Java clojure emits bytecode that runs on the jvm


the argument against running the server on prod the way you do locally is that clj is a build tool, and it's more reliable to have a standalone thing to run (a fat jar with deps) as opposed to a program that will fetch your deps and run your code


it removes a layer of complexity / source of errors


last I checked / tried, depstar was the best way to make an uberjar with clj


bonus, the author is very active here and responsive to errors or questions


You don't want to rely on a dependency manager because it can't handle reproducibility and rollbacks as well


So like if you just installed clj on the server, than ran your code, its possible the dependencies it pulls down will be different on one node from another


Its possible to do it though, if you use something like Artifactory


Where you can make sure your own repo of artifacts is immutable, and the version you depend on will always be exactly the same thing


Then with git deps using shas, that covers your own code base.


But short of being sure that your deps.edn deps are guaranteed to be the same every time you pull them down, for the same version you specified, it could get you in trouble.


That and, if you depend on public repos too, like Clojars, that can go down at any time, someone can pull out their lib from it, or someone could take it over and replace some of the lib their with compromised ones


That said... I never thought about this, but maybe if you packaged the clj local deps and classpath cache, and replicated that across your servers, but that is actually exactly what an UberJar is, just a lot more convenient then this


Like the act of creating an UberJar is just taking all your dependencies and your code and putting it in a Zip file.


Just one you don't have to unzip, and can just launch your app straight from the zip


@andyfry01 try depstar for creating uberjars with clj, works wonders


You don't have to build a jar though, you can start the app with clojure -M -m myapp.main as well


right from the git repo


Right, I figure I can fall back on this if I can't get it to work. I assume the downside is that the app won't be as performant and/or use more memory? Are there other downsides to that approach?


@andyfry01 there is no real difference if your source is loaded from an uberjar or from disk, but people also AOT their code for faster startup during deploys


this can be independently done from an uberjar though. uberjars are just easy for copying things around


but you often see the combination of both. none of this is mandatory


so if your old AOT-ed uberjar is still running and you can stop it + swap the uberjars real quickly and then start the other one, you can have less downtime


Ahhh, ok. This is great info.


this is actually one of the questions in the Clojure survey: how do you start your production app?


Looks like I'm headed over there next 👀


question 20


Thank you! This is exactly what I needed. Sounds like my approach is going to change over to installing a process runner which can invoke my app directly with clojure on startup and reboot, and maybe upgrading to building JAR files if/when I need the advantages you mentioned above.


You've saved me hours of googling :man-bowing:


There shouldn’t be any hit to performance. Clojure -M is just a script that builds a class path. So largely saving you the trouble


@andyfry01 I'm going to emphatically disagree with the others about the benefit of using an uberjar - the main advantage as I see it is that you have a single artifact to recreate a state of your code, which eliminates many sources of uncertainty (eg. - I've had prod errors caused by the programmer having a "customized" version of a lib locally). when you have an uberjar you can directly find this sort of discrepancy (as of a specific release artifact), while using the build tool in prod means investigating the caches and other server state


Also it separates concerns - if you use a build tool on prod, you now have tooling config / proxys, auth to private repos etc. that leak into multiple environment configs. Ideally you only need these things in the local dev machine and the CI / build server.


(to elaborate more about the specific example: if you are using an uberjar, you can verify the code that runs based on what's present in the jar, to do the same debugging when using a build tool in prod, you need to somehow guarantee that your editor / dev environment sees the exact same cache of packages, and sources all artifacts from the same places that the server did - maven shenanigans are rare, but between local dev mucking and private maven server admin they do happen)


Hey, thank you so much for the very detailed writeup. I was doing some more thinking over the weekend, and I think that just running things with CLJ both locally and in production is going to be the way to go. Coming from a JS perspective, I was very fixated on the concept of needing to build a production version of the app just as a matter of course. You could run any old JS Node server in development mode, but JS is slow, and anything you can do to make it faster is necessary in most cases. But if sticking this thing into a JAR/Uberjar delivers no performance benefits and adds complexity, then why would I? JARs come with other benefits, but they aren't going to be important for my specific case. Thanks again 🙂


using a jar reduces complexity, running a build tool on prod increases it


(but it reduces work in the short term, because you need to do the same work (installing the build tool, running it) on the server as you do locally. but now you need to eg. ensure that the build tool version and config matches your local, and factor the build tool run time into your startup time)


(a jar doesn't make your code run faster, but skips the run of the clj tool, which probably takes longer to run than your code does to start)


@andyfry01 if you get stuck with depstar ask for help in the #depstar channel -- happy to explain/walk you through everything there.

🙇 3
Harley Waagmeester21:01:48

what is a transducer ?


it's a transform like "map" or "filter" removed from the concrete detail of a collection as input


they are mainly an optimization, but they also mean you can eg. filter a channel without turning it into a collection first


if you run (map f (map g (map h coll))) there are two lazy seqs created that are only needed in order to provide data to another immediate step. clojure doesn't optimize that away, but transducers allow building the same pipeline without that overhead.


so you can use (sequence (comp (map h) (map g) (map f)) coll) to get the same result as the nested map calls, but it's cheaper on resources


as a beginner, you can ignore transducers for a while

Harley Waagmeester21:01:21

mmm... thanks again, and yes i don't see any need for using transducers at this time

Steve Lombardi05:01:31

transducers clicked for me after reading rich's history of clojure paper

Steve Lombardi05:01:35

there's some good info in there


@noisesmith I don't disagree with you. I'm just curious how "(eg. - I've had prod errors caused by the programmer having a "customized" version of a lib locally)" is fixed by an uberjar vs running using a dependency tool in an environment that should be resilient to dev-local changes.


because my coworkers periodically, locally install changes to libraries into a cache without changing version numbers, so they are unable to recreate or understand the bug


or worse, they blame the wrong thing and introduce bad changes via shotgun debugging


but this problem still manifests with an uberjar I guess?


considering that the uberjar is also built using the same deps that would be have been resolved in the prod env if you used a build tool there


no - because you open the source file that's in the uberjar when you go to source


but it sounded like they lein install some deps?


but any resultant uberjar from their machine would be similarly tainted right?


which is borkdude's point. build machine needs to be clean. which is largely the same problem as the prod machine needs to be clean


But that's not the only issue, there's also cases where a lib does AOT, so acording to project.clj you are using one artifact, but in the deployed artifact you use another. This can fool an editor that hooks into your build config, it doesn't fool editor+uberjar.


but my point was about dev replication


true. i don't have a strong opinion in this fight and lean towards uberjar as default


it's a pragmatic concern - the expensive thing is developer time


uberjars eliminate many factors that I've seen waste days of dev time


I think one of the key points is around how to debug what happens when there is an issue. with the uberjar, it's much easier to inspect the uberjar artifact to debug what happened. if it's just a command that was run using the state on the prod machine, it's a different story. if there's an issue, you can redeploy, but how to figure out the root cause so it doesn't happen again?

💯 3

when you use an uberjar with a lib that's already AOT-ed... that doesn't really help the fact that that other lib was already AOT-ed. I had a problem with a lib that was AOT-ed and transitively included AOT-ed sources of cheshire, while we needed a newer version. This was a major headache, but not helped by using an uberjar.


when I want to know what's being used, I do: (slurp (io/resource "foo/bar.clj")) or (slurp (io/resource "foo/bar_init_.class")) which is how I managed to find the AOT issue


this works regardless of uberjar or not


@borkdude it's a pain, but when I use an uberjar, I can see via the files in the jar that the wrong thing was included - not using an uberjar slows finding issues like that


there's also cases where you deploy, it corrupts some state in the database or some other persistent storage, but you don't find out about it until a week later. if you have the actually deployed artifacts, then it makes debugging, reproducing, and fixing those types of issues much easier.


I guess I could have saved a lot of words and said "it reduces many layers of complexity when diagnosing prod issues"


it can also cause issues, e.g include a conflicting resource. I don't see an uberjar as a cure-all, although they may cure some things. (slurp (io/resource ...)) is the cure for "what is being used".


right, but looking at the jar I can see the two resources (the .class vs. the .clj) and that they come from different artifacts


also, I tend to use io/resource to get coords, and open that coord in my editor rather than using slurp, but that's a trivial difference


(io/copy (io/resource ...) "/tmp/foo.clj") is also something I sometimes do ;)


my editor opens jars so I don't need to do that


mine does too, but read-only


yeah, it sounds like we found relatively equivalent process- I often use io/resource by itself to verify where the ns comes from (which is often more interesting than the precise contents at a first glance)


(ins)user=> (io/resource "clojure/set.clj")
#object[ 0x545607f2 "jar:file:/home/justin/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar!/clojure/set.clj"]


yeah, it took me a while to figure out that our version of cheshire was using the equivalent of:

user=> (io/resource "clojure/set__init.class")
#object[ 0x3e134896 "jar:file:/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar!/clojure/set__init.class"]


so next time I'll be looking for that darned thing first


it would be nice to have a tool that replicated this logic - finding the location of first of class / cljc / clj to be found


Perhaps this can be added to the output of *loading-verbosely*:

user=> (binding [clojure.core/*loading-verbosely* true] (require '[clojure.set] :reload-all))
(clojure.core/load "/clojure/set")


yeah, it would be great if loading-verbosely was more specific about where it got that thing it was loading


currently that's decided at the innermost layer (in Rt/load), which makes it harder to instrument like that


well, a small function that just tries to find the first non-nil resource using this logic is quickly written


you can hook that up with the output from loading-verbosely to print the exact locations


right, for something like this there's no harm in calculating it twice


Btw, I also use an uberjar even for GraalVM native-image builds, using depstar now. I can also just pass the classpath using $(clojure -Spath):classes but I hit a problem on Windows (too long classpath, or some other gnarly problem) which was fixed by using an uberjar.


I still need lein for uberjars in other projects due to some errors I'm getting with clj (some OS-specific .tar.gz error message)


Wondering about this


I can reproduce it if you want


Is it an issue with a native lib?


I think so yes.


it is with the SVM dependency from GraalVM


I've got a repro here:

{:deps {borkdude/sci {:mvn/version "0.2.0"}
        cheshire/cheshire {:mvn/version "5.10.0"}
        borkdude/clj-reflector-graal-java11-fix {:mvn/version "0.0.1-graalvm-20.3.0"}}
 :aliases {:uberjar
           {:replace-deps ; tool usage is new in 2.x
            {seancorfield/depstar {:mvn/version "2.0.165"}}
            :ns-default hf.depstar
            :exec-fn uberjar
            :exec-args {:jar plsci.jar
                        :compile-ns [plsci.core]
                        :aliases [:native]}}
           :native {:jvm-opts [""]
                    :extra-deps {org.clojure/clojure {:mvn/version "1.10.2-rc2"}}}}}
$ clojure -X:native:uberjar
[main] INFO hf.depstar.uberjar - Compiling plsci.core ...
[main] INFO hf.depstar.uberjar - Building uber jar: plsci.jar
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/nativeimage/svm-hosted-native-darwin-amd64/20.3.0/svm-hosted-native-darwin-amd64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/nativeimage/svm-hosted-native-linux-amd64/20.3.0/svm-hosted-native-linux-amd64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/nativeimage/svm-hosted-native-windows-amd64/20.3.0/svm-hosted-native-windows-amd64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/truffle/truffle-nfi-native-darwin-amd64/20.3.0/truffle-nfi-native-darwin-amd64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/truffle/truffle-nfi-native-linux-aarch64/20.3.0/truffle-nfi-native-linux-aarch64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/truffle/truffle-nfi-native-linux-amd64/20.3.0/truffle-nfi-native-linux-amd64-20.3.0.tar.gz"}


^ @U04V70XH6 does this look familiar?


it works with lein


@borkdude Sorry, been head down in a code spike all afternoon (after getting my brain swabbed for COVID earlier today which was unpleasant...)... So, er, what does Leiningen actually do here?


Does lein just binary-copy those files into the uberjar?


@U04V70XH6 To be honest, I don't know what's the matter with these .tar.gz files. In lein classpath they don't even show up, but with clojure -Spath they do. When I try to continue with the depstar uberjar, I get this error from graalvm native-image:

Error: No manifest in /Users/borkdude/Dropbox/dev/clojure/plsci/plsci.jar$NativeImageError: No manifest in /Users/borkdude/Dropbox/dev/clojure/plsci/plsci.jar


If there's a repo with a small repro case in, with instructions, I can take a look. Needs to be something I can run locally fairly easily.


Maybe native image is looking for more stuff in the MANIFEST.MF file than depstar puts in? Can you build it with lein uberjar and then extract MANIFEST.MF and see what's in that generated file? (and compare it to depstar generating the same-named file)


Will do! The repo is here: The branch is deps.edn. To build, set GRAALVM_HOME to a GraalVM installation, so the script can find native-image. To "install" GraalVM, you just need to download and unzip this: I am assuming you are running this in linux/WSL2.


And then run script/compile-libplsci


Can you drop that info into an issue on depstar so I can take a look when I'm at a computer and "working"?


(and better to assume I'm doing it on macOS, although it would have to work on 10.12 which is really old)


This is the manifest from lein btw:

Manifest-Version: 1.0
Created-By: Leiningen 2.9.3
Built-By: borkdude
Build-Jdk: 11.0.8
Leiningen-Project-ArtifactId: plsci
Leiningen-Project-GroupId: borkdude
Leiningen-Project-Version: 0.0.1-SNAPSHOT
Main-Class: plsci.core
Depstar doesn't seem to have any manifest...


Yes, it does. Inside the JAR.


That's why I said you needed to extract the generated file.


But you must specify a main class and you need a pom.xml (which depstar can build for you if you don't already have one).


Nothing in that repo you linked seems to be using depstar so I can't see how you are invoking it.


Ah, just went back to your original deps.edn:

{:replace-deps ; tool usage is new in 2.x
            {seancorfield/depstar {:mvn/version "2.0.165"}}
            :ns-default hf.depstar
            :exec-fn uberjar
            :exec-args {:jar plsci.jar
                        :compile-ns [plsci.core]
                        :aliases [:native]}}
You're missing :main-class!


So it would use clojure.main in the manifest.


Ah, it works now with:

:sync-pom true
                        :group-id plsci
                        :artifact-id plsci
                        :version "0.0.1-SNAPSHOT"
                        :main-class plsci.core


If you specify :main-class plsci.core you don't need :compile-ns


Yup. There we go!


I was confused by the docs since :main-class was suggested to have -main, which I don't have. It's a native library, not an app.


If you have suggestions to improve the docs around that, I'd love an issue created with them in!


If you want to write up how to use depstar with GraalVM to make native images and send a PR, that would also be great.


ok, I'll try to wrap up this branch and I'll do that


Maybe a different error message would have helped me: > [main] WARN hf.depstar.uberjar - Ignoring :main-class because no 'pom.xml' file is present! > [main] WARN hf.depstar.uberjar - Ignoring :main-class because no 'pom.xml' file is present! See :sync-pom true option... but this gets really long and the README is clear.


I'm planning a spike at the end of the month on depstar and honeysql (v2) as I'm taking four days off... and there are two open issues that touch on pom.xml:


Specifically 56 and 59. So those will get addressed in a few weeks.


Ah right, I think 59 would have helped me here. This is good default behavior probably. Maybe if main-class is a clojure file, also AOT it automatically?


Then it's pretty much the ease of use that you get from lein uberjar


Well :main-class is "just" a class name so it can't necessarily tell but I suppose it could look for a matching namespace...


Yeah, and skip it when there's already __init.class or Java class. Not sure how hard this is, since maybe depstar isn't running in the same classpath context as the libs you're building the jar for.


Anyway :compile-ns [] I found more explicit and maybe nicer than :aot true since it isn't obvious to me what this will do without looking at the docs every time


And once you figure this out once, it's just copy paste for the next project.


Strangely enough I already used depstar in another GraalVM project: I did not specify the main class but it worked well that time. I now see why... You can set the main class upon compilation, so you don't need a manifest at all facepalm

😄 3
Harley Waagmeester22:01:18

@borkdude it;s like with C code, somebody will copy and modify a system header file and include it with a sneaky #include "stupidly-modified-system-header-file.h" , which uses a locally modified "standard" file, a real bummer

Harley Waagmeester22:01:45

well, maybe a "dependency tool" will stop that in clojure

Harley Waagmeester22:01:52

but it's really all based on filenames

Harley Waagmeester22:01:20

ok, fixed by an uberjar, ok... i see some greater stgability now


@codeperfect An uberjar is not a cure-all for this. An uberjar can be built from patched files. "Reproducible environment" is the proper answer here.

Harley Waagmeester22:01:53

@borkdude heh, as usual you reduce the dataset to the essential content ;0

Harley Waagmeester22:01:16

my wifi really sucks, i appologize for the lag


Let's see if I can express this. I have a grouping of vector of maps. On each group, I'm invoking (mapv :fookey my-vector-of-maps) and it's sorta working, it's pulling out from the maps and constructing a vector of the values I'm interested in. However, in certain groups, the value for the key is nil, thus I end up with [nil] on occasions. Any suggestions on how to avoid constructing the vector (or returning an empty vector []) if the value of the key is nil?


you can wrap it in filter identity or use (into [] (mapcat ...) coll) to skip maps with no results

Harley Waagmeester22:01:40

yes, filter is your friend

Harley Waagmeester22:01:18

i eat millions of nils with my filter


the fact that mapcat can return 0 or more results for each input is under-utilized, it can be an elegant replacement for a separate filter step


Not sure if I've described that correctly, a moment, let me whip up a data shape...


user=> (mapcat (fn [m] (let [[fst :as result] (:k m)] (when fst [result]))) [{:k [nil]} {:k [:a :b :c]} {:k [:d :e :f]}])
([:a :b :c] [:d :e :f])



user=> (remove #{[nil]} (map :k [{:k [nil]} {:k [:a :b :c]} {:k [:d :e :f]}]))
([:a :b :c] [:d :e :f])


Hi, sorry it took some time...


(def my-coll-of-maps [{:k :a
                       :x nil
                       :y "456"}
                      {:k :b
                       :x "123"
                       :y "456"}
                      {:k :b
                       :x "789"
                       :y "456"}
                      {:k :c
                       :x nil
                       :y "456"}])

(for [[_ my-maps] (group-by :k my-coll-of-maps)]
  (let [x-vec (mapv :x my-maps)]


["123" "789"]


that's without the surrounding [] of course


my examples above both remove [nil] - the second one is better actually


Ideally I would like to end up with either [] or nothing, just one entry, ie., ["123" "789"]


oh, so you do want mapcat


(mapcat :x my-maps)


I end up with


(\1 \2 \3 \7 \8 \9)


yeah, I missed that for


yeah, you probably just want (remove #{[nil]} (for ...))


or turn the let into :let in the for, then add a :when that excludes [nil]


the remove works, but interested in the second option too


(for [[_ my-maps] (group-by :k my-coll-of-maps)
      :let [x-vec (mapv :x my-maps)]
      :when (not= [nil] x-vec)]


that works too


thank you @noisesmith that was very helpful 🙂


and educational too!

Andy Nortrup23:01:48

Hi there, I’m confused by an error message that I keep getting. I have function that ends with:

(remove empty? ....) 
that outputs
Which I then want to merge so that I have one map that looks like: {:CONNECT [:PLAT :CHART :TRENDS]}

Andy Nortrup23:01:14

I can take the straight output and do: (reduce (fn [one two] (merge-with union one two)) '({:CONNECT [:PLAT]} {:CONNECT [:CHART, :TRENDS]})) and get the output I want.

Andy Nortrup23:01:33

But when I try to put it straight into the larger function I’m building up (reduce (fn …) (remove empty?….) I get an error > Execution error (ClassCastException) at prodops.core/eval31234$fn (form-init5220355977580653078.clj:1705). > ; class [Ljava.lang.Object; cannot be cast to class clojure.lang.IPersistentCollection ([Ljava.lang.Object; is in module java.base of loader ‘bootstrap’; clojure.lang.IPersistentCollection is in unnamed module of loader ‘app’)

Robert Mitchell23:01:36

Can you share more context for how you’re putting it into the larger function? Also, merge-with can apply to more than two maps at once, so this is a bit shorter:

(apply (partial merge-with union)
       '({:CONNECT [:PLAT]} {:CONNECT [:CHART, :TRENDS]}))

Andy Nortrup00:01:56

(remove empty? (for [issue ((run-jql-query
                             "project = \"Connect\"  and issueLinkType in (\"relates to\") and resolution is empty"
                             :expand "issueLinks"
                             :fields "issuelinks"
                             :max-results 10)

                 (let [links (filter #(not= % (first (string/split (issue :key) #"-")))
                              (for [outboundLink (remove nil? (for [link (-> issue :fields :issuelinks)]
                                                                (get-in link [:outwardIssue :key])))]
                                (first (string/split outboundLink #"-")))))]
                       (if (not-empty links)
                         (hash-map (keyword (first (string/split (issue :key) #"-")))
                                   (to-array (for [link links]
                                     (keyword link))))))))
I’m a little ashamed of how messy that probably is. But such is life as a beginner. It starts as a query to the JIRA API, grabbing some pieces of the response, and trying to build a map of projects to the other projects they link out to. Goal is to build a graphviz graph from that.


what does the rest of the stacktrace say?


you are making primitve java arrays with to-array


(the [Ljava.lang.Object; in the error message)


and that is the error message you get when you try to use clojure.set/union with primitive arrays


stuff in clojure.set is generally intended to work on clojure's native set datatype, but it doesn't really do any input type checking, so sometimes people use it with collections that are not sets, and that sometimes works


but primitive java arrays are not collections or sets

Andy Nortrup00:01:45

That fixed it.