Fork me on GitHub
#beginners
<
2021-01-15
>
Stuart00:01:21

ah, i think its :integratedSecurity

seancorfield00:01:46

@qmstuart You can pass any properties that the JDBC driver understands -- so you won't find them in the docs specifically, but there is an explanation of passing additional properties to the driver as part of the db-spec hash map...

Claudio Ferreira01:01:51

Clojurians, what are we really doing here when we " { :keys [ minimum maximum ] } " as a parameter inside that function? I became lost here

Stuart01:01:33

That is destructuring a map into bindings by specifying the names of the keys in the map. So you function expects a parameter that is a map that contains the keys minimum and maximum After the destructuring you can just use the names minimum and maximum rather than having to do something like (:minimum your-map) e.g. Say you have a map

{:foo 5 :bar 6}
And a function
(defn foo [{:keys [foo bar]}]
   (prn foo bar))
ANd calling it with our map
(foo {:foo 5 :bar 6})
5 6 
=> nil
If foo didn't destructure it's map argument, I would have to do something like this
(defn foo [my-map]
   (prn (my-map :foo) (my-map :bar)))
See here, https://clojure.org/guides/destructuring, you want the section on associative destructuring.

Claudio Ferreira01:01:10

HMMM, thank you @qmstuart! And when we use ":as" after it? Are we binding the keys to next element? e. g. Acc in the next image?

Matthew Curry02:01:02

:as allows you to also bind the entire map as well, in addition to the key names specified

Stuart02:01:45

That is so you can still reference the entire map. So in that method your map argument is called acc, and its keys are current-bag and bags. The method also has a second parameter called stream. Im assuming this is a method to pass to reduce.

seancorfield02:01:10

Also worth noting: if you have :or to set a default value, it binds the symbol to that value but the :as symbol is bound to the original data.

seancorfield02:01:20

(let [{:keys [a b] :as data :or {a 1 b 2}} {:a 42 :c "Sea"}] (prn a b data))
42 2 {:a 42, :c "Sea"}

seancorfield02:01:44

So b is bound to 2 -- the default -- but data still doesn't contain b.

Claudio Ferreira15:01:08

Thanks all. Finally i've understood that!

zackteo06:01:37

Quick (?) question: what idioms should I be using instead if I'm planning to use map inside another map function

seancorfield06:01:52

@zackteo For some problems, that might well be the right solution. Can you provide some context?

zackteo06:01:33

Give me a moment 🙂

zackteo06:01:06

So I current have a list of sheets like ( "Sheet1" "Sheet2" ) and a list of lists of java objects (( org.apache.poi.ss.util.CellRangeAddress [A1:C1] org.apache.poi.ss.util.CellRangeAddress [A2:B3] org.apache.poi.ss.util.CellRangeAddress [C2:C3] )( org.apache.poi.ss.util.CellRangeAddress [A1:C1] )) and I want to transform it into a format like so ...

{[0 0 "Sheet1"] {:lrow 0, :lcol 2},
 [1 0 "Sheet1"] {:lrow 2, :lcol 1},
 [1 2 "Sheet1"] {:lrow 2, :lcol 2},
 [0 0 "Sheet2"] {:lrow 2, :lcol 2}}
where I can grab the numbers using .getFirstRow .getFirstColumn .getLastRow .getLastColumn on the java objects.

zackteo06:01:32

And currently I have

#(hash-map (vector (.getFirstRow %)
                   (.getFirstColumn %))
           (hash-map
                   :lrow (.getLastRow %)
                   :lcol (.getLastColumn %)))
which works on a list of java objects (not a list of list) to produce
{[0 0] {:lrow 0, :lcol 2},
 [1 0] {:lrow 2, :lcol 1},
 [1 2] {:lrow 2, :lcol 2}}

zackteo06:01:03

Sorry about the mess. I'm trying to figure out how to best approach the problem

seancorfield06:01:22

Can you explain what the actual problem is that you're trying to solve?

zackteo06:01:15

Am trying to add merged cells into https://github.com/zero-one-group/fxl/blob/develop/src/zero_one/fxl/read_xlsx.clj#L107 by adding :lrow and :lcol into :coord when it is a merged-cell . Which should seem easy to do. But the only/best access point to find out about merged-cells seems to be using getMergedRegions on the sheet https://poi.apache.org/apidocs/dev/org/apache/poi/ss/usermodel/Sheet.html#getMergedRegions--

seancorfield06:01:27

I think you're struggling because you can't explain the problem very well...

zackteo06:01:36

So I was thinking the way to approach it is by having a list of merged-regions in the let block of throwable-read-xlsx! https://github.com/zero-one-group/fxl/blob/develop/src/zero_one/fxl/read_xlsx.clj#L116 which I then can feed into poi-cell->fxl-cell to check if the current cell (it is mapping through) is a merged cell and adding the right keys accordingly

seancorfield06:01:44

You're too deep in the weeds of the implementation.

seancorfield06:01:04

Take a few steps back, away from the code, and think about the data at a higher level.

zackteo06:01:03

I guess I this might still be implementation level but broadly my problem seems to be a 2 step one 1. Get a hash-map which I can use to identify which cells are merged-cells. (which also contains the extra parameters I want, to make a cell a merged-cell) 2. Update the list of cells based on that hash-map

seancorfield06:01:11

That's still implementation. You haven't actually described the problem yet.

dpsutton06:01:42

"i have a workbook with two spreadsheets in them. i would like to ..."

seancorfield07:01:25

Right now I have no idea what you mean by "merged-cells" and what "extra parameters" means, even in that context.

zackteo07:01:54

Hmmmmm :thinking_face: I wondering if this is too high level. I am adding merged-cell reading capabilities to an excel read/write library (that wraps apache poi) Merged-cells are a combination of multiple cells but only display the value of the top leftmost one

zackteo07:01:13

A merged cell is a range/region of cells. So a cell typically only has 1 pair of coordinates. I will need the other pair to give me the region

zackteo07:01:59

Like cell A1 (:row 0 :col 0) to merged-cell A1:B2 (:row 0 :col 0 :lrow 1 :lcol 1)

zackteo07:01:15

Maybe I was trying to jump steps by considering multiple sheets, without successfully handling a single sheet first

Yang Xu07:01:15

Hi, Who knows the best way to invoke HoneySQL from java? or some suggestions?

Yang Xu09:01:56

How to solve if I face the ’macros/usetime‘ functions or some special logics in the Clojure, like macro ... etc. Should I only resolve it using Clojure?And before I invoke something from Java I need to use the Clojure wrap it?

noisesmith16:01:51

you can use the clojure compiler to load and run clojure library code, it will handle all of those things via its own read and eval steps https://clojure.org/reference/java_interop#_calling_clojure_from_java

noisesmith17:01:29

here's a small example I made, that loads up a namespace and runs it when a java daemon starts: https://github.com/noisesmith/clj-jsvc-adapter/blob/master/src/java/org/noisesmith/Cljsvc.java

noisesmith17:01:07

clojure.java.api.Clojure handles all the setup, compilation etc. you just need to use the require method and the invoke method on any resulting functions you want to use

Yang Xu09:01:18

Thank you.

dumrat07:01:20

Hi guys, need some opinion if possible. Not 100% related to clojure. I want to do an excel-like grid at front end, but evaluate it (the formulas) at backend. I'm thinking of casting the formulas into variable assignments like so: `A.1 = 30 + 20` , concatenating them and sending them to the backend when the user clicks a button. So, the request would contain something like this (Assuming a 2x2 grid):

A.1 = nil;
A.2 = 10;
B.1 = A.2;
B.2 = B.1 + 50;
I can parse this using instaparse and have played with it a bit here (Incomplete so far and assignment is not fully handled yet) : https://github.com/nakiya/efev/blob/main/src/duminda/efev.clj I have two general questions: 1. Is this a reasonable way to handle this problem? 2. How should I handle Stuff like this: A.1 = A.2; A.2 = 40; Because at evaluation I need to evaluate the second statement first and then only the first. 3. How should I handle circular references? i.e. A.1 = A.2; A.2 = A.1; It is enough to identify and stop evaluation on encountering circular references.

aratare07:01:34

@dumrat Can't you send a map of cells to their values instead? So something like {:A.1 nil, :A.2 10, :B.1 :A.2, :B.2 [:B.1 + 50]}?

dumrat07:01:03

@U013F1Q1R7G Yes I can. Still I'd have to parse the input because the users expect excel-like formulas. But I don't think this would solve #2 and #3 issues if I understand correctly.

aratare07:01:23

To handle (2) and (3), I suggest building a topology graph to identify the leaves (e.g. A2 = 10) vs the internal nodes (e.g. B1 = A2) so you can start evaluating them in the correct order.

aratare07:01:03

You can also detect loop while building topology graph iirc.

dumrat07:01:54

@U013F1Q1R7G Thanks. Do you have any suggested reading for building a topology graph? Not familiar with the concept.

dumrat07:01:05

Great, will check it out. Thanks!

Grigory Shepelev07:01:08

Hello there. Could you please help with understanding of the async lib ans subscriptions? Suppose I want to send https req to website every second and subscribe to it's result. And do some action each time as I get response. Suppose it's just printing the response's body. So having the following code does not produce desirable effect:

(ns async.core
  (:require [clojure.core.async
             :as async
             :refer [chan <!! <! >! >!! timeout pub sub unsub unsub-all go go-loop]]
            [clj-http.client
             :as h]))

(def updates-channel (chan))

(def publishing
  (pub updates-channel :status))

(defn req [url]
  (h/get url {:async? true}
         (fn [res] 
           (go (>! updates-channel res)))
         (fn [_] nil)))

(defn listen!
  [publisher topic]
  (let [l (chan 1)]
    (sub publisher topic l)
    (go-loop []
      (when-let [{:keys [body]} (<! l)]
        (println (str "got incoming message: " "\n" body))
        (recur)))))

(listen! publishing :200)

(go-loop [seconds 1]
  (req "")
  (<! (timeout 1000))
  (recur (inc seconds)))
As far as I undestood that's because of double go-loop.

Grigory Shepelev08:01:10

I am an idiot. sorry. It should be:

(listen! publishing 200)

Mark11:01:49

Hi folks, is there a fn to pop a key from a map, similar to dissoc but should also return the value of the key provided. e.g.

(pop-key {:a 1 :b 2 :c 3} :a)
;;=>[{:b 2 :c 3} 1]  

👍 1
bronsa11:01:26

nothing builtin

bronsa11:01:40

(juxt dissoc get)

🙌 1
😁 1
agata_anastazja (she/her)11:01:23

Hello, I am trying to find a way to write an acceptance test for my app. I want to verify it prints to the terminal the correct strings, but the last thing that the app needs to do is to exit. So I tried capturing it with with-out-str and leaving the program using System/exit 0, but it seems the program shuts down before any output is produced or captured. I reproduced a simpler version in repl, like so:

(defn print-many []
  (prn "here")
  (prn "there"))
=> #'user/print-many

(let [output (with-out-str (do (print-many)
                               (System/exit 0)))]
  (= output "here"))


Process finished with exit code 0
Any idea how I can test the correct output?

simongray12:01:36

I think you are capturing the output of (System/exit 0) since that is the last form of your do.

agata_anastazja (she/her)13:01:33

but does it mean that if I want to exit the program there is no way for me to capture what it prints?

Eamonn Sullivan13:01:24

The way I've handled that in the past is to split out the code that puts the string together into a separate function and unittest that function. So that would mean changing print-many to return a string and not call prn directly. Just a thought. I try to keep (defn -main ...) very tiny, for just this reason.

💜 1
bronsa13:01:43

well, for one you're not flushing the output stream actually nevermind, prn does autoflush

bronsa13:01:03

invoke (flush) before exiting

bronsa13:01:26

and invoke System/exit after the =

bronsa13:01:34

you can't do any computation after the program has quit -- System/exit needs to be the last expression evaluated

👍 1
agata_anastazja (she/her)13:01:27

I guess the problem is that I use System/exit to close the program

agata_anastazja (she/her)13:01:41

and I want to run a test on a whole program

bronsa13:01:53

your program code should not invoke System/exit anywhere but in the top level

👍 1
agata_anastazja (she/her)13:01:19

the above is just a simplified version of how my app behaves

bronsa13:01:20

if you need to return from a place deep in your stack, consider using an exception

bronsa13:01:53

e.g in your app instead of having (if my-condition (continue-running) (System/exit 0)) consider (if my-condition (continue-running) (throw ..))

agata_anastazja (she/her)13:01:55

but the idea is that it prints something, and then when it’s done processing, quits

bronsa14:01:08

then at the very top level you can catch the exception and invoke System/exit

bronsa14:01:32

and in your tests you can capture the exception and return nil, so that you can capture the output

alexmiller14:01:47

in general, things like (System/exit) and (shutdown-agents) are world-ending kinds of calls and should be factored to the very outside of an application either by wrapping or via swappable callback so that you can test stuff without actually ending the world

👍 1
roelof17:01:32

if I want to make this to a back-end of a website (https://gist.github.com/RoelofWobben/98066daa4042ad9e7e5db0b0956c00c3) am I right I only need ring and cheshire so no framework

noisesmith17:01:26

the bare minimum with ring is a web server backend (which only shows up directly in one line of code), ring core, and a router

noisesmith17:01:59

the web server does the actual work, ring has an adaptor to that which makes it more clojurey, and the router decides which function to call for each request

noisesmith17:01:26

the default back end is jetty, the default router is compojure

noisesmith17:01:27

@roelof the ring wiki has an even simpler example, just ring and backend and no router - it's a "misbehaving" server with no routes, it just uses the same function for every request ignoring the route https://github.com/ring-clojure/ring/wiki/Getting-Started

roelof17:01:45

I know I read that page several times and did some experiments

roelof17:01:17

that is why I think things like luminus is a overkill for this back-end

noisesmith17:01:36

looking at your code again, it looks like you are only interested in doing one thing - parsing a request URL to calculate a result - the issue you'd come up with is your function will get called with a lot of paths that won't make sense to that code

noisesmith17:01:47

eg. browsers will ask for favicon.ico

noisesmith17:01:31

with a router, you can have separate 404 response (or even a meaningful one), otherwise your function gets complex as it tries to handle all these unexpected strings

noisesmith17:01:46

(or just blatantly fails for all unexpected input)

roelof17:01:24

so maybe ring and computure ?

noisesmith17:01:01

with compojure, yeah

roelof17:01:03

first study those two again and try to make it work

noisesmith17:01:52

@roelof this is a minimal compojure router, you can plug in any function that takes a ring request map and returns something ring knows how to use as a response https://github.com/weavejester/compojure-example/blob/master/src/compojure/example/routes.clj

roelof17:01:28

oke, I have not decided if the front-end will be hiccup or maybe something like reframe

noisesmith17:01:26

with reframe your back end router will be managing requests from the cljs code for data used in rendering the page, with hiccup your back end router ends up calling code that generates the page, the router will look similar to what's above in both cases

roelof17:01:41

but I will like i said study ringa nd compujure

noisesmith17:01:45

using hiccup is easier for a first draft

roelof17:01:03

before Im going to make my own code work

noisesmith17:01:04

or even skipping hiccup and just returning some html string

roelof17:01:32

for example

roelof17:01:59

in the end I hope I can build a front-end like this : https://codepen.io/rwobben/full/xxVepxG

roelof17:01:08

but then without the movement

roelof17:01:50

See how long that will costs me . maybe weeks or maybe months but that does not matter

Harley Waagmeester18:01:10

i can run lein uberjar just fine, but when do java -classpath ... target/snapshot.jar i get the error /tmp/form-init1109030110411510610.clj ( no such file or directory )

Harley Waagmeester18:01:31

i use the correct classpath

noisesmith18:01:13

the uberjar task makes two jars

noisesmith18:01:20

you are running the wrong one

Harley Waagmeester18:01:49

java -jar uberjar works fine

noisesmith18:01:50

(this is because it treats making a jar for your app as a subtask for making the fat jar, and doesn't delete that pre-req)

Harley Waagmeester18:01:31

ok, i need to do lein jar ?

noisesmith18:01:45

no, lein uberjar creates two jars, one of which is created by using lein jar

noisesmith18:01:03

I'm just saying that when you get an error like that, the usual cause is running the regular jar instead of the uberjar

noisesmith18:01:32

"correct classpath" with an uberjar is just one thing- the jar itself

noisesmith18:01:52

it contains all other things that it needs

Harley Waagmeester18:01:07

i want to use the jar because it is smaller

noisesmith18:01:51

but it's not smaller than all the separate classpath items you need to gather to make it usable

Harley Waagmeester18:01:40

the file is smaller, the system loaded into ram will be the same i imagine

noisesmith18:01:54

the convention is to use uberjar for application deployment, and regular jars for libraries

noisesmith18:01:24

otherwise you need to deploy and run a build tool as part of app deployment, which is not a good idea

dorab18:01:26

The "skinny" jar is smaller because it only contains the files from your project. You want to use the "uber" jar that contains all the transitive dependencies as well.

Harley Waagmeester18:01:39

i usually use cider

noisesmith18:01:54

cider is another thing you don't want to use when deploying an app

noisesmith18:01:11

if you don't deploy, and everything is just cider usage with your editor, you don't need jar or uberjar at all

Harley Waagmeester18:01:16

but still i wonder why a trasient file in /tmp is required

noisesmith18:01:42

that's an implementation detail of lein, it creates a bootstrap to load up your project

Harley Waagmeester18:01:44

yes, tmux, emacs, cider is my stack

Harley Waagmeester18:01:31

ah well.. i was just wondering

dorab18:01:35

For development, that is fine. But for deploying, you (likely) just want an uberjar.

noisesmith18:01:40

OK - it would be foolish to deploy an app for others to use that is run that way, but if you aren't doing that you can ignore jar / uberjar entirely

Harley Waagmeester18:01:11

this is code for my private web server, deployment from me to others would be source only

noisesmith18:01:29

OK - running cider and emacs as a container for your app in a production environment is considered a problem. It can work but you can save yourself a lot of problems by just not doing that.

Harley Waagmeester18:01:40

the uber is 27M, that is a lot of man hours of coding

Harley Waagmeester18:01:20

lately i've been using the uberjar more often

Harley Waagmeester18:01:02

not my coding, other persons, it's really amazing

Harley Waagmeester18:01:38

sure, and now i'm looking at GraalVM, more candy to tempt me

Harley Waagmeester18:01:33

is it ok to run "lein snapshot.uberjar &", backgrounded is what i call it

roelof18:01:36

one last question For a new project can I the best use leiningen or switch to the new kid clojure clj ?

dorab18:01:11

For a new project I'd use clojure CLI over lein. But be aware that you may need other tools (or aliases) to build and run. See https://github.com/clojure/tools.deps.alpha/wiki/Tools

dorab18:01:31

@codeperfect If you have already created an uberjar that has a main function, then java -jar youruberjar.jar should work.

noisesmith18:01:00

switching between the two is relatively easy, and most tutorials (especially with ring) specify lein commands for building / running, I'd recommend lein for a beginner

noisesmith18:01:06

(unless you are following a tutorial or lesson that uses clj, then use that of course)

afry20:01:15

Hey all, I'm having a tough time wrapping my head about how to deploy a Clojure app for production. I'm hoping to explain what I'm trying to do and get pointed in the right direction! So I've got a simple ring-based HTTP server which connects to a database and serves up an index.html file for requests to "/". When I'm working on it locally, I run it with the clj tool, but now the time is coming to put it online. Coming from a Javascript background, the way I'd do that is: - install NGINX on a web server - start up a Node process running my backend server, this runs on port 3000. - the NGINX server reverse proxies requests on port 80 to port 3000, and bam, you're on the internet. This is a Clojure app though, and Clojure compiles to Java, so I need something that can build a JAR, which when invoked will start the server on port 3000. It looks like Boot and/or Leinigen will do this for you out of the box. I'm open to refactoring to use one of those tools, but I'd rather stay away from them if I can so I can stay current with language tooling. When it comes to clj though, it's unclear how you go about this. I've tried: - https://github.com/EwenG/badigeon , and - https://github.com/tonsky/uberdeps But it isn't working out great so far. Do I even need to build a JAR in the first place? Can I just run the app on the production server the same way I do locally?

noisesmith21:01:45

> Clojure compiles to Java clojure emits bytecode that runs on the jvm

noisesmith21:01:53

the argument against running the server on prod the way you do locally is that clj is a build tool, and it's more reliable to have a standalone thing to run (a fat jar with deps) as opposed to a program that will fetch your deps and run your code

noisesmith21:01:02

it removes a layer of complexity / source of errors

noisesmith21:01:38

last I checked / tried, depstar was the best way to make an uberjar with clj https://github.com/seancorfield/depstar

noisesmith21:01:52

bonus, the author is very active here and responsive to errors or questions

didibus04:01:55

You don't want to rely on a dependency manager because it can't handle reproducibility and rollbacks as well

didibus04:01:52

So like if you just installed clj on the server, than ran your code, its possible the dependencies it pulls down will be different on one node from another

didibus04:01:49

Its possible to do it though, if you use something like Artifactory

didibus04:01:29

Where you can make sure your own repo of artifacts is immutable, and the version you depend on will always be exactly the same thing

didibus04:01:02

Then with git deps using shas, that covers your own code base.

didibus04:01:41

But short of being sure that your deps.edn deps are guaranteed to be the same every time you pull them down, for the same version you specified, it could get you in trouble.

didibus04:01:22

That and, if you depend on public repos too, like Clojars, that can go down at any time, someone can pull out their lib from it, or someone could take it over and replace some of the lib their with compromised ones

didibus04:01:27

That said... I never thought about this, but maybe if you packaged the clj local deps and classpath cache, and replicated that across your servers, but that is actually exactly what an UberJar is, just a lot more convenient then this

didibus04:01:14

Like the act of creating an UberJar is just taking all your dependencies and your code and putting it in a Zip file.

didibus04:01:44

Just one you don't have to unzip, and can just launch your app straight from the zip

borkdude20:01:11

@andyfry01 try depstar for creating uberjars with clj, works wonders

borkdude20:01:36

You don't have to build a jar though, you can start the app with clojure -M -m myapp.main as well

borkdude20:01:43

right from the git repo

afry20:01:20

Right, I figure I can fall back on this if I can't get it to work. I assume the downside is that the app won't be as performant and/or use more memory? Are there other downsides to that approach?

borkdude20:01:36

@andyfry01 there is no real difference if your source is loaded from an uberjar or from disk, but people also AOT their code for faster startup during deploys

borkdude20:01:12

this can be independently done from an uberjar though. uberjars are just easy for copying things around

borkdude20:01:48

but you often see the combination of both. none of this is mandatory

borkdude20:01:49

so if your old AOT-ed uberjar is still running and you can stop it + swap the uberjars real quickly and then start the other one, you can have less downtime

afry20:01:54

Ahhh, ok. This is great info.

borkdude20:01:09

this is actually one of the questions in the Clojure survey: how do you start your production app?

afry20:01:28

Looks like I'm headed over there next 👀

borkdude20:01:02

question 20

afry20:01:23

Thank you! This is exactly what I needed. Sounds like my approach is going to change over to installing a process runner which can invoke my app directly with clojure on startup and reboot, and maybe upgrading to building JAR files if/when I need the advantages you mentioned above.

afry20:01:43

You've saved me hours of googling :man-bowing:

dpsutton20:01:07

There shouldn’t be any hit to performance. Clojure -M is just a script that builds a class path. So largely saving you the trouble

noisesmith21:01:28

@andyfry01 I'm going to emphatically disagree with the others about the benefit of using an uberjar - the main advantage as I see it is that you have a single artifact to recreate a state of your code, which eliminates many sources of uncertainty (eg. - I've had prod errors caused by the programmer having a "customized" version of a lib locally). when you have an uberjar you can directly find this sort of discrepancy (as of a specific release artifact), while using the build tool in prod means investigating the caches and other server state

noisesmith21:01:57

Also it separates concerns - if you use a build tool on prod, you now have tooling config / proxys, auth to private repos etc. that leak into multiple environment configs. Ideally you only need these things in the local dev machine and the CI / build server.

noisesmith21:01:26

(to elaborate more about the specific example: if you are using an uberjar, you can verify the code that runs based on what's present in the jar, to do the same debugging when using a build tool in prod, you need to somehow guarantee that your editor / dev environment sees the exact same cache of packages, and sources all artifacts from the same places that the server did - maven shenanigans are rare, but between local dev mucking and private maven server admin they do happen)

afry16:01:17

Hey, thank you so much for the very detailed writeup. I was doing some more thinking over the weekend, and I think that just running things with CLJ both locally and in production is going to be the way to go. Coming from a JS perspective, I was very fixated on the concept of needing to build a production version of the app just as a matter of course. You could run any old JS Node server in development mode, but JS is slow, and anything you can do to make it faster is necessary in most cases. But if sticking this thing into a JAR/Uberjar delivers no performance benefits and adds complexity, then why would I? JARs come with other benefits, but they aren't going to be important for my specific case. Thanks again 🙂

noisesmith18:01:15

using a jar reduces complexity, running a build tool on prod increases it

noisesmith18:01:52

(but it reduces work in the short term, because you need to do the same work (installing the build tool, running it) on the server as you do locally. but now you need to eg. ensure that the build tool version and config matches your local, and factor the build tool run time into your startup time)

noisesmith18:01:44

(a jar doesn't make your code run faster, but skips the run of the clj tool, which probably takes longer to run than your code does to start)

seancorfield21:01:20

@andyfry01 if you get stuck with depstar ask for help in the #depstar channel -- happy to explain/walk you through everything there.

🙇 1
Harley Waagmeester21:01:48

what is a transducer ?

noisesmith21:01:27

it's a transform like "map" or "filter" removed from the concrete detail of a collection as input

noisesmith21:01:51

they are mainly an optimization, but they also mean you can eg. filter a channel without turning it into a collection first

noisesmith21:01:53

if you run (map f (map g (map h coll))) there are two lazy seqs created that are only needed in order to provide data to another immediate step. clojure doesn't optimize that away, but transducers allow building the same pipeline without that overhead.

noisesmith21:01:04

so you can use (sequence (comp (map h) (map g) (map f)) coll) to get the same result as the nested map calls, but it's cheaper on resources

noisesmith21:01:30

as a beginner, you can ignore transducers for a while

Harley Waagmeester21:01:21

mmm... thanks again, and yes i don't see any need for using transducers at this time

Steven Lombardi05:01:31

transducers clicked for me after reading rich's history of clojure paper https://dl.acm.org/doi/pdf/10.1145/3386321

Steven Lombardi05:01:35

there's some good info in there

borkdude22:01:32

@noisesmith I don't disagree with you. I'm just curious how "(eg. - I've had prod errors caused by the programmer having a "customized" version of a lib locally)" is fixed by an uberjar vs running using a dependency tool in an environment that should be resilient to dev-local changes.

noisesmith22:01:21

because my coworkers periodically, locally install changes to libraries into a cache without changing version numbers, so they are unable to recreate or understand the bug

noisesmith22:01:43

or worse, they blame the wrong thing and introduce bad changes via shotgun debugging

borkdude22:01:59

but this problem still manifests with an uberjar I guess?

borkdude22:01:18

considering that the uberjar is also built using the same deps that would be have been resolved in the prod env if you used a build tool there

noisesmith22:01:19

no - because you open the source file that's in the uberjar when you go to source

dpsutton22:01:40

but it sounded like they lein install some deps?

dpsutton22:01:01

but any resultant uberjar from their machine would be similarly tainted right?

dpsutton22:01:31

which is borkdude's point. build machine needs to be clean. which is largely the same problem as the prod machine needs to be clean

noisesmith22:01:43

But that's not the only issue, there's also cases where a lib does AOT, so acording to project.clj you are using one artifact, but in the deployed artifact you use another. This can fool an editor that hooks into your build config, it doesn't fool editor+uberjar.

noisesmith22:01:07

but my point was about dev replication

dpsutton22:01:15

true. i don't have a strong opinion in this fight and lean towards uberjar as default

noisesmith22:01:23

it's a pragmatic concern - the expensive thing is developer time

noisesmith22:01:42

uberjars eliminate many factors that I've seen waste days of dev time

phronmophobic22:01:30

I think one of the key points is around how to debug what happens when there is an issue. with the uberjar, it's much easier to inspect the uberjar artifact to debug what happened. if it's just a command that was run using the state on the prod machine, it's a different story. if there's an issue, you can redeploy, but how to figure out the root cause so it doesn't happen again?

💯 1
borkdude22:01:13

when you use an uberjar with a lib that's already AOT-ed... that doesn't really help the fact that that other lib was already AOT-ed. I had a problem with a lib that was AOT-ed and transitively included AOT-ed sources of cheshire, while we needed a newer version. This was a major headache, but not helped by using an uberjar.

borkdude22:01:16

when I want to know what's being used, I do: (slurp (io/resource "foo/bar.clj")) or (slurp (io/resource "foo/bar_init_.class")) which is how I managed to find the AOT issue

borkdude22:01:26

this works regardless of uberjar or not

noisesmith22:01:30

@borkdude it's a pain, but when I use an uberjar, I can see via the files in the jar that the wrong thing was included - not using an uberjar slows finding issues like that

phronmophobic22:01:35

there's also cases where you deploy, it corrupts some state in the database or some other persistent storage, but you don't find out about it until a week later. if you have the actually deployed artifacts, then it makes debugging, reproducing, and fixing those types of issues much easier.

noisesmith22:01:41

I guess I could have saved a lot of words and said "it reduces many layers of complexity when diagnosing prod issues"

borkdude22:01:50

it can also cause issues, e.g include a conflicting resource. I don't see an uberjar as a cure-all, although they may cure some things. (slurp (io/resource ...)) is the cure for "what is being used".

noisesmith22:01:07

right, but looking at the jar I can see the two resources (the .class vs. the .clj) and that they come from different artifacts

noisesmith22:01:52

also, I tend to use io/resource to get coords, and open that coord in my editor rather than using slurp, but that's a trivial difference

borkdude22:01:44

(io/copy (io/resource ...) "/tmp/foo.clj") is also something I sometimes do ;)

noisesmith22:01:30

my editor opens jars so I don't need to do that

borkdude22:01:44

mine does too, but read-only

noisesmith22:01:41

yeah, it sounds like we found relatively equivalent process- I often use io/resource by itself to verify where the ns comes from (which is often more interesting than the precise contents at a first glance)

noisesmith22:01:19

(ins)user=> (io/resource "clojure/set.clj")
#object[java.net.URL 0x545607f2 "jar:file:/home/justin/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar!/clojure/set.clj"]

borkdude22:01:08

yeah, it took me a while to figure out that our version of cheshire was using the equivalent of:

user=> (io/resource "clojure/set__init.class")
#object[java.net.URL 0x3e134896 "jar:file:/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar!/clojure/set__init.class"]

borkdude22:01:16

so next time I'll be looking for that darned thing first

noisesmith22:01:34

it would be nice to have a tool that replicated this logic https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L427 - finding the location of first of class / cljc / clj to be found

borkdude22:01:00

Perhaps this can be added to the output of *loading-verbosely*:

user=> (binding [clojure.core/*loading-verbosely* true] (require '[clojure.set] :reload-all))
(clojure.core/load "/clojure/set")
nil

noisesmith22:01:13

yeah, it would be great if loading-verbosely was more specific about where it got that thing it was loading

noisesmith22:01:52

currently that's decided at the innermost layer (in Rt/load), which makes it harder to instrument like that

borkdude22:01:17

well, a small function that just tries to find the first non-nil resource using this logic is quickly written

borkdude22:01:53

you can hook that up with the output from loading-verbosely to print the exact locations

noisesmith22:01:37

right, for something like this there's no harm in calculating it twice

borkdude22:01:01

Btw, I also use an uberjar even for GraalVM native-image builds, using depstar now. I can also just pass the classpath using $(clojure -Spath):classes but I hit a problem on Windows (too long classpath, or some other gnarly problem) which was fixed by using an uberjar.

borkdude22:01:10

I still need lein for uberjars in other projects due to some errors I'm getting with clj (some OS-specific .tar.gz error message)

ghadi22:01:36

Wondering about this

borkdude22:01:23

I can reproduce it if you want

ghadi22:01:22

Is it an issue with a native lib?

borkdude22:01:52

I think so yes.

borkdude22:01:09

it is with the SVM dependency from GraalVM

borkdude22:01:15

I've got a repro here:

{:deps {borkdude/sci {:mvn/version "0.2.0"}
        cheshire/cheshire {:mvn/version "5.10.0"}
        borkdude/clj-reflector-graal-java11-fix {:mvn/version "0.0.1-graalvm-20.3.0"}}
 :aliases {:uberjar
           {:replace-deps ; tool usage is new in 2.x
            {seancorfield/depstar {:mvn/version "2.0.165"}}
            :ns-default hf.depstar
            :exec-fn uberjar
            :exec-args {:jar plsci.jar
                        :compile-ns [plsci.core]
                        :aliases [:native]}}
           :native {:jvm-opts ["-Dclojure.compiler.direct-linking=true"]
                    :extra-deps {org.clojure/clojure {:mvn/version "1.10.2-rc2"}}}}}
$ clojure -X:native:uberjar
[main] INFO hf.depstar.uberjar - Compiling plsci.core ...
[main] INFO hf.depstar.uberjar - Building uber jar: plsci.jar
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/nativeimage/svm-hosted-native-darwin-amd64/20.3.0/svm-hosted-native-darwin-amd64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/nativeimage/svm-hosted-native-linux-amd64/20.3.0/svm-hosted-native-linux-amd64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/nativeimage/svm-hosted-native-windows-amd64/20.3.0/svm-hosted-native-windows-amd64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/truffle/truffle-nfi-native-darwin-amd64/20.3.0/truffle-nfi-native-darwin-amd64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/truffle/truffle-nfi-native-linux-aarch64/20.3.0/truffle-nfi-native-linux-aarch64-20.3.0.tar.gz"}
{:warning "ignoring unknown file type", :path "/Users/borkdude/.m2/repository/org/graalvm/truffle/truffle-nfi-native-linux-amd64/20.3.0/truffle-nfi-native-linux-amd64-20.3.0.tar.gz"}

borkdude22:01:56

^ @U04V70XH6 does this look familiar?

borkdude22:01:13

it works with lein

seancorfield00:01:49

@borkdude Sorry, been head down in a code spike all afternoon (after getting my brain swabbed for COVID earlier today which was unpleasant...)... So, er, what does Leiningen actually do here?

seancorfield00:01:30

Does lein just binary-copy those files into the uberjar?

borkdude21:01:55

@U04V70XH6 To be honest, I don't know what's the matter with these .tar.gz files. In lein classpath they don't even show up, but with clojure -Spath they do. When I try to continue with the depstar uberjar, I get this error from graalvm native-image:

Error: No manifest in /Users/borkdude/Dropbox/dev/clojure/plsci/plsci.jar
com.oracle.svm.driver.NativeImage$NativeImageError: No manifest in /Users/borkdude/Dropbox/dev/clojure/plsci/plsci.jar

seancorfield21:01:43

If there's a repo with a small repro case in, with instructions, I can take a look. Needs to be something I can run locally fairly easily.

seancorfield21:01:40

Maybe native image is looking for more stuff in the MANIFEST.MF file than depstar puts in? Can you build it with lein uberjar and then extract MANIFEST.MF and see what's in that generated file? (and compare it to depstar generating the same-named file)

borkdude21:01:49

Will do! The repo is here: https://github.com/borkdude/plsci The branch is deps.edn. To build, set GRAALVM_HOME to a GraalVM installation, so the script can find native-image. To "install" GraalVM, you just need to download and unzip this: https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-20.3.0/graalvm-ce-java11-linux-amd64-20.3.0.tar.gz I am assuming you are running this in linux/WSL2.

borkdude21:01:19

And then run script/compile-libplsci

seancorfield21:01:28

Can you drop that info into an issue on depstar so I can take a look when I'm at a computer and "working"?

seancorfield21:01:53

(and better to assume I'm doing it on macOS, although it would have to work on 10.12 which is really old)

borkdude21:01:31

This is the manifest from lein btw:

Manifest-Version: 1.0
Created-By: Leiningen 2.9.3
Built-By: borkdude
Build-Jdk: 11.0.8
Leiningen-Project-ArtifactId: plsci
Leiningen-Project-GroupId: borkdude
Leiningen-Project-Version: 0.0.1-SNAPSHOT
Main-Class: plsci.core
Depstar doesn't seem to have any manifest...

seancorfield21:01:05

Yes, it does. Inside the JAR.

seancorfield21:01:49

That's why I said you needed to extract the generated file.

seancorfield21:01:20

But you must specify a main class and you need a pom.xml (which depstar can build for you if you don't already have one).

seancorfield21:01:00

Nothing in that repo you linked seems to be using depstar so I can't see how you are invoking it.

seancorfield21:01:10

Ah, just went back to your original deps.edn:

{:replace-deps ; tool usage is new in 2.x
            {seancorfield/depstar {:mvn/version "2.0.165"}}
            :ns-default hf.depstar
            :exec-fn uberjar
            :exec-args {:jar plsci.jar
                        :compile-ns [plsci.core]
                        :aliases [:native]}}
You're missing :main-class!

seancorfield21:01:38

So it would use clojure.main in the manifest.

borkdude21:01:00

Ah, it works now with:

:sync-pom true
                        :group-id plsci
                        :artifact-id plsci
                        :version "0.0.1-SNAPSHOT"
                        :main-class plsci.core

seancorfield21:01:04

If you specify :main-class plsci.core you don't need :compile-ns

seancorfield21:01:27

Yup. There we go!

borkdude21:01:39

I was confused by the docs since :main-class was suggested to have -main, which I don't have. It's a native library, not an app.

seancorfield21:01:29

If you have suggestions to improve the docs around that, I'd love an issue created with them in!

seancorfield21:01:03

If you want to write up how to use depstar with GraalVM to make native images and send a PR, that would also be great.

borkdude21:01:25

ok, I'll try to wrap up this branch and I'll do that

1
borkdude22:01:16

Maybe a different error message would have helped me: > [main] WARN hf.depstar.uberjar - Ignoring :main-class because no 'pom.xml' file is present! > [main] WARN hf.depstar.uberjar - Ignoring :main-class because no 'pom.xml' file is present! See :sync-pom true option... but this gets really long and the README is clear.

seancorfield22:01:51

I'm planning a spike at the end of the month on depstar and honeysql (v2) as I'm taking four days off... and there are two open issues that touch on pom.xml: https://github.com/seancorfield/depstar/issues

seancorfield22:01:25

Specifically 56 and 59. So those will get addressed in a few weeks.

borkdude22:01:50

Ah right, I think 59 would have helped me here. This is good default behavior probably. Maybe if main-class is a clojure file, also AOT it automatically?

borkdude22:01:10

Then it's pretty much the ease of use that you get from lein uberjar

seancorfield22:01:03

Well :main-class is "just" a class name so it can't necessarily tell but I suppose it could look for a matching namespace...

borkdude22:01:40

Yeah, and skip it when there's already __init.class or Java class. Not sure how hard this is, since maybe depstar isn't running in the same classpath context as the libs you're building the jar for.

borkdude22:01:57

Anyway :compile-ns [foo.bar] I found more explicit and maybe nicer than :aot true since it isn't obvious to me what this will do without looking at the docs every time

borkdude22:01:23

And once you figure this out once, it's just copy paste for the next project.

borkdude22:01:36

Strangely enough I already used depstar in another GraalVM project: https://github.com/babashka/pod-babashka-buddy/blob/8ac874b44e3700666ce77aa94fee374a6cfae7df/deps.edn#L12 I did not specify the main class but it worked well that time. I now see why... https://github.com/babashka/pod-babashka-buddy/blob/8ac874b44e3700666ce77aa94fee374a6cfae7df/script/compile#L25 You can set the main class upon compilation, so you don't need a manifest at all facepalm

😄 1
Harley Waagmeester22:01:18

@borkdude it;s like with C code, somebody will copy and modify a system header file and include it with a sneaky #include "stupidly-modified-system-header-file.h" , which uses a locally modified "standard" file, a real bummer

Harley Waagmeester22:01:45

well, maybe a "dependency tool" will stop that in clojure

Harley Waagmeester22:01:52

but it's really all based on filenames

Harley Waagmeester22:01:20

ok, fixed by an uberjar, ok... i see some greater stgability now

borkdude22:01:55

@codeperfect An uberjar is not a cure-all for this. An uberjar can be built from patched files. "Reproducible environment" is the proper answer here.

Harley Waagmeester22:01:53

@borkdude heh, as usual you reduce the dataset to the essential content ;0

Harley Waagmeester22:01:16

my wifi really sucks, i appologize for the lag

dharrigan22:01:24

Let's see if I can express this. I have a grouping of vector of maps. On each group, I'm invoking (mapv :fookey my-vector-of-maps) and it's sorta working, it's pulling out from the maps and constructing a vector of the values I'm interested in. However, in certain groups, the value for the key is nil, thus I end up with [nil] on occasions. Any suggestions on how to avoid constructing the vector (or returning an empty vector []) if the value of the key is nil?

noisesmith22:01:28

you can wrap it in filter identity or use (into [] (mapcat ...) coll) to skip maps with no results

Harley Waagmeester22:01:40

yes, filter is your friend

Harley Waagmeester22:01:18

i eat millions of nils with my filter

noisesmith22:01:54

the fact that mapcat can return 0 or more results for each input is under-utilized, it can be an elegant replacement for a separate filter step

dharrigan22:01:03

Not sure if I've described that correctly, a moment, let me whip up a data shape...

noisesmith22:01:27

user=> (mapcat (fn [m] (let [[fst :as result] (:k m)] (when fst [result]))) [{:k [nil]} {:k [:a :b :c]} {:k [:d :e :f]}])
([:a :b :c] [:d :e :f])

noisesmith22:01:58

or

user=> (remove #{[nil]} (map :k [{:k [nil]} {:k [:a :b :c]} {:k [:d :e :f]}]))
([:a :b :c] [:d :e :f])

dharrigan22:01:18

Hi, sorry it took some time...

dharrigan22:01:21

(def my-coll-of-maps [{:k :a
                       :x nil
                       :y "456"}
                      {:k :b
                       :x "123"
                       :y "456"}
                      {:k :b
                       :x "789"
                       :y "456"}
                      {:k :c
                       :x nil
                       :y "456"}])

(for [[_ my-maps] (group-by :k my-coll-of-maps)]
  (let [x-vec (mapv :x my-maps)]
    x-vec))

dharrigan22:01:50

[nil]
["123" "789"]
[nil]

noisesmith22:01:03

that's without the surrounding [] of course

noisesmith22:01:19

my examples above both remove [nil] - the second one is better actually

dharrigan22:01:32

Ideally I would like to end up with either [] or nothing, just one entry, ie., ["123" "789"]

noisesmith22:01:42

oh, so you do want mapcat

noisesmith22:01:52

(mapcat :x my-maps)

dharrigan22:01:54

I end up with

dharrigan22:01:56

()
(\1 \2 \3 \7 \8 \9)
()

noisesmith22:01:11

yeah, I missed that for

noisesmith22:01:35

yeah, you probably just want (remove #{[nil]} (for ...))

noisesmith22:01:05

or turn the let into :let in the for, then add a :when that excludes [nil]

dharrigan23:01:01

the remove works, but interested in the second option too

dharrigan23:01:51

(for [[_ my-maps] (group-by :k my-coll-of-maps)
      :let [x-vec (mapv :x my-maps)]
      :when (not= [nil] x-vec)]
  x-vec)

dharrigan23:01:03

that works too

dharrigan23:01:45

thank you @noisesmith that was very helpful 🙂

dharrigan23:01:52

and educational too!

Andy Nortrup23:01:48

Hi there, I’m confused by an error message that I keep getting. I have function that ends with:

(remove empty? ....) 
that outputs
({:CONNECT [:PLAT]} {:CONNECT [:CHART, :TRENDS]})
Which I then want to merge so that I have one map that looks like: {:CONNECT [:PLAT :CHART :TRENDS]}

Andy Nortrup23:01:14

I can take the straight output and do: (reduce (fn [one two] (merge-with union one two)) '({:CONNECT [:PLAT]} {:CONNECT [:CHART, :TRENDS]})) and get the output I want.

Andy Nortrup23:01:33

But when I try to put it straight into the larger function I’m building up (reduce (fn …) (remove empty?….) I get an error > Execution error (ClassCastException) at prodops.core/eval31234$fn (form-init5220355977580653078.clj:1705). > ; class [Ljava.lang.Object; cannot be cast to class clojure.lang.IPersistentCollection ([Ljava.lang.Object; is in module java.base of loader ‘bootstrap’; clojure.lang.IPersistentCollection is in unnamed module of loader ‘app’)

Robert Mitchell23:01:36

Can you share more context for how you’re putting it into the larger function? Also, merge-with can apply to more than two maps at once, so this is a bit shorter:

(apply (partial merge-with union)
       '({:CONNECT [:PLAT]} {:CONNECT [:CHART, :TRENDS]}))

Andy Nortrup00:01:56

(remove empty? (for [issue ((run-jql-query
                             "project = \"Connect\"  and issueLinkType in (\"relates to\") and resolution is empty"
                             :expand "issueLinks"
                             :fields "issuelinks"
                             :max-results 10)
                            :issues)]

                 (let [links (filter #(not= % (first (string/split (issue :key) #"-")))
                             (dedupe
                              (for [outboundLink (remove nil? (for [link (-> issue :fields :issuelinks)]
                                                                (get-in link [:outwardIssue :key])))]
                                (first (string/split outboundLink #"-")))))]
                       (if (not-empty links)
                         (hash-map (keyword (first (string/split (issue :key) #"-")))
                                   (to-array (for [link links]
                                     (keyword link))))))))
I’m a little ashamed of how messy that probably is. But such is life as a beginner. It starts as a query to the JIRA API, grabbing some pieces of the response, and trying to build a map of projects to the other projects they link out to. Goal is to build a graphviz graph from that.

hiredman00:01:13

what does the rest of the stacktrace say?

hiredman00:01:54

you are making primitve java arrays with to-array

hiredman00:01:11

(the [Ljava.lang.Object; in the error message)

hiredman00:01:53

and that is the error message you get when you try to use clojure.set/union with primitive arrays

hiredman00:01:18

stuff in clojure.set is generally intended to work on clojure's native set datatype, but it doesn't really do any input type checking, so sometimes people use it with collections that are not sets, and that sometimes works

hiredman00:01:34

but primitive java arrays are not collections or sets

Andy Nortrup00:01:45

That fixed it.