Fork me on GitHub
#clojure
<
2020-04-13
>
danielglauser00:04:45

If you are going to parse command line args is tools.cli still the suggested way to go? https://github.com/clojure/tools.cli

seancorfield00:04:14

@danielglauser I might be a little bit biased but, yes, I don't know of a better alternative 🙂

😀 4
danielglauser00:04:51

Bias is fine by me. Thanks Sean 🙂

Chris Lester02:04:14

A question about record use: I'm going through some kata's (like kruskal's and prims algorithms, etc.) where I solved the need to update and keep in sync multiple data structures with mutation (atoms) or via simply passing that problem off to the caller .. but for a few immutable versions online I see some people are recursively generating a new record and passing the new structure to it when needing to build up a map or other data structure. Is this performant? It seems like it would create potentially thousands of objects.

ataggart02:04:22

Maps, vectors, etc. are real classes, too.

Chris Lester02:04:28

Those are data structures, I'd be surprised to see creation of thousands of map instances as you conj onto one. Perhaps that's what happens, seems unlikely.

ataggart02:04:11

That's about how many instances, not whether the instance is a record or a map, etc.

Chris Lester02:04:51

Yes. So when someone is recursively creating Records inside the record to build up a map that feels non-performant. Is that a correct assumption?

Chris Lester02:04:23

..or at least overly memory intensive.

Chris Lester03:04:09

[updated so I don't have that discussion with a ton of people but not the point of the question]

ataggart03:04:14

What about recursively creating maps inside the map to build up a map?

ataggart03:04:09

I mean, I don't know what this algorithm is doing.

ataggart03:04:12

but there's not really much difference between (assoc {} :k :v) and (->R :v).

ataggart03:04:33

both create a new instance of a thing

Chris Lester03:04:02

Interesting. Would have thought the left would not create new overhead (instances) where the right clearly would.

ataggart03:04:07

That's what happens when you have persistent data structures.

Chris Lester03:04:08

I was under the impression that a persistent data structure (the bit partitioned hash tries) would simply join new roots onto the tree. Will find the assoc code and see what's going on.

Chris Lester03:04:31

Or join new structures into the tree. Not create new trees all the time.

ataggart03:04:47

Yep, it does do structural sharing, but the root is new, as is the path from the root to the changed value. https://upload.wikimedia.org/wikipedia/commons/thumb/5/56/Purely_functional_tree_after.svg/1920px-Purely_functional_tree_after.svg.png

ataggart03:04:52

And the JVM is ridiculously good at dealing with short-lived objects.

Chris Lester03:04:43

Yes, had assumed that under the hood a single map instance would represent that structure as it changed. So the assertion is that both assoc, by extension reduce, et al. .. and a record creating new instances of itself (recursively) are equivalent in performance and memory footprints. If so then I'll add that as something that isn't odd.

Chris Lester03:04:32

Re: the jvm ... I've seen enough JVM OOM / memory pressure caused by similar recursive object creation (in web scale platforms) to be leery when I see it.

ataggart03:04:32

> Yes, had assumed that under the hood a single map instance would represent that structure as it changed. That behaviour is available via transient. See https://clojure.org/reference/transients

seancorfield03:04:34

@US893PCLF The OOM happens when there is a memory leak, not in normal usage with this sort of data structure.

ataggart03:04:35

And thus, by definition, not "short-lived". 😉

seancorfield03:04:40

We run a dozen Clojure apps in production, many with just 1GB heaps and they run for weeks and weeks solidly, despite all this object creation.

Chris Lester03:04:39

That's good to know. The system I'm refactoring off of Java to Clojure can't run for a day without OOM (well, it can now of sorts) and we're running 16-20GB heaps (granted, it's mostly the data model). So OOM's are near to my heart lol. Thx @U5ZNLFCQ7 ... transients seem useful for some of the things we actually do (transform large maps).

ataggart03:04:57

Note that transients are for when a fn is making repeated modifications to the "same" data structure. They should not be passed around.

ataggart03:04:37

And as with all things performance-related, profile first.

Chris Lester03:04:11

Agree, if you don't measure you shouldn't be optimizing... not that you should fly blind anyway.

seancorfield03:04:41

We've also moved from large legacy systems that required 10-15GB heaps to run for any length of time. We started using Clojure a decade ago and we've been very happy as Clojure has replaced each of our legacy systems.

Chris Lester03:04:25

Very cool. I am moving one of the largest media companies content management and streaming systems to Clojure (and thus will bear the cost if it isn't successful). It's good to have that kind of anecdotal (from experience) information to point to.

emccue05:04:42

The only area you should be wary of with clojure's data structures is laziness. If you have a tail-recursive process that builds up a data structure eagerly, generally maps, vectors, and sets perform pretty well

emccue05:04:52

for all the reasons mentioned above (JVM does pointer bumps for allocations most of the time, so new, small, objects are cheap so each operation is generally fast and structural sharing means its linear too)

emccue05:04:36

but if you build up something in a lazy sequence and expanding that sequence would take up stack space proportional to the size of the data structure, then you can run into issues

emccue05:04:37

recursively calling concat is the most well documented culprit

emccue05:04:30

I would not jump to transients immediately

emccue05:04:56

thats "tight loop, we profiled, needs to be faster" sort of stuff

Ben Sless06:04:18

Another point about performance, if it's really critical - avoid merge on your tight loop

Chris Lester08:04:54

Both interesting observations, how would merge by worse than map or reduce? ... and on lazy sequences and concat, That is pretty wild bug (read Stuart Sierra's comments on it) .. wonder why it was never patched (beyond this https://gist.github.com/jondistad/2a4971fe8948ca2f6ba0)

ataggart16:04:48

> how would merge by worse than map or reduce? That's odd to me too, given that merge is implemented via reduce.

ataggart16:04:22

lazy sequences and concat is usually dealt with via lazy-cat. The more common problem I've seen is holding onto the head of a lazy seq. Often this isn't noticed in testing, but then production-sized data causes a realized seq that comsumes a lot of memory. E.g.:

; bad
(let [xs  (some-lazy-seq ...)
      ... ...]
  (doseq [x xs]
    ...))
vs
; good
(let [... ...]
  (doseq [x (some-lazy-seq ...)]
    ...))

Ben Sless17:04:33

Why is merge slow? • It always iterates over an unknown sequence of maps • it merges maps via conj • instead of discarding nil values it conjs an empty map • Just to ballpark how slow merge is, merging two maps with 10 members takes ~1.5us. That's fast in human time but in relation to other clojure functions it's super slow

Ben Sless17:04:38

For comparison, this is about 20% faster:

;;; Credit Metosin
;;; 
(defn fast-assoc
  "Like assoc but only takes one kv pair. Slightly faster."
  {:inline
   (fn [a k v]
     `(.assoc ~(with-meta a {:tag 'clojure.lang.Associative}) ~k ~v))}
  [^clojure.lang.Associative a k v]
  (.assoc a k v))

;;; Credit Metosin
;;; 
(defn fast-map-merge
  "Returns a map that consists of the second of the maps assoc-ed onto
  the first. If a key occurs in more than one map, the mapping from
  te latter (left-to-right) will be the mapping in the result."
  [x y]
  (reduce-kv
   (fn [m k v]
     (fast-assoc m k v))
   x
   y))

seancorfield18:04:21

@UK0810AQ2 That's premature optimization unless you've already measured/profiled and established merge is a bottleneck (unlikely). For the vast majority of programs, that sort of optimization is pointless and moves you away from standard, readable, maintainable Clojure. The Metosin folks do a lot of crazy stuff to speed Clojure up -- they're kind of obsessed with performance-at-any-cost. Telling people to "avoid merge" is not helpful advice -- there are lots of core functions that can be optimized for specific cases but you wouldn't want Clojure programs to be made up of customized non-core functions everywhere 😞

Ben Sless20:04:26

@U04V70XH6 I agree, which is why I did not say to avoid merge at any cost, but: > avoid `merge` on your tight loop i.e you already know that's where you're spending cycles I guess my perspective is also colored by my day to day experience where I develop in Clojure and performance of long running processes is a concern > customized non-core functions everywhere That's sort of the capstone of clj-fast which I'm working towards - providing the most idiomatic extensions to core functions via inlining such that the API will stay exactly the same and the customized code will be generated by macros opportunistically. But I digress. I said in another thread that Clojure's core is good. I don't think I'd replace any part of it. I was more trying to shine a spotlight on areas of interest for people who might be concerned with performance, even at the cost of writing some custom code

seancorfield21:04:47

Fair enough. Tommi Reiman provided a lot of awesome performance-related feedback on next.jdbc while I was developing that and I've certainly got nothing against making stuff faster but the core team are very conservative about changes and always want to ensure that something that makes "use case A" faster doesn't also make "use case B" slower. In the end, with next.jdbc, the "happy path" isn't quite the fastest possible but there are extension points that allow faster result set builders so Tommi's use case could be satisfied.

👍 4
ataggart22:04:57

> Why is merge slow? This would be more helpful as "Why is merge slower than X?" Prior to the example code above, I would not have thought "X" was "using a bespoke implementation of merge/`assoc`".

👍 4
Setzer2207:04:27

I asked this yesterday but I think my comment got occluded by other discussions. Reposting to see if someone has any useful input :) A question about transients: I have a grid data structure represented as a vector of vectors, and I find myself updating it quite frequently. I would like to use transients to make the tight loops faster (and avoid unnecessary allocations!), but since it's a nested structure, I'm a bit confused. Should I recursively convert all the inner vectors into transients, then the outer vector, and then recursively convert everything back to persistent?

Timur Latypoff08:04:28

How bad is naive non-transient performance?

Ben Sless08:04:24

If you really care about performance, why not use arrays?

👍 4
Setzer2208:04:09

Just for a bit of context (even though I must admit it's a bit tiring having to justify myself each time I ask about optimization techniques...) I'm working on a game with Arcadia (Clojure CLR on top of Unity). I have some code that runs on the update loop, and is generating a ton of garbage (CLR's GC is worse than JVM's). In a frame where this particular function is run, there is GC pauses creating huge performance spikes, going down from roughly 60fps (more than acceptable) to less than 30 (noticeably worse). Since my function is a reduction that basically assoc-ins to an vector "grid", in the shape I described, I was hoping transients would help me alleviate all the unnecessary intermediate allocations. I'm still interested in keeping the full grid before and after the operation as an immutable object, so using arrays would be a huge downgrade in usability.

Ben Sless08:04:38

Don't feel bad, optimization has its place and Clojure is a hosted language and it would be a disservice to it not to reach down to the host platform when needed

Ben Sless08:04:12

second, even if using an array, you can populate a new array with the results instead of doing it in-place

Ben Sless08:04:34

And later do the switch if/when needed

Ben Sless08:04:53

if you look at amap it does exactly that

Ben Sless08:04:12

Just do a nested amap, it's fine 🙂

4
Ben Sless08:04:24

That way you can still save all the grid states, because yo always get a new grid, and you generate less intermediate garbage

andy.fingerhut08:04:26

I suspect that trying to use nested transients in Clojure might require a bit of care.

andy.fingerhut08:04:41

For example, if you have a vector of vectors, calling transient on one of the child/inner vectors returns a new object, which should ideally be replaced in the parent/outer vector.

andy.fingerhut08:04:34

Every assoc! call can perhaps return a different object than the one you passed to it, and so to be safest ought to cause a replacement in the parent/outer transient vector after every assoc! call on a child/inner vector.

andy.fingerhut08:04:32

You could 'batch' many assoc! calls to a single inner/child vector, and then do a singe assoc! call on the parent/outer vector to replace it, rather than doing assoc! on the parent/outer vector each time.

Setzer2208:04:12

@UK0810AQ2 that makes sense. Probably arrays are the way to go for those tight loops. I was asking more out of curiosity than anything else, since I would still prefer having a more functional-like but still performant solution.

Setzer2208:04:10

@U0CMVHBL2 thanks! It indeed looks a bit messy having to do all this... It would be awesome having built-in support for something like this in the language though... Having "flattened" data structures is so uncommon for me in day-to-day clojure that I always give up every time I try to reach for transients.

andy.fingerhut08:04:11

If you know the dimensions of your 2d vector, e.g. every row is the same number of elements as each other, so m rows of n elements each, you could create a single Clojure vector where accessing element (x,y) means accessing element (x*n + y) of the 1d vector.

andy.fingerhut08:04:25

Then there would never be two 'levels' of Clojure vector to worry about transients for.

andy.fingerhut08:04:48

but may not be a good fit for your use case for other reasons.

Setzer2208:04:35

Indeed, that's what I meant by flattened structures. I could do this, but then I need to choose between a more cumbersome API, or encapsulate everything behind a set of accessor functions. And if I have to encapsulate things anyway, I can just go the full way and use arrays.

Ben Sless08:04:13

@UP0Q30S10 I don't know if there is a more functional-like solution with reasonable performance. To be functional you need immutable values. For immutability you need either copying or structural sharing, both of which are terrible for your use-case. If even suggest a further optimization assuming you only need a before and after array: never copy the 2d array. Allocate two 2d arrays in advance, when you calculate the new array, just swap new to be the old, and do the manipulations on the old-now-new array. Zero garbage or new allocations

Setzer2209:04:20

I feel it goes against the philosophy of the language: We have lots of standard functions that operate seamlessly across different data structures. But as soon as you need performance, you have to abandon all that and resort to a very reduced set of low level functions

Setzer2209:04:15

@UK0810AQ2 Indeed! In fact, that trick is used a lot in the excellent book "Deep Learning for Programmers (in Clojure)".

Ben Sless09:04:43

I wouldn't say it goes directly against the philosophy of the language because clojure is also very pragmatic

Ben Sless09:04:05

And that assumption holds reasonably well as long as you don't build up huge collections in your tight loop 🙂

Ben Sless09:04:38

I also experimented with it myself and was able to implement sort of "loop-unrolling" macros for core clojure functions for more performance

Ben Sless09:04:10

But you can say that Clojure's abstractions aren't "zero cost"

Ben Sless09:04:20

which is fine, it's still very powerful

Setzer2209:04:43

Nice! I was aware of the clj-fast library but I didn't know I was speaking to the author 😅

Setzer2209:04:34

As a side note, do you think it might be easy to port it to another platform?

Ben Sless09:04:09

"the author" 🙃 I really think anyone could have done it, nothing magical (besides the lenses, maybe)

Ben Sless09:04:25

There's no reason it shouldn't work on the CLR, if that's what you're implying

Setzer2209:04:22

I was indeed implying that 😁

Setzer2209:04:24

I'll have a go later and see if I can make it run there...

Ben Sless09:04:37

I think even the interfaces are the same so the interop code should work just fine, but I never gave CLR clojure a deep look

Ben Sless09:04:45

There are some specific namespaces which rely on java collections, but the inline namespace should work

Setzer2209:04:48

Yes, it's more or less a 1:1 translation from Java, so all the interfaces should match. I see just a bit of JVM-specific stuff in concurrent_map.clj and map.clj

Ben Sless09:04:06

also in core.clj

Ben Sless09:04:28

ah, no, only clojure interfaces there

Ben Sless09:04:36

maybe it'll be portable as well

Setzer2209:04:34

it indeed looks portable... I will try to see if I can make it work with a few reader conditionals 😄

Ben Sless09:04:00

Go ahead. PRs welcome, etc

Setzer2209:04:18

nice! Can't promise I'll succeed, but if I do I'll let you know 🙂

👍 4
Ben Sless09:04:59

But back to your original issue, I don't know how it would look different, for example, in a fully-lazy language. Maybe a different way to think about it is of every index as a series developing in time, if it's independent of other indices, but if not you're getting into matrix operations and at that point you're doing place-oriented-programming anyway and there it's better to accept mutability as a fact of life

Setzer2209:04:01

Yeah.. I don't think looking for a "fully functional" approach is required. IMO, as long as we can treat it as pure from outside I don't care what happens on the inside to make it run faster. In fact that's a common promise that people make when you are taught any functional language (e.g., Haskell): "Look at all the crazy optimizations we can do if we embrace immutability!" But then none of them are actually done in practice... That local mutability is the promise behind transients, but then I guess it falls a bit short on delivering it fully 😅 I know it's not easy either. I would just love if the compiler analyzed the code and if it knew I'm not going to reference back the old value after an "assoc" so it just mutated it behind the scenes.

Ben Sless09:04:58

The good and bad of the clojure compiler is that it's dead simple. It has no way of knowing that value isn't going to be referenced outside

phronmophobic16:04:55

I’d be interested to see how a implementation using specter would compare. https://github.com/redplanetlabs/specter > In addition, Specter has performance rivaling hand-optimized code (see https://gist.github.com/nathanmarz/b7c612b417647db80b9eaab618ff8d83). Clojure’s only comparable built-in operations are get-in and update-in, and the Specter equivalents are 30% and 85% faster respectively (while being just as concise).

Setzer2212:04:15

I think specter does a lot of this internal optimizations itself, like unrolling a (get-in m [k1 k2 k3]) into (-> m k1 k2 k3) wen things are known statically

slipset08:04:33

For reasons https://github.com/borkdude/babashka/issues/263#, I changed https://github.com/clj-commons/ordered/commit/9f7d36352827777d5f2316c133fe62ddb27a18d3. From the perspective of flatland/ordered, this seems totally fine. However, when using this in clj-yaml (https://github.com/clj-commons/clj-yaml/pull/10) The build breaks with

Reflection warning, flatland/ordered/map.clj:113:5 - reference to field size on clojure.lang.IPersistentMap can't be resolved.
Reflection warning, flatland/ordered/map.clj:117:5 - reference to field isEmpty on clojure.lang.IPersistentMap can't be resolved.
Reflection warning, flatland/ordered/map.clj:119:5 - reference to field keySet on clojure.lang.IPersistentMap can't be resolved.
This leads me to at least two questions: 1. What introduced these reflection warnings? 2. How do I call size, isEmpty , and keySet on an ordered-map.`(size (ordered-map))` tells me
flatland.ordered.map> (size (ordered-map))
CompilerException java.lang.RuntimeException: Unable to resolve symbol: size in this context, compiling:(*cider-repl ) 
flatland.ordered.map> 

borkdude09:04:14

@slipset It seems the ordered library is missing reflection warnings. I put (set! *warn-on-reflection* true) at the top and those warnings appear when I run the tests. This doesn't fix your problem, but it might be a good practice to include it.

borkdude09:04:19

@slipset Maybe just doing this fixes it:

user=> (def x {})
#'user/x
user=> (.size ^java.util.Map x)
0
user=> (.size x)
Reflection warning, NO_SOURCE_PATH:1:1 - reference to field size can't be resolved.
0

borkdude09:04:54

@slipset I see there are more such type hints, like here:

(if-let [^MapEntry e (.get ^Map backing-map k)]
      (.val e)
      not-found)

borkdude09:04:25

@slipset I also opened two follow up PRs, 52 and 53 which clean up the code.

andy.fingerhut09:04:48

I do not know the answer yet. Have you tried looking at a macro-expanded version of the older definition of the ordered map data structure that used the delegating-deftype macro, to see how that differs with the new deftype?

borkdude09:04:54

I think I understand. this delegating-deftype takes an options map:

{backing-map {IPersistentMap [(count [])]
                Map [(size [])
                     (containsKey [k])
                     (isEmpty [])
                     (keySet [])]}}
which it probably uses to figure out which methods belong to which interfaces and uses that to "delegate".

andy.fingerhut09:04:51

I am looking at the output of (pprint (macroexpand-1 (delegating-deftype ...))) from the former version of the ordered library, and comparing it against the latest version of the ordered library. I am pretty sure that if there is any metadata in the macro expansion, pprint will not show it, so I could be missing that in what I am looking at.

andy.fingerhut10:04:34

I do see one difference that I do not know whether it makes a difference yet: the delegating-deftype version of the code produces this definition for the Map interface's method size : (size [_] (. backing-map (size))) . The latest version of the ordered lib has this in its place: (size [this] (.size backing-map)) . I doubt the _ vs. this makes a difference, but I wonder whether the other syntax differences might make a difference.

andy.fingerhut10:04:00

It isn't clear to me yet what size method in what JVM class or interface was being resolved to in the former version of the ordered lib code there.

andy.fingerhut10:04:29

Adding a type hint ^Map to the three occurrences of backing-map in the three methods that give the reflection warning, eliminates the reflection warnings, and might be what the former version of the ordered library code was effectively doing?

slipset12:04:24

RTFS’ing a bit shows that that is exactly what the the useful library was doing:

slipset12:04:03

And macroexpansion doesn’t show meta on the things.

andy.fingerhut10:04:59

Sorry, I'm slow. I see that the PR you mention above suggests this change. I do not see any problems with that change.

pinkfrog11:04:03

I am concerning about the efficiency of a typical pipeline-is clojure code.

pinkfrog11:04:12

the solution iterates over all possible subset, and filters out those are needed. this style is typical in clojure, but lacks branch cutting , hences redundant work are performed.

victorb11:04:00

I'm about to do something highly forbidden and dark magic. I receive bunch of JSON data and I want to generate functions in namespaces and write that to disk. Current solution I'm using is having string templates and interpolation, then just spit that to disk. Is there any cleaner solutions? Previously tried to get this to work differently by having lists and pprint it into disk but the files I'm writing needs to have function calls that are not available in the context that reads/spits the files, so ended up with the string interpolation solution. But, might be a better solution out there

danielglauser12:04:46

Sounds fun. Templates are probably your safest bet, otherwise you’re getting into parsing your own language territory. If you want it to run faster and load the new code into memory you can look at this with the requisite caveat emptor warning: https://clojuredocs.org/clojure.core/read

victorb06:04:03

Thanks for that pointer! :thumbsup:

Ramon Rios12:04:10

Is there any option to wrote logs into a file using clojure.tools.logging ?

hindol12:04:41

Yes. You select an implementation from here: https://github.com/clojure/tools.logging#selecting-a-logging-implementation Then configure that implementation to write to a file.

slipset13:04:36

While I’m at it. Is there a difference between (. backing-map (size)) and (.size backing-map)) apart from the syntax?

slipset13:04:37

Any history on why we have the two forms?

Alex Miller (Clojure team)13:04:16

the latter is sugar for the formerr

Alex Miller (Clojure team)13:04:57

the first one is the . special form, which supports many interop cases (instance fields, static fields, instance invocation, static invocation)

Alex Miller (Clojure team)13:04:20

the other stuff is all sugar to make it nicer for specific cases

slipset13:04:00

But not macro-sugar, right?

Alex Miller (Clojure team)13:04:15

it's implemented in the compiler, not in a macro, if that's what you're asking

slipset13:04:44

Exactly what I was asking 🙂 Thanks.

amalantony13:04:12

What is the idiomatic way to maintain config files in a Clojure project? I’m currently thinking of having a `config.edn` file in the root of my lein project that I’ll parse with `(edn/read-string (slurp "config.edn"))` . Are there better ways?

p-himik13:04:54

If you decide to go that route, https://github.com/juxt/aero/ can help with that.

👍 4
tvaughan21:04:39

I too would recommend aero

FlavaDave13:04:04

Trying to sort a map of maps based on the value of a key. also the value in the key needs to be pulled from a string that is an alphacode and then the integer I want to sort by. I have already gotten the integer out with a regex. Just having a hard time with the sort

p-himik13:04:40

Can you give an example of such data?

FlavaDave14:04:51

{:expand "example string",
           :self "",
           :key "DRO-61",
           :fields {:description "example string",
                    :duedate nil,
                    
                             :iconUrl "example.url",
                             :name "To Do",
                                                             :id 2,
                                              :key "new",
                                              :colorName "blue-gray",
                                              :name "To Do"}}}}

FlavaDave14:04:19

I want to sort by the number at the end of :key

p-himik14:04:56

You said "map of maps". All I see is a nested map that's hard to read due to formatting and that has unbalanced closing braces.

p-himik14:04:45

You either need sorted-map-by with the right comparator or sort-by with the right key function that extracts that number from :key.

FlavaDave15:04:01

ahh sorry i can clean that up

FlavaDave15:04:20

that was an example of one map. in the map of maps

p-himik15:04:33

What is "map of maps"? Is it map -> some_value, some_value -> map, or map -> map?

FlavaDave18:04:44

here is a better example:

FlavaDave18:04:51

{{:a "ABC-03"
  :b "string"
  :c "string"}
 {:a "ABC-01"
  :b "string"
  :c "string"}
 {:a "ABC-02"
  :b "string"
  :c "string"}}

p-himik18:04:27

Alas, it is not. Try to evaluate this form to see why.

FlavaDave18:04:17

crap im sorry it is

[{:a "ABC-03"
  :b "string"
  :c "string"}
 {:a "ABC-01"
  :b "string"
  :c "string"}
 {:a "ABC-02"
  :b "string"
  :c "string"}]

FlavaDave18:04:11

not a map. I apologize

p-himik18:04:05

Ah, now it's clear. You need sort-by with a key function that just extracts that number. Do you know how to write that?

FlavaDave18:04:35

(re-find #"\d+$" :a) is what I was trying to do but I feel like im close but no cigar

p-himik18:04:48

How do you want to compare those numbers? As regular numbers without leading zeros, or as 8-based numbers (because of the leading zero), or as strings?

FlavaDave18:04:28

reg numbers without leading zeros

p-himik18:04:26

Try (sort-by #(->> % (re-find #"[1-9]\d*$") Integer/parseInt) your-collection-of-maps).

FlavaDave18:04:39

Classcastexception clojure.lang.PersistentArrayMap cannot be cast to java.lang.CharSequence clojure.core/re-matcher (core.clj:4789)

p-himik18:04:47

Ah, sorry, replace that % with (:a %) or whatever your key is.

FlavaDave18:04:02

lol you know, in my head i was like “what kinda clojure magic is he doing where he doesn’t need the key?”

FlavaDave18:04:20

returns

({:a "ABC-01", :b "string", :c "string"}
 {:a "ABC-02", :b "string", :c "string"}
 {:a "ABC-03", :b "string", :c "string"})

p-himik18:04:44

Seems like exactly what you need, no?

FlavaDave19:04:18

lol sorry i forgot the thank you part. was in a meeting trying to multitask

FlavaDave19:04:25

Thank you!

👍 4
borkdude14:04:51

I didn't have an answer to this, so I'm going to try again: In normal JVM Clojure when executing a script, is there a way of detecting whether you're being called with a main argument -m foo or just executing as a script file?

(defn -main [& args]
  (println "args" args))

(when ... ;; we're running as a script
  (apply -main *command-line-args*))
I'm considering setting a system property in babashka so you can do (when-not (System/getProperty "babashka.main") (apply -main *command-line-args*)) unless there are already patterns in Clojure to handle this.

slipset14:04:58

In shell-scripts you can take a look at $0 to see the name under which the script was invoked. Maybe you can do something similar?

borkdude15:04:59

This appears to work:

$ bb -e '(System/getenv "_")'
"/Users/borkdude/Dropbox/bin/bb"
But you only get the program name, not all the raw arguments.

borkdude15:04:18

hmm, that doesn't appear to work with clj.

rutledgepaulv02:04:29

Could you check the stack trace and see which entrypoint you came through?

jumar10:04:40

@U04V15CAJ would the sun.java.command work for you? These two print its content:

$ clj hello.clj
...
clojure.main hello.clj

$ clj -m hello
...
clojure.main -m hello

borkdude10:04:20

Everything starting with sun is considered private / implementation detail. Therefore it's not supported / present in GraalVM binaries.

borkdude10:04:47

I've already chosen the babashka.main system property.

Risque14:04:10

I have the following build.boot file,

(set-env!
 :resource-paths #{"src"}
 :dependencies '[[me.raynes/conch "0.8.0"]
                 [boot.core :as boot]])
(task-options!
 pom {:project 'myapp
      :version "0.1.0"}
 jar {:manifest {"Foo" "bar"}})



(boot/deftask cider "CIDER profile"
  []
  (require 'boot.repl)
  (swap! @(resolve 'boot.repl/default-dependencies)
         concat '[[org.clojure/tools.nrepl "0.2.12"]
                  [cider/cider-nrepl "0.15.0"]
                  [refactor-nrepl "2.3.1"]])
  (swap! @(resolve 'boot.repl/default-middleware)
         concat '[cider.nrepl/cider-middleware
                  refactor-nrepl.middleware/wrap-refactor])
  identity)
following this documentation: https://github.com/boot-clj/boot/wiki/Cider-REPL However, upon doing cider-jack-in I get the error "refusing to run as root. set BOOT_AS_ROOT=yes to force.", and yet after doing export BOOT_AS_ROOT=yes, I get the same error. What's wrong?

borkdude15:04:00

maybe you can insert (prn (System/getenv "BOOT_AS_ROOT")) in your build.boot file just to make sure it's set correctly. maybe also try in #boot

Risque17:04:13

Hi my boot repl starts in this boot.user namespace

Risque17:04:25

And then I change the namespace in cider to user

Risque17:04:42

And my user.clj has this nice (go) function (def go reloaded.repl/go) that reruns my server

Risque17:04:05

However, when I change the namespace and run (go), I get: java.lang.RuntimeException: No such var: user/go

Risque17:04:13

why would that be?

Risque17:04:16

The user.clj file is in the dev/ directory

Risque17:04:34

and my boot config is

(set-env!
 :resource-paths #{"src" "dev"}
 :source-paths #{"dev"}
 :dependencies '[[me.raynes/conch "0.8.0"]
                 ])

Risque17:04:59

so the dev directory is in the resource-path and the source-path too.

hiredman17:04:19

the user.clj needs to be on the classpath when the clojure runtime is initialized to be loaded

hiredman17:04:29

so adding it after the fact with set-env! does nothing

Risque17:04:28

what's the fix then

Risque17:04:49

How to put the user.clj in the class path beforehand?

hindol17:04:24

Some info here: https://clojureverse.org/t/how-are-user-clj-files-loaded/3842 EDIT: Sorry this does not address the issue specific to boot.

Risque17:04:31

@hiredman also, this isn't the case when in a lein project, where

:source-paths ["dev"]
just works

hiredman17:04:27

lein and boot are not the same

Risque17:04:08

@hindol.adhya that looks too complicated for such a small issue

hindol17:04:37

Yeah, sorry, that link is not specific to boot.

hindol17:04:36

Not used boot, but what I feel hiredman is saying is, you need to compile user.clj beforehand and then run the boot task.

hindol17:04:04

Using something like (compile 'dev.user).

Risque17:04:55

I get: http://java.io.FileNotFoundException: Could not locate dev/user__init.class or dev/user.clj on classpath.

hindol17:04:31

Sorry that should be (compile 'user).

hindol17:04:48

Provided user is your namespace.

Risque18:04:30

that gives:

Risque18:04:31

java.lang.ClassCastException: http://java.io.File cannot be cast to java.lang.String clojure.lang.Compiler$CompilerException: java.lang.ClassCastException: http://java.io.File cannot be cast to java.lang.String, compiling:(user.clj:1:1)

hindol18:04:45

I think I get what he said. He said when Clojure runtime is initialized, dev itself is not in your classpath.

hindol18:04:21

You are injecting it later with set-env! and that won't work.

Risque18:04:36

so how to set it before?

hindol18:04:04

What happens when in boot REPL you evaluate the whole user.clj file? Instead of just (go)?

hindol18:04:37

And looking at the error you posted, seems there's some issue with your user.clj as well. Possible to post it somewhere?

Risque18:04:34

Could not locate myapp/application__init.class

Risque18:04:48

or myapp/application.clj on classpath

Risque18:04:03

in cider-error buffer

Risque18:04:01

and that myapp/application.clj is in the src directory, which I have in my resource-paths

hindol18:04:56

Seems something wrong with the setup itself. Do you have a particular reason to use boot?

Risque18:04:26

What could go wrong with the setup/

hindol18:04:49

Sorry, I am out of ideas actually.

hindol18:04:06

But if you are willing to migrate to deps.edn based setup, I can help.

seancorfield18:04:16

There's a #boot channel where folks are more likely to be able to help. Not many people use Boot these days.

Risque17:04:53

There's got to be a way that doesn't feel like an overkill

zane19:04:16

I recognize this is probably just a question of taste, but I'm curious: Which of the following feels clearer? Option 1:

(if (test? x)
  (change x arg)
  x)
Option 2:
(cond-> x (test? x) (change arg))

1️⃣ 12
2️⃣ 16
seancorfield19:04:34

(I wish there were a variant of cond-> that threaded through the predicate as well as the expression: (condp-> x test? change) )

🎉 12
4
4
jjttjj19:04:39

I did this the other day :grinning_face_with_one_large_and_one_small_eye:

(as-> x x
       (cond-> x (or (tspec/year? x)
                   (tspec/year-month? x)) t/beginning)
       (cond-> x (tspec/local-date? x) t/midnight)
       (cond-> x (or (tspec/local-date-time? x)
                   (tspec/instant? x)
                   (inst? x))
               (t/in (t/zone))))

jjttjj19:04:55

which isn't too bad for not having to use a custom macro (imo)

ghadi19:04:10

note: the word "change" doesn't usually appear in Clojure source, because it implies mutate-in-place

4
ghadi19:04:04

("update" implies mutation somewhat, too. Words are hard)

zane20:04:02

While we're talking about cond->, is folding updates that should always happen into cond-> forms good style or bad style?

(cond-> x 
  (test1? x) (sometimes-update1 args1)
  (test2? x) (sometimes-update1 args2)
  true (always-update))

zane20:04:14

Could also consider replacing true with :always

manutter5120:04:00

I usually just do (cond-> (always-update x) ,,,) or similar

zane20:04:37

Right on! I'm looking at how to handle situations where the order of updates matters, though.

manutter5120:04:43

I have sometimes used true (always-update) inside a cond->, so I wouldn’t say it’s terrible.

4
seancorfield21:04:01

I tend to use :always instead of true but I think it's reasonable for occasional use. Anything more complex, I break down into multiple bindings or I write specific transformation functions that encapsulate the conditional logic.

☝️ 8
zane01:04:37

Makes sense. Thanks, Sean!

nick20:04:13

What are my options if I want to control certain aspects of a running program in runtime? For example, "custom backoff multiplier" config value or smth like that. JMX (writeable) operations are not currently supported in Clojure. Remote repl to production to change some atom - this doesn't seem to work for clustered environments(or does it?) Yes, I can probably store this data in the database but I was looking for something on a higher level(web server? JVM?) with faster access time.

Ben Sless20:04:09

especially in a cluster environment, how about writing the updates to something like a kafka topic which all the instances will subscribe to, and will keep a process which will update the configuration atom from there?

💡 4
Ben Sless20:04:09

But I wonder if it's possible to implement a sort of multi-repl which connects to multiple machines in parallel :thinking_face:

noisesmith20:04:41

there's also zookeeper, which kafka builds on top of, and is directly meant for synchronized shared changing state

💡 4
4
noisesmith20:04:57

but the kafka API is nicer to use (if more complexity ops wise...)

nick20:04:21

Thanks for the answers!

noisesmith20:04:01

I'm also not convinced that one can't use JMX from Clojure, but I don't know JMX well enough to have a counterexample handy

ghadi20:04:15

shouldn't do anything with push model IMHO

ghadi20:04:39

easier to reason about+secure pull model

👍 4
ghadi20:04:14

could have an atom that has current config in it, and a background process that periodically swaps the latest value from $database

👌 4
Ben Sless20:04:31

Can you elaborate on that? I find myself being a fan of push over pull

ghadi20:04:15

push requires you to open a connection

ghadi20:04:25

pull requires a Control Plane

Ben Sless20:04:03

Would you consider subscribing to a queue topic (kafka or some mq) as push or pull?

ghadi20:04:23

kafka would be a pull, but I wouldn't recommend it because it's not enough on its own

ghadi20:04:45

don't want to scan the log to figure out what the current value should be

ghadi20:04:57

(yes, compaction, etc....but Kafka is generally overkill)

ghadi20:04:41

JMX is a bag of sad, anyways

ghadi20:04:07

"Config DB" could be periodically reading an EDN file in S3

ghadi20:04:27

(you want versioning on that object, but the point still stands)

👍 4
nick20:04:48

Thanks Ghadi! Very helpful

ghadi20:04:15

btw this is how the Envoy Proxy works, too

ghadi20:04:25

there are some brains in the middle,

ghadi20:04:45

and all the individual proxies download their http route tables and config from the brains, periodically

ghadi20:04:21

eventual consistency for db transactions is a bad idea, but eventual consistency for cluster config is a great model, IMHO

lukasz21:04:30

Consul is a popular solution for this kind of stuff

nick22:04:45

@U0JEFEZH6 I guess you were referring to the "Consul KV" service, right? HTTP API interface must be pretty handy for the cases when it is not requested too frequently. Thanks for the idea, last time I used Consul they didn't have that product

lukasz22:04:51

@U0113AVHL2W yes, http://consul.io - if I understand requirements correctly, it fits the bill. It's really solid and easy to deploy, something I cannot say about Kafka (I wouldn't recommend it for this use case). If you want push/pull - Consul supports notifications/watches as well

👍 4