Fork me on GitHub
#clojure
<
2020-02-04
>
doby16214:02:20

Hey friends, what are the performance characteristics of swap! ? I have a program that stores a bunch of state in a single atom, (roughly 30 kb of data once exported to json, but that should grow substantially as the simulation becomes more complex) and calls swap! on that atom a minimum of once per frame, which is roughly 20 times a second. I wonder partly because I seem to have a pretty bad memory leak, I haven't had time to really search it out yet but I thought I would check if my whole premise was an issue before I search too hard simple_smile profiling indicates that it comes from the hottest loops, which is where the global atom is being processed and swapped. Thanks!

andrzej.fricze14:02:15

swap! is, most likely, not your bottleneck. It’s pointer change. How do you change data structure kept in atom? Problem would rather be in processing data before swap! call, or any code that listens to atom change and processes new value.

doby16214:02:56

I change it lots and lots of ways! Thanks, that makes sense, glad to know I'm not totally misusing swap 😄

teodorlu14:02:06

Have you tried benchmarking (swap! your-atom your-change-fn) vs (your-change-fn some-data)? I'd suspect your-change-fn might be a major contributor.

doby16214:02:51

I haven't yet! The profiling made it look like swap was the hotspot and I didn't find any information on if it was ok to run swap dozens of times a second, this narrows my search a lot!

retrogradeorbit15:02:57

Hi everyone. Is there any way to convey a dynamic var's binding into the body of a list comprehension without using another var?

user=> (def ^:dynamic *foo* 0)
#'user/*foo*
user=> (binding [*foo* 1]  *foo*)
1
user=> (binding [*foo* 1] (for [n [1]] *foo*))
(0)
user=> (binding [*foo* 1] (let [p *foo*] (for [n [1]] p)))
(1)
I want the (0) to show (1) without doing the last line. I want *foo* itself to resolve to 1. Is this possible?

p-himik15:02:12

(binding [*foo* 1] (doall (for [n [1]] *foo*)))

p-himik15:02:34

The issue here is that the lazy collection returned by for is realized only outside of the binding.

retrogradeorbit15:02:25

thanks @p-himik !

kwladyka15:02:32

What is the fastest way to reset the REPL to the state when it is freshly runed? I want be sure all ns are unloaded and I don’t want to lose time for reset REPL each time to be sure if dependencies are correctly reduced.

kwladyka15:02:06

@borkdude I don’t think this is what I need. Does it demand Sierra component? Also the issue is not reloading, but unloading in this case.

kwladyka15:02:33

I remove dependency from ns dependencies and reload ns. It works. But if I kill REPL and load ns then not.

kwladyka15:02:44

ok I found manually which dependency make the issue, but anyway I am interested in how to reset the REPL without kill and run again

nickmbailey15:02:04

But first, it will unload (remove) the namespaces that changed to clear out any old definitions.

nickmbailey15:02:14

which i think is what you want

kwladyka15:02:07

Hmm it could work. I have to try. Thanks.

vemv15:02:37

yes, I think you seek refresh (only affects namespaces), as opposed to reset (affects a System)

mac0102116:02:44

What does one typically do when your deps.edn includes dependencies that have transitive dependencies that are only available in 3rd party Maven repositories? Is deps expected to be able to handle this case?

ghadi16:02:41

@mac01021 there is a :mvn/repos option in the deps.edn

ghadi16:02:47

you should add the repository there

ramon.rios18:02:29

Does anyone here uses postal for sending email by SMTP?

grounded_sage18:02:43

What libraries are people using for testing http calls?

dpsutton18:02:25

we use postal to create messages but just use javax.mail to send it

ramon.rios19:02:00

My doubt it was i was able to send a attachment by a inputstream

ramon.rios19:02:32

Because i saw that i can send with http://java.io.File

kwladyka19:02:51

I was using Postal in the past, but I don’t remember too much

noisesmith19:02:57

I haven't use .File with postal, but my experience with the jvm generally is that people often assume they need a File when really they could use something more abstract, or not reliant on using a fileystem

noisesmith19:02:47

eg. io/resource to read from inside a jar, or ByteArrayInputStream to read directly from bytes in memory

hiredman19:02:47

I think postal shells out so it may actually need a file

kwladyka19:02:19

If I remember postal has :attachments key in map so you can set paths to files.

kwladyka19:02:23

I didn’t use any other method

kwladyka20:02:30

oh actually it is not a path ,but http://java.io.File so probably you can use stream too

jakob.riishede.moller20:02:12

Remember that (+ [1 2 3]) is erroneous - I wonder why (max [1 2 3]) is not - the result is [1 2 3] even though no ordering of vectors is defined: (max [1 2 3] [4 5 6]) is erroneous as well. I think it would have made more sense to return an error for (max [1 2 3])?

hiredman20:02:15

There is a defined order for vectors, max just doesn't use it

jakob.riishede.moller21:02:06

OK? For all pairs of vectors? Which is greater [1 50 0] or [-100 1000 9]?

hiredman21:02:29

Use compare and see

jakob.riishede.moller21:02:52

OK - I tried and got some 1 and minus 1 results - does it make sense? - to me it does not - please explain.

jakob.riishede.moller21:02:16

And why does max not use that order then?

hiredman21:02:49

The doc string for compare explains the returned numbers

hiredman21:02:06

max only works on numbers

jakob.riishede.moller21:02:46

It seems that compare is another buggy function - unless you can make sense of that order for any given pair of vectors?

hiredman21:02:19

it is a well defined total order

jakob.riishede.moller21:02:36

Ha ha - it seems to compare strings?

jakob.riishede.moller21:02:00

Yes - I read that - try to explain the math

hiredman21:02:11

it isn't math

jakob.riishede.moller21:02:31

so max is not a mathematical function?

hiredman21:02:45

I forget the exact algorithm vectors using when comparing

hiredman21:02:12

like I said, max is only for numbers, which is why max doesn't use compare, which is a generic way to compare things

jakob.riishede.moller21:02:43

OK - which is intuitively biggest: [0 1] or [1 0]?

hiredman21:02:52

sort for example (which doesn't actually use clojure.core/compare, but is built on Comparable) can sort lists of vectors

hiredman21:02:04

it doesn't matter to some degree

hiredman21:02:13

there is a defined order though

jakob.riishede.moller21:02:21

But one must be biggest if there is an order?

hiredman21:02:57

and clojure.core/compare tells you

jakob.riishede.moller21:02:58

And I guess it is [1 0]

jakob.riishede.moller21:02:35

But it makes no sense from a mathematical viewpoint - remember I asked about max

hiredman21:02:13

the algorithm is, a vector with fewer elements is always less than a vector with more, and if vectors have the same number they are compared element by element until they differ, returning the first difference

hiredman21:02:47

you said "even though no ordering of vectors is defined"

jakob.riishede.moller21:02:52

Yes - fine - I wonder what the use of that is

hiredman21:02:54

and I am correcting that

hiredman21:02:09

sorting things is extremely useful

jakob.riishede.moller21:02:18

OK - from a mathematical viwepoint there is no ordering

hiredman21:02:52

I dunno why you keep saying mathematical viewpoint

jakob.riishede.moller21:02:13

Becausemax is a mathematical function

hiredman21:02:15

math is perfectly fine with non-numeric objects and arbitrarily defined orderings

jakob.riishede.moller21:02:16

Do you know what a mathematical ordering is?

hiredman21:02:53

I think you may be saying math and mathematical when you mean numeric, but please enlighten me

jakob.riishede.moller21:02:13

Anyway - there does not seem to be a such for vectors in clojure - at least it does not apply to the max function

jakob.riishede.moller21:02:53

And that was my point - sorry if it was not clear

hiredman21:02:57

still isn't clear, you keep saying there isn't a mathematical ordering for vectors, and as far as I know math would be fine with the ordering that is defined

hiredman21:02:52

> The notion of order is very general, extending beyond contexts that have an immediate, intuitive feel of sequence or relative quantity. In other contexts orders may capture notions of containment or specialization. Abstractly, this type of order amounts to the subset relation, e.g., "Pediatricians are physicians," and "Circles are merely special-case ellipses."

jakob.riishede.moller21:02:23

So again - an order can be defined that makes [1 0] larger than [0 1] - but it does not apply to tha max function

jakob.riishede.moller21:02:42

No - exactly max does not work for vectors

hiredman21:02:58

I see, yes, by "max doesn't use it" I meant: there is a defined order for vectors and max doesn't use it, and since max doesn't use the defined order for vectors, it doesn't work on vectors

jakob.riishede.moller21:02:28

Fine - I know that you can define orderings and partial orderings, but that was not my point here

hiredman21:02:53

no idea what your point is, I was just correcting a specific part of what you said, that there is no ordering of vectors defined, which as demonstrated by both compare and sort there is

jakob.riishede.moller21:02:13

Like reading out of context?

noisesmith20:02:23

@jakob.riishede.moller many clojure functions return identity for a single arg without consuming it or checking its validity in any way

noisesmith20:02:17

user=> (conj :a)
:a

noisesmith20:02:49

this is useful for cases where you would reduce across an input collection and need a sensible base case

jakob.riishede.moller20:02:55

Is [1 2 3] a sensible base case for a function that computes the maximum?

noisesmith20:02:21

and identity of [1 2 3] is [1 2 3]

noisesmith20:02:19

it works accidentally, whether clojure should eagerly reject invalid inputs is a ship that has sailed already

jakob.riishede.moller20:02:25

Well, if at least there was consistency, so that (+ [1 2 3]) returned [1 2 3]?

noisesmith20:02:39

I agree that the inconsistency is annoying, I doubt there's an easy way out. That said I find clojure generally more consistent than most languages I've used.

noisesmith20:02:58

Pragmatically, I think it's better to avoid invalid inputs (knowing you can't count on being saved from them by the compiler or the runtime). I have my own pet peeves (eg. merge accidentally working, sometimes, on things that are not hash-maps)

noisesmith20:02:27

or (Math/PI) working in the place of Math/PI (despite not being an invocation)

jakob.riishede.moller20:02:15

Well - thank you @noisesmith - at least: now i know :+1:

lilactown20:02:13

Are there any libs or examples of neat ways of interacting with nav-able structures at a REPL

alexmiller20:02:36

REBL works with Cognitect's AWS lib, dirs/files, data for charting, Java class hierarchies, jdbc data, etc

seancorfield20:02:57

@lilactown next.jdbc produces datafyable/`nav`able result sets by default from execute! and execute-one!

lilactown20:02:08

I wasn’t clear enough. I am not looking for examples of datafy and nav, and I know about REBL. I’m wondering if anyone has explored a more ergonomic way of interacting with datafy/nav directly at a REPL. Sort of like https://github.com/eggsyntax/datawalk

lilactown20:02:37

(preferably that you could evaluate in a buffer though, so datawalk doesn’t fit exactly what I’d like)

lilactown20:02:27

I guess the “semi-interactive” version of datawalk is basically what I’m thinking of

ghadi21:02:40

omg I'm dying waiting for that datawalk gif to load

ghadi21:02:42

you can totally make something like datawalk for nav, but then you'd be re-building REBL @lilactown

lilactown21:02:14

yeah it takes forever

lilactown21:02:33

I understand. the tradeoffs I’m thinking about for REBL are:

lilactown21:02:54

• doesn’t work in CLJS (which is where I actually want to use this right now) • have to leave my editor to use it

aisamu21:02:27

You might already know about it, but shadow-cljs has a an "inspector" which is quite handy!

lilactown21:02:26

I do know that. like I said, there’s already a purpose built external devtool for this. I specifically want something I can use in my editor / REPL

lilactown21:02:47

for my use case, there’s already a purpose built set of devtools for visualizing the objects I want to datafy and nav and I’m interested in building some utilities to interact with them at the REPL

lilactown21:02:22

I was hoping I could implement some wrappers around it with datafy + nav and call it a day 😉 but looks like I’d have to build it myself anyway

dpsutton21:02:40

i think @thheller is cooking something exciting up along these lines

ghadi21:02:01

datawalk doesn't seem work in nREPL, womp womp

lilactown21:02:40

thheller’s inspector looks awesome but, again, there is already an excellent external devtool for inspecting this data; I specifically want something that works at a REPL. I have my answer, which is, this does not exist yet as far as we know 😄

thheller21:02:49

I don't think you can get very far if you constrain yourself to a text interface

noisesmith21:02:40

as long as you can assume no background thread is trying to print, you could have a curses UI inspector

noisesmith21:02:14

not sure if it would be worth the trouble to implement...

lilactown21:02:16

There are certain use cases that are better at a REPL imo

lilactown21:02:48

I'm trying look at the current state and props of a react component instance without leaving my editor

lilactown21:02:45

It's simply walking the react fiber tree and finding where the element type equals the component I want to inspect

lilactown21:02:40

But react fibers are opaque data (to CLJS) so I thought I could buy myself some time and effort and use datafy/nav. But datafy/nav is too raw to really be usable at the REPL for that

hiredman21:02:34

sounds like you just want tree-seq and filter

hiredman21:02:45

navigating is such a low level thing, don't do it, be lazy, search

ctamayo22:02:13

This code:

(defn- grant->pin [grant]
  {:id (:id grant)
   :amount_recommended (:amount_recommended grant)
   :organization_name (:name grant)})

(defn aggregate-pin-data [data]
  (-> data
      (get-in [:records :grant_request])
      #(map grant->pin %)))
is throwing:
class clojure.lang.Symbol cannot be cast to class clojure.lang.IPersistentVector (clojure.lang.Symbol and clojure.lang.IPersistentVector are in unnamed module of loader 'app')
Which symbol is it trying to cast to a vector? Seems to be complaining about the #(map ...) lambda but beyond that I'm a bit lost...

lukaszkorecki22:02:20

(defn aggregate-pin-data [data]
  (-> data
      (get-in [:records :grant_request])
      (#(map grant->pin %))))

lukaszkorecki22:02:23

try this ☝️

ctamayo22:02:07

that works!

ctamayo22:02:02

I could've sworn I've passed lambdas like that to -> , does this not work at all? Or only in specific cases?

lukaszkorecki22:02:28

It shouldn't work

noisesmith22:02:29

@ctamayo it helps to look at the macroexpansion of the -> form

ctamayo22:02:58

@noisesmith I should've thought of that 😛

noisesmith22:02:03

user=> (macroexpand '(-> x #(+ % %)))
(fn* x [p1__224#] (+ p1__224# p1__224#))

noisesmith22:02:22

notice how x becomes the name of the fn, and doesn't appear in the body at all

ghadi22:02:44

don't try to fit everything into a -> chain

ghadi22:02:55

it's not like a Street Fighter combo

noisesmith22:02:19

here's how the (()) hack fixed it:

user=> (macroexpand '(-> x (#(+ % %))))
((fn* [p1__228#] (+ p1__228# p1__228#)) x)
and agreed with @ghadi, it's cleaner not to try to wrap it like that

emccue22:02:52

->->><?as->cond->!

emccue22:02:01

haaadouken

ctamayo22:02:39

that'll finish him.

ctamayo22:02:26

anyway, the (#(,,,)) hack definitely feels gross...what's the suggested style?

lukaszkorecki22:02:36

The simplest: extract a function

lukaszkorecki22:02:48

it took me a while to get that to be honest

noisesmith22:02:11

use a let, or arguably (-> x (f y) (->> (map g))) but even this is controversial

noisesmith22:02:02

->> / as-> etc. are intentionally designed so that they can nest inside -> fwiw, but opinions about actually doing so are mixed

emccue22:02:13

(->->><?as->cond->! x
 :-> (f y)
 :->> (map g))

noisesmith22:02:45

OK poe's law definitely got me on that one

ctamayo22:02:12

I'm gonna go with "yikes"

mark54023:02:35

Without threading it's pretty simple.

(defn aggregate-pin-data [data]
    (map grant->pin
      (get-in [:records :grant_request] data)))

borkdude23:02:13

if you're not using randomcorp's thread-first-thread-last-backwards-question-mark-as-arrow-cond-arrow-bang, you're missing out: https://github.com/randomcorp/thread-first-thread-last-backwards-question-mark-as-arrow-cond-arrow-bang Also check out the video: https://youtu.be/jlPaby7suOc?t=699

ctamayo23:02:07

Thanks for the much-needed reminder. I've literally seen every clojure talk ever....But I guess I still haven't learned my lesson because I'm not modeling this in XML.