Fork me on GitHub
#clojure
<
2019-04-17
>
currentoor00:04:32

I have in coming datastructures for requests like this

[{(ucv.ui.root.signon/login {:email "", :password "letmein"})
  [:session-key :logged-in? :user-id :firm-id :pos/id :user/id :firm/id]} :com.wsscode.pathom/trace]
And I want to log them, but omit any whitelisted keys (like :password), but otherwise log the datastructure completely as is (don’t want to change vectors to lists or anything). What the recommended way of doing that?

currentoor00:04:51

I tried clojure.walk/poswalk but even with identity that changes the datastructure

currentoor00:04:48

oh wait no, i forgot to prevent the list from being treated as code

currentoor00:04:22

so, seems like postwalk can work, is that the recommended approach for something like this?

schmidt7301:04:17

what do you want to log them to? plaintext?

currentoor01:04:08

just logging with timbre

schmidt7302:04:18

yeah to remove the :password field in an arbitrarily nested structure postwalk is your best bet

jaihindhreddy03:04:26

postwalk should do well. If you want to preserve the datastructure though (i.e, not everything should be coerced to seqs which print as lists) check out the library specter. Code for things like this also ends up being faster and less ceremonious.

leonoel06:04:56

I want to write a clojure function in java, and that function may throw an exception that is unknown at compile time. in clojure I can throw anything, anywhere :

(fn [error] (throw error))
So clearly the JVM bytecode allows this and it does the expected thing at runtime. But if I try to do that in java, javac gives the finger :
new AFn() {
            public Object invoke(Object error) {
                throw (Throwable) error;
            }
        };
unreported exception java.lang.Throwable; must be caught or declared to be thrown is there a compiler or language trick to make javac happy there ?

Alex Miller (Clojure team)12:04:50

You can find various utilities Clojure uses for this in clojure.lang.Util, like the sneakyThrow() method.

leonoel12:04:15

exactly what I need, thanks !

borkdude11:04:54

With clojure.tools.reader, how do I set the “current ns” when reading auto-namespaced keywords?

(binding [r/*alias-map* {? 'foo}] (read-expressions "(ns foo) #::{:b 1}"))

borkdude11:04:22

or do I set *ns* to 'foo?

borkdude11:04:37

that works:

(binding [*ns* (create-ns 'foo)
            r/*alias-map* {'foo 'foo}] (read-expressions "(ns foo) #::{:b 1}"))
but I’ve no idea if that’s how it should work

bronsa11:04:55

read-expressions can't really work consistently in clojure jvm

bronsa11:04:40

consider "(ns foo (:use bar)) (foo) #::{:b 1}" with (ns bar) (defn foo [] (in-ns 'bar))

bronsa11:04:08

if you accept this limitaion, then yes, bind *ns* manually

bronsa11:04:14

no need for alias-map

bronsa11:04:38

but the model of clojure is read-eval-loop

bronsa11:04:45

not read-loop

borkdude11:04:55

read-expressions is my own function, which should emulate the read-eval loop

borkdude11:04:09

so I’m going to mess with *ns* there

bronsa11:04:28

then if you eval the ns form before reading the map it should set! *ns* automatically

bronsa11:04:42

and there should be no need to bind it

borkdude11:04:02

I’m not going to eval anything, just static code analysis

bronsa11:04:10

ok, that's what I'm saying

bronsa11:04:17

it cannot work in the general case in clojure

bronsa11:04:33

because clojure is not static like clojurescript

bronsa11:04:43

and the read/eval phases are not split

borkdude11:04:51

because there can be non-top-level in-ns expressions you mean?

bronsa11:04:03

same reason why you can't macroexpand a full file w/o step by step evaluation

bronsa11:04:18

this can work in clojurescript, not in clojure

bronsa11:04:29

it can work in clojure for the 90% of the real world code, but by convention

borkdude11:04:42

yeah, I guess I’m going to have to live with that

bronsa11:04:11

right, then yes, (binding [*ns* (create-ns ..)] ..) is the best you can do

bronsa11:04:36

modulo doing some crazy analysis but I don't think you want to go there

borkdude11:04:50

90% is good for now

bronsa11:04:05

I'm just inventing that % btw :)

borkdude11:04:13

I think you’re not far off 😉

bronsa11:04:14

being able to do this stuff is why CL has eval-when and racket has defined phases

bronsa11:04:25

clojure has nothing equivalent

bronsa11:04:52

it's not the easiest language to build tooling/analysis for

bronsa11:04:15

but often it doesn't really matter to be fair

abdullahibra11:04:19

what is the best way to override protected java method ?

abdullahibra11:04:53

i would like to write this piece of java in clojure

mpenet11:04:52

In theory with gen-class I think, in practice writing it in java feels cleaner/faster

mpenet11:04:32

there was also once a discussion about this on jira, but it's gathering dust: https://dev.clojure.org/jira/browse/CLJ-1255

Alex Miller (Clojure team)12:04:56

Well aware of it, hoping to consider for 1.11 when we get there

mloughlin13:04:41

how useful are code reloading libraries like Component, Mount or Integrant? I worry that I'm diving into using a workflow I don't really need.

stathissideris14:04:28

as soon as you have resources that need to be initialized and restarted (database connection pools, HTTP server, job scheduler etc), it starts to make sense to use this pattern. I found Component to have too much bureaucracy and mount to be too “magical”, so these days I feel that integrant is a very good middle way

💯 4
didibus16:04:40

They're not very useful to be honest

didibus16:04:55

Don't use them until you feel a need for it

didibus16:04:25

Especially if.you have this worry, it probably means you don't need them.

greywolve10:04:53

just roll your own version of component, it's very easy, focusing just on getting a minimal system map. i've found it useful in all my projects. it helps things not to get out of hand, and makes testing simpler

greywolve10:04:26

but as with anything, try it out, and see if it helps you

greywolve10:04:58

you probably can't just rely on people's opinions, because like this thread, some people will say don't use them, others will swear by them. 🙂

mloughlin13:04:24

at the moment I'm defining start and stop functions in a top level namespace, and putting stuff like my http server into atoms using defonce

mloughlin13:04:26

it seems to work

stathissideris13:04:17

but I guess you have to start and stop them in a particular order, right?

mloughlin14:04:00

I suppose so. Isn't that built into the order I evaluate the start functions?

mloughlin14:04:27

I can see where there'd be a tipping point in system size, and it's just simpler to define integrant keys

orestis14:04:13

You could start with just an atom holding your database connection pool - then go with one of those libraries when you need a second atom and dependencies between them.

steveh201814:04:10

Just throwing this out there as a GUI alternative. I'm slow, still learning Clojure, time-constrained but I think a productive exercise is to translate the guiserver.lsp file written in newlisp to clojure. The actual GUI part is written in Java and spun up separately from a jar and the guiserver.lsp talks to it over a socket. I think it would be a very nice addition as a clojure librsry. If you play with it first in newlisp, you'll find its plenty responsive and complete enough.

chancerussell16:04:21

Anyone know of something like clojure.core/eduction except where the input is itself a reducible?

chancerussell16:04:09

Another way to say it would maybe be “something that could wrap a given reducible with a transducer xform”

hiredman16:04:41

that is what eduction does

hiredman16:04:08

my mistake, actually eduction doesn't call seq when printed either

ghadi16:04:42

it's the print mechanism that calls seq on it

hiredman16:04:47

oh, yes it does

ghadi16:04:48

I think I logged a bug on this

hiredman16:04:54

yeah, print-sequential

ghadi16:04:06

(defn less-promises-eduction [xform coll]
  (reify clojure.lang.IReduceInit
   (reduce [_ f init]
     ;; NB (completing f) isolates completion of inner rf from outer rf
      (transduce xform (completing f) init coll)))

chancerussell16:04:41

@hiredman Heh, how about that. I assumed from the docstring it needed a coll, and I saw that error when trying it at the repl 🙂

chancerussell16:04:56

Thanks very much

roklenarcic18:04:34

I have a function that returns a lazy sequence, and I want to lazily concat invocations of this function. So I wanted to use concat, but I heard that it has performance issues and that each concatenation deepens the call stack. Is there some other way I can achieve this?

roklenarcic18:04:09

For instance if you had a function (get-page n) and it returns 10 items or nil, and you wanted to return a sequence of items that lazily calls get-page in the background as needed, what would you do?

roklenarcic18:04:08

and each call has a cost so you don't want to chunked-seq 32 calls at a time

hiredman18:04:10

there are kind of 2 things there: 1. paginated api calls 2. concat

hiredman18:04:22

for concat I wouldn't worry

roklenarcic18:04:43

I heard a couple of people complain about that

hiredman18:04:16

the problem with concat occurs when you use it in particular ways, but for example (apply concat (get-pages ...)) is generally safe

roklenarcic18:04:43

And it's not (concat ...seqs...) it's really (concat page 1 (lazy-seq (concat page2 (lazy-seq (concat page-3...

roklenarcic18:04:05

A sequence of concats on 2 args

hiredman18:04:12

yeah, don't do it that way

hiredman18:04:23

build a seq of seqs, then apply concat

roklenarcic18:04:43

but I don't know in advance how many subsequences I need to fetch

hiredman18:04:54

that is fine

roklenarcic18:04:56

I'm not going to load all pages then apply

hiredman18:04:14

apply won't load all the pages

roklenarcic18:04:15

that defeats the purpose altogether

hiredman18:04:28

(apply concat s) forces the first two elements of s (the first two pages) and then the rest will be forced as you walk the seq

hiredman18:04:21

but in general, lazy-seqs are not the best abstraction for io, which is why in my initial proposal on CLJ-1906 linked above I suggested something that just implements IReduceInit

hiredman18:04:42

exactly how and when pages are forced if you use lazy-seqs will also depend a lot on where/if you put the (lazy-seq ...) inside get-page (assuming it has one)

roklenarcic18:04:24

yeah I am aware of that

roklenarcic18:04:38

I think I'll read the discussion in that ticket

hiredman18:04:27

https://ce2144dc-f7c9-4f54-8fb6-7321a4c318db.s3.amazonaws.com/reducers.html is a thing I wrote a while ago about using reduce for this kind of thing, it talks about reducers which are kind of old news these days, clojure has grown even more reduce related features since then (transducers, the interfaces Reduce and ReduceInit)

fenton19:04:18

i seem to recall that there was a way to inspect or print out the intermediate values of the threading macros... like an audit or something... does that ring a bell for anyone?

noisesmith19:04:32

(-> foo (f) (doto println) (g)) is one pattern for a quick hack

noisesmith19:04:26

user=> (-> 42 (+ 10) (doto println) (* 2))
52
104

👍 12
fenton19:04:52

How do I add metadata, this doesn't work:

(-> (map (fn [x] ^{:a 1} x) [1 2 3])
    first
    meta)
;; ==> nil

noisesmith19:04:40

numbers don't accept metadata, you need a type which supports it

noisesmith19:04:23

the metadata is implemented as a field on the object that has it, if the object doesn't define the field there's nowhere for metadata to go

👍 4
noisesmith19:04:39

it might be more straightforward to use (fn [x] {:a 1 :val x}) and then use (map :val) at the last step, rather than putting the metadata on the thing, you mix the thing in with the metadata

g19:04:54

hey everyone. i noticed that my heap usage according to visualvm has been increasing monotonically since i launched my application in the repl. is there anything going on under the hood that could be contributing to this outside of my application, or do i definitely have a leak somewhere?

g19:04:13

about 1mb every two seconds or so

noisesmith19:04:03

the biggest gotcha I can think of is an indefinite lazy-seq where you hold onto a reference to the head

g19:04:37

oh, weird. it just dropped to about half and started increasing again

noisesmith19:04:45

but also it's normal for the jvm gc to let your app usage ramp up indefinitely until some limit is hit - you can test this by specifying a lower max usage

noisesmith19:04:03

OK, that's normal jvm behavior, it's batching some things for perf reasons

g19:04:10

interesting

g19:04:33

so this isn’t indicative of a problem?

noisesmith19:04:45

a normal run of a well-behaved app will look like a sawtooth, where garbage slowly accumulates then all goes down at once

g19:04:52

yeah, that’s exactly what it looks like

g19:04:02

cool. thanks

g19:04:15

so i wonder if i need to keep my maximum steady-state usage to less than the vertical offset of this sawtooth

g19:04:40

or if it’ll clean up before pushing me over the edge

Alex Miller (Clojure team)19:04:53

depends a lot on your app. how much resident memory does the app actually need? how fast does it produce garbage objects? etc. there are a few hundred knobs to turn for GC.

g19:04:05

hmm, ok. are there any resources you could point me to in learning to formulate the right questions to ask here?

Alex Miller (Clojure team)19:04:10

it's a big topic, probably worth doing some googling. I don't have a canonical resource to point you to.

seancorfield20:04:34

@gtzogana Folks who are new to the JVM are often shocked by how much memory it takes up but it's a highly tuned platform and its ability to transparently use and release memory is part of what gives it the power it has... In production, we run processes with between 1GB and 4GB max heap typically -- mostly toward the low end of that -- and those are pretty high traffic processes.

seancorfield20:04:17

(we have multiple, small services in Clojure now -- previously we had a monolithic web app in a different tech on the JVM and it needed 10-15GB of heap to stay high performance)

hipster coder21:04:21

What kind of tech did you have on the JVM to run the monolith? A web framework? I am just wondering which stack takes 15 GB to run.

seancorfield23:04:08

CFML. It would run in a lot less, but the best performance was when we cranked the heap up to 15GB, so the GC had plenty of leeway to do its work (we we using the G1 collector so we could minimize pauses and have reliable throughput).

hipster coder23:04:15

wow, Cold Fusion. I remember that framework. It was just before Rails started gaining traction.

seancorfield00:04:24

These days CFML has both a commercial implementation (Adobe) and an open source implementation (Lucee). It evolved into a JS-like scripting language that compiles-on-demand to JVM bytecode (like Clojure), with a traditional OO model (unlike JS's prototype stuff), and it supports some metaprogramming, closures, map/filter/reduce etc. And it changed from Cold Fusion to ColdFusion some time in the 90's 🙂 I first encountered it when I worked at Macromedia and they bought Allaire (who created it in '95).

seancorfield00:04:39

We migrated piecemeal to Clojure -- I wrote a small CFML library that did some classpath "magic" and used Clojure's Java API to require and call into arbitrary Clojure code:

;; Clojure code
(some.namespace/func arg1 arg2)

// CFML code
clj.some.namespace.func( arg1, arg2 );
🙂

seancorfield00:04:06

Then we rewrote our code from the bottom up, until we had enough built to start spinning off individual Ring/Compojure apps.

seancorfield00:04:42

I even have an adapter, written in CFML, that can call Ring handlers written in Clojure, and bridges cookies and session management across the two, so we can rewrite our remaining app one controller method at a time 🙂

hipster coder01:04:45

I know that companies will hire you on the spot for Cold Fusion… because the talent pool is so tiny.

seancorfield01:04:24

That's how I got hired into my current company -- where I introduced Clojure to replace CFML 🙂

hipster coder01:04:55

is it a good strategy as one grows in their career? Opposed to working for startups just doing full stack dev?

seancorfield01:04:26

I'm server side only and I don't like startup / Silicon Valley culture 🙃

seancorfield01:04:37

Front end stuff is just too annoying for an old dog like me to want to put up with!

hipster coder01:04:12

ohhh believe me too… JS fatigue is hitting me too. I think JS is driving tons of people nuts.

hipster coder02:04:55

6 ways… omg wow

seancorfield02:04:27

I'm so out of touch -- I know only two ways in JS! :rolling_on_the_floor_laughing:

seancorfield03:04:26

Articles like that are why I detest JavaScript 🙃 (but I now know a lot more about JS than I did before I read that!).

hipster coder03:04:46

I spent half the day looking for a Javascript function in the codebase… only to find out that some weird function definition was used… so ya… JS is a pain in that regard. I think that’s why so many people love TypeScript.

hipster coder03:04:23

I was feeling older today too. But a girl cheered me up. She said that us seniors are actually very valuable. The reason we feel like a small group is that the industry is on a 30% growth rate. So you and I started out when the entire workforce was smaller. She pointed out there’s a huge population size skew. My dad always felt like his age was against him. But it’s simply not true. Experience matters a ton in this field. Looking back, I’ve made so many mistakes that only a senior version of me could understand now. For starters… jumping on a technology too soon only to end up using it for six months. Mostly JS tech stuff that keeps changing.

g20:04:30

cool, much appreciated

didibus21:04:44

The default GC for the JVM are optimized for performance, not for memory

didibus21:04:43

Every GC run, that is, having to loop over objects, identify garbage, and clean it up, takes CPU away from your app

didibus21:04:18

And also will hit your CPU caches and all. Making them cold again for your own computations

didibus21:04:54

Thus what it generally does is wait until absolutely necessary before cleaning up

didibus21:04:14

So it'll just let garbage accumulate until you run out of memory

didibus21:04:23

That's a typical server workload style. Where it assumes your process is the main thing your OS is running, and thus it's better to maximise all your memory use and trade it for better performance

g00:04:46

thanks!

hcarvalhoaves22:04:40

It's a bit old, so it's leaving out a new GC coming in Java 11 (https://www.opsian.com/blog/javas-new-zgc-is-very-exciting/)

ahungry23:04:52

Is there any concept of inheritance ala common lisp CLOS defclass extending another class and fulfilling its type checks of the parent (particularly for multi method dispatch)? Like, if I have a defrecord Person, and want multi dispatched on class. And want an Employee to extend person defrecord without redundant list of field names in arg vector, and would like a defmethod foo Person to work on an Employee instance

hiredman23:04:06

what you are describe is a typical oop type/class hierarchy, which to some degree clojure rejects(while still having to interop with that kind of thing on the java side) as a good way to organize data.

Alex Miller (Clojure team)23:04:42

Clojure rejects concrete inheritance