Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2021-11-08

Channels

#clojure
<
2021-11-08
>
seancorfield04:11:59

You really shouldn't cross-post unless you aren't getting an answer after a decent amount of time. Two minutes isn't very patient 🙂

zackteo04:11:24

Oops, my bad. Was kinda unsure if I should be posting here or integrant

seancorfield04:11:20

It isn't a big deal. I suspect you won't get an answer until Monday "morning" US time in #integrant...

zackteo04:11:38

And I guess I was also thinking how I asked a slightly related question in #clojure earlier and this was my attempt at making it more specific (/another angle where I might get responses)

zackteo04:11:59

But yes, I will try to remember that! Actually, are there a list of rules/guidelines for clojurian slack? o:

seancorfield04:11:45

There's a Code of Conduct, but it doesn't cover stuff like this. So we manually encourage folks not to x-post from time to time, and to be patient 🙂

zackteo04:11:28

Okay 🙂 thank you!

qqq07:11:43

What is the correct way to parse / validate untrusted data in Clojure? In the past, I thought "user data => parse as edn => spec"; but now I am concerned about algorithmic denial of service attacks.

Linus Ericsson08:11:48

I would use malli because it allows closed specs. And be careful what I put in those specs of course.

delaguardo12:11:53

Do you know how other languages/platforms handle such problem?

qqq18:11:45

@U04V4KLKC: In languages with static types and serialization / deserialization generators, one can often get "size of struct + O(1)" validators; (though might still grow the stack too much, and not nearly as flexible as spec).

qqq08:11:24

"Algorithmic" here as in: the attacker constructs input designed to maximize RAM / CPU usage.

hanDerPeder12:11:33

given this: 

(set! *warn-on-reflection* true)

(def http-client-1 ;; reflection warning: reference to field build can't be resolved.
  (.build (doto (HttpClient/newBuilder)
            (.version HttpClient$Version/HTTP_1_1))))

(def http-client-2 ;; no reflection warning
  (let [builder (doto (HttpClient/newBuilder)
                  (.version HttpClient$Version/HTTP_1_1))]
    (.build builder)))
http-client-1 generates a reflection warning, but http-client-2 does not. in my mind these are equivalent, but seems not. anybody know why the latter does not generate a warning but the former does?

meatballs21:11:52

I don't have an answer, and these are just observations rather than a suggestions, but I found it curious that if your http-client-1 innards are bound to a local first, it also doesn't reflect. It seems like something unusual in the behavior of def, but that is speculative. 

(def http-client-1* ;; no reflection warning
  (let [client (.build (doto (HttpClient/newBuilder)
                              (.version HttpClient$Version/HTTP_1_1)))]
    client))

(let [client (.build (doto (HttpClient/newBuilder)
                       (.version HttpClient$Version/HTTP_1_1)))]
  (.sslParameters client)) ;; no reflection warning

dharrigan13:11:11

@(promise) will suffice here I think (if you're just looking for a blocking call) - but on second reading, I think you want an operation within the loop)

FiVo17:11:46

Is there a way to explicitly destroy an object in Clojure? I have a repl session where I created some huge DS in memory. In the end only the GC was running. I niled the var with all the references to the data, thinking that would take care of it. It didn't. Does this mean I still have some lingering references to the data? Is there a way to resolve an issue like this without killing the repl session?

Alex Miller (Clojure team)17:11:29

the jvm does not have a way to explicitly destroy an object

Alex Miller (Clojure team)17:11:58

hard to say what exactly was happening without more info

FiVo18:11:39

This is what it looked like (visualvm heap size and clj-async-profiler during the end of the graph)

Ben Sless18:11:15

can you pop open the memory inspector, too?

FiVo18:11:30

I killed the session now.

FiVo18:11:21

can probably reproduce it

Ben Sless18:11:09

what kind of application is this? a web server? something else?

FiVo18:11:55

a crawler

Ben Sless18:11:31

hm. If you're doing a traversal there's certainly a chance you're holding on to stuff you shouldn't. Are you able to share the code? If not, recreating and examining the heap is also good

FiVo18:11:19

I actually know where the issue is. I was more interested in why I couldn't save the the session even after I stopped the crawler.

FiVo18:11:28

As Joshua mentioned outside of the thread it might have been *1 of which I did not think of at that moment

Ben Sless18:11:19

You could try to force gc with (System/gc)

Ben Sless18:11:45

that would at least tell you if you were still holding references

Alex Miller (Clojure team)17:11:21

could have been a lingering object reference somewhere - there are ways to debug that with either memory dumps or live profiling tools that do "path to root" analysis. or could have been something like a gc that had not yet found the dead object or was collecting but taking a long time

Joshua Suskalo18:11:19

on the repl if you def ed something you can use ns-unmap to remove it from your current ns. If you're on the repl then the last three return results and the last exception are kept around in the *1, *2, *3 , and *e vars. Once you've made sure your huge value is in none of these, you can call (.gc (Runtime/getRuntime)) and this will hint the garbage collector to run, which may collect your value.

FiVo18:11:19

True, I did not think about *1...

Joshua Suskalo18:11:40

If after all of that the object is still around, you probably have retained a reference to it some other way.

Joshua Suskalo18:11:01

But that said, there's still no guarantee that it will be removed.