Fork me on GitHub
#clojure
<
2021-10-01
>
plexus08:10:49

I'm noticing this in the pom.xml of some projects (tools.namespace, tools.deps.alpha)

<properties>
  <clojure.warnOnReflection>true</clojure.warnOnReflection>
</properties>
Where does that get picked up exactly? Do these become available as system properties when you have this as a dependency?

plexus08:10:12

And then what does Clojure do with it? I only find this in the clojure source, which is a different property name

private static final String REFLECTION_WARNING_PROP = "clojure.compile.warn-on-reflection";

p-himik08:10:57

Those aren't picked by Clojure or even JVM - Maven <properties> specifies values to be used in the pom.xml itself or by Maven plugins.

plexus08:10:25

makes sense, thanks

👍 3
Alex Miller (Clojure team)11:10:03

it's used by the clojure-maven-plugin

jmckitrick12:10:02

Not sure if there's a better place for a clojure crypto question, so here goes....

jmckitrick12:10:25

I need to handle a message with AES 256 CBC encryption.

jmckitrick12:10:48

The closest I found in buddy is :aes256-cbc-hmac-sha512

jmckitrick12:10:14

Is that a functionally different cipher?

roklenarcic12:10:32

yes that is an AEAD cypher

Dynom12:10:33

I'm not familiar with buddy itself, but it should be different.

Dynom12:10:17

The hmac indicates that it is a verified cipher. It's good to use if you need verification. If you specifically need just AES than you should use block-cipher.

Dynom12:10:32

(I just quickly looked at the buddy documentation)

jmckitrick12:10:14

That's what I started with, block-cipher. Let me try again.

jmckitrick12:10:48

Will the key size choose AES 128/192/256?

Dynom12:10:55

Looking at the docs you should probably make sure that your key is the correct length.

Dynom12:10:56

However, I do think that a different AES cipher is chosen depending on the size of your key as I cannot see a way to tell buddy that you want a specific level.

jmckitrick13:10:12

I'm very close... but my decrypted result is truncated to block size. Does this mean I have to split my cleartext into blocks, encrypt them, then join them at the end?

Dynom14:10:33

If you are able to, use one of the higher level primitives. Such as AES-GCM. That should take care of padding.

roklenarcic12:10:45

Is there a way to make lein test stop on first failure/error?

borkdude12:10:32

@roklenarcic there's a plugin for this, but I'm doing it the basic way like this:

(defmethod clojure.test/report :begin-test-var [m]
  (println "===" (-> m :var meta :name))
  (println))

(defmethod clojure.test/report :end-test-var [_m]
  (when-let [rc *report-counters*]
    (let [{:keys [:fail :error]} @rc]
      (when (and (= "true" (System/getenv "FAIL_FAST"))
                 (or (pos? fail) (pos? error)))
        (println "=== Failing fast")
        (System/exit 1)))))

roklenarcic13:10:34

thanks I will try eftest first

jcromartie14:10:09

I’m having trouble understanding compile-time errors when running with lein run. I looked at the report .edn file but the line numbers are incorrect. I figured out the problem with my code but I’m wondering if there’s something wrong with my setup.

Syntax error (NullPointerException) compiling at (/private/var/folders/2l/l9v846s11fngrc129svcg3300000gn/T/form-init7016381723268623267.clj:1:125).
Cannot invoke "java.lang.Character.charValue()" because "x" is null

Full report at:
/var/folders/2l/l9v846s11fngrc129svcg3300000gn/T/clojure-2081245308366011348.edn

ghadi14:10:39

@jcromartie the stack frames a few beneath the null pointer exception should point to somewhere in your code

jcromartie14:10:58

it’s close but a few lines off

dgb2315:10:59

Does the mismatch happen “inside” a macro?

ghadi15:10:25

dump the whole trace, otherwise we're trickling information back and forth

jcromartie15:10:38

{:clojure.main/message
 "Syntax error (NullPointerException) compiling at (/private/var/folders/2l/l9v846s11fngrc129svcg3300000gn/T/form-init7016381723268623267.clj:1:125).\nCannot invoke \"java.lang.Character.charValue()\" because \"x\" is null\n",
 :clojure.main/triage
 {:clojure.error/phase :compile-syntax-check,
  :clojure.error/line 1,
  :clojure.error/column 125,
  :clojure.error/source "form-init7016381723268623267.clj",
  :clojure.error/path
  "/private/var/folders/2l/l9v846s11fngrc129svcg3300000gn/T/form-init7016381723268623267.clj",
  :clojure.error/class java.lang.NullPointerException,
  :clojure.error/cause
  "Cannot invoke \"java.lang.Character.charValue()\" because \"x\" is null"},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.Compiler$CompilerException,
    :message
    "Syntax error compiling at (/private/var/folders/2l/l9v846s11fngrc129svcg3300000gn/T/form-init7016381723268623267.clj:1:125).",
    :data
    {:clojure.error/phase :compile-syntax-check,
     :clojure.error/line 1,
     :clojure.error/column 125,
     :clojure.error/source
     "/private/var/folders/2l/l9v846s11fngrc129svcg3300000gn/T/form-init7016381723268623267.clj"},
    :at [clojure.lang.Compiler load "Compiler.java" 7648]}
   {:type java.lang.NullPointerException,
    :message
    "Cannot invoke \"java.lang.Character.charValue()\" because \"x\" is null",
    :at [clojure.lang.RT intCast "RT.java" 1220]}],
  :trace
  [[clojure.lang.RT intCast "RT.java" 1220]
   [my_namespace.core$_main invokeStatic "core.clj" 102]
   [my_namespace.core$_main doInvoke "core.clj" 98]
   [clojure.lang.RestFn invoke "RestFn.java" 397]
   [clojure.lang.Var invoke "Var.java" 380]
   [user$eval140 invokeStatic "form-init7016381723268623267.clj" 1]
   [user$eval140 invoke "form-init7016381723268623267.clj" 1]
   [clojure.lang.Compiler eval "Compiler.java" 7177]
   [clojure.lang.Compiler eval "Compiler.java" 7167]
   [clojure.lang.Compiler load "Compiler.java" 7636]
   [clojure.lang.Compiler loadFile "Compiler.java" 7574]
   [clojure.main$load_script invokeStatic "main.clj" 475]
   [clojure.main$init_opt invokeStatic "main.clj" 477]
   [clojure.main$init_opt invoke "main.clj" 477]
   [clojure.main$initialize invokeStatic "main.clj" 508]
   [clojure.main$null_opt invokeStatic "main.clj" 542]
   [clojure.main$null_opt invoke "main.clj" 539]
   [clojure.main$main invokeStatic "main.clj" 664]
   [clojure.main$main doInvoke "main.clj" 616]
   [clojure.lang.RestFn applyTo "RestFn.java" 137]
   [clojure.lang.Var applyTo "Var.java" 705]
   [clojure.main main "main.java" 40]],
  :cause
  "Cannot invoke \"java.lang.Character.charValue()\" because \"x\" is null",
  :phase :compile-syntax-check}}

jcromartie15:10:00

I’m assuming the compiler uses temp files

jcromartie15:10:30

is the [user$eval… part of the stacktrace normal?

jcromartie15:10:10

anyway, not to dwell on it, I unfortunately can’t share the code, and I worked it out

jcromartie15:10:24

I don’t understand why it’s a syntax check error, though, or what that’s about

jcromartie15:10:58

maybe because the namespace is not compiled and “lein run” loads it dynamically?

ghadi15:10:04

looks like this was encountered during a (require) or (load)

jcromartie15:10:35

it’s actually invoking the -main function

jcromartie15:10:57

The error occurs inside an anonymous function

respatialized15:10:10

I have a question about lazy sequences, side effects, and queues that I'm trying to think through. I have a sequence of operations that needs to be performed sequentially because the operation, which produces side effects, needs to block until it's complete. Once each of those operations is complete, I need to add it to another sequence of operations that needs to be performed in a separate thread/in the background because those operations, while non-blocking, also take a bit of time to complete and I want to make sure that the sequence of blocking ops spends as much time as possible doing useful work. (I'm trying to asynchronously separate generation of some files with further processing of those files). I was thinking initially that seque would be the right thing to reach for here, as I could create a sequence that depends on the result of the first sequence that's processed in the background, but I'm not sure if it meshes well with the need to perform side effects. What should I be using to tie these processes together? Does seque work for this use case? core.async channels? promesa? I guess I'm hoping that I won't need to pull in another library, but it's not that big a deal if I have to.

didibus23:10:50

An operation that blocks doesn't mean it needs to be done sequentially? I suspect you're thinking of it wrong

didibus00:10:24

It sounds like you want to do something like: In parallel: Generate file 1 then process file 1 Generate file 2 then process file 2 Generate file 3 then process file 3 ??

respatialized22:10:44

No, file generation can't be parallelized. File processing can be parallelized and done for any number of files once they're generated. The model in my head is of the CEO signing a bunch of documents and handing them off to be stamped and stuffed into envelopes. The boss only can sign one document at a time, but if there's a bunch of signed letters in the stack, multiple people can put them into envelopes concurrently and independently of whatever the boss is currently signing.

didibus23:10:42

Why can't the boss sign more than one thing at a time (assuming this boss is a machine it would be trivial)? The question is, does signing one document require you to know about the previous document? If not, than you can parallelize it

didibus23:10:30

So like, does the generation of the second file require the result of the generation of the first?

didibus23:10:20

Anyway, here's like the simplest version:

(defn generate-file
  [input]
  (println "Generated" input)
  {:file input})

(defn process-file
  [file]
  (locking *out*
    (println "Processed" file))
  file)

(let [generated-files (mapv generate-file [1 2 3 4])
      processed-files-futures (mapv #(future (process-file %)) generated-files)]
  (mapv deref processed-files-futures))

respatialized23:10:54

I don't have control over the file generation process. It's external to my program, happening in a separate program that is single-threaded and only has one entry point. The trigger to generate the file comes from my program, but the output is created by the other program and I have to wait for it. Very much a "constraints of a legacy system" type of problem.

didibus23:10:16

I see, so you get a bunch of processed files synchronously? And once you've received them, you want to process them in parallel?

respatialized23:10:38

Yeah, and independently of the thread that sends out the trigger to generate a file / wait on the result.

respatialized23:10:27

Your solution above wouldn't start processing the results until all of the files have been generated, because mapv is eagerly evaluated, right?

didibus23:10:54

Ya, I mean, its not so much the eagerness, but its a bit harder to reason about what happens in a lazy context. But basically the call to generate-file and mapv are blocking, so they won't return until all of it is done. Once that's done, it fires a bunch of future, one per generated file which will process them in parallel.

didibus23:10:35

And finally the last mapv with deref waits for all parallel processing to be done

respatialized23:10:52

The problem is that processing files is also time consuming (but not as slow as file generation). I'm ok with harder to reason about if I can get faster processing time overall by not waiting to process results; there are hundreds of these operations and sometimes they take around 30 seconds, so even a modest improvement helps because it adds up.

didibus00:10:11

My example code would have them processed in parallel, so it will save you time.

didibus00:10:27

This line: (mapv #(future (process-file %)) generated-files) means each file will be processed in its own thread

dgb2316:10:55

Does anything depend/need to be synced with the second pipeline?

Alys Brooks16:10:10

Could the first sequence of operations individually use future to start the second set of operations at the end in a separate thread?

respatialized17:10:33

I'd prefer not to do that, as there's nothing in theory that prevents individual items from getting "pushed" to the second sequence of operations to be processed concurrently (this is a long running operation with up to thousands of elements, so doing things concurrently where I can is pretty important)

potetm17:10:19

Probably something like:

(a/pipeline-blocking n-init   c1          (map initial-operation) (a/to-chan input-seq))
(a/pipeline-blocking n-second output-chan (map second-operation) c1)

(a/<!! (a/into [] output-chan))

potetm17:10:12

@afoltzm Does that^ make sense?

Alys Brooks17:10:31

Periods are officially https://clojure.org/reference/reader#_literals in keyword names but not namespaces (so :com.example/foo is okay but :example/foo.bar is not). However, we use periods in keyword names in Kaocha because the name actually refers to a namespace (such as :kaocha.type/clojure.test). We haven't noticed any problems; the reader seems to accept them just fine. Are there any plans to make the reader stricter? I'm guessing not because it would be a breaking change, but I thought I'd ask.

potetm17:10:04

@U01FJUDL57C My first instinct is to ask whether you considered just using a symbol? (i.e. 'clojure.test)

potetm17:10:10

I have no insight into the actual question tho 😕 My gut is to say they won’t willy-nilly make a change that will break a significant number of people, but they won’t promise not to.

Alys Brooks17:10:17

Thanks for sharing! Maybe I'll open an issue to get the page on the reader changed.

Alys Brooks17:10:12

@U07S8JGF7 That might have worked originally, but we're trying to avoid making any breaking changes ourselves. (Kaocha users can specify :kaocha.type/clojure.test in config)

potetm18:10:04

yeah figured

kenny17:10:42

What's the best approach to sort something in reverse order without passing a comparator that reverses the args? e.g., the following is true

(= '("b" "a") (sort ["a" "b"]))
=> true

p-himik17:10:29

Pretty sure supplying #(compare %2 %1) is the best approach. What would be a reason not to use it?

kenny17:10:23

I know 🙂 Special case. I am writing a custom keyfn that defines a special sort order. The sort order returns a vector of things for a multi-field sort. All fields follow regular sort order except one -- it must be reversed.

p-himik17:10:25

So it's a different question altogether then. If I understood you correctly, given items like

[{:a 1 :b 1}
 {:a 1 :b 2}
 {:a 2 :b 1}]
you want to e.g. sort by :a and by :b but in reverse. Like ORDER BY a ASC, b DESC in SQL. Is that right?

kenny17:10:03

Though the vals of a and b are strings in my case.

p-himik17:10:05

I think you'll have to write a custom comparator still, but embed the key function into it. The comparator would reduce over the fields, compare the corresponding value in a manner that you want, and return the first non-0 result.

kenny17:10:57

Hmm, that makes sense. No way to negate the string sort "inline" ?

p-himik17:10:13

Have never seen it.

p-himik17:10:58

But it's not that hard to implement it:

(defn sql-sort [order-by coll]
  (sort (fn [x y]
          (loop [order-by order-by]
            (if (seq order-by)
              (let [c (first order-by)
                    [field asc?] (if (keyword? c) [c true] c)
                    a (field x)
                    b (field y)
                    result (if asc? (compare a b) (compare b a))]
                (if (zero? result)
                  (recur (next order-by))
                  result))
              0)))
        coll))

R.A. Porter18:10:38

I'm sad to see there's no analog to Java's thenComparing in Clojure.

R.A. Porter18:10:59

Oh, I'm aware. That's a LOT more verbose than it would be in Java. Which I almost never say about Clojure.

kenny18:10:58

Could you write the Java version?

p-himik18:10:46

It will be more verbose in Clojure because you would have to deal with creating multiple java.util.function.Fuction instances. AFAICT, Java comparators work well with things like Item::getItemField - which require you to deal with all the other boilerplate first.

R.A. Porter18:10:49

Hmm...I might have been wrong...

R.A. Porter18:10:54

(let [input [{:a 1 :b 3}
               {:a 5 :b 2}
               {:a 1 :b 6}
               {:a 1 :b 1}
               {:a 5 :b 4}]
        c1    #(< (:a %1) (:a %2))
        c2    #(> (:b %1) (:b %2))]
    (sort (.thenComparing c1 c2) input))

R.A. Porter18:10:51

Would only work if you were running on Java 8 or above, of course.

p-himik18:10:08

That's neat - clojure.lang.AFunction is a Comparator by itself.

💯 2
p-himik18:10:08

Which also means that #(compare %2 %1) can be replaced with (.reversed compare). Although it makes it not portable with Clojure CLR and ClojureScript.

R.A. Porter18:10:25

Right. I'm always so focused on JVM that I pretty much always forget other platforms. 🙂

john17:10:29

@afoltzm agents are essentially work queues, so you could try looking at using those

1
phronmophobic20:10:00

Not sure if there's a better channel for this. Does anyone know if the flamegraphs from https://github.com/clojure-goes-fast/clj-async-profiler are guaranteed to preserve any sort of order of the calls they represent? It seems like no, but just wanted to double check.

Ben Sless20:10:54

Flame graphs are histograms with the stacks being the bins. They visually group overlapping data. You can set it to display different threads separately which sometimes helps if two pools are doing different kinds of work

👍 1
phronmophobic20:10:04

I'm trying to write my own viz tools so I'm just trying to understand the underlying format.

Ben Sless20:10:32

The underlying format is the perf file. Have you looked at Brendan Gregg's site?

phronmophobic20:10:11

nope. I guess I should check it out

Ben Sless20:10:17

He's the original author of the flame graph script. Lots of explanations on his site

👍 1
phronmophobic20:10:36

fyi, Brendan Gregg's presentation addresses my question directly, https://www.youtube.com/watch?v=D53T1Ejig1Q. The x-axis does not represent time and the call stacks are sorted to facilitate merging

borkdude20:10:17

@smith.adriane No, it just measures what stack frames are present at a given time and then adds that up

thanks2 1
👍 2
borkdude20:10:27

If I understand correctly