Fork me on GitHub
#clojure
<
2022-12-17
>
Drew Verlee02:12:32

How would i read a string into clojure where part of it is valid clojure and another part isnt. E.g "{:a [///fobar]"} Like i would like to at least end up with {:a ["////foobar"]} . The actual goal is to read a string which has seralized stack trace in it. e.g {:msg "UncaughException" ex "#error" {:cause "blah" via ... it complains that some parts of it can't be handled by read-string or edn/read-string telling me there is do dispatch macro No dispatch macro for: ' Which makes sense because of this part of the string: calling #'com.stuartsierra.component/start I was sort of hoping there would be a way to make my stack traces readable in clojure so i could pull them apart using clojure. Any ideas in this direction would be interesting to me. I usually don't worry about stack traces because i so rarely make mistakes /sarcasm.

Drew Verlee04:12:33

i can probably pass edn/read-string readers to handle it...

p-himik08:12:43

//// is not valid Clojure, but #' is. But, it's not valid EDN. But you say that clojure.core/read-string didn't work for you as well. What was the error? It works just fine for me:

user=> (read-string "#'a")
(var a)
user=> (require '[clojure.edn :as edn])
nil
user=> (edn/read-string "#'a")
Execution error at user/eval7 (REPL:1).
No dispatch macro for: '
user=> 

Drew Verlee02:12:01

(-> "errors/error.edn"
    slurp
    read-string)

;; 1. Unhandled java.lang.RuntimeException
;;    Unsupported character: \"UncaughtException

p-himik07:12:00

Can you share that EDN file?

Valentin Mouret14:12:16

Hello 🙂 I am trying to build an uber JAR using build, but I am getting the rather cryptic:

Building uberjar target/eve-standalone.jar...
Execution error (IllegalArgumentException) at clojure.tools.build.tasks.uber/explode (uber.clj:152).
/ is not a relative path
Does someone have a clue of what’s happening?

Valentin Mouret14:12:19

Can it somehow be because my src directory contains .clj{,s,c} files?

Alex Miller (Clojure team)14:12:46

No, that shouldn’t matter. Do you have any :local deps?

Alex Miller (Clojure team)14:12:17

If you can share your deps.edn that would help

Valentin Mouret14:12:00

I don’t have :local deps. Here are my industrial secrets:

{:paths ["src"  "resources"]
 :deps  {org.clojure/clojure               {:mvn/version "1.11.1"}
         org.clojure/clojurescript         {:mvn/version "1.11.51"}
         org.clojure/data.csv              {:mvn/version "1.0.1"}
         ;; Malli is a data-driven schema library.
         ;; 
         metosin/malli                     {:mvn/version "0.9.2"}
         ;; Third-party library for dealing with SQL and Postgre.
         ;; Not sure we need it since we have `next.jdbc`.
         ;; 
         clj-postgresql/clj-postgresql     {:mvn/version "0.7.0"}
         org.postgresql/postgresql         {:mvn/version "42.3.7"}
         ;; Third-party for dealing with JDBC.
         ;; 
         com.github.seancorfield/next.jdbc {:mvn/version "1.3.834"}
         ;; Deals with JSONs.
         ;; 
         cheshire/cheshire                 {:mvn/version "5.11.0"}
         ;; HTTP server.
         ;; 
         ring/ring-core                    {:mvn/version "1.9.6"}
         ring/ring-jetty-adapter           {:mvn/version "1.9.6"}
         ;; Routing library that integrates with ring.
         metosin/reitit                    {:mvn/version "0.5.18"}
         nrepl/nrepl                       {:mvn/version "1.0.0"}
         ;; Frontend
         thheller/shadow-cljs              {:mvn/version "2.20.14"}
         reagent/reagent                   {:mvn/version "1.1.1"}
         re-frame/re-frame                 {:mvn/version "1.3.0"}
         clj-commons/pushy                 {:mvn/version "0.3.10"}
         cljs-ajax/cljs-ajax               {:mvn/version "0.8.4"}
         day8.re-frame/http-fx             {:mvn/version "0.2.4"}
         ;; Dev dependencies
         ;;
         ;; Dev dependencies to have tracing inside the application.
         ;; 
         day8.re-frame/tracing             {:mvn/version "0.6.2"}
         day8.re-frame/re-frame-10x        {:mvn/version "1.5.0"}
         binaryage/devtools                {:mvn/version "1.0.6"}
         ;; Not sure
         com.cognitect/transit-cljs        {:mvn/version "0.8.280"}
         com.lambdaisland/dom-types        {:mvn/version "0.5.37"}
         ;; Since 2.20.11 guava is not explicitly required by
         ;; shadowcljs anymore but apparently it's still needed
         ;; because it doesn't start without this pinned version.
         com.google.guava/guava            {:mvn/version "31.0.1-jre"}}
 :aliases
 {;; Builds the backend as an uberjar. Unused for now.
  :build {:deps       {io.github.seancorfield/build-clj
                       {:git/tag "v0.8.5" :git/sha "de693d0"
                        ;; since we're building an app uberjar, we do not
                        ;; need deps-deploy for  deployment:
                        :deps/root "slim"}}
          :ns-default build}

  ;; Runs the tests. No tests for now. #yolo
  :test {:extra-paths ["test"]
         :main-opts   ["-m" "cognitect.test-runner"]
         :exec-fn     cognitect.test-runner.api/test
         :extra-deps  {org.clojure/test.check               {:mvn/version "1.1.1"}
                       io.github.cognitect-labs/test-runner {:git/tag "v0.5.0" :git/sha "48c3c67"}}}}}

Valentin Mouret14:12:00

Maybe some deps are cljs only and need to be moved in an alias?

Alex Miller (Clojure team)14:12:43

Nah, that doesn’t matter

Valentin Mouret14:12:50

Yeah, turns out if I comment out the frontend deps it builds fine.

Alex Miller (Clojure team)14:12:28

It looks to me like that’s probably a jar that has a / path in it, which is probably legal but weird

Alex Miller (Clojure team)14:12:11

I mean if you can exclude the dep that’s weird, then do so, might be good to figure out which one it is

Valentin Mouret14:12:54

I’ll start the hunt

Alex Miller (Clojure team)14:12:01

You could clone tools.build locally, add a debug print statement to the uber task, then include the local dep for tools.build on your build alias

Valentin Mouret14:12:59

I’ll comment them out one by one until it breaks 😄

Valentin Mouret14:12:44

Looks like it’s shadow-cljs

Valentin Mouret14:12:28

Thanks for your time and feedback. :man-bowing:

Alex Miller (Clojure team)15:12:49

Huh, well would be good to report that

thheller15:12:42

FWIW it is not shadow-cljs, it is the closure compiler. they switched their entire jar creation to bazel and ever since it has been weird. either basically releasing uberjars or apparently including a / entry.

Clojure 1.11.1
(require ')
nil
( "/")
#object[java.net.URL 0x5c09d180 "jar:file:/home/thheller/.m2/repository/com/google/javascript/closure-compiler-unshaded/v20221102/closure-compiler-unshaded-v20221102.jar!/"]

thheller15:12:57

but for what its worth. none of your CLJS dependencies (including shadow-cljs) should be part of an uberjar build. these are development dependencies that just create unnecessary bloat (and conflicts) in uberjars

Alex Miller (Clojure team)17:12:07

ah, I was wondering if it was something transitive. maybe I'll exclude this specific / in uber

Valentin Mouret17:12:58

Yes, that’s what I figure. This is a project I just created and I am learning Clojure(Script). It’s not always easy to navigate all the different pieces and how they fit together, for the language(s) and the tooling most of all.

orestis15:12:49

So I'm reading through this: https://death.andgravity.com/pwned -- and out of curiosity, I'd like to see if the results can be replicated in Clojure/JVM. I've started with the naive linear approach: • First using a RandomAccessFile, but that seems to be quite slow, as there's no buffering going on • Then using a FileReader (setting file encoding to ascii), wrapped in a BufferedReader. The times I get for around 8000 iterations are: • Python: ~2ms • Clojure RAF: ~380ms • Clojure Buffered/FileReader: ~10ms Are there any other IO optimisations I can look into to bring the Java implementation closer to python?

Drew Verlee02:12:06

im confused about the goal before i try to read the whole thing. > Instead, we'll do it the hard way – we'll check passwords offline. That seems fairly useless, one of the key features in any good security system is that it throttles attempts.

ghadi03:12:41

I'd like to see your code for RAF or FileReader

orestis08:12:48

@U050ECB92 here's the whole thing. I haven't profiled yet but from online reading it seems that RAF does no buffering whatsoever.

(ns powned
  (:require [ :as io]
            [clojure.string :as str]))

(set! *warn-on-reflection* true)

#_
(def random-file (java.io.RandomAccessFile. "resources/passwords-sample.txt" "r"))

(def random-file ^java.io.RandomAccessFile (java.io.RandomAccessFile. "resources/pwned-passwords-sha1-ordered-by-hash-v8.txt" "r"))

(defn get-hash [type data]
  (.digest (java.security.MessageDigest/getInstance type) (.getBytes data)))

(def hex (-> (java.util.HexFormat/of) (.withUpperCase)))

(defn sha1-hash [^String data]
  (let [digest
        (get-hash "sha1" data)]
    (.formatHex hex digest)))

(defn find-line-linear [^String prefix]
  (with-open [reader (io/reader (java.io.FileReader.  "resources/pwned-passwords-sha1-ordered-by-hash-v8.txt" (java.nio.charset.Charset/forName "US-ASCII")))]
    (loop [line (.readLine reader)
           counter 0]
      (tap> ["line2" line])
      (if false #_(> counter 10000)
          (println "giving up")
          (when (some? line)
            (cond
              (str/starts-with? line prefix)
              (do
                (println "found in "  counter)
                line)
              (pos? (compare prefix line))
              (recur (.readLine reader)
                     (inc counter))))))))

(defn find-line-linear-random [^String prefix]
  (.seek random-file 0)
  (loop [line (.readLine random-file)
         counter 0]
    (tap> ["line" line])
    (if false #_ (> counter 10000)
      (println "giving up")
      (when (some? line)

        (cond 
          (str/starts-with? line prefix)
          (do
              (println "found in "  counter)
            line)
          (pos? (compare prefix line))
          (recur (.readLine random-file)
                 (inc counter)))))))


(defn find-password [password]
  (let [sha1 (sha1-hash password)
        _ (println "looking for sha" sha1)
        found (find-line-linear  sha1)]
    (if found
      (-> found
          (str/split #":")
          (second)
          (parse-long))
      0)))


(comment
    (add-tap println)
    (remove-tap println)
    (time
        (sha1-hash "blocking")
     )
    (time
     (find-line-linear (sha1-hash "blocking")))
    (time
     (find-line-linear-random (sha1-hash "blocking")))
    (find-line-linear  "00000DBE43074309BC753F183A067ED42BC0DEA0")
    (time
     (find-password "blocking"))
  )

orestis08:12:04

@U0DJ4T5U1 this is about finding a password in a list of compromised passwords, nothing to do with security or throttling etc. In more detail, it's just an excuse to explore the boundaries of I/O performance, for academic curiosity on my part.

orestis08:12:31

The Random File bit is to support a later attempt of doing a binary search by jumping around in the file. I can see obvious optimisations like applying a custom buffering layer on top of the native RAF, and trying to move away from strings entirely, but I'm looking to stay with whatever the JVM provides natively without heaps of custom code just yet.

Takis_17:12:22

what is best to use for a clojure mixed java spring-boot project? leiningen,maven or gradle? does anyone has experience or a sample project in like github?

didibus01:12:26

Gradle with Clojurephant works well for that. https://github.com/clojurephant/clojurephant

Takis_02:12:24

thank you, i tried it in past and i had some problems i dont remember exactly, if you have a sample project example it would help alot, thank you anyways i will retry it

emccue03:12:14

> clojure mixed java spring-boot project? Can you explain the reasoning here?

Takis_19:12:35

java is so popular and spring has became functional and reactive with spring 5, webflux etc, so i am thinking to have do functional java and clojure when possible

Takis_19:12:48

main reason is the job, i dont want to be worry if i will get clojure job

emccue19:12:25

Yeah...so most of what spring provides is a DI framework

emccue19:12:36

That is what spring is, really

Takis_19:12:05

you think is bad idea? i was thinking to do functional/reactive java, and where possible to use clojure

Takis_19:12:37

after all knowing the JVM is important for clojure

emccue19:12:50

Yeah, I think it's a bad idea

emccue19:12:15

It's the kind of "synthesis" that you can only really think about after you understand both kinds of systems

emccue19:12:37

And it's not a super practical one

emccue19:12:58

Your goals are to learn stuff and be prepared for a job/interview right?

Takis_19:12:23

yes in my country there are like 0-2 clojure jobs

emccue19:12:32

And you want to learn/use clojure for your fun thing while learning spring?

Takis_19:12:52

if i could get clojure job, i wouldnt read java things

Takis_19:12:16

but functional/reactive java is ok to me, but still clojure way nicer

emccue19:12:32

Are you actively trying to get a job right now or just learning with "buffer"

Takis_19:12:43

i am learning

Takis_19:12:57

and its on schedule after some time

emccue19:12:08

Then, do both

emccue19:12:20

1 clojure project, 1 spring project

emccue19:12:39

Once you figure out how to do something in clojure, do it in the spring project

Takis_19:12:12

yes thats what i am thinking to go java/clojure java for safety, and clojure where possible, and if possible only clojure

emccue19:12:29

I mean two entirely separate projects

emccue19:12:55

Clojure calling into Java is fine, but Java calling into clojure is a relatively hard thing

emccue19:12:18

And if you want to be learning spring, that would be Java calling into clojure, and that's not worth your time

Takis_19:12:50

i did that wasnt so hard, but i dont see people using spring and clojure

emccue19:12:11

I do understand the appeal of polyglot projects, believe me, I've tried and succeeded occasionally, but it's water and oil

Takis_19:12:26

its ok i will see what i can do, thank you from your experience, worst case learn the jvm using java, and sometime move completly to clojure

didibus02:12:45

I will disagree with @U3JH98J4R here. Lots of people, including me, use Clojure with a Java web or API framework. It's not as nice as going full Clojure, but it's a great way to do Clojure and remain working inside a Java shop. Spring Boot doesn't just do DI, it'll handle the server, the routes, the authentication, the wire protocol, any interceptor chain processing you need on the request and response, etc. And it also does DI in a way that also manages initializing and maintaining the state, setting up your logger, your metrics publisher, and injecting all that on a per request basis. Again, I'd rather be using Ring with http-kit and some Clojure router, but it's a very viable route to piggy back on a Java framework, and it's much nicer than using Java still.

didibus02:12:45

At my work, we use an in-house fork of Spring boot and Spring MVC, that's used company wide, it means there's a lot of internal tooling based around it and standard interceptors for it, custom internal auth, tracing, client generators, etc. Most teams use Java with it, some use Scala, my team uses Clojure.

didibus02:12:56

This is a good read as well: https://github.com/stuarthalloway/clojure-from-java Though it uses maven, which is also a viable option.

didibus02:12:52

If you go that route I do recommend a Java build tool, maven or Gradle or even Ant, over a Clojure one, because building Clojure is relatively simple, but Java not so much, so you benefit more from the build tool being specialized for Java.

Mark Wardle10:12:49

I am mixing Java and Clojure as I move a legacy Java application to Clojure. I don’t try to have a single build however but instead consume my Clojure code as libraries. For some, I have a dedicated Java interface so in Clojure I simply implement that interface. In other cases, I build a Java glue layer that calls invoke and wraps up types so that Clojure code is easier to call. For some libraries, I only build and install locally rather than publishing to maven central or elsewhere. But I keep build tools separate using maven for Java.

Takis_02:12:40

i tried maven and clojure based on the github project above in past it worked fine, i tried also the clojure way to make a java class to be call from java like normal java functions, worked very good also, keeping clojure code in leinigein

Takis_02:12:25

i am thinking to spring reactive functional programming with webflux, core reactore, and i made 2 clojure libraries for mongodb and one on top of jooq

Takis_02:12:09

hoping to mix them somehow, than you for your information, i will try and see