Fork me on GitHub
#clojure
<
2023-04-07
>
Noah Bogart01:04:12

I've run into a bit of a conundrum with clojure.core/compile: if I call compile from the repl, it fails because it can't find the class file. but if I write a function in build.clj, calling b/compile-clj does work. any ideas why that might be?

2
Noah Bogart01:04:16

$ clojure -M:dev -e "(require 'noahtheduke.splint) (compile 'noahtheduke.splint)"
Syntax error (ClassNotFoundException) compiling at (noahtheduke/splint.clj:5:1).
noahtheduke.splint

Full report at:
/var/folders/5w/z50rbwg546x94k8p8jzqfbl00000gn/T/clojure-15092840249394826547.edn

Noah Bogart01:04:33

$ clojure -T:build compile-
Compiling io.github.noahtheduke/splint

Noah Bogart01:04:44

with implementation:

(defn compile- [opts]
  (b/delete {:path class-dir})
  (let [basis (b/create-basis {:aliases [:dev]})]
    (b/compile-clj {:basis basis
                    :class-dir class-dir}))
  opts)

Noah Bogart01:04:48

I just double checked and the clojure -M call works on my linux machine but it doesn't work on my mac.

hiredman01:04:23

You should checkout the docstring for compile

hiredman01:04:04

Anything about directories required to exist and be on the class path jump out at you?

Noah Bogart01:04:00

$ ls classes/
noah@Noahs-MacBook-Pro ~/personal/splint  [main ↑2 +0 ~3 -0 !]
$ ls
bb.edn  build.clj  CHANGELOG.md  classes  deps.edn  dev  docs  justfile  LICENSE  pom.xml  README.md  resources  src  target  tasks  test  tests.edn
noah@Noahs-MacBook-Pro ~/personal/splint  [main ↑2 +0 ~3 -0 !]
$ head deps.edn
{:paths ["src" "resources" "classes"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        org.clojure/tools.cli {:mvn/version "1.0.214"}
        borkdude/edamame {:mvn/version "1.3.20"}}
 :aliases
 {:run {:main-opts ["-m" "noahtheduke.splint"]}
  :dev {:extra-paths ["dev"]
        :extra-deps {org.clojure/tools.namespace {:mvn/version "1.3.0"}
                     criterium/criterium {:mvn/version "0.4.6"}
                     com.clojure-goes-fast/clj-async-profiler {:mvn/version "1.0.3"}
noah@Noahs-MacBook-Pro ~/personal/splint  [main ↑2 +0 ~3 -0 !]
$ clojure -M:dev -e "(require 'noahtheduke.splint) (compile 'noahtheduke.splint)"
Syntax error (ClassNotFoundException) compiling at (noahtheduke/splint.clj:5:1).
noahtheduke.splint

Full report at:
/var/folders/5w/z50rbwg546x94k8p8jzqfbl00000gn/T/clojure-9125859686944406454.edn

Noah Bogart01:04:22

the directory exists and is on the class path

hiredman01:04:50

What's on line 5 of splint.clj?

Noah Bogart01:04:22

the namespace declaration. i have a copyright header above it

Noah Bogart01:04:32

(ns noahtheduke.splint
  "Entry-point for splint. Loads all of the lints up front, delegates all work
  to library functions."
  (:gen-class)
  (:require

hiredman01:04:58

And does it still work on your Linux machine if you empty out the classes directory there?

hiredman01:04:54

Hard to tell with how things wrapped, but are you sure that is line 5?

Noah Bogart02:04:10

okay, i just figured it out. calling require first is what tripped it up. i'm not sure why

Noah Bogart02:04:25

$ clojure -M:dev -e "(compile 'noahtheduke.splint)"
noahtheduke.splint
noah@Noahs-MacBook-Pro ~/personal/splint  [main ↑2 +0 ~3 -0 !]
$ ls classes/
clojure  edamame  noahtheduke

hiredman02:04:26

compiling forces loading the code, and there are many ways that double loading code can trip things up

Noah Bogart02:04:58

oh interesting, that makes sense. thanks for the help

Kimo11:04:39

Designing a "theme" system in my framework, where a user can extend the render function for a set of dispatch types. In this example, a "theme" is a namespace. The user prefers the :u theme. If a :u/TYPE method isn't defined, it falls back to :me/TYPE and then :default/TYPE. Does this sound reasonable? I like multimethods, but maybe this reverse-engineers them too much. Is there a simpler construct I should consider? Should I try something like https://github.com/camsaul/methodical?

p-himik11:04:14

It feels like exactly something that multimethods are designed for.

🎉 2
Kimo14:04:44

Thanks @U0WL6FA77, see the snippet for how I've used multimethods so far.

jpmonettas15:04:47

why do I get a reflection warning when accessing a mutable field on a deftype? Is there a way to avoid the reflection?

user=> (set! *warn-on-reflection* true)
true
user=> (deftype T [^:unsynchronized-mutable a b])
user.T
user=> (defn get-a [^T t] (.-a t))
Reflection warning, NO_SOURCE_PATH:1:20 - reference to field a on user.T can't be resolved.
#'user/get-a
user=> (defn get-b [^T t] (.-b t))
#'user/get-b 

hiredman15:04:25

Mutable fields are made private

hiredman15:04:39

Not only do you get a reflection warning, but if you tried to call those functions you'd get an exception about t not having the field

hiredman15:04:53

Inline protocol extension and interface implementation can access the private field

jpmonettas15:04:27

what is the fastest way of accessing those mutable fields then?

hiredman15:04:42

There is no fast external access to private fields

jpmonettas15:04:11

so accessing a mutable field in a deftype will always go through an invokeinterface, am I right?

hiredman15:04:09

Depends what you mean by accessing

jpmonettas15:04:15

given t of type T reading/setting the value of t.a being a a mutable field

hiredman15:04:37

if you have your deftype implement an interface and those methods use the mutable field, the methods access as a field on the object using get and put field instructions

hiredman15:04:19

Same for an inline protocol extension

jpmonettas15:04:27

but I mean, if you want to create a getter/setter, you will need to go through the interface instead of the obj method which is slower, or am I missing something?

hiredman15:04:55

It's just like a private field in java

jpmonettas15:04:30

(definterface TI
  (getA []))

(deftype T [^:unsynchronized-mutable a b]
  TI
  (getA [this] a))

(defn getB [^T o]
  (.-b o))

(def t (->T 5 7))

(.getA t) ;; Execution time mean : 567.532462 ns
(getB t) ;; Execution time mean : 3.407326 ns
That is what I meant, that accessing mutable fields is much slower than non mutable ones on Clojure deftype while they should be the same on Java

hiredman15:04:31

It is a private mutable field, not just a mutable field

jpmonettas15:04:29

oh, nvm, I measured it wrong, I should have said (.getA ^T t) the slowness was coming from the reflector

agorgl16:04:54

Quick poll, how would you prefer to organize your sources in (small/medium sized) fullstack projects? With the source-sets on project root like:

project-root
  dev
  src
    clj
      ecorp
        fullstack
          core.clj
    cljs
      ecorp
        fullstack
          core.cljs
  test
    clj
      ecorp
        fullstack
          core_test.clj
    cljs
      ecorp
        fullstack
          core_test.cljs
Or with source-sets under src like:
project-root
  src
    dev
    main
      clj
        ecorp
          fullstack
            core.clj
      cljs
        ecorp
          fullstack
            core.cljs
    test
      clj
        ecorp
          fullstack
            core_test.clj
      cljs
        ecorp
          fullstack
            core_test.cljs
Any other recommendations?

p-himik16:04:50

The top one without separation by file type.

agorgl16:04:34

You mix clj and cljs sources on same folders?

p-himik16:04:57

Yes, absolutely. And cljc.

💯 2
seancorfield16:04:08

The separation arose mostly due to limitations in tooling way back when, as I recall, but is not needed now.

agorgl16:04:38

Hmm I might switch to common folders too then

seancorfield16:04:56

If you feel the need to have a single (mono)repo that builds multiple, separate artifacts for deployment, you could consider #C013B7MQHJQ but I don't think that's what you're really asking here?

agorgl16:04:47

No, I'm mostly searching to settle on a nice fullstack (shadow-cljs + some backend) project structure

agorgl16:04:00

For simple / medium sized projects mostly

seancorfield16:04:55

Then just src and test with namespaces/folders that are well-named for concerns within the application/domain (and mixed .clj/`.cljc`/`.cljs` files in them).

p-himik16:04:13

There was a similar question around a year ago or so. IIRC, in the end people who were keen on keeping file types separated by folders were doing that because it felt nice. To each their own. And since you've mentioned shadow-cljs, thheller has wrote an article about the fact that the separation is not useful. :)

hiredman16:04:29

for my very small clojure and clojurescript projects, I keep everything in one file

p-himik16:04:15

@U03PYN9FG77 The first one, but the second one is also worth reading.

hifumi12317:04:18

For full-stack i normally separate the backend and frontend repositories into subprojects part of a monolithic project, so it is less likely i accidentally fuse the two codebases together. In the past I’ve used lein-monolith so I can control several projects under one root project. I would add subprojects as needed (e.g. specs shared between backend and frontend). I also avoid separating source code by file extension. For some reason all of the lein templates (e.g. luminus) do it, but I never use such templates anyway. Always build from scratch 😄

isak17:04:41

> For full-stack i normally separate the backend and frontend repositories so it is less likely i accidentally fuse the two codebases together. That would also prevent you from benefiting from the possibility of sharing some code between the frontend and backend, or making a change that affects both the frontend and the backend in one commit. Seems like a bad idea to me. Better to just get your source paths under control (at least one path for each app, and then one that is shared).

2
seancorfield17:04:11

At work, we have separate frontend / backend repos -- but the frontend is a massive React.js / Redux app and the backend is a massive Clojure monorepo building 20 separate services and processes, so there's a lot of shared code between backend processes but nothing shared with the frontend (although there are times when that would be convenient...).

hifumi12317:04:23

> That would also prevent you from benefiting from the possibility of sharing some code between the frontend and backend Extract common things into subprojects. > making a change that affects both the frontend and the backend in one commit. How? The monolithic project is what has the .git folder, not the individual subprojects.

hifumi12317:04:57

Neither of these issues brought up happened in a fairly large codebase I was working on with some dozen other people..

seancorfield17:04:29

Ah, so you have a monorepo with sub-"projects" for front, back, and shared stuff?

seancorfield17:04:15

You said "normally separate the backend and frontend repositories" so folks assumed you meant you had multiple git repos.

hifumi12317:04:57

It’ll be laid out as such. The root project.clj contains managed dependencies and common settings used across projects.

root-folder/
  subproject-1/
   ...
    project.clj
  ...
  subproject-n/
    ...
    project.clj
  project.clj
You’re right, I should edit my message to say “separate subprojects” instead

2
Drew Verlee17:04:40

Do any users of http://clojure-goes-fast.com/kb/profiling/clj-async-profiler/allocation-profiling/ know what I should do to resolve this AgentInitializationException? (it's below in the code block). It only seems to work when i want to pass the {:event :alloc} option. I ran sudo sysctl -w kernel.perf_event_paranoid=1\nsudo sysctl -w kernel.kptr_restrict=0 and i passed the required jvm param for allowing attach self (which i tried to verify below).

(defn foobar [x] (range x))
;; => #'centriq-web.jobs.import.fix.import-all-staging-march-19/foobar

(require '[clj-async-profiler.core :as prof])
;; => nil

(prof/profile {:event :alloc} (foobar 100))
;; => Execution error (AgentInitializationException) at sun.tools.attach.HotSpotVirtualMachine/loadAgentLibrary (HotSpotVirtualMachine.java:106).
;;    Agent_OnAttach failed

(defn jvm-options []
  (let [runtime-bean (ManagementFactory/getRuntimeMXBean)
        input-args (.getInputArguments runtime-bean)]
    input-args))
;; => #'centriq-web.jobs.import.fix.import-all-staging-march-19/jvm-options

(jvm-options)
;; => #<java.util.Collections$UnmodifiableRandomAccessList@773279f [-XX:-OmitStackTraceInFastThrow, -Djdk.attach.allowAttachSelf, -Xmx20480m, -Dclojure.basis=.cpcache/3428257497.basis]>

;; using the default of :event :CPU seems to work...
(prof/profile  (foobar 1))
;; => (0)

Drew Verlee17:04:41

maybe my version java matters?

➜  backend git:(etl-with-adaptions) ✗ java --version
openjdk 17.0.6 2023-01-17
OpenJDK Runtime Environment (build 17.0.6+10-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 17.0.6+10-Ubuntu-0ubuntu122.04, mixed mode, sharing)

Ben Sless18:04:35

I remember some difficulties with alloc profiling. Do you have debug symbols installed? Do you have patience to try jvm 8 and 11?

Drew Verlee18:04:17

@UK0810AQ2 ty for the reply. I didn't explicitly install any debug symbols. Should i have? I can probably try jvm 11... This code is currently living in a mono repo so I'm worried something might complain though.

Ben Sless19:04:38

iirc I only got alloc profiling working on jvm 11 and 8, but don't quote me on that not sure debug symbols are related

Ben Sless19:04:11

on .deb the debug symbols package is openjdk-17-dbg

p-himik19:04:34

FWIW I wasn't able to make the profiler pick those symbols up, despite having that package installed.

p-himik19:04:53

But the error also seem to have nothing to do with debug symbols. Don't quote me on that though.

👀 2
Ben Sless19:04:01

yeah I don't think debug symbols are related either

👀 2
Ben Sless19:04:02

start with trying 11 and 8 (crossing fingers etc), if 11 gives you trouble maybe 15 but then you're cooking with gas

Drew Verlee19:04:39

That looks promising. I'll install those and report back. Tyty

Ben Sless19:04:31

I had trouble where debug symbols weren't picked up when I started the process, I used colorful language but to no avail

Drew Verlee21:04:50

@UK0810AQ2, installing the debug symbols did the trick. That is, running:

apt install openjdk-11-dbg

Drew Verlee21:04:51

Though...the resulting flamegraph looks wrong. ill figure it out. 😆

Drew Verlee22:04:10

works with java version 11.

Drew Verlee22:04:28

also works with 17.

clojure-spin 4