2025-05-06 clerk | Clojure Slack Archive

clerk 2025-05-06

pieterbreed 2025-05-06T07:21:24.519089Z

Hi, I want to do a "side-effectful" call to load some messages from kafka into a clerk notebook, similar to what's explained in the book under section Cached Evaluation. My code looks like this:

^{::clerk/visibility {:code :show :result :hide}}
(def msgs
  (let [_run-at #inst "2025-05-06T07:16:47.060791288Z"]
    (-kafka-utils/load-messages-from-confluent-topic :bootstrap-server (:kafka-bootstrap-server cfg)
                                                     :api-key           (:confluent-api-key cfg)
                                                     :api-secret        (:confluent-api-secret cfg)
                                                     :topics            ["v2-update-product"]
                                                     :extra-strategies  [(-kafka/StringSerializer :consumer :key)
                                                                         (-kafka/EdnSerializer :consumer :value)
                                                                         (-kafka/SeekToTimestampOffset msgs-when-to-start)
                                                                         ]
                                                     :collect-messages? true
                                                     :nr-msgs           1000)))


(first msgs)

This code works in the repl, but in the clerk window I'm getting:

Unhandled clojure.lang.ExceptionInfo
Cannot find field parseLong for class class java.lang.Long
{:class java.lang.Long 
 :column 60 
 :end-column 99 
 :end-line 528 
 :field parseLong 
 :file "NO_SOURCE_PATH"  
 :line 15}

borkdude 2025-05-06T07:46:36.332499Z

Can you paste more of the stacktrace (e.g. is it coming from tools analyzer) and can you try clerk from the main branch last commit? We switched out the analyzer for a custom home-built one

pieterbreed 2025-05-06T07:49:39.896099Z

The stacktrace is quite long, I didn't want to lead with that. This below looks like it's recursed deep into PersistentVector.java: 418

analyze_host_expr.clj:	73	clojure.tools.analyzer.passes.jvm.analyze-host-expr/analyze-host-field
analyze_host_expr.clj:	68	clojure.tools.analyzer.passes.jvm.analyze-host-expr/analyze-host-field
analyze_host_expr.clj:	172	clojure.tools.analyzer.passes.jvm.analyze-host-expr/analyze-host-expr
analyze_host_expr.clj:	142	clojure.tools.analyzer.passes.jvm.analyze-host-expr/analyze-host-expr
Var.java:	386	clojure.lang.Var
passes.clj:	167	clojure.tools.analyzer.passes/compile-passes/fn--8852/fn--8857
passes.clj:	169	clojure.tools.analyzer.passes/compile-passes/fn--8852/fn--8859
passes.clj:	169	clojure.tools.analyzer.passes/compile-passes/fn--8852/fn--8859
passes.clj:	169	clojure.tools.analyzer.passes/compile-passes/fn--8852/fn--8859
passes.clj:	169	clojure.tools.analyzer.passes/compile-passes/fn--8852/fn--8859
passes.clj:	169	clojure.tools.analyzer.passes/compile-passes/fn--8852/fn--8859
passes.clj:	169	clojure.tools.analyzer.passes/compile-passes/fn--8852/fn--8859
passes.clj:	169	clojure.tools.analyzer.passes/compile-passes/fn--8852/fn--8859
core.clj:	2648	clojure.core/partial/fn--5929
ast.clj:	102	clojure.tools.analyzer.ast/walk/walk--8741
ast.clj:	96	clojure.tools.analyzer.ast/walk/walk--8741/walk--8742
utils.clj:	208	clojure.tools.analyzer.utils/mapv'
utils.clj:	202	clojure.tools.analyzer.utils/mapv'
ast.clj:	51	clojure.tools.analyzer.ast/-update-children/fn--8730
PersistentVector.java:	418	clojure.lang.PersistentVector
core.clj:	6964	clojure.core/reduce
core.clj:	6947	clojure.core/reduce
ast.clj:	49	clojure.tools.analyzer.ast/-update-children
ast.clj:	46	clojure.tools.analyzer.ast/-update-children
ast.clj:	64	clojure.tools.analyzer.ast/update-children-reduced
ast.clj:	58	clojure.tools.analyzer.ast/update-children-reduced
ast.clj:	99	clojure.tools.analyzer.ast/walk/walk--8741
ast.clj:	96	clojure.tools.analyzer.ast/walk/walk--8741/walk--8742
ast.clj:	51	clojure.tools.analyzer.ast/-update-children/fn--8730
...

Bottom of the stack:

...
PersistentVector.java:	418	clojure.lang.PersistentVector
core.clj:	6964	clojure.core/reduce
core.clj:	6947	clojure.core/reduce
ast.clj:	49	clojure.tools.analyzer.ast/-update-children
ast.clj:	46	clojure.tools.analyzer.ast/-update-children
ast.clj:	64	clojure.tools.analyzer.ast/update-children-reduced
ast.clj:	58	clojure.tools.analyzer.ast/update-children-reduced
ast.clj:	99	clojure.tools.analyzer.ast/walk/walk--8741
ast.clj:	96	clojure.tools.analyzer.ast/walk/walk--8741/walk--8742
ast.clj:	51	clojure.tools.analyzer.ast/-update-children/fn--8730
PersistentVector.java:	418	clojure.lang.PersistentVector
core.clj:	6964	clojure.core/reduce
core.clj:	6947	clojure.core/reduce
ast.clj:	49	clojure.tools.analyzer.ast/-update-children
ast.clj:	46	clojure.tools.analyzer.ast/-update-children
ast.clj:	64	clojure.tools.analyzer.ast/update-children-reduced
ast.clj:	58	clojure.tools.analyzer.ast/update-children-reduced
ast.clj:	99	clojure.tools.analyzer.ast/walk/walk--8741
ast.clj:	95	clojure.tools.analyzer.ast/walk
ast.clj:	84	clojure.tools.analyzer.ast/walk
ast.clj:	115	clojure.tools.analyzer.ast/postwalk
ast.clj:	110	clojure.tools.analyzer.ast/postwalk
ast.clj:	113	clojure.tools.analyzer.ast/postwalk
ast.clj:	110	clojure.tools.analyzer.ast/postwalk
passes.clj:	171	clojure.tools.analyzer.passes/compile-passes/analyze--8864
core.clj:	2586	clojure.core/comp/fn--5895
core.clj:	2586	clojure.core/comp/fn--5895
jvm.clj:	476	clojure.tools.analyzer.jvm/run-passes
jvm.clj:	468	clojure.tools.analyzer.jvm/run-passes
jvm.clj:	519	clojure.tools.analyzer.jvm/analyze/fn--11408/fn--11413
jvm.clj:	517	clojure.tools.analyzer.jvm/analyze/fn--11408
AFn.java:	152	clojure.lang.AFn
AFn.java:	144	clojure.lang.AFn
core.clj:	667	clojure.core/apply
core.clj:	1990	clojure.core/with-bindings*
core.clj:	1990	clojure.core/with-bindings*
RestFn.java:	428	clojure.lang.RestFn
jvm.clj:	506	clojure.tools.analyzer.jvm/analyze
jvm.clj:	486	clojure.tools.analyzer.jvm/analyze
analyzer.clj:	138	nextjournal.clerk.analyzer/analyze-form/fn--37043
core.clj:	7706	clojure.core/with-redefs-fn
core.clj:	7690	clojure.core/with-redefs-fn
analyzer.clj:	135	nextjournal.clerk.analyzer/analyze-form
analyzer.clj:	128	nextjournal.clerk.analyzer/analyze-form
analyzer.clj:	172	nextjournal.clerk.analyzer/analyze
analyzer.clj:	164	nextjournal.clerk.analyzer/analyze
analyzer.clj:	397	nextjournal.clerk.analyzer/analyze-doc/fn--37167
PersistentVector.java:	418	clojure.lang.PersistentVector
core.clj:	6964	clojure.core/reduce
core.clj:	6947	clojure.core/reduce
analyzer.clj:	381	nextjournal.clerk.analyzer/analyze-doc
analyzer.clj:	374	nextjournal.clerk.analyzer/analyze-doc
analyzer.clj:	442	nextjournal.clerk.analyzer/analyze-file
analyzer.clj:	436	nextjournal.clerk.analyzer/analyze-file
analyzer.clj:	598	nextjournal.clerk.analyzer/build-graph/fn--37251
protocols.clj:	42	clojure.core.protocols/iterator-reduce!
protocols.clj:	52	clojure.core.protocols/iter-reduce
protocols.clj:	74	clojure.core.protocols/fn--8256
protocols.clj:	74	clojure.core.protocols/fn--8256
protocols.clj:	13	clojure.core.protocols/fn--8203/G--8198--8216
core.clj:	6965	clojure.core/reduce
core.clj:	6947	clojure.core/reduce
analyzer.clj:	585	nextjournal.clerk.analyzer/build-graph
analyzer.clj:	568	nextjournal.clerk.analyzer/build-graph
eval.clj:	271	nextjournal.clerk.eval/+eval-results
eval.clj:	263	nextjournal.clerk.eval/+eval-results
clerk.clj:	72	nextjournal.clerk/show!/fn--41728
clerk.clj:	71	nextjournal.clerk/show!
clerk.clj:	22	nextjournal.clerk/show!
clerk.clj:	143	nextjournal.clerk/file-event
clerk.clj:	135	nextjournal.clerk/file-event
clerk.clj:	509	nextjournal.clerk/serve!/fn--41811
beholder.clj:	13	nextjournal.beholder/fn/reify--31727
DirectoryWatcher.java:	402	io.methvin.watcher.DirectoryWatcher
DirectoryWatcher.java:	349	io.methvin.watcher.DirectoryWatcher
DirectoryWatcher.java:	232	io.methvin.watcher.DirectoryWatcher
CompletableFuture.java:	1768	java.util.concurrent.CompletableFuture/AsyncSupply
CompletableFuture.java:	1760	java.util.concurrent.CompletableFuture/AsyncSupply
ForkJoinTask.java:	387	java.util.concurrent.ForkJoinTask
ForkJoinPool.java:	1312	java.util.concurrent.ForkJoinPool/WorkQueue
ForkJoinPool.java:	1843	java.util.concurrent.ForkJoinPool
ForkJoinPool.java:	1808	java.util.concurrent.ForkJoinPool
ForkJoinWorkerThread.java:	188	java.util.concurrent.ForkJoinWorkerThread

pieterbreed 2025-05-06T07:50:43.997699Z

I'll try with main and revert...

pieterbreed 2025-05-06T07:55:59.045909Z

It also breaks with main branch from clerk. I think it has to do with how -kafka-utils/load-messages-from-confluent-topic is implemented. https://github.com/Afrolabs/afrolabs-clj/blob/5222f9cade48491906744f5c924749f6c69b7803/src/afrolabs/components/kafka/utilities.clj#L34.

pieterbreed 2025-05-06T07:56:21.919259Z

Unhandled clojure.lang.ExceptionInfo
Could not resolve var: msgs
{:column 93 :file "NO_SOURCE_PATH"
 :line 35 :var msgs}

pieterbreed 2025-05-06T07:56:31.650809Z

The stacktrace looks extremely similar.

borkdude 2025-05-06T08:19:26.170449Z

If the stacktrace looks similar you don't have the main branch perhaps, since we don't use tools.analyzer in there anymore

borkdude 2025-05-06T08:20:20.338799Z

can you maybe also post the stacktrace from the main branch?

borkdude 2025-05-06T08:23:18.831709Z

or even better a full repro including deps.edn and standalone notebook that I can run locally

teodorlu 2025-05-06T08:31:14.382819Z

Not a solution to your problem (analyzer errors), but sharing how I'm working with "providing data" into Clerk notebooks as it can ensure a smooth Clerk experience by separating data generation Clerk analysis, and ensure quick Clerk feedback loops. — I do the following when I read data into Clerk from somewhere outside of the Clerk cache: 1. Push new data into some place outside of Clerk 2. Do a ^::clerk/no-cache (def new-analysis-timestamp (load-analysis-timestamp-from-local-data-outside-of-clerk-cache)) 3. Then define Clerk-controlled defs based on new-analysis-timestamp. As long as load-analysis-timestamp-from-local-data-outside-of-clerk-cache returns quickly, I can play nicely with Clerk's caching and get a quick feedback loop.

pieterbreed 2025-05-06T08:36:59.236739Z

Thank you Teodor, that's maybe addressing the issue I really have. My "load data from kafka" takes loooooong. Minutes, sometimes 10's of minutes. I think I'll manually create a json file, and then slurp in a way you suggest.

💯 1

❤️ 1

borkdude 2025-05-06T08:56:17.792819Z

@pieterbreed Still would like to have the longer stacktrace and/or repro so the analyzer that's not been released could possibly be improved :)

➕ 1

pieterbreed 2025-05-06T08:57:01.445379Z

Thank you @borkdude, I'll respond within hours 👍🏽 🙂

borkdude 2025-05-06T08:57:14.657479Z

👍 thanks

borkdude 2025-05-06T09:01:26.617459Z

Hmm, when I try to add that library via add-lib I'm getting:

Could not find artifact org.apache.kafka:kafka-clients:jar:7.9.0-ce in
   central ()
   {:via [{:type clojure.lang.ExceptionInfo, :message "Could not find artifact org.apache.kafka:kafka-clients:jar:7.9.0-ce in central ()", :data {:lib org.apache.kafka/kafka-clients, :coord {:mvn/version "7.9.0-ce", :deps/manifest :mvn, :dependents [afrolabs/afrolabs-clj], :parents #{[afrolabs/afrolabs-clj]}}}, :at [clojure.tools.deps.extensions.maven$get_artifact invokeStatic "maven.clj" 167]}], :trace [[clojure.tools.deps.extensions.maven$get_artifact invokeStatic "maven.clj" 167] [clojure.tools.deps.extensions.maven$get_artifact invoke "maven.clj" 155] [clojure.tools.deps.extensions.maven$eval1225$fn__1228 invoke "maven.clj" 178] [clojure.lang.MultiFn invoke "MultiFn.java" 244] [clojure.tools.deps$download_libs$fn__802$fn__803 invoke "deps.clj" 466] ...], :cause "Could not find artifact org.apache.kafka:kafka-clients:jar:7.9.0-ce in central ()", :data {:lib org.apache.kafka/kafka-clients, :coord {:mvn/version "7.9.0-ce", :deps/manifest :mvn, :dependents [afrolabs/afrolabs-clj], :parents #{[afrolabs/afrolabs-clj]}}}}
               interop.clj:   81  clojure.tools.deps.interop/invoke-tool
               interop.clj:   41  clojure.tools.deps.interop/invoke-tool
                  deps.clj:   48  clojure.repl.deps/add-libs
                  deps.clj:   59  clojure.repl.deps/add-lib
                  deps.clj:   59  clojure.repl.deps/add-lib
                      REPL:    1  user/eval60501
                      REPL:    1  user/eval60501
             Compiler.java: 7700  clojure.lang.Compiler/eval
    interruptible_eval.clj:  106  nrepl.middleware.interruptible-eval/evaluator/run/fn
    interruptible_eval.clj:  101  nrepl.middleware.interruptible-eval/evaluator/run
               session.clj:  230  nrepl.middleware.session/session-exec/session-loop
        SessionThread.java:   21  nrepl.SessionThread/run

borkdude 2025-05-06T09:03:35.530089Z

also not using regular deps.edn and then restarting the REPL:

afrolabs/afrolabs-clj {:git/url ""
                                                     :git/sha "5222f9cade48491906744f5c924749f6c69b7803"}

pieterbreed 2025-05-06T09:06:21.982849Z

You need this top-level in deps.edn :

:mvn/repos {"confluent" {:url ""}}

👍 1

borkdude 2025-05-06T09:10:57.320969Z

How to "preplib" a library that is in an alias?

$ clj -X:deps:dev prep
Error building classpath. The following libs must be prepared before use: [afrolabs/afrolabs-clj]
borkdude@Mac ~/dev/clerk (main*) $ clj -X:dev:deps prep
Error building classpath. The following libs must be prepared before use: [afrolabs/afrolabs-clj]

pieterbreed 2025-05-06T09:11:10.252319Z

lol - if you know then you may tell me too

borkdude 2025-05-06T09:12:47.607359Z

lol, ok. I moved it temporarily to the main deps

borkdude 2025-05-06T09:14:55.162269Z

What namespace is the -kafka/ alias from?

pieterbreed 2025-05-06T09:15:19.773909Z

[afrolabs.components.kafka :as -kafka]
   [afrolabs.components.kafka.utilities :as -kafka-utils]

borkdude 2025-05-06T09:17:51.104919Z

ok. I don't get an analysis failure with this repro:

(ns scratch
  (:require [afrolabs.components.kafka :as -kafka]
            [afrolabs.components.kafka.utilities :as -kafka-utils]
            [nextjournal.clerk :as clerk]))

(def cfg)
(def msgs-when-to-start)

^{::clerk/visibility {:code :show :result :hide}}
(defn dude []
  (let [msgs
        (let [_run-at #inst "2025-05-06T07:16:47.060791288Z"]
          (-kafka-utils/load-messages-from-confluent-topic :bootstrap-server (:kafka-bootstrap-server cfg)
                                                           :api-key           (:confluent-api-key cfg)
                                                           :api-secret        (:confluent-api-secret cfg)
                                                           :topics            ["v2-update-product"]
                                                           :extra-strategies  [(-kafka/StringSerializer :consumer :key)
                                                                               (-kafka/EdnSerializer :consumer :value)
                                                                               (-kafka/SeekToTimestampOffset msgs-when-to-start)
                                                                               ]
                                                           :collect-messages? true
                                                           :nr-msgs           1000))]
    (first msgs)))

Do you? (if you paste it in a scratch.clj file)

pieterbreed 2025-05-06T09:39:52.471619Z

Locally, I'm still seeing LOTS of clojure.tools.analyzer.* in the stacktrace. I have to understand why that's happening first. 🤔

borkdude 2025-05-06T09:40:09.950729Z

then you're not on the main branch of clerk

borkdude 2025-05-06T09:40:27.134119Z

or you have some junk in "target" or so

👀 1

borkdude 2025-05-06T09:41:12.750899Z

io.github.nextjournal/clerk {:git/sha "f4c5488e36c8df11fe352889544e7deb9af73cb7"}

pieterbreed 2025-05-06T09:42:42.117889Z

I have this active alias:

:notebooks
           {:extra-paths ["notebooks"]
            :extra-deps {io.github.nextjournal/clerk {#_#_:mvn/version "0.17.1102"
                                                      :git/url "git@github.com:nextjournal/clerk.git"
                                                      :git/sha "f4c5488e36c8df11fe352889544e7deb9af73cb7"}
                         ;; :exclusions [org.clojure/tools.analyzer
                         ;;              org.clojure/tools.analyzer.jvm
                         ;;              borkdude/edamame
                         ;;              io.methvin/directory-watcher
                         ;;              org.clojure/core.memoize
                         ;;              org.ow2.asm/asm
                         ;;              org.clojure/core.cache
                         ;;              org.clojure/data.priority-map]
                         }}

Those exclusions used to be required in a previous iteration using project.clj.

borkdude 2025-05-06T09:43:15.490849Z

and you don't accidentally have some other clerk somewhere else in the deps.edn?

borkdude 2025-05-06T09:44:56.879269Z

Perhaps also try clj -X:notebooks:deps tree from the command line and paste the output here?

pieterbreed 2025-05-06T09:53:49.409479Z

https://gist.github.com/pieterbreed/59076e5eb4f44bb225fb295a2f3957be

borkdude 2025-05-06T09:55:28.089229Z

ok core.async still pulls in tools.analyzer . org.clojure/core.async 1.8.741 . org.clojure/tools.analyzer.jvm 1.3.2 . org.clojure/tools.analyzer 1.2.0 but it's no longer used by clerk

borkdude 2025-05-06T09:56:01.666399Z

hey wait, clerk isn't even part of the output, what's going on?

pieterbreed 2025-05-06T10:05:46.510359Z

I don't know.

borkdude 2025-05-06T10:06:55.946419Z

Oh I might know, -X:deps requires you to supply aliases as an argument facepalm

borkdude 2025-05-06T10:08:48.235009Z

clj -X:deps tree :aliases '[:notebooks]'

🌟 1

pieterbreed 2025-05-06T10:16:00.169349Z

I updated https://gist.github.com/pieterbreed/59076e5eb4f44bb225fb295a2f3957be, it now shows clerk.

pieterbreed 2025-05-06T10:16:24.659569Z

and it's still core.async that is pulling it in...

borkdude 2025-05-06T10:18:26.585689Z

that's expected. but it's not expected that you would get a tools.analyzer stacktrace from clerk (unless you or one of your dependencies is using core.async in your notebook)

pieterbreed 2025-05-06T10:18:58.204589Z

yes, definitely core.async in that -kafka-utilities/load-... fn in the notebook

borkdude 2025-05-06T10:19:38.464769Z

borkdude 2025-05-06T10:20:41.295859Z

feel free to still provide a repro though. not sure why it would work in the REPL but not in the notebook

👍🏽 1

borkdude 2025-05-06T11:43:42.303619Z

Ah, regarding the preplib + aliases thing, same issue:

clj -X:deps prep :aliases '[....]'

🙏🏽 1

pieterbreed 2025-05-07T10:39:44.398669Z

Hi @borkdude, thanks for your help yesterday. I made the repro repo here: https://github.com/pieterbreed/repro-clerk-analyzer , but in the repro it works fine! It's not crashing like with whatever setup I've got going in the main (work) project.

borkdude 2025-05-07T10:40:22.886309Z

hmmm thanks for the repro though! did you try to turn your REPL off and on again? 😅

pieterbreed 2025-05-07T10:40:58.436739Z

If some inspiration strikes and I know what to change on my work project, or how to force the regression on the repro-repo, then I'll ping you again, but I honestly think it's probably the former. Edit: Meaning it's probably a configuration issue on my side. I just have to find it.

pieterbreed 2025-05-07T10:41:23.730529Z

It triggers repeatably in the work repository.

borkdude 2025-05-07T10:42:05.513589Z

thanks, happy to be pinged again

🙏🏽 1

Clojurians Log v2

clerk 2025-05-06