Fork me on GitHub
#cljdoc
<
2021-08-07
>
deleted03:08:05

gosh these data structures are so huge

lread15:08:01

the nippied blobs in the db?

martinklepsch20:08:40

I used interceptors pretty heavily back in the day, could be there’s some usages where a function might have been just as good 😄

lread02:08:56

Yeah, that cache bundle can be a bit big and mysterious to me too. Probably because I'm not in the code super often.

martinklepsch10:08:00

No worries, on the contrary I wanted to acknowledge that things might be strange 😄

martinklepsch10:08:27

There’s a schema for the bundle in the analyzer code btw. Maybe that helps you find your way around.

martinklepsch10:08:47

A lot of it is directly taken from the structure codox provided, with a few tweaks here and there.

lread13:08:21

Given how old cljdoc is and how many people have worked on it, I find the code quite good!

🙂 2
rkiouak14:08:58

My apologies, I had asked about this failing job: https://app.circleci.com/pipelines/github/cljdoc/builder/20050/workflows/d1b47500-b9e5-40e7-bb42-dc1cdf994953/jobs/36425 (where the failure looks to be, seemingly inexplicably, on this line: https://github.com/protojure/lib/blob/master/src/protojure/internal/grpc/codec/io.clj#L99) a while back and have now lost the slack history where the request was that I add a test reproducing this to the cljdoc repo. Could someone point me at the best place to reproduce this for debugging?

lread15:08:58

I can take a peek at this sometime later today

🙇 3
lread19:08:43

I can reproduce the symptom you are seeing with https://github.com/cljdoc/cljdoc-analyzer by running:

clojure -M -m cljdoc-analyzer.main analyze --project protojure/protojure --version 1.6.1 --output-filename protojure-test.edn

lread19:08:33

Not that it is necessarily related, but a strange thing I see is clojure core class files in the protojure jar file:

❯ unzip -l protojure-1.6.1.jar
Archive:  protojure-1.6.1.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
...deleted...
     1320  06-11-2021 16:50   clojure/tools/reader/default_data_readers$fn__3726.class
     2530  06-11-2021 16:50   clojure/tools/reader/reader_types$fn__3447.class
     3694  06-11-2021 16:50   clojure/tools/reader/default_data_readers$validated$fn__3690.class
     1847  06-11-2021 16:50   clojure/tools/reader/reader_types$fn__3355$G__3350__3360.class
     1014  06-11-2021 16:50   clojure/tools/reader/default_data_readers$fn__3693$fn__3694.class
        0  06-11-2021 16:50   clojure/tools/reader/impl/
      699  06-11-2021 16:50   clojure/tools/reader/impl/inspect$fn__3581.class
     1127  06-11-2021 16:50   clojure/tools/reader/impl/utils$newline_QMARK_.class
     1250  06-11-2021 16:50   clojure/tools/reader/impl/errors$throw_invalid_octal_len.class
     1980  06-11-2021 16:50   clojure/tools/reader/impl/utils$compile_when.class
     2380  06-11-2021 16:50   clojure/tools/reader/impl/inspect$fn__3565.class
     1490  06-11-2021 16:50   clojure/tools/reader/impl/errors$throw_invalid_unicode_digit_in_token.class
     1265  06-11-2021 16:50   clojure/tools/reader/impl/inspect$inspect.class
     2836  06-11-2021 16:50   clojure/tools/reader/impl/inspect$dispatch_inspect.class
     1173  06-11-2021 16:50   clojure/tools/reader/impl/errors$throw_bad_escape_char.class
      681  06-11-2021 16:50   clojure/tools/reader/impl/inspect$fn__3583.class
...etc...

lread20:08:46

I’ll stop poking around until I hear back from you.

rkiouak20:08:09

Yeah, I am in the process of getting the bits that require aot pulled out, though i imagine there is probably an easier way to configure the current aot setup yo only aot the needed bits, not sure what that is though. Do you think those are the cause?

rkiouak22:08:09

IIUC, this is roughly a description of what may be causing the issue? https://clojurians.slack.com/archives/C0AB48493/p1628374759011400

rkiouak22:08:41

heh, it now occurs to my very slow brain that we may be uploading the -standalone jar rather than the non-standalone we intend, will check this I don’t think this is the issue, as I suspect these class files have always been in a jar, and it was only recent builds where the cljdoc job started failing (though I’ve posted in leiningen and can track that thread in case it turns out to be related. Any other things we can look at trying in the meanwhile?

lread02:08:01

I'll have another peek tomorrow. I can manually delete the clojure classes from the jar and retry. Interesting to know that this project used to work on cljdoc. Curious: I'm guessing that aot is a must for some interop reasons?

lread02:08:18

I'm not a lein user, so likely won't be of any great help there.

rkiouak10:08:54

Don’t feel compelled to pull in, I suspect you are right and the analyzer run will work without the class files included based on some testing I did last night. We’ll try to get a set of PRs in that move the aot’d nses to another lib, and see if that resolves. Yeah, we need the AOT for a couple of gen-classes to interop with Undertow (a Java based web server).

lread13:08:02

Wether removing those Clojure classes solves the problem or not, it would at least remove a big variable from the equation. I'm curious enough to try it sometime today.

rkiouak17:08:59

To be clear, im fairly certain you are right.

lread18:08:05

Ya, but I’m not sure of the absolute cause! simple_smile

lread18:08:14

So I’m no lein guy… (this might be very hacky, please tell me if I’ve gone way astray anywhere here)… if I clone protojure, then add the following to project.clj before :profiles

:jar-exclusions [#"\.class$" 
                   #"^clojure"]
  :jar-inclusions [#"^protojure/internal/grpc/io/InputStream\.class"
                   #"^protojure/internal/grpc/io/OutputStream\.class"
                   #"^protojure/pedestal/io/InputStream\.class"]
This effectively removes the clojure class files and only includes protojure’s 3 gen-class class (that right?) files in the jar. If I then do a:
lein clean
lein install
And the from cljdoc-analyzer, if I try:
clojure -M -m cljdoc-analyzer.main analyze --project protojure/protojure --version 1.6.2-SNAPSHOT --output-filename protojure-test.edn
I get the same “Can only recur from tail position” error you originally reported. So… not great to include unnecessary class files in in a lib jar, but not the contributor (or only contributor) to this issue?

lread21:08:59

Okey dokey… so there seems to be a problem around the version of org.clojure/core.async? Pedestal service refs https://github.com/pedestal/pedestal/blob/6b518f1403132e0474daa5edaf08b1de1df267b5/service/project.clj#L26, and protojure refs https://github.com/protojure/lib/blob/7ab7e4fac6231dee9300fe5a86896e148e0711c1/project.clj#L14 (scope “provided”). If I downgrade protojure to core.async 0.4.490, I get much further in analysis. Not verified but my guess is that core.async 0.4.490 is fine with your go macro usage but 1.3.618 finds exception with it (literally!).

lread21:08:11

And then we have the next issue. The cljdoc analyzer will automatically determine if your project is clj, cljs or both by the types of files it finds within it. It is assuming that protojure is both Clojure and ClojureScript because it contains .cljc files. And it is now failing when trying to analyze protojure as a CloureScript project. If protojure is only a Clojure project, then renaming those .cljc files to .clj files would probably make sense. If protojure is used from ClojureScript… well… lemme know.

lread22:08:21

(not sure I understand core.async problem, might be version+aot?)

rkiouak00:08:32

I very much suspect moving the aot of files out of the leiningen project will resolve the analyzer issues, just waiting for a +1 from another maintainer on those PRs to get merged. Will let you know (to your investigative points, moving the aot’d nses to another lib “fixes” the behavior where all of the dependency class files are included in the jar.

lread02:08:59

I thought you needed a few aoted classes for interop? Anyway, I will leave this in your capable hands for now. Don't forget about the "next issue" I mentioned above.

rkiouak12:08:31

Yes, the aot’d classes are needed for interop, but they have a relatively small footprint, and if aot’d in a separate library and required in the protojure/lib project.clj, the extraneous class files aren’t being pulled into the jar by leiningen. The .cljc files are needed for cljs interop.

lread13:08:12

Right, gotcha. Just a thought: If you decide to leave the necessary aot-ed *Stream classes in the protojure jar itself, and if you are still having issues, you might consider making them much thinner, to have them only as calls to clojure code and nothing else. That way less code is aoted. To recap my hacky explorations, I did get cljdoc analysis working for Clojure, but it was failing for ClojureScript analysis. So after you solve your aot problem, there is likely a second problem for you to solve.

lread16:08:06

Huh, an interesting coincidence, just noticed this new video on protojure: https://youtu.be/cGeWBJtOz2Y

rkiouak16:08:30

Yeah, I'm trying to get the cljdocs issue resolved in light of knowing the talk/video may prompt visits to the project and tire kicking

rkiouak16:08:47

At this point I'm just waiting on another maintainers approval of PRs, and some follow up from why leiningen includes dependencies in it's aot pass (its actually clojure aot compiler, but thats probably more detail than you care for ;))

lread16:08:01

I watched the vid and noticed that the presenter stated ClojureScript is not yet supported but planned, so cljdoc ClojureScript analysis might fail for protojure. Lemme know when/if you hit this. We can temporarily hardcode Clojure only analysis for protojure or... maybe we'll add a feature to allow lib authors to specify what platforms (Clojure and/or ClojureScript) they'd like cljdoc to analyze for in cljdoc.edn

rkiouak11:08:02

one thing I’d like to test — with this diff applied to the project.clj, does the analyzer run still fail on the generated jar?

git diff
diff --git a/project.clj b/project.clj
index 339a8bd..168d9ad 100644
--- a/project.clj
+++ b/project.clj
@@ -24,6 +24,9 @@
                  [commons-io/commons-io "2.9.0"]
                  [funcool/promesa "3.0.0"]
                  [lambdaisland/uri "1.4.70"]]
+  ;; Workaround for  by deleting
+  ;; compilation artifacts for namespaces that come from dependencies.
+  :clean-non-project-classes true
   :aot [
         ]
   :codox {:metadata {:doc/format :markdown}

lread12:08:24

@U7F53SKPD you can try this yourself locally by cloning then using https://github.com/cljdoc/cljdoc-analyzer against a locally installed protojure jar (`lein install` from protojure project dir). See some of my examples in this thread for how I did this.

lread12:08:45

Please let us know how it goes - and ping back if you get stuck or have questions.

rkiouak20:08:43

@UE21H2HHD — I do see a cljc related error once the aot’d resources are moved to a new library (removing the non-project class files seemed to be insufficient to resolve, fwiw):

Reflection warning, protojure/internal/grpc/client/providers/http2/core.clj:148:44 - reference to field close can't be resolved.
{:clojure.main/message
 "Unexpected error (IllegalArgumentException) compiling at (/private/var/folders/9j/stfkqkvj07vfzzlsd2f9lwh00000gn/T/cljdoc-protojure-protojure-1.6.2-SNAPSHOT14612348720605984274/src/main/clojure/protojure/protobuf/serdes/complex.cljc:6:1).\nDon't know how to create ISeq from: clojure.lang.Keyword\n",
 :clojure.main/triage
 {:clojure.error/source nil,
  :clojure.error/line 6,
  :clojure.error/column 1,
  :clojure.error/phase :compilation,
  :clojure.error/path
  #object[java.io.File 0x3cc669cc "/private/var/folders/9j/stfkqkvj07vfzzlsd2f9lwh00000gn/T/cljdoc-protojure-protojure-1.6.2-SNAPSHOT14612348720605984274/src/main/clojure/protojure/protobuf/serdes/complex.cljc"],
  :clojure.error/class java.lang.IllegalArgumentException,
  :clojure.error/cause
  "Don't know how to create ISeq from: clojure.lang.Keyword"},
 :clojure.main/trace
 {:via
  [{:type clojure.lang.ExceptionInfo,
    :data
    {:clojure.error/source
     #object[java.io.File 0x3cc669cc "/private/var/folders/9j/stfkqkvj07vfzzlsd2f9lwh00000gn/T/cljdoc-protojure-protojure-1.6.2-SNAPSHOT14612348720605984274/src/main/clojure/protojure/protobuf/serdes/complex.cljc"],
     :clojure.error/line 6,
     :clojure.error/column 1,
     :clojure.error/phase :compilation},
    :at
    [cljs.analyzer$analyze_seq_STAR__wrap
     invokeStatic
     "analyzer.cljc"
     3978]}
   {:type clojure.lang.ExceptionInfo,
    :message
    "Don't know how to create ISeq from: clojure.lang.Keyword at line 6 /private/var/folders/9j/stfkqkvj07vfzzlsd2f9lwh00000gn/T/cljdoc-protojure-protojure-1.6.2-SNAPSHOT14612348720605984274/src/main/clojure/protojure/protobuf/serdes/complex.cljc",
    :data
    {:file
     #object[java.io.File 0x3cc669cc "/private/var/folders/9j/stfkqkvj07vfzzlsd2f9lwh00000gn/T/cljdoc-protojure-protojure-1.6.2-SNAPSHOT14612348720605984274/src/main/clojure/protojure/protobuf/serdes/complex.cljc"],
     :line 6,
     :column 1,
     :tag :cljs/analysis-error},
    :at [cljs.analyzer$error invokeStatic "analyzer.cljc" 762]}
   {:type java.lang.IllegalArgumentException,
    :message
    "Don't know how to create ISeq from: clojure.lang.Keyword",
    :at [clojure.lang.RT seqFrom "RT.java" 557]}],
  :trace
  [[clojure.lang.RT seqFrom "RT.java" 557]

rkiouak20:08:52

thanks for all the help on this btw!

lread21:08:16

Yeah I think that protojure cljc code does not evaluate successfully under ClojureScript?

rkiouak23:08:13

Possibly, though is there a way to disable the cljs evaluation on the repository?

lread00:08:32

Not currently (edit: except via hardcoding exceptions as a support request). But it might be a helpful feature to add for repos like this where ClojureScript support is not, or only partly, working.

lread01:08:47

Raised a git issue: https://github.com/cljdoc/cljdoc/issues/468. In any case, we do have the ability to hardcode config override for clj only analysis server side for protojure.

rkiouak12:08:14

yes, lets please override for now, if it's reasonably low effort for you to do so. Do I need to make a formal request somewhere?

lread12:08:34

Nope, I can do... but I think I’ll expose the feature to give this power to library authors instead. Will likely work on this today.

lread13:08:42

Or tomorrow... I’ve got a bicycle that needs to be repaired...

rkiouak13:08:54

enjoy the ride if you get the repair in today!

🚲 2
simple_smile 2
lread15:08:41

@deleted-user yep, deploys to prod are automatic after merge to master. Surprised me the first time, but really like it.

👍 3