polylith 2022-09-02 | Slack Archive

An update on our migration:

projects: 20   interfaces: 77
  bases:    16   components: 79

which is 81,436 lines (out of 141,364 -- about 58%). After the PR to fix the Clojure initialization issue was merged, which had blocked us on our large app migration (see August 18th post above), we ran into a new problem that we believe is related to classloaders and threads but it's intermittent and so it's hard to diagnose (and likely hard to fix). If we re-run poly test multiple times, we do tend to get a passing run "eventually". Here's what we think is happening: • poly test sets up a classloader for each project's test dependencies • it uses the original parent of Clojure's classloader as the parent for these new classloaders • we have code that relies on Java's async features that in turn calls reify -- so it's a Clojure to Java-async to Clojure code path • sometimes this works just fine (yay!) • sometimes we get ClassCastExceptions (or other strange exceptions from the Clojure compiler itself) that suggest the class being reify'd is in a different classloader than the code that is receiving it Our hypothesis is that there are thread pools within the overall JVM that end up trying to refer to classes from multiple classloaders. We've partially gotten around some of this by trying to move the reify calls onto the same thread that invokes the Java async code but we haven't managed to catch all occurrences yet. Judicious use of bound-fn also seems to help in a few places (although that may be coincidental at this point -- we don't have enough test run data to be certain of anything yet!).

👍 3

seancorfield17:09:46

#Also sent to the channel

It's been a busy week at World Singles Networks:

projects: 20   interfaces: 87
  bases:    16   components: 89

That's 85,287 lines out of 141,887 which is just over 60% now! We have four legacy apps to refactor to bases and two legacy subprojects to refactor to (a lot of) components at this point.

seancorfield04:09:41

(just as a side note, since *use-context-classloader* defaults to true, we're also trying a fork of Polylith that binds it to false in the eval code for test runners -- it defaults to true so I'm not sure why the test runner eval code has binding in the first place? Anyway, so far, with that bound to false, we've had test passes every time... we'll keep y'all posted)

hiredman05:09:24

Something fun: that binding will effect the clojure runtime polylith is running with, but not the clojure runtime in the isolated classloader, because of course that has its own separate vars

seancorfield05:09:09

Yeah, figured. But it's another variable to play with/eliminate as a factor.

Thomas Moerman07:09:45

Q: I use Docker to package up my polylith app. One of the steps in the build stage of the Dockerfile is to run the test suite, as I want the build to fail when the test suite fails. I noticed that when running tests using RUN clj -M:poly test in the Dockerfile, no tests are executed. Is this perhaps because there is no .git folder in the Docker image? If so, is there a flag or a workaround to tell Polylith to ignore .git when running tests? Thanks.

imre08:09:41

Do your tests get packaged into the docker image?

imre08:09:07

could also try passing :all to poly test

Daniel Gerson10:09:30

Q: I assume that adopting the polylith tool can't be done piecemeal? I.e. I'd like to turn on just interface checking, and ignore projects that aren't ready?

tengstrand10:09:58

When you say ignoring projects, do you mean ignoring the tests for specific projects, or maybe ignoring the “top namespace” check?

tengstrand10:09:48

One way of doing it could be to put all code into the src directory of the project, instead of putting the code into a base, and then migrating the code into that base from the project source directory, in steps.

👍 1

Daniel Gerson10:09:45

Thanks for the reply. I guess I meant ignoring tests for specific projects, or perhaps a particular project doesn't have a deps.edn because it's irregular (but it still makes sense for it to look for shared code in its relative place). I guess I'll just have to try it, but thought it would be good to ask first in case there was anything obvious I was missing.

seancorfield21:09:25

@U03B2SRNYTY We've migrated a large monorepo to Polylith over time and poly simply ignores all non-`bases`, non-`components` stuff -- so legacy (unmigrated) code is not affected. I've written a series of blog posts about it -- not sure if there's enough detail in there to help? https://corfield.org

Daniel Gerson13:09:57

@U04V70XH6 Thanks, I'll need some time to experiment. I have read your blogposts. My codebase is largely Clojurescript, so there's that. Also still working on getting Dockerised LocalStack testing fully working (rolled my own solution given didn't want to pay for commercial offering of Websocket & ApiGatewayV2 Localstack). All that complicates the ordering of testing the whole stack.

seancorfield17:09:46

replied to a thread:An update on our migration: projects: 20 interfaces: 77 bases: 16 components: 79 which is 81,436 lines (out of 141,364 -- about 58%). After the PR to fix the Clojure initialization issue was merged, which had blocked us on our large app migration (see August 18th post above), we ran into a new problem that we _believe_ is related to classloaders and threads but it's intermittent and so it's hard to diagnose (and likely hard to fix). If we re-run `poly test` multiple times, we do tend to get a passing run "eventually". Here's what we _think_ is happening: • `poly test` sets up a classloader for each project's test dependencies • it uses the original parent of Clojure's classloader as the parent for these new classloaders • we have code that relies on *Java's* async features that in turn calls `reify` -- so it's a Clojure to Java-async to Clojure code path • sometimes this works just fine (yay!) • sometimes we get ClassCastExceptions (or other strange exceptions from the Clojure compiler itself) that suggest the class being reify'd is in a different classloader than the code that is receiving it Our hypothesis is that there are thread pools within the overall JVM that end up trying to refer to classes from multiple classloaders. We've partially gotten around some of this by trying to move the `reify` calls onto the same thread that invokes the Java async code but we haven't managed to catch all occurrences yet. Judicious use of `bound-fn` also seems to help in a few places (although that may be coincidental at this point -- we don't have enough test run data to be certain of anything yet!).

It's been a busy week at World Singles Networks:

projects: 20   interfaces: 87
  bases:    16   components: 89

That's 85,287 lines out of 141,887 which is just over 60% now! We have four legacy apps to refactor to bases and two legacy subprojects to refactor to (a lot of) components at this point.

2022-09-02

Channels