Sean I just had a rogue idea. Would it be possible to implement this as an optional wrapper around a classloader-based test runner?

It could live in poly core then, and one could configure, say, the default test runner but wrapped in this.

That way, if, say I want kaocha+separate processes I don't have to copy code over and reimplement the process forking

and you wouldn't have to duplicate the default test runner's test discovery/execution/color coding

The classloader is the problem I need to avoid.

I mean the wrapper that creates the new process could just pass the process' own classloader along

Classpath. You can't pass a classloader across processes.

And the process needs to be started with that classpath.

wrapper->classpath->newprocess->classloader->testrunner

I meant that the subprocess could pass its own classloader to the testrunner. Not sure how to properly put it into words

But then that new process becomes pretty heavyweight since it has to duplicate all of the classloader and test orchestration logic from Polylith itself.

Not the classloader stuff, the process (which has the entire test classpath now) could just pass its own (default?) classloader

Orchestration: probably yes but I'd say that's probably lighter weight

so basically the default-test-runner implementation would have to be added to the subprocess classpath

this would separate concerns though: poly assembles classpath and eventually a classloader which is by default in-process but if the wrapper is used then it's in a subprocess, testrunner works off that classloader and figures out which tests to execute and how

I think you'd have to drag enough of Polylith into that new process that it wouldn't be worth doing this at all. It would slow the process down and reintroduce the exact problem I'm trying to avoid: classloader isolation which doesn't work properly.

See all my previous posts about problems with classloader isolation -- it's a flawed approach.

I'm not suggesting classloader isolation

You want minimal dependencies in your test runner process. You do not want a whole bunch of Polylith code (and its dependencies) in that process.

Building a classloader and trying to execute just the tests in it is a flawed approach -- it does not work in large, multi-threaded situations like ours.

I understand that and I agree that your suggestion of using a subprocess per project-test-run is most probably a good solution to this. What I'm saying is that whether or not the tests run in only a separate classloader or a separate process is one concern, and how and what tests are run is a different one. If you can separate the two then a developer can choose either the default or the kaocha runner, plus they can choose whether they want to stay in-process or use subprocesses. Right now: • poly ◦ classloader1 ▪︎ test-runner ▪︎ test1 ◦ classloader2 ▪︎ test-runner ▪︎ test2 suggestion: • poly ◦ subprocess1 ▪︎ classloader of subprocess1 • test-runner • test1 ◦ subprocess2 ▪︎ classloader of subprocess2 • test-runner • test2

The test runner doesn't need to have lots of polylith dependencies. The kaocha one only uses the testrunner interface for example

It does depend on kaocha on the other hand but that would be the case with a barebones single depsedn project tested with kaocha as well

Having a subprocess that is passed the classpath and then builds a classloader and then runs the tests inside it, is not a solution to the problem we have at World Singles.

It might be an interesting, elegant way to structure things, but it is the wrong solution to this problem.

If someone else wants to build that "solution" and submit a PR, that's fine. But it won't be something World Singles' folks do.

If the Polylith team don't want a simple, clean subprocess test runner, then we'll continue to use our fork indefinitely.

I can't speak for the Polylith team but I understand your reasons

Trying to run tests in a classloader inside an already running process breaks for some of our code at work. We have reported several problems with it, we've had one very small PR accepted that mitigates one small aspect of the problems, and we've already had to modify our code in... unfortunate... ways to work around another aspect of the problems. But, fundamentally, having the test runner create a classloader and attempt to run arbitrary test code inside it is broken for us.

Cognitect's test-runner works for us: it has very minimal dependencies and runs the tests in-process without a custom classloader. I don't know how the Kaocha test runner works but I see it uses dynapath and add-classpath stuff so I suspect we'd have problems with that too...

Sure I understand, won't be pressing further

(it still isn't Cognitect's test runner that's used though)

I know that. Prior to using poly test, we were using Cognitect's test-runner. And we still use it for legacy subprojects, where it works for our complex, multi-threaded test code.

When we migrated our chat server to a Polylith base, that's when we first started seeing problems with the Polylith test runner -- we hadn't had problems up until then. We worked around those problems. Then we migrated our rebilling engine to Polylith and ran into more problems. Hence this whole discussion and the subprocess test runner.

Both of those projects' tests ran just fine with the Cognitect test-runner (no classloader stuff). Both of those projects' tests failed somewhat randomly with Polylith's test runner (often only in CI).

We still have a few, equally complex, equally multi-threaded legacy projects to migrate (whose tests run just fine with the Cognitect runner and may or may not run fine with Polylith's test runner but I'm going to be a lot more confident about them running with the subprocess test runner).