Fork me on GitHub
#clojure-uk
<
2021-07-20
>
djm05:07:45

πŸ‘‹

dharrigan06:07:05

Good Morning!

yogidevbear12:07:12

Morning β˜€οΈ

Ben Hammond15:07:05

Hi guys; I have a (slightly off topic) Docker / TeamCity Agent / Gradle problem that I am running out of ideas on I am trying to run http://testcontainers.org mongo db from within an integration test but it is blowing up on init(), essentially because it is trying to check System.getenv("PATH") but it is getting a null response This is only happening when it runs on the Dockerized TeamCity agent I can echo $PATH on a bash process within the agent container, and I get a sensible result There's a load of sequentially forked java processes: 1. jetbrains.buildServer.agent.Launcher 2. jetbrains.buildServer.agent.AgentMain 3. org.gradle.wrapper.GradleWrapperMain 4. org.gradle.launcher.daemon.bootstrap.GradleDaemon 5. http://worker.org.gradele.process.internal.worker.GradleWorkerMain I think its that last one that is actually failing Any suggestions?

Ben Hammond15:07:20

I'm expecting the system environment variables to get passed between forked processes

Ben Hammond15:07:35

but perhaps I dont understand anything...

dominicm15:07:51

between fork() they do

Ben Hammond15:07:53

well I don't actually know how they got spawned I just know that ps reports a parentage

dominicm15:07:00

When doing a fork() you can choose to clear the environment too.

djm15:07:15

Which container are you running the echo in? The agent container, or the mongo container?

dominicm15:07:32

Oh, if this is the JVM spawning other JVMs, then I bet it doesn't convey the environment.

Ben Hammond15:07:33

I am running the echo in the agent container

dominicm15:07:38

The JVM ChildProcess thing is a bit weird.

Ben Hammond15:07:20

I'm naively ignoring the mongo container and hoping that http://test-container.org knows what it is about

Ben Hammond15:07:41

but I think I get the NPE before testcontainers does any real work

Conor15:07:42

I assume you've checked that you can run Docker-in-Docker on the build agent, if it's already a Docker container?

Conor15:07:57

We had to jump through some hoops to get Testcontainers working with Concourse

dominicm15:07:37

Just checked, java.lang.Process does keep the environment unless you explicitly change it.

Ben Hammond15:07:18

> I assume you've checked that you can run Docker-in-Docker on the build agent, if it's already a Docker container? I am expecting this to be a problem but I was tacking the NullPointerException first unless they are somehow the same problem..?

dominicm15:07:27

Oh, Gradle uses a background daemon

dominicm15:07:32

so it might be running where PWD isn't set.

Ben Hammond15:07:05

but I think the Gradle Daemon is just a plain old jvm isn't it

dharrigan15:07:24

On the CI, it's usually recommened to disable the deamon

dharrigan15:07:25

When we were using gitlab (in my previous job) we always had this on our build steps

dharrigan15:07:40

GRADLE_OPTS: "-Dorg.gradle.daemon=false -Dorg.gradle.caching=true"

Ben Hammond15:07:24

yes I think that IS set in my case - but it stll runs the org.gradle.launcher.daemon.bootstrap.GradleDaemon class

dharrigan15:07:37

maybe also include .gradle/gradle.properties and have that on too?

dharrigan15:07:42

other than that, no idea 😞

Ben Hammond15:07:36

it run TeamCity agent on java8 and flips to java13 for gradle dunno if that is important...

Conor15:07:53

I haven't used Gradle in ages, but I seem to recall that you need to explicitly make environment variables available to it

Ben Hammond12:07:12

thanks for suggestions

Ben Hammond12:07:12

I can manually trigger gradlew -Dorg.gradle.daemon=false --info integration-test:test from bash shell on the docker instance and my test class successfully printout all the System.getenv() values I would have expected... so it must be somthing about how TeamCity Agent spawns the gradlewrapper that is depriving it of environment vars

Ben Hammond15:07:28

Update: I have just discovered the delightfully named

fun hackilySetStuffInTheEnv(passwords: Map<String,String>) {
whos running seems strongly correlated with the results of
System.getenv("PATH")
going tits-up

Ben Hammond15:07:11

thanks for all your suggestions; in this case I suspect that Docker - TeamCityAgent - Gradle - TestContainer are all behaving themselves and its my team's dodgy code that is the issue

Ben Hammond15:07:18

same old story really

πŸ˜… 3
dominicm21:07:51

Where was that!

dominicm21:07:56

Is it using JNI to do that? πŸ˜„

Ben Hammond20:07:57

no it was using some horrible non public api I think

Ben Hammond20:07:08

will report back on Monday

Ben Hammond07:07:03

that code fragment does a hard-to-understand thing in the catch block; I suspect that this does not work, (probaly has never worked)

protected static void setEnv(Map<String, String> newenv) throws Exception {
  try {
...
  } catch (NoSuchFieldException e) {
    Class[] classes = Collections.class.getDeclaredClasses();
    Map<String, String> env = System.getenv();
    for(Class cl : classes) {
      if("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
        Field field = cl.getDeclaredField("m");
        field.setAccessible(true);
        Object obj = field.get(env);
        Map<String, String> map = (Map<String, String>) obj;
        map.clear();
        map.putAll(newenv);
      }
    

Ben Hammond07:07:12

this is exactly why I am so dubious about github copilot

Ben Hammond07:07:34

I believe the phrase is > flood the zone with shit

Conor07:07:44

It's simply abusing reflection and the JVM internals in a perfectly clear and horrifying way

Ben Hammond07:07:19

but how the hell is it set setting environment variables by abusing unmodifiable map?

Ben Hammond07:07:33

well, makes me glad I'm charging by the day

Ben Hammond08:07:11

It annoys me because it is so completely unnecessary It can be simplified by using plain old System.setProperty() https://docs.oracle.com/en/java/javase/13/docs/api/java.base/java/lang/System.html#setProperty(java.lang.String,java.lang.String)

dominicm08:07:38

Only if there's a corresponding env variable right?

Ben Hammond08:07:45

and then the password accessors just have to read from eith Environment OR java property

dominicm08:07:51

As in, whatever you are using checks env and property.

Ben Hammond08:07:13

and thats a pattern you always want, imho

Ben Hammond08:07:35

because it gives you so much more flexibility

Ben Hammond08:07:03

(and works properly)

djm08:07:17

CoPilot is worse than StackOverflow - at least with StackOverflow you get context about what probably is being solved, comments on different approaches, pros and cons etc, and people will explain why a bad solution is a bad. With CoPilot you don’t get that

βœ… 3
πŸ’― 3
dominicm08:07:56

I still think it's silly that java doesn't let you set env.

dominicm08:07:59

It's a simple posix call 😜

Ben Hammond08:07:19

but java doesn't know where it is running java cannot assume where it will eventually be running

Ben Hammond08:07:28

might not be a posix env

Conor08:07:33

The real pro move is to assume you're running on Linux like Go does and silently fail to do things properly if you're not

Ben Hammond08:07:18

so thats where I've been going wrong all this time

Ben Hammond08:07:56

possessing a conscience can be so disabling

Ben Hammond08:07:29

back to

rotected static void setEnv(Map<String, String> newenv) throws Exception {
  try {
...
  } catch (NoSuchFieldException e) {
    Class[] classes = Collections.class.getDeclaredClasses();
    Map<String, String> env = System.getenv();
    for(Class cl : classes) {
      if("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
        Field field = cl.getDeclaredField("m");
        field.setAccessible(true);
        Object obj = field.get(env);
        Map<String, String> map = (Map<String, String>) obj;
       ** map.clear(); **
        map.putAll(newenv);
      }
I reckon if get rid of that map.clear() my immediate problem might go away

Ben Hammond08:07:05

I dont understand what the clear is trying to achieve, other than destroying everthing

Conor08:07:26

I presume the intent is to overwrite whatever's there with the new environment map. It would be better to merge the two maps

Ben Hammond08:07:58

yeah if we just do the putAll on top of what was there before

Ben Hammond08:07:08

I'll try it as a quick fix

Ben Hammond08:07:09

whilst I contemplate replacing with JVM Properties

Ben Hammond08:07:07

when you are copy-and-pasting StackOverflow code, do you not always credit it with the relevant SO url in a comment? surely that is common courtesy to Those Who Come After You