For other commands, it works

user=> (shell "where clj-kondo")
C:\Users\super\scoop\shims\clj-kondo.exe
{:proc #object[java.lang.ProcessImpl 0x7e17f120 "Process[pid=11688, exitValue=0]"], :exit 0, :in #object[java.lang.ProcessBuilder$NullOutputStream 0x19faa00c "java.lang.ProcessBuilder$NullOutputStream@19faa00c"], :out #object[java.lang.ProcessBuilder$NullInputStream 0x57fe8cde "java.lang.ProcessBuilder$NullInputStream@57fe8cde"], :err #object[java.lang.ProcessBuilder$NullInputStream 0x57fe8cde "java.lang.ProcessBuilder$NullInputStream@57fe8cde"], :prev nil, :cmd ["where" "clj-kondo"]}       
user=> (shell "clj-kondo")       
clj-kondo v2021.06.18


Options:
....... 
Lots of lines at the end removed 

I'm pretty sure all are in the path (as where could find it), but some commands seem to work while others get "The system cannot find the file specified"

Not much experience with windows, but it does sound like an issue with paths, even if where can find it outside of babashka. You may want to check if the path has been exported to babashka - which runs a sub-shell. In *nix shells, typically custom set path/env variables will not be passed on to subshells if they are not exported. It depends on what startup files a shell uses for config - some come from /etc, some from $HOME.

In babashka shell, try echo-ing the $PATH to confirm, maybe?

The thing is, I couldn't even get echo to work without calling the command prompt from within babashka

user=> (shell "echo %PATH%")               
java.io.IOException: Cannot run program "echo": CreateProcess error=2, The system cannot find the file specified [at <repl>:2:1]

user=> (shell "cmd /C \"echo %PATH%\"")  
C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Python39\Scripts\;C:\Python39\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;D:\Program Files (x86)\Brackets\command;C:\Program Files\PuTTY\;D:\Program Files\PostgreSQL\12\bin;D:\Program Files\PostgreSQL\12\lib;C:\Program Files (x86)\gettext-iconv\bin;D:\Users\super\AppData\Roaming\npm;D:\Program Files (x86)\Yarn\bin\;C:\Program Files\dotnet\;D:\src\flutter\bin;D:\Program Files\Calibre2\;D:\Program Files\nodejs\;C:\ProgramData\chocolatey\bin;D:\Users\super\AppData\Roaming\npm-cache;C:\Users\super\scoop\shims;C:\Users\super\AppData\Local\Programs\Python\Python37\Scripts\;C:\Users\super\AppData\Local\Programs\Python\Python37\;C:\Users\super\AppData\Local\Programs\Python\Python38\Scripts\;C:\Users\super\AppData\Local\Programs\Python\Python38\;C:\Users\super\AppData\Local\Microsoft\WindowsApps;C:\Users\super\AppData\Local\Programs\Microsoft VS Code\bin;D:\Users\super\AppData\Local\Programs\Git\cmd;C:\Users\super\AppData\Roaming\npm
{:proc #object[java.lang.ProcessImpl 0x443ea63f "Process[pid=16732, exitValue=0]"], :exit 0, :in #object[java.lang.ProcessBuilder$NullOutputStream 0x19faa00c "java.lang.ProcessBuilder$NullOutputStream@19faa00c"], :out #object[java.lang.ProcessBuilder$NullInputStream 0x57fe8cde "java.lang.ProcessBuilder$NullInputStream@57fe8cde"], :err #object[java.lang.ProcessBuilder$NullInputStream 0x57fe8cde "java.lang.ProcessBuilder$NullInputStream@57fe8cde"], :prev nil, :cmd ["cmd" "/C" "echo %PATH%"]}

Ah. Don’t have a windows machine handy to help, but honestly, I don’t know what is the default shell when invoking in windows, and whether that is configurable. And as I re-read your original, I realize I did not completely parse your output. So, interestingly, where worked from babashka alright! Interesting.

No worries @U06KGH6LB, maybe someone else will be able to help Now that I've explored it more, the problem can also be seen when using clojure.java.shell/sh , which makes me think it is a Java / java config thing in windows.

user=> (shell/sh "where" "npm")
{:exit 0, :out "D:\\Program Files\\nodejs\\npm\r\nD:\\Program Files\\nodejs\\npm.cmd\r\n", :err ""}

user=> (shell/sh "npm")         
Execution error (IOException) at java.lang.ProcessImpl/create (REPL:-2).
CreateProcess error=2, The system cannot find the file specified 

user=> (shell/sh "where" "clj-kondo")
{:exit 0, :out "C:\\Users\\super\\scoop\\shims\\clj-kondo.exe\r\n", :err ""}

user=> (shell/sh "clj-kondo")         
{:exit 0, :out "clj-kondo v2021.06.18\r\n\n\nOptions:\n\n  --lint <file>: a file can either be a normal file, directory or classpath. In the\n    case of a directory or classpath, only .clj, .cljs and .cljc will be\n    processed. Use - as filename for reading from stdin.\n\n  --lang <lang>: if lang cannot be derived from the file extension this option will be\n    used. Supported values: clj, cljs, cljc.\n\n  --filename <file>: in case stdin is used for linting, use this to set the\n    reported filename.\n\n  --cache-dir: when this option is provided, the cache will be resolved to this\n    directory. If --cache is false, this option will be ignored.\n\n  --cache: if false, won't use cache. Otherwise, will try to resolve cache\n  using `--cache-dir`. If `--cache-dir` is not set, cache is resolved using the\n  nearest `.clj-kondo` directory in the current and parent directories.\n\n  --config <config>: config may be a file or an EDN expression. See\n    \n\n  --config-dir <config-dir>: use this config directory instead of auto-detected\n    .clj-kondo dir.\n\n  --parallel: lint sources in parallel.\n\n  --dependencies: don't report any findings. Useful for populating cache while linting dependencies.\n\n  --copy-configs: copy configs from dependencies while linting.\n\n  --fail-level <level>: minimum severity for exit with error code.  Supported values:\n    warning, error.  The default level if unspecified is warning.\n\r\n", :err ""}

Perhaps npm isn’t a normal executable?

Perhaps try npm.cmd ?

Damn that was it

Thanks @U04V15CAJ!

Adding .CMD to a command would probably make it stop working on Linux/macos though. Do you know of any elegant solutions? ( Right now can't think of much more than an if statement 😅 )

I usually use a predicate that checks for Windows + if

See borkdude/jayfu’s bb.edn

Thanks 😁

something similar was posted by @U04V15CAJ https://gist.github.com/borkdude/335c9911cabf4db6d47cc772b4c69d4d

probably needs a custom bb build?

This is unsupported, it doesn't work in combination with System/getenv

unless you write your own C-based getenv, but libraries all go through System/getenv, so it doesn't compose

I need something which does work with System/getenv

Java caches the environment at startup, this is why it doesn't work

I have seen: https://stackoverflow.com/questions/318239/how-do-i-set-environment-variables-from-java

I don't know whether it could work that way

well yes, you can spawn a new process of course, that always works. but it's a new process

What about this one:

@SuppressWarnings({ "unchecked" })
  public static void updateEnv(String name, String val) throws ReflectiveOperationException {
    Map<String, String> env = System.getenv();
    Field field = env.getClass().getDeclaredField("m");
    field.setAccessible(true);
    ((Map<String, String>) field.get(env)).put(name, val);
  }

I will check that out

This one works

But not for babashka..

Stackoverflow mentions it also doesn't work on Windows

That's not a problem

It will run on lambda environment

I need to figure it out how to use it on babashka side

For obvious reasons I cannot import that class in babashka. Any ideas how to overcome this?

which class?

I have made this class:

package fierycod.holy_lambda;

import java.lang.reflect.Field;
import java.util.Map;

public class UpdateEnv {
  @SuppressWarnings({ "unchecked" })
  public static void updateEnv(String name, String val) {
    try {
      Map<String, String> env = System.getenv();
      Field field = env.getClass().getDeclaredField("m");
      field.setAccessible(true);
      ((Map<String, String>) field.get(env)).put(name, val);
    } catch (ReflectiveOperationException e) {
      System.err.println("[holy-lambda] Unable to patch in memory environment variable: " + name + ".Exiting!");
      System.exit(1);
    }
  }
};

I have compiled it an checked whether it works on Clojure side

It worked, so I switched to babashka and I couldn't import this class

Of course I know why

you can translate this code to Clojure perhaps

so you don't have to compile Java

Good idea

This works:

(set! *warn-on-reflection* true)

(defn set-env [name val]
  (let [env (System/getenv)
        clazz (class env)
        field (.getDeclaredField clazz "m")
        _ (.setAccessible field true)
        the-map (.get field env)
        _ (.put ^java.util.Map the-map name val)]))

(set-env "FOO" "BAR")

in clojure

but the output is:

$ clj -M /tmp/setenv.clj
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by user$set_env to field java.util.Collections$UnmodifiableMap.m
WARNING: Please consider reporting this to the maintainers of user$set_env
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

so I don't think this is a good idea ;)

Yeah. I have tried it. It works on Clojure side. On bb I got Unable to resolve classname: java.lang.reflect.Field

I have to somehow change the environment variable. I know that it's a hack. I really do, but I don't know how to do it otherwise

it's a hack that will only work in java 11, but when the next LTS comes out it won't work anymore, so it's better to not go this way

I just listened to a podcast "Java Insiders" about this whole topic

In lambda context it doesn't matter I think. I'm forcing use of java8 in HL

"Inside Java" it's called

well, bb is compiled with Java 11 already

and it will probably upgrade to the newest LTS. Note that on macOS only Java 11 is available

not 8

btw, in bb I get:

$ bb /tmp/setenv.clj
----- Error --------------------------------------------------------------------
Type:     java.lang.NoSuchFieldException
Message:  m
Location: 4:3

so I bet this won't work in a native image anyway

😞

unless you perhaps allow reflection on UnmodifiableMap something something in the reflection config

but it's a hack

Which you probably won't accept on bb side

well, perhaps first try to make it work in a normal native image and see what happens. I can't promise anything

why don't you launch another process with the correct environment set?

Wouldn't it be slower?

I think I don't much understand your question

why do you need to set an environment variable during the duration of the program. can you set it before launching the program?

I cannot

Because in the loop I'm fetching new AWS events, which contain a AWS_TRACE_ID, that I have to attach to environment variable

don't all Java / native image programs have this problem? why must this be solved through env vars? which libraries are reading these env vars?

Some code for context.

(defn- next-iter
  [routes env-vars]
  (let [runtime (get env-vars "AWS_LAMBDA_RUNTIME_API")
        handler-name (get env-vars "_HANDLER")
        aws-event (fetch-aws-event runtime)
        handler (get routes handler-name)
        iid (:invocation-id aws-event)]

    ;; aws-event contains a header with trace-id
    ;; I need bound the trace id to environment ariable right here.

    (when-not handler
      (send-runtime-error runtime iid (->ex (str "Handler " handler-name " not found!")))
      (System/exit 1))

    (when-not iid
      (send-runtime-error runtime iid (->ex (str "Failed to determine new invocation-id. Invocation id is:" iid)))
      (System/exit 1))

    (when (and (:invocation-id aws-event) (u/success-code? (:status aws-event)))
      (process-event runtime iid aws-event env-vars (u/call handler)))))

aws-xray now does not allow to overwrite trace id AFAIK 😂

Well, this design obviously doesn't play well with Java programs. I suggest discussing this with this AWS architect we had this discussion with on the GraalVM slack

For instance quarkus substitutes the xray class which their own implementation of TraceHeader class which takes the trace id not from environment variable, but from outer context 😂

Yeah I'm talking with Mark right now about it.

Thank you @U04V15CAJ! You convinced me about not making a hack!

It seems that I don't longer need setting an environment variable. 😄 AWS x-ray is going to introduce a system property check for trace id! @U04V15CAJ, you always save me from making those little dirty hacks! Thank you so much for you patience and hard work! I do learn a lot from you 🙂

:-D

$ uname -a
Linux moria 3.2.40 #25556 SMP PREEMPT Thu Mar 4 17:49:46 CST 2021 i686 GNU/Linux synology_evansport_415play

What is special about this linux? os it glibc based?

ah i686, this is 32 bit, probably not supported

yeah 😕

too bad, I was hoping with replacing my groovy scripts with bb ones

bb depends on graalvm which only has compilers for 64 bit: https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-21.1.0

yes I understand oh well, I'll make do and hope one day there will be a graalvm for IoT or older architectures 😅

https://github.com/oracle/graal/issues/1329

btw, there is a talk tonight about mirabelle at the Dutch Clojure Meetup, a monitoring system in clojure, might be interesting for you

I am not faimilar with the meetup

do you have a link?

the talk is already done

found the link

thanks

I joined the group for future

London Clojurians is also a nice online meetup

(I'm doing a talk next week about bb tasks...)

I replaced clj-http with httpkit (API seems to be compatible) - at least tests pass

I'm thining how I can test this since vault-clj provides leiningen, not deps.edn

maybe make a deps.edn

I guess I could provide deps.edn myself locally or just add vault-clj/src to bb paths?

yes, both should work

httpkit returns delays btw, not direct responses

so there is a difference in API regarding that

babashka.curl might be closer

but if you have to touch every spot anyway, it's only adding an @ which makes the difference with httpkit

ok, thanks

there is just one case where this needs to be done and a lot in tests

so no issue here I believe

I've ended up getting to log errors:

Checking out:  at 2d9a59d9d29bea963c6371b45bf93e25fd3ff297
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Could not resolve symbol: log/trace
Location: /home/ieugen/.gitlibs/libs/amperity/vault-clj/2d9a59d9d29bea963c6371b45bf93e25fd3ff297/src/vault/lease.clj:202:4
Phase:    analysis

----- Context ------------------------------------------------------------------
198: 
199: 
200: (defn ^:no-doc maintain-leases!
201:   [client window]
202:   (log/trace "Checking for renewable leases...")
        ^--- Could not resolve symbol: log/trace
203:   ; Check auth token for renewal.
204:   (let [auth @(:auth client)]
205:     (when (and (:renewable auth)
206:                (expires-within? auth window))
207:       (try

https://github.com/ieugen/vault-clj/tree/53-support-babashka-use-httpkit

ah is this from tools logging?

yes

trace should be available on master

I've built babashka locally from master

master newest?

bb --version 
babashka v0.4.7-SNAPSHOT

commit?

commit 2932e246522f976af1409ee9346a56fc1439bd55 (HEAD -> master, upstream/master)
Author: Michiel Borkent <[email protected]>
Date:   Wed Jul 14 16:31:56 2021 +0200

    v0.5.0

will build again

ok, I just pushed

another commit

which fixes the BLOAT PROBLEM

ok, will test with this, rebuild and test again. if that does not work maybe you can clone https://github.com/ieugen/vault-clj/tree/53-support-babashka-use-httpkit and run

bb vault-get-dev

first build same issue, building with latest commit

;; Type bb tasks to see all tasks
;; Type bb <task-name> or bb run <task-name> to run a task
{:min-bb-version "0.4.0"
 :paths ["script"]
 :deps {amperity/vault-clj {:git/url ""
                            :sha "2d9a59d9d29bea963c6371b45bf93e25fd3ff297"}
        org.clojure/tools.logging {:mvn/version "1.1.0"}}
 :tasks {:requires (
                    ;; [vault.core :as vault]
                    ;; [vault.client.http]
                    [clojure.tools.logging :as log])
        ;;  :init     (do
        ;;              (def client (vault/new-client "")))
        ;;  ;; Helpers
        ;;  vault-get-dev {:doc "Get dev secrets from vault - requires vault cli."
        ;;                 :task (do
        ;;                         (client))}
         log-me {:doc "Log test"
                       :task (do
                               (log/info "hello")
                               (log/trace "hmm"))}}}

bb log-me
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Could not resolve symbol: log/trace
Location: <expr>:24:25
Phase:    analysis

----- Context ------------------------------------------------------------------
20: nil
21: (def log-me (binding [
22:   babashka.tasks/*task* '{:name log-me, :doc "Log test", :task (do (log/info "hello") (log/trace "hmm"))}]
23:   nil
24: (do (log/info "hello") (log/trace "hmm")))) log-me

there is an issue with trace

do you have a way to add tests for bb features ?

tests that can run after build?

@eugen.stan Yes. trace is missing. I will add it

I do have tests, but not for everything regarding logging yet, since it is so now

I added all of these:

['debug 'debugf 'info 'infof 'warn 'warnf 'error 'errorf
                                              'logp 'logf '*logger-factory* 'log*]

but trace wasn't there yet

@eugen.stan pushed the trace commit

I'm debugging vault-clj

ok, I will finish the release and will continue tomorrow.

ok

so vault-clj works with babashka now and I hope to upstream some of the changes there https://github.com/amperity/vault-clj/pull/54

cool :)

it's going to require some cleanup though

nice work on your side too

it's nice you got so far. I guess you can already start using your fork locally for testing right

btw, the IllegalArgumentException change might not be needed: this is available in bb

yep, that one works now - IllegalStateException does not

ah

I'm taking very very tiny steps and compiling to native at every step...

a very exciting activity I imagine 🙂

very tedious and tiring. I have spent half my day on this :-S

but here's the link: https://www.meetup.com/The-Dutch-Clojure-Meetup/events/rfkzfsycckbsb/

and @U051T4Q05 might make the talk available later

oh wrong thread

;)