Fork me on GitHub
#beginners
<
2023-04-05
>
Chris Lester02:04:31

I'm having a strange problem where running under zsh vs under bash (both local and container). I have a config.edn that pulls in other configs using juxt/aero..

;;local files to include
 :env/config #include #join [#env EXTERNAL_CONFIG "env_config.edn"]
 :secrets/config #include #join [#env EXTERNAL_CONFIG "secrets.edn"]}
... and under zsh it runs fine, finds the files and everything is OK. Under bash (same env var, machine, etc) it fails:
;;local files to include
 :env/config #include #join [#env EXTERNAL_CONFIG "/env_config.edn"]
 :secrets/config #include #join [#env EXTERNAL_CONFIG "/secrets.edn"]} (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:158)
	at $fn__11496.invokeStatic(io.clj:229)
	at $fn__11496.invoke(io.clj:229)
	at $fn__11409$G__11402__11416.invoke(io.clj:69)
	at $fn__11508.invokeStatic(io.clj:258)
	at $fn__11508.invoke(io.clj:254)
	at $fn__11409$G__11402__11416.invoke(io.clj:69)
	at $fn__11470.invokeStatic(io.clj:165)
	at $fn__11470.invoke(io.clj:165)
	at $fn__11422$G__11398__11429.invoke(io.clj:69)
	at $reader.invokeStatic(io.clj:102)
	at $reader.doInvoke(io.clj:86)
	at clojure.lang.RestFn.invoke(RestFn.java:410)
	at aero.core$read_config_into_tagged_literal.invokeStatic(core.cljc:194)
	at aero.core$read_config.invokeStatic(core.cljc:414)
Any ideas on why that could happen?

hiredman02:04:21

You have autoexport on in zsh and haven't exported the var in bash

hiredman02:04:04

Looking at the error, the way it includes that snippet of config, it looks sort of like it is trying to use that whole string as the file name

2
hiredman02:04:21

Which suggests something is really wrong, interestingly some of the quotes on the string are escaped and some are not, which again suggests something funky

hiredman02:04:36

Makes me doubt the difference is bash vs. zsh

seancorfield02:04:15

You have secrets.edn in one place and /secrets.edn in the other -- I doubt that file is in the root of the file system?

hiredman02:04:33

I think, but hard to tell from what is pasted, that is part of the error message from the exception, it looks sort of like the fragment of the config was half way pr-str and then that was attempt to be opened as a file

hiredman02:04:22

Oh, no, of course what am I thinking, wrong slashes for escapes on the wrong side of the quotes

seancorfield02:04:26

@US893PCLF what is in your EXTERNAL_CONFIG variable? Pretty sure the "no such file" error is because of that /...

Chris Lester03:04:44

I've tried both .. which is why it's different between pastes (didn't catch that earlier).

EXTERNAL_CONFIG=/tmp/resources/
in the container...
EXTERNAL_CONFIG=/Users/chris/Local/ixi/projects/ixi/resources
... however I can repro this problem with zsh and bash in the container .. so the default shell has something setup such that this works.

Chris Lester03:04:05

... default on my Macbook Pro. I'll diff the env vars and see what differences there are.

hiredman03:04:51

How are you setting the env var?

Chris Lester03:04:06

Main config file (for completeness)

{
 :http/pedestal {:async-supported? true
                 :auth-validate #ig/ref :auth/validate
                 :auth-login #ig/ref :auth/login
                 :auth-login-user #ig/ref :auth/login-user
                 :auth-logout #ig/ref :auth/logout
                 :db #ig/ref :storage/dummy
                 :env #ref [:env/config :env]
                 :join? false
                 :openai-chat #ig/ref :openai/chat
                 :openai-completions #ig/ref :openai/completions
                 :openai-embeddings #ig/ref :openai/embeddings
                 :openai-edits #ig/ref :openai/edits
                 :port #or [#env PORT 8080]
                 :container-options {:ssl?      #ref   [:env/config :ssl/ssl?]
                                     :ssl-port   #ref  [:env/config :ssl/port] }
                 :server-type :immutant}
 
 :auth/configuration {:client {:client-id #ref [:secrets/config :auth0/clientid]
                               :client-secret #ref [:secrets/config :auth0/client-secret]}
                      :m2m {:audience #ref [:secrets/config :auth0/m2m-audience]
                            :client-id  #ref [:secrets/config :auth0/m2m-clientid]
                            :client-secret #ref  [:secrets/config :auth0/m2m-secret]}

                      ;; this will change only if we pay Auth0 $$ to have one that masquerades as our domain
                      :base-url "<url>"
                      :endpoints {:authorize "/authorize"
                                  :jwk "/.well-known/jwks.json"
                                  :logout "/logout"
                                  :oauth-token "/oauth/token"}
                      :response-type "code"}
:auth/login {:config #ig/ref :auth/configuration
              :redirect-uri #ref [:env/config :login/redirect-uri]}

 ;; consider moving the auth0 routes and redirect-uri's into the auth config section as they don't change
 ;; for a particular flow.
 ;;
 :auth/login-user {:config #ig/ref :auth/configuration
                   :grant-type "client_credentials"
                   :redirect-uri #ref [:env/config :login-user/redirect-uri]
                   :required? true}

 :auth/logout {:config #ig/ref :auth/configuration
               :return-to #ref [:env/config :logout/redirect-uri]} ;; must match redirect URI in Auth0 dashboard

 :auth/validate {:config #ig/ref :auth/configuration
                 :required? true}

 :account/signup {}

 :openai/configuration {:api-key #ref [:secrets/config :openai/apikey]
                        :org "engrammicai"}

 :openai/chat {:route :chat
               :config #ig/ref :openai/configuration}

 :openai/completions {:route :completions
                      :config #ig/ref :openai/configuration}

 :openai/edits {:route :edits
                :config #ig/ref :openai/configuration}

 :openai/embeddings {:route :embeddings
                     :config #ig/ref :openai/configuration}

 :storage/dummy {}

 ;;local files to include
 :env/config #include #join [#env EXTERNAL_CONFIG "/env_config.edn"]
 :secrets/config #include #join [#env EXTERNAL_CONFIG "/secrets.edn"]}

Chris Lester04:04:10

Using export <name>=<val> locally, for the container I'm using ENV dockerfile command.

hiredman04:04:14

Are you setting it via docker or whatever container runtimes environment settings? Is your process pid 1 in the container?

hiredman04:04:52

Is the environment variable not getting set? Getting set to something incorrect? Or is the file not there?

Chris Lester04:04:17

I can repro this locally (using any shell started from vs code) so I'm not sure it's a container issue now. It's something in my default shell config that seems to have it working vs any shell started inside VS Code (also happens in containers.. )

hiredman04:04:18

How are you setting the env?

Chris Lester04:04:22

locally ..

export EXTERNAL_CONFIG=$HOME/Local/ixi/projects/ixi/resources
which resolves to:
EXTERNAL_CONFIG=/Users/chris/Local/ixi/projects/ixi/resources

hiredman04:04:24

You can check using System/getenv to see if the environment variable is set

hiredman04:04:06

Resources are usually bundled inside an uberjar, so they are not files on disk

Chris Lester04:04:53

Yes it is set (can get it from the repl launched in one of those shells). I don't want to deploy again for config changes, so I'm moving the configs out of the jar. They'll eventually be projected into that location from somewhere else (pushed down as configmaps, etc.).

seancorfield04:04:14

Aero can include files from the class path (relative path) or file system (absolute path). It's what we use at work. When you are running the program inside a container, those EDN files are inside the container too, yes? Or at least those paths are mounted into the container...

Chris Lester04:04:51

They are. However I just found and fixed the problem. It was as stupid as I feared it would be. I'd left a slurp in the pipeline that fed Aero's read-config (when pulling the code from another project that didn't use Aero) ... the imported files were a red herring based on the exception logging from the container and the fact that it looked like Aero was calling out it couldn't find the secrets file. Was just an artifact of the filename being the entire config file lol. Thank you everyone for piling in.

seancorfield04:04:52

So @U0NCTKEV8 was basically right at the beginning of this thread:grin:

seancorfield04:04:19

Sometimes the bug is the weirdest possible suggestion:smile:

Chris Lester04:04:49

True, it never seems to be the cool bug that makes you look smart 😆

vncz16:04:21

Stylistic question: Which of the two do you prefer?

andre16:04:52

map + reduce if performance doesn't matter

vncz16:04:11

That was also my thought; is the former slower than the latter?

vncz16:04:21

but in general no, I do not care

andre16:04:30

I think it is, because you have to iterate the vector twice

andre16:04:17

Maybe https://clojure.org/reference/transducers could help, but I don't know much about it yet

vncz16:04:29

Transducers would fix that problem

vncz16:04:35

Yes exactly that 🙂

andre16:04:22

so ((comp (map :size) (reduce +)) blobs) would work and use transducers?

vncz16:04:16

reduce does not return a transducer

vncz16:04:19

It would be more like (transduce (map :size) + blobs)

vncz16:04:05

Or `(->> blobs (transduce (map :size) +))`

andre16:04:57

Hmmm interesting... Guess I'd go for the reduce if I needed performance, but I see the value if it becomes too big, like multiple maps+filters, then transducers might look much nicer

👍 2
Alex Miller (Clojure team)16:04:38

transducers is really not going to matter much here. personally, I think reduce (or transduce) is the clearest way to convey the intent of what you are doing here and so I would start with that

👍 2
Alex Miller (Clojure team)16:04:47

it won't matter much because: you're only doing one operation, and you are eagerly doing all the work either way. the sequence variant will do a bit more work in memory, but due to chunking, it's probably not really going to make much difference (and might even be faster)

vncz17:04:07

Wonderful, thanks

Alex Miller (Clojure team)17:04:29

chunked sequence amortization is annoyingly fast, and in a case where you're using all of it, that's all win

vncz17:04:18

Ok this last part I do not understand

Alex Miller (Clojure team)18:04:40

sequence chunking is a form of amortization - doing stuff more efficiently by using batches to gain efficiency / performance. The potential downside is that if you only need the first thing in the batch, you might be doing work that is unnecessary (esp bad if the unit op is slow). but if you are going to add ALL the numbers, there is no downside, it's just faster

💯 4
teodorlu11:04:57

You might also get some good feedback on this question in #C03M5U2LLAC if you ask there 🙂

teodorlu11:04:50

Personally, I’d be curious about the context around this code! More context tends to surface more problems, at least in my experience. I’m not too experienced with transducers, so (->> blobs (map :size) (reduce +)) is what I can read and understand the “quickest”. It’s a good fit for ->>.

vncz16:04:11

There is not much context to add, it was just that snippet that I wrote on the fly to explain a small thing to a coworker 🙂