This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-04-05
Channels
- # aleph (2)
- # announcements (3)
- # architecture (5)
- # beginners (51)
- # biff (5)
- # cider (1)
- # clerk (8)
- # clj-kondo (6)
- # cljsrn (5)
- # clojure (31)
- # clojure-europe (42)
- # clojure-nl (1)
- # clojure-norway (21)
- # clojure-uk (3)
- # emacs (11)
- # fulcro (2)
- # graphql (6)
- # hugsql (1)
- # jobs (2)
- # leiningen (3)
- # lsp (3)
- # malli (13)
- # missionary (1)
- # off-topic (7)
- # pathom (7)
- # polylith (27)
- # reagent (14)
- # reitit (3)
- # remote-jobs (7)
- # shadow-cljs (20)
- # spacemacs (4)
- # sql (3)
- # tools-build (4)
- # xtdb (7)
I'm having a strange problem where running under zsh vs under bash (both local and container). I have a config.edn that pulls in other configs using juxt/aero..
;;local files to include
:env/config #include #join [#env EXTERNAL_CONFIG "env_config.edn"]
:secrets/config #include #join [#env EXTERNAL_CONFIG "secrets.edn"]}
... and under zsh it runs fine, finds the files and everything is OK. Under bash (same env var, machine, etc) it fails:
;;local files to include
:env/config #include #join [#env EXTERNAL_CONFIG "/env_config.edn"]
:secrets/config #include #join [#env EXTERNAL_CONFIG "/secrets.edn"]} (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:158)
at $fn__11496.invokeStatic(io.clj:229)
at $fn__11496.invoke(io.clj:229)
at $fn__11409$G__11402__11416.invoke(io.clj:69)
at $fn__11508.invokeStatic(io.clj:258)
at $fn__11508.invoke(io.clj:254)
at $fn__11409$G__11402__11416.invoke(io.clj:69)
at $fn__11470.invokeStatic(io.clj:165)
at $fn__11470.invoke(io.clj:165)
at $fn__11422$G__11398__11429.invoke(io.clj:69)
at $reader.invokeStatic(io.clj:102)
at $reader.doInvoke(io.clj:86)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at aero.core$read_config_into_tagged_literal.invokeStatic(core.cljc:194)
at aero.core$read_config.invokeStatic(core.cljc:414)
Any ideas on why that could happen?Looking at the error, the way it includes that snippet of config, it looks sort of like it is trying to use that whole string as the file name
Which suggests something is really wrong, interestingly some of the quotes on the string are escaped and some are not, which again suggests something funky
You have secrets.edn
in one place and /secrets.edn
in the other -- I doubt that file is in the root of the file system?
I think, but hard to tell from what is pasted, that is part of the error message from the exception, it looks sort of like the fragment of the config was half way pr-str and then that was attempt to be opened as a file
Oh, no, of course what am I thinking, wrong slashes for escapes on the wrong side of the quotes
@US893PCLF what is in your EXTERNAL_CONFIG
variable? Pretty sure the "no such file" error is because of that /
...
I've tried both .. which is why it's different between pastes (didn't catch that earlier).
EXTERNAL_CONFIG=/tmp/resources/
in the container...
EXTERNAL_CONFIG=/Users/chris/Local/ixi/projects/ixi/resources
... however I can repro this problem with zsh and bash in the container .. so the default shell has something setup such that this works.... default on my Macbook Pro. I'll diff the env vars and see what differences there are.
Main config file (for completeness)
{
:http/pedestal {:async-supported? true
:auth-validate #ig/ref :auth/validate
:auth-login #ig/ref :auth/login
:auth-login-user #ig/ref :auth/login-user
:auth-logout #ig/ref :auth/logout
:db #ig/ref :storage/dummy
:env #ref [:env/config :env]
:join? false
:openai-chat #ig/ref :openai/chat
:openai-completions #ig/ref :openai/completions
:openai-embeddings #ig/ref :openai/embeddings
:openai-edits #ig/ref :openai/edits
:port #or [#env PORT 8080]
:container-options {:ssl? #ref [:env/config :ssl/ssl?]
:ssl-port #ref [:env/config :ssl/port] }
:server-type :immutant}
:auth/configuration {:client {:client-id #ref [:secrets/config :auth0/clientid]
:client-secret #ref [:secrets/config :auth0/client-secret]}
:m2m {:audience #ref [:secrets/config :auth0/m2m-audience]
:client-id #ref [:secrets/config :auth0/m2m-clientid]
:client-secret #ref [:secrets/config :auth0/m2m-secret]}
;; this will change only if we pay Auth0 $$ to have one that masquerades as our domain
:base-url "<url>"
:endpoints {:authorize "/authorize"
:jwk "/.well-known/jwks.json"
:logout "/logout"
:oauth-token "/oauth/token"}
:response-type "code"}
:auth/login {:config #ig/ref :auth/configuration
:redirect-uri #ref [:env/config :login/redirect-uri]}
;; consider moving the auth0 routes and redirect-uri's into the auth config section as they don't change
;; for a particular flow.
;;
:auth/login-user {:config #ig/ref :auth/configuration
:grant-type "client_credentials"
:redirect-uri #ref [:env/config :login-user/redirect-uri]
:required? true}
:auth/logout {:config #ig/ref :auth/configuration
:return-to #ref [:env/config :logout/redirect-uri]} ;; must match redirect URI in Auth0 dashboard
:auth/validate {:config #ig/ref :auth/configuration
:required? true}
:account/signup {}
:openai/configuration {:api-key #ref [:secrets/config :openai/apikey]
:org "engrammicai"}
:openai/chat {:route :chat
:config #ig/ref :openai/configuration}
:openai/completions {:route :completions
:config #ig/ref :openai/configuration}
:openai/edits {:route :edits
:config #ig/ref :openai/configuration}
:openai/embeddings {:route :embeddings
:config #ig/ref :openai/configuration}
:storage/dummy {}
;;local files to include
:env/config #include #join [#env EXTERNAL_CONFIG "/env_config.edn"]
:secrets/config #include #join [#env EXTERNAL_CONFIG "/secrets.edn"]}
Using export <name>=<val> locally, for the container I'm using ENV dockerfile command.
Are you setting it via docker or whatever container runtimes environment settings? Is your process pid 1 in the container?
Is the environment variable not getting set? Getting set to something incorrect? Or is the file not there?
I can repro this locally (using any shell started from vs code) so I'm not sure it's a container issue now. It's something in my default shell config that seems to have it working vs any shell started inside VS Code (also happens in containers.. )
locally ..
export EXTERNAL_CONFIG=$HOME/Local/ixi/projects/ixi/resources
which resolves to:
EXTERNAL_CONFIG=/Users/chris/Local/ixi/projects/ixi/resources
Yes it is set (can get it from the repl launched in one of those shells). I don't want to deploy again for config changes, so I'm moving the configs out of the jar. They'll eventually be projected into that location from somewhere else (pushed down as configmaps, etc.).
Aero can include files from the class path (relative path) or file system (absolute path). It's what we use at work. When you are running the program inside a container, those EDN files are inside the container too, yes? Or at least those paths are mounted into the container...
They are. However I just found and fixed the problem. It was as stupid as I feared it would be. I'd left a slurp in the pipeline that fed Aero's read-config (when pulling the code from another project that didn't use Aero) ... the imported files were a red herring based on the exception logging from the container and the fact that it looked like Aero was calling out it couldn't find the secrets file. Was just an artifact of the filename being the entire config file lol. Thank you everyone for piling in.
So @U0NCTKEV8 was basically right at the beginning of this thread:grin:
Yes 🙂
Sometimes the bug is the weirdest possible suggestion:smile:
True, it never seems to be the cool bug that makes you look smart 😆
Maybe https://clojure.org/reference/transducers could help, but I don't know much about it yet
Hmmm interesting... Guess I'd go for the reduce if I needed performance, but I see the value if it becomes too big, like multiple maps+filters, then transducers might look much nicer
transducers is really not going to matter much here. personally, I think reduce (or transduce) is the clearest way to convey the intent of what you are doing here and so I would start with that
it won't matter much because: you're only doing one operation, and you are eagerly doing all the work either way. the sequence variant will do a bit more work in memory, but due to chunking, it's probably not really going to make much difference (and might even be faster)
chunked sequence amortization is annoyingly fast, and in a case where you're using all of it, that's all win
sequence chunking is a form of amortization - doing stuff more efficiently by using batches to gain efficiency / performance. The potential downside is that if you only need the first thing in the batch, you might be doing work that is unnecessary (esp bad if the unit op is slow). but if you are going to add ALL the numbers, there is no downside, it's just faster
You might also get some good feedback on this question in #C03M5U2LLAC if you ask there 🙂
Personally, I’d be curious about the context around this code! More context tends to surface more problems, at least in my experience.
I’m not too experienced with transducers, so (->> blobs (map :size) (reduce +))
is what I can read and understand the “quickest”. It’s a good fit for ->>
.