we're using ~this approach, using a logback.xml in resources, and also setting up the additional config in a integrant/init-key fn

but now we want to configure the papertrail port etc. inside that integrant function, depending on the integrant configuration (because different app configs might be loaded at boot)

ideally, we pass the port to integrant and configure it there. alternatively we could have multiple logback.xml/log4j.properties and load the correct one depending on config. what is the best way to achieve that?

It's possible, I use this for Sentry:

;; Potentially useful things to consider:
;; - 
;; - 
;; - 
;; The last two have an example usage at .
(defmethod ig/init-key ::logback
  [_ {:keys [sentry]}]
  (when-let [dsn (not-empty (:backend-dsn sentry))]
    ;; Sentry must be configured before its appender,
    ;; otherwise this call to `init!` will override the appender settings.
    (sentry/init! dsn (dissoc sentry :backend-dsn))
    (let [^LoggerContext context (LoggerFactory/getILoggerFactory)
          ^Logger root-logger (LoggerFactory/getLogger Logger/ROOT_LOGGER_NAME)
          appender (doto (SentryAppender.)
                     (.setMinimumEventLevel Level/WARN)
                     (.setMinimumBreadcrumbLevel Level/DEBUG)
                     (.setContext context))]
      (.addAppender root-logger appender)
      (.start appender)
      appender)))

Here is an example for logback: https://github.com/igrishaev/teleward/blob/master/src/teleward/logging.clj

+1 for unilog if you're not dead set on using logback.xml files

can you clarify what you mean by "mirror"? Mirrors are a thing that Maven explicitly supports in ~/.m2/settings.xml and CLI/deps do use those if configured. but it almost sounds like you do not mean that

My mirror is an intern Artifactory server

do you have it configured as a mirror in settings.xml?

if so, you should not have to do anything else

https://jfrog.com/help/r/jfrog-artifactory-documentation/maven-repository

I see in the Artifactory UI that it mirror various repos: clojars, central, ... but other command like clj -M:test do get their dependencies from the mirror and use the project local .m2 directory

the local .m2 directory is a cache, and is somewhat orthogonal

I'm not sure if it would help, but in lein, you specify the :local-repo path, and all the packages get downloaded into it. For example, in our lein project we have a :docker profile with settings:

:docker
   {:local-repo ".docker/m2"} 

the local repo is independent I think of the question being asked here, which is about where to get things from in the first place

@U1WAUKQ3E know that 😉 But this project uses CLI deps and to have a similar feature I use a custom deps string as arg of clojure cli

the command line args you pass to clj will not affect what happens in your build, which creates its own basis

you can pass :extra arg to create-basis to do the equivalent

but I still question whether you need to do any of that if you've configured maven properly to use artifactory

@alexmiller because everything run in a docker image in Gitlab-CI with cache configured for project local m2. Each run I can see my cache populated and every command using it, only clojure -T:build dont wants to use it 😁

well, see my answer on that above

I can clear the cash and see on the first clojure -M the download of packages from the mirror and my cache populated but clojure -Tbuild just do it again ignoring my -Sdeps config

As far as I got the question, the main problem is that CI downloads the libraries again and again. I believe, setting a local-repo would solve it. I've had the same issue on CI, the only difference is I have lein.

as I said, "the command line args you pass to clj will not affect what happens in your build, which creates its own basis. you can pass :extra arg to create-basis to do the equivalent"

@U1WAUKQ3E project using lein works fine for me too. My issues with my other project using CLI deps and tools.build 😉

ok @alexmiller , think I ll try the extra-arg solution to see where it gets me 😁

clojure -Tbuild is running your build program. inside the build program, it determines the basis for your project - that job is completely separate from the args you passed when running the build program itself.

Just to let you know, I did removed my -Sdeps and call my tools.build functions with a deps map to create a basis with :extra deps. It did work, but I could still see downloads from central. After few tests, I saw that I needed to keep my -Sdeps custom map because there is the initial download of jars for tools.build itself. But now at last, what a joy to see my initial run to download only from my mirror and my second run with a populated jars cache to get to work immediately without downloading a single jar! 🙌 Thank you Alex

"Minimal" deps.edn for me:

{:deps
 {org.clojure/clojure #:mvn{:version "1.12.0-alpha2"},
  com.google.cloud.opentelemetry/exporter-trace #:mvn{:version "0.23.0"},
  ring/ring-devel #:mvn{:version "1.9.6"},
  com.google.cloud/google-cloud-bigquery #:mvn{:version "2.14.6"},
  com.taoensso/carmine #:mvn{:version "3.2.0"},
  }}

Shell history:

clojure -version  
Clojure CLI version 1.11.1.1273
√ $ clj               
Clojure 1.12.0-alpha2
user=> (add-lib 'metosin/malli)
Execution error (IOException) at java.lang.ProcessImpl/forkAndExec (ProcessImpl.java:-2).
error=7, Argument list too long
user=> 

commenting out any of the bottom four deps in the list and then re-running works without error

a subset of the basis gets passed to the forked command

can you just put the general problem on https://ask.clojure.org ?

there are also impacts if you have a large set of aliases in ~/.clojure/deps.edn. curious if that's true for you?

yep. Thanks

ahh, good point I didn't think about that

there's a couple ways to address this, I wasn't sure how common of a problem it would be

I have a 9 aliases in ~/.clojure/deps.edn

well that doesn't seem like too many, probably more the deps then.

when adding a lib, you have to resolve against the current lib set, which means you need to pass the current lib set out to tools.deps, so that's where that's coming in. I'll try to get a fix for this into alpha3

https://ask.clojure.org/index.php/12852/execution-error-java-processimpl-forkandexec-argument-using

thx!

I bet I'll run into that. I'm off today but will test it at work on Monday.

FWIW, I might be hitting a related issue. I don’t have an easy way to repro yet, but I’m using a couple of aliases that, when combined, amount to a pretty big chunk of deps, and I get Unreadable arg: "{:existing {<snip big dep map>}}" when calling e.g. (clojure.repl.deps/add-lib 'com.github.seancorfield/next.jdbc). Not sure if that’s useful — I can try to come up with a repro if necessary.

Such as, perhaps, configure your tools.deps-or-whatever command as a Java Property for the core Clojure. There would then be a fig leaf that "it's not really Clojure" (and also it could actually be Leiningen)

have you looked at the add-lib stuff?

there is explicitly an var bound to try and limit its use to within the repl

and it shells out for dependency resolution and fetching (all done in another process)

Clojure has always let you load and execute arbitrary code, and has had built-in support via clojure.java.shell for shelling out to run arbitrary things. we really haven't added much that's new to that very open set of doors. I think it's well scoped due to the process separation.

but open to thinking about whether to have knobs to turn on or off capabilities

@U0HG4EHMH You've been able to do this programmatically via clojure.tools.deps.alpha for ages (years, at this point maybe) -- in addition to just generally loading and executing arbitrary code from anywhere at runtime (via core Clojure functionality) for even longer. If anything, this makes it easier to go through the proper channels for code -- Clojars & Maven -- than to workaround it with the less secure options we had previously...

That requires a library to do that though right? I think the worry is that that surface is now inseparable from the language itself

read, eval, slurp, load-file, etc are core Clojure functions...

I ask only whether the bundling or advertisement of a high-risk feature as part of core Clojure puts its reputation at risk. The REPL limit and the separate-process hedge are good but they will not quell a firestorm of fud if someone finds a hole in behavior that you get out-of-the-box by installing Clojure. The CVE will cite Clojure. Might there be a more "open", less-coupled, way to accomplish add-lib that distances core Clojure from the dependency-resolution and acquisition-of-jars tar pits? An opt-in which, even if you opt in, won't be core Clojure's problem if it turns into the next Log4j disaster?

I’m not advocating either way but I do see the point that a CVE in all of that apparatus means there is a CVE in Clojure. It’s true eval and it’s ilk let’s you do zany things, but CVEs generally mean bad things can happen when you didn’t intend to. I think it’s a good point and I’m glad @U0HG4EHMH brought it up.

Having looked at the implementation, I'm much less worried about this new feature than the existing stuff already in the language: • add-lib etc rely on the basis • the basis only exists in processes run via the Clojure CLI -- not via java • the way the dependencies are analyzed/fetched is via shelling out to the clojure command, using the :deps alias So if you build uberjars and run them via java then none of this new stuff will be available inside your running program. If you don't have the clojure CLI installed on a target system, you can't use these new clojure.repl.deps functions

> If you don't have the clojure CLI installed on a target system, you can't use these new clojure.repl.deps functions > Except CLI Tools is bundled with Clojure. Is it even possible to run Clojure without having CLI Tools installed? (Genuinely don't know the answer because I've never thought to try, but only raising the question rhetorically either way.) If I understand correctly?, phill's concern is not about actual risk being assumed. Rather the concern is that if CLI Tools inherits some major exploit through Maven, Clojure-the-language would also be labeled as exploitable because of the bundling.

I'd expect most people do not have the Clojure CLI installed in production: they build (uber) JARs and deploy those and run them via just java.

The Clojure CLI is one way to run Clojure, but not the only way.

Not trying to argue. Just raising the following pseudo-philosophical question in regards to wondering how it affects the concern of Clojure possibly getting labeled with CVEs from CLI Tools: > they build (uber) JARs and deploy those and run them via just java. > Is that even "running Clojure" anymore? In some sense, you're just running Java at that point.

Well, that's how most production systems run I think...

According to the 2022 survey results: 52% use java and a JAR. 25% use lein run (surprised it's that high). ~24% use clojure. So that's higher than I expected...

Maybe concern about this potential vector will drive more folks to use the java startup approach? But let's consider the "CVE in a Maven artifact" scenario: if your code is using that artifact, it's vulnerable no matter how it starts up -- because you have to load it somehow in order to use it. So when do you load it? Either at build time and bundle it into your app, or (currently) at startup time via clojure which fetches it and loads it before your code runs. This new machinery potentially allows you to load it after startup. Either way (well, in all three ways), you're still loading and running that CVE. For folks who worry about network connectivity etc, they're going to want as few network ops as possible to get their code running -- which is why most folks prefer the build-time bundling (in addition to those three ways to run Clojure in production, there a bunch of other options that did not involve using clojure-- it was multiple choice).

So the "real concern" here I think, that @U0HG4EHMH is expressing is that your code might load unknown/unexpected CVEs at runtime through this machinery? So your code would need to be loading "arbitrary" dependencies at runtime -- either through code you've deliberately written to load new dependencies based on... what? User input? ...or through arbitrary code injected into your app somehow (again, where would it come from?). In the latter case -- arbitrary code injection -- Clojure already provides direct filesystem I/O (and HTTP access), the ability to run shell processes, and full code evaluation and that's likely to be a pretty easy way for hackers to achieve their objective, without needing to rely on fetching and loading some arbitrary library from Maven or Clojars.

> So the "real concern" here I think, that @U0HG4EHMH is expressing is that your code might load unknown/unexpected CVEs at runtime through this machinery? > I hope @U0HG4EHMH will return to clarify the issue under discussion, because I read a completely different concern than that. My interpretation is that @U0HG4EHMH worries that if Clojure CLI Tools inherits a bunch of CVEs from its interactions with Maven, then Clojure itself be inadvertently listed as vulnerable on public CVE lists because CLI Tools happens to be bundled with it.

except ... it's not bundled with it

"Clojure CLI Tools inherits a bunch of CVEs from its interactions with Maven" -- that's like saying Gradle is inherently unsafe because it interacts with Maven... which doesn't make any sense to me...

Clojure (the language) is a jar that you put on the classpath of a program run in the Java runtime. The Clojure CLI is a separate tool you can optionally install on a machine that provides a CLI for downloading/managing dependencies, creating classpaths, and running Clojure programs

Ahh. Good point. If I understood the OP correctly, then I think that should solve the concern outright without doing anything.

Clojure itself does not depend on the CLI, or Maven, or dynamic library loading either as a published dependency or as a runtime dependency

the end

This is a piece of functionality I will not be telling our security engineering team about. They'd have a conniption and need a lot of explanation, and proof that our production runtime (`java -jar uberjar.jar`) isn't able to load dependencies dynamically (it doesn't matter what you can currently technically achieve). That conversation would be much easier if the docstring for add-lib(s) explains the circumstances in which it's available, and those in which it isn't.

I urge an overt opt-in step, like "clojure --with-add-lib-from-tools", or better still, limit add-lib to jars you already have on your disk, because the recipe for reputational damage is pretty short: Step 1: "Install Clojure" (https://clojure.org/guides/install_clojure). Step 2: Something happens that someone thinks is unexpected or bad, which is not absurdly unlikely in the realm of finding and fetching jars from Maven repos on the internet. Step 3: The CVE says "Clojure" because that's what you installed. FUD riots trample the nuances. P.S. Can you rebind repl?

The first two steps already existed in the Clojure CLI and every user of Maven, Gradle, sbt etc. This is just FUD.

I'm stepping out of this thread. @U0HG4EHMH You are not addressing any of the specifics that you've been asked about how you imagine a real-world exploit to actually happen here, with this new feature, that can't already happen with a Clojure program.

i assumed that the feature brought in more jars like maven libraries to the clojure.jar and thought it was a good point at first. Seeing that that is not the case I’m not worried and quite excited to try the new functionality. I think i gave the FUD more life than it should have an i apologize for that.

s/def is a macro by itself - you can't pass it a symbol and have it resolved that way. You have to compose a keyword in your macro itself, and then insert it into the (s/def ...) form so it actually becomes (s/def :msmve.core/... ...) instead of (s/def tke__1 ...). Just in case - if you ever feel like you need to create specs in run time or simply have to apply a lot of logic during spec creation, it might be better to use some other spec'ing library, e.g. Malli.

BTW, it's better to create shorter questions and provide more details in a thread. This way, a question doesn't span a few pages on everyone's screen and lets people see whether they can answer it immediately.

Oh, and just a tiny thing - inserting manual line breaks to keep lines short is not necessary. :) It might be your editor doing that, of course, but in general it's not necessary here at all because everyone can adjust the width of the chat area in Slack to their preference.

So to help out a bit more, you can do this by lifting the creation of the keyword to outside of your syntax quote.

(let [tke (keyword "msmve.core" (str term "-enum"))]
  `(s/def ~tke ~heads))

@U2FRKM4TW thanks for the meta-advice, too 🙂 I find these web-app embedded editors maddening because my fingers automatically type Emacs keybindings and they wreak havoc, so I pre-edit and then paste. I’ll not shorten lines and I will shorten questions in the future!

> inserting manual line breaks to keep lines short ... in general it's not necessary here at all because everyone can adjust the width of the chat area in Slack to their preference. > Very short line lengths, when it's reasonable to do so, can be extremely helpful for people on cell phones, where line wrapping can often make large code blocks almost unreadable. (I am reading right now on a Fold 3, and even when opened to the larger tablet-like screen, I still see too much wrapping in the code above.)

> wrapping can often make large code blocks almost unreadable I wasn't talking about code. And I'd argue that inserting extra line breaks to make some code more readable for mobile users will also make it less readable for desktop/laptop users. Otherwise such code should've been formatted that way from the get go.

> I wasn't talking about code. > Oh, then yeah. 💯 agreed. > inserting extra line breaks to make some code more readable for mobile users will also make it less readable for desktop/laptop users. > Definitely depends on the circumstance. That falls under my "when it's reasonable to do so" qualification, where the two main factors to "reasonable" are 1. how much code would need to be edited just for Slack and 2. how ridiculous would it make that particular code sample look with everything slammed over to the left using extra lines.

not sure if this is the correct method (but feels pretty solid) but the docstring tells you how to achieve this:

clojure.java.shell/sh
([& args])
  Passes the given strings to Runtime.exec() to launch a sub-process.

  Options are

  :in      may be given followed by any legal input source for
           , e.g. InputStream, Reader, File, byte[],
           or String, to be fed to the sub-process's stdin.

Like this? (sh "clojure" "-M:my-fn" :in "data.json"). Not working for me.

clojure.java.shell is misnamed

No shell involved at all

And pipes and redirects like that are shell features

What’s a good way to test that then?

You can write the contents of data.json to the stdin of the process, which is what is being suggested up above

You need to make an inputsream and pass it in as :in

When you pass a string the string itself is passed as input, not the file named by the string

user=> (with-open [in ( "input.txt")]
         (clojure.java.shell/sh "cat" :in in))
{:exit 0, :out "hello from clojure java shell\n", :err ""}

(the with-open is not needed. i was checking with something else at first)

user=> (clojure.java.shell/sh "cat"
                              :in ( "input.txt"))
{:exit 0, :out "hello from clojure java shell\n", :err ""}

I generally prefer using ProcessBuilder directly, you can get an output steam that connects to the processes stdin and write whatever you want to

It

And in Clojure 1.12.0 Alpha 2, there's clojure.java.process which wraps ProcessBuilder 🙂

https://clojure.github.io/clojure/branch-master/clojure.java.process-api.html

This is what I have in my -main fn, but I can’t get any of the suggestions to work

(defn -main [& input-data]
  (println  (or input-data (-> *in* slurp (str/split #"\n")))))

Output is different when called from the shell vs the options above

Different how?

(~/clojure)-(!2011)-> clojure -M:my-fn < deps.edn
[{:deps {org.clojure/clojure {:mvn/version "1.12.0-alpha2"}}  :aliases  {   :my-fn {:main-opts ["-m" "diego"]}   :hiccup {:extra-deps {hiccup/hiccup {:mvn/version "RELEASE"}}}   :build {:deps {io.github.clojure/tools.build
{:git/tag "v0.9.4" :git/sha "76b78fe"}}           :ns-default build}   }}]

Fri Apr 14 17:16:31
(~/clojure)-(!2012)-> clj
Clojure 1.12.0-alpha2
user=> (require '[clojure.java.process :as p])
nil
user=> (p/exec {:in (p/from-file "deps.edn")} "clojure" "-M:my-fn")
"[{:deps {org.clojure/clojure {:mvn/version \"1.12.0-alpha2\"}}  :aliases  {   :my-fn {:main-opts [\"-m\" \"diego\"]}   :hiccup {:extra-deps {hiccup/hiccup {:mvn/version \"RELEASE\"}}}   :build {:deps {io.github.clojure/tools.build
         {:git/tag \"v0.9.4\" :git/sha \"76b78fe\"}}           :ns-default build}   }}]"
user=>

That's with

(~/clojure)-(!2013)-> cat src/diego.clj
(ns diego
  (:require [clojure.string :as str]))

(defn -main [& input-data]
  (println  (or input-data (-> *in* slurp (str/split #"\n")))))

yes, cat works fine but I am using < (as in clojure -M:my-fn < data.json ) and the behavior is different depending on the suggestion.

ended up with this (sh "bash" "-c" "clojure -M:my-fn < data.json")

Huh? I only used cat to show what's in src/diego.clj

What I'm saying is that clojure -M:my-fn < deps.edn and (p/exec {:in (p/from-file "deps.edn")} "clojure" "-M:my-fn") are equivalent -- but I'm not sure what you're saying is different?

Oh I see @U04V70XH6, missed one of your messages. Thanks, I’ll keep that in mind for the future. Though I think I will stay with the bash call as it better represents the use case that I am trying to test.

@U7AMPCPU2 Well, @U11BV7MTK’s suggestion seems to work too (with Clojure 1.11):

(~/clojure)-(!2006)-> clj -M:1.11
Clojure 1.11.1
user=> (require '[clojure.java.shell :as sh] '[ :as io])
nil
user=> (:out (sh/sh "clojure" "-M:my-fn" :in (io/file "deps.edn")))
"[{:deps {org.clojure/clojure {:mvn/version \"1.12.0-alpha2\"}}  :aliases  {   :my-fn {:main-opts [\"-m\" \"diego\"]}   :hiccup {:extra-deps {hiccup/hiccup {:mvn/version \"RELEASE\"}}}   :build {:deps {io.github.clojure/tools.build
         {:git/tag \"v0.9.4\" :git/sha \"76b78fe\"}}           :ns-default build}   }}]\n"
user=>

You are right, it does, must have made a mistake.