Fork me on GitHub
#clojure
<
2019-07-11
>
hlolli03:07:20

is there any way of loading clojure code on runtime that is outside of the classpath? I guess this is security issue, but I was hoping to add some preloaded code to my application in similar fashion to the #'user namespace.

dpsutton03:07:05

I think Stu Does this in the day of datomic repo

andy.fingerhut03:07:11

Calling load-file can do it, as well as eval

hlolli03:07:47

I'll try further, but I'm trying to call functions from the classpath namespaces. And from load-file, I can't call the symbols defined in the file. Maybe I'm too quick asking for help.

andy.fingerhut03:07:12

By classpath namespaces, you mean Clojure namespaces whose source files are on the classpath?

hlolli03:07:24

yes, specifially a macro

andy.fingerhut03:07:52

Macros are called and executed during eval of the data structures representing the code, so not sure whether that will prevent you from doing what you wish.

andy.fingerhut03:07:47

You are calling load-file on a file containing Clojure source code, and you say you "can't call the symbols defined in the file" -- from where can't you call the symbols defined in the file?

hlolli03:07:58

yes, this is a file I'm trying to load outside of the classpath

(ns panaeolus.csound.instruments.spacedrone
  (:require [panaeolus.csound.macros :as c]))
(c/definst spacedrone
  :orc-internal-filepath
  "instruments/drone/spacedrone.orc"
  :instr-form  [{:name :dur :default 4}
                {:name :nn :default 48}
... etc....
` (load-file "/full/path/to/spacedrone") or read-string + eval from the namespace I call load-file, I can't see the symbol spacedrone defined. But maybe there's error with the macro not related to load-file.

hlolli03:07:42

sorry nor the namespace panaeolus.csound.instruments.spacedrone

hlolli03:07:50

the symbol spacedrone would have to be required ofc

andy.fingerhut03:07:58

from which namespace do you call load-file?

hlolli03:07:13

in thie case panaeolus.all, it could be any namespace

andy.fingerhut03:07:20

Have you tried removing most of the code in namespace p.c.i.spacedrone, leaving only say one defn of a simple function, and see if that exists after doing load-file?

hlolli03:07:50

no, not with namespace, but a single file with single function works fine

andy.fingerhut03:07:26

just for testing purposes, to see if it works as you hope, maybe try adding an ns form to the file you are calling load-file on.

hlolli03:07:44

yup, will do. Sounds like this should be possible 🙂

hlolli03:07:03

yup you're right, this works with ns plus single function

hlolli03:07:22

I bet there's an error with the macro, so I'll work form it there

andy.fingerhut03:07:24

yep. For me as well with a simple try from the REPL

dacopare10:07:02

Is there a place one can find open-source Clojure projects that would like contributions?

jumar10:07:26

You can also have a look at the clj-commons github org: https://github.com/clj-commons

dacopare10:07:21

I've got some spare time

vlaaad11:07:27

(-> x
    (blah :y)
    (cond-> (pred? x) (blop :z))
    (bleep))
vs
(cond-> x
        :always (blah :y)
        (pred? x) (blop :z)
        :always (bleep))
what do you think?

Stefan12:07:58

I think as a reader the second form is easier to understand…

👍 8
joelsanchez12:07:58

I think the second one is redundant and I find the first one easier to read

✔️ 20
joelsanchez12:07:50

plus you can keep adding functions like bleep without needing :always

borkdude14:07:58

Interesting.

(ns private.tmp.import
  (:import (java.util.regex.Pattern))) ;; not allowed by core.specs

(ns private.tmp.import
  (:import (java.util regex.Pattern))) ;; allowed by core specs

👀 4
jeroenvandijk15:07:47

@U04V15CAJ maybe even more surprising is that you don't even need the import if you don't need the alias Pattern and use it with package prefix.. I've learnt this recently after asking a similar question here in slack. Not sure if it has always been there

borkdude15:07:42

always been like that

borkdude15:07:53

I think it's the same in Java: importing classes don't change anything to the classpath, it's just a syntactic convenience

borkdude15:07:27

this makes it a little harder on the linter: you don't know if something is a class or a namespace (that was or wasn't required)

jeroenvandijk16:07:59

I guess the linter isn't on the same classpath? if so, you could try to resolve

borkdude16:07:19

it's only static analysis

jeroenvandijk16:07:59

unless you have better options. A good heuristic would be camelcase vs non-camelcase <=> java vs non-java

bronsa14:07:58

ew

☝️ 4
bronsa14:07:29

and it's no more

jonpither14:07:37

looking for an example resource of how to lazily stream a resultset from next.jdbc...

borkdude14:07:58

specifically next.jdbc or is current.jdbc also fine?

Charles Fourdrignier15:07:41

I came here with very "Clojure way of doing things" question. Clojure is great for data pipelines. I would like to prove this to my coworkers for maybe one day, introduce Clojure has an option for some of our tasks. But I miss some full pratical example. The use case should be the following : - Receive a data file (e.g. CSV) - Parse it - Validate datas - Insert valid ones into a database, Write others into a file (with some explain-data) What's come to mind could be some sort of "Railway" (with https://github.com/funcool/cats ?). But I'm not sure that's the more "Clojurey" solution. From this talk (https://www.youtube.com/watch?v=oOON--g1PyU), I know halt-when is good to stop a transduction operation. Which is good for "one shot" loading like the one expose by Stuart Halloway, but not for my imaginary (but realistic) use case. Do you have any recommended solution (or reading) ?

waffletower16:07:10

Before you dive into Cats, I recommend reading this blog about it: https://medium.com/@functionalhuman/functional-programing-with-cats-105d666b260e

waffletower16:07:57

Do note that the examples at the end utilize clojure.spec.alpha2 which is still in prerelease.

Charles Fourdrignier15:07:12

Just discover your answer. I'll have a look, thanks.

borkdude15:07:03

Hmm. In Clojure when you write:

(ns foo)
bar/baz
bar/baz could in theory be a public field in a class bar right? so it's hard to say this is an unresolved symbol by linting statically

bronsa15:07:15

there's no ambiguity between Klass/field and ns/var

borkdude15:07:07

in CLJS-land there are also things like goog/isFunction

borkdude15:07:18

that are just "there" without requiring

bronsa15:07:10

yes, w/o keeping a runtime mapping of the available classes it's not easy to distinguish the two

borkdude15:07:25

currently clj-kondo doesn't report anything with a namespace as unresolved, but thinking about this more, I think it's the correct behavior because of the above

👍 4
borkdude15:07:44

(given that underreporting is better than overreporting)

john15:07:49

@charles.fourdrignier You could maybe draw some inspiration from Grammarly's experience optimizing some data munging with transducers https://tech.grammarly.com/blog/building-etl-pipelines-with-clojure

Charles Fourdrignier15:07:27

Thanks ! To be honest, I already found it. It's very interesting for the performance point of view. But at the moment my biggest concern is about 1 entry -> 2 outputs, which seems not to be a so common pattern in Clojure. By looking with a fresh eye to the article, maybe introducing a second "out" channel should be a good idea ?

john15:07:16

Yeah, depends on business requirements I think, so you'd have to experiment.

john15:07:34

If data has lots of interdependencies, it maybe harder to parallelize or design transducers around it. But in general, I think forking data out to different places can be an orthogonal concern.

carkh16:07:22

a solution to this problem that i use sometimes is to map over your data, returning command data structures (often just a pair vector) that will then be "interpreted"

carkh16:07:48

easily testable, pure...

carkh16:07:56

sounds like a lot as i read it, but it's actually pretty simple, and you segregate IO in your "interpreter" function. Interpreter is a big word for just a case statement =)

Charles Fourdrignier18:07:07

Thank you for the ideas.

danielneal07:07:56

Yeah, I use the @U13RRNF2M pattern a fair bit - create a sequence of ops like [{:op/type :write-to-error-log :op/data …} {:op/type :store-in-db :op/data …} ...] so all the clever stuff happens in the generation of the ops sequence, and can check it works without doing any io

borkdude15:07:07

@jonpither If you're also interested in current.jdbc, I asked this question 3 years ago on SO: https://stackoverflow.com/questions/39765943/clojure-java-jdbc-lazy-query

borkdude15:07:28

we used this for a while, but later on we used the transducer/Reducible approach

seancorfield16:07:18

@jonpither I would recommend next.jdbc/plan (or clojure.java.jdbc/reducible-query). There are several DB-specific concerns as well to get true streaming. On some DBs you need the connection to have auto-commit turned off and/or :fetch-size 0 I think.

jonpither16:07:00

@seancorfield do you happen to have an example of next.jdbc/plan being used to produce a lazy seq of results?

jonpither16:07:29

currently working with plan and trying to cajole the reducible into a lazy seq

seancorfield16:07:58

No, you can't produce a lazy sequence from it. Laziness cannot escape the scope of the connection. But "lazy seq" is not the same as "lazily streaming results from the DB".

seancorfield16:07:41

You have to process the whole result set while the connection is still open. With plan (and its forefather reducible-query), you reduce the whole result set, but you can process more rows than will fit in memory by setting the appropriate (DB-specific) set of options. So exactly how you do it, will depend on your database.

jonpither17:07:10

thanks @seancorfield, will ponder

seancorfield17:07:24

And if you follow-up in #sql I'm more likely to see the questions. I'd missed your original question here but happened to see Michiel's response.

jonpither17:07:49

ah ok, sorry, had looked for jdbc @seancorfield

joshkh20:07:22

how might i go about deploying a deps.edn based project to a private s3 repo without using lein?

theeternalpulse20:07:39

since where on deps.edn topic, is there a good set of practices for managing a monorepo with deps. Do monorepos with tools.deps?

andy.fingerhut20:07:49

There was a discussion on tools.deps and monorepos recently on the #tools-deps channel, perhaps yesterday. Should still be in the history on Slack, but if not, https://clojurians.zulipchat.com has it in its unlimited history.

theeternalpulse20:07:22

Ah, I need to start using the search for channels