Fork me on GitHub
#clojure
<
2021-10-21
>
quoll02:10:50

Hi everyone. I have a question on macros: I have a predefined list of items. I have a macro that takes a variadic argument. Is there any way to spread my predefined list into the macro? e.g.

=> (defmacro v [& args] `(vector ~@args))
=> (v 1 2 3)
[1 2 3]
=> (def a '(4 5 6))
The value of a is constant, and def’ed, as above. Is there any way to use the macro v with the contents of a?

quoll02:10:35

(note: I cannot redefine the macro v)

dpsutton02:10:15

you want to be able to call (v a) ?

quoll02:10:08

I’d like to do (v ~@a), but of course, that doesn’t work

phronmophobic02:10:24

as long as a is not a local binding, you could do something like:

(defmacro vapply [macro xs]
  `(~macro ~@(eval xs)))

> (vapply v a)
[4 5 6]

(let [b [1 2 3]]
  (vapply v b))
;; Unexpected error (UnsupportedOperationException) macroexpanding vapply 
;; Can't eval locals

quoll02:10:55

That’s ideal! Thank you!

phronmophobic02:10:37

another alternative is (eval (apply list 'v a ))

hiredman02:10:21

You don't need eval at all

hiredman02:10:54

The above macro causes xs to be evaled, then the result of that to be evaled

hiredman02:10:21

You can just use apply, inside an apple macro

hiredman02:10:59

(defmacro mapply [m args] (apply (resolve m) nil nil args))

hiredman03:10:56

Depending you may have to replace those nils with actually values but I forget the order of &form and &env, env you can pass through, form http://will.be goofy

hiredman03:10:08

But just don't do that

hiredman03:10:51

applying macros isn't the answer, the answer is stepping back and avoiding whatever it is that is making you want to apply macros

quoll03:10:30

Well, I have a macro that I don’t control, and I have a seq of predefined data that I want to give to it, and I want to use that seq elsewhere in the code as well. Until now, that very large seq is copy/pasted between the 2 areas. I really want to let the macro use it

quoll03:10:14

If I had control of the macro, it would be very easy. Just redefine it to use a seq. It actually turns the arguments into a seq, so it’s trivial.

quoll03:10:09

At that point, I could write a macro just like the one that current exists (taking a variadic argument), collect the args, and pass them into my new, better-designed macro.

quoll03:10:17

But… That’s not what I have to work with

👍 2
hiredman03:10:23

Sounds horrific, a macro that takes a large literal coll

hiredman03:10:09

I would double check the library docs or whatever to make sure that is actually the case

quoll03:10:51

It’s a DSL. The idea is to call it with something like:

(defprog [a b c]
  operation1
  operation2
  (op3 a)
  (op4 b c))
I’m trying to get fancy and construct programs to run in it

hiredman03:10:37

My experience with that sort of thing is largely around core.logic, which ends up having function versions of the operations you need for that sort of thing, just not well documented

hiredman03:10:42

I usually have to dig through the source to find them

didibus04:10:25

Just stumbled upon an old Clojure mailing list thread: https://groups.google.com/g/clojure/c/o43RyH_ua8I

(defmacro let-map
  "Equivalent of (let [a 5 b (+ a 5)] {:a a :b b})."
  [kvs]
  (let [keys (keys (apply hash-map kvs))
        keyword-symbols (mapcat #(vector (keyword (str %)) %) keys)]
    `(let [~@kvs]
       (hash-map ~@keyword-symbols))))

user=> (let-map [a 5 b (inc a) c [a b] d {:foo c :bar (* a b)}])
{:a 5, :c [5 6], :b 6, :d {:foo [5 6], :bar 30}}
That's awesome!

😮 1
😎 1
borkdude08:10:58

@didibus I have this one in a work project:

(defmacro ->map
    "(->map a b) ;;=> {:a a :b b}"
    [& symbols]
    (assert (every? symbol? symbols))
    `(hash-map
      ~@(interleave (map keyword symbols) symbols)))

pinkfrog11:10:44

Hi. Is there any framework (java also considered) that supports this functionality: 1. one master and multiple slave instances 2. the master sends a task to a queue, and only one slave instance will fetch task from the queue, process it 3. the master waits and read the result of that task

pinkfrog11:10:08

An example in Python would be: Celery

dominicm11:10:17

#proletarian is very good.

👌 1
msolli12:10:14

@UGC0NEP4Y I’m the author of Proletarian (https://github.com/msolli/proletarian). It’s a Postgres-backed job queuing and worker system for Clojure. Please share your use case in #proletarian, and let’s see if it can be of use to you.

thom12:10:44

Nice, any plans to support listen/notify vs polling?

p-himik12:10:49

FWIW I was actually considering proletarian at some point, but decided against it given that I already have a listen/notify implementation of my own.

pinkfrog12:10:01

> FWIW I was actually considering proletarian at some point, but decided against it given that I already have a listen/notify implementation of my own. Thinking on this, I feel Redis as a middleware might achieve this too.

p-himik13:10:48

Redis is just a transport. It's neither framework not a library. It's definitely possible to use it to achieve what you want, but it would also be possible with many other transports.

pinkfrog13:10:33

Sure. Mention because it offers notification and pubsub.

msolli19:10:49

@UTF99QP7V Re. listen/notify: No plans yet. Polling works great, it’s simple and performant. I’d love to look into it if there’s interest and valid use cases for it.

msolli19:10:18

@U2FRKM4TW I’m interested to know more about your implementation. Do you have anything to share?

p-himik19:10:58

Alas, not really - all project-specific stuff in a closed-source project.

msolli19:10:46

Fair enough. Would you mind if I pinged you at some time in the future to pick your brain about this if/when I venture into implementing listen/notify in Proletarian?

p-himik19:10:21

I doubt I would be the right person but sure, I don't mind. :) On the topic of polling vs listen/notify - if I'm reading the code of Proletarian correctly, you're essentially busy-looping with the polling. There are no configurable delays of any kind. It's a problem in an of itself - apart from an unnecessary load (albeit, probably a low one), it will pollute PostreSQL logs if you have configured it to log queries.

msolli20:10:58

No, it’s not quite how it works. While there are jobs to be done, yes, it’s going to loop around and get the next item. But when there are no jobs, the Queue Worker (backed by a ScheduledExecutorService) is going to wait the configured amount of milliseconds (`:proletarian/polling-interval-ms`) before polling.

p-himik21:10:42

Ah, right, I see now, thanks for the explanation. Although it will still pollute the logs.

msolli07:10:07

Sure. Though I wouldn’t run Postgres with log_statement = 'all'i production, so the query statements wouldn’t be logged at all.

Proctor14:10:38

Has anybody done anything with pulling from OSX Keychain in Clojure(Script)? Thinking about how to have secrets in local dev config but not stored as plain text, or plain text in environment variables, fetched using something like an edn reader macro… e.g.

{:username 'my-user'
 :password #keychain 'secret-name'
...
}

dpsutton14:10:57

i've always treated dev credentials as not super secret. The dev setup should be recreateable and throw awayable if necessary

Proctor14:10:34

in this case, it is things like AD credentials

Proctor14:10:18

e.g. AD -> Vault to get other secrets

p-himik14:10:44

If there are CLI tools, I would definitely just shell out to them.

p-himik14:10:11

In ClojureScript, if it's for browser, you can do it during build time.

Proctor15:10:59

mostly just repl env stuff, when we are trying to hit dev environment when not using fakes

Proctor15:10:59

so was looking at calling out to shell with something like:

(-> (sh "security" "find-generic-password" "-s" "AD Creds" "-w") :out)
but was checking if any recommendations… 😉

Proctor15:10:16

(or here there be dragons….)

p-himik15:10:39

Probably dragons indeed. As dpsutton mentioned, usually treating dev credentials (assuming you don't have your dev setup available from the outside world) as something not that secret is fine, and using plain text files with dev credentials is an approach I've seen incomparably more times than any other.

Proctor15:10:30

sadly they are dev credentials, and not local credentials at this point… plus it is Active Directory credentials on top of that

p-himik15:10:28

Seems like there's a slight mixup of terms. At least in my experience "dev" in "dev credentials" means "used during development". Which usually implies that a single developer has everything that's needed to work on the software in question on their PC, thus automatically making dev credentials to also be local ones. But given that you use AD + Vault for that, I think in your case "dev" means something different or that implication of a single developer having all the needed pieces locally does not hold. Either way, that's just a rambling attempt at clarification. Regardless of the specifics, I'm absolutely sure that shelling out is the simplest and the easiest way to go here.

Proctor15:10:49

yeah, sorry… they are the Deployed Shared Development Environment credentials, not local dev credentials… fell into the terms used in the job, which I do think are goofy and confusing

👍 1
Jakub Holý (HolyJak)15:10:01

Is it possible to extend a protocol to another protocol, i.e. say that everything that satisfies LowerLevelProtocol also satisfies HigherLevelProtocol (and provide its implementation in terms of the lower-level protocol functions)? I guess not?

Alex Miller (Clojure team)15:10:04

there are a couple of techniques though

Alex Miller (Clojure team)15:10:08

one path people sometimes use is to extend using the generated interface, probably the most common, but I'm not in favor of it because you will miss extensions via metadata etc

Alex Miller (Clojure team)15:10:32

another is to have a default extension in Object that determines whether the concrete type you have supports the protocol and then installs the extension dynamically (to the concrete type), after which it can route directly

Jakub Holý (HolyJak)08:10:00

Could you be so kind and explain that a little more? 🙏

Alex Miller (Clojure team)12:10:34

There's a section on this technique in Clojure Applied

🔍 1
Jakub Holý (HolyJak)12:10:35

Found it, Extending Protocols to Protocols at p.22 Thanks!

Alex Miller (Clojure team)15:10:04

and another very advanced technique is to directly manipulate the internal protocol map

borkdude15:10:31

is the latter "supported" or relying on impl details?

dpsutton15:10:22

another way is if the protocol isn't used directly but rather there's a function that itself dispatches to the protocol you can do whatever you want before invoking the protocol. Requires more of a design before you begin though

Alex Miller (Clojure team)15:10:12

we've taken to treating this as a best practice and you can see this prevalent in spec for example

dpsutton15:10:09

yeah i was digging through the clojure source code for a good example

indy19:10:45

Would be nice to see some examples, please post them if you find any.

didibus00:10:55

You see how the protocol defines the methods name with a * at the end

didibus00:10:44

And then here: https://github.com/clojure/spec.alpha/blob/master/src/main/clojure/clojure/spec/alpha.clj#L167-L184 You can see that there are functions defined with the same name but without the * at the end, and they just call the protocol methods.

Alex Miller (Clojure team)01:10:55

in spec 2, we're using those outer functions to do more special handling

indy15:10:19

Thanks, still trying to wrap my head around how this technique is a way of extending ProtocolB from ProtocolA (the original question, if I understood it correctly). I’ve faced this problem before and what I’ve wanted is a way to use all the methods in say ProtocolA in another ProtocolB with B having some more methods.

if (satisfies? ProtocolA x) => true
then (satisfies? ProtocolB x) => true
(I understand this is stepping into OOPs territory)

Alex Miller (Clojure team)15:10:23

it relies on impl details, but Clojure leaves those details accessible for those that understand the tradeoffs

emccue15:10:38

You can also have an into-thing-a on the protocol, depending on why you are doing it

emccue15:10:20

Like, if you have protocols A and B, and you want all Bs to support A's methods, you can always have

(defprotocol A
  (do-thing []))
(defprotocol B
  (do-other [])
  (as-a []))
And then your B protocol requires that implementors have some way to produce an implementation of A

p-himik15:10:52

This whole discussion sounds like a perfect idea for a blog post, wink-wink.

emccue15:10:53

Which for things that implement A and B can be just an identity function

borkdude15:10:24

Or use multi-methods + derive ;)

emccue15:10:25

It has the pro of encoding the relationship between the protocols explicitly, with the con of requiring explicit conversions by convention (that might be missed since the basic code might work regardless)

emccue15:10:50

Not to derail, but does anyone have any experience using ImageIO, maybe with twelvemonkeys, in production?

seancorfield16:10:55

Yes, we use this combination for all our image manipulation.

seancorfield16:10:38

We use it to auto-correct rotation when a member uploads their photos. We also do cropping and resizing with it.

seancorfield16:10:10

(for privacy reasons, we strip EXIF after initial processing)

emccue16:10:45

Have you seen a need to put this image processing on its own separate server?

emccue16:10:07

That is our arrangement right now, but it was kinda done preemptively

emccue16:10:57

(ffmpeg on a lambda, basically)

emccue17:10:11

Also I think you must have ran into this exact issue - will imageio be enough/have a way to interpret and later strip the exif data?

seancorfield17:10:53

Yes to the latter Q -- that's exactly what we do. As to a separate server, no, we do a bunch of ImageIO/TwelveMonkeys stuff inline in a couple of our processes. We do have a separate "photo queue" process that does more of the heavy lifting, but the reason for that is that it also uses external AI processes to analyze images and so we have to throttle the calls due to rate limiting thresholds on those services.

seancorfield17:10:51

Oh, on the EXIF stuff, we recently added com.drewnoakes/metadata-extractor {:mvn/version "2.16.0"} since ImageIO didn't give us everything or didn't work for all image types. I'd have to dig in JIRA to see exactly what the root issue was there.

Jakub Holý (HolyJak)08:10:43

I believe it is @U07QKGF9P who mentions somewhere in https://clojurescriptpodcast.com/ that image processing should run in its own VM because there is no way to make the processing C libraries bug free and thus you open yourself to be hacked by a well-crafted image. So I would run anything like ffmpeg in a lambda. ImageIO is Java so it is likely far safer than C but I am not sure it is 100% safe.

emccue15:10:56

We have some code that relies on ffmpeg for doing image resizing and it's missing the rotation metadata on some jpegs, which causes wierd rotated preview images

emccue15:10:43

So I'm looking into alternatives

chrisn16:10:33

There are a few opencv wrappers for clojure that may be useful.

winsome16:10:02

I'm trying to figure out a bug with a cli4clj application, which I understand is basically a wrapper around clojure.main/repl. I have an input that looks number-like (ex. 1234/abc), but it uses clojure's read and that interprets the input as an invalid number. Is there a simple way to hack the reader to treat inputs as strings instead?

p-himik16:10:22

Have you tried passing '"1234/abc"'?

winsome16:10:16

That works just fine, and I have other workarounds too, but I wanted to know if there was a simple approach I could take to avoid that

noisesmith16:10:08

not with the clojure reader - namespaces can't start with numbers in symbol literals (though the symbol function will happily turn any string into a symbol)

dpsutton17:10:52

if the reader interpreted input as strings, it would just return the input, and strings eval to themselves to that repl would be identical to identity right?

winsome17:10:56

Well, in a cli application you'd hand those strings off to the eval function to be processed anyways, and I don't really care about data types there, I just need strings

dpsutton17:10:11

i don't follow. if you eval the string "1234/abc" you're gonna get either the exact same string back or the same error as above

hiredman17:10:48

yeah, there seems to be a fundamental misunderstanding about what read and eval do

hiredman17:10:13

eval doesn't operate on strings, it operates on datastructures

winsome17:10:18

I have a cli prompt that goes something like this:

# mycli > arg1 my-id/here
# (output)
# mycli > arg2 1234/abc
# (<crashes>)
I'm just using the inputs as strings to figure out which commands to run, all the inputs are treated as strings anyways. So I don't actually want "reading" to happen. I've got my own custom eval - that's what the cli4clj framework supplies.

winsome17:10:33

But unfortunately it doesn't allow you to plug the "read", just the "eval"

winsome17:10:59

The "read" unfortunately crashes on valid program inputs because not all valid program inputs are valid edn. I'll file an issue to allow plugging in your own reader.

dpsutton17:10:32

i'd question why you are using the reader at all here. Why are you using clojure.main/repl ?

winsome17:10:15

I'm just investigating a bug report on a legacy tool we have that crashes on unusual inputs. This application was made with cli4clj, which wraps clojure.main/repl.

Ivan Fedorov19:10:03

Hey, does Clojure community has any preferences regarding the best approach to sessions for web? I usually went with a stored session + session-token + cookie, and now looking towards a JWT stored in the same cookie If this was discussed already – a pointer would be enough, thanks!

ghadi19:10:36

this is a great article -- not clojure-specific

p-himik19:10:42

Can't speak for the whole community but it has been argued time and time again that JWTs are largely misused, especially when it comes to scenarios that can be solved with the solution you're already using. I haven't seen the article above (and thanks for the link, I'll check it out), but here's another one, JWT-specific: http://cryto.net/~joepie91/blog/2016/06/13/stop-using-jwt-for-sessions/

❤️ 1
☝️ 1
ghadi19:10:14

personally I'd love to see a good clojure library for macaroons

1
ghadi19:10:51

(speaking idly into the ether...)

Ivan Fedorov20:10:49

@U2FRKM4TW not following the critique there. I think JWT that contains only a {user-id: 1} and is stored in a secure cookie is alright, isn’t it? But yeah if we want an individual token invalidation we’ll have to store them somewhere. UPD: Ok, I got it, yeah, I’m reinventing the session. Thanks!

👍 1
Ivan Fedorov20:10:08

Now reading the second part