Fork me on GitHub
#clojure
<
2020-01-28
>
macrobartfast04:01:20

I asked a pedestal/docker question in #pedestal if anyone feels like taking look.

seancorfield04:01:58

@macrobartfast That Pedestal issue talks about ::server/host -- you mentioned ::http/host -- is your host an alias to the same namespace that server is aliased to in that issue?

macrobartfast04:01:56

hmm... let me look.

macrobartfast04:01:22

(slowly grokking what you saying, too)

seancorfield04:01:17

@macrobartfast Looking at the commits around that security issue, it seems like you should just be able to specify :host "0.0.0.0" in the hash map you pass into the Pedestal server at startup. Can you share how you're starting the server?

macrobartfast04:01:54

I start the docker container with docker run -d -p 8080:8080 general-service (general-service is the name I gave my docker image)

macrobartfast04:01:18

the -d just makes it a daemon... I had the same problem with or without it.

seancorfield04:01:34

No, I mean the Clojure code that starts the server.

macrobartfast04:01:14

oh, gotcha... stand by...

seancorfield04:01:45

I've never used Pedestal but all the Clojure servers I've seen get passed an options map when they start, where you can pass :host and that can be "0.0.0.0"

macrobartfast04:01:26

how do you post longer code in here in a socially acceptable way?

macrobartfast04:01:36

i.e. the slack version of pastebin

seancorfield04:01:03

Post it to a service like pastebin and put the link here?

macrobartfast04:01:22

haha ok I had thought there was some slacky way to do it...

macrobartfast04:01:45

so, I call a main function

(defn -main
  "The entry-point for 'lein run'"
  [& args]
  (println "\nCreating your server...")
  (server/start runnable-service))

seancorfield04:01:53

I was only expecting a few lines ๐Ÿ™‚

macrobartfast04:01:04

which of course is calling a whole bunch of other code.

macrobartfast04:01:10

was figuring out how to summarize that.

seancorfield04:01:25

And that server/start -- does it accept other options?

macrobartfast04:01:36

I don't know... lemme look.

macrobartfast04:01:04

and this is helpful to me in experiencing how you would start solving an issue like this, rather than my usual mode of flailing my arms around the web.

macrobartfast04:01:12

I'm reading that now.

seancorfield04:01:09

OK, so what is runnable-server in your code?

seancorfield04:01:30

You got it by calling create-server yes?

macrobartfast04:01:25

haha just catching up with you stand by

seancorfield04:01:49

And you pass a hash map to create-server with options in to describe the server?

macrobartfast04:01:55

(defonce runnable-service (server/create-server service/service))

seancorfield04:01:40

OK... and what is service/service then? A default service description, I assume?

macrobartfast04:01:03

lemme look, stand by...

macrobartfast04:01:41

[general-pedestal-service.service :as service]

seancorfield04:01:25

Is that your code or something in Pedestal? (remember I've never used Pedestal)

macrobartfast04:01:27

which is the other principal file.

macrobartfast04:01:04

I know, which makes this so extraordinary to me... you are walking me through my code, and you don't have it. ;-D

macrobartfast05:01:36

I'm reading service.clj right now, and trying to put it all together. Even if the original issue isn't resolved, I'm learning a lot about how Pedestal works.

seancorfield05:01:41

OK, so in that file, there's a symbol service that is a hash map?

seancorfield05:01:04

So general-pedestal-service.service is Pedestal code, not your code?

seancorfield05:01:17

If so, the fix to all this is to change your defonce:

macrobartfast05:01:22

and yes, a hash map.

macrobartfast05:01:39

it's a def, but yes, probably.

seancorfield05:01:40

(defonce runnable-service (server/create-server (assoc service/service :host "0.0.0.0")))

seancorfield05:01:49

No, your defonce

seancorfield05:01:04

(you don't go changing library code)

macrobartfast05:01:52

haha sounds like a good idea, especially in my case.

seancorfield05:01:59

Since service/service is a "service map" per Pedestal API docs, that's what can have a :host key (per the commit I looked at) and it defaults to "localhost" for security reasons.

macrobartfast05:01:05

ok, modified the defonce.

seancorfield05:01:20

But if you add your :host key, it overrides that default.

macrobartfast05:01:27

ok, very cool.

seancorfield05:01:12

I searched the Pedestal repo for localhost and looked at the commits

macrobartfast05:01:56

was wondering how you found that!

seancorfield05:01:34

My initial search was for :host, followed by localhost in the code, but those returned too many results. I just figured it must be doing something similar to what other Clojure http servers do.

macrobartfast05:01:17

building a new docker container now.

seancorfield05:01:24

The question is: did it fix your problem? ๐Ÿ™‚

macrobartfast05:01:30

we'll find out!

macrobartfast05:01:58

while the container builds, I can say the answer is yes...

macrobartfast05:01:25

as the problem is my unwillingness to walk the code and look a the source. ๐Ÿ˜…

seancorfield05:01:49

Have you watched Debugging With The Scientific Method, the talk by Stu Halloway?

seancorfield05:01:40

You should ๐Ÿ™‚

macrobartfast05:01:45

I'll look it up tonight.

macrobartfast05:01:22

so, the port is still unreachable... but I was rushing.

macrobartfast05:01:46

let me make sure I implemented the possible fix right.

macrobartfast05:01:22

oh, I might have done something silly, stand by, building docker again.

macrobartfast05:01:03

well, hmm.... still not reachable, but we've made great progress, I suspect.

macrobartfast05:01:19

I just need to continue walking the code, probably.

macrobartfast05:01:42

also, they support tomcat and another server in pedestal, maybe I'll try those.

seancorfield05:01:00

You can run it standalone outside of Docker?

seancorfield05:01:35

Another test you can run: build the Docker container and run it, then shell into it and try to telnet to localhost on the port you believe the server is running on -- that would confirm whether your server is really starting up and binding to the right port (on the loopback).

macrobartfast05:01:19

omg, just realized I wasn't rebuilding the uberjar before building the container... ๐Ÿ˜ณ

macrobartfast05:01:42

but, to answer the question, yes, it did work outside of the docker container.

macrobartfast05:01:33

rebuilding the docker image with the fix we came up with in the actual uberjar. unbelievable.

macrobartfast05:01:16

which still doesn't work. I'll try the telnet route now.

seancorfield05:01:54

If you can run it outside Docker, you can test the port access via telnet (both on localhost and on the IP that your machine has to test how the server started up and whether it bound to the right port).

seancorfield05:01:50

That would establish whether the problem is the server code / config or the Docker setup.

seancorfield05:01:07

You have a lot of variables in play the way you are trying to run it right now.

macrobartfast05:01:45

and I realized just now when you meant 'standalone outside docker' you meant the java -jar <standalone> I had thought you meant via the repl in cider... testing the standalone now.

macrobartfast05:01:24

yep, standalone jar works.

seancorfield05:01:53

Via both localhost and the main IP of your machine?

seancorfield05:01:36

Then the next test will be to telnet/ssh into the running Docker container and then try that same telnet test inside the container..

seancorfield05:01:09

(basically trying to go step by step through the system, verifying each piece of infrastructure)

macrobartfast05:01:43

re standalone: only checked localhost, I'll check that, then I'll go the ssh route (just got into the container, turns out it's /bin/sh)

macrobartfast05:01:50

so, works localhost, but not the LAN ip of the machine on my home router...

macrobartfast05:01:00

(the standalone, that is)

macrobartfast05:01:09

my LAN address seems to be 10.0.0.236, and 10.0.0.236:8080 does not work.

seancorfield05:01:15

That the LAN IP doesn't work might be a firewall setting locally so perhaps don't worry about that too much.

macrobartfast05:01:15

and I opened the firewall.

macrobartfast05:01:27

messages crossed.

macrobartfast05:01:53

next step is the ssh into docker route.

macrobartfast05:01:29

so, was able to telnet to the port inside the container (cool trick).... got the blank screen which means the port is open.

macrobartfast05:01:03

and while I'm in there, I see that the jar running.

seancorfield05:01:36

"blank screen" from telnet?

macrobartfast05:01:50

well, ran telnet 0.0.0.0 8080 and enter and the there is no response, which means it's open, according to https://www.opcsupport.com/s/article/How-do-I-ping-and-telnet-to-test-a-tunneller-connection, anyway ;-D

macrobartfast05:01:33

I guess if the port isn't open, you get 'telnet: can't connect to remote host' which I confirmed by trying 8081

seancorfield06:01:01

Right but you can send an HTTP command like GET / (and press enter twice)

macrobartfast06:01:29

haha, well, learning away here... that did produce a response

macrobartfast06:01:40

<h1>Bad Message 400</h1><pre>reason: HTTP/0.9 not supported</pre>Connection closed by foreign host

seancorfield06:01:41

And just to confirm, you are running that telnet command inside the container after shelling into it? Just want to make sure you're not accidentally still testing the standalone JAR running outside Docker.

macrobartfast06:01:05

I am... apparently osx doesn't have telnet, as a side note.

seancorfield06:01:19

OK, so GET / HTTP/1.1 (should prevent the HTTP/0.9 problem)

seancorfield06:01:28

I'm on a Mac and I have telnet

macrobartfast06:01:31

hmmm, not sure why I don't,

andy.fingerhut08:01:09

I have heard that some recent versions of macOS / OSX have ssh/scp/sftp, but not unencrypted telnet/ftp, unless you go out of your way to install them. They were installed by default as of the latest macOS from 2-3 years ago -- I forget the exact version where this changed.

macrobartfast06:01:45

however, inside the container...

GET / HTTP/1.1

HTTP/1.1 400 No Host
Content-Type: text/html;charset=iso-8859-1
Content-Length: 50
Connection: close

<h1>Bad Message 400</h1><pre>reason: No Host</pre>Connection closed by foreign host

macrobartfast06:01:59

iiiinteresting.

seancorfield06:01:14

Ah, OK, so it needs a Host: header before it will accept a connection.

seancorfield06:01:45

That could be how Pedestal is configured for security?

macrobartfast06:01:14

aha ok... although the jar works outside the container.

seancorfield06:01:14

GET / HTTP/1.1
Host: 
maybe?

macrobartfast06:01:39

oh, I understand. this is cool, btw... always wanted to telnet.

seancorfield06:01:43

If you telnet to the JAR running outside the container? You can issue that GET and it works?

seancorfield06:01:16

OK, so here's a thought: when you try to run the Docker container, do you already have something on your Mac running on that port (8080), such as the standalone JAR?

seancorfield06:01:29

You can't have two processes on the same port.

macrobartfast06:01:29

and now I understand why you press enter twice in telnet (so you can create multiline commands)

macrobartfast06:01:48

let me telnet to the jar, stand by.

macrobartfast06:01:58

regarding two things running: oops, I had left the jar running, but that was only for the last couple of minutes.

macrobartfast06:01:28

I'll now attempt to telnet to the jar, after stopping the docker container, for good measure.

seancorfield06:01:30

And more importantly, make sure the JAR is not running when you start the Docker container ๐Ÿ™‚

macrobartfast06:01:04

gotcha (I promise that hasn't been running this whole time)

seancorfield06:01:15

unless you map the Docker container to a different port (e.g., 8888)... can't remember which way round that goes 8888:8080 or 8080:8888

macrobartfast06:01:30

Mojave doesn't have telnet; installing via brew

macrobartfast06:01:55

I think the first one is the host machine.

kulminaator06:01:19

Are you sure this is the best channel for all of this ?

macrobartfast06:01:09

@kulminaator fair question, and sorry for the volume of messages to all; I did post to #pedestal but, as is sometimes the case on clojurians, the answer/help comes in here, but I think it can be made a side conversation or something (not sure how slack works, long live IRC)

seancorfield06:01:20

@kulminaator Sorry, we probably should have gone to a thread but since there were no other conversations and it's late at night US time (when it's normally quiet here), it seemed like it should have been a quicker problem to solve ๐Ÿ™‚

vlaaad14:01:06

I remember I saw a github repo that collected problems in clojure repls (like protocol redefinition requiring re-evaluating extend-protocol calls), but can't find it now, does anyone remembers that?

vlaaad15:01:26

This is it, thanks!

โœŒ๏ธ 4
noisesmith18:01:49

haha the -p at the end of the name is very revealing :D

mac0102114:01:36

Do you guys write code that looks like this?

(defn  env-use! [[name]]
  (let [envs (read-settings)
        envs (for [[name settings] envs]
               [name (dissoc settings :active)])
        _      (assert (envs name))
        envs (into {} envs)
        envs (assoc-in envs [name :active] true)]
    (write-settings! envs)))
Or am I failing at idiomaticity?

Daniel Stephens15:01:28

(defn env-use! [name]
    (let [envs (->> (read-settings)
                    (map (fn [[n s]] [n (dissoc s :active)]))
                    (into {}))]
      (assert (get envs name))
      (-> envs
          (assoc-in [name :active] true)
          write-settings!)))
Not completely sure of the intent of the method but from what I can tell I'd write something like this

slipset15:01:09

Not coming up with a solution, but I would argue that your function does too much:

slipset15:01:28

(defn env-use! [[name]]
  (-> (read-settings)
      (transform-settings name)
      (write-settings)))

๐Ÿ‘ 4
slipset15:01:25

Now you can define your transform-settings (which is kindโ€™a hard ๐Ÿ™‚

slipset15:01:22

(defn transform-settings [envs name]
  (let [envs (for [[name settings] envs]
               [name (dissoc settings :active)])
        _      (assert (envs name))
        envs (into {} envs)]
    (assoc-in envs [name :active] true)))

slipset15:01:21

This gives you the possibility to give transform-setting a reasonable name which would describe why youโ€™re doing this, and not how.

slipset15:01:46

It also makes it easier to write tests around this, and play with it from the repl, now that youโ€™ve separated the sideeffecting bits from the pure bits (at least if you squint a bit, since the assert would make this function non-pure (at least for some purists))

hindol18:01:41

I have a follow up question. Do you guys rebind the same variable in let? I tend to avoid it because the code can get confusing that way.

๐Ÿ‘ 4
borkdude14:01:39

that's a lot of envs

borkdude14:01:29

the assert looks a bit weird, you're calling the lazy sequence from the for expression with an argument?

mac0102115:01:48

oops, didn't test the code yet. I should do that assert first while envs is still a map

Alex Miller (Clojure team)14:01:29

usually a threading macro would make this a lot cleaner

mac0102115:01:59

The problem I have with the threading macro is that, if I want to add a transform in the middle that isn't just a simple function call, then I have to break everything up.

mac0102115:01:06

(not that I'm arguing - i've obviously written a lot less clojure than you have. just wondering what the pros do, normally.)

orestis15:01:26

Perhaps as-> ?

โœ”๏ธ 8
mac0102115:01:38

that is nice, i did not know about as->

Lone Ranger19:01:19

I could use some suggestions for diagnosing how I screwed up a macro implementation.

(go
    (py..
     requests
     (get "")
     -content)
    ) ;;=> Syntax error macroexpanding go at (src/python/dev2.clj:13:3).
Could not resolve var: content
(py..
   requests
   (get "")
   -content) ;;=> works as intended
(macroexpand-1 
   '(py..
     requests
     (get "")
     -content)) 
;;=> (#'libpython-clj.python/py.- (#'libpython-clj.python/py. requests get "") content)
I'm guessing I'm probably supposed to be using gensyms or something?

noisesmith19:01:17

where is "content" supposed to come from?

Lone Ranger19:01:47

correct. The Python equivalent code would be

requests.get("").content

noisesmith19:01:48

you are creating something that uses a symbol that isn't bound

Lone Ranger19:01:41

(py..
   requests
   (get "")
   -content) 
this works fine. It only breaks when I put it in a go block

noisesmith19:01:18

oh, I misunderstood - so py.- is a macro?

ghadi19:01:52

you're going to have a bad time mixing core.async with libpython-clj

๐Ÿ˜‚ 4
Lone Ranger19:01:54

correct, sorry. (py.. obj (method arg) -attr) expands to (py.- (py. obj method arg) attr)

Lone Ranger19:01:37

awww but I want it to be a good time!

ghadi19:01:39

make it work outside of core.async first

Lone Ranger19:01:50

it works outside of core.async already

ghadi19:01:53

core.async does its own macroexpansion

Lone Ranger19:01:01

ahh interesting.

Lone Ranger19:01:16

should we move this to a thread to be polite?

noisesmith19:01:22

the easy fix is to move the macro call into a function, and call the function inside go

8
Lone Ranger19:01:43

this is a great workaround, thank you

ghadi19:01:45

I'm still suspicious of moving single-threaded python into core.async

ghadi19:01:51

(which uses multiple threads)

noisesmith19:01:44

oh that is a potential issue

Lone Ranger19:01:19

@ghadi You should be suspicious! I'm investigating those edge cases right now ๐Ÿ™‚

ghadi19:01:36

probably move to #libpython-clj ?

Lone Ranger19:01:38

Yes indeed. Although in this case it's more about proper macro writing than the code itself -- the code works fine ๐Ÿ™‚ Thank you for the insight that core.async is doing something special

Lone Ranger19:01:14

Do you happen to know if this same issue occurs with

(go (.. obj method -attr))
?

ghadi19:01:42

not sure, but i know historically we had issues with the .- form

Lone Ranger19:01:51

ahhh okay, interesting. Thank you

Lone Ranger19:01:13

Yep, this works fine

(go
    (println (py.. requests (get ""))))
So we're experiencing the same issues with the py.- form then. I'll make sure to update the pitfalls section with that info

hiredman19:01:04

defining a name than ends in '.' is going to be a bad time

hiredman19:01:12

and the libpython stuff is basically an rpc call (one runtime to another) and the other famously has locks around everything, doing it from a go block is bad

โœ… 8
andy.fingerhut19:01:20

Clojure reference docs anti-recommend it in as strong a language as it ever uses, here: https://clojure.org/reference/reader. "Symbols beginning or ending with '.' are reserved by Clojure."

andy.fingerhut19:01:40

Definitely seems worth an issue

hiredman19:01:21

ugh, and that library actually recommends calling System/gc

andy.fingerhut20:01:26

do you have a link to that part?

hiredman19:01:24

this is gross

ghadi19:01:37

It's a very, very cool library. I've seen several amazing demos using it

hiredman20:01:00

sure, and I have no doubt there is a lot of interest in it, no doubt it will blow in popularity, so it would be great if the insides were not gross

hiredman20:01:10

I just wish (and I understand why it doesn't happen) that sometimes cool libraries that take off were written by people both deep into both domain X and clojure, and not mostly by people that are deep into domain X and maybe just wading into clojure a little.

andy.fingerhut20:01:08

Understand what you are saying. I haven't ever tried to design language interop systems like this before, but JVM/Python seems like it has plenty of issues to find a solution for. Rich Hickey did several Java/Common-Lisp interop systems before developing Clojure.

andy.fingerhut20:01:24

You can make suggestions where you see things could be better, but in the end, these things are built by people with the time/interest/determination.

andy.fingerhut20:01:17

(You may also have that time/interest/determination, but I'm not trying to volunteer you for projects you don't want to do ๐Ÿ™‚

hiredman20:01:34

(a lot of the macros in this library could just be functions)

seancorfield20:01:31

I've just discovered that (deftest foo) is legal (and produces an empty test, essentially, with no assertions. That was surprising to me. Can anyone think of any reason why that should be supported, rather than throwing an exception?

seancorfield20:01:02

(I ask because I'm looking at differences in behavior between clojure.test's deftest and Expectations' defexpect)

andy.fingerhut20:01:31

Same reason that defn and do with an empty body are not an error? (and most macros that take a body, I think, e.g. let )

seancorfield20:01:09

Well, defn without a body just returns nil and that's perfectly reasonable. But an empty test?

andy.fingerhut20:01:18

Of course the author of deftest could have added an explicit check to give an error for an empty body, but they probably chose to just pass it on to whatever other defn /etc. macro that they used to implement the deftest body, without additional checks

andy.fingerhut20:01:05

You can have two deftest in the same namespace with the same name, due to copy/paste errors, and one of them will never run its tests. No errors, no warnings, no tests run.

andy.fingerhut20:01:12

Not even the failing ones ๐Ÿ™‚

seancorfield20:01:42

Which is exactly like two defns with the same name...

seancorfield20:01:02

...but not the same as specifically writing an empty test.

seancorfield20:01:26

It just surprised me. But then deftest doesn't enforce the presence of is/`are` either so...

andy.fingerhut20:01:28

sure, just an example I like to toss in where the system isn't doing what you want, even if it is doing what you told it to, whether you knew that or not.

andy.fingerhut20:01:25

I doubt anyone is likely to write empty bodies on any of those things, by hand, except as an accident. Having a macro that expands into one of those things with an empty body might be one of the reasons they are allowed.

seancorfield20:01:12

Background: (defexpect foo (bar)) is currently treated as (deftest foo (is (bar)) which can produce surprising failures when the intention is just to write a "sanity check" that runs code but contains no assertions. (deftest foo (bar)) is perfectly reasonable as a "test" in that context. (defexpect foo) currently blows up (because it expands to (deftest foo (expect)) which is an illegal arity -- same as (deftest foo (is))) but (deftest foo) is legal. So I'm debating whether to make defexpect more like deftest in those cases.

andy.fingerhut20:01:43

This might not clarify anything, and you may already be aware, but (deftest foo (bar)) I have used myself (i.e. function calls inside a deftest body), where those functions contain is calls in their bodies (perhaps nested several function calls deep)

andy.fingerhut20:01:24

i.e. is and are work just fine in the dynamic scope of the deftest even if they are outside the lexical scope

andy.fingerhut20:01:49

which can be useful in writing tests that share logic in a common fn between them.

Derek20:01:51

@seancorfield Is this for the clojure.test compatible expectations?

seancorfield20:01:29

@dpassen1 yes. "Classic" Expectations doesn't have named tests (which is why it isn't compatible with most test tooling out there).

Derek20:01:03

Would it be simpler to just remove defexpect and assume all expectations can and will be called within deftest?

seancorfield20:01:54

@dpassen1 well you can already use deftest as a wrapper for expect forms -- this is just the syntactic sugar for migration.

seancorfield20:01:07

I was probably "trying too hard" to provide a migration path from "Classic" Expectations ๐Ÿ™‚

Derek21:01:34

I canโ€™t fault you for that

seancorfield22:01:00

I'm starting on 2.x for the clojure.test-compatible version hence the changes being discussed here. I may well stop documenting the (defexpect my-name pred (expr)) format as a shortcut (but continue supporting it) so as to discourage its use. I suspect we may be the heaviest use of the new version (at World Singles Networks) anyway so I probably don't need to be as worried about backward compatibility as I am ๐Ÿ™‚

Derek20:01:44

AFAICT we only lose that single expression form

seancorfield20:01:08

@andy.fingerhut Yeah, good point. I think I just made an error when I originally wrote the defexpect macro to auto-wrap bodies containing two or fewer expressions in expect, as a migration from "Classic" Expectations: (expect (bar)) was a sensible test there -- an unnamed assertion that (bar) produces a truthy result. So (expect (bar)) could just become (defexpect my-name (bar)) rather than requiring you to write (defexpect my-name (expect (bar)))

seancorfield20:01:44

(for (expect pred (expr)) it sort of made sense that (defexpect my-name pred (expr)) should be equivalent but...)

seancorfield20:01:18

OK, I'll just special case the two form case and otherwise make defexpect behave like deftest.

seancorfield20:01:33

Thanks for the input!

Eduardo Mata21:01:21

Hello! Is there a way of passing a java option such as -Dsegment=True and use the java option as a variable and value inside the code

ghadi21:01:24

(System/getProperty "segment")

Eduardo Mata21:01:35

Could I then use the value in a function? i.e.

(defn something [] (when (true? (System/getProperty "segment")) DoSomething))

souenzzo17:02:40

I use

(-> (System/getProperty "segment") edn/read-string)

andy.fingerhut21:01:52

I think the return value is always a string, so perhaps (= "true" (System/getProperty "segment")) is closer to what you want, as opposed to calling true?

Eduardo Mata21:01:38

oh I see. That make sense

Alex Miller (Clojure team)21:01:15

you might actually look at (Boolean/getBoolean "segment") for an alternative weird tributary of Java

โ€ผ๏ธ 4
๐Ÿ˜ 4
ghadi22:01:16

Apparently Booleans didnโ€™t have a rich enough API

noisesmith22:01:59

holy wow

$ clj -J-Dbar=1
Clojure 1.10.1
(ins)user=> (Long/getLong "bar")
1
(ins)user=> (type *1)
java.lang.Long

Alex Miller (Clojure team)22:01:56

these functions are weird, but I find they are actually the least error-prone way to safely read non-string system properties

andy.fingerhut00:01:51

It does seem odd that they chose to put them into classes Boolean, Long, etc.

noisesmith00:01:37

at least http://java.net.URL/getURL doesn't exist

Alex Miller (Clojure team)01:01:14

URL has its own sins :)

emccue22:01:35

This might be maybe the 4th time ive written something like this for a random hobby project

emccue22:01:37

I've either stumbled upon an antipattern or a pattern

emccue22:01:59

but i find myself nesting with-opens with lets a bit more than is legible

emccue22:01:54

so usage would be like

emccue22:01:30

well, actually direct example from what im doing

emccue22:01:16

where both Demuxer and MediaPicture are from a library thats wrapping some JNI pointers

emccue22:01:00

and .close is used in conjunction with /make to manage reference counts

isak22:01:20

do you actually need on-enter? Maybe you could do those acquisitions in the constructor of some object, because then you can use AutoCloseable (built in)

emccue22:01:03

I'm truly not sure

emccue22:01:16

All my use cases come down to closables, usually

emccue22:01:52

but theres also things like with-transaction and co. that, in my imagination, could fit under the with-* style macro pattern

emccue22:01:55

if that makes sense

emccue22:01:21

the on-enter and on-exit come from the python design for context managers

noisesmith22:01:42

user=> (with-open [foo (reify java.lang.AutoCloseable (close [this] (println "bye")))] (println "hi"))
hi
bye
nil

โž• 4
emccue22:01:10

the issue isnt so much making it work as it is doing it nested

emccue22:01:04

which i guess is my primary motivator

emccue22:01:19

(and which may or may not be reasonable)

isak22:01:36

good point

noisesmith22:01:06

what about a with-open that calls close for things that have the method, and ignores args that don't have it?

โž• 4
emccue22:01:26

that makes sense, but that would have to use reflection no?

noisesmith22:01:35

or hints, yeah

emccue22:01:02

I'm not concerned with performance at all really, but the benefits of narrowing the interface to .close doesn't seem to outweigh the downsides of using reflection everywhere

emccue22:01:07

at least in my brain

emccue22:01:36

where the usecase here would be to intersperse some normal lets with resource acquiring calls

isak22:01:46

In F#, they have let conn = makeConnection() vs. use conn = makeConnection() , where the latter will call .Dispose on conn at the end of the scope

noisesmith22:01:56

you can also use with-open with a wrapper that reifies closeable as a no-op

emccue22:01:34

that wouldnt work entirely, at least in this case

isak22:01:35

you can make a macro that uses 3 forms for bindings instead of just 2 like normal

emccue22:01:36

(with-open [resource-a (...)
            normal-thing (noop 5)
            resource-b (... normal-thing)]
  ...)

emccue22:01:41

that would work

emccue22:01:53

but in my example, I am using destructuring

emccue22:01:06

which with-open doesnt support

emccue22:01:45

and while making a 3 form macro would work, I generally work in cursive, so at least for selfish reasons "looking like let" makes the plugin work

โœ”๏ธ 4
noisesmith22:01:51

(cmd)user=> (defmacro as-closeable [v] `(reify java.lang.AutoCloseable (close [_#]) clojure.lang.IRef (deref [_#] ~v)))
#'user/as-closeable
(ins)user=> (with-open [a (as-closeable 1) b (as-closeable 2)] (+ @a @b))
3
just "coding out loud" here so to speak, dunno if that leads to anything usable

emccue22:01:07

whereas custom syntax for binding would make everything come up as undefined

isak22:01:36

yea it is a good point, I have the same concern

emccue22:01:33

skimming around libraries for self-justification -

emccue22:01:51

(with-open [con (jdbc/get-connection ds)]
  (jdbc/execute! con ...)
  (jdbc/execute! con ...)
  (into [] (map :column) (jdbc/plan con ...)))
(jdbc/with-transaction [tx ds]
  (jdbc/execute! tx ...)
  (jdbc/execute! tx ...)
  (into [] (map :column) (jdbc/plan tx ...)))

emccue22:01:17

at least with next.jdbc there could be a use

emccue22:01:46

but its not really strong - those macros seem to do just fine

emccue22:01:49

user> (def ignored (time (py/with-gil-stack-rc-context
                           (->> (repeatedly 1000 #(py/->py-dict {:a 1 :b 2}))
                                (py/->py-list)
                                (py/as-jvm)
                                (mapv py/->jvm)))))

emccue22:01:35

and this kind of macro is a counterexample to the thought that this could replace with-* macros

emccue22:01:06

since, not every with-* macro yields anything to bind

emccue22:01:22

or benefits from being "horizontal" with code that would otherwise be nested

emccue22:01:14

(with [_ (py/gil-stack-rc-context)
       py-list (->> (repeatedly 1000 #(py/->py-dict {:a 1 :2}))
                                (py/->py-list)
                                (py/as-jvm)
                                (mapv py/->jvm))]
  py-list)

emccue22:01:48

which, while it would work, doesn't exactly add value

emccue22:01:23

which probably generalizes to most "tons of resources allocated" types of things

emccue22:01:49

just thinking out loud

isak22:01:15

another option:

(with* [conn (make-connection)
          _ (defer* (.close conn))
          something (something-else)]
  (foo conn something))

isak22:01:48

so defer* would be a special thing inside the binding that would run that code at the end of the scope

emccue22:01:55

well, at least in this system

emccue22:01:06

(defmacro defer [statement]
  `(reify ContextManager
     (on-enter [_#])
     (on-exit [_# _#] ~statement)))

ghadi22:01:26

not reading the scrollback, but why not try/finally?

emccue22:01:47

I'm not sure how else to make defer* special than to do some runtime type checking regardless

isak22:01:33

you would just check the symbol at macro expansion time, no?

emccue22:01:57

@ghadi Its mainly about nesting with multiple resources and lets

emccue22:01:20

clojure.test does that with some things with is

isak22:01:35

so with* would be the macro, not defer*

emccue22:01:55

it breaks the ide since the symbol isn't actually anywhere, but also its an annoying suprise of semantics

emccue22:01:33

plus the defer style thing implies a side effect, like in go

emccue22:01:51

defer is only needed as a seperate statement if it might not be called

emccue22:01:54

if that makes sense

hiredman22:01:18

don't manually write code that is a linearized graph, write a graph and have the computer linearize it ๐Ÿ™‚

emccue22:01:22

if cond {
   x = ...
   defer x.close()
}
...

p-himik22:01:35

Just to make sure - I can't write a macro that accepts unbalanced {, [, and (, right?

emccue22:01:44

though that does seem like a strawman argument

emccue22:01:53

@p-himik not to my knowledge

emccue22:01:15

@hiredman which concept is that statement in favor of?

isak22:01:23

well I assumed you wanted in called even in the case of exceptions, no?

hiredman22:01:28

none, I guess

emccue22:01:37

@isak yes, but thats somewhat orthogonal to the syntax/metaphysics of the thing

emccue22:01:46

the hypothetical with* looks like let to satisfy the IDE, but the only feature it adds carries the syntactic burden of the [thing binding] part of let

emccue22:01:58

so it feels ...wrong

emccue22:01:07

or at the very least "non-native"

emccue22:01:21

though I do see the benefit of not needing any runtime cost

emccue23:01:06

since the try, except, finally triplets need only exist for the places that actually need them

emccue23:01:39

and if no forms actually use the features it would still allow recur

emccue23:01:52

which makes an easier case for "just use instead of let"

hiredman23:01:21

(def g (as-> {} graph
         (assoc graph :open-context #{})
         (assoc graph :some-operation #{:open-context})
         (assoc graph :some-operation2 #{:some-operation})
         (assoc graph :close-context (set ((fn f [k]
                                             (cons k (for [k (get graph k) i (f k)] i)))
                                           :open-context)))))

(defn topo [graph]
  (when-not (empty? graph)
    (lazy-seq
     (let [n (first (for [[node s] graph
                          :when (not (seq s))]
                      node))]
       (assert n "if this fails there is a cycle")
       (cons n (topo (into {} (for [[node deps] graph
                                    :when (not= n node)]
                                [node (disj deps n)]))))))))


(topo g)

;;=> (:open-context :some-operation :some-operation2 :close-context)

hiredman23:01:00

the code that computes the dependencies isn't exactly right

emccue23:01:27

@hiredman I'm lost, is that on this topic or something else

emccue23:01:03

are you advocating for making a state machine or similar?

hiredman23:01:40

you define your contexts and operations on same as a graph of dependencies and then use a topo sort to flatten it out in to something to run, and never write nested lets and with-opens

4
hiredman23:01:53

because you have a literal dependency graph, a computer can easily insert the close after the final depedency

ghadi23:01:59

this is the content I'm here for

noisesmith23:01:34

that's basically the skeleton of stuartsierra/component

emccue23:01:13

thanks i hate it

isak23:01:47

@hiredman that is cool and makes sense for some cases like assembling a running system, but do you still think it makes sense for smaller things?

emccue23:01:07

at the risk of offending @hiredman, I think its a joke

hiredman23:01:52

I've written things using it, but not for resource management, more for parallel task dependency things

4
emccue23:01:54

"make not only code data, but the way your code runs data"

emccue23:01:00

turtles all the way down

hiredman23:01:38

because the code that builds the graph is responsible for composition you can do interestings with it like composing in some monad, like seqs, so your graph can become a weird kind of sequence processing thing

hiredman23:01:55

or futures, or whatever, pick an M

emccue23:01:02

You know how most animals evolve an aversion to spicy or bitter foods because in the wild those are signs of poison?

emccue23:01:13

I feel the same way about monads

hiredman23:01:02

see, the thing with poison is it kills you if you know about it or not, where as people with aversions to monads are happy to use them until someone points out it is a monad

borkdude23:01:50

some poisons act like nootropics in small doses

emccue23:01:26

yeah like that guy who can eat glass and bullets

emccue23:01:33

at one point he couldn't do that

emccue23:01:35

and now he can

borkdude23:01:02

sorry, too off topic ๐Ÿ˜›

emccue23:01:06

no that makes sense

emccue23:01:15

if I start using monads maybe I'll seem smarter

hiredman23:01:35

a way monads are often talked about is as a computation in some context, which seems like it might be particular interesting to learn about given how you are looking around for ways to deal with contexts

seancorfield23:01:42

I consider Monads to be a bit like Design Patterns these days -- they evolved from observation of common code structure in solutions to certain problems and then got a little more formalized and given names.

seancorfield23:01:05

You're basically already using monads. You just don't identify them as such.

seancorfield23:01:37

(and, yeah, monads have more underlying formalism in math than design patterns)

emccue23:01:55

at least in this context, I don't think going to the level of composing monads makes sense, since alot of builtins kinda classify as monads

noisesmith23:01:08

on the other hand, show an actual category theory expert from the math world what we call monads and they'll call you mad

๐Ÿ˜ 4
emccue23:01:15

like I'm pretty sure cats treats Object/nil as a "maybe monad"

emccue23:01:02

which infects the code a bit more than I would like in that cats/mlet isn't exactly a drop in solution

emccue23:01:45

because if i were to make "closable" a monadic thing

emccue23:01:53

Closing[A] or whatever

emccue23:01:10

then my whole expression would need to end up wrapped in one of the monads in the end

emccue23:01:30

if that checks out

emccue23:01:15

my understanding of that side of the world isn't exactly clean

emccue23:01:27

though i do know thats how scala handles it more or less

emccue23:01:39

since their for is more or less a monadic let

emccue23:01:09

there are libraries that do for (file <- managed(inputStream)) yield { ... }

emccue23:01:51

I'll try rubbing some monads on it later

๐Ÿ˜ 4
isak23:01:00

See any problems with this?

(with* [conn (make-connection)
          :defer (.close conn)
          something (make-something)]
         (foo conn something))
(Since this is fine for IDEs:)
(for [i (range 5)
        :when (= 0 (rem i 2))
        j (range 5)
        :when (= 1 (rem j 2))]
    (str i j))

emccue23:01:57

not conceptually, but cursive isnt liking it

emccue23:01:57

definitely less convoluted semantics than the protocol dispatch and exception rethrowing

emccue23:01:46

maybe even :closing conn would make sense