This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-01-28
Channels
- # babashka (28)
- # beginners (252)
- # bristol-clojurians (2)
- # calva (28)
- # cider (11)
- # clj-kondo (15)
- # cljs-dev (7)
- # clojure (378)
- # clojure-europe (4)
- # clojure-italy (4)
- # clojure-nl (3)
- # clojure-norway (4)
- # clojure-uk (32)
- # clojurescript (128)
- # cursive (39)
- # data-science (18)
- # docker (37)
- # figwheel-main (10)
- # fulcro (45)
- # ghostwheel (7)
- # graalvm (2)
- # hugsql (1)
- # jobs (2)
- # joker (5)
- # kaocha (5)
- # luminus (12)
- # off-topic (37)
- # onyx (4)
- # pathom (22)
- # pedestal (70)
- # re-frame (7)
- # reagent (30)
- # ring (4)
- # shadow-cljs (12)
- # spacemacs (1)
- # sql (26)
- # tools-deps (7)
- # vrac (2)
- # vscode (7)
- # xtdb (27)
I asked a pedestal/docker question in #pedestal if anyone feels like taking look.
@macrobartfast That Pedestal issue talks about ::server/host
-- you mentioned ::http/host
-- is your host
an alias to the same namespace that server
is aliased to in that issue?
hmm... let me look.
(slowly grokking what you saying, too)
@macrobartfast Looking at the commits around that security issue, it seems like you should just be able to specify :host "0.0.0.0"
in the hash map you pass into the Pedestal server at startup. Can you share how you're starting the server?
I start the docker container with docker run -d -p 8080:8080 general-service
(general-service is the name I gave my docker image)
the -d just makes it a daemon... I had the same problem with or without it.
No, I mean the Clojure code that starts the server.
oh, gotcha... stand by...
I've never used Pedestal but all the Clojure servers I've seen get passed an options map when they start, where you can pass :host
and that can be "0.0.0.0"
how do you post longer code in here in a socially acceptable way?
i.e. the slack version of pastebin
Post it to a service like pastebin and put the link here?
haha ok I had thought there was some slacky way to do it...
so, I call a main function
(defn -main
"The entry-point for 'lein run'"
[& args]
(println "\nCreating your server...")
(server/start runnable-service))
I was only expecting a few lines 🙂
which of course is calling a whole bunch of other code.
was figuring out how to summarize that.
And that server/start
-- does it accept other options?
I don't know... lemme look.
and this is helpful to me in experiencing how you would start solving an issue like this, rather than my usual mode of flailing my arms around the web.
server is io.pedestal.http http://pedestal.io/api/pedestal.service/io.pedestal.http.html
I'm reading that now.
OK, so what is runnable-server
in your code?
You got it by calling create-server
yes?
haha just catching up with you stand by
And you pass a hash map to create-server
with options in to describe the server?
(defonce runnable-service (server/create-server service/service))
OK... and what is service/service
then? A default service description, I assume?
lemme look, stand by...
[general-pedestal-service.service :as service]
Is that your code or something in Pedestal? (remember I've never used Pedestal)
which is the other principal file.
I know, which makes this so extraordinary to me... you are walking me through my code, and you don't have it. ;-D
I'm reading service.clj right now, and trying to put it all together. Even if the original issue isn't resolved, I'm learning a lot about how Pedestal works.
OK, so in that file, there's a symbol service
that is a hash map?
So general-pedestal-service.service
is Pedestal code, not your code?
indeed
If so, the fix to all this is to change your defonce
:
and yes, a hash map.
it's a def, but yes, probably.
(defonce runnable-service (server/create-server (assoc service/service :host "0.0.0.0")))
No, your defonce
oh right.
(you don't go changing library code)
haha sounds like a good idea, especially in my case.
Since service/service
is a "service map" per Pedestal API docs, that's what can have a :host
key (per the commit I looked at) and it defaults to "localhost"
for security reasons.
ok, modified the defonce.
But if you add your :host
key, it overrides that default.
ok, very cool.
This was the commit I found https://github.com/pedestal/pedestal/commit/a9a0845736944836f5cd352e965a9c012d3a0de6
I searched the Pedestal repo for localhost
and looked at the commits
ah ok...
was wondering how you found that!
My initial search was for :host
, followed by localhost
in the code, but those returned too many results. I just figured it must be doing something similar to what other Clojure http servers do.
building a new docker container now.
The question is: did it fix your problem? 🙂
we'll find out!
while the container builds, I can say the answer is yes...
as the problem is my unwillingness to walk the code and look a the source. 😅
Have you watched Debugging With The Scientific Method, the talk by Stu Halloway?
not yet
You should 🙂
I'll look it up tonight.
so, the port is still unreachable... but I was rushing.
let me make sure I implemented the possible fix right.
oh, I might have done something silly, stand by, building docker again.
well, hmm.... still not reachable, but we've made great progress, I suspect.
I just need to continue walking the code, probably.
also, they support tomcat and another server in pedestal, maybe I'll try those.
You can run it standalone outside of Docker?
Another test you can run: build the Docker container and run it, then shell into it and try to telnet to localhost on the port you believe the server is running on -- that would confirm whether your server is really starting up and binding to the right port (on the loopback).
omg, just realized I wasn't rebuilding the uberjar before building the container... 😳
but, to answer the question, yes, it did work outside of the docker container.
rebuilding the docker image with the fix we came up with in the actual uberjar. unbelievable.
which still doesn't work. I'll try the telnet route now.
If you can run it outside Docker, you can test the port access via telnet (both on localhost and on the IP that your machine has to test how the server started up and whether it bound to the right port).
That would establish whether the problem is the server code / config or the Docker setup.
You have a lot of variables in play the way you are trying to run it right now.
and I realized just now when you meant 'standalone outside docker' you meant the java -jar <standalone> I had thought you meant via the repl in cider... testing the standalone now.
yep, standalone jar works.
Via both localhost and the main IP of your machine?
Then the next test will be to telnet/ssh into the running Docker container and then try that same telnet test inside the container..
(basically trying to go step by step through the system, verifying each piece of infrastructure)
re standalone: only checked localhost, I'll check that, then I'll go the ssh route (just got into the container, turns out it's /bin/sh)
so, works localhost, but not the LAN ip of the machine on my home router...
(the standalone, that is)
http://127.0.0.1:8080/ works, http://localhost:8080/ works, http://0.0.0.0:8080/ works...
my LAN address seems to be 10.0.0.236, and 10.0.0.236:8080 does not work.
That the LAN IP doesn't work might be a firewall setting locally so perhaps don't worry about that too much.
and I opened the firewall.
Hmm...
messages crossed.
next step is the ssh into docker route.
so, was able to telnet to the port inside the container (cool trick).... got the blank screen which means the port is open.
and while I'm in there, I see that the jar running.
"blank screen" from telnet?
well, ran telnet 0.0.0.0 8080
and enter
and the there is no response, which means it's open, according to https://www.opcsupport.com/s/article/How-do-I-ping-and-telnet-to-test-a-tunneller-connection, anyway ;-D
I guess if the port isn't open, you get 'telnet: can't connect to remote host' which I confirmed by trying 8081
Right but you can send an HTTP command like GET /
(and press enter twice)
haha, well, learning away here... that did produce a response
<h1>Bad Message 400</h1><pre>reason: HTTP/0.9 not supported</pre>Connection closed by foreign host
And just to confirm, you are running that telnet
command inside the container after shelling into it? Just want to make sure you're not accidentally still testing the standalone JAR running outside Docker.
I am... apparently osx doesn't have telnet, as a side note.
OK, so GET / HTTP/1.1
(should prevent the HTTP/0.9 problem)
ok cool.
I'm on a Mac and I have telnet
hmmm, not sure why I don't,
I have heard that some recent versions of macOS / OSX have ssh/scp/sftp, but not unencrypted telnet/ftp, unless you go out of your way to install them. They were installed by default as of the latest macOS from 2-3 years ago -- I forget the exact version where this changed.
however, inside the container...
GET / HTTP/1.1
HTTP/1.1 400 No Host
Content-Type: text/html;charset=iso-8859-1
Content-Length: 50
Connection: close
<h1>Bad Message 400</h1><pre>reason: No Host</pre>Connection closed by foreign host
iiiinteresting.
Ah, OK, so it needs a Host:
header before it will accept a connection.
That could be how Pedestal is configured for security?
aha ok... although the jar works outside the container.
GET / HTTP/1.1
Host:
maybe?oh, I understand. this is cool, btw... always wanted to telnet.
If you telnet
to the JAR running outside the container? You can issue that GET
and it works?
OK, so here's a thought: when you try to run the Docker container, do you already have something on your Mac running on that port (8080), such as the standalone JAR?
You can't have two processes on the same port.
and now I understand why you press enter twice in telnet (so you can create multiline commands)
let me telnet to the jar, stand by.
regarding two things running: oops, I had left the jar running, but that was only for the last couple of minutes.
I'll now attempt to telnet to the jar, after stopping the docker container, for good measure.
And more importantly, make sure the JAR is not running when you start the Docker container 🙂
gotcha (I promise that hasn't been running this whole time)
unless you map the Docker container to a different port (e.g., 8888)... can't remember which way round that goes 8888:8080 or 8080:8888
Mojave doesn't have telnet; installing via brew
I think the first one is the host machine.
Are you sure this is the best channel for all of this ?
@kulminaator fair question, and sorry for the volume of messages to all; I did post to #pedestal but, as is sometimes the case on clojurians, the answer/help comes in here, but I think it can be made a side conversation or something (not sure how slack works, long live IRC)
@kulminaator Sorry, we probably should have gone to a thread but since there were no other conversations and it's late at night US time (when it's normally quiet here), it seemed like it should have been a quicker problem to solve 🙂
I remember I saw a github repo that collected problems in clojure repls (like protocol redefinition requiring re-evaluating extend-protocol calls), but can't find it now, does anyone remembers that?
haha the -p
at the end of the name is very revealing :D
Do you guys write code that looks like this?
(defn env-use! [[name]]
(let [envs (read-settings)
envs (for [[name settings] envs]
[name (dissoc settings :active)])
_ (assert (envs name))
envs (into {} envs)
envs (assoc-in envs [name :active] true)]
(write-settings! envs)))
Or am I failing at idiomaticity?(defn env-use! [name]
(let [envs (->> (read-settings)
(map (fn [[n s]] [n (dissoc s :active)]))
(into {}))]
(assert (get envs name))
(-> envs
(assoc-in [name :active] true)
write-settings!)))
Not completely sure of the intent of the method but from what I can tell I'd write something like this(defn env-use! [[name]]
(-> (read-settings)
(transform-settings name)
(write-settings)))
(defn transform-settings [envs name]
(let [envs (for [[name settings] envs]
[name (dissoc settings :active)])
_ (assert (envs name))
envs (into {} envs)]
(assoc-in envs [name :active] true)))
This gives you the possibility to give transform-setting
a reasonable name which would describe why you’re doing this, and not how.
It also makes it easier to write tests around this, and play with it from the repl, now that you’ve separated the sideeffecting bits from the pure bits (at least if you squint a bit, since the assert
would make this function non-pure (at least for some purists))
I have a follow up question. Do you guys rebind the same variable in let? I tend to avoid it because the code can get confusing that way.
the assert looks a bit weird, you're calling the lazy sequence from the for expression with an argument?
oops, didn't test the code yet. I should do that assert first while envs is still a map
usually a threading macro would make this a lot cleaner
The problem I have with the threading macro is that, if I want to add a transform in the middle that isn't just a simple function call, then I have to break everything up.
(not that I'm arguing - i've obviously written a lot less clojure than you have. just wondering what the pros do, normally.)
really great overview of threading techniques here: https://medium.com/@hlship/confessions-of-a-threading-macro-addict-5a026dae4af7
I could use some suggestions for diagnosing how I screwed up a macro implementation.
(go
(py..
requests
(get "")
-content)
) ;;=> Syntax error macroexpanding go at (src/python/dev2.clj:13:3).
Could not resolve var: content
(py..
requests
(get "")
-content) ;;=> works as intended
(macroexpand-1
'(py..
requests
(get "")
-content))
;;=> (#'libpython-clj.python/py.- (#'libpython-clj.python/py. requests get "") content)
I'm guessing I'm probably supposed to be using gensyms or something?where is "content" supposed to come from?
requests?
correct. The Python equivalent code would be
requests.get("").content
you are creating something that uses a symbol that isn't bound
(py..
requests
(get "")
-content)
this works fine. It only breaks when I put it in a go blockoh, I misunderstood - so py.-
is a macro?
correct, sorry. (py.. obj (method arg) -attr)
expands to (py.- (py. obj method arg) attr)
awww but I want it to be a good time!
it works outside of core.async already
ahh interesting.
should we move this to a thread to be polite?
the easy fix is to move the macro call into a function, and call the function inside go
this is a great workaround, thank you
oh that is a potential issue
@ghadi You should be suspicious! I'm investigating those edge cases right now 🙂
Yes indeed. Although in this case it's more about proper macro writing than the code itself -- the code works fine 🙂 Thank you for the insight that core.async is doing something special
Do you happen to know if this same issue occurs with
(go (.. obj method -attr))
?ahhh okay, interesting. Thank you
Yep, this works fine
(go
(println (py.. requests (get ""))))
So we're experiencing the same issues with the py.-
form then. I'll make sure to update the pitfalls section with that infoand the libpython stuff is basically an rpc call (one runtime to another) and the other famously has locks around everything, doing it from a go block is bad
Clojure reference docs anti-recommend it in as strong a language as it ever uses, here: https://clojure.org/reference/reader. "Symbols beginning or ending with '.' are reserved by Clojure."
Definitely seems worth an issue
do you have a link to that part?
sure, and I have no doubt there is a lot of interest in it, no doubt it will blow in popularity, so it would be great if the insides were not gross
I just wish (and I understand why it doesn't happen) that sometimes cool libraries that take off were written by people both deep into both domain X and clojure, and not mostly by people that are deep into domain X and maybe just wading into clojure a little.
Understand what you are saying. I haven't ever tried to design language interop systems like this before, but JVM/Python seems like it has plenty of issues to find a solution for. Rich Hickey did several Java/Common-Lisp interop systems before developing Clojure.
You can make suggestions where you see things could be better, but in the end, these things are built by people with the time/interest/determination.
(You may also have that time/interest/determination, but I'm not trying to volunteer you for projects you don't want to do 🙂
I've just discovered that (deftest foo)
is legal (and produces an empty test, essentially, with no assertions. That was surprising to me. Can anyone think of any reason why that should be supported, rather than throwing an exception?
(I ask because I'm looking at differences in behavior between clojure.test
's deftest
and Expectations' defexpect
)
Same reason that defn
and do
with an empty body are not an error? (and most macros that take a body, I think, e.g. let
)
Well, defn
without a body just returns nil
and that's perfectly reasonable. But an empty test?
Of course the author of deftest
could have added an explicit check to give an error for an empty body, but they probably chose to just pass it on to whatever other defn
/etc. macro that they used to implement the deftest
body, without additional checks
You can have two deftest
in the same namespace with the same name, due to copy/paste errors, and one of them will never run its tests. No errors, no warnings, no tests run.
Not even the failing ones 🙂
Which is exactly like two defn
s with the same name...
...but not the same as specifically writing an empty test.
It just surprised me. But then deftest
doesn't enforce the presence of is
/`are` either so...
sure, just an example I like to toss in where the system isn't doing what you want, even if it is doing what you told it to, whether you knew that or not.
I doubt anyone is likely to write empty bodies on any of those things, by hand, except as an accident. Having a macro that expands into one of those things with an empty body might be one of the reasons they are allowed.
Background: (defexpect foo (bar))
is currently treated as (deftest foo (is (bar))
which can produce surprising failures when the intention is just to write a "sanity check" that runs code but contains no assertions. (deftest foo (bar))
is perfectly reasonable as a "test" in that context. (defexpect foo)
currently blows up (because it expands to (deftest foo (expect))
which is an illegal arity -- same as (deftest foo (is))
) but (deftest foo)
is legal. So I'm debating whether to make defexpect
more like deftest
in those cases.
This might not clarify anything, and you may already be aware, but (deftest foo (bar))
I have used myself (i.e. function calls inside a deftest body), where those functions contain is
calls in their bodies (perhaps nested several function calls deep)
i.e. is
and are
work just fine in the dynamic scope of the deftest
even if they are outside the lexical scope
which can be useful in writing tests that share logic in a common fn between them.
@seancorfield Is this for the clojure.test compatible expectations?
@dpassen1 yes. "Classic" Expectations doesn't have named tests (which is why it isn't compatible with most test tooling out there).
Would it be simpler to just remove defexpect and assume all expectations can and will be called within deftest?
@dpassen1 well you can already use deftest
as a wrapper for expect
forms -- this is just the syntactic sugar for migration.
I was probably "trying too hard" to provide a migration path from "Classic" Expectations 🙂
I'm starting on 2.x for the clojure.test
-compatible version hence the changes being discussed here. I may well stop documenting the (defexpect my-name pred (expr))
format as a shortcut (but continue supporting it) so as to discourage its use. I suspect we may be the heaviest use of the new version (at World Singles Networks) anyway so I probably don't need to be as worried about backward compatibility as I am 🙂
@andy.fingerhut Yeah, good point. I think I just made an error when I originally wrote the defexpect
macro to auto-wrap bodies containing two or fewer expressions in expect
, as a migration from "Classic" Expectations: (expect (bar))
was a sensible test there -- an unnamed assertion that (bar)
produces a truthy result. So (expect (bar))
could just become (defexpect my-name (bar))
rather than requiring you to write (defexpect my-name (expect (bar)))
(for (expect pred (expr))
it sort of made sense that (defexpect my-name pred (expr))
should be equivalent but...)
OK, I'll just special case the two form case and otherwise make defexpect
behave like deftest
.
Thanks for the input!
Hello!
Is there a way of passing a java option such as -Dsegment=True
and use the java option as a variable and value inside the code
Could I then use the value in a function? i.e.
(defn something [] (when (true? (System/getProperty "segment")) DoSomething))
I think the return value is always a string, so perhaps (= "true" (System/getProperty "segment"))
is closer to what you want, as opposed to calling true?
oh I see. That make sense
you might actually look at (Boolean/getBoolean "segment")
for an alternative weird tributary of Java
https://docs.oracle.com/javase/8/docs/api/java/lang/Boolean.html#getBoolean-java.lang.String-
holy wow
$ clj -J-Dbar=1
Clojure 1.10.1
(ins)user=> (Long/getLong "bar")
1
(ins)user=> (type *1)
java.lang.Long
these functions are weird, but I find they are actually the least error-prone way to safely read non-string system properties
It does seem odd that they chose to put them into classes Boolean, Long, etc.
at least http://java.net.URL/getURL doesn't exist
URL has its own sins :)
This might be maybe the 4th time ive written something like this for a random hobby project
where both Demuxer and MediaPicture are from a library thats wrapping some JNI pointers
do you actually need on-enter
? Maybe you could do those acquisitions in the constructor of some object, because then you can use AutoCloseable
(built in)
but theres also things like with-transaction
and co. that, in my imagination, could fit under the with-*
style macro pattern
user=> (with-open [foo (reify java.lang.AutoCloseable (close [this] (println "bye")))] (println "hi"))
hi
bye
nil
what about a with-open that calls close
for things that have the method, and ignores args that don't have it?
or hints, yeah
I'm not concerned with performance at all really, but the benefits of narrowing the interface to .close
doesn't seem to outweigh the downsides of using reflection everywhere
where the usecase here would be to intersperse some normal lets with resource acquiring calls
In F#
, they have let conn = makeConnection()
vs. use conn = makeConnection()
, where the latter will call .Dispose on conn
at the end of the scope
you can also use with-open with a wrapper that reifies closeable as a no-op
(with-open [resource-a (...)
normal-thing (noop 5)
resource-b (... normal-thing)]
...)
and while making a 3 form macro would work, I generally work in cursive, so at least for selfish reasons "looking like let" makes the plugin work
(cmd)user=> (defmacro as-closeable [v] `(reify java.lang.AutoCloseable (close [_#]) clojure.lang.IRef (deref [_#] ~v)))
#'user/as-closeable
(ins)user=> (with-open [a (as-closeable 1) b (as-closeable 2)] (+ @a @b))
3
just "coding out loud" here so to speak, dunno if that leads to anything usable(with-open [con (jdbc/get-connection ds)]
(jdbc/execute! con ...)
(jdbc/execute! con ...)
(into [] (map :column) (jdbc/plan con ...)))
(jdbc/with-transaction [tx ds]
(jdbc/execute! tx ...)
(jdbc/execute! tx ...)
(into [] (map :column) (jdbc/plan tx ...)))
user> (def ignored (time (py/with-gil-stack-rc-context
(->> (repeatedly 1000 #(py/->py-dict {:a 1 :b 2}))
(py/->py-list)
(py/as-jvm)
(mapv py/->jvm)))))
and this kind of macro is a counterexample to the thought that this could replace with-* macros
(with [_ (py/gil-stack-rc-context)
py-list (->> (repeatedly 1000 #(py/->py-dict {:a 1 :2}))
(py/->py-list)
(py/as-jvm)
(mapv py/->jvm))]
py-list)
another option:
(with* [conn (make-connection)
_ (defer* (.close conn))
something (something-else)]
(foo conn something))
so defer*
would be a special thing inside the binding that would run that code at the end of the scope
(defmacro defer [statement]
`(reify ContextManager
(on-enter [_#])
(on-exit [_# _#] ~statement)))
I'm not sure how else to make defer*
special than to do some runtime type checking regardless
it breaks the ide since the symbol isn't actually anywhere, but also its an annoying suprise of semantics
don't manually write code that is a linearized graph, write a graph and have the computer linearize it 🙂
Just to make sure - I can't write a macro that accepts unbalanced {
, [
, and (
, right?
the hypothetical with*
looks like let to satisfy the IDE, but the only feature it adds carries the syntactic burden of the [thing binding]
part of let
since the try, except, finally
triplets need only exist for the places that actually need them
(def g (as-> {} graph
(assoc graph :open-context #{})
(assoc graph :some-operation #{:open-context})
(assoc graph :some-operation2 #{:some-operation})
(assoc graph :close-context (set ((fn f [k]
(cons k (for [k (get graph k) i (f k)] i)))
:open-context)))))
(defn topo [graph]
(when-not (empty? graph)
(lazy-seq
(let [n (first (for [[node s] graph
:when (not (seq s))]
node))]
(assert n "if this fails there is a cycle")
(cons n (topo (into {} (for [[node deps] graph
:when (not= n node)]
[node (disj deps n)]))))))))
(topo g)
;;=> (:open-context :some-operation :some-operation2 :close-context)
you define your contexts and operations on same as a graph of dependencies and then use a topo sort to flatten it out in to something to run, and never write nested lets and with-opens
because you have a literal dependency graph, a computer can easily insert the close after the final depedency
that's basically the skeleton of stuartsierra/component
@hiredman that is cool and makes sense for some cases like assembling a running system, but do you still think it makes sense for smaller things?
I've written things using it, but not for resource management, more for parallel task dependency things
it is very similar to https://github.com/plumatic/plumbing#graph-the-functional-swiss-army-knife
there's a lib for that here too: https://github.com/stuartsierra/dependency
because the code that builds the graph is responsible for composition you can do interestings with it like composing in some monad, like seqs, so your graph can become a weird kind of sequence processing thing
You know how most animals evolve an aversion to spicy or bitter foods because in the wild those are signs of poison?
see, the thing with poison is it kills you if you know about it or not, where as people with aversions to monads are happy to use them until someone points out it is a monad
a way monads are often talked about is as a computation in some context, which seems like it might be particular interesting to learn about given how you are looking around for ways to deal with contexts
I consider Monads to be a bit like Design Patterns these days -- they evolved from observation of common code structure in solutions to certain problems and then got a little more formalized and given names.
You're basically already using monads. You just don't identify them as such.
(and, yeah, monads have more underlying formalism in math than design patterns)
at least in this context, I don't think going to the level of composing monads makes sense, since alot of builtins kinda classify as monads
on the other hand, show an actual category theory expert from the math world what we call monads and they'll call you mad
which infects the code a bit more than I would like in that cats/mlet
isn't exactly a drop in solution
See any problems with this?
(with* [conn (make-connection)
:defer (.close conn)
something (make-something)]
(foo conn something))
(Since this is fine for IDEs:)
(for [i (range 5)
:when (= 0 (rem i 2))
j (range 5)
:when (= 1 (rem j 2))]
(str i j))