This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-06-06
Channels
- # announcements (1)
- # beginners (147)
- # boot (9)
- # calva (28)
- # cider (3)
- # circleci (18)
- # cljdoc (54)
- # cljs-dev (55)
- # cljsrn (22)
- # clojure (179)
- # clojure-canada (9)
- # clojure-dev (74)
- # clojure-europe (1)
- # clojure-italy (15)
- # clojure-nl (7)
- # clojure-spec (30)
- # clojure-uk (55)
- # clojurescript (65)
- # core-async (15)
- # cursive (12)
- # datomic (16)
- # events (4)
- # fulcro (25)
- # graalvm (3)
- # joker (2)
- # kaocha (15)
- # keechma (94)
- # off-topic (12)
- # qlkit (2)
- # re-frame (15)
- # reagent (11)
- # reitit (29)
- # remote-jobs (15)
- # rewrite-clj (16)
- # shadow-cljs (73)
- # spacemacs (151)
- # sql (3)
- # tools-deps (11)
- # unrepl (19)
- # vim (35)
Does anyone know of a best practice for taking a non-namespaced map (example: (def x {:a 1, :b 2, :nested {:h 3 :f 4}})
and producing the namespaced keyword
version given a spec:
example:
(s/def ::result (s/keys :req [:thing/a :thing/b :thing/nested]))
(s/def :thing/nested (s/keys :req [:other/f :other/g]))
and produces given x
and the spec ::result
=> {:thing/a 1 :thing/b 2 :nested {:other/h 3 :other/f 4}}
Or does a project already do this? For example, I would think the ring swagger coercion would do something like this.
Maybe I am being silly and (s/keys :req-un [:thing/a])
is the right choice
I would do the last - spec the data you have
@tonsky hey if someone wanted to implement the kind of clojure highlighting that you describe in your recent talk, but in emacs, is there anywhere they could steal bits of implementation from...?
I watched the talk too and was curious to try out the tonsky color scheme! All I found was this https://github.com/uzhne/alabaster-emacs but did not try it out yet.
I really like the idea of less noisy, more minimal but meaningful highlighting. At the moment I even don't use any highlighting at all, for performance when files have large maps 😄
Hello guys! What is the easiest way of calling clojure functions from an already compiled java class?
I can think of a lot of common instances (eg. (Thread. #(foo))
- but really the answer depends on what the existing Java Class does and how it goes about finding something to invoke and invoking it.
I tend to just use #()
or something like https://github.com/we-shop/arrows
Maybe i should've asked how do you deal with partial fns
let say i have defined partial that i will use later on in run time
how do i find out information about that partial function #function[clojure.core/partial/fn--5561]
when it like this? Is there anyway to 'expand' it ?
Is it bad practice to have a lot of partial fns all over the place that are defined during startup but used during run time?
there is no way to expand it, it is opaque to users
usually I prefer passing explicit environment over passing partials that close over it
Is there way with spec to validate the content of a string? I.e., can I use spec to ensure that a string is made of two parts separated by a ‘:’ (e.g. “procotol:domain”)?
there's an example of this in the spec guide via regex for email https://clojure.org/guides/spec
Does the usage of future
has any implications/gotcha's when running the program within a JAR ?
It seems that it's not being called when running as a JAR
whether your code is in a jar or on disk should not effect how it executes, aside from usage of File
APIs that don't work for resources inside jars
how are you starting the future, and what's your evidence it doesn't run?
yeah i'm having a problem with Files inside the resources too, and i'm trying to figure out how to create folders/files outside of the project folder/jar
my evidence is just a simple println
inside the future, when running on disk it prints on console, and when running as a JAR it doesn't
there are no Files inside resources, use http://clojure.java.io/resource instead of file
@cybersapiens97 note that when an exception is thrown inside a future, it's reserved and doesn't do anything until the future is dereffed
likely the code inside the future is broken when in a jar (perhaps trying to use a file, which isn't valid for things inside jars), and the exception never gets thrown because you don't access the return value of the future
If you're using timbre
already for logging, it has logged-future
which is a replacement for future
that logs any failures -- very handy!
you can also use try/catch inside future for use cases where you intentionally never deref
(try/catch plus log or print that is)
Is there a ballpark memory consumption for a pretty trivial clojure “microservice” that has: - a few routes - very moderate traffic - returns some json 😄
I'd use visualvm and see how much of the heap your program uses
the java vm checks how much memory it's allowed to use and just grabs it whether it needs it or not
but visualvm shows how much is actually being used
Yes I’m aware, I’m thinking how much is expected for something pretty simple to run. We have java services running with -Xmx=128mb
that's definitely too low for clojure
I guess you can also make and endpoint and expose (/ (.freeMemory (Runtime/getRuntime)) (* 1024.0 1024))
?
use visualvm because libraries can really change the baseline, I'd expect 500megs at least, in the low gigs is normal
I'm successfully running a small Clojure app with -Xmx256m but the only way to check is to run bunch of different settings and workloads and see how it works.
Different GCs may give you different results especially in low-memory settings. I've heard that Shenandoah may be pretty good in such situations.
Kind of conflicts with my current desire to write a bunch of small focused services.
FWIW, most of the processes on our QA servers run with 256m heaps just fine. Even in production with heavy traffic, most of our processes run with 1g heaps. Only a few have larger heaps (and that's just to avoid GC bottlenecks if there's a really big spike in traffic).
Prior to Clojure, we used a web stack that really needed 10-15g to run smoothly so Clojure seems very lightweight to us 🙂
Hehe I guess we all have different contexts - I’ve written the last few services in Go which have been very easy on the RAM
Yeah, non-JVM systems seem to use tiny amounts of memory by comparison. I know lots of places that run JVM processes with heaps in the tens of GB 🙂
@U6T7M9DBR I haven't found much info on the topic but this might help: https://plumbr.io/blog/memory-leaks/how-to-estimate-memory-consumption
In the "Fearless JVM Lambdas" they also say that with Clojure you have to use at least 256M: https://youtu.be/GINI0T8FPD4?t=2262
It’s pretty much what we thought of yesterday - measuring or guessing are the two options. 🙂
@borkdude I prefer visualvm because the actual numbers are constantly oscillating in a sawtooth pattern, but it's easy to see the peaks in the graph
(I guess with some stats code you could capture peak / mean values but they are easy to eyeball)
I think a proper test would involve performance testing the service and lowering the memory allowed for the JVM?
since the tool differentiates used and allocated, you can get pretty close without lowering the numbers (and also when you lower the allocation numbers below a certain point you sacrifice performance since the GC is doing more work)
I’ve got some pretty zealous colleagues when it comes to RAM memory usage, and I kind of can see their point since our software runs on prem at the customers data centers
they won't like clojure
there are many places where clojure explicitly decides to use more heap space (for performance, for simplicity / consistency...)
@noisesmith the problem was indeed the future throwing an exception. It happens that i'm trying to execute a shell command inside a folder that is inside the JAR. and this seems to be impossible
there is no folder inside a jar
there's a resource path
if the shell needs to use your resources, you need to expand the jar
(or a system level tool that knows how to mount a zip file as a file system, but that would be OS specific)
i don't necessarily need to access the files within the jar, i could first create them outside of it, but i'm having problems accessing anything outside the scope of the project, it doesn't accept to create a folder with the absolute path like: /home/user/folder
for programmatically expanded things I'd opt for /tmp
or some other file system designed for ephemeral resources created by a program
i was planning on using docker volumes
because i'm using docker to get prod ready
OK, there's a /tmp inside the docker image
and more importantly, it's mounted to a real file system that you can alter at runtime with a program in that image
alright, i'll try to write the folder/files to a /tmp
(real as far as anything in your image is concerned at least)
@cybersapiens97 a more specific gotcha is if you don't drop permissions, root's home dir isn't /home/root, it's /root
but /tmp is more reliable anyway
this (.mkdir (java.io.File. folder-path))
should work with a folder-path like this: /tmp
?
well, if folder-path is nested, don't use mkdir - there's one for nested dirs but it's different
ops, /tmp/my-folder
but sure, with a single level, mkdir should work
https://docs.oracle.com/javase/7/docs/api/java/io/File.html#mkdirs() creates all parents as needed
just an extra s in your code!
thanks for sharing this
but it really is a single level nesting
it's just a folder that i transform into a zip file later on, in the same function
thanks @noisesmith it worked on the JAR now, and i'm using the /tmp/ folder
now my Ring middleware only serve files inside the resource folder, so i need to point this files from /tmp/ and make them available 😂
@cybersapiens97 just use wrap-resource
instead of wrap-file
i just saw it on the documentation hehe
i'll use both, since i need static files on the resources folder
wrap-resource works on files
(as long as the directory is on classpath, but that's easy to arrange)
given a symbol 'a.b.c
, how can I ask Clojure "from which file/files you would you fetch this ns?"
would help me debug a possible classpath issue
to be clear, I can just ps aux | grep classpath
and see the process' classpath, but I want to see the exact thing Clojure would do
rarely you should also check foo.cljc
there is a munge function for that
it would be convenient if there was a function that did the exact lookup and showed the resource it found
dev=> ( "clojure/core/specs/alpha.clj")
#object[java.net.URL 0x675c1f6b "jar:file:/Users/vemv/.m2/repository/clojure-future-spec/clojure-future-spec/1.9.0-beta4/clojure-future-spec-1.9.0-beta4.jar!/clojure/core/specs/alpha.clj"]
that was it! very likely I debugged the source of my issue (this library was overwriting the latest-and-greatest clojure.core.specs.alpha
)
thanks @ghadiDo you know why Clojure in google trends going down since 2017? it is like 40% less.
it seems that on 2017 or so, there was a giant hype about Functional programming
now it's becoming something that all major languages are implementing in some way or another
but that's just a guess
it seems that google trends do really show a trend on functional programming from 2016 to the present
interesting that china has a lead on this trend, but probably because there is a billion of chinese people
fortunately, I do not program with Google, so all my Clojure programs still work great
What do you do on solutions like kubernetes where you create many services to reduce memory consumption in Clojure?
Have you already tried trimming the dependencies on a per-service basis? e.g. all services use the same deps b/c you have some internal framework, but maybe each service actually needs a fraction of that
just to be clear I'm talking about plain project.clj / deps.edn dependencies. No much surgery needed Are you using a large web framework?
> I don’t use frameworks at all. I prefer to use small libraries instead.
then it might be worth a shot. lein deps :tree
might reveal where the bloat comes from
(if you are complaining about baseline RAM consumption, then I have no idea)
Probably Clojure is not a good choice for such services, but I don’t feel confident to write it in for example golang
perhaps there is some useful insight in the Microservices with Clojure
book. Haven't got my hands to it yet
But yeah… Probably the true is we should use another languages for small services or pay more for RAM 🙂
I am thinking about it, because I am doing my own hobby project, which will became something serious maybe. So I will pay for everything from my own pocket.
Hi. I have no idea about this stuff, but I found that maybe OpenJ9 could help: https://technology.amis.nl/2018/12/09/jvm-performance-openj9-uses-least-memory-graalvm-most-openjdk-distributions-differ/#prettyPhoto?
...the issue above left me quite frustrated (~90m wasted) so probably I will implement a linter based on this snippet:
(defn resources
([n]
(resources n (.getContextClassLoader (Thread/currentThread))))
([n ^ClassLoader loader]
(.getResources loader n)))
(->> "clojure/core/specs/alpha.clj"
resources
enumeration-seq
(map str)
(clojure.pprint/pprint))
;; the above results in:
'("jar:file:/Users/vemv/.m2/repository/clojure-future-spec/clojure-future-spec/1.9.0-beta4/clojure-future-spec-1.9.0-beta4.jar!/clojure/core/specs/alpha.clj"
"jar:file:/Users/vemv/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar!/clojure/core/specs/alpha.clj")
• list every ns declared/required in your project, fetch their resources
• if for any given ns there is more than one entry, issue a warningI tried to find where RELEASE
and LATEST
are documented but then I read they are deprecated in maven 3?
How does this relate exactly to clojure deps and lein, since they use the maven stuff under the hood?
clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "LATEST"}}}' -e '(clojure-version)'
they still work
but I consider them to be unsupported from a clj / tools.deps pov
because they are not fixed versions
yeah, I usually only use it for one-off things in the command line when I don’t feel like looking up a version number. I didn’t even know LATEST
existed until today
clj-new
produces deps like that so that I don't have to keep updating clj-new
's templates when new versions of libraries drop. I figured folks would change the deps to specific versions if they wanted.
At work, we have some dev-only tooling that uses RELEASE
so we always pick up the newest versions. If anything breaks in dev, we can easily pin a tool back to a working version.
(I don't use LATEST
for anything tho')
Hi, what would you say is the essential Clojure testing stack? spec, test.check, clojure.test - anything else?
etaoin if developing web apps aero or such for having clean per-env config component/integrant for having clean per-env systems expound more exotic: circleci.test, kaocha
@robertfrederickwarner Im talking about CI for medium-to-large commercial projects.
@hi135 We have about 80k lines of Clojure at work. About 20k of that is test code. We use clojure.test
and Cognitect's test-runner
(since we're all clj
/`deps.edn`-based). We use clojure.spec
but mostly in production code for data validation, although we do use some instrumentation in dev/test, and we do some generative testing with it (so we use test.check
under the hood as well).
We also use https://github.com/clojure-expectations/clojure-test (I prefer the expect
style syntax) -- we used to use the core Expectations library but it wasn't compatible with clojure.test
-based tooling.
We used to have Clojure-based Selenium/WebDriver tests -- and we used clj-webdriver
for that but we've killed off those web apps and that library is pretty much unmaintained now. I'd probably look at etaoin if I needed that these days. We do have some HtmlUnit
-based tests, just using clojure.test
for assertions and plain Java interop.
And, like @robertfrederickwarner, we use test.chuck
for some stuff above and beyond test.check
(mostly for string regex generators).
Noted. I'm checking, because I'm preparing a pitch for the company. The big focus is on testing. Right now a lot of code is in Python which, as far as property-based testing is concerned, only has hypothesis
.
also, for circleci specifically, there's a circleci test namespace that runs clojure.test tests but has some extra ci-friendly features
circle uses clojure internally and supports it well