Fork me on GitHub
#clojure
<
2019-06-06
>
johnnyillinois00:06:02

Does anyone know of a best practice for taking a non-namespaced map (example: (def x {:a 1, :b 2, :nested {:h 3 :f 4}}) and producing the namespaced keyword version given a spec: example:

(s/def ::result (s/keys :req [:thing/a :thing/b :thing/nested]))
(s/def :thing/nested (s/keys :req [:other/f :other/g]))
and produces given x and the spec ::result => {:thing/a 1 :thing/b 2 :nested {:other/h 3 :other/f 4}}

johnnyillinois00:06:07

Or does a project already do this? For example, I would think the ring swagger coercion would do something like this.

johnnyillinois00:06:41

Maybe I am being silly and (s/keys :req-un [:thing/a]) is the right choice

Alex Miller (Clojure team)01:06:52

I would do the last - spec the data you have

danielneal11:06:30

@tonsky hey if someone wanted to implement the kind of clojure highlighting that you describe in your recent talk, but in emacs, is there anywhere they could steal bits of implementation from...?

lread12:06:43

I watched the talk too and was curious to try out the tonsky color scheme! All I found was this https://github.com/uzhne/alabaster-emacs but did not try it out yet.

danielneal11:06:32

I really like the idea of less noisy, more minimal but meaningful highlighting. At the moment I even don't use any highlighting at all, for performance when files have large maps 😄

p4ulcristian12:06:23

Hello guys! What is the easiest way of calling clojure functions from an already compiled java class?

noisesmith16:06:52

I can think of a lot of common instances (eg. (Thread. #(foo)) - but really the answer depends on what the existing Java Class does and how it goes about finding something to invoke and invoking it.

lepistane13:06:17

hi guys what is your opinion on partial functions? Do you use them often?

Olical13:06:14

I tend to just use #() or something like https://github.com/we-shop/arrows

lepistane14:06:51

Maybe i should've asked how do you deal with partial fns let say i have defined partial that i will use later on in run time how do i find out information about that partial function #function[clojure.core/partial/fn--5561] when it like this? Is there anyway to 'expand' it ? Is it bad practice to have a lot of partial fns all over the place that are defined during startup but used during run time?

Alex Miller (Clojure team)14:06:42

there is no way to expand it, it is opaque to users

Alex Miller (Clojure team)14:06:08

usually I prefer passing explicit environment over passing partials that close over it

Alex Miller (Clojure team)14:06:14

as then it's visible

👍 4
grischoun15:06:24

Is there way with spec to validate the content of a string? I.e., can I use spec to ensure that a string is made of two parts separated by a ‘:’ (e.g. “procotol:domain”)?

Alex Miller (Clojure team)15:06:04

there's an example of this in the spec guide via regex for email https://clojure.org/guides/spec

✔️ 12
grischoun15:06:46

Right. Thank you!

CyberSapiens9717:06:51

Does the usage of future has any implications/gotcha's when running the program within a JAR ?

CyberSapiens9717:06:33

It seems that it's not being called when running as a JAR

noisesmith17:06:04

whether your code is in a jar or on disk should not effect how it executes, aside from usage of File APIs that don't work for resources inside jars

noisesmith17:06:20

how are you starting the future, and what's your evidence it doesn't run?

CyberSapiens9717:06:14

yeah i'm having a problem with Files inside the resources too, and i'm trying to figure out how to create folders/files outside of the project folder/jar

CyberSapiens9717:06:43

my evidence is just a simple println inside the future, when running on disk it prints on console, and when running as a JAR it doesn't

dpsutton17:06:01

could this be the process exiting before the future is run?

noisesmith17:06:15

there are no Files inside resources, use http://clojure.java.io/resource instead of file

noisesmith17:06:51

@cybersapiens97 note that when an exception is thrown inside a future, it's reserved and doesn't do anything until the future is dereffed

🙂 4
noisesmith17:06:35

likely the code inside the future is broken when in a jar (perhaps trying to use a file, which isn't valid for things inside jars), and the exception never gets thrown because you don't access the return value of the future

seancorfield17:06:25

If you're using timbre already for logging, it has logged-future which is a replacement for future that logs any failures -- very handy!

noisesmith17:06:50

you can also use try/catch inside future for use cases where you intentionally never deref

noisesmith17:06:03

(try/catch plus log or print that is)

emilaasa17:06:53

Is there a ballpark memory consumption for a pretty trivial clojure “microservice” that has: - a few routes - very moderate traffic - returns some json 😄

noisesmith17:06:18

I'd use visualvm and see how much of the heap your program uses

noisesmith17:06:36

the java vm checks how much memory it's allowed to use and just grabs it whether it needs it or not

noisesmith17:06:45

but visualvm shows how much is actually being used

emilaasa17:06:18

Yes I’m aware, I’m thinking how much is expected for something pretty simple to run. We have java services running with -Xmx=128mb

emilaasa17:06:29

Will it be equal or more?

noisesmith17:06:31

that's definitely too low for clojure

borkdude17:06:43

I guess you can also make and endpoint and expose (/ (.freeMemory (Runtime/getRuntime)) (* 1024.0 1024))?

borkdude17:06:53

or usedMemory or whatnot

noisesmith17:06:55

use visualvm because libraries can really change the baseline, I'd expect 500megs at least, in the low gigs is normal

emilaasa17:06:13

@borkdude yeah, the problem is it’s very hard to monitor the jvm like that

emilaasa17:06:19

The gc will take whatever you give it

jumar17:06:54

Exactly, it's also not that useful to use visualvm for the same reason.

jumar17:06:43

I'm successfully running a small Clojure app with -Xmx256m but the only way to check is to run bunch of different settings and workloads and see how it works.

jumar17:06:32

Different GCs may give you different results especially in low-memory settings. I've heard that Shenandoah may be pretty good in such situations.

emilaasa17:06:45

Interesting!

emilaasa17:06:00

I think I could probably get away with 512mb but not much more

jumar17:06:14

beware that JVM memory consumption isn't just heap though

emilaasa17:06:30

Atleast not for services that are also trivial to solve in plain java or kotlin.

emilaasa17:06:22

The downside of a really high level language I guess. 🙂

emilaasa17:06:46

Kind of conflicts with my current desire to write a bunch of small focused services.

seancorfield17:06:55

FWIW, most of the processes on our QA servers run with 256m heaps just fine. Even in production with heavy traffic, most of our processes run with 1g heaps. Only a few have larger heaps (and that's just to avoid GC bottlenecks if there's a really big spike in traffic).

seancorfield17:06:32

Prior to Clojure, we used a web stack that really needed 10-15g to run smoothly so Clojure seems very lightweight to us 🙂

emilaasa17:06:21

Hehe I guess we all have different contexts - I’ve written the last few services in Go which have been very easy on the RAM

emilaasa17:06:49

And the CSP implementation actually make it sort of workable with concurrency

emilaasa18:06:59

Kind of feels like brutalist programming tho compared to clojure 🙂

seancorfield18:06:36

Yeah, non-JVM systems seem to use tiny amounts of memory by comparison. I know lots of places that run JVM processes with heaps in the tens of GB 🙂

jumar08:06:08

In the "Fearless JVM Lambdas" they also say that with Clojure you have to use at least 256M: https://youtu.be/GINI0T8FPD4?t=2262

emilaasa09:06:54

Interesting!

emilaasa09:06:21

Thanks for linking that stuff

emilaasa09:06:31

I think I have that java performance book lying around somewhere too

emilaasa09:06:45

It’s pretty much what we thought of yesterday - measuring or guessing are the two options. 🙂

noisesmith17:06:33

@borkdude I prefer visualvm because the actual numbers are constantly oscillating in a sawtooth pattern, but it's easy to see the peaks in the graph

noisesmith17:06:53

(I guess with some stats code you could capture peak / mean values but they are easy to eyeball)

emilaasa17:06:06

I think a proper test would involve performance testing the service and lowering the memory allowed for the JVM?

emilaasa17:06:16

and seeing when stuff stops working

noisesmith17:06:17

since the tool differentiates used and allocated, you can get pretty close without lowering the numbers (and also when you lower the allocation numbers below a certain point you sacrifice performance since the GC is doing more work)

emilaasa17:06:18

I’ve got some pretty zealous colleagues when it comes to RAM memory usage, and I kind of can see their point since our software runs on prem at the customers data centers

noisesmith17:06:30

they won't like clojure

emilaasa17:06:45

Hehe I’m getting that feeling 🙂

noisesmith17:06:20

there are many places where clojure explicitly decides to use more heap space (for performance, for simplicity / consistency...)

CyberSapiens9717:06:53

@noisesmith the problem was indeed the future throwing an exception. It happens that i'm trying to execute a shell command inside a folder that is inside the JAR. and this seems to be impossible

noisesmith17:06:07

there is no folder inside a jar

noisesmith17:06:12

there's a resource path

noisesmith17:06:24

if the shell needs to use your resources, you need to expand the jar

noisesmith17:06:55

(or a system level tool that knows how to mount a zip file as a file system, but that would be OS specific)

CyberSapiens9717:06:18

i don't necessarily need to access the files within the jar, i could first create them outside of it, but i'm having problems accessing anything outside the scope of the project, it doesn't accept to create a folder with the absolute path like: /home/user/folder

noisesmith17:06:51

for programmatically expanded things I'd opt for /tmp

noisesmith17:06:04

or some other file system designed for ephemeral resources created by a program

CyberSapiens9717:06:17

i was planning on using docker volumes

CyberSapiens9717:06:27

because i'm using docker to get prod ready

noisesmith17:06:35

OK, there's a /tmp inside the docker image

noisesmith17:06:14

and more importantly, it's mounted to a real file system that you can alter at runtime with a program in that image

CyberSapiens9717:06:31

alright, i'll try to write the folder/files to a /tmp

noisesmith17:06:31

(real as far as anything in your image is concerned at least)

noisesmith17:06:32

@cybersapiens97 a more specific gotcha is if you don't drop permissions, root's home dir isn't /home/root, it's /root

noisesmith17:06:40

but /tmp is more reliable anyway

CyberSapiens9717:06:18

this (.mkdir (java.io.File. folder-path)) should work with a folder-path like this: /tmp ?

noisesmith17:06:40

well, if folder-path is nested, don't use mkdir - there's one for nested dirs but it's different

CyberSapiens9717:06:41

ops, /tmp/my-folder

noisesmith17:06:04

but sure, with a single level, mkdir should work

noisesmith17:06:43

just an extra s in your code!

CyberSapiens9717:06:19

thanks for sharing this

CyberSapiens9717:06:27

but it really is a single level nesting

CyberSapiens9717:06:50

it's just a folder that i transform into a zip file later on, in the same function

CyberSapiens9718:06:28

thanks @noisesmith it worked on the JAR now, and i'm using the /tmp/ folder

👍 4
CyberSapiens9718:06:04

now my Ring middleware only serve files inside the resource folder, so i need to point this files from /tmp/ and make them available 😂

noisesmith18:06:37

@cybersapiens97 just use wrap-resource instead of wrap-file

CyberSapiens9718:06:12

i just saw it on the documentation hehe

CyberSapiens9718:06:23

i'll use both, since i need static files on the resources folder

noisesmith18:06:44

wrap-resource works on files

noisesmith18:06:57

(as long as the directory is on classpath, but that's easy to arrange)

vemv18:06:06

given a symbol 'a.b.c, how can I ask Clojure "from which file/files you would you fetch this ns?" would help me debug a possible classpath issue

vemv18:06:42

to be clear, I can just ps aux | grep classpath and see the process' classpath, but I want to see the exact thing Clojure would do

ghadi18:06:30

either clojure will load it from a/b/c.clj or if AOT'ed from a/b/c__init.class

ghadi18:06:37

so run this:

ghadi18:06:24

(require '[ :as io]) (io/resource "a/b/c.clj") (io/resource "a/b/c__init.class")

noisesmith18:06:31

rarely you should also check foo.cljc

hiredman18:06:47

and you might need to do the underscores for dashes thing

Alex Miller (Clojure team)18:06:00

there is a munge function for that

noisesmith18:06:26

it would be convenient if there was a function that did the exact lookup and showed the resource it found

hiredman18:06:43

or if verbose loading included it

ghadi18:06:53

^ that could be nice

vemv18:06:55

dev=> ( "clojure/core/specs/alpha.clj")
#object[java.net.URL 0x675c1f6b "jar:file:/Users/vemv/.m2/repository/clojure-future-spec/clojure-future-spec/1.9.0-beta4/clojure-future-spec-1.9.0-beta4.jar!/clojure/core/specs/alpha.clj"]
that was it! very likely I debugged the source of my issue (this library was overwriting the latest-and-greatest clojure.core.specs.alpha) thanks @ghadi

ghadi18:06:45

np @vemv classpath conflicts are turrible

kwladyka18:06:14

Do you know why Clojure in google trends going down since 2017? it is like 40% less.

CyberSapiens9719:06:04

it seems that on 2017 or so, there was a giant hype about Functional programming

👍 8
CyberSapiens9719:06:31

now it's becoming something that all major languages are implementing in some way or another

CyberSapiens9719:06:56

but that's just a guess

kwladyka19:06:48

I see how other languages grow, but Clojure going down. Just it worry me a little.

kwladyka19:06:59

Well, probably more, than that 🙂

CyberSapiens9719:06:28

it seems that google trends do really show a trend on functional programming from 2016 to the present

CyberSapiens9719:06:07

interesting that china has a lead on this trend, but probably because there is a billion of chinese people

kwladyka19:06:23

yes, I am surprised too

Alex Miller (Clojure team)19:06:56

fortunately, I do not program with Google, so all my Clojure programs still work great

💯 16
4
kwladyka20:06:02

What do you do on solutions like kubernetes where you create many services to reduce memory consumption in Clojure?

4
vemv20:06:31

Have you already tried trimming the dependencies on a per-service basis? e.g. all services use the same deps b/c you have some internal framework, but maybe each service actually needs a fraction of that

kwladyka20:06:13

no, it is new topic for me

kwladyka20:06:32

But I am not sure I would like to risk to do it and want to maintenance it 😉

kwladyka20:06:25

Probably will be easier paying for RAM heh 😉

vemv20:06:25

just to be clear I'm talking about plain project.clj / deps.edn dependencies. No much surgery needed Are you using a large web framework?

kwladyka20:06:54

I don’t use frameworks at all. I prefer to use small libraries instead.

kwladyka20:06:17

So for example I have service only to login user as auth-provider

kwladyka20:06:35

but while it is Clojure it will consume so much RAM for such simple thing

vemv20:06:09

> I don’t use frameworks at all. I prefer to use small libraries instead. then it might be worth a shot. lein deps :tree might reveal where the bloat comes from (if you are complaining about baseline clj RAM consumption, then I have no idea)

kwladyka20:06:18

Probably Clojure is not a good choice for such services, but I don’t feel confident to write it in for example golang

kwladyka20:06:43

yes, it is a little about baseline 😉

kwladyka20:06:32

but not only, just I would like to listen somebody experienced in this topic

👍 4
vemv20:06:32

perhaps there is some useful insight in the Microservices with Clojure book. Haven't got my hands to it yet

kwladyka20:06:58

But yeah… Probably the true is we should use another languages for small services or pay more for RAM 🙂

👍 4
kwladyka20:06:03

there will be no magic

vemv20:06:35

worst-case scenario you can resort to cljs / node.js 😜

kwladyka20:06:11

Paying for RAM is not so bad 😛 😉

👌 4
kwladyka20:06:02

I am thinking about it, because I am doing my own hobby project, which will became something serious maybe. So I will pay for everything from my own pocket.

kwladyka20:06:42

Then you start to count this “small” bills and it matter heh

kwladyka20:06:35

I am going sleep. Goodnight.

🌙 4
vemv20:06:18

...the issue above left me quite frustrated (~90m wasted) so probably I will implement a linter based on this snippet:

(defn resources
  ([n]
   (resources n (.getContextClassLoader (Thread/currentThread))))
  
  ([n ^ClassLoader loader]
   (.getResources loader n)))

(->> "clojure/core/specs/alpha.clj"
     resources
     enumeration-seq
     (map str)
     (clojure.pprint/pprint))

;; the above results in:
'("jar:file:/Users/vemv/.m2/repository/clojure-future-spec/clojure-future-spec/1.9.0-beta4/clojure-future-spec-1.9.0-beta4.jar!/clojure/core/specs/alpha.clj"
  "jar:file:/Users/vemv/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar!/clojure/core/specs/alpha.clj")
• list every ns declared/required in your project, fetch their resources • if for any given ns there is more than one entry, issue a warning

borkdude20:06:52

I tried to find where RELEASE and LATEST are documented but then I read they are deprecated in maven 3? How does this relate exactly to clojure deps and lein, since they use the maven stuff under the hood?

clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "LATEST"}}}' -e '(clojure-version)'

Alex Miller (Clojure team)20:06:20

but I consider them to be unsupported from a clj / tools.deps pov

Alex Miller (Clojure team)20:06:38

because they are not fixed versions

borkdude20:06:10

yeah, I usually only use it for one-off things in the command line when I don’t feel like looking up a version number. I didn’t even know LATEST existed until today

seancorfield21:06:47

clj-new produces deps like that so that I don't have to keep updating clj-new's templates when new versions of libraries drop. I figured folks would change the deps to specific versions if they wanted.

seancorfield21:06:42

At work, we have some dev-only tooling that uses RELEASE so we always pick up the newest versions. If anything breaks in dev, we can easily pin a tool back to a working version.

seancorfield21:06:53

(I don't use LATEST for anything tho')

Iwo Herka21:06:55

Hi, what would you say is the essential Clojure testing stack? spec, test.check, clojure.test - anything else?

robertfw21:06:14

There's a lot to factor into that - what are you testing? where? (repl, cli, ci?)

robertfw21:06:14

We currently use all of the above + test.chuck and eftest

vemv21:06:06

etaoin if developing web apps aero or such for having clean per-env config component/integrant for having clean per-env systems expound more exotic: circleci.test, kaocha

Iwo Herka21:06:57

@robertfrederickwarner Im talking about CI for medium-to-large commercial projects.

Iwo Herka21:06:08

Thanks for the info.

seancorfield21:06:14

@hi135 We have about 80k lines of Clojure at work. About 20k of that is test code. We use clojure.test and Cognitect's test-runner (since we're all clj/`deps.edn`-based). We use clojure.spec but mostly in production code for data validation, although we do use some instrumentation in dev/test, and we do some generative testing with it (so we use test.check under the hood as well).

seancorfield22:06:49

We also use https://github.com/clojure-expectations/clojure-test (I prefer the expect style syntax) -- we used to use the core Expectations library but it wasn't compatible with clojure.test-based tooling.

Iwo Herka22:06:14

Thanks, that's very helpful. I know what to research.

seancorfield22:06:40

We used to have Clojure-based Selenium/WebDriver tests -- and we used clj-webdriver for that but we've killed off those web apps and that library is pretty much unmaintained now. I'd probably look at etaoin if I needed that these days. We do have some HtmlUnit-based tests, just using clojure.test for assertions and plain Java interop.

seancorfield22:06:19

And, like @robertfrederickwarner, we use test.chuck for some stuff above and beyond test.check (mostly for string regex generators).

Iwo Herka22:06:36

Noted. I'm checking, because I'm preparing a pitch for the company. The big focus is on testing. Right now a lot of code is in Python which, as far as property-based testing is concerned, only has hypothesis.

Iwo Herka22:06:15

Clojure seems to be much more powerful in this area.

noisesmith22:06:06

also, for circleci specifically, there's a circleci test namespace that runs clojure.test tests but has some extra ci-friendly features

noisesmith22:06:17

circle uses clojure internally and supports it well

Iwo Herka22:06:41

What about Midje? Is it recommended?

ghadi22:06:45

Not recommended @hi135

👍 8