Fork me on GitHub
#clojure
<
2020-07-07
>
quadron00:07:25

is there a library of specs for frequently occuring entities (for example representation of a person username password email ...) ?

seancorfield00:07:42

@veix.q5 Not that I know of -- and I really don't think you could have reusable specs for most of those things in any meaningful way.

seancorfield00:07:22

Pretty much every system is going to have slightly different rules about such things.

seancorfield01:07:24

Username, for example, is going to have all sorts of site-specific rules around it -- different maximum/minimum lengths, different sets of allowable characters. Same with passwords.

seancorfield01:07:39

Email could potentially be reused but the RFC for what's valid is extremely complex so getting an accurate spec is hard -- and I know of several sites that do not accept perfectly valid email addresses, just because they didn't know the RFC.

✔️ 3
phronmophobic01:07:44

i dunno. that sounds like it might be an interesting idea. I don't think you'll ever have everybody use the exact same specifications for some subdomain (eg. user accounts), but I could see the use for some sort of standard for a given subdomain and having multiple projects that reuse it.

👍 3
seancorfield01:07:12

This regex is fairly close to the RFC I believe (as an example of the complexity):

(def email-regex
  "Sophisticated regex for validating an email address."
  (re-pattern
   (str "(([^<>()\\[\\]\\\\.,;:\\s@\"]+(\\.[^<>()\\[\\]\\\\.,;:\\s@\"]+)*)|"
        "(\".+\"))@((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\])|"
        "(([a-zA-Z\\-0-9]+\\.)+[a-zA-Z]{2,}))")))
and even this doesn't account for allowable non-Latin characters in domains (I think, maybe... I dunno, regexes are hard!).

eskos07:07:46

Ah, this good old thing 🙂 Funnily enough there’s even a domain about this. https://emailregex.com/

seancorfield01:07:42

Also, the generators for such specs probably couldn't be agreed on... I mean, we already have different generators for different test contexts...

3
miro09:07:45

Hi folks! A quick (yet hardcore ) question: any idea why is clojure eval much much faster than java compilation? Running clojure eval takes milliseconds whereas using javax.tools.JavaCompiler to compile and run a tiny java code takes seconds in general. I am suspecting that since JavaCompiler class seems to be reliant on StandardJavaFileManager , some 💾 I/O might be at play here (even though the java code itself is in memory, is maybe the .class created on disk?)... Just curious if anybody around have ever played with it or investigated further?

jumar09:07:49

What implementation of JavaCompiler do you use and how do you call it? JavacTool seems to be an internal class.

miro10:07:17

@U06BE1L6T seems like it is using the internal JDK one - obtained via ToolProvider.getSystemJavaCompiler() . Also seems like the I/O might not be the issue, I am trying to use an in memory file manager...

jumar10:07:52

It could help if you provide a specific example. I quickly tried this and it takes about ~500 ms on my machine: https://github.com/jumarko/java-experiments/blob/master/src/main/java/net/curiosprogrammer/javaexperiments/compiler/JavaCompiler.java

jumar11:07:37

And here's the same thing in Clojure - subsequent runs are usually about 300 ms: https://github.com/jumarko/clojure-experiments/blob/master/src/clojure_experiments/java/compilation.clj#L12

jumar11:07:59

So I'd say it has a lof of overhead (plugins initialization, walking file system, zip file handling) for tiny things but should get better once you try to compile larger set of files.

miro11:07:48

Thanks @U06BE1L6T, seems like you are observing the same thing - I basically have hard time optimizing the compilation below 100s of ms. Seems like JavaCompiler is basically just built that way and not optimized performance wise (and I understand that historically there was no need for that). It is quite striking difference when you then take clojure which also runs on JVM and load any clojure source code using load-string or eval it often just takes <10ms. And it includes class generation so basically it does everything java compiler does... Just 10x-100x faster...

Mikael Andersson11:07:19

Did heinous acts that included class loaders and java compilation years ago, and I concur. The overhead for small file sets is quite big. I seem to recall there either was some (reasonable) trick to speed up subsequent compilations, or a different compiler that could do incremental compile. The reason however is probably less mysterious, I assume eval use the existing class loaders instead of loading the dependent classes from the jar files again?

Mikael Andersson11:07:49

Javac needs to be able to run on one set of dependencies / classpath, and compile using another. More obvious when annotation processors are used.

miro11:07:35

@UQRPN1XGW yeah definitely the class loading adds to that - basically if you want to make it work with your current classpath, you have to parse the classpath string, get cannonical path for each jar on it and the compiler will then load them all over again! But it seems that even with just very few jars the compilation time is in 100s of ms, so I would expect more bottlenecks...

Mikael Andersson12:07:20

How portable/sane do you need the solution to be? If I simply were investigating, I would try to run using unpacked jars (raw classdiles) from a memory backed device on linux as a first step.

Mikael Andersson12:07:35

If I only wanted a proof of concept, I'd look into if I could somehow reuse the loaded classes between runs, possibly using terrifying reflection/class loader shenanigans if the API doesn't support it. Mostly to get a lower bound on the actual compilation. For a while there were several different java compilers to try, but since generics I believe most disappeared. Have been out of the Java loop for a bit, so I'm not exactly sure of the state.

miro18:07:41

@UQRPN1XGW basically my conclusion at this stage is that it is not worth the effort 🙂 just wanted to compile new pieces of java code during runtime and to make it perform better than what JavaCompiler currently offers seems like way too much effort at this stage - will just accept it as a limitation for now

Mikael Andersson20:07:39

@U5L1P2D9U As I haven't done a lot of Java interop from clojure I thought it could be a good idea to try some, and as I noticed the "new" compiler API (not new at all, but new since I touched it) actually provided a bit of machinery to cache paths I went ahead and cobbled together an ugly hack that does that: https://gist.github.com/mickronome/27d3df3898924608facaf0584b55ec5c Doesn't make a huge difference using the sample in question, afaict, on my machine the compilation stabilises at about 40-50ms.

miro20:07:22

Thanks @UQRPN1XGW, will have a look!

👍 3
Mikael Andersson12:07:35

If I only wanted a proof of concept, I'd look into if I could somehow reuse the loaded classes between runs, possibly using terrifying reflection/class loader shenanigans if the API doesn't support it. Mostly to get a lower bound on the actual compilation. For a while there were several different java compilers to try, but since generics I believe most disappeared. Have been out of the Java loop for a bit, so I'm not exactly sure of the state.

Yehonathan Sharvit13:07:55

I just listened to a great episode of Clojurescript Podcast with @alexmiller where he describes the differences between tools.deps and maven when it comes to transitive deps resolution. Is the exact deps resolution implemented by tools.deps documented somewhere?

Alex Miller (Clojure team)13:07:43

no, my conj talk is probably the best overview

Yehonathan Sharvit13:07:00

Would you like to include a written documentation somewhere if I would write it?

Alex Miller (Clojure team)13:07:16

I would much prefer to write it myself :)

dpsutton13:07:27

I've wanted to come up with an example where the same deps under maven's resolution does one behavior and under deps resolution does another.

Alex Miller (Clojure team)13:07:12

it's pretty easy to do that, but you have to have at least one level of transitive dep

Alex Miller (Clojure team)13:07:05

b/c maven prefers dep versions at the top, and tools.deps treats top deps specially as well, they should always give you the same top level deps

Alex Miller (Clojure team)13:07:26

given:

->A->Cv1
->B->Cv2

Alex Miller (Clojure team)13:07:51

Maven will encounter Cv1 first and use it. tools.deps will use Cv2

Alex Miller (Clojure team)13:07:11

that's about the simplest case

dpsutton13:07:14

yeah. there are some plugins and some scripts in project.clj which translate between the two. just wondering how hard it was to create one "in the wild". I know meander and clojure-lsp both do this off the top of my head in project.clj. In addition to any apps that might dev with deps and deploy/jar with lein

dpsutton13:07:04

feel like a PoC might point this out to people. As it stands the deployed artifacts are potentially distinct from the dev time artifacts

Alex Miller (Clojure team)13:07:42

I've debugged through a few way more complicated ones. in general, I'd say many large dep graphs will end up with different results, but they are the same range of differences you would get with Maven too, depending on order and in general most people don't notice those either.

Alex Miller (Clojure team)13:07:35

the artifacts are the same, but transitive resolution at dev and use time are almost always different

Alex Miller (Clojure team)13:07:30

for libs, we rarely test with the same lib set that our lib is run with, which should be disturbing to you

dpsutton13:07:24

i hadn't considered that and now it is haha

Alex Miller (Clojure team)14:07:26

and really highlights how a growth not breakage mindset is even more important

Yehonathan Sharvit13:07:35

I see. What are you waiting for @alexmiller 🙂 ?

Alex Miller (Clojure team)13:07:43

having 47 hours in a day

Yehonathan Sharvit13:07:03

I can borrow you some hours from my day (I am currently at 20/24)

Alex Miller (Clojure team)13:07:15

I keep getting interrupted by people asking me questions on slack ;)

😄 12
Yehonathan Sharvit13:07:26

Hahaha. Sorry about that

Yehonathan Sharvit13:07:47

Stop creating awesome tools and people will ask less questions

❤️ 3
wombawomba17:07:25

does anyone know if there’s an easy way to set up a named pipe in Clojure?

noisesmith17:07:28

there might be something in java.nio, I wouldn't look for a dedicated clojure wrapper for something this specific

noisesmith17:07:15

there's java.nio.Pipe for inside one jvm, but I'm sure what you want is the OS level feature

wombawomba17:07:45

yeah I was after the OS level feature

wombawomba17:07:04

and yeah, if there’s a (non-Clojure) Java solution that’s fine too

noisesmith17:07:57

you can use jna to create one, or if one already exists (or was created via eg. a shell call) you can use it via RandomAccessFile

wombawomba18:07:15

alright, I’ll look into that – thanks!

danielglauser21:07:21

Dumb question, how do I get a single backslash to show up in a string?

(str "\ ")
Unsupported escape character: \

(str "\\ ")
\\ "

(str "\\\ ")
Unsupported escape character: \

(str "\\\\ ")
"\\\\ "

nahuel21:07:02

(println "\\")

ghadi21:07:40

user=> (println "\\")
\
nil
user=> (prn "\\")
"\\"
nil

nahuel21:07:53

when it shows as an string object, you see it quoted "\\" , but it really only have a single \

danielglauser22:07:00

Gotcha, the double backslash will work with println.

ghadi22:07:03

when you eval a string, it's as if it was prn ed.

ghadi22:07:25

No -- neither println nor prn care about their argument

ghadi22:07:17

to represent a backslash in a string, you have to escape it (using the escape char, which is a backslash)

ghadi22:07:13

prn prints a readable representation (which has the escape)

ghadi22:07:29

so does typing a literal string at the REPL

3
lilactown22:07:41

how do I implement count on my own data type?

dpsutton22:07:22

(count (reify clojure.lang.Counted (count [_] 500)))

👍 3
❤️ 3