Fork me on GitHub
#clojure
<
2019-12-18
>
yuhan06:12:50

I get uncomfortable about using :type or :ns-qualified/typeas a map key, because it means that destructuring it using :keys [type] would locally shadow the clojure.core/type function. Is this a justified concern or something people regularly do? Having to do something like {typ :type} each time is even clunkier.

potetm15:12:02

I basically always {t :type} for this. I don’t find it clunky.

potetm15:12:28

In fact, it’s fewer characters than {:keys [type]}, not more 😉

potetm15:12:37

It is a justified concern, IMO. I’ve been bitten far too many times by this avoidable mistake.

potetm15:12:21

(I also usually work in an editor that highlights un-resolved symbols, and I put a lot of trust into it.)

seancorfield06:12:10

@qythium I think there are certain code function names that people regularly shadow. I don't worry about shadowing type since I almost never use it in production code (just in the REPL while exploring).

4
seancorfield06:12:24

I think name is another thing that people regularly shadow. And if you're destructuring a map, you can always use the form that maps keys to different symbols (although I don't like that much).

seancorfield06:12:54

If your functions are short, shadowing core names isn't really a problem -- the context is obvious (fits easily on the screen) and you're unlikely to have both invocations of the core function and usage of your local binding that shadows it in the same function, since you can delegate to another small function with an argument name that doesn't shadow the core function you plan to use on it.

vemv07:12:26

While I wouln't oppose shadowing for the sake of it, it's also true that shadowing code can be a bit fragile; as code changes over time, you might remove or break the destructuring, and type would suddenly refer to clojure.core's So I tend to {user-type :type}, which isn't as ugly as {typ :type} (`user` being some business-specific concept) Aside from type and name, suspects include identity, key... the list might go on, so for me it's easier to avoid shadowing altogether than keeping a mental list of acceptable cases

vemv07:12:53

https://github.com/jonase/eastwood provides automated shadowing analysis btw!

joefromct08:12:52

I feel like i'm missing something with clojure.java-time.... does anyone understand why this returns 2020 as the year? I'm very confused...

(defn change-ds-format [in-ds]
  {:pre [string? in-ds ]}
  (let [input-format "yyyy-MM-dd'T'HH:mm:ss"  
        output-format "MM/dd/YYYY hh:mm:ss a"]
    (->> in-ds
         (jt/local-date-time input-format)
         (jt/format output-format)))) 

(change-ds-format "2019-12-29T00:00:00" )
;=> "12/29/2020 12:00:00 AM"
;; Why is this one 2020 ???

solf08:12:56

This is a wild guess as I don't remember the date format strings: is it correct to have the year the input yyyy and YYYY in the output?

joefromct08:12:24

yeah you are right... "week base era " vs "year of era" Thanks for the tip.

dabrazhe09:12:15

Hi, Can someone recommend good resources on building scaleable and resilient Clojure apps, with modern CI/CI, Kubernetes deployments etc.? I can't seem to find much on the subject.

hkjels10:12:03

I can’t really recommend it, since I haven’t looked at it yet. But it’s atleast on topic: https://github.com/resilience4clj

vemv10:12:03

Recently i subscribed to https://learning.oreilly.com/home/ which offers a free trial. You can find all (?) Clojure books over there, including Microservices with Clojure (which piques my interest, haven't read it though)

noisesmith17:12:30

most of this will be the same instructions you follow for java - you are deploying a library that runs on the jvm

dabrazhe09:12:41

Thank you all! @U051SS2EU This is about the same conclusion I am drawing: create a modular app and deploy it to Kubernetes as you would with a Java app

vemv10:12:35

https://polylith.gitbook.io/ (from clj-minded folks) has some thoughts on modular apps I think it influenced me (I do modularity) but don't remember tbh

kwladyka11:12:34

@dennisa What exactly are you looking? ci/cd and k8s are independent things from Clojure and you wouldn’t find anything as Clojure k8s or something like that.

dabrazhe12:12:41

I guess I am looking for best practices on architecture and deployment, and how do they play together . Most of the articles I was able to find involve ring or luminous on the servers side, and they end with basic examples. Re deployment, most of the stuff is about deploying java apps with docker. I am not sure if these all are up to date and reflect the modern Clojure vision.

jaihindhreddy12:12:40

AFAIK if you make an uberjar of your Clojure app, then the deploy and ops aspect of that app is exactly the same as any Java app, and so all the best practices would apply. There are good arguments as to why not to make uberjars and instead run your app with the clojure CLI program. Even still, most of the things are the same. It looks to me like DevOps is moving towards some of the same ideas that underly Clojure. Immutable architecture and GitOps in particular is functional programming at scale IMHO. As for concrete tutorials, there definitely aren't as many that talk about Clojure as there are other languages. Sorry for being so wordy 😅

kwladyka15:12:49

You have to get experience from ci/cd, k8s (DevOps) and Clojure separately and connect it by wisdom. Seriously, you wouldn’t find anything deep in google about that,.

kwladyka15:12:11

As a first step learn how to make Dockerfiles from Clojure app

kwladyka15:12:27

Learn k8s, it will take you 1-2 years to do it good heh. Seriously 🙂

kwladyka15:12:49

But to start with that probably 3 months?

kwladyka15:12:59

Mainly it is about choose tools and make choices

kwladyka15:12:31

but it is not like a special choices for Clojure, it doesn’t matter what language you will use

hindol20:12:05

Building a Docker image can be super simple with JUXT's pack.alpha. Thought I would mention it.

hindol20:12:09

If you are using deps.edn.

markxnelson15:12:01

hi all, i am having a problem with re-find - on some files it just seems to hang forever and nothing happens.. here is a way to reproduce the issue:

;; here is my problem
;; using Clojure 1.10.1

;; download this file and untar/zip it
;; 

;; you need to update the form below to point to the babel.min.js file
;; in the directory where you unzipped the archive

;; according to `file`, this file is:
;; babel.min.js: UTF-8 Unicode text, with very long lines, with no line terminators, with escape sequences
(def line
  (first
    (with-open [r ( "/tmp/mark/package/babel.min.js")]
      (doall
        (line-seq r)))))

(def re #".*[Cc]opyright.*(19|20)[0-9][0-9].*")

;; when you run this form, it will hang the repl/program and never return
(re-find re line)

;; for contrast - if you run this on the file `babel.js` instead, it will
;; return quickly (with no output) which is correct
i did take a thread dump, and the interesting part is this:
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f669c08c800 nid=0x8be in Object.wait() [0x00007f6669704000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000719608ed0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x0000000719608ed0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f669c08a000 nid=0x8bd in Object.wait() [0x00007f6669805000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000719606bf8> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x0000000719606bf8> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007f669c00a800 nid=0x8b3 runnable [0x00007f66a27f2000]
   java.lang.Thread.State: RUNNABLE
	at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3799)
	at java.util.regex.Pattern$Curly.match0(Pattern.java:4274)
	at java.util.regex.Pattern$Curly.match(Pattern.java:4236)
	at java.util.regex.Pattern$Start.match(Pattern.java:3463)
	at java.util.regex.Matcher.search(Matcher.java:1248)
	at java.util.regex.Matcher.find(Matcher.java:637)
	at clojure.core$re_find.invokeStatic(core.clj:4905)
	at clojure.core$re_find.invokeStatic(core.clj:4898)
so it looks like it is stuck in the match method

andy.fingerhut15:12:04

I could check this myself soon, but out of curiosity, how long is the string you are doing re-find on?

andy.fingerhut15:12:18

And have you tried leaving out the ".*" at the beginning and end of the regex?

andy.fingerhut15:12:43

If you have individual text lines that are tens or hundreds of megabytes long, then the ".*" in the middle of the regex might become an performance issue, but that would surprise me.

andy.fingerhut15:12:22

Ah, I see you already answered the question before I asked about having very long lines.

markxnelson15:12:41

its 1654599 chars

markxnelson15:12:50

good point - let me try removing the .*

andy.fingerhut15:12:14

and according to text search using the 'less' command, does not contain any occurrences of the string "opyright".

markxnelson15:12:46

that was the problem - the .*

markxnelson15:12:48

thanks!!!! 🙂

andy.fingerhut15:12:49

Basically you were asking it to match the entire string of 1.5 million chars, which I'm a little surprised hung it, but certainly it seemed to be unlikely what you needed.

andy.fingerhut15:12:32

It does seem like a possible performance bug that JVM devs would want to know about, or perhaps already do

markxnelson15:12:06

thank you very much for your help 🙂

markxnelson15:12:13

i ❤️ the clojure community

andy.fingerhut16:12:55

Thinking on it a bit more, likely that regex starting with the greedy ".*" causes backtracking, i.e. every time it finds a c or C in the string, it checks to see if it can complete a match from the beginning of the entire string. If not, try one shorter. If not, try one shorter, etc. until finally a 0 length string before the c or C also does not match, and only then does it give up and find the next c or C character, and start that whole process over again.

andy.fingerhut16:12:27

Probably a non-greedy ".?" at the beginning would also avoid that, as would anchoring the beginning of match to the beginning of a line using a regex starting with "^."*

markxnelson17:12:17

removing the ".*" at both ends was fine, and i still got the behavior i needed

john-shaffer18:12:06

Weird situation with a protocol: I have the protocol definition

(defprotocol API
  (get-db [this request table])
  (get-model [this request table])
  (get-role [this request doc]))
and later in the same file this record implementing the protocol
(defrecord Site [db-f role-f models]
  API
  (get-db [this request table]
    (db-f this request table))
  (get-model [this request table]
    (first (filter #(= (:type %) table) models)))
  (get-role [this request doc]
    (role-f this request doc)))
I have a test suite for this library, and everything works great when I run the tests. But when I include the library in another project and use it, I get java.lang.IllegalArgumentException: No implementation of method: :get-model of protocol: #'cchesty.api/API found for class: I would think that maybe old code is being loaded somehow, but there is no old implementation of Site. I get the same error with CIDER and via lein run, so I'm out of ideas.

vemv18:12:56

is there any macro magic going on? or plain protocol + defrecord

john-shaffer18:12:11

plain Clojure

👍 4
hiredman18:12:52

you are loading the namespace twice

hiredman18:12:02

are you aot compiling either project?

john-shaffer18:12:07

Only for uberjar... but after I ran lein uberjar, the error went away, so maybe the implicit lein clean did the trick? Should have thought of that, but I still don't understand the cause

hiredman18:12:39

I would turn all aot compiling off and see what errors you get

hiredman18:12:53

turn it off, and run lein clean, and see

hiredman18:12:28

my guess is you are importing the type created by defrecord without requiring the namespace that creates the defrecord somewhere

hiredman18:12:28

I strongly recommend never importing the type and always use the constructor functions generated by defrecord since you are less likely to make that kind of error

hiredman18:12:29

similarly I strongly recommend against aot compilation because it can hide those kind of errors and manifest them has hard to understand and debug errors

john-shaffer18:12:53

lein clean && lein run works fine. The import isn't the problem here (almost everything relevant in one ns for dev) but that's really good advice. I forget about the ->Record constructor

john-shaffer18:12:07

Saves a line off the ns form too, nice bonus

john-shaffer18:12:13

Working great now, thanks 🙂

hiredman18:12:19

the import is almost certrainly the problem

hiredman18:12:09

there are other ways it can fail too, that have to do with compilation order, the way lein aot compiles doesn't match with how the compiler works entirely

john-shaffer18:12:45

There is only one import outside of the tests, and it requires the ns also

(ns cchesty.models
  (:require [cchesty.acl :as acl]
            [cchesty.api :as api]
            [cchesty.db :as db]
            ...etc...)
  (:import ))
The only :aot I have is under :uberjar, although I don't know all the details of how lein works be certain nothing is being aot'd during dev

hiredman18:12:16

the classfiles end up in target/classes or something and that is always on the classpath

hiredman18:12:40

my suggestion is always not to aot compile

john-shaffer18:12:02

Is it not more efficient for deployment?

hiredman18:12:44

it is slight faster to start, and that is it

john-shaffer19:12:04

Oh, startup time isn't a big concern. I'll see if I can run it without AOT

hiredman19:12:38

clojure always compiles to bytecode before running anything, aot compilation just tees that off to disk to be re-used

john-shaffer19:12:15

Got it packaging without AOT. Thanks, I learned a lot.

lilactown18:12:42

how are you including it in another project?

john-shaffer18:12:48

I'm using S3 as a repository, via leiningen

borkdude20:12:49

Is this letfn-ish macro conceptually correct?

(defmacro letfn2 [fnspecs & body]
  (let [syms (map first fnspecs)
        state-sym (gensym "state")
        fns (map (fn [sym]
                   `(fn [& args#]
                      (apply (get (deref ~state-sym) '~sym) args#))) syms)]
    `(let [~state-sym (volatile! {})
           ~@(interleave syms fns)
           ~@(mapcat (fn [sym fnspec]
                       [sym `(fn ~@(rest fnspec))]) syms fnspecs)]
       (do ~@(map (fn [sym]
                    `(vswap! ~state-sym assoc '~sym ~sym)) syms)
           ~@body))))
It seems to work:
(letfn2 [(f [x] (g x)) (g [x] (inc x))] (f 10))
Just wondering if I haven't missed anything

borkdude20:12:51

One incompatibility I spotted is that multi-arity doesn't work (fixed)

hiredman20:12:04

you can do it without the mutation

noisesmith20:12:38

the fns are "named" anonymous fns, so I'd expect (fn ~some-name [...] ...) - this effects self-calls that are not recurs

borkdude20:12:05

I'll add that, thanks

hiredman20:12:00

(but using mutation will be more similar to the built in letfn for things like function identity)

borkdude20:12:25

@hiredman I didn't see how while making forward declarations work

borkdude20:12:56

interesting, thanks!

hiredman20:12:19

it is a fun party trick

borkdude20:12:41

why is vswap! a macro and not a function like:

borkdude20:12:54

(defn vswap!*
  [vol f & args]
  (.reset ^clojure.lang.Volatile vol (apply f @vol args)))

dominicm20:12:55

It looks wrong as defined. It will use vol twice

👀 4
noisesmith20:12:44

right, volatiles make no attempt to be thread safe, that's the point of volatiles (moved this into a thread from a top level comment because of conversation flow)

dominicm20:12:33

This isn't related to that. This is about doubling side effects and improperly written macros.

noisesmith20:12:17

but that was defn, not defmacro

noisesmith20:12:12

or do you mean clojure.core vswap has this problem?

borkdude20:12:07

For comparison this is the macro:

(defmacro vswap!
  [vol f & args]
  (let [v (with-meta vol {:tag 'clojure.lang.Volatile})]
    `(.reset ~v (~f (.deref ~v) ~@args))))

borkdude20:12:37

would:

(defn vswap!*
  [vol f & args]
  (vreset! vol (apply f @vol args)))
have the same effect?

borkdude20:12:00

maybe a macro is used for some inlining effect?

👍 8
andy.fingerhut20:12:09

I would not be surprised if maximum efficiency was a goal, given the kinds of places volatiles are used in transducer implementation.

dominicm20:12:44

(vswap! (do (println :print) (volatile! 0)) inc) Produces

:print
:print
1

👍 4
borkdude20:12:12

@dominicm I bet there isn't a single place where vswap! is used like that, since you'll lose the reference to the volatile! (but still you're right)

noisesmith20:12:09

consider (if (some-test-with-side-effects?) v0 v1) - I could see that being used inline

borkdude20:12:16

I mean a volatile-creating expression

noisesmith20:12:20

(ins)user=> (def v0 (volatile! 0))
#'user/v0
(ins)user=> (def v1 (volatile! 1))
#'user/v1
(ins)user=> (vswap! (if (println :foo) v0 v1) inc)
:foo
:foo
2

noisesmith20:12:36

the bug happens here even without the creating expression

borkdude20:12:50

damn, you're right. make a JIRA

noisesmith20:12:37

@dominicm if you don't want to make a jira for the above I can probably do so today - that's the closest to a non-contrived version of the behavior I can think of at the moment

borkdude20:12:56

(vswap! (fetch-volatile-using-very-expensive-calculation) inc)

noisesmith20:12:53

or "fetch volatile via non-idempotent API call"

borkdude20:12:37

(vswap! (we-can-do-this-only-once-or-Apollo-13-wont-be-able-to-make-it-back-to-earth!) inc)

dominicm21:12:23

I'm on mobile, go ahead

vemv20:12:58

Can I get repl-y, rebel-readline or such to write closing cparens for me?

hiredman21:12:09

I think volatile! and vswap! are just bad. introduced to squeeze the last bit of performance out of transducers, but I wouldn't use them

hiredman21:12:40

vswap! is just silly, it acts like it provides clojure's time model over volatile! cells, which it absolutely does not

ghadi21:12:08

i think people are too eager to reach for volatile!, but I don't think it's "bad", and it does exactly what the docstring says:

"Non-atomically swaps the value of the volatile as if:
   (apply f current-value-of-vol args). Returns the value that
   was swapped in."

borkdude21:12:12

It's pretty clear: Non-atomically swaps. I do find them useful

hiredman21:12:23

Use atoms for concurrent mutation, use clojure.lang.Box otherwise (Edit: for clarity since people keep misreading this, I mean if you use multiple threads at all, and your mutable reference is visible to different threads, use an atom. full stop, multiple threads, use an atom. If you have a single thread, there is really no need for mutation you can write a nice functional loop and do whatever you want. However, if you insist, do not use volatile! cells for that.)

Alex Miller (Clojure team)21:12:28

there are approx 0 good reasons to ever do unsafe concurrent mutation

hiredman21:12:55

I am not sure how to read that as a commentary on my suggestion to use atoms for concurrent stuff

Alex Miller (Clojure team)21:12:46

I read it as "otherwise" == unsafe concurrent mutation

Alex Miller (Clojure team)21:12:08

but maybe you just meant mutation with identity

Alex Miller (Clojure team)21:12:34

(and in that case, I'd suggest an object array instead)

hiredman21:12:42

I just meant whatever silly non-concurrent mutation people end up doing where they could use loop/recur

hiredman21:12:44

like somewhere above in the channel there is a reimplementation of letfn using volatile! cells to tie the knot

leonoel09:12:27

a valid use case for unsynchronized concurrent mutation : lazy memoization of a referentially transparent expression, cf java.lang.String/hashCode

vemv21:12:13

Dunno. Even if a single thread ever mutates a the val of a Box (https://github.com/clojure/clojure/blob/653b8465845a78ef7543e0a250078eea2d56b659/src/jvm/clojure/lang/Box.java#L17), since that val is not volatile (Java lang keyword), other threads are not guaranteed to see those changes (JCIP stuff) So Clojure's volatile seems like life insurance

hiredman21:12:57

feel free to reread where I said for concurrency use an atom

vemv21:12:16

you said concurrent mutation :) writes performed by a single thread aren't concurrent.

borkdude21:12:24

what does Box offer compared to volatile! except that it isn't a public API and doesn't work in CLJS?

hiredman21:12:54

box doesn't pretend to be something it isn't, and the fact that it is so weird should make you step back and think if you need mutation in that situation at all

Alex Miller (Clojure team)21:12:18

you should rank not doing this at all well above using Box

hiredman21:12:49

entirely agree, don't mutate things, it is the best

didibus05:12:58

Wild guess, it doesn't bypass the CPU caches? As volatile does

hindol20:12:25

I read somewhere that this is not entirely true. Volatiles do not bypass cache for performance reasons. Instead it uses a tagging system to maintain cache coherence.

didibus20:12:51

I think that's probably architecture dependent

didibus20:12:33

I can imagine some CPUs having mechanisms now to synchronize the caches between cores within themselves that might be faster then going to ram

didibus20:12:25

In such case it might still use a CPU cache. The point being the volatile guarantees read-after-write consistency

didibus20:12:52

And Box would not

arohner21:12:58

I have an app that is reliably segfaulting on kubernetes, but not locally using the same docker image. Is there a way to make the SIGSEGV error report print to stderr so kube logs pick it up?

dpsutton21:12:09

is there a notion in spec similar to cut in prolog to prevent backtracking? The problem being that if you fail some spec it could be marked as the correct failed spec? It seems if there are several possible specs for something, to mark it as a failed one of the several failed specs rather than failing all of the specs?

p-himik21:12:23

Can't answer your question, but there's #clojure-spec. Maybe someone in there knows the answer.

dpsutton21:12:37

thanks i will

ghadi21:12:14

@arohner fun! isn't there a *.err file that gets output with a segfault?

ghadi21:12:53

I wonder if either a) you could control its location or b) you could have a sidecar that ran cat *.err on a loop

arohner21:12:50

I’m trying to use -XX:ErrorFile, but it’s not being picked up yet

ghadi21:12:28

at least on adoptopenjdk 13, without that option, it writes to /dev/stdout when I kill -SEGV the process

ghadi21:12:45

if I use ErrorFile, it also writes to the error file

arohner21:12:46

hrm, I’m on adoptopenjdk12

ghadi21:12:37

java  -cp $(clojure -Spath) clojure.main

ghadi21:12:45

i tested by kill -SEGV that process ^

ghadi21:12:50

and it wrote some stuff to stdout

ghadi22:12:45

the output wasn't super useful @arohner

arohner22:12:51

Yeah, the contents of stdout on sigsegv are much less interesting than that hs_err file

arohner22:12:33

Apparently -XX:ErrorFile must be in the java command line, not $JVM_OPTS

ghadi22:12:06

I still don’t think you need it at all

ghadi22:12:22

All of it was written to stdout for me

ghadi22:12:39

But i didn’t do extensive testing

ghadi21:12:50

@borkdude userspace callbacks will not happen on a segfault

arohner21:12:50

I’m trying to use -XX:ErrorFile, but it’s not being picked up yet