Fork me on GitHub
#clojure
<
2019-12-27
>
noisesmith00:12:03

with clj / deps.edn is the only way to add default jvm opts to use a profile explicitly?

noisesmith00:12:08

(or of course supply -J... every time I start the process...)

seancorfield01:12:49

@noisesmith I believe that is correct. I think there is a JIRA ticket to consider allowing :jvm-opts at the top-level.

noisesmith01:12:20

OK, well that's very inconvenient

noisesmith01:12:55

@seancorfield so there's no implicit base alias, or repl alias that gets merged onto?

didibus01:12:40

You can kind of fake a default implicit alias by using a shell alias

noisesmith01:12:03

yeah, I'm setting up project tooling for a new app, I dont' want to force my team to all use the same shell aliases or custom config

noisesmith01:12:51

what I want is to make this lib work by providing the right system properties at vm startup, for now I guess my best option is to make everyone switch to a custom shell script instead of using clj/clojure directly

noisesmith01:12:59

but that's still ugly

seancorfield01:12:40

Is there a reason you're using JVM properties rather than some other approach?

seancorfield01:12:09

Those seem pretty volatile to me since anyone can override them at startup, in their own environment.

noisesmith01:12:19

it's the only way the config library accepts setting its config source

seancorfield01:12:48

And it doesn't have a default source?

noisesmith01:12:02

nope, weirdly enough

noisesmith01:12:23

I guess I could set outpace.config.bootstrap/explicit-config-source but that still needs to be done in a way that all tasks end up invoking the code

noisesmith01:12:58

oh - I see, the default is in current directory, not classpath

noisesmith01:12:05

so it does have a default I can use

seancorfield01:12:24

If your library code has control before outpace runs, you can set a default value for the JVM property if none is present.

seancorfield01:12:32

(System/setProperty ,,,)

noisesmith01:12:45

right, but I'd have to do this at every entrypoint

noisesmith01:12:39

looks like just moving the file to the expected place would solve it (but then I need some other incompatible setup for running from a jar...)

seancorfield01:12:38

(this sort of thing is why we wrote our own config library 🙂 )

seancorfield01:12:15

At some point we will refactor it out of our code and open source it... but I've been saying that for a couple of years already so...

1
zignd02:12:06

do you guys use cheshire for strucutred logging, as in logging JSON to the stdout to capturing it using another tool to send it to a logs aggregator?

noisesmith02:12:31

I'd use something more dedicated to logging - eg. cambium

noisesmith02:12:48

also, clojure.data.json has a cleaner deps tree than cheshire

zignd02:12:50

thanks, i will look into this cambium, i'm currently trying to combine timbre and cheshire

potetm02:12:36

pretty big speed hit for clj.json

potetm02:12:54

(i.e. cheshire is faster)

noisesmith02:12:57

if your logging is a perf bottleneck you have bigger problems

noisesmith02:12:04

but yes, cheshire is marginally faster

noisesmith02:12:21

it also brings in a bunch of apache / jackson nonsense

noisesmith02:12:57

jackson is the biggest source of painful / breaking dep tree problems in my entire software career

☝️ 1
potetm02:12:09

but, imo, for well-defined, not-likely-to-change things, there’s a fair argument for “go with the faster thing”

potetm02:12:22

@noisesmith there a much worse offenders

potetm02:12:27

in the clojure space alone

potetm02:12:38

(not to mention some java-centric things)

potetm02:12:49

java logging alone is sig worse

zignd02:12:02

there's one problem i'm facing with both org.clojure/data.json and cheshire, maybe you guys could help, i'm kinda new to clojure. timbre has an option that allows you to provide a function that will be responsible for generating the output based on the data sent to it to log, i provided a function like this one:

; for cheshire
(s/defn timbre-output [data]
  (-> data
      (json/generate-string data)
      (println)))
; for clojure json.data
(s/defn timbre-output [data]
  (-> data
      (json/write-str)
      (println)))

potetm02:12:32

I dunno nuthin bout no timbre :derp:

noisesmith02:12:53

println returns nil

potetm02:12:12

I usually use tools.logging + logback

zignd02:12:10

hmmm, this combination lets you send whatever object comes to you to your logging function and it manages to serialize it without exceptions?

noisesmith02:12:22

@zignd your -> macro is providing two args to cheshire, data twice

noisesmith02:12:55

I forget what its generate-string does with a second arg, but that's clearly not what you want

noisesmith02:12:16

also, if that function is returning the thing you want to log, take out the println call

zignd02:12:30

woah, that is true! @noisesmith thanks for mentioning!

noisesmith02:12:34

if it is meant to do the output, println is a weird choice but I guess it could work

noisesmith02:12:18

you can use doto if printing for debugging purposes (-> data json/write-str (doto println))

zignd02:12:43

oh, i didn't know about doto :thinking_face: gonna check it out, thanks for mentioning!

zignd02:12:26

the problem i'm facing is that both clojure/json.data and cheshire throws exceptions when they encounter an object they can't seem to convert to json, currently an instance of clojure.lang.Delay, i'd like to knows if there's any option to call .toString for objects in this condition, but couldn't find anything on the docs. although there's an option in both libraries that allows you to check the object's type and manually serialized them

noisesmith02:12:53

each one accepts a global config offering rules to output custom types

noisesmith02:12:14

the one with clojure.data.json is nicer to work with IMHO

potetm02:12:39

If all you want is to call .toString, then maybe a wrapper ns?

potetm02:12:15

e.g. a ns with (defn info [obj] (log/info (str obj)))

noisesmith02:12:40

with clojure.data.json you extend the JSONWriter protocol for each type - it has one method -write

noisesmith02:12:52

so (extend-type clojure.lang.Delay JSONWriter (-write [this out] ...))

zignd02:12:33

nice! now i do have a few things to look into, thank you @noisesmith and @potetm. cambium seems to be a better option in order to not reinvent the wheel, at least from what i could quickly read on their web site, in case it don't i will try the clojure.data.json approach maybe checking for the protocol implementation with satisfies? and calling .toString when it doesn't :thinking_face:

didibus06:12:49

Is there a good reason why println isn't concatenating on single-arity before it writes the newline to out ?

didibus06:12:20

The current behavior means that newline could be out of order in a multi-thread scenario

noisesmith19:12:55

yeah, println simply isn't thread safe, I wouldn't be surprised if that was intentional

noisesmith19:12:06

like - if you need thread safe output, you need logging not println

didibus20:12:18

Once in a while, I get burned trying to debug a concurrent algorithm at the repl, and realize the println isn't thread safe

didibus20:12:10

Generally my fix has been to concatenate before I call println, but I realized today even the newline is not thread safe

didibus20:12:51

So now I need to use (print (println-str a b c))

didibus20:12:11

Or (print (str a " " b " " c \newline))

didibus20:12:37

I wonder why println just doesn't do that in its implementation though

didibus20:12:45

My only small guess is maybe it is assumed that .write on a stream is itself not thread safe, so they figured no guarantees exists and so didn't bother

didibus20:12:27

But System.out appears to have a threadsafe write in OpenJDK at least

didibus08:12:21

And I guess why it wouldn't also do the same for the multi-arity variant

hmaurer12:12:19

Nevermind, this is in ClojureScript; I’ll post in the right channel 🙂

kenny16:12:57

Anyone know of a good algorithm to group programmatically created string together? For example, "gke-qa-stable-06e4523d-grp" and "gke-qa-stable-06e4525d-grp" would belong together. "cachefiler01" and "l2cachefiler01" would not belong together. I've tried using levenshtein distance under some threshold to group strings together, but it gets several strings wrong. Ideally the algorithm would weight earlier letters more than later letters when computing the distance.

delaguardo16:12:55

https://en.wikipedia.org/wiki/Jaccard_index have a look at this one. I found it more accurate for purpose of calculating weights of “autocomplete suggestions” that looks close to what you describe in example

kenny16:12:17

Thanks, reading now!

codonnell16:12:10

If the strings all have a regular structure, it seems like parsing them and grouping them based on their parsed attributes would be more reliable than trying to use a string similarity algorithm.

kenny16:12:30

They don't have a regular structure 😞

😳 1
Sascha16:12:26

For address matching (street, city, …) we used quite successful the n-gram matching. (Similar to levenshtein). If you then add additional information about your strings, like grouping the sections separated by a ‘-’ you could get good predictions. Or have the first n-grams higher weighted than the last n-grams. In clojure it will be pretty easy to be implmented with the partition function.

kenny17:12:28

Interesting. About 50% of my sample of names have a hyphen and the other 50% do not. For example "devices01" and "devices02".

Sascha17:12:32

This might give you an idea about an unweighted n-gram match. Good luck. But really interesting task you have. 😉

kenny16:12:52

This won't really work because I have a giant list of strings that needs to be grouped. I'd need to know where in that sorted list to break apart into groups.

kenny17:12:44

There may be something here though. Sort, partition into groups of 2, find groups where the levenshtein is less than some threshold or percentage of word size. May work well if the groups are often pretty dissimilar.

potetm17:12:15

Requirements are pretty vague

potetm17:12:41

Why are the strings being “grouped?”

potetm17:12:32

Are you trying to derive some sort of partitioning scheme?

kenny17:12:48

We have customers running VMs in the cloud. They have hundreds to thousands of instances running which are not grouped in any way besides name similarity. The goal is to help them understand what they are running by grouping things together.

potetm17:12:41

:thinking_face:

potetm17:12:10

so, my reframing might be: “Please parse these strings for metadata about the VM”

kenny17:12:47

Sounds right

potetm17:12:50

Which, if that’s accurate, is much easier (or only, really possible?) if you have a proper grammar

potetm17:12:23

idk - at this point I’m pushing the convo way outside what you were asking

kenny17:12:37

Definitely. It’s hard to enforce that at large enterprises though. These VMs cross several teams. They are interested in fixing what they have now and then work on the cultural problem.

potetm17:12:30

Yeah I wonder how far a “simple” sort+filter might get you

potetm17:12:46

At any rate, curious to see what you come up with. What you said about sort + levenshtein might just work gud enuf.

Joe Lane17:12:54

@kenny Look up Chris Oakman's talk on probabilistic record linkage on youtube from the last conj. It seems highly relevant to your situation except for domain differences. He even has source code in a repo on his github account.

kenny17:12:21

Oh cool! I've done some work in the healthcare realm and know the data can be quite nasty. I'll be he's got some good ideas.

kenny18:12:22

The sort + levenshtein only gets about 2% wrong. I’m going to watch that talk for fun but I’m guessing that error rate is acceptable.

erik17:12:01

what's the appropriate channel for questions regarding metabase-datomic?

zignd17:12:43

is there a cider+company operation/function to list parameters of the function at the cursor?

zignd17:12:46

something like this on cursive

Joe Lane17:12:07

The tool you're looking for is eldoc , which cider supports.

zignd17:12:51

oh, it is even already enabled by default, it was displaying the doc at the bottom of the window and i didn't notice. sorry and thanks at the same time @U0CJ19XAM hahahah

tcrawley17:12:49

FYI: http://clojars.org is currently down. I'm working on fixing it now. The repo is readable, but search, uploads, and the API are unavailable.

tcrawley18:12:20

Ok, it's back up. I was iterating on the ansible config and make a mistake there.

👍 3
deep-symmetry20:12:48

Are there any thoughts about upgrading the Clojure license to EPL 2.0 from 1.0? I have been nudged to do that in one of my own Clojure projects to continue benefiting from the free open-source project tier at Netlify.

dpsutton20:12:15

i saw something about netlify changing how they offer free tier. i think i remember they will require a code of conduct in the future as well?

didibus20:12:51

Doesn't EPL 1.0 allow you to use 2.0 at the user discretion?

alexmiller21:12:35

Rich prefers 1.0

alexmiller21:12:01

So it’s unlikely Clojure itself will change

didibus21:12:26

By the way, EPL 1.0/is an approved OSI license. So it seems it would fit the Netlify qualification

dpsutton21:12:15

I didn't see it in the list

dpsutton21:12:04

ah yes. it is included in the "All Approved Licenses"

dpsutton21:12:18

@U0EHA00G1 according to Netlify's terms they should allow EPL 1.0 since its on https://opensource.org/licenses/alphabetical

dpsutton21:12:37

> Includes a license listed on the Open Source Initiative approved license list or a Creative Commons license that includes “attribution” or places the work in the public domain.

deep-symmetry21:12:49

This is for the Open Source Plan which gives pro plan benefits for free. I did need to add my code of conduct to the project, and EPL 1.0 is not one of the available licenses because it’s deprecated to reduce license proliferation.

deep-symmetry21:12:23

I don’t need Clojure to change, I was just curious about the thinking because of that license proliferation question. It’s probably inevitable because I’m sure Rich isn’t the only person to prefer 1.0.

didibus21:12:23

The Program (including Contributions) may always be distributed subject to the version of the Agreement under which it was received. In addition, after a new version of the Agreement is published, Contributor may elect to distribute the Program (including its Contributions) under the new version.

didibus21:12:50

"Contributor" means any person or entity that distributes the Program.

didibus21:12:34

Because of that, EPL 1.0 allows anyone to redistribute an EPL 1.0 program as an EPL 2.0 program.

vemv21:12:14

Is there a strong drawback to using EPL2?

didibus21:12:04

IANAL, but I'd say no. It all depends in your intent

didibus21:12:06

EPL 2.0 clarifies that source distributed code as part of a program making use of an EPL 2.0 dependency can have its own code be distributed under a different license

vemv21:12:48

:thumbsup:

didibus21:12:55

Because EPL 1.0 doesn't have that, it isn't clear if say a JavaScript app which links against the EPL 1.0 code can itself be non EPL 1.0

vemv21:12:18

...I was a bit worried because I upgraded epl1->2 mindlessly

didibus21:12:22

Or to be honest, any Clojure code which doesn't do AOT

didibus22:12:16

The part I'm not sure is why EPL 1.0 isn't suitable for scripting languages. I believe it is related to the wording using "module" instead of "file"

didibus22:12:18

It makes it sound like by bundling your own files with EPL 1.0 files together in one module this would be considered a modified work of the original and therefore your new files should also be EPL 1.0

didibus22:12:05

Which would be arguable for some scripting languages like JS

didibus22:12:34

Since it doesn't have a clear concept of module

didibus22:12:06

In Java, you could argue module means a given JAR, but even there it isn't super clear. Could it be the package? Or what is it in Clojure? Is it the namespace?

didibus22:12:40

For example, if I take Clojure core and add 5 new functions to closure.core namespace? Is that derivative or modified?

didibus22:12:40

In EPL 2.0 the language is very clear, if I modified an existing file, even if by only adding to it, it is modified and my additions fall under EPL as well

👍 1
didibus22:12:52

Otherwise it isn't

didibus22:12:04

In EPL 1.0 it is up to the courts.

didibus22:12:36

In EPL 1.0, if I just add functions to the clojure.core namespace at runtime, even if I do this in my own file bundled in a seperate jar, someone could claim it is a modification of the closure.core module and thus needs to be licensed as EPL as well

👍 1
1
seancorfield21:12:41

(I'm trying to eliminate all the Illegal Reflective Access warnings that JDK 11 highlights and this is the last one -- and I sort of expected it from clojure.xml)