Fork me on GitHub
#clojure
<
2020-11-13
>
Jack Arrington00:11:47

If I have a custom list type that I've implemented with deftype, and I want to make it so that I can call (map) on it, what do I have to do? Implement ISeq, ISeqable, both?

didibus00:11:55

ISeqable I think is all you need

👍 3
thanks2 3
cursande00:11:22

Hi 👋 Bit of a crapshoot, but anyone had success instrumenting async web requests with newrelic and netty server running via clojure? I'm looking to add in newrelic instrumentation on specific endpoints for a netty server run via aleph. I'm following this old guide from sean https://corfield.org/blog/2013/05/01/instrumenting-clojure-for-new-relic-monitoring/, and this lib https://github.com/TheClimateCorporation/clj-newrelic implements the same approach and works great for us when instrumenting code running on clojure services running jetty. Following the newrelic docs on async I get either transactions with incorrect times (as deferred value is returned immediately I assume) or I can see in logs transactions are not created despite adding :dispatcher and :async to true on the method annotation.

seancorfield00:11:20

@adfarries No, unfortunately. Our Jetty-based services are monitored just fine via New Relic, but our one Netty-based service shows almost nothing useful.

seancorfield00:11:51

We've used New Relic's "metrics publish" library (I think you can d/l the source from their GitHub repo and build it yourself), and the NR agent to allow us to write plugins that send all sorts of specific metrics to New Relic -- they don't show up in APM, only under Plugins, but you have a lot of control over what you publish.

👍 3
danielglauser04:11:58

I used to really enjoy using New Relic. We wrote some code to send statsd metrics to DataDog, that seems to be working well.

seancorfield05:11:41

We're very heavily in bed with NR at this point, both front end and back end. It would take a seismic shift for us to switch to a different metrics solution I think...

seancorfield05:11:23

Not to say NR is "perfect" (it's not!) but it's "good enough" and easy for everyone to use, from backend to frontend to product management etc...

3
seancorfield00:11:47

https://github.com/newrelic/metrics_publish_java -- I see V2 is archived, we're on V1 from ages ago.

seancorfield01:11:16

(I suspect, at some point, we'll have to revisit/update all our plugins)

cursande01:11:06

@seancorfield thanks. I was hopeful that updating to latest agent, agent-api etc. and following NR's docs on custom async instrumentation would get me there, but unfortunately I'm just not seeing much of value. Segments 'work' in a very naive sense but make no sense in the transaction context

cursande01:11:00

will keep chipping away + try with plugins, if I find a sensible solution I might post it up somewhere

rgm01:11:21

I have a core.cache question (I think, or at least I’m musing about implementing something with core.cache): I want to generate on-disk temp PDF files and have them disappear after say, an hour. The cache holds an identifier and a path to the file, to be fed out of a jetty server with the identifier in the request URL. It’s fine if the URL 404's after an hour. I got to thinking: would a TTL cache be able to handle this use case? I’m trying to figure out if I could directly supply a function that unlinks the on-disk PDF on eviction, or if this is best done by implementing a custom version of the TTL cache with CacheProtocol evict wrapped?

phronmophobic01:11:11

does the pdf have to be written to disk or could it be cached in memory?

rgm01:11:39

I wasn’t excited about keeping them in memory; they’re running about 200-300kb each.

rgm01:11:48

but not impossible, I suppose.

phronmophobic02:11:36

another option I might consider is to have a temporary folder and have a background process that runs every hour and deletes any files older than an hour

phronmophobic02:11:36

it makes worrying about "what happens if the server crashes or restarts" type of issues less of a problem

rgm02:11:54

Yeah, that’s probably OK too… just yank the existing keys out of the cache as a set and if it’s not in the set, farewell PDF.

hiredman02:11:20

I would not recommend core.cache for that kind of thing, look at guava's caches

🙏 3
💯 3
hiredman02:11:18

core.cache is built around the idea of immutable caches as value

rgm02:11:58

/ looks at https://github.com/google/guava/wiki, expires from unexpectedly broadened horizons /

rgm02:11:10

yeah, I had just got to thinking an evict callback would let me be lazier but it sounds like I should probably think this through a little better.

seancorfield03:11:37

Yeah, as the maintainer of core.cache, I would agree with @U0NCTKEV8 on this @U08BW7V1V -- I don't think it's a great match for what you're trying to do.

🙏 3
potetm14:11:08

https://github.com/ben-manes/caffeine looked extremely promising to me

🙏 3
Stuart13:11:44

what is a current library for CLojure for building REST APIs ? And does anyone have any decent documentation ?

Mathieu Corbin13:11:42

My stack is ring+jetty (aleph at work but I think jetty is enough for most use cases), https://github.com/exoscale/interceptor for the request lifecycle and https://github.com/exoscale/ex for error handling. For routing I use https://github.com/juxt/bidi

Stuart13:11:12

thank you! I'll take a look at those

Stuart13:11:44

I worry when I goto github page for yada and no check ins in years, or does this jsut mean its pretty stable / finished ?

st3fan13:11:03

that is the case with most clojure packages i am using right now 🙂

st3fan13:11:14

being new to clojure, not sure what to think of that yet

Alex Miller (Clojure team)13:11:50

why does everyone only trust software that changes all the time? seems counter-intuitive... :)

👍 12
Stuart13:11:15

not used to using libraries that arent full of bugs... and need constant patching 😄

😂 6
st3fan13:11:58

@alexmiller there is a difference between ‘all the time’ and ‘not updated in 9 years’ 😉

st3fan13:11:30

in any case - i read the source and very often libraries that look like abandonware are just simple enough to not be worried

☝️ 3
Jonathan Doane13:11:38

I think that there is a difference between not being updated because it’s been abandoned and not being updated because it’s stable.

st3fan13:11:12

the world is not stable 🙂

Jonathan Doane13:11:20

There are libraries that were written for 1.3 that still work with 1.10, so why update it if it isn’t broken and it works?

Jonathan Doane13:11:59

The world might not be stable, but Clojure seems pretty solid.

👍 3
lukas.rychtecky13:11:12

@st3fan I would say that in Clojure ecosystem it’s common that a library is not being updated for years and it’s still compatible with the latest Clojure and etc.

Jonathan Doane13:11:23

Cognitect has been very good about not breaking backwards compatibility.

borkdude13:11:58

@qmstuart We have been using yada for years. It's powering https://covid-search.doctorevidence.com/.

Stuart13:11:29

cool, i only need this api to be dead simple. It wont be public, it will be running on a machine and only taking a single POST request from that same machine.

st3fan13:11:40

i think where it gets tricky is with libraries depend on Java packages or have to interact with the outside world - as an example, I was just looking at clj-yml, which wraps a 8 year old version of SnakeYAML - it is entirely possible that it has security issues

borkdude13:11:42

fyi, the company behind yada is working on another framework (apex) but that's not production ready

dominicm14:11:28

And also spin

dominicm14:11:20

This is an active research area for Malcolm, but I've also been backporting pieces into yada where possible (e.g.I got reap working, and fixed a few bugs)

mpenet15:11:12

is it the one using vertx?

mpenet15:11:56

we also have a lab thing that wraps vertx server (it's called helix internally), I guess there could be crossed interests here

dominicm15:11:40

I think so, yeah. Although I think (personally) that vertx isn't too interesting in the bigger picture. Just as a precedent for guiding the design of ring2's web sockets and such.

mpenet15:11:36

right, not in apex context specifically

borkdude15:11:16

I think it would be great if apex could be bigger than just juxt, so I would welcome this co-operation from a community perspective

borkdude15:11:47

also the work that is happening in malli seems to co-incide with all the json schema stuff maybe

borkdude15:11:11

so maybe a metosin / exoscale / juxt backed framework, could be great :)

👍 3
borkdude13:11:32

@st3fan you should be looking at https://github.com/clj-commons/clj-yaml for the most recent version which has a recent version of SnakeYAML (also available in bb)

borkdude13:11:09

@qmstuart for something super basic, ring jetty + compojure is probably the "easiest"

👍 3
st3fan13:11:12

@borkdude i just pulled in snakeyaml and wrote a 3 line parse .. that is also another great option IMO - do not pull in massicve deps if your needs are simple

borkdude13:11:43

@qmstuart Actually, babashka will let you create a ring app with it's built-in http server. if it's not performance critical, that might be the "cheapest" option in terms of project setup, etc: you only need one file

st3fan13:11:09

@qmstuart the thing i am working on only has one endpoint do i did not even bother with a router .. i am just using ring+jetty

Stuart13:11:45

yeah, we have a bash script / ps script on windows. It collects data and just want it to go to our local api, which can do some processing locally and fire it off to our rabbitmq queue

st3fan13:11:06

if i add another endpoint i may just add a mini dispatcher myself, because a few lines of code is much simpler than new DSLs 🙂

borkdude13:11:44

babashka doesn't even have a router at the moment. Just use (:uri request) and (:method request) and write your own thing ;)

👍 3
Stuart13:11:52

@borkdude, single small file would be nice. Current alternative is to write it all in Go (to get single small file), but I hate Go

borkdude14:11:05

@qmstuart So this would be a small web-app with instant startup.

borkdude14:11:29

One caveat: I don't know if babashka can talk to your RabbitMQ. It doesn't have a built-in library specifically for that.

Stuart14:11:39

thanks, i'll have a look. bb seems a great alternative to trying to get graal native stuff working

st3fan13:11:53

(where mini-dispatcher could very well be a defmulti dispatching on the path .. clojure has enough on board to do a lot of things very easily)

st3fan13:11:17

i’m rewriting a GitHub app from Go to Clojure .. I love Go, but I’m currently at 50% functionality converted to Clojure in like 10% of code 🙂 🙂 🙂

Stuart13:11:05

using Go, after using things like C#, F#, clojure just feels like i'm missing so many features. Stuff that in those languages is easy feels really complicated in go when all you have is for loops and ifs

potetm15:11:01

I’m curious what features you feel like you’re missing in clojure.

Stuart15:11:34

I dont feel missing any features in Clojure. I mean I feel a lot of things I use regularly in Clojure, C# and F# are missing from Golang 🙂

potetm15:11:56

yeah, same

Stuart15:11:19

and it feels like every other line is if err != nil { }

👆 3
alpox16:11:32

Im in the same boat. Also: working with dynamic datastructures in Golang is such a pain. Reflect much? 😬

Stuart13:11:51

maybe getting used to higher level features has dumbed me down

borkdude13:11:55

I know a project that uses Go but generates the Go code in Clojure: https://github.com/tzzh/pod-tzzh-aws

fenton17:11:28

can I do: #:person{name "joe"} instead of {:person/name "joe"} without having a valid namespace persondefined? Seems to work sometimes, but not others.

fenton17:11:11

Warning :undeclared-ns in app/client.cljs at 17:1 `No such namespace: person, could not locate person.cljs, person.cljc, or JavaScript source providing "person"`

dpsutton17:11:37

are you using #::person when that error happens?

fenton17:11:13

{:person/id [#:person {id 1 name "Fenton" age 40}]}

fenton17:11:28

shoot missed the second colon.

fenton17:11:37

sorry/thanks.

fenton17:11:19

{:person/id [#:person {:id 1 :name "Fenton" :age 40}]}

dpsutton18:11:09

hard to tell. is everything clear now or have you found the example that errors that you don't understand? also this might be better served in #beginners as its related to a bit of syntax

👍 3
hoynk18:11:17

Is there any profiling lib that can easily point me to the slowest functions on the code? I am using ptaoussanis/tufte which is almost what I want, except I have to manually instrument my code.

noisesmith18:11:04

there's nothing easy in my experience - laziness can move the cost of expensive calculations around at runtime

hoynk18:11:37

Yup, I know, But I am used to sprinkle some doall's around for those times 🙂

noisesmith18:11:35

but a proper profiler (yourkit, visualvm) can help you find hot spots, once you get past the ways clojure breaks java assumptions

jacklombard18:11:15

Facing this possible memory leak in one of our Clojure apps, has anyone come across something similar before?

noisesmith18:11:13

it looks like something is creating a lot of nio channels and never closing them http://www.docjar.com/docs/api/java/nio/channels/SelectionKey.html

noisesmith18:11:07

perhaps you have a global value holding connections to clients?

jacklombard18:11:32

Can’t really tell, its a pretty big app but this has started happening in the last month.

jacklombard18:11:33

I’ll read the docs

noisesmith18:11:50

the clojure libs that would use nio channels would be eg. ring implementations, websocket libs

noisesmith18:11:12

or maybe someone started doing prematurely optimized file IO and isn't closing the files

jacklombard19:11:16

Is there a pattern of code that I can look for? Those channels being stored in some var?

noisesmith19:11:51

the first place to look would be something that holds onto io handles so they can't be collected

noisesmith19:11:44

but with nio you could well have some internal state that isn't cleaned up and closed on gc, so you'd want to look for file / network handles that get created but never get closed

noisesmith19:11:54

also check if you have some process that accepts connections or watches files, that would be a likely culprit for creating all the io handles

jacklombard19:11:55

Thanks for the pointers, going to have an interesting weekend hunting this

noisesmith19:11:35

also, yourkit / visualvm have a graph view of who allocates / holds objects by type, so you could look for the owners of those objects

noisesmith19:11:01

(likely that will be some internal of a lib like netty, and then you figure out who is using that lib's api etc.)

jacklombard19:11:05

Right, but we haven’t enabled visual vm in prod and so can’t set it up any time soon. Would be really hard to pin point the culprit in local. But I think I already sort of know where the problem might be

mafcocinco19:11:58

Is there a reason the clojure.set/intersection does not have zero-arity version which returns an empty set? This would make it usable in combination with reduce to find values that appear in every entry of a sequence. I’m guessing it is because returning the empty set is not always something one would want to do (i.e. there is some ambiguity in what the behavior of the zero-arity version would be).

noisesmith19:11:54

user=> (reduce set/intersection [#{:a :b :c} #{:b :c :d} #{:a :b :d}])
#{:b}
I don't see a problem here

noisesmith19:11:16

one of the few cases where implicitly using the first item in coll as the accumulator is a win

noisesmith19:11:07

oh, right

(reduce set/intersection [])
Execution error (ArityException) at user/eval176 (REPL:1).
Wrong number of args (0) passed to: clojure.set/intersection

😪 3
Derek19:11:11

The intersection of an empty set and a populated set is an empty set

noisesmith19:11:09

but if set/intersection actually behaved correctly for zero args, requiring an init arg would make things worse

noisesmith19:11:21

having implicit first arg acc actually fixes it

noisesmith19:11:57

I guess your init acc could be "the set of all possible clojure values", but we have no way to represent it currently :D

borkdude19:11:04

hmm good point :)

mafcocinco19:11:07

@noisesmith exactly. @dpassen1 That is true but I’m not sure it is relevant for this discussion. The 1-arity version of intersection returns the set you pass in. The zero arity (which would be called when the sequence to reduce is empty) would correctly return the empty set.

borkdude19:11:49

I guess the identity function is the set of all elements

noisesmith19:11:17

@mafcocinco the hacky evil way to do this would be to reify the right interfaces so as to create an object that acts like it contains all values

noisesmith19:11:26

the right fix is an improved set/intersection :D

borkdude19:11:32

or use (when (seq ...) (apply set/intersection ...))

mafcocinco19:11:07

Yeah. I solved it by creating a local intersects function with the proper arity as it is internal code. Just struck me as odd and I suppose it is good sign of how well clojure was put together that I assumed there was some logical reason vs. an oversight/mistake.

borkdude19:11:54

@mafcocinco The usual reason is that in many cases there is no obvious identity element. E.g. - has no identity element, so (-) doesn't work. intersection is similar

mafcocinco19:11:39

is it wrong that the (intersection) is #{}?

noisesmith19:11:55

haha, my hack idea (never a good one in the first place), won't work because it wants to walk the first arg

mafcocinco19:11:42

perhaps mathematically it is but I’m interested in what the pragmatic reason is for not doing it.

dpsutton19:11:45

usually the return of that is the identity element. (+) is 0 because x + 0 = 0 + x = x for all x

dpsutton19:11:06

#{} is a zero element, not an identity under intersection

noisesmith19:11:14

oh right, #{} isn't the identity, just a fixed point

mafcocinco19:11:41

right. That makes sense. So in order to support (intersection) we would need a set-of-everything.

💯 6
noisesmith19:11:44

the identity would be my above (impossible) set of all possible clojure values

borkdude19:11:07

if intersection supported functions in addition to set objects, identity would be the set of all things

borkdude19:11:18

but that is not really how it works

borkdude19:11:53

btw, using apply is sometimes better than reduce since reduce will use the 0 or 2 arg of the function, while in many cases the varargs version is optimized.

borkdude19:11:20

e.g. in the case of the set functions it will usually begin with the set that is of optimal size

mafcocinco19:11:50

Good to know. thanks!

borkdude19:11:26

similar for str: when you use reduce I think you will have many StringBuilders under the hood, but with apply only one

noisesmith19:11:50

yeah, reduce would create a new StringBuilder for each arg

dpsutton19:11:56

reduce str is almost always wrong

noisesmith19:11:47

and (reduce str [x]) wouldn't even make a string

mpenet19:11:56

It's also quite easy to write a reducing fn for turning things into a stringbuilder in a single pass. Then you can transduce with it

noisesmith19:11:47

(ins)user=> (reduce str [42])
42
(ins)user=>  (apply str [42])
"42"

dpsutton19:11:32

i always felt like clojure.string/join would be more optimized but now i see it just delegates to (apply str coll) lol

borkdude19:11:45

but str is optimized

dpsutton19:11:34

i've never actually looked at the source until now to be honest

didibus23:11:26

Hum, so you can't do?:

(create-ns 'foo.bar)
(require 'foo.bar)

noisesmith23:11:48

what would the require do?

noisesmith23:11:05

or would you be running it for :as

3
noisesmith23:11:36

use alias, that's what :asdoes under the hood

didibus23:11:07

Fair enough, I just thought it was strange.

noisesmith23:11:33

clojure uses a mutable database internally to keep track of what's been loaded

noisesmith23:11:43

just creating the ns doesn't update the db

didibus23:11:32

What's weird though is that create-ns will look it up and return the existing ns if there is one, or create a new one. So I'd have assume if creating a new one, it be added as well

didibus23:11:54

Also, something doesn't add up, because: (alias 'foo 'foo.bar) fails if you don't first call (create-ns 'foo.bar)

noisesmith23:11:07

alias needs the ns to exist

didibus23:11:17

So clearly create-ns adds it to some global collection of namespaces

noisesmith23:11:27

no, that's not what alias is using

noisesmith23:11:30

$ clj
Clojure 1.10.1
(cmd)user=> (contains? (loaded-libs) 'clojure.set)
false
(ins)user=> (require 'clojure.set)
nil
(cmd)user=> (contains? (loaded-libs) 'clojure.set)
true
(ins)user=> (create-ns 'foo.bar)
#object[clojure.lang.Namespace 0x2bffa76d "foo.bar"]
(ins)user=> (contains? (loaded-libs) 'foo.bar)
false

noisesmith23:11:46

well, it's not using the one loaded-libs provides, at the very least

didibus23:11:17

Right, so it means there are two different collections of namespaces. I guess one for loaded and one for not loaded ?

noisesmith23:11:17

it's likely using the list returned by all-ns

noisesmith23:11:49

all-ns is all namespaces, only some are considered loaded

noisesmith23:11:11

but all-ns is a list of namespace objects, it's not keyed for lookup

didibus23:11:11

Those would only be the ones created with create-ns 😛

noisesmith23:11:33

it is all the namespaces period, no matter how you create them

didibus23:11:55

But what other scenario is a namespace created but not loaded?

noisesmith23:11:20

wait, by "those" in "those would only be the ones created by create-ns" what do you mean?

noisesmith23:11:30

because all-ns contains every namespace object period

didibus23:11:37

Like, I'd expect that I can: create-ns which creates the ns, and then load it with require, but it seems you can't

Alex Miller (Clojure team)23:11:08

require is about load

💯 3
Alex Miller (Clojure team)23:11:23

create-ns is about in memory

Alex Miller (Clojure team)23:11:41

Which happens as a side effect of load

didibus23:11:17

But ns can create a namespace in memory, and then let you require it?

Alex Miller (Clojure team)23:11:40

But require is about load and there is nothing to load

didibus23:11:57

What is there to load with ns ?

didibus23:11:31

;; At repl:
(ns foo.bar)
(in-ns 'some-other)
(require 'foo.bar)

Alex Miller (Clojure team)23:11:51

If it’s already loaded, the load is short circuited

noisesmith23:11:52

@didibus things like :as and :refer etc. in require are conveniences, and that usage of require is a no-op

didibus23:11:22

So (ns foo.bar) will create the namespace and load it?

noisesmith23:11:45

and put you in it too