Fork me on GitHub
#clojure
<
2021-06-29
>
grav06:06:21

Why is there a #queue tagged literal in ClojureScript, but not in Clojure?

quoll11:06:23

Clojure has a built-in persistent queue but it’s awkward to work with it. You need to do things like: (into clojure.lang.PersistentQueue/EMPTY [1 2 3]) I’m guessing that Rich built it because it’s a described structure in Okasaki, but he didn’t see enough general utility in the structure to expose it via functions in core, or (eventually) via a reader. (Personally I’ve only needed it once). When ClojureScript readers were written, I’m thinking that David (or Antonio, who’s name is also attached to that line) added queue to the readers thinking that it would be handy to have. After all, queues were ported over, like everything else, so why not make them easier to use? Different developers, different motivations.

quoll11:06:44

The only way I know to find out for sure is to ask them 🙂

grav12:06:37

Makes sense! Thanks for the answer 🙂

grav12:06:04

Oh, btw great defn episode, really enjoyed it @U051N6TTC :)

quoll12:06:54

Thank you

quoll12:06:14

(I’ve no idea what I said 😆)

😄 2
grav06:06:44

Wouldn't you need to tokenize beforehand so that each element is a pair of string,number? Something like

(->> ["item_10" "item_2"] (map #(re-matches #"(.+?)(\d+)" %)) (map (fn [[_ a b]] [a (Integer/parseInt b)])) sort)

grav06:06:23

If there isn't a regular pattern in each input element, then I'd immediately think that sorting would be ambiguous

grav06:06:03

Eg ["10_item" "item_2"]

robert-stuttaford07:06:50

yeah @U066U8JQJ’s gist is what i needed!

🙏 2
Endre Bakken Stovner14:06:36

It is usually called version sort or natural sort.

thanks3 2
henryw37410:06:08

I'm looking for a function that gives me a snapshot of jvm stats (heap, threads, classloading, gc...) as clojure data. I'm aware of jvm metrics libs like micrometer and dropwizard, but I'm looking for something dependency free. not too hard to create myself, but ofc if it exists I'll use it 😉

colinkahn11:06:11

is there anyway to get clojure.pprint/pprint to work with *print-meta*?

delaguardo11:06:58

(binding [*print-meta* true]
  (clojure.pprint/pprint (with-meta {:foo 1} {:bar 2})))
;; => ^{:bar 2} {:foo 1}

💡 2
colinkahn12:06:25

I found this - https://clojure.atlassian.net/browse/CLJ-1445 and suspecting I’m on an older Clojure version perhaps since the above example doesn’t print the meta for me

colinkahn12:06:15

Yep, Clojure 1.10.1

kenny15:06:19

We have started to get a "java.io.IOException: Connection reset by peer" in our production environment (stacktrace in thread). The exception is coming from a call using the Java 11 HTTP client. We have not changed our Java version (running Azul 11.0.10-11.45.27) in some time. I have seen the exception come from a call to two different services. One that we are running internally (AWS API Gateway -> Datomic NLB -> Ion HTTP), and the other is Auth0's management API. This exception indicates that the remote server has dropped the connection. The part I cannot figure out is why it has "suddenly" started occurring. We have been continuously deploying new releases, and some of those releases may have changed the classpath deps, but I cannot see how any dep change would affect the Java 11 HTTP client library. I'm a bit at a loss on how to debug this. Does anyone have any insights into this particular exception or what could provoke something like this?

kenny15:06:06

elided internal code calls

Alex Miller (Clojure team)16:06:18

while you may not have changed anything, AWS may have

Alex Miller (Clojure team)16:06:47

just another variable to consider

kenny16:06:13

My thought as well. Likelihood slightly decreased by the two different services causing the issue. No idea if Auth0 runs on AWS & uses the same services.

kenny16:06:13

My current thought on a fix is to just retry those exceptions since it seems that a subsequent call will go through. It feels so wrong since I have such a poor understanding of the root problem.

Alex Miller (Clojure team)16:06:09

seems similar to things datomic team spent a whole lot of time looking at (was aws related), but I don't know the details

👀 2
seancorfield16:06:42

We've also seen an increase of this error with our AWS-based services (mostly Elastic Search but also a few third party services that are hosted on AWS).

😮 2
kenny16:06:58

I have found some references to AWS services on Auth0's blog, so I think they have at least some things running there. I could totally see this being related to an AWS level change. If that is so, tracking that down to a repro to present to them seems like a very time consuming task 😬

rutledgepaulv16:06:14

there are a lot of subtle things around nlbs... let me dig up some posts

seancorfield16:06:42

We also see more random timeouts as well, so we are having to build in more retry logic etc.

kenny16:06:09

@U04V70XH6 thanks for that info. Very helpful. How are you thinking to tackle it? Retry IOException w/ "Connection reset by peer" string?

rutledgepaulv16:06:17

when did medium add a paywall? 😞

kenny16:06:32

Thanks @U5RCSJ6BB. Will read through. I'm definitely not an expert on the intimate details of NLBs.

seancorfield16:06:10

@U083D6HK9 Two-fold approach: yes, retry on certain IOExceptions and also add explicit timeouts and retries on some calls. Not all calls can safely be retried so there's no one-size-fits-all approach 😐 It's very annoying...

seancorfield16:06:37

@U5RCSJ6BB Medium's had a $5/month paywall for a long time. I pay. Lots of great content on there.

kenny16:06:10

If you have the info readily available @U04V70XH6, what other IOExceptions are you retrying?

seancorfield16:06:25

(I think I pay about $40/month now for written content online?)

seancorfield16:06:39

@U083D6HK9 I don't have specifics I can share, sorry.

2
kenny16:06:36

@U04V70XH6 would you be able to share which HTTP client library you are using that you've been seeing these connection reset messages?

seancorfield16:06:20

(we have legacy code that still requires Java 8 even though most of our code is on Java 11)

kenny18:06:36

@U5RCSJ6BB That first article was great! I am definitely seeing a good number of TCP_Target_Reset_Count in the CW metrics. However, they don't align as perfectly as I'd like with when I am seeing the connection reset exception thrown. I am now wondering if the IOException "Connection reset by peer" exception can be thrown in a situation that isn't the remove server sending a RST packet.

👍 2
nate sire20:06:18

It sounds like you are exceeding the limit on the # of connections allowed in a connection pool

nate sire20:06:42

if one more IO client connects... it can bump an earlier client out of the pool

nate sire20:06:05

if this is for Solr/Elastic Search... there is a config to set the pool limit

nate sire20:06:28

I have seen that error when concurrently calling either Solr/ElasticSearch pools or PostGres pools

kenny20:06:16

It's the Java 11 HTTP client.

nate sire20:06:16

the client is saying... something closed it's connection

nate sire20:06:49

since it's happening with both AWS services... it sounds like a scaling issue...

nate sire20:06:05

are you scaling AWS up and down? burst traffic?

nate sire20:06:19

a retry is a good idea... no API calls are ever 100% especially when AWS scales up and down

nate sire20:06:19

but if the error is happening frequently... I would examine the types of tcp connections... persistant versus non persistant

nate sire20:06:57

have to look at the % failure rate

seancorfield20:06:00

@natesire for us this is happening against third-party services that are hosted on AWS so we can't adjust their services 🙂

seancorfield20:06:35

But, yes, retries is the most effective approach we've found 🙂 for service calls that can safely be retried.

2
nate sire20:06:24

ahh... AWS built in services... no controls over your Ec2 or NLB

nate sire20:06:29

does it happen frequently? or just .01%

nate sire20:06:09

I mostly run my own stuff on Ec2 and beanstalk... interested to know about it

seancorfield20:06:50

For us it happens in short bursts "occasionally". We'll get a flurry of either connection reset and/or connection timed out errors from a particular service and then it's just fine. For a while we were actually suspecting some sort of network glitches in the data center we use but then we saw that, while that was causing some of this behavior, it seemed that more of it was caused by something at AWS since we were only seeing connection issues for AWS-hosted services in most of the interruption windows.

2
seancorfield20:06:55

(funnily enough, we just saw a short burst of connection timeouts while this thread has been active: immediately before and after, everything looked just fine, latency was good, response times were good, then a few dozen connection timeouts for maybe a minute, and back to normal)

kenny20:06:57

In the middle of one of these "bursts" of failures, I actually got an email from AWS on "Amazon EC2 Instance Retirement" on one of the instances behind the NLB, which confused me even more, haha. I don't think that's related anymore though.

nate sire20:06:32

sounds like AWS is using their burstable instances to run these services... I'd have to check if you can configure those...

nate sire20:06:05

I am guessing that's the root cause. I need to update my AWS certs....this is the exact stuff they ask on their tests

nate sire20:06:59

I can give you the ultimate solution...

nate sire20:06:16

use GraphQL... it has a que that handles this exact problem

nate sire20:06:48

the que can backup during AWS scale up events... and retry the call

seancorfield20:06:49

Heh, if only the services we used had a GraphQL API... 🙂

nate sire20:06:24

yea... it might only work for some services... I will think on a full proof solution

nate sire20:06:23

it's really the failed calls that are unsafe to retry... that you need solved

nate sire20:06:01

I have an idea for you...

nate sire21:06:21

check what size your ElasticSearch instances are....you can try smaller instances that scale up/down more frequently. It could be that your instance size is too large... so when another instance is spun up... the spin up time takes too long.

nate sire21:06:53

hence the burst of failed connections

nate sire21:06:45

you can adjust many config options on services like ElasticSearch....to match the cpu, memory, shards with the needed instance sizes

nate sire21:06:37

you should be able to spin up an extra instance ahead of time... before the burst of failed connections

nate sire21:06:53

CloudWatch on AWS can help you tune the config

Emi Gal19:06:02

New to Clojure and have been using Atom + Chlorine, but have heard great things about VS Code + Calva. I was wondering what differences / pros / cons there are between the two? Should I switch?

seancorfield20:06:47

Chlorine and Clover are essentially the same, using a Socket REPL, which I like. Calva is great: it provides a lot of useful functionality and you can use it with Clover. In the Calva settings, turn the nREPL UI off and then you get all of Calva's great features with Clover's Socket REPL instead of nREPL.

seancorfield20:06:04

I switched because Atom just doesn't have the same level of support as VS Code: VS Code is updated more often, has much better/broader extension support, and is much more widely used. Since you're already using Chlorine, you should be able to switch to VS Code + Clover and retain the exact same editing experience -- you can have the same hot keys, the same config.cljs customization, and so on.

seancorfield20:06:07

Then you can add Calva (and turn off the nREPL UI), and benefit from built-in LSP (static analysis), clj-kondo, auto-formatting, great paredit support, rainbow parens, etc. Without needing a whole bunch of extensions added -- just Clover and Calva.

p-himik20:06:35

@U04V70XH6 Since you're here - do you work with jar files in any way? With IDEA+Cursive it's so easy, I sometimes don't get it how people don't look at the sources of libraries they use. It's as easy as looking at your own files, even if a library is in a jar with compiled classes. Last time I checked, there was no Java support for VSCode that would be advanced enough for that kind of navigation/search.

seancorfield20:06:43

Non-Clojure JAR files you mean?

seancorfield20:06:10

I can see the docs or source for any Clojure code, even in JAR files...

jcsims20:06:01

Java dependencies, in particular - IntelliJ has first-class support for downloading (and decompiling, if necessary) java deps

jcsims20:06:14

something I miss in... anything else, really

p-himik20:06:35

Sorry, right - I meant Java jars in particular.

seancorfield20:06:55

In over a decade of working with Clojure, I've never found a need to do that...

p-himik20:06:42

Just like some people never seem to see how a REPL-DD could be useful. ;) Just as a single example - for me personally, I just find it to be so much faster, easier, and more convenient in general to understand how to work with some library as compared to reading docs. It's not always the case, but it usually is.

p-himik20:06:06

Even to just better understand how Clojure itself works - I'd rather read Compile.java and know exactly how something works then rely on my ability to find the doc that's just right and not outdated or incomplete. And being able to navigate to all the .java files of Clojure right from its .clj files is a bliss.

seancorfield20:06:56

I have the Clojure repo locally so it's easy enough to browse if I ever want to look at the Java code. ¯\(ツ)

seancorfield20:06:22

I just don't find I need to look at the source of libraries very much...

p-himik20:06:30

Fair. Alas, for me it's pretty much the only reason why my move from IntelliJ to VSCode is so slow. I've started using the latter but for projects that don't involve Clojure.

seancorfield20:06:19

I only installed the Java extensions to VS Code very recently. I'd resisted for quite a while. I was working on some tools.deps.alpha stuff and finally got sufficiently annoyed at being prompted about Java extensions that I gave in and let it install them 😐

seancorfield20:06:26

Now it just pesters me to "import Java projects" which I'm mostly ignoring 🙂

pez20:06:20

Since you are new to Clojure, @U01UV7G7PJ9, give Calva a spin. See https://calva.io/getting-started/ and feel very welcome to chat us up in #calva

🙏 2
Emi Gal21:06:27

That’s fantastic, thank you for the response @U04V70XH6! Found your blog post super useful too. Gonna give VS Code a spin right now. Quick question: if I understand your setup correctly, you’re basically using Calva for formatting, linting, tooling, etc., and Clover for running a socket REPL instead of Calva’s default nREPL. Apologies if this is a super noob question, but what’s the benefit of using socket REPL vs nREPL?

seancorfield22:06:05

@U01UV7G7PJ9 Since you are using Atom + Chlorine, I assumed you were already using a Socket REPL?

seancorfield22:06:38

If you were using Chlorine with nREPL, you might just as well use Calva on its own, without Clover (and leave the nREPL UI enabled).

Emi Gal23:06:27

@U04V70XH6 sorry - yes, was using Atom + Chlorine with socket REPL obviously, was just wondering whether there are any benefits to using nREPL?

Emi Gal23:06:08

BTW I’m playing with VS Code + Calva + Clover right now and it’s awesome

seancorfield23:06:06

I prefer simpler tools, so I avoid nREPL/Orchard stuff, and I like that I can start up any process that includes Clojure and specify a JVM option and get a Socket REPL with no code needed.

seancorfield23:06:08

There are definitely advocates of nREPL here who will extol the virtues of it and the tooling based on it, but I just don't want that much code between my editor and my program. Definitely a YMMV situation.

Emi Gal23:06:04

Thought that might be the case 🙂. As a beginner, I found a socket REPL much easier / simpler to grasp than an nREPL, which is what drew me to the Atom + Chlorine setup.

Emi Gal23:06:23

Thanks Sean, appreciate your time. Loving the Clojure community!

seancorfield23:06:41

I have my .clojure/deps.edn file up on GitHub, along with my config for both Atom/Chlorine and VS Code/Clover/Calva, in case you want to take a look...

Emi Gal01:06:36

Cloned your vs code / clover github - love it! (had to make the code font smaller though 😀)

seancorfield02:06:24

Haha... I did that partly for doing screen shares... And partly because I'm old and my eyes are going 👀

🙃 3
mauricio.szabo13:07:24

Charlie, about the nREPL and Socket-REPL: nREPL is more powerful, Socket-REPL is simpler. In fact, most (if not all) Clojure implementations implement Socket REPLs (Lumo, Plank, BB, Clojure, Clojerl, ClojureCLR, etc) and only a few implement nREPL (Clojure, ClojureCLR and BB AFAIK). You can do everything you do in nREPL with a socket REPL, but it's way harder and sometimes it doesn't work that well. That being said, I also only use a socket-repl so maybe it's not a big deal 😄

🙏 3
nate sire22:06:28

I am still working on a way to format lein's output... so the java stack and errors sits at the bottom of my screen. I had to switch my monitors to vertical....to see the java errors. Anyone try using tail or reverse to pretty format the lein output?

nate sire22:06:18

I would really like to switch my monitors back to landscape (wide) and find way to focus the output on the java error... not 100s of lines

nate sire22:06:55

maybe I should write a grep script to filter all java errors out of the stack trace?

dpsutton22:06:13

if you're at a repl clojure.repl/pst is quite handy and does some of this. not sure what your dev setup looks like though

nate sire22:06:36

yes, REPL helps... I mean at the command line

nate sire22:06:03

when I am auto testing some tests... or looking at the raw stack trace of leiningen

nate sire22:06:34

does almost everyone just rely on the REPL to eval their code? not stack traces?

Alex Miller (Clojure team)22:06:37

can I ask which version of leiningen and Clojure you are using? some of the default behavior changed in Clojure 1.10.1

nate sire22:06:57

ok... I will check that now

nate sire22:06:25

Leiningen 2.9.1 on Java 11.0.11 OpenJDK 64-Bit Server VM

nate sire22:06:37

let me check on the latest version if it's different...

dpsutton22:06:47

(clojure-version) will get you the clojure version

Alex Miller (Clojure team)22:06:53

the difference is from your Clojure version

Alex Miller (Clojure team)22:06:14

but it might depend exactly what command you're running as well (lein test, lein run, something else)

nate sire22:06:54

clojure version

Alex Miller (Clojure team)22:06:37

and are you running lein test or something else?

nate sire22:06:29

I am running tests with lein... inside a shell

Alex Miller (Clojure team)22:06:38

and this is an exception thrown during the test?

nate sire22:06:06

and normally... the useful java error will be 100 lines above the bottom of the shell

Alex Miller (Clojure team)22:06:10

ok, so it's really clojure.test error reporting in the context of lein test (which iirc monkey patches part of clojure.test)

🙌 2
nate sire22:06:13

so I switched my monitors to vertical

Alex Miller (Clojure team)22:06:23

I'm just trying to be clear on what you're seeing. we've spent a lot of time trying to reduce places where you get thrown a stack trace out of clojure and instead format a useful error but in some cases the tooling is doing its own thing

nate sire22:06:45

let me write a quick demo... I will show you a photo of it

nate sire22:06:49

give me a few mins

Alex Miller (Clojure team)22:06:15

that's ok, I've repro'ed it

Alex Miller (Clojure team)22:06:19

in this case, it's clojure.test error reporting

nate sire22:06:44

now if I scroll up... I can see the useful error... like this...

nate sire22:06:07

Exception in thread "main" Syntax error compiling at (hello/core.clj:7:3).

nate sire22:06:55

but the java stack trace can be hundreds of lines... so I switched my monitor to vertical... I was going to try some ideas to format and filter out the most useful part... or maybe... reverse it

Alex Miller (Clojure team)22:06:26

there is actually a dynamic var to set the max stack trace depth at clojure.test/*stack-trace-depth* but I'm not exactly sure how to bind that appropriately

nate sire22:06:27

bash tail or reverse has a ton of options... with grep

Alex Miller (Clojure team)22:06:39

I guess you could use a fixture in your tests

nate sire22:06:49

ahhh. ok I will try that first...

dominicm22:06:06

lein has a thing for setting dynamic vars iirc.

nate sire22:06:06

I think the bash approach could become tedious... maybe not 99% effective either

Alex Miller (Clojure team)22:06:52

I'm not sure that dynvar exists yet to set there

nate sire22:06:35

does most engineers just use some IDE/Vim plugins to run tests?

Alex Miller (Clojure team)22:06:37

(defn short-stack [f]
  (binding [clojure.test/*stack-trace-depth* 5]
    (f)))

(use-fixtures :once short-stack)

Alex Miller (Clojure team)22:06:56

^^ that should work if you put it in each test file (which is tedious I admit)

nate sire22:06:24

ok... I will play with it and try to bind it into my tooling

nate sire22:06:48

I will follow up... I have always run my tests from the shell on any stack... so it's just a nice thing to have with the tooling

Alex Miller (Clojure team)22:06:56

I think there are a wide range of approaches so hard to say "most" do any particular thing

Alex Miller (Clojure team)22:06:08

but this is something I'm interested in improving the default case

nate sire22:06:33

you bet... I think it helps with the java error handling that everyone often complains about

nate sire22:06:56

it's not a show stopper... just makes it more friendly... tooling... when first learning Clojure

dominicm22:06:16

Hmm, with lein test I was unable to get a stacktrace. I only got the new .edn file stuff.

Alex Miller (Clojure team)22:06:17

yep. the other option here would be to redefine the clojure.test/report :error method to set a default

nate sire22:06:45

@dominicm you can also just try "lein run"

nate sire22:06:57

scaffold a lein app... lein new app helloworld

Alex Miller (Clojure team)22:06:10

lein run with clojure 1.10.1 should not print a stack trace anymore if an exception is thrown out of main

nate sire22:06:37

I purposely created an error in main... just to demo this

dominicm22:06:29

I got this:

/tmp/helloapptest 8s
❯ lein test                
Syntax error compiling at (helloapptest/core.clj:7:3).
Unable to resolve symbol: i in this context

Full report at:
/tmp/clojure-12954933124682389143.edn
Tests failed.

Alex Miller (Clojure team)22:06:43

that's a compilation error on load

borkdude22:06:48

that doesn't even get to the test runner

nate sire22:06:51

(ns hello.core (:gen-class)) (defn short-stack [f] (binding [clojure.test/stack-trace-depth 5] (f))) (use-fixtures :once short-stack) (defn -main "I don't do a whole lot ... yet." [& args] (println "Hello, World!"i))

Alex Miller (Clojure team)22:06:59

but if you have something like (throw (ex-info "I suck" {:a 1})) in that test that will work

dominicm22:06:35

@natesire Same thing for lein run:

/tmp/helloapptest
❯ lein run 
Syntax error compiling at (helloapptest/core.clj:7:3).
Unable to resolve symbol: i in this context

Full report at:
/tmp/clojure-14415791333264548638.edn

Alex Miller (Clojure team)22:06:49

these are all not what was reported above

Alex Miller (Clojure team)22:06:58

you need a deftest that throws at runtime

Alex Miller (Clojure team)22:06:15

then clojure.test reports the error with full stack in clojure.test/report :error

nate sire22:06:16

I have not upgraded anything in a few months...

dominicm22:06:18

@alexmiller I'm going off the screenshots above, which are a syntax error.

nate sire22:06:44

it sounds like it's already being addressed... awesome

nate sire22:06:49

wish I could help more

dominicm22:06:51

Actually, also the example above ^ which is exactly what I'm doing 😛

Alex Miller (Clojure team)22:06:13

oh, I didn't look at the screenshots. yeah, that case should already be handled.

nate sire22:06:24

I just need to upgrade.... awesome

nate sire22:06:01

I really like Clojure... I think once we have a web toolchain... this language could skyrocket

didibus23:06:49

FYI, there are multiple web toolchains already

j07:06:35

> so many that's the problem like there are tens of them in ruby, but, also, theres basically Rails and Sinatra, both of which are fairly complete, and then theres a bunch of mess if you wanna go your own way whereas in clj you are by default going your own way, and can have a second class citizen experience by using one of a few weird frameworks, last i looked insert xkcd standards here

Drew Verlee11:06:07

You can produce a rails like app using luminous.

Drew Verlee11:06:08

I'm not sure what would be missing as I haven't messed with rails for a number of years.

Drew Verlee11:06:59

I recall a lot of logic I wouldn't want now being bunded into core rails. Orm, etc..

JohnJ14:06:18

It hasn't in 10 years, it won't magically skyrocket now ;)

nate sire20:06:07

Python was around almost 20 years before it skyrocketed... I think Clojure has an awesome future. It just is not hyped. It is following a more steady growth path.

didibus21:07:48

Maybe because I know how to do web apps, but I really don't see what's so complicated about it. You don't need any framework, all you need is a good webserver, and there are plenty good ones like Jetty or Netty.

didibus21:07:24

Than for routing and managing the HTML reply/responses, people pretty much universally use Ring with Compojure or Reitit or Bidi, or you just use Pedestal with Pedestal router

didibus21:07:53

For HTML templating you can go Hiccup or Selmer. And for DB access use whatever Java client for noSQL, or use next.jdbc for SQL

Drew Verlee16:07:06

Given the combined documentation for fulcro and shadow is close to half a thousand pages, I think its fair to say the range of how complex even a front end application can be is fairly wide. And thats within just one context (shadow + fulcro) if you expand that to cover the ecosystem, it's impossible to know the specifics of it all.

didibus07:07:27

Those include a whole set of things related to client side SPA JavaScript applications though, which something like Rails doesn't have. I wouldn't be surprised if 95% of their documentation revolved around that.