Fork me on GitHub
#clojure
<
2021-09-10
>
raspasov01:09:27

If possible, I try to avoid closing channels. It makes for a simpler code, IMO

raspasov01:09:18

Closing a channel feels a bit akin to setting the state of an object. It can be used, but make sure you really need it. Perhaps there’s a way to avoid closing the channel in the first place.

zackteo02:09:43

Hello 🙂 perhaps a long shot but are there any public examples of Clojure being used for a server requiring high concurrency? Like for example that of messaging applications like whatsapp (but not nearly that insane levels)?

zackteo02:09:03

Am thinking in most of such cases people use things like erlang when factoring downtime?

seancorfield04:09:28

@zackteo define "high concurrency". http-kit claims to have very high throughput, as I recall.

seancorfield04:09:01

We have a "high concurrency" messaging app at work, using nettyio but of course not public.

seancorfield04:09:23

(I think many people think they need WhatsApp/Facebook/Twitter scale but they really don't, and it's amazing how much traffic you can handle on a pretty basic JVM-based HTTP server these days)

2
zackteo05:09:07

@seancorfield am looking at something like a customer engagement suite that will need to send for example whatsapp messages to perhaps an average of 100k people (with a higher upper limit)

zackteo05:09:15

I think the sending is not the issue as much dealing with the responses. Also because the from what i understand with the example of whatsapp api, the one using the api needs to handle the open connections? Like something like 1 phone number to 10k people or something

seancorfield05:09:27

Perhaps, yeah. We send hundreds of thousands of emails a day and track deliverability and that's really no big deal (we used to send 2-3M emails a day but deliverability analysis showed that was not worthwhile -- better to send fewer, more targeted messages). So we have a dedicated process for just handling email deliverability reports -- which are posted by Perl scripts running on a PowerMTA box. Focus on stateless POSTs and avoid open HTTP connections if you can.

seancorfield05:09:49

(a dedicated stateless process running on three servers, to clarify)

seancorfield05:09:47

We also send a lot of web push notifications and deal with deliverability of those. Again, it's all done outside of HTTP keep-alive connections.

zackteo05:09:20

Actually could I ask what are the stateless POSTs?

seancorfield05:09:46

I mean that the POST is self-contained and can go to any server in the cluster.

seancorfield05:09:02

Each POST is "atomic".

seancorfield05:09:19

(i.e., avoid sticky sessions etc)

zackteo05:09:50

Am also wondering if messaging has some difference in that the sent messages and replies need to be written to a database to track the conversation

zackteo05:09:40

Hmmm right, I think i still lack the knowledge and experience to understand what that might fully entail in a real world application

seancorfield05:09:50

Our messaging server process writes everything to the DB. Uses a GraphQL API and WebSockets and NettyIO.

seancorfield05:09:21

(and just three instances running in the cluster -- but they are fairly high memory)

zackteo05:09:02

I see I see! I guess I will try to do more of my own research. But do agree, think we often think the use cases need the same infrastructure of that of the tech giants like perhaps use of cassandraDB but usually that is not true

seancorfield05:09:04

We got very enamored of MongoDB at one point because of the "web scale" hype but after a few years we migrated everything back into our "huge" Percona (MySQL) 5.7 instance and we were better off.

seancorfield05:09:34

It's truly amazing what you can achieve with some pretty boring software and hardware.

zackteo05:09:34

But yeah, want to see if there a good reason to propose the use of clojure somewhere in the tech stack in such an application. But reasonably will end up using the standard like that of java, nodejs etc

seancorfield05:09:01

Clojure wins on the development side, not on the scale/performance side, in general.

Ben Sless07:09:13

I must say wrt performance, just like most applications are not on Twitter FB scale, neither are most performance problems. You're more likely to find blocking calls on the event loop before you need to solve Clojure's performance

💯 2
Ben Sless07:09:50

And even when you do, Clojure can go much farther than most would initially guess. Yes, you might have to lie cheat and steal to make the JVM happy but it can mostly be hidden away at libraries while the business layer remains ergonomic

❤️ 2
2
seancorfield05:09:02

There are some areas where the immutability is a win -- ClojureScript does better here with Reagent etc compared to plain React.js -- but mostly Clojure wins on simplicity and maintainabiity.

seancorfield05:09:38

For us, when I introduced Clojure a decade ago it was about abstraction, first and foremost, and immutability second.

zackteo05:09:24

I realise also that now i have started work, some of the issues with adopting new technologies like clojure, is that it along with other jvm languages might have no proper testing analysis pipeline to go through even in a Java shop

seancorfield05:09:48

I'm taking this to a thread at this point...

seancorfield05:09:03

That's a cultural issue tho' (testing) not a technical one.

seancorfield05:09:33

We can got from commit to QA in about 20-25 minutes and then to production in another 15.

zackteo05:09:41

but i guess those are metrics management uses to measure how well departments are doing

seancorfield05:09:17

We're aiming to get that down to about 10 minutes (commit to QA) but the 15 minutes is a reasonable rollout across our production cluster, fully automated.

seancorfield05:09:49

Clojure/Java doesn't really affect that.

seancorfield05:09:05

Clojure's value prop is entirely in dev/maintenance.

seancorfield05:09:46

Well, it eliminates an entire class of production errors (due to state), but really it's about productivity in terms of adding features.

seancorfield05:09:36

We've measured a lot of stuff over the years, and we can get new features to production with Clojure faster than other stuff we've used in the past -- even supposedly Rapid Application Development stuff.

seancorfield05:09:49

Language is really a very minor part of overall company efficiency. Automated testing, automated (continuous) integration, and automated (continuous) deployment have much bigger impacts on a company's ability to move nimbly forward.

seancorfield05:09:40

We adopted Clojure a decade ago but it's really only been in the last couple of years that we've managed to implement reliable CI/CD and that has been the game-changer for overall impact.

seancorfield05:09:34

Sure, Clojure has allowed us to make software changes much faster -- and at much bigger scale in terms of business logic -- but the real productivity boosts have come from that fast CI/CD cycle.

zackteo05:09:44

and guess without the CI/CD pipeline it doesn't make sense to use other languages, not to mention issues of hiring and maintenance

seancorfield05:09:18

Yeah, to be honest, the language is irrelevant if you don't have CI/CD in place, for the most part.

seancorfield05:09:39

And, yeah, Clojure makes some aspects of hiring harder (but also some aspects easier).

seancorfield05:09:58

Does where you work have a fully automated testing/deployment process? A fully automated production rollout?

seancorfield05:09:39

Does it have a formal pull request review process?

seancorfield05:09:51

Those are fundamental.

zackteo05:09:51

I think they do .... i haven't been too involved with any projects at the moment to really know

zackteo05:09:32

But there is a fully automated production rollout from what i understand

seancorfield06:09:22

A fully-automated CI/CD system is far more important than the language you're using.

seancorfield06:09:01

The language is about developer pain/happiness, so it's important too -- but not as important as some of the other "basics".

seancorfield06:09:01

Although, if a different language makes your developers a lot happier... 🙂

zackteo06:09:54

I've been trying to figure out where I get fulfilment out of work - having just graduated about a month ago and just started work

seancorfield06:09:57

Yeah, that's a tricky one. I graduated university in '83, did PhD research for three years, and have worked professionally ever since, and I will say not all jobs are fulfilling and even the best job has some miserable times.

seancorfield06:09:26

I mostly love my job right now -- ten years for the same company, 100% remote, a lot of autonomy (I introduced Scala but it didn't stick and I introduced Clojure and it did stick).

seancorfield06:09:13

I've been very fortunate: most of my jobs have been very interesting and very challenging -- which is not true for most people in IT. I think a lot of people are a bit rose-tinted about the IT world 😞

zackteo06:09:42

Is quite silly that I'm already having a small existential crisis of sorts

seancorfield06:09:39

Yeah, I don't want to be insensitive but you have decades ahead of you and you're already worrying too much 🙂

zackteo06:09:09

Perhaps it is also fuelled by a worry if my current role will lead me to where i want to be couple of years from now

zackteo06:09:34

Yeap, no offence taken 😄

zackteo06:09:37

I guess it is about perspective and just making best of my situation

seancorfield06:09:23

I think career growth is orthogonal to language choice?

zackteo06:09:04

Right 😅 that's true hmmm

Ben Sless07:09:36

It's also not the end of the world to change direction later in life. Don't let it bother you.

👍 2
raspasov06:09:19

@zackteo Erlang has a built-in framework to do distributed actors, as far as I know (never used Erlang myself).

raspasov06:09:30

But I have worked on chat applications, in Clojure.

raspasov06:09:21

On the JVM, the equivalent of Erlang-style distributed actors programming would be Akka https://akka.io (also haven’t used it, have only read about it a bit)

raspasov06:09:20

Actors are a big trade-off, as far as I understand. They can achieve massive scale, but the trade-off is that any communication with actors is akin to being “locked inside a mailbox and communicating to the outside world by sending and receiving mail”

raspasov06:09:10

For small to medium/large scale, I would recommend using a message queue.

raspasov06:09:09

Amazon SQS is a nice managed option; alternatively, there’s RabbitMQ (which is actually built in Erlang itself)

raspasov06:09:27

I have a bit of experience with both, but it has been a while.

zackteo06:09:54

Thank you! 😄

👍 2
hiredman06:09:33

even if you use actors you'll need a queue

hiredman06:09:30

for almost any application the choice of implementation language is never the limiting factor, in general the limit is some dumb decision you made last year which means you have to poll for updates instead of getting them pushed to you

😂 8
raspasov06:09:37

(have not used this AWS service myself)

simongray08:09:54

I have an endpoint that I want to serve files based on the URL params, so I’m constructing a file path using string concatenation. How can I make sure that syntax like . or .. doesn’t work? And these aren’t public files, so they don’t go into the regular public resources directory with all of the other static files.

2
simongray08:09:42

Now I’m doing

(io/file (str files-dir fmt "/" filename))
but this is unsafe as filename can easily contain ../../other.file

p-himik08:09:02

It's worth mentioning that the first line of defense should be the right file permissions. Regarding dots - you can get the relative path from the URL, make it absolute, then make it canonical. Compare the canonical path to the absolute one - if they're different then something is not right.

simongray08:09:46

Point taken about file permissions.

opqdonut08:09:21

also if you're using ring you can just use a handler somebody else made and hopefully considered security

simongray08:09:22

I’m using pedestal

p-himik08:09:26

To be more clear - just file permissions are not enough of course since the process has to be able to access its own classpath. Which, I imagine, you don't want to serve via URLs.

opqdonut08:09:13

ring.middleware.file does the canonical path check that p-himik mentioned: https://github.com/ring-clojure/ring/blob/master/ring-core/src/ring/util/response.clj#L91

🙏 2
simongray08:09:11

Yeah, like I mentioned these are not public resources

simongray08:09:33

I use the resource path in pedestal for public resources like css files and so on.

simongray09:09:06

@US1LTFF6D I think I’ll just include Ring as a dependency and use those functions rather than copy them

hiredman09:09:13

Why user given names for the files at all? Use uuid names, keeping a map of user given file names to uuids somewhere else

simongray09:09:17

thanks for linking to the specific functions

👍 2
simongray09:09:06

@hiredman All of the files reference each other and already use the filenames as identifiers. I would have to make a two-way mapping which seems like a pointless exercise.

simongray09:09:29

I don’t control the dataset, it’s provided to me

simongray09:09:27

Hmm.. actually maybe I can just use the resource path interceptor from Pedestal and put some auth interceptors in front of it.

simongray09:09:06

(which incidentally reuses Ring)

simongray09:09:35

Pretty simple solution

(defn ->handler
  [files-dir]
  (fn [{:keys [path-params]}]
    (let [{:keys [fmt filename]} path-params
          filepath  (str "/" fmt "/" filename)]
      (ring/file-response filepath {:root files-dir}))))

2
noisesmith16:09:27

btw io/file doesn't require string concatenation:

user=> (io/file "/home" "justin")
#object[java.io.File 0x2b97cc1f "/home/justin"]

noisesmith16:09:38

> just file permissions are not enough of course since the process has to be able to access its own classpath there's no requirement that any part of classpath be openable via http://java.io.File, in a normal app your classpath roots should be jars, the entries of which cannot be opened via FIle

p-himik16:09:14

What do permissions have to do with File ? If the user that launched a Java process doesn't have an access to one of the files on the classpath, the process will fail, will it not?

p-himik16:09:47

Ah, I get what you mean now. But still, if /tmp/x.jar is on my classpath and my process works, won't it be able to just open /tmp/x.jar as a file?

noisesmith16:09:01

that's true, fair point

vemv17:09:37

I'm facing https://github.com/jeremylong/DependencyCheck/issues/3502#issuecomment-916589172 locally (and not in CI). Generally this functionality has worked just fine for a long time. tldr the JDK's native http client can fail opaquely saying nothing much other than unable to connect. From similar issues it looks like a SSL Cert thing. Is there anything one can do to fix certs?

Jon Boone17:09:15

Presumably you have no control over the endpoint with the bad cert?

Jon Boone17:09:17

I think there is a JVM option you can pass to have it ignore bad certs, but you should never use such a thing in production.

vemv17:09:19

I don't think the cert is bad, as the GH comment says the link can be visited w/ Chrome / url

vemv17:09:35

I wonder if this sort of thing would help i.e. upgrade root certs https://community.boomi.com/s/article/howtomigratejavacertificatestonewversionofjava

hiredman17:09:21

doesn't seem to be clojure related, so maybe #clojure is not the best place for this. It seems premature to assume it is an ssl issue

vemv17:09:42

I know it's not clojure related, but people with nice JVM skills hang out here. Also would serve https://github.com/rm-hull/nvd-clojure > It seems premature to assume it is an ssl issue I made an educated guess by also looking at similar, recent issues in the same repo

hiredman17:09:50

the issues on the repo with the fail to download where it was concluded to be an ssl issue have stacktraces that mention ssl on them, do you?

vemv17:09:44

the messages were slightly different (those included SSL, this one not), but the url being hit was the same.

hiredman17:09:32

so what stacktrace do you get?

hiredman17:09:33

it looks like the format of the error reporting is some exception the tool creates (not very useful) followed by the caused by stacktrace (very useful)

hiredman17:09:29

so an abbreviated version that only grabs the first part is not going to tell you anything

hiredman17:09:34

arch or fedora? are you using systemd-resolved?

vemv17:09:18

macOS ...ultimately I can see {:type java.net.SocketTimeoutException :message connect timed out i.e. the original exception and stacktrace. So their ex handling doesn't seem too terrible

hiredman17:09:39

anyway, that stacktrace is a failure to connect, not an ssl cert connected error at all

hiredman17:09:07

so a network connectivity issue, or my wild guess (which is why I asked about the os and whatnot) dns

vemv17:09:36

why else would it fail on java but not curl or Chrome? And only for that particular url?

hiredman17:09:19

I've seen dns setups where java will fail but other tools (chrome, nslookup, ping, etc) work fine

hiredman17:09:30

there isn't really a single standard dns interface that is standard on linux, so different tools can end up behaving differently based on your setup, which is why I was asking about your os

👍 2
hiredman17:09:14

in particular I've seen that in combination when using systemd-resolved and unbound

hiredman17:09:38

but whatever the case, doesn't look to be ssl at all

vemv17:09:39

will try out a few things, like bleeding-edge jdks thanks!

hiredman17:09:14

maybe try messing with the socket timeout settings

hiredman17:09:22

-Dsun.net.client.defaultReadTimeout=30000
-Dsun.net.client.defaultConnectTimeout=5000

hiredman17:09:00

hrrm, those might not effect the httpclient

hiredman17:09:46

oh, they are using URLConnection, and not the cool new java http client, so those settings should work

vemv18:09:59

I eyed those but the failure I'm getting is immediate. It would surprise me if something changed the default 10000 timeout to sth lower good variety of things to try anyway, thanks!

vemv18:09:40

for extra context I kinda don't care about this issue, in CI it works. Most of all I was wondering if there are any "quick wins" or usually-applied fixes. For sake of adding more stuff to the thread, I also tried these for debugging. Didn't show a smoking gun unfortunately

"-Djavax.net.debug=all" ;; or:
"-Djavax.net.debug=ssl"

pez18:09:06

I have this feeling that Clojure is picking up steam. Job postings everywhere and I meet a lot of people interested in the language and community. What if 2021 will be the year where Clojure is breaking to the next level of adaption? Happy Friday everyone!

clojure-spin 48
❤️ 6
💥 4
metal 4
djblue18:09:57

Feels the same way to me. A few years ago I thought it would be impossible to get a clojure job. Today, I have one and so do a bunch of my friends.

metal 8
clojure-spin 6
vemv18:09:43

definitely #jobs #remote-jobs feels much different than circa 2017

Ben Sless19:09:24

Happy indeed. Feels different. Less like a hype and more like "slow and steady"

2
pez19:09:43

Slow and steady wins the race.

🐢 10
Ben Sless19:09:57

If this is going to end up being the year of the linux desktop and the year of clojure I'm throwing a party

🎉 8
Thomas Moerman19:09:57

Cheers to that!

🍸 4
zackteo08:09:15

I haven't had success of getting a Clojure role yet. Perhaps because I'm geographically distant from the Clojure boom 😅 But maybe 2022/2023 will be the year 😄

pez09:09:40

Are you in China, @zackteo ? According to Calva usage the last month, Clojure is picking up at least some steam there.

zackteo09:09:16

@U0ETXRFEW I'm in Singapore! Was close to getting a rare Clojure role in Singapore but perhaps some lack of experience with some policies (like "non-negotiable salary arrangement") costed me the role. And had some others leads but no news soon after :x

pez09:09:08

Crossing my fingers for you landing a Clojure gig soon!

clojure-spin 4
simongray12:09:01

Sweden up by 20%!? Have you been asking all your friends to download it, @U0ETXRFEW? :P

simongray12:09:53

And Canada has Sweden beat!

Ben Sless12:09:18

How is that measured?

pez12:09:43

You mean the increase, @UK0810AQ2?

pez12:09:15

It’s the past four weeks compared to the four-week period before that. Total users as reported by Google Analytics.

pez12:09:16

I think I might be driving Sweden’s figures by starting Calva sessions from new Gitpod workspaces at least 10 times daily, @U4P4NREBY 😃

😁 6
pez12:09:26

The usage pattern over the weeks tells us that Clojure is very much used for business. 😀

🙃 2
2
2
2
Ben Sless12:09:07

Have you tried looking at trends geographically at a lower resolution? I usually find it's centered around companies which use it

pez14:09:09

I haven’t studied it very deeply at all. I just start all days by looking at that graph I just posted. 😃 It makes me happy because most often the current period is visibly above the previous one. It is super rare with that thing you can see there from last Monday, when usage was way below the previous month’s corresponding Monday. Anyone know of a holiday in Brazil last Monday that might explain. 😃

genRaiy21:09:05

I have never looked at datafy and nav and am feeling like asking a dumb question will help me out…

genRaiy21:09:28

(clojure.datafy/nav (clojure.datafy/datafy {:foo {:bar 1}}) :foo :bar)

genRaiy21:09:13

gives back :bar rather than 1

genRaiy21:09:54

I must be holding it wrong … what is the right way … with a trivial example like the above

ghadi21:09:03

Check @seancorfield ‘s articles on it. I don’t have a link handy

genRaiy21:09:24

I’ve read it but there are no simple examples

genRaiy21:09:53

I know he has done it for JDBC but it’s late and I don’t want to pore over complex code to grok something … I have assumed … is simple

hiredman21:09:29

the v you pass to nav there should be {:bar 1}

genRaiy21:09:45

ok, that gives back {:bar 1} which also feels odd

p-himik21:09:16

nav and datafy are not for walking regular Clojure collections. They are for making it possible to walk some non-Clojure objects as if they were Clojure collections. The mentioned article by Sean Corfield: https://corfield.org/blog/2018/12/03/datafy-nav/ Examples and discussions: • https://nextjournal.com/zampino/clojure-datafyhttps://www.reddit.com/r/Clojure/comments/c1w33x/examples_of_datafy/https://ask.clojure.org/index.php/8550/can-someone-explain-the-arguments-to-nav

p-himik21:09:53

You see your v back because nothing extends Navigable:

(extend-protocol Navigable
  Object
  (nav [_ _ x] x))

genRaiy21:09:57

yes, have seen it … again, examples are thin on the ground and that’s how I learn best

hiredman21:09:17

(let [m {:foo {:bar 1}}
      d (clojure.datafy/datafy m)
      v (get d :foo)
      m2 (clojure.datafy/nav d :foo v)
      d2 (clojure.datafy/datafy m2)
      v (get d2 :bar)]
  (clojure.datafy/nav d2 :bar v))

hiredman21:09:08

in :foo {:bar 1}} 1 is 2 levels deep in maps, you are trying to pull it out naving down once

hiredman21:09:13

which is not how it works

genRaiy21:09:56

ah, ok … this is a nice example … thanks @hiredman

genRaiy21:09:26

@p-himik my understanding was that nav works out of the box for maps and vectors … I just hadn’t grokked the model until now 🙂

p-himik22:09:19

Technically, it works for absolutely everything because of that extend-protocol on Object. It's just that it's probably not what you want for types other than Clojure's collections.

seancorfield22:09:10

The model I try to give folks is: datafy takes you from Thing to pure data. get navigates in pure data. nav takes you from that pure data back to a different Thing and you have to give it the key and the value from the get navigate so it knows how/where to go.

seancorfield22:09:28

Thing -> datafy -> data
               /    |
          nav      get
      /             |
Thing'             data'

p-himik22:09:18

Shouldn't Thing' be derived from data' instead of data?

seancorfield22:09:22

No, you nav from data using key/value from the get to a new Thing

seancorfield22:09:44

But it isn't necessarily true that Thing' would datafy back to data' in all cases.

seancorfield22:09:47

A lot of times, nav implementations ignore the value (i.e., they ignore data').

seancorfield22:09:30

In other words, when you call datafy on Thing, and it produces data, it's common to have the implementation of nav attached as metadata.

seancorfield22:09:34

Because nav usually has to know about Thing and it may use either the key and/or value to determine what new Thing to navigate you from the original Thing.

Fredrik22:09:02

https://gist.github.com/sashton/56a14d275c71d7ce3e224c11c9a16bd4 is a really nice example of all the things @seancorfield mentions above.

❤️ 2
seancorfield22:09:15

It's very non-intuitive until you've implemented the protocols on several things and used datafy/`nav` with, say, Cognitect REBL.

seancorfield22:09:42

Yeah, that gist is helpful -- note that both datafy and nav set up the next nav and datafy implementations respectively via metadata and the context of the "database" is carried through them as a closed-over binding.

seancorfield22:09:59

Here's what next.jdbc adds (optionally): https://github.com/seancorfield/next-jdbc/blob/develop/src/next/jdbc/datafy.clj -- beyond the regular datafy/`nav` on result sets that it provides out of the box (that does FK navigation through the DB).

seancorfield22:09:01

(excuse my terrible ascii art skillz @raymcdermott!)

😂 2
ghadi22:09:03

What a k + v navs to is your responsibility

seancorfield22:09:39

☝️:skin-tone-2: Right, nav can use coll and/or k and/or v or ignore any or all of them. [updated to add coll!]

seancorfield22:09:57

This is why it's a bit misleading to try to use datafy and nav with pure data and try to deduce their intent from that because (datafy d) is just d if it is pure data, and (nav d k v) is just v in that case.

Fredrik22:09:39

nav can even ignore coll , like in the gist I linked to above. The reason, as @seancorfield points out, is that the db serving as the collection is closed over in the datafy method

seancorfield22:09:12

In next.jdbc there's a datafy namespace that adds datafy/`nav` to a whole bunch of JDBC classes so you can navigate around the metadata of your database (or the metadata of your result set) and you can navigate across connections and statements and so on.