the last place I worked use chef, but you don't need to install lein

build a jar, deploy the jar

gotcha so no need for jenkins as well to deploy

oof, well, that is a whole other thing, if you are building a ci box, for whatever reason our "devops" team refused to take care of our ci box at the last job so it wasn't managed using chef, just a shell script

so I have very little good advice there

yea i'm in the process of putting our clojure app on a server so looking for best practices since its a new job and the previous place i worked we only used the jar method you said but the devops want to use jenkins and chef

create a lein uberwar and drop it on a java container

my generator isn't working, I'm trying to run this line: https://github.com/cognitect-labs/aws-api/blob/master/examples/s3_examples.clj#L34

I have a map: '({:a 1 :b2}) collection .. I want to take the first map and get all the keys ... but what I'm tryin' ain't workin'

my uuid-string function looks identical to this one ... does this mean I need to add the APACHE license to my project? https://github.com/cognitect-labs/aws-api/blob/master/src/cognitect/aws/util.clj#L250-L253

@johnjelinek I would expect everyone who needs a UUID string has written that exact same function independently (so, no, you only need to keep the license when you copy in code from an OSS project).

what gives? my :gen-class from yesterday doesn't work today when I AOT?

@deleted-user CSP -- Communicating Sequential Processes -- is the background to read for this.

Yeah, Go comes from CSP as well.

Yeah, I learned it via Occam in the early 80's.

A lot of solid production code is built on core.async...

I wouldn't worry too much about futures, given what you can already do 🙂

BEAM is an interesting environment. I took a workshop back at an FP conference years ago and was impressed by what Erlang can do. But I've been on the JVM for... 22 years at this point... so I both love it and hate it and I've learned to live with its limitations in order to take advantage of everything else.

I've never felt the loss of that -- in 22 years on the JVM. Make of that what you will.

Hum.. interesting aside, Java actually used to have only green threads

Like in Java 1 or some old version like that

But it later moved to OS threads, because they are better for true parallelism and thus performance

And I guess convenient

Shows how the world goes in circle

That said, I think you're missing the point of CSP

Or I don't know about CSP, but the point of the "go" macro

Blocking is a feature

Its how you synchronize producers and consumers and coordinate the concurrent processes

From my understanding, you have to go to great lengths of additional complexity in Actor models to perform such coordination

And I think that was one reason Rich Hickey went with CSP over Actors

Also, isn't Erlang a preemptive scheduler? Are you sure its threads are lightweight?

Oh. Sorry, I thought you meant parking, or blocking the go-process

Ya, unfortunately, Clojure can't prevent the actual thread running the process from being accidentally blocked

So you just have to know what you're doing

Is project loom implementing a preemptive scheduler?

So, that's exactly what core.async does

I don't think the issue has anything to do with that, but all to do with blocking IO

Does erlang have blocking io?

If so, it would also block one of the threads in its pool. Meaning there is one less thread to run the processes on.

So unless Project Loom can magically rewrite calls to Blocking IO or other blocking operations, it would suffer the same fate.

Well... except core.async isn't preemptive. So your go block won't park on its own

Erlang has its own IO implementations so that you can do IO without blocking the underlying OS thread.

Ya, so, that's really the difference here (minus preemption)

The issue is just Java legacy blocking operations

And how so many things rely on them

When using core.async go block, you just need to make sure you're only using non blocking IO

That’s the appeal of the Loom - it aims to make the legacy IO stuff just work (tm)

Okay, that's where it gets interesting

But how?

I guess a virtual layer that handles the callback for you

That, I don’t remember. But the videos that they have put out so far are really interesting and full of the nitty gritty.

> The implementation of the networking APIs in the http://java.net and java.nio.channels packages have as been updated so that fibers doing blocking I/O operations park, rather than block in a system call, when a socket is not ready for I/O. When a socket is not ready for I/O it is registered with a background multiplexer thread. The fiber is then unpacked when the socket is ready for I/O. These same blocking I/O are also updated to support cancellation. If a fiber is cancelled while in a blocking I/O operation then it will abort with an IOException.

Hum... still requires the use of nio then

But I guess ya, they can just update all IO code and have a if called by Fiber.. then park instead of blocking

Ya, that be neat

It’s a huge undertaking but they seem to be chipping at it and making good progress.

Regarding blocking IO, isn’t it true that most file access IO is blocking at the OS level and it’s hard to overcome that? That’s what I’ve been hearing for years. So if you make socket calls parking you get huge practical value.

I had gone down a big rabbit whole about this a while ago

It seems only Windows can do non blocking file IO

Yay for Windows? 🙂

Ya, it is really advanced in its async IO

In this case

You can actually hand off a callback to the OS and register a user thread with it. And on interrupt from the peripheral, it'll switch to your registered user thread, hand of execution to your callback and pass you the IO result

So it's truly push all the way down

Where as linux only really supports multiplexed io

Does this low-level stuff really matter tho' in the grand scale of things? (serious question)

But it provides an API called like Asyncio which abstract it away, but it's still multiplexed under the hood

I mean, we build production scale stuff on Windows, Linux... without worrying about this...?

Well, if you wanted to go from say 10k requests/s to 100 000k maybe 😋

I.mean, probably not, probably the gain are minor in performance but wtv

But how many people need to do that?

I think the vast majority "don't care". Regular JVM stuff is "fast enough", right?

Ya, I'd agree

I worry that we go down rabbit holes in the name of (unnecessary) performance.

That's why I'm with you, for all practical purpose, core.async solves all the use cases I have. When I need it, I'm just careful not to block

Google, Facebook, Twitter... they care but most of us don't.

(and I'm not even sure they care really, in all of these cases)

I doubt they even care. All this is about scaling single host concurrency, and for most things, especially big tech, adding a few more hosts is way cheaper then the maintenance cost of more complex software

I actually think startup care more

I see some people's "obsession" with async I/O and I'm like "really?". I get asked about it with clojure.java.jdbc and next.jdbc for example...

Shops trying to run everything on like 1 box

Cause they can't afford more servers

Async JDBC is horribly non-portable. Why would anyone inflict that pain upon themselves?

I have no experience with it. Is that just an issue with async JDBC kinda just sucking? Or the async-ness makes it inherently less portable?

Only some DBs have drivers that support async.

(and each of them are different)

So when folks ask me to support async in clojure.java.jdbc or next.jdbc I'm like, yeah, send me a PR for portable async 🙂

PostgreSQL has a fairly well-established async driver I believe... and some (many?) PG users tend to assume the world is like PG 🙂

Hum... I mean. I can imagine how in 10 years, everything will just have moved to async IO, and it'll just be the new io standard

And everything might be a little faster.

Yup, and in ten years, I'll be happy to release jdbc.async 🙂

Haha 👌

I've actually never used async IO in Java and/or Clojure yet teehi

I wonder if the APIs for it are simple

I'm hoping it's just like takes a callback which is a callable

And you could just therefore give it a Clojure fn which puts to the channel when called back

Actually that's the part I can't quite imagine for Loom

Could it park and like... block the return, hum. I guess it could. So you'd basically be doing a blocking call, but the thread is a lightweight one. Ya that's nice

I guess it could be leveraged by http servers and what not

But, really, these are implementation details in the grand scale of things...

async io just sounds like another horrible trap for hard to reproduce, non deterministic, bugs

Ya, I mean, Java can already scale. So all it could further do is save you a few extra hosts

If you look at this from a higher level, you can't tell how a lot of the low-level stuff actually operates, and that's a good thing.

We should be focusing on higher-level stuff, not lower-level stuff.

That's true.

I think that would be the defining factor for Loom as well. If its invisible, then great. You'll save some memory and shave of a few milliseconds in context switches without having to do anything. But if it's like a whole buy in, redesign everything, etc. Then meh

I wonder how many people use go blocks in Clojure (not ClojureScript). I've used thread, but never had a real need for go.

We have a few things that are very heavy on async and rely on core.async

But out of 80k+ lines it's not much

Cool! I guess I did maybe have a use case for it recently. Had to process a millions+ csv file and make a request to an API for each row.

And then aggregate back their results

And that's probably not a good use case for it

Used an executor for it. But I guess async IO and go could have worked

Ah OK

Well.. actually no lll

Frankly an executor is better

Lol

Because its not like I can call the API with a million concurrent requests

The fleet maybe handles a few hundreds max concurrent, of which other prod systems call into. That's the thing with real life, you always need throttling, back offs, circuit breakers, etc.

Seriously, Executors are great. There's one for every use case too. That with just futures take you really far already

I'm always sad when beginners just jump on core.async. And don't start with future, executor and pmap

What's an example use case you use go blocks for?

@deleted-user By the way, you should also have a look at https://clojureverse.org/t/cloroutine-v1/3300 At this point, you might know more about the whole space of coroutines vs fibers vs green threads vs cps, etc. But apparently core.async aren't full generalized coroutines and cloroutine implements a full genral coroutine in Clojure.

@deleted-user And for completeness since this is the beginner channel. In ClojureScript land, when targeting NodeJS, you don't have to worry about accidentally blocking the go-blocks, since Node only supports async non-blocking IO. Some people do prefer NodeJS over the JVM as their backend runtime and use ClojureScript for that reason instead.

Hi , My map is like {2004 [{:id 1, :name water, :attribute_name Color, :attribute_value blue} ,{:id 2, :name water, :attribute_name formula, :attribute_value H2O} ,{:id 3, :name water, :attribute_name State, :attribute_value liquid}] } i want to create following map like this {2004 [{:Color blue, :formula H2O ,:State liquid}]} Please guys help me

Hello, has anyone used Cucumber with Clojure? If so, which command do I have to use to run a tagged Test Case?

Hey, does anyone have any resources on how to include, import, and wrap homegrown Java classes in Clojure projects? I'm going to start working on wrapping an object model that a colleague wrote in Clojure. The classes are pretty involved, they're used to model CME market data for live feeds, so I'm looking for some resources on how to import the jar from the Java project in my Clojure project and how to properly implement a wrapper so that we can work with it more idiomatically on the Clojure side. Any resources would be a huge help! Thank you! Edit: I'm reading through the following things for a primer as well 1. https://clojure.org/reference/java_interop 2. http://clojure-doc.org/articles/language/interop.html 3. https://www.braveclojure.com/java/ 4. https://www.reddit.com/r/Clojure/comments/aqy19n/questions_about_wrapping_a_java_library/

If you can get the jar in the class path (differs based on your build tool, I guess?) then you’re off to the races, everything should work with the Java interop syntax.

As for more idiomatic, most Clojure apis would like to deal with maps, collections and seqs - so are if you can write some wrappers that do this kind of translation.

is there a macro to aesthetically improve this (foo value (:bar (baz x y))) ?

What's wrong with that?

nothing, wanted to play with -> ->> and cousins 😉

I actually prefer the nested one showed above but almost every clojure I see in the wild is obsessed with -> ->>

the straightforward translation via arrows is (-> x (bar y) (:bar) (->> (foo value)))

Or maybe with as->.

or alternately (->> y (bar x) (:bar) (foo value)) - it really depends on what x and y are and what they mean

Speaking for myself, -> or ->> in the context of some other threading operator gives me the heebie jeebies.

the other threading macros are designed to be nested under ->

Oh yeah? That's interesting.

that's why as-> takes a value first then a binding name for example

and the semantics are simple - if -> is the parent, you just transition from the -> rule to the threading of the child threading macro

line breaks help

x and y are single values. For small compositions stuff I guess its personal preference but seem handy for much bigger comps

@lockdown- everything is a value - what I mean is in (bar x y) are you conceptually doing something to x that passes forward, or are you doing something to y that passes forward, or is bar something that combines x and y as "peers"

yeah, the latter

x and y end with the call to bar

So are threading macros akin to the pipe operator in Elixir and easy to overuse?

there is also comp which I like but don't see it used much

@lockdown- in that case (->> (bar x y) (:bar) (foo value)) is how I'd do it, since neither x or y is conceptually carried forward to the next step

@buddhamagnet they are not an operator, they are a syntax transform

they can be overused, but usually (when combined with good formatting) they make code clearer

I have a string iso8601 date, and wanna add or subtract from the year ending up with an otherwise identical string. What's the best path there? I'm worried parse/format could get caught in funny formatting problems. String manip makes me feel guilty. Is there already a function for that somewhere? I see clj-time.core and clj-time.format in a few places... so maybe I should go learn those?

if it is really iso8601, the format is standard and well supported, it should be pretty straightforward

I don't know if I've ever seen dates straightforward... -_- But yeah, ok, how would you do that?

I would unequivocally recommend clj-time - it’s great.

I would anti recommend clj-time because it's based on a deprecated library (Joda time)

user=> (str (.plusYears (java.time.ZonedDateTime/parse "2019-06-12T20:18:17.744-04:00" java.time.format.DateTimeFormatter/ISO_OFFSET_DATE_TIME) 5))
"2024-06-12T20:18:17.744-04:00"
user=>

^^^yes do that

the new time stuff built into java 7+ (if I recall, maybe 8+?) is very good

8

@seancorfield have you considered re-implementing clj-time in terms of Java’s built-in time classes?

Requiring beginners to learn Java interop simply to handle basic data types (i.e. dates / times) seems a bit naff.

There was a very long discussion thread about that in an issue over several years -- and the consensus was that it would not be possible without breaking backward compatibility.

Makes sense.

clj-time's readme suggests looking at clojure.java-time if you want syntactic sugar.

There's also a new cljc-based wrapper for Java Time that has a compatible JS implementation inside it. I have not looked at that yet.

Nice - was not aware of that until now.

FYI https://github.com/clj-time/clj-time/issues/196

(the discussion took nearly two years!)

I wonder if clojars supports a deprecated flag, at the library level? That would be quite handy for situations like this.

It does not, unfortunately.

awww, dang

(although I guess updating the project's name/description which would end up in pom.xml and therefore display on Clojars might be a good idea)

Yeah - shame there isn’t something that build tools (`lein`, tools.deps, etc.) can consume.

Also, is the cljc library you mentioned https://github.com/henryw374/cljc.java-time ?

Yes, thank you! Launched recently at Clojure/North.

By the way, beginners should learn Java interop, and they should do so relatively early on.

Java interop is not like an esoteric thing that you almost never reach for. It's one of the main component of Clojure

Wrappers over it should only be used when they add additional value. Like make it considerably easier to use, provide additional functionality on top, etc.

Or if you need a common abstraction between dialects like Clojure and ClojureScript

Or learn Java and interop will come naturally 😉

This is my first post here, so hello all. Talking of Java interop. I was looking at the examples from the Java Interop chapter of Programming Clojure and playing around with them. I came across something I don't understand. One example given is the following:

Hi 👋 I'm trying to use the following lib https://github.com/mbuczko/embodie in a sample project. However, I got an error about:

1. Unhandled java.io.FileNotFoundException
   Could not locate embodie/core__init.class, embodie/core.clj or
   embodie/core.cljc on classpath.

(defproject hello_world_embody "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url ""
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url ""}
  :dependencies [[org.clojure/clojure "1.10.0"]
                 [defunkt/embodie "1.0.0"]]
  :repl-options {:init-ns hello-world-embody.core}
  :main ^:skip-aot hello-world-embody.core
  :target-path "target/%s"
  :profiles {:uberjar {:aot :all}})

did you restart your repl after adding the dependency?

@noisesmith Yes. I did

In addition, I got the same error when I ran lein run

trying again, haven't got the hang of entering new lines in Slack... This is my first post here, so hello all. Talking of Java interop. I was looking at the examples from the Java Interop chapter of Programming Clojure and playing around with them. I came across something I don't understand. One example given is along the lines of the following: (defn sax_parse [source handler] (.. SAXParserFactory newInstance newSAXParser (parse (InputSource. (StringReader. source)) handler))) that works if called with an XML string as source and something extending org.xml.sax.helpers.DefaultHandler as handler. If I expand the .. member access threading macro I get: (defn sax_parse [source handler] (. (. (. SAXParserFactory newInstance) newSAXParser) (parse (InputSource. (StringReader. source)) handler))) which also works. But if I try to rewrite the last line using the -> thread-first macro then whether I try: (defn sax_parse [source handler] (.. SAXParserFactory newInstance newSAXParser (-> source StringReader. InputSource. (parse handler)))) or: (defn sax_parse [source handler] (. (. (. SAXParserFactory newInstance) newSAXParser) (-> source StringReader. InputSource. (parse handler)))) then it doesn't work. The defn can't be evaluated, I get Syntax error (ClassNotFoundException) compiling at (test.clj:16:3). StringReader. It seems to be taking StringReader. as a class name. Can anyone tell me why (please)?

try wrapping it in parens

(-> source (StringReader.) etc.

yeah, StringReader. without parens will not read properly

OK, thanks, but if I try: (defn sax_parse [source handler] (. (. (. SAXParserFactory newInstance) newSAXParser) (-> source (StringReader.) InputSource. (parse handler)))) I get a different error: Syntax error (IllegalArgumentException) compiling new at (test.clj:17:17). No matching ctor found for class java.io.StringReader

or you can use (new StringReader) which is what (StringReader.) expands to when read

What's confusing me is that (macroexpand '(-> source StringReader. InputSource. (parse handler)))

(macroexpand '(-> source StringReader. InputSource. (parse handler)))gives

also this would be much easier to read with sugared . forms - so (.newSAXParser (SAXParserFactory/newInstance) ...)`

<sorry keep forgetting about shit enter> What's confusing me is that (macroexpand '(-> source StringReader. InputSource. (parse handler))) gives (parse (InputSource. (StringReader. source)) handler) which is what worked in the first place?

yes, that's actually weird - that should work even without the parens actually

are you sure you are evaluating that code in the right environment?

I think so, everything else works. I'm just using C-c C-e in Emacs CIDER

@alex.sumner you might want to look at Slack's formatting with triple-backticks so you get

code that looks like this

Yes, that would be better! Thanks.

And under Preferences > Advanced, check the option that pressing enter inside triple-backticks does not send the message. That way you can type three backticks, press enter, type lots of code with newlines, then three more backticks (and then enter will send the message).

OK, so like this? `

I guess not!

You need that preferences change too.

` to start

and ` to end

triple-backslash CODE CODE CODE triple-backslash

you can also use control-enter to make a new line without sending a message

OK, now we're getting somewhere
Still don't know about the problem with ->
but getting somewhere with Slack!

I think in

(defn sax_parse [source handler]
 (. (. (. SAXParserFactory newInstance) newSAXParser)
    (-> source (StringReader.) InputSource. (parse handler))))

(.newSAXParser (SAXParserFactory/newInstance))

oh yeah, that is an issue

Oh, .parse is a method on the parser created?

the issue is the latter

(defn sax_parse [source handler]
 (. (. (. SAXParserFactory newInstance) newSAXParser)
    (-> source (StringReader.) InputSource. (parse handler))))

(defn sax-parse [source handler]
  (.parse (.newSAXParser (SAXParserFactory/newInstance))
          (-> source (StringReader.) (InputSource.))
          handler))

^

so nice

(with kebab-case name instead of snake_case BTW for idiomatic Clojure)

What I actually wanted to try was

(defn sax_parse [source handler]
  (.. SAXParserFactory newInstance newSAXParser
      (-> source StringReader. InputSource. (parse handler))))

(let [factory (SAXParserFactory/newInstance)
      parser (.newParser factory)
.....

naming things will bring some clarity, too

Agreed.

Mixing .. and -> can be pretty confusing to read (and, as we see here, often confusing to write).

(defn sax-parse [source handler]
  (let [factory (SAXParserFactory/newInstance)
        parser (.newParser factory)
        input (-> source (StringReader.) (InputSource.))]
    (.parse parser input handler)))

(whatever the method signature is for parse, I can't remember)

.. is good for method chains in Java:

Foo.getOrder().getLineItem(2).getPrice()

(.. Foo getOrder (getLineItem 2) getPrice)

OK, thanks. I am still worried that

(A (B))

(A (C))

(macroexpand '(C))

(B)

.. SAXParserFactory newInstance newSAXParser

. (. (. SAXParserFactory newInstance) newSAXParser)

(hit up arrow to edit the last thing you wrote, btw)

B is

parse (InputSource. (StringReader. source)) handler

-> source StringReader. InputSource. (parse handler)

Right, cool!

(-> source StringReader. InputSource. (parse handler))

(parse PREVIOUS handler)

where you need

(.parse PREVIOUS handler)

note the .

(assuming that parse is a java method)

this is a difference between .. and other forms

Yes parse is a method

right

so without the leading . it thinks it's either a clojure function or a local in a let-binding

https://docs.oracle.com/javase/8/docs/api/javax/xml/parsers/SAXParser.html#parse-java.io.InputStream-org.xml.sax.HandlerBase-

@ghadi But he has an outer (. parser (-> ,,,)) form in the above

SAXParser.parse takes an input and a handler, so if you want to translate this to the "fancy style" https://clojurians.slack.com/archives/C053AK3F9/p1560381542398400

it would become:

(defn sax-parse
  [source handler]
  (let [input (-> source (StringReader.) (InputSource.))]
    (.. (SAXParserFactory/newInstance)
        (newParser)   ;;  no parens necessary here, but I'm including them
        (parse input handler))))

sorry, edited a typo

OK thanks. That looks better. I am still confused as to why

(defn sax_parse [source handler]
 (.. SAXParserFactory newInstance newSAXParser
     (parse (InputSource. (StringReader. source)) handler)))

I think it could have to do with .. being expanded before -> ?

there are two differences between that and what I pasted: 1) the first argument to parse (the input thingy) is inlined

2) (.. SAXParserFactory newInstance) is equivalent to (. SAXParserFactory newInstance) which is equivalent to (SAXParserFactory/newInstance)

in practice I never see the first two styles for calling a static method -- always (Class/staticmethod)

If it helps

user=> (macroexpand-all '(.. SAXParserFactory newInstance newSAXParser (-> source StringReader. InputSource. (parse handler))))
(. (. (. SAXParserFactory newInstance) newSAXParser) (parse (new InputSource (new StringReader source)) handler))
user=> (macroexpand '(.. SAXParserFactory newInstance newSAXParser (-> source StringReader. InputSource. (parse handler))))
(. (. (. SAXParserFactory newInstance) newSAXParser) (-> source StringReader. InputSource. (parse handler)))
user=> 

but the middle style (. Class methodName) is the "final" macroexpansion that the compile operates on

user=> (macroexpand '(java.util.Objects/nonNull a b c))
(. java.util.Objects nonNull a b c)

I think this is an issue too

user=> (macroexpand '(.. foo (-> x (bar y) (someMethod))))
(. foo (-> x (bar y) (someMethod)))

user=> (. "hello" (-> "l" (replace "L")))
Execution error (IllegalArgumentException) at user/eval163 (REPL:1).
No matching method __GT_ found taking 2 args for class java.lang.String
user=> (. "hello" (replace "l" "L"))
"heLLo"
user=> 

and it definitely shouldn't try to invoke the initial symbol in a form

(. "hello" (-> "l" (replace "L"))) doesn't trigger macroexpansion

on the arguments, that is

yeah, there's no sane way it could, for the operator position at least

So it turns it into (.-> "hello" "l' (replace "L")) which is munged to (.__GT_ "hello" "l" (replace "L"))

So that's why you can't mix .. and -> like this.

i don't think it does yes, misread you Sean

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L970-L974

those are the parsed forms for lists beginning with .

Yup. And going back to .. to be more like @alex.sumner’s original code

user=> (.. "HELLO" toLowerCase (-> "l" (replace "L")))
Execution error (IllegalArgumentException) at user/eval170 (REPL:1).
No matching method __GT_ found taking 2 args for class java.lang.String
user=> (.. "HELLO" toLowerCase (replace "l" "L"))
"heLLo"
user=> 

(because you get (.__GT_ (.toLowerCase "HELLO") "l" (replace "L")) because -> is munged to __GT_ in JVM code)

extra dot at the beginning of the last line^ reading skillz no good right now

Thank you for the question/puzzle @alex.sumner -- it's been a great learning experience for us old 'uns too!

indeed

OK, thanks all of you! So, to check my understanding, you can't put macros inside

.

..

you can, but not in the method position

because it intentionally doesn't expand that position - you don't expect it to look up parse and replace it right?

it's used as is

you can safely use -> to construct args to one of the methods

OK, thanks again.

this all makes me wonder if explicit message passing OO isn't a lot more sane, I'm sure it doesn't optimize as nicely as what the JVM does, but it would make higher order code much simpler to construct

Going back to your original code @alex.sumner this should work:

(defn sax-parse [source handler]
  (.. SAXParserFactory newInstance newSAXParser
      (parse (-> source StringReader. InputSource.) handler)))

(but I haven't tested it)