Fork me on GitHub
#clojure
<
2018-08-07
>
lemontea05:08:34

so I finally have time now to do catch up with devops and cloud stuff (even though it’s 2018 now and serverless is the cool kid in town)… what options are there in clojure? There is pallet http://palletops.com/ but seems the project’s dead?

seancorfield05:08:56

@lemontea Does there need to be anything in Clojure? There are plenty of solid, well-maintained DevOps projects out there that aren't Clojure.

seancorfield05:08:38

(we tried to use Pallet -- we even worked with the creators of Pallet -- and it just wasn't up to snuff for what we needed: if you aren't doing something that is really mainstream and standard, Pallet doesn't really have an answer... That said, it's a very cool project and was a great concept... it just wasn't really flexible enough or simple enough)

lemontea06:08:31

it’s just for personal learning

lemontea06:08:13

being able to do things end-to-end with clojure will be more satisfying, but ya I expect the big player out there to be better supported

lemontea06:08:59

still, since this is just learning I don’t need anything fancy at all - just “work” is plenty good enough

seancorfield06:08:59

I don't know of anything in Clojure in that space...

seancorfield06:08:10

...at least, nothing that is still actively maintained.

lemontea06:08:50

oh no. 😐 https://twitter.com/dorrismccomics (pardon the off topic link… “oh no” is a great answer to many situations)

4
lemontea06:08:25

for career development going the mainstream route (for things that ain’t your “core competency” or differentiator) is pretty much a no-brainer, but I think I can give it a chance/try…

lemontea06:08:08

those are the authoritative reference I find aside from official doc~

seancorfield06:08:11

Yeah, and that latter is pretty much ancient history in Clojure terms.

seancorfield06:08:03

And Adam Bard's blog seems like a huge amount of work to deploy those things (and nearly all from forks or ad hoc variants of old, unmaintained projects).

lemontea06:08:05

thanks for the advise! gotta work now

myguidingstar10:08:39

@lemontea if you're into devops, I think Nixos is worth the effort. It's a huge thing to learn, but it's just elegant

jonahbenton11:08:50

I do a lot of devops-type work and would never do it in clojure. Interfacing with the messy details of the operating system is not a strength of the jvm, and the abstractions one wants to use in clojure do not fit. The "reconciliation" process kubernetes uses is much more appropriate, and the ergonomics of go for OS work are highly productive. Different spaces, different tools.

lemontea13:08:10

I see. I wonder why jvm is weak at that though. Because it’s designed to be platform independent and so can’t handle each OS’s idiosyncrasies easily?

jonahbenton13:08:47

Yes, that design decision means that there will be a certain amount of implementation detail exposure under the abstraction- "works this way on this platform" and "works that way on that platform."" But even dealing with the platform abstractions themselves- the file system, the network- is just ugly and inefficient. The recent kerfuffle around the removal of sun.misc.Unsafe is a good example- basically every networking library was using it to work around the abstraction (and not just one abstraction, there are 3 or 4 different network IO abstractions in the standard library now). Take that and move up a layer with Clojure, where you have immutable data structures, and infinite seqs- these things just do not align with the very concrete systems world.

jonahbenton13:08:34

In comparison, Go's ergonomics match closely to the linux kernel's, so much so that Google wrote a system call layer in userspace as a security measure in Go. Kubernetes very effectively bridges the gap between declarative and operational. You specify a declarative model of some system thing- how many pods you want running, what memory or io policy you want- and then the Controllers iteratively compare the state of the system against the state of the model and go through state transition mechanisms to get them aligned over time. The gravity and physics of the system are first class considerations.

jonahbenton13:08:16

One more point before calling the horse truly dead- it is not as though the linux kernel is a model of beautiful design, the interfaces to which should be preserved all the way up the stack. Quite the contrary. There are a half dozen different ways to do everything, each with their own unique special cases and historical oddities. But I'd put it this way- in the same way that Clojure is "hosted" on the JVM and the JSVM, Go is "hosted" on the interface exposed by the kernel to userspace.

lemontea14:08:17

the nio story in Java is a bit sad compared to, say, C#/F#…

lemontea14:08:58

you’ve got a well beaten horse there :face_with_rolling_eyes:. I do intend to learn Kubernetes though - logically it precedes devops scripting/programming tool like puppet/chef/ansible?

jonahbenton14:08:34

Interesting, I have not used them, but I have worked with the windows apis, which have more design thinking behind them, and can imagine a managed runtime on top being sane. Wrt k8s, it is orthogonal and to some extent competitive with ansible, puppet and co. K8s, combined with a container runtime, is much more like a multi-machine platform/programming abstraction.

jonahbenton14:08:17

People use ansible to deploy kubernetes, and maybe do some system level management with puppet. But a lot of tasks that were also in those tools bailiwick in terms of allocating resources to workloads are better done directly in k8s

jonahbenton14:08:34

*allocating resources and especially managing workload/app lifecycle

myguidingstar14:08:06

are you guys aware of NixOS?

jonahbenton15:08:28

I am, followed it early on when they were just starting, maybe a decade ago, and poke at it from time to time, but haven't used it in anger. Glad to see they are still at it.

myguidingstar15:08:01

I think they're quite mature now 🙂

jonahbenton15:08:45

Is there industry traction? Revenues + support model?

jonahbenton15:08:11

Technically I don't doubt they're solid

jgh11:08:18

Go is pretty nice, the directory structure thing really bugs me though....but it is what it is....i wish i could like Rust.

jgh11:08:49

I like Go's simplicity, they did a good job with that.

jonahbenton11:08:41

Yes. One way to do things. One of the ergonomics-at-scale that I really like is that I can read anyone's go and understand what it is doing. With clojure, everyone writes it differently, builds their own semantics and abstractions. As much as that makes it fun to write, the ecosystem suffers, I believe, because reading is impeded.

jgh11:08:20

C++ will be in my life forever though I'm sure because it's the only thing one can reasonably use on pretty much any platform (well that or C, sideshow_bob_shuddering.gif).

jgh11:08:32

I wish there was a more FP-style systems language. I do a lot of video systems development and there's a lot of state, both intentionally and not, and I wonder if an FP style approach may reduce bugs.

jgh11:08:40

although Wowza and Red5 are both Java media servers so maybe Clojure would work for that sort of thing.

jgh11:08:17

might be an interesting experiment since with graal you could interact with ffmpeg libs to do transcoding and so on.....hm

roklenarcic11:08:47

I hope someone can help me explain why I get OutOfMemoryError. I am trying to copy data from a huge table in database A to database B using clojure.jdbc. It blows my heap, and I don't understand why. https://gist.github.com/RokLenarcic/cc1562e376c0d29371a224724def0153

roklenarcic11:08:19

As far as I can see I am only copying 10k rows at a time

roklenarcic11:08:03

After copying about 3M rows I get OutOfMemoryError

Empperi11:08:38

doesn't that code try to create a resultset which is huge from the query made at line 6?

Empperi11:08:56

the resultset will contain only nils since your :result-set-fn doesn't return anything sensible

Empperi11:08:59

but it's still there

Empperi11:08:50

is there a reason why you can't just query 10k rows at a time and use LIMIT and OFFSET to go through the database?

Empperi11:08:11

guess it could get a bit slow dunno

Empperi11:08:30

I once implemented a lazy sequence implementation via db cursors

Empperi11:08:53

anyway, not sure if this really helped. I know you are using cursors there but I think you are creating a huge empty resultset

Empperi11:08:57

which blows up your memory usage

puzzler11:08:11

Is there a way to preserve type hints in macroexpansion? I want to do something like this:

(defmacro unwrap-solution [s]
  `(.solution ^Solution ~s))
but am getting reflection warnings. Thanks.

roklenarcic11:08:24

Well the result-set is empty because it's not the point

roklenarcic11:08:41

the result set handling function is performing inserts

roklenarcic11:08:44

which are the point

Alex Miller (Clojure team)11:08:15

@puzzler it requires basically applying the type hint using with-meta

Alex Miller (Clojure team)11:08:14

If that’s not enough, I can dig up an example

Empperi11:08:46

@roklenarcic yes I know that it's empty. But it still consumes memory. Consider (vec (take 3e6 (repeat nil))). That will consume quite a lot of memory without doing anything meaningful

Empperi11:08:19

Like I said, I'm not 100% certain if that is the cause of your problems but just giving some ideas

Empperi12:08:38

since you are querying data from the database clojure.jdbc tries to return a resultset. Your query will end up returning all the data your query would return, however you are dropping most of that out in your :result-set-fn. This leaves however a nil per row in your database which is stored in memory of you Clojure app. At least, this is what I think is happening. It's a different version of lazy sequences and keeping on the sequence head

roklenarcic12:08:12

Possibly, but I don't see a way to do it better.

roklenarcic12:08:32

Also 3M element vector of nil shouldn't really take gigabytes of RAM

Empperi12:08:14

are you sure you are not just running out of heap?

roklenarcic12:08:32

yes gigabytes of heap gone

roklenarcic12:08:50

Also I think you are not correct about how the result-set-fn works

Empperi12:08:54

so you've already given tons of mem for jvm heap?

roklenarcic12:08:14

it is applied to resultset as a whole

roklenarcic12:08:51

:row-fn is applied to each row, but result-set-fn is applied to whole result set

Empperi12:08:41

let's say, execute your select query but set LIMIT 1000

Empperi12:08:53

and check what comes out of your call starting at line 5

roklenarcic12:08:43

my guess would be nil

Empperi12:08:46

that is my main idea what could be going wrong. I'm assuming you've already set -Xmx4g or similar to your JVM

roklenarcic12:08:11

but I'll check it

Empperi12:08:36

something somewhere is piling up data into your memory

Empperi12:08:00

if this doesn't help then I would suggest grabbing a profiler and retrieve some real data to see what kind of objects are being gathered

roklenarcic12:08:10

result of that jdbc/query is nil

Empperi12:08:48

It's been a too long time since I've done anything like that with clojure.jdbc

roklenarcic12:08:12

I think I understand what's wrong

Empperi12:08:12

anyway, profiler will give you unbiased real world data which should give you some hints

roklenarcic12:08:25

my destructuring expression ruins everything

roklenarcic12:08:37

(fn [[keys & values :as rs]]

Empperi12:08:43

you are processing the whole resultset

roklenarcic12:08:51

keeps whole lazy sequence in values

roklenarcic12:08:02

I need to delete that

Empperi12:08:39

@roklenarcic did it solve your problem?

roklenarcic12:08:26

So the function parameter is a lazy sequence and the parameter declaration itself is holding on to the head of it. How do I get around that?

roklenarcic12:08:22

in other words, when you have (fn [result-set] .... ) and result-set is large lazy sequence, is there even a way to process that without blowing heap?

roklenarcic12:08:55

or if it's easier, imagine it's infinite sequence

Empperi12:08:56

(doseq [row result-set]
    (comment "do stuff with row"))

roklenarcic12:08:33

but won't result-set local hold to head of lazy sequence and ruin everything?

Empperi12:08:56

I would guess it shouldn't cause that

roklenarcic12:08:03

I'll try again

roklenarcic12:08:07

I'll let you know

Empperi12:08:23

but if it does then maybe you can do a hackish solution to bypass that

roklenarcic12:08:40

No I think I had (first rs) somewhere down the line

Empperi12:08:51

yeah I think so too

roklenarcic12:08:59

I moved that up into a new local

Empperi12:08:13

since if parameter declaration would cause lazy seq to hold the head then lazy sequences would be pretty much useless

roklenarcic12:08:41

well, usually they get passed directly from fn to fn

roklenarcic12:08:53

and things like doseq are a macro

Empperi12:08:54

yes but within that fn there are parameters too

Empperi12:08:13

if that would hold the head then you would not be able to do:

(doseq [x (map inc (range))] (println x))

manutter5112:08:23

What about something like this?

roklenarcic12:08:13

I think partition-all into doseq makes sure that batches get dropped

Empperi12:08:16

line 2 holds the head

roklenarcic12:08:40

not really, since first returns the element

roklenarcic12:08:48

which has no link to sequence itself

roklenarcic12:08:13

I'm running the new version.... gimme 5 min

roklenarcic12:08:50

looks like another failure

roklenarcic12:08:00

Slowed down to a crawl, GC working overtime

roklenarcic12:08:15

I changed the Gist to latest version

Empperi12:08:31

just an idea, what if you take the write connection opening within your result set fn

Empperi12:08:56

it shouldn't really affect performance that much since you have (I hope?) connection pools in place anyway

Empperi12:08:24

Not exactly certain if this would affect anything but oh well

Empperi12:08:30

I suggest you grab that profiler at this point

roklenarcic12:08:15

looks like I'm going to have to do that

roklenarcic12:08:34

but first I'll see if I can write a fn that consumes a massive sequence of integers

roklenarcic13:08:12

jesus christ... the problem was that running REPL as Debug in IntelliJ prevents locals from being cleared (despite the button that prevents locals clearing not being selected at all). Seems to be a Cursive bug or something.

Empperi13:08:34

but that DOES make sense really

Empperi13:08:11

since for Cursive and IntelliJ to be able to provide local value debugging it needs to hold on to those...

Empperi13:08:40

but glad you got that sorted out

dabrazhe14:08:13

@dominicm Re: string cycling: Thank you for answering. @andy.fingerhut your last solution is exactly what I meant, you are a star )

dabrazhe14:08:24

although the first one works as well, just more cryptic )

hjrnunes15:08:08

hi all, quick question. I'm trying to write the simplest of macros: like (comment) with condition: (include cond? body) I tried:

(defmacro include [cond? body]
  (when cond?
    body))
But macroexpand only prints the result of evaluating the body, not the body itself. e.g.
(include true (+ 1 2))
; => 3 
; instead of '(+ 1 2)

scriptor15:08:43

you need to quote and unquote the relevant parts

manutter5115:08:56

That is correct: the when is executed at macro-expand time, and is not included in the result. Only body will be returned as the result of the macro, and only when cond? evaluates to true

hjrnunes15:08:16

Yep, but that's what I want, I think

manutter5115:08:33

Sorry, the body will be evaluated also

manutter5115:08:49

What you're missing are the backtick and tilde operators

scriptor15:08:09

and even once you’ve added them, remember that calling a macro will also cause the returned code to be evaluated

scriptor15:08:24

so even when it’s correct you’ll still see 3

hjrnunes15:08:30

Yeah, but I couldn't figure it out. I tried unquoting, but I don't know what to quote, since I only want what's in body, nothing else

scriptor15:08:38

you’ll want macroexpand or macroexpand-1

scriptor15:08:48

have you read any resources on how to write a macro?

scriptor15:08:49

ok, first question, do you want the condition to be checked at compile (macro-expand) time or at runtime?

hjrnunes15:08:59

compile time

manutter5115:08:58

(defmacro include [cond? body]
  (when cond?
    `~body))
=> #'user/include
(macroexpand '(include true (+ 1 2)))
=> (+ 1 2)

👍 4
hjrnunes15:08:31

@manutter51 is that any different from

(defmacro include [cond? body]
  (when cond?
    body))
no quote/unquote?

manutter5115:08:51

Yes, they are different

manutter5115:08:12

without the backtick and tilde, you’re defining code that executes at macroexpand-time, so everything will be evaluated at macroexpand-time, and so expanding the macro will give you 3

manutter5115:08:07

With the backtick and tilde, you’re writing a macro that returns an expression that will be evaluated after macroexpand time, so when you expand the macro you get back (+ 1 2) [EDIT: nope, read the OP wrong, they’re not different in fact.]

moxaj15:08:58

@manutter51 I don't think that's the case

moxaj15:08:20

I believe he wants something like

(when cond `'~body)

bronsa15:08:32

there's no difference between foo and

`~foo

bronsa15:08:12

syntax-quote is not what makes "a macro"

bronsa15:08:56

when you unquote from a syntax-quote in a macro, the unquoted code is executed at macroexpansion time

manutter5115:08:48

I think if you want macroexpand to return '(+ 1 2) and not just (+ 1 2), then yes, it should be

`'~body 

bronsa15:08:24

yes, that's different

bronsa15:08:58

altho I'm not sure that macro makes much sense

bronsa15:08:08

it's testing the compile-time value of cond? for truthiness

bronsa15:08:28

(let [nope false] (include nope ..)) won't do what you likely want

bronsa15:08:48

since cond? is bound to the 'nope symbol there

manutter5115:08:13

@bronsa You’re right, I thought the OP was saying they got back 3 when they expanded the macro, but I misread

hjrnunes15:08:27

@bronsa I realise that. The idea is to include that code if an external condition is true, which I still haven't figured what it'll be, probably something off lein project map

bronsa15:08:53

how are you going to get the condition passed to include tho?

moxaj15:08:15

@benzap he wants it quoted, yours is evaluated

bronsa15:08:29

cond? needs to be a literal false or nil if you don't want it to test truthy

manutter5115:08:41

Yeah, the problem with macros is that you don’t get any runtime values, you only get compile-time symbols

benzap15:08:53

I thought he wanted something like #ifndef in C macroprocessing

benzap15:08:03

but with a condition, ifncond

hjrnunes15:08:13

OK, thanks guys. First I wasn't quoting the code to macroexpand to begin with. So that's why I always got the eval result. As for @manutter51 example, I get this:

(defmacro include1 [cond? body]
  (when cond?
    `~body))

(defmacro include2 [cond? body]
  (when cond?
    body))    

(macroexpand '(include1 true (+ 1 2)))
=> (+ 1 2)

(macroexpand '(include2 true (+ 1 2)))
=> (+ 1 2)
So I can't see the difference between quote-unquote or do nothing.

bronsa15:08:25

there isn't any

manutter5115:08:37

Yes, that was my mistake, I misread your original post

bronsa15:08:51

`~foo
literally expands to foo

bronsa15:08:20

a nice trick to see how a syntax-quote expression expands to is to quote it

bronsa15:08:29

user=> '`~foo
foo

bronsa15:08:50

user=> '`[~foo]
(clojure.core/apply clojure.core/vector (clojure.core/seq (clojure.core/concat (clojure.core/list foo))))

💡 8
manutter5115:08:28

Heh, gonna file that last example under “Be careful what you wish for.”

bronsa15:08:00

don't try

'
``foo
then ;)

bronsa15:08:40

(that actually generates so much code that the bytecode excedes the classfile limits of the jvm)

benzap15:08:40

sigh, going to need to end that process now lol

hjrnunes15:08:49

So, maybe a wider question. I need to do "editions" or "cuts" of my program. i.e. different editions have different features. I'm not quite sure how to do it. We distribute a uberjar, so I was thinking using a macro like the above to include code based on a lein project map value, or something like that.

benzap15:08:48

yeah, you could have different profiles, and inject a var to determine which one to generate

xiongtx16:08:36

Why does this give a reflection warning:

(let [id-1 3
      id-2 1
      id-3 2
      ids [id-1 id-2 id-3]]
  #(compare (.indexOf ids %1) (.indexOf ids %2)))
;; => call to method indexOf on clojure.lang.IPersistentVector can't be resolved (no such method).
But this doesn't?
(let [ids [3 2 1]]
  #(compare (.indexOf ids %1) (.indexOf ids %2)))
The only difference is in the values inside the vector, which doesn't seem like it should affect calling indexOf on the vector.

kenny16:08:04

@xiongtx No exception for me:

(let [id-1 3
      id-2 1
      id-3 2
      ids [id-1 id-2 id-3]]
  #(compare (.indexOf ids %1) (.indexOf ids %2)))
=> #object[user$eval1506$fn__1507 0x5c10d9e9 "user$eval1506$fn__1507@5c10d9e9"]

kenny16:08:36

Perhaps you are eliding the call to the returned function?

noisesmith16:08:08

it's not an exception, it's a reflection warning

noisesmith16:08:14

you need to turn those on to see them

noisesmith16:08:05

Clojure 1.9.0
(ins)user=> (set! *warn-on-reflection* true)
true
(let [id-1 3
      id-2 1
      id-3 2
      ids [id-1 id-2 id-3]]
  #(compare (.indexOf ids %1) (.indexOf ids %2)))
Reflection warning, NO_SOURCE_PATH:6:13 - call to method indexOf on clojure.lang.IPersistentVector can't be resolved (no such method).
Reflection warning, NO_SOURCE_PATH:6:31 - call to method indexOf on clojure.lang.IPersistentVector can't be resolved (no such method).
#object[user$eval149$fn__150 0x2072acb2 "user$eval149$fn__150@2072acb2"]

kenny16:08:06

That's what I get for skimming the message 🙃

noisesmith16:08:39

@xiongtx my hunch is that the method is specifically parameterized on the second arg's type, and in one case the clojure compiler knows that the type is Long, in the other it doesn't carry that info when compiling

noisesmith16:08:49

the clojure compiler doesn't try to be especially clever

xiongtx16:08:46

Which method? indexOf?

xiongtx16:08:12

But how does the Long type affect whether the vector is recognized as a subtype of List or only an IPersistentVector?

noisesmith16:08:16

hmm no, that's on the List API and it takes Object

noisesmith16:08:42

that wouldn't be the thing, I was wrong anyway - I thought maybe the arg type was parameterized but it's not

noisesmith17:08:14

sometimes methods are not found because the type of some arg isn't matched

noisesmith17:08:39

anyway, looks like a minor bug in the clojure compiler to me

🐛 4
xiongtx17:08:00

I created https://dev.clojure.org/jira/browse/CLJ-2382 Can't seem to edit the issue to fix the code formatting, however.

noisesmith17:08:22

you should probably specify your clojure version also

noisesmith17:08:33

(and for bonus, maybe try it on the latest alpha)

bronsa17:08:58

that's a fun one

bronsa17:08:07

I think I know what's going on

👀 4
bronsa17:08:13

ah yeah @alexmiller beat me to it :)

bronsa17:08:02

a minimal repro is

user=> #(.indexOf [:a] %1)
#object[user$eval187$fn__188 0x3b0c9195 "user$eval187$fn__188@3b0c9195"]
user=> #(.indexOf [%1] %1)
Reflection warning, NO_SOURCE_PATH:18:2 - call to method indexOf on clojure.lang.IPersistentVector can't be resolved (no such method).
#object[user$eval192$fn__193 0x8c11eee "user$eval192$fn__193@8c11eee"]

taojang21:08:06

hi I'm rather new to clojure and I'm trying to ues plumatic/schema. In test I would love to define a generator based on schema, but I'm getting error "unable to resolve symbol: Person" Could you please give me a hint what's going wrong here?

emil0r22:08:41

Tried an import? It's late, but iirc records create classes which need to be imported in addition to requiring the namespace