so I finally have time now to do catch up with devops and cloud stuff (even though it’s 2018 now and serverless is the cool kid in town)… what options are there in clojure? There is pallet http://palletops.com/ but seems the project’s dead?

@lemontea Does there need to be anything in Clojure? There are plenty of solid, well-maintained DevOps projects out there that aren't Clojure.

(we tried to use Pallet -- we even worked with the creators of Pallet -- and it just wasn't up to snuff for what we needed: if you aren't doing something that is really mainstream and standard, Pallet doesn't really have an answer... That said, it's a very cool project and was a great concept... it just wasn't really flexible enough or simple enough)

it’s just for personal learning

being able to do things end-to-end with clojure will be more satisfying, but ya I expect the big player out there to be better supported

still, since this is just learning I don’t need anything fancy at all - just “work” is plenty good enough

I don't know of anything in Clojure in that space...

...at least, nothing that is still actively maintained.

oh no. 😐 https://twitter.com/dorrismccomics (pardon the off topic link… “oh no” is a great answer to many situations)

for career development going the mainstream route (for things that ain’t your “core competency” or differentiator) is pretty much a no-brainer, but I think I can give it a chance/try…

https://adambard.com/blog/deploying-a-clojure-project-with-pallet/

and https://cemerick.com/2010/11/02/continuous-deployment-of-clojure-web-applications/

those are the authoritative reference I find aside from official doc~

Yeah, and that latter is pretty much ancient history in Clojure terms.

And Adam Bard's blog seems like a huge amount of work to deploy those things (and nearly all from forks or ad hoc variants of old, unmaintained projects).

thanks for the advise! gotta work now

@lemontea if you're into devops, I think Nixos is worth the effort. It's a huge thing to learn, but it's just elegant

I do a lot of devops-type work and would never do it in clojure. Interfacing with the messy details of the operating system is not a strength of the jvm, and the abstractions one wants to use in clojure do not fit. The "reconciliation" process kubernetes uses is much more appropriate, and the ergonomics of go for OS work are highly productive. Different spaces, different tools.

Go is pretty nice, the directory structure thing really bugs me though....but it is what it is....i wish i could like Rust.

I like Go's simplicity, they did a good job with that.

C++ will be in my life forever though I'm sure because it's the only thing one can reasonably use on pretty much any platform (well that or C, sideshow_bob_shuddering.gif).

I wish there was a more FP-style systems language. I do a lot of video systems development and there's a lot of state, both intentionally and not, and I wonder if an FP style approach may reduce bugs.

although Wowza and Red5 are both Java media servers so maybe Clojure would work for that sort of thing.

might be an interesting experiment since with graal you could interact with ffmpeg libs to do transcoding and so on.....hm

I hope someone can help me explain why I get OutOfMemoryError. I am trying to copy data from a huge table in database A to database B using clojure.jdbc. It blows my heap, and I don't understand why. https://gist.github.com/RokLenarcic/cc1562e376c0d29371a224724def0153

As far as I can see I am only copying 10k rows at a time

After copying about 3M rows I get OutOfMemoryError

doesn't that code try to create a resultset which is huge from the query made at line 6?

the resultset will contain only nils since your :result-set-fn doesn't return anything sensible

but it's still there

is there a reason why you can't just query 10k rows at a time and use LIMIT and OFFSET to go through the database?

guess it could get a bit slow dunno

I once implemented a lazy sequence implementation via db cursors

anyway, not sure if this really helped. I know you are using cursors there but I think you are creating a huge empty resultset

which blows up your memory usage

Is there a way to preserve type hints in macroexpansion? I want to do something like this:

(defmacro unwrap-solution [s]
  `(.solution ^Solution ~s))

Well the result-set is empty because it's not the point

the result set handling function is performing inserts

which are the point

@puzzler it requires basically applying the type hint using with-meta

If that’s not enough, I can dig up an example

@roklenarcic yes I know that it's empty. But it still consumes memory. Consider (vec (take 3e6 (repeat nil))). That will consume quite a lot of memory without doing anything meaningful

Like I said, I'm not 100% certain if that is the cause of your problems but just giving some ideas

since you are querying data from the database clojure.jdbc tries to return a resultset. Your query will end up returning all the data your query would return, however you are dropping most of that out in your :result-set-fn. This leaves however a nil per row in your database which is stored in memory of you Clojure app. At least, this is what I think is happening. It's a different version of lazy sequences and keeping on the sequence head

Possibly, but I don't see a way to do it better.

Also 3M element vector of nil shouldn't really take gigabytes of RAM

are you sure you are not just running out of heap?

yes gigabytes of heap gone

Also I think you are not correct about how the result-set-fn works

so you've already given tons of mem for jvm heap?

it is applied to resultset as a whole

:row-fn is applied to each row, but result-set-fn is applied to whole result set

let's say, execute your select query but set LIMIT 1000

and check what comes out of your call starting at line 5

my guess would be nil

that is my main idea what could be going wrong. I'm assuming you've already set -Xmx4g or similar to your JVM

but I'll check it

something somewhere is piling up data into your memory

if this doesn't help then I would suggest grabbing a profiler and retrieve some real data to see what kind of objects are being gathered

result of that jdbc/query is nil

ok

It's been a too long time since I've done anything like that with clojure.jdbc

years

I think I understand what's wrong

anyway, profiler will give you unbiased real world data which should give you some hints

my destructuring expression ruins everything

haha true

(fn [[keys & values :as rs]]

you are processing the whole resultset

keeps whole lazy sequence in values

I need to delete that

@alexmiller thanks

@roklenarcic did it solve your problem?

So the function parameter is a lazy sequence and the parameter declaration itself is holding on to the head of it. How do I get around that?

in other words, when you have (fn [result-set] .... ) and result-set is large lazy sequence, is there even a way to process that without blowing heap?

or if it's easier, imagine it's infinite sequence

(doseq [row result-set]
    (comment "do stuff with row"))

but won't result-set local hold to head of lazy sequence and ruin everything?

thinking

I would guess it shouldn't cause that

I'll try again

I'll let you know

but if it does then maybe you can do a hackish solution to bypass that

No I think I had (first rs) somewhere down the line

yeah I think so too

I moved that up into a new local

since if parameter declaration would cause lazy seq to hold the head then lazy sequences would be pretty much useless

well, usually they get passed directly from fn to fn

and things like doseq are a macro

yes but within that fn there are parameters too

for example https://github.com/clojure/clojure/blob/clojure-1.9.0/src/clj/clojure/core.clj#L2737

if that would hold the head then you would not be able to do:

(doseq [x (map inc (range))] (println x))

What about something like this?

I think partition-all into doseq makes sure that batches get dropped

line 2 holds the head

not really, since first returns the element

which has no link to sequence itself

ok

I'm running the new version.... gimme 5 min

looks like another failure

Slowed down to a crawl, GC working overtime

I changed the Gist to latest version

just an idea, what if you take the write connection opening within your result set fn

it shouldn't really affect performance that much since you have (I hope?) connection pools in place anyway

Not exactly certain if this would affect anything but oh well

I suggest you grab that profiler at this point

looks like I'm going to have to do that

but first I'll see if I can write a fn that consumes a massive sequence of integers

jesus christ... the problem was that running REPL as Debug in IntelliJ prevents locals from being cleared (despite the button that prevents locals clearing not being selected at all). Seems to be a Cursive bug or something.

wow

but that DOES make sense really

since for Cursive and IntelliJ to be able to provide local value debugging it needs to hold on to those...

but glad you got that sorted out

@dominicm Re: string cycling: Thank you for answering. @andy.fingerhut your last solution is exactly what I meant, you are a star )

although the first one works as well, just more cryptic )

hi all, quick question. I'm trying to write the simplest of macros: like (comment) with condition: (include cond? body) I tried:

(defmacro include [cond? body]
  (when cond?
    body))

(include true (+ 1 2))
; => 3 
; instead of '(+ 1 2)

you need to quote and unquote the relevant parts

That is correct: the when is executed at macro-expand time, and is not included in the result. Only body will be returned as the result of the macro, and only when cond? evaluates to true

Yep, but that's what I want, I think

Sorry, the body will be evaluated also

What you're missing are the backtick and tilde operators

and even once you’ve added them, remember that calling a macro will also cause the returned code to be evaluated

so even when it’s correct you’ll still see 3

Yeah, but I couldn't figure it out. I tried unquoting, but I don't know what to quote, since I only want what's in body, nothing else

you’ll want macroexpand or macroexpand-1

have you read any resources on how to write a macro?

yes

ok, first question, do you want the condition to be checked at compile (macro-expand) time or at runtime?

compile time

(defmacro include [cond? body]
  (when cond?
    `~body))
=> #'user/include
(macroexpand '(include true (+ 1 2)))
=> (+ 1 2)

@manutter51 is that any different from

(defmacro include [cond? body]
  (when cond?
    body))

Yes, they are different

without the backtick and tilde, you’re defining code that executes at macroexpand-time, so everything will be evaluated at macroexpand-time, and so expanding the macro will give you 3

With the backtick and tilde, you’re writing a macro that returns an expression that will be evaluated after macroexpand time, so when you expand the macro you get back (+ 1 2) [EDIT: nope, read the OP wrong, they’re not different in fact.]

@manutter51 I don't think that's the case

I believe he wants something like

(when cond `'~body)

there's no difference between foo and

`~foo

syntax-quote is not what makes "a macro"

when you unquote from a syntax-quote in a macro, the unquoted code is executed at macroexpansion time

I think if you want macroexpand to return '(+ 1 2) and not just (+ 1 2), then yes, it should be

`'~body 

yes, that's different

altho I'm not sure that macro makes much sense

it's testing the compile-time value of cond? for truthiness

(let [nope false] (include nope ..)) won't do what you likely want

since cond? is bound to the 'nope symbol there

@bronsa You’re right, I thought the OP was saying they got back 3 when they expanded the macro, but I misread

@bronsa I realise that. The idea is to include that code if an external condition is true, which I still haven't figured what it'll be, probably something off lein project map

how are you going to get the condition passed to include tho?

@benzap he wants it quoted, yours is evaluated

cond? needs to be a literal false or nil if you don't want it to test truthy

Yeah, the problem with macros is that you don’t get any runtime values, you only get compile-time symbols

I thought he wanted something like #ifndef in C macroprocessing

but with a condition, ifncond

OK, thanks guys. First I wasn't quoting the code to macroexpand to begin with. So that's why I always got the eval result. As for @manutter51 example, I get this:

(defmacro include1 [cond? body]
  (when cond?
    `~body))

(defmacro include2 [cond? body]
  (when cond?
    body))    

(macroexpand '(include1 true (+ 1 2)))
=> (+ 1 2)

(macroexpand '(include2 true (+ 1 2)))
=> (+ 1 2)

there isn't any

Yes, that was my mistake, I misread your original post

`~foo

a nice trick to see how a syntax-quote expression expands to is to quote it

user=> '`~foo
foo

user=> '`[~foo]
(clojure.core/apply clojure.core/vector (clojure.core/seq (clojure.core/concat (clojure.core/list foo))))

Heh, gonna file that last example under “Be careful what you wish for.”

don't try

'

``foo

😆

(that actually generates so much code that the bytecode excedes the classfile limits of the jvm)

sigh, going to need to end that process now lol

So, maybe a wider question. I need to do "editions" or "cuts" of my program. i.e. different editions have different features. I'm not quite sure how to do it. We distribute a uberjar, so I was thinking using a macro like the above to include code based on a lein project map value, or something like that.

yeah, you could have different profiles, and inject a var to determine which one to generate

Why does this give a reflection warning:

(let [id-1 3
      id-2 1
      id-3 2
      ids [id-1 id-2 id-3]]
  #(compare (.indexOf ids %1) (.indexOf ids %2)))
;; => call to method indexOf on clojure.lang.IPersistentVector can't be resolved (no such method).

(let [ids [3 2 1]]
  #(compare (.indexOf ids %1) (.indexOf ids %2)))

@xiongtx No exception for me:

(let [id-1 3
      id-2 1
      id-3 2
      ids [id-1 id-2 id-3]]
  #(compare (.indexOf ids %1) (.indexOf ids %2)))
=> #object[user$eval1506$fn__1507 0x5c10d9e9 "[email protected]"]

Perhaps you are eliding the call to the returned function?

it's not an exception, it's a reflection warning

you need to turn those on to see them

Clojure 1.9.0
(ins)user=> (set! *warn-on-reflection* true)
true
(let [id-1 3
      id-2 1
      id-3 2
      ids [id-1 id-2 id-3]]
  #(compare (.indexOf ids %1) (.indexOf ids %2)))
Reflection warning, NO_SOURCE_PATH:6:13 - call to method indexOf on clojure.lang.IPersistentVector can't be resolved (no such method).
Reflection warning, NO_SOURCE_PATH:6:31 - call to method indexOf on clojure.lang.IPersistentVector can't be resolved (no such method).
#object[user$eval149$fn__150 0x2072acb2 "[email protected]"]

That's what I get for skimming the message 🙃

@xiongtx my hunch is that the method is specifically parameterized on the second arg's type, and in one case the clojure compiler knows that the type is Long, in the other it doesn't carry that info when compiling

the clojure compiler doesn't try to be especially clever

Which method? indexOf?

right

But how does the Long type affect whether the vector is recognized as a subtype of List or only an IPersistentVector?

hmm no, that's on the List API and it takes Object

that wouldn't be the thing, I was wrong anyway - I thought maybe the arg type was parameterized but it's not

sometimes methods are not found because the type of some arg isn't matched

anyway, looks like a minor bug in the clojure compiler to me

lol

that's a fun one

I think I know what's going on

ah yeah @alexmiller beat me to it :)

a minimal repro is

user=> #(.indexOf [:a] %1)
#object[user$eval187$fn__188 0x3b0c9195 "[email protected]"]
user=> #(.indexOf [%1] %1)
Reflection warning, NO_SOURCE_PATH:18:2 - call to method indexOf on clojure.lang.IPersistentVector can't be resolved (no such method).
#object[user$eval192$fn__193 0x8c11eee "[email protected]"]

hi I'm rather new to clojure and I'm trying to ues plumatic/schema. In test I would love to define a generator based on schema, but I'm getting error "unable to resolve symbol: Person" Could you please give me a hint what's going wrong here?

Tried an import? It's late, but iirc records create classes which need to be imported in addition to requiring the namespace