Fork me on GitHub
#clojure
<
2019-01-09
>
kwladyka10:01:55

Did something change about Clojure memory consumption? Hello worlds still consume 256 MB? I am asking in the context of making N instances in environment like kubernetes. Small services can consume very fast too much memory.

Lennart Buit10:01:33

does it consume or request 256m of memory

Lennart Buit10:01:52

right, starting a repl here indeed has a initial heap size of 256 of which only 27 is in use. You could probably use the default memory flags of the JVM to change that

kwladyka10:01:05

I was trying it a few years ago and my conclusion was Clojure wrapper for Java takes around 256 MB. I have no idea how it looks today.

Lennart Buit10:01:43

(I checked with VisualVM, the (initial) heapsize is indeed 256, but its not all used, you could probably further restrict that to 64)

Lennart Buit10:01:08

You seem to be able to pass :jvm-opts in your leiningen project file (similarly for deps.edn), but the exact correct flags for the JVM are beyond me for now. Check the usual JVM resources

kwladyka10:01:30

So how many MB are “lost” only to run Clojure app?

fahrrad10:01:07

JVM will use more memory than it requires by default. If you want to see how low you can go with memory, you have to limit the memory that is available to the jvm. For example by passing this -Xmx64m

fahrrad10:01:55

also, keep in mind that java 8 cannot determine the memory limits of a container reliably (it will in stead take the memory limit of the host).

kwladyka10:01:03

but then wouldn’t clojure app have bad performance or other downside?

fahrrad10:01:12

You can get past that with -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

fahrrad10:01:24

that is what you have to figure out

fahrrad10:01:03

all I wanted to say is that JVM will take more memory that strictly needed by default

kwladyka10:01:04

heh ok thx 😉 Just curious what is people experience about memory consumption for services in clouds

fahrrad10:01:24

do you use java8?

kwladyka10:01:44

now I use 11

kwladyka10:01:03

I feel confuse how it will behave in practice after such cut of memory

rovanion11:01:17

If you wanted to write as many bytes as possible per second to a file on the file system on clojure-jvm, how would you do it?

kwladyka11:01:49

too general question, but mainly I will buy fast hard drive 😉

rovanion12:01:24

Let me specify then: Would you recommend any specific programmatic route? I've tried out http://clojure.java.io/output-stream but it's not anywhere close what I get out of dd.

WhoNeedszZz15:01:13

I wouldn't compare performance to dd.

kwladyka12:01:44

I never did performance research for such case. I don’t feel competent to answer. From general point of view I will try to read stream from memory if possible, if know size before save create X MB empty file and write to it (not really sure how and if it performance better, but I think it should), if you can use dd use it instead clojure library.

kwladyka12:01:22

Maybe use bash input stream instead? But it shouldn’t be the case. I guess it is something about dd copy whole blocks which is something different than stream.

rovanion12:01:18

I'll probably look at java.nio next.

urzds13:01:30

How would I get the var from my function or macro argument? (defmacro b [f] (var f)) throws CompilerException java.lang.RuntimeException: Unable to resolve var: f in this context, compiling:(null:1:17) and defn has the exact same behaviour.

narendraj913:01:28

This is interesting. Looks like the symbol needs to resolve to a var at compile time not runtime. Something like this works:

(defmacro b [f] `(var ~f)) (b x)
given that x is already a defined var.

narendraj913:01:10

Both defn and defmacro are calling var special form during evaluation of a defn or defmacro definition (unless it is quoted).

narendraj913:01:16

This makes sense because vars represent identities that must not change even though the value that is pointed at by them may change at runtime.

narendraj913:01:06

@U066J7E2U can probably give a better or more accurate explanation here.

Shantanu Kumar16:01:59

var is a special form - it expects a symbol literal that corresponds with a var.

narendraj906:01:44

@U37339DBP I am interested in knowing the exact use case here. What are you trying to accomplish with a macro like that?

urzds11:01:40

@UDHE6D287 That macro is an MWE. The end goal is to retrieve specs with s/get-spec, which requires a var as argument. I was wondering how to get that var.

victorb13:01:39

Anyone know of any recent libraries similar to pallet and/or re-ops? Interested in managing infrastructure from Clojure directly

kwladyka14:01:25

I don’t but I have this feeling I shouldn’t use Clojure as tool for DevOps. I was trying in the past, but… I had bad experience. Just because of such solutions are not enough popular to be well supported and integrated.

kwladyka14:01:37

Better to learn good DevOps tool

kwladyka14:01:59

Unfortunately I have this impression all open source cicd tools are just bad 🙂 https://concourse-ci.org is open source and very interesting solution - unfortunately it is doing a little to slow progress while paid competitors are much faster. https://codefresh.io and https://circleci.com are good paid alternative IMO if you have budget.

kwladyka14:01:15

@victorbjelkholm429 but doing this in Clojure is a bad idea, because of the reasons which I mentioned above - just my opinion

victorb14:01:04

@kwladyka yes, so I'm not interested in ci/cd really, but just to manage instances from different cloud hosters. I personally already know Terraform for managing infra, but this project is a bit different. In the end, the UI I'm building is backed by a Clojure application, that app should be able to create/destroy cloud instances for the user, so can't really ask them to just learn Terraform instead 🙂

kwladyka14:01:06

oh and gitlab cicd

victorb14:01:29

probably will end up with some common interface, then wrappers around existing cloud libs to support that interface in the end. Seems easy enough but wanted to avoid writing it from scratch

kwladyka14:01:46

hmm I don’t know details to try help

kwladyka14:01:06

It looks like you have some kind of not standard needs

victorb14:01:08

eh, actually quite simple. Just need to be able to create/destroy/stat(get metrics) from hosts. pallet looked really nice for exactly this but seems outdated. If interested, here is the project described: https://github.com/ipfs-shipyard/cube

kwladyka14:01:37

I remember there was a library for Clojure to create pipelines in Clojure code. But what it does was draw UI. But I don’t remember the name and can’t find it in google.

kwladyka14:01:43

If it help in any way (from UI perspective)

kwladyka14:01:46

https://github.com/flosell/lambdacd this one, but it doesn’t look like it is well supported

kwladyka14:01:22

But idea was nice 🙂

kwladyka14:01:58

@victorbjelkholm429 as alternative maybe you can use kubernetes do deploy / destroy etc. ?

kwladyka14:01:15

then you can use the same way for all cloud providers

victorb14:01:47

@kwladyka thanks, but kubernetes is the wrong abstraction, still need to create the kubernetes cluster somehow, and that's the missing piece I'm out after right now

kwladyka14:01:29

ok so as another alternative you can run bash commands. Most of the popular providers let manage instances by bash commands

victorb14:01:15

@kwladyka well, bash is a horrible language. I'll probably end up going the route described in https://clojurians.slack.com/archives/C03S1KBA2/p1547043029371400 (with http APIs)

kwladyka14:01:50

You can run bash commands from Clojure. You don’t have to write bash scripts

victorb14:01:04

@kwladyka you mean with clojure.java.shell? In the end the bash commands you talk about end up doing http requests, so can just do the requests directly from clojure instead...

kwladyka14:01:53

yes, but it could be easier than interop and update, just saying an alternative

👍 5
myguidingstar14:01:09

@victorbjelkholm429 if you're serious about functional programming and linux, there's Nixos https://nixos.org/

victorb14:01:59

@myguidingstar yeah, I looked at nixos (specifically nixops) before. Didn't find anything that would concretely help me, but I did get some useful ideas from reading and understanding it

kwladyka14:01:02

@myguidingstar Interesting, what is the real difference in practice

victorb14:01:42

nixpkg is basically a immutable package manager, which is super useful (especially for work machines). Do an upgrade and something broke? Just rollback the entire system to the previous version and stuff is working again. nixos is a operating system using nixpkgs and the immutable ideas

kwladyka14:01:18

Hmm but considering for everything I use Docker images

victorb14:01:10

@kwladyka nixops is for the instance setup itself, not the application runtime you run on the instance.

victorb14:01:40

to run your images, you need a host + docker daemon somewhere. Nixops would help you operate that host + daemon

kwladyka14:01:45

I mean with docker images there is no issues about rollback / update

kwladyka14:01:12

Just thinking if NixOS can has benefits for me

myguidingstar14:01:27

@kwladyka nixpkgs can be used for building package ie your own app, due to its immutability even your app is reproducible (except database and file system)

myguidingstar14:01:03

anyway, it's a lot to learn, so you may consider cost vs benefit

kwladyka14:01:58

Well in this case I am thinking about it more in the sense to be early adopter 😉 Just curious what they did, but probably you are right.

victorb14:01:05

personally I hope nix (the package manager) takes over the world. Seems to be the only package manager who does things correctly.

myguidingstar14:01:52

the same here 🙂

kwladyka14:01:04

True, but I am afraid it is good for developers, not for common people 😉

victorb14:01:36

eh, common people don't use package managers anyways 🙂

kwladyka14:01:56

that is why it doesn’t matter for most of the people making it hard to became popular 😉

rovanion15:01:27

Surely it can be popular within certain populations. As long as we are speaking package managers it makes sense to speak of popularity with users of package managers.

kwladyka16:01:22

@whoneedszzz did you finish research about https://github.com/juxt/yada ? I am wonder if it is worth to switch to use it

WhoNeedszZz17:01:07

Yeah, I'm using it right now. My site is super basic, but it's been a really nice experience and the amount of code I had to write was tiny

kwladyka17:01:42

Is it public repo?

kwladyka17:01:25

How did you find out this repo exist?

kwladyka17:01:27

Asking because maybe I should start read some blog or something, because I miss that info

WhoNeedszZz17:01:55

Oh. I saw it in my research of popular libraries for creating an API server. It was mentioned in multiple places so I just read the docs at https://juxt.pro/yada/manual/index.html

👍 5
kwladyka18:01:28

https://github.com/juxt/edge this could be interesting place with best practices

kwladyka18:01:54

at least examples 🙂

WhoNeedszZz07:01:03

Got everything working now with yada. Final code is nice and small.

netpyoung17:01:17

I have a question. How can I get arglist from fn

clojure
(defn func1
  ([a])
  ([a b]))

(:arglists (meta (var func1)))
;;=> ([a] [a b])


(def func2 (fn
             ([a] (inc a))
             ([a b] (+ a b))))

(????? func2)
;; => ([a] [a b])

WhoNeedszZz17:01:15

Slack is kinda weird about triple backticks. To actually get Clojure syntax highlighting you need to add a code snippet with the '+' button.

netpyoung17:01:24

even using reflection?

bronsa17:01:02

you could reflect on the class backing that anonymous function and look for the invoke methods, and you'd get the arg numbers

bronsa17:01:05

but blargh

bronsa17:01:12

why do you need this?

netpyoung17:01:18

I want to call that function, using map. (do-someting func1 {:a 1 😛 2})

netpyoung17:01:30

(do-someting func1 {:a 1 :b 2})

dpsutton17:01:24

the doc-string for sort-by doesn't mention that it is stable but I thought i remember hearing this? Am i mistaken?

schmidt7317:01:29

"Guaranteed to be stable"

dpsutton17:01:53

thanks. wanted to make sure i wasn't just going on rumor

dpsutton17:01:18

oh. i'm in cljs. and that is conspicuously absent from the docstring

Alex Miller (Clojure team)17:01:59

that was added in clojure at some point a couple releases ago, maybe corresponding needs to be verified and added in cljs

dpsutton17:01:41

yeah. just put in #cljs-dev. It's a comment from Stu Halloway from 8 years ago that it is stable but the doc string purposefully doesn't promise it

WhoNeedszZz17:01:44

Why not? Is it stable or not?

dpsutton17:01:18

it's stable right now. But this is asking if cljs will promise this for all future versions essentially

WhoNeedszZz17:01:57

So i'd think that would be the better documentation than being ambiguous

dpsutton17:01:07

it's not ambiguous i suppose. it doesn't promise stability even though it has it. it just seems like this is something that it can promise though is all

WhoNeedszZz17:01:55

It's ambiguous by not stating. I'm suggesting it state exactly what you said - currently stable, but not guaranteed in future versions

WhoNeedszZz17:01:37

Then update the documentation if that changes...

seancorfield18:01:13

If the docstring is updated, it should be to just add "Not guaranteed to be stable" -- saying it's stable now but might change in the future isn't useful, IMO.

seancorfield18:01:34

If a docstring says "property X is true but might change", folks can't rely on property X remaining true so either they rely on it today (and might have their code broken in subtle/unexpected ways) or they explicitly don't rely on it at all (even tho' it may never actually change).

seancorfield18:01:15

It's better to either explicitly guarantee property X or explicitly not guarantee it.

dpsutton18:01:32

agreed. although i'm hoping that the 8 years of stable sort means it will just be acknowledged as a promise. But sean that's a very good comment you should leave on the ticket. Mark it as not guaranteed stable or guaranteed stable to remove confusion?

dpsutton18:01:01

Just ran to lunch. New cljs one

seancorfield18:01:54

Comment added 🙂

dpsutton18:01:52

appreciate your experience and sharing it 🙂

WhoNeedszZz19:01:04

I don't agree with that as "not guaranteed to be stable" is ambiguous. It is not stating that is or isn't stable, which was Dan's original question. If the documentation were to state that it is "Stable, but not guaranteed" now it's not ambiguous.

dpsutton19:01:14

all i'm looking for is what i can bank on for the future. if it is reasonable to promise stability I'm good. If there is no commitment to stability I need to find another solution regardless of current behavior

WhoNeedszZz19:01:03

I think it would be helpful if we had an explanation on why it may not be stable in the future

seancorfield19:01:21

I'm answering this as someone who spent nearly a decade working on ANSI/ISO standards committees. Saying "Stable, but not guaranteed" is not useful. Saying "Not guaranteed to be stable" provides more useful information.

☝️ 5
seancorfield19:01:15

(because it makes it clear that the stability of the current implementation is just an implementation detail and not part of the contract)

☝️ 5
WhoNeedszZz19:01:44

Please explain to me how something that is ambiguous is more useful than something that is not ambiguous

WhoNeedszZz19:01:12

The documentation should reflect the current behavior. If that behavior changes then the documentation should change with it. If it is stable now, then it should state that it is stable. If it becomes unstable then the documentation should change with that.

WhoNeedszZz19:01:14

Behavior in software changes. I don't subscribe to the idea that the documentation should try to predict the future.

seancorfield19:01:17

Stability of a sort is either part of the contract or part of the implementation. You need to know whether it is part of the contract. Either it is intended to be stable or it is not. That is true across Clojure's APIs in general -- if the implementation happens to provide "more" than the docstring, what's in the docstring is all you can rely on.

seancorfield19:01:56

In this case, the docstring provides no guidance (although it does not guarantee stability today so it's actually just fine as it is).

seancorfield19:01:11

If a Clojure docstring provides a guarantee, that is considered when changes are made in the future -- because the core team have a principle of not breaking guaranteed behavior.

seancorfield19:01:52

If the docstring provides no guarantee, behavior can change without needing to update the documentation (since the behavior was never guaranteed).

seancorfield19:01:36

It's a courtesy for the docstring to be explicit about not guaranteeing a specific behavior. That's kind of Clojure's basic position on a lot of things.

WhoNeedszZz19:01:21

Documentation of goog.array/stableSort: "Sorts the specified array into ascending order in a stable way. If no opt_compareFn is specified, elements are compared using goog.array.defaultCompare, which compares the elements using the built in < and > operators. This will produce the expected behavior for homogeneous arrays of String(s) and Number(s)."

seancorfield19:01:12

Right, and if the decision is to guarantee a stable sort -- as the Clojure docstring does -- then updating the (ClojureScript) docstring to make that same guarantee about current and future behavior would be the correct thing to do.

seancorfield19:01:51

You're missing my point that if there is no intended guarantee of that behavior, then the current docstring is accurate: it makes no guarantee about stability.

seancorfield19:01:47

Clojure's docstrings are generally very careful about what they do and don't guarantee. If they guarantee something, it is a contract that can be relied on and will not break in the future (because Rich says breakage is bad).

seancorfield19:01:21

If a docstring does not guarantee a specific behavior, then that behavior can change (because it wasn't part of the contract).

seancorfield19:01:37

Does that make what I'm saying clearer?

Lennart Buit19:01:17

(Maybe put it like this, that it uses stable sort is an implementation detail, it could as well use quicksort, which is not stable, but that wouldn’t matter as the stability is not required by the contract between caller (you calling sort) and the callee (cljs’s sort))

WhoNeedszZz19:01:20

I've understood what you're saying this whole time; I just don't agree with it. I don't believe my point is being understood, which is that the Clojure version is guaranteed stable and there is no reason that can't be true of the ClojureScript version as the function it is using is guaranteed to be stable. The goog.array library could disappear off of the face of the Earth and we'd still be fine because we could recreate the implementation as it is open source and relies only on essential JS operations that aren't going to change. The fact the CLJS code hasn't changed in 8 years should be a testament to that.

seancorfield20:01:48

You're still missing the point: the question isn't how it is implemented but whether the intent of the design is for a stable sort or not on cljs.

seancorfield20:01:16

If David Nolen (or whoever can make that decision) says "Yeah, we intended it to guarantee stability" then there's no question: the docstring should be updated to reflect that design intention as a guarantee -- and any future implementation changes should reflect that. If he/they say "Nope, we are not guaranteeing that for cljs" then the current docstring is correct, but I think it would be a courtesy to update the docstring to reflect that stability is (explicitly) not guaranteed.

WhoNeedszZz20:01:59

Please stop accusing me of missing the point when I've stated I just don't agree with what you're saying. I just referred to that indirectly in my above message. Obviously we still need to check with the powers that be what their intentions are. I stated that, objectively, there isn't any reason it couldn't be guaranteed given the expected and current functionality. I am not speaking for the powers that be as that is not my place.

WhoNeedszZz20:01:10

That being said it also begs the question as the implementation that has survived 8 years uses an, explicitly, stable sort function so I would think it should be pretty clear what the intention was when that code was written as they could have used any other sort function, but deliberately chose that one.

bcambel18:01:44

hey peeps, any advice which library to use with Spark ?

bcambel18:01:17

both flambo and sparkling haven't done any commits the last 6+ months..

WhoNeedszZz19:01:00

That may or may not be meaningful. Clojure libraries have a tendency to be completed for their stated objective(s) and haven't needed to change anything since then.

bcambel19:01:29

I am aware of that. However in this specific case Spark is already at 2.4 and the libs are in 2.1

Lennart Buit19:01:04

huh, googles impl of stableSort is actually quite interesting. Expected an actual stable sort algorithm, but instead they change the comparator to also take index into account.

Lennart Buit19:01:05

So it gains 2n extra time complexity for looping the array twice to add/remove indices, and n memory for the array of value/index pairs that is actually sorted

WhoNeedszZz19:01:05

Just O(n) extra: "Runtime: Same as Array.prototype.sort, plus an additional O(n) overhead of copying the array twice."

Lennart Buit19:01:27

right, O(n) and O(2n) are the same complexity class: O(n), it really loops over the array twice

dpsutton19:01:36

that's an interesting find @lennart.buit

Lennart Buit19:01:35

Sorting in javascript is a complete can of worms

Lennart Buit19:01:19

I guess that calling into v8s default sort algorithm is so much faster, that implementing a stability guarantee in javascript is more performant than implementing a complete stable sort alg.

Lennart Buit19:01:57

Or; it was easiest. But I am not yet prepared to accept that google engineers take shortcuts too.

dpsutton19:01:00

yes. it's using the built in sort on arr which probably changes algo depending on size. i've heard of quick sorts bottoming out into insertion sort at small n, etc

dpsutton19:01:28

(not saying that is what is happening here but could be the reason not to duplicate that kind of stuff)

Lennart Buit19:01:32

yeah, lists of less than 10 are actually instertion-sorted

andrewboltachev20:01:27

Hi all! I've built a small tool (based on rewrite-clj) that allows to search for a certain values in Clojure files (like grep) Idea is that one can try it from command line in seconds (no need to install): https://github.com/andrewboltachev/fo0001#clojure-value-finder-like-grep Not sure if alternatives exist. Feedback appreciated 🙂

Alex Miller (Clojure team)20:01:58

someone go make that for Clojure! :)

jumar07:01:51

not that fancy but debux works for me in those cases: https://github.com/philoskim/debux#81-dbg-inside-dbgn-or-vice-versa

cfleming21:01:27

Not for Clojure yet, but it’s on some list I have somewhere.

👍 5
Kyle Cavalieri22:01:28

Anybody around?

dguay22:01:43

Can someone ban this guy (Tariq Nasheed)

☝️ 85
ghadi22:01:31

I thought at-everyone is disabled

dpsutton22:01:54

yeah i was surprised sa well

Tariq Nasheed22:01:50

white supremacist slack

fellshard22:01:57

The spammers probably found a way to bypass it

fellshard22:01:40

Given they already have robust impersonation utilities.

eggsyntax23:01:53

Huh, the above comments from "Tariq" are interesting. That makes it seem like either a) it was a human troll rather than a bot, or b) the bot watches for its (impersonated) name and then says the stuff above. Bit more sophisticated than I expected.

fellshard23:01:33

It's definitely a human at the wheel, there were direct back-and-forths in the original attack as well

idiomancy22:01:49

lol 3...2...1...

duncanmak22:01:54

yeah, it’s annoying

Kyle Cavalieri22:01:00

Admins know about it

robert.spurrier22:01:00

dude!!!!! dont do that!

fellshard22:01:24

Last time this happened, they were impersonating an existing user, too.

g22:01:45

that has to be the most confusing meme i’ve ever seen

duncanmak22:01:47

I didn’t type that

boomboom22:01:11

how do you block someone?

👍 5
👌 5
fellshard22:01:12

just like that

fellshard22:01:20

UFA3W5N4U is the account that posted

Alex Miller (Clojure team)22:01:41

fyi, /collapse can be used to collapse all media in the room

😍 10
👏 5
shaunxcode22:01:49

maybe this the catalyst for the clojure community moving on from slack?

👆 10
ghadi22:01:56

and btw: do not download any attachments posted

jaide22:01:39

Perhaps that’s how they’ve been able to impersonate?

fellshard22:01:04

Turn off inline image display in your preferences, too; it's not often needed in this slack, and better safe than sorry

💯 10
robert.spurrier22:01:06

Preferences > Messeges & Media

👍 10
seancorfield22:01:41

User deactivated. We're working on deleting the messages wherever they got posted. Feel free to post in #slack-help with links to ones we haven't gotten to yet. And, yes, these bots are able to get around the restrictions that Slack lets us put in place to prevent non-admins using @ everyone/channel/here...

spieden22:01:58

yeesh, client-side permissions enforcement?

seancorfield23:01:20

Confirmed via an article one of the other members here sent me (thank you @jayzawrotny!). You can hack the permission client-side in JS and then spam with immunity it seems. Slack have known about it for two and a half years and confirmed a year and a half ago that they won't fix it. Sigh.

🚮 15
💩 40
dpsutton23:01:07

Also if you remap caps lock to control the caps lock button still makes the text box take focus rather than copying 😞

seancorfield23:01:10

Probably the only thing we can do at this point is to change the auto-inviter to require moderation somehow.