Fork me on GitHub
#clojure
<
2017-06-05
>
noisesmith00:06:38

right - all your code should be behind some kind of rate limiting, if it's open to public requests

macalimlim01:06:39

Hi 🙂 may I know if there is way to hotload jars at runtime?

noisesmith01:06:52

yes, pomegranate makes it relatively straightforward, or pallet/alembic is even easier, as long as you are using lein

macalimlim02:06:29

how about for clojurescript? hotloading the application (which is already in production) at runtime..

kevinludwig05:06:24

using https://github.com/funcool/struct, how do I validate the type of objects nested inside an array, e.g. {:addresses [{:street "blah" :city "wherever" :state "NY" :zip 12345} {:street "another street" :city "another city" :state "MA" :zip 34234}]}

kevinludwig05:06:00

I've figured out that I can do a schema like {:addresses [st/vector]} but that's hardly useful since it allows anything in the addresses vector.

kevinludwig05:06:24

the documentation conveniently leaves out any example of using st/vector

kevinludwig05:06:07

as a more general question, I was going to use this for input validation for some REST APIs, e.g. to validate a POST or PUT body. I started by looking at this because it's apparently what Luminus uses, http://www.luminusweb.net/docs/input_validation.md.

kevinludwig05:06:13

after looking some more I see there is some basic support for checking every method of an array, e.g. st/every although I was unable to get that working even for a trivial case like this (st/validate {:xs ["a" "b"]} {:xs [st/every string?]})

kevinludwig05:06:24

^^ that just throws ClassCastException clojure.core$string_QMARK___4365 cannot be cast to clojure.lang.Associative clojure.lang.RT.assoc (RT.java:792)

kwladyka07:06:44

@kevinludwig why not clojure.spec?

dominicm08:06:55

kwladyka: Not OP but: - Better messages than spec can provide - Possibly (not overly familiar with struct) but error messages which validate across keys - Before passing a map to spec, you have to validate it, because there's potential for security flaws due to the open-ness of maps (solved via select-keys, but too easy to forget)

dominicm08:06:22

Oh, and struct validators can be generated on the fly, meaning you can check in the db for validity (e.g. email uniqueness)

kevinludwig14:06:15

Yeah mostly the error messages.

pesterhazy08:06:10

I'm looking for a lightweight documentation tool to generate a single html page manual. I like this output: http://funcool.github.io/struct/latest/ but it uses Asciidoctor, thus asciidoc. Is there anything like this that uses Markdown?

pesterhazy08:06:47

(Or explain to me why adoc is worth learning?)

dominicm09:06:40

@pesterhazy asciidoctor is really good. It's very much like markdown, but with enough features for use in actual technical documentation. Native cross-references & bibliography generation make it very useful for our internal documentation. The yada manual is also in Asciidoc for similar reasons.

pesterhazy09:06:47

I think I can live without bibliography. Cross-references I think I can do in md too?

pesterhazy09:06:19

I'm just wondering if Markdown lowers the bar for contributors to the manual

pesterhazy09:06:36

Any other advantages for adoc @dominicm? I know adoc is also used for the Clojure guides

dominicm09:06:47

@pesterhazy those were just a few off the top of my head. There's numerous additional features available. They're all conspire to make it actually useful for documentation. It can pull out of source files to do embedded code examples. I ❤️ this: http://asciidoctor.org/docs/user-manual/#callouts The cross-references are better in adoc than md.

dominicm09:06:42

Table of contents are really useful too.

pesterhazy09:06:19

The callouts feature is neat

dominicm09:06:53

You can combine callouts with http://asciidoctor.org/docs/user-manual/#include-directive so you can have your code snippets all tested to ensure they never break.

pesterhazy09:06:19

ok I'll think about it

lxsameer09:06:38

spec/valid? says that the given data is not valid for spec ::x but spec/assert does not throw an exception. is it a bug ?

pesterhazy11:06:32

@metametadata thanks. The default styles are a bit off-putting

pesterhazy11:06:42

do you know any project that uses this?

metametadata12:06:39

pesterhazy: yeah, I use it for my projects, e.g.: http://metametadata.github.io/clj-fakes/user-guide/

pesterhazy12:06:28

ah I like the style there

pesterhazy12:06:32

did you customize it?

pesterhazy12:06:00

nice, I really like this

pesterhazy12:06:30

did you run into any limitations?

metametadata12:06:39

the biggest hurdle was that TOC is not scrollable in some themes by default

metametadata12:06:08

and making GitHub's README a part of site generation

pesterhazy12:06:39

scrolling seems to work in the theme you picked

metametadata12:06:40

thus I have a script that temporarily copies it into the docs folder before site generation

metametadata12:06:51

yeah, it's because I fixed it in CSS, hopefully

pesterhazy12:06:05

heh, works for me 🙂

taylor13:06:40

I want to use what is basically a Result monad to handle some request validation/processing in a Compojure handler. I suppose this would be conceptually similar to Ring middleware handlers, except on a per-route basis. I implemented this using reduce and a series of functions that take/return “result” maps e.g. {:result :success :state foo}, and the reduce function exits early on error. It works fine but I feel like there’s probably a better/more Clojure-y way? Basically, I’m doing this to avoid having a big, nasty, deeply-nested control flow structure in the request handler.

souenzzo14:06:25

There is a smarter way to do (reduce #(%2 %1) ctx chain)? Some like (high-order-fn chain ctx )

captainlexington14:06:01

And you can replace your reduce with a bespoke loop/recur construct so you can control flow a little more elegantly than with exiting early on errors

captainlexington14:06:22

It may even be possible to follow @souenzzo 's advice and mine a the same time!

captainlexington14:06:41

And find some kind of extensible pattern in your loop/recur that you can reuse

mccraigmccraig14:06:04

you could also use a monad from cats @taylor - i don't know the Result monad, but i'm guessing it's a different name for https://github.com/funcool/cats/blob/master/src/cats/monad/maybe.cljc or https://github.com/funcool/cats/blob/master/src/cats/monad/either.cljc

taylor14:06:01

thanks, and yes @mccraigmccraig this would be synonymous with the Either monad

taylor14:06:01

I suppose I could also implement this as a special threading macro that checks the output of each function and returns early if it’s an “error”

joelsanchez15:06:30

Still don't know how to do the same for cljs

noisesmith16:06:55

yeah, I think you need to coordinate the backend finding the dep and downloading it, and sending it to the frontend to be evaluated, and that’s a gaping security issue if you do it sloppy

noisesmith16:06:23

I guess you could make it happen via dev websocket (figwheel) only and assume that’s never connected in the real app… ?

carocad18:06:29

Hey guys, does anybody has some experience with inductive graph? As defined in the "functional graph library" in Haskell

carocad18:06:12

I'm trying to check if it makes sense (performance-wise) to implement some graph traversal algorithms using that approach but I'm a bit worried about is performance. Any advice is more than welcome :)

nmax19:06:29

Hi, is there a channel for novices? I've some troubles with hello world)

captainlexington19:06:09

nmax: There's a #beginners

husain.mohssen20:06:21

My JVM is dying with an out of memory loading up a big file. Here is my code:

(def x (slurp "/data/3GBfile.txt"))

             java.lang.OutOfMemoryError: 
clojure.lang.Compiler$CompilerException: java.lang.OutOfMemoryError, compiling:(form-init1107474503944394809.clj:1:8)

husain.mohssen20:06:31

I have more than enough memory in the JVM:

husain.mohssen20:06:44

(. (java.lang.management.ManagementFactory/getRuntimeMXBean) getInputArguments)
["-Dfile.encoding=UTF-8" "-Xmx12g" "-Dclojure.compile.path=/Users/halmohssen/src/etl/target/classes" "-Detl.version=0.1.0-SNAPSHOT" "-Dclojure.debug=false"]

husain.mohssen20:06:55

What's going on here?

hiredman20:06:41

you should check the stracktrace, but it is likely either in stringbuilder or bytearrayoutputstream, both of which, if I recall, grow their internal buffer by doubling

husain.mohssen20:06:50

yes @hiredman seems to be the problem

husain.mohssen20:06:57

{:type java.lang.OutOfMemoryError
   :message nil
   :at [java.lang.AbstractStringBuilder hugeCapacity "AbstractStringBuilder.java" 161]}

husain.mohssen20:06:24

what's going on, I have 4x RAM

souenzzo20:06:43

clojure encourages to use namespaced and not nested map (I agree it's a great idea.) There is some (plans?) tools to operate with namespace'd maps? Example (qualify :a {:foo :bar}) ;=> {:a/foo :a/bar} (split-ns {:foo/bar 1 :bar/foo 2}) ;=> {:foo {:foo/bar 1} :bar {:bar/foo 2}} (dissoc-ns {:foo/bar 33 :bar/foo 55} :foo) ;=> {:bar/foo 55}

mfikes20:06:29

Perhaps 2 bytes per character in that 3GB file doubles it to 6GB, and then double that is 12GB?

hiredman20:06:50

I would recommend using a randomaccessfile and using map to make a mmap'ed bytebuffer

souenzzo20:06:03

(@nathanmarz I know that #specter do that. I'm using and loving 😉 )

hiredman20:06:25

oh, you have to randomaccess file -> channel -> map, so you can avoid the raf by using a filechannel directly, but that

husain.mohssen20:06:50

@hiredman can you elaborate more? I'm a big confused.

hiredman20:06:00

which will return an implementation of ByteBuffer backed by the file mapped (as with mmap) in to memory

hiredman20:06:13

there are a few ways to get a filechannel, the randomaccess file route is just what I usually do https://docs.oracle.com/javase/7/docs/api/java/io/RandomAccessFile.html

hiredman20:06:57

the memory mapped file also won't count against the jvm heapsize, the garbage collector doesn't have to care about it

mfikes20:06:52

Looks like a lib exists, that would make it trivial to try the approach: https://github.com/thebusby/clj-mmap

hiredman20:06:29

slurp is basically a toy convenience for hacking around sometimes, generally not someone you want to use

qle-guen20:06:57

Hey I'm doing a game of life to learn Clojure. Which display library should I use?

qle-guen20:06:41

Seems to be exactly what I need thanks

husain.mohssen20:06:23

@hiredman @mfikes slurp was a way of showing the problem. I actually was getting OOM errors when doing lazy reads using csv-read.

husain.mohssen20:06:36

I'll play with it more and report back.

husain.mohssen20:06:55

I'm trying to build a pipline that reads from csv's and writes to other csv's

husain.mohssen20:06:02

and the input ones are a few GB in range.

nadejde20:06:24

Hi! Has anyone successfully used ring with datomic client? I’m really struggling with dependency conflicts. Can’t seem to find a version of jetty that works with both datomic client and lein-ring plugin

hiredman20:06:51

I haven't used csv-read, but my guess there (ignoring anything in data.csv itself), is using strings, for large data sets String is pretty terrible

hiredman20:06:09

java characters are utf-16, so 10 bytes of data, when represented as a String becomes 20 bytes

husain.mohssen20:06:24

If my data is just ASCII is there a simple way for strings not to use UTF16 ?

joelsanchez21:06:06

@nadejde Using ring and datomic but not jetty - httpkit. I put a nginx server in front of it to use https and static resources. Assuming you want to use Jetty, what's the error / conflict you are getting?

hiredman21:06:30

newer jvms have a flag for compressed strings that will be utf-8, but I am not sure if how reliable that is

Alex Miller (Clojure team)21:06:37

@husain.mohssen in Java 9 they are automatically compressing strings

hiredman21:06:14

the mapped bytebuffer approach gives you bytes, so great for ascii

hiredman21:06:30

but you will have to roll csv yourself unless you can find a library for it

Alex Miller (Clojure team)21:06:56

And they also dedupe identical strings in the gc (that's available as an option in Java 8)

alex-dixon21:06:19

Anyone willing to help a noob deploy to clojars? 🙂

joelsanchez21:06:49

In my experience it is "almost" just making an account in Clojars and executing lein deploy clojars

joelsanchez21:06:54

What problems are you facing?

alex-dixon21:06:17

Eh heh. Well it’s asking me to sign with gpg and I’m having problems with that. Avoding signing for now would be fine I guess but I can’t figure that out either

mobileink21:06:27

@alex-dixon deploy from where? boot, leiningen, other?

mobileink21:06:25

alex-dixon: my advice is to use boot, heh. seriously, sorry i can't help with lein.

alex-dixon21:06:31

It’s cool. Thanks. I actually tried with boot. I got some kind of error. What’s the command? Do I need a build.boot file?

mobileink21:06:54

alex-dixon: but fwiw, i did spend the time online figuring out how to set up gpg for boit, i imagine it's similar. annoying, but worth the effort.

alex-dixon21:06:36

Yikes. Ok. Thank you 🙂

alex-dixon21:06:08

Awesome. Thank you so much

alex-dixon21:06:41

But at this point I’ll take anything 🙂

joelsanchez21:06:20

I guess you'd need to setup GPG following the instructions given by your distro

nadejde21:06:27

@joelsanchez I’l look into httpkit. I don’t have to use jetty i guess that’s only the default i guess. The error I get is: java.lang.ClassNotFoundException: org.eclipse.jetty.http.HttpParser$ProxyHandler. datomic client forces an old version of jetty. If I try to use a newer one datomic client breaks and ring works:)

joelsanchez21:06:23

@nadejde You can solve that easily using an exclusion, you know your way around them?

alex-dixon21:06:35

I’m on Mac. I followed clojars-web’s instructions using brew install. gpg: signing failed: Inappropriate ioctl for device

mobileink21:06:28

alex-dixon: try $ export GPG_TTY=$(tty). I had a similar problem with boot.

alex-dixon21:06:07

@U0LGCREMU OMFG worked. Ha-ha! Thank you!!

mobileink21:06:45

alex-dixon: i tried putting that in .bash_profile but i still have to do it by hand. no prob since i rarely deploy, but please let me know if you figger sth like that out. cheers!

nadejde21:06:00

Tried ever witch way. The problem is there’s not any version of jetty that works with both datomi client and ring (or have not been able to find it)

joelsanchez21:06:04

Damn I have 0 Mac knowledge, can't help, but maybe stackoverflow can - after all it's not clojure specific

nadejde21:06:57

if exclude at datomic client level datomic client breaks as it cant work with what ring brings in. If I exclude at ring level ring breaks as it can’t work with what datomic brings in…

noisesmith21:06:18

@alex-dixon I’ve found that on a mac the only usable thing is to install the GUI client MacGPG

nadejde21:06:27

even tried excluding at both and manually putting jetty in. just cant get it to work

noisesmith21:06:37

using the command line gpg from mac does not work with lein at all because lein doesn’t know how to give up the tty

noisesmith21:06:46

and it would need to for the homebrew gpg to work

joelsanchez21:06:01

Yikes @nadejde looks tough, let me see if I can find something

noisesmith21:06:40

@alex-dixon there’s a workaround where you decode a random file using your credentials first so that the auth is in cache, but that’s just a hack and MacGPG fixes the issue

joelsanchez21:06:47

The "solution" they propose is to forget jetty and use http-kit

joelsanchez21:06:03

Maybe try that? After all you don't mind about jetty

nadejde21:06:23

@joelsanchez I will do just that!

alex-dixon21:06:29

@noisesmith Thanks. Trying that now. Why do we put up with this though? 😕

nadejde21:06:58

@joelsanchez Thank’s for you help:) Will try it with http-kit. Looked at their website should be really straight forward to setup

captainlexington21:06:05

Rich Hickey talked about that last conj. He said spec is an attempt to fix the problem

captainlexington21:06:14

I still feel like it maybe has an easier solution?

captainlexington21:06:35

Like maven please just use two different versions of jetty okay

joelsanchez21:06:44

@nadejde Yes, it's a breeze, also it's pretty damn fast (but remember to setup a reverse proxy like nginx in production, since httpkit is quite limited in some areas, for example I think it can't do https - not designed for that)

nadejde21:06:30

@captainlexington I think I saw that talk:) Good talk it was

captainlexington21:06:57

@nadejde He convinced me. I'll never release breaking changes in a library! :fist:

nadejde21:06:14

@joelsanchez I’m only playing around for now. Trying to learn some clojure, ring , datomic etc. Not going into production anytime soon

mobileink21:06:18

@alex-dixon: try $ export GPG_TTY=$(tty). I had a similar problem with boot.

nadejde21:06:49

@captainlexington If only everyone could do that 🙂

josh.freckleton21:06:40

using clojure's bindings to jdbc, can I get back the auto-incremented id from an execute!

josh.freckleton21:06:42

specifically, I have an "INSERT ... ON DUPLICATE KEY UPDATE ..." and it also has an auto inc'd key, can I get that back after the jdbc/execute!?

currentoor21:06:03

Is there anyway to tell if a a thread created by a future has died?

dpsutton21:06:31

@josh.freckleton if you're using insert! you've got the following contract: > When inserting a row as a map, the result is the database-specific form of the generated keys, if available (note: PostgreSQL returns the whole row).

dpsutton21:06:59

wasn't sure if there was a wat to change to isnert!

dpsutton21:06:06

sorry if you already knew that

josh.freckleton21:06:10

oh sorry, i'm on mysql (wish it were postgres 🙂 )

dpsutton21:06:27

well on mysql you get back the id entered

josh.freckleton21:06:32

can I achieve an "upsert" with jdbc/insert!?

josh.freckleton21:06:45

(IE "INSERT ... ON DUPLICATE KEY UPDATE ..."

josh.freckleton21:06:59

or could I get "REPLACE" to return an id?

dpsutton21:06:18

is there no way to know if it'll have a duplicate key?

dpsutton21:06:27

why would there be a duplicate key?

josh.freckleton21:06:12

the key is made out of 2 foreign keys, and our system has been throwing duplicate key errors

josh.freckleton21:06:52

ya, in this case, I think it's just as easy to "upsert" it

josh.freckleton21:06:10

the alternative was (try insert... (catch update...))

tanzoniteblack21:06:54

@josh.freckleton , I believe you can do this in mysql with INSERT INTO table SET unique1=value1 AND unique2=value2… ON DUPLICATE KEY UPDATE <anything you want to update>, id=LAST_INSERT_ID(id); the last_insert_id(<my unique id>) should get it to behave such that mysql will return either the newly inserted id or the id of the row which you updated

tanzoniteblack21:06:05

I think; I’m far more familiar with postgres

josh.freckleton21:06:21

and then jdbc/insert! instead of jdbc/execute!? and, I'll test this, but do you know if that id update could reset the auto inc'd id?

tanzoniteblack21:06:27

I think you’d need to use execute!, since it’s not a straight forward insert!

josh.freckleton21:06:51

oh yep, just saw that in the signature of insert!

tanzoniteblack21:06:59

and I believe that mysql will always auto increment id’s when you do on duplicate key update, if I remember mysql correctly?

josh.freckleton22:06:33

so, IIUC, it increments the counter, so future inserts have a higher id, but shouldn't inc the current one I don't think

tanzoniteblack22:06:17

(I do recommend you test this, like I said, I’m far more familiar with postgres than mysql)

josh.freckleton22:06:16

hm, the execute just returns the "number of rows effected", i think

bcbradley22:06:37

i wonder if anyone has ever implemented an ISeq that is memory mapped, so as it grows it moves from the jvm heap to the filesystem

bcbradley22:06:36

i figure making something like that with standard IO wouldn't be hard, but page swapping is so much more efficient

josh.freckleton22:06:07

dpsutton: 1.

Unhandled java.sql.SQLException
   Can not issue data manipulation statements with executeQuery().

tanzoniteblack22:06:01

@josh.freckleton did you verify that the code I gave you actually returns the id (like through the mysql command line?)

tanzoniteblack22:06:11

I think if you get that to work right, then execute! should work?

josh.freckleton22:06:01

hm, from the mysql cli, it just returns "rows affected"

josh.freckleton22:06:39

1=insert, 2=update, 0=update was the same as current row

bcbradley22:06:12

my use case is a video game where i store a seq of all past game states-- the idea is that you never have to "save" the game because it stores its entire history

bcbradley22:06:50

that would also make backtracking to a previous state easy, which enables a lot of cool designs for the gameplay

grzm22:06:18

I'm trying to learn how to eventually extend Java classes. I'm having issues figuring out how to use gen-class in Cider. Here's my example:

(ns ex.Example
  (:gen-class))

(defn -toString
  [this]
  "Hello, World!")
I'm using https://kotka.de/blog/2010/02/gen-class_how_it_works_and_how_to_use_it.html as a guide. I get a ClassNotFoundException when I try to instantiate it via (ex.Example.). I get "java.lang.ClassCastException: java.io.File cannot be cast to java.lang.String" when I try to compile it via (compile 'ex.Example). I'm probably missing something basic. Any pointers to better references most welcome!

hiredman22:06:38

that compile exception sounds like you set whatever the compile path var is to a File object instead of a String

hiredman22:06:53

in generally, gen-class doesn't work well with the repl

hiredman22:06:16

unless you really need gen-class (which you usually don't) I would avoid using it for anything

hiredman22:06:57

if you must, a good thing to read is https://clojure.org/reference/compilation because gen-class requires aot compilation

grzm22:06:02

@hiredman thanks. My understanding is that if I need to extend a Java class that will be referenced by other Java classes, I need to use gen-class. In this case, I'm extending logback's AbstractDiscriminator.

grzm22:06:43

I was just copying something out of the channel topic, and it seems I accidentally updated it as well? If I screwed it up, apologies.

joelsanchez22:06:00

Surprised that the channel-topic var is not ^:private

hiredman22:06:56

grzm: yeah, that is likely the case, I would check out the aot compilation documentation for whatever build tool you use, and checkout that link

grzm22:06:34

@joelsanchez You and me both!

grzm22:06:45

@hiredman sounds like a plan. Thanks.

grzm22:06:21

@hiredman looks like calling (boot (aot :namespace '#{ex.Example})) in my repl (using boot and Cider) has moved me forward at least one step.

hiredman22:06:10

the problem is, aot compilation generates class files on disk, and per classloader hierarchy (and you only get one without backflips) you can only load them once

hiredman22:06:45

so if you want to change the compiled ex.Example, and need to recompile, you will likely need to restart your repl

qqq22:06:54

is user.clj in src/ or elsewhere

grzm22:06:00

@hiredman Yup. Been working on that as well. From what I'm gathering, as long as I'm not changing the signature of the class, I can update the internals of the functions/methods without restarting the repl.

hiredman22:06:08

user.clj is loaded by clojure.lang.RT from anywhere on the classpath

qqq23:06:14

@hiredman: okay, so it's meant for stuff that are repl specific rather than project specific ?

hiredman23:06:53

I have rarely seen it used

seancorfield23:06:42

(I don’t think user.clj works with Boot, BTW?)

seancorfield23:06:38

(because Boot starts Clojure up before it’s even read build.boot so there’s no user.clj on the classpath for Clojure to load — as I understand it)

grzm23:06:53

@seancorfield You mean that user.clj won't be evaluated on start up, correct? The classpath used within the repl is going to be what's defined in build.boot, right?

seancorfield23:06:27

@grzm Leiningen creates two JVMs: the second one is in the context of the project and will execute user.clj as part of that. Boot only creates one JVM and it modifies the context once it has read build.boot. So user.clj won’t be loaded by Clojure with Boot, because Clojure starts before any user.clj could be on the classpath.

seancorfield23:06:48

(not sure whether that’s clearer or not?)

seancorfield23:06:08

With Boot, you can modify the context after the REPL has started (so saying the “classpath used within the repl is going to be what’s defined in build.boot” is a bit squishy).

grzm23:06:41

Yup, it is. I was getting at the "execute" part. user.clj can certainly be on the classpath as defined by :resource-paths in build.boot, right? You just don't get the felicity of having it already executed prior to the repl prompt appearing.

seancorfield23:06:27

Right, the automatic behavior of Clojure (to load and run user.clj) doesn’t happen with Boot, but you can load and run it yourself in the REPL.

grzm23:06:53

Cool. That's how I've understood the behavior I'm seeing. Thanks for confirming.