This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-06-05
Channels
- # aws (6)
- # beginners (10)
- # boot (33)
- # cider (23)
- # cljs-dev (50)
- # cljsjs (2)
- # cljsrn (10)
- # clojars (1)
- # clojure (215)
- # clojure-czech (1)
- # clojure-dev (5)
- # clojure-italy (23)
- # clojure-russia (18)
- # clojure-spec (11)
- # clojure-uk (53)
- # clojurescript (157)
- # core-async (29)
- # cursive (12)
- # data-science (15)
- # datascript (16)
- # datomic (68)
- # graphql (2)
- # jobs (5)
- # jobs-discuss (1)
- # juxt (17)
- # lein-figwheel (2)
- # luminus (3)
- # off-topic (155)
- # om (3)
- # pedestal (1)
- # portkey (1)
- # re-frame (7)
- # reagent (4)
- # ring (3)
- # ring-swagger (2)
- # rum (11)
- # unrepl (11)
- # vim (1)
- # yada (2)
right - all your code should be behind some kind of rate limiting, if it's open to public requests
Hi 🙂 may I know if there is way to hotload jars at runtime?
yes, pomegranate makes it relatively straightforward, or pallet/alembic is even easier, as long as you are using lein
how about for clojurescript? hotloading the application (which is already in production) at runtime..
using https://github.com/funcool/struct, how do I validate the type of objects nested inside an array, e.g. {:addresses [{:street "blah" :city "wherever" :state "NY" :zip 12345} {:street "another street" :city "another city" :state "MA" :zip 34234}]}
I've figured out that I can do a schema like {:addresses [st/vector]}
but that's hardly useful since it allows anything in the addresses vector.
the documentation conveniently leaves out any example of using st/vector
as a more general question, I was going to use this for input validation for some REST APIs, e.g. to validate a POST or PUT body. I started by looking at this because it's apparently what Luminus uses, http://www.luminusweb.net/docs/input_validation.md.
after looking some more I see there is some basic support for checking every method of an array, e.g. st/every
although I was unable to get that working even for a trivial case like this (st/validate {:xs ["a" "b"]} {:xs [st/every string?]})
^^ that just throws ClassCastException clojure.core$string_QMARK___4365 cannot be cast to clojure.lang.Associative clojure.lang.RT.assoc (RT.java:792)
@kevinludwig why not clojure.spec?
kwladyka: Not OP but: - Better messages than spec can provide - Possibly (not overly familiar with struct) but error messages which validate across keys - Before passing a map to spec, you have to validate it, because there's potential for security flaws due to the open-ness of maps (solved via select-keys, but too easy to forget)
Oh, and struct validators can be generated on the fly, meaning you can check in the db for validity (e.g. email uniqueness)
Yeah mostly the error messages.
I'm looking for a lightweight documentation tool to generate a single html page manual. I like this output: http://funcool.github.io/struct/latest/ but it uses Asciidoctor, thus asciidoc. Is there anything like this that uses Markdown?
(Or explain to me why adoc is worth learning?)
Hm, CIDER uses readthedocs and markdown: https://github.com/clojure-emacs/cider/tree/master/doc
@pesterhazy asciidoctor is really good. It's very much like markdown, but with enough features for use in actual technical documentation. Native cross-references & bibliography generation make it very useful for our internal documentation. The yada manual is also in Asciidoc for similar reasons.
I think I can live without bibliography. Cross-references I think I can do in md too?
I'm just wondering if Markdown lowers the bar for contributors to the manual
Any other advantages for adoc @dominicm? I know adoc is also used for the Clojure guides
@pesterhazy those were just a few off the top of my head. There's numerous additional features available. They're all conspire to make it actually useful for documentation. It can pull out of source files to do embedded code examples. I ❤️ this: http://asciidoctor.org/docs/user-manual/#callouts The cross-references are better in adoc than md.
The callouts feature is neat
You can combine callouts with http://asciidoctor.org/docs/user-manual/#include-directive so you can have your code snippets all tested to ensure they never break.
ok I'll think about it
thanks!
spec/valid?
says that the given data is not valid for spec ::x
but spec/assert
does not throw an exception. is it a bug ?
@pesterhazy take a look at http://www.mkdocs.org/
@metametadata thanks. The default styles are a bit off-putting
do you know any project that uses this?
pesterhazy: yeah, I use it for my projects, e.g.: http://metametadata.github.io/clj-fakes/user-guide/
ah I like the style there
did you customize it?
It's a theme from here: http://sourcefoundry.org/cinder/ And these are the only customizations I added: https://github.com/metametadata/clj-fakes/blob/master/docs/custom.css
nice, I really like this
did you run into any limitations?
the biggest hurdle was that TOC is not scrollable in some themes by default
and making GitHub's README a part of site generation
scrolling seems to work in the theme you picked
thus I have a script that temporarily copies it into the docs folder before site generation
yeah, it's because I fixed it in CSS, hopefully
heh, works for me 🙂
nice 🙂
I want to use what is basically a Result monad to handle some request validation/processing in a Compojure handler. I suppose this would be conceptually similar to Ring middleware handlers, except on a per-route basis. I implemented this using reduce
and a series of functions that take/return “result” maps e.g. {:result :success :state foo}
, and the reduce
function exits early on error. It works fine but I feel like there’s probably a better/more Clojure-y way? Basically, I’m doing this to avoid having a big, nasty, deeply-nested control flow structure in the request handler.
There is a smarter way to do (reduce #(%2 %1) ctx chain)
? Some like (high-order-fn chain ctx )
And you can replace your reduce
with a bespoke loop/recur
construct so you can control flow a little more elegantly than with exiting early on errors
It may even be possible to follow @souenzzo 's advice and mine a the same time!
And find some kind of extensible pattern in your loop/recur
that you can reuse
you could also use a monad from cats
@taylor - i don't know the Result
monad, but i'm guessing it's a different name for https://github.com/funcool/cats/blob/master/src/cats/monad/maybe.cljc or https://github.com/funcool/cats/blob/master/src/cats/monad/either.cljc
thanks, and yes @mccraigmccraig this would be synonymous with the Either monad
I suppose I could also implement this as a special threading macro that checks the output of each function and returns early if it’s an “error”
@macalimlim A useful macro for clj (you need to have pomegranate): https://gist.github.com/JoelSanchez/b553ee49ad8490d351b4526bd38e76ca
Still don't know how to do the same for cljs
yeah, I think you need to coordinate the backend finding the dep and downloading it, and sending it to the frontend to be evaluated, and that’s a gaping security issue if you do it sloppy
I guess you could make it happen via dev websocket (figwheel) only and assume that’s never connected in the real app… ?
Hey guys, does anybody has some experience with inductive graph? As defined in the "functional graph library" in Haskell
I'm trying to check if it makes sense (performance-wise) to implement some graph traversal algorithms using that approach but I'm a bit worried about is performance. Any advice is more than welcome :)
FYI: great guide on concurrency in Clojure: https://purelyfunctional.tv/guide/clojure-concurrency-the-ultimate-guide/
nmax: There's a #beginners
My JVM is dying with an out of memory loading up a big file. Here is my code:
(def x (slurp "/data/3GBfile.txt"))
java.lang.OutOfMemoryError:
clojure.lang.Compiler$CompilerException: java.lang.OutOfMemoryError, compiling:(form-init1107474503944394809.clj:1:8)
I have more than enough memory in the JVM:
(. (java.lang.management.ManagementFactory/getRuntimeMXBean) getInputArguments)
["-Dfile.encoding=UTF-8" "-Xmx12g" "-Dclojure.compile.path=/Users/halmohssen/src/etl/target/classes" "-Detl.version=0.1.0-SNAPSHOT" "-Dclojure.debug=false"]
What's going on here?
you should check the stracktrace, but it is likely either in stringbuilder or bytearrayoutputstream, both of which, if I recall, grow their internal buffer by doubling
yes @hiredman seems to be the problem
{:type java.lang.OutOfMemoryError
:message nil
:at [java.lang.AbstractStringBuilder hugeCapacity "AbstractStringBuilder.java" 161]}
I'm confused
what's going on, I have 4x RAM
clojure encourages to use namespaced and not nested map (I agree it's a great idea.)
There is some (plans?) tools to operate with namespace'd maps? Example
(qualify :a {:foo :bar}) ;=> {:a/foo :a/bar}
(split-ns {:foo/bar 1 :bar/foo 2}) ;=> {:foo {:foo/bar 1} :bar {:bar/foo 2}}
(dissoc-ns {:foo/bar 33 :bar/foo 55} :foo) ;=> {:bar/foo 55}
Perhaps 2 bytes per character in that 3GB file doubles it to 6GB, and then double that is 12GB?
I would recommend using a randomaccessfile and using map to make a mmap'ed bytebuffer
(@nathanmarz I know that #specter
do that. I'm using and loving 😉 )
oh, you have to randomaccess file -> channel -> map, so you can avoid the raf by using a filechannel directly, but that
@hiredman can you elaborate more? I'm a big confused.
so it sort of depends on a lot, but likely the optimal thing for you to do is to call https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode,%20long,%20long)
which will return an implementation of ByteBuffer backed by the file mapped (as with mmap) in to memory
there are a few ways to get a filechannel, the randomaccess file route is just what I usually do https://docs.oracle.com/javase/7/docs/api/java/io/RandomAccessFile.html
the memory mapped file also won't count against the jvm heapsize, the garbage collector doesn't have to care about it
Looks like a lib exists, that would make it trivial to try the approach: https://github.com/thebusby/clj-mmap
slurp is basically a toy convenience for hacking around sometimes, generally not someone you want to use
quil is nice http://quil.info/
@hiredman @mfikes slurp was a way of showing the problem. I actually was getting OOM errors when doing lazy reads using csv-read.
I'll play with it more and report back.
I'm trying to build a pipline that reads from csv's and writes to other csv's
and the input ones are a few GB in range.
Hi! Has anyone successfully used ring with datomic client? I’m really struggling with dependency conflicts. Can’t seem to find a version of jetty that works with both datomic client and lein-ring plugin
I haven't used csv-read, but my guess there (ignoring anything in data.csv itself), is using strings, for large data sets String is pretty terrible
java characters are utf-16, so 10 bytes of data, when represented as a String becomes 20 bytes
If my data is just ASCII is there a simple way for strings not to use UTF16 ?
@nadejde Using ring and datomic but not jetty - httpkit. I put a nginx server in front of it to use https and static resources. Assuming you want to use Jetty, what's the error / conflict you are getting?
newer jvms have a flag for compressed strings that will be utf-8, but I am not sure if how reliable that is
@husain.mohssen in Java 9 they are automatically compressing strings
And they also dedupe identical strings in the gc (that's available as an option in Java 8)
Anyone willing to help a noob deploy to clojars? 🙂
Start here (EDITED: more friendly link): https://github.com/clojars/clojars-web/wiki/Tutorial
In my experience it is "almost" just making an account in Clojars and executing lein deploy clojars
What problems are you facing?
Eh heh. Well it’s asking me to sign with gpg and I’m having problems with that. Avoding signing for now would be fine I guess but I can’t figure that out either
@alex-dixon deploy from where? boot, leiningen, other?
alex-dixon: my advice is to use boot, heh. seriously, sorry i can't help with lein.
It’s cool. Thanks. I actually tried with boot. I got some kind of error. What’s the command? Do I need a build.boot file?
alex-dixon: but fwiw, i did spend the time online figuring out how to set up gpg for boit, i imagine it's similar. annoying, but worth the effort.
Yikes. Ok. Thank you 🙂
yeah you would have to make a build.boot. then, see https://github.com/boot-clj/boot/wiki/Repository-Credentials-and-Deploying
Awesome. Thank you so much
But at this point I’ll take anything 🙂
I guess you'd need to setup GPG following the instructions given by your distro
@joelsanchez I’l look into httpkit. I don’t have to use jetty i guess that’s only the default i guess. The error I get is: java.lang.ClassNotFoundException: org.eclipse.jetty.http.HttpParser$ProxyHandler. datomic client forces an old version of jetty. If I try to use a newer one datomic client breaks and ring works:)
@nadejde You can solve that easily using an exclusion, you know your way around them?
I’m on Mac. I followed clojars-web’s instructions using brew install
. gpg: signing failed: Inappropriate ioctl for device
@U0LGCREMU OMFG worked. Ha-ha! Thank you!!
alex-dixon: i tried putting that in .bash_profile but i still have to do it by hand. no prob since i rarely deploy, but please let me know if you figger sth like that out. cheers!
Tried ever witch way. The problem is there’s not any version of jetty that works with both datomi client and ring (or have not been able to find it)
Damn I have 0 Mac knowledge, can't help, but maybe stackoverflow can - after all it's not clojure specific
if exclude at datomic client level datomic client breaks as it cant work with what ring brings in. If I exclude at ring level ring breaks as it can’t work with what datomic brings in…
@alex-dixon I’ve found that on a mac the only usable thing is to install the GUI client MacGPG
using the command line gpg from mac does not work with lein at all because lein doesn’t know how to give up the tty
and it would need to for the homebrew gpg to work
Yikes @nadejde looks tough, let me see if I can find something
Hey @nadejde look at this https://stackoverflow.com/questions/43291069/lein-ring-server-headless-fails-when-including-datomic-dependency
@alex-dixon there’s a workaround where you decode a random file using your credentials first so that the auth is in cache, but that’s just a hack and MacGPG fixes the issue
The "solution" they propose is to forget jetty and use http-kit
Maybe try that? After all you don't mind about jetty
@joelsanchez I will do just that!
@noisesmith Thanks. Trying that now. Why do we put up with this though? 😕
@joelsanchez Thank’s for you help:) Will try it with http-kit. Looked at their website should be really straight forward to setup
Rich Hickey talked about that last conj. He said spec
is an attempt to fix the problem
I still feel like it maybe has an easier solution?
Like maven please just use two different versions of jetty okay
@nadejde Yes, it's a breeze, also it's pretty damn fast (but remember to setup a reverse proxy like nginx in production, since httpkit is quite limited in some areas, for example I think it can't do https - not designed for that)
@captainlexington I think I saw that talk:) Good talk it was
@nadejde He convinced me. I'll never release breaking changes in a library! :fist:
@joelsanchez I’m only playing around for now. Trying to learn some clojure, ring , datomic etc. Not going into production anytime soon
@alex-dixon: try $ export GPG_TTY=$(tty)
. I had a similar problem with boot.
@captainlexington If only everyone could do that 🙂
using clojure's bindings to jdbc, can I get back the auto-incremented id from an execute!
specifically, I have an "INSERT ... ON DUPLICATE KEY UPDATE ..."
and it also has an auto inc'd key, can I get that back after the jdbc/execute!
?
Is there anyway to tell if a a thread created by a future has died?
@josh.freckleton if you're using insert!
you've got the following contract:
> When inserting a row as a map, the result is the database-specific form of the generated keys, if available (note: PostgreSQL returns the whole row).
oh sorry, i'm on mysql (wish it were postgres 🙂 )
can I achieve an "upsert" with jdbc/insert!
?
(IE "INSERT ... ON DUPLICATE KEY UPDATE ..."
or could I get "REPLACE"
to return an id?
the key is made out of 2 foreign keys, and our system has been throwing duplicate key errors
ya, in this case, I think it's just as easy to "upsert" it
the alternative was (try insert... (catch update...))
@josh.freckleton , I believe you can do this in mysql with INSERT INTO table SET unique1=value1 AND unique2=value2… ON DUPLICATE KEY UPDATE <anything you want to update>, id=LAST_INSERT_ID(id)
; the last_insert_id(<my unique id>)
should get it to behave such that mysql will return either the newly inserted id or the id of the row which you updated
I think; I’m far more familiar with postgres
and then jdbc/insert!
instead of jdbc/execute!
?
and, I'll test this, but do you know if that id update could reset the auto inc'd id?
I think you’d need to use execute!
, since it’s not a straight forward insert!
oh yep, just saw that in the signature of insert!
and I believe that mysql will always auto increment id’s when you do on duplicate key update
, if I remember mysql correctly?
so, IIUC, it increments the counter, so future inserts have a higher id, but shouldn't inc the current one I don't think
correct
(I do recommend you test this, like I said, I’m far more familiar with postgres than mysql)
hm, the execute just returns the "number of rows effected", i think
i wonder if anyone has ever implemented an ISeq that is memory mapped, so as it grows it moves from the jvm heap to the filesystem
i figure making something like that with standard IO wouldn't be hard, but page swapping is so much more efficient
dpsutton: 1.
Unhandled java.sql.SQLException
Can not issue data manipulation statements with executeQuery().
@josh.freckleton did you verify that the code I gave you actually returns the id (like through the mysql command line?)
I think if you get that to work right, then execute!
should work?
hm, from the mysql cli, it just returns "rows affected"
1=insert, 2=update, 0=update was the same as current row
my use case is a video game where i store a seq of all past game states-- the idea is that you never have to "save" the game because it stores its entire history
that would also make backtracking to a previous state easy, which enables a lot of cool designs for the gameplay
I'm trying to learn how to eventually extend Java classes. I'm having issues figuring out how to use gen-class in Cider. Here's my example:
(ns ex.Example
(:gen-class))
(defn -toString
[this]
"Hello, World!")
I'm using https://kotka.de/blog/2010/02/gen-class_how_it_works_and_how_to_use_it.html as a guide. I get a ClassNotFoundException when I try to instantiate it via (ex.Example.)
. I get "java.lang.ClassCastException: java.io.File cannot be cast to java.lang.String" when I try to compile it via (compile 'ex.Example)
. I'm probably missing something basic. Any pointers to better references most welcome!that compile exception sounds like you set whatever the compile path var is to a File object instead of a String
unless you really need gen-class (which you usually don't) I would avoid using it for anything
if you must, a good thing to read is https://clojure.org/reference/compilation because gen-class requires aot compilation
@hiredman thanks. My understanding is that if I need to extend a Java class that will be referenced by other Java classes, I need to use gen-class. In this case, I'm extending logback's AbstractDiscriminator.
I was just copying something out of the channel topic, and it seems I accidentally updated it as well? If I screwed it up, apologies.
Surprised that the channel-topic
var is not ^:private
grzm: yeah, that is likely the case, I would check out the aot compilation documentation for whatever build tool you use, and checkout that link
@joelsanchez You and me both!
@hiredman looks like calling (boot (aot :namespace '#{ex.Example}))
in my repl (using boot and Cider) has moved me forward at least one step.
the problem is, aot compilation generates class files on disk, and per classloader hierarchy (and you only get one without backflips) you can only load them once
so if you want to change the compiled ex.Example, and need to recompile, you will likely need to restart your repl
@hiredman Yup. Been working on that as well. From what I'm gathering, as long as I'm not changing the signature of the class, I can update the internals of the functions/methods without restarting the repl.
@hiredman: okay, so it's meant for stuff that are repl specific rather than project specific ?
(I don’t think user.clj
works with Boot, BTW?)
(because Boot starts Clojure up before it’s even read build.boot
so there’s no user.clj
on the classpath for Clojure to load — as I understand it)
@seancorfield You mean that user.clj won't be evaluated on start up, correct? The classpath used within the repl is going to be what's defined in build.boot, right?
@grzm Leiningen creates two JVMs: the second one is in the context of the project and will execute user.clj
as part of that. Boot only creates one JVM and it modifies the context once it has read build.boot
. So user.clj
won’t be loaded by Clojure with Boot, because Clojure starts before any user.clj
could be on the classpath.
(not sure whether that’s clearer or not?)
With Boot, you can modify the context after the REPL has started (so saying the “classpath used within the repl is going to be what’s defined in build.boot” is a bit squishy).
Yup, it is. I was getting at the "execute" part. user.clj can certainly be on the classpath as defined by :resource-paths in build.boot, right? You just don't get the felicity of having it already executed prior to the repl prompt appearing.
Right, the automatic behavior of Clojure (to load and run user.clj
) doesn’t happen with Boot, but you can load and run it yourself in the REPL.