Fork me on GitHub
#clojure
<
2018-04-16
>
Empperi05:04:39

hmm, this change in Transit CLJ seems to be causing a NPE for me https://github.com/cognitect/transit-clj/blame/5fcd8e421ef7cd2e6240ab39c693ec1b8a34b3d3/src/cognitect/transit.clj#L160 with following code that previously worked just fine:

(let [stream (ByteArrayOutputStream.)]
  (t/write (t/writer stream :json) {:wow "such doge"})
  (.toString stream (.toString StandardCharsets/UTF_8)))

Empperi05:04:44

anyone else seeing this?

Empperi05:04:27

version [com.cognitect/transit-clj "0.8.307"]

Empperi05:04:25

at java 9.0.4

Empperi05:04:30

(should upgrade)

Empperi06:04:10

previous version (0.8.303) does not get affected by this

Empperi06:04:16

guess it's a regression bug, I'll report it

Empperi06:04:55

right, there is an issue about this already https://github.com/cognitect/transit-clj/issues/41

Empperi06:04:17

@dnolen ping ^ - transit-clj broken

urzds09:04:18

Hello! Is there a tool to extract a list of environment variables that are being read from our code through calls to @weavejester's environ's env function? Or maybe even a similar library that allows to specify a documentation string and default values and generates --help output and possibly *roff / mandoc / CommonMark / MarkDown?

devurandom08:04:01

Thanks, @U051TMSBY! That perfectly fits our requirements! 🙂

xtreak2911:04:05

I am trying to list the doc string with spec for all the functions in a namespace using (map doc (keys (ns-publics 'foo.core))) . It fails with CompilerException java.lang.RuntimeException: Can't take value of a macro: #'clojure.repl/doc . Any idea how to list the docstring of all the public functions ?

pesterhazy11:04:26

try something like (doseq [x xs] (doc x))

pesterhazy11:04:40

although not sure that actually works... better to look at the implementation of doc (`source doc`) and do what it does

xtreak2911:04:41

Thanks. What is the recommended tool to generate docs like http://clojure.org docs ?

xtreak2911:04:10

I tried codox but I don't get the specs generated in the HTML

ajs12:04:27

I'm having quite some difficulty figuring out how to combine java.util.zip.GZIPInputStream with java.io.PushbackReader. so I can read compressed edn right into a clojure data structure. I'm trying various variations of this:

(defn from-zip-file
  [file]
  (with-open [in (java.util.zip.GZIPInputStream.
                  (java.io.PushbackReader.
                   (
                    file)))]
    (clojure.edn/read in)))

ajs12:04:13

reading directly from (java.io.PushbackReader. (io/reader file)) on non-zipped edn is fine, but I have some gzipped stuff

Alex Miller (Clojure team)12:04:20

In Java, Readers are for character data and Streams are for binary data so generally something like the above won’t work.

ajs12:04:37

I'm having a heck of time since I don't really have the Java background

ajs12:04:04

I know it's something simple but i've been looking at countless examples and none quite do this basic thing

Alex Miller (Clojure team)12:04:09

Usually you want to wrap the reader around the stream

ajs12:04:46

isn't my pushbackreader wrapping the input-stream? or do you mean the gzipinputstream? I tried swapping the placement of the gzipinputstream and the pushbackreader also

Alex Miller (Clojure team)12:04:45

I think you definitely want a reader on the outside

Alex Miller (Clojure team)12:04:26

But I haven’t used the gzip stuff enough to know exactly what you need there

ajs12:04:30

how would I cast/move data from an input stream into a reader? I've tried this also:

(java.io.PushbackReader.
                  (java.util.zip.GZIPInputStream.
                   (
                    file)))

ajs12:04:12

I could throw away the gzip stuff entirely if there is a more clojure-native way of compressing edn. I have half a gig of edn and reading and writing is fine but storage is a problem.

ajs12:04:43

The gzipped write is a 90% reduction in the size of the edn on disk, but I just can't read it back in

Alex Miller (Clojure team)12:04:52

I think you’re on the right track

ajs12:04:42

i found this, lol, the comment >Boy, I'm absolutely in love with Java, but this question comes up so often you'd think they'd just figure out that the chaining of streams is somewhat difficult and either make helpers to create various combinations or rethink the whole thing. https://stackoverflow.com/questions/309424/read-convert-an-inputstream-to-a-string

ajs12:04:37

is UTF_8 the proper string encoding to read in edn?

ajs12:04:12

looks like this doing it:

(defn gzipped-input-stream->str [input-stream ]
  (with-open [out (StringWriter.)]
    (IOUtils/copy input-stream out Charsets/UTF_8)
    (.toString out)))

(defn from-zip-file
  [file]
  (with-open [in (java.util.zip.GZIPInputStream.
                  (
                   file))]
    (clojure.edn/read-string (gzipped-input-stream->str in))))

ajs12:04:43

reminds me of C++

roklenarcic13:04:03

StandardCharsets/UTF_8 is the standard way

roklenarcic13:04:30

Charsets is probably some kind of apache commons class probably, or guava

roklenarcic13:04:38

StandardCharsets is pure java

roklenarcic13:04:09

the typical way to resolve byte stream into a charset is to convert input stream to a reader

roklenarcic13:04:01

new InputStreamReader(in, StandardCharsets.UTF_8)

roklenarcic13:04:08

and then use a stream reading variant of the function which is clojure.edn/read

ajs13:04:11

@roklenarcic thanks trying this now. Having trouble resolving StandardCharsets. I'm importing [java.nio.charset.StandardCharsets] and then using java.nio.charset.StandardCharsets.UTF_8 or also trying StandardCharsets/UTF_8 but it is a class not found

roklenarcic13:04:06

that's core java api

roklenarcic13:04:22

java.nio.charset.StandardCharsets/UTF_8

roklenarcic13:04:38

of course you should move package into import statement

Alex Miller (Clojure team)13:04:01

I think you want (:import [java.nio.charset StandardCharsets]) and StandardCharsets/UTF_8

Alex Miller (Clojure team)13:04:21

note the missing . in the first one

ajs13:04:33

got it now, was using wrong syntax java.nio.charset.StandardCharsets.UTF_8 my java interop is terrible thanks to no java background

ajs13:04:07

last dot should have been a slash

ajs13:04:25

@roklenarcic this is my interpretation of your suggestion

(defn from-zip-file2
  [file]
  (with-open [in (java.util.zip.GZIPInputStream.
                  (
                   file))]
    (clojure.edn/read (java.io.InputStreamReader. in java.nio.charset.StandardCharsets/UTF_8))))

roklenarcic13:04:25

right.. I wrote a lot of java, because I assume everyone knows java I guess

roklenarcic13:04:41

at least it should

ajs13:04:52

i'm getting same error as many other variations: java.io.InputStreamReader cannot be cast to java.io.PushbackReader

roklenarcic13:04:10

right... edn expects a pushback readers

roklenarcic13:04:47

(java.io.PushbackReader. (java.io.InputStreamReader. .... ))

roklenarcic13:04:15

you should add class imports

ajs13:04:05

you mean to avoid the full java.io.....

roklenarcic13:04:46

here's an example from a project

(ns myns
  (:import ( PushbackReader InputStreamReader)
           (java.util.zip GZIPInputStream)
           (java.nio.charset StandardCharsets)))

roklenarcic13:04:10

then you can just use the class names

ajs13:04:33

it appears my original implementation using read-string and the IOUtils/copy is about 20% faster than this new version

ghadi13:04:53

gzip is slow

ajs14:04:02

they are both gzip versions

ajs14:04:16

compare, the second one is slower for me:

(defn gzipped-input-stream->str [input-stream]
  (with-open [out (java.io.StringWriter.)]
    (IOUtils/copy input-stream out Charsets/UTF_8)
    (.toString out)))

(defn from-zip-file
  [file]
  (with-open [in (java.util.zip.GZIPInputStream.
                  (
                   file))]
    (clojure.edn/read-string (gzipped-input-stream->str in))))

;; --------- alternate read technique ---------

(defn from-zip-file2
  [file]
  (with-open [in (java.util.zip.GZIPInputStream.
                  (
                   file))]
    (clojure.edn/read (java.io.PushbackReader. (java.io.InputStreamReader. in java.nio.charset.StandardCharsets/UTF_8)))))

roklenarcic14:04:43

yes the difference is

roklenarcic14:04:54

that first one loads whole string first

ajs14:04:58

just timing them like (time (dotimes [x 10] (from-zip-file "somefile.gz")))

ajs14:04:18

so for very very large files, the second one should be better, to avoid memory problems

roklenarcic14:04:41

also, read-string will read first object

roklenarcic14:04:47

read will read next object in stream

roklenarcic14:04:16

which means you can call read multiple times and read multiple objects

roklenarcic14:04:54

provided you do it inside with-open of course

roklenarcic14:04:08

if you leave that macro it will close the stream

roklenarcic14:04:35

same goes if you return a lazy sequence

ajs14:04:07

if I'm reading a single form that is gigantic, should the second one be better for memory use?

Bravi14:04:56

hi everyone. is there a code review -like channel here to get some suggestions / recommendations of what I should / should not do in clj?

ghadi14:04:13

#code-reviews

Bravi14:04:17

ah nice 😄

Denis G18:04:33

Maybe it’s an off-topic question, but it’s still written in clojure and developed for clojure as well, so the question basically is, why did LightTable die?

borkdude18:04:40

@denisgrebennicov I think that’s a good question for the #off-topic channel

😅 4
👍 4
ghadi18:04:08

A lot of interactive eval features live on in CIDER and SPIRAL

idiomancy19:04:02

how do I change the value of a single property of a mutable object? e.g.:`velocity.y = -1;` using set! doesn't work here the way I want it to, because y is just an int. so (-> velocity .y (set! -1) isn't doing the trick, because at that point it's just trying to mutate an integer

Sallide19:04:17

From the docs, looks like this is the answer

Sallide19:04:20

(set! (. instance-expr instanceFieldName-symbol) expr) (set! (. Classname-symbol staticFieldName-symbol) expr)

Sallide19:04:45

I just tried doing (def p (java.awt.Point.)) (set! (. p x) 4) which seems to have worked

idiomancy23:04:25

Sorry I missed your response. Thanks... I guess this is an artifact of the CLR stuff I'm doing. :thinking_face:

leontalbot19:04:27

Hey guys, can’t find the answer online: I have this:

(with-open [in (io/input-stream uri)
            out (io/output-stream file)]
    (io/copy in out))
this all works except if the uri is redirected, in my case it is from http://
 to https://....

leontalbot19:04:10

Is there a way to first let the redirection happen (if any) and then “slurp” the data with io/input-stream

ghadi19:04:12

slurp doesn't handle any of that stuff, because the underlying default java URL readers don't

ghadi19:04:29

you're better off pulling in a more robust http client

leontalbot19:04:50

@ghadi Thanks, where should I look for code example of the equivalent to io/input-stream with http-kit?

ghadi19:04:32

dunno about http-kit, but with clj-http you can do (http/get ... {:as :stream}) and get the :body as an InputStream (don't forget to close it)

leontalbot19:04:19

@ghadi And how would I close it? (sorry for rocky question here) Much thanks

leontalbot20:04:17

(defn copy-file
  [uri file]
  (with-open [in (-> uri (client/get {:as :stream}) :body)
              out (io/output-stream file)]
    (io/copy in out)))

ghadi20:04:25

wrap your usage with with-open

ghadi20:04:30

basically

lxsameer20:04:19

hey guys, I'm looking for a stream implementation beside manifold.streams, any suggestion ?

hiredman20:04:23

not sure what that question means, manifold.stream is an abstraction, not a concrete implementation. the manifold library comes with a number of implementations of that abstraction as a bunch of namespaces like manifold.stream.async

lwhorton20:04:33

the reason I ask is my build-tool environment can’t seem to find clj if it’s installed as a lein dep 😕

hiredman20:04:36

assuming you are asking about using lein, :dependencies doesn't "install" clojure

lwhorton20:04:04

yes that’s what i’m asking. how can i go get clojure 1.9, for example, from a build tool where I don’t have scripting if something like lein doesn’t do an install for me?

dpsutton20:04:50

the clj command line tool is entirely different beast from any lein dependency. https://clojure.org/guides/getting_started

hiredman20:04:14

it is going to depend on your os, there are different packagers and installers people have created for install the clj tool

dpsutton20:04:54

has windows been finished? i know @alexmiller has been focusing on that recently

Alex Miller (Clojure team)20:04:14

not finished yet but getting close

👍 4
lwhorton20:04:27

cool, thanks for clearing that up for me

👍 4
hiredman20:04:54

if you are using lein fine with 1.8, I would suggest continuing to do so with 1.9. the clj tool has basically spawned a new ecosystem of tooling which is all brand new

lwhorton20:04:31

maybe i should just use lein run or whatever it is instead of clj my-file -m ...

bfabry22:04:36

I have java projects where gradle is the build tool and the dependencies are specified in gradle files. what would be the simplest way for me to spin up repls in those projects (with the same dependencies and the local source trees available) for dev time exploration?

hiredman22:04:14

clojure has the ability to spin up a socket repl when the clojure runtime is initialize

hiredman22:04:13

so if you pass -Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}" and reference clojure.lang.RT somewhere that causes the class to be loaded, you'll get a socket repl

bfabry22:04:58

that's right one of the main options is a socket repl. I should be able to work with that, thanks

idiomancy23:04:03

so... what is the interop equivelant of bracket notation, i.e. anim["walk"].speed = 2.0f;

hiredman23:04:30

for clojurescript?

hiredman23:04:28

I guess it wouldn't be, since 2.0f doesn't make sense in javascript, but anim["walk"] doesn't make sense in java, but I guess lots of new syntax has landed in java recently

idiomancy23:04:41

ahh, actually its C#. So I'm totally asking in the wrong place 😆

idiomancy23:04:11

I forgot that syntax is just straight up not in Java

alice23:04:47

Can anyone in here help?