Fork me on GitHub
#clojure
<
2018-10-05
>
kingcode00:10:41

I am trying to fetch some HTML at the REPL over the network, but keep getting 504s for a reasonably sized file, which comes up fast in Chrome. I am using enlive-html/html-resource to fetch - has anyone come across this issue? Thanks in advance…

cgrand05:10:07

@kingcode html-resource use the default java http client which is blocked by some sites. Try using clj-http and pass the response body to enlive.

rymndhng05:10:31

anyone know of a clojure library that can take a file and return string contents that allow me to highlight a particular line, to have a similar effect to linter squiggly underlines?

seancorfield05:10:07

@rymndhng Do you just want to get a sequence of lines of text from a file?

rymndhng05:10:02

not quite, I wondered if there's something a bit more, i.e given a file + {:line 25 :col-start 10 :col-end 35 :context 5}, it'd be able to extract the 5 lines before/after line, and apply some magic highlighting from column 10->35, i.e underlining or ansi escape code definitely something I could whip up, just wondering if someones aware of this existing out there!

andy.fingerhut05:10:58

Like the squiggly-clojure Emacs package can visually display such things, you mean?

andy.fingerhut05:10:03

If you don't mind the display being inside an Emacs window, that might do what you want. If you want it in ansi escape codes on a terminal window, I don't think that particular thing will help you.

rymndhng05:10:06

yes, kind of like squiggly-clojure there's two parts to the thing I'm trying to do: 1. figuring out the best data structure to represent this (which is where I'm hoping to lean on an existing solution) 2. rendering it -- I'm planning to spit it to a terminal, it's mostly to help show errors in a json/yaml file in a console output

andy.fingerhut05:10:44

For the example you gave, I can't think of a better data structure than the Clojure map you showed. For rendering, I'm not familiar with the internals of the pretty library, but it might have something you could reuse: https://github.com/AvisoNovate/pretty

rymndhng05:10:27

totally, that came to mind too 😄 thanks

andy.fingerhut05:10:49

Google search for terms: clojure ansi terminal gives a few hits, but nothing I have used, and not necessarily anything that makes your life easier.

vandr0iy07:10:32

Hi clojurians! How would one call a java bit shifting operator from clojure? I'm perfectly aware of the existence of functions like bit-shift-left and such in the core namespace, but they do some implicit type conversion, and I don't want that. Any ideas?

val_waeselynck08:10:56

You could define a static method with a primitive signature in a java class and call it via interop (if frequently called you can trust the JIT to inline it, so don't worry about the method calling overhead). You will probably still need to pass boxed values across clojure functions, there's no way to avoid that. Still, I would consider using the clojure standard libraries functions: they are inlined too, so I wouldn't be surprised if you got no type conversion at all in the end - if you call them with primitive arguments, you will directly call the primitive static methods in clojure.lang.Numbers

schmee08:10:02

those operators don’t do type conversion if you use primitive type hints

schmee08:10:35

in fact, they use the same specialized JVM bytecodes that Java uses

schmee08:10:48

here’s the function and the bytecode:

(defn bsl ^long [^long a ^long b]
  (bit-shift-left a b))
->
public static long invokeStatic(long a, long arg1);
    0  lload_0 [a]
    1  lload_2 [arg1]
    2  l2i
    3  lshl
    4  lreturn

vandr0iy08:10:45

(defn bsl ^Byte [^Byte a ^Byte b]
    (bit-shift-left a b))
=> #'asn1.core/bsl
(type (bsl 127 4))
=> java.lang.Long
@U3L6TFEJF nah

vandr0iy08:10:47

@U06GS6P1N that's what I ended up doing

schmee08:10:58

well, you have two errors: 1. the primitive type hint is ^byte, not ^Byte 2. the REPL can’t handle primitive values so they will be boxed before printing. if you use primitive type hint in a let or a loop they will not get boxed

schmee08:10:54

but either way, I agree with doing this in Java

schmee08:10:22

there are so many things that can go wrong when trying to use primitive bytes without boxing in Clojure

schmee08:10:33

ahh, just recalled that there is no primitive byte type hint…

vandr0iy08:10:34

^byte is not supported, though:

(defn bsl ^byte [^byte a ^byte b]
    (bit-shift-left a b))
CompilerException java.lang.IllegalArgumentException: Only long and double primitives are supported, compiling 

schmee08:10:48

yeah, all these things only work for long and double unfortunately

andy.fingerhut08:10:14

I don't know whether this lib can do what you want, but worth checking: https://github.com/ztellman/primitive-math

carocad11:10:04

hey guys, I am trying to create a zip file with several csv files inside. I am using clojure.data.csv and the function correctly writes the zipfile but most files inside are empty and the ones that have content are just full of gibberish. Does anyone have any idea what could be happening? 😕

misha11:10:42

I can't use :keys destructuring inside let inside my macro, can I?

(defmacro foo [m]
  `(let [{:keys [foo bar]} ~m]
    ,,,))
I don't see how even clojure.core/destructure would help :(

tristefigure11:10:40

This works fine like this. You can't use gensyms (`bar#`) with the :keys syntax (since that would require the key in the map to match that gensym), so you have to use a raw symbol (`~'bar`) in order to avoid it to be expanded into my-namespace/bar.

lloydshark11:10:58

@carocad Might still be an issue with how you are writing to the zip. Or perhaps some lazy sequences not being resolved. But I'm guessing - can you show some code? Or otherwise I would get in a REPL and try out some of the csv functions and see whether you are getting the right output at that level before the zipping part happens.

tristefigure11:10:32

Or you can use the symbol :keyword map-destructuring syntax. E.g

(defmacro foo [m]
  `(let [{bar# :bar} ~m]
    (inc bar#)))
@misha

carocad11:10:34

@lloydshark here is the code that I am using … I have already tried realizing all lazy sequences but so far no success 😕.

carocad11:10:47

(defn dump!
  ^ZipOutputStream
  [network outstream]
  (with-open [zipstream (ZipOutputStream. outstream)]
    (doseq [feed (:feeds reference/gtfs-spec)
            :when (some :unique (:fields feed))]
      (println "writing" (:filename feed))
      (let [filename  (:filename feed)
            zip-entry  (ZipEntry. ^String filename)
            header     (map :field-name feed)
            field-id   (first (filter :unique (:fields feed)))
            id         (reference/get-mapping filename (:field-name field-id))
            attributes (eduction (map :field-name)
                                 (map #(reference/get-mapping filename %))
                                 (map :keyword)
                                 (:fields feed))]
        (. zipstream (putNextEntry zip-entry))
        (csv/write-csv (io/writer zipstream)
          (cons header
            (for [d (data/datoms network :avet (:keyword id))
                  :let [entity (data/entity network (:e d))]]
              (for [attr attributes]
                (get entity attr)))))
        (.flush zipstream)
        (. zipstream (closeEntry))))))

(dump! @firstlast @(:networks (:router hiposfer.kamal.dev/system)))
        (io/output-stream "resources/test/foo.zip"))

lloydshark11:10:19

Hmmm... there's a bit going on there 🙂

carocad11:10:32

😅 sorry for that

lloydshark11:10:42

Ha, that's cool.

carocad11:10:31

the gist of it is: create a series of file, and for each file I extract the content from a datascript structure, then write it to the csv file

lloydshark11:10:49

For me I would probably change the implementation to take a (lazy) sequence to produce the zip.

lloydshark11:10:01

ie Then you could pass in fixed data - and see it work.

lloydshark11:10:27

Then separately create the function that generates your list of csv's.

lloydshark11:10:42

That way you should be able to test the 2 things separately.

lloydshark11:10:51

And then join them together.

carocad11:10:11

jummm that is a good point. I will try that. I was doing it like this since I am returning the zipstream as a response in a ring server

lloydshark11:10:17

Sorry that's not really an immediate fix to your problem.

carocad11:10:14

hey @lloydshark I followed your advice and ended up with this:

(with-open [outstream (io/output-stream "resources/test/foo.zip")
            zipstream (ZipOutputStream. outstream)]
  (doseq [[filename content] [["foo.txt" (repeat 5 (range 10))]
                              ["bar.txt" (repeat 10 (repeat 5 "hello"))]]
          :let [zip-entry  (ZipEntry. ^String filename)]]
    (println "writing" filename)
    (. zipstream (putNextEntry zip-entry))
    (csv/write-csv (io/writer zipstream) content)
    (.flush zipstream)
    (.closeEntry zipstream))
  (.finish zipstream))

carocad11:10:38

that reproduces my problem … the content is just not writen to each file 😕

lloydshark12:10:04

Just curious - do you need the (.flush zipstream) ?

lloydshark12:10:18

I just can't remember having to do a raw flush when I wrote to a zip.

carocad12:10:30

no idea 😅 I am kind of desperate now so I started trying whatever I found in internet

carocad12:10:44

to the best of my knowledge I dont need it but again … desperate 😞

lloydshark12:10:26

ok, so I just looked at some previous zipping code of mine.

lloydshark12:10:52

And it did a setSize on the entry.

lloydshark12:10:22

Before the putNextEntry

lloydshark12:10:37

Not saying that's the issue - but its at least a difference.

carocad12:10:01

oh I got it !!!

carocad12:10:21

I had to flush the writer not the stream

carocad12:10:07

this does the trick

(with-open [outstream (io/output-stream "resources/test/foo.zip")
            zipstream (ZipOutputStream. outstream)]
  (doseq [[filename content] [["foo.txt" (repeat 5 (range 10))]
                              ["bar.txt" (repeat 10 (repeat 5 "hello"))]]
          :let [zip-entry (ZipEntry. ^String filename)
                writer    (io/writer zipstream)]]
    (println "writing" filename)
    (.putNextEntry zipstream zip-entry)
    (csv/write-csv writer content)
    (.flush writer)))

carocad12:10:15

just in case you run into the same issue 😉

lloydshark12:10:24

Ah so you are getting content now.

orestis11:10:04

I see that it’s common for uberjar-generating libs to generate a -standalone flavor, that contains everything and a, well, non-standalone flavor. Presumably you have to provide some of those dependencies manually, right? But how?

orestis11:10:49

I tried java -cp ~/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:~/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar -jar _build/october-1.0.0-SNAPSHOT.jar, but I’m getting

Exception in thread "main" java.lang.NoClassDefFoundError: clojure/lang/Var
	at nosco.main.<clinit>(Unknown Source)

misha11:10:31

@tristefigure true, thanks. but all I wanted is to save me some (:foo ~m), and {foo# :foo} ~m kinda the same amount of code, so I'll just go with (:foo ~m) opieop

misha11:10:50

good to know, thanks. but not gonna pull whole dep to save me 3 (:foo ~m) in one throw away macro : )

misha11:10:57

and {:keys [~'bar]} ~m would be too cryptical for colleagues (and me in, like, 3 weeks from now)

deliciousowl14:10:25

wait what does this macro do?

JH16:10:06

Does anybody have any recommendations for how to open a pdf returned by a get call in a new window? That caviot is that we need to send a *tolkien in the request header*. Libraries being used [cljs-http "0.1.45"] Without the header, this is done easily with just (.open js/window "uri-to-pdf), integrating the header is the tricky part.

dpsutton16:10:10

what is ILookupThunk?

ghadi16:10:47

it's part of an inline cache that optimizes expressions that look like this: (:k map)

ghadi16:10:22

It optimizes for the potential case that you're accessing a record that has a declared field named k

dpsutton16:10:07

so without this defrecord's fields would not be much faster?

ghadi16:10:48

yeah, when accessed like so (:key record)

dpsutton16:10:51

(or a custom type similar to defrecord)

dpsutton16:10:57

thanks so much

ghadi16:10:09

when accessed (.-key record) that's a direct field access, but you can't write generic code like that

markw17:10:16

Question: Is there any way to determine if a symbol is reserved? E.G. I want to postwalk a form, and convert all symbols which are NOT used by the language (macros, functions, special forms) to keywords. I am aware of special-symbol? but that only covers a small subset of what I need.

markw17:10:48

for example I wouldn't want a let inside a form to be keywordized

roti17:10:06

@markw what do you consider a reserved symbol?

markw17:10:30

Yes I know that is vague - basically anything that is a special-form (covered by special-symbol?), a function, or a macro

seancorfield17:10:04

@markw You can redefine almost everything in Clojure. It has very few special forms https://clojure.org/reference/special_forms

markw17:10:22

I'm not sure how that's related to the question

seancorfield17:10:48

I'm saying there are almost no "reserved" symbols.

markw17:10:09

I admit my choice nomenclature might not be appropriate

markw17:10:46

basically I want to walk a form and convert any symbol that isn't defined by the language to a keyword

markw17:10:57

it's hard to explain with correct vocabulary so maybe an example is best

seancorfield17:10:22

"any symbol that isn't defined by the language" <-- but that is an extremely small set of symbols.

markw17:10:31

ok - or anything in core then

markw17:10:42

but that isn't truly what i'm after because there is stuff beyond core

noisesmith17:10:43

you can use resolve to check if something has a binding, and find out where it comes from

markw17:10:15

yeah i thought about some combination of special-symbol, resolve, and bound?

noisesmith17:10:20

sounds like the hard part here is deciding the scope of what you consider "built in" for your purposes

markw17:10:30

yes agreed

seancorfield17:10:24

There are a lot of namespaces bundled with Clojure. And would you consider the Contrib namespaces to be "used by the language" or not?

seancorfield17:10:37

(`clojure.core.cache` etc)

markw17:10:09

OK I thought about this a bit more... I framed the question poorly

markw17:10:40

I think really what I'm after is to convert anything within an arbitrary form which does not already have a binding outside of the form, to a keyword

markw17:10:07

so resolve may be a solution

seancorfield17:10:30

You'll still need to consider the special forms I linked to above: if, def, let ... those resolve to nil.

seancorfield17:10:38

(but at least that's a short list)

noisesmith17:10:17

user=> (resolve 'let)
#'clojure.core/let

noisesmith17:10:32

now good luck using the value it returns - but it does return something

seancorfield17:10:14

Hah, that'll teach me to type more items in a list than I actually tested in the REPL!

noisesmith17:10:16

the union of special-symbol? plus resolve plus some namespace whitelisting magic should suffice

noisesmith17:10:06

@seancorfield the reason I checked was the earlier convo revealed let was not considered a special-symbol? which got me curious...

seancorfield17:10:56

I'd tested if and def and just foolishly assumed let since it was also listed on the special forms page...

noisesmith17:10:23

I'm seeing a pattern

(ins)user=> (special-symbol? 'let*)
true
(cmd)user=> (resolve 'let*)
nil
(cmd)user=> (special-symbol? 'let*)
true
(ins)user=> (resolve 'fn*)
nil
(ins)user=> (special-symbol? 'fn*)
true
(ins)user=> (resolve 'fn)
#'clojure.core/fn
(ins)user=> (special-symbol? 'fn)
false

markw17:10:41

Hmm i hadn't considered that symbols like let* would not resolve

markw17:10:14

hopefully the combination of special-symbol? and resolve as @noisesmith suggests should get it done

markw17:10:37

I don't need to use the return values, just know that they are truthy

noisesmith18:10:53

@markw this may or may not also be useful

(ins)user=> (defmacro local-binds [] (vec (map (comp keyword name) (keys &env))))
#'user/local-binds
(cmd)user=> (let [x 0] ((fn [v] (local-binds)) 42))
[:x :v]

noisesmith18:10:17

there's variations of that macro that can also give you the values of the locals but it didn't sound like you need it

markw18:10:20

yeah don't need the values, just that they exist

markw18:10:35

i did play around with &env actually

tristefigure18:10:32

I'm currently working on a version of postwalk that keeps track of the local variables within a form. That might help you: https://github.com/TristeFigure/dance/blob/master/src/dance/choreography.clj#L277 doc: https://github.com/TristeFigure/dance/blob/master/src/dance/core.clj#L211

andy.fingerhut18:10:18

Depending on your purposes, you may end up at tools.analyzer 🙂

tristefigure06:10:52

aha true. I tried using tools.analyzer but it looked daunting. If I recall correctly, there is a plan to provide a simpler interface to tools.analyzer, maybe I could share my experience ?

the2bears18:10:00

I sense a corollary to Greenspun's 10th rule coming up 🙂

Renata Soares18:10:06

I need suggestions of libraries (or others solutions) to implement a soap server in a clojure project. The project is long-term and i saw some libraries that are no longer maintained. So anyone uses one of them? Which one is recommended?

seancorfield19:10:48

@renata.santos Having gone down that path several times over the years, I recommend just using the Apache Axis Java libraries directly via interop.

seancorfield19:10:03

(I don't think Clojurians use SOAP enough to have produced any effective stable solutions)

dpsutton19:10:07

we are using defrecord and implementing nth on this record to destructure the arguments. But count and nth won't necessarily agree. Ie, count could return an arbitrarily large number but nth is only established for a subset of the record fields. Will clojure.lang.RT assume things based on count or are these largely independent?

Alex Miller (Clojure team)19:10:51

are you implementing Indexed?

Alex Miller (Clojure team)19:10:57

if so, then I think it will work

Alex Miller (Clojure team)19:10:30

(but in general this is weird and is a case where deftype is probably the ideal choice, although it’s a lot more effort)

dpsutton19:10:23

yes Indexed

dpsutton19:10:54

yes we are in prototype mode right now. So don't want to do a full deftype before we know which way to go. But worried count and nth might blow up somewhere in the runtime

Alex Miller (Clojure team)19:10:58

I can’t think of any place it would be an issue

dpsutton20:10:02

thanks alex. that was our suspicion. just a little worried that somewhere it might

lgillette20:10:01

Can someone give me/point me to a code snippet example for connecting to a Redshift cluster with JDBC? What deps do I need in project.clj and what connection parameters to I need in and db connection map?

noisesmith20:10:31

there's a jdbc driver, hopefully clojure.java.jdbc is able to use it (maybe @seancorfield knows?) https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html

noisesmith20:10:56

there is an example in the README.md https://github.com/clojure/java.jdbc

;; if the dbtype is not known to clojure.java.jdbc, or you want to override the
;; default choice of JDBC driver class name, you can provide :classname and the
;; name of the class to use:

(def redshift42 {:dbtype "redshift"
                 :dbname "myredstore"
                 :classname "com.amazon.redshift.jdbc42.Driver"
                 ...})

noisesmith20:10:42

you would need to add the redshift jdbc connector to your project.clj as a dep

lgillette21:10:57

Thanks noisesmith...sorry should have mentioned that I used these references as well, in various permutations of configs...but the connections just hang, finally ended in "ConnectException Connection timed out sun.nio.ch.Net.connect0 (Net.java:-2)". I was hoping for a rather full code example specific to Redshift that someone actually uses (with private db credentials redacted of course), because all the ones online I've found - including these docs - are only partial code examples for Redshift.

noisesmith21:10:32

have you verified that your creds work from the host you are trying to use redshift from?

lgillette21:10:20

Yeah, all my other sql client tools connect just fine, so I know the creds are good.

lgillette21:10:10

In project.clj :dependencies: [org.clojure/java.jdbc "0.7.8"] [com.amazon.redshift/redshift-jdbc42-no-awssdk "1.2.12.1017"] In my core.clj" (ns foo (:require [clojure.java.jdbc :as jdbc])) (def redshift {:dbtype "redshift" :host "..." :port 5432 :dbname "..." :user "..." :password "..." :ssl true :sslfactory "org.postgresql.ssl.NonValidatingFactory" :classname "com.amazon.redshift.jdbc42.Driver" }) (jdbc/query redshift "select 42") ;; => ConnectException Connection timed out sun.nio.ch.Net.connect0 (Net.java:-2)