Fork me on GitHub
#clojure
<
2017-06-16
>
bradford00:06:10

I can't figure out which fn is throwing it 😉

hiredman00:06:29

that is a macro expansion error

hiredman00:06:37

not a function call error

bfabry00:06:50

compiler errors can be hard

hiredman00:06:50

you have a defn somewhere that is ill formed

hiredman00:06:23

{:keys [name content mime-type charset transfer-encoding], :or {:transfer-encoding :quoted-printable}}

hiredman00:06:31

that is the arg list that is ill formed

hiredman00:06:55

should be {transfer-encoding :quoted-printable}

hiredman00:06:26

I think, I doubt really use :or so I am not overly familiar with the syntax off hand

bfabry00:06:42

indeed, and here's the error you would have got pre-spec:

(let [{:keys [foo] :or {foo :bar}} {}] foo)
=> :bar
(let [{:keys [foo] :or {:foo :bar}} {}] foo)
=> nil

bfabry00:06:45

silent failure

bradford00:06:37

Weird, that defn definitely isn't anywhere in my code. I wonder if a dependency got out of date somewhere...

bradford00:06:57

that line of code is from aleph

bradford00:06:11

but it was working yesterday....

hiredman00:06:27

my guess is you are getting a different aleph version or a different clojure version than you were yesterday

bradford00:06:49

Welp, bumped it and it all worked 🙂

bradford00:06:45

Unrelated question: Is there anything 'wrong' with creating atoms that contain a maps of core.async chans? I'm seeing some weird kind of thread starvation in my code, and that's the only area that seems problematic.

hiredman00:06:06

my guess is you are doing things with channels using swap!, which of course will re -run the function if it needs to, so doing side effecting things, like channel manipulation, is bad

bradford00:06:23

Yup! that's what I'm ... oh god

bradford00:06:31

But it seems innocuous? I have a few:

(swap! proxy-session-ip (fn [v x y]
                                                  (assoc v x y)) session-id (afghan.service/find-ip country-id))
and
(do
      (info "Creating recieve channel: " reqid)
      (swap! service/proxy-recieve-pool (fn [v r]
                                          (assoc v r (async/chan 1))) reqid)
      (service/add-work-item {:requestid reqid
                              :request   {:headers heads
                                          :request clean-req}} forward)

      (info "Getting channel from recieve pool: " reqid)
      (let [c (get @service/proxy-recieve-pool reqid)]
        (info "Getting obj from recieve channel: " reqid)
        (let [obj (async/<!! c)]
          (do
            (info "Closing recieve channel: " reqid)
            (async/close! (get @service/proxy-recieve-pool reqid))
            (info "Swapping out empty recieve channel: " reqid)
            (swap! service/proxy-recieve-pool (fn [v r]
                                                (dissoc v r)) reqid)

            obj))))

bradford00:06:18

Oh boy. I have the feeling I should be using refs

qqq00:06:11

is there a pattern for this: I want a "global registry" which is a dynamic map of :keywords to objects this map is modified as the program runs (things register and unregistering) is there a "clojure pattern" for doing this?

hiredman00:06:33

you can use an atom (around an immutable map), or a ConcurrentHashMap, but globals are not better an idea in clojure than in any other language, so I would recommend against them

qqq00:06:55

my other option is to pass, to every functiko ninvolved a field called "global-registry"

qqq00:06:01

is that a better way t do so ?

qqq00:06:15

by "global-registry" I really mean something like "directory for looking up other objects/services"

qqq00:06:08

it'd kinda weird; if we look at humans, we can refer to each other by name but by modules, we'd ahve to (1) sort people by when they were born, and (2) older people can refer to younger people but younger people could not refer to other people // <-- this would be our communication pattern if we did not have a "name -> human" mapping

hiredman00:06:12

well, if you want to use that analogy, the name -> human mapping is local to each person, and populated by another person introducing a human with a name

qqq01:06:37

well, there's facebook

bradford04:06:02

What could cause a core.async put! to succeed instantly, but a <!! to block for 40+ seconds, even if the buffer is empty? (Yeah, it's weird) There's also a async/alts! [event-channel (async/timeout), but that seems fine to me, I thought that wouldn't be blocked by the timeout.

joshjones13:06:48

bradford: so you're: (put! ch 42) and then you're (<!! ch) and this is blocking? @U085HE1PB

tomaas10:06:21

Hi, using clj-http I make a simple get request. It request returns xml content in its body. If I make that request with postman plugin I get the xml, however clj-http response returns body="" with content-length being correct (the same as with postman)

gaverhae11:06:27

tomaas: I suspect you might have a stream problem

gaverhae11:06:45

By default the Ring spec says body is an InputStream

gaverhae11:06:57

most InputStreams can only be read once

gaverhae11:06:17

so you might have some middleware or something reading the stream before the point you're looking at it

tomaas12:06:01

@U061HGP8C, thank you, I will check this

tomaas12:06:14

has to be what you say

tjtolton11:06:28

what's the lispy variable name for a transform function? xf?

curlyfry11:06:02

@tjtolton In what context?

tjtolton11:06:45

I mean, if I have to name the variable something that's appropriate for context its not a very lispy variable name 😂

tjtolton11:06:00

its like naming a string "s" its just a placeholder that preserves the shape of the code and gets the implementer out of the variable name discussion

tjtolton11:06:41

im really just joking. I wanted to be lazy and give it a lispy name, but I'll name it something descriptive and useful if I have to i guess

curlyfry11:06:31

@tjtolton In clojure core most function inputs are simply called f https://clojuredocs.org/clojure.core/update Transducers are usually called xf or xform https://clojuredocs.org/clojure.core/transduce

curlyfry11:06:46

But descriptive is always best if possible! 😄

tjtolton11:06:59

okay, it was transducers I was thinking of, gotcha

tjtolton11:06:25

well, I guess xform implies transform too

tjtolton11:06:27

so xf it is

hjrnunes15:06:15

@yogthos in selmer, can I access nested structures using vars? As in: item.var1 for var1 = :foo and item = {:foo "bar"}

yogthos15:06:23

not sure I understand, can you give a complete example?

hjrnunes16:06:26

@yogthos class and inner_class contain keywords

yogthos16:06:18

oh I see, yeah that’s not currently possible

yogthos16:06:44

you probably could add a tag that did this though

yogthos16:06:15

the tag could accept the item and the name of the key, then select it from the item

hjrnunes16:06:51

@yogthos got it. thanks!

lmergen16:06:25

what’s typically the best way to convert a heavily “nested map of vector of maps” — i think i’m looking for a nice lens library. a transducer seems to be able to do this quite easily due to the callback nature of it, but there’s a bit of boilerplate involved here i wonder if there’s any decent clojure lens library ? is this where specter comes into play ?

jeff.terrell16:06:11

Yep, Specter can do lens-y things, so that's probably what you want.

lmergen16:06:42

right, now i guess i have to get over myself and learn to accept its UPPERCASE syntax things 🙂

captainlexington16:06:10

I am really interested in learning Specter, but I'm also curious about his lens pattern in general. I haven't heard about it before.

lmergen16:06:59

@captainlexington it’s commonly used in Haskell, not sure whether it’s used in many other languages

jeff.terrell16:06:58

I think he actually kinda invented it himself then realized later that it was called lensing, so I wouldn't be surprised if it were a bit different than what you might find elsewhere.

jeff.terrell16:06:13

I'm pretty ignorant about both lensing and Specter, though, so take that with a grain of salt. :-)

lmergen17:06:27

it’s very powerful, and saves you a lot of boilerplate — but on the other hand, requires a whole new vocabulary for tree traversals

lmergen17:06:38

didn’t know about that one, looks good as well

lmergen17:06:01

i like the syntax a lot better

nathanmarz17:06:11

@jeff.terrell that's right, although I do think "navigation" is a much better name

nathanmarz17:06:36

haskell has multiple abstractions for navigation (lens, traversal) while specter just has one

nathanmarz17:06:08

not sure why haskell complicates things like that, as there's no performance benefit to distinguishing – I think it may have something to do with the type system

souenzzo17:06:18

traversy replaces the original update function! Does traversy works with datomic entity maps?

stuarthalloway17:06:12

@souenzzo if it does not, you might be happier with Datomic pull, which returns ordinary maps

josh_tackett18:06:41

Does anyone know a hardware that allows for GPS tracking with an API to pull location information from?

zylox19:06:23

i think there are gps modules for things like rasp pi

zylox19:06:40

unless you want something that would connect to pc over usb or something...im sure that exists

alricu21:06:00

Hi all, I am trying to find a way to create a REST client in clojure in which I can send client certificates to the server!!!! I have the client client.crt and the client.key and I need to send them in a GET request

alricu21:06:26

I tried with Alepho but nothing happens!!!

alricu21:06:52

not working!!!

alricu21:06:00

I do not know why!!!

qqq22:06:02

(defn entry-<-file [file]
  (let [fname                     (.getName file)
        [_ book_name page_number] (re-matches #"(.*).pdf-([0-9]*).png" fname)]
    (println "entry-<-file: " fname)
    (when book_name
      [book_name (Integer. page_number) (slurp file)])))

(defn load-book-pages! []
  (j/execute! db-spec ["drop table book_pages"])
  (j/execute! db-spec
              [(str
                "create table book_pages "
                "( book_name VARCHAR(20) "
                ", page_number BIGINT "
                ", png MEDIUMBLOB "
                ", PRIMARY KEY(book_name, page_number)) ")]) 
  (let [entries      (keep entry-<-file (file-seq ( "/tome/tb/data/calc1")))
        groups-of-10 (partition 10 entries)] 
    (doseq [group-of-10 groups-of-10] 
      (jdbc/execute!
       db-spec 
       (hc/format {:insert-into :book_pages
                   :columns     [:book_name :page_number :png]
                   :values      group-of-10})))))


I am inserting a bunch of (book_name, page_number, png_of_page) triples into a db.

qqq22:06:14

Currently, this is a bit slow as the png is loaded, serialized intoa . string, put into the sql statement, then executed.

qqq22:06:21

Is tehre a better / more efficient way of dealingwith blobs ?

bfabry22:06:16

I hate answering simple questions with a "maybe change your design" buuut... I've never heard of anyone having good experiences with blob/clob fields. most projects I've worked on in the last 5 years have stored a varchar with the path to a randomly generated unique path on disk instead

qqq22:06:49

damn it 🙂

qqq22:06:55

so I'm storing a bunch of machine learning data

qqq22:06:09

and I was hoping to stuff it all in the db, instead of (1) having a separate kv store and (2) stroing the shasum of the blobs

qqq22:06:23

why can't db providers just attach a blob store? 🙂

bfabry22:06:49

well I think that is basically what they do underneath, it's just not having access to that underneath tends to end up restricting you. ie if I want to send that blob to a http requester I'd much prefer to just point apache at the path than have to have it go through my application memory

qqq22:06:36

hmm, I didn't even consider caching

qqq22:06:55

it's directly from file, possibly cdn-able vs "load into db memory; load into app memory; stream over http / unable to be cached"

bfabry22:06:26

yes, it would be cdnable (depending on lots of things to do with setting it up of course)

qqq22:06:51

alright; I'm convinced

qqq22:06:59

any recommended hashing functin to use from clojure ?

bfabry22:06:04

not really sorry. I would say you need to make very sure that the path is both unique and unpredictable, as if it's predictable it's an attack vector. you also probably want it to be in some sort of place on disk that can't be traversed out of

csm22:06:06

nit on that gist, why not just (format "%02x" (bit-and % 0xFF)) instead of the substring stuff?

qqq22:06:31

to be clear, that's not my gist

qqq22:06:38

taht's the first result . found for "clojure shasum"

csm22:06:44

yeah, got that, just sayin

qqq22:06:23

(.digest (java.security.MessageDigest/getInstance "sha-512") (.getBytes (slurp "build.boot")))

qqq22:06:31

that's what I'm goingto use; the rest of the gist, no idea what it's there for

qqq22:06:12

edrr, IK don't need that

qqq22:06:18

I'll just use the sha512 hash as the fname

qqq22:06:31

it's guaranteed to be 0-9, abcdef for filename

hiredman22:06:40

using slurp to read a binary file is a terrible thing to do, and turning binary data in to a string before sticking it in the db is also terrible

hiredman22:06:43

Strings are pretty much garbage for anything except for data you are printing out as a displayed string to a person

hiredman22:06:42

slurp is bad for reading in binary data because it creates strings (which are garbage) and assumes a character encoding (or you can pass one in) because to build a String (from characters) from bytes you need an encoding, but binary data doesn't have one

qqq22:06:49

(.toString (.digest (java.security.MessageDigest/getInstance "sha-512") (.getBytes (java.nio.file.Files/readAllBytes (java.nio.file.Paths/get (java.net.URI/create "build.boot"))))))

qqq22:06:53

gah, I'm getting a "missing scheme" error

hiredman22:06:11

^- creating a string from binary data again

qqq22:06:24

no, the digest is hex

hiredman22:06:39

hex is encoding of bytes

hiredman22:06:45

the digest is a byte array

lucasbradstreet22:06:49

Has anyone had any difficulties with repl refresh when using cljc files with reader conditionals? I find clojure.tools.namespace.repl/refresh ignores the namespaces completely when using the conditionals within the ns form

qqq22:06:43

@hiredman: I am putting quite a bt of google-fu into this; can you please tell me the RIGHT way to do this instead of saying " it's wrong?" 🙂

hiredman22:06:45

(let [baos (ByteArrayOutputStream.)] (io/copy some-file baos) (.toByteArray baos))

hiredman22:06:18

the nio way may be faster, but it may not, and you will need to expand some google juice working on the syntax for it and reading the javadocs

hiredman22:06:29

java.nio.file.Files/readAllBytes returns a byte array, so you cannot call getBytes on it

hiredman22:06:55

you seem to be randomly button mashing stuff together, that is not how you write a program

bfabry22:06:40

depends on the program and the context, I've button-mashed out plenty of stuff over the years

qqq22:06:42

copying/pasting SO snippets is a completely reasonable approalch to learning a new API

hiredman22:06:05

not if you can't make the edges line up

qqq22:06:20

that's when you ask for help in #clojurians slack 🙂

qqq22:06:29

I can completely be replaced by a bot that just (1) gets snippets from gist / SO and (2) runs the vodce, and (3) pastes the error into slack

qqq22:06:53

(.digest (java.security.MessageDigest/getInstance "sha-512") (java.nio.file.Files/readAllBytes (java.nio.file.Paths/get "." (into-array ["build.boot"]))))

qqq22:06:07

button mashing success! 🙂

kaosko22:06:35

hey guys, does anyone know how to configure immutant/undertow to respond properly to byterange requests?

qqq22:06:19

@csm: what is the use of (format "%02x" (bit-and % 0xFF)) ?

csm22:06:41

it turns the digest byte array to a hex string

qqq22:06:32

(format "%02x" (bit-and 88 0xFF))

qqq22:06:36

okay, can you explain to me how this works?

qqq22:06:42

because it clearly works, byt I don't understand why it works

qqq22:06:09

bit-and = strips all but last 8 bits

csm22:06:12

the bit-and is since the input is bytes, which are signed; so that turns it into a “unsigned byte”, if you will

csm22:06:41

the %02x formats it as hex, padding with 0 prefix if needed

qqq22:06:45

isn't 0xFF the same as 11111111

noisesmith22:06:00

I really wish our bot could handle strings

noisesmith22:06:15

@qqq it’s pretty idiomatic for byte munging to represent 255 as 0xff

noisesmith22:06:33

but yeah, it’s eight bits, all of them ones

qqq22:06:39

sure, my questiho is, isn't x the same as (bit-and x 0xff)

qqq22:06:43

since 0xff is 11111111

qqq22:06:55

okay, tell me why I'm wrong

noisesmith22:06:26

as csm said, we need to handle signed bytes

csm22:06:46

hmm, it seems to work without the bit-and, but I’m not sure why

noisesmith22:06:12

@csm not for me, for negative bytes

noisesmith22:06:31

peregrine.circle=> (format "%02x" -88 0xFF)
"ffffffffffffffa8"
peregrine.circle=> (format "%02x" (bit-and -88 0xFF))
"a8"

noisesmith22:06:59

oh - for a byte though it might just work

noisesmith22:06:18

peregrine.circle=> (format "%02x" (byte -88) 0xFF)
"a8"

noisesmith22:06:25

yeah, it’s very type dependent I guess

csm22:06:47

yep, Java’s String/format behaves the same on int vs. byte

noisesmith22:06:58

that’s all clojure’s format is 😄

csm22:06:59

learning something new every day 🙂

noisesmith23:06:37

(defn format
  "Formats a string using java.lang.String.format, see java.util.Formatter for format
  string syntax"
  {:added "1.0"
   :static true}
  ^String [fmt & args]
  (String/format fmt (to-array args)))
- much smaller function than I first expected

noisesmith23:06:10

I bet that one function prevented an otherwise inevitable wrapper lib from being written, lol