Fork me on GitHub
#clojure
<
2016-09-06
>
tom01:09:21

Does plumatic/schema have a type for a function? ClojureScript context

tom01:09:22

{:a s/Str :b s/Fn}? The docs are sparse

pythonruby03:09:08

Hey there, I have a function which works when runs in "lein repl" but only runs "partially" when runs with "lein run" or uberjar. By "partially" I mean (-main) calls (a) calls (b) calls (c) and stops, while the should be (d) and (e) in the calling stack. If I call (a) from "lein repl" it will calls (b) all way through to (e) as expected. Any ideas?

csm03:09:45

is the result lazy?

csm04:09:06

I think the repl will evaluate the entire result, in order to print it

pythonruby04:09:43

ok, I'll just post my code out, wait a minute

pythonruby04:09:02

anything lazy to you?

pythonruby04:09:36

If I call (rename-pngs-in-folder) directly from repl, it works. However, if I call (-main) it will stops at (rename-files)

csm04:09:45

possibly; just wrapping the calls to rename-pngs-in-folder with doall would tell you for sure

pythonruby04:09:04

ok, let me try

csm04:09:05

I saw filter in one of the fns, which is definitely lazily evaluated: https://clojuredocs.org/clojure.core/filter

pythonruby04:09:39

lein run and uberjar both work. Thanks for the hint, saved my day.

csm04:09:55

I’ve run into that exact thing a bunch of times already 🙂

pythonruby04:09:12

experience shines

ghadi04:09:04

(defn supply
  [f eof]
  (reify clojure.lang.IReduceInit
    (reduce [_ rf init]
      (loop [res init]
        (let [v (f)]
          (if (identical? v eof)
            res
            (let [res (rf res v)]
              (if (reduced? res)
                @res
                (recur res)))))))))

;; dual to (line-seq rdr)
(supply #(or (.readLine rdr) ::eof) ::eof)

ghadi04:09:24

Need a better name for it ^ but I find myself creating that function over and over

dpsutton04:09:19

this is equivalent to (reduce f init (take-while #(not-sentinel %) coll)?

dpsutton04:09:36

supply reduces until it hits a predetermined thing to not reduce?

dpsutton04:09:40

unless i missed something (quite possible)

ghadi05:09:43

almost, it doesn't do the reduction. it's equivalent to (take-while #(not-sentinel %) (repeatedly f))

didibus08:09:58

Any way to extend a protocol for a particular sequence of precise type? Like I have a sequence of string and a sequence of int, and I want each to have a different implementation. Does Clojure know the type to that degree, or is all sequences typed as sequences of Objects?

didibus08:09:55

Also, is there a way I can set assert to false but only for one namespace and not the whole app?

mpenet09:09:14

Sequences are heterogeneous, so no. You can do that with arrays but that s it

grounded_sage11:09:23

what is the best library for reading from Excel documents?

Alex Miller (Clojure team)11:09:06

@grounded_sage there is a Java lib called Apache Poi you can use. I’m not sure if there is a Clojure wrapper for that.

grounded_sage11:09:12

thanks. I'm still new to programming and wanting to apply it to a small produce business I am starting. Trying to assess Elixir vs Clojure ecosystem. Elixir has a nicer runway but once in the air I'm sure Datomic would be an amazing asset to be able to use.

ikitommi11:09:33

@jannikko if you want bi-directional routing for your Compojure app, you could try compojure-api. It’s reverse routing is described here: https://github.com/metosin/compojure-api/wiki/Routing#bi-directional-routing

grounded_sage11:09:24

That one looks great! Man Clojure just has so much clarity in it's syntax. Might finish up the chapter I am on with Programming Elixir and go back to Brave and True haha.

stunami12:09:17

insert usual newbie disclojure. Im a bit stuck and this is when I usually learn the most.. but I am having a bit of a mental block here after trying for a while. I have built a map with this structure, and I want to evaluate it all to determine if :a has x attributes, and 😛 has y attributes, and :c has z attributes do one thing. If :a has y, and 😛 has x, and :c has z something else, and so on.

stunami12:09:04

here’s my map for the above example

stunami12:09:21

Im stuck on how to for/doseq/if/cond through the map and treat them collectively, i can only evaluate one k v at a time , but i want to evaluate k v k1 v1 k2 v2

stunami12:09:02

spamming done...

caio12:09:24

transform the whole map before and then check the conditions

caio12:09:23

i’d implement something like this:

(defn parse-cup [attrs] <parse the attributes here>)
(defn check-cup-lifted [cups]
  (let [parsed-cups (zipmap (keys cups) (map parse-cup (vals cups)))]
     (cond
         (and (= (:a parsed-cups) :x) (= (:b parsed-cups) :y) (= (:c parsed-cups) :z)) (do-one-thing)
         (and (= (:a parsed-cups) :y) (= (:b parsed-cups) :x) (= (:c parsed-cups) :z)) (do-something-else))))

stunami12:09:13

Cool, thanks @caio . That gives me a different angle, and I was definitely thinking a loop of sorts wasnt quite it, so the above makes sense. I’ll give that a go in the morning

ccann16:09:34

Looking for advice on adhering to http://jsonapi.org/ with Clojure. Right now I’ve built my API with compojure-api

Alex Miller (Clojure team)16:09:34

@nathanmarz (and you should get with the Java 1.8 :)

rickmoynihan17:09:46

@grounded_sage: There's also https://github.com/Swirrl/clj-excel which is our fork of clj-excel... Not sure of the differences to docjure, it might be better... but if you want another fallback this one fixes some issues with the original clj-excel, which fell into lack of maintainance

rickmoynihan17:09:30

there are also a bunch of other forks of it on github though - all I can say is this one works for us 🙂

didibus18:09:12

Anyway I can create a protocol that is compatible with LazySeq?

ghadi18:09:18

like a thing that behaves like a LazySeq?

pjstadig18:09:49

@didibus you can create a class that will implement the interfaces necessary to behave like a LazySeq. I'm not sure it makes sense to talk about a protocol that is compatible with LazySeq? Perhaps I'm misunderstanding you. Can you expand some more details about what you want to accomplish?

didibus18:09:46

@pjstadig I guess I'm kind of trying to subclass it. Basically, I want a LazySeq, but I want to add more behaviour to it. But when I do type polymorphism, I want the type of the one with more behaviour to be different then LazySeq.

pjstadig18:09:19

@didibus what behavior would like like to add to LazySeq?

didibus18:09:33

@pjstadig Maybe I need to explain my situation a little better. Basically, I have a protocol which I want to extend LazySeq with. But, if the LazySeq contains Tokens, I need a different implementation then if the LazySeq contains other types. So I was thinking of creating a TokenLazySeq which is compatible with LazySeq, but in my code, I would always use that type when I create a seq of Tokens. This way, my protocol can have a different implementation for that.

pjstadig18:09:04

@didibus what is the protocol that you are extending to LazySeq?

papachan18:09:49

Hello. is there some works with clojure/clojurescript and cartodb?

didibus19:09:08

@pjstadig A custom protocol I have. Think of it as adding a fuzzyEqual fuction

didibus19:09:23

@pjstadig So I want to define what it means for two LazySeq to be fuzzy-equal. But a LazySeq that contains Token should have a different meaning, so it means something different for it to be fuzzy-equal

pjstadig19:09:27

@didibus so you want to compare two lazy sequences where = would return false, but your fuzzy equal would return true because it is more lenient?

pjstadig19:09:33

@didibus what is an example of something that = would say is not equal, but your lenient equal would say is?

didibus19:09:35

@pjstadig And depending on the type of objects inside LazySeq I need a different kind of lenience.

majenful19:09:44

If I want to save data (text, some files/images) for a desktop app, could you suggest me a good strategy ? A SQL database seems too heavy (and painful for a user to install)

didibus19:09:43

@pjstadig One can be that only the majority of elements must be equal. Another one can force all elements to be equal. Another one can force at least one element to be equal, etc.

pjstadig19:09:11

@didibus does order matter? could your lenient equal return true, but if you switch the order of the sequence it would return false?

hlolli19:09:12

Is (defn symbol ^{:some :metadata}) deprecated? Seemed to have read that on stackoverflow, and adding metadata with the carrot-map doesnt come in when doing (meta #'symbol)

didibus19:09:27

@pjstadig Either way, I feel like there should be a way to create custom sequences. What if I want to have a list of receipes. Can't I make this explicitly typed to be a ListOfRecipies instead of LazySeq? Or must I resort to metadata

didibus19:09:13

@pjstadig I mean, its much more complicated then that. Fuzzy-equal also calls fuzzy-equals on the elements. And yes order does matter.

pjstadig19:09:14

@hlolli if you remove the carat I think it would work

hlolli19:09:00

yes you re right @pjstadig easy one to solve 🙂

pjstadig19:09:10

@hlolli you can add metadata to a defn with either (defn ^{:some :metadata} symbol ,,,) or (defn symbol {:some :metadata} ,,,)

pjstadig19:09:18

@didibus what you are describing is functions, and I think you'll find that you don't need to subclass LazySeq to add behavior, you just need to write a fuzzy-equals function that operates on (possibly lazy) sequences

didibus19:09:04

@pjstadig No, I need a function which varies in implementation based on the type of objects contained inside a LazySeq

pjstadig19:09:35

@didibus correct, and that can be done with functions. there's no need to subclass LazySeq for that

didibus19:09:20

@pjstadig How can it be done?

pjstadig19:09:33

equality isn't even something that is compatible with a lazy seq, you need to fully consume a lazy seq to come to a conclusion about equality

didibus19:09:57

@pjstadig Yes, which my function does.

didibus19:09:21

@pjstadig But in the case of my fuxxy-equal, I don't always need to consume everything.

didibus19:09:48

@pjstadig Sometimes, it means equal as soon as one element is equal. So you don't have to consume everything

didibus19:09:47

@pjstadig I guess it sounds like I need to create a type which manually implements all of the LazySeq protocol.

pjstadig19:09:35

(some (fn [[x y]] (fuzzy-equals x y)) (map vector seq1 seq2)) would return falsy if fuzzy-equals returns false for at least one pair of elements

pjstadig19:09:43

no need to subclass lazyseq

pjstadig19:09:25

similarly (every? (fn [[x y]] (fuzzy-equals x y)) (map vector seq1 seq2)) would return truthy if fuzzy-equals returns true for every pair of elements

didibus19:09:25

@pjstadig You don't understand my problem. The issue I have is that, I need a different type of equality logic for different kinds of LazySeq. That is, depending on the type contained inside the LazySeq

pjstadig19:09:53

I guess I'm suggesting that it is not useful to think of LazySeqs with different contents as being different kinds of LazySeqs, but perhaps you are right and I've misunderstood your problem.

didibus19:09:18

@pjstadig So I'd need something like: (fn [lazySeq] (cond (every? #(instance? String %) lazySeq) (do-string-sequence-fuzzy-equal lazySeq) (every? #(instance? Token %) lazySeq) (do-token-sequence-fuzzy-equal lazySeq)

didibus19:09:48

Well, its because I'm simplifying the whole thing. But basically, I have different types of things grouped together. And I want to fuzzy-equal these groups together. But, only groups which have similar things can be fuzzy-equaled, and fuzzy-equaling each group together means something different to my domain depending on the type of things they contain.,

didibus19:09:41

I could group them using records, and then it would all work. Each group will have its on record type, and then I can extend my fuzzy-equal protocol for each of these types.

didibus19:09:21

But, I don't want record semantics, because I need them grouped in order. Also, I like the performance gains I get from the Lazyness. And I don't care about keys, only values.

didibus19:09:53

Ideally, I'd have another method to create say a person-seq and a token-seq and a vehicle-seq. Each would return a sequence of person, token or vehicle based on which one you called. Those function will assert that everything inside it is of the same type. So I get back a bunch of LazySeq which my code guarantees contain the same type. One has tokens, one has vehicle and one has persons. Now I want to call (fuzzy-equal person-seq1 person-seq2) And have it do what should be done when its persons. If I call (fuzzy-equal vehicle-seq1 vehicle-seq2) I need it to do what is required for vehicles.

didibus19:09:36

@pjstadig Does that make more sense?

pjstadig19:09:57

i think it makes sense to have functions that would create homogeneous sequences. I don't think fuzzy-equals needs to care. It can compare each pair of elements from a couple of sequences, and how you would combine those element comparisons (i.e. did at least one of them match? did all of them match?) is a different question that can be answered by a function like some or every?

pjstadig19:09:17

if fuzzy-equals was trying to compare a vehicle to a person, couldn't it just return false?

pjstadig19:09:47

and i guess what i mean my fuzzy-equals is a function that compares two single elements

pjstadig19:09:01

i'm trying to separate the idea of comparing elements with the idea of comparing sequences

didibus19:09:22

Well, the behaviour I need is that a sequence of data that relates to the same thing is all used together to decide equality. But, a sequence of sequences is different.

pjstadig19:09:25

if you want to define some sequence comparison functions that assert that their arguments are sequences of elements that are all the same type, that could work, too

pjstadig19:09:07

the point is you are talking about defining behavior, and behavior is what functions do. you do not need to subclass something to get behavior

didibus19:09:35

I know that. I need to subclass to get type polymorphism

didibus19:09:27

It doesn't really matter to me that I use a protocol or not. But I need a way to identify the type of a LazySeq in a way that's more specific then the LazySeq type. I need to distinguish between different LazySeq

didibus19:09:09

I was trying to not use metadata, because sometimes it gets inadvertently lost as you apply transorms on the seq

pjstadig19:09:57

i'm still not sure i agree with the approach, but if you want to create a new class that behaves like a LazySeq but has a different type depending on the type of its elements, but you don't want record semantics, then you can use deftype instead of defrecord

didibus20:09:09

Hum, could I use deftype to simply subclass LazySeq but have the type be different?

didibus20:09:31

But it would still work as a LazySeq everywhere LazySeq works

pjstadig20:09:13

@didibus you would not subclass LazySeq. If you want to subclass you should use proxy. You could use deftype and implement a bunch of different interfaces (like Seqable and IPersistentCollection).

didibus20:09:06

Hum, ok, can I deftype and implement LazySeq, but just reuse the clojure implementation of it?

didibus20:09:57

All I need is just for my LazySeq to have a different type, so I can switch on it

didibus20:09:08

But still works wherever LazySeq would

majenful20:09:27

Could you suggest me a simple way to handle file uploading from a user in a web app ?

val_waeselynck20:09:23

@majenful if that fits your requirement, S3 provides a way to directly upload from the client without the payloag going through your servers

val_waeselynck20:09:03

@majenful google "S3 presigned upload url"

majenful20:09:44

nop I don’t use S3,

majenful20:09:33

but I would like to handle it into my app

majenful20:09:12

I saw @yogthos solution in luminus app, but there is too much java interop for my taste

val_waeselynck20:09:51

it seems reasonable to me

rickmoynihan20:09:17

At the bottom in ring the body of an http request is an inputstream - so you can read/write it where you want... or mix in a middleware to e.g. coerce x/www-form-encoded (or whatever it's called) into a java tempfile etc..

rickmoynihan20:09:45

most default middleware stacks have something like that included

majenful20:09:16

@val_waeselynck I though in one of libraries there were functions already

rickmoynihan20:09:34

if the file is small enough and its text - you can just slurp the body too

majenful20:09:38

It will be image

idiomancy20:09:19

general clojure convention question: should non-referentially transparent functions be marked with a bang(!)?

idiomancy20:09:11

like, if I have something that asks for the current mouse pointer location, should it be, e.g. (pointer-location) or (pointer-location!)

pjstadig20:09:45

@idiomancy maybe. Clojure core itself is not consistent

idiomancy20:09:33

im conflicted. It almost seems like it should be preceded by one

idiomancy20:09:42

(!pointer-location)

idiomancy20:09:09

indicating the two different non-pure function types

pjstadig20:09:11

I would recommend the interrobang: pointer-location‽

idiomancy20:09:22

Ill get right on that

rickmoynihan20:09:56

majenful: pretty sure you could just use (with-open [upload (:body request)] ( upload (file "/tmp/uploaded-file.jpg"))

idiomancy20:09:39

@pjstadig oh my god you were serious

rickmoynihan21:09:01

one difference is the luminus stuff is using the nio apis though... so it might be more efficient - depending on the webserver. The above will block the thread handling the request

pjstadig21:09:24

look unicode exists

rickmoynihan21:09:34

pjstadig: Hey it's funny you've posted that - I've been toying with with using reducer/transducer streams too

pjstadig21:09:07

@rickmoynihan i wrote a blog post about it. I think the readme contains much of the same information http://paul.stadig.name/2016/08/reducible-streams.html

rickmoynihan21:09:03

pjstadig: I've been looking at it from a performance (and of course resource handling) perspective. Initially I've been looking at CSV processing... basically the standard clojure CSV parsers all abstract over them as lazy-sequences... my benchmarks show that reimplementing one in terms of reducers/transducers should be able to improve performance by between 2-10x - and that's before you start layering transformations onto the parsed results

rickmoynihan21:09:45

but I've not actually implemented a real parser yet

rickmoynihan21:09:23

just a crude simplistic one using (.readLine buf) and a split on ,... though I've been thinking also about trying to parallelise the processing

Alex Miller (Clojure team)21:09:49

it’s trickier than that due to quoted and multi-line cells

rickmoynihan21:09:13

alexmiller: yeah I know 🙂 like I said I haven't implemented a proper parser yet

Alex Miller (Clojure team)21:09:21

but it seems like you reuse the guts of an existing one to leverage those parts

Alex Miller (Clojure team)21:09:41

adding something like this to data.csv for example would be a great addition

rickmoynihan21:09:46

that's why my estimate is so large 🙂 - I was actually planning on taking your clojure.data.csv and reducerifying it

Alex Miller (Clojure team)21:09:31

(not my clojure.data.csv btw :)

rickmoynihan21:09:35

but was trying to get a ballpark on performance before I did

rickmoynihan21:09:59

ahhh sorry - contribs one 🙂

ghadi21:09:32

on that note, I posted this yesterday that will come in handy @rickmoynihan https://clojurians.slack.com/archives/clojure/p1473135664000261

ghadi21:09:19

IMHO reducefication is worth it, but parallelization is not

rickmoynihan21:09:40

but also saw there's a java one (univocity) that's supposed to be one of the fastest

rickmoynihan21:09:09

@ghadi: I think you're probably right... about reducification being the biggest win... but I recon there's some mileage in parallelising... I've been researching it a little... and the results from e.g. widefinder/widefinder2 some years back indicate it's possible. I have a hunch that one trick is that you still (even with SSDs) need to read the file sequentially (otherwise you blow cachelines etc)... so you basically have a single thread reading it as a sliding buffer/window using a bufferedinputstream - then because readers are a lot slower than inputstreams (I'm guessing due to 16bit characters in java) you run each reader on a separate thread... assigning them a chunk of the underlying buffer, the csv parsing itself happens in parallel - along with the object allocations. Computations then get layered ontop of each reader as transducers.

rickmoynihan21:09:13

you'll probably need quite a large buffer to make it worth while... coordination overhead could be a killer though

ghadi21:09:28

(yeah the old tim bray post is good. early days of clojure too)

ghadi21:09:19

There's a lot of great talks from Guy Steele where he talks about parallel prefix algorithms... But for a large amount of applications, I think it's really worthwhile to get rid of seqs and just do eager reductions on the stack

rickmoynihan21:09:25

yeah - its a realshame most of that widefinder stuff is no longer online 😞 oracle destroyed sun

rickmoynihan21:09:11

yeah I've seen a lot of Steele's talks

ghadi21:09:02

seqs are useful but reducibles are much faster

rickmoynihan21:09:06

yeah - I was planning on doing something similar with sparql

ghadi21:09:00

if you reify Seqable and IReduceInit, then you get to choose between a caching seq and a non caching reducible

ghadi21:09:17

reducible implementations have a definite pattern, kind of like the formula that people learn for making lazy-seqs, just a tad more boilerplate.

rickmoynihan21:09:21

ghadi: that's exactly what I was doing! 🙂

rickmoynihan21:09:11

I really wished clojure.lang.Seq was a protocol though

ghadi21:09:13

an old gist of regex splitting as a reducibles https://gist.github.com/ghadishayban/7002262

rickmoynihan21:09:53

but c'est la vie

rickmoynihan21:09:06

thanks these links are useful

ghadi21:09:44

I think i need to write a post about the reducible "formula"

rickmoynihan21:09:39

yeah - it's definitely not as popular as it should be... also once you do it, it's infectious... as you want everything to behave the same

rickmoynihan21:09:23

e.g. - could really benefit from defining CollReduce on Reader etc... I'd do it myself, but I don't own the protocol or the type

jazzydag21:09:18

Hi, I've a HugSQL question with a JDBC driver over a postgres db. I wanted to carry out a date operation https://www.postgresql.org/docs/current/static/functions-datetime.html In a where clause such as WHERE date = :date - integer '7' (note that date is a column of my table). When I execute the generated HugSQL function with a :date parameter (joda date type), I've got

PSQLException ERROR: operator does not exist: date = integer
  Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
  Position: 249  org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse (QueryExecutorImpl.java:2182)

jazzydag21:09:53

When I'm trying to specify the type with a WHERE date = date :date - integer '7', I've got another Exception

PSQLException ERROR: syntax error at or near "$1"
  Position: 256  org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse (QueryExecutorImpl.java:2182)

jazzydag21:09:07

I fixed it to put this date operation in a ON after a JOIN. But I would like to understand the previous errors

dpsutton22:09:33

ghadi can you explain what you mean "get rid of seqs and just do eager reductions on the stack?"

dpsutton22:09:14

i can only guess reduce lazy into a single value so that you don't have a huge memory allocation, and just recur on the acc value of the reducer?

ghadi22:09:40

lazy-seqs basically consist of a function that produces a value and a thunk -- when you call the thunk it produces another seq (value + thunk). All those values require storage and memory allocation. Instead if you alter the definition of a collection into "something you can reduce over" you can divorce yourself from the need to allocate so much. The only thing you need to store is a couple of things on the stack to accumulate. Here is a loose version of range as a reducible collection:

ghadi22:09:36

(defn rrange [start end]
  (reify clojure.lang.IReduceInit
    (reduce [_ rf init]
      (loop [acc init
             i start]
        (if (< i end)
          (let [acc (rf acc i)]
            (if (reduced? acc)
              @acc
              (recur acc (inc i))))
          acc)))))

Alex Miller (Clojure team)22:09:54

(if only it were that easy :)

ghadi22:09:21

so it's a bit boilerplate-y... but as you can see, acc and i are the only thing that are ever stored, and they're only stored on the stack and mutated

dpsutton22:09:32

alex you use invalid parenthetical emoticons

dpsutton22:09:47

i like that shorthand syntax

ghadi22:09:47

'mutated'

Alex Miller (Clojure team)22:09:53

says you - I’m old school

dpsutton22:09:23

oh no, i'm saying that's a closing parenthesis doing double duty

Alex Miller (Clojure team)22:09:34

efficiency was more important in my day

dpsutton22:09:03

ghadi, you said get away from seqs, but doesn't this embrace them?

ghadi22:09:15

no seqs in that implementation at all

dpsutton22:09:39

i'm thinking its gonna operate on a lazy seq. and if its realized, you would gain no benefits?

ghadi22:09:54

(reduce + 0 (rrange 5 10)) <- all work is done by the reducing function, in this case +

ghadi22:09:42

I think there is a misunderstanding here -- rrange above is a function that returns a 'virtual collection' i.e. something that can be reduced

ghadi22:09:00

it doesn't actually realize the collection

dpsutton22:09:25

it's just a lazy sequence, correct?

ghadi22:09:27

calling reduce does that. But it does it in a way that only one item is produced at a time

ghadi22:09:12

Nope still, no lazy seqs at all. It's just a implementation of an interface that defines 'something that can be reduced over'

dpsutton22:09:38

that's IReducedInit?

dpsutton22:09:52

ok. this ads some more clarity to what you were posting yesterday

dpsutton22:09:28

i was the one who asked if it was equivalent to (reduce f init (take-while not-sentinel coll))

dpsutton22:09:39

thanks for taking the time

ghadi22:09:41

using a reducible and a seq have some similarities -- you 'consume' both. But lazy-seqs can be realized bit by bit, then you can pause, then realize some more. Reducibles are one-shot, eager

ghadi22:09:10

np. reducibles could use some more attention.

dpsutton22:09:16

ahh thus your supply

bwstearns23:09:31

Does anyone have a recent open project or gist that reads text from PDFs? I found a couple examples but they're from a while ago and the APIs they use seem deprecated and I was wondering if I am missing the "modern" way of doing it.