Fork me on GitHub
#beginners
<
2020-05-17
>
metehan01:05:55

What is the clever way to sort values descending order for now I am doing like this ->

(fn [ledger _]
   (reverse (sort-by (comp :date val) ledger)))

dpsutton01:05:52

a common idiom is to

(sort-by (comp :date val) #(compare %2 %1) ledger)
where you just swap the compare args

👍 1
EmmanuelOga03:05:37

sup, have a simple thread loop I was wondering what's a more idiomatic way to write something like this: https://gist.github.com/EmmanuelOga/0ca240eee458fb8e039c5089c43d904a Thx!

Michael W04:05:27

I'm trying to solve this problem: http://www.4clojure.com/problem/128

(fn [c]
  (let [suit [:diamond :heart :club :spade]
        suitsym ["D" "H" "C" "S"]
        rank (mapv str (range 0 13))
        ranksym (mapv str (concat (range 2 10) ["T" "J" "Q" "K" "A"]))
        suits (zipmap suitsym suit)
        ranks (zipmap ranksym rank)
        s (clojure.string/split c #"")]
    (hash-map :suit (suits (first s)) :rank (read-string (ranks (second s))))))

Michael W04:05:45

That works in my repl, but the website gives me a NullPointerException

andy.fingerhut05:05:03

The 4clojure site uses an old version of Clojure (1.4 maybe?), so likely there is some function you are using that doesn't exist in that version, or behaved differently somehow.

andy.fingerhut05:05:02

or perhaps it is the use of clojure.string/split ? I don't know for sure -- guessing what might be the difference.

AC05:05:42

My guess is that the NPE is from nil being passed to read-string. clojure.string/split is likely doing something funky.. manually binding ["D" "Q"] to s causes it to pass the first test.

Michael W05:05:30

Thanks I actually got it working by turning it into a direct lookup without all the string conversions

👍 1
Michael W05:05:55

Created a map from char to string for both lookups and skipped the intermediate conversions, it worked and was much shorter.

Timofey Sitnikov12:05:02

How do you start a ring server with just clj, not using boost or lein?

craftybones12:05:49

@timofey.sitnikov Do you have a main that starts it? If so, its just a question of setting up an alias in your deps.edn with :main-opts to [“-m” <name_of_your_main_namespace>]

craftybones12:05:57

and then invoking that

craftybones12:05:09

with

clj -A:server

craftybones12:05:01

For example:

{:deps {ring/ring-jetty-adapter {:mvn/version "1.6.3"}
        ring/ring-mock {:mvn/version "0.4.0"}
        compojure {:mvn/version "1.6.1"}}
 :paths ["src"]
 :aliases {:test {:extra-paths ["test"]}
           :runner {:extra-deps {com.cognitect/test-runner {:git/url ""
                                                            :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}}
                    :main-opts ["-m" "cognitect.test-runner"]}
           :server {:main-opts ["-m" "foo.main"]}}}

craftybones12:05:10

Now I start it with clj -A:server

👍 1
Timofey Sitnikov13:05:03

@srijayanth, oh awesome, so there is nothing all that special, that will certainly work. Looking at the ring readme, it looks like there is some magic that goes on underneath with lein and boost. This is why I like clj - fewer abstractions.

Dieter17:05:58

Hi, I'm wondering how to best organize data in a full stack project. I don't find a lot of resources on it but I suppose it is documented somewhere? I'm starting to learn re-frame and doing a project with sample data, but eventually I want to use crux or datomic on the backend. It seems like those databases organize data as [{:person/id "1" :person/name "xx" }, {..}] (flat?), while in frontend it seems like more nested structures are used {:person [{:id 1 :name "xx"}, ..]} (or even subdivision for id as recommended here https://purelyfunctional.tv/guide/database-structure-in-re-frame/). What is a commonly used strategy? is it useful to keep the same structure on backend as frontend? (databases can use queries, but not possible with re-frame)?

krzyz17:05:30

It’s mostly performance related, and depends a lot on the kind of data and what you are planning to do with it. Of course it is best to have your db data match how you store data in your app (the less work to convert, the better), but again, you often need data in your app structured a particular way for performance reasons, and converting to/from db storage happens less frequently than data access and manipulation within your app. Just as one example, consider your flat/nested difference. If I am working in my app and I want to access a collection of people by ID or name, I could filter over the collection by name, but that is an O(n) operation. If it is just a few names and doesn’t happen frequently, who cares, but I may also want to structure it like this {id person id person}, in which case accessing an individual person by ID becomes an O(1) operation. Again it depends on the size of your collection and how frequently the operation is done, whether or not something like this matters. Then you have the tradeoff of being able to outsource processing to clients, thereby reducing load on your server, vs fluidity of your app for the user. There are no simple answers, just a lot of tradeoffs to consider. I should also mention that choosing the right data representation can either simplify or complicate your code, so that's another thing to look out for. You'll build up an intuition about it over time. Particularly with re-frame, you'll notice if your db accessors are very long or complicated and hard to read, that's probably because your data could be structured better.

Dieter18:05:49

Thanks for your reply. Performance won't be an issue, so I'll try the 'flat' structure and see how it works out 🙂

💯 1
dvingo18:05:27

Even if you're not using fulcro, the db model used there is where you'll likely end up after much iterating and headaches: http://book.fulcrologic.com/#_fulcro_client_database

dvingo18:05:11

I have a .cljc data-model file that uses clojure specs to define my data contract and then I use that on the client and server. combined with guardrails which lets you define fn specs inline https://github.com/fulcrologic/guardrails you can iterate very quickly on a data model and then add some light type checking along the way

dvingo18:05:42

there's been some recent exploration using fulcro just for the normalized db and re-frame for ui layer: https://www.youtube.com/watch?v=ng-wxe0PBEg

Dieter19:05:44

So if I understand correctly, you can store data in graph style (similar to crux and datomic), but than you can also use and manipulate the data, as in a nested structure?

dvingo20:05:25

data is stored normalized on the client - so all updates are 3-levels deep: (assoc-in state-db [:users/id :my-user :name] "new username") at render time the fulcro db->tree takes a query and a state db and returns a tree of data that you can pass to your components. so you don't have to update nested structures, but you get to render with nested structures, the best of both worlds

Dieter10:05:20

I notice they use 'pathom' https://github.com/wilkerlucio/pathom but doing this without fulcro does not seem so easy to implement. I like the concept though, I'll keep it in mind for later. Thx for sharing this

Viktor18:05:43

Hi there, I can run (. Byte MAX_VALUE) ; => 127 But I cannot do (map #(. % MAX_VALUE) [Byte Integer Long]) it gives me: Error printing return value (IllegalArgumentException) at clojure.lang.Reflector/getInstanceField (Reflector.java:397). No matching field found: MAX_VALUE for class java.lang.Class Why, any ideas ?

andy.fingerhut18:05:20

. is a 'special form' in Clojure, which has some similarities to Clojure macros, in that some arguments are not evaluated, or evaluated differently than arguments to a function call.

andy.fingerhut18:05:20

If you look on the reference docs page for Java interop here: https://clojure.org/reference/java_interop

andy.fingerhut18:05:08

I believe (. Byte MAX_VALUE) falls into the category of (. Classname-symbol member-symbol)

andy.fingerhut18:05:12

If it is very important for you to be able to do something like that code and have it work as you wish, you can use the Java reflection API explicitly

Cas Shun20:05:22

So I'm still on this InputStream reading thing, just because I don't understand how things work fully yet. I have a function that tries to read an input stream into an array of byte-arrays. The byte-arrays are a given size. I'm inflexible on this just because I want to learn how to learn about how to troubleshoot what I've done so far. Code example coming.

Cas Shun20:05:19

(defn read-mem-3
    "Returns an array of byte arrays of size available / frame-size"
    ^"[[B"
    [stream frame-size]
    (let [available (.available stream)
          frames (/ available frame-size)
          ma (make-array Byte/TYPE frames 6)]
      (dotimes [i frames]
        (.read ^InputStream stream ^bytes (aget ^"[[B" ma i)))
      ma))

  (with-out-str 
    (time 
     (let [as (FileInputStream. "big-file-here") ]
                        (read-mem-3 ^InputStream as 6))))
  ;; => "\"Elapsed time: 1432.400101 msecs\"\n"

hiredman21:05:36

Don't use available, read in the loop until read returns -1

Cas Shun21:05:15

how do I create the array to hold the data then?

phronmophobic21:05:42

just using a BufferedInputStream gives me a >10x speed boost on my laptop:

(time 
 (let [as (BufferedInputStream. (FileInputStream. "testfile_10MB")) ]
   (read-mem-3 ^BufferedInputStream as 6)))

hiredman21:05:50

The buffers inputstream speedup is showing that "frame-size" of 6 is the example is absurdly small

hiredman21:05:43

A buffered inputstream will read in larger chunks at once, and then trickle them out to you, so you can achieve the same effect using a larger buffer directly

hiredman21:05:44

You can use something more flexible that you don't need to allocate all at once to hold your byte arrays

hiredman21:05:15

Like a clojure vector or Java arraylist

Cas Shun21:05:25

the downstream callee wants the data as an array of byte-arrays

hiredman21:05:21

Then I would use the length of the file

hiredman21:05:35

Which is not what available on an inputstream means

Cas Shun21:05:10

hmm, ok. When I get back to my computer I'll investigate this further then.

Cas Shun21:05:25

available always returned the file length, so I found that confusing

Cas Shun21:05:52

Regardless, even if I remove available and use a long directly, it doesn't speed anything up. I tried that before.

hiredman21:05:59

I imagine most the time it does

hiredman21:05:35

Right, because 6 is an absurdly small buffer size for reading

hiredman21:05:54

So use a buffered inputstream that manages a larger buffer for you, or use a buffer in the mb range (I like to start with 5mb see from there)

Cas Shun21:05:10

So I should use BufferedInputStream or read it all at once, and then what would be the fast way to split it into an array of byte-arrays of length 6?

hiredman21:05:43

Make byte arrays and copy the data into it

Cas Shun21:05:49

ok, I'll explore that. Probably back with more. Thank you for your patience and help

hiredman21:05:11

arrays are fairly inflexible, so you can't really turn a byte array of size 10 into 5 byte arrays of size 2 without making new arrays and copying

hiredman21:05:10

Which is why things like nio's Bytebuffer and netty's Bytebuf which are similar to what other langues sometimes call slices

Cas Shun21:05:13

That's what I figure. This seems to be what I get for learning clojure without having written a line of java in my life

phronmophobic21:05:33

coarsely, just (make-array Byte/TYPE frames 6) accounts for about 2/3 of the execution time on my laptop

phronmophobic21:05:51

so as hiredman noted, making a ton of 6 byte buffers is definitely not the most efficient approach

phronmophobic21:05:52

not sure if there’s a quicker way to allocate a 2d byte array from clojure

Cas Shun22:05:36

So if I wanted to rewrite this callee (which wants a 2d byte-array) myself, what would be the "fast" way to process a large byte-array in chunks myself?

phronmophobic22:05:36

you can just use a single byte array and use offsets

phronmophobic22:05:54

generally, the java.nio package has lots of goodies for working with bytes, https://docs.oracle.com/javase/8/docs/api/java/nio/package-summary.html

Cas Shun22:05:31

I experimented with ByteBuffer, but consuming a large byte-array with it efficiently kinda did my head in.

phronmophobic22:05:48

using a single Byte Array with offsets should work well enough

phronmophobic22:05:11

any more specific advice would probably require more info about your use case

krzyz22:05:06

Yeah I figured the allocations were taking up the most time.

Cas Shun22:05:07

Ok cool. Out of curiousity, what did you use to profile the code to find that?

phronmophobic22:05:09

time 😞

(defn read-mem-3
  "Returns an array of byte arrays of size available / frame-size"
  ^"[[B"
  [stream frame-size]
  (let [available (.available stream)
        frames (int (/ available frame-size))
        ma (time (make-array Byte/TYPE frames 6))]
    (time (dotimes [i frames]
            (.read ^InputStream stream ^bytes (aget ^"[[B" ma i))))
    ma))

phronmophobic22:05:21

I actually have a project open with com.clojure-goes-fast/clj-async-profiler setup, but since the results were in >10ms range, it seemed like time would be good enough

Cas Shun22:05:30

ok cool. I'm still learning these things. I'll investigate clj-async-profiler as well.

phronmophobic22:05:14

fyi, if you’re looking for tuning resources, http://clojure-goes-fast.com/ is pretty good. I’ve also used these projects in the past: • jvisualvm, https://visualvm.github.io/https://github.com/clojure-goes-fast/clj-async-profilerhttps://github.com/hugoduncan/criterium/

Cas Shun22:05:40

super cool, thanks

Cas Shun20:05:11

10mb file I'm using here. 1.4s is crazy. If I read the entire file into a single large byte array, it takes a fraction of a milisecond.

Cas Shun20:05:42

I would like some hints as to what I'm screwing up here. I've removed as much reflection as I saw, and this is my best result so far.