The 4clojure site uses an old version of Clojure (1.4 maybe?), so likely there is some function you are using that doesn't exist in that version, or behaved differently somehow.

or perhaps it is the use of clojure.string/split ? I don't know for sure -- guessing what might be the difference.

My guess is that the NPE is from nil being passed to read-string. clojure.string/split is likely doing something funky.. manually binding ["D" "Q"] to s causes it to pass the first test.

Thanks I actually got it working by turning it into a direct lookup without all the string conversions

Created a map from char to string for both lookups and skipped the intermediate conversions, it worked and was much shorter.

It’s mostly performance related, and depends a lot on the kind of data and what you are planning to do with it. Of course it is best to have your db data match how you store data in your app (the less work to convert, the better), but again, you often need data in your app structured a particular way for performance reasons, and converting to/from db storage happens less frequently than data access and manipulation within your app. Just as one example, consider your flat/nested difference. If I am working in my app and I want to access a collection of people by ID or name, I could filter over the collection by name, but that is an O(n) operation. If it is just a few names and doesn’t happen frequently, who cares, but I may also want to structure it like this {id person id person}, in which case accessing an individual person by ID becomes an O(1) operation. Again it depends on the size of your collection and how frequently the operation is done, whether or not something like this matters. Then you have the tradeoff of being able to outsource processing to clients, thereby reducing load on your server, vs fluidity of your app for the user. There are no simple answers, just a lot of tradeoffs to consider. I should also mention that choosing the right data representation can either simplify or complicate your code, so that's another thing to look out for. You'll build up an intuition about it over time. Particularly with re-frame, you'll notice if your db accessors are very long or complicated and hard to read, that's probably because your data could be structured better.

Thanks for your reply. Performance won't be an issue, so I'll try the 'flat' structure and see how it works out 🙂

Even if you're not using fulcro, the db model used there is where you'll likely end up after much iterating and headaches: http://book.fulcrologic.com/#_fulcro_client_database

I have a .cljc data-model file that uses clojure specs to define my data contract and then I use that on the client and server. combined with guardrails which lets you define fn specs inline https://github.com/fulcrologic/guardrails you can iterate very quickly on a data model and then add some light type checking along the way

there's been some recent exploration using fulcro just for the normalized db and re-frame for ui layer: https://www.youtube.com/watch?v=ng-wxe0PBEg

So if I understand correctly, you can store data in graph style (similar to crux and datomic), but than you can also use and manipulate the data, as in a nested structure?

data is stored normalized on the client - so all updates are 3-levels deep: (assoc-in state-db [:users/id :my-user :name] "new username") at render time the fulcro db->tree takes a query and a state db and returns a tree of data that you can pass to your components. so you don't have to update nested structures, but you get to render with nested structures, the best of both worlds

I notice they use 'pathom' https://github.com/wilkerlucio/pathom but doing this without fulcro does not seem so easy to implement. I like the concept though, I'll keep it in mind for later. Thx for sharing this

If you look on the reference docs page for Java interop here: https://clojure.org/reference/java_interop

I believe (. Byte MAX_VALUE) falls into the category of (. Classname-symbol member-symbol)

If it is very important for you to be able to do something like that code and have it work as you wish, you can use the Java reflection API explicitly

Thanks!

Don't use available, read in the loop until read returns -1

how do I create the array to hold the data then?

just using a BufferedInputStream gives me a >10x speed boost on my laptop:

(time 
 (let [as (BufferedInputStream. (FileInputStream. "testfile_10MB")) ]
   (read-mem-3 ^BufferedInputStream as 6)))

The buffers inputstream speedup is showing that "frame-size" of 6 is the example is absurdly small

A buffered inputstream will read in larger chunks at once, and then trickle them out to you, so you can achieve the same effect using a larger buffer directly

You can use something more flexible that you don't need to allocate all at once to hold your byte arrays

Like a clojure vector or Java arraylist

the downstream callee wants the data as an array of byte-arrays

Then I would use the length of the file

Which is not what available on an inputstream means

hmm, ok. When I get back to my computer I'll investigate this further then.

available always returned the file length, so I found that confusing

Regardless, even if I remove available and use a long directly, it doesn't speed anything up. I tried that before.

I imagine most the time it does

Right, because 6 is an absurdly small buffer size for reading

So use a buffered inputstream that manages a larger buffer for you, or use a buffer in the mb range (I like to start with 5mb see from there)

So I should use BufferedInputStream or read it all at once, and then what would be the fast way to split it into an array of byte-arrays of length 6?

Make byte arrays and copy the data into it

ok, I'll explore that. Probably back with more. Thank you for your patience and help

arrays are fairly inflexible, so you can't really turn a byte array of size 10 into 5 byte arrays of size 2 without making new arrays and copying

Which is why things like nio's Bytebuffer and netty's Bytebuf which are similar to what other langues sometimes call slices

That's what I figure. This seems to be what I get for learning clojure without having written a line of java in my life

coarsely, just (make-array Byte/TYPE frames 6) accounts for about 2/3 of the execution time on my laptop

so as hiredman noted, making a ton of 6 byte buffers is definitely not the most efficient approach

not sure if there’s a quicker way to allocate a 2d byte array from clojure

So if I wanted to rewrite this callee (which wants a 2d byte-array) myself, what would be the "fast" way to process a large byte-array in chunks myself?

you can just use a single byte array and use offsets

generally, the java.nio package has lots of goodies for working with bytes, https://docs.oracle.com/javase/8/docs/api/java/nio/package-summary.html

I experimented with ByteBuffer, but consuming a large byte-array with it efficiently kinda did my head in.

using a single Byte Array with offsets should work well enough

any more specific advice would probably require more info about your use case

Yeah I figured the allocations were taking up the most time.

Ok cool. Out of curiousity, what did you use to profile the code to find that?

time 😞

(defn read-mem-3
  "Returns an array of byte arrays of size available / frame-size"
  ^"[[B"
  [stream frame-size]
  (let [available (.available stream)
        frames (int (/ available frame-size))
        ma (time (make-array Byte/TYPE frames 6))]
    (time (dotimes [i frames]
            (.read ^InputStream stream ^bytes (aget ^"[[B" ma i))))
    ma))

I actually have a project open with com.clojure-goes-fast/clj-async-profiler setup, but since the results were in >10ms range, it seemed like time would be good enough

ok cool. I'm still learning these things. I'll investigate clj-async-profiler as well.

thank you

fyi, if you’re looking for tuning resources, http://clojure-goes-fast.com/ is pretty good. I’ve also used these projects in the past: • jvisualvm, https://visualvm.github.io/ • https://github.com/clojure-goes-fast/clj-async-profiler • https://github.com/hugoduncan/criterium/

super cool, thanks