@andrewboltachev: just a tip, having an upload folder as part of your uberjar could be troublesome, I wouldn't do that

@andrewboltachev try to find the answer in the Java / JVM ecosystem, it will be the same for Clojure for this kind of topic

@andrewboltachev: https://ring-clojure.github.io/ring/ring.util.response.html#var-file-response could help maybe. Either as-is or by lookign at the source.

#C03RZGPG1 appears to have not seen my query, so I would like to see if I have some input as to where I should look to ask

It's a pretty simple question.

Now, assuming I have a vector of maps that are all of the same structure, what would be the best approach to processing that vector so that I can write to database? Ignoring the type of database

(Not looking for implementation details but more on logical and high-level thinking)

Well, depends on what sort of processing do you think you need

@trancehime: I know it sounds like @jaen is ducking the question but there are a number of options and it really depends on your use case. For example, is the vector likely to be large, does 'processing' (upsert?) the maps involve considerable latency, does the processing have to be 'synchronous', is it strictly transactional in nature....etc.

I know there's at least one use-case where the vector has 10 items in it

This is actually a wider architectural question.

Do you need every map in the vector to be 'processed' as one transaction? I.e. all succeed or all fail?

Basically yes

@agile_geek: well, that too, but I was thinking of the somewhat narrower architectural question - do you want to translate keys between db and Clojure, only change it to kebab case, or also change names, are you content with Java dates or want JodaTime. But those are good points as well.

my concerns are more in line with @agile_geek's points

I mean, there will be cases where I'm going to be working with a sizeable chunk of data at once

I have to admit to not having a huge amount of experience about doing this myself in Clojure (although huge amounts in other languages at scale).

I would guess that generically you need a 'transaction' to close over a doseq

@trancehime: is this a one time thing like importing something, or you expect to need to do bulk insert often?

Latter

Well, "often"

You would use something like doseq (or doall) for the reasons why to use these fn's instead of a 'for' or 'map' see http://stuartsierra.com/2015/08/25/clojure-donts-lazy-effects

I actually use doseq for a different operation that's basically initializing empty records to be filled in later on

(I figured this out through trial and error D:)

Yes. Although the lazyness of Clojure sequences often isn't discovered in experimentation with the REPL as the P (Print) tends to force realisation of the full sequence.

@trancehime: with postgres if you want to import a lot of data there's copy - http://www.postgresql.org/docs/current/interactive/sql-copy.html - you can just pass over some CSV as file and it'll import it. It's the fastest way, but might not be right for you, hence I was asking how often would you do that and what for.

purpose is data submission of statistics in some table which will be compared against similar records input by different peoples

Hmm, if the volume of those statistics is high

This might actually be a sensible choice

The downside is the file needs to be reachable for the DB

MySQL seems to support something similar - https://dev.mysql.com/doc/refman/5.1/en/load-data.html - since I think you mentioned you're usign MySQL

I think you'll find a 'bulk insert' operation in just about all RDBMS and for large datasets it should be your default option.

I mean it's kind of like a setup where the data input is occurring during observation of a given event

and also I have to provide a feedback to the user

Since your sample data spoiled you seem to be doing something like LoL match analysis - the question is would data come as things happen (someone is killed, someone gets the buff and so on), or would you import the match wholesale after it ended?

First of all, the user in this case would do data submission once the individual set/game is over

Ok, so I guess that's a point towards using an import mechanism like copy

Secondly, the idea is that multiple people will be submitting data for the same game and then there will be some checks made to see if the submitted data is the same

Now here's the caveat - this all has to happen within a period of time since the data submission

I see. I guess in that case you could have a table for submissions; when a submission happens you do a single insert to create a submission id to associate the data with then serialise your vector as a CSV file with rows having a foreign key to that submission row, and import it with the copy mechanism.

Then, and I suppose that depends on how you do the checking exactly

You could just select all valid rows from the db and use copy again to insert them wherever they are needed after you confirmed they are valid.

But that's just a rather rough guess on how that could be done

in Clojure can you have a mutable (transient) nested vector, or should I look at core.matrix?

But what if I have multiple people submitting data for different schedules at once, this means I'll just have a ton of CSV files wouldn't I?

@borkdude: I think I've read somewhere that transience isn't transitive if that's what you mean. That is with (transient [1 2 [3 4]]) only the outer vector is transient. But I'm not 100% sure.

I tried core.matrix, but it didn't quite do what I expected. When I googled, I saw someone had the same problem with set-selection!

(Anyway, submission only happens once. It's modifications after that)

@jaen: that's correct

@trancehime: hm, but would modifications be in bulk as well or not?

@jaen: I could of course use mutable arrays

It's essentially a "resubmission" so it might be bulk it might not be

@borkdude: hmm, what's wrong with set-selection!? I've been fighting with it today for Advent of Code, curious what you had problems with.

@jaen: Exactly what I wanted to use it for

Well, I can tell it worked out for me.

@jaen: I played around with it, like this (matrix/set-selection! m [[0 0] [4 4]] (matrix/new-matrix 5 5)) (where m is a :double-array matrix), but I keep getting: RuntimeException Inconsistent count of selection arguments

@trancehime: I see. Well yes, but after you copy the file into the database you can just get rid of those temporary files, so I don't think that's much of a problem?

@borkdude: well, the docs aren't entirely clear to be honest. The coords should be exploded I think for starters, that is (matrix/set-selection! m [0 0] [4 4] (matrix/new-matrix 5 5)) but I'm not sure if that solves everything.

@jaen Well I suppose I'll just figure something out. Thanks

@jaen: "RuntimeException Mismatched shapes in assign!" ... I guess I should read the source. The docs are in fact not clear.

@jaen: I hate to say it, but this is the kind of library where static typing would help 😉

@borkdude: here, have an example from my solution (that works) - (matrix/set-selection! m [[0 0] [4 4]] (matrix/new-matrix 5 5))

whoops

I suck at copying & pasting xD

(m/set-selection! grid coords-x coords-y 1)

Where

coords-x (range start-x (inc stop-x))
coords-y (range start-y (inc stop-y))

you basically have to specify all rows that are affected

ah, I see

I'll try that later then. thanks

I don't know if you can do something like you want with setting a matrix

No problem

@borkdude: (m/set-selection (m/zero-matrix 10 10) [4 5 6] [4 5 6] (m/fill (m/zero-matrix 3 3) 125)) seems to work

@jaen thanks

@jaen cool, it worked https://www.refheap.com/112401

@borkdude: nice, though I must say I'm surprised you could just assign a keyword like that.

you can assign any object there right

Maybe depends on the underlying implementation

if I understood correctly, ndarray stores objects of any type: https://github.com/mikera/core.matrix/wiki/Matrix-implementations

Yeah, I've used vectorz, it can only deal with doubles IIRC.

Yeah it is implementation dependent. There is an implementation that allows complex numbers, for example

@mikera: could I also initialize an nd-array with a default value?

Hmmmm sure.... you can use assign!

@mikera: oh, since you're here - how would you logically index a matrix to overwrite negative elements? I couldn't exactly figure this out and ended reshaping to 1D array and back to 2D, but I guess that's inefficient.

Or if you don't need to mutate it, you can broadcast a single-element NDArray to the shape you want

@mikera: I mean, now I have:

(defn start-state []
  (-> (m/new-matrix :ndarray 10 10)
      (set-grid! 0 0 9 9 :off)))

Right now I have something like:

(defn clamp-to-zero! [matrix]
  (let [linear (m/reshape matrix [(m/ecount matrix)])
        clamped (ms/set-sel! linear (ms/where-slice neg?) 0)]
    (m/reshape clamped [(m/row-count matrix) (m/column-count matrix)])))

just (assign! :off)

Clamping is best to use emax I think

sorry clojure.core.matrix.operators/max

(clojure.core.matrix.operators/max [-1 1 -10 10] 0) => [0 1 0 10]

@mikera: seems to work, thanks; I kind of couldn't get selectors to select negative elements in a 2D matrix for some reason.

The selector stuff is a bit tricky for sure. I was was thinking about replacing it with something more like specter

is there a thing like interleave that always exhausts the supplied colls? i.e. colls don’t have to be same length?

not great (filter identity (interleave (concat [:a 😛 :c] (repeat nil)) [:1 :2 :3 :4 :5]))

hi, all! please remind me if any convenient data structure does already exist in clojure libs for many-to-many relation? naive implementation is to keep two maps of sets and keep them in sync., but i'm sure that somebody have done better

I’m developing a library in the context of an app, how do I make clojure pick the code of the library instead of the installed jar (so I don’t have to re-install it every change I make)?

@pupeno: leiningen has a feature called checkouts. It means you have a checkouts folder where you link to your other project. You still have to install it once and declare it as dependency in your project.clj file: https://github.com/technomancy/leiningen/blob/master/doc/TUTORIAL.md#checkout-dependencies

Thanks.

why is this so terribly slow for a 1000x1000 matrix? it makes my Emacs become unresponsive:

(defn to-string [matrix]
  (let [row-to-str (fn [row]
                     (apply str (for [e row]
                                  (case e :on "*" " "))))]
    (string/join "\n"
                 (map row-to-str (m/to-nested-vectors matrix)))))

with a StringBuilder and doseq it's better, probably because 1000 vectors don't fit in memory easily? (it surprised me)

what's the easiest way to tell how much memory a data structure occupies in Clojure

I suppose it's rather has to do with the fact that Java strings are immutable, so you're generating a million of throwaway strings and copying the contents each time.

But that's just a hunch

could be yes

You're trying to visualise the lights exercise?

@jaen: yeah. Darn, my answer isn't correct, too low

I'm not sure if visualising will help, it's a big, big matrix after all

Does it work for the sample data?

I mean, the examples

@jaen yeah, just tried it for a smaller set, that worked, but big matrix is indeed too big to display

@jaen: let me see

Visualising a smaller matrix, like 10x10

Can be a good idea

And trying if that works well

turning on all lights works: (count-lights (apply process-line (start-state) (parse-line "turn on 0,0 through 999,999"))) equals a million

Question is how well does toggling work

It's probably the trickiest part

I actually went a bit of a clever way

And had 1 represent on, -1 represent off and I just multiplied by -1 to toggle.

this equals a 1000: (count-lights (apply process-line (start-state) (parse-line "toggle 0,0 through 999,0")))

I think that with a keyword representation it might be harder to flip it

hmm

not really, I'm doing it like this:

(defn swap-value! [matrix x y f]
  (let [v (m/mget matrix x y)
        v' (f v)]
    (m/mset! matrix x y v')))

but working with numbers would be a lot faster of course

since you can then use matrix multiplication

but that would still yield the same result

Can you make a 10x10 or so matrix and check if, say, turning on (1,1) to (5,5) and then toggling (0,0) to (4,4) and see if that equals IIRC 18

That should probably show you if toggle works ok quite fast

You could also display such a small matrix easily.

Well, yeah, I guess I just didn't want to flip cells one at a time subconsciously

this equals 999996: (count-lights (apply process-line (m/assign! (start-state) :on) (parse-line "turn off 499,499 through 500,500")))

That seems okay as well

@jaen: I could try to run over your input and see if I have a different answer than you?

I suppose there's no harm in that

@borkdude: catch - https://gist.github.com/jaen/da81e8cb77b7a04cf4a3

thanks

It's 569999 for the first part

I'm getting 501660

what do you mean with 'first part'?

Each exercise has two parts

I missed that at first, only noticed it after the first exercise

@jaen: I've only started this one today

On OS X, does anyone know how to get lein to load what I think is a java extension? There's an issue with MIDI on OS X and I've read that this library is required

http://www.humatic.de/htools/mmj.htm

I've placed it into /Library/Java/Extensions, but I'm getting an error about my classpath in the REPL, and I'm not sure how to instruct lein to load it

Caused by java.lang.UnsatisfiedLinkError no mmj in java.library.path

@jaen lol, I rewrote in Scala, now the number is too high

@jaen: ah got it right now, I counted the lights that were off, though. Very fast in Scala too btw

only 2 seconds

@borkdude: core.matrx + vectorz ~6secs for the first part

For me at least

@jaen: In Scala I'm using a nested mutable array of Int

@jaen: thus not thread safe, but I hope Santa will forgive me