Welcome to #notepad_plus_plus Channel (notepad++ for Clojure)

@ghadi thats great, bookmarked!

anyone here using thinktopic/cortex ? I'm looking for a deep learning library that I can train in java land, and this looks the closest

I've played with it a while back. Likely to still be pretty alpha. Pretty awesome though

You'll have more tested/documented solutions in java though. I believe there's Java bindings for TensorFlow now.

aaaaand, it looks like there's a wrapper already: https://github.com/kieranbrowne/clojure-tensorflow-interop

@john: are those bindings for training + inference or inference only? last I checked, java is only calable of inference and not training because for training, we need to compute the gradient via backprop .... but doing so in TF requires some hack of attaching some python code to nodes or something, so it ends u ptaht java bindings can only eval ^^ this is just random stuff I read on the internet, so it may be blatantly false

Not sure, but yea, I heard the java bindings aren't feature complete yet either

support5ing this theory is the fact that the java example https://www.tensorflow.org/versions/master/install/install_java only does inference, and not training (it loads a model someone else trained)

right

I'd definitely play around with cortex. My preference is always pure clj/cljs when it's an option.

come to think of it, all I really need is a fast tensor library on cuda with splicing support

yeah, I tried tensorflow in python, missed lisp

then I tried tensorflow + Hy ... which was nice, but just didn't quite feel like clojure

right. Not sure if it's possible, but it'd be neat if Dragan Djuric's Neanderthal (http://dragan.rocks/articles/17/Neanderthal-090-released-Clojure-high-performance-computing) could be helpful to cortex. Seems like some pretty interesting stuff he's doing there.

They're already using cuda, but my understanding is that Dragan has really smoothed over all the hardware/software platforms.

When (swap! some-map-atom assoc ... is run in another thread, is the new structure stored local to that thread, with a pointer back to the structure back in the main thread? Or is all the structure stored back in the main thread?

threads don't store data

I was wondering that. "thread local storage" was throwing me off

well, not atoms at least - the jvm has no stack storage of objects

I was reading that vars have something called "thread local storage"

atoms are not vars

vars can have thread local bindings, in that case it's possible to have thread local binding of a var containing an atom

but the atom is never thread-local, it's the var that is

a var (in clj on the jvm) is a mutable container owned by a namespace

Gotcha

if a namespace defines an atom, the atom is in the heap, referenced in the var in the namespace - changes to the var (eg. giving it a different atom) can happen thread locally, changes to the atom are synchronized across all threads

Ahhh, okay. I'm ruminating over the possibility of a distributed persistent data structure. It would end up being a structure with pointers to potentially many different locations. Sounds highly inefficient.

Resolving any particular mutation would potentially update multiple locations. And coordinating that sounds hairy.

But it definitely could be more efficient than distributed non-persistent data structures.

But then computations over that structure would also have to be performed in a distributed fashion. Otherwise you'd have to materialize the whole structure locally. Or at least all the nodes you plan on mutating.

Which I guess would be the way to do it. Lazily materialize only the nodes you need. Then submit yourself as the new owner of the head to who ever the root owner is, coordinating the pointers.

you could look at eg. zookeeper, (not only how it does things, but it's features and limitations due to being distributed)

there's avout, which uses zookeeper, and almost acts like clojure atoms / refs / agents

oh nice

yessss. This is what I was wanting.

I'm writing a DSL that compiles to cuda.

I need to come up with a good name for my DSL.

(the relation to #clojure is that the compiler for my DSl is written in clojure and I'm using jcuda)

and the entire compiler is written as a series of nested core.logic/match

phalanx = good name, all threads act in unison, like soliders in a phalanx 🙂

hhhm, looks like avout does distributed MVCC, but I think each version is a full copy. I don't think avout is splitting the structure's internal nodes across the net. Still reviewing.

@qqq what other names are you thinking of?

best way to take something like this: ["a" [1 2] [3 4] [5 6] "b" [99 1] "c" [32 43] [89 76] [12 13]] and get this: {"a" [[1 2] [3 4] [5 6]], "b" [[99 1]], "c" [[32 43] [89 76] [12 13]]}?

IOW, if i encounter a string, I want to assoc it with all the vectors following it in the sequence

nevermind, figured it out, but i would be open to other suggestions

i'm rusty!

i am trying to do queue with async, but it confuse me. I have queue and i want get from this queue max 20 items on each iteration (when 10 items i want get 10 items and do job, not wait for 20). How to do it in most simple way?

wound up just using something like (reduce (fn [[m current-string] e] (if (string? e) [(assoc m e []) e] (update m current-string conj e) current-string])) [{} nil] ["a" [1 2] [3 4] "b" [89 23] [43 54] [23 12]])

ugh

(map #(if (string? (first %)) (first %) %) (map vec (partition-by type ["a" [1 2] [3 4] "b" [99 1]])))

@john oh, right, good call on partition-by

thanks

np. Might be an interesting exercise for a transducer too

https://twitter.com/danlebrero/status/862605376168095744

@john I wonder if there's a cleaner way to do handle those multiple calls to first

if-let

no

that'd return a boolean

(map (fn [[x :as e]]
       (if (string? x)
         x
         e))
     (partition-by type ["a" [1 2] [3 4] "b" [89 23] [43 54] [23 12]]))

ah

nice

haven't messed with transducers much, but this might be a transducer: (into [] (map (fn [[x :as e]] (if (string? e) x e))) (map vec (partition-by type ["a" [1 2] [3 4] "b" [99 1]])))

ew

think you still need a first in there

Yeah, I'm not sure if that binding form is doing what you wanted

Better transducer version: (into [] (comp (map vec) (map #(if (string? (first %)) (first %) %))) (partition-by type ["a" [1 2] [3 4] "b" [99 1]]))

Even betterer: (into [] (comp (partition-by type) (map vec) (map #(if (string? (first %)) (first %) %))) ["a" [1 2] [3 4] "b" [99 1]])

I need to start using transducers 🙂

@john cool, tbh, i have never really used them, and i've been using clojure for something like 7 years?

i've found that i use about 1/3rd of clojure 90% of the time

same here. But that last version ought to perform better, for speed and memory. And it doesn't look unreadable either.

@john i was doing some dumb hobby stuff, so i'm not focused on perf. though I do often focus (sometimes unnecessarily) on that kind of thing at work

im moving back to an individual contributor role from a management role in a week's time, and i am trying to get back into writing some more code

nice

get your hands dirty 🙂

@john indeed. i forgot how much i missed it until a couple weeks ago

that said, management pain or programming pain, both are inevitable in my experience 🙂

@john : no other names, I think it kind of makes sense since cuda is all about multipe threads doing the same thing on different data

I prefer programming pain. But maybe that's because I've never worked in a dev shop 🙂

I've done mgmt though. Computers are far more predictable than people.

@qqq yeah, I don't seen any other clojure phalanx projects out there. Good name.

@john so here's a query for you: which of the above do you think is easiest to read?

for my money i sort of prefer the explicit reduce, if for nothing else the names

i wonder if there's a way to do this with reductions...

I prefer the map version, but a reductions version would probably be pretty slick

I have queue and i want get from this queue max 20 items on each iteration (when 10 items i want get 10 items and do job, not wait for 20). How to do it in most simple way? After each iteration i want do X seconds of break. What would you use to code it? I was trying with async, but it is maybe not the right choice.

@kwladyka I'd partition it

@john if you use partition-all with chan it works almost like that. It iterate after 20 items, but i want do job even with 10.

it is harder than it looks 🙂

eagerly take as many items as possible first. Then partition it. The last partition should be (could be) less than 20.

You need some kind of way to detect that the queue is done or it's waiting

maybe a nil, or close the channel

whats the source? do you need a chan?

i need a queue with conditions which i described

i tried with async, but i failed

but you say you might not need a chan. can you control the source?

yes

whatever solution i will make, but with this conditions

i can use something else than async

Why not use partition-all without the chan? I missed the conditions you were talking about.

in what way? <! will freeze it when less than 20 items

or you mean without async? But then how i will add things to queue and get from qeueue

to avoid additional side-effects

well, in clj, you could have a thread constantly consuming the queue

not such a great idea in cljs

sure, it is what async do

and it is easy to get from queue parts by 20

but hard to get 20 or less - i don’t know how to do it

My core.async-foo is not that great. I'd look for a function that allows you to detect whether you're in a waiting condition. Perhaps checking for a nil on the take would be enough.

nil is when channel is closed, but i don’t want close queue 🙂

i was trying with channel consuming another channel, but still i failed

Yeah, you tried sending a chan in the chan, and then closing the inner chan after each put?

(def queue (chan))
(def queue-portion (async/chan 100 (partition-all 20)))
(async/pipe queue queue-portion)

and it is almost what i want, unless it takes 20, not 20 or less

You mean it hangs on less than 20?

yes

Are you closing queue-portion after each set of puts?

i have to close queue to “unhang” it

in what way? I can close it only after it will get 20 items, because only then it will unfreeze

so it doesn’t solve the problem

maybe i can add next chan with timeout

so chan > chan > chan heh

I'm saying, make a new queue-portion for each new set of items you want to send. Have a receiver that receives new queue-portions (chan). On the producer side, dump your data into the newly created and sent queue-portion, then close it. On the receive side, pass the new queue-portion to a function that consumes it. Which would close when the producer closes it.

Though, again, my core.async-foo is not great, so that may not be the idiomatic solution. Sounds like a really simple problem that probably has a really simple solution in core.async.

Might want to wait a bit and ask again when more folks are around.

it sounds easy, but code it is not so obviously 🙂

but probably it has to be solved by chan > chan with timeout > chan with parition-all

it could work

but it is a little complex i would say

I really don't think you need a timeout

i need some kind of timeout to not wait to fill 20 items

It'd probably work. But it feels wrong

Unless you don't care about potentially unnecessary wait times.

Do that a few thousand times though and those timeouts may get nasty.

wait between iteration is my intention

oh

Yeah, if you're doing pauses anyway...

like i wrote on the beginning:

I have queue and i want get from this queue max 20 items on each iteration (when 10 items i want get 10 items and do job, not wait for 20).  How to do it in most simple way? After each iteration i want do X seconds break.

but thank so much for try to help 🙂

np. Good luck!

hey y'all! i just launched a website for building git-based programming tutorials. the free one is on Clojure basics, so if anyone would like to try it out i'd love hear feedback: https://gitorials.com/id/1/

Does mapv force lazy seqs? I forget.

yes

Kinda wish that the (filter|map)v fns said so in the docstring

in the same way that do* fns are explicit about it

err, maybe those docstrings don't after all

Well, the map docstring explicitly states it returns a lazyseq

and mapv says vector.

And you don't get a vec out of seq without realizing it, afaik

I suppose it could be implemented lazily though :thinking_face:

@sekao looks awesome!

oh, nice

Interested in making one? Login with a GitLab account to get started. All gitorials are $3, and you get 70% of every sale.

I want to specify some function(s) via configuration/EDN for my program to use. The functions would be defined in the program. Could I define them as symbols in the EDN and somehow resolve/invoke them at runtime?