Clojurians Log v2

ah ok, I’ve “fixed” it locally anyway

was thinking about with-resources, if a setup throws, the body isn’t run and the setup itself would need to cleanup any incomplete state

try/finally in with-resources guards body so that teardown is run

was thinking about #26

now that testing was mentioned 🙂

rearranged things a bit, but overall would be neat to be able to run all tests against local and docker, both 1.5 and 2.1

I broke the build 😕

hum, didn’t find a way to see build log

ah, had “my builds” button ticked so didn’t see any at https://circleci.com/gh/HCADatalab/powderkeg

gah this test fixtures

something like “lein run-tests-in-docker” but when locally in repl, use local-spark

realizing that this would fail on a remote cluster https://github.com/viesti/powderkeg/blob/sql/test/powderkeg/sql_test.clj, either two deftests with different name, one with ^:integration meta and different setup or another way of saying the same

to actually run remotely that is

can you provide more context on wy it would fail?

this .collect is begging for into support 🙂

the spec registry

yep

ah stupid me

Several suggestions: • keep an eye on all atoms transferred and if changed at next barrier, update them (WeakRef ftw), is it overkill? Is it going to create more bugs than it fixes?

• have a whitelist, initially populated with common atoms to migrate

• no more ideas

last one :D

second one sounds reasonable

The first suggestion is my plan for multimethods

usually a worker is not going to change a multimethod

while it may change an atom

and ruining caches stored in atoms at each barrier sounds mean (”hey replace your nice cache that you worked hard to populate with this empty one from this lazy driver")

true :)

keeping distributed execution obvious but simple would be neat

huh? what do you have in mind?

just that I've made similar mistakes in Spark with Scala without realizing where code is executed :)

hmm, actually spec registry might not be a problem with DataSet, at least in the that I made, since a DataSet is returned to the driver, so specs themselves aren’t used by the workers

True