Can you share the code that reads from file?

I have only tried something like this but then I would have to do a new line-seq on the reader and read the entire file again.

(with-open [rdr (io/reader "file.log")]
    (doseq [line (line-seq rdr)]
           (line-handler line)))

I'd search online for something like "tail -f" in java. Or, if your program is intended to be running on *nix, I'd simply pipe the output of tail -f into it - no need to reinvent the wheel.

(require '[ :as io])
(import '[ BufferedReader])

(with-open [^BufferedReader rdr (io/reader (io/file "/tmp/log.txt"))]
  (loop []
    (let [line (.readLine rdr)]
      (case line
        ;; Nothing is ready to read yet, wait a second and continue
        nil (do (Thread/sleep 1000)
                (recur))
        ;; Handle any stop reading condition
        "stop" (prn line)
        ;; Overwise, handle line and continue the loop
        (do (prn line)
            (recur))))))

There is also https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/input/Tailer.html which is exact what @U2FRKM4TW suggested: "tail -f" implementation in java

Yes, this solves the issue that I had. Thanks for the suggestions.

Make a data reader

(including a printer)

Oh, duh. Thanks!

Er - I may have misunderstood both the question and the answer, but - isn't it a matter of, what do you want computed at compile time, vs run time, and if the best you can do at compile time is emit a readable thing which will execute that .newXpath constructor at run time, then is it very important to use a macro?

That's the point - the best thing is better than emitting (.compile ...). It's actually calling that before run time.

if you embed an object in code, it is recreated at code load time

For a bit more detail, I want to encrypt and/or sign a browser-side payload so it can be verified on the server. Symmetric encryption would expose the key. Asymmetric signing also exposes a private key. It seems the only option is encryption, where the private key is on the server, and the public key is in the browser. However, I cannot find any clojure libraries that will do just that: decrypt the payload without signing or using any other ciphers.

that all seems the wrong way around

for asymmetric encryption the private key is used to encrypt or sign

I actually think I might ditch it all and use ECDSA… it works in SubtleCrypto and in Buddy

but for signing, you should be able to sign with a private key and verify with a public key

Yes, I’m back to where I started: exposing a private key in the browser that is signing/sending the data

> It seems the only option is encryption, where the private key is on the server, and the public key is in the browser. this seems like the reverse of what you are saying you want, but reads like that is what you think your only option is

Well, it’s ok to use a public key in the browser, right? So the only option is encryption. Then the server decrypts with the private key.

It seems ‘incorrect’ to use asymmetric keys to encrypt the payload itself. That’s how the libraries make it seem.

https://stackoverflow.com/a/59678340

This is promising, let me take a look, thanks…

looks like you can specify the hash option in either on the js side or the java crypto side, and you just have to make sure they match

ok here goes..

If you care about when things happen, don't use lazy sequences. Even if it does work, it's not guaranteed to continue to work and is brittle.

I know. But in this particular case, I would like to understand why the whole collection is traversed.

Debugging it is rather painful, especially given that -Dclojure.compiler.disable-locals-clearing=true doesn't seem to work for me (assuming it's useful at all without an explicit call to compile).

Other than curiosity, is there a particular reason for trying to learn the implementation details of the lazy sequence?

Uhm... maybe? Why does it matter?

Just wondering if there's an X/Y question here and maybe there's an easier way to solve your problem.

No, no XY question. It's not my first post on this server. ;) I'm applying the "XY heuristic" to my own questions before I post them.

So it doesn't seem to be 32 to the power of 2, 3, and even 4. And a plain

(first (sequence (map (fn [i]
                          (println "item" i)
                          i))
                   (range 1000)))

pretty sure that it has something to do with the mapcat:

(first (sequence (mapcat (fn [i]
                           (println "item" i)
                           i))
                 [(range 1000)]))

Transducers are push, lazy chunking is pull. It's not possible to pause a transducing step, therefore the full range has to be traversed. The only way to get a partial result from expanding transducers like mapcat is to use early reduction. This will print once :

(let [get-first (fn [xform coll]
                  (reduce (comp reduced {}) nil
                    (eduction xform
                      (map (fn [t]
                             (println "item" t)
                             t)) coll)))
      data [{}
            {:a []}
            {:a [{:b []} {:b [1 2 3]}]}
            {:a [{:b (range 1000)}]}]]
  (get-first (comp (mapcat :a)
               (mapcat :b))
    data))

I'm not sure how to reconcile your first two statements with the fact that a simple map in my last code block doesn't result in printing of more than 32 items.

The simple map doesn't use cat. 🐈

map is not expanding, each transducing step produces a single item

what version of Clojure are you using?

particularly, is it 1.12.0-alpha1

1.11.1

Tried on 1.12.0-alpha1 - same behavior.

@U7RJTCH6J It does work correctly though, at least the way I understand it. mapcat chunks over the collection that it processes (I think), so [(range 1000)] prints out the whole range, because there's just one item in the outer collection, while (map vector (range 1000)) prints out just 32 single-element vectors. That's why I was checking for powers of 32, because I have a combination of two mapcats.

@U053XQP4S Your (comp reduced {}) broke my mind a bit, worthy of code golfing. :)

without digging into the details of this code, what is (comp reduced {}) doing in there?

is {} a short-cut for a 1-and-2 arity fn that always returns nil?

The 2-arity always returns its second argument. The 1-arity isn't used by reduce but yeah, would return nil.

oh right

@U2FRKM4TW I don't know why you think the transducer form of mapcat chunks. You can see in the implementation of cat that it immediately reduces over each sequence it sees.

Well, I was thinking that because I had a wrong mental model of cat. :) Thanks for pointing me in the right direction.

In such cases, I don't use profiles and instead use simple config vars. Usually as env vars. I've seen other people implement custom aero reader tags that use more intricate logic but with the same end result.

Shouldn't you use derive in that case? https://clojuredocs.org/clojure.core/derive

@U2FRKM4TW Yeah in other codebases we’ve implemented config value overrides that apply custom per-test values after the Aero config is read in. But that’s always been kind of a hacky solution.

It certainly beats massive config map duplication though

@UK0810AQ2 Aero does not rely on keyword hierarchies. It's a simple get when it comes to profiles.

@U02FU7RMG8M It should be possible to implement your own profile-like Aero reader.

hm, bummer. Although you could add your own tag, but that's not trivial

Could you resolve that with a #ref for the single nested config var, or am I misunderstanding?

@UK0810AQ2 It's actually very trivial. :)

(defmethod aero.core/eval-tagged-literal 'instrumentation
  [tl opts env ks]
  (aero.alpha.core/expand-case (:instrumentation opts) tl opts env ks))

@U2FRKM4TW you know the trick of posting wrong answered to nerd snipe someone to post the correct answer? 🙃

God dammit.

Just a note, but a pattern I’ve found useful for situations where you have different environments is to consider each EDN document to be a profile, and to meta-merge them together in a profile chain. This means you can keep things DRY by having the latter profiles easily override definitions in the earlier profiles. It’s https://github.com/lambdaisland/kaocha/pull/128/commits/0403b58b2e0467526a766449772720b2836b9579 to add a #meta-merge tagged reader to aero too, allowing you to write profile chains like:

#meta-merge [#include "base.edn" #include "test.edn"]

https://github.com/weavejester/meta-merge

So we ultimately went with the post-config-read override since we didn’t want to get too fancy for a feature that’s only used in testing

It’s basically a post-read, ad-hoc version of meta-merge with the ^:displace flag

If you need more predictable control over parallel resource usage then I'd recommend using the claypoole library, which makes it easy to limit a parallel operation to a fixed size threadpool

I think the idea with pmap is that it wants to saturate the available CPU. My guess is that 100 items isn’t enough to do that, but a larger sequence is.

you can search this slack to find critiques of pmap, in particular suggestions that pmap is not great for parallel io, which is often what people that ask about pmap are trying to do with it

That makes sense, i think in this case i'll be happier with regular old map!

try async/pipeline if you want to control parallelism if pmap not satisfy you.

and if for some reason you need the most detailed way of control you can do something like that to create each worker and control it.

(dotimes [_ pool-size]
      (async/go-loop []
        (when-let [f (async/<! fns-chan)]
          (try
            (f)
            (catch Exception e
              (l/error e "sync-db go-loop worker" )))
          (recur))))

this 3 options (pmap, go-loop, pipeline) should be all what you need

just choose one which fit into your needs

oh and pmap doesn’t use all CPU, it use constant number of parallelism based on your CPU. I don’t remember the pattern but let’s say it is number of cores / 4 +2 or something like that.

what I usually do is: 1) pmap to prove of work and it can be all what you need depends what you do 2) if not satisfy me, then pipeline 3) if I need more control, than I create workers with go-loop I don’t think I have ever needed anything else, than above

thanks @U0WL6FA77 That makes sense, thanks a lot for the advice 🙂

javax.crypto.spec.PSource$PSpecified/DEFAULT inner classes take a dollar sign https://docs.oracle.com/en/java/javase/19/docs/api/java.base/javax/crypto/spec/PSource.html

Excellent! That was it, thanks!

Quick side question: This type of reference requires the fully specified class name, right? I cannot just require it as PSource and then refer to PSource$PSpecified? I’m getting compilation errors when I do, but having that entire class name looks kludgy.