If your builds are tied to commits, don't you already know all the inputs, just from the source tree?

Of course it is, but we're using monorepo, and :local/root deps are used quite a bit, so I'd like to discover it based on deps.edn.

How exactly do you produce a build?

We are using GitHub and the push trigger in actions cause builds and deploys the main branch. The repository is Polylith based and uberjar is created by https://github.com/polyfy/polylith/blob/master/examples/doc-example/build.clj script.

Polylith can tell you which projects need building so you only have to build what needs to be deployed. Check http://corfield.org - I've blogged about how to do this. I'm on my phone right now.

that's fantastic. I am going to find the post right now

See part 4 of https://corfield.org/blog/2022/12/07/deps-edn-monorepo-10/

If that doesn't give you enough detail, ping me tomorrow when I'm at work and I can give you more details

It seems that even without relying on Polylith it shouldn't be hard to figure out whether a new uberjar will be different from the old one. AFAICT, that build script just uses the top-level :paths provided in your deps.edn and doesn't copy over any resources. If that's correct, you can simply check whether the stuff in contained in the directories referred to by :paths has changed or not.

It's not 100% accurate of course. You can have some x.clj and never require it - it won't affect the uberjar (if your build script is indeed that script) but you wouldn't be able to tell it. You can also potentially have some macros that do crazy stuff during compilation. Though I can't see how Polylith could possibly help in those scenarios either.

@U04V70XH6 that was helpful enough. Polylith & seancorfield solved my problem again. 🙇

@U2FRKM4TW paths is only a small part of how Polylith projects are built - there's all the transitive local deps too. Polylith knows which have changed and which haven't.

Right, good point. It's :paths all the way down. :)

Maybe this? https://github.com/borkdude/speculative

No - the core macros all have spec enabled for them - let and friends and this was done by the core team I think. Where I am coming from is that compiling ham-fisted's main api namespace is 1200ms with spec enabled and 84ms with it disabled.

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L305

https://github.com/clojure/core.specs.alpha

Thinking aloud - sure, macro specs are slow, spec implementations are also slow-ish Can they be made faster?

Well yes but specifically for macro checking like what is in core.specs.alpha (specifically let and fn destructuring) it is just smarter to move the check into the macro expansion code itself. Spec's implementation could be made faster especially if we want to generate bytecode but it won't be free. The compiler already has a number of passes - the lisp reader which can be significantly faster, the analysis phase and the emit phase. All of these can be quicker.

Don't threaten me with a good time 😁

Even without bytecode generation, I believe spec could be faster, malli is proof of a way lower bound atm

Right I was thinking honestly the best way to get spec faster is convert it to malli 🙂.

At least start there. Then you can file issue with ikitommi and they will get addressed and you don't have to do it yourself.

You mean port spec forms to malli?

yes - on another note here is an implementation of clojure.core/map that is faster in all non-chunking situations especially in higher arities where you are looking at 10x or more - https://github.com/cnuernber/clojure/blob/faster-map/src/clj/clojure/core.clj

> Don't have to do it yourself > _Looks at all my merged MRs to malli Oh well_

haha - you are a champ for doing the work though.

Also faster every?, some, and filter (in nonchunking case).

If you add in chunking to the above map implementations in all arities then it really gets a bit faster but the implementation is a lot longer - https://github.com/cnuernber/ham-fisted/blob/master/src/ham_fisted/lazy_caching.clj#L15

In higher arities that makes the seq-based map implementation equivalent to the lazy-noncaching implementation in ham-fisted which completely surprised me.

In the malli utils you'll find hidden versions of every and some which unroll their args

I thought about every? as a macro if the input is a persistent vector.

But I need to investigate how your implementation gets these speedups

Typehinting direct calls to ISeq so you aren't calling RT/seq, RT.first, RT.next all over the place. This allows hotspot to inline the call.

simple. Could have been done long ago.

> I thought about every? as a macro if the input is a persistent vector. > That's most of clj-fast

clj-fast has a lot of good stuff.

Passing through RT is important though for dynamism Although you can get many speedups by specializing earlier, which is sort of what you did

Right I don't bypass RT - I just call 'seq' exactly once so I am not sacrificing any dynamisim that I can see.

https://github.com/metosin/malli/blob/master/src/malli/impl/util.cljc#L42

(deftype TransformedCountedSeq [qseq-result transform-fn]
  clojure.lang.Seqable
  (seq [_] (map transform-fn qseq-result))
  clojure.lang.Counted
  (count [_] (count qseq-result)))

(def out (java.io.ByteArrayOutputStream. 2000))
(def w (cognitect.transit/writer out :json {}))
(cognitect.transit/write w (TransformedCountedSeq. [1 2 3] inc))
(ex-message *e);;=> "java.lang.NullPointerException: Cannot invoke \"com.cognitect.transit.WriteHandler.tag(Object)\" because \"h\" is null"

Your need to specify a handler for your type, transit doesn't know you want to encode it via transforming it into a seq

That is not a custom sequence, that is a custom seqable

A custom seq implements ISeq

would it work if I implement ISeq?

It would work if you implement java.util.List since there's a default writer for that.

Not sure, depends if transit has a built in handler for ISeq

It wouldn't surprise me if it didn't though

right, saw the java.util.List writer. Does implementing that make sense or does that come with other downsides?

The downside is that your custom type will be serialized as if it were a plain array. If you want to be able to reconstruct that type, you should go with a custom handler. If not, I'd just implement java.util.List.

hmm, thinking I might want to implement clojure.lang.LazySeq this would imply java.util.List , right?

I’d like my type to behave like the lazy seq that (map transform-fn qseq-result) returns, only also be counted?

clojure.lang.LazySeq is a final class. What do you mean by "implementing" it?

I see, not what I want then 🙃

I'd probably create a class that implements/extends exactly the same things as the lazy seq class, plus the counted interface, and then delegates everything but the count to the wrapped instance of lazy seq.

I’ll try that, so ISeq, Sequential, List, IPending, IHashEq thanks @U2FRKM4TW!

PersistentList is a counted seq

And I think transit handles them as Lists (they get read back as ArrayList)

@U0NCTKEV8 are you suggesting I can drop my custom type in favor of PersistentList?

but this constructor isn’t public? https://github.com/clojure/clojure/blob/08a2d9bdd013143a87e50fa82e9740e2ab4ee2c2/src/jvm/clojure/lang/PersistentList.java#L79C37-L79C51

https://github.com/cnuernber/ham-fisted/blob/master/java/ham_fisted/IMutList.java

Just implement size and get and everything else works

Which is how the lazy noncaching version of map works

@UDRJMEFSN thanks!

Or yes, just use persistentlist

That is the type that clojure.core/list creates, is the type the reader usually produces for things in parens

The way to convert a lazy sequence (output of map) to a persistentlist is apply list

I assume that the desire goes beyond the question and it's not only about Transit but also about having a custom counted lazy seq. In that case, just creating a list isn't sufficient.

It isn't going to be lazy if you are serializing it

But it will be if the type is used elsewhere, not just for serialization, as the previous thread by the OP seems to imply. Otherwise, one could just use mapv or sequence + map instead of map.

Probably #C05006WDW or #C06B40HMY

https://clojure.org/community/training

#C5FBP6A5V also has a link to contact info http://www.clojureconsultants.org/

thank you all