There's a Clojureverse thread about it. I think this particular post very useful: https://clojureverse.org/t/best-library-for-querying-html/1103/18

jsoup is really good

(if you don't mind doing interop)

yeah, i was searching for clojure library but there's no good. I'm unfamiliar with java ecosystem, so thanks for the suggestion, I'll try it out!

yep, jsoup

enlive uses jsoup doesn't it? https://github.com/cgrand/enlive

What about Hickory?

We use Jsoup (cleaning) + hickory (traversing, parsing, converting to other formats) - both are great

what makes it mutable though? I was testing the idea with

(defn uuid-seq
  []
  (lazy-seq
    (cons (str (UUID/randomUUID))
          (uuid-seq))))

(def us (uuid-seq))

(take 10 us)

its not mutable. so recurring and doing all the mapping and taking 1 will get the same result from your queue-items

ok really dense question then but what makes a mutable lazy sequence then?

you can't make a clojure sequence mutable. you can look at java.util types like blocking queue and some others. this can block waiting on an item. or you can look at core.async channels

k guess async it is then

probably now as good as ever to wrap my head around it

as to the dedicated thread stuff, i don't think its true you need two dedicated threads. using async will use whatever thread is available in the async pool i believe. but you certainly need two concurrent threads of execution. you have a loop just watching the queue. if you don't have two threads i don't see how you would keep watching the queue and also populate the queue but with two threads of execution

the idea was something like have the "queue" be a producer that yields when someone takes from it

Core.async is a big jump. PersistentQueue might be all you need: https://admay.github.io/queues-in-clojure/

I guess MessageDigest via Java Interop is a good start?

Yes (we do something similar)

a counterpoint, you may want to add a library. because you need these hashes to never change and i doubt you can depend on that from anything that's included already. Clojure has changed its hashing library in the past in which case you would be pretty sunk, right?

MessageDigest comes from the JDK, AFAIR it doesn't relate in any way how Clojure did it's underlying hashing

Will that be identical across newer jvms? I don’t know. Just wondering how you could ensure that things hash consistently as time goes by

I believe that depends on the type of the hashing algorithm used.

the thing with databases is you usually want to index things

so the hash is a way to reduce the size of what gets indexed (index the hash instead of all the text)

but any hash function can have collisions, so the best thing to do is to basically build a hash table in the database

index on the hash, but don't enforce a unique constraint

and search by hash, and then scan through possible results there to see if the text actually exists

Good point about consistency, but I’d assume that e.g. the SHA256 is implemented in a consistent way across JVMs. Good point to research, assuming is not enought 😉

Good point about the collisions, @U0NCTKEV8. I’d almost tend to accept the probability of a SHA256 collision, but implementing a way to handle the collision is relatively easy, so there is no reason not to do it.

If you are willing to accept collisions you can also use smaller/weaker hash functions too https://ankane.org/large-text-indexes

the probability of a collision with SHA256 is for all intents and purposes non-existent, and can be safely ignored

@U0N9SJHCH you can certainly rely on SHA256 to be consistent across JVMs, and using MessageDigest with SHA2-256 sounds like a good solution to me :thumbsup:

Thanks. So just call that and things will be good?

You will have clojure.core symbols referred, not any code defined in the namespace

I suggest reading about how others are tackling similar problems. https://cs224d.stanford.edu/reports/ManiArathi.pdf

To do deep learning that understands time (or sequential stuff, like words) I think https://en.wikipedia.org/wiki/Recurrent_neural_network are pretty useful. Not really my area of expertise though, but maybe this gives you some ideas

There's a lot of NLP stuff that does not use deep learning, too, but I think the state of the art is some form of RNN

thanks, it's been about a decade since I looked at RNN stuff! 😃