I've done keyword extraction, I.e. given a block of text, pick a few words from that text that are representative of the content (with tf-idf), not sure if that's exactly what you're describing here though? tf-idf might be a good first step though. fastText is pretty convenient for getting word embeddings may be useful too.

is there a library you’ve used?

Just python actually

I generate a csv with clojure and then call the python script as a subprocess. then I also call fastText as a subprocess

oh very cool

probably overkill to use libpython-clj?

when I tried libpython-clj several years ago, it ran about 30% slower than calling python a subprocess 🤷 in general I find csv + subprocess is pretty convenient

looks like spark mllib can do tf-idf, I would definitely check that out first: https://spark.apache.org/docs/latest/mllib-feature-extraction.html I've been using spark mllib for its collaborative filtering algorithm, after previously using python for that, and it's awesome