https://github.com/dazld/r11y is now publicly available - it’s a pure Clojure URL->Markdown extraction tool, compiled in Graal with a couple of super powers:
• Preserves whitespace inside pre-formatted blocks (this was a big one for me!)
• Extracts metadata as structured data from URLs
• Pretty fast 🚀
Caveat - it’s not even remotely as battle-tested as tools like Trafilatura - but it works pretty well for me on personal projects. Hope it might be helpful for others too.
I’m not publishing a macOS build release (yet..) but you can make one locally by pulling the repo and running make build-macos
Linux x86 builds are available on the releases page.
This one is quite interesting! Talking with @zikajk he mentioned we could maybe use in #eca offering a web-fetch tool I always wanted to support, so LLM could fetch urls without any extra work.
Q: how's that work briefly? does it just delegate to normal http calls and manual parsing, if so there is no service/saas dependencies right?
correct, all self contained - and I use this for exactly the same purpose, combining web search and a web fetch tool that delegates to r11y - inside an agentic loop
amazing! I do want to use in #eca! How you manage web-search? I suppose in another way
got it! yeah there is no free lunch, in the future I may consider something like that optionally
it’s honestly not that bad - but not sure if it’s in a bit of a grey area with regards to TOS
I can see that
well thanks, I will try to find some time to add that web-fetch tool to eca using r11y, great project!
lmk if I can help or if there’s something that’s a bit wonky
this is a pretty naive tool in many respects, but just about valuable enough to liberate from private repo hell.. so.. 😛
working with graal is already a plus as eca is compliled with graal as well
fair enough, if you have interest in maintaining it, I have interest in using it
I actively use it, and think it plugs a gap. If there was an equivalent, I’d just have used that - but couldn’t really spot something. I’ll keep an eye open for PRs.
Adding my support for an eca web search tool! gratitude