@lee suppose I want to make a test project in the cljdoc repo, to have it analyzed during tests so I can make a test search docset... I guess I'm not sure where to put that or how to set that up using deps.edn

do you have a minute to talk about it?

I have done something similar for visual inspection. There is the stand-alone https://github.com/cljdoc/cljdoc-exerciser…

Is that what you mean by test project? A test lib/repo?

yes, essentially

but I'd like this automated and within cljdoc itself, so I can exercise generating a cache bundle and transforming that into a searchset

if that seems like a good idea

Do you want to go through full integration? Like have your test project analyzed by cljdoc-analyzer?

right now I have stuck an edn cache bundle in a file

yes

if possible

it feels like something that we should have in general?

but I need to do things like have a markdown file and an asciidoc file and multiple namespaces and such so I can confirm that my code is breaking those down into chunks correctly (for searching)

And you’d prefer not to use a stand-alone github repo because you want it more tightly coupled to cljdoc source base, yeah?

yes

I mean, stop me if I'm doing something ridiculous

or something ill-advised

Sounds reasonable if you want to do a full integration test.

Our current integration test already ingests a project. But one from the wild.

I'm less focused on the analysis portion and exercising that fully as much as generating the cache bundle input to this searchset creation

(for client-side search)

I don't want to just stick static edn in the project for this since that could break without us knowing

The ingest cycle is not very speedy, I’m guessing you are ok with that for this particular test.

does it take a lot to analyze a tiny already-local project?

Well… you can get an idea by trying an ingest from the command line. But assuming you are ok with speed, I don’t see a problem with creating some sort of cljdoc test project. Ingest only works from jars, so you’d have to jar it up.

ahhh there's the rub

maybe I'll stick with static for the moment

and let people report it if it breaks

rub=too slow?

yes

ya, the feedback loop might drive you a bit mad there…

btw, I am working on search on the server side. So might have some searchy questions for you sometime soon!

oh! well, this is only search within an individual :group-id/:artifact-id/:version, and only client-side

something I've had on the backburner forever

but the server-side is mostly done

Ya, but your brain is wired on search and I might have general ideas to bounce.

cool 🙂

I'm definitely up for it

(by server side search I mean the lucene stuff btw)

👋 I've recently been thinking about problems and ideas that it seems like cljdoc either already solves or is in the process of solving. I've been trying to get up to speed on all the cool things cljdoc is up to. Just wanted to say hi and say that cljdoc is really neat!

hope it's not a dumb question. what is client side search?

Hey buddy! Nice to see you here!

I’ll let @corasaurus-hex answer that one. Here’s the PR https://github.com/cljdoc/cljdoc/pull/466 which has a demo link.

@smith.adriane I was thinking your https://github.com/phronmophobic/dewey might come in handy for cljdoc someday/somehow.

Gotta run for now…

oh, hi 👋:skin-tone-2:

sorry, had to step away, and now I cut my left pointer so typing is fun

wheeee

I actually am nixing that branch from the PR, @smith.adriane, but it has the beginnings of it

it was just too out of date and rebasing was too painful

I was just curious what was considered client-side since I'm still getting acquainted with cljdoc

the idea is to feed the docs for a given :group-id/:artifact-id/:version into the browser and populate an in-browser full-text search engine with it. then you can search within just that docset

you can try searching here to see what I mean https://corasaurus-hex.github.io/cljdoc-search/

that's all client-side

I've been working on "client-side" library search with "client-side" meaning the developer's computer. https://github.com/phronmophobic/add-deps

Yea, I was checking out that link from the PR. Looks cool!

oh nice

looks fun 🙂

Both the search and the static/dynamic analysis that cljdocs does is really interesting

it definitely is

I'm trying to avoid having to index every docset server-side, especially when some may only be searched a handful of times before there's a new version (and therefore new docset) and then never searched again

and so having the client index it, cheaply and quickly, seems like a good trade-off maintenance-wise

just a heads-up, I'm not sure how up to date the specs are in the project

the cache-bundle doesn't seem to be up to date

or perhaps there are two things called cache-bundle in the project

defining specs for some of these huge data structures is paaaaaainful

Yeah, https://github.com/cljdoc/cljdoc/issues/532. I started describing data structures in docstrings, whenever I was scratching my head.

So @corasaurus-hex, I’m learning how server-side search currently works. There’s details around how text is tokenized, but basically it seems like we have prefix searching. So a search for thi will match this and thing but not rethink. I’m gonna guess that client-side search is more along the lines of simple character matching?

no, it's as full as we want

https://github.com/nextapps-de/flexsearch

specifically the indexing options here https://github.com/nextapps-de/flexsearch#index-options

you can choose how you want it to tokenize https://github.com/nextapps-de/flexsearch#tokenizer-prefix-search

I'll bet the server-side things can do this as well?

Thanks! The search technology we are using server side (lucene) is geared for speed. It can be tweaked programmatically in tons of ways. It has evolved over decades… and still actively maintained… and widely used. Powerful but not trivial to use. At first glance flexsearch seems a whole lot more end-user-focuced. But anyway, just wondering out loud how consistent our client vs server side matching techniques are or should be.

the server-side is mostly for matching project names?

Yeah, it searches on artifact-id group-id and description (the one from the pom).

is it worth investigating alternatives? those needs seem super light

It will soon rank results by clojars download count.

Maybe. But it is super-fast. Which is nice for suggest typing.

And there are probably enough lucene geeks out there who understand the tech well… I’m just not one of them… yet!

Clojars also uses lucene but https://github.com/clojars/clojars-web/wiki/Search-Query-Syntax. I was considering offering that syntax, and while it makes sense for clojars, I decided it probably adds more end-user complexity than value for cljdoc.

Related tangent: We search on pom description but don’t display it in results. I personally find that kind of confusing. Any opinion?

so a lot of search tools offer returning the relevant section of text with the matches highlighted

which is a nice feature

in general I like it to feel obvious why something matched, that way I can assess relevance without having to click through

in this case, though, are people searching mostly for projects they already know about?

or is this more about project discovery?

it feels like the former to me but the latter definitely has some value (however someone needs to have generated docs for the description to be searched?)

sorry, I realize that i'm not offering a concrete opinion

if I had to do it myself I'd just make it match artifact and group id. I'd want to hear a use case and get some inspiration before taking it further

Yeah, I am muddling through myself, thanks for muddling with me! I feel the same way about what has matched being obvious. And I also like highlighting (which we don’t yet do but can do).

I’m guessing that we search the description for discovery. Which seems like an ok thing to do. The description comes from clojars so no need for it to have been already built by cljdoc yet. But if we are searching on it, I feel we should show it in results.

Oh neat, https://github.com/cljdoc/cljdoc/blob/master/doc/adr/0019-use-custom-search.md.