How are you using llava? I've found google's vision API for OCR to be unreasonably effective when feeding into LLMs.

Here's some example code: https://github.com/phronmophobic/slackcommands/blob/7c6db9a8775a5559a3173ddbdb0d033c0e317954/src/slackcommands/ai/vision.clj#L266

I found llava to be amazing, but super unreliable — it was sort of fun learning to to make it work using ollama, but when I got a bit frustrated with the poor results after a couple of days, I went back to GPT-4-vision, and was startled at how good it was. Will write more soon, and always amazed at your work!

PS: I’m 3 steps into the rewrite, and am amazed at how much cleaner it is already!!!!

@U6VPZS1EK I look forward to seeing the rewrite!

I haven't used llava that much, but I've found it mostly not that useful. Would be interested to hear what it's good at.

I'll probably update llama.clj with llava support at some point since I think llama.cpp already supports it.

OMG, I’m so ridiculously happy. Here’s what it generated! I was able to add a step that detected whether the image matched dimensions of an iPhone, and count the red pixels below the YouTube player window (something that I spent 2 hours trying to get gpt-4-vision to do.) It was amazing to be able to work on each of these steps in isolation from each other. Here’s some images: • youtube video screenshot being analyzed • the summary of it being put into the database • the bag of data being displayed in Portal Here’s the rewritten version: it’s not what I expected it to be — I didn’t want to deal with having to write function names, so I just kept it in a big long function. But I was amazed at how easy it was to change the steps. I even introduced a new one, to write the 2nd LLM prompt as a pure function, pulling it out of the 2nd LLM stage. It made the process of writing and understanding the code 100x easier and so much more fun! THANK YOU, @neumann and @U0510902N!

Some other comments

The revised code is in a comment in the gist: https://gist.github.com/realgenekim/17f9a7ae48aaf2e03df3cc80326a5094?permalink_comment_id=4943675#gistcomment-4943675 • This big bag of “vars” inside the bag: I thought it would be confusing to have so many of them, but it wasn’t a problem. (I kept thinking I should group them by {:pass1 {:db _ :photo-id _}, :pass2 {:photo-b64-string _…} …}. But I didn’t want the overhead of having to number and renumber them. So, big ol flat map it is! • At the bottom is the comment block manually saving states, passing it to the next. It was super easy to test, revise, and run/re-run. • I have yet the make the wrapper, which will just recur until state is done. This is just night and day compared to before. Thank you!

@U7RJTCH6J My initial reaction to llava was initially “amazing!” But I was constantly surprised by how poor quality the interpretations are, and how toxic some of its confabulations/hallucinations are — my reaction was often, “my goodness, what have you been trained on!” 😆

In contrast, see the GPT4-vision interpretation here (the picture with the arrow in it): https://clojurians.slack.com/archives/CKKPVDX53/p1708999673659119?thread_ts=1708991492.816989&cid=CKKPVDX53 Am just amazed at how much relevant details in can pull out. Aside from the iPhone YouTube mobile app, it can reliably pull out the current playtime of the screenshot of almost any screenshot I give it.

I can’t wait to try running your code — or better yet, wanna get together sometime in the next week or so? I’d love to get a demo of what you’ve built. this looks incredible!]

I'm also pretty impressed that chatgpt can extract the approximate percent played. I'll have to try it out sometime.

Check this one out — gpt4-vision extracted current play position with no help from me. Prompt was as follows (I will take out the stuff asking it to estimate the red progress bar — I just counted red pixels instead for iPhone portrait mode pics, which it couldn’t handle.) >

You are a helpful AI assistant. I am trying to remember what was said in a video or podcast, so I can write an article about it.
> 
> Describe the contents of the provided image in detail. The image is a paused video on a screen or a screenshot. Mention the apparent video player interface, the progress bar status, any visible text or titles, the individual visible in the video frame, and any discernible actions or expressions. Also, note any visible tabs or browser elements that could indicate the context or content of the video. Provide a clear and coherent description of all elements without speculating about the content beyond what is visible.
> 
> I want to know whether this is a YouTube video or a podcast.  I want you to extract all visible text.
> 
> 
> Right below the in the YouTube video window, which is at the very top of the screen, you will find the episode name (e.g., podcast name), with the channel name right below it.
> 
> If the current playback time is not available, estimate the percentage viewed instead by looking at the length of the red progress indicator positioned at the bottom of the main video window  in relation to the full width of the screen. Assume the full red progress bar represents a 100% completion. Measure or make a visual estimation of the red part of the progress bar. Then, divide this length by the total width of the screen to obtain a rough percentage of the video completion. Provide this percentage as the progress indicator value.
> 
> Do not use code interpreter -- only analyze the images.
> 
> Analyze the screenshot and extract all the relevant information, and put it into
> a Clojure EDN map.  Here is an example:
> 
> 
> {:date-of-photo "2023-12-30"
>  :podcast-app "Overcast, screenshot of mobile interface"
>  :time "5:08"
>  :podcast-episode "Episode 161 — 'There I was!': Real Life Stories from the Cockpit with the Mitchell Institute Part II"
>  :podcast-name "THE AEROSPACE ADVANTAGE"
>  :length "58:06"
>  :current-playback-time "34:11"
>  :percentage-viewed "92%, based on the red bar progress indicator"
>  :name-of-podcast-app "Apple Podcasts"
>  :notes "12/30 was not extracted from the screenshot, but was added manually; and any other comments go here. Man with green shirt standing in front of a presentation. The interface suggests that the video is possibly part of a series or playlist, as the text 'Mix - PapersWeLove' is visible, indicating more content from this channel is available"
>  :all-extracted text "Comments are turned off."}
> 
>  Do not use code interpreter -- only analyze the images.

Interesting. I've found the vision tools that do OCR to be pretty effective and most other vision tools to be kinda mediocre. > I just counted red pixels instead for iPhone portrait mode pics, which it couldn’t handle. Does that mean the LLMs couldn't count the red pixels? I would be surprised if they could.

Wow, this is soo cool.

Late to the party and stuff, so sorry about that. Looking at your doit fn, I’d deffo split that thing into a fn for each of the case statements, be that a multi method or just normal fns:

(defn load-photo-from-db [...]...)
(defn url->b64 [...] ...)
;; etc

(def empty-bag {:photo-b64-string "" ;; or should it be nil?
                ...}

You will most likely also see that a bunch of these fns (if you choose to extract them) are nice, pure fns, and just a few of them be sideffecting.

pass2-summary-to-edn could also reaonably be split in to several fns. One which creates

{:summary (-> pass1-summary :summary)
 :prompt pass2-prompt
 :is-screenshot? is-screenshot?
 :youtube-percentage youtube-percentage}

Mindblowing thread this!

I’ve updated the gist, more comming

I’ve somewhat come to my end now with what I see as fruitful refactors, but I must admit, there is more to be done. I think it’s fair to point out that your interaction with the LLMs could be modeled more strongly. Like, the prompt and the prompt-photo fns seem quite similar, and i’d be curious if we couldn’t have just a prompt datastructure to describe the different prompts we’d need.

So rather than specializing on what kind of prompt you’re sending I’d have fns like:

(defn ->photo-prompt [....] returns data that describes a photo prompt)

(defn ->screenshot-prompt [....] returns data that describes a screenshot prompt)

(gpt/prompt (->photo-prompt ...))

(gpt/prompt (->screenshot-prompt ...))

{:prompt :summary :success}

@U04V5VAUN I’m in complete awe of what you did — thank you so much! I suspect I’ll be studying and replicating this form of code for the next decade. I will be copying this into my codebase later today, but some reactions (more reflections coming later).

When I was reading your note, I was extremely skeptical that even the first refactoring would be worth it — as @U0609QE83SS noted in another thread, pulling out the code out of the case statements runs the risk of now having to look in two places instead of one, decreasing coherence, and increasing the risk of the pieces no longer fitting together in the end. But wow, there is no doubt that your version is much, much better than mine, on so many dimensions: readability, testability, and getting rid of the stupid “next state” in the case state machine is just 🔥 And the explicit shifts to a monoid pattern is also 🔥 — I wrote about how I loved @U050P0ACR use of monoids and “algebraic thinking” here: https://clojurians.slack.com/archives/CKKPVDX53/p1700849580265389 I wish I had been able to watch you refactor this code, because I have a potentially silly-sounding question, which actually I think subconsciously is deterring me from doing what you did: • wasn’t it a lot of typing? It sounds laughable, but the idea of having to give each one of those tiny functions names and typing it out could seem like a lot of work! I’d love to know your thought process and response to that question! (Rest assured, I have no doubt that it’s worth it — since I started this thread, I’ve gone on to create 5+ more of these “monster let block” functions, which I now must struggle with. I think seeing your code pattern gives me something that I can build or refactor to much earlier!) Thank you again! Am in awe of what you did!

Thank you 🙂

This was actually not a lot of work. As for the fn-names, you already had them spelled out in the case statement. Basically, the three/four comments are the iterations that I took. I am a heavy user of paredit, so that makes stuff so much simpler.

This is a small(!) recording of the first steps of such a refactor.

Wow, I watched that whole 5.5m video — thank you so much for posting it! (I’m so glad you recorded it!!) The notion that “refactoring would take a LOT of typing” is clearly false. And the increase in clarity is stunning. I also love the fact that the arguments are now explicit — instead of a “bag of values in a map,” they’re explicitly defined, which makes understanding the function so much easier. In 5.5 minutes, you refactored two functions — so, what a huge payback for 2.5m of work per function! I also LOVE that you turned doit function into this:

(defn doit [db photo-id]
  (-> (xtp/photo-xtdb-fresh-url-uuid db photo-id)
      (url->b64)
      (write-to-file "tmp/decoded.jpg")
      (prompt-vision-llm-pass1 default-prompt)
      (store-summary :pass1)
      (analyze-screenshot "tmp/decoded.jpg")
      (pass2-generate-prompt)
      (pass2-summary-to-edn)
      (write-to-database db photo-id)))

You make me blush 🙂 Two things: One, I think you’ve picked up on most of the things I was looking to achieve, there’s still some things to be done, but I think we’ve gotten to at least 80% improvement. Two, I could never have written the thing that you wrote, as in I would never be able to realize that I store a bunch of screenshots of talks on my phone (which I used to do), and that there is a way to solve the problem with LLMs and what not. And even if I had realized that it was a solvable problem, I would never have gotten around to even try solving it because there is just too much icky stuff to deal with. So well done!