portal 2021-09-09 | Slack Archive

genekim16:09:34

Hi, @djblue ‚ thanks for writing Portal — I love using it! And your suggestion is that it’s most helpful for “medium sized data” rings so true. I’ve been using it recently to parse out all the diffs in a git repo, so that I can eventually visualize it in vega. One of the data structures that I’m printing out is a sequence of 17K nested maps (medium data! 🙂 . I’ve found that when inspecting it, I want to go something in the middle of that sequence, or search for a specific string. This is made a little more difficult because it seems like Portal lazily loads the data — I.e., I can’t hit Control-F and find an element that hasn’t been loaded in, unless I force the data to load by holding down the Page Down key to get to the bottom. Ideally, I’d just hit End or Command-Down, and it would go to the very last entry. But instead, I have to hit Page Down 50+ times, to force the viewer to load and display all of the entries. (I’d use Filter, but in this case, the data is in a vector, not a map, so searching for :delete only shows that element, not the entire vector.) My question: Is there a way to force load all the data, so I can Find in browser? Or is there a better way to do this type of searching?

djblue19:09:44

Hi @U6VPZS1EK, thanks for the kind words! I do agree that fits the definition of medium sized data. You are correct, portal does lazy load and lazy render values from the runtime. I think serializing 17k items isn't an issue so I could bump the lazy load limit. However, that still leaves the lazy render. Getting react / browser to render that many items is challenging and would severely reduce the ux. I've also been considering adding a "show more" button to explicitly load more items since large values can be a challenge to scroll past. I think the ideal solution would be to improve the search algorithm. What if it was simplified to only filter one level deep? It could also filter the currently selected item if you wanted to further filter found values. How does that sound?

genekim21:09:55

I just posted screenshot of the data — Your notion of filter being the answer seems promising — I’m certainly not visually scanning for “1309”. In the screenshot, I would have love to filter for “:modify” or “1309”, and see the entire vector that it’s in, and ideally be able to then see the parent map. OTOH, I do think it would be great to be able to manually force load and render everything, and just search using Ctrl-F in browser. (A command: “load and render all” ?) This would help in the situation I was in yesterday: “why is the graph not showing all the data? wait. why is the shape of the data wrong? where is the shape of the data wrong?” In this scenario, I’m not really filtering — I’m exploring the data, searching to find anomalies, looking for other instances of it. If that can be done by filter and unfiltering, that would be fantastic! Thanks!

djblue17:09:37

@U6VPZS1EK I pushed up some filtering updates to the https://djblue.github.io/portal/. I'm hoping the ux makes sense but will also solve your problem. If you try it out, let me know what you think. I'm going to play around with removing lazy loading and see what happens. That way you wouldn't need to think about it at all 👌

genekim15:09:20

I will try it over the weekend — thank you!!

genekim21:09:26

Oh, shoot! I forgot to post the screenshot! (Reply coming in 30m)

djblue03:09:43

https://clojurians.slack.com/archives/C06MAR553/p1632106557378800 is out, hopefully this resolves some of the issues you mentioned 🤞

genekim03:09:10

Holy cow, I will update the deps right now, and give it a whirl tomorrow! THANK YOU! @djblue

genekim00:12:51

#Also sent to the channel

Hey, Chris — hope all goes well. Back in September, I had mentioned how difficult it is to get to the bottom of a large data structure. Ran into this again today, and made this video to show you a concrete example of me using portal, trying to get to the bottom element of the big data structure. I hope this helps!!! https://www.dropbox.com/s/s2bvlk37dxwvwyh/for%20djblue%20large%20data%20structure%20Screen%20Recording%202021-12-13%20at%204.32.54%20PM.mov?dl=0

djblue04:12:09

Hi @U6VPZS1EK, I think a pagination UI might be the eventual solution, but a nice short term solution would be to use something like clojure.core/last. Or maybe something like:

(defn last-10
  "Take last 10 elements of a collections."
  [coll]
  (take-last 10 coll))

(portal.api/register! #'last-10)

genekim04:12:34

Whoa! That’s awesome — do pagination and selection outside of portal. I’ll play around with that. Thx!

djblue04:12:15

I might look into adding a number input prompt so you could use take-last directly and it would be useful for any other function that would like numbers :thinking_face:

genekim00:12:51

replied to a thread:Oh, shoot! I forgot to post the screenshot! (Reply coming in 30m)

2021-09-09

Channels