Fork me on GitHub

A question about a good storage mechanism and indexing. I have 31000 of unique text IDs with length ranging from, say, 3 to 20 characters. For each unordered pair of IDs, there's a float value (most of the values are 0). And while there are almost half a billion such pairs, due to the nature of the values there are only about 12 million unique values. So many pairs have the same value assigned to them. The data is not static - in the future, new IDs will appear, so the 31001st ID will bring in 31000 new pairs and, hence, 31000 values. I need to be able to store it persistently in such a way so that extracting the value for a random pair of IDs is as quick as possible. Seems like in terms of data structures, a trie would be ideal here. But I may be wrong. Do you have any suggestions?


What about a Clojure map, where the keys are sets of two text IDs, and the associated values are the numbers? If the map does not contain a particular key, then it has the 'default' common value, which I believe you say is 0.


When you say store persistently, you mean in a database? Or you mean persistent as in persistent data structure in memory, like Clojure's persistent data structures?

Mark Gerard18:08:59

Is there a term for that feeling you have when you write something in clojure, then wonder if it is idiomatic enough? I see it cropping up a lot. I propose clojure beginner syndrome


Often I get that feeling, but usually I'm asking "Is there a function for that?"


@cattabanks That affects folks who've been using Clojure for years too! 🙂


it's not just beginners. what's the feeling when you just tried to answer "is this idiomatic?" and immediately see "alexmiller is typing..." 😰

😂 11

Even after a decade of production use, I'll still sometimes post a bit of code to #clojure and ask for opinions/suggestions.

Mark Gerard18:08:53

@seancorfield tbh that is reassuring


I often have the Clojure Cheatsheet open just for this reason 😆


@borkdude Oh my god that's awesome!


(reset! days-since-last-impressed 0)


@cattabanks It's rare at this point that someone suggests a function I'm not aware of but they often suggest a combination of functions that I hadn't considered 🙂

💯 1

In precisely that way, I've wondered if Clojure deliberately shares one of Larry Wall's aims with Perl: the basics are dead simple (you don't need to learn a lot to get started), but the more you learn, the more you can do with it. Just like with natural languages.

👍 1