A few “bits” from a more personal lens and intuition on the relationship between complexity, learning, and compression in modern ML. Including, perhaps, general facets of academia that are, or have been, revealed to me throughout my PhD.
Everybody “connects the dots” differently based on their personal story. Here, I share some of mine. Everything is, and will be, written exclusively by me (if that is ever not obvious).
/ Choose a bit.
When I’m introduced to a new board game, each friend takes a turn explaining their most essential strategy, so I end up with virtually every single game detail. Sometimes, we even use the first 20 minutes reading said details from the rulebook. As soon as we start playing, however, nearly all these details end up being irrelevant for actually playing the game. We really only need to look at the rulebook so I can convince Jonas that it is indeed possible to achieve world domination in a single turn, although I seemingly did nothing for 3 hours and swept in at the very last minute to take it all (Board game: Risk, 6-players).
Anyways, for that reason, I skip almost all details when introducing someone to a new game and try to mention only the 2 or 3 compressed scenarios I think are the most important to understand (filling in details as needed). When it works, it’s certainly much more effective, yet my friends’ approach, in contrast, always works. I don’t think it’s a lack of detail; it’s more a question of how the scenarios are structured (also in the rulebooks). And even when it fails, just repeating the information, the same (word-for-word) information, once or twice, usually gets everyone on the same page.
To the same tune, many approaches towards making neural networks more efficient identify seemingly redundant or repeated details and remove them. Here, as with many other things in life, we might have slightly overlooked the significance of the details, especially those that are repeated to us. If we can’t see the forest for the trees, we tend to burn a few of them, increasing our FOV. Rather, I’d argue that, many times, the forest is easier to see were we to plant trees among those already there. In that sense, repeatability and similarity between details can also help reveal the bigger picture, not just obfuscate it.
Related to Algorithmic Simplification of Neural Networks with Mosaic-of-Motifs.