Searching

“God told us to search, so I did.”

This is a line I have been repeating a lot throughout the last couple of months while working on ML. We at the company have spent a ludicrous amount of time going through different areas of AI research, from GOFAI to transformers to meta-learning and neuro-symbolic approaches. What we concluded is not relevant here. What is more important is that I have come to understand, thanks to my colleagues and friends, that almost everything we work on can be understood as a search problem.

We spent a lot of time looking at searching for programs, exploring approaches like Dreamer-code and AlphaDev. But funnily enough, gradient descent is a form of search too. I had never really thought about it that way, but it is. The loss measures how wrong the network is, and its gradient gives us a direction to adjust the weights. That is what makes this huge search space more manageable. Not relevant right now, lol. I know I am rambling.

Anyways, this led me to a sort of late and probably obvious epiphany: we are always searching. Everything we do as a species, and so much of what every other organism on this planet does, can be understood as a search problem.

The difference is in what we are searching through, what guides us, and what counts as finding something. For a neural network, the search space is the possible configurations of its weights. For someone writing a program, it is the possible combinations of instructions. For an animal looking for food, it might literally be the environment around it. The things being searched are different, but there is still this process of exploring possibilities and responding to what works.

Even biology and chemistry can be looked at through this lens. Evolution explores variations across generations, with natural selection shaping which ones persist. Molecules interact according to physical laws, exploring possible arrangements without any intention of finding something. With language, we search through words and structures to express what we mean. With philosophy, we search through ideas and arguments to make sense of things we do not understand. These are obviously different processes, and calling them search does not mean they all work the same way. But I find it interesting how often this pattern appears in things that seem completely unrelated.

So fundamentally, I have arrived at this weird optimum that, for now, I consider my global minimum: I think all the world’s problems can be understood in terms of search. We are constantly learning how to search better, how to narrow down the possibilities, or how to recognize when we are looking in the wrong space entirely. Every discovery gives us something to build on, so we do not have to start from nothing again.

But progress sometimes means expanding the search space too, realizing that something we had ruled out or never considered was possible. That is how we push our own frontier. We get better at exploring what we know, and occasionally figure out how to look beyond it.

In a funny sense, it makes me feel like we, as in every single thing on this planet and in the universe, are all doing some version of the same thing.

Searching.