Shallow Progress

Take a sentence and convert its words into numbers so the computer can process them, then organize them so it understands the order. Next, look at the sentence as a whole to capture the overall meaning, while also letting each word relate to others to identify what matters most. At the same time, process the sentence in multiple ways to refine the understanding without losing context. Then begin forming the response one word at a time, without looking ahead, while continuously checking back with the original sentence to stay aligned. At last, choose the most likely next word at each step to build a clear and meaningful answer.

Now, this is the core idea behind transformers that are used in tools like ChatGPT and Google Translate, and also form a key part of voice assistants such as Siri and Alexa, helping them understand language, generate responses, and power many of the AI systems we interact with every day. Here comes the most important question: is this the only way to construct a sentence? There is probability involved, but should it be the only process we rely on for everything? Is it the only way forward? Or can we step back and explore entirely different ways of thinking?

We often optimize what already exists without asking why it exists that way. Is that a problem? Yes. Is it something to worry about? Not always. A healthy system needs different kinds of thinkers. Not everyone has to wear a research hat. Some build, some refine, some question, some use and very few really research. That said, many of us skip the foundations. We don’t sit with the problem long enough to understand it. We explore, tweak, and add our cents, but that is not the same as solving. Replacing two layers with three is not research. Making something slightly faster or slightly better is not always insight. We rush to improve before we understand. We optimize outcomes without questioning the assumptions underneath.

Take ReLU as an example. It is a simple function that turns all negative values to zero and keeps positive values as they are. This helps models learn faster and avoid certain issues like vanishing gradients. Did we question why and how? Did we think of alternate ways of learning faster? Why do we need a learning rate? Are we really trying to understand machine learning or are we working with the statistics? It works well, so we use it everywhere. Why don’t we think whether something entirely different could do better?

Did you ever think of transformers from first principles? Did you ever question what and why for each layer that made it so? Let us return to the core question: do you accept what we call AI today? Not the definition we write down, but the way we have realized it and works in practice. I mean yes, it has solved a lot of our problems but can we use it to solve all of our problems?