A Bittensor Lesson

In 2019, Richard Sutton began his prescient essay, ‘The Bitter Lesson’, like this:

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.

Without fail, since the inception of the field, systems equipped with a priori information (in a way analogous to some theories of intelligence) have lost out to those armed with massive computational budgets, and left to find solutions on their own. This was the zero to one realization for machine intelligence.

This principle can be observed across every field of AI. For example, early approaches conceived of vision as searching for edges, angles and cylinders but were later discarded in favour of deep learning approaches that allowed neural networks to find invariances (like edges) in the data themselves. Recently, approaches have prevailed that remove human direction altogether and allow systems to discover meaning from unlabeled images.

In language we see a similar pattern. Early researchers tried to build knowledge of ontological concepts and syntax into their systems in order to orchestrate language comprehension, but these systems were replaced by those those who built their own syntax trees, and later, by those that could learn from unlabeled corpora without any prior ontological framework or notion of syntax to speak of.

The direction here is clear. The more withheld the lock maker, teacher, engineer, the better the result. Solutions, with their endless complexities, cannot be manufactured linearly. Instead, we must include only the meta-methods, step back, and allow the interaction of data and compute to uncover intelligence independently.

We must also step back from the definitions of intelligence that currently guide our advancement. We must relinquish benchmarks, paper writing, and conferences. We must think of the field itself as a meta system upon which AI can grow.

Subscribe to Bittensor

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.