Nested Learning: How Google’s New AI Architecture Learns Without Forgetting

TL;DR:

Contemporary AI systems often struggle to retain learned knowledge when exposed to new data — a problem known as catastrophic forgetting. Google’s new Nested Learning paradigm proposes a brain-inspired architecture with multi-level learning processes operating at different update frequencies. This aims to enable continual learning without overwriting foundational knowledge, moving AI closer to genuinely accumulative learning rather than static pattern-matching.

In recent years, AI– and especially AI agents– has shown remarkable abilities in generating content, automating tasks, and mimicking human-like behaviour. Yet the gap between what AI appears to do and what it actually does is wider than most people think.

Large Language Models (LLMs) produce step-by-step “reasoning” that looks like human problem-solving. But when faced with genuinely novel or highly complex problems, their performance often degrades abruptly rather than gracefully. They don’t gradually decline; they collapse completely.

Why? Because AI isn’t truly thinking or learning like the human brain. It relies on statistical pattern recognition trained on large datasets. So the next time a model like GPT claims it is “thinking,” it’s best to remember that it is predicting the most likely continuation of tokens based on what it has seen, not introspecting like a human mind.

You may also have noticed that sometimes GPT seems to answer or even behave worse than before. A big contributor is catastrophic forgetting: when a model is updated with new data that shifts in distribution, previously learned capabilities can degrade sharply. This happens because fine-tuning often overwrites parameters that encoded older knowledge.

This is one of the central limitations of current AI systems.

Google has recently introduced a potential path forward called Nested Learning, a framework inspired by how the brain learns across multiple timescales rather than trying to update everything at once. ( Google’s paper on Nested Learning )

The Core Problem: Why AI Forgets

To understand why Nested Learning matters, let’s start with an example.

Suppose you train a model on Task A and it achieves 80% accuracy. Great. Now you retrain it on Task B, and it achieves 70%. During the process of learning Task B, performance on Task A drops to, say, 75%. The gains you made earlier are partially lost — this is catastrophic forgetting.

Today’s LLMs effectively split their “memory” into two parts:

1. Immediate Context (short-term memory): Information within the current input sequence, available during inference (aka prompting) but quickly forgotten after processing.

2. Static Pre-training (long-term memory): Knowledge encoded in the model’s parameters during training and rarely updated. (every major release of a chatbot)

This leads to a kind of AI version of anterograde amnesia: after training, the system can’t readily incorporate new knowledge into its stable memory without retraining the whole model, risking earlier knowledge loss.

Think about what this means practically: if you deploy an AI system to a hospital, a law firm, or a research lab where it encounters novel cases daily, it can’t actually learn from those cases in any meaningful way. It can only pattern-match against its pre-training data. Any genuinely new situation falls outside its expertise.

Researchers have thrown various patches at this problem—regularisation techniques, memory replay mechanisms, additional neural resources—but these are external fixes to a fundamental architectural flaw.

Learning from Biology: How the Brain Does It

Unlike artificial systems that must be retrained from scratch, the brain continuously acquires new knowledge while preserving the existing knowledge. This process is called neuroplasticity. Neuroplasticity is the brain’s ability to reorganise itself by forming new neural connections, adapting to life, learning or injuries.

At the core of biological learning is the stability-plasticity dilemma. The stability-plasticity dilemma describes the challenge in neural networks and learning systems of balancing the ability to retain previously learned information (stability) with the capacity to adapt to new information (plasticity).

The brain implements this balance through the Complementary Learning Systems (CLS) framework:

• Fast Learning (Hippocampus): rapidly captures new, episodic experiences.

• Slow Learning (Neocortex): gradually builds stable, structured long-term knowledge.

When you learn something new, your hippocampus rapidly encodes it into short-term memory. This is fast, flexible, and highly plastic. But the same information doesn’t immediately get written into your neocortex, your long-term knowledge repository. Instead, through a slower process—often during sleep—that information is consolidated, integrated with existing knowledge, and stored stably.

Your brain maintains separate, nested learning mechanisms operating at different timescales. Fast learning for immediate needs. Slow learning for foundational knowledge. They work together without interfering.

Your prefrontal cortex can learn a new shortcut today without destroying your understanding of basic mathematics learned in childhood. That’s because those foundational concepts are stored and updated at a different, much slower frequency than your working memory.

This multi-timescale approach to learning is the opposite of how current AI works. Most language models treat all parameters as equally plastic (updatable), or conversely, lock most of them down completely. There’s no intelligent hierarchy.

Nested Learning: Google’s Brain-Inspired Solution

Nested Learning (NL) is Google Research’s attempt to formalise multi-level learning into deep learning architectures. Rather than viewing a model as a single optimisation problem, NL treats it as a system of nested, interconnected optimisation sub-problems, each with its own context flow and update frequency.

In conventional neural networks, you first design the architecture and then apply a single optimiser across all parameters. Nested Learning unifies these two pieces: the architecture and optimisation process become part of the same system, and different components are intentionally designed to update at different speeds.

This hierarchy is organised around update frequency:

• Inner Levels (High Frequency): Parameters that update rapidly, capturing immediate context and fast-changing patterns.

• Outer Levels (Low Frequency): Slowly-updated parameters that preserve foundational knowledge, analogous to the slow learning rate observed in the neocortex.

This introduces a new design dimension called computational depth, not just depth in layers, but depth across time, allowing models to maintain a much larger context without being overwhelmed.

This creates what Google calls the Continuum Memory System (CMS), a spectrum of memory modules operating at different update speeds, extending beyond simple short-term and static long-term memory into a richer memory hierarchy.

Nested Learning, therefore, organises a model into a chain of components, each updated at its own frequency. This mirrors the brain’s Complementary Learning Systems: working memory updating every step, intermediate systems adapting over days or weeks, and foundational knowledge changing only very slowly.

Proof of Concept: The HOPE Architecture

To validate these ideas, Google researchers developed a proof-of-concept model called HOPE (Hierarchical Optimisation with Persistent Experience), built on earlier long-term memory architectures and augmented with CMS blocks.

HOPE implements a series of nested learning levels, each updating at a distinct frequency, and even learns to modify its own update rules. Effectively learning how to learn on its own.

HOPE’s early results look very promising because it:

• Achieves lower perplexity and higher accuracy than standard transformer and recurrent models on various tasks,

• Handles long-context predictions more effectively by managing memory across the CMS spectrum

• Demonstrates better continual learning behaviour in benchmarks designed to test long-range and sequential knowledge retention

These results suggest that a hierarchy of update speeds, rather than a flat learning process, can significantly mitigate catastrophic forgetting.

A Practical Example: Autonomous Medical Diagnostic Agent

To understand how Nested Learning solves the catastrophic forgetting problem in practice, consider an Autonomous Medical Diagnostic Agent (AMDA) designed for continuous deployment in a hospital environment.

This system needs to:

1. Remember basic anatomy, drug interactions, and medical ethics (foundational knowledge)

2. Learn about new disease patterns emerging in the hospital

3. Adapt to each new patient’s unique case

4. Do all of this without forgetting what it already knows

A traditional LLM fine-tuned on recent pathology data would immediately suffer catastrophic forgetting of foundational medical knowledge. The system might learn to recognise newly emerging diseases or regional treatment variations, but in doing so, it would degrade its understanding of basic human anatomy, fundamental drug interactions, and universal medical ethics.

With Nested Learning and its Continuum Memory System, the AMDA’s architecture would be like this:

• Level 1 (Working Memory): Updates constantly (every patient interaction).

  • Stores: live patient data, conversational context, immediate symptoms. Highly plastic, allowing rapid adaptation to the specific case.

• Level 2 (Recent Experience): Updates daily or weekly based on Local Surprise Signals (a technical term for “when the model made an error”).

  • Stores: meta-learning rules about optimisation, effectiveness of recent strategies, and regional drug efficacy.

• Level 3 (Domain Expertise): Updates periodically (monthly or quarterly, based on medical journals).

  • Stores: disease patterns, standard treatment protocols, and language understanding.

• Level 4 (Foundational Knowledge): Updates very slowly (maybe once a year when medical consensus changes).

  • Stores: human anatomy, fundamental drug interactions, and medical ethics. This layer is locked down tight, resistant to interference from new data.

By implementing this design, your AI system can process a new patient’s complex case at Level 1 while guaranteeing that foundational understanding at Level 4 remains stable and accurate.

This is the promise of Nested Learning: AI systems that finally learn the way brains actually do.

Why This Matters Now

The Nested Learning paradigm represents a meaningful step forward in how we think about deep learning. By treating model architecture and optimisation as parts of a single, coherent process–rather than independent design choices– it opens up a new space where multiple levels of learning can coexist.

This approach suggests that when different learning processes are organised hierarchically and allowed to interact, models can become more expressive, more capable, and potentially more efficient.

Nested Learning doesn’t solve everything, of course. It requires more sophisticated thinking about how to structure these multi-level learning systems. But it addresses something fundamental limitation in today’s AI systems: the inability to learn continuously without overwriting what they already know.

If this approach proves scalable and efficient, we may see AI systems that can genuinely accumulate knowledge over time and not just pattern-match against pre-training data.

That’s the difference between having a tool that looks smart and having a tool that can actually learn. Between AI that simulates intelligence and AI that builds it.

Let’s Connect!

• 🐦 X/Twitter : alex_kazos

• 💼 LinkedIn: alex-kazos

• 👨🏻‍💻 GitHub: alex-kazos