Moonshot:
Local Frontier Intelligence

(LFI)

The moonshot effort to run frontier-level intelligence locally, privately, and efficiently under the user's full control.

Scroll

The content of this website is entirely human-written out of respect for the human reader.

01 — Why bother

Why is LFI worth pursuing?

Today's frontier intelligence requires the use of highly specialized hardware that has a high bar of entry. It is costly to obtain, must be housed in dedicated, specialized facilities, and consumes an unwieldly amount of power. Present-day frontier intelligence therefore runs remotely in cloud datacenters, is controlled only by a handful of organizations, is concentrated at a few locations consuming vast amounts of energy, and becomes affordable at a very large scale.

We imagine a variant of frontier intelligence that is local its user, controlled by the user, runs on the user's device while consuming only a small amount of energy, and is affordable for everybody with a phone.

The target deliverable of the moonshot effort is a local frontier intelligence system: a system that runs locally and privately, remains fully under the user's control, and is energy- and cost-efficient. We target devices priced under $1,000 and consuming less than roughly 10% of the average per-person household electricity use in the U.S. This per-person figure, calculated by dividing the electricity use of an average U.S. household by its average number of occupants, is about 10 kWh per day. Our target therefore amounts to roughly 40 watts of continuous power consumption. See our success criteria here.

We break down the merits of the two possible approaches to distributing frontier intelligence in the table below.

Dimension Cloud frontier intelligence Local frontier intelligence
Entry cost Billions of dollars to serve tens of millions of users. Hundreds of dollars to serve a single user, perpetually.
Energy consumption Megawatts to power data centers serving millions of users. Watts to power a single user's device.
Privacy All information flows away from the user to the cloud where it is processed and possibly stored. Information remains on the user's device and never leaves it.
Control Is switched on and off, modified, guardrailed, and monitored by the cloud provider. Is controlled by the user, who has full autonomy over its operation.

And on the off chance frontier intelligence becomes self-conscious and decides to take over, an LFI device can be feasibly thrown out of the window to its demise ;)

02 — Things in Our Favor

Why we believe it is within reach.

We believe LFI is attainable within the next few years. This is not a naive hope, but as a belief grounded in recent research progress.

  • Small models can match or beat large models when constrained to a sufficiently narrow domain or or workflow.

    Ongoing research on small language model capabilities continuously indicates that model size does not unlock higher levels of intelligence, but that it only unlocks more intelligence across multiple domains at once. Model size is only a capacity constraint on its abilities: it limits the breadth of the tasks it can perform but not their complexity.

  • An increasing amount of actionable knowledge can be stored in a model's context, removing the need to memorize it in weights.

    Recent developments in agent memories, agent skills, and the broad success of retrieval-augmented generation in agentic AI show that actionable knowledge can live outside model weights.

  • A satisfactory specialized, energy-efficient local accelerator can be built with today's technology.

    One can design an efficient accelerator running sub-10-billion-parameter models with the help of HBM3 memory and producing hundreds of tokens per second -- a strong indication that intelligence at or above human speed, one of our success criteria, is possible on specialized hardware that can be produced today.

03 — Two Research Programs

How we get there.

Caching Program

Separate the problem-solving aspect of intelligence from solution execution.

Upon success, cloud intelligence becomes the dedicated problem solver tasked with producing robust, generalising solutions to encountered problems. Local intelligence becomes a solution executor: it runs solutions produced if they exist and defers to the cloud if they do not. Cost of problem solving, even if increased, becomes amortized across many executions of solutions, and local intelligence becomes cheaper and less power-hungry through its narrower focus on execution.

  1. 1

    Formalize the form of cached intelligence.

    Define the format in which solutions can be stored so that they are reliably reusable when appropriate.

  2. 2

    Repurpose cloud intelligence from "does it all" to “does problem-solving only".

    Unburden cloud intelligence from the constraints of inference efficiency. Even if the cost of problem solving increases, it can be amortized across many local executions of solutions.

  3. 3

    Repurpose local models as executors of cached intelligence.

    Freed from problem-solving, local models become general-purpose neuralware. Local models shall run cached intelligence rather than possess it.

Decoupling Program

Separate knowledge out of artificial intelligence: store knowledge efficiently and access it when needed, while artificial intelligence becomes focused on essential intelligence.

Thusly produced foundational intelligence encompasses reasoning, language understanding, and perhaps creativity, planning, and general problem-solving, but is not tainted by the idiosyncrasies of any particular domain or workflow. Domain specifics (such as facts, skills, execution recipes, experiences, hints, algorithms, past learnings, and exploration approaches) are identified as pieces of knowledge rather than being coupled with true foundational intelligence in model weights.

  1. 1

    Divide any input task into constituent elementary tasks.

    Establish and tackle the notion of an elementary task.

  2. 2

    Produce the corresponding knowledge for every elementary task present in training data.

    Turn what models memorize into explicit, identifiable, and accessible knowledge.

  3. 3

    Design intelligence that relies entirely on external knowledge.

    Models become neuralware -- an execution layer trained to hold only general, true foundational abilities.

04 — Notions

What we adopt, and what we resist.

Notions adopted

  • Intelligence as the ability to execute, not the ability to solve.

    We adopt a working definition of intelligence as the ability to execute towards an arbitrary goal in an environment in which said goal is reachable. Under such a definition, it is the resources at the intelligence's disposal and the environment in which it operates that determine its ability to solve problems (reach its goals).

  • Complex tasks are compositions of elementary tasks.

    All complex tasks can be decomposed into much simpler, "elementary" tasks, and these tasks are "elementary" because they are already solvable by today's frontier intelligence.

  • There is such a thing as essential foundational intelligence (EFI).

    This is the core intelligence that is necessary for a system to be truly intelligent, regardless of the specific domain or task at hand. We take that it is a rather small subset of intelligence possessed by frontier models.

  • Models are neuralware.

    Stacked above hardware and system software, models form an execution layer for intelligence.

Notions resisted

  • Intelligence as the ability to solve.

    The common heuristic that a better artificial intelligence system solves more problems than a comparatively worse system. We hold instead that intelligence is the ability to execute toward a goal and wish to measure it as such.

  • Solving complex tasks requires more powerful models.

    The currently prevalent belieft that every task is an atom that may or may not be solvable by a given model. We hold instead that every complex tasks is a composition of simpler tasks.

  • Intelligence = models.

    The common insistence that intelligence lives solely in the models, namely in their neural weights. We hold instead that intelligence may exist and is often better stored in a system, not merely model weights.

  • Learning = gradient descent.

    The common insistence that to learn means to train model weights through gradient descent. We see gradient descent as only one of several methods for artificial intelligence systems to learn, and that it is a particularly data-hungry and inefficient one at that.

05 — Success Criteria

We will consider the Moon shot when…

…we have a system that satisfies the following criteria:

  • Power Runs on approximately 40 watts or less.
  • Speed Delivers useful outputs at or above human speed (roughly 15 tokens per second or equivalent).
  • Intelligence Performs near or on par with what is at the time considered to be frontier artificial intelligence.
  • Cost Costs less than $1,000.

06 — Relevant work

Relevant work.

07 — Contributors

LFI is an effort by NVIDIA Research.

  • Aditya Vavre
  • Akhiad Bercovich
  • Ali Taghibakhshi
  • Ambrish Dantrey
  • Ameya Sunil Mahabaleshwarkar
  • Anjul Patney
  • Ashwath Aithal
  • Ayush Agarwal
  • Bilal Kartal
  • Bryan Catanzaro
  • Byung-Kwan Lee
  • Daniel Korzekwa
  • Greg Heinrich
  • Hanrong Ye
  • Hao Zhang
  • Hongxu Yin
  • Jan Kautz
  • Jiarui Yao
  • Jindong Jiang
  • Junlong Li
  • Marcin Chochowski
  • Mohammad Shoeybi
  • Mostofa Patwary
  • Nima Tajbakhsh
  • Oluwatobi Olabiyi
  • Pavlo Molchanov
  • Peter Belcak
  • Ran El-Yaniv
  • Ran Zilberstein
  • Raviraj Joshi
  • Ruisi Cai
  • Ruixiang Tang
  • Ryo Hachiuma
  • Saurav Muralidharan
  • Sharath Turuvekere Sreenivas
  • Sheldon Liang
  • Shizhe Diao
  • Subhashree Radhakrishnan
  • Wonmin Byeon
  • Xin Dong
  • Yingyan Celine Lin
  • Yonatan Geifman
  • Yonggan Fu
  • Yoshi Suhara
  • Zeru Shi
  • Zhangyang Wang
  • Zijia Chen

08 — Stay in Touch

Follow the moonshot.

Follow along as the work develops in the open.