My CVPhD ThesisContactGoogle ScholarSemantic Scholar

Papers

Talks

Mentoring

Representative papers where I am a primary contributor are highlighted

Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2

Nov 20, 2023
H. Ivison and Y. Wang et al.
ArXiv
Advancements in instruction tuning and RLHF! Empirical studies.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

Oct 31, 2023
Nathan Lambert, Roberto Calandra
How the optimization setup of RLHF is limiting the steerability of LLMs.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Zephyr: Direct Distillation of LM Alignment

Oct 25, 2023
HuggingFace H4 Team
The report for a small and powerful chat model trained with DPO!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Entangled Preferences: The History and Risks of Reinforcement Learning and Human Feedback

Oct 20, 2023
Nathan Lambert, Thomas Krendl Gilbert, Tom Zick
The complicated historical past underpinning reinforcement learning from human feedback!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning

Oct 12, 2023
Ran Wei, Nathan Lambert, Anthony McDonald, Alfredo Garcia, Roberto Calandra
ArXiv
Where is model-based RL heading 4 years after the seminal paper of my Ph.D.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Measuring Data

Dec 12, 2022
Margaret Mitchell, Alexandra Sasha Luccioni, Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, Douwe Kiela
ArXiv Preprint
When you "measure data", you quantify its characteristics to support dataset comparison & curation. You also begin to know what systems will learn. Many ML systems don't reason with this, we posit you should.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Synergy of Prediction and Control in Model-based Reinforcement Learning

May 12, 2022
Nathan Lambert
My thesis on model-based RL. Let's make models work with tasks!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Reward Reports for Reinforcement Learning

Apr 25, 2022
Thomas Krendl Gilbert, Sarah Dean, Nathan Lambert, Tom Zick, Aaron Snoswell
We propose a new type of documentation for dynamic machine learning (and reinforcement learning) systems!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systems

Feb 8, 2022
Thomas Krendl Gilbert, Sarah Dean, Tom Zick, Nathan Lambert
Center for Long-Term Cybersecurity White paper Series
We detail why reinforcement learning systems pose a different type of (dynamic) risks to society. This paper outlines the different types of feedback present in RL systems, the risks they pose, and a path forward for policymakers.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

The Challenges of Exploration for Offline Reinforcement Learning

Feb 1, 2022
Nathan Lambert, Markus Wulfmeier, William Whitney, Arunkumar Byravan, Michael Bloesch, Vibhavari Dasagi, Tim Hertweck, Martin Riedmiller
We flip the script on Offline RL research and ask the question of "what is the best dataset to collect?" rather than "what is the best algorithm?"
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Investigating Compounding Prediction Errors in Learned Dynamics Models

Dec 7, 2021
Nathan Lambert, Kristofer Pister, Roberto Calandra
arXiv preprint
In this paper we set out to understand the causes of compounding prediction errors in one-step learned models. With this, we hope a next generation of models can be used to improve model-based reinforcement learning.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

BotNet: A Simulator for Studying the Effects of Accurate Communication Models on Multi-agent and Swarm Control

Sep 1, 2021
Mark Selden, Jason Zhou, Felipe Campos, Nathan Lambert, Daniel Drew, Kristofer S. J. Pister
IEEE International Symposium on Multi-Robot and Multi-Agent Systems
A simulator for studying high-agent-count networked systems!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Best Student Paper Finalist!

Axes for Sociotechnical Inquiry in AI Research

May 1, 2021
Sarah Dean, Thomas Krendl Gilbert, Nathan Lambert, Tom Zick
IEEE Transactions on Technology and Society
We present a concise set of directions for understanding the societal risks of new directions of AI research.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

MBRL-Lib: A Modular Library for Model-based Reinforcement Learning

Apr 20, 2021
Luis Pineda, Brandon Amos, Amy Zhang, Nathan O Lambert, Roberto Calandra
An open-source PyTorch repository designed from the bottom up for model-based reinforcement learning research.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

Feb 26, 2021
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, Roberto Calandra
2021 International Conference on Artificial Intelligence and Statistics
We showed that advancements in AutoML when paired with common deep RL tasks, MBRL algorithms perform so well they break the simulator.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

AI Development for the Public Interest: From Abstraction Traps to Sociotechnical Risks

Feb 4, 2021
McKane Andrus, Sarah Dean, Thomas Krendl Gilbert, Nathan Lambert, Tom Zick
IEEE International Symposium on Technology and Society
We study three developing subfields of AI research and their growing relationship with the sociotechnical: AI Safety, Fair Machine Learning, and Human-in-the-loop Autonomy.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Nonholonomic Yaw Control of an Underactuated Flying Robot with Model-based Reinforcement Learning

Dec 21, 2020
Nathan Lambert, Craig Schindler, Daniel S Drew, Kristofer SJ Pister
IEEE Robotics and Automation Letters
We explored how MBRL can learn multi-step, nonlinear controllers!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning

Dec 16, 2020
Nathan O Lambert, Albert Wilcox, Howard Zhang, Kristofer SJ Pister, Roberto Calandra
2021 IEEE Conference on Decision and Control
Trying to reframe the MBRL framework with long-term predictions instead of one-step predictions!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Learning for Microrobot Exploration: Model-based Locomotion, Robust Navigation, and Low-Power Deep Classification

Jul 13, 2020
Nathan Lambert, Fahran Toddywala, Brian Liao, Eric Zhu, Lydia Lee, Kristofer S.J. Pister
2020 Conference on Manipulation, Automation and Robotics at Small Scales (MARSS)
A collections of steps towards a data-driven autonomous microrobot.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Objective Mismatch in Model-based Reinforcement Learning

Feb 11, 2020
Nathan Lambert, Brandon Amos, Omry Yadan, Roberto Calandra
2020 Conference on Learning for Decision and Control
Studying the numerical effects of a dual-optimization problem in model-based reinforcement learning -- control and dynamics. When optimizing model accuracy, there is no guarantee on improving task performance!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning

Sep 26, 2019
Tianyu Li, Nathan Lambert , Roberto Calandra , Franziska Meier , Akshara Rai
2020 IEEE International Conference on Robotics and Automation (ICRA)
Learning how to walk with a real-world hexapod using a hierarchy of model-free RL for basic motion primitives with model-based RL for higher level planning.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning

Jan 11, 2019
Nathan Lambert, Daniel Drew, Joseph Yaconelli, Roberto Calandra, Sergey Levine, Kristofer Pister
IEEE Robotics and Automation Letters
We used deep model-based reinforcement learning to have a quadrotor learn to hover from less than 5 minutes of all experimental training data.
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Toward Controlled Flight of the Ionocraft: A Flying Microrobot Using Electrohydrodynamic Thrust With Onboard Sensing and No Moving Parts

Jul 18, 2018
Daniel Drew, Nathan Lambert, Craig Schindler, Kristofer SJ Pister
IEEE Robotics and Automation Letters
A collection of steps towards controlled flight of The Ionocraft, a completely silent microrobot with ion thrust!
[pdf][arxiv][code][video (<5 min)][talk (>15min)]

Bridging RLHF from LLMs back to control

November 6, 2023
|
CoRL LangRob Workshop 2023
What the LLM community should learn from roboticists, and how we can all RLHF better together.
[Watch Me]
[Slides]

Reinforcement Learning from Human Feedback: A Tutorial

July 24, 2023
|
International Conference on Machine Learning 2023
Introduction to the basics of RLHF :)!
[Watch Me]
[Slides]

Reinforcement Learning from Human Feedback: Open and Academic Perspectives

March 9, 2023
|
UCL Deciding, Acting, and Reasoning with Knowledge (DARK) Lab
A more technical version of my RLHF talk!
[Watch Me]
[Slides]

Intro to Reinforcement Learning from Human Feedback

December 13, 2022
|
HuggingFace Deep Reinforcement Learning Course
I try and take you from 0 to ChatGPT on RLHF with language models!
[Watch Me]
[Slides]

Planning through Exploration and Exploitation in Model-based Reinforcement Learning

April 25, 2022
|
University of Pennsylvania – Perception, Action, and Learning Group
A talk on the links between exploration, model exploitation, and intrinsic curiosity from the lens of model-based reinforcement learning.
[Watch Me]
[Slides]

(Dissertation Talk) Synergy of Prediction and Control in Model-based Reinforcement Learning

April 15, 2022
|
UC Berkeley EECS
My "defense" at Berkeley. As this is a talk I can only pass, I took liberties in trying to reflect the journey of a Ph.D.!
[Watch Me]
[Slides]

Machine Learning for Microsystem Control

April 6, 2022
|
Berkeley Sensor & Actuator Center Spring Review
An applied ML talk I gave at a sensor & actuator industry review event. I won best talk!
[Watch Me]
[Slides]

Improving Model Predictive Control Used in Model-based Reinforcement Learning

March 4, 2021
|
Cornell Robotics Seminar (Virtual)
How can we use a better understanding of dynamics models to improve data-driven model predictive control (MPC).
[Watch Me]
[Slides]

Bringing Model-based RL to Novel Robots

April 30, 2020
|
Practice Qualifying Exam
My practice quals talk! Learn about how I got into model-based reinforcement learning and what I wanted to do in the end of my Ph.D.
[Watch Me]
[Slides]

Model Learning for Low-level Control in Robotics

April 10, 2020
|
Semiautonomous Seminar (UC Berkeley)
A mixed talk discussing the research challenges of controlling microrobots and how model-learning can be used to synthesize highly specific controllers.
[Watch Me]
[Slides]

I have been lucky to work with many brilliant, younger students:

Berkeley Undergrads

Other Students

  • Brian Li (link forthcoming)