April 2021: Research Foundations

Greetings from Bend, Oregon. Vaccines are wonderful.


My next conundrum of life and research is packaging everything I have learned and care about into a dissertation. It is hard to part with the ideas that I will not have time to pursue. I’ve put on my data scientist hat big time in the last month analyzing trends in how distributions and core values propagate through model-based reinforcement learning systems.

That is: what does that data look like? Rather than: what is the system doing?

I am lucky to have been early in working on reinforcement learning, as I think the systems all technology companies use with deploying models reduce to the loop of: deploy a new model, collect new data (which affects the environment — normally user behavior), and train a new model that is the next iteration of that framework.

RL as a framework is where you can learn how to study the long-term effects of these data-driven technologies. Studying the underlying numerical happenings in these systems feels like learning a new language that the future will be built on — whether many people are fluent in it.

Research & Compounding

I have worked really hard during my Ph.D. During COVID it has been compounded by my main mental coping mechanism being to find satisfaction (normally of the temporary variant) through my work. I’m reaching a transition point in this: coping through healthier mechanisms and my hard work being apparent.

This year has started as one when many of my long-term projects and collaborations come to fruition. In academia, you only get credit when your work functions and is done, so the timeline for that is very unpredictable.

Let it be said for all the students out there earlier in a Ph.D.: it does work out. Learning and creating knowledge takes way longer than it may initially seem. An undergraduate expects to try research for a semester (4 months) and in reality most projects take around 2-3 years of work to start kicking into steady drive. It’s remarkable how poorly this is communicated to younger students and how snapshots of research early in careers are hard to read.


I have been reading a lot more recently. I have two recommendations relevant to this audience:

  1. Genius Makers by Cade Metz (brief review). If you work in technology, this book shows how and why the biggest companies are investing so heavily in deep learning. Through observing the pace of it in the last decade, it is easy to see that both a) there will be many, many more new industries centered around deep learning in the next decade (it’s on a sigmoid growth) and b) there is so much potential harm of this technology.
  2. "Surely You're Joking, Mr. Feynman!” by Richard Feynman (brief review). This book made me feel like and love the goals of science again. After a big integration of my work online, being on Twitter, and more, I am shifting back in the direction of the Slow Science Manifesto (interesting read). As many things in life go in phases, I think going in cycles of deep work and collaboration followed by promotion and new connections would be optimal in the long term.

Some done things:

Media & Academic

2021 has so many of my projects being released! At 4 already 🚀, 3 more are close.:

  • We released MBRL-Lib: A Modular Library for Model-based Reinforcement Learning (page link), which has been in the works for a couple years. I wish I had this tool for more of my Ph.D.
  • Another peer-reviewed piece on the sociotechnics of AI: Axes for Sociotechnical Inquiry in AI Research (page link). We discuss how practitioners can evaluate the relative risk of new data-driven technologies!


I’ve been helping write so many papers this semester, the bandwidth for blogging has been low. Though, it’s coming back with hopefully a higher quality and more sustainable format (leveraging friend-pre-review!):

  • The main tools of the trade of my Ph.D.: Debugging model-based reinforcement learning systems. This is a long blog post where you can learn a lot about how MBRL systems work if you invest the time. It pairs well with the open-source repository!
  • Democratizing Automation: Coming back into the fold. See you this summer! Some of my ideas are so clearly replicated in the RL Book, so I am zooming out and thinking about where my contributions lay.

See you next month! Search for more!