Motivation

Learning goal-directed behavior is one of the classical problems in AI, one that has received renewed interest in recent years and currently sits at the crossroads of many seemingly-disparate research threads: self-supervised learning , representation learning, probabilistic inference, metric learning, and duality.

Our workshop focuses on these goal-conditioned RL (GCRL) algorithms and their connections to different areas of machine learning. Goal-conditioned RL is exciting not just because of these theoretical connections with different fields, but also because it promises to lift some of the practical challenges with applying RL algorithms: users can specify desired outcomes with a single observation, rather than a mathematical reward function. As such, GCRL algorithms may be applied to problems varying from robotics to language models tuning to molecular design to instruction following.

Our workshop aims to bring together researchers studying the theory, methods, and applications of GCRL, researchers who might be well posed to answer questions such as:

• How does goal-directed behavior in animals inform better GCRL algorithmic design?
• How can GCRL enable more precise and customizable molecular generation?
• Do GCRL algorithms provide an effective mechanism for causal reasoning?
• When and how should GCRL algorithms be applied to precision medicine?

Goal

The workshop aims to foster an inclusive environment where researchers and practitioners from all backgrounds can engage in discussions and build collaborations on the theory, methods, and applications of GCR.

Broadly, the workshop will focus on the following topics and problems:

• Connections: What are the connections between GCRL and representation learning, few-shot learning, and self-supervised learning? When does (say) effective representation learning emerge from GCRL?
• Future directions: What are limitations of existing methods, benchmarks, and assumptions?
• Algorithms: How might we improve existing methods, and do this in a way that enables applications to broader domains (e.g., molecular discovery, instruction-following robots)?

Speakers

Panelists

Schedule

Time (GMT-6)
09:00 am - 09:55 am		Poster Session 1
09:55 am - 10:00 am		Opening Remark
10:00 am - 10:40 am		Invited Speaker 1 Jeff Clune
10:40 am - 11:20 am		Invited Speaker 2 Reuth Mirsky
11:20 am - 12:00 pm		Invited Speaker 3 Olexandr Isayev
12:00 pm - 13:30 pm		Lunch Break
13:30 pm - 14:15 pm		Panel Discussion
14:20 pm - 15:00 pm		Invited Speaker 4 Yonatan Bisk
15:00 pm - 15:15 pm		Coffee Break
15:15 pm - 15:50 pm		Contributed Talks GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models Entity-Centric Reinforcement Learning for Object Manipulation from Pixels Is feedback all you need? Leveraging natural language feedback in goal-conditioned RL Causality in Goal Conditioned RL: Return to No Future? Automata Conditioned Reinforcement Learning with Experience Replay Empowering Clinicians with MeDT: A Framework for Sepsis Treatment
15:50 pm - 16:30 pm		Invited Speaker 5 Susan Murphy
16:30 pm - 17:30 pm		Poster Session 2

Call for Contributions
▻ 5-min video or 2-page report (You choose!)

Areas of Interest

We solicit submissions related to (but not limited to) the following topics:

• Algorithms.
  We encourage both proposals of new methods, as well as analyses and/or evaluations of existing ones.
• Connections between goal-conditioned RL and other ML areas.
  Examples might include representation learning, self-supervised learning, adversarial training, probabilistic inference, metric learning, duality, etc.
• Applications of goal-conditioned decision making.
  In addition to common decision making tasks (e.g., robotics and games) and goal-conditioned applications (e.g., instruction-following, molecular discovery), we especially encourage work in goal-conditioned domains where GCRL is not (yet) the mainstream strategy

NOTE: While there isn't an RL-specific workshop at NeurIPS this year, we will reject papers that only focuses on RL but not GCRL. We encourage authors to take methods developed for other problem settings and try applying them to GCRL domains (e.g., online robotics benchmark, offline robotics benchmark, language-conditioned agents).

Accessible Submission Format

The workshop will accept submissions in a more casual format to encourage new ideas and to be more accessible.

• A submission choose one of two tracks, which have different submission formats:
  • Video Track: Submit a video (e.g., narrated slide deck, recorded talk) in mp4 format with at most 5 minutes.
    Video track submissions must provide a pdf appendix, containing sufficient details of the method and experiments. Reviewers are instructed to evaluated submission based on the video, and consult the appendix if details aren't clear.
    The appendix should be in the NeurIPS style. At most 2-pages of appendix can be details of methods and experiments.
  • Short-Report Track: Submit A report in the NeurIPS style with at most 2 pages (unlimited on appendices and references).
• Submissions should maximize clarity and explanation. E.g., concise clear bullet points are preferred (even with incomplete sentences) over long detailed paragraphs.
• Submissions should still properly justify their significance.
• Accepted submissions will be required to submit a 4-page (or shorter) report containing sufficient details for future reference and reproducibility.

Submission Instructions

• All submissions will be managed through OpenReview.
• The review process is double-blind so the submission should be anonymized. For video submission, natural voice narration is still considered anonymized. However, the video must not show authors' faces or names.
• Format: 5-minute video or 2-page report. See above.
• One author of each submission must serve as a reviewer, responsible for reviewing up to 3 submissions.
• We have a small number of free conference registrations, which we will offer to authors from historically underrepresented groups.
• All participants must adhere to the NeurIPS Code of Conduct.

Review Guideline

• Submissions will be evaluated based on clarity, novelty, soundness, and relevance to theme of the workshop. Both empirical and theoretical contributions are welcomed.
• Reviewers are instructed to give feedbacks and ideas for future improvements.
• (To be finalized)

Important Dates

• Submission deadline: October 4th, 2023, AoE.
• Author Notifications: October 17th, 2023, AoE.
• Workshop: December 15th, 2023.

Papers

• Bi-Directional Goal-Conditioning on Single Value Function for State Space Search Problems [link]
  Vihaan Akshaay Rajendiran, Yu-Xiang Wang, Lei Li
• Does Hierarchical Reinforcement Learning Outperform Standard Reinforcement Learning in Goal-Oriented Environments? [link]
  Ziyan Luo, Yijie Zhang, Zhaoyue Wang
• Backward Learning for Goal-Conditioned Policies [link]
  Marc Höftmann, Jan Robine, Stefan Harmeling
• Numerical Goal-based Transformers for Practical Conditions [link]
  Seonghyun Kim, Samyeul Noh, Ingook Jang
• METRA: Scalable Unsupervised RL with Metric-Aware Abstraction [link]
  Seohong Park, Oleh Rybkin, Sergey Levine
• Simple Data Sharing for Multi-Tasked Goal-Oriented Problems [link]
  Ying Fan, Jingling Li, Adith Swaminathan, Aditya Modi, Ching-An Cheng
• Score-Models for Offline Goal-Conditioned Reinforcement Learning [link]
  Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, Scott Niekum
• Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets [link]
  Anirudhan Badrinath, Allen Nie, Yannis Flet-Berliac, Emma Brunskill
• GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models [link] (spotlight)
  Mianchu Wang, Rui Yang, Xi Chen, Meng Fang
• Hierarchical Empowerment: Toward Tractable Empowerment-Based Skill Learning [link]
  Andrew Levy, Sreehari Rammohan, Alessandro Allievi, Scott Niekum, George Konidaris
• Universal Visual Decomposer: Long-Horizon Manipulation Made Easy [link]
  Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, Luca Weihs
• Efficient Value Propagation with the Compositional Optimality Equation [link]
  Piotr Piękos, Aditya Ramesh, Francesco Faccio, Jürgen Schmidhuber
• Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data [link]
  Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, Sergey Levine
• Contrastive Difference Predictive Coding [link]
  Chongyi Zheng, Ruslan Salakhutdinov, Benjamin Eysenbach
• Using Proto-Value Functions for Curriculum Generation in Goal-Conditioned RL [link]
  Henrik Metternich, Ahmed Hendawy, Pascal Klink, Jan Peters, Carlo D'Eramo
• Entity-Centric Reinforcement Learning for Object Manipulation from Pixels [link] (spotlight)
  Dan Haramati, Tal Daniel, Aviv Tamar
• An Investigation into Value-Implicit Pre-training for Task-Agnostic, Sample-Efficient Goal-Conditioned Reinforcement Learning [link]
  Samyeul Noh, Seonghyun Kim, Ingook Jang, Hyun Myung
• Multi-Resolution Skill Discovery for Hierarchical Reinforcement Learning [link]
  Shashank Sharma, Vinay Namboodiri, Janina Hoffmann
• Middle-Mile Logistics Through the Lens of Goal-Conditioned Reinforcement Learning [link]
  Onno Eberhard, Thibaut Cuvelier, Michal Valko, Bruno Adrien De Backer
• Is feedback all you need? Leveraging natural language feedback in goal-conditioned RL [link] (spotlight)
  Sabrina McCallum, Max Taylor-Davies, Stefano Albrecht, Alessandro Suglia
• Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View. [link]
  Raj Ghugare, Matthieu Geist, Glen Berseth, Benjamin Eysenbach
• GROOT: Learning to Follow Instructions by Watching Gameplay Videos [link]
  Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, Yitao Liang
• Causality in Goal Conditioned RL: Return to No Future? [link] (spotlight)
  Ivana Malenica, Susan Murphy
• Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models [link]
  Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, Sergey Levine
• Automata Conditioned Reinforcement Learning with Experience Replay [link] (spotlight)
  Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, Sanjit Seshia
• Empowering Clinicians with MeDT: A Framework for Sepsis Treatment [link] (spotlight)
  Aamer Abdul Rahman, Pranav Agarwal, Vincent Michalski, Rita Noumeir, Samira Kahou
• Goal-Conditioned Recommendations of AI Explanations [link]
  Saptarashmi Bandyopadhyay, Vibhu Agrawal, Sarah Savidge, Eric Krokos, John P Dickerson
• STEVE-1: A Generative Model for Text-to-Behavior in Minecraft (Abridged Version) [link]
  Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, Sheila McIlraith
• Asymmetric Norms to Approximate the Minimum Action Distance [link]
  Lorenzo Steccanella, Anders Jonsson
• Goal-Conditioned Predictive Coding for Offline Reinforcement Learning [link]
  Zilai Zeng, Ce Zhang, Shijie Wang, Chen Sun
• Goal Misgeneralization as Implicit Goal Conditioning [link]
  Diego Dorn, Neel Alex, David Krueger

Organizers

Contact
Reach out to gcrlworkshop@gmail.com for any questions.

Program Committee
We would like to thank the following people for their agreeing to review and making this workshop a success!

Sponsors