Welcome to OpenRLHF’s documentation!

OpenRLHF is the first easy-to-use, high-performance RLHF framework built on Ray, vLLM, ZeRO-3 and HuggingFace Transformers, designed to make RLHF training simple and accessible:

• Distributed Architecture with Ray OpenRLHF leverages Ray for efficient distributed scheduling. It separates the Actor, Reward, Reference, and Critic models across different GPUs, enabling scalable training for models up to 70B parameters. It also supports Hybrid Engine scheduling, allowing all models and vLLM engines to share GPU resources—minimizing idle time and maximizing GPU utilization.

• vLLM Inference Acceleration + AutoTP RLHF training spends 80% of the time on the sample generation stage. Powered by vLLM and Auto Tensor Parallelism (AutoTP), OpenRLHF delivers high-throughput, memory-efficient samples generation. Native integration with HuggingFace Transformers ensures seamless and fast generation, making it the fastest RLHF framework available.

• Memory-Efficient Training with ZeRO-3 Built on DeepSpeed’s ZeRO-3 and deepcompile, OpenRLHF enables large model training without heavyweight frameworks. It works directly with HuggingFace for easy loading and fine-tuning of pretrained models.

• Optimized PPO Implementation Incorporates advanced PPO tricks inspired by practical guides and community best practices, enhancing training stability and reward quality in RLHF workflows. Referencing Zhihu and Advanced Tricks for Training Large Language Models with Proximal Policy Optimization.

For more technical details, see our technical report and slides.

Check out the Quick Start section for further information, including how to Installation the project.

Note

This project is under active development.

Contents

• Quick Start
  • Installation
  • Prepare Datasets
  • Pretrained Models
  • RLHF with Ray and vLLM
  • One-Click Installation Script of Nvidia-Docker
• Reinforcement Learning from Human Feedback (RLHF)
  • Common Options
  • Supervised Fine-tuning
  • Reward Model Training
  • Process Reward Model (PRM) Training
  • PPO with Ray (vLLM)
  • REINFORCE++ /RLOO with Ray (vLLM)
• Non-RL Methods
  • Direct Preference Optimization (DPO)
  • Kahneman-Tversky Optimization (KTO)
  • Rejection Sampling & RAFT
    • Options for openrlhf.cli.batch_inference
  • Iterative DPO
  • Conditional SFT
  • Knowledge Distillation (MiniLLM)
• Hybrid Engine
  • Ray RLHF using Hybrid Engine
• Performance Tuning Guide
  • Ray PPO
  • SFT/RM/DPO/PPO training
  • Batch Inference
• Multi-node Training
  • How to launch Ray PPO on Slurm?
  • How to launch SFT/RM/DPO training on Slurm?
  • How to specify a particular node for a model in Ray PPO?
• Remote Reward Server & Reinforced Fine-tuning
  • How to specify the reward model on a remote server?
  • Reinforced Fine-tuning
• Checkpoint
  • How to save training checkpoints?
• Sequence Parallelism
  • Ring Attention
  • Examples