OpenAI

This page serves as a locus for everything related to OpenAI.

Software stack

OpenAI has disclosed the following about their software stack:

• They use a private monorepo for their code. This was stated in some video they posted about testing with data that they know wasn’t in the training dataset.
• They have used Ray for training GPT 3.5 and GPT 4.0.¹ It is unclear if they have used it for training since then, or if they use it for inferencing at all.
• They have used Kubernetes on their large training clusters.²
• They have used Apache Spark for data preprocessing. This was mentioned in the GPT-3 paper.

Training techniques

• They used multicluster training for GPT-4.5.

Footnotes

1. https://www.anyscale.com/glossary/what-is-ray ↩

2. https://openai.com/index/scaling-kubernetes-to-7500-nodes/ ↩