Watchtower

OpenAI published a new system card on Oct. 7 for GPT-6 Sol and GPT-6 Luna, covering new versions of the two models available in the basic ChatGPT interface. It makes a distinction between these new versions and the previous GPT-6 Sol and GPT-6 Luna, described by Appendix A of the GPT-6 Astra system card, which are still available in Codex and via ChatGPT Work. The versions detailed in this system card are referred to as GPT-6 Sol (October) and GPT-6 Luna (October). 

Many evaluations are changed or not included between Appendix A and the new system card. For many that remain the same, GPT-6 Sol (October) and GPT-6 Luna (October) are benchmarked against GPT-5.6 Sol (August) and GPT-5.6 Luna (August), making it difficult to compare results against the evaluations of these models found in Appendix A of the GPT-6 Astra system card.

Both models retain the “High” capability rating in Cybersecurity and Biological/Chemical domains, and below High in AI self-improvement. On “Tacit Knowledge and Troubleshooting,” a Biological capability evaluation, GPT-6 Luna (October) crossed a High bio threshold the September version didn’t, moving from a score of 69.2% in September to 94.0% in this card.

Despite this, the card repeats a sentence from the previous Appendix A, stating that Critical testing “was not required for GPT-6 Luna because it scored below GPT-5.6 Sol on all High capability evaluations.” This claim is not true for GPT-6 Luna (October) due to the jump on its Tacit Knowledge score. Critical Biological testing for GPT-6 Luna (October) was still performed, with Luna scoring below the threshold on the three evaluations. 

In safety evaluations, GPT-6 Sol (October) and GPT-6 Luna (October) both handled harmful requests less safely than its September versions, although fewer benign requests were refused.

In alignment evaluations, the card contains a new caveat. OpenAI observed the models “cite the synthetic nature of this evaluation as justification for the performance of improper actions.”

This is not an exhaustive list of the differences between the two cards.