Today, OpenAI released OpenAI o1, a new series of AI models equipped with advanced reasoning capabilities to solve hard problems. Like you, we are excited to put the new o1 model through its paces and have tested integrating o1-preview with GitHub Copilot. While we are exploring many use cases with this new model, such as debugging large-scale systems, refactoring legacy code, and writing test suites, our initial testing showed promising results in code analysis and optimization. This is because of o1-previewโs ability to think through challenges before responding, which enables Copilot to break down complex tasks into structured steps.
In this blog, weโll describe two scenarios showcasing the new modelโs capabilities within Copilot and how it could work for your day to day. Keep reading for an inside look at what happens when a new model launches, what we test, and how we approach AI-powered software development at GitHub.
Optimize Complex Algorithms With Advanced Reasoning
In our first test, we wanted to understand how o1-preview could help write or refine complex algorithms, a task that requires deep logical reasoning to find more efficient or innovative solutions. Developers need to understand the constraints, optimize edge cases, and iteratively improve the algorithm without losing track of the overall objective. This is exactly where o1-preview excels. With this in mind, we developed a new code optimization workflow that benefits from the modelโs reasoning capabilities.
In this demo, a new built-inย Optimizeย chat command provides rich editor context out of the box, like imports, tests, andย performance profiles. We tested how well o1-preview could analyze and iterate code to come up with a more thorough and efficient optimization in one shot.
We optimised the performance of a byte pair encoderย used in Copilot Chatโsย tokenizer libraryย (yes, this means we use AI to optimize a key AI development building block).
This was a real problem the VS Code team faced, as Copilot needs to repeatedly tokenize large amounts of data while it assembles prompts.
The results highlight how o1-previewโs reasoning capability allows a deeper understanding of the codeโs constraints and edge cases, which helps produce a more efficient and higher quality result. Meanwhile, GPT-4o sticks to obvious optimizations and would need a developerโs help to steer Copilot towards more complex approaches.
Beyond handling complex code tasks, o1-previewโs math abilities shine as it effortlessly calculates the benchmark results from the raw terminal output, then summarizes them succinctly.
Optimize Application Code to Fix a Performance Bug
In this next demo on GitHub, o1-preview was able to identify and develop a solution forย a performance bug within minutes. The same bug took one of our software engineers a few hours before they came up with the same solution. At the time, we wanted to add a folder tree to the file view in GitHub.com, but the number of elements was causing our focus management code to stall and crash the browser.
With 1,000 elements managed by this code, it was hard to isolate the problem. Eventually we implemented a change that improved the runtime of this function from over 1,000ms to about 16ms. If we had Copilot with o1-preview, we could have quickly identified the problem and fixed it faster.
Through this experimentation, we found a subtle but powerful difference, which is how deliberate and purposeful o1-previewโs responses are, making it easy for the developer to pinpoint problems and quickly implement solutions. With GPT-4o, a similar prompt might result in a blob of code instead of a solution with recommendations broken down line by line.
Bringing The Power of o1-preview to Developers Building on GitHub
Not only are we excited to experiment with integrating o1-preview into GitHub Copilot, we canโt wait to see what youโll be able to build with it too. Thatโs why weโre bringing the o1 series toย GitHub Models. Youโll find o1-preview and o1-mini, a smaller, faster, and 80% cheaper model, in our marketplace later today, but because it is still in preview youโll need toย sign up for Azure AIย for early access.
Stay Tuned
As part of Microsoftโs collaboration with OpenAI, GitHub is able to constantly explore how we can leverage the latest AI breakthroughs to drive developer productivity, and, most importantly, increase developer happiness. Although these demos showcase o1-previewโs enhanced capabilities for two specific optimization problems, weโre still early in our experimentation and are excited to see what else it can do.
Weโre currently exploring more use cases across Copilotโin IDEs,ย Copilot Workspace, and on GitHubโto leverage o1-previewโs strong reasoning capabilities to accelerate developer workflows even further. The advancements weโre showcasing today barely scratch the surface of what developers will be able to build with o1-preview in GitHub Copilot. And with the expected evolution of both the o1 and GPT series, this is just the beginning.
