How we evaluate AI models and LLMs for GitHub Copilot

There are so many AI models to choose from these days. From the proprietary foundation models of OpenAI, Google, and Anthropic to the smaller, more open options from the likes of Meta and Mistral. It’s tempting to hop immediately to the latest models. But just because a model is newer doesn’t mean it will perform better for your use case. We recently expanded the models available in GitHub Copilot by adding support for Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5 Pro, and OpenAI’s o1-preview and o1-mini. While adding models to GitHub Copilot, performance, quality, and safety was always kept top of mind. In this article, we’ll share some of the GitHub Copilot team’s experience evaluating AI models, with a focus on our offline evaluations—the tests we run before making any change to our production environment. Hopefully our experience will help guide your own evaluations.

AI models are systems that combine code, algorithms, and training data to simulate human intelligence in some way. GitHub Copilot gives users the choice of using a number of AI models trained on human language, which are known as large language models, or LLMs. OpenAI’s models are some of the most well-known LLMs due to ChatGPT’s popularity, though others such as Claude, Gemini, and Meta’s Llama models are increasingly popular as well.AI models are systems that combine code, algorithms, and training data to simulate human intelligence in some way. GitHub Copilot gives users the choice of using a number of AI models trained on human language, which are known as large language models, or LLMs. OpenAI’s models are some of the most well-known LLMs due to ChatGPT’s popularity, though others such as Claude, Gemini, and Meta’s Llama models are increasingly popular as well.

Automating code quality tests We run more than 4,000 offline tests, most of them as part of our automated CI pipeline. We also conduct live internal evaluations, similar to canary testing, where we switch a number of Hubbers to use a new model. We test all major changes to GitHub Copilot this way, not just potential new models.

GitHub Models makes it simple to use, compare, experiment, and build with AI models from OpenAI, Cohere, Microsoft, Mistral, and more. More Details

Choose the Right AI Model for Your Needs – Just Chase the Latest!

Copilot is now included in Microsoft 365 Personal and Family

Since we launched Microsoft 365 (formerly Office 365) to consumers in 2013, we’ve steadily delivered new apps, features, and benefits to our subscribers. These include advanced security with Microsoft Defender, creative tools like Microsoft Clipchamp, and countless enhancements to Word, Excel, PowerPoint, OneNote, and Outlook. We’re building on those 12 years of innovation by bringing Microsoft Copilot and Microsoft Designer to Microsoft 365 Personal and Family subscribers in most markets worldwide.1 These changes bring the transformative power of AI to the personal productivity tools that millions of people use every day. Note: these changes only apply to our consumer subscription plans.

Microsoft 365 Personal and Family subscribers will receive a monthly allotment of AI credits to use Copilot in Word, Excel, PowerPoint, Outlook, and OneNote. The credits can also be used for AI image generation and editing in Designer and in other AI-powered apps like Paint, Photos, and Notepad on a device running Windows. The monthly allotment should be enough for most subscribers. For Microsoft 365 Family subscribers, Copilot will be available to the subscription owner and cannot be shared with others. Those who frequently use Copilot can upgrade to Copilot Pro, without worrying about usage limits. Learn more about AI credits.

We recognize that our customers have questions about AI privacy, including how we train our AI models. Transparency is one of our six responsible AI principles, so we’re sharing more details here. To protect your privacy, we do not use your prompts, responses, or file content (such as Word documents or Excel spreadsheets) when you use Copilot in the Microsoft 365 apps to train our foundation models.... Read More

Unlock AI-powered tools with Microsoft 365 Personal and Family — Upgrade today!

Boost processing performance by combining AI models

Leveraging the strengths of different AI models and bringing them together into a single application can be a great strategy to help you meet your performance objectives. This approach harnesses the power of multiple AI systems to improve accuracy and reliability in complex scenarios. In the Microsoft model catalog, there are more than 1,800 AI models available. Even more models and services are available via Azure OpenAI Service and Azure AI Foundry, so you can find the right models to build your optimal AI solution. Let’s look at how a multiple model approach works and explore some scenarios where companies successfully implemented this approach to increase performance and reduce costs.

Let’s suppose you want to pair a fine-tuned vision model with a large language model to perform several complex imaging classification tasks in conjunction with natural language queries. Or maybe you have a small model fine-tuned to generate SQL queries on your database schema, and you’d like to pair it with a larger model for more general-purpose tasks such as information retrieval and research assistance. In both of these cases, the multiple model approach could offer you the adaptability to build a comprehensive AI solution that fits your organization’s particular requirements.

One innovative application of multiple model processing is to route tasks simultaneously through different multimodal models that specialize in processing specific data types such as text, images, sound, and video. For example, you can use a combination of a smaller model like GPT-3.5 turbo, with a multimodal large language model like GPT-4o, depending on the modality. This routing allows an application to process multiple modalities by directing each type of data to the model best suited for it, thus enhancing the system’s overall performance and versatility.

To learn more about enhancing the reliability, security, and performance of your cloud and AI investments, explore the additional resources below. Read More

Unlock 1,800+ AI models to build your solution today! — Get started today!

Deep dive into TMDL view for Power BI Desktop (Preview)

We’re excited to introduce TMDL view for Power BI Desktop. This new feature lets you script, modify, and apply changes using Tabular Model Definition Language (TMDL), providing an alternative experience to semantic modeling using code, instead of a graphical user interface such as Model view. Future updates will integrate Copilot, laying the foundation for advanced Copilot modeling interactions handling time-intensive tasks such as model translations, bulk operations, among others. Using TMDL view to view and comprehend the semantic model metadata can be highly beneficial, even if modifications are not intended. For example, it may be necessary to review and reuse a specific Power Query expression from a table in another file or verify that all measures have the correct format string configuration.

TMDL view introduces a new capability that allows users to share and reuse semantic model objects easily within Power BI Desktop. For example, if you want to share a calculation group you created with a colleague, you just need to follow these steps:

• Script the calculation group into TMDL view.
• Copy the text and share it with your colleague via Teams chat or email.
• Your colleague can then paste the script into a new TMDL view tab and run it to get exactly the same calculation group created on his semantic model.

Learn more about TMDL view features and its limitations in our documentation: Work with TMDL view in Power BI Desktop (preview)... Read More

Unlock TMDL View in Power BI Desktop: Streamline Modelling with Code — Connect with MTC Experts! 

The future of retail with Dynamics 365 AI-powered ERP solutions

This week, thousands of professionals will gather at the National Retail Federation (NRF) 2025: Retail’s Big Show, for insights into the changing retail landscape. Top of mind for many attendees is how AI will impact the retail industry. What retail-specific AI innovation is on the horizon, and can those solutions solve the myriads of challenges retailers face, from improving customer experiences to meeting customer demand on time? To help answer these questions, Microsoft leaders and industry partners will be on hand to showcase the very latest AI innovation for retailers—including how agents can open doors to unprecedented customer experiences, help meet customer demand on time, and drive ethical sourcing and sustainability.

At the Microsoft expo booth and our featured session, NRF attendees will learn how organizations can achieve these priorities by infusing AI across every business process and workflow—within both customer-facing and backend operations. By integrating AI solutions like Microsoft Copilot and agents within ERP systems—the traditional foundation for managing critical business processes—retailers can scale the delivery of personalized customer experiences and flexible shopping options.

Future-proof your retail operations in 2025 AI-powered solutions for ERP and service systems are more than a technological upgrade. They are a strategic imperative for retailers aiming to deliver exceptional customer experiences, build lasting brand loyalty, and achieve sustainable profitability. In today’s business landscape, you need autonomous systems that can anticipate market changes, customer needs, or operational risks. The risk of clinging to legacy systems extends beyond inefficiency. It leads to missed opportunities, slower time-to-market, and the inability to remain competitive. Companies using AI-first processes through AI-powered solutions for ERP systems gain the foresight and agility necessary to not just react to changes but to lead through them... Read More

Join us at NRF 2025 to explore AI innovations transforming retail! — Get started now!