> ## Content Index > Fetch the complete content index at: https://codetraveler.io/llms.txt > Use this file to discover other available public pages before exploring further. # Unit Testing LLMs: Evaluating LLM Quality in .NET with `Microsoft.Extensions.AI.Evaluation` - URL: https://codetraveler.io/2026/08/03/unit-testing-llms-evaluating-llm-quality-in-net-with-microsoft-extensions-ai-evaluation/ - Published: 2026-08-03T09:00:41.000Z - Updated: 2026-08-03T16:57:02.000Z - Author: Brandon Minnick How do we unit test an LLM that never gives the same answer twice? [Microsoft.Extensions.AI.Evaluation](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation?ref=codetraveler.io) makes it easy. This is Part 4 of our `Microsoft.Extensions.AI` series. In [Part 1](https://codetraveler.io/2026/07/16/adding-ai-to-your-net-app-with-ichatclient/) we built a chat app with `IChatClient` — but how do we know its answers are any *good*? This week, we'll write a unit test that grades our LLM! You're welcome to skip the definitions and jump directly to the code below. ## Why Can't We Just Write Normal Unit Tests? `Assert.AreEqual` is useless for LLMs because the exact wording changes on every run. Instead, we grade the *meaning*. `Microsoft.Extensions.AI.Evaluation` uses a second LLM to score the first LLM's answer against our criteria. It has ready-made evaluators that each return a score from 1 (bad) to 5 (great). ## What is EquivalenceEvaluator? `EquivalenceEvaluator` answers one question: "Does the LLM's response mean the same thing as my expected answer?" We provide the expected answer (the "ground truth") via `EquivalenceEvaluatorContext`, and the judge compares the two on that 1-5 scale. Score a 4 or 5, and our test passes! ## Code Now that we have those pesky definitions out of the way, let's write some code! You can find the [completed code sample here](https://github.com/Dometrain/from-zero-to-hero-microsoft-extensions-ai/tree/main/4.%20Unit%20Testing%20LLMs?ref=codetraveler.io). ### Prerequisites If you'd like to follow along, here are the steps. We'll use [Ollama](https://ollama.com/?ref=codetraveler.io) as both the LLM under test AND the judge, so this entire test suite runs completely free! 1. [Download + install Ollama](https://ollama.com/download?ref=codetraveler.io) and pull a model: `ollama pull qwen3.5` 2. Create a [File -> New NUnit Test Project in Visual Studio](https://learn.microsoft.com/dotnet/core/testing/unit-testing-csharp-with-nunit?ref=codetraveler.io) 3. Add the [Microsoft.Extensions.AI](https://www.nuget.org/packages/Microsoft.Extensions.AI?ref=codetraveler.io), [Microsoft.Extensions.AI.Evaluation](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation?ref=codetraveler.io), [Microsoft.Extensions.AI.Evaluation.Quality](https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Quality?ref=codetraveler.io) and [OllamaSharp](https://www.nuget.org/packages/OllamaSharp?ref=codetraveler.io) NuGet Packages to the test project. > If you're unfamiliar with adding NuGet Packages, you can find more information [here](https://learn.microsoft.com/nuget/consume-packages/install-use-packages-visual-studio?ref=codetraveler.io#find-and-install-a-package). Now we can finally write some code! ### Unit Test ```csharp using Microsoft.Extensions.AI; using Microsoft.Extensions.AI.Evaluation; using Microsoft.Extensions.AI.Evaluation.Quality; using OllamaSharp; namespace AiEvaluation.UnitTests; public class ChatClientTests { [Test] public async Task LlmResponse_MatchesExpectedAnswer() { // Arrange IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "qwen3.5"); var equivalenceEvaluator = new EquivalenceEvaluator(); var equivalenceContext = new EquivalenceEvaluatorContext("The sky is blue"); var question = new ChatMessage(ChatRole.User, "What color is the sky on a clear day? Answer in one short sentence."); // Act ChatResponse response = await client.GetResponseAsync([question]); EvaluationResult result = await equivalenceEvaluator.EvaluateAsync( [question], response, new ChatConfiguration(client), [equivalenceContext]); NumericMetric equivalenceMetric = result.Get(EquivalenceEvaluator.EquivalenceMetricName); Assert.That(equivalenceMetric.Value, Is.GreaterThanOrEqualTo(4)); // 1 = completely different, 5 = fully equivalent } } ``` Now let's click **Run** and see what gets printed to the console: ``` Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1 ``` It works! The LLM answered in its own words, the judge compared the *meaning* to our ground truth, and the score cleared our threshold. No brittle string comparisons required! 🎉 ### More Evaluators `EquivalenceEvaluator` is just one member of the family. `Microsoft.Extensions.AI.Evaluation.Quality` also ships the following evaluators: - `RelevanceEvaluator` - `CoherenceEvaluator` - `FluencyEvaluator` - `CompletenessEvaluator` - `GroundednessEvaluator` - `TaskAdherenceEvaluator` - `ToolCallAccuracyEvaluator` All use the same `EvaluateAsync` pattern above. Learn one, and you've learned them all 💯 ## Conclusion As .NET developers, we can finally bring our unit testing discipline to AI features, thanks to `Microsoft.Extensions.AI.Evaluation`. And we've only scratched the surface! The [Microsoft.Extensions.AI.Evaluation.Reporting generates reports](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries?ref=codetraveler.io#reporting) that you can use to augmentadd to your CI pipeline. You can find the completed code sample here: [https://github.com/Dometrain/from-zero-to-hero-microsoft-extensions-ai](https://github.com/Dometrain/from-zero-to-hero-microsoft-extensions-ai/tree/main/4.%20Unit%20Testing%20LLMs?ref=codetraveler.io) Next week, we'll check whether our AI's answers are *safe* to ship using `Microsoft.Extensions.AI.Evaluation.Safety`. Make sure to subscribe to this blog so you don't miss it! 👇 To learn even more about Microsoft Extensions AI, check out my course on Dometrain: [Learn to use Microsoft.Extensions.AI in .NET - DometrainMaster the main AI abstraction library for .NET AI development![](https://storage.ghost.io/c/dc/c2/dcc2e43e-24f8-46bb-a899-9c446f3ef794/content/images/icon/favicon-32x32-04d20a86-5edd-481a-9ddb-d09f72f19e88.png)DometrainBrandon Minnick![](https://storage.ghost.io/c/dc/c2/dcc2e43e-24f8-46bb-a899-9c446f3ef794/content/images/thumbnail/thumbnail-e14b39ca760b44139b0be9246741f6cb-ad5148f9-591f-4bcf-8d9e-5cd8bde3242e.jpeg)](https://dometrain.com/course/from-zero-to-hero-microsoft-extensions-ai-in-dotnet?ref=codetraveler.io)