[MEVD] Specification/conformance integration test suite for providers

westey (@westey-m)

The current integration tests for NEVM providers are implemented separately for each provider, with no support from the abstraction. To promote better universal coverage and reduce the per-provider work needed, we could expoes a "specification test suite" with the abstraction, which providers implement, and which execute a set of standardized tests to ensure the provider works correctly. A good model here would be the Entity Framework Core specification test suite, which exercises an EF provider in a myriad of ways.

/cc westey (@westey-m)

Small note: I already refactored the integration tests for IVectorStore interface to have a single set of tests for each provider:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/IntegrationTests/Connectors/Memory/BaseVectorStoreTests.cs

I tried to do the same with IVectorStoreRecordCollection interface and it appeared that it's more complicated than IVectorStore, since each provider has some limitations (e.g. the requirements for collection/table/field names, supported index kinds, dimensions etc), which makes it harder to unify into a single test suite to cover all methods in that abstraction and all available connectors. At the end, I noticed that I still need to override some test cases for specific connectors to ensure high coverage and this put the idea of a single test suite under the question.

As an alternative solution, we can keep current tests for specific providers to ensure high coverage but add separate set of test cases (maybe basic ones) using abstraction only and enable it for as many connectors as we can.

Dmytro Struk (@dmytrostruk)

Dmytro Struk (@dmytrostruk) thanks for the context, yeah, that all makes sense. I plan to take a look at this, will share what i come up with. I agree that it's OK if an implementation needs to have its own tests in addition to the "standard" ones it can get for free - I do hope we'll be able to provide a meaningful set of standard ones, without being prevented from testing too much by differences between the database etc.

Note: the basic infrastructure for this and first test implementation are being done as part of #10273 (LINQ-based filtering).

Base test classes and fixtures to be extended by each provider (and published via a separate nuget). This is the main point of the exercise.
Use of testcontainers wherever possible; this simplifies setting up the Docker container, manages waiting for the service to actually be available (reduces flakiness), etc.
Where container-based testing isn't possible (since there's no Docker image), have the test suite check whether the required configuration for accessing the service is defined - and skip if not. This allows a dev to simply define e.g. an endpoint URL environment variable to activate testing a connector, while not failing tests where that config is missing (just skip).
Separate integration test project for each provider. This makes sense since we eventually would like connectors to be owned by their respective vector database vendors, and so ideally move out of the SK repo. This also separates the tests for different containers, allowing easier work on one without affecting others, etc.

Assuming we're all OK with this direction (we should discuss), am keeping this issue to track porting over the other non-filter integration tests over into this new integration test structure.

Base test classes and fixtures to be extended by each provider (and published via a separate nuget)

Do we need to publish test-related code via NuGet? If we need to share something internally between test projects - we have a concept of InternalUtilities in a repo, where we define some shared classes and then include them into specific projects where needed in compile time.

Add MEVD TestSuiteImplementationTests

.Net: Add MEVD TestSuiteImplementationTests (#12574)

.Net MEVD: More Conformance tests (microsoft#10944)

.Net: Add MEVD TestSuiteImplementationTests (microsoft#12574)

This is now fully complete.

Navigation Menu

Repository navigation

[MEVD] Specification/conformance integration test suite for providers #10194

39 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[MEVD] Specification/conformance integration test suite for providers #10194

Description

Activity

dmytrostruk commented on Jan 21, 2025

roji commented on Jan 22, 2025

roji commented on Feb 6, 2025

dmytrostruk commented on Feb 6, 2025

39 remaining items

roji commented on Jan 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions