Auto-Test Provider Extensibility

BerryCrush’s auto-test feature supports custom test providers through Java’s ServiceLoader pattern. This allows users to add custom invalid request tests, security tests, and multi-request idempotency tests without modifying the core library.

Overview

The auto-test system uses three types of providers:

InvalidTestProvider - Generates complete invalid AutoTestCase entries
SecurityTestProvider - Generates complete security AutoTestCase entries
MultiTestProvider - Executes multi-request idempotency tests

All types are discovered automatically via ServiceLoader, allowing you to add custom providers by simply adding classes to your project.

Built-in Providers

Invalid Test Providers

Test Type	Description
minLength	Strings shorter than minLength constraint
maxLength	Strings longer than maxLength constraint
pattern	Strings not matching the pattern
format	Invalid format values (email, uuid, date, etc.)
enum	Values not in the enum list
minimum	Numbers below the minimum
maximum	Numbers above the maximum
type	Wrong type values (e.g., string for number field)
required	Missing required fields
minItems	Arrays with fewer items than minItems
maxItems	Arrays with more items than maxItems

Security Test Providers

Test Type	Display Name	Description
SQLInjection	SQL Injection	SQL injection attack payloads
XSS	XSS	Cross-site scripting payloads
PathTraversal	Path Traversal	Path traversal attack patterns
CommandInjection	Command Injection	Shell command injection payloads
LDAPInjection	LDAP Injection	LDAP query injection payloads
XXE	XXE	XML External Entity payloads
HeaderInjection	Header Injection	HTTP header injection payloads
NoSQLInjection	NoSQL Injection	MongoDB-style operator payloads
SSTI	Template Injection	Server-side template injection payloads
JWT	JWT Attacks	JWT tampering and malformed token payloads
AuthorizationBypass	Authorization Bypass	Missing/invalid auth header payloads

Built-in provider inventory is intentionally stable so existing excludes usage remains compatible.

Multi Test Providers

Test Type	Display Name	Description
sequential	Sequential Idempotency	Executes requests one after another
concurrent	Concurrent Idempotency	Executes requests in parallel using a thread pool

Creating Custom Providers

Custom Invalid Test Provider

Create a class implementing InvalidTestProvider:

package com.example

import org.berrycrush.autotest.provider.InvalidTestProvider
import org.berrycrush.autotest.AutoTestCase
import org.berrycrush.autotest.AutoTestType
import org.berrycrush.autotest.provider.InvalidTestRequest
import io.swagger.v3.oas.models.media.Schema

class NumericOverflowProvider : InvalidTestProvider {
    // Unique identifier for test reports and excludes
    override val testType: String = "numericOverflow"
    
    // Higher priority overrides built-in providers with same testType
    override val priority: Int = 100

    override fun canHandle(schema: Schema<*>): Boolean =
        schema.type == "integer" || schema.type == "number"

    override fun generateTestCases(request: InvalidTestRequest): List<AutoTestCase> {
        if (request.location != org.berrycrush.autotest.ParameterLocation.BODY) return emptyList()

        val overflowBody = request.baseBody.toMutableMap().apply {
            this[request.fieldName] = Long.MAX_VALUE
        }

        val infinityBody = request.baseBody.toMutableMap().apply {
            this[request.fieldName] = Double.POSITIVE_INFINITY
        }

        return listOf(
            AutoTestCase(
                type = AutoTestType.INVALID,
                fieldName = request.fieldName,
                invalidValue = Long.MAX_VALUE,
                description = "Numeric overflow value",
                location = request.location,
                body = overflowBody,
                pathParams = request.basePathParams,
                headers = request.baseHeaders,
                tag = "Invalid request - $testType",
            ),
            AutoTestCase(
                type = AutoTestType.INVALID,
                fieldName = request.fieldName,
                invalidValue = Double.POSITIVE_INFINITY,
                description = "Infinity value",
                location = request.location,
                body = infinityBody,
                pathParams = request.basePathParams,
                headers = request.baseHeaders,
                tag = "Invalid request - $testType",
            ),
        )
    }
}

Custom Security Test Provider

Create a class implementing SecurityTestProvider:

package com.example

import org.berrycrush.autotest.ParameterLocation
import org.berrycrush.autotest.AutoTestCase
import org.berrycrush.autotest.AutoTestType
import org.berrycrush.autotest.provider.SecurityTestProvider
import org.berrycrush.autotest.provider.SecurityTestRequest

class NoSqlInjectionProvider : SecurityTestProvider {
    // Unique identifier for excludes
    override val testType: String = "NoSQLInjection"
    
    // Human-readable name for test reports
    override val displayName: String = "NoSQL Injection"
    
    // Higher priority overrides built-in providers with same testType
    override val priority: Int = 100

    override fun applicableLocations(): Set<ParameterLocation> =
        setOf(ParameterLocation.BODY, ParameterLocation.QUERY)

    override fun generateTestCases(request: SecurityTestRequest): List<AutoTestCase> {
        if (request.location !in applicableLocations()) return emptyList()

        return listOf(
            "MongoDB \$ne injection" to "{\"\$ne\": null}",
            "MongoDB \$where injection" to "{\"\$where\": \"sleep(5000)\"}",
        ).map { (name, payload) ->
            val body =
                if (request.location == ParameterLocation.BODY) {
                    request.baseBody.toMutableMap().apply { this[request.fieldName] = payload }
                } else {
                    request.baseBody
                }

            AutoTestCase(
                type = AutoTestType.SECURITY,
                fieldName = request.fieldName,
                invalidValue = payload,
                description = "$displayName: $name",
                location = request.location,
                body = body,
                pathParams = request.basePathParams,
                headers = request.baseHeaders,
                tag = "security - $displayName",
            )
        }
    }
}

Registering via ServiceLoader

Create a service configuration file in your project:

src/main/resources/META-INF/services/org.berrycrush.autotest.provider.InvalidTestProvider:

com.example.NumericOverflowProvider

src/main/resources/META-INF/services/org.berrycrush.autotest.provider.SecurityTestProvider:

com.example.NoSqlInjectionProvider

The providers will be automatically discovered and registered when the auto-test system initializes.

Migration Notes

The provider contract now expects complete AutoTestCase generation.

Preferred API:
- InvalidTestProvider.generateTestCases(request)
- SecurityTestProvider.generateTestCases(request)
Legacy APIs (generateInvalidValues, generatePayloads) are deprecated compatibility adapters.
AutoTestGenerator orchestrates provider calls only and no longer assembles provider-specific cases.
Missing-required cases are provider-owned via the built-in required provider.

Provider Priority

Each provider has a priority property (default: 0 for built-in, 100 for user providers):

Higher priority providers override lower priority ones with the same testType
Equal priority: later registration wins
Built-in providers have priority 0
User providers should use priority >= 100 to override built-in

Excluding Test Types

Use the excludes option in your scenario to skip certain test types:

auto-test:
  operations: [createPet]
  types: [invalid, security]
  excludes: [SQLInjection, maxLength, myCustomType]

The excludes option works with both built-in and custom provider test types.

Programmatic Registration

You can also register providers programmatically:

val registry = AutoTestProviderRegistry.withDefaults()
registry.registerInvalid(MyCustomInvalidProvider())
registry.registerSecurity(MyCustomSecurityProvider())

val generator = AutoTestGenerator(openApi, registry)

Or create an empty registry for full control:

val registry = AutoTestProviderRegistry.empty()
// Only your custom providers will be used
registry.registerInvalid(MyOnlyProvider())

Test Type Naming Conventions

Use camelCase for testType (e.g., numericOverflow, NoSQLInjection)
Use human-readable names for displayName (e.g., “Numeric Overflow”, “NoSQL Injection”)
The testType is used for:
- Test identification and deduplication
- excludes configuration
- Provider override matching
The displayName is used for:
- Test reports (IntelliJ, JUnit XML)
- Scenario output logs

Multi-Test Providers

Multi-test providers support idempotency testing by executing requests multiple times in different modes. They are used with the auto: [multi] directive to verify API operations produce consistent results.

MultiTestProvider Interface

interface MultiTestProvider {
    /**
     * Unique identifier for this multi-test mode.
     * Used for display names in test reports: [multi:{[mode]}]
     * Used in excludes configuration: excludes: [{[mode]}]
     */
    val mode: String

    /**
     * Human-readable display name for test reports.
     * Defaults to testType if not overridden.
     */
    val displayName: String get() = testType

   /**
    * Default number of requests to execute for this multi-test mode.
    * Can be overridden via configuration parameters:
    * 
    * ```
    * parameters:
    *   multiTest:
    *     ${mode}:
    *       count: 10
    * ```
    */
    val defaultCount: Int
    
    /**
     * Priority for provider override. Higher values = higher priority.
     * User-provided providers default to 100, built-in default to 0.
     */
    val priority: Int get() = 0

    /**
     * Execute multi-request test.
     * @param count Number of requests to execute
     * @param executor Function that executes a single request
     * @return Results of all requests and aggregate information
     */
    fun executeMultiTest(
        count: Int,
        executor: (requestIndex: Int) -> RequestResult,
    ): MultiTestResult
}

Built-in Multi-Test Providers

Test Type	Display Name	Description
`sequential`	Sequential Idempotency	Executes requests one after another, verifying sequential idempotency
`concurrent`	Concurrent Idempotency	Executes requests in parallel using a thread pool (max 20 threads), verifying concurrent access safety

Custom Multi-Test Provider

Create a class implementing MultiTestProvider:

package com.example

import org.berrycrush.autotest.MultiMode
import org.berrycrush.autotest.MultiTestResult
import org.berrycrush.autotest.RequestResult
import org.berrycrush.autotest.provider.MultiTestProvider

class RetryMultiTestProvider : MultiTestProvider {
    // Unique identifier for this provider
    override val mode: String = "RETRY"
    
    // Human-readable name for test reports
    override val displayName: String = "Retry Test"
  
    // Default retry count
    override val defaultCount: Int = 3
    
    // Higher priority overrides built-in providers
    override val priority: Int = 100

    override fun executeMultiTest(
        count: Int,
        executor: () -> RequestResult,
    ): MultiTestResult {
        val results = mutableListOf<RequestResult>()
        var lastResult: RequestResult? = null
        
        // Execute with exponential backoff
        repeat(count) { attempt ->
            if (attempt > 0) {
                Thread.sleep((100 * attempt).toLong())
            }
            lastResult = executor()
            results.add(lastResult!!)
            
            // Stop if successful
            if (lastResult!!.statusCode in 200..299) {
                return@repeat
            }
        }
        
        val totalDuration = results.sumOf { it.durationMs }
        val passed = lastResult?.statusCode in 200..299
        
        return MultiTestResult(
            mode = MultiMode.SEQUENTIAL, // or create custom mode
            requestCount = results.size,
            results = results,
            totalDurationMs = totalDuration,
            passed = passed,
            failureReason = if (!passed) "All retry attempts failed" else null,
        )
    }
}

Registering Multi-Test Providers

Create a service configuration file:

src/main/resources/META-INF/services/org.berrycrush.autotest.provider.MultiTestProvider:

com.example.RetryMultiTestProvider

Programmatic Registration

val registry = AutoTestProviderRegistry.withDefaults()
registry.registerMulti(RetryMultiTestProvider())

Configuration Parameters

Multi-test execution can be configured via parameters:

Parameter	Default	Description
`multiTest.sequential.count`	3	Number of sequential requests
`multiTest.concurrent.count`	5	Number of concurrent requests

Set in scenario files (file-level):

parameters:
  multiTest:
    sequential.count: 5
    concurrent.count: 10

At the feature level:

feature: Idempotency Tests
  parameters:
    # legacy format
    multiTestSequentialCount: 10
    multiTestConcurrentCount: 20
  
  scenario: Multi test
    when: I test the API
      call ^operation
        auto: [multi]

Or at the step level (in the call directive):

when: I stress test with custom counts
  call ^operation
    auto: [multi]
    multiTest.sequential.count: 5
    multiTest.concurrent.count: 10

Step-level parameters take precedence over file-level and feature-level parameters.

Kotlin DSL Auto-Test API

Auto-tests can also be configured using the Kotlin DSL. The DSL provides a type-safe way to enable auto-test generation.

Basic Usage

scenario("Create pet with auto-tests") {
    whenever("I create a pet") {
        call("createPet") {
            body("""{"name": "Fluffy", "category": "cat"}""")
            // Enable specific auto-test types
            autoTest(AutoTestType.INVALID, AutoTestType.SECURITY)
        }
    }
}

Using Boolean Parameters

call("createPet") {
    // Alternative API using booleans
    autoTest(invalid = true, security = true, multi = false)
}

Multi-Request Idempotency Tests

scenario("Idempotency test") {
    whenever("I send multiple requests") {
        call("listPets") {
            // Enable multi-request testing
            autoTest(AutoTestType.MULTI)
        }
    }
}

Custom Multi-Test Parameters

Configure request counts at the suite/configuration level:

val suite = BerryCrushSuite.create()
suite.configuration.multiTestConfig["sequential.count"] = 5
suite.configuration.multiTestConfig["concurrent.count"] = 10

suite.scenario("Custom counts") {
    whenever("I stress test the API") {
        call("listPets") {
            autoTest(AutoTestType.MULTI)
        }
    }
}

Excluding Test Categories

Exclude specific test categories from auto-generation:

call("createPet") {
    autoTest(AutoTestType.INVALID, AutoTestType.SECURITY)
    // Exclude specific categories
    excludes("XSS", "minLength")
}

Test Display Names

DSL auto-tests use the same display name format as scenario files:

Test Type	Display Name Format
Invalid	`[invalid - minLength]`, `[invalid - pattern]`
Security	`[security - SQL Injection]`, `[security - XSS]`
Multi (Sequential)	`[multi:sequential] N requests`
Multi (Concurrent)	`[multi:concurrent] N requests`