# Auto-Test Provider Extensibility

BerryCrush's auto-test feature supports custom test providers through Java's ServiceLoader pattern. This allows users to add custom invalid request tests, security tests, and multi-request idempotency tests without modifying the core library.

## Overview

The auto-test system uses three types of providers:

1. **InvalidTestProvider** - Generates complete invalid `AutoTestCase` entries
2. **SecurityTestProvider** - Generates complete security `AutoTestCase` entries
3. **MultiTestProvider** - Executes multi-request idempotency tests

All types are discovered automatically via ServiceLoader, allowing you to add custom providers by simply adding classes to your project.

## Built-in Providers

### Invalid Test Providers

| Test Type | Description |
|-----------|-------------|
| minLength | Strings shorter than minLength constraint |
| maxLength | Strings longer than maxLength constraint |
| pattern | Strings not matching the pattern |
| format | Invalid format values (email, uuid, date, etc.) |
| enum | Values not in the enum list |
| minimum | Numbers below the minimum |
| maximum | Numbers above the maximum |
| type | Wrong type values (e.g., string for number field) |
| required | Missing required fields |
| minItems | Arrays with fewer items than minItems |
| maxItems | Arrays with more items than maxItems |

### Security Test Providers

| Test Type | Display Name | Description |
|-----------|--------------|-------------|
| SQLInjection | SQL Injection | SQL injection attack payloads |
| XSS | XSS | Cross-site scripting payloads |
| PathTraversal | Path Traversal | Path traversal attack patterns |
| CommandInjection | Command Injection | Shell command injection payloads |
| LDAPInjection | LDAP Injection | LDAP query injection payloads |
| XXE | XXE | XML External Entity payloads |
| HeaderInjection | Header Injection | HTTP header injection payloads |
| NoSQLInjection | NoSQL Injection | MongoDB-style operator payloads |
| SSTI | Template Injection | Server-side template injection payloads |
| JWT | JWT Attacks | JWT tampering and malformed token payloads |
| AuthorizationBypass | Authorization Bypass | Missing/invalid auth header payloads |

Built-in provider inventory is intentionally stable so existing `excludes` usage remains compatible.

### Multi Test Providers

| Test Type | Display Name | Description |
|-----------|--------------|-------------|
| sequential | Sequential Idempotency | Executes requests one after another |
| concurrent | Concurrent Idempotency | Executes requests in parallel using a thread pool |

## Creating Custom Providers

### Custom Invalid Test Provider

Create a class implementing `InvalidTestProvider`:

```kotlin
package com.example

import org.berrycrush.autotest.provider.InvalidTestProvider
import org.berrycrush.autotest.AutoTestCase
import org.berrycrush.autotest.AutoTestType
import org.berrycrush.autotest.provider.InvalidTestRequest
import io.swagger.v3.oas.models.media.Schema

class NumericOverflowProvider : InvalidTestProvider {
    // Unique identifier for test reports and excludes
    override val testType: String = "numericOverflow"
    
    // Higher priority overrides built-in providers with same testType
    override val priority: Int = 100

    override fun canHandle(schema: Schema<*>): Boolean =
        schema.type == "integer" || schema.type == "number"

    override fun generateTestCases(request: InvalidTestRequest): List<AutoTestCase> {
        if (request.location != org.berrycrush.autotest.ParameterLocation.BODY) return emptyList()

        val overflowBody = request.baseBody.toMutableMap().apply {
            this[request.fieldName] = Long.MAX_VALUE
        }

        val infinityBody = request.baseBody.toMutableMap().apply {
            this[request.fieldName] = Double.POSITIVE_INFINITY
        }

        return listOf(
            AutoTestCase(
                type = AutoTestType.INVALID,
                fieldName = request.fieldName,
                invalidValue = Long.MAX_VALUE,
                description = "Numeric overflow value",
                location = request.location,
                body = overflowBody,
                pathParams = request.basePathParams,
                headers = request.baseHeaders,
                tag = "Invalid request - $testType",
            ),
            AutoTestCase(
                type = AutoTestType.INVALID,
                fieldName = request.fieldName,
                invalidValue = Double.POSITIVE_INFINITY,
                description = "Infinity value",
                location = request.location,
                body = infinityBody,
                pathParams = request.basePathParams,
                headers = request.baseHeaders,
                tag = "Invalid request - $testType",
            ),
        )
    }
}
```

### Custom Security Test Provider

Create a class implementing `SecurityTestProvider`:

```kotlin
package com.example

import org.berrycrush.autotest.ParameterLocation
import org.berrycrush.autotest.AutoTestCase
import org.berrycrush.autotest.AutoTestType
import org.berrycrush.autotest.provider.SecurityTestProvider
import org.berrycrush.autotest.provider.SecurityTestRequest

class NoSqlInjectionProvider : SecurityTestProvider {
    // Unique identifier for excludes
    override val testType: String = "NoSQLInjection"
    
    // Human-readable name for test reports
    override val displayName: String = "NoSQL Injection"
    
    // Higher priority overrides built-in providers with same testType
    override val priority: Int = 100

    override fun applicableLocations(): Set<ParameterLocation> =
        setOf(ParameterLocation.BODY, ParameterLocation.QUERY)

    override fun generateTestCases(request: SecurityTestRequest): List<AutoTestCase> {
        if (request.location !in applicableLocations()) return emptyList()

        return listOf(
            "MongoDB \$ne injection" to "{\"\$ne\": null}",
            "MongoDB \$where injection" to "{\"\$where\": \"sleep(5000)\"}",
        ).map { (name, payload) ->
            val body =
                if (request.location == ParameterLocation.BODY) {
                    request.baseBody.toMutableMap().apply { this[request.fieldName] = payload }
                } else {
                    request.baseBody
                }

            AutoTestCase(
                type = AutoTestType.SECURITY,
                fieldName = request.fieldName,
                invalidValue = payload,
                description = "$displayName: $name",
                location = request.location,
                body = body,
                pathParams = request.basePathParams,
                headers = request.baseHeaders,
                tag = "security - $displayName",
            )
        }
    }
}
```

### Registering via ServiceLoader

Create a service configuration file in your project:

**`src/main/resources/META-INF/services/org.berrycrush.autotest.provider.InvalidTestProvider`**:
```
com.example.NumericOverflowProvider
```

**`src/main/resources/META-INF/services/org.berrycrush.autotest.provider.SecurityTestProvider`**:
```
com.example.NoSqlInjectionProvider
```

The providers will be automatically discovered and registered when the auto-test system initializes.

### Migration Notes

The provider contract now expects complete `AutoTestCase` generation.

- Preferred API:
    - `InvalidTestProvider.generateTestCases(request)`
    - `SecurityTestProvider.generateTestCases(request)`
- Legacy APIs (`generateInvalidValues`, `generatePayloads`) are deprecated compatibility adapters.
- `AutoTestGenerator` orchestrates provider calls only and no longer assembles provider-specific cases.
- Missing-required cases are provider-owned via the built-in `required` provider.

## Provider Priority

Each provider has a `priority` property (default: 0 for built-in, 100 for user providers):

- Higher priority providers override lower priority ones with the same `testType`
- Equal priority: later registration wins
- Built-in providers have priority 0
- User providers should use priority >= 100 to override built-in

## Excluding Test Types

Use the `excludes` option in your scenario to skip certain test types:

```scenario
auto-test:
  operations: [createPet]
  types: [invalid, security]
  excludes: [SQLInjection, maxLength, myCustomType]
```

The `excludes` option works with both built-in and custom provider test types.

## Programmatic Registration

You can also register providers programmatically:

```kotlin
val registry = AutoTestProviderRegistry.withDefaults()
registry.registerInvalid(MyCustomInvalidProvider())
registry.registerSecurity(MyCustomSecurityProvider())

val generator = AutoTestGenerator(openApi, registry)
```

Or create an empty registry for full control:

```kotlin
val registry = AutoTestProviderRegistry.empty()
// Only your custom providers will be used
registry.registerInvalid(MyOnlyProvider())
```

## Test Type Naming Conventions

- Use **camelCase** for `testType` (e.g., `numericOverflow`, `NoSQLInjection`)
- Use **human-readable names** for `displayName` (e.g., "Numeric Overflow", "NoSQL Injection")
- The `testType` is used for:
  - Test identification and deduplication
  - `excludes` configuration
  - Provider override matching
- The `displayName` is used for:
  - Test reports (IntelliJ, JUnit XML)
  - Scenario output logs

## Multi-Test Providers

Multi-test providers support idempotency testing by executing requests multiple times in different modes. They are used with the `auto: [multi]` directive to verify API operations produce consistent results.

### MultiTestProvider Interface

```kotlin
interface MultiTestProvider {
    /**
     * Unique identifier for this multi-test mode.
     * Used for display names in test reports: [multi:{[mode]}]
     * Used in excludes configuration: excludes: [{[mode]}]
     */
    val mode: String

    /**
     * Human-readable display name for test reports.
     * Defaults to testType if not overridden.
     */
    val displayName: String get() = testType

   /**
    * Default number of requests to execute for this multi-test mode.
    * Can be overridden via configuration parameters:
    * 
    * ```
    * parameters:
    *   multiTest:
    *     ${mode}:
    *       count: 10
    * ```
    */
    val defaultCount: Int
    
    /**
     * Priority for provider override. Higher values = higher priority.
     * User-provided providers default to 100, built-in default to 0.
     */
    val priority: Int get() = 0

    /**
     * Execute multi-request test.
     * @param count Number of requests to execute
     * @param executor Function that executes a single request
     * @return Results of all requests and aggregate information
     */
    fun executeMultiTest(
        count: Int,
        executor: (requestIndex: Int) -> RequestResult,
    ): MultiTestResult
}
```

### Built-in Multi-Test Providers

| Test Type | Display Name | Description |
|-----------|--------------|-------------|
| `sequential` | Sequential Idempotency | Executes requests one after another, verifying sequential idempotency |
| `concurrent` | Concurrent Idempotency | Executes requests in parallel using a thread pool (max 20 threads), verifying concurrent access safety |

### Custom Multi-Test Provider

Create a class implementing `MultiTestProvider`:

```kotlin
package com.example

import org.berrycrush.autotest.MultiMode
import org.berrycrush.autotest.MultiTestResult
import org.berrycrush.autotest.RequestResult
import org.berrycrush.autotest.provider.MultiTestProvider

class RetryMultiTestProvider : MultiTestProvider {
    // Unique identifier for this provider
    override val mode: String = "RETRY"
    
    // Human-readable name for test reports
    override val displayName: String = "Retry Test"
  
    // Default retry count
    override val defaultCount: Int = 3
    
    // Higher priority overrides built-in providers
    override val priority: Int = 100

    override fun executeMultiTest(
        count: Int,
        executor: () -> RequestResult,
    ): MultiTestResult {
        val results = mutableListOf<RequestResult>()
        var lastResult: RequestResult? = null
        
        // Execute with exponential backoff
        repeat(count) { attempt ->
            if (attempt > 0) {
                Thread.sleep((100 * attempt).toLong())
            }
            lastResult = executor()
            results.add(lastResult!!)
            
            // Stop if successful
            if (lastResult!!.statusCode in 200..299) {
                return@repeat
            }
        }
        
        val totalDuration = results.sumOf { it.durationMs }
        val passed = lastResult?.statusCode in 200..299
        
        return MultiTestResult(
            mode = MultiMode.SEQUENTIAL, // or create custom mode
            requestCount = results.size,
            results = results,
            totalDurationMs = totalDuration,
            passed = passed,
            failureReason = if (!passed) "All retry attempts failed" else null,
        )
    }
}
```

### Registering Multi-Test Providers

Create a service configuration file:

**`src/main/resources/META-INF/services/org.berrycrush.autotest.provider.MultiTestProvider`**:
```
com.example.RetryMultiTestProvider
```

### Programmatic Registration

```kotlin
val registry = AutoTestProviderRegistry.withDefaults()
registry.registerMulti(RetryMultiTestProvider())
```

### Configuration Parameters

Multi-test execution can be configured via parameters:

| Parameter                    | Default | Description |
|------------------------------|---------|-------------|
| `multiTest.sequential.count` | 3 | Number of sequential requests |
| `multiTest.concurrent.count` | 5 | Number of concurrent requests |

Set in scenario files (file-level):
```berrycrush
parameters:
  multiTest:
    sequential.count: 5
    concurrent.count: 10
```

At the feature level:
```berrycrush
feature: Idempotency Tests
  parameters:
    # legacy format
    multiTestSequentialCount: 10
    multiTestConcurrentCount: 20
  
  scenario: Multi test
    when: I test the API
      call ^operation
        auto: [multi]
```

Or at the step level (in the call directive):
```berrycrush
when: I stress test with custom counts
  call ^operation
    auto: [multi]
    multiTest.sequential.count: 5
    multiTest.concurrent.count: 10
```

Step-level parameters take precedence over file-level and feature-level parameters.

## Kotlin DSL Auto-Test API

Auto-tests can also be configured using the Kotlin DSL. The DSL provides a type-safe way to enable auto-test generation.

### Basic Usage

```kotlin
scenario("Create pet with auto-tests") {
    whenever("I create a pet") {
        call("createPet") {
            body("""{"name": "Fluffy", "category": "cat"}""")
            // Enable specific auto-test types
            autoTest(AutoTestType.INVALID, AutoTestType.SECURITY)
        }
    }
}
```

### Using Boolean Parameters

```kotlin
call("createPet") {
    // Alternative API using booleans
    autoTest(invalid = true, security = true, multi = false)
}
```

### Multi-Request Idempotency Tests

```kotlin
scenario("Idempotency test") {
    whenever("I send multiple requests") {
        call("listPets") {
            // Enable multi-request testing
            autoTest(AutoTestType.MULTI)
        }
    }
}
```

### Custom Multi-Test Parameters

Configure request counts at the suite/configuration level:

```kotlin
val suite = BerryCrushSuite.create()
suite.configuration.multiTestConfig["sequential.count"] = 5
suite.configuration.multiTestConfig["concurrent.count"] = 10

suite.scenario("Custom counts") {
    whenever("I stress test the API") {
        call("listPets") {
            autoTest(AutoTestType.MULTI)
        }
    }
}
```

### Excluding Test Categories

Exclude specific test categories from auto-generation:

```kotlin
call("createPet") {
    autoTest(AutoTestType.INVALID, AutoTestType.SECURITY)
    // Exclude specific categories
    excludes("XSS", "minLength")
}
```

### Test Display Names

DSL auto-tests use the same display name format as scenario files:

| Test Type | Display Name Format |
|-----------|---------------------|
| Invalid | `[invalid - minLength]`, `[invalid - pattern]` |
| Security | `[security - SQL Injection]`, `[security - XSS]` |
| Multi (Sequential) | `[multi:sequential] N requests` |
| Multi (Concurrent) | `[multi:concurrent] N requests` |