Testing Strategy for Bedrock Budgeteer

Overview

Comprehensive testing strategy covering unit tests, integration tests, and validation procedures for the CDK infrastructure and application components.

Testing Levels

1. Unit Tests (CDK Infrastructure)

Purpose: Validate CloudFormation template synthesis and resource configurations Framework: pytest + aws_cdk.assertions Location: app/tests/unit/

Test Categories

Stack Synthesis Tests

Verify stack synthesizes without errors
Validate resource counts and types
Check environment-specific configurations

Resource Configuration Tests

DynamoDB table properties and GSIs
IAM roles and policy attachments
CloudWatch log groups and retention
SNS topics and subscriptions
SSM parameters and values

Security Tests

IAM permission validation
Encryption configuration verification
Resource access patterns

Compliance Tests

Required tag validation
Removal policy verification
Environment isolation checks

Running Unit Tests

cd app
python -m pytest tests/unit/ -v

2. Integration Tests

Purpose: Test component interactions and AWS service integrations Framework: pytest + boto3 Location: app/tests/integration/

Test Categories

Service Integration Tests

Lambda → DynamoDB operations
EventBridge → Lambda triggers
Step Functions → Lambda invocations

Configuration Tests

SSM Parameter Store access
Environment-specific behavior
Cross-service communication

Monitoring Tests

CloudWatch metrics collection
Alarm trigger validation
Dashboard functionality

Running Integration Tests

cd app
AWS_PROFILE=dev python -m pytest tests/integration/ -v

3. End-to-End Tests

Purpose: Validate complete workflows in deployed environments Framework: pytest + boto3 + custom test harness Location: tests/e2e/

Test Scenarios

User budget setup and monitoring
Usage tracking and cost calculation
Budget threshold alerts
Account suspension workflow
Budget restoration process

Test Data Management

Synthetic Data Generation

# Example test data factory
class TestDataFactory:
    @staticmethod
    def create_test_user(user_id: str = None) -> dict:
        return {
            "user_id": user_id or f"test-user-{uuid.uuid4()}",
            "budget_limit": 100.00,
            "current_usage": 0.00,
            "status": "active",
            "created_at": datetime.utcnow().isoformat()
        }
    
    @staticmethod
    def create_usage_event(user_id: str, service: str = "bedrock") -> dict:
        return {
            "user_id": user_id,
            "service_name": service,
            "operation": "InvokeModel",
            "cost": round(random.uniform(0.01, 1.00), 4),
            "timestamp": datetime.utcnow().isoformat()
        }

Test Environment Isolation

Use separate AWS accounts for testing
Environment-specific test data cleanup
Isolated DynamoDB tables with test prefixes

Test Automation

CI/CD Integration

# .github/workflows/test.yml
name: Test Suite
on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          cd app
          pip install -r requirements.txt
          pip install -r requirements-dev.txt
      - name: Run unit tests
        run: |
          cd app
          python -m pytest tests/unit/ -v --cov=app --cov-report=xml
      - name: Upload coverage
        uses: codecov/codecov-action@v3

  integration-tests:
    needs: unit-tests
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v3
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: $
          aws-secret-access-key: $
          aws-region: us-west-2
      - name: Run integration tests
        run: |
          cd app
          python -m pytest tests/integration/ -v

Pre-commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: unit-tests
        name: Run unit tests
        entry: bash -c 'cd app && python -m pytest tests/unit/ --tb=short'
        language: system
        pass_filenames: false
        always_run: true

Test Validation Criteria

Unit Test Requirements

Coverage: Minimum 80% code coverage
Performance: Tests complete in < 30 seconds
Reliability: No flaky tests, 100% pass rate
Maintainability: Clear test names and documentation

Integration Test Requirements

Environment: Test against deployed dev environment
Data: Use synthetic test data only
Cleanup: Automated test data cleanup
Isolation: Tests can run in parallel without conflicts

Security Test Requirements

IAM: Validate least-privilege permissions
Encryption: Verify data encryption at rest/transit
Access: Test unauthorized access prevention
Compliance: Validate SOC2/GDPR requirements

Test Infrastructure

Test Dependencies

# requirements-dev.txt
pytest>=7.4.0
pytest-cov>=4.1.0
pytest-mock>=3.11.0
moto>=4.2.0  # AWS service mocking
aws-cdk-lib>=2.211.0
boto3>=1.35.0

Mock Services

# Example using moto for AWS service mocking
import boto3
from moto import mock_dynamodb, mock_ssm

@mock_dynamodb
@mock_ssm
def test_parameter_access():
    # Create mock AWS resources
    dynamodb = boto3.resource('dynamodb', region_name='us-west-2')
    ssm = boto3.client('ssm', region_name='us-west-2')
    
    # Test logic with mocked services
    pass

Test Utilities

# tests/utils/helpers.py
class TestHelper:
    @staticmethod
    def wait_for_stack_status(stack_name: str, status: str, timeout: int = 300):
        """Wait for CloudFormation stack to reach specified status"""
        pass
    
    @staticmethod
    def cleanup_test_resources(environment: str):
        """Clean up test resources in specified environment"""
        pass
    
    @staticmethod
    def validate_tags(resource_arn: str, expected_tags: dict):
        """Validate resource tags match expected values"""
        pass

Performance Testing

Load Testing

Simulate high-volume usage events
Test DynamoDB capacity scaling
Validate Lambda concurrency limits
Monitor CloudWatch metrics under load

Stress Testing

Test system behavior at limits
Validate error handling and recovery
Test backup and restore procedures
Verify monitoring and alerting

Security Testing

Penetration Testing

Test IAM permission boundaries
Validate network security controls
Test encryption implementation
Verify audit trail integrity

Compliance Testing

SOC2 control validation
GDPR data protection verification
Audit log completeness
Access control validation

Continuous Improvement

Test Metrics

Test execution time trends
Code coverage trends
Defect escape rates
Test reliability metrics

Test Review Process

Regular test suite review
Test case effectiveness analysis
Performance bottleneck identification
Security test gap analysis

Documentation Updates

Keep test documentation current
Update test procedures for new features
Maintain test environment documentation
Document known issues and workarounds