AISquare-Studio-QA

# AISquare Studio AutoQA

[![GitHub Action](https://img.shields.io/badge/GitHub%20Action-AutoQA-2088FF?logo=github-actions&logoColor=white)](https://github.com/AISquare-Studio/AISquare-Studio-QA)
[![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white)](https://www.python.org/)
[![Playwright](https://img.shields.io/badge/Playwright-Enabled-2EAD33?logo=playwright&logoColor=white)](https://playwright.dev/)
[![CrewAI](https://img.shields.io/badge/CrewAI-Multi--Agent-FF6B6B)](https://www.crewai.com/)

`ai` `automation` `testing` `playwright` `github-action` `crewai` `openai` `qa` `test-generation` `multi-agent`

**AI-powered GitHub Action that converts natural language test descriptions in pull request bodies into fully automated Playwright tests.** Write what you want to test in plain English — AutoQA generates, executes, and commits production-ready test code using CrewAI multi-agent orchestration and OpenAI GPT-4.

---

## Features

- **AI-Powered Test Generation** — Natural language steps become executable Playwright Python tests
- **Active Execution Mode** — Iterative step-by-step generation with real-time browser context
- **Smart Selector Discovery** — Auto-discovers optimal selectors from live pages via DOMInspectorTool
- **Intelligent Retry** — Automatic error recovery with alternative selectors and failure analysis
- **AST-Based Security Validation** — Prevents unsafe code patterns before execution
- **Cross-Repository Architecture** — Deploys as a GitHub Action, runs in any repository
- **Comprehensive Reporting** — PR comments with screenshots, HTML reports, and JSON artifacts
- **ETag-Based Idempotency** — Prevents duplicate test generation for unchanged PR descriptions
- **Multi-Tier Test Organization** — Categorize tests into A/B/C tiers by criticality
- **Caching Strategy** — Pip and Playwright browser caching for fast CI runs

---

## How It Works

```
PR Description          AutoQA Action              Your Repository
┌──────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  ```autoqa   │     │ 1. Parse PR body │     │ tests/autoqa/    │
│  flow: login │────▶│ 2. Generate code │────▶│   A/auth/        │
│  tier: A     │     │ 3. Validate AST  │     │     test_login.py│
│  area: auth  │     │ 4. Execute tests │     └──────────────────┘
│  ```         │     │ 5. Commit on pass│
│              │     │ 6. Comment on PR │
│  1. Go to /  │     └──────────────────┘
│  2. Login    │
│  3. Verify   │
└──────────────┘
```

1. A developer writes numbered test steps in the PR description inside a fenced `autoqa` block
2. The GitHub Action triggers on PR open/edit/sync events
3. AutoQA parses the PR body for metadata (`flow_name`, `tier`, `area`) and test steps
4. CrewAI agents generate Playwright Python test code from the steps
5. Generated code is validated via AST analysis and executed against your staging environment
6. On success, the test file is committed to `tests/autoqa/{tier}/{area}/test_{flow_name}.py`
7. Results and screenshots are posted as a PR comment

---

## Quick Start

### 1. Add the workflow

Create `.github/workflows/autoqa.yml` in your repository:

```yaml
name: AutoQA Test Generation

on:
  pull_request:
    types: [opened, synchronize, edited]

jobs:
  autoqa:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Generate and Execute Tests
        uses: AISquare-Studio/AISquare-Studio-QA@main
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          staging-url: ${{ secrets.STAGING_URL }}
          staging-email: ${{ secrets.STAGING_EMAIL }}
          staging-password: ${{ secrets.STAGING_PASSWORD }}
```

### 2. Configure secrets

Add the following secrets in your repository's **Settings → Secrets and variables → Actions**:

| Secret             | Description                       |
| ------------------ | --------------------------------- |
| `OPENAI_API_KEY`   | OpenAI API key (GPT-4 access)     |
| `STAGING_URL`      | Staging environment login URL     |
| `STAGING_EMAIL`    | Test account email                |
| `STAGING_PASSWORD` | Test account password             |

### 3. Write test steps in a PR

Include a fenced `autoqa` block in your pull request description:

````markdown
```autoqa
flow_name: user_login_success
tier: A
area: auth
```

1. Navigate to the login page
2. Enter valid email address
3. Enter valid password
4. Click the login button
5. Verify the dashboard appears
````

Open the PR and AutoQA takes care of the rest.

---

## PR Format Reference

The `autoqa` code block defines metadata. Numbered steps below it describe the test scenario.

````markdown
```autoqa
flow_name: <snake_case_test_name>
tier: <A|B|C>
area: <feature_area>
```

1. First test step in plain English
2. Second test step
3. ...
````

| Field       | Required | Description                                               |
| ----------- | -------- | --------------------------------------------------------- |
| `flow_name` | Yes      | Snake-case identifier used for the generated file name    |
| `tier`      | Yes      | `A` (critical), `B` (important), or `C` (nice-to-have)   |
| `area`      | Yes      | Feature area used as subdirectory (e.g., `auth`, `billing`) |

---

## Configuration Reference

### Action Inputs

| Input               | Required | Default          | Description                                       |
| ------------------- | -------- | ---------------- | ------------------------------------------------- |
| `openai-api-key`    | **Yes**  | —                | OpenAI API key                                    |
| `staging-url`       | **Yes**  | —                | Staging environment URL                           |
| `qa-github-token`   | No       | `github.token`   | GitHub token (for private repo access)             |
| `staging-email`     | No       | `test@example.com` | Test account email                              |
| `staging-password`  | No       | —                | Test account password                             |
| `target-repo-path`  | No       | `.`              | Path to the target repository                     |
| `git-user-name`     | No       | `AutoQA Bot`     | Git user name for test commits                    |
| `git-user-email`    | No       | —                | Git user email for test commits                   |
| `pr-body`           | No       | *(auto-detected)* | PR description text                              |
| `test-directory`    | No       | `tests/autoqa`   | Base directory for generated tests                |
| `create-pr`         | No       | `false`          | Create a PR for tests instead of pushing directly |
| `execution-mode`    | No       | `generate`       | Execution mode: `generate`, `suite`, or `all`     |

### Action Outputs

| Output                | Description                             |
| --------------------- | --------------------------------------- |
| `test_generated`      | Whether a test was generated (`true`/`false`) |
| `test_file_path`      | Path to the generated test file         |
| `test_results`        | JSON object with execution results      |
| `generation_metadata` | JSON object with generation metadata    |
| `screenshot_path`     | Path to captured screenshots            |
| `etag`                | Idempotency hash of the PR description  |
| `flow_name`           | Parsed flow name                        |
| `tier`                | Parsed tier                             |
| `area`                | Parsed area                             |
| `error`               | Error message (if failed)               |

### Execution Modes

| Mode       | Behavior                                                          |
| ---------- | ----------------------------------------------------------------- |
| `generate` | Parse PR, generate a new test, execute it, and commit on success  |
| `suite`   

[truncated…]