dspy-compounding-engineering

# Compounding Engineering (DSPy Edition)

![Python](https://img.shields.io/badge/python-3.10+-blue.svg)
![License](https://img.shields.io/badge/license-MIT-green.svg)
![Code Style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)
![DSPy](https://img.shields.io/badge/DSPy-Powered-blueviolet)
![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)

A Python implementation of the [Compounding Engineering Plugin](https://github.com/EveryInc/compounding-engineering-plugin) using [DSPy](https://github.com/stanfordnlp/dspy).

[**📚 Documentation**](https://strategic-automation.github.io/dspy-compounding-engineering/)

## What Is Compounding Engineering?

**Each unit of engineering work should make subsequent units of work easier—not harder.**

This CLI tool provides AI-powered development tools for code review, planning, and workflow automation, following the compounding engineering philosophy. It is designed as a **Local-First** tool that runs on your machine, keeping your code secure while leveraging powerful LLMs.

## Features

- **🧠 Compounding Engineering**: True learning system where every operation makes the next one easier

  - **Auto-Learning**: Every todo resolution automatically codifies learnings
  - **KB Auto-Injection**: Past learnings automatically inform all AI operations
  - **Pattern Recognition**: Similar issues are prevented based on past resolutions
  - **Knowledge Accumulation**: System gets smarter with every use

- **🔍 Multi-Agent Code Review**: Run 10+ specialized review agents in parallel

  - **Security Sentinel**: Detects vulnerabilities (SQLi, XSS, etc.)
  - **Performance Oracle**: Identifies bottlenecks and O(n) issues
  - **Architecture Strategist**: Reviews design patterns and SOLID principles
  - **Data Integrity Guardian**: Checks transaction safety and validation
  - **KB-Augmented**: All agents benefit from past code review learnings
  - And many more...

- **🤖 ReAct File Editing**: Intelligent file operations with reasoning

  - **Smart Context Gathering**: Relevance-scored file selection and token budget management
  - **Iterative Reasoning**: Think → Act → Observe → Iterate pattern
  - **Zero Hallucination**: Direct file manipulation, not text generation

- **🛡️ Secure Work Execution**: Safely execute AI-generated plans

  - **Isolated Worktrees**: High-level isolation for safe parallel execution via `--worktree`
  - **Parallel Processing**: Multi-threaded todo resolution with `--workers`
  - **Auto-Codification**: Every resolution creates learnings for future use

- **📋 Smart Planning**: Transform feature descriptions into detailed plans

  - Repository research & pattern analysis
  - **🌐 Internet Search**: Access live sources and current standards
  - **Documentation Fetcher**: Deep-read official documentation from URLs
  - SpecFlow user journey analysis
  - **KB-Informed**: Plans leverage past architectural decisions

- **✅ Interactive Triage**: Manage code review findings
  - **Batch Operations**: Approve multiple findings at once
  - **Smart Priorities**: Auto-detection of P1/P2/P3 severity
  - **Work Logs**: Tracks decisions and rationale automatically
  - **KB-Augmented**: Triage decisions informed by past patterns

## Installation

### Prerequisites

Install [uv](https://github.com/astral-sh/uv) (fast Python package installer):

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

### Setup

```bash
# Clone repository
git clone https://github.com/Strategic-Automation/dspy-compounding-engineering.git
cd dspy-compounding-engineering

# Configure environment
cp .env.example .env
# Edit .env with your API keys (OpenAI, Anthropic, or Ollama)

# Install dependencies
uv sync
```

### Repo-Agnostic Installation (Recommended)

To use `compounding` in **other repositories**, install it globally using `uv tool`:

```bash
# Install globally from source
uv tool install .
# or from git once published
# uv tool install git+https://github.com/Strategic-Automation/dspy-compounding-engineering.git
```

This exposes the `compounding` command globally, allowing you to run it inside any project folder.

### Vector Database Setup (Qdrant)

This project uses [Qdrant](https://qdrant.tech/) for semantic search. A Docker Compose configuration is provided.

1. **Start Qdrant**:

   ```bash
   docker compose up -d qdrant
   ```

   This will start Qdrant on `localhost:6333`.

2. **Configure Embeddings**:
   Ensure your `.env` has the correct `EMBEDDING_BASE_URL` if you are using a non-standard provider (like OpenRouter). See `.env.example`.

> **Note:** If Qdrant is not running, the system will automatically fall back to keyword-based search using local JSON files.

### Temporary runner: `uvx`

The repository includes a small wrapper script `uvx` that runs the CLI via `uv run` without doing a global install. It's useful for quickly trying commands locally:

```bash
./uvx -h
./uvx generate-agent "create an agent to check for hardcoded secrets"
```

The script is in the repository root and is executable; run it from the repo directory.

You can also prefix commands with `compounding`, for example `./uvx compounding generate-agent`.

## Example run

Quick example using the temporary runner `uvx` to see the `generate-agent` help:

```bash
./uvx generate-agent -h
```

Expected excerpt:

```
Usage: compounding generate-agent [OPTIONS] DESCRIPTION

Generate a new Review Agent from a natural language description.

Options:
  --dry-run  -n   Show what would be created without writing files
  --help     -h   Show this message and exit.
```

You can run other commands similarly, for example:

```bash
./uvx -h
./uvx review --project
./uvx work p1 --env-file test.env
```

## Global Options

The tool supports several global options that can be used with any command:

- `-e, --env-file PATH`: Explicitly specify a `.env` file to load.
- `-h, --help`: Show help for any command.

## Configuration Priority

The tool loads configuration from multiple sources in the following priority order:

1. **Explicit Flag**: `--env-file` / `-e` on the command line.
2. **Environment Variable**: `COMPOUNDING_ENV` pointing to a `.env` file path.
3. **Local Override**: `.env` in the current working directory (CWD).
4. **Tool-Specific Global**: `~/.config/compounding/.env`.
5. **System Fallback**: `~/.env` in the user's home directory.

> [!TIP]
> This priority allows you to have a global `~/.config/compounding/.env` with your API keys while using local `.env` files for project-specific model selections.

## The Compounding Engineering Loop

This implementation embodies the core philosophy: **each unit of work makes subsequent work easier**.

```mermaid
graph LR
    A[Plan] -->|KB Context| B[Execute Work]
    B -->|KB Context| C[Review Code]
    C -->|KB Context| D[Triage Findings]
    D -->|Auto-Codify| KB[(Knowledge Base)]
    KB -->|Auto-Inject| A
    KB -->|Auto-Inject| B
    KB -->|Auto-Inject| C
    KB -->|Auto-Inject| D

    style KB fill:#4CAF50,stroke:#333,stroke-width:3px
    style D fill:#FFC107,stroke:#333,stroke-width:2px
```

**How it works:**

1. **Auto-Injection**: All AI operations (`review`, `triage`, `plan`, `work`) automatically receive relevant past learnings
2. **Auto-Codification**: Every todo resolution automatically extracts and stores learnings
3. **Pattern Recognition**: The system identifies similar issues and suggests solutions based on past successes
4. **Continuous Improvement**: The more you use it, the smarter it gets

### Knowledge Base Features

- **Persistent Learning**: Learnings stored in `.knowledge/` as structured JSON
- **Smart Retrieval**: Keyword-based similarity matching (extensible to vector embeddings)
- **Auto-Documentation**: `AI.md` automatically updated with consolidated learnings
- **Tagged Search**: Filter learnings by category, source, or topic

### Roadmap

See our [detailed Road

[truncated…]