smartrag

<div id="top"></div>

<br />
<div align="center">
    <img src="assets/multi-agent.png">

  <h1 align="center">SmartRAG: Elevating RAG with Multi-Agent Systems</h1>
  <p align="center">
    🐞 <a href="https://github.com/aymenfurter/smartrag/issues">Report Bug</a>
    ·
    💡 <a href="https://github.com/aymenfurter/smartrag/issues">Request Feature</a>
  </p>
  <br/> 
  <p>
  <a href="https://github.com/aymenfurter/smartrag/actions/workflows/python-tests.yml">
    <img height="30" src="https://github.com/aymenfurter/smartrag/actions/workflows/python-tests.yml/badge.svg" alt="Python Tests"></a>&nbsp; <a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Faymenfurter%2Fsmartrag%2Fmain%2Finfrastructure%2Fdeployment.json">
  <img height="30" src="https://aka.ms/deploytoazurebutton" alt="Deploy to Azure">
</a>
  </p>

</div>

## In the Wake of the Generative AI Revolution

We've seen a surge in GenAI-powered apps. While these apps promise a completely new way to interact with computers, they often don't meet user expectations. SmartRAG is a demonstration app that showcases various concepts to improve Retrieval-Augmented Generation (RAG) applications.

## Main Features

1. **Multiple Query Approaches**: Explore different data ingestion and querying methods, from simple and fast Azure OYD to the advanced GraphRAG approach.

2. **Voice Mode**: Utilize Azure OpenAI's Text-to-Speech (TTS) and Whisper for Speech-to-Text, enabling natural voice conversations.

3. **Advanced Querying**: Use Langchain summarizer or GraphRAG for complex queries in the "Ask" section.

4. **Advanced Indexing**: Enhance retrieval accuracy through multi-modal indexing techniques.

5. **Multi-Agent Research**:
   - **Multi-Agent**: Combine different indexes with agents (critic and researcher) working together for extended durations to find answers.
   - **AutoGen Integration**: Utilize Microsoft's AutoGen framework to create an ensemble of AI agents for collaborative research on complex topics.
   - **Time-Bounded Research**: Specify research duration to balance depth of analysis with response time.
   - **Citation and Verification**: Responses include citations for accuracy verification.

# Deploying SmartRAG

SmartRAG can be easily deployed using the Azure Developer CLI (azd):

1. Ensure you have the Azure Developer CLI installed.
2. Clone the SmartRAG repository.
3. Navigate to the project directory.
4. Run the following command:

   ```
   azd up
   ```
   
5. Some featurse may not be available until the app is restarted once.

## Voice Mode Deployment Considerations

SmartRAG includes a Voice Mode feature that uses Azure OpenAI's Text-to-Speech (TTS) and Whisper for Speech-to-Text capabilities. Please note:

- The TTS feature is currently available only in the Sweden Central and North Central US regions.
- If you want to use Voice Mode, ensure you deploy to one of these regions.
- If you don't need Voice Mode, you can modify the deployment script to remove this component and deploy the rest of the application in any supported Azure region.

The deployment process uses Bicep scripts (in the `infra` folder) and ARM templates (in the `infrastructure` folder) to set up the necessary Azure resources. 

# Multi-Agent Research for RAG

SmartRAG's experimental "Multi-Agent Research" feature uses Microsoft's [AutoGen](https://microsoft.github.io/autogen/) framework to create an ensemble of AI agents that collaborate on complex topics:

<img src="assets/agents.png" width="350">

## Key Components

1. **Researcher Agent**: Created for each data source, allowing independent research across various indexes.
2. **Reviewer Agent**: Oversees the process, guiding research and synthesizing findings.
3. **Collaborative Querying**: Agents ask follow-up questions, reframe queries, and synthesize information from multiple sources.

Here's a snippet of how the reviewer agent works:

```python
def create_reviewer_agent(llm_config: Dict[str, Any], list_of_researchers: str, single_data_source: bool = False) -> AssistantAgent:
    system_message = (
        "I am Reviewer. I review the research and drive conclusions. "
        "Once I am done, I will ask you to terminate the conversation.\n\n"
        "My job is to ask questions and guide the research to find the information I need. I always ask 10 questions at a time to get the information I need. "
        "and combine it into a final conclusion.\n\n"
        "I will make sure to ask follow-up questions to get the full picture.\n\n"
        "Only once I have all the information I need, I will ask you to terminate the conversation.\n\n"
        "I will keep an eye on the referenced documents, if it looks like not the right documents were referenced, ask the researcher to reframe the question to find additional data sources.\n\n"
        "I will use follow-up questions in case you the answer is incomplete (for instance if one data source is missing data).\n\n"
        "My researcher is: " + list_of_researchers + "\n\n"
        "To terminate the conversation, I will write ONLY the string: TERMINATE"
    )

    return AssistantAgent(
        name="Reviewer",
        llm_config=llm_config,
        is_termination_msg=lambda msg: "TERMINATE" in msg["content"].upper(),
        system_message=system_message,
    )
```

# GraphRAG: Advanced Querying

SmartRAG implements GraphRAG, a powerful approach for complex querying across multiple data sources. This feature allows for more nuanced and comprehensive answers by leveraging graph-based representations of knowledge.

## Key Features of GraphRAG

1. **Global Search**: Perform searches across multiple interconnected data sources.
2. **Community Context**: Utilize community structures within the data for more relevant results.
3. **Token-based Processing**: Efficiently manage and process large amounts of text data.

Here's a glimpse of how GraphRAG is implemented:

```python
async def global_query(self, query: str):
    # ... [setup code omitted]

    global_search = GlobalSearch(
        llm=llm,
        context_builder=context_builder,
        token_encoder=token_encoder,
        max_data_tokens=3000,
        map_llm_params={"max_tokens": 500, "temperature": 0.0},
        reduce_llm_params={"max_tokens": 500, "temperature": 0.0},
        context_builder_params={
            "use_community_summary": False,
            "shuffle_data": True,
            "include_community_rank": True,
            "min_community_rank": 0,
            "max_tokens": 3000,
            "context_name": "Reports",
        },
    )

    result = await global_search.asearch(query=query)
    # ... [result processing omitted]
```

# Voice Mode: Natural Conversation Interface

SmartRAG's Voice Mode creates a seamless, conversational interface using Azure OpenAI's Text-to-Speech and Whisper for Speech-to-Text capabilities.

## Key Features

1. **Text-to-Speech**: Convert AI responses to natural-sounding speech.
2. **Speech-to-Text**: Automatically transcribe user voice input for processing.
3. **Continuous Listening**: Enable hands-free, natural conversation flow.

# The Foundation: Quality Data and Mature Frameworks
<img src="assets/screenshot.png?raw=true">

## Indexing Quality Improvements

1. **Document Intelligence**: Convert unstructured files to structured Markdown format.
2. **Multimodal Post-processing**: Additional processing for documents with images or graphs.
3. **Table Enhancement**: Implement strategies for better handling of table content.
4. **Page-Level Splitting**: Split documents at page-level during preprocessing for easier citation verification.

Here's an example of how document intelligence is implemented:

```python
def convert_pdf_page_to_md(pdf_path: str, page_num: int, output_dir: str, prefix: str, refine_markdown: bool = False) -> str:
    # ... [initialization code omitted for brevity]
    
    # Use Azure's Document Intelligence to convert PDF to Markdown
    with o

[truncated…]