backend

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![opensource](https://img.shields.io/badge/open-source-blue)](https://en.wikipedia.org/wiki/Open_source)
![GitHub package.json version](https://img.shields.io/github/package-json/v/flowlitics/backend)

# Flowlitics Backend

This project is a **Node.js RESTful backend API server** designed to manage and execute complex workflows (Flows). It provides a robust API for defining, managing, and running workflows composed of various node types, including those interacting with AI agents and requiring human intervention. The system supports asynchronous, event-driven execution with persistence and scheduling capabilities.

## ⚠️ Not Production-Ready

This project is currently under active development and is not ready for production use. Expect frequent and potentially breaking changes to the codebase, features, and APIs. Use with caution and be prepared for instability.

❤️ Interested in **collaborating** or **sponsoring**? You're welcome to join! For more information, please see the bottom of this README.

## Objective

The primary goal is to create a flexible and scalable engine for orchestrating complex processes that require coordination between automated tasks (like AI agent interactions) and manual steps (human-in-the-loop).

## Features

- **Workflow Definition**: Define flows composed of interconnected nodes (Start, Action, Decision, Delay, End, Custom) and edges with optional conditions.

- **Agent Integration**: Integrate various AI agents and define 'human' agents to represent manual steps. See the [Available Agents](#available-agents) section for details.

- **Asynchronous Execution**: Execute workflows asynchronously using a queue-based, event-driven model. The executor processes nodes until a waiting state is encountered (e.g., human node pending), freeing up resources. Execution resumes upon external events (e.g., human node resolution).

- **State Management**: Persistent storage of flow definitions, executions, and execution steps using MongoDB.

- **Human-in-the-Loop**: Explicit support for nodes requiring manual input or approval, pausing execution until resolved via API.

- **Pluggable Node Types**: Extend functionality by adding custom node types via a plugin system.

- **Scheduling**: Schedule flow executions using cron expressions.

- **API**: Comprehensive RESTful API for managing flows, nodes, agents, executions, plugins, and schedules.

- **Logging**: Structured logging for monitoring system activity and execution progress.

- **TypeScript**: Developed using TypeScript for improved code quality and maintainability.

## Architecture Highlights

The system employs an event-driven architecture centered around the `queueManager` and `flowExecutionService`.

- **Queueing**: New executions or resumed executions are added to a queue.

- **Execution**: A limited number of concurrent workers (`flowExecutionService`) pick up jobs from the queue.

- **Partial Processing**: The worker executes nodes sequentially within a flow until it encounters a node that cannot proceed (e.g., a 'pending' human node or a node waiting for input from a pending upstream node).

- **Passive Waiting**: When a flow is blocked, the worker releases it. The execution status remains 'running' in the database, but it's passively waiting for an external event.

- **Event Trigger**: An external event (e.g. an API call resolving a human node) updates the state of a specific node step.

- **Re-queueing/Resumption**: The event processing logic checks if the flow can now proceed. If so, the execution is re-enqueued, and a worker will pick it up to continue processing from the newly unblocked node(s).

- **Finalization**: When no more nodes can be processed and no nodes are pending, the execution is finalized as 'completed' or 'failed'.

This model ensures efficient resource utilization by not blocking workers on long-waiting tasks.

## Key Components

- **Example API Client**: The `ApiClient` class in `client-demo.ts` provides methods to interact with the backend APIs for managing agents, flows, nodes, and executions.
- **Flow Execution**: Demonstrates creating a complex workflow with AI and human-controlled nodes, starting execution, and monitoring its progress.
- **Mermaid Syntax**: Thanks to [Mermaid](https://mermaid.js.org/) [Live Editor](https://mermaid.live/edit) [Syntax](https://mermaid.js.org/syntax/flowchart.html), it is also possible to graphically represent the flow of all involved processes.

## How to Run

1. **Install Dependencies**:

   ```bash
   git clone git@github.com:flowlitics/backend.git flowlitics-backend
   cd flowlitics-backend
   npm install
   ```

2. **Set Up Environment**:

   - Create a `.env` file with the necessary configuration. See `.env.example` for a template of all required variables.

3. **Start the Server**:

   ```bash
   npm start # or npm run dev
   ```

4. **Run the Client Demo**:

   Optionally, run the `client-demo.ts` to test workflow creation and execution:

   ```bash
   npm run client:demo
   ```

## Run the Backend with Docker

If you prefer, you can start the backend with Docker. We've prepared a quick guide with a few commands that you can find here: [Docker Guide](./docs/DOCKER.md).

## Tech Stack Overview

Flowlitics Backend is built on a robust, modern tech stack that ensures scalability, maintainability, and efficient performance. Our core technologies include:

- **Node.js, Express.js & TypeScript:** For a fast, asynchronous, and strongly-typed backend.
- **MongoDB & Mongoose:** For NoSQL document storage and data modeling.
- **Advanced Agent Integrations:** Supporting AI SDKs (Google Gemini, Mistral AI, OpenAI, Anthropic, etc.).
- **Workflow Management & Scheduling:** Utilizing a custom queue system and node-cron.
- **Utilities & Middleware:** Including Joi for validation, Winston for logging, and more.

For a complete overview of all the technologies and integrations used, please see the full [Tech Stack Documentation](./docs/TECH_STACK.md).

## Available Agents

Flowlitics Backend integrates multiple agent types to automate AI tasks and support human interactions within workflows.

The following agents are supported out-of-the-box:

- **Human**: Requires manual intervention for workflows.
- **Gemini**: Integrates with Google's Gemini AI language models.
- **Codestral**: Provides coding assistance via Mistral AI.
- **Mistral**: Access to open-source language model capabilities.
- **Mistral Media**: Extends capabilities to process media files.
- **Ollama**: Local execution of AI models.
- **Tavily Search**: AI-powered search engine.
- **Tavily Extract**: Extract web page content from one or more specified URL.
- **OpenAI**: A variety of models from OpenAI.
- **Anthropic Claude**: Utilizes Anthropic's Claude models.
- **Groq**: Integrates with advanced models via the Groq API (compound, llama, deepseek, gemma, playai, qwen, mistral, allam ..)

For detailed information on each agent, please see the [Agents Documentation](./docs/AGENTS.md).

## Example Workflow Execution

The [basic/client.ts](https://github.com/flowlitics/backend/blob/main/examples/basic) script provides a practical example of how to use the API to:

1. Create AI and Human agents.
2. Define a complex flow structure with various node types (start, custom, decision, action, end).
3. Connect nodes with edges, including conditional edges.
4. Start an execution of the flow with initial data.
5. Simulate the resolution of a human-controlled node via an API call, triggering the flow to resume.
6. Poll the execution status until completion.
7. Retrieve the flow definition in Mermaid syntax for visualization.

![Visual Example](https://github.com/flowlitics/backend/blob/main/screenshots/visual-example-demo.png?raw=true)

Here you can find [More Examples](https://github.com/flowlitics/backend/blob/main/examples)

## Visual Representation Example

Flows ca

[truncated…]