Data Harbor

Cursor for data scientists

Comment

AI-Powered Multi-Agent Research System

Data Harbor is an advanced, autonomous AI-powered platform for data analysis and research. It employs a multi-agent system built with LangChain, LangGraph, and OpenAI to orchestrate a complete research workflow, from hypothesis generation to final report compilation, with strategic human oversight.

Table of Contents

System Architecture

The multi agent system operates as a stateful graph where each node represents an agent or a human interaction point. The system is designed to iteratively process research tasks, ensuring quality at each step through review cycles and maintaining context via a dedicated note-taking agent.

Workflow Diagram

This diagram illustrates the flow of control and data between the different agents and human review stages.

graph TD
    subgraph "Phase 1: Hypothesis Generation"
        Start[START] --> Hypothesis;
        Hypothesis --> HumanChoice{"Human Choice"};
        HumanChoice -- "Regenerate" --> Hypothesis;
    end

    HumanChoice -- "Continue" --> Process;

    subgraph "Phase 2: Iterative Research Cycle"
        Process -- "Assign Task" --> Code;
        Process -- "Assign Task" --> Search;
        Process -- "Assign Task" --> Visualization;
        Process -- "Assign Task" --> Report;

        Code --> QualityReview;
        Search --> QualityReview;
        Visualization --> QualityReview;
        Report --> QualityReview;

        QualityReview -- "Rejection (to specific agent)" --> Code;
        QualityReview -- "Rejection (to specific agent)" --> Search;
        QualityReview -- "Rejection (to specific agent)" --> Visualization;
        QualityReview -- "Rejection (to specific agent)" --> Report;

        QualityReview -- "Approval" --> NoteTaker;
        NoteTaker --> Process;
    end

    subgraph "Phase 3: Finalization"
        Process -- "FINISH" --> Refiner;
        Refiner --> HumanReview{"Human Review"};
        HumanReview -- "Needs Revision" --> Process;
        HumanReview -- "Approve & End" --> End[END];
    end

    %% Styling optimized for dark mode visibility
    style Start fill:#4ade80,stroke:#e5e7eb,stroke-width:2px,color:#000000
    style End fill:#f87171,stroke:#e5e7eb,stroke-width:2px,color:#000000
    classDef humanNode fill:#a78bfa,stroke:#e5e7eb,stroke-width:2px,color:#ffffff
    class HumanChoice,HumanReview humanNode
    classDef specialAgent fill:#38bdf8,stroke:#e5e7eb,stroke-width:2px,color:#000000
    class Hypothesis,Code,Search,Visualization,Report,QualityReview,Refiner specialAgent
    classDef normalAgent fill:#facc15,stroke:#e5e7eb,stroke-width:2px,color:#000000
    class Process,NoteTaker normalAgent

Architectural Concepts

  • Workflow START/END Nodes: Represent the entry and exit points of the entire process.
  • Human Nodes: Points where the system pauses for human input and validation (HumanChoice, HumanReview). This ensures control over the research direction and final approval.
  • Special Agents: Agents designed to perform specific, isolated tasks like coding, searching, or generating visualizations.
  • Normal Agents: Core agents that manage the workflow.
    • Process: The supervisor or project manager. It analyzes the overall state and delegates tasks to the appropriate special agents.
    • NoteTaker: The state historian. It observes the outputs of other agents, summarizes the progress, and maintains a coherent, structured representation of the entire research state. This prevents context loss in long-running tasks.
  • Edges: The connections between nodes.
    • Definite Edge: A direct, unconditional path (e.g., after Hypothesis generation, the system always proceeds to HumanChoice).
    • Conditional Edge: A branching path where the next step is determined by the output of the current node (e.g., QualityReview can either approve and move to NoteTaker or reject and loop back to a special agent).

Core Concepts

Multi-Agent Collaboration

The system is built on the principle of "divide and conquer." Instead of a single monolithic AI, it uses a team of specialized agents. Each agent has a distinct role, a specific set of tools, and a unique prompt defining its expertise. This modular design makes the system more robust, easier to debug, and allows for the use of different LLMs optimized for different tasks (e.g., a powerful model for coding, a JSON-enforced model for state management).

State-Driven Workflow

The entire research process is managed as a state machine using LangGraph. A central State object is passed between nodes, and each agent's execution updates this state.

The State object (defined in core/state.py) contains fields that track every aspect of the research:

  • messages: The history of conversation and agent outputs.
  • hypothesis: The current research hypothesis.
  • process_decision: The next action decided by the Process agent.
  • visualization_state, searcher_state, code_state, report_section: Content generated by the respective agents.
  • quality_review: Feedback from the QualityReview agent.
  • needs_revision: A boolean flag that controls the revision loop.

The NoteTaker agent plays a critical role by reading the entire state and generating a clean, summarized JSON representation of it, ensuring consistency as the workflow progresses.

Human-in-the-Loop

The tool is a human-AI collaboration tool, not a fully autonomous system. There are two key points for human intervention:

  1. Hypothesis Review (HumanChoice): After the initial hypothesis is generated, a human must approve it or request modifications before the main analysis begins.
  2. Final Review (HumanReview): Once the Process agent determines the research is complete, it passes the final report to a human for review. The human can either approve and end the process or send it back for further revisions.

Key Features

Rate Limit Resilience

The system includes sophisticated rate limit handling to ensure continuous operation even with API restrictions:

  • Exponential Backoff: Automatic retry with increasing wait times
  • Model Fallback: Automatically switches to gpt-4o-mini when gpt-4o hits rate limits
  • Request Tracking: Monitors usage per model to proactively switch to fallback models
  • Jitter: Adds randomization to retry timing to avoid thundering herd effects

Workflow Persistence

Built-in checkpointing ensures no work is lost during interruptions:

  • SQLite Checkpointing: Persistent storage of workflow state across sessions
  • Thread Management: Support for multiple concurrent research workflows
  • Resume Capability: Restart from any checkpoint after interruptions or errors
  • State Recovery: Full state restoration including agent outputs and decisions

Quality Control

Rigorous quality assurance throughout the research process:

  • Structured Review Process: Quality review agent with standardized APPROVE/REJECT responses
  • Idempotency Protection: Prevents duplicate task execution
  • Output Validation: Ensures agent outputs meet quality standards before proceeding
  • Revision Loops: Automatic routing back to specific agents for improvements

Enhanced Logging

Comprehensive logging and observability for debugging and monitoring:

  • Structured Logging: Separate log files for different severity levels
  • Workflow Tracking: Detailed logs of agent transitions and state changes
  • Error Context: Full stack traces with contextual information
  • Rate Limit Monitoring: Tracking of API usage and fallback events
  • Colored Console Output: Easy-to-read console logs with color coding

Directory Structure

.
├── agent/                  # Definitions for all specialized agents
│   ├── code_agent.py
│   ├── hypothesis_agent.py
│   ├── note_agent.py
│   ├── process_agent.py
│   ├── quality_review_agent.py
│   ├── refiner_agent.py
│   ├── report_agent.py
│   ├── search_agent.py
│   └── visualization_agent.py
├── core/                   # Core logic for the workflow, state, and routing
│   ├── language_models.py
│   ├── node.py
│   ├── router.py
│   ├── state.py
│   └── workflow.py
├── tools/                  # Tools available to the agents (file I/O, code execution, web search)
│   ├── basetool.py
│   ├── FileEdit.py
│   └── internet.py
├── vue-frontend/           # (Placeholder for Vue.js frontend)
├── .env.example            # Template for environment variables
├── .gitignore
├── create_agent.py         # Factory functions for creating agents
├── load_cfg.py             # Loads environment variables from .env file
├── logger.py               # Logging configuration
├── main.py                 # Main script for command-line execution
├── README.md
└── requirements.txt

Codebase Deep Dive

The System's Core (core/)

This directory contains the essential machinery that drives the multi-agent system.

  • workflow.py: The central orchestrator. The WorkflowManager class initializes all agents and assembles the StateGraph. It defines all the nodes and edges shown in the architecture diagram, effectively building the application's logic flow.
  • node.py: Defines the functions that are executed at each node in the graph.
    • agent_node: A generic wrapper that invokes an agent with the current state and updates the state with the agent's output.
    • note_agent_node: A special node for the NoteTaker agent. It processes the agent's JSON output to update the entire state object, ensuring consistency.
    • human_choice_node / human_review_node: Functions that pause the graph and prompt for user input from the console.
    • refiner_node: A node that gathers all generated artifacts (markdown files, images) and passes them to the Refiner agent for final polishing.
  • router.py: Contains the logic for conditional branching in the graph. These functions inspect the current state and decide which node to visit next.
    • hypothesis_router: Directs the flow after HumanChoice—either back to Hypothesis or forward to Process.
    • process_router: The main router that interprets the Process agent's decision (e.g., 'Coder', 'Search', 'FINISH') and routes to the corresponding node.
    • QualityReview_router: Routes based on the QualityReview agent's output, either to NoteTaker (approval) or back to the original agent (rejection).
  • state.py: Defines the data structure for the shared state (State TypedDict) that circulates through the graph. This is the single source of truth for the research project at any given time.
  • language_models.py: The LanguageModelManager centralizes the configuration and initialization of the different ChatOpenAI models used throughout the system (e.g., gpt-4o for complex tasks, gpt-4o-mini for simpler ones, and a JSON-mode model for the NoteTaker).

The Agent Workforce (agent/)

Each file in this directory defines a specialized agent by providing a system prompt and a set of tools.

Agent Responsibilities Key Tools
hypothesis_agent.py Generates an initial research hypothesis based on the user's prompt and initial data. collect_data, google_search, wikipedia
process_agent.py Supervisor. Oversees the entire research cycle, assigns tasks to other agents, and decides when to finish. (None - It's a routing agent)
code_agent.py Writes, debugs, and executes Python code for data processing and analysis. execute_code, read_document
visualization_agent.py Generates Python code to create data visualizations (e.g., charts, plots) and saves them as files. execute_code, read_document
search_agent.py Conducts literature reviews and web searches to gather supporting information and references. google_search, scrape_webpages, arxiv
report_agent.py Drafts and structures the final research report, integrating text, findings, and visualizations. create_document, read_document, edit_document
quality_review_agent.py Reviews the outputs of other agents for accuracy, clarity, and adherence to standards. read_document, edit_document
note_agent.py State Manager. Observes the workflow, summarizes progress, and outputs the entire state as a clean JSON. read_document
refiner_agent.py Polishes the final compiled report, improving readability, flow, and coherence before final human review. read_document, edit_document, web tools

The Agent Factory (create_agent.py)

This module provides helper functions to standardize agent creation.

  • create_agent: A generic function that builds a standard agent. It combines a system prompt, a list of tools, and placeholders for the shared state into a prompt template compatible with LangChain's create_openai_functions_agent.
  • create_supervisor: A specialized function for creating the Process agent. It configures the LLM with a route function definition, compelling it to output its decision in a structured format that the process_router can parse.
  • create_note_agent: A specialized function for the NoteTaker agent. It injects Pydantic-based JSON formatting instructions into the prompt to ensure the agent's output is always a valid NoteState object.

The Agent's Toolkit (tools/)

This directory contains the functions that agents can invoke to interact with the outside world.

  • basetool.py:
    • execute_code: Executes Python code in a sandboxed Conda environment.
    • execute_command: Executes shell commands in the same environment, primarily used for installing dependencies.
  • FileEdit.py:
    • Provides a suite of tools for file system operations within the WORKING_DIRECTORY: collect_data (from CSV), create_document, read_document, write_document, and edit_document.
  • internet.py:
    • Provides tools for web access: google_search, scrape_webpages (using WebBaseLoader), and FireCrawl_scrape_webpages. It includes a fallback mechanism to ensure robustness.

Entrypoints & Configuration

  • main.py: The primary script for running the application. It initializes the MultiAgentSystem class and starts the workflow stream.
  • load_cfg.py & .env.example: Manage the loading of secrets and configuration from a .env file.

Setup and Usage

Prerequisites

  • Python 3.10+
  • Conda package manager

Installation

  1. Clone the repository:

  2. Create and activate a Conda environment:

    conda create -n hack_env python=3.10
conda activate hack_env

  3. Install dependencies:

    pip install -r requirements.txt

Configuration

  1. Create .env file: Create a .env file in the project root with your configuration:

    # Copy the example file (if it exists)
cp .env.example .env
# Or create a new one
touch .env

  2. Fill in the required environment variables: Open the .env file and provide the following values:

    # REQUIRED: Your data storage path
WORKING_DIRECTORY=./data_storage/

# REQUIRED: Anaconda installation path
# Windows: C:\Users\YourUsername\anaconda3 or C:\Users\YourUsername\miniconda3
# macOS/Linux: /home/yourusername/anaconda3 or /opt/anaconda3
CONDA_PATH=/path/to/your/anaconda3

# REQUIRED: Conda environment name (must match the one you created)
CONDA_ENV=hack_env

# REQUIRED: OpenAI API key (get from https://platform.openai.com/account/api-keys)
OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# OPTIONAL: ChromeDriver executable path (for web search functionality)
# Download from https://chromedriver.chromium.org/
# Windows: ./chromedriver/chromedriver.exe
# macOS/Linux: ./chromedriver/chromedriver
CHROMEDRIVER_PATH=./chromedriver/chromedriver

# OPTIONAL: Firecrawl API key (for enhanced web scraping)
FIRECRAWL_API_KEY=fc-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# OPTIONAL: LangChain API key (for LangSmith tracing and debugging)
LANGCHAIN_API_KEY=ls__xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# OPTIONAL: User agent for web requests
USER_AGENT=AI-Research-System/1.0

  3. Verify your configuration: The system will validate your configuration on startup and provide helpful error messages if anything is missing.

Running the System

  1. Place your data: Put your data file (e.g., MyData.csv) inside the data_storage directory specified in your .env file.

  2. Choose an execution method:

    A) Using the Python Script (Recommended for runs)

    Modify the user_input variable in the main() function of main.py:

    # in main.py
def main():
    system = MultiAgentSystem()
    user_input = '''
    datapath:MyData.csv
    Please analyze this data to find sales trends and create a report with visualizations.
    '''
    system.run(user_input)

    Then, run the script from your terminal:

    python main.py

Built With

Share this project:

Updates