MarketMind Graph

MarketMind Graph transforms complex market data into actionable insights by mapping real-time relationships between Mag7 using AI-powered knowledge graphs, helping investors make smarter decisions.

Comment

💡 Inspiration

The idea for MarketMind Graph was born from my frustration with traditional market analysis tools. As someone deeply interested in both technology and financial markets, I noticed a significant gap: while there was abundant market data available, understanding the intricate relationships between major tech companies, their initiatives, and market impacts required piecing together information from countless sources.

The "Magnificent Seven" tech companies (Apple, Microsoft, Alphabet, Amazon, NVIDIA, Meta, and Tesla) particularly caught my attention. These giants don't exist in isolation – their decisions, products, and strategies create a complex web of relationships that significantly impact the market. I wanted to make a tool that automatically maps and analyzes these relationships in real-time.

📚 What it does

MarketMind Graph integrates AI and Snowflake to analyze financial news and construct a dynamic knowledge graph. The system intelligently classifies and processes queries to provide the most relevant insights:

For sentiment-focused questions, it analyzes market mood and emotional reactions, providing:

  • Overall market sentiment (Positive/Negative/Neutral)
  • Key sentiment drivers and market implications
  • Temporal sentiment trends and changes

For general market queries, it focuses on:

  • Factual relationships and event analysis
  • Company partnerships and strategic moves
  • Market impact assessments
  • Historical context and precedents

Core features available to users include:

  • Real-time insights with inline citations for transparency
  • Pro Search mode for advanced analysis and refined queries
  • Dynamic knowledge graph exploration of entity relationships
  • Trending questions and intelligent follow-up suggestions
  • Comprehensive sentiment metrics and market implications

💪 How we built it

  1. Foundation Setup

    • Created Snowflake database architecture
    • Set up Streamlit for the frontend
    • Established basic news fetching pipeline

  2. Knowledge Graph Development

    • Implemented entity extraction
    • Designed relationship mapping
    • Built semantic search functionality

  3. Enhanced Analysis Features

    • Developed question classification system
    • Integrated sentiment analysis pipeline
    • Created dynamic sentiment visualization
    • Implemented confidence scoring

  4. AI Integration

    • Integrated Mistral Large 2 model
    • Implemented embedding-based search
    • Created query refinement system
    • Added sentiment analysis capabilities

🤔 Challenges and Solutions

1. Data Consistency

Challenge: News articles often contain duplicate information or inconsistent entity references.

Solution: Implemented a robust deduplication system and standardized entity naming. This included creating a unified naming convention for entities and developing a sophisticated matching system to identify and merge duplicate entries while preserving unique information.

2. Real-time Processing

Challenge: Processing large volumes of news data in real-time while maintaining system responsiveness.

Solution: Implemented parallel processing and caching mechanisms. The system now processes multiple queries simultaneously and maintains a smart cache of recent results, significantly improving response times while reducing computational load.

3. Knowledge Graph Accuracy

Challenge: Early versions of the system created many low-confidence or spurious relationships.

Solution: Developed a confidence scoring system and validation pipeline. Each relationship is now evaluated based on multiple factors including entity recognition confidence, relationship pattern matching, and evidence text relevance. Only relationships with high confidence scores are included in the graph.

4. Sentiment Analysis Integration

Challenge: Implementing accurate and contextual sentiment analysis for market-specific content.

Solution: Developed a specialized sentiment analysis pipeline that:

  • Considers market-specific terminology and context
  • Evaluates sentiment across multiple dimensions
  • Provides confidence scores for sentiment predictions
  • Identifies key drivers of sentiment
  • Analyzes potential market implications

5. GraphRAG Implementation in Snowflake

Challenge: Implementing an effective GraphRAG system within Snowflake's structured environment.

Solution: Developed a hybrid approach that:

  • Created a custom embedding system combining entity attributes with relationship context
  • Implemented a two-stage retrieval process
  • Developed a custom scoring mechanism
  • Built a context assembly system
  • Implemented a caching layer for graph-based retrievals

🎯 Accomplishments

  • Successfully processed over 1,000 articles
  • Built a knowledge graph with over 1000 entities and relationships
  • Developed GraphRAG inside the traditional relationship database
  • Implemented accurate sentiment analysis with market context
  • Generated content as a summary of chat query and converted in Markdown
  • Query caching for frequently asked questions by users.
  • Question classifications
  • Sentiment analysis

🔧 What we learned

  • The importance of optimizing data pipelines for real-time performance
  • Effective ways to extract and visualize complex entity relationships
  • How to leverage AI model Mistral for financial insights
  • Techniques for accurate market sentiment analysis
  • Methods for classifying and routing different types of queries

🔄 What's next for MarketMind Graph

  • Expand Stock Coverage: Include more companies and historical data
  • Enhanced Sentiment Analysis: Deeper sentiment tracking and prediction
  • Enhanced Visualizations: Develop interactive graph views for better user engagement
  • Predictive Analytics: Use AI to forecast market trends based on historical data and relationships
  • API Access: Provide third-party integrations for broader applications
  • Advanced Classification: Expand query classification types and response templates
  • Sentiment Trends: Track and visualize sentiment changes over time and avoid multiple refreshes to the screen.
  • Performance: Need to decrease the query execution time.

Built With

  • ai
  • graphrag
  • llm
  • mistral
  • python
  • snowflake
  • streamlit
Share this project:

Updates