Panoptes

Connected Organizational Memory

Comment

Inspiration

In any organization, context switching and information fragmentation are major productivity killers. On any given workday, teams communicate across various disconnected tools: Slack threads, Gmail messages, Notion spec documents, Jira tickets, GitHub pull requests, Zoom meeting transcripts, Google Drive files, Microsoft 365 apps, and WhatsApp chats.

When someone asks a critical question, finding the answer requires searching across various different applications, sifting through long chat histories, and interrupting teammates or waiting for them to respond.

We built Panoptes (named after the hundred-eyed all-seeing figure in Greek mythology) to solve this. Panoptes is an AI organizational memory teammate that synthesizes information across every communication and collaboration channel. It transforms scattered enterprise noise into a single, unified, privacy-first intelligence graph that can reason and answer complex questions with full contextual awareness.

──────

What it does

Panoptes serves as a unified intelligence engine and AI teammate for your entire organization:

• Universal Multi-Channel Integration: Connects with 11+ major enterprise tools:

  • Communication: Email (Gmail, IMAP, Microsoft 365), Messaging (Slack, Microsoft Teams, WhatsApp Business API).

  • Documents & Knowledge: Google Workspace (Docs, Drive, Calendar), Notion, Dropbox.

  • Issue Tracking & Code: Jira Cloud, Linear, GitHub.

  • Meetings: Zoom Cloud recordings and automated transcript ingestion.

• Privacy-First Local Embedding Engine: Embeds organizational data 100% locally on-device using sentence transformers (BAAI/bge-small-en-v1.5). Proprietary internal discussions and documents never leave your local environment to generate vectors.

• Hybrid Retrieval System: Combines full-text lexical keyword search (SQLite FTS5) with dense vector cosine similarity. This ensures exact matches for specialized terms (error codes, commit hashes, ticket IDs) alongside deep semantic understanding of natural language questions. It also offers an option to use ColPali and ColQwen models if the system resources permit. These state-of the-art models for multi-vector embeddings from images of document pages allow for enhanced retrieval from documents with images, graphs, tables and infographics.

• AI Reasoning & Answer Synthesis: Reranks top supporting records across disparate channels and synthesizes clear, structured answers backed by direct citations and source references.

• Real-Time & Background Sync: Supports both pull-based background batch indexing for API-backed providers and push-based real-time webhook ingestion (e.g., HMAC-verified WhatsApp Business events).

• Encrypted Credential Vault: Secures API keys, OAuth tokens, and secrets at rest using symmetric Fernet encryption (cryptography.fernet).

──────

How we built it

Panoptes was built with a privacy-focused, single-node architecture designed for high throughput and security:

• Backend Core: Built in Python with Flask (server.py), providing API endpoints for source connections, OAuth callbacks, background worker dispatching, credential vault management, and hybrid search execution.

• Modular Connector Suite: Created a unified connector package (connectors.py) featuring provider-specific integration modules for Google Workspace OAuth, Microsoft Graph API, Slack Web API, Notion API, Jira REST API, Linear GraphQL API, GitHub API, Dropbox SDK, Zoom Server-to-Server OAuth, IMAP mailbox parsers, and WhatsApp Webhooks.

• Local Embedding & Vector Pipeline: Integrated Hugging Face SentenceTransformers to generate normalized dense vector embeddings on-device, storing vector representations alongside document metadata.

• Database & Hybrid Retrieval: Leveraged SQLite with Write-Ahead Logging (WAL) enabled, using SQLite FTS5 for BM25-style keyword search combined with vector cosine similarity scoring for hybrid search ranking.

• Security & Cryptography: Implemented Fernet symmetric key encryption (.panoptes.key) to lock down provider configuration tokens at rest, paired with SHA-256 HMAC signature verification for incoming webhooks.

• Frontend Dashboard: Designed an interactive, modern visual frontend (index.html, styles.css, app.js) with live sync status reporting, real-time source management, and interactive query-answering interface.

──────

Challenges we ran into

  1. Heterogeneous Data Normalization: Every enterprise tool structures data differently—Slack has threaded message trees, Zoom provides timed transcript VTTs/JSON, Jira has complex issue schemas, and Notion relies on block trees. Normalizing all of these into a single, unified Document schema (title, author, created_at, body, metadata) without losing valuable context was a major design challenge.
  2. API Rate Limits & Deep Sync Depth: Enterprise APIs enforce strict rate limits (e.g., Slack conversations.history tier limits and Microsoft Graph throttle thresholds). We designed bounded batch syncs and incremental cursors to stay within platform quotas while preserving fast sync speeds.
  3. High-Performance Local Vector Search: Running dense vector calculations without heavy, complex vector database dependencies required careful optimization. By pre-normalizing embedding vectors during ingestion, we reduced cosine similarity to fast dot product operations directly over memory arrays.
  4. Multi-Tenant OAuth & Real-Time Security: Handling multi-provider OAuth authorization flows (Google Workspace, M365, Slack) alongside real-time webhook validation (WhatsApp Cloud API) required robust security primitives to protect token lifetimes and verify incoming signatures.
    ──────

Accomplishments that we're proud of

• Integrated Enterprise Connectors: Successfully built and unified 11 distinct enterprise connectors into one cohesive platform.
• Zero-Trust On-Device Embeddings: Engineered an embedding pipeline that runs completely locally, allowing teams to unlock AI search capabilities without exposing proprietary organizational memory to third-party embedding providers.
• Instant Hybrid Search: Delivered lightning-fast response times combining FTS5 keyword indexing and vector search across thousands of cross-channel documents.
• End-to-End Encrypted Vault: Built a seamless credential storage vault that automatically handles token encryption and refresh cycles securely.

──────

What we learned

• Hybrid Search is Essential for Enterprise RAG: Neither vector search nor full-text keyword search is enough on its own. Keyword search excels at pinpointing specific IDs, code symbols, and jargon, while vector search captures natural language intent. Combining both is vital for enterprise queries.

• Privacy is the #1 Blocker for Organizational AI: Organizations want AI intelligence, but they cannot risk leaking confidential emails or internal chat logs. Building a local-first embedding system proved to be the single biggest factor in earning trust.

• Clean Abstractions Simplify Expansion: Defining a strict, modular provider interface made adding new connectors (e.g., moving from Slack to WhatsApp or Linear) straightforward.
──────

What's next for Panoptes

• Granular RBAC & Source ACLs: Implement query-time Access Control Lists (ACLs) so users only search documents and messages they have explicit permission to view in the underlying source tools.
• Graph RAG & Entity Relationships: Build a knowledge graph layer connecting related entities across platforms (e.g., linking a Jira ticket to its corresponding Slack thread, GitHub PR, and Zoom recording).
• Proactive AI Executive Briefings: Evolve Panoptes from a reactive query assistant into a proactive teammate that delivers daily cross-departmental briefings, flags project blockers, and alerts teams to key decisions made across channels.

Built With

Share this project:

Updates