Rin AI Multimodal Intents Agent

RinAI
Architecture Overview Diagram
Intents State Machines Diagram

Inspiration

NEAR Intents is a powerful protocol for multi-chain financial products that allows information, requests, assets, and actions to be exchanged between AI agents, services, and end users. The protocol enables cross-chain swaps by allowing users to submit "intents" that are solved by the Intents solver bus and executed with the intents.near contract.

Currently, the Intents user experience is fragmented and ephemeral:

Deposit tokens into intents.near.
Submit the swap order (quote, then accept).
Wait for cross-chain bridging or solver fill.
Manually withdraw resulting tokens from the contract.
Swap must be executed at time of submission, requiring a quote from the solver before submitting the intent.

This results in multiple manual steps that feel cumbersome—especially if you want repeated or limit-based swaps.

Our solution was to build an AI agent that:

Gathers user swap details (tokens, chain, target price, etc.) via an analysis function.
Manages sub-states (deposit → quote → approve → swap → withdraw) with a hierarchical state machine (HSM).
Enables limit orders or automatic trades by hooking into a schedule manager + monitoring:
- We can poll the Solver (or a price feed) every X minutes, and execute the swap once conditions are met.

What it does

🤖 RinAI is an open-source multimodal AI companion and desktop agent that combines speech processing, LLMs, tool automation, and hierarchical state machines.

Features include:

🎙️ Real-time STT/TTS with Groq & 11Labs
💸 Near Intents Integration
🐦 Twitter scheduling & automation
🧠 GraphRAG memory system
🔧 Extensible tool framework
🎮 VTube Studio integration
💬 YouTube chat interaction
🐥 ElizaOS Twitter Client Integration

RinAI Architecture

Key 💸NEAR Intents features include:

Collect All Parameters in a single flow (e.g., "I want to swap 5 NEAR for ETH when NEAR hits $1.20").
Approval Manager ensures user reviews all swap details (e.g. chain, rate) before finalizing.
Scheduling & Monitoring for:
- Immediate swaps (market orders).
- Future or limit-based swaps (price triggers).
Full Deposit/Execute/Withdraw flow is automated—the agent orchestrates each step, so the user only sees one "approve" action.

The user can utilize either the speech-to-text functionality or the interface to interact with the agent and create a command; for example: 'Swap 5 NEAR for ETH on Ethereum.'

Analyze (LLM-based)
- Our AI agent identifies this as a Intents swap command via the Trigger Detector, extracting:
  - Source token (e.g., NEAR)
  - Destination token (e.g., ETH)
  - Amount (e.g., 5)
  - Target price (optional limit)
  - Chain (if bridging to another blockchain)
  - User's wallet (for final withdrawal)
Confirm & Approve
- The agent organizes these details into a Tool Operation.
- The Approval Manager presents a summary of the details: "You want to swap 5 NEAR for ETH on Ethereum if NEAR >= $1.20."
- Approval Manager ensures the user is good to proceed.
Scheduling Activation
- If it's a limit order or a timed operation, we spin up a Monitoring Service that:
  - Polls the Solver for quotes every X minutes (or uses a price feed first).
  - If the solver quote meets user conditions → triggers the swap immediately.
Hierarchical State Machine Execution
- Sub-states handle the ephemeral nature of NEAR Intents:
  - Deposit user's tokens to intents.near.
  - Fetch solver quote → If user's price is met, accept & finalize the swap.
  - Monitor cross-chain bridging & fill.
  - Withdraw final tokens to user's wallet.
- The user doesn't manually do each step; the agent orchestrates them under the hood.
Completion
- Once the swap is done, our system marks the Tool Operation as COMPLETED.
- The user sees a final message: "Your 5 NEAR swapped into ETH on Ethereum. Tokens withdrawn to 0xYourWallet."

This improves the current NEAR Intents experience in the following ways:

Better UX: Hides the multiple deposit/swap/withdraw steps behind a single "approve or schedule" operation.
Limit Orders: Users can automatically execute swaps only if the solver's price meets their target.
Repeating Trades: Our scheduling logic supports multiple tool items (e.g. "Swap 1 NEAR daily for 5 days at $2.50+").
Scalable: The same approach can be extended to other NEAR-based actions (NFTs, DeFi, multi-chain bridging), harnessing the same hierarchical state machine pattern.

How we built it

State Machines Diagram

RinAI is built with Python, TypeScript, modern AI services, and integration with the NEAR Intents solver bus api.

Project Components

Intents Tool Script
- Manages the complete lifecycle of tool operations
- Our core bridging logic with NEAR Intents
- Submits/accepts solver quotes, handles deposit/withdraw calls
Scheduling Manager
- Manages conditional operations like limit orders
- The monitoring that checks solver quotes (or price feeds) every X minutes
- Triggers the intent execution once conditions are met
Approval Manager
- Handles user authorization workflow
- States: PENDING → APPROVED/REJECTED
- Allows asynchronous approvals (users can approve hours or days later)
Agent State Manager
- Controls high-level conversation state (NORMAL_CHAT ↔ TOOL_OPERATION)
- Manages the transition between conversational interactions and tool executions
- Determines when to parse commands vs. engage in regular dialogue
Orchestrator
- Manages the stateful workflow of tool operations
- Routes tool operations to the respective managers/state machines
AI Command Parsing
- Uses a large language model (LLM) or any structured approach to parse user's text into parameters (tokens, chain, price limit, etc.)
Advanced Features
- Multi-User Approval Flows: Collaborative workflows where different users handle setup vs. approval
- Scheduled Operations: Time-based and condition-based transactions
- Recurring Transactions: Support for "Swap 1 NEAR daily for 5 days"
- Extensibility: Same architecture supports other NEAR-based actions (NFTs, DeFi)

Tech Stack

Backend: Python, Node.js/TypeScript
LLMs: Role-Playing LLM, Claude 3.5
Speech Processing: Groq Whisper AI (STT), 11Labs (TTS)
Frontend: [Vtube Studio, OBS]
Audio: FFmpeg, VoiceMeeter Banana (Windows)

Architecture

1. Multiple Managers, Each with a State Machine

Approval Manager

Manages user sign-off (approval/rejection) for sensitive actions (like big trades or social media posts).
States: PENDING, PARTIAL_APPROVAL, REJECTED, APPROVED.
Value: The user can come back any time to approve—even hours or days later. No ephemeral lockouts.

Scheduling/Monitoring Manager

Handles limit orders or time-based triggers for trades.
States: SCHEDULED, WAITING_FOR_TRIGGER, TRIGGERED, EXECUTING, COMPLETED.
Value: Automates logic like "Swap my NEAR only if price hits $1.20," checking every X minutes until success or expiration.

Tool-Specific Manager (e.g. NEAR Intents tool)

Breaks down ephemeral steps (deposit → swap → withdraw) into persistent states.
States: DEPOSIT_PENDING, DEPOSIT_SUCCESS, SWAP_PENDING, SWAP_SUCCESS, WITHDRAW_PENDING, WITHDRAW_SUCCESS, ERROR, etc.
Value: Tracks partial progress; if bridging times out, it auto-retries or prompts user without losing previous steps.

2. Database-Backed Workflows

All manager state transitions are stored in a MongoDB (or any DB) document, so:

No ephemeral memory: A crash or restart won't lose your progress.
Automatic re-tries: If DEPOSIT fails, the system remains in DEPOSIT_PENDING or DEPOSIT_ERROR until it recovers or the user cancels.
Collaboration: One user can set up the tool operation, another user can finalize it (via Approval Manager).

3. Error Handling & Automatic Retries

Because each manager's state transitions are persisted in the DB:

Timeouts or Failures

If a blockchain deposit fails (e.g., NEAR transaction times out), the state machine remains in an error or re-try state.
The system can resume from that exact sub-state (rather than re-doing all steps).

No More Ephemeral States

You don't lose your progress if the AI agent restarts.
The next time the manager wakes up, it sees "We were in DEPOSIT_PENDING with an error. Let's re-try deposit."
This is a huge upgrade over ephemeral agent-based solutions, especially for unpredictable or long-running crypto tasks.

4. Powerful Approval State Machine

Approval on Your Own Timeline

Some trades or social posts might require a final sign-off from a manager or lead. Our Approval Manager keeps that operation in AWAITING_APPROVAL indefinitely (or until a timeout) in the DB.
No ephemeral expiry: The user can come back hours or days later to confirm or reject.

Multi-User Collaboration

Especially valuable in real world work scenarios where employee performs the agentic action and the manager provides final approval, on their own time-from anywhere in the world
An intern can do the "draft" (e.g. propose a tweet or set up a trade).
The manager (or final approver) logs in later and approves.
The state machine then transitions automatically to the Execution sub-state without missing a beat.

5. Synergy with NEAR Intents Sub-Steps

NEAR Intents inherently require 3 distinct on-chain sub-steps:

Deposit
Swap (quote & accept)
Withdraw

Each can be a sub-state in our architecture, and each can fail or time out. A hierarchical state machine is perfect:

Deposit might fail if user's balance is insufficient or if the NEAR transaction times out.
Swap might fail if the solver rejects the price.
Withdraw might fail if cross-chain bridging is delayed.

Because we store every sub-state in the DB, we can retry deposit or handle partial fills. The user sees a smooth single operation: "Swap 5 NEAR for ETH," but behind the scenes:

DEPOSIT_PENDING → DEPOSIT_SUCCESS → SWAP_PENDING → SWAP_SUCCESS → WITHDRAW_PENDING → WITHDRAW_SUCCESS

6. Unified, Persistent Workflow = Zero Clunkiness

By combining:

Approval Manager (collect user sign-off at any stage)
Scheduling Manager (trigger trades at time or price conditions)
NEAR Intents (multi-chain bridging steps)
Persistent DB (no ephemeral memory)

...we turn "clunky ephemeral transactions" into a smooth user experience:

User: "Limit sell 5 NEAR for ETH at $1.20."
System: Waits for user approval.
User approves a day later.
System automatically deposits, swaps, and withdraws—retrying any step if needed.
Done, with final state tracked in DB.

Challenges we ran into

Complex DeFi strategies require multiple planned steps and can be risky if trusted entirely to AI, as a users assets are at risk

Intents architecture poses some minimal limitations that had to be overcome to provide a smooth workflow for limit orders.

What we learned

Complex tasks like DeFi trades and limit-orders required us to implement state machines to manage sensitive workflows and actions that require human approval. This influenced a large part of the submission.

Price oracle rates vs intents solver bus rates: The intents solver quote for our implemented limit order operations will give us denomination in NEAR not USD. We utilize coingecko api to verify the solver quote meets our limit order operations however the price delta between the limit order and solver quote may be high for larger trades.

Accomplishments that we're proud of

The addition of hierarchical state machines allows for user managed approvals, scheduling, and tool specific operations enabling sensitive/complex workflows (DeFi, Intents, limit orders, etc). In short, ephemeral transaction-based logic is a huge hurdle in crypto. Our hierarchical state machine approach conquers that hurdle, making long or complex on-chain tasks feel straightforward, robust, and collaborative. It's the perfect synergy with NEAR Intents or any multi-step blockchain process!

The multimodal and social components of the RinAI stack advance AI agents to the next level. The framework we have created paves the way for new innovations with web3 and for NEAR protocol.

Project Tracks

DeFi Track: RinAI integration with the NEAR Intents protocol allows for cross trading and limit order scheduling via the Intents solver bus and the RinAI state machine.

Social Media Track: With our custom version of the ElizaOS Twitter Client, RinAI enables automated tweet scheduling and social media management. The RinAI agent uses this client's API server layer to manage Twitter interactions without requiring direct Twitter API access.

Proximity Cross-chain Trading Agents (Sponsor Track): RinAI utilizes user input to interact with NEAR intents contract (intents.near) and the Intents solver bus to instantly swap assets between supported chains or schedule trades (limit orders) to execute at a later date.