Native Agent Orchestrator Documentation

Overview

The Native Agent is a client-side AI orchestrator built with LangChain.js libraries that runs directly within the React Native application. Unlike other agent types in the app (multi-agent, browser-agent, custom) that communicate via REST APIs, the Native Agent executes locally using LangChain's tool-calling capabilities to orchestrate multiple integrations and provide intelligent responses.

Architecture

High-Level Architecture

Key Characteristics

• Client-Side Execution: Runs entirely within the React Native app, no backend required
• LangChain Integration: Uses @langchain/core and @langchain/openai libraries
• Model: OpenAI GPT-4o-mini (gpt-4o-mini)
• Agentic Loop: Implements iterative tool calling with a maximum of 5 iterations
• Streaming Support: Supports both streaming and non-streaming response modes
• Observability: Integrated with Langfuse for tracing and monitoring

Core Components

1. Main Orchestrator

File: lib/utils/langchain-helpers.ts

The orchestrator contains two main functions:

• streamAgentResponse(): Handles streaming responses with tool support
• getNonStreamingAgentResponse(): Handles non-streaming responses (complete at once)

Both functions implement the same agentic loop pattern:

// Simplified flow
1. Create LangChain model with tools bound
2. Build message history (system + chat history + user input)
3. Agentic loop (max 5 iterations):
   - Invoke model with messages
   - If tool calls requested → execute tools → add results to messages
   - If no tool calls → return final response
4. Stream or return complete response

2. Tool Definitions

File: lib/services/langchain.ts

Defines all available tools using LangChain's DynamicStructuredTool:

• createWebSearchTool() - Web search via Tavily
• createBolSearchTool() - Bol.com product search
• createJetSearchTool() - Just Eat Takeaway food search
• createSofiaSearchTool() - Sofia travel search

3. System Prompt

File: lib/constants/langchain.ts

Dynamic system message that:

• Includes current date for context
• Provides tool usage guidelines
• Enforces visual result display protocol (brief responses when cards appear)
• Instructs markdown formatting

4. React Integration

File: hooks/use-chat-native-agent.ts

Provides hooks for UI integration:

• handleNativeAgentStreamingResponse() - For streaming mode
• handleNativeAgentNonStreamingResponse() - For non-streaming mode

Agentic Loop Flow

The Native Agent uses an iterative reasoning pattern:

Example Flow: Product Search

1. User: "What are the best noise-cancelling headphones?"
2. Agent: Calls web_search tool with query "best noise-cancelling headphones 2024"
3. Tool Result: Returns article with product recommendations (Sony WH-1000XM5, Bose QuietComfort 45, etc.)
4. Agent: Calls bol_search tool with product names ["Sony WH-1000XM5", "Bose QuietComfort 45"]
5. Tool Result: Returns products with prices, images, and purchase links
6. Agent: Generates brief response: "I found some great options for you!" (products displayed in carousel)

Available Tools (Integrations)

1. Web Search (web_search)

Purpose: Search the web for current information, news, facts, or real-time data.

Integration: Tavily API

When to Use:

• Up-to-date information not in training data
• Product recommendations or comparisons
• Current prices, reviews, or availability
• News and recent events
• Finding specific product names/models before using bol_search

Input Schema:

{
  query: string  // The search query to look up on the web
}

Output Format:

[1] Title
URL: https://example.com
Content summary...

[2] Title
URL: https://example.com
Content summary...

API Details:

• Endpoint: https://api.tavily.com/search
• Method: POST
• Configuration:
  • search_depth: 'basic'
  • include_answer: false
  • max_results: 5

Environment Variable:

• EXPO_PUBLIC_TAVILY_API_KEY - Tavily API key (from .env file)

Implementation: lib/services/tavily.ts

---

2. Bol.com Search (bol_search)

Purpose: Search for products on Bol.com (Dutch online retailer).

Integration: Bol.com Partner API

When to Use:

• User wants to buy products from Bol.com
• After using web_search to find specific product names
• User provides specific product names directly

Important: This tool requires specific product names. Use web_search first to find product names/models.

Input Schema:

{
  productNames: string[]  // Array of specific product names
  // Example: ["Sony WH-1000XM4", "Bose QuietComfort 45"]
}

Output Format:

{
  "products": [
    {
      "id": "string",
      "ean": "string",
      "title": "string",
      "price": number,
      "urls": {
        "desktop": "string"
      },
      "images": [
        {
          "url": "string"
        }
      ]
    }
  ]
}

API Details:

• Base URL: https://api.bol.com
• Endpoint: /products/search
• Method: GET
• Authentication: OAuth 2.0 Bearer token
• Parameters:
  • search-term: Product name
  • country-code: 'nl' (default)
  • page-size: 20 (default)

Environment Variables:

• EXPO_PUBLIC_BOL_CLIENT_ID - Bol.com API client ID (from .env file)
• EXPO_PUBLIC_BOL_CLIENT_SECRET - Bol.com API client secret (from .env file)

Implementation: lib/services/bol.ts

Visual Response: Products appear in a carousel below the agent's message. The agent must keep text response to 1-2 sentences maximum.

---

3. JET Search (jet_search)

Purpose: Search for food and restaurants on JET (Just Eat Takeaway / Thuisbezorgd).

Integration: JET API

When to Use:

• User wants to order food for delivery
• Finding nearby restaurants
• Searching for specific dishes or cuisines
• Discovering top-rated or popular food nearby

Input Schema:

{
  query: string  // Food type or cuisine
  // Examples: "pizza", "burger", "sushi", "thai food"
  // Use "food" for generic discovery queries
}

Output Format:

{
  "query": "string",
  "totalRestaurants": number,
  "restaurants": [
    {
      "restaurantId": "string",
      "restaurantSeoName": "string",
      "restaurantName": "string",
      "restaurantLogoUrl": "string",
      "restaurantRating": {
        "count": number,
        "starRating": number
      },
      "products": [
        {
          "id": "string",
          "name": "string",
          "price": number,
          "priceFormatted": "string",
          "imageUrl": "string"
        }
      ]
    }
  ],
  "products": [...]  // Flattened products array
}

API Details:

• Base URL: https://rest.api.eu-central-1.production.jet-external.com
• Web Proxy: http://localhost:3001/jet (for web platform to bypass CORS)
• Endpoints:
  • /search/restaurants/nl - Restaurant search
  • /discovery/nl/restaurants/enriched - Restaurant details
• Method: GET
• Headers:
  • accept-tenant: 'nl'
  • accept-language: 'nl-NL'
• Parameters:
  • searchTerm: Food query
  • latlong: User coordinates (from user profile)
  • limit: 300 (default)
  • serviceType: 'delivery'

Location Requirements: Uses user's latitude/longitude from user profile store.

Environment Variables: None required (public API, but requires user location)

Implementation: lib/services/merchants/jet/jet.ts

Visual Response: Restaurant cards appear below the agent's message. The agent must keep text response to 1-2 sentences maximum.

---

4. Sofia Search (sofia_search)

Purpose: Search for flights and hotels using Sofia, Despegar's AI travel assistant.

Integration: Sofia (Despegar) API

When to Use:

• User wants to search for flights to a destination
• Finding hotels in a city or area
• Planning a trip or vacation
• Getting travel recommendations

Input Schema:

{
  query: string  // Natural language travel query
  // Examples: 
  // "flights to Paris next week"
  // "hotels in Barcelona for 2 adults"
  // "cheap flights from Amsterdam to New York"
}

Output Format:

{
  "query": "string",
  "response": "string",  // Sofia's text response
  "components": {
    "components": [
      {
        "component": "string",  // "ACCOMMODATION" or "FLIGHT"
        "data": {
          "items": [
            {
              "productType": "HOTEL" | "FLIGHT",
              "hotel_name": "string",  // For hotels
              "price": {...},  // For flights
              "segments": [...],  // For flights
              // ... extensive hotel/flight data
            }
          ]
        }
      }
    ]
  }
}

API Details:

• Base URL: https://sofia.despegar.com/chappie
• Endpoints:
  • POST /chats - Create new chat session
  • PUT /chats/{chatId} - Send message to chat
• Method: POST/PUT
• Headers:
  • Content-Type: 'application/json'
• Body:
  • text: User query
  • temporalUserId: Session user ID
  • context.assistantId: 'despegar'
  • xLocale: 'en_US'

Session Management:

• Creates a new Sofia chat session on first use
• Persists session (chatId and temporalUserId) per conversation
• Reuses session for subsequent queries in the same conversation

Rate Limiting:

• Implements retry logic with exponential backoff
• Maximum 3 retries
• Handles 429 (Too Many Requests) responses

Environment Variables: None required (public API)

Implementation: lib/services/sofia.ts

Visual Response: Hotel/flight cards with images, prices, amenities appear below the agent's message. The agent must keep text response to 1-2 sentences maximum.

---

Observability

Langfuse Integration

The Native Agent includes comprehensive observability through Langfuse, a LLM observability platform.

Implementation: lib/services/langfuse.ts

Features:

• Custom React Native-compatible implementation (avoids dynamic import issues)
• Trace creation for each agent response
• Span tracking for tool calls
• Session-based grouping (uses conversation ID as session ID)
• Automatic event batching and flushing

Trace Structure:

Trace (agent-response or agent-response-streaming)
├── Input: { userInput, historyLength }
├── Spans: tool-web_search, tool-bol_search, etc.
└── Output: { content, hasProducts, maxIterationsReached? }

Environment Variables (Optional):

• EXPO_PUBLIC_LANGFUSE_SECRET_KEY - Langfuse secret key (from .env file)
• EXPO_PUBLIC_LANGFUSE_PUBLIC_KEY - Langfuse public key (from .env file)
• EXPO_PUBLIC_LANGFUSE_HOST - Langfuse host (default: https://cloud.langfuse.com)

Usage:

// Automatically created in orchestrator
const trace = createLangfuseTrace("agent-response", sessionId);
trace?.update({ input: { userInput, historyLength } });
trace?.span({ name: `tool-${toolName}`, input: toolArgs });
trace?.update({ output: { content, hasProducts } });
await flushLangfuse();

Communication Modes

The Native Agent supports two communication modes:

1. Streaming Mode

Function: streamAgentResponse()

Behavior:

• Executes tool calls first (non-streaming)
• Streams the final response token-by-token after tools complete
• Provides real-time feedback to users
• Better UX for longer responses

Usage:

const result = await streamAgentResponse(
  input,
  chatHistory,
  signal,
  onChunk,  // Callback for each content update
  onToolStart,  // Callback when tool is called
  sessionId,
  sofiaContext
);

2. Non-Streaming Mode

Function: getNonStreamingAgentResponse()

Behavior:

• Executes tool calls
• Returns complete response at once
• Useful when streaming is disabled in settings
• Simpler implementation

Usage:

const result = await getNonStreamingAgentResponse(
  input,
  chatHistory,
  signal,
  onToolStart,
  sessionId,
  sofiaContext
);

Note: Polling mode is not used by the Native Agent (only for external agent types).

Session Management

Sofia Session Persistence

Sofia (travel search) maintains session state across messages in a conversation:

Storage:

• Stored in conversation record in local database
• Fields: sofia_chat_id, sofia_temporal_user_id

Lifecycle:

1. First Sofia query in conversation → Creates new session
2. Subsequent queries → Reuses existing session
3. Session persists until conversation is deleted

Implementation:

// Load existing session
const existingSession = await loadSofiaSession(convId);

// Use in agent context
const sofiaContext = { existingSession };

// Save new session if created
if (result.sofiaSession) {
  await saveSofiaSession(convId, result.sofiaSession);
}

Files:

• Session loading/saving: hooks/use-chat-native-agent.ts
• Database operations: Conversation repository

Chat History Management

• Messages stored in local SQLite database
• History passed to agent as LangChain BaseMessage[] format
• System message automatically prepended to each request
• Conversation context maintained across messages

Configuration & Environment Variables

All configuration is done through environment variables in .env.local:

Required Variables

Variable	Description	Source
EXPO_PUBLIC_OPENAI_API_KEY	OpenAI API key for GPT-4o-mini	OpenAI Platform
EXPO_PUBLIC_TAVILY_API_KEY	Tavily API key for web search	Tavily
EXPO_PUBLIC_BOL_CLIENT_ID	Bol.com Partner API client ID	Bol.com Partner Platform
EXPO_PUBLIC_BOL_CLIENT_SECRET	Bol.com Partner API client secret	Bol.com Partner Platform

Optional Variables

Variable	Description	Default
EXPO_PUBLIC_LANGFUSE_SECRET_KEY	Langfuse secret key for observability	Not configured
EXPO_PUBLIC_LANGFUSE_PUBLIC_KEY	Langfuse public key for observability	Not configured
EXPO_PUBLIC_LANGFUSE_HOST	Langfuse host URL	https://cloud.langfuse.com

Important: Never commit API keys to version control. All keys should be stored in .env.local (which is git-ignored).

Important Note on API Endpoints

⚠️ The Native Agent does NOT expose REST API endpoints.

The Native Agent runs entirely client-side within the React Native application. It uses LangChain libraries to directly call:

• OpenAI API (for the LLM)
• Tavily API (for web search)
• Bol.com API (for product search)
• JET API (for food search)
• Sofia API (for travel search)

All API calls are made directly from the client application.

Other Agent Types

The app supports other agent types that DO use REST APIs:

• Multi-Agent: Connects to external multi-agent API (A2A protocol)
• Browser-Agent: Connects to browser automation API
• Custom Agent: Connects to user-configured API endpoints

These agent types are configured in the Agent Settings screen and use different code paths (use-chat-custom-agent.ts, use-chat-polling-agent.ts).

System Architecture Diagram

Code Examples

Example 1: Basic Agent Query Flow

// User sends: "What are the best noise-cancelling headphones?"

// 1. Message received in useChat hook
const handleSendMessage = async (content: string) => {
  // ... create assistant message
  
  // 2. Route to Native Agent handler
  if (config.agentType === 'native') {
    await handleNativeAgentStreamingResponse(
      content,
      chatHistory,
      abortSignal,
      assistantMessageId,
      convId,
      updateMessage,
      handleStreamComplete
    );
  }
};

// 3. Inside handleNativeAgentStreamingResponse
const result = await streamAgentResponse(
  content,  // "What are the best noise-cancelling headphones?"
  chatHistory,
  signal,
  onContentUpdate,  // Updates UI as tokens arrive
  onToolCall,  // Shows "Searching web..." indicator
  convId,
  sofiaContext
);

// 4. Agent decides to call web_search
// Tool executed: web_search({ query: "best noise-cancelling headphones 2024" })

// 5. Agent receives results, decides to call bol_search
// Tool executed: bol_search({ productNames: ["Sony WH-1000XM5", "Bose QuietComfort 45"] })

// 6. Agent generates final response
// Response: "I found some great options for you!"
// Products displayed in carousel below message

Example 2: Tool Execution Pattern

// Inside executeToolCall function
async function executeToolCall(toolCall, onToolStart, sofiaContext) {
  // Notify UI about tool usage
  onToolStart(toolCall.name, toolCall.args.query);
  
  let toolResult: string;
  let products: Product[] | undefined;
  
  if (toolCall.name === "web_search") {
    const tool = createWebSearchTool();
    toolResult = await tool.invoke({ query: toolCall.args.query });
  } 
  else if (toolCall.name === "bol_search") {
    const tool = createBolSearchTool();
    toolResult = await tool.invoke({ productNames: toolCall.args.productNames });
    
    // Parse and extract products
    const bolResult = JSON.parse(toolResult);
    products = bolResult.products.map(p => ({
      title: p.title,
      price: p.price,
      image: p.images?.[0]?.url,
      url: p.urls?.desktop,
      type: "bol"
    }));
  }
  // ... other tools
  
  return { result: toolResult, products };
}

Example 3: Agentic Loop Implementation

// Simplified agentic loop from streamAgentResponse
const messages = [
  new SystemMessage(getLangChainSystemMessage()),
  ...chatHistory,
  new HumanMessage(input)
];

for (let iteration = 0; iteration < MAX_TOOL_ITERATIONS; iteration++) {
  const response = await modelWithTools.invoke(messages);
  
  if (response.tool_calls && response.tool_calls.length > 0) {
    // Execute tools
    for (const toolCall of response.tool_calls) {
      const { result, products } = await executeToolCall(toolCall);
      
      // Add tool call and result to message history
      messages.push(new AIMessage({ tool_calls: [toolCall] }));
      messages.push(new ToolMessage({ content: result, tool_call_id: toolCall.id }));
    }
    // Continue loop to let agent process tool results
  } else {
    // No more tool calls - stream final response
    const streamingModel = createLangChainModel(true);
    const content = await streamLangChainResponse(streamingModel, messages, signal, onChunk);
    return { content, products };
  }
}

Example 4: Sofia Session Management

// Loading existing session
async function loadSofiaSession(convId: string): Promise<SofiaSession | null> {
  const conversation = await conversationsDb.getById(convId);
  if (conversation?.sofia_chat_id && conversation?.sofia_temporal_user_id) {
    return {
      chatId: conversation.sofia_chat_id,
      temporalUserId: conversation.sofia_temporal_user_id,
    };
  }
  return null;
}

// Using session in agent
const existingSofiaSession = await loadSofiaSession(convId);
const sofiaContext = { existingSession: existingSofiaSession };

const result = await streamAgentResponse(
  content,
  chatHistory,
  signal,
  onChunk,
  onToolCall,
  convId,
  sofiaContext  // Passed to sofia_search tool
);

// Saving new session
if (result.sofiaSession) {
  await saveSofiaSession(convId, result.sofiaSession);
}

Tool Response Protocol

When tools return visual results (products, restaurants, hotels, flights), the agent follows a strict protocol:

Visual Results Display Protocol

Tools that trigger visual components:

• bol_search → Product carousel
• jet_search → Restaurant cards
• sofia_search → Hotel/flight cards

Agent Response Rules:

1. ✅ Maximum 1-2 sentences
2. ✅ Generic acknowledgment only (e.g., "I found some great options!")
3. ❌ NO product/hotel/restaurant names
4. ❌ NO prices, ratings, or amenities
5. ❌ NO descriptions or details

Why: Visual components below the message show all details. Repeating information makes responses unnecessarily long and redundant.

Example Good Response:

"I found some great options for you!"

Example Bad Response:

"I found several hotels in Barcelona. Hotel A is 4 stars and costs €120 per night. Hotel B is 5 stars with a pool and costs €200 per night. Hotel C is..."

Error Handling

Abort Signal Support

All agent functions support cancellation via AbortSignal:

const controller = new AbortController();
const signal = controller.signal;

// User cancels request
controller.abort();

// Agent checks abort signal at multiple points:
// - Before starting
// - Before each tool call
// - During streaming

Tool Execution Errors

• Tools handle errors gracefully and return error messages
• Agent continues with error information in tool result
• Maximum iterations prevent infinite loops

Rate Limiting

• Sofia API implements retry logic with exponential backoff
• Other APIs rely on standard HTTP error handling
• Langfuse events are queued and batched to avoid rate limits

Performance Considerations

Tool Execution

• Tools execute sequentially (not in parallel)
• Each tool call adds latency
• Maximum 5 iterations prevents long-running loops

Streaming

• Streaming provides better UX for longer responses
• Tool execution happens before streaming starts
• Users see tool call indicators during tool execution

Caching

• JET restaurant details are cached to reduce API calls
• Sofia sessions are persisted to avoid recreation
• Chat history is stored locally

Limitations

1. Client-Side Only: All processing happens on the device, which may have performance implications on slower devices
2. API Rate Limits: Subject to rate limits of all integrated APIs
3. Network Dependency: Requires internet connection for all tool calls
4. Maximum Iterations: Limited to 5 tool call iterations to prevent infinite loops
5. Sequential Tool Execution: Tools execute one at a time, not in parallel

Future Enhancements

Potential improvements:

• Parallel tool execution where possible
• Tool result caching
• Configurable maximum iterations
• Additional tool integrations
• Backend proxy option for sensitive API keys

Related Documentation

• Project README: See the main project README in the repository root
• Agent Settings UI: app/agent-settings.tsx in the main application
• Database Schema: lib/database/types.ts in the main application