ai/studio

Lightning ⚡ RAG User Documentation

Introduction

Lightning RAG (L⚡RAG) is an advanced Retrieval Augmented Generation platform designed to transform how you interact with your data. By combining powerful document processing, database connectivity, and natural language understanding, L⚡RAG enables you to have intelligent conversations with all your information sources through an intuitive chat interface.

This documentation provides comprehensive guidance on how to use Lightning RAG effectively, from creating your first collection to building sophisticated data interactions.

Getting Started

System Requirements

Accessing Lightning RAG

  1. Navigate to your organization's Lightning RAG instance URL
  2. Log in with your credentials
  3. You'll be directed to the Collections dashboard

Core Concepts

Collections

Collections are the fundamental building blocks in Lightning RAG. A collection is a set of related data from a specific source type that has been processed and optimized for conversational AI interaction.

Collection Types

Lightning RAG supports three primary data structure categories:

  1. Unstructured Data

    • PDF Collections
      • Document-based collections with free-form text, tables, images
      • Supports multiple document formats (PDF, DOCX, TXT)
      • Uses OCR and document understanding technology

  2. Semi-structured Data

    • MongoDB Collections
      • Works with NoSQL document databases
      • Handles nested document structures
      • Supports aggregation pipelines
    • API Collections
      • Connects to external APIs
      • Auto-generates schema from OpenAPI/Swagger definitions
      • Handles authentication and parameter mapping

  3. Structured Data

    • SQL Collections
      • Connects to relational databases with rigid schemas
      • Supports schema understanding and query generation
      • Compatible with PostgreSQL, MySQL, SQL Server, and more
    • Excel Collections
      • Processes tabular spreadsheet data
      • Handles multiple sheets and complex formulas
      • Auto-converts to optimized SQL structures internally

Embedding Types

For unstructured data collections (PDF), Lightning RAG offers three embedding technologies:

  1. PaddleOCR

    • High-accuracy document processing optimized for complex layouts
    • Best for documents with mixed content (text, tables, images)
    • Default processing engine

  2. Llama Parse (Cloud)

    • Advanced cloud-based parsing for sophisticated document structures
    • Superior handling of tables and structured data
    • Requires internet connectivity

  3. Docling (On-Premise)

    • Secure, locally-hosted solution for sensitive document processing
    • Ensures data never leaves your infrastructure
    • Ideal for confidential or regulated information

User Interface Overview

Navigation

Collections Dashboard

The Collections dashboard displays all your available collections with key information:

Filtering and Sorting

Creating Collections

Step 1: Initiate Collection Creation

  1. Click the "+ New Collection" button in the top right corner
  2. Select the data structure category and collection type
  3. Enter a name for your collection

Step 2: Configure Source

Depending on the collection type, you'll see different source options:

For Unstructured Data (PDF Collections)

  1. Choose a source:

    • Upload: Upload files from your computer
    • Web Scraper: Extract content from websites
    • URL: Import documents from direct links

  2. Select embedding type:

    • PaddleOCR: Best for general documents
    • Llama Parse: Optimal for complex structures
    • Docling: For sensitive information

For Semi-structured Data

MongoDB Collections:

  1. Enter connection details:
    • Connection URI
    • Database name
    • Collection names
    • Authentication credentials

API Collections:

  1. Enter API details:
    • API name and description
    • Base URL
    • Authentication method
    • Endpoint configuration

For Structured Data

SQL Collections:

  1. Enter database connection details:
    • Database type (PostgreSQL, MySQL, etc.)
    • Host and port
    • Database name
    • Authentication credentials
    • Table selection (optional)

Excel Collections:

  1. Upload Excel files or provide URL
  2. Select sheets to include
  3. Choose embedding options (Schema only or Data enhanced)

Step 3: Create and Build

  1. Click "Create Collection" to initialize your collection
  2. The system will automatically begin the build process
  3. Building includes:
    • Document parsing and OCR (for unstructured data)
    • Schema analysis (for structured and semi-structured data)
    • Vector embedding generation
    • Index optimization

Working with Collections

Collection States

Collections exist in one of two states:

Collection Actions

Build/Rebuild

The Build process prepares your collection for chat interaction:

  1. Click the "Build" button on a collection card
  2. Monitor progress in the detail view
  3. Building time varies based on collection size and complexity

Chat

Start a conversation with your collection:

  1. Click the "Chat" button on a Ready collection
  2. Type natural language questions in the chat interface
  3. Receive AI-generated responses based on your collection data
  4. Follow-up with additional questions for context-aware responses

Share

Share collections with team members:

  1. Click the "Share" button on a collection
  2. Set access permissions (View, Chat, Edit)
  3. Enter recipient email addresses or copy shareable link
  4. Optional: Add expiration date or password protection

Collection Details

Access detailed information and settings by clicking on a collection card:

Overview Tab

Content Tab

Settings Tab

Analytics Tab

Advanced Features

Dynamic Collection Mapping

For enterprise users, Lightning RAG supports dynamic collection mapping:

  1. Create collection templates with variable placeholders
  2. Set up mapping rules based on user roles or session parameters
  3. Collections automatically adapt to the current user context

Published Links

Create embeddable Lightning RAG dashboards:

  1. Configure a collection for publishing
  2. Generate a MINDSHARE_KEY for secure access
  3. Set refresh rate for dashboard data
  4. Embed the published link in other applications

RBAC Settings

Control access with role-based permissions:

  1. Configure organization-level access policies
  2. Assign roles to team members
  3. Set granular permissions for collections and features

Troubleshooting

Common Issues

Collection Building Fails

  1. Check source file formats and compatibility
  2. Verify database connection details
  3. Ensure API endpoints are accessible
  4. Review error logs in the detail view

Chat Responses Are Inaccurate

  1. Rebuild the collection with updated content
  2. Try refining your question with more specific details
  3. Check if the information exists in your collection
  4. For unstructured data, consider changing the embedding type

Performance Issues

  1. Split large collections into smaller, focused collections
  2. Optimize database queries for structured data collections
  3. Reduce the scope of semi-structured collections
  4. Use local embedding types for faster processing of unstructured data

Best Practices

Collection Organization

Data Type Selection Guidelines

Effective Querying

Data Management

Glossary