> ## Documentation Index
> Fetch the complete documentation index at: https://ago.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# AI Ingestion
> Import and organize documentation using AI-powered content extraction
Import and organize documentation automatically using AI-powered content extraction. AGO analyzes uploaded files or text, extracts knowledge entries, deduplicates against existing content, and organizes documents into your knowledge hierarchy.
## Overview
AI Ingestion provides:
* **Smart Extraction**: AI analyzes content and extracts meaningful knowledge entries
* **Automatic Deduplication**: Detects duplicates and updates existing documents
* **Hierarchy Organization**: Places documents in the appropriate location
* **Multiple Formats**: Supports PDF, Word, PowerPoint, Excel, HTML, and more
## Supported File Formats
### Documents
| Format | Extensions | MIME Types |
| ---------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| PDF | .pdf | application/pdf |
| Word | .doc, .docx, .docm | application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/vnd.ms-word.document.macroEnabled.12 |
| PowerPoint | .ppt, .pptx | application/vnd.ms-powerpoint, application/vnd.openxmlformats-officedocument.presentationml.presentation |
| Excel | .xls, .xlsx | application/vnd.ms-excel, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet |
| Plain Text | .txt | text/plain |
| CSV | .csv | text/csv |
| JSON | .json | application/json |
| XML | .xml | application/xml, text/xml |
| HTML | .html | text/html |
| EPUB | .epub | application/epub+zip |
| ZIP | .zip | application/zip |
### Images (with OCR)
| Format | Extensions |
| ------ | ----------- |
| JPEG | .jpg, .jpeg |
| PNG | .png |
| GIF | .gif |
| WebP | .webp |
| BMP | .bmp |
| TIFF | .tiff |
***
## How It Works
```mermaid theme={null}
flowchart TD
A[Upload File/Text] --> B[Content Parsing]
B --> C[Granularity Analysis]
C --> D[Knowledge Extraction]
D --> E[Deduplication Check]
E --> F{Decision}
F -->|New| G[Create Document]
F -->|Duplicate| H[Skip]
F -->|Update| I[Update Existing]
G --> J[Organize in Hierarchy]
I --> J
J --> K[Generate Embeddings]
K --> L[Ready for Search]
```
Files are converted to Markdown text. The system computes a content hash (SHA-256) to detect re-processing of identical content.
AI analyzes the content structure to determine the best extraction strategy:
| Strategy | Description | Best For |
| ------------ | ---------------------------------------- | -------------------------------------------- |
| **Atomic** | Individual facts, FAQs, definitions | Reference documentation, glossaries |
| **Thematic** | Cohesive narratives, connected concepts | Guides, tutorials, explanations |
| **Mixed** | Both atomic facts and narrative sections | Documentation with both facts and narratives |
AI extracts knowledge entries with metadata:
| Entry Type | Description |
| -------------- | ---------------------------------- |
| Atomic Fact | Standalone piece of information |
| Thematic Block | Connected narrative or explanation |
| Procedure | Step-by-step instructions |
| Definition | Term or concept definition |
| Example | Code sample or use case |
Each entry includes:
* Title
* Content
* Summary
* Entry type
* Confidence score (0.0 - 1.0)
For each extracted entry, the system checks for duplicates:
1. **Semantic Search**: Find similar documents using embeddings (threshold: 70% similarity)
2. **LLM Analysis**: Compare content and determine action
| Decision | Action |
| --------- | ----------------------------------------------- |
| Duplicate | Skip — identical content exists |
| Update | Update existing document with newer content |
| Related | Create new document (different aspect of topic) |
| Different | Create new document (no relation) |
New documents are placed in the appropriate location:
* AI determines the best parent category
* Creates new category nodes if needed
* Applies relevant tags
* Sets position in hierarchy
***
## Ingesting Content
### From File Upload
**Via Admin Interface:**
1. Navigate to **Knowledge** → **Sources**
2. Select a source
3. Click **Import** > **Upload File**
4. Select your file(s)
5. Monitor progress in extraction logs
### From Raw Text
You can also ingest raw text content. The system supports both synchronous processing (waits for completion) and asynchronous processing (runs in the background while you continue working).
> **ℹ️ NOTE**: File upload and text ingestion are also available via API. See the [Public API v1 Reference](../api/public-api-v1) for endpoint details.
***
## Monitoring Extractions
### Extraction Status
Track the progress of an ingestion operation from the extraction logs in the admin interface. Each extraction shows its current status, source file, and processing statistics (documents created, updated, and skipped).
### Status Values
| Status | Description |
| ------------- | ------------------------------------------------ |
| Pending | Queued, waiting to start |
| Extracting | Parsing file and analyzing content |
| Deduplicating | Checking for duplicates |
| Organizing | Placing documents in hierarchy |
| Completed | Successfully finished |
| Failed | Error occurred (check error message for details) |
### List Extractions
View all extraction operations in the admin interface by navigating to the source's extraction logs. You can filter extractions by source and status.
### View Created Documents
After an extraction completes, you can view the documents that were created or modified by clicking on the extraction entry in the logs.
***
## Extraction Results
### Document Metadata
Documents created via AI ingestion include additional metadata such as the extraction source, entry type, confidence score, and an AI-generated summary. This metadata helps you identify and manage AI-imported content.
### Hierarchy Changes
The extraction log records hierarchy modifications, including new categories that were created and where documents were placed in the hierarchy.
***
## Best Practices
### Content Preparation
* **Clean formatting**: Remove headers/footers from PDFs
* **Structured content**: Use headings for better extraction
* **Reasonable size**: Very large documents may be split
### Source Organization
* Create a dedicated source for AI-imported content
* Use meaningful source identifiers for tracking
* Review extracted documents for accuracy
### Deduplication
* Import to sources with existing content for best deduplication
* Re-running import on same file skips unchanged content (hash detection)
* Review "Updated" documents to verify changes
***
## Related
* [Knowledge Source Management](./knowledge-source-management) — Managing sources
* [Knowledge Overview](./knowledge-overview) — Feature overview