AI Ingestion

Import and organize documentation using AI-powered content extraction

AI Ingestion

Import and organize documentation automatically using AI-powered content extraction. AGO analyzes uploaded files or text, extracts knowledge entries, deduplicates against existing content, and organizes documents into your knowledge hierarchy.

Overview

AI Ingestion provides:

• Smart Extraction: AI analyzes content and extracts meaningful knowledge entries
• Automatic Deduplication: Detects duplicates and updates existing documents
• Hierarchy Organization: Places documents in the appropriate location
• Multiple Formats: Supports PDF, Word, PowerPoint, Excel, HTML, and more

Supported File Formats

Documents

Format	Extensions	MIME Types
PDF	.pdf	application/pdf
Word	.doc, .docx, .docm	application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/vnd.ms-word.document.macroEnabled.12
PowerPoint	.ppt, .pptx	application/vnd.ms-powerpoint, application/vnd.openxmlformats-officedocument.presentationml.presentation
Excel	.xls, .xlsx	application/vnd.ms-excel, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Plain Text	.txt	text/plain
CSV	.csv	text/csv
JSON	.json	application/json
XML	.xml	application/xml, text/xml
HTML	.html	text/html
EPUB	.epub	application/epub+zip
ZIP	.zip	application/zip

Images (with OCR)

Format	Extensions
JPEG	.jpg, .jpeg
PNG	.png
GIF	.gif
WebP	.webp
BMP	.bmp
TIFF	.tiff

---

How It Works

1. Content Parsing

   Files are converted to Markdown text. The system computes a content hash (SHA-256) to detect re-processing of identical content.

2. Granularity Analysis

   AI analyzes the content structure to determine the best extraction strategy:

   Strategy	Description	Best For
   Atomic	Individual facts, FAQs, definitions	Reference documentation, glossaries
   Thematic	Cohesive narratives, connected concepts	Guides, tutorials, explanations
   Mixed	Both atomic facts and narrative sections	Documentation with both facts and narratives

3. Knowledge Extraction

   AI extracts knowledge entries with metadata:

   Entry Type	Description
   Atomic Fact	Standalone piece of information
   Thematic Block	Connected narrative or explanation
   Procedure	Step-by-step instructions
   Definition	Term or concept definition
   Example	Code sample or use case

   Each entry includes:

   • Title
   • Content
   • Summary
   • Entry type
   • Confidence score (0.0 - 1.0)
4. Deduplication

   For each extracted entry, the system checks for duplicates:

   1. Semantic Search: Find similar documents using embeddings (threshold: 70% similarity)
   2. LLM Analysis: Compare content and determine action

   Decision	Action
   Duplicate	Skip — identical content exists
   Update	Update existing document with newer content
   Related	Create new document (different aspect of topic)
   Different	Create new document (no relation)

5. Organization

   New documents are placed in the appropriate location:

   • AI determines the best parent category
   • Creates new category nodes if needed
   • Applies relevant tags
   • Sets position in hierarchy

---

Ingesting Content

From File Upload

Via Admin Interface:

1. Navigate to Knowledge → Sources
2. Select a source
3. Click Import > Upload File
4. Select your file(s)
5. Monitor progress in extraction logs

From Raw Text

You can also ingest raw text content. The system supports both synchronous processing (waits for completion) and asynchronous processing (runs in the background while you continue working).

ℹ️ NOTE: File upload and text ingestion are also available via API. See the Public API v1 Reference for endpoint details.

---

Monitoring Extractions

Extraction Status

Track the progress of an ingestion operation from the extraction logs in the admin interface. Each extraction shows its current status, source file, and processing statistics (documents created, updated, and skipped).

Status Values

Status	Description
Pending	Queued, waiting to start
Extracting	Parsing file and analyzing content
Deduplicating	Checking for duplicates
Organizing	Placing documents in hierarchy
Completed	Successfully finished
Failed	Error occurred (check error message for details)

List Extractions

View all extraction operations in the admin interface by navigating to the source’s extraction logs. You can filter extractions by source and status.

View Created Documents

After an extraction completes, you can view the documents that were created or modified by clicking on the extraction entry in the logs.

---

Extraction Results

Document Metadata

Documents created via AI ingestion include additional metadata such as the extraction source, entry type, confidence score, and an AI-generated summary. This metadata helps you identify and manage AI-imported content.

Hierarchy Changes

The extraction log records hierarchy modifications, including new categories that were created and where documents were placed in the hierarchy.

---

Best Practices

Content Preparation

• Clean formatting: Remove headers/footers from PDFs
• Structured content: Use headings for better extraction
• Reasonable size: Very large documents may be split

Source Organization

• Create a dedicated source for AI-imported content
• Use meaningful source identifiers for tracking
• Review extracted documents for accuracy

Deduplication

• Import to sources with existing content for best deduplication
• Re-running import on same file skips unchanged content (hash detection)
• Review “Updated” documents to verify changes

---

Related

• Knowledge Source Management — Managing sources
• Knowledge Overview — Feature overview