Skip to content

Fixes: crawl code storage issue with <think> tags for ollama models. #775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
coleam00 merged 2 commits into from
Oct 10, 2025
Merged

Fixes: crawl code storage issue with <think> tags for ollama models. #775

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
81c0f49
Fixes: crawl code storage issue with <think> tags for ollama models.
sean-esk Oct 10, 2025
ab853b7
updates from code rabbit review
sean-esk Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 76 additions & 1 deletion migration/complete_setup.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -782,6 +782,12 @@ CREATE POLICY "Allow public read access to archon_code_examples"
TO public
USING (true);

CREATE POLICY "Allow public read access to archon_page_metadata"
ON archon_page_metadata
FOR SELECT
TO public
USING (true);
Comment on lines +785 to +789
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Move archon_page_metadata policy after the table is created

This CREATE POLICY runs before archon_page_metadata exists (the table is only created later in Section 6.5), so complete_setup.sql will fail on clean installs. Please relocate the policy definition to immediately after the table creation/RLS enablement in Section 6.5.

-CREATE POLICY "Allow public read access to archon_page_metadata"
-  ON archon_page_metadata
-  FOR SELECT
-  TO public
-  USING (true);
...
 -- Enable RLS on archon_page_metadata
 ALTER TABLE archon_page_metadata ENABLE ROW LEVEL SECURITY;
+
+CREATE POLICY "Allow public read access to archon_page_metadata"
+  ON archon_page_metadata
+  FOR SELECT
+  TO public
+  USING (true);
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
CREATE POLICY "Allow public read access to archon_page_metadata"
ON archon_page_metadata
FOR SELECT
TO public
USING (true);
-CREATE POLICY "Allow public read access to archon_page_metadata"
- ON archon_page_metadata
- FOR SELECT
- TO public
- USING (true);
-- Enable RLS on archon_page_metadata
ALTER TABLE archon_page_metadata ENABLE ROW LEVEL SECURITY;
CREATE POLICY "Allow public read access to archon_page_metadata"
ON archon_page_metadata
FOR SELECT
TO public
USING (true);
🤖 Prompt for AI Agents 
In migration/complete_setup.sql around lines 785 to 789, the CREATE POLICY for
archon_page_metadata is defined before the archon_page_metadata table exists
which causes failures on clean installs; move this CREATE POLICY statement to
immediately after the archon_page_metadata table creation and its RLS (row-level
security) enablement in Section 6.5 so the table exists when the policy is
created, ensuring ordering: create table -> enable RLS -> CREATE POLICY.


-- =====================================================
-- SECTION 7: PROJECTS AND TASKS MODULE
-- =====================================================
Expand Down Expand Up @@ -954,6 +960,73 @@ COMMENT ON COLUMN archon_document_versions.change_type IS 'Type of change: creat
COMMENT ON COLUMN archon_document_versions.document_id IS 'For docs arrays, the specific document ID that was changed';
COMMENT ON COLUMN archon_document_versions.task_id IS 'DEPRECATED: No longer used for new versions, kept for historical task version data';

-- =====================================================
-- SECTION 6.5: PAGE METADATA FOR PAGE-BASED RAG
-- =====================================================

-- Create archon_page_metadata table
-- This table stores complete documentation pages alongside chunks for improved agent context retrieval
CREATE TABLE IF NOT EXISTS archon_page_metadata (
-- Primary identification
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
source_id TEXT NOT NULL,
url TEXT NOT NULL,

-- Content
full_content TEXT NOT NULL,

-- Section metadata (for llms-full.txt H1 sections)
section_title TEXT,
section_order INT DEFAULT 0,

-- Statistics
word_count INT NOT NULL,
char_count INT NOT NULL,
chunk_count INT NOT NULL DEFAULT 0,

-- Timestamps
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),

-- Flexible metadata storage
metadata JSONB DEFAULT '{}'::jsonb,

-- Constraints
CONSTRAINT archon_page_metadata_url_unique UNIQUE(url),
CONSTRAINT archon_page_metadata_source_fk FOREIGN KEY (source_id)
REFERENCES archon_sources(source_id) ON DELETE CASCADE
);

-- Add page_id foreign key to archon_crawled_pages
-- This links chunks back to their parent page
-- NULLABLE because existing chunks won't have a page_id yet
ALTER TABLE archon_crawled_pages
ADD COLUMN IF NOT EXISTS page_id UUID REFERENCES archon_page_metadata(id) ON DELETE SET NULL;

-- Create indexes for query performance
CREATE INDEX IF NOT EXISTS idx_archon_page_metadata_source_id ON archon_page_metadata(source_id);
CREATE INDEX IF NOT EXISTS idx_archon_page_metadata_url ON archon_page_metadata(url);
CREATE INDEX IF NOT EXISTS idx_archon_page_metadata_section ON archon_page_metadata(source_id, section_title, section_order);
CREATE INDEX IF NOT EXISTS idx_archon_page_metadata_created_at ON archon_page_metadata(created_at);
CREATE INDEX IF NOT EXISTS idx_archon_page_metadata_metadata ON archon_page_metadata USING GIN(metadata);
CREATE INDEX IF NOT EXISTS idx_archon_crawled_pages_page_id ON archon_crawled_pages(page_id);

-- Add comments to document the table structure
COMMENT ON TABLE archon_page_metadata IS 'Stores complete documentation pages for agent retrieval';
COMMENT ON COLUMN archon_page_metadata.source_id IS 'References the source this page belongs to';
COMMENT ON COLUMN archon_page_metadata.url IS 'Unique URL of the page (synthetic for llms-full.txt sections with #anchor)';
COMMENT ON COLUMN archon_page_metadata.full_content IS 'Complete markdown/text content of the page';
COMMENT ON COLUMN archon_page_metadata.section_title IS 'H1 section title for llms-full.txt pages';
COMMENT ON COLUMN archon_page_metadata.section_order IS 'Order of section in llms-full.txt file (0-based)';
COMMENT ON COLUMN archon_page_metadata.word_count IS 'Number of words in full_content';
COMMENT ON COLUMN archon_page_metadata.char_count IS 'Number of characters in full_content';
COMMENT ON COLUMN archon_page_metadata.chunk_count IS 'Number of chunks created from this page';
COMMENT ON COLUMN archon_page_metadata.metadata IS 'Flexible JSON metadata (page_type, knowledge_type, tags, etc)';
COMMENT ON COLUMN archon_crawled_pages.page_id IS 'Foreign key linking chunk to parent page';

-- Enable RLS on archon_page_metadata
ALTER TABLE archon_page_metadata ENABLE ROW LEVEL SECURITY;

-- =====================================================
-- SECTION 7: MIGRATION TRACKING
-- =====================================================
Expand Down Expand Up @@ -991,7 +1064,9 @@ VALUES
('0.1.0', '006_ollama_create_indexes_optional'),
('0.1.0', '007_add_priority_column_to_tasks'),
('0.1.0', '008_add_migration_tracking'),
('0.1.0', '009_add_cascade_delete_constraints')
('0.1.0', '009_add_cascade_delete_constraints'),
('0.1.0', '009_add_provider_placeholders'),
('0.1.0', '010_add_page_metadata_table')
ON CONFLICT (version, migration_name) DO NOTHING;

-- Enable Row Level Security on migrations table
Expand Down
Loading