RidhiD. commited on
Commit
1ebc860
Β·
1 Parent(s): c36e692

Ingestion script and test file

Browse files
.gitignore ADDED
@@ -0,0 +1 @@
 
 
1
+ cuad-demo-quadrant/.env
.vscode/settings.json ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "python-envs.defaultEnvManager": "ms-python.python:venv",
3
+
4
+ // ── File watcher exclusions (reduces inotify load & CPU usage) ──────────
5
+ "files.watcherExclude": {
6
+ "**/.git/objects/**": true,
7
+ "**/.git/subtree-cache/**": true,
8
+ "**/__pycache__/**": true,
9
+ "**/*.pyc": true,
10
+ "**/*.pyo": true,
11
+ "**/.venv/**": true,
12
+ "**/venv/**": true,
13
+ "**/.tox/**": true,
14
+ "**/node_modules/**": true,
15
+ "**/dist/**": true,
16
+ "**/build/**": true,
17
+ "**/cuad_data/**": true,
18
+ "**/full_contract_pdf/**": true,
19
+ "**/*.pdf": true,
20
+ "**/*.PDF": true
21
+ },
22
+
23
+ // ── Explorer exclusions (hides clutter from the sidebar) ─────────────────
24
+ "files.exclude": {
25
+ "**/__pycache__": true,
26
+ "**/*.pyc": true,
27
+ "**/*.pyo": true,
28
+ "**/.pytest_cache": true,
29
+ "**/*.egg-info": true
30
+ },
31
+
32
+ // ── Search exclusions (speeds up Ctrl+Shift+F) ───────────────────────────
33
+ "search.exclude": {
34
+ "**/__pycache__": true,
35
+ "**/.venv": true,
36
+ "**/venv": true,
37
+ "**/node_modules": true,
38
+ "**/dist": true,
39
+ "**/build": true,
40
+ "**/cuad_data": true,
41
+ "**/full_contract_pdf": true,
42
+ "**/*.pdf": true,
43
+ "**/*.PDF": true,
44
+ "**/*.json": false
45
+ },
46
+
47
+ // ── Python analysis β€” limit what Pylance indexes ─────────────────────────
48
+ "python.analysis.exclude": [
49
+ "**/cuad_data/**",
50
+ "**/full_contract_pdf/**",
51
+ "**/__pycache__/**",
52
+ "**/.venv/**",
53
+ "**/venv/**"
54
+ ],
55
+ "python.analysis.indexing": true,
56
+ "python.analysis.packageIndexDepths": [
57
+ { "name": "", "depth": 2 }
58
+ ]
59
+ }
cuad-demo-quadrant/CHUNKING_STRATEGY.md ADDED
@@ -0,0 +1,224 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Text Chunking Strategy for CUAD Contract Embeddings
2
+
3
+ ## Overview
4
+
5
+ The CUAD PDF ingestion pipeline uses a **smart hierarchical boundary-aware chunking strategy** to split contract text into semantically meaningful chunks before embedding and uploading to Qdrant vector database.
6
+
7
+ ## Chunking Strategy
8
+
9
+ ### Core Algorithm: `split_text_with_offsets()`
10
+
11
+ Located in `upload_to_qdrant.py`, the chunking function implements a **sliding window with intelligent boundary detection**:
12
+
13
+ ```python
14
+ def split_text_with_offsets(text: str, chunk_size: int, chunk_overlap: int) -> list[dict]:
15
+ separators = ["\n\n", "\n", " ", ""]
16
+ # ... splits at best available boundary
17
+ ```
18
+
19
+ ### Strategy Details
20
+
21
+ #### 1. **Hierarchical Boundary Detection**
22
+ The algorithm attempts to break text at natural boundaries in this priority order:
23
+ - **Paragraph boundaries** (`"\n\n"`) β€” highest priority
24
+ - **Line boundaries** (`"\n"`)
25
+ - **Word boundaries** (` ` space)
26
+ - **Character boundaries** (`""`) β€” fallback only
27
+
28
+ This ensures chunks respect document structure and semantic coherence.
29
+
30
+ #### 2. **How It Works**
31
+
32
+ ```
33
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
34
+ β”‚ Full Contract Text β”‚
35
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
36
+ ↓
37
+ [Start at offset 0]
38
+ ↓
39
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
40
+ β”‚ Try to fit chunk_size (500 chars) of text β”‚
41
+ β”‚ Then search backwards for nearest separator β”‚
42
+ β”‚ [next_chunk_boundary] ← found at \n\n β”‚
43
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
44
+ ↓
45
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
46
+ β”‚ Yield Chunk 1: { text, char_start, char_end } β”‚
47
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
48
+ ↓
49
+ [Advance by (end - overlap)]
50
+ [Move to next chunk with 50-char overlap]
51
+ ↓
52
+ [Repeat until text exhausted]
53
+ ```
54
+
55
+ #### 3. **Overlap for Context Preservation**
56
+
57
+ - Default overlap: **50 characters**
58
+ - Ensures semantic continuity between chunks
59
+ - Important for vector search β€” similar content appears in adjacent chunks
60
+ - Example:
61
+ ```
62
+ Chunk 1 ends: "...payment shall be made within 30 days."
63
+ Chunk 2 starts: "...within 30 days. Any late payment will incur..."
64
+ ↑ overlapping context from Chunk 1
65
+ ```
66
+
67
+ ### Configuration Parameters
68
+
69
+ | Parameter | Default | Purpose |
70
+ |-----------|---------|---------|
71
+ | `CHUNK_SIZE` | 500 chars | Target size per chunk |
72
+ | `CHUNK_OVERLAP` | 50 chars | Context overlap between chunks |
73
+ | `MAX_DOCS` | 1000 chunks | Total chunks to upload |
74
+ | `ENCODE_BATCH_SIZE` | 32 | Chunks per embedding batch |
75
+ | `UPLOAD_BATCH_SIZE` | 100 | Chunks per Qdrant upsert |
76
+
77
+ ## Rationale for This Strategy
78
+
79
+ ### 1. **Semantic Preservation**
80
+ - Contracts are highly structured documents with clear logical boundaries (clauses, sections, subsections)
81
+ - Breaking at paragraph/line boundaries keeps related sentences together
82
+ - Avoids splitting mid-sentence or mid-clause, which would lose meaning
83
+
84
+ ### 2. **RAG Optimization**
85
+ - For retrieval-augmented generation (RAG), chunks should be semantically cohesive
86
+ - When a chunk is retrieved as context, the model expects complete thoughts
87
+ - Boundary-aware chunking improves relevance of retrieved context
88
+
89
+ ### 3. **Legal Document Specificity**
90
+ - Contract clauses often span multiple lines
91
+ - Paragraph breaks typically separate distinct legal obligations or definitions
92
+ - Natural language models (embedding models) perform better on semantically complete units
93
+
94
+ ### 4. **Overlap for Robustness**
95
+ - Query might match content spanning two chunks
96
+ - Overlap ensures retrieval captures complete relevant context
97
+ - Trade-off: 50 chars is minimal overhead (10% of 500), benefits retrieval accuracy significantly
98
+
99
+ ### 5. **Fallback Handling**
100
+ - If a 500-char window has no natural boundaries within the middle 50% (chunk_size // 2 = 250 chars), the algorithm falls back to character-level splitting
101
+ - This prevents infinite loops on text without natural boundaries (e.g., URLs, base64, code blocks)
102
+ - Rare in contract text but handles edge cases gracefully
103
+
104
+ ## Implementation Flow
105
+
106
+ ```
107
+ β”Œβ”€ Extract Text from PDF (all pages) ─────────────────┐
108
+ β”‚ β”‚
109
+ β”œβ”€ Concatenate pages with \n\n separator β”‚
110
+ β”‚ β”‚
111
+ β”œβ”€ Build page_map: map character offsets β†’ page # β”‚
112
+ β”‚ β”‚
113
+ β”œβ”€ For each PDF: β”‚
114
+ β”‚ β”œβ”€ Split full text into chunks using boundaries β”‚
115
+ β”‚ β”‚ β”‚
116
+ β”‚ └─ For each chunk: β”‚
117
+ β”‚ β”œβ”€ Calculate which pages it spans β”‚
118
+ β”‚ β”œβ”€ Store metadata: β”‚
119
+ β”‚ β”‚ - doc_id, title, text β”‚
120
+ β”‚ β”‚ - char_start/end (for PDF highlighting) β”‚
121
+ β”‚ β”‚ - page_start/end (for citation) β”‚
122
+ β”‚ β”‚ - pdf_path (for traceability) β”‚
123
+ β”‚ β”‚ β”‚
124
+ β”‚ └─ Yield for embedding β”‚
125
+ β”‚ β”‚
126
+ └─ Batch encode embeddings & upsert to Qdrant β”€β”€β”€β”€β”€β”€β”€β”€β”˜
127
+ ```
128
+
129
+ ## Metadata Attached to Each Chunk
130
+
131
+ ```python
132
+ {
133
+ "doc_id": "contract-name-chunk-42", # Unique ID
134
+ "title": "contract-name", # PDF filename stem
135
+ "text": "...", # Actual text (500 chars)
136
+ "char_start": 1250, # Start offset in full text
137
+ "char_end": 1750, # End offset in full text
138
+ "page_start": 3, # Page(s) this chunk spans
139
+ "page_end": 4,
140
+ "pdf_path": "path/to/contract.pdf", # Relative path for retrieval
141
+ }
142
+ ```
143
+
144
+ This metadata enables:
145
+ - **Exact citation**: Link search results directly to PDF pages
146
+ - **Traceability**: Know which contract and location each embedding came from
147
+ - **Transparency**: Show users the source text chunk alongside similarity scores
148
+
149
+ ## Performance Characteristics
150
+
151
+ | Metric | Value | Notes |
152
+ |--------|-------|-------|
153
+ | Chunk size | ~500 chars | β‰ˆ 150–200 tokens for language models |
154
+ | Overlap | 50 chars | β‰ˆ 15–20 tokens |
155
+ | Effective unique content per chunk | ~450 chars | (500 - 50 overlap) |
156
+ | Max chunks per contract | Varies | Depends on contract length |
157
+ | Processing batches | 100-chunk upserts | Balances memory vs. network round-trips |
158
+
159
+ ## Why NOT Other Strategies?
160
+
161
+ ### ❌ Fixed Character-only Splitting
162
+ - Would break contracts mid-clause
163
+ - "...party shall" | "...indemnify" (semantically broken)
164
+ - Poor retrieval quality
165
+
166
+ ### ❌ Sentence-based Splitting
167
+ - Contract "sentences" can be very long (100+ tokens)
168
+ - Would create inconsistent chunk sizes (too large for some)
169
+ - "Sentence" tokenization unreliable in legal text (e.g., "U.S." is not end of sentence)
170
+
171
+ ### ❌ No Overlap
172
+ - Query might start at the end of one chunk and continue into next
173
+ - Without overlap, key context would be missing from retrieved chunk
174
+ - Significantly degrades RAG quality
175
+
176
+ ### βœ… This Strategy: Hierarchical Boundaries with Overlap
177
+ - Respects document structure (legal precision)
178
+ - Consistent encoding/retrieval (same chunk sizes across collection)
179
+ - Robust against edge cases (fallback to character level)
180
+ - Context-aware retrieval (overlap ensures complete thoughts)
181
+
182
+ ## Usage Example
183
+
184
+ ```bash
185
+ # Set environment variables
186
+ export QDRANT_API_KEY="your_api_key"
187
+ export CLUSTER_URL="https://your-qdrant-cluster"
188
+ export MAX_DOCS=5000 # Upload 5000 chunks (instead of default 1000)
189
+ export CHUNK_SIZE=800 # Larger chunks (instead of default 500)
190
+ export CHUNK_OVERLAP=100 # More overlap (instead of default 50)
191
+
192
+ # Run ingestion
193
+ python upload_to_qdrant.py
194
+ ```
195
+
196
+ ## Monitoring & Tuning
197
+
198
+ When running uploads, watch for:
199
+
200
+ ```
201
+ [DEBUG] {title}: {N} pages, {M:,} chars
202
+ ```
203
+
204
+ **If chunks seem too large:**
205
+ - Reduce `CHUNK_SIZE` (e.g., 300–400) for more granular retrieval
206
+ - Increases total chunks and computation time
207
+
208
+ **If chunks seem too small:**
209
+ - Increase `CHUNK_SIZE` (e.g., 700–1000) for broader context
210
+ - Reduces total chunks but may lose fine-grained retrieval
211
+
212
+ **If retrieval is imprecise:**
213
+ - Increase `CHUNK_OVERLAP` to 100–150 for richer context overlap
214
+ - Improves recall at small storage cost
215
+
216
+ ## Summary
217
+
218
+ This chunking strategy balances:
219
+ - **Semantic coherence** (hierarchical boundaries)
220
+ - **Retrieval accuracy** (overlap for context)
221
+ - **Practical efficiency** (reasonable batch sizes)
222
+ - **Legal precision** (metadata for exact citations)
223
+
224
+ The result is chunks that embeddings models can meaningfully represent, and that users can trace back to exact contract locations.
cuad-demo-quadrant/upload_to_qdrant.py ADDED
@@ -0,0 +1,350 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # %% Imports & docstring
2
+ """
3
+ upload_to_qdrant.py
4
+ ───────────────────
5
+ Extracts text from CUAD PDF contracts, generates embeddings with
6
+ all-MiniLM-L6-v2, and uploads up to 1 000 chunks to a Qdrant Cloud
7
+ collection named 'cuad_contracts'.
8
+
9
+ Reuses the PDF extraction and chunking logic from:
10
+ project1/open_search_tuning/cuad_opensearch/notebooks/extract_index_cuad_pdfs.py
11
+
12
+ Usage
13
+ ─────
14
+ python upload_to_qdrant.py
15
+
16
+ Environment (loaded from .env in this directory)
17
+ ─────────────────────────────────────────────────
18
+ QDRANT_API_KEY – Qdrant Cloud API key
19
+ CLUSTER_URL – Qdrant Cloud cluster URL (without port)
20
+
21
+ Optional env vars
22
+ ─────────────────
23
+ MAX_DOCS – max chunks to upload (default: 1000)
24
+ CHUNK_SIZE – characters per chunk (default: 500)
25
+ CHUNK_OVERLAP – overlap between chunks (default: 50)
26
+ ENCODE_BATCH_SIZE– embedding batch size (default: 32)
27
+ UPLOAD_BATCH_SIZE– upsert batch size (default: 100)
28
+ """
29
+
30
+ import os
31
+ import sys
32
+ import uuid
33
+ from pathlib import Path
34
+
35
+ from dotenv import load_dotenv
36
+ from sentence_transformers import SentenceTransformer
37
+ from qdrant_client import QdrantClient
38
+ from qdrant_client.models import Distance, VectorParams, PointStruct
39
+ from tqdm import tqdm
40
+
41
+ print("[INFO] Imports loaded successfully.")
42
+
43
+ # %% Configuration β€” load .env and set constants
44
+ load_dotenv(Path(__file__).resolve().parent / ".env")
45
+
46
+ QDRANT_API_KEY = os.environ["QDRANT_API_KEY"]
47
+ CLUSTER_URL = os.environ["CLUSTER_URL"].strip()
48
+
49
+ COLLECTION_NAME = "cuad_contracts"
50
+ VECTOR_SIZE = 384 # all-MiniLM-L6-v2
51
+
52
+ MAX_DOCS = int(os.getenv("MAX_DOCS", "1000"))
53
+ CHUNK_SIZE = int(os.getenv("CHUNK_SIZE", "500"))
54
+ CHUNK_OVERLAP = int(os.getenv("CHUNK_OVERLAP", "50"))
55
+ ENCODE_BATCH_SIZE = int(os.getenv("ENCODE_BATCH_SIZE", "32"))
56
+ UPLOAD_BATCH_SIZE = int(os.getenv("UPLOAD_BATCH_SIZE", "100"))
57
+
58
+ PDF_ROOT = Path(
59
+ "/home/ridhi/projects/project1/open_search_tuning"
60
+ "/cuad_opensearch/cuad_data/CUAD_v1/full_contract_pdf"
61
+ )
62
+
63
+ print(f"[INFO] Config loaded:")
64
+ print(f" CLUSTER_URL = {CLUSTER_URL}")
65
+ print(f" COLLECTION_NAME = {COLLECTION_NAME}")
66
+ print(f" VECTOR_SIZE = {VECTOR_SIZE}")
67
+ print(f" MAX_DOCS = {MAX_DOCS}")
68
+ print(f" CHUNK_SIZE = {CHUNK_SIZE} | CHUNK_OVERLAP = {CHUNK_OVERLAP}")
69
+ print(f" ENCODE_BATCH_SIZE= {ENCODE_BATCH_SIZE} | UPLOAD_BATCH_SIZE = {UPLOAD_BATCH_SIZE}")
70
+ print(f" PDF_ROOT exists = {PDF_ROOT.exists()}")
71
+
72
+
73
+ # %% PDF extraction backend β€” auto-selects best available library
74
+ def _make_extractor():
75
+ try:
76
+ import fitz
77
+
78
+ def _extract(path: Path):
79
+ pages = []
80
+ with fitz.open(str(path)) as doc:
81
+ for i, page in enumerate(doc):
82
+ pages.append({"page": i + 1, "text": page.get_text("text") or ""})
83
+ return pages
84
+
85
+ print("[DEBUG] PDF backend selected: pymupdf (fitz)")
86
+ return "pymupdf", _extract
87
+ except ImportError:
88
+ print("[DEBUG] pymupdf not available, trying pdfplumber …")
89
+
90
+ try:
91
+ import pdfplumber
92
+
93
+ def _extract(path: Path):
94
+ pages = []
95
+ with pdfplumber.open(str(path)) as pdf:
96
+ for i, page in enumerate(pdf.pages):
97
+ pages.append({"page": i + 1, "text": page.extract_text() or ""})
98
+ return pages
99
+
100
+ print("[DEBUG] PDF backend selected: pdfplumber")
101
+ return "pdfplumber", _extract
102
+ except ImportError:
103
+ print("[DEBUG] pdfplumber not available, trying pdfminer …")
104
+
105
+ try:
106
+ from pdfminer.high_level import extract_pages
107
+ from pdfminer.layout import LTTextContainer
108
+
109
+ def _extract(path: Path):
110
+ pages = []
111
+ for i, layout in enumerate(extract_pages(str(path))):
112
+ text = "".join(
113
+ el.get_text()
114
+ for el in layout
115
+ if isinstance(el, LTTextContainer)
116
+ )
117
+ pages.append({"page": i + 1, "text": text})
118
+ return pages
119
+
120
+ print("[DEBUG] PDF backend selected: pdfminer")
121
+ return "pdfminer", _extract
122
+ except ImportError:
123
+ pass
124
+
125
+ raise RuntimeError("No PDF library found. Install pymupdf, pdfplumber, or pdfminer.six.")
126
+
127
+
128
+ EXTRACTOR_NAME, extract_pages_from_pdf = _make_extractor()
129
+ print(f"[INFO] PDF extractor : {EXTRACTOR_NAME}")
130
+
131
+
132
+ # %% Text chunking helpers
133
+ def split_text_with_offsets(text: str, chunk_size: int, chunk_overlap: int) -> list[dict]:
134
+ separators = ["\n\n", "\n", " ", ""]
135
+ chunks: list[dict] = []
136
+ start = 0
137
+ while start < len(text):
138
+ end = min(start + chunk_size, len(text))
139
+ chunk_text = text[start:end]
140
+ if end < len(text):
141
+ split_at = -1
142
+ for sep in separators:
143
+ idx = chunk_text.rfind(sep)
144
+ if idx > chunk_size // 2:
145
+ split_at = idx + len(sep)
146
+ break
147
+ if split_at > 0:
148
+ end = start + split_at
149
+ chunk_text = text[start:end]
150
+ chunks.append({"text": chunk_text, "char_start": start, "char_end": end})
151
+ if end >= len(text):
152
+ break
153
+ next_start = end - chunk_overlap
154
+ if next_start <= start:
155
+ next_start = start + 1
156
+ start = next_start
157
+ return chunks
158
+
159
+
160
+ # ── Page-boundary helpers ────────────────────────────────────────────────────
161
+ def build_page_map(pages: list[dict]) -> list[tuple[int, int, int]]:
162
+ page_map = []
163
+ pos = 0
164
+ for p in pages:
165
+ end = pos + len(p["text"])
166
+ page_map.append((pos, end, p["page"]))
167
+ pos = end + 2
168
+ return page_map
169
+
170
+
171
+ def char_range_to_pages(char_start: int, char_end: int, page_map: list) -> tuple[int, int]:
172
+ first = last = None
173
+ for seg_start, seg_end, pg in page_map:
174
+ if char_end <= seg_start:
175
+ break
176
+ if char_start < seg_end:
177
+ if first is None:
178
+ first = pg
179
+ last = pg
180
+ return (first or 1, last or 1)
181
+
182
+
183
+ # %% PDF discovery & chunk iterator
184
+ def find_pdfs(root: Path) -> list[Path]:
185
+ return sorted(p for p in root.rglob("*") if p.is_file() and p.suffix.upper() == ".PDF")
186
+
187
+
188
+ def iter_chunks(all_pdfs: list[Path], limit: int):
189
+ """Yield chunk dicts until *limit* chunks have been emitted."""
190
+ count = 0
191
+ skipped_pdfs = 0
192
+ for pdf_path in all_pdfs:
193
+ if count >= limit:
194
+ return
195
+ title = pdf_path.stem
196
+ print(f"[DEBUG] Extracting: {pdf_path.name}", end="\r")
197
+ try:
198
+ pages = extract_pages_from_pdf(pdf_path)
199
+ except Exception as exc:
200
+ print(f"\n[WARN] Could not extract {pdf_path.name}: {exc}")
201
+ skipped_pdfs += 1
202
+ continue
203
+
204
+ pages = [p for p in pages if p["text"].strip()]
205
+ if not pages:
206
+ print(f"[WARN] No text extracted from {pdf_path.name} β€” skipping")
207
+ skipped_pdfs += 1
208
+ continue
209
+
210
+ full_text = "\n\n".join(p["text"] for p in pages)
211
+ page_map = build_page_map(pages)
212
+ rel_path = str(pdf_path.relative_to(PDF_ROOT))
213
+ print(f"[DEBUG] {title}: {len(pages)} pages, {len(full_text):,} chars", end="\r")
214
+
215
+ for chunk_idx, chunk in enumerate(
216
+ split_text_with_offsets(full_text, CHUNK_SIZE, CHUNK_OVERLAP)
217
+ ):
218
+ if count >= limit:
219
+ return
220
+ pg_start, pg_end = char_range_to_pages(
221
+ chunk["char_start"], chunk["char_end"], page_map
222
+ )
223
+ yield {
224
+ "doc_id": f"{title}-chunk-{chunk_idx}",
225
+ "title": title,
226
+ "text": chunk["text"],
227
+ "char_start": chunk["char_start"],
228
+ "char_end": chunk["char_end"],
229
+ "page_start": pg_start,
230
+ "page_end": pg_end,
231
+ "pdf_path": rel_path,
232
+ }
233
+ count += 1
234
+
235
+ if skipped_pdfs:
236
+ print(f"\n[INFO] PDFs skipped due to extraction errors: {skipped_pdfs}")
237
+
238
+
239
+ # %% Connect to Qdrant and create collection if needed
240
+ print(f"[INFO] Connecting to Qdrant: {CLUSTER_URL}")
241
+ qdrant = QdrantClient(
242
+ url=f"{CLUSTER_URL}:6333",
243
+ api_key=QDRANT_API_KEY,
244
+ timeout=30, # seconds β€” avoids silent hangs if cluster is unreachable
245
+ prefer_grpc=False,
246
+ )
247
+ print("[INFO] Qdrant client created.")
248
+
249
+ existing = [c.name for c in qdrant.get_collections().collections]
250
+ print(f"[DEBUG] Existing collections: {existing}")
251
+ if COLLECTION_NAME in existing:
252
+ print(f"[INFO] Collection '{COLLECTION_NAME}' already exists β€” skipping creation.")
253
+ else:
254
+ qdrant.create_collection(
255
+ collection_name=COLLECTION_NAME,
256
+ vectors_config=VectorParams(size=VECTOR_SIZE, distance=Distance.COSINE),
257
+ )
258
+ print(f"[INFO] Collection '{COLLECTION_NAME}' created (dim={VECTOR_SIZE}, distance=Cosine).")
259
+
260
+
261
+ # %% Load embedding model
262
+ # NOTE: first run downloads ~90 MB from HuggingFace β€” this can take a minute.
263
+ # Subsequent runs load from the local cache and are fast.
264
+ print("[INFO] Loading embedding model all-MiniLM-L6-v2 (downloading if not cached) …")
265
+ model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
266
+ print(f"[INFO] Model loaded. Max sequence length: {model.max_seq_length} tokens.")
267
+
268
+
269
+ # %% Discover PDFs
270
+ all_pdfs = find_pdfs(PDF_ROOT)
271
+ print(f"[INFO] PDFs discovered : {len(all_pdfs)}")
272
+ print(f"[INFO] Target chunks : {MAX_DOCS} | chunk_size={CHUNK_SIZE} overlap={CHUNK_OVERLAP}")
273
+ if not all_pdfs:
274
+ raise FileNotFoundError(f"No PDFs found under {PDF_ROOT}. Check PDF_ROOT path.")
275
+
276
+ # %% Encode & upload in batches
277
+ chunk_buffer: list[dict] = []
278
+ uploaded = 0
279
+ errors = 0
280
+
281
+ def flush_buffer(buf: list[dict]) -> int:
282
+ """Encode and upsert a batch; returns number of points uploaded."""
283
+ texts = [d["text"] for d in buf]
284
+ print(f"[DEBUG] Encoding batch of {len(texts)} chunks …", end="\r")
285
+ embeddings = model.encode(
286
+ texts,
287
+ batch_size=ENCODE_BATCH_SIZE,
288
+ show_progress_bar=False,
289
+ normalize_embeddings=True,
290
+ )
291
+ print(f"[DEBUG] Embeddings shape: {embeddings.shape} β€” upserting …", end="\r")
292
+ points = [
293
+ PointStruct(
294
+ id=str(uuid.uuid5(uuid.NAMESPACE_DNS, d["doc_id"])),
295
+ vector=embeddings[i].tolist(),
296
+ payload={
297
+ "doc_id": d["doc_id"],
298
+ "title": d["title"],
299
+ "text": d["text"],
300
+ "char_start": d["char_start"],
301
+ "char_end": d["char_end"],
302
+ "page_start": d["page_start"],
303
+ "page_end": d["page_end"],
304
+ "pdf_path": d["pdf_path"],
305
+ },
306
+ )
307
+ for i, d in enumerate(buf)
308
+ ]
309
+ qdrant.upsert(collection_name=COLLECTION_NAME, points=points)
310
+ return len(points)
311
+
312
+
313
+ with tqdm(total=MAX_DOCS, desc="Uploading chunks", unit="chunk") as pbar:
314
+ for chunk in iter_chunks(all_pdfs, MAX_DOCS):
315
+ chunk_buffer.append(chunk)
316
+ if len(chunk_buffer) >= UPLOAD_BATCH_SIZE:
317
+ try:
318
+ n = flush_buffer(chunk_buffer)
319
+ uploaded += n
320
+ pbar.update(n)
321
+ print(f"[INFO] Batch upserted β€” total uploaded so far: {uploaded}", end="\r")
322
+ except Exception as exc:
323
+ print(f"\n[ERROR] Upsert failed: {exc}")
324
+ errors += 1
325
+ chunk_buffer.clear()
326
+
327
+ # Flush remainder
328
+ if chunk_buffer:
329
+ print(f"\n[INFO] Flushing final batch of {len(chunk_buffer)} chunks …")
330
+ try:
331
+ n = flush_buffer(chunk_buffer)
332
+ uploaded += n
333
+ pbar.update(n)
334
+ except Exception as exc:
335
+ print(f"\n[ERROR] Final upsert failed: {exc}")
336
+ errors += 1
337
+ chunk_buffer.clear()
338
+
339
+
340
+ # %% Summary
341
+ info = qdrant.get_collection(COLLECTION_NAME)
342
+ print("\n====== UPLOAD COMPLETE ======")
343
+ print(f"[INFO] Collection : {COLLECTION_NAME}")
344
+ print(f"[INFO] Chunks uploaded : {uploaded}")
345
+ print(f"[INFO] Errors : {errors}")
346
+ print(f"[INFO] Vectors in coll.: {info.points_count}")
347
+ if errors:
348
+ print(f"[WARN] {errors} batch(es) failed β€” check [ERROR] lines above.")
349
+
350
+ # %%
tests/verify_collection.py ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+ """
3
+ verify_collection.py
4
+ ────────────────────
5
+ Test script to verify that documents are properly uploaded to the Qdrant collection.
6
+
7
+ Connects to the Qdrant cluster and retrieves collection stats and sample documents.
8
+
9
+ Usage
10
+ ─────
11
+ python tests/verify_collection.py
12
+ """
13
+
14
+ import os
15
+ import sys
16
+ from pathlib import Path
17
+
18
+ from dotenv import load_dotenv
19
+ from qdrant_client import QdrantClient
20
+
21
+ # Load environment variables from parent directory's .env
22
+ env_path = Path(__file__).resolve().parent.parent / "cuad-demo-quadrant" / ".env"
23
+ load_dotenv(env_path)
24
+
25
+ # Configuration
26
+ QDRANT_API_KEY = os.environ.get("QDRANT_API_KEY")
27
+ CLUSTER_URL = os.environ.get("CLUSTER_URL", "").strip()
28
+ COLLECTION_NAME = "cuad_contracts"
29
+
30
+ if not QDRANT_API_KEY or not CLUSTER_URL:
31
+ print("[ERROR] Missing QDRANT_API_KEY or CLUSTER_URL in .env")
32
+ sys.exit(1)
33
+
34
+ print(f"[INFO] Connecting to Qdrant cluster: {CLUSTER_URL}")
35
+
36
+ try:
37
+ qdrant = QdrantClient(
38
+ url=f"{CLUSTER_URL}:6333",
39
+ api_key=QDRANT_API_KEY,
40
+ timeout=30,
41
+ prefer_grpc=False,
42
+ )
43
+ print("[βœ“] Connected to Qdrant cluster successfully.")
44
+ except Exception as e:
45
+ print(f"[ERROR] Failed to connect: {e}")
46
+ sys.exit(1)
47
+
48
+ # Get collection info
49
+ print(f"\n[INFO] Retrieving collection '{COLLECTION_NAME}' info...")
50
+ try:
51
+ info = qdrant.get_collection(COLLECTION_NAME)
52
+ print(f"[βœ“] Collection found.")
53
+ print(f" Points count : {info.points_count}")
54
+ print(f" Vector size : {info.config.vectors_config.size if hasattr(info.config, 'vectors_config') else 'N/A'}")
55
+ except Exception as e:
56
+ print(f"[ERROR] Failed to get collection info: {e}")
57
+ sys.exit(1)
58
+
59
+ # List all collections
60
+ print(f"\n[INFO] Available collections:")
61
+ try:
62
+ collections = qdrant.get_collections().collections
63
+ for coll in collections:
64
+ print(f" - {coll.name}")
65
+ except Exception as e:
66
+ print(f"[ERROR] Failed to list collections: {e}")
67
+
68
+ # Retrieve and display sample documents
69
+ print(f"\n[INFO] Retrieving sample documents...")
70
+ try:
71
+ # Scroll through the collection to get sample points
72
+ points, next_page_offset = qdrant.scroll(
73
+ collection_name=COLLECTION_NAME,
74
+ limit=5,
75
+ with_payload=True,
76
+ with_vectors=False,
77
+ )
78
+
79
+ if not points:
80
+ print("[WARN] No points found in collection!")
81
+ else:
82
+ print(f"[βœ“] Retrieved {len(points)} sample documents:\n")
83
+ for i, point in enumerate(points, 1):
84
+ payload = point.payload
85
+ print(f" [{i}] ID: {point.id}")
86
+ print(f" Doc ID : {payload.get('doc_id', 'N/A')}")
87
+ print(f" Title : {payload.get('title', 'N/A')}")
88
+ print(f" PDF Path : {payload.get('pdf_path', 'N/A')}")
89
+ print(f" Pages : {payload.get('page_start', 'N/A')}-{payload.get('page_end', 'N/A')}")
90
+ text_preview = payload.get('text', '')[:100].replace('\n', ' ')
91
+ print(f" Text : {text_preview}...")
92
+ print()
93
+ except Exception as e:
94
+ print(f"[ERROR] Failed to retrieve documents: {e}")
95
+ sys.exit(1)
96
+
97
+ print("[βœ“] Verification complete!")
98
+ print(f"\n[INFO] Summary:")
99
+ print(f" Collection: {COLLECTION_NAME}")
100
+ print(f" Total uploaded documents: {info.points_count}")