Tatlatat dc8a882f1f fix(rag): use a stable hash for document IDs so dedup survives restarts (#1098) ...

add_document() and add_documents_batch() derive the persistent ChromaDB
document id from Python's built-in hash():

    doc_id = f"doc_{hash(text) % 10**16}"

str hashing is randomized per process (PYTHONHASHSEED is on by default), so
the same document text gets a different doc_id on every restart. The dedup
check right after — self._collection.get(ids=[doc_id]) — therefore misses
on restart, and identical documents are re-embedded and re-added as
duplicates each time the app restarts, bloating the vector store and
skewing retrieval.

Derive the id from a stable hashlib.sha256 of the text via a shared
_generate_doc_id() helper, used by both add paths so they agree.

tests/test_rag_vector_id_stability.py runs _generate_doc_id in subprocesses
under PYTHONHASHSEED=0/1/random and asserts the id is identical across all
of them (and differs for different text). Fails before this change.

2026-06-02 22:42:23 +09:00

1.1 KiB

Raw Blame History

View Raw