Prepare a paper for model input
The document loader connects PDF downloading, text extraction, and chunking. Chunk size, overlap, and the maximum number of chunks can be configured for an experiment.
PROJECT 03 / RESEARCH TOOLS
Paper Chain.
Reading papers, one question at a time.
Experiments in loading research papers, splitting their text, and asking language models questions about selected passages.
- Python
- Transformers
- PDF processing
- Retrieval
PAPERS → PASSAGES → QUESTIONSSCHEMATIC
01 / OVERVIEW
The idea.
Paper Chain explores the pieces of a paper-reading assistant: getting text out of a PDF, splitting it into manageable passages, and passing selected context to a question-answering model. The repository also contains separate experiments in embedding-based document retrieval.
- PDF loading and text chunking
- Questions over selected passages
- Separate semantic-retrieval experiments
02 / IMPLEMENTATION
How it works.
- PDF source
Download a paper and extract its text.
- Text chunks
Split the document into configurable passages.
- Context + question
Send selected passages and a question to the model.
- Answer text
Collect the responses from the selected passages.
This flow follows the Llama-based example. Semantic retrieval is explored in a separate module.
Ask questions with selected context
The Llama-based example passes a question together with explicitly selected text chunks to a local model. It collects the generated responses and joins them as answer text.
Explore retrieval as a separate component
A separate module experiments with sentence-transformer embeddings and similarity ranking. The Q&A example selects its chunks explicitly, so retrieval and answering are presented as distinct experiments.
PROJECT CONTEXT
An exploratory collection of Python scripts for local model and document-processing experiments, with model and configuration dependencies.
03 / CODE
Explore the code.
A few useful places to start in the public repository.