Skip to content
Gradmarc Adtech LLP

Case studyIllustrative write-up · no real client

Answers from private documents, with sources shown

An experiment in retrieval-based question answering where every answer can be checked.

Client
None: illustrative
Industry
Internal experiment
Screens from
Paperlane (concept)

01Challenge

The problem, as it was found

Useful answers existed in a set of private documents, but finding them meant searching files by hand, and a plain chatbot could not be trusted to stay faithful to them.

  • Answers were buried across many documents
  • A general-purpose assistant could state things the documents did not say
  • Users needed to see where an answer came from
  • The document set changed over time

Context

This was a prototype to understand what retrieval over private documents does well and where it fails, before recommending it to anyone.

The working rule was that an answer without a visible source was treated as a failure, however fluent it sounded.

02System

How it was put together

A retrieval pipeline in front of a language model. Generation is constrained to what retrieval returns.

L1 · Ask and verify

Interface

  • Question box
  • Answer
  • Source panel

L2 · Find, then answer

Retrieval

  • Passage search
  • Ranking
  • Source tracking

L3 · Constrained generation

Model

  • AI / LLM integration
  • Prompt rules

L4 · Keeps the index current

Ingestion

  • Splitting
  • Embeddings
  • Re-indexing

Path through the system

  1. 1Question asked
  2. 2Passages retrieved
  3. 3Answer drafted from passages
  4. 4Sources shown
  5. 5Answer reviewed

03What was built

The parts, in the order they shipped

  1. 01

    Ingestion

    Documents are split into passages and indexed, and re-indexed when they change.

  2. 02

    Retrieval

    A question finds the most relevant passages before any text is generated.

  3. 03

    Answer with sources

    The assistant answers from the retrieved passages and shows which ones it used.

  4. 04

    Review view

    A way to compare answers with their sources and note where they fell short.

Paperlane question and answer screen: a question about contract renewals, an answer with a table of three suppliers and notice periods, numbered citations and a sources panel with highlighted clauses.
Every answer cites the clause it came from. Concept screen.
Paperlane invoice review: a document list, an invoice page with extracted fields highlighted, and a panel of extracted values with confidence bars, one flagged for review.
Extracted fields are highlighted on the page, with low-confidence values flagged. Concept screen.

04Decisions

The choices that shaped it

  1. Decision 1

    No source, no answer

    Why

    An answer that cannot be traced is not useful in a setting where accuracy matters.

    Trade-off

    The assistant declines more often than a general chatbot.

  2. Decision 2

    Evaluate on real questions

    Why

    Quality can only be judged against the questions people actually ask.

    Trade-off

    It meant collecting examples first, before tuning anything.

  3. Decision 3

    Keep a person in the loop

    Why

    Retrieval narrows the risk but does not remove it.

    Trade-off

    Slower than a fully automated flow, and more trustworthy.

05Technology

What it was built with

Drawn from our confirmed capability stack. Illustrative for this demo.
Backend
Node.jsExpress.jsREST APIs
Database
PostgreSQL
Cloud and DevOps
Docker
AI and automation
AI / LLM IntegrationsRAG

06Outcome

Showed where retrieval-based answering is reliable, and where it still needs a person to check.

  • Every answer links back to the passages behind it
  • Questions outside the documents are declined rather than guessed
  • The experiment informed how we scope retrieval projects

Demo case study. Outcomes are qualitative. No metrics, savings, revenue or user figures are claimed.

Start a projectAI Development & Automation

More case studies

More case studies

Related services

Have something worth building?

Tell us what you want to build. A short brief is enough to start the conversation.

Start a projectAll case studies