Searching documents

An agent rarely knows which document it needs; it knows the user’s question. POST /v1/docs/search takes that question in plain words and returns the semantic documents most likely to help write the SQL, best first. Any authenticated user may search.

curl -u analyst:analyst-pw localhost:8080/v1/docs/search \
  -d '{"q": "monthly revenue per customer", "limit": 3}'

{"ranker": "jev",
 "results": [{"score": 0.93, "judged": true,
              "doc": {"id": "...", "header": {...}, "context": {...}}}]}

New in v0.5.0

Search needs curral v0.5.0 or later.

Request

Field
q	the question, required
table, schema	narrow the candidates, matched as in the listing
limit	results to return; default 5, at most 50
fields	full (default) or header, which leaves out body and sqls

Three layers

question ──▶ filters      table= / schema=, exact name or its last parts
             ├─ BM25      words of the question, weighted by field
             └─ Jev       (with --jev-key) would the SQL use this document?

1. Filters

table and schema keep only documents about those names, before any ranking.

2. BM25

BM25 scores each document by the words it shares with the question. Matching ignores case and accents, splits identifiers on _ and . (so monthly_revenue matches “monthly revenue”) and drops a plural s. Where a word appears matters:

Field	Weight
title	3
used_for	2.5
tables, schemas	2
description	1.5
body, sqls	1

Without a Jev key this is the whole search: the response says "ranker": "bm25", documents sharing no word with the question are left out, and score is useful only for ordering. It runs in memory and needs no network, so search works offline.

3. Jev

Shared words are a weak signal. “Employee turnover” and a document about job openings share a topic but no data; “cancelled sales” needs a document that says cancelled orders are excluded, so the query can invert that rule.

With --jev-key, the best BM25 candidates (--search-shortlist, 20, and at least limit) are each sent to TypeSafe’s Jev model with the question. Jev answers one question per document: would the SQL that answers this question use data or rules described in this document?, and returns a probability.

• Matches: documents at or above --jev-threshold (0.7) are returned, ranked by that probability, with "judged": true.
• Weak matches: if none reaches the threshold, the judged ones at or above --jev-floor (0.2) come back with "below_threshold": true on the response. A partial fit (around 0.4) shows up here.
• No match: below the floor documents are left out, so a question no document fits gets an empty list instead of filling the agent’s context with unrelated notes.
• Not judged: a document Jev could not rate within --jev-timeout (5s) keeps its BM25 place after the judged ones, with "judged": false. The search never fails because Jev did.
• Other words, other languages: Jev understands that the question and the document may use different words, or different languages, for the same thing. When a filtered set has no more documents than the shortlist, all of them reach Jev, even those that share no word with the question.

export CURRAL_JEV_KEY=...        # prefer the environment over --jev-key
curral serve ... --docs-dir /var/lib/curral/docs

Data leaves your network

With a Jev key, the question and each candidate document (title, description, used_for, tables, schemas, the body up to 4,000 characters and each SQL up to 500) are sent to TypeSafe’s API. Query results and table data are never sent. Leave the key unset to keep search fully local with BM25.

Reading the results

Response	Meaning
"ranker": "jev", "judged": true	Jev judged the document needed; score is its probability
"below_threshold": true	nothing reached the threshold: treat the results as weak matches
"judged": false	not judged; score is BM25, so check the document is really about the question
"ranker": "bm25"	no Jev key: every result is ranked by shared words only
"results": []	no document fits; write the SQL from /v1/schema alone

BM25 and Jev scores are not comparable with each other.

Tuning

Flag	Default
--jev-key	off	TypeSafe API key; prefer CURRAL_JEV_KEY
--jev-threshold	0.7	minimum probability for a match, above 0 and at most 1
--jev-floor	0.2	minimum for a weak match, below the threshold; 0 returns the best whatever their probability
--jev-timeout	5s	how long a search waits for Jev
--search-shortlist	20	best BM25 candidates handed to Jev

The defaults come from a labeled evaluation in the curral repository (internal/rerank/testdata/eval.json): questions, the documents that must match and those that must not. To check them against your own documents (as GET /v1/docs/{id} returns them), write a file in the same format, keep it out of the repository, and run from a curral checkout:

CURRAL_JEV_KEY=... CURRAL_JEV_EVAL=$PWD/my-cases.json \
  go test ./internal/rerank -run TestJevEval -v

The log shows the probability Jev gives each document, so a threshold change can be compared with the previous run.

A larger shortlist lets Jev rescue documents BM25 ranked low, at the cost of more calls per search; they run 8 at a time within the timeout.