The short version: vector search finds passages that are semantically close to a question rather than ones sharing its keywords, which is why it answers “how do I stop being charged” with a document titled “Cancelling your subscription”. It is not a replacement for keyword search (it is worse at exact identifiers), and the quality of what it returns depends far more on how you chunked your source documents than on which model produced the embeddings.
The mechanism, briefly
Text is converted into a vector — a list of numbers positioning it in a space where similar meanings sit near each other. Your documents are chunked, each chunk embedded, and the vectors stored. At query time the question is embedded the same way and the nearest chunks are returned.
Two consequences follow immediately, and they explain nearly every surprise:
- Nothing is matched literally. There is no guarantee a returned chunk contains any word from the question.
- The chunk is the unit. Retrieval can only return what chunking created. A fact split across two chunks may be findable in neither.
Chunking is where the quality is
This is the part teams underinvest in and then blame the model for.
Too small and chunks lose the context that makes them meaningful — a paragraph that says “this does not apply to enterprise customers” is dangerous without the section it qualifies.
Too large and each chunk covers several topics, so its vector sits in the average of all of them and matches none of them well.
Split on structure, not on character count. Headings, sections, list boundaries. A fixed-size split through the middle of a table produces chunks that are actively misleading.
Carry the context down. Prefixing each chunk with its document title and heading path costs a few tokens and meaningfully improves both retrieval and the model’s ability to use what comes back.
Overlap a little so a fact near a boundary appears whole in at least one chunk.
If retrieval quality is poor, look here before changing anything else. In my experience chunking accounts for most of the difference between a retrieval system that works and one that does not.
Where vector search is worse than keyword search
| Query type | Better served by |
|---|---|
| “What is the refund policy?” | Vector |
| “Case 00123456” | Keyword or a direct lookup |
| “Error code ERR-4521” | Keyword |
| “Why is my invoice wrong?” | Vector |
| “Policy documents updated this month” | Structured filter |
Exact identifiers are the clear weakness. An error code is a token with no semantic neighbourhood, so nearest-neighbour search on it returns things that look vaguely technical. Users type error codes constantly.
The practical answer is hybrid retrieval: run both, and combine. Most production systems that work well are not purely vector-based, and treating the choice as either/or is a common early mistake.
Filtering before searching
Semantic similarity does not know about permissions, dates, or language. A chunk from a document the user may not see is just as close in vector space as one they may.
So metadata filters have to be applied as part of retrieval, not after it — filtering afterwards means you may retrieve ten chunks, discard nine on permissions, and answer from one. Filter on access, on document type, on effective date, and on language, then search within what remains.
Access control in particular is not optional and is not the model’s job. If the retrieval layer can return a chunk the user should not see, you have an access problem regardless of what the agent does with it.
Evaluating retrieval on its own
Evaluate retrieval separately from generation, or you cannot tell which one is failing.
- Write thirty real questions, taken from actual user queries rather than invented.
- For each, identify by hand which chunks should be returned.
- Measure whether they are, and at what position.
- Only then look at what the model produced from them.
The common finding is that the right chunk was retrieved third and the model used the first. That is a generation or ranking problem, and no amount of re-chunking fixes it, but you cannot know that without separating the two measurements.
Keeping it current
Embeddings are computed from content at a point in time. Change the document and the vector is stale until it is re-embedded, so there has to be a process (triggered on publish, or a scheduled sweep), and a way to remove chunks for deleted documents.
Orphaned chunks are a real failure mode: a policy is withdrawn, the document is deleted, and the agent keeps citing it because nobody removed its vectors.
Frequently asked
Do I need to choose an embedding model?
Where the platform provides one, using it is usually the right default. Model choice matters far less than chunking, and a mismatch between how documents and queries are embedded matters far more. They must use the same model.
How many chunks should I retrieve?
Enough to cover the answer and few enough that the model is not distracted. Three to five is a common starting range. Tune it against your evaluation set rather than a rule.
Is this the same as semantic search in Salesforce?
Salesforce has been adding semantic retrieval capability across its search and grounding surfaces, and the details differ by product and release. Check the current documentation for what is available in your org rather than assuming a general answer applies.
Can I use it for structured data?
Generally you should not. If the answer is “the balance on account X”, that is a query, not a retrieval problem. Embedding structured records to answer questions a query would answer exactly is a common and expensive detour.
Why does it return confidently irrelevant results?
Nearest-neighbour search always returns something. There is no concept of “no good match” unless you impose a similarity threshold. Set one, and handle the below-threshold case explicitly rather than passing weak matches to the model.
Reference: Data Cloud, Einstein Trust Layer and Prompt Builder grounding documentation, Salesforce; general retrieval-augmented generation practice. Product capabilities in this area change quickly. Verify specifics against current documentation.
