Why Can't Traditional Search Engines Find What You Want to Buy?

Before understanding how RAG systems find their "cheat sheets," let's first examine the shortcomings of traditional search technology.

Imagine you run an e-commerce clothing store. A customer wants to find "a coat that keeps me warm in snowy weather." They type into your search bar: "extremely cold-resistant clothing".

If your website uses a traditional search engine (i.e., "keyword matching"), the result would be: "Sorry, no products found." Why? Because in your database, the product is named "Premium Down Extreme Cold Jacket." Traditional databases are extremely rigid—they only recognize exact word matches. Since the six characters "extremely cold-resistant clothing" don't appear in "Premium Down Extreme Cold Jacket," the system assumes these two things are completely unrelated.

But in the human mind, we know "cold-resistant" and "extreme cold" are synonyms, and "clothing" includes "jackets." We rely on an understanding of "semantics."

If we want RAG systems to accurately locate the relevant paragraphs in vast company documents that can answer questions, we can't rely on traditional keyword matching. We must teach computers to "understand semantics." This is where a groundbreaking invention in AI comes into play: Embeddings (Vectorization).

---

What Are Embeddings? The Magic of Turning Text into "Coordinates"

Embeddings might sound abstract, but think of them as playing a "3D positioning game" with words.

Suppose we have three words: "apple," "banana," and "dog." If we ask an AI model to generate embeddings for these words, it will convert them into three sets of "numerical coordinates" placed in a three-dimensional space.

• The coordinates for "apple" might be: [0.9, 0.1, 0.2] (indicating it's very fruity and somewhat sweet)
• The coordinates for "banana" might be: [0.8, 0.2, 0.1] (also a fruit but with a different shape)
• The coordinates for "dog" might be: [0.1, 0.9, 0.9] (very animal-like and can bark)

Here’s where the magic happens! In this mathematical space, the coordinates for "apple" and "banana" are extremely close together, while "dog" is miles away!

This is the power of embeddings. They transform the complex "semantics" humans understand into "numerical coordinates (vectors)" that computers excel at processing. With embeddings, when a customer searches for "extremely cold-resistant clothing," the AI converts it into a set of coordinates and compares them to the coordinates of "Premium Down Extreme Cold Jacket" in the database. It then realizes they are very close in this space! Thus, the AI successfully recommends this jacket to the customer.

This search technique, which focuses on meaning rather than exact wording, is called "Semantic Search." It’s the heart of RAG systems!

---

The Rise of Vector Databases

Since we need to break down thousands of pages of company documents into "hundreds of thousands of coordinate sets (vectors)," traditional relational databases (e.g., PostgreSQL, MySQL) simply can't compute these geometric distances quickly enough.

Thus, a new type of database designed for the AI era has emerged: Vector Databases. The most famous vector databases include:

• Chroma: Lightweight, open-source, and can be installed locally—ideal for RAG development and testing.
• Pinecone: The cloud powerhouse, the top choice for enterprises—extremely fast but not cheap.
• Qdrant / Milvus: Also powerful open-source vector databases.

In our RAG system, these vector databases act as "super librarians." When you convert a question into coordinates and send it to the database, it can find the 5 most semantically relevant paragraphs (closest in coordinate space) from millions in just fractions of a second.

---

Vibe Prompt in Action: Using LangChain to Call Embedding Models

To turn text into coordinates, we need language models (e.g., OpenAI's text-embedding-3-small). With LangChain, this takes just two lines of code!

【Embedding Demo Prompt】 I'm learning about LangChain and OpenAI's embedding technology. Please write a short Python example:

1. Use the OpenAIEmbeddings model from langchain_openai.
2. Generate embeddings (vectorize) for the text: "This is a high-speed cyclone fan."
3. Print the first 5 numbers of the resulting vector so I can see what the "coordinates" look like.
4. Include Chinese comments.

AI-Generated Example Code:

import os
from langchain_openai import OpenAIEmbeddings

# Make sure you've set your OpenAI API Key (in real projects, store it in .env)
os.environ["OPENAI_API_KEY"] = "sk-your-key"

def demo_embedding():
    # 1. Initialize OpenAI's Embedding model (defaults to the high-performance text-embedding-3-small)
    embeddings_model = OpenAIEmbeddings()
    
    text = "This is a high-speed cyclone fan"
    print(f"Converting text to vector coordinates: '{text}'")
    
    # 2. Perform the vectorization magic!
    vector = embeddings_model.embed_query(text)
    
    # 3. Print the result (OpenAI vectors typically have 1536 dimensions; we'll just print the first 5)
    print(f"Success! The text is now a {len(vector)}-dimensional coordinate set.")
    print(f"The first five numbers are: {vector[:5]}")

demo_embedding()

# Terminal output:
# Converting text to vector coordinates: 'This is a high-speed cyclone fan'
# Success! The text is now a 1536-dimensional coordinate set.
# The first five numbers are: [0.012, -0.045, 0.088, 0.001, -0.023]

See? This is how AI perceives the world. That Chinese sentence about the fan has been transformed into a long series of precise floating-point coordinates.

Now that we have the magic of semantic understanding and the databases to store them, a new question arises: How do we automatically feed our company's 500-page PDF manual into this system? In the next chapter, we’ll explore LangChain’s most practical building blocks: Document Loaders and Text Splitters, to create an automated knowledge base processing pipeline!