That is both its biggest strength and one of its biggest problems.

Ask an LLM a question and you can often get an impressive answer within seconds. The response is well structured, grammatically correct, and sometimes even more detailed than you expected.

But there is a problem:

A confident answer isn't necessarily a correct answer.

I learned this while experimenting with AI applications. At first, I thought the solution to inaccurate responses was simply better prompting.

It wasn't.

The problem with asking an LLM to "just know"

Imagine building an AI assistant for a company.

You give it a prompt:

"You are an intelligent company assistant. Answer the user's questions accurately."

It works surprisingly well for general questions.

Then someone asks:

"What is our company's refund policy for orders older than 30 days?"

The model doesn't actually know your company's refund policy.

It has two choices:

Admit that it doesn't know. Generate something that sounds reasonable.

And language models are extremely good at option two.

This is where the difference between using an LLM and building an AI system becomes important.

The idea behind RAG

One approach I found particularly useful is Retrieval-Augmented Generation (RAG).

The basic idea is quite simple.

Instead of expecting the model to have all the information it needs, we give it relevant information at the time of the query.

A simplified flow looks like this:

User question → Retrieve relevant information → Give information to LLM → Generate answer

For example, suppose a company has thousands of documents containing:

HR policies Product documentation Internal processes Customer support information Technical documentation

Instead of putting everything into one enormous prompt, we can process those documents, convert their content into embeddings, and store them in a vector database.

When a user asks a question, the system searches for the most relevant pieces of information.

Those pieces are then provided to the LLM as context.

The model isn't being asked to magically know the answer.

It is being asked to reason over information that we retrieved for it.

That distinction is important.

But RAG doesn't magically solve hallucinations

This was another lesson that became obvious after actually working with the architecture.

You can build a RAG pipeline and still get bad answers.

Why?

Because there are several places where things can go wrong.

1. The wrong document gets retrieved

If the retrieval system doesn't find the relevant information, the model has bad context.

Garbage in, garbage out.

1. The retrieved context is incomplete

The answer might require information from multiple documents, but the system may only retrieve one.

1. The context can be noisy

Giving the model 10 irrelevant documents isn't necessarily better than giving it two useful ones.

1. The model can still make things up

Even with good context, an LLM can sometimes infer something that isn't actually supported by the provided information.

So RAG shouldn't be treated as a magic anti-hallucination button.

It's a system that reduces the amount of information the model has to invent.

Prompt engineering is still important

This doesn't mean prompting is irrelevant.

The prompt can tell the model how to use the retrieved information.

For example, instead of:

"Answer the question."

we can instruct it to:

Use only the provided context to answer the question. If the answer cannot be found in the context, say that the information is unavailable.

That small change can make the system much more reliable.

But there's an important engineering lesson here:

You can't prompt your way out of every architectural problem.

If your retrieval system consistently returns irrelevant documents, rewriting the prompt 50 times probably isn't going to fix the root cause.

Sometimes the solution isn't a better prompt.

It's better retrieval.

The part people don't talk about enough: evaluation

This is probably one of the most important parts of building AI applications.

A demo can look amazing.

You ask five questions.

The AI answers all five correctly.

You think you've built something great.

Then 500 real users start asking questions you never considered.

Suddenly you discover:

Some questions retrieve the wrong documents. Some answers contain unsupported claims. Some responses are unnecessarily long. Some queries are extremely expensive. Some questions have no answer in your knowledge base.

This is why AI applications need evaluation.

You need actual test questions and expected behaviour.

Instead of asking:

"Does my AI application work?"

ask:

"How often does it work, where does it fail, and why?"

That is a much more useful question.

What I find exciting about GenAI

The interesting thing about generative AI isn't simply that a model can write text.

It's that we can combine models with traditional software engineering.

An LLM can become one component inside a larger system involving:

APIs + databases + retrieval + tools + authentication + business logic + UI + monitoring

That's where things get interesting.

The model doesn't have to do everything.

It just needs to do the things it is good at.

For example, instead of asking an LLM to calculate a customer's invoice, I would rather let normal software perform the calculation and let the LLM explain the result to the customer.

Instead of asking an LLM to remember an entire knowledge base, retrieve the relevant information.

Instead of trusting every generated answer, evaluate it.

The best AI systems I've seen don't replace engineering.

They combine AI with good engineering.

What I'm taking away from this

My biggest change in perspective has been moving from:

"How can I make the model give a better answer?"

to:

"How can I design a system that gives the model the right information and the right tools to produce a reliable answer?"

That shift sounds small, but it changes how you approach AI development.

Generative AI makes it incredibly easy to build a prototype.

The harder part is building something people can actually trust.

And I think that's where the real engineering challenge begins.
/