Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 8

Enterprise Knowledge & Data Architecture

Last updated · 10 min read

8.1 Retrieval-Augmented Generation (RAG), Embeddings & Vector Databases

Priority: Must Understand

Executive Definition: Retrieval-augmented generation (RAG) grounds a model's answer by first pulling relevant passages from your own data, rather than relying only on what the model learned in training. Text is converted into embeddings (numeric vectors capturing meaning) and stored in a vector database that finds semantically similar content at query time. This is the standard way enterprises connect a general-purpose model to proprietary knowledge without retraining it.

Why It Matters: RAG output quality is now understood to be dominated by retrieval and data quality, not model choice: Anthropic found that adding generated context to chunks before embedding ("contextual retrieval") cut retrieval failure rates by up to 49%, and combining it with reranking cut failures by 67% versus baseline vector search. Most "hallucination" complaints in production RAG systems are retrieval or chunking failures, not model failures: which is why this is the foundational topic for the entire domain.

What I Need to Understand:

  • Embeddings are lossy numeric compressions of meaning; retrieval quality depends on chunking strategy, embedding model choice, and index freshness: not on the LLM.
  • Vector databases (pgvector, Pinecone, Weaviate, etc.) are storage/retrieval infrastructure, not reasoning engines; choosing one is an infrastructure decision, not an AI strategy decision.
  • Naive "chunk and embed" RAG degrades on long, structured, or technical documents because chunking destroys context: this is why contextual retrieval and hybrid retrieval (next topic) exist.
  • Reranking (a second, more expensive relevance pass over retrieved candidates) materially improves precision and is usually worth the added latency/cost for high-stakes use cases.
  • Retrieval accuracy and generation accuracy should be measured and evaluated separately; a wrong answer is often a retrieval-stage failure, not a model-stage one.

Questions I Should Be Able to Ask My Team:

  1. How do we measure retrieval recall/precision separately from final answer quality, and what's our current baseline?
  2. Are we using contextual retrieval and/or reranking, and what lift did we actually measure versus plain vector search?
  3. What's our re-indexing cadence for documents that change frequently, and how stale can the index get before it's a risk?

Technologies / Standards / Companies to Know: pgvector, Pinecone, Weaviate, Qdrant, Azure AI Search, Amazon Kendra/OpenSearch, OpenAI/Cohere/Voyage embedding models, Anthropic Contextual Retrieval, Cohere Rerank.

Recommended Learning:

Time Investment: 2-3 hours

8.2 Hybrid Retrieval, Knowledge Graphs, GraphRAG & Text-to-SQL

Priority: Monitor

Executive Definition: Pure vector search retrieves by semantic similarity but is blind to exact keywords, IDs, and relationships between entities. Hybrid retrieval combines vector search with keyword search (e.g., Best Matching 25 (BM25)); knowledge graphs and GraphRAG structure information as entities and relationships so a system can reason across an entire corpus, not just within single documents; text-to-SQL lets a model query structured databases in natural language instead of treating tables as unstructured text.

Why It Matters: Many enterprise questions ("who approved this vendor, and what else are they connected to") are relationship or corpus-wide questions that plain vector RAG answers poorly: Microsoft Research built GraphRAG specifically because standard RAG cannot answer "global" queries requiring synthesis across many documents. Text-to-SQL benchmark accuracy also drops sharply on real, messy production schemas compared to curated academic benchmarks, so vendor accuracy claims need validation against your own data before you trust them.

What I Need to Understand:

  • Hybrid search (vector + keyword) is close to default practice now, because pure semantic search misses exact matches like IDs, names, and codes.
  • GraphRAG builds a knowledge graph and community summaries at indexing time: it costs meaningfully more to build and maintain than plain vector RAG and is justified only for corpus-wide relationship reasoning, not general document Q&A.
  • Text-to-SQL accuracy on real enterprise schemas (ambiguous columns, undocumented joins) is typically much lower than published benchmark numbers: test on your own schema before trusting it.
  • These techniques are additive: production systems commonly combine vector, keyword, and structured (SQL/graph) retrieval behind one interface rather than choosing just one.

Questions I Should Be Able to Ask My Team:

  1. Which of our use cases genuinely require corpus-wide relationship reasoning (GraphRAG-shaped) versus simple document lookup (RAG-shaped)?
  2. What is our text-to-SQL accuracy on our actual production schema, and what's the fallback when a generated query is wrong: silent execution or confirm-before-run?
  3. What's the ongoing cost of keeping a knowledge graph or hybrid index synchronized with source systems as they change?

Technologies / Standards / Companies to Know: Microsoft GraphRAG, Neo4j, Elasticsearch/OpenSearch (BM25 + vector), Weaviate hybrid search, the BIRD-SQL (Big Bench for Large-Scale Database Grounded Text-to-SQL Evaluation) and Spider benchmarks.

Recommended Learning:

Time Investment: 1 hour

8.3 Permissions-Aware Retrieval & Access Control Propagation

Priority: Must Understand

Executive Definition: When an AI system retrieves from enterprise content, it must enforce the same access permissions the source systems already have: who can see which documents, folders, or records. Permissions-aware retrieval means access decisions are checked at query time against retrieved content, not just at the login screen of the application in front of the model.

Why It Matters: This is one of the most common and dangerous production failures in enterprise RAG: a vector index built once from "all documents" and queried by everyone, regardless of source permissions, turns a chatbot into a data exposure path, as OWASP's RAG security guidance details explicitly. Standard identity and access management (Role-Based Access Control (RBAC)/Attribute-Based Access Control (ABAC)) built for applications does not automatically propagate into a vector database or embeddings pipeline: it has to be deliberately reimplemented at the retrieval layer, and getting it wrong is a governance and legal exposure issue, not a minor bug.

What I Need to Understand:

  • A vector index has no inherent concept of "who can see what" unless permissions metadata is attached to every chunk and enforced at retrieval time (pre-filtering, not just post-filtering).
  • Filtering after retrieval is weaker than filtering during retrieval: ranking on content a user shouldn't see can leak information even if the final response is blocked.
  • Permission changes (offboarding, role changes, reclassification) must propagate into the retrieval layer promptly; sync lag is itself a risk window.
  • Document-level and row-level security at the database/vector-store layer is different from, and complementary to, application-layer authorization: both are needed for defense in depth.
  • Multi-tenant AI deployments (one system serving many business units or customers) need real namespace/partition isolation in the vector store, not just query-time filters.

Questions I Should Be Able to Ask My Team:

  1. Does our retrieval layer enforce source-system permissions per query, or was the index built once from a static export at a single access level?
  2. What is the sync lag between a permission change in the source system and that change taking effect in retrieval?
  3. Have we actually red-teamed whether a user can retrieve content they shouldn't see through prompt-based probing, not just through the normal UI?

Technologies / Standards / Companies to Know: OWASP LLM/RAG security guidance, RBAC/ABAC, Postgres row-level security, Pinecone namespaces, Weaviate multi-tenancy, Microsoft Purview, Amazon Kendra/Q with ACL-aware retrieval.

Recommended Learning:

Time Investment: 1 hour

8.4 Data Classification, Residency, Freshness & Source Provenance

Priority: Must Understand

Executive Definition: Before content reaches a model, it needs metadata: sensitivity classification (public/internal/confidential/regulated), residency (where it may legally be stored/processed), freshness (how current it is), and provenance (source system, author, version). Without this metadata layer, an AI system cannot reliably enforce policy, explain its answers, or avoid presenting outdated information as current.

Why It Matters: Regulated environments increasingly require AI outputs to be traceable to a specific, current, authorized source: an answer with no provenance is an audit and liability problem, not just a UX gap. Data residency requirements constrain which cloud regions and model providers are legally usable for a given data set, which makes this a governance-driven architecture constraint rather than an engineering preference; NIST's AI Risk Management Framework explicitly calls for documenting data provenance and quality as part of AI governance (Govern/Map functions), signaling this is becoming a compliance expectation rather than optional hygiene.

What I Need to Understand:

  • Classification, residency, freshness, and provenance are metadata problems solved in the data layer before retrieval: they cannot be bolted onto the model afterward.
  • Residency decisions determine which cloud regions and model providers are usable for a given data set, end-to-end: including where embeddings are computed and cached, not just where the final answer is generated.
  • "Freshness" needs an explicit owner and update cadence per source; an index with no last-verified date is a silent risk.
  • Provenance (source system, author, timestamp, version) should be attached to every retrieved chunk and, ideally, surfaced as a citation in the answer.

Questions I Should Be Able to Ask My Team:

  1. Can we trace any AI-generated answer back to the specific source document, version, and last-verified date it came from?
  2. Which of our data sources have residency constraints, and does our current retrieval/model deployment actually satisfy them end-to-end?
  3. Who owns freshness for each major knowledge source, and what's the process when a source becomes stale or deprecated?

Technologies / Standards / Companies to Know: NIST AI Risk Management Framework, ISO/IEC 42001, data catalogs (Microsoft Purview, Collibra, Atlan), cloud region/residency controls (Azure, AWS, GCP).

Recommended Learning:

Time Investment: 1 hour

8.5 Enterprise, Personal & Agent Memory Architecture

Priority: Should Understand

Executive Definition: "Memory" refers to information retained across sessions rather than supplied fresh each time: enterprise memory (shared organizational knowledge), personal memory (per-user preferences/history), and agent memory (an agent's record of what it has done in a task). Unlike RAG, which retrieves from a fixed corpus, memory is written and updated by the system itself over time: which raises its own accuracy, staleness, and governance questions.

Why It Matters: Memory expands what an agent can do across long-running or multi-session tasks, but introduces a distinct risk: the system now writes and later trusts its own prior conclusions, so errors can compound: which is why Anthropic's agent-memory approach treats memory as something to curate deliberately, not log unconditionally. Memory design overlaps directly with the permissions and provenance questions above: whose memory is it, who can read it, and does it expire?

What I Need to Understand:

  • Memory is not the same as context window size (see Context Windows topic): it's about what persists after a task ends, versus what fits in one request.
  • Agent memory architectures (e.g., the MemGPT "OS-inspired" model) keep a small working context plus a larger external store the agent retrieves from: this is retrieval applied to an agent's own history, subject to the same quality risks as RAG.
  • Enterprise/shared memory needs the same permission and provenance controls as any other knowledge source: memory is a data asset, not just a convenience feature.
  • Unreviewed agent-written memory can drift or encode errors that reinforce over time; production systems need review/expiry mechanisms, not indefinite accumulation.

Questions I Should Be Able to Ask My Team:

  1. When an agent "remembers" something across sessions, who can see that memory, and does it inherit the access controls of the source data it was derived from?
  2. What's our process for reviewing, correcting, or expiring agent-written memory so errors don't compound?
  3. Is personal/user memory scoped and deletable in line with our data retention and privacy obligations?

Technologies / Standards / Companies to Know: Anthropic memory tool / Claude Managed Agents memory, MemGPT, Letta, Mem0.

Recommended Learning:

Time Investment: 1 hour