Skip to content

Knowledge base

Engineering context products

How documents, conversations and source code become context products, step by step from ingestion to a code knowledge graph, for platform and data teams.

Written by

Updated:

Agentic Engineering 5

Our position. We hold that the constraint is no longer the code but the context (opens in a new tab): the standards, conventions and decisions that live only in senior developers’ heads. Every wrong agent answer is a documentation audit result, and the fix is writing the standard down where both the agent and the next human can find it.

The pipelines below differ by source, but each keeps the structure and metadata a raw dump would lose. An agent can then reason over the result instead of guessing from fragments.

How do you turn raw enterprise data into context products?

Ingestion and transformation pipelines turn the enterprise’s most critical sources into structured assets, which are then registered in the context catalog. Extraction alone is not the goal. The pipelines “cognify” raw information into an interconnected knowledge base that agents can reason over, a principle of memory frameworks such as Cognee (topoteretes - Cognee - The Open-Source AI Memory Platform for Agents (opens in a new tab)).

Ingesting Enterprise Knowledge

A significant portion of enterprise knowledge resides in unstructured or semi-structured formats. Engineering robust pipelines to process this information is a foundational capability.

Unstructured Text (Confluence, Jira)

Documents from knowledge bases like Confluence and issue trackers like Jira form the core of the Domain Context. An effective ingestion pipeline for these sources involves several steps:

  1. Extraction: Use the official REST APIs to pull content from the source systems. For Confluence, this means extracting the content of pages within specific spaces; for Jira, it involves fetching descriptions, comments, and key fields from issues matching a JQL query (Medium - Building a Confluence-Powered Search Engine Using LangChain and LLMs (opens in a new tab)).
  2. Chunking: Raw documents are often too large for an LLM’s context window. They must be split into smaller, semantically coherent chunks. Instead of naive fixed-size splitting, intelligent strategies should be used, such as splitting by HTML headings (<h1>, <h2>), paragraphs, or Jira issue sections (Shakudo - Building a Confluence Q&A App with LangChain and ChatGPT (opens in a new tab)).
  3. Enrichment: During processing, crucial metadata must be preserved and attached to each chunk. This includes the source URL, page title, author, creation date, and any relevant labels or tags (e.g., Confluence space key, Jira project key). This metadata is vital for filtering and providing citations in the final AI response.
  4. Vectorization & Storage: Each chunk is then passed through an embedding model to create a numerical vector representation. These vectors, along with the text and metadata, are stored in a retrieval system, such as an enterprise search engine, for later querying (Shakudo - Building a Confluence Q&A App with LangChain and ChatGPT (opens in a new tab)).

Conversation Histories

Conversation histories are a unique and dynamic form of context, essential for conversational agents. Unlike static documents, this context grows with each interaction, necessitating active management to avoid exceeding token limits while preserving continuity (Mem0 - LLM Chat History Summarization Guide October 2025 (opens in a new tab)). The management strategy chosen represents a trade-off between fidelity and cost:

  • Conversation Buffer Memory: The simplest approach, where the entire raw conversation history is retained. This offers perfect recall but is not scalable and will quickly exceed model token limits (Pinecone - Conversational Memory for LLMs with Langchain (opens in a new tab)).

  • Conversation Window Memory: A more practical approach that only keeps the last k interactions. This is efficient but risks losing important information from earlier in the conversation (Pinecone - Conversational Memory for LLMs with Langchain (opens in a new tab)).

  • Conversation Summary Memory: This strategy uses an LLM to periodically create a running summary of the conversation. It is highly efficient with tokens but can lose nuance and specific details during the summarization process (Managing Context History Best Practices (opens in a new tab)).

    A sophisticated agent architecture might employ a hybrid approach, maintaining a short-term window buffer for immediate context and using a background process to “reflect” on the conversation, extracting key facts and decisions into a more structured, summarized long-term memory. This active management is a key function of the Agent Context layer.

Advanced Memory: Beyond Simple Summarization

While basic memory strategies are useful, state-of-the-art agentic systems require a more sophisticated approach that mirrors human cognition. Platforms like Zep and Cognee are pioneering this field by building persistent, long-term memory stores that go beyond simple conversation logs (arXiv - Zep: A Temporal Knowledge Graph Architecture for Agent Memory (opens in a new tab)).

These systems operate on a principle similar to the human brain’s distinction between episodic and semantic memory:

  • Episodic Memory: Captures the raw, sequential events of an interaction (e.g., the transcript of a chat session).
  • Semantic Memory: Extracts and synthesizes key facts, entities, and relationships from these events into a structured knowledge base.

Frameworks like Zep achieve this by building a temporal knowledge graph for each user or session (getzep - Zep Cloud: Examples & Integrations (opens in a new tab)). As new conversations happen, the system automatically extracts entities and facts, integrating them into the graph and updating existing information. This allows the agent to reason about how information changes over time (e.g., a user’s preference changing from one session to the next) (Zep: Context Engineering Platform for AI Agents (opens in a new tab)). When the agent needs context, it doesn’t just retrieve a raw chat log; it queries this rich graph to assemble a structured, token-efficient block of the most relevant facts, preferences, and past interactions (Zep: Context Engineering Platform for AI Agents (opens in a new tab)). This “memory formation” approach provides a much deeper and more reliable context than simple summarization, enabling truly personalized and context-aware agent behavior (arXiv - Zep: A Temporal Knowledge Graph Architecture for Agent Memory (opens in a new tab)).

Modeling Code as Context: From Source Files to Semantic Understanding

For AI agents tasked with software development, raw source code is insufficient context. They require a deep, structural understanding of the codebase. Creating a high-fidelity “Technical Context” product is a multi-stage process of progressive abstraction, moving from raw text to syntactic structure and finally to semantic relationships.

Syntactic Analysis with Abstract Syntax Trees (ASTs)

Standard text-based chunking methods are ineffective for source code because they disregard the grammatical structure, often splitting code in the middle of a function or class definition, rendering the chunk syntactically invalid and meaningless (Enhancing LLM Code Generation with RAG and AST-Based Chunking (opens in a new tab)). The correct approach is to parse the code into an Abstract Syntax Tree (AST). An AST is a tree representation of the code’s syntactic structure, where each node represents a construct like a function declaration, an if statement, or a variable assignment (Wikipedia - Abstract syntax tree (opens in a new tab)).

The practical workflow for leveraging ASTs is as follows:

  1. Parser Selection: Choose a robust code parser. tree-sitter is an excellent choice due to its high performance, error tolerance, and extensive library of grammars for various programming languages (arXiv - Abstract Syntax Tree for Programming Language Understanding and Representation: How Far Are We? (opens in a new tab)). Other tools like ANTLR can be used for languages with custom or complex grammars (Reddit - AST for any code : r/Compilers (opens in a new tab)).
  2. Code Parsing: For each source file in a repository, the selected parser is invoked to generate its corresponding AST.
  3. Node Extraction: The AST is then traversed programmatically to identify and extract semantically meaningful units of code. Instead of arbitrary chunks, the output is a collection of complete functions, classes, methods, and import blocks. This ensures that every piece of context provided to the LLM is a syntactically complete and logically coherent unit (Enhancing LLM Code Generation with RAG and AST-Based Chunking (opens in a new tab)).

Building the Code Knowledge Graph

While an AST provides a deep view of a single file, it cannot capture the complex web of relationships that exist between files in a large codebase. To model these dependencies, the structured data from the ASTs must be used to construct a Code Knowledge Graph (DZone - Enhancing Code Analysis With Code Graphs (opens in a new tab)). This graph represents the entire software project as a network of interconnected entities, enabling powerful queries about structure and impact.

The process of building this graph is systematic:

  1. Schema Definition: A graph schema is defined to model the code entities and their relationships. Nodes in the schema would include types like Module, File, Class, Function, and Variable. Relationships would include IMPORTS, DEFINED_IN, CALLS, INHERITS_FROM, and INSTANTIATES (FalkorDB - Code Graph: From Visualization to Integration (opens in a new tab)).
  2. Graph Population from ASTs: The ingestion pipeline iterates through the AST of each file. For each relevant AST node (e.g., a function definition), it creates a corresponding Function node in the graph. It then creates relationships based on the code’s structure, such as a CALLS edge to another function or an INHERITS_FROM edge to a parent class.
  3. Dependency Enrichment: The graph is further enriched with external dependency information. Tools like the GitHub Dependency Graph API can be used to add nodes for third-party libraries and create DEPENDS_ON edges from the project’s modules to these external packages (GitHub Docs - About the dependency graph (opens in a new tab)).
  4. Storage and Querying: The completed graph is loaded into a native graph database such as Neo4j. This provides a powerful and efficient query language (e.g., Cypher) to traverse the graph and answer complex questions about the codebase that are intractable with other methods (Neo4j - Generative AI - Ground LLMs with Knowledge Graphs (opens in a new tab)).

Resources

Articles

Documentation

Papers and reports

Posts

Tools and repositories