Note

Go to the end to download the full example code.

RAG Agent#

This notebook takes inspiration from Pydantic-AI RAG tutorial and simplifies its approach.

Context#

Our objective is to create an assistant that answers questions about Logfire (developed by Pydantic) using the project’s documentation as a knowledge base. We make this RAG agentic by providing a retrieve function as a tool for the LLM.

        flowchart TD
   A(User) -->|Ask a question| B(Agent)
   B --> |Messages + retrieve tool|C{LLM}
   C --> |Text or Tool Calling response|B
   B --> |Vectorize user query |D(Knowledge base)
   D --> |Return documents closest to query|B
   B --> |Return answer|A

In this system, the LLM has access to the “retrieve” tool, which it may or may not invoke in its response. If invoked, the tool call is parsed by the LLM client and returned as a structured response to the agent, which executes the requested function.

This differs from workflow-based RAG, where the retrieved function is always executed before calling the LLM. In workflow-based RAG, the LLM prompt is a concatenation of the initial prompt and the retrieved content. For more detailed information, we recommend exploring Ragger-Duck, a RAG implementation developed by scikit-learn core developers.

../_images/rag_workflow.png — A RAG workflow (source: Ragger-Duck)#

RAG Workflow vs Agent-Based RAG#

We revisit the “workflow vs agent-based systems” tradeoff mentioned in the previous notebook. This assistant use-case requires a high degree of flexibility, as user queries are arbitrary. To reduce costs and latency, we only query the knowledge base when necessary based on user input.

Implementation#

Design choices#

We simplify the original pydantic tutorial with the following optimizations:

No vector database:
- For small knowledge bases, overall latency can be reduced by keeping the data in memory using a dataframe or a numpy array, persisted with diskcache.
- Approximate nearest neighbors (ANN) operations, typically provided by a vector database, can be replaced by a simple Nearest Neighbors estimator from scikit-learn.
Batch vectorization:
- The content of the knowledge base is vectorized in a single batch rather than looping through each element individually.
Local vectorization:
- To reduce API costs, we vectorize content locally by downloading a text vectorizer from HuggingFace. This is achieved using skrub, which wraps the sentence-transformers library to provide a scikit-learn-compatible transformer. No GPU is required.

Overall, this approach is faster to execute while remaining scalable for reasonably sized knowledge bases, making it more efficient than the original Pydantic tutorial.

Building the knowledge base#

We begin by fetching the Logfire documentation archived online. Next, we use skrub’s dataframe visualization to display long text more efficiently (click on any cell to view its text content).

import pandas as pd
import requests
from skrub import patch_display


# Replace pandas' dataframe visualisation with skrub's
patch_display()

DOCS_JSON = (
    'https://gist.githubusercontent.com/'
    'samuelcolvin/4b5bb9bb163b1122ff17e29e48c10992/raw/'
    '80c5925c42f1442c24963aaf5eb1a324d47afe95/logfire_docs.json'
)

doc_pages = pd.DataFrame(
    requests.get(DOCS_JSON).json()
)
doc_pages

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

We now use a text embedding model to get a vectorized representation of each page of our knowledge base. We can choose among various type of vectorizer:

Type	Example	Advantages	Caveats
Ngram-based	BM25, TF-IDF, LSA, MinHash	Fast and cheap to train. Good baselines.	Lack flexibility, corpus dependent.
Pre-trained text encoder, open weight	BERT, e5-v2, any model on sentence-transformers	More powerful embedding representations, local inference.	Requires installing pytorch and extra dependencies.
Pre-trained text encoder, commercial API	open-ai text-embedding-3-small	Most powerful representations, using techniques like Matryoshka representation learning (also available on sentence-transformers). Easy API integration.	Inference costs, reliance on a third party, closed weights, batch size < 2048.

For this example, we choose the second option, as it reduces inference cost. skrub’s TextEncoder downloads the specified model locally using sentence-transformers, transformers and pytorch before generating the embeddings for the knowledge base.

from skrub import TextEncoder


text_encoder = TextEncoder(
    model_name="sentence-transformers/paraphrase-albert-small-v2",
    n_components=None,
)
embeddings = text_encoder.fit_transform(doc_pages["content"])
embeddings.shape

(299, 768)

Next, we use scikit-learn NearestNeighbors to perform exact retrieval. Note that this operation has a time complexity of \(O(d \times N)\), where \(d\) is the dimensionality of our embedding vectors, and \(N\) the number of elements to scan. For larger knowledge bases, using Approximate Nearest Neighbors, with techniques like HNSW (implemented by faiss) or random projections (implemented by Annoy) is recommended, as these reduce the retrieval time complexity to \(O(d \times log(N))\).

We return the indices of the 8 closest match and their distances for two queries: one related query to our knowledge base topic (Logfire) and another one unrelated (cooking).

from sklearn.neighbors import NearestNeighbors


nn = NearestNeighbors(n_neighbors=8).fit(embeddings)

query_embedding = text_encoder.transform(
    pd.Series([
        "How do I configure logfire to work with FastAPI?",
        "I'm a chef, explain how to bake Lasagnas.",
    ])
)

distances, indices = nn.kneighbors(query_embedding, return_distance=True)

print(distances[0])
doc_pages.iloc[indices[0]]

[12.45328426 13.05936241 13.30593491 13.38497925 13.43500328 13.65298176
 13.69662762 13.72200966]

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

We observe that we successfully retrieved content related to FastAPI in the documentation. What are the results for the unrelated query?

print(distances[1])
doc_pages.iloc[indices[1]]

[15.43010139 15.45553875 15.55194283 15.60620213 15.61275005 15.61703777
 15.67152786 15.82261467]

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

For the second query, we can hardly discern a link between the retrieved items and the original question. However, notice that their distances are higher compared to the first query. This means that no article closely match the second query.

The average distances between the first and second queries are quite similar, though. This issue is commonly referred as the curse of dimensionality, where items in high-dimensional spaces tends to all appear “far” from each other due to the hyper-volume growing exponentially with the number of dimensions. Real-world implementations require a careful evaluation of retrieval system performance, which we skip here.

A possible filtering method would be to set a radius, i.e., a maximum distance beyond which retrieved elements are discarded. As shown below, the second query results in an empty set, as all euclidean distances exceed 14.

nn.radius_neighbors(
    query_embedding,
    radius=14,
    return_distance=True,
    sort_results=True,
)

(array([array([12.45328426, 13.05936241, 13.30593491, 13.38497925, 13.43500328,
              13.65298176, 13.69662762, 13.72200966, 13.92671013, 13.94815063,
              13.99003315])                                                   ,
       array([], dtype=float64)], dtype=object), array([array([  8, 111, 113,  15, 112,  13,  35, 114,  67,  94,  33]),
       array([], dtype=int64)], dtype=object))

We can emulate persistence on disk using diskcache. Originally designed as a fast key-value storage solution for Django, it can also be applied in our agentic context. Here, we serialize the knowledge base content, our text encoder, and the fitted nearest neighbors estimator.

import diskcache as dc


cache = dc.Cache('tmp')
cache["doc_pages"] = doc_pages
cache["text_encoder"] = text_encoder
cache["nn"] = nn

Defining the Agent#

We defined our pydantic-ai Agent with its retrieve function set as a tool. Notice how pydantic-ai enables you to specify a schema for the dependency Deps, which is used as a RunContext during tool execution.

For this example, we use OpenAI GPT-4o-mini rather than Llama3.3-70B with Groq’s free tier as Groq currently struggles with tool calling.

import logfire
from dotenv import load_dotenv
from dataclasses import dataclass
from pydantic_ai import RunContext
from pydantic_ai.agent import Agent
import nest_asyncio


# Load the 'OPENAI_API_KEY' variable environment from a source file.
load_dotenv()

# Enable nested event loop in jupyter notebooks to run pydantic-ai
# asynchronous coroutines.
nest_asyncio.apply()

# Some boilerplate around logging.
logfire.configure(scrubbing=False)

@dataclass
class Deps:
    text_encoder: TextEncoder
    nn: NearestNeighbors
    doc_pages: pd.DataFrame

system_prompt = (
    "You are a documentation assistant. Your objective is to answer user questions "
    "by retrieving the right articles in the documentation. "
    "Don't look-up the documentation if the question is unrelated to LogFire "
    "or Pydantic. "
)

agent = Agent(
    'openai:gpt-4o-mini',
    system_prompt=system_prompt,
    deps_type=Deps,
)

def make_prompt(pages):
    return "\n\n".join(
        (
            "# " + pages["title"]
            + "\nDocumentation URL:" + pages["path"]
            + "\n\n" + pages["content"] + "\n"
        ).tolist()
    )

@agent.tool
async def retrieve(context: RunContext[Deps], search_query: str) -> str:
    """Retrieve documentation sections based on a search query.

    Args:
        context: The call context.
        search_query: The search query.
    """
    with logfire.span(f'create embedding for {search_query=}'):
        query_embedding = context.deps.text_encoder.transform(
            pd.Series([search_query]),
        )

    indices = (
        context.deps.nn.kneighbors(query_embedding, return_distance=False)
        .squeeze()
    )

    pages = context.deps.doc_pages.iloc[indices]
    doc_retrieved = make_prompt(pages)

    print(doc_retrieved)

    return doc_retrieved

Finally, we define our main coroutine entry point to run the agent.

async def run_agent(question: str):
    """
    Entry point to run the agent and perform RAG based question answering.
    """
    logfire.info(f'Asking "{question}"')

    cache = dc.Cache('tmp')

    deps = Deps(
        text_encoder=cache["text_encoder"],
        nn=cache["nn"],
        doc_pages=cache["doc_pages"],
    )
    return await agent.run(question, deps=deps)

Results#

We are now ready run our system! Logfire will generate logs for the different steps to help us observe the different internal steps.

import asyncio


answer = asyncio.run(
    run_agent("Can you summarize the roadmap for Logfire?")
)

18:22:17.971 Asking "Can you summarize the roadmap for Logfire?"
18:22:17.974 agent run prompt=Can you summarize the roadmap for Logfire?
18:22:17.974   preparing model and tools run_step=1
18:22:17.975   model request
Logfire project URL:
https://logfire.pydantic.dev/vincent-maladiere/testing-pydantic
18:22:18.963   handle model response
18:22:18.965     running tools=['retrieve']
18:22:18.966     create embedding for search_query='LogFire roadmap'
# Roadmap
Documentation URL:roadmap.md

Here is the roadmap for **Pydantic Logfire**. This is a living document, and it will be updated as we progress.

If you have any questions, or a feature request, **please join our [Slack][slack]**.


# Features 💡
Documentation URL:roadmap.md

There are a lot of features that we are planning to implement in Logfire. Here are some of them.


# Language Support
Documentation URL:roadmap.md

Logfire is built on top of OpenTelemetry, which means that it supports all the languages that OpenTelemetry supports.

Still, we are planning to create custom SDKs for JavaScript, TypeScript, and Rust, and make sure that the
attributes are displayed in a nice way in the Logfire UI — as they are for Python.

For now, you can check our [Alternative Clients](guides/advanced/alternative-clients.md) section to see how
you can send data to Logfire from other languages.

See [this GitHub issue][language-support-gh-issue] for more information.


# Next steps
Documentation URL:index.md

Ready to keep going?

- Read about [Tracing with Spans](get-started/traces.md)
- Complete the [Onboarding Checklist](guides/onboarding-checklist/index.md)

More topics to explore

- Logfire's real power comes from [integrations with many popular libraries](integrations/index.md)
- As well as spans, you can [use Logfire to record metrics](guides/onboarding-checklist/add-metrics.md)
- Logfire doesn't just work with Python, [read more about Language support](https://opentelemetry.io/docs/languages/){:target="_blank"}


# Pydantic Logfire
Documentation URL:index.md

From the team behind Pydantic, Logfire is a new type of observability platform built on the same belief as our open source library — that the most powerful tools can be easy to use.

Logfire is built on OpenTelemetry, and supports monitoring your application from any language, with particularly great support for Python! [Read more](why-logfire/index.md).


# or
Documentation URL:integrations/psycopg.md

logfire.instrument_psycopg('psycopg')


# Usage
Documentation URL:integrations/system-metrics.md

```py
import logfire

logfire.configure()

logfire.instrument_system_metrics()
```

Then in your project, click on 'Dashboards' in the top bar, click 'New Dashboard', and select 'Basic System Metrics (Logfire)' from the dropdown.


# To instrument the whole module:
Documentation URL:integrations/psycopg.md

logfire.instrument_psycopg(psycopg)

18:22:20.000   preparing model and tools run_step=2
18:22:20.000   model request
18:22:22.924   handle model response

Let’s now display the final response:

print(answer.data)

The roadmap for Logfire is a living document that outlines planned features and development directions. Here are the key points:

1. **Features**: Logfire is set to include various features aimed at enhancing logging and observability. Specific details are available in the documentation.

2. **Language Support**:
   - Logfire is built on OpenTelemetry, supporting all languages compatible with it.
   - Custom SDKs are planned for JavaScript, TypeScript, and Rust to ensure attributes are well-represented in the Logfire UI, similar to Python.
   - More information can be found in the Alternative Clients section.

3. **Next Steps**: Users are encouraged to explore topics such as:
   - Tracing with spans
   - Completing the onboarding checklist
   - Integrations with popular libraries
   - Recording metrics

For ongoing updates and more detailed information, you can refer to the [full roadmap documentation](roadmap.md).

The display below shows the sequence of messages from top to bottom (most recent).

The LLM correctly responded to our first query by calling a retrieval tool.
After retrieving the content queried by the LLM, we make another call with this content.
Finally, the LLM sends back its text response, completing the message loop.

import json
from pprint import pprint


pprint(
    json.loads(answer.all_messages_json())
)

[{'kind': 'request',
  'parts': [{'content': 'You are a documentation assistant. Your objective is '
                        'to answer user questions by retrieving the right '
                        "articles in the documentation. Don't look-up the "
                        'documentation if the question is unrelated to LogFire '
                        'or Pydantic. ',
             'part_kind': 'system-prompt'},
            {'content': 'Can you summarize the roadmap for Logfire?',
             'part_kind': 'user-prompt',
             'timestamp': '2025-01-05T18:22:17.974732Z'}]},
 {'kind': 'response',
  'parts': [{'args': {'args_json': '{"search_query":"LogFire roadmap"}'},
             'part_kind': 'tool-call',
             'tool_call_id': 'call_EMpVwW5Z89eoIa8UBb5LJfox',
             'tool_name': 'retrieve'}],
  'timestamp': '2025-01-05T18:22:18Z'},
 {'kind': 'request',
  'parts': [{'content': '# Roadmap\n'
                        'Documentation URL:roadmap.md\n'
                        '\n'
                        'Here is the roadmap for **Pydantic Logfire**. This is '
                        'a living document, and it will be updated as we '
                        'progress.\n'
                        '\n'
                        'If you have any questions, or a feature request, '
                        '**please join our [Slack][slack]**.\n'
                        '\n'
                        '\n'
                        '# Features 💡\n'
                        'Documentation URL:roadmap.md\n'
                        '\n'
                        'There are a lot of features that we are planning to '
                        'implement in Logfire. Here are some of them.\n'
                        '\n'
                        '\n'
                        '# Language Support\n'
                        'Documentation URL:roadmap.md\n'
                        '\n'
                        'Logfire is built on top of OpenTelemetry, which means '
                        'that it supports all the languages that OpenTelemetry '
                        'supports.\n'
                        '\n'
                        'Still, we are planning to create custom SDKs for '
                        'JavaScript, TypeScript, and Rust, and make sure that '
                        'the\n'
                        'attributes are displayed in a nice way in the Logfire '
                        'UI — as they are for Python.\n'
                        '\n'
                        'For now, you can check our [Alternative '
                        'Clients](guides/advanced/alternative-clients.md) '
                        'section to see how\n'
                        'you can send data to Logfire from other languages.\n'
                        '\n'
                        'See [this GitHub issue][language-support-gh-issue] '
                        'for more information.\n'
                        '\n'
                        '\n'
                        '# Next steps\n'
                        'Documentation URL:index.md\n'
                        '\n'
                        'Ready to keep going?\n'
                        '\n'
                        '- Read about [Tracing with '
                        'Spans](get-started/traces.md)\n'
                        '- Complete the [Onboarding '
                        'Checklist](guides/onboarding-checklist/index.md)\n'
                        '\n'
                        'More topics to explore\n'
                        '\n'
                        "- Logfire's real power comes from [integrations with "
                        'many popular libraries](integrations/index.md)\n'
                        '- As well as spans, you can [use Logfire to record '
                        'metrics](guides/onboarding-checklist/add-metrics.md)\n'
                        "- Logfire doesn't just work with Python, [read more "
                        'about Language '
                        'support](https://opentelemetry.io/docs/languages/){:target="_blank"}\n'
                        '\n'
                        '\n'
                        '# Pydantic Logfire\n'
                        'Documentation URL:index.md\n'
                        '\n'
                        'From the team behind Pydantic, Logfire is a new type '
                        'of observability platform built on the same belief as '
                        'our open source library — that the most powerful '
                        'tools can be easy to use.\n'
                        '\n'
                        'Logfire is built on OpenTelemetry, and supports '
                        'monitoring your application from any language, with '
                        'particularly great support for Python! [Read '
                        'more](why-logfire/index.md).\n'
                        '\n'
                        '\n'
                        '# or\n'
                        'Documentation URL:integrations/psycopg.md\n'
                        '\n'
                        "logfire.instrument_psycopg('psycopg')\n"
                        '\n'
                        '\n'
                        '# Usage\n'
                        'Documentation URL:integrations/system-metrics.md\n'
                        '\n'
                        '```py\n'
                        'import logfire\n'
                        '\n'
                        'logfire.configure()\n'
                        '\n'
                        'logfire.instrument_system_metrics()\n'
                        '```\n'
                        '\n'
                        "Then in your project, click on 'Dashboards' in the "
                        "top bar, click 'New Dashboard', and select 'Basic "
                        "System Metrics (Logfire)' from the dropdown.\n"
                        '\n'
                        '\n'
                        '# To instrument the whole module:\n'
                        'Documentation URL:integrations/psycopg.md\n'
                        '\n'
                        'logfire.instrument_psycopg(psycopg)\n',
             'part_kind': 'tool-return',
             'timestamp': '2025-01-05T18:22:19.999467Z',
             'tool_call_id': 'call_EMpVwW5Z89eoIa8UBb5LJfox',
             'tool_name': 'retrieve'}]},
 {'kind': 'response',
  'parts': [{'content': 'The roadmap for Logfire is a living document that '
                        'outlines planned features and development directions. '
                        'Here are the key points:\n'
                        '\n'
                        '1. **Features**: Logfire is set to include various '
                        'features aimed at enhancing logging and '
                        'observability. Specific details are available in the '
                        'documentation.\n'
                        '\n'
                        '2. **Language Support**:\n'
                        '   - Logfire is built on OpenTelemetry, supporting '
                        'all languages compatible with it.\n'
                        '   - Custom SDKs are planned for JavaScript, '
                        'TypeScript, and Rust to ensure attributes are '
                        'well-represented in the Logfire UI, similar to '
                        'Python.\n'
                        '   - More information can be found in the Alternative '
                        'Clients section.\n'
                        '\n'
                        '3. **Next Steps**: Users are encouraged to explore '
                        'topics such as:\n'
                        '   - Tracing with spans\n'
                        '   - Completing the onboarding checklist\n'
                        '   - Integrations with popular libraries\n'
                        '   - Recording metrics\n'
                        '\n'
                        'For ongoing updates and more detailed information, '
                        'you can refer to the [full roadmap '
                        'documentation](roadmap.md).',
             'part_kind': 'text'}],
  'timestamp': '2025-01-05T18:22:20Z'}]

Let’s now observe the agent behavior for an unrelated query.

unrelated_answer = asyncio.run(
    run_agent("I'm a chef, explain how to bake a delicious brownie?")
)

22:22.930 Asking "I'm a chef, explain how to bake a delicious brownie?"
22:22.932 agent run prompt=I'm a chef, explain how to bake a delicious brownie?
22:22.933   preparing model and tools run_step=1
22:22.933   model request
22:23.746   handle model response

Since we specified in the agent system prompt not to perform a retrieval operation for unrelated questions, the agent responds with a plain text message indicating its inability to answer.

print(unrelated_answer.data)

I'm sorry, but I can only assist with questions related to LogFire or Pydantic. If you have any questions about these topics, feel free to ask!

As expected, the message loop is smaller since the LLM didn’t invoke the retrieve function, resulting in less latency and lower inference costs.

pprint(
    json.loads(unrelated_answer.all_messages_json())
)

[{'kind': 'request',
  'parts': [{'content': 'You are a documentation assistant. Your objective is '
                        'to answer user questions by retrieving the right '
                        "articles in the documentation. Don't look-up the "
                        'documentation if the question is unrelated to LogFire '
                        'or Pydantic. ',
             'part_kind': 'system-prompt'},
            {'content': "I'm a chef, explain how to bake a delicious brownie?",
             'part_kind': 'user-prompt',
             'timestamp': '2025-01-05T18:22:22.932962Z'}]},
 {'kind': 'response',
  'parts': [{'content': "I'm sorry, but I can only assist with questions "
                        'related to LogFire or Pydantic. If you have any '
                        'questions about these topics, feel free to ask!',
             'part_kind': 'text'}],
  'timestamp': '2025-01-05T18:22:23Z'}]

Finally, we cleanup the tmp diskcache folder.

import shutil

shutil.rmtree('tmp')

Total running time of the script: (0 minutes 29.137 seconds)

Gallery generated by Sphinx-Gallery

	id	parent	path	level	title	content
	id	parent	path	level	title	content
0	0		roadmap.md	1	Roadmap	Here is the roadmap for Pydantic Logfire. This is a living document, and it will be updated as we progress. If you have any questions, or a feature request, please join our [Slack][slack].
1	1	0.0	roadmap.md	2	Features 💡	There are a lot of features that we are planning to implement in Logfire. Here are some of them.
2	2	1.0	roadmap.md	3	Server side Scrubbing of Sensitive Data	The Logfire SDK scrubs sensitive data from logs on the client side before sending them to the server. We are planning to implement similar scrubbing on the server side for other OpenTelemetry clients. We'll also support adhoc scrubbing of rows.
3	3	1.0	roadmap.md	3	Create Teams	You'll be able to create teams with organization. See [this GitHub issue][teams-gh-issue] for more information.
4	4	1.0	roadmap.md	3	Alerts & Notifications	The following features are planned for the alerts and notifications system: - [X] Slack integration - [ ] Email integration - [X] Webhook integration Alerts are based on SQL queries (with canned templates for common cases) that are run periodically, and decide if a new event has occurred. See [this GitHub issue][alerts-email-gh-issue] for more information.

294	294		guides/onboarding-checklist/add-manual-tracing.md	2	f-strings	Instead of this: ```python logfire.info('Hello {name}', name=name) ``` it's much more convenient to use an f-string to avoid repeating `name` three times: ```python logfire.info(f'Hello {name}') ``` Contrary to the previous section, this _will_ work well in Python 3.11+ because Logfire will use special magic to both set the `span_name` to `'Hello {name}'` and set the `name` attribute to the value of the `name` variable, so it's equivalent to the previous snippet. Here's what you need to know about this: - The feature is enabled by default in Python 3.11+. You can disable it with [`logfire.configure(inspect_arguments=False)`][logfire.configure(inspect_arguments)]. You can also enable it in Python 3.9 and 3.10, but it's more likely to not work correctly. - Inspecting arguments is expected to always work under normal circumstances. The main caveat is that the source code must be available, so e.g. deploying only `.pyc` files will cause it to fail. - If inspecting arguments fails, you will get a warning, and the f-string argument will be used as a formatting template. This means you will get high-cardinality span names such as `'Hello Alice'` and no `name` attribute, but the information won't be completely lost. - If inspecting arguments is enabled, then arguments will be inspected regardless of whether f-strings are being used. So if you write `logfire.info('Hello {name}', name=name)` and inspecting arguments fails, then you will still get a warning. - The values inside f-strings are evaluated and formatted by Logfire a second time. This means you should avoid code like `logfire.info(f'Hello {get_username()}')` if `get_username()` (or the string conversion of whatever it returns) is expensive or has side effects. - The first argument must be an actual f-string. `logfire.info(f'Hello {name}')` will work, but `message = f'Hello {name}'; logfire.info(message)` will not, nor will `logfire.info('Hello ' + name)`. - Inspecting arguments is cached so that the performance overhead of repeatedly inspecting the same f-string is minimal. However, there is a non-negligible overhead of parsing a large source file the first time arguments need to be inspected inside it. Either way, avoiding this overhead requires disabling inspecting arguments entirely, not merely avoiding f-strings.
295	295		guides/onboarding-checklist/add-manual-tracing.md	2	Exceptions	The `logfire.span` context manager will automatically record any exceptions that cause it to exit, e.g: ```python import logfire logfire.configure() with logfire.span('This is a span'): raise ValueError('This is an error') ``` If you click on the span in the Live view, the panel on the right will have an 'Exception Traceback' tab: ![Traceback in UI](../../images/guide/manual-tracing-traceback.png) Exceptions which are caught and not re-raised will not be recorded, e.g: ```python with logfire.span('This is a span'): try: raise ValueError('This is an acceptable error not worth recording') except ValueError: pass ``` If you want to record a handled exception, use the [`span.record_exception`][logfire.LogfireSpan.record_exception] method: ```python with logfire.span('This is a span') as span: try: raise ValueError('Catch this error, but record it') except ValueError as e: span.record_exception(e) ``` Alternatively, if you only want to log exceptions without creating a span for the normal case, you can use [`logfire.exception`][logfire.Logfire.exception]: ```python try: raise ValueError('This is an error') except ValueError: logfire.exception('Something went wrong') ``` `logfire.exception(...)` is equivalent to `logfire.error(..., _exc_info=True)`. You can also use `_exc_info` with the other logging methods if you want to record a traceback in a log with a non-error level. You can set `_exc_info` to a specific exception object if it's not the one being handled. Don't forget the leading underscore!
296	296		guides/onboarding-checklist/add-manual-tracing.md	2	Convenient function spans with `@logfire.instrument`	Often you want to wrap a whole function in a span. Instead of doing this: ```python def my_function(x, y): with logfire.span('my_function', x=x, y=y): ... ``` you can use the [`@logfire.instrument`][logfire.Logfire.instrument] decorator: ```python @logfire.instrument() def my_function(x, y): ... ``` By default, this will add the function arguments to the span as attributes. To disable this (e.g. if the arguments are large objects not worth collecting), use `instrument(extract_args=False)`. The default span name will be something like `Calling module_name.my_function`. You can pass an alternative span name as the first argument to `instrument`, and it can even be a template into which arguments will be formatted, e.g: ```python @logfire.instrument('Applying my_function to {x=} and {y=}') def my_function(x, y): ... my_function(3, 4)
297	297		guides/onboarding-checklist/add-manual-tracing.md	1	Logs: Applying my_function to x=3 and y=4	``` !!! note - The [`@logfire.instrument`][logfire.Logfire.instrument] decorator MUST be applied first, i.e., UNDER any other decorators. - The source code of the function MUST be accessible.
298	298	297.0	guides/onboarding-checklist/add-manual-tracing.md	2	Log levels	The following methods exist for creating logs with different levels: - `logfire.trace` - `logfire.debug` - `logfire.info` - `logfire.notice` - `logfire.warn` - `logfire.error` - `logfire.fatal` By default, `trace` and `debug` logs are hidden. You can change this by clicking the 'Default levels' dropdown in the Live view: ![Default levels dropdown](../../images/guide/manual-tracing-default-levels.png) You can also set the minimum level used for console logging with [`logfire.configure`][logfire.configure], e.g: ```python import logfire logfire.configure(console=logfire.ConsoleOptions(min_log_level='debug')) ``` To log a message with a variable level you can use `logfire.log`, e.g. `logfire.log('info', 'This is an info log')` is equivalent to `logfire.info('This is an info log')`. Spans are level `info` by default. You can change this with the `_level` argument, e.g. `with logfire.span('This is a debug span', _level='debug'):`. You can also change the level after the span has started but before it's finished with [`span.set_level`][logfire.LogfireSpan.set_level], e.g: ```python with logfire.span('Doing a thing') as span: success = do_thing() if not success: span.set_level('error') ``` In the Live view, spans are colored based on the highest level of them and their descendants. So e.g. this code: ```python import logfire logfire.configure() with logfire.span('Outer span'): with logfire.span('Inner span'): logfire.info('This is an info message') logfire.error('This is an error message') ``` will be displayed like this: ![Spans colored by level](../../images/guide/manual-tracing-level-colors.png) Here the spans themselves still have their level set to `info` as is the default, but they're colored red instead of blue because they contain an error log. If a span finishes with an unhandled exception, then in addition to recording a traceback as described above, the span's log level will be set to `error`. This will not happen when using the [`span.record_exception`][logfire.LogfireSpan.record_exception] method. In the database, the log level is stored as a number in the `level` column. The values are based on OpenTelemetry, e.g. `info` is `9`. You can convert level names to numbers using the `level_num` SQL function, e.g. `level > level_num('info')` will find all 'unusual' records. You can also use the `level_name` SQL function to convert numbers to names, e.g. `SELECT level_name(level), ...` to see a human-readable level in the Explore view. Note that the `level` column is indexed so that filtering on `level = level_num('error')` is efficient, but filtering on `level_name(level) = 'error'` is not.

Column	Column name	dtype	Null values	Unique values	Mean	Std	Min	Median	Max
0	id	Int64DType	0 (0.0%)	299 (100.0%)	149.	86.5	0	149	298
1	parent	Float64DType	118 (39.5%)	66 (22.1%)	139.	86.0	0.00	143.	297.
2	path	ObjectDType	0 (0.0%)	61 (20.4%)
3	level	Int64DType	0 (0.0%)	4 (1.3%)	1.99	0.766	1	2	4
4	title	ObjectDType	0 (0.0%)	244 (81.6%)
5	content	ObjectDType	0 (0.0%)	285 (95.3%)

Column 1	Column 2	Cramér's V
id	parent	0.749
id	path	0.570
parent	level	0.546
parent	path	0.505
path	level	0.372
title	content	0.336
level	title	0.316
parent	title	0.286
id	level	0.265
path	content	0.256
id	title	0.227
level	content	0.203
id	content	0.179
path	title	0.177
parent	content	0.151

Column	Column name	dtype	Null values	Unique values	Mean	Std	Min	Median	Max
0	id	Int64DType	0 (0.0%)	8 (100.0%)	65.1	51.2	8	111	114
1	parent	Float64DType	1 (12.5%)	4 (50.0%)	55.1	53.1	1.00	32.0	111.
2	path	ObjectDType	0 (0.0%)	4 (50.0%)
3	level	Int64DType	0 (0.0%)	3 (37.5%)	2.00	0.535	1	2	3
4	title	ObjectDType	0 (0.0%)	8 (100.0%)
5	content	ObjectDType	0 (0.0%)	8 (100.0%)

Column 1	Column 2	Cramér's V
title	content	1.00
level	content	1.00
level	title	1.00
path	content	1.00
path	title	1.00
parent	content	1.00
parent	title	1.00
id	title	1.00
parent	level	1.00
parent	path	1.00
id	content	1.00
id	path	1.00
id	level	1.00
id	parent	1.00
path	level	0.750

	id	parent	path	level	title	content
	id	parent	path	level	title	content
75	75	72.0	integrations/starlette.md	2	Excluding URLs from instrumentation	<!-- note that this section is duplicated for different frameworks but with slightly different links --> - [Quick guide](use-cases/web-frameworks.md#excluding-urls-from-instrumentation) - [OpenTelemetry Documentation](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/starlette/starlette.html#exclude-lists)
290	290	287.0	guides/onboarding-checklist/integrate.md	3	Structlog	To integrate with Structlog, check out the [Structlog] page. [inspect-command]: ../../reference/cli.md#inspect-inspect [integrations]: ../../integrations/index.md [Loguru]: ../../integrations/loguru.md [Structlog]: ../../integrations/structlog.md
82	82	81.0	integrations/django.md	2	Excluding URLs from instrumentation	<!-- note that this section is duplicated for different frameworks but with slightly different links --> - [Quick guide](use-cases/web-frameworks.md#excluding-urls-from-instrumentation) - [OpenTelemetry Documentation](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/django/django.html#exclude-lists)
116	116	111.0	integrations/fastapi.md	2	Excluding URLs from instrumentation	<!-- note that this section is duplicated for different frameworks but with slightly different links --> - [Quick guide](use-cases/web-frameworks.md#excluding-urls-from-instrumentation) - [OpenTelemetry Documentation](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/fastapi/fastapi.html#exclude-lists)
64	64		integrations/system-metrics.md	2	Usage	```py import logfire logfire.configure() logfire.instrument_system_metrics() ``` Then in your project, click on 'Dashboards' in the top bar, click 'New Dashboard', and select 'Basic System Metrics (Logfire)' from the dropdown.
126	126	123.0	integrations/flask.md	2	Excluding URLs from instrumentation	<!-- note that this section is duplicated for different frameworks but with slightly different links --> - [Quick guide](use-cases/web-frameworks.md#excluding-urls-from-instrumentation) - [OpenTelemetry Documentation](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/flask/flask.html#exclude-lists)
46	46		integrations/mysql.md	1	To instrument the whole module:	logfire.instrument_mysql() connection = mysql.connector.connect( host="localhost", user="user", password="secret", database="database", port=3306, use_pure=True, )
165	165	164.0	integrations/use-cases/web-frameworks.md	2	Integrations	If you're using one of the following libraries, check out the integration docs: - [FastAPI](../fastapi.md) - [Starlette](../starlette.md) - [Django](../django.md) - [Flask](../flask.md) Otherwise, check if your server uses [WSGI](../wsgi.md) or [ASGI](../asgi.md) and check the corresponding integration.

RAG Agent#

Context#

RAG Workflow vs Agent-Based RAG#

Implementation#

Design choices#

Building the knowledge base#

id

parent

path

level

title

content

id

parent

path

level

title

content

Please enable javascript

id

parent

path

level

title

content

id

parent

path

level

title

content

Please enable javascript

id

parent

path

level

title

content

id

parent

path

level

title

content

Please enable javascript

Defining the Agent#

Results#

This Page