Conversation analytics for LLM apps

Bring the conversations from your LLM app into Ivy Analytics with one API call.

How data gets in

Ivy Analytics receives conversation data through the Ivy Analytics API. The ivyanalytics Python SDK wraps the API: "This SDK is used to ingest data into Ivy Analytics."

REST endpoint: POST /v1/conversations, "Log a batch of conversations to BigQuery"
Base URL: https://api.ivyanalytics.io
Authentication: HTTP Basic
Python 3.7+

Conversation data model

Field	Description
`conversation_id`	Your ID for the conversation
`tenant_id`	Your unique tenant ID
`conversation_created_at`	When the conversation started (ISO 8601)
`messages`	List of messages, each with id, role (user or assistant), body and created_at
`metadata`	Free-form key/value data, for example model and user_id

Quick start

Installation

pip install ivyanalytics

Configuration

import ivyanalytics
from ivyanalytics.rest import ApiException
from pprint import pprint

# Defining the host is optional and defaults to https://api.ivyanalytics.io
configuration = ivyanalytics.Configuration(
    host = "https://api.ivyanalytics.io"
)

# Configure HTTP basic authorization: HTTPBasic
configuration = ivyanalytics.Configuration(
    username = os.environ["USERNAME"],
    password = os.environ["PASSWORD"]
)

Log a batch of conversations

with ivyanalytics.ApiClient(configuration) as api_client:
    api_instance = ivyanalytics.ConversationApi(api_client)
    conversation_data_schema = ivyanalytics.ConversationDataSchema(conversations=[
        {
            "conversation_id": "my_historical_conv_id_1",
            "tenant_id": "<my_unique_tenant_id>",
            "conversation_created_at": "2020-02-20T20:19:34Z",
            "messages": [
                {"id": "message_id", "role": "user",
                 "body": "What time does the team arrive?",
                 "created_at": "2020-02-20T20:19:34Z"},
                {"id": "message_id", "role": "assistant",
                 "body": "The team arrives at 6pm.",
                 "created_at": "2020-02-20T20:20:23Z"},
            ],
            "metadata": {"model": "gpt-3.5-turbo-16k", "user_id": "2947451"},
        },
    ])
    try:
        # Log a batch of conversations to BigQuery
        api_response = api_instance.ingest_conversations(conversation_data_schema)
        pprint(api_response)
    except ApiException as e:
        print("Exception when calling ConversationApi->ingest_conversations: %s\n" % e)

From the ivyanalytics 0.0b0 README on PyPI (March 2024).