This page shows you how to evaluate responses from Pinecone Assistant or other RAG systems using the Evaluation API.

To learn about the concepts related to the Evaluation API, see Evaluation API.

This feature is in public preview.

​
Install the Pinecone Assistant Python plugin

To interact with Pinecone Assistant using the Python SDK, upgrade the client and install the pinecone-plugin-assistant package as follows:

HTTP
pip install --upgrade pinecone pinecone-plugin-assistant

​
Calculate evaluation metrics

To evaluate a response, use the https://prod-1-data.ke.pinecone.io/assistant/evaluation/metrics/alignment endpoint. This operation returns a JSON object.

Response
{
  "metrics": {
	  "correctness": 0.5,
	  "completeness": 0.333,
	  "alignment": 0.398,
  },
  "reasoning":{
	  "evaluated_facts": [
	    {
	      "fact": {"content": "Paris is the capital of France"},
	      "entailment": "entailed",
	    },
	    {
	      "fact": {"content": "London is the capital of England"},
	      "entailment": "neutral"
	    },
	    {
	      "fact": {"content": "Madrid is the capital of Spain"},
	      "entailment": "contradicted",
	    }
	  ]
  },
  "usage": {
     "prompt_tokens": 22,
     "completion_tokens": 33,
     "total_tokens": 55
  }
}

Was this page helpful?

Understanding Evaluation APIMove to production