Support QA — claude-sonnet-5

Customer Support QA · claude-sonnet-5 · Jul 7, 11:55 PM

Status
failed
Pass rate
13%
Duration
44.82s

What is your refund policy?

failed
Expected output

Refunds are available within 30 days of purchase with a valid receipt.

Actual output

Refunds are available within 30 days of purchase with a valid receipt. (Response omitted key detail from context.)

Metric scores
Answer Relevancy0.39

Output scored below the answer relevancy threshold of 0.5.

Faithfulness0.57

Output meets the faithfulness threshold of 0.5.

Hallucination0.67

Output scored above the hallucination threshold of 0.5.

Contextual Precision0.47

Output scored below the contextual precision threshold of 0.5.

How do I reset my password?

failed
Expected output

Go to Settings > Security and click 'Reset Password', then follow the emailed link.

Actual output

Go to Settings > Security and click 'Reset Password', then follow the emailed link. (Response omitted key detail from context.)

Metric scores
Answer Relevancy0.35

Output scored below the answer relevancy threshold of 0.5.

Faithfulness0.38

Output scored below the faithfulness threshold of 0.5.

Hallucination0.45

Output meets the hallucination threshold of 0.5.

Contextual Precision0.30

Output scored below the contextual precision threshold of 0.5.

Can I upgrade my plan mid-cycle?

failed
Expected output

Yes, upgrades take effect immediately and are prorated for the remainder of the billing cycle.

Actual output

Yes, upgrades take effect immediately and are prorated for the remainder of the billing cycle. (Response omitted key detail from context.)

Metric scores
Answer Relevancy0.53

Output meets the answer relevancy threshold of 0.5.

Faithfulness0.54

Output meets the faithfulness threshold of 0.5.

Hallucination0.61

Output scored above the hallucination threshold of 0.5.

Contextual Precision0.64

Output meets the contextual precision threshold of 0.5.

Do you support single sign-on (SSO)?

failed
Expected output

Yes, SSO via SAML and OIDC is available on the Enterprise plan.

Actual output

Yes, SSO via SAML and OIDC is available on the Enterprise plan. (Response omitted key detail from context.)

Metric scores
Answer Relevancy0.31

Output scored below the answer relevancy threshold of 0.5.

Faithfulness0.65

Output meets the faithfulness threshold of 0.5.

Hallucination0.50

Output meets the hallucination threshold of 0.5.

Contextual Precision0.65

Output meets the contextual precision threshold of 0.5.

What happens to my data if I cancel my subscription?

failed
Expected output

Your data is retained for 90 days after cancellation, after which it is permanently deleted.

Actual output

Your data is retained for 90 days after cancellation, after which it is permanently deleted. (Response omitted key detail from context.)

Metric scores
Answer Relevancy0.32

Output scored below the answer relevancy threshold of 0.5.

Faithfulness0.31

Output scored below the faithfulness threshold of 0.5.

Hallucination0.34

Output meets the hallucination threshold of 0.5.

Contextual Precision0.41

Output scored below the contextual precision threshold of 0.5.

Is there a free trial available?

failed
Expected output

Yes, a 14-day free trial is available on all paid plans, no credit card required.

Actual output

Yes, a 14-day free trial is available on all paid plans, no credit card required. (Response omitted key detail from context.)

Metric scores
Answer Relevancy0.46

Output scored below the answer relevancy threshold of 0.5.

Faithfulness0.51

Output meets the faithfulness threshold of 0.5.

Hallucination0.69

Output scored above the hallucination threshold of 0.5.

Contextual Precision0.30

Output scored below the contextual precision threshold of 0.5.

How do I invite team members?

passed
Expected output

Navigate to Team Settings and click 'Invite Member' to send an email invitation.

Actual output

Navigate to Team Settings and click 'Invite Member' to send an email invitation.

Metric scores
Answer Relevancy0.56

Output meets the answer relevancy threshold of 0.5.

Faithfulness0.64

Output meets the faithfulness threshold of 0.5.

Hallucination0.44

Output meets the hallucination threshold of 0.5.

Contextual Precision0.55

Output meets the contextual precision threshold of 0.5.

What regions do you have data centers in?

failed
Expected output

We operate data centers in US-East, US-West, EU-Central, and AP-Southeast.

Actual output

We operate data centers in US-East, US-West, EU-Central, and AP-Southeast. (Response omitted key detail from context.)

Metric scores
Answer Relevancy0.34

Output scored below the answer relevancy threshold of 0.5.

Faithfulness0.48

Output scored below the faithfulness threshold of 0.5.

Hallucination0.66

Output scored above the hallucination threshold of 0.5.

Contextual Precision0.32

Output scored below the contextual precision threshold of 0.5.