Research & Insights

2 Related Work

This paper builds on prior research across three themes: (1) CVA tools, which examine how users generate visualizations via natural language dialogue; (2) CVA evaluation tools, which offer frameworks and interfaces to systematically assess conversational outputs for VA; and (3) visualization and analytical language evaluation methods, which propose both quantitative and qualitative metrics to judge the quality of generated visualizations and analytical explanations.

2.1 CVA Tools

A growing body of work has explored Conversational Visual Analytics (CVA) tools, i.e., systems that enable users to interact with data and create visualizations through natural language dialogue [71, 89]. These tools are designed to lower the technical barriers to data exploration by allowing users to issue queries in natural language, which the tool interprets to: retrieve relevant data fields, select appropriate chart types, assign encodings, and generate visualizations. Early CVA tools used keyword recognition and clarifications [69, 76, 93], interactive widgets [24, 81], and gesture-based input [77] during the conversation so that users can intuitively interact with their data without technical or programming expertise. With recent advances in LLMs, there has been a marked shift toward more expressive and capable CVA tools that can comprehend colloquial, flexible queries, and generate diverse output formats, including structured code, visualization specifications, rendered charts, and natural language explanations.

2.2 CVA Evaluation Methods

This section outlines existing benchmarks and evaluation tools, highlighting their limitations in addressing real-world CVA complexities like multi-turn interactions, diverse output formats, and the need for interpretable, graded metrics. Traditional NLP metrics and even recent visualization-specific metrics often fall short in capturing the nuances of CVA outputs, emphasizing the need for a user-centered approach.

3 Formative Studies: Eliciting Real-World Use Cases, Evaluation Criteria & Workflows

We conducted two complementary formative studies: interviews with 22 CVA tool developers and an observational study with 16 end-users. These studies revealed practitioners' real-world use of CVA tools, their evaluation criteria, and workflows, along with the challenges they faced.

3.1 Study 1: Tool Developers' Use Cases, Evaluation Criteria & Workflows

Semi-structured interviews with 22 professionals revealed four key areas: CVA tool use cases, evaluation criteria, workflows, and challenges. Insights included the need for systematic evaluation across the LLM-mediated interaction components, such as conversational coherence and inferred assumptions.

3.2 Study 2: End-Users' Use Cases, Evaluation Criteria & Workflows

Lab-based sessions with 16 professional data analysts involved multi-turn CVA interactions. Participants rated output quality and provided corrections, informing the design of LEXARA's test case library. Key findings included diverse visualization types, various forms of ambiguity (syntactic, semantic, pragmatic), and the importance of context carryover.

3.4 Evaluation Criteria for CVA Use Cases

Through thematic analysis, participants consistently evaluated responses along three key categories: Visualization Quality (Data, Chart Type, Functionality, Design), Natural Language Quality (Factual Grounding, Analytical Thinking), and Conversation Quality (Coherence, Follow-up Relevance).

4 Design Considerations For A CVA Evaluation Toolkit

Guided by observed needs and challenges, seven design goals (D1-D7) were defined for LEXARA:

• D1: Lower the barrier to systematic benchmarking. Enable low-code setup, run, and interpretation.
• D2: Tailor evaluations to real-world CVA use cases. Support benchmarking on user-specific data, tasks, and prompts.
• D3: Scale evaluations with speed and reliability. Support scalable, repeatable experiments across many utterances, prompts, and models.
• D4: Compare across formats. Support reasoning on alignment across rendered visualizations, natural language explanations, and chart specifications.
• D5: Link overviews to instance-level insights. Allow fluid navigation between high-level aggregate metrics and fine-grained utterance-level results.
• D6: Support context-aware diagnostic analysis. Enable interpretation of model behavior in relation to specific analytic contexts.
• D7: Make metrics interpretable and actionable for handling multiple plausible answers. Employ transparent, graded metrics that accommodate multiple valid outputs.

5 LEXARA: A User-Centered CVA Evaluation Toolkit

Building on the design considerations, we developed LEXARA, a user-centered evaluation toolkit that operationalizes the findings from our formative studies. The toolkit comprises three complementary components:

5.1 LEXARA's Test Cases

Test cases are in YAML/JSON format, sourced from end-user interactions and prior benchmarks, reviewed by experts, and support multiple plausible answers. This ensures real-world complexity and context-awareness.

5.2 LEXARA's User-Centered Evaluation Metrics

Metrics for visualization quality (Data Fidelity, Field Similarity, Chart Type, Functionality, Design) and natural language quality (Factual Grounding, Analytical Thinking, Conversational Quality) are implemented. These metrics are graded, accommodating partial correctness, and are interpretable, anchoring on rubric-based assessment theory. Examples show how scores differentiate between exact, plausible, and poor choices for data representation, chart type, axes, filters, and encodings.

5.3 LEXARA's Interactive CVA Evaluation Tool

The interactive interface supports low-code benchmarking, side-by-side comparison of multi-format outputs (visualizations, text, JSON specs), and drill-down from aggregate metrics to turn-level diagnostics. Features include test case upload, prompt specification, model selection, metric selection, and hierarchical metric drill-down with explanations.

6 Field Deployment Diary Study: Method & Findings

We deployed LEXARA with six CVA tool developers for a two-week diary study. Participants conducted 38 evaluation experiments across 57 test cases, comparing 10 LLMs and 6 system prompts. Findings demonstrated LEXARA's effectiveness in:

• Capturing Real CVA Use: Participants valued the realism and variety of test cases, including multi-turn follow-ups.
• Nuanced and Interpretable Metrics: Drilldowns and on-hover explanations clarified scores, making results actionable.
• Supporting Experiments at Scale: Enabled varied comparisons (models, prompts) and follow-up on edge cases.
• Facilitating Granular, Multi-Format Evaluation: Side-by-side views and JSON diffs reduced cognitive load and helped diagnose score mismatches.

7 Validating LEXARA's Metrics

To assess alignment with expert judgment, we conducted a quantitative validation study comparing LEXARA's metric outputs against human ratings of CVA responses. We sampled 120 CVA responses, stratified by metrics, score ranges, ambiguity labels, and task types.

7.2.1 Inter-Rater Reliability

Human raters showed moderate to high agreement (Cohen's κ median = 0.65 for visualization, 0.63 for natural language), confirming reliability. Interactivity metrics showed lower agreement due to subjectivity, guiding them to be used as diagnostic signals rather than pass/fail criteria.

7.2.2 Metric-Human Correlation

LEXARA's metrics aligned well with human judgments. Data Fidelity, Field Similarity, Chart Type Similarity, and Factual Grounding showed strong rank correlations (Spearman's p ranging from 0.68-0.82). Other NL metrics correlated at p = 0.57-0.71, comparable to human-human agreement. Safeguards were implemented to reduce LLM-as-a-Judge biases.

7.2.3 Model Alignment with Human Preferences

Models perceived as stronger by participants generally obtained higher mean LEXARA scores, with rank correlation for overall visualization score at p = 0.79 and for natural language response score at p = 0.74, providing a sanity check for LEXARA's metrics.

8 Limitations and Future Work

LEXARA contributes to a growing body of research on evaluating LLMs, with a particular focus on the unique demands of CVA. While LEXARA makes significant progress in addressing limitations of existing evaluation approaches, several limitations suggest future directions for sustained use and broader impact:

• Test Suite Coverage: The current test suite, while comprehensive, is bounded by datasources and domains from formative studies. Expanding to dashboard authoring or data stories would be valuable.
• Multimodal & External Tools: LEXARA currently assumes text-only chat endpoints and JSON specifications. Future work involves native multimodal perception and tool-use capabilities.
• Subjectivity in Metrics: While metrics are graded, some subjective judgments (e.g., interactivity) need further refinement and independent validation.
• Authoring Workflow: The YAML/JSON authoring workflow, while transparent, can be a barrier for non-technical users. CSV templates or point-and-click builders are desired.
• Actionable Sensemaking: LEXARA diagnoses issues but does not yet close the loop with semi-automated prompt repair or training data augmentation to create a feedback-driven improvement loop.
• Operational Concerns: Aspects like cost, latency, and prompt/model drift are not currently addressed.

9 Conclusion

As LLMs increasingly mediate analytical reasoning and visual exploration, rigorous and user-centered evaluation becomes critical. Through formative studies with practitioners, we identified key challenges in evaluating LLMs for CVA, including test cases misaligned with real-world use cases, a lack of interpretable, graded metrics, and ad hoc fragmented evaluation workflows. We operationalize these insights into LEXARA, a user-centered CVA evaluation toolkit including test cases grounded in real-world CVA use cases, interpretable metrics that account for multiple or partially correct responses, and supporting low-code benchmarking balancing human and automatic evaluation methods. By enabling scalable, nuanced, and CVA-specific evaluation, our work contributes both conceptual and practical advances toward more transparent, trustworthy, and user-centered assessment of LLM behavior in CVA systems. The toolkit is publicly available at https://lexara-6b38293fcdac.herokuapp.com/ with open-source code at https://anonymous.4open.science/r/Lexara-CVA-Eval-280B/README.md, to support broader adoption and extension by the HCI and visual analytics communities.