Enterprise AI Analysis: Human-centric Evaluation of Semantic Resources: A Systematic Mapping Study
Bridging the Gap in Human-Centric Semantic Resource Evaluation
This systematic mapping study addresses the critical need for a theoretical framework and empirical understanding of Human-centric Evaluation of Semantic Resources (HESR). By analyzing 144 papers over 15 years, it provides foundational definitions, identifies key trends, and offers practical guidelines for researchers and practitioners in developing intelligent systems.
Executive Impact & Key Findings
Understand the critical metrics and overarching insights that define the landscape of human-centric semantic resource evaluation.
0
Studies Analyzed
0
HESR Instances Identified
0
Semantic Accuracy Verified
0
Human Knowledge as Reference
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
RQ1: Semantic Resource Characteristics
RQ2: Evaluation Process Performance
RQ3: Evaluation Population Characteristics
34.95%
Of Evaluated Resources are Ontologies (Glossary-Mapped)
| Resource Type | Original Terminology (%) | Glossary Terminology (%) |
|---|---|---|
| Ontology | 39.25 | 34.95 |
| Ontology Triples | 13.44 | 15.05 |
| Knowledge Graph | 12.37 | 9.14 |
| Knowledge Base | 8.6 | 11.83 |
| Linked Data | 5.91 | 8.6 |
| Dataset | 7.53 | 10.22 |
| Taxonomy | 2.15 | 2.15 |
| Rules | 1.61 | 1.61 |
| Thesaurus | 0.54 | 3.76 |
48.04%
Formalism Not Reported for Evaluated Resources
<500
Triples (27.12%) - Most Common Resource Size
41.52%
Most Popular Verified Aspect: Semantic Accuracy
89.50%
Evaluations Rely on Human Knowledge Frame of Reference
Enterprise Process Flow: Evaluation Contexts
Automatic Extraction Verification
→
Ontology Construction
→
Resource Quality Improvement
→
Task Support
→
Theory Building
→
Training Data Creation
| Method Typology | Method | Usage (%) |
|---|---|---|
| Hevner | Static Analysis | 80.46 |
| Hevner | Dynamic Analysis | 8.05 |
| Peffers | Subject-based Experiment | 47.19 |
| Peffers | Expert Evaluation | 33.71 |
| Pesquita | Custom Questionnaire | 67.01 |
Case Study: S47 - Human Cardiovascular Ontology Evaluation
Study S47 demonstrates HESR within ontology construction. 20 3rd-year medical students and 51 Latin American medical experts evaluated an OWL ontology for completeness, duplication errors, disjunction errors, and consistency. They used a survey with Yes/No questions and assessed improvements between versions. This highlights the importance of expert participation and multi-stage evaluation in verifying domain conceptualization. The HESR was complemented by OOPS! and competency questions.
12.35%
HESR Studies Discussing Potential Biases
65.29%
Evaluations with Less Than 50 Participants
Enterprise Process Flow: Evaluator Task Types
Binary Classification (35.29%)
→
Multi-class Classification (22.94%)
→
Rating/Ranking (21.18%)
→
Create Annotations (10.59%)
→
Task Execution (9.41%)
Case Study: S42 - Crowdsourced Medical Knowledge Verification
Study S42 utilizes HESR for verifying automatically extracted medical knowledge. Crowdworkers assessed the domain correctness of triples extracted from PubMed abstracts. A pilot study with 193 workers and a main study with 101 workers confirmed triples, measured by F-measure against a gold standard. This highlights crowdsourcing as a viable approach for large-scale verification of semantic facts.
<20%
Demographics Weakly Reported
Calculate Your Potential AI ROI
Estimate the potential annual savings and reclaimed human hours by implementing AI solutions in your enterprise, tailored to insights from this study.
Potential Annual Savings
$0
Reclaimed Human Hours
0
Your AI Implementation Roadmap
A strategic approach to integrating human-centric AI, based on best practices and research insights.
Phase 1: Strategic Assessment & Framework Definition
Leverage the theoretical framework of HESR to define evaluation goals, identify relevant semantic resources, and outline human participation requirements. Focus on aspects like semantic accuracy and completeness (RQ2), and align with your enterprise's specific application domains (RQ1).
Phase 2: Pilot HESR & Method Selection
Conduct pilot human-centric evaluations using appropriate methods (e.g., Static Analysis, Subject-based Experiments, Custom Questionnaires) and modalities (e.g., Crowdsourcing Platforms) (RQ2). Start with smaller-scale resources (e.g., <500 triples) and participant groups (<50) (RQ1, RQ3) to refine your approach.
Phase 3: Data-Driven Optimization & Bias Mitigation
Analyze evaluation data to identify trends and optimize resource quality. Actively address potential biases (RQ2) in study design and participant selection (RQ3). Explore strategies like majority voting or tailoring study design to minimize their impact, ensuring robust and generalizable findings.
Phase 4: Scaled Implementation & Continuous Improvement
Scale HESR to larger resources and diverse populations, building upon insights from initial pilots. Integrate HESR into your broader data processing workflows for continuous quality improvement. Focus on automating repetitive tasks while reserving human input for complex, knowledge-intensive evaluations.
Ready to Transform Your Enterprise with AI?
Don't navigate the complexities of AI implementation alone. Schedule a free consultation with our experts to discuss how these insights can drive your organization's success.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat