ENTERPRISE AI ANALYSIS

LOGICAL-COMMONSENSEQA addresses these limitations by reframing commonsense reasoning as a logical composition task. It leverages a dataset of multiple-choice questions where each option is a composition of two atomic statements linked by 'AND', 'OR', or 'NEITHER/NOR' operators. This design explicitly models joint plausibility, partial plausibility, and joint implausibility. The benchmark's construction pipeline involves generating candidate options, refining them for logical consistency and contextual relevance, and deterministically composing them with symbolic operators. Human validation ensures socially grounded plausibility judgments.

Our experiments with various LLMs (zero-shot, few-shot, fine-tuned) reveal significant performance disparities. While models perform reasonably well on 'AND' (conjunctive) and 'OR' (disjunctive) reasoning, their accuracy sharply declines on 'NEITHER/NOR' (negation-based) questions. This 'negation inversion' indicates models struggle to correctly process implausibility and composite logical structures, often relying on surface-level heuristics rather than genuine compositional reasoning. Fine-tuned models show improved performance, suggesting the task is learnable with explicit supervision, but core challenges remain for instruction-tuned models.

The findings highlight persistent gaps in LLMs' ability to perform robust compositional commonsense reasoning, particularly concerning negation and nuanced plausibility. LOGICAL-COMMONSENSEQA serves as a crucial diagnostic tool, pushing the boundaries beyond simple factual retrieval to complex relational judgments. Future work will involve expanding the operator set to include richer logical structures like implication and causality, exploring generative settings, and studying transferability to real-world applications such as dialogue and planning, further enhancing LLMs' understanding of the world.