Lupinus Inc. and JCOM Corporation presented at the Artificial Intelligence Society

Lupinus Inc. (Headquarters: Minato-ku, Tokyo; President and CEO: Hiroki Sano, hereinafter "the Company") is pleased to announce that, as part of a joint research project with JCOM Corporation (Headquarters: Chiyoda-ku, Tokyo; President and CEO: Yoichi Iwaki, hereinafter "J:COM"), they developed a question category classification method combining Contextual Bandit and LLM-as-a-Judge. This was presented orally at the 40th Annual Conference of the Japanese Society for Artificial Intelligence, held from June 8 (Mon) to June 12 (Fri), 2026, at G Messe Gunma in Takasaki City, Gunma Prefecture. The Company will continue to contribute to the improvement of service quality utilizing AI through technological cooperation with partner companies, including J:COM.

Presentation Overview

Conference Name

The 40th Annual Conference of the Japanese Society for Artificial Intelligence

Date

June 8 (Mon) - June 12 (Fri), 2026

Venue

G Messe Gunma, Takasaki City, Gunma Prefecture

Presentation Title

Question Category Classification Combining Contextual Bandit and LLM-as-a-Judge

Presenters

Katsuya Wada (Company), Shohei Kibe (Company), Reon Hata (J:COM)

Presentation Format

Oral Presentation

Background and Objectives

[Social Background]

In recent years, with the widespread adoption of AI-powered conversational systems, the importance of appropriately responding to diverse user inquiries has been increasing. Especially in services with frequent customer touchpoints, a wide variety of inquiries are received daily, including those with different phrasing and notation. In cable TV and telecommunications services like J:COM, there are many users aged 60 and over, requiring the appropriate understanding of intent for diverse inquiries from a broad range of users. In such an environment, the accuracy of distinguishing between "questions requiring a search" and "conversational responses" in conversational systems incorporating RAG is directly linked to response quality and processing efficiency. Therefore, a mechanism for continuous online learning is necessary to improve this classifier. To achieve online learning, actual user reactions must be used as feedback signals. However, there was a structural problem where user Good/Bad ratings were not for "correctness of category selection" but for "satisfaction with the overall response," and using them directly as learning signals would update the classifier in the wrong direction.

[Objectives of the Initiative]

The main objective of this research is to demonstrate how to extract a pure learning signal related to the correctness of category selection from Good/Bad ratings that are mixed with factors unrelated to the correctness of category selection, such as response generation quality or the presence/absence of search results, and to show that continuous online learning using this signal actually functions. Specifically, we proposed and verified an approach to construct an evaluation signal dedicated to category classification, independent of user satisfaction, by having an LLM-as-a-Judge compare responses for each category and output its confidence.

Presentation Content and Key Findings

[Overview of Presentation Content]

The proposed system consists of five components. The QueryPreprocessor replaces proper nouns and dates with generalized symbols, after which the QuestionEmbedder converts the question into a 384-dimensional vector. Based on this vector, LinUCB selects one of three categories (Chat / VOD Search / Linear Broadcast Search), and the AnswerGenerator generates responses for all three categories in parallel. Finally, LLM-as-a-Judge compares the three responses and outputs its confidence for each category.

The update reward for LinUCB is designed to combine user Good/Bad ratings with the agreement/disagreement between the Judge's selection and LinUCB's selection. Learning and penalties are applied strongly (5 times the differential reward Δt) only when both agree, while they are treated weakly (Δt) when they disagree. This design is intended to prevent excessive reliance on the Judge.

The experiment was conducted in two phases. Phase 1 verified whether LinUCB could learn a classification policy under ideal conditions where the correctness of category selection is directly fed back. Phase 2 verified this under conditions closer to actual operation, using only Good/Bad ratings for the overall response.

[Key Achievements and Findings]

Achievement 1: Confirmed that the classification policy learning by LinUCB effectively functions under ideal conditions where the correctness of category selection is directly fed back. An overall accuracy of 70.56% was achieved, an improvement of +30.95 percentage points compared to random selection.

Achievement 2: Confirmed that classification policy learning is possible even with evaluation signals that include factors unrelated to the correctness of category selection, such as Good/Bad ratings for the overall response, by using LLM-as-a-Judge. An excluded-Good rate of 64.00% was achieved, an improvement of +24.39 percentage points compared to random selection.

Company Profile

Company Name: Lupinus Inc. Established: August 2021 Representative: Hiroki Sano, President and CEO Location: 4F, THE PORTAL MITA, 3-2-8 Mita, Minato-ku, Tokyo 108-0073 HP: https://www.lupinus.com

The Company aims to provide the highest quality business consulting services, focusing on top-line growth through thorough data utilization and deepening customer understanding. With the corporate philosophy of "Creating a New Era for Japan," we promote cutting-edge corporate transformation necessary for Japanese companies to strengthen their competitiveness.

---------------------------------------- ▼Inquiries here https://www.lupinus.com/contact ----------------------------------------