Development of a High-Performance Japanese LLM to Support Medical Administrative Tasks

Under the project "Research and Development/Verification of Safety for Japanese Medical LLMs" promoted by NEDO, a consortium of 10 organizations has developed a high-performance Japanese LLM for medical support. This model is capable of operating in environments where patient information can be managed securely, such as on-premises or domestic cloud environments managed by medical institutions, and possesses performance approaching that of the world's most advanced commercial LLMs.

Verification using a proprietary benchmark showed a maximum accuracy rate of 90.8% in academic exams simulating specialist medical certification, reaching a level close to major commercial LLMs (91.4%). Additionally, safety verification based on Japanese medical characteristics was conducted, confirming the balance between performance and safety required for use in clinical settings.

The results of this R&D are intended to contribute to the efficiency and quality improvement of medical services, with plans for phased social implementation.

1. Background
Medical institutions face three structural challenges when utilizing AI:
1) Patient Data Management: Most AI services process data on overseas servers or under third-party control, making it difficult for institutions to track and manage data.
2) Data Standardization: Terminology and coding systems vary by institution, hindering interoperability.
3) Safety Standards: There is a lack of established safety criteria for LLM use in clinical settings, leaving institutions without a basis for adoption decisions.
This project aimed to develop an AI capable of secure operation with performance comparable to major AI models, focusing on (1) LLM development, (2) safety verification, and (3) use-case verification.

2. Results
(1) Development of a High-Performance Japanese LLM for Secure Environments
Using open LLMs as base models, the team performed additional training using medical textbooks, clinical guidelines, and case studies. The resulting model maintains high performance while ensuring secure data management.
Key results include a 90.8% accuracy rate in mock specialist exams using RAG (Retrieval-Augmented Generation). Furthermore, performance improved by up to 10.8 points compared to the base model in metrics evaluating adherence to Japanese clinical guidelines.

(2) Safety Verification Based on Japanese Medical Characteristics
To ensure safety, the team implemented: 1) Quantitative risk assessment of patient data memorization, 2) Automatic detection and masking of patient data, 3) Development of a conversational safety benchmark (over 50,000 items), and 4) Red teaming (6,000 items) to evaluate attack resistance.
Verification confirmed that high safety levels are maintained even after additional training.

(3) Use-Case Verification
To support medical staff, the team verified feasibility in: 1) Automatic conversion of test names to JLAC11 codes (80.3% accuracy), 2) Automatic organization of case data (92.2% accuracy), 3) Drafting discharge summaries (quality comparable to commercial LLMs), and 4) Natural language queries for electronic medical records.
These tools assist administrative tasks and do not perform diagnosis or treatment. Final decisions remain with medical professionals.

3. Future Plans
The LLM will be implemented in society in phases, prioritizing safety and reliability while maintaining dialogue with medical institutions.