Uni-Mate

A Retrieval-Augmented Generation (RAG) system designed to provide accurate, personalized academic guidance for high school students navigating university options.

AIUX ResearchLLM
2 min read
Uni-Mate

Image Credit: Samuele Mazzei

Situation

High school students often experience significant confusion and a sense of being lost when navigating fragmented and inconsistent university websites. A foundational survey of 183 students confirmed that 74% of respondents faced major difficulties in the orientation process, with fewer than 10% finding existing digital tools to be effective.

Task

The goal was to design and evaluate Uni-Mate (formerly MyVision), a Retrieval-Augmented Generation (RAG) system aimed at providing accurate, personalized academic guidance. The project sought to create a unified platform for orientation, comparison, and reviews to fill a prominent gap in the European academic ecosystem.

Action

To ensure a rigorous, user-centered development process, I implemented the following HCI methodologies:

  • User Research & Co-Design: Built upon user-driven innovation models—similar to those involving high school and university students—to align the platform with the specific skills and interests of the target demographic.
  • Information Architecture (IA): Utilized Markdown parsing to structure fragmented web data into well-defined sections. This technical choice prioritized the user experience by enabling the system to present complex information through clear, readable tables.
  • Dual-Method Evaluation: Orchestrated a robust evaluation framework consisting of automated AI assessment (using a “golden model”) and blinded human evaluation.
  • Statistical Reliability: Employed Krippendorff’s Alpha to calculate inter-annotator agreement among human evaluators. We achieved a score of 0.90, demonstrating extremely high data reliability and objectivity in our assessment of system accuracy.

Result

  • European Recognition: Awarded first place in the Expert Category at DigiEduHack, a prestigious innovation challenge promoted by the European Commission.
  • Stakeholder Impact: Invited to present the project at the Digital Education Stakeholder Forum 2025 in Brussels to discuss digital education policy.
  • Performance Metrics: The system achieved a 79.22% average accuracy score in human evaluations and an 83.63% score in automated AI evaluations.
  • Market Validation: Evaluation results indicated strong latent demand, with 81% of students expressing willingness to use the platform and 67% indicating a readiness to pay for the service.

Mazzei, S., Zambotto, L., Tealdo, G., Macagno, A., & Aprosio, A. P. (2025, September). Uni-Mate: A Retrieval-Augmented Generation System to Provide High School Students with Accurate Academic Guidance. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025) (pp. 699-709).

Check the Github Repository