Development of the Rational Thinking, Emotion Regulation, and Problem-Solving Mental Fitness Mobile App for US Navy Sailors: Qualitative and Quantitative Usability Evaluation

Abstract

Abstract Background US Navy sailors experience a range of personal and interpersonal stressors due to challenging operational tempos that can contribute to psychological distress and increased suicide risk. Digital mental fitness tools offer scalable, stigma-reducing solutions to support psychological readiness. However, many existing digital mental health interventions use top-down designs that lack input from the target population and focus on treating psychological symptoms rather than improving overall mental fitness. Objective This study aimed to design a mobile app version of the Rational Thinking, Emotion Regulation, and Problem Solving (REPS) program and evaluate usability and user experience. REPS is grounded in cognitive-behavioral and rational emotive behavior therapy principles. We examined sailor feedback across two evaluation phases to make iterative improvements to the app. Methods Two rounds of usability testing were conducted with 13 first-term Navy sailors over a 2-month period in spring of 2024. Sailors on duty at a naval ship were recruited by their commanding officer. Evaluation phase 1 (n=8) included exploratory interviews and concept testing using an initial prototype that covered a learning objective of the first module. Sailor feedback from phase 1 informed iterative refinements, including a gamified badge feature and dedicated skill practice session. Evaluation phase 2 (n=5) included a think-aloud protocol using an updated, fully functioning prototype of the first module, with three structured learning objectives. Both phases assessed sailor feedback through qualitative interviews and the System Usability Scale (SUS). Thematic analysis identified core themes related to user needs and app characteristics, and SUS scores were summarized to describe usability. Results Sailors found the REPS app prototype to be culturally relevant and relatable to their daily life within the Navy. Sailors also commented that the app had an intuitive, user-friendly design and praised the modern and visually appealing interface. Sailors reported liking the scenario-based situations and clear explanation of how thoughts, feelings, and actions are intertwined. Sailors also identified ways to enhance user engagement, including clearly defining psychological terminology, introducing gamified rewards, outlining learning objectives, and incorporating more prominent visual cues to enhance the discoverability of interactive content. Quantitatively, SUS scores for both evaluation phases were in the excellent range (phase 1: median 92.50, range 72.5‐95.0; phase 2: median 85.00, range 80.0‐92.5). Conclusions The REPS mobile app was rated as highly relevant and usable by newly enlisted Navy sailors. Scenario-based activities, intuitive design, and relatability with lived military experiences were identified as core strengths. Iterative feedback emphasized the importance of human-centered design in refining app functionality, esthetics, and information. These findings support REPS as a promising, scalable digital tool to enhance mental fitness and contribute to universal suicide prevention in the Navy.