User Experience Analysis on Bakamla Messenger Applications Using User Experiences Questionnaire (UEQ)

User experience describes the experience a user gets when using a software product. This research aims to measure the user experience when using the Bakamla Messenger application. Measurements were carried out using the User Experience Questionnaire (UEQ) method. The research was carried out by distributing online questionnaires to users of the Bakamla Messenger application, with a total of 117 respondents. The measurement results for the attractiveness aspect of 2.26, clarity of 2.30, efficiency of 2.24, accuracy of 2.27, and stimulation of 2.28 have a positive impression value and are included in the excellent criteria. However, the novelty aspect gets a value of 0.02, meaning it has a negative impression value and is included in the bad criteria, so the innovation of the product needs to be increased. Thus, we recommend that Bakamla messenger application developers focus on improving aspects of the novelty value of the application, such as the level of security of confidential data and the messenger system being able to provide new features beyond messenger in general.


Introduction
The Indonesian Maritime Security Agency (Bakamla) is vital in ensuring the safety of Indonesian seas, safeguarding national interests when at sea, and enhancing marine management capacities across the country [1].To carry out its duties, Bakamla must be able to communicate effectively with related agencies such as the Indonesian Navy, the Ministry of Maritime Affairs and Fisheries, the Ministry of Transportation, Polri, and other institutions [2].In the current digital era, messenger applications can play a very vital role in increasing the efficiency of Bakamla's communications with other agencies in monitoring and dealing with various illegal activities in Indonesian waters, such as illegal fishing, illegal actions at sea, and terrorist activities [3].Bakamla has a special messenger application called Bakamla Messenger.The existence of the Bakamla Messenger application allows users to communicate in real-time via text messages, images, sound, video, and location sharing and is able to create communication groups in accordance with the mission of implementing Indonesian maritime security operations [4].
A key factor in the success of a software product is focusing on the user's needs and emotions when interacting with the product [5].User experience (UX) has a very important role in exploring and meeting the needs of user-oriented product development.Therefore, UX should be a primary concern in IICS SEMNASTIK 2023 E-ISSN: 2774-5899 | P-ISSN: 2774-5880 ■ 2 the early stages of product development [6].When designing a Messenger app interface, user experience can be an invaluable guide [7].One indication of the failure of an application or product is the user's discomfort with using it.This research aims to measure the level of user satisfaction when using the Bakamla Messenger application using the User Experience Questionnaire (UEQ) method.
The user experience can be measured both qualitatively and quantitatively.One method used to measure user experience is by using the User Experience Questionnaire (UEQ).The use of UEQ is considered beneficial because it is able to provide comprehensive measurement results of the user experience.This research aims to evaluate the user experience when using Bakamla Messenger.This application was developed in 2022 and began testing in 2023.Evaluation was carried out through the distribution of the User Experience Questionnaire (UEQ) to a number of application users so that we could understand the experiences they have had while using Bakamla Messenger.
Several researchers previously used UEQ to evaluate the user experience of an application or their product.Previous research has applied UEQ for the UI/UX evaluation of developed applications.Research desktop-based applications such as game applications and information systems [8], [9], [10].Other research involves web-based applications such as e-commerce [11], e-report, siakad, e-learning [12] [13].Mobile-based applications such as Gojek and Grab applications [14], [15], [16], [17].The User Experience Questionnaire (UEQ) method has several advantages compared to other user experience evaluation methods, such as simplicity, speed of application, the ability to compare results with other products or applications, and the ability to produce reliable data.This method helps developers better understand the user experience and improve the quality of their products or applications by focusing on user needs and preferences.
Measuring the user experience of the Bakamla Messenger application as a means of communication at the Republic of Indonesia Maritime Security Agency needs to be carried out in order to evaluate the extent to which the quality of the application supports the duties of the Republic of Indonesia Maritime Security Agency and provide input for improving the design and development of the application in the future.It is hoped that this research will make a significant contribution to improving the quality of the Bakamla Messenger application as a communication and coordination tool between state institutions responsible for maintaining Indonesia's maritime security.

Literature Review 2.1. User Experience
User Experience refers to an individual's overall experience when interacting with a product, service or system.It involves various aspects such as perception, emotions, value judgments and user response to the product or service.User Experience consists of several main components: 1. User Interface (UI): The visual and interactive parts of a product used by users, such as display design, icons, buttons, and other elements.2. Function and Performance: The product's ability to perform the desired tasks quickly and efficiently.3. Security and Privacy: It is important for users to feel safe when using the product and confident that their data will not be misused.4. Sustainability: how the product performs and meets user needs in the long term. 5. Emotions and Satisfaction: The user experience is also related to the user's emotions and satisfaction when interacting with the product or service.
The user experience has a significant impact on the success of a product or service.A positive user experience can increase user satisfaction, user loyalty [18], user retention, user error rate, product reputation, and recommendations from other users.
User experience evaluation is a key aspect in the development and maintenance of mobile applications, including messenger applications.A good user experience is an important factor in retaining users, increasing loyalty, and ensuring an app remains competitive.Research by Hazenzal underlines the importance of positive user experiences in increasing satisfaction and continued use of applications [19].
The user experience is a very important concept in successful product and service design.Understanding user needs and preferences and focusing on creating a positive user experience can improve product quality and provide benefits to the company or organization that creates it.Overall, user experience is a factor that should not be ignored in today's digital era.

User Experience Questionnaire (UEQ)
UEQ is a questionnaire specifically designed to comprehensively measure user experience.This questionnaire contains a number of questions that reflect various aspects of the user experience, including

IICS SEMNASTIK 2023
E-ISSN: 2774-5899 | P-ISSN: 2774-5880 ■ 3 satisfaction, usability, aesthetics, and emotions that arise when interacting with the product or service.The use of UEQ aims to: 1. Objectively measure the quality of the user experience.
2. Understand how users experience and evaluate products or services.
3. Identify the strengths and weaknesses of the product or service in the context of the user experience.4. Find areas for improvement in product or service design.
UEQ can be downloaded at www.ueq-online.org.There are six scales with a total of twenty-six elements, which are categorized based on the measurement scales in the UEQ.The user experience scale in this questionnaire is as follows: 1. Attractiveness: To what extent is this product or service visually attractive?It includes elements such as design, color, and appearance that make the product attractive.2. Efficiency: the ability of a product to be used quickly and efficiently.how easily the user can complete the task without much effort or difficulty.3. Perspicuity: Is it easy for users to understand how to use this product?how clear the instructions or appearance of a product are.4. Dependability: Are interactions with this product safe and predictable?how much trust the user has in this product through the control it has. 5. Stimulation: Do users feel interested in and encouraged to use the product?how much motivation the product provides to users.6. Novelty: Is this product design innovative and creative?Is the product able to attract users' attention with its new characteristics?
The six UEQ scales can be grouped into three main aspects, namely the attractiveness aspect, the practical quality aspect, and the hedonic quality aspect.Practical aspects of quality relate to perceived benefits, efficiency, and ease of use.Clarity, efficiency, and accuracy are included in the practical aspects of quality.Meanwhile, the hedonic quality aspect is related to stimulation and innovation.UEQ data can be analyzed using UEQ analysis tools.To understand the meaning of the values produced by each UEQ scale, we can compare the calculated values for each scale with the values contained in Table 1.

Bakamla Messenger
Bakamla Messenger is a messenger application developed since 2022 specifically for the Indonesian Maritime Security Agency and institutions that have legal authority in Indonesian seas, as well as Sea Guard Volunteers, as a means of communication in their operational activities to protect and maintain the security and sovereignty of Indonesia's seas.unwanted users in the application.Bakamla Messenger is a newly developed application; therefore, an analysis is needed to determine the advantages, disadvantages, and level of user satisfaction with using the application.

Research Method
This research will evaluate the user experience of users of the Bakamla Messenger application.This application has not been uploaded to the Play Store because the users are institutions that have legal authority in Indonesian seas and only Marine Guardian Volunteers.The Bakamla Messenger application can be seen in Figure 2. The evaluation of user experience at Bakamla Messenger is carried out using the User Experience Questionnaire (UEQ) method.The factors measured using UEQ are attractiveness, efficiency, perspicuity, dependability, and stimulation.The stages of the research carried out can be seen in Figure 3 as follows:

Literature Study
The initial stage of this research was to conduct a literature review related to user experience, user experience questions, and previous research that could be used as a reference for this research.Literature studies are taken from reputable national and international journals from the last three years.

Determination of Research Population and Sample
Research respondents were users of the Bakamla Messenger application, including Bakamla, POLRI, TNI, the Ministry of Transportation, the Ministry of Maritime Affairs and Fisheries, and Marine and Coast Guard volunteers.The number of samples is determined using the following equation:

Data Collection with UEQ
After calculating the minimum sample size, data collection was then carried out using UEQ.The data collection process uses the Google Form application by referring to the twenty-six questions in the UEQ, as in Figure 4.The evaluation process begins by asking respondents to use the Bakamla Messenger application.The goal is for users to get good feedback and accurate UEQ results.Users can fill in the UEQ according to the impression they feel when using the Bakamla Messenger application, both in terms of function, color, type of writing, speed, accuracy, service, layout, and so on.After the user uses the application, they are asked to fill out the questionnaire provided.
Users fill out a questionnaire via a Google form that has been distributed to them.Completing the UEQ will be directed in accordance with the UEQ provisions, where point 7 is not always the highest score on the questionnaire.In this research, user experience measurement was carried out using the UEQ questionnaire, which consists of 26 questions, as seen in Figure 5.There are six user experience factors measured using the UEQ, namely (1) attractiveness, (2) clarity, (3) efficiency, (4) accuracy, ( 5) stimulation, and (6) novelty.The data that has been collected from respondents is then entered into the UEQ Data Analysis Tool and used as input values to calculate the experience value of Bakamla Messenger users.Reliability testing of the questionnaire was carried out using the Cronbach alpha coefficient.The Cronbach alpha coefficient describes the consistency for all acceptable scales (>0.7), namely attractiveness (0.9), clarity (0.79),

Processing and Analysis of Data
Processing data obtained from respondents is then processed using UEQ tools.The results of data processing are then analyzed to obtain user experience information from the Bakamla Messenger application.The stages of the data processing process can be seen in Figure 6.The data processing process in Figure 6 is carried out in 5 stages, namely: first, entering the questionnaire results data into UEQ tools; second, carrying out data transformation; third, calculating the individual average (means per person); and fourth, calculating the average for each aspect.The fifth is analyzing the final UEQ results.

Results and Analysis
Evaluation of the user experience (UX) on the Bakamla Messenger application was carried out by distributing questionnaires via Google Form to all users of the Bakamla Messenger application from several state institutions that have legal authority in Indonesian seas, namely Bakamla, TNI AL, POLRI, KKP, the Ministry of Transportation, Customs, and Excise, Ministry of Foreign Affairs and Sea Guard Volunteers From the link distributed, 117 respondents filled out the questionnaire.The data obtained from the questionnaire was entered into UEQ data analysis, and then data transformation was carried out by calculating the value obtained from UEQ minus 4 and obtaining positive and negative values for each UEQ item.The value +3 is the highest positive value, and -3 is the lowest negative value.The results of the data transformation are then processed to obtain an average value for each UEQ aspect.The results of UEQ data processing can be seen in Table 2 and Figure 7. Based on the processing results in Table 2. the highest value is the pragmatic quality aspect, namely 2.27.In the pragmatic quality aspect, the highest value is the clarity aspect, namely 2.30.This shows that the Bakamla Messenger application is easy for users to understand and use because of the clarity and ease of getting positive scores.
The accuracy aspect received a positive impression with a value of 2.27; this shows that users send messages very quickly, precisely, and safely to the recipient, meaning that the application has the advantage of a very good accuracy aspect.
The efficiency aspect received a positive impression with a value of 2.24; this shows that users who send messages in the form of text, photos, videos, and audio are easily received quickly and well by users without requiring additional costs, only requiring an internet connection and quota.
The attractiveness aspect received a positive impression with a score of 2.26, meaning that many users from several state institutions that have legal authority at sea are very interested in using the Bakamla Messenger Application to communicate when carrying out security operations at sea because the conversations are very safe, comfortable, and cheap.The test results showed that the average value of the hedonic quality aspect was 1.15.The stimulation and novelty aspects are included in hedonic quality.The result of calculating the UEQ value for the stimulation aspect is 2.28.This shows that the Bakamla Messenger application is useful for users and motivates them to use it.Meanwhile, the UEQ value for the novelty aspect is 0.02, which indicates that Bakamla Messenger has negative novelty.This is because the application features are similar to those of other Messengers, namely Whatsapp and Telegram.

IICS SEMNASTIK
Based on the research results, it was found that the evaluation results for the aspects of attractiveness, clarity, efficiency, accuracy, and stimulation had a positive value, where all UX aspect values obtained a value above 0.8.This is in accordance with the statement [18] that "the average impression value is between -0.8 and 0.8, which are normal evaluation values."Values >0.8 are positive evaluations, and values <-0.8 are negative evaluations.Of all aspects, the average value for the novelty aspect is lower than for other aspects (<1).This is because, according to users, Bakamla Messenger is almost similar to other messenger applications.So the novelty of the Bakamla Messenger Application is considered very low.Therefore, developers must innovate to provide more value from the Bakamla Messenger application to be able to answer the problems that exist at the Indonesian Maritime Security Agency.The average value of the measurement results for each variable from the UEQ novelty aspect can be seen in Table 3.

Table 3. Average results of measuring aspects and novelty
To find out the meaning of the values produced by each UEQ scale, this is done by comparing the calculated values for each scale with the benchmark values as shown in Table 1.The UEQ scale measurement results compared with the benchmark values can be seen in Figure 8.
Based on the comparison results with the benchmark scale, the aspects of attractiveness, clarity, efficiency, accuracy, and stimulation are included in the excellent category with an average value of >2.00.Meanwhile, the novelty aspect is included in the bad category because the value obtained is below <0.7.Therefore, the results of the benchmark value comparison recommend updating existing applications, especially focusing on the additional ChatBot feature for violation information, security information, and weather condition information, as well as the availability of patrol boats in several areas to help communication between institutions and improve Indonesia's maritime security.Therefore, the development of this application needs to be carried out systematically, especially to add confidential and fast communication features and facilities.

Conclusion
The results of the user experience evaluation using UEQ on the Bakamla Massenger application found that the level of user satisfaction with the system was very high.The measurement results for the attractiveness aspect of 2.26, clarity of 2.30, efficiency of 2.24, accuracy of 2.27, and stimulation of 2.28 have a positive impression value and are included in the excellent criteria.However, the novelty aspect gets a value of 0.02, meaning it has a negative impression value and is included in the bad criteria, so the innovation of the product needs to be increased.Thus, we recommend that Bakamla Messenger application developers focus on improving aspects of the novelty value of the application, such as the level of security of confidential data and the messenger system being able to provide new features beyond general manager.It is hoped that this research can provide benefits for developers in improving the quality of the Bakamla Messenger application and user satisfaction.However, it is important to remember that the use of UEQ must be applied wisely, taking into account the specific context of use.

4 Figure 1 . 8 .
Figure 1.Bakamla Messenger Application Features According to Figure 1, several Bakamla Messenger application features can be explained: 1. Group Chatting: Allows members to form chat groups with various parties involved in maritime security operations.This facilitates rapid coordination and information sharing between all relevant parties.2. Voice and Video Call: Voice and video calling features enable real-time, in-depth discussions and faster decisions in emergency or critical coordination situations.3. Voice Message: Members can send voice messages, which is useful in situations where typing a message is impractical, such as while on a ship.4. Real-time Message Indicator: Displays message status in real time, allowing users to see when a message has been sent or has been read by the other party.5. Copy, Forward, Share, and Delete: This feature makes it easier to share information and documents, as well as manage messages that are no longer relevant.6.Last Seen and Online Indicator: Members can see when other members were last active or online, helping in better communication management.7. Search Particular Message: Allows members to search for specific messages or information in a conversation, saving time and making decisions easier based on historical data.8. Location Sharing: Members can share important locations or coordinates, which is very useful in positioning ships or responding to incidents at sea. 9. File Sharing: Allows members to share documents, action plans, reports, or other important files needed in operations.10.Mute and Block Users: This facility allows members to lock or block annoying orunwanted users in the application.Bakamla Messenger is a newly developed application; therefore, an analysis is needed to determine the advantages, disadvantages, and level of user satisfaction with using the application.

5 Figure 2 .
Figure 2. Display of the Bakamla messenger application

Figure 4 .
Figure 4. Data collection process with UEQ

Figure 6 .
Figure 6.Data processing process with UEQ

9 Figure 8 .
Figure 8.Comparison results with the Benchmark Scale

Table 1 .
Interval Benchmarks for the UEQ Scale

Table 2 .
Results of User Experience Questionnaire Experience (UEQ)