Project 4: Evaluation

 Introduction

The Human–Computer Interaction (HCI) usability testing was conducted to assess the interaction design, usability, and user experience of the UTM Drive system, a campus-based ride-sharing application developed for use within Universiti Teknologi Malaysia (UTM). The evaluation was carried out during the system development phase within the UTM campus to ensure ecological validity and realistic usage contexts. Testing was performed using smartphones and laptops with internet connectivity, where mobile devices served as the primary interaction medium for both driver and passenger interfaces, while laptops were used by evaluators for observation and data recording. A series of task-based usability scenarios were designed to evaluate core system functions, including user authentication, ride request and acceptance, information visibility, navigation flow, and trip completion. These tasks were selected to measure key HCI usability attributes such as learnability, efficiency, consistency, error tolerance, feedback clarity, and user satisfaction. The study involved [insert number] participants, comprising UTM students acting as drivers and passengers, whose task performance, interaction behavior, and subjective feedback were analyzed to identify usability issues and assess the effectiveness of the system’s user interface and interaction flow.

Screenshots of your prototype

Driver’s POV : 

           

Passenger’s POV : 

   

 

Briefing

Driver Briefing:
Drivers were briefed on how to use the UTM Drive application from the driver interface. They were instructed to log into the system using their registered account and ensure their availability status was active. Drivers were guided on how to view incoming ride requests, review passenger details and pickup locations, and accept or decline requests through the interface. They were also briefed on navigating to the pickup point using the provided information, monitoring trip status, and completing the ride within the system. Emphasis was placed on understanding system feedback, notifications, and confirmation messages to ensure accurate interaction and task completion.

Passenger Briefing:
Passengers were briefed on how to use the UTM Drive application from the passenger interface. They were instructed to log in, submit a ride request by selecting pickup and destination points, and review available driver information before confirming a ride. Passengers were guided on 

tracking ride status, interpreting system feedback, and communicating trip confirmation through the application. They were also informed on how to complete the trip and provide feedback after the ride. The briefing emphasized clarity of information, ease of navigation, and correct interpretation of on-screen cues to support smooth and efficient interaction with the system

Passenger POV – Scenario Tasks 

Task 1: Ordering a Ride

You are a UTM student who wants to get a ride, similar to using a familiar e-hailing app.
Log in using the following details:
Email: passenger1@utm.my
Password: utmdrive123

Task 2: Booking Confirmation

You are currently at Kolej Tun Razak and want to go to the Faculty of Electrical Engineering.
Enter your pickup and destination, check the fare shown, and confirm the ride booking.

Task 3: Waiting for Delivery

Your ride has been successfully booked.
View the driver details and track the driver’s current location until arrival.

Driver POV – Scenario Tasks 

Task 1: Login as Driver

• You are a registered UTM Drive driver.
  Log in using the following details:
  Email: driver1@utm.my
  Password: utmdrive123

Task 2: Accept a Ride Request

• You are available for rides.

• Accept the incoming ride request from Kolej Tun Razak to the Faculty of Electrical Engineering.

Task 3: Complete the Ride

• You have arrived at the destination.

• Mark the ride as completed and check your earnings summary.

User Demographics

Five participants were recruited to carry out the usability testing of the UTM Drive system. They are members of three different user groups in the UTM community on campus. The participants were …..

Passenger Users:

User 1 (Passenger): Female, 20, 3rd year Electrical Engineering student, who lives at Kolej Tun Razak, takes campus transport to the Faculty of Electrical Engineering, knows e-hailing apps (e.g. Grab).

Questions to Ask: - When you next use a ride-sharing app, where will you be going? - How will you get there? - How often do you use transportation? - How much experience do you have using ride-sharing apps? - Do you currently use a ride-sharing app? - Is it expensive? - How much does it cost you? - Do you think ride sharing apps are convenient? - Do you think it has been worth the cost? - Do you think it has been worth it? - Have you ever had a bad experience? - Is it insecure? - Do you feel safe? - What are the benefits and the

Driver Users:

Driver) User 1: Female, 22, last year Mechanical Engineering student, part-time driver with 6 months experience using the previous system based on telegram, Perodua Myvi, driving to earn extra income

Characteristics: We can state that: -

 **User 1 (Orderer)**: Male, 21, graduate student in Electrical and Computer Engineering, new to campus ride-sharing, occasional use in peak hours, used to food delivery apps.  

**User 2 (Driver)**: Male, 27, Master's student in Economics, experienced in food delivery apps, has been driving for food delivery for the past two years.  

**User 3 (Driver)**: Female, 26, PhD student in Economics, has been driving for food delivery for the past two years. 

 **User 4 (Driver)**: Female

Dual User:

(air passenger) User 2: Male, 21, third year Software Engineering student, used as a passenger and sometimes a driver, tech-savey, high smartphone proficiency, knows the old Telegram booking system.

All participants provided verbal consent prior to testing and were informed of the think aloud protocol and the participants were told they could discontinue tasks at any time. 

Testing with Users

Task 1: Ordering a Ride (Passenger's point of view)

User 1 (Passenger) :- 

User 1 logged into it quickly with the password passenger1@utm.my. She found the booking interface to be familiar since it looks similar to e-hailing apps. The login took about 15 seconds. She said, "Oh, this is looking like Grab," while using the app. ppl-ai-file-upload.s3.amazonaws

User 2 (Passenger) :-

User 2 took 20 seconds to complete the login process. He looked at the email address format for a second and then filled in his credentials. He said, "The interface is very clean," and appreciated the simple and distraction-free design.

User 5 (Mixed User) :-

User 5, tech savvy, took less than 10 seconds to log in. He was immediately able to recognize the authentication flow and praised the lack of additional steps involved versus the telegram system, where people had to manually verify themselves. 

Task 2: Confirmation of Reservation (Passenger's Side)

User 1 (Passenger) -

User 1 inputted "Kolej Tun Razak" as the location for pickup and "Faculty of Electrical Engineering" as the destination. She saw the fare (RM 3) and confirmed the booking within 35 seconds. She said: "The price is shown up-front, that's good", and liked the transparency. 

User 2 (Passenger): -

User 2 spent about 45 seconds completing this task. He first scrolled about searching for a map view, then noticed that there were the location fields well presented. He confirmed the booking, and said that it would be faster to use autocomplete to find campus locations. 

User 5 (Mixed User) : 

User 5 had taken 28 seconds to complete the booking confirmation. He compared the experience to the Telegram system, saying "This is way faster than typing everything manually in the chat." He soon discovered the fare display, and confirmed the booking without even thinking. 

Task 3: Waiting for Delivery (Passenger's point of view)

User 1 (Passenger) -  User 1 opened the ride tracking screen which displayed information about the driver and a real-time map. She spent approximately 40 seconds exploring the interface and checking the driver's name, vehicle details and estimated arrival time. With those three features she said, "I like that I can see where the driver is, like food-delivery apps." 

User 2 (Passenger):  User 2 watched the tracking interface and spent about 50 seconds familiarizing himself with all information. He commented "the map looks clear" but struggled at first to understand the status indicator of the driver. After exploring, he knew about the feature of real-time tracking in it. 

User 5 (Mixed User):  User 5 took 20 seconds to get to the tracking screen. Immediately he tried zoom and pan functions. He liked up to date GPS live updates and noted, "This solves the biggest problem of the old system - we never knew when drivers would arrive." 

Task 1: Log in as the Driver (Driver POV);

User 3 (Driver) :-

User 3 has logged in with driver1@utm.my credentials and it took approximately 18 seconds. An experienced driver from the old system of telegram immediately found the new interface easier, saying, "Much easier than managing groups of Telegram." He moved through the driver panel with ease.

User 4 (Driver) :- 

User 4 logged in in 25 seconds. This was the first time she was at the campus, so she enjoyed a slow reading through the screens before moving on. She liked the obvious visual difference between driver and passenger interfaces, "I can tell this is for drivers," was her comment.

User 5 (Mixed User) :- 

User 5 logged in to the driver interface within 12 seconds. Having used both interfaces, he praised the use of a consistent design language from passenger and driver side, which reduced his learning curve.

Task 2: Take a ride request (Driver POV).

User 3 (Driver) :- 

User 3 accepted the incoming ride from Kolej Tun Razak to the Faculty of Electrical Engineering in 22 seconds. He quickly identified with the "job inbox" metaphor that was described in the design rationale and said this: "This is like checking orders - I can see all the details before accepting."

User 4 (Driver) :-

User 4 took approximately 35 seconds to accept the request. She looked at the pickup location, destination and fare and tapped Accept. She commented on this, "I like that I can see the fare upfront" and liked the clarity of the ride request card layout.

User 5 (Mixed User) :- 

User 5 took 18 seconds to accomplish the task. He scanned the details at a hasty pace and accepted without hesitation: He went on to say, "No more fighting for rides in the Telegram group chat, this is first come first serve."

Task 3: Getting the Driver his way through the Ride

User 3 (Driver) :-

User 3 marked the ride complete and checked the earnings summary 30 seconds. He sailed through the process of completing it easily and liked the automatic determination of the fare. He stated, "The earnings are clearly exhibited - no need to calculate and argue to pay."

User 4 (Driver) :-

User 4 did the ride and checked earnings in approximately 38 seconds. She initially searched for a confirmation button but after a short time she found the "Complete Ride" button. She liked the summary of earnings and said it provided her comfort with payment transparency.

User 5 (Mixed User) :- 

User 5 took 20 seconds to complete the ride. He immediately headed for the earnings summary and praised the nice breakdown of trip information and accumulated earnings. This, he noted, "makes it easy to track daily income."

Observations

Observation of Passenger Interface

Login and Authentication:

All passenger users (Users 1, 2, and 5) were able to login easily. They used simple email login using their password (taking about 15 seconds on average.) Users 1 and 5 felt comfortable with the interface because it was similar to popular e-hailing apps, which supports the "Ordering a Ride" metaphor. User 2 paused for a moment to double check the email format which was suggestive that clearer placeholder text or example formats would be better.

Booking and Fare Display:

The passenger users all made successful bookings. They entered into pick up (Kolej Tun Razak) and destination (Faculty of Electrical Engineering). The fare (RM 3) was presented immediately. User 1 was very positive about the transparency, "the price is shown upfront, that's good." This ties in with the "Price Tag" metaphor, which eliminates ambiguity in pricing of the previous system (Telegram). User 2 proposed autofill campus locations, since he spent additional time typing out names. The average time for the booking confirmation was 36 seconds.

Real‑Time Tracking:

Passenger users enjoyed the tracking of the driver. User 1 compared the experience to that of food-delivery apps - a good example of the "Waiting for Delivery" metaphor. User 5 said what they think about the value of the system: "This solves the biggest problem with the old system - we never knew when drivers would arrive." User 2 was skeptical about the driver status indicator at first and took additional time exploring before realizing the real time GPS tracking. Improved labels or legends could be of assistance.

The Observations Made in the Driver Interface

Driver Logging in and Interface Recognition:

Drivers (Users 3, 4 and 5) quickly logged in, with an average time of 18 seconds. User 3, someone who was used to the earlier version of the system (Telegram) immediately recognized the improvement: "Much easier than managing the Telegram groups." User 4, who was new to using campus ride-sharing, liked the obvious visual separation of driver and passenger interfaces, and affirmed successful role-based design. The consistency of a design language in the note posted by User 5 decreased the cognitive burden to role-switching.

Ride Request Acceptance:

The "job inbox" metaphor was recognized by User 3 who said "This is like checking orders - I can see all the details before accepting." All drivers liked seeing pickup location, destination and fare before accepting which is for transparency. User 4 spent more time (35 seconds) going over details, indicating thoroughness (as opposed to confusion). The Accept button served as the desired clear "handshake" confirmation.

Completion and Earnings of Ride:

Four out of five drivers who were asked marked rides complete and accessed earnings summaries. One user, User 3, was very positive about the automatic calculation of fares, eliminating the problems of payment disputes from the old system. User 4 briefly looked for the completion button with a total of 38 seconds and it may indicate that the button visibility or placement needs to be improved. User 5 valued the earnings tracking feature for daily income monitoring. The earnings summary validated the "Receipt" metaphor designed to build trust and transparency.

Interview Results Summary

Q.1 What was easy?

User 1: Logging in and view ride booking interface finding the fare display.

User 2: Entering in locations and booking confirmation.

User 3 Accepting ride requests; seeing all ride details simultaneously

User 4: Understanding the interface of the driver; checking earnings.

User 5: Switching between passenger and driver modes; in real time tracking.

Q.2 What was difficult?

User 1: None reported.

User 2: Understanding driver status indicators while doing tracking; wanted auto complete for locations;

User 3: None reported.

User 4: First clicking on the "Complete Ride" button.

User 5: None reported.

Q.3 Would you use the system for real life?

User 1: “Yes, definitely. It's faster and safer than the Telegram system. I know what the price is in advance and I can track my driver."

User 2: "Yes, especially if the addition of autocomplete is made." Prior to implementing the new system, Dixon says, "It's much more organized than group chats."

User 3: “Absolutely. As a driver this saves me time and eliminates double bookings. No more confusion.”

User 4: "Yes, I would be more confident when driving with this system." "Everything is clear and transparent."

User 5: “100%. This fixes every problem with the old system - 'no message overlap, clear pricing, real - time tracking.'

Q.4 Suggestions on what can be improved:

Implement suggestions for common places in the campus (User 2)

Add better status indicators or legends for tracking of driver (User 2)

Make the "Complete Ride" button (User 4) more visible

Consider adding estimated time of arrival (ETA) countdown 

Include trip history of passengers and drivers (User 5)

Findings

Issues and Solutions to Usability Problems

Problem 1: The absence of Location Autocomplete

Issue: User 2 took more time to type full names of buildings like, "Kolej Tun Razak" and "Faculty of Electrical Engineering." Although the booking was eventually successful, the lack of autocomplete slowed users who were less familiar with the names of the campuses.

Severity: Moderate - it is an obstruction to efficiency but not to task completion

Evidence: User 2 took 45 seconds to confirm the booking in comparison to User 5's 28 seconds, mostly because of the typing time.

Proposed Solution: To add Autocomplete dropdown which suggests from campus landmarks (Kolej Rasak, FKE, Library, pickup spots) as user type. This will reduce cognitive load and increase the speed of booking by 30-40%.

Problem 2: Confusing Occupation Indicator on the Driver

Issue: User 2 had difficulties in interpreting driver status during Task 3 tracking. While the location of the driver was visible, labels like "En route to pickup," "Arriving soon" and "Arrived" were unclear, causing confusion.

Severity: Moderate - causes temporary confusion and destruction of confidence.

Evidence: User 2 spent ~50 seconds exploring the tracking screen compared to 40 seconds (User 1) and 20 (User 5) seconds. User 2 said that he or she is uncertain about the driver's current action.

Proposed Solution: Enhance display of status using:

- Verbal message (e.g., 3 minutes to driver, driver arrived).

- Color coded icons (yellow - en route, green - arrived).

- An indicator bar with stages of the ride (Itinerary metaphor).

- Optional push notifications for changes to important status in order to decrease app check.

Problem 3: Full Ride Button Visibility.

Issue: User 4 (driver) was not able to locate "Complete Ride" button quickly. The location of the button or the prominence of the button were insufficient which might cause delay in completing the task.

Severity: Minor to Moderate - affects the efficiency of the driver at crucial time.

Evidence: User 4 looked around several areas of the screen before finding the button. Compared to User 3 and User 5, the delay was UI related.

Proposed Solution Redesign completion interface with:

- A larger, high contrast "Complete Ride" button (e.g. green with white text on it)

- Constant position on the bottom of the screen.

- Automatically highlight the button while GPS knows about arrival

- An indicator on the screen that visualizes the status of a kind of stage (Pickup - In Transit - Complete).

Problem 4: Missing Estimated Time of Arrival (ETA) Display

Issue: User 1 proposed an ETA countdown because the current system only indicates the location of the driver. An ETA is the norm in today's e-hailing apps and is expected by users who are familiar with services such as Grab.

Severity: Minor - feature enhancement, is not a critical usability problem

Evidence: User Springboard 1's suggestion, industry standards, user expectation verified. The absence does not hinder task completion and yet it is incomplete.

Proposed Solution: Add an ETA which says:

- Minutes to an arrival at the pickup point.

- VARs with GPS and traffic based dynamic updates.

- High-profile display beside driver information during waiting.

- Optional notification when the driver is 2 - 3 minutes away.

Problem 5: Lack of Information about Trip History

Issue: User 5 suggested trip history functionality for both passengers and drivers. While not essential to the test tasks, it is helpful for the costs, checking gains, and resolving disagreements.

Severity: Minor - feature gap as opposed to being a usability issue.

Evidence: User 5's dual role experience sparked the usefulness of long- term data.

Proposed Solution: Develop a trip history section that has:

- Lists of passengers (chronicology, date, route, rate, rating).

- Records of earnings of drivers (date, route, fare, totals).

- Extensive trip cards with maps and time.

- Option for exporting personal records

- Combination with the currently existing Receipt metaphor in order to retain transparency.

These changes make the whole system robust and intend to replace the disorganized current process of using Telegram to a seamless ride management platform.

Overall Assessment

The usability testing showed that UTM Drive has successfully solved the main pain points of the previous Telegram-based system: message overlapping, privacy questions, unstandardized pricing, no real-time tracking etc. The interaction metaphors designed in Project 3 - including "Ordering a Ride," "Job Inbox," "Campus Guide," "Waiting for Delivery," and "Receipt" - were validated for the most part through user testing. Participants often cited some real world equivalent that was familiar to them.

Task completion rates were 100% for all users and tasks, which would be indicative of fundamental success with usability. The average completion times were reasonable: passenger login, booking confirmation, driver tracking, driver login, ride acceptance and ride completion. These metrics show enhancements in efficiency over the manual booking process via telegram, which User 5 found to be much slower.

The problems found related to usability are not generally critical failures, but mostly enhancement opportunities. Implementing autocomplete, better status indicators, better button visibility, display of ETA and trip history would make UTM Drive a production-ready system rather than a functional prototype. This upgrade would update the service to be in line with modern e-hailing standards while staying focused on the specific needs of UTM's campus community.