GO TO PARENT TASK ↗

---

Table of contents

1. Participants
2. Summary
3. Detailed analysis

1. Participants

• AR Female (21, Morocco)
• FR Female (29, France)
• HI Female (21, India)
• JA Female (24, USA)
• ID Male (27, Indonesia)

2. Summary

😍 – What works better than before?

1. Understanding of article talk pages: 5/5 understood that article talk pages are about discussions related to the article. The more conversational design we choose seems to work.

2. Understanding of the diff view: 4/5 understood what the diff is all about. The recent design updates improved understanding. The strike-through and summary design updates for additional clarity made a difference.

3. Performance: There were drastic improvements in the perceived performance when publishing messages. It much more feels like instant publishing than before.

4. Orientation: People navigate and orient themselves perfectly in the new modernized interface for talk pages. The continuity between the notifications home page and talk pages works. Comparing it with the before state, the new design has been a huge improvement.

5. Notifications discovery: 5/5 had no issues finding notifications.

😡 – Where is room for improvement?

1. Talk page discovery (T321076):
   • Problem: Participants associate Wikipedia mainly with editing articles, also when asked to discuss edits. They had issues finding the talk page entry point. Most used the edit pencil.
   • Solutions:
     1. Add the “Discuss edits” item to the menu when tapping the first edit pencil
     2. Add a discovery tooltip for the talk pages option in the overflow menu
     3. Introduce the talk page bubble next to the article title (T296084 | Figma)
     4. Introduce it as a default in the bottom toolbar
2. Orientation (T321263):
   • Problem: Certain notification types take users outside the app with a different design (e.g. Wikimedia Commons). This confused participants.
   • Solution: Make sure to keep users in the app at all times when they tap notifications (use mobile web view. In the test, notifications coming from Wikimedia Commons took users outside the app.
3. Understanding of user talk pages (T297865):
   • Problem: 3/5 had issues understanding user talk pages when coming from notifications. From observing the tests, participants didn’t ultimately realize that it is a different space *(RS: I’m wondering how important the distinction is though since users did not have any issues replying to user talk page messages from our new notification center)*
   • Solution: Contextual tooltip onboarding when users interact with a user talk page for the first time, e.g. when receiving a user talk page message. We should explain the purpose of a user talk page clearly and concisely.
4. The publicness of talk pages (T297865):
   • Problem: 2/5 described user talk pages as direct messages or private messages. These types of messages are usually private, and can’t be seen by anyone.
   • Solution: Educate users that talk page messages are public. E.g. when they first post a message on a user talk page. This can be achieved with a dialog that contains a “don’t show this message again” checkbox.
5. Understanding of a revert (T297865):
   • Problem: Even though participants understood the diff view, they needed a lot of time processing the task that uses the word “revert”.
   • Solution: Use more newcomer-friendly language for “revert” or explain a revert with contextual onboarding.
6. Cache answers when creating new topics (T321269):
   • Problem: A participant accidentally pressed back while creating a new topic. The subject and description were deleted after going back and creating a new topic
   • Solution: Make sure to cache/store the contents of an input field for a specific session to avoid the loss of the user’s input.
7. The reply button is not shown in the Japanese test (T321268):
   • Problem: Reply button within a talk page thread is not shown in the JA test.
   • Solution: [SPIKE] to investigate why the ‘Reply’ button is not shown.
8. The ✓-icon to publish a topic might be unclear to some users (T321270):
   • Problem: The ID participant had issues identifying ✓ at the top to confirm the input.
   • Solution: Change the ✓ button everywhere in the app to a label like PUBLISH, as an icon might be hard to understand in/for multiple cultures.
9. Meta-level improvements for the next protocol (Comment: we did it the same way as last time for comparison reasons)
   • Problem: Participants had issues processing the task description (English) into their other native language they speak.
   • Solution: Translate the English usability protocol into the participant's other native language.

3. Detailed analysis

AR Female

• 😡 Article text at the top is not shown in dark mode
• 😡 Misunderstands task (edit instead of discuss)
• 😡 Taps article edit button to discuss first
• 😍 Uses article footer talk page entry point
• 😍 Understands talk page concept, compares it with Quora.
• 😍 Likes that people can “criticize” content on Wikipedia through this feature
• 😍 Has no issues starting a discussion. Everything goes as expected
• 😍 Knows that other people can join a discussion
• 😍 Understands a discussion thread
• 😍 Has no issues replying in a thread
• 😍 Has no issues navigating to notifications
• 😍 Uses search functionality intuitively
• 😡 Does not understand the edit reverted notification at first
• 😡 Has trouble understanding the diff screen
• 🤔 Does not completely understand the difference between user and article talk pages. But realizes it’s something different
• 😍 Likes article talk pages
• 😡 It seemed like she understood the concept of article talk better than user talk pages.

FR Female

• 😡 Misunderstands task (edit instead of discuss)
• 😡 Taps article edit button to discuss first
• 😡 Taps the article edit button to discuss
• 😍 Compares editing Wikipedia to editing content on a blog
• 😍 Understand the concept of an article talk page (a forum to discuss how to make the article better).
• 😍 Finds article talk pages really cool
• 😍 Has no issues starting a discussion.
  • 🤔 Her previous input got deleted *(RS: can we cache the subject and message? Also, should we ask if people are tapping the back button and have stuff written?)*
• 😍 Understands a discussion thread
• 😍 Has no issues replying in a thread
• 😍 Finds notifications right away
• 😍 Understands the revert notification
• 😍 Understands the diff
• 😍 Understands user talk pages and the difference between article talk pages

HI Female

• 😡 Misunderstands task (edit instead of discuss)
• 😡 Taps the article edit button to discuss first
• 😡 Does not find the entry point for discussion by herself
• 😍 Uses article footer talk page entry point after being prompted
• 😍 Has no issues starting a discussion.
• 😍 Understands that the talk page is related to the Cat article
  • 🤔 Thinks of a talk page more like a discussion forum about Cats
• 😍 Has no issues replying in a thread
• 😍 Understands the revert notification
• 😍 Understands the diff. Mentions though it’s a bit confusing.
• 😡 Diff empty spaces/lines are white in dark mode
• 😡 Probably doesn’t completely understand the difference between user and article talk pages
• 😡 Mentions that the test has been a bit hard

JA Female

• 😡 Taps the article edit button to discuss first
• 😍 Uses article footer talk page entry point
• 😍 Has no issues starting a discussion
• 😍 More or less understand the concept of an article talk pages
• 😡 Does not see the “Reply” button in a discussion thread
• 😍 Understands the revert notification
• 😍 Understands the diff
• 🤔 Not sure if they’re aware of a user talk page (also, the participant tapped on a Commons talk page notification)

ID Male

• 😡 Does not understand the task of accessing talk pages
• 😡 Does not find entry point to access a talk page at all
• 😍 Has no issues starting a discussion
• 🤔 Suggests changing the ✓ button to DONE or CONTINUE
• 😍 Understands the discussion thread/talk page purpose
• 😍 Has no issues replying in a thread
• 😍 Understands the revert notification
• 😍 Understands the diff
• 😍 Understands the difference between article and user talk pages. Describes user talk pages as “private messages”.
  • 🤔 We need to make sure that people are aware of the fact that talk page messages are public

---

VIEW VIDEOS AND MORE ON NOTION ↗

You can request access to the document here.