Reddit.com is a social media aggregation, content rating and discussion website. Registered users may post interesting links, texts, images that can be voted up or down by other users. Posts are organized by subject into user-created boards called "subreddits", covering a variety of topics including news, science, movies, video games, music, books, fitness, food, and much more.

The interface of Reddit's website was recently redesigned, and we set out to determine whether its new design helps users perform typical Reddit tasks more efficiently than the old design. The old interface is available at www.old.reddit.com, at the time of testing, and the new site is www.reddit.com (alternatively www.new.reddit.com).


To determine which Reddit design users can use more efficiently, we narrowed our objective to the following typical Reddit user functions:

1. Which interface is better to submit a post? (Can users create and submit a post with images efficiently?)

2. Which interface is better for commenting? (Can users efficiently submit a comment? Can users efficiently collapse a comment thread?)

3. Which interface is better for searching and saving a post? (Can users efficiently search and save a post?)

4. Which interface is better for finding a category or board ("subreddit") within Reddit? (Can users find a particular category/board ("subreddit") efficiently?)

Summary

We conducted a usability test with 16 representative users on five typical Reddit user task combinations: create and submitting posts; formatting post comments; searching and saving articles; collapsing comment threads; and searching for the board of popular content (r/popular). We evaluated the efficiency of performing tasks on Reddit's old and new designs by measuring task completion rate, time to successfully perform task, and user's perceived ease for performing the task.

Our findings suggest that new Reddit design does allow users to perform these common tasks more efficiently than the old design. There were no significant differences in successful task completion among the participants. However, we found statistically significant evidence that participants performed tasks (4 of 5 tasks) on the new design more quickly. We also found statistically significant evidence that participants perceive tasks (3 of 5 tasks) as easier on the new design.

This report details the comparative results for the old and new designs of Reddit.com on each of the five typical user tasks. We identify usability issues that compromised user efficiency in the old design and highlight the design strengths of the new Reddit version. Although findings suggest the new design is more efficient, we identified several usability issues in the new design that affected user performance: no error feedback in comment post form; journeying outside of Reddit to view articles; labeling commands in collapsing/expanding comments; and confusion with the hierarchical structure of Reddit while performing searches. We also provide recommendations to address these issues in the new design.

Usability issues in the new design are rated according to their level of severity and our recommendations are provided in consideration of an issue's severity level:

- High Severity: Potential fatal issue that prevents the user from using the system and greatly compromises the user experience.
- Medium Severity: Issue compromises the user experience although the user may still be able to operate the system.
- Low Severity: Issue may impact the user experience for some users.

Participants
We recruited a convenience sample of 16 representative users for the study based on the following screening criteria:

1. Adults 18 years of age or older

2. New or inexperienced Reddit users:
- New: Never used Reddit and does not have a Reddit account
- Inexperienced: Only logged in to account a couple times and no more than 10 posts

3. Self-reported familiarity with Chrome browser (somewhat familiar or very familiar)

Participants were screened to include only new or inexperienced Reddit users to minimize the potential impact of previous Reddit experience on participant performance and avoid any bias criticisms regarding the two versions of Reddit. Inexperienced Reddit users were included with new Reddit users due to the difficulty of finding only users that have never visited/logged on to Reddit at least a couple times.

Task order was not varied as the tasks built off of one another. Reddit version presentation order was counterbalanced across participants to account for possible ordering effects. Task 2 statement presentation order was also counterbalanced across participants and within version presentation order to account for possible impact of typing the provided statement in a timed task. Counterbalanced version presentation order and statement presentation order was also evenly distributed (further counterbalanced) across four lead moderators to control for possible moderator bias.

After completing a study, the moderator determined whether the participant successfully completed each task according to the following criteria:

- Task Success: Participant completes all parts of task on own within 2 minutes
- Task Failure: Participant did not complete all parts of task within 2 minutes

After all testing was completed, all participant task completion data was compiled by version and task for analysis.

Time on Task and Ease of Use Rating
Task times for each participant, version, and task were recorded by moderator. Task ease ratings were marked by participants on handouts for each version and task. After participants' tests were complete, all participant task times and task ease ratings were then compiled by version and task for analysis.

After participant tests were completed, averages were calculated for completion rates, time on task, and ease of use ratings for tasks on both Reddit versions. Data for participants was removed from time on task and ease of use rating task averages if they had failed that task. Fisher's Exact Test was used to compare task completion rates at alpha level .10. Paired t-tests were used to compare time on task and ease of use ratings at alpha level .10. As the study objective is to determine whether Reddit B (new) is a more efficient design, significance was determined by referencing one-tailed p-values.

Observed participant behavior and interview comments were reviewed and analyzed using observation notes and recordings to determine:

- how differences between Reddit versions and possible usability issues impacted participant ability to timely complete tasks

- reasoning behind participant task ease ratings for the two Reddit versions

- why participants might prefer one Reddit version over the other

For Task 1, there was no difference in participants' ability to successfully create and submit a post on either Reddit design. We conducted Fisher's Exact Test to compare the Task 1 completion rate with the old Reddit design (75%) to the Task 1 completion rate with the new Reddit design (69%) and found no statistically significant difference at alpha level .10. However, participants were able to complete the task more quickly with the new design and also rated the new design as easier to use than the old design. A paired t-test was conducted to compare the time on task for Task 1 between the old and new Reddit designs. We found a significant difference (t(9) = 2.33, p < .10), such that participants took longer to complete Task 1 when using the old design (M = 80 s, SD = 26.3 s) than when using the new design (M = 58.9 s, SD = 22.8 s). We also conducted a paired t-test to compare participants' ease of use ratings for task 1 between the old and new Reddit designs. We found a significant difference (t(9) = 3, p < .10), such that participants rated the new design as easier (M = 1.7, SD = 0.9) than the old design (M = 2.8, SD = 1.3) for performing Task 1.

Due to issues with the labeling and positioning of the button for creating a post in the old design, participants found creating a post more efficient on the new design. In the old design, 10 out of 16 participants reported or were observed to have had problems with finding the proper link to create a post with an image. There are two links that can be used to post, but to post in different formats. Users do not know this until after they click their selection; labeling is problematic. One participant said, "I thought that uploading an image would be a text post and within the text you could upload an image. So, first I created a text post. And then when I didn't see anywhere to attach the image I saw it was in link. [...] I thought it was a little hard." Another participant complained, "I wish it would tell you the kinds of content you can post, like on tumblr, there's text, photo, audio, video..."

7 out of 16 participants reported or were observed to have had problems with the positioning of the button to create a post with an image. They found it strange and non-intuitive that the button would be on the right side of the page, amongst pop-up advertisements, making it difficult to locate the button. Many participants reported or were observed not even looking on the right side because they had assumed that everything on the right was advertisements. One participant commented, "I didn't like that it was underneath an ad. I had no clue because I don't even look at the ads." Another participant reported, "I didn't look at the right-hand side because it seemed at first to be exclusively ads so I didn't bother to even look over there. That took a while. If you're going to have generally ads on one side, don't switch it over all of a sudden to be something that's useful." Another participant speculated as to why the mixture of pop-up ads and critical Reddit commands on the right side of the screen might exist, "As a user I think that's dumb, but I guess smart on advertisers so you can accidentally click on them."

Conversely, in the new design, participants appeared to perform the task faster as there was only one button for creating a post, and it was clearly labeled: "Create post". Despite the button's location on the right side of the page, participants' higher average ease of use rating for the new design suggests that the task was still perceived as easier on the new design. Participants' (6 out of 10) reported perceptions on the task and button for creating a post support the ratings and findings that the new design is perceived to be easier for Task 1, based on the below participant commentary:

- "Much easier and obvious, and more streamlined. I liked how choosing the subreddit to post to was the first thing I saw and decided on, which is nice. It also clearly says 'Post image'."

- "I like that there's a universal icon at the top next to the username that indicates 'create a post.'"

- "I thought it was easy, a lot like social media."

- "Easy to see and find what to do on the page."

- "Easy to see where to begin, what to click on."

- "I like this. Post types are labeled, just as I expected."

Despite the evidence of a more efficient performance overall on the new design for Task 1 in terms of time on task and perceived ease of use, 31% (5 out of 16) of the participants failed to perform the task on the new design. Almost half of the participants (7 out of 16) reported or were observed having difficulty with finishing the actual submission of the image post after it had been created. This is because the submit button was greyed out whenever the title field was empty. However, there was no error message or denotation that the title field was required. Hence, several participants failed the test or gave up in frustration simply because they could not figure out how to "un-grey" the submit button, even though they had done everything correctly up until then. We rate this issue as High in severity as it will greatly limit a user's ability to interact with Reddit via the core operation of posting. Participant comments during debriefing interviews reveal that this was a source of frustration on the new design:

- "So I clicked on 'Create Post' and I uploaded an image, but then I couldn't post it. The post button wasn't working. That was frustrating. I didn't know how to create my post."

- "I don't like that it wouldn't let me click Post until I added a title. It should've told me that it's a required thing or that I can't post until there's a title for it."

- "The only thing I stumbled on, I didn't give it a title. Once I gave it a title, the Post button activated."

Fix the Post form. Add required field indicators and guide users with meaningful error messages. It is not enough to deactivate and grey out the Post button until all information is entered without more guidance. Look to old design and other industry-standard error handling strategies for required field indicators and error messages, such as asterisks and dialog boxes. If post form in the old version is utilized as a model for the new version, then improve upon the generic error message given in the old design.

Similar to Task 1, there was no difference in participants' ability to successfully format a post in Task 2 between the two designs. We conducted Fisher's Exact Test to compare the Task 2 completion rate with the old Reddit design (94%) to the Task 2 completion rate with the new Reddit design (94%) and found no statistically significant difference at alpha level .10. However, participants performed the task faster and perceived the task as easier on the new design for Task 2. We found a significant difference (t(14) = 3.4, p < .10) while conducting a paired t-test to compare the time on task for Task 2 between the old and new Reddit designs, such that participants took longer to complete Task 2 when using old design (M = 52 s, SD = 24.8 s) than when using the new design (M = 36.8 s, SD = 14.2 s). We also conducted a paired t-test to compare participants' ease of use ratings for Task 2 between the old and new Reddit designs. We found a significant difference (t(14) = 2.38, p < .10), such that participants rated the new design as easier to use (M = 1.3, SD = 0.6) than the old design (M = 2.1, SD = 1.3) for performing Task 2.

For Task 2 on the old design, half of the participants reported or were observed to have had problems operating the text editor to stylize their text. Many complained that it was not what they were expecting because the comment box was a plain text box without the typical text customization options that are usually visible to the user. The accessibility to the Formatting window is inconvenient and an unnecessary extra step, and the actual act of manually bolding the word in the text was confusing. This extra step and the cognitive tax it places on a user is the likely source of the longer Task 2 completion time for the old design.

In contrast, the new Reddit design presents text formatting options as readily accessible icon buttons along the bottom of the comment field. This design better comports to user expectations as exemplified by participant debriefing interview comments which likened it to previous experiences with Microsoft word or other comment driven applications.

We recommend keeping the comment formatting options design as is in new design.

There was no difference in participants' ability to successfully perform Task 3 between the old and new designs. We conducted Fisher's Exact Test to compare the Task 3 completion rate with the old Reddit design (88%) to the Task 3 completion rate with the new Reddit design (88%) and found no statistically significant difference at alpha level .10. There was also no significant difference between the time on task for the old design (M = 54.4 s, SD = 33.2 s) and the new design (M = 46.3 s, SD = 30.6 s) on Task 3 upon performing a paired t-test at alpha level .10. We also conducted a paired t-test to compare participants' ease of use ratings for Task 3 between the old and new Reddit designs. We found a significant difference (t(13) = 2.51, p < .10), such that participants rated the new design as easier (M = 1.8, SD = 0.7) than the old design (M = 2.4, SD = 0.9) for performing Task 3.

For Task 3 qualitative analysis, see Task 5 qualitative analysis, as the findings and recommendations are very closely linked.

There was no difference on the Fisher's Exact Test to compare the Task 4 completion rate with the old Reddit design (63%) to the Task 4 completion rate with the new Reddit design (50%). We found no statistically significant difference at alpha level .10. On the paired t-test, while comparing the time on task for Task 4 between the old and new Reddit designs, we found a significant difference (t(7) = 1.52, p < .10), such that participants took longer to complete Task 4 when using the old design (M = 39 s, SD = 27.7 s) than when using the new design (M = 21.8 s, SD = 13.2 s). We conducted a paired t-test to compare participants' ease of use ratings for Task 4 between the old and new Reddit designs. There was no significant difference between participants' Task 4 ease of use ratings for the old design (M = 1.8, SD = 0.7) and the new design (M = 2.3, SD = 0.9).

Within the old version of Reddit, 7 of 16 participants were observed to have clicked on the article first, figuring that it was the proper way to read the article and ultimately view the comments on it. However, clicking on the article brings the user to the original source of the article, in a separate tab away from Reddit. Thus, they cannot read the article and then view the comments of the article within the same tab. This resulted in a very disorienting user experience. One participant succinctly illustrated this disorientation: "No comments on this... Is it on the Reddit thread?"

The new version of Reddit attempts to improve this problem through an in-page popup. However, there could still be some additional improvements. The new Reddit design uses an in-page popup feature to show the Reddit thread of a post. In this case, clicking the post title opens an in-page popup containing the Reddit user interactions (comments, points, etc.) and a link to the original source. Clicking on the source article link opens the article page on a new tab and leaves the Reddit interaction page in a separate tab. In old Reddit, clicking the post title takes the user to the source link in the same tab that they were browsing Reddit so they lose their place and flow by having to backtrack between the Reddit page and source link. New Reddit improves that experience by making that easier for users since they don't lose their place when browsing and makes the flow smoother. However, for new users, the source link opening in a tab could surprise them, making it a minor inconvenience.

To provide the user—especially a new Reddit user—with additional context as to what is happening once they click on the article's source link, a dialog can appear informing the user, "Redirecting to (article link) in new tab…" with an OK button and a checkbox then can mark that says, "Do not show this message again". This provides some additional context for what is happening within the user’s browser, so they can easily find their way back to the Reddit interface and feel less disoriented by alleviating the sudden jump from Reddit to an outside source link.

This section discusses findings that were not particularly borne out in the quantitative data of Task 3 and Task 5, but were unmistakably so in the qualitative data and post-test interviews.

During testing of the old and new version of Reddit, there was obvious confusion and disorientation amongst many participants as to where they were located within the site any given moment. During searches, many questions were surfacing from participants during and after the tests: Were they in the global Reddit level, or were they in a subreddit? Does this impact search results? What exactly is a subreddit? In many participants' cases, despite successfully using the search bar and pull-down menu to accomplish Task 3 and Task 5, there was a significant number of behavioral observations that reflected the above questions. Frustrations participants gave were:

- "I thought it would give me r/science as a suggestion when I was searching in the search bar. I didn't realize it would search that term in the subreddit I'm currently in."
- "I think r/popular is not an official reddit subscription. It's kind of like a hashtag...I feel it's like a hashtag [...] r/popular can be found in any posts, so I couldn't find it, I couldn't search for it. When I finally clicked on a post, I found it. It wasn't that easy to figure out."
- "Super hard! Subreddit, I was trying to read what it is, because I'm not familiar with the lingo."

- Have a mini-tutorial for a new user so they can understand how Reddit is set up and how to navigate and use subreddits and their subscriptions. For example, giving users an explanation on navigating, searching, and browsing through Reddit as a whole, subreddits, and their subscribed subreddits. This way it will be easy for users to identify and recognize site areas within Reddit.

- Revert to the old Reddit component within the Search bar in which the user is specifically being told whether or not they are currently searching global or within a subreddit, and the checkbox option is clearly shown so that the user can make the choice of whether to search global or within a particular topic.

- Alternative Option: When the user clicks inside the Search bar, a temporary pull-down menu appears asking where exactly they would like to search (e.g., All of Reddit, Subreddit A, Subreddit B, Subreddit X, Subscriptions, etc.)