Defining Requirements for Social Web Applications – Part 6: Collective Intelligence

This is the 6th post in a series on Defining Requirements for Social Web Applications. As with previous posts in this series, the content is largely borrowed from Joshua Porter’s book Designing for the Social Web. Porter’s book is a gem, and if the topic of social web design is of interest to you, I highly recommend you pick up a copy.

This post also borrows significantly from Satnam Alag’s book Collective Intelligence in Action.

Click on the following links to access previous posts in this series:

• Part 1: An Introduction
• Part 2: The Framework
• Part 3: Social Objects
• Part 4: Defining Core User Actions
• Part 5: Motivations for User Participation

Introduction

To my knowledge, the term Collective Intelligence was first coined – in the sense we mean it here – in a seminal paper published by Tim O’Reilly titled What is Web 2.0, published in September 2005. In this paper, O’Reilly states the following:

The central principle behind the success of the giants born in the Web 1.0 era who have survived to lead the Web 2.0 era appears to be this, that they have embraced the power of the web to harness collective intelligence

I rather like Joshua Porter’s comments which come close to capturing, IMO, the essence of Collective Intelligence. Porter states that Collective Intelligence is all about:

[Aggregating] the individual actions of many people in order to surface the best or most relevant content. … Collective Intelligence is based on the idea that by aggregating the behavior of many people, we can gain novel insights.

Satnam Alag in his excellent book Collective Intelligence in Action, comments that the Collective Intelligence of Users in essence is:

• The intelligence that’s extracted out from the collective set of interactions and contributions made by your users.
• The use of this intelligence to act as a filter for what’s valuable in your application for a user.

The common thread is “aggregated opinion”. Quoting Porter:

Digg and other aggregation systems rely on the fact that while no individual is right all the time, in the collective a large number of users can be amazingly accurate in their decisions and behavior. Amazon, Digg, Google, Netflix, and many other sites base their recommendations of products, news, sites, movies, etc. on aggregated opinion.

One result of Web 2.0-style applications that use Collective Intelligence is that, to quote Tim O’Reilly, the applications get better the more people use them.

The insights and patterns gleaned from Collective Intelligence are the product of algorithms of various degress of sophistication. Alag lists the following ways to harness Collective Intelligence in your application:

• Aggregate information lists
• Ratings, reviews, and recommendations
• User-generated content: blogs, wikis, message boards
• Tagging, bookmarking, voting, saving
• Tag Cloud navigation
• Analyze content to build user profiles
• Clustering and predictive models
• Recommendation engines
• Search
• Harness external content – provide relevant information from the blogosphere and external sites

Alag comments that:

Web applications that leverage Collective Intelligence develop deeper relationships with their users, provide more value to users who return more often, and ultimately offer more targeted experiences for each user according to her personal need.

Amazon, Yelp, Netflix, Google Search, Google News, Del.iciou.us, and Digg are just some of the more popular sites that leverage Collective Intelligence to target relevant content to their users.

Applying Collective Intelligence in your application

Alag states that there are three things that need to happen to apply collective intelligence in your application.

You need to:

1. Allow users to interact with your site and with each other, learning about each user through their interactions and contributions.
2. Aggregate what you learn about your users and their contributions using some useful models.
3. Leverage those models to recommend relevant content to your users.

Joshua Porter refers to these three steps as:

1. Initial Action
2. Display
3. Feedback

He provides the following table to illustrate the different forms these three steps take at various popular social websites:

Let’s see what Josha Porter has to say about these 3 steps.

Initial Action

The first step is for users to add content. Porter takes Digg as his case study.

On Digg, like on many social sites, you need an account to submit stories. Then, the process of submitting stories has two steps.

The first step is to enter the link you’re submitting. This is a normal URL. You also choose the type of content it is: a news story, image, or video. Digg helps people by providing a nice set of guidelines.

After you click “Continue” in step 1, Digg takes a moment to analyze the line to see if it’s a duplicate. This helps keep the system clean. When Digg thinks you’ve submitted duplicate content, it notifies you that the story has already been submitted.

Porter continues:

If the submission is not a duplicate, Digg analyzes the page and grabs any relevant content from it, including the page title, a description, and any images on the page. It then allows you to choose which elements are appropriate as part of your submission. This step makes it much easier to digg content, as you don’t have to do any heavy lifting of grabbing the content yourself.

Finally, Digg checks to make sure that the submitter of content is indeed a human being.

The initial action on Digg is a crucial step in the system. It determines what content is allowed, makes sure the content is unique, adds data that supports the story, and determines how can and cannot submit content. These decisions act as a barrier of entry to the system. The quality of content Digg that receives entry into the Digg system depends on the checks at this stage.

Adding Tags

Some services allow people to tag content, which allows aggregation of the content in additional, helpful ways. Porter uses the example of Del.iciou.us, which lets you add tags to bookmarks as you enter them into the system.

Aggregate Display

Quoting Porter:

The display of content is crucial to how people will interact with it. If content is displayed prominently then people will consider it more important. Content displayed less prominently will be considered less important.

In general, content is deemed more important when it is displayed:

• On a home page. The home page is visited the most of any page, and therefore it garners the most attention from both site owners and readers.
• More often. The more content is displayed and repeated, the more it is considered valuable.
• At the top of a page. Just like on the front of a newspaper, above the fold is the prime real estate. The top of a web page is where the most important content is placed.
• Higher in ranked displays. When content is ranked, such as in a “most emailed” list, the content at the top is deemed most valuable.

Porter continues:

When content first gets added to an adapative system, it is usually displayed in an appropriately less prominent location. Digg, for example, has what they call an Upcoming page, which displays all new submissions into the system in reverse-chronological order. These freshly-submitted stories stay on the upcoming page a short period of time, getting pushed off in favor of even fresher content. The Upcoming page is crucial to the functioning of the Digg site because it forces each story to gain its own popularity.

All of these stories aspire to reach the Digg home page, the ultimate place for grabbing attention, where they will be seen by thousands of people in a very short period of time. In fact, the burst of attention resulting from being on the Digg homepage often makes the site unreachable. So many people visit the site from Digg that the web server is overwhelmed and either slows to a crawl or breaks outright.

Types of Aggregation Order

Porter goes on to list some of the more popular ways that applications built for collective intelligence display content to their users to ensure that it is relevant and compelling to their audience. These are:

• Chronological order
• Popularity within a time range
• Participant ranking
• Collaborative filtering – filtering content based on your preferences and the recommendations of others
• Relevance
• Social

– displaying content based on who it’s from

• User-based views – so the user can see their own content

Feedback

Types of Feedback

Finally, social applications that leverage Collective Intelligence are dependent on feedback to provide value. Porter highlights some different types of feedback: Implicit vs. Explicit, and Positive vs. Negative.

I’ll quote Porter’s comments of Implicit vs. Explicit feedback:

Typically, a combination of implicit and explicit feedback is used to create a picture of popularity. For example, Amazon’s bestseller list (based on implicit feedback) also show ratings (based on explicit feedback).

Implicit feedback is based on user behavior that is captured while someone moves through a site. Examples include downloading, bookmarking, and purchases.

Explicit feedback comes from someone’s explicitly-declared preferences, including ratings, reviews, and comments. While this sort of feedback tends to be more accurate in reflecting user taste, it also requires more work from the user and so less data can be collected.

Make Feedback easy

Finally, Porter has a few words to say about the importance of making feedback an easy, simple task for the user.

In Summary

Wow, that was a decent-sized post as well. So that’s a brief journey into how some of the more popular sites on the web leverage collective intelligence to keep their users engaged, and deliver interesting and relevant content.

In the next post, we’ll look at one more chapter from Porter’s book, that being devoted to application functionality designed to make it easy to share content with your friends and the world.

glenn

Also in this series

• Defining Requirements for Social Web Applications – Part 1: Introduction
• Defining Requirements for Social Web Applications – Part 2: The Framework
• Defining Requirements for Social Web Applications – Part 3: Social Objects
• Defining Requirements for Social Web Applications – Part 4: Defining Core User Actions
• Defining Requirements for Social Web Applications – Part 5: Motivations for User Participation
• Defining Requirements for Social Web Applications – Part 7: Enabling Sharing