Hadoop at Twitter – Dmitriy Ryaboy from April 2010
Another in a series of posts on Big Data, NoSQL, and Hadoop. Previous recent related posts include:
- Steve Huffman on scaling Web Apps at Reddit – FOWA 2010 Miami
- Analyzing Big Data – Wonderful SDForum panel on NoSQL and Big Data
- Facebook’s Architectural Stack – designing for Big Data
- Designing Large-Scale Retrieval Systems at Google – Jeff Dean from 2009
- Introduction to Hadoop – understanding Big Data
- Keen Insight on Big Data from Cloudera CEO Mike Olson
In the video below, Dmitriy Ryaboy of Twitter provides an overview of the Hadoop stack at Twitter (from April 2010):
In a nutshell, here’s Twitter’s Big Data/Hadoop technology stack:
Also some nice discussion on the use of Pig for large-scale data analysis in Hadoop (without having to write sequences of data manipulation in MapReduce). The following slide summarize the benefits of using Pig:
And here’s a summary slide highlighting why Twitter uses Pig over SQL for expressing data query and transformation syntax:
glenn
Hyperlocal – Core Dimensions (Part 2)
This is the third in a series of posts on key dimensions of Hyperlocal. Other posts in this series are:
In the previous post, we explored the dimensions of Hyperlocal News and Commerce. In this post, we will explore Local Advertising and Hyperlocal Community.
Local Advertising
Local Advertising is definitely a key part of Local Business/Commerce, which I explored in the previous post. But local advertising can also be embedded within Local News and Local Community portals. Thus I’ve chose to deal with it as a separate topic.
Insights into Local/Hyperlocal Advertising
First off, I have a few favorite resources for keeping informed in the Local/Hyperlocal advertising space. These are:
- Borrell Associates blog
- BIA/Kelsey blog
- Screenwerk – Greg Sterling’s blog
Borrell Associates – headed up by CEO Gordon Borrell – also sponsors the Local Online Advertising Conference, which was held in New York city early this month.
Jeff Jarvis also frequently has compelling insights into Advertising strategies for Local News Media. For example, see his recent blog posts from February 2010: Stop selling scarcity and NewBizNews: What ad sales people hear.
Search Engine Marketing/SEO for SMEs
Obviously, SEM strategies are critical for any local online business on the web. My top go-to resources for local SEM/SEO insights are:
- Small Business Search Marketing by Matt McGee
- SEO by the Sea – Bill Slawski’s blog
- Understanding Google Maps and Local Search – Mike Blumenthal’s blog
- Chris Silver Smith
- The Noisy Channel – Daniel Tunkelang’s blog
- David Mihm
- Local SEO Guide – Andrew Shotland’s blog
- Search Engine Land
- John Battelle’s Searchblog
Big Ad Networks
On the solution provider front, you have the big ad networks around Search Engine marketing, some of which include:
Local Advertising Media/Platforms
A number of application/media providers – many with a mobile focus – are positioned to be significant players, including:
Niche/Regional-based Ad Networks and Services
Increasingly, however, you also have your niche/regional-based ad networks and service providers. Here’s some examples:
- Adify
- Clickable
- GrowthSpur
- ReachLocal
- Yodle
- WebVisible
- Spot Runner
- Sacremento Press
- Village Voice Media
Bargains and Deals
Numerous vendors provide applications to notify consumers of bargains and deals in the local vicinity, including:
Additional Local Advertising Solution Providers
One more advertising solution provider I’ll mention:
So there you have it, a sampling of Local Advertising solution providers. Local Advertising should be a very interesting space to watch in 2010, particularly when it comes to mobile, location-based tools and technologies.
Local Community
The Local Community view of HyperLocal is about information and events of interest to the Community. Information and Events around the Local Community may be contributed by businesses, community organizations, or municipal governmental sources, or it may be user-generated content contributed by the Community.
When you talk Community, by definition you are talking about Social Networks. Therefore, you have to consider the various social networking platforms, and particularly those that host large social graphs. I’m thinking here most specifically of:
Many of the HyperLocal News platforms are also positioning themselves as Local Community platforms. For example:
You also have open city initiatives/discussions such as those initiated by:
For additional information on open city initiatives, see here.
Then there are do-it-yourself City initiatives and tools, for example:
You have Local Event platforms, such as:
And finally, organizational and community tools around local causes. See:
- The Extraordinaries, also see here
- Localgiving
This is really just a very small sampling of possible ways/platforms for organizing people within a geographic community. I look for a lot of innovation in this space over the next several years.
HyperLocal Business Models
This viewpoint explores various ways to make a HyperLocal business commercially viable. There’s some great pioneering work being done by Jeff Jarvis and the folks at CUNY here – see the New Business Models for News Project at CUNY, and Jarvis’ overview of the work on HyperLocal business models here.
More on this to come.
Hyperlocal – Core Dimensions (Part 1)
This is the second in a series of posts on key dimensions of Hyperlocal. Other posts in this series are:
In the initial post in this series, I introduced the following core dimensions of Hyperlocal/Local:
- Local News/Journalism
- Local Business/Commerce
- Local Advertising
- Local Community
- HyperLocal Business Models
This post will briefly explore the first 2 dimensions – Local News and Local Business/Commerce.
Local/Hyperlocal News
An important service to the Local community is News about the Community, or Community perspectives and reactions to the News. A nice definition of HyperLocal News is provided by Christopher Wink here: Hyperlocal news: a definition.
Keeping Informed about HyperLocal News Media
Here are a few of the sites I regularly visit to keep track of happenings in the Local/Hyperlocal News media space:
- BuzzMachine – Jeff Jarvis’ blog
- News Innovation
- Journalism 2.0 – Mark Briggs’ blog
- Lost Remote
- Media Transparent
- HyperlocalBlogger – Matt McGee
Sites
There are many, many Hyperlocal blogs and websites currently servicing their local communities. Wink provides a nice overview of some of the major ones in his post Hyperlocal news sites worth following.
Platform Providers
This is a core area of innovation at present, and a slew of recent acquisitions by major media companies of hyperlocal news platform providers illustrates this point.
Some of the key innovators:
Citizen Journalism
Often Hyperlocal news sites leverage Citizen Journalists as contributors to their site. A very interesting example of this approach is Examiner.com, which I blogged about here.
Real-time News Feeds
Finally, with the rise of Twitter, real-time geotagged feeds are also breaking onto the scene. Pat Kitano‘s Hyperlocal Curation of Real Time News post from November 2009 provides some interesting examples. Kitano’s Breaking News Network is a powerful example of location-relevant breaking news.
And of course, Twitter has launched location-based Trending Topics, would should add additional impetus to this trend.
Local Business/Commerce
Here, we’re talking about the commercial aspect of HyperLocal. A profound insight into the potential de-coupling of “core” Local News from Local Business services was recently delivered by Clay Shirky at the Harvard’s Shorenstein Center on the Press, Politics, and Public Policy in September 2009. A must-listen-to presentation IMHO.
There are many, many players and platforms that service local business needs, with many more soon to come no doubt. Here are a smattering of players in a very competitive space:
Business Listings – General
- YellowPages.com
- Superpages
- Yellowbook
- Yell.com
- InfoUSA
- Acxiom
- Localeze
- CitySearch
- CitySquares
- Local.com
- Yodle
- City.com
- Business.com
- Insider Pages
- ThinkLocal
- CityVoter
Business Listings – Niche
Services – Niche
- Craigslist
- Oodle
- AutoTrader.com
- Monster.com
- MLSlistings
- Realtor.com
- Zillow
- Trulia
- HelpHive
- My Builder
- ServiceMagic
Shopping
As you can see, there are a number of different spaces by which software providers and hosting sites are seeking to provide services for local commerce. In the next post, we’ll look at two additional hyperlocal/local dimensions: Advertising and Community.
glenn
Create more value than you Capture – Tim O’Reilly’s prescription for Publishing companies
A nice presentation from Tim O’Reilly on tips for using Twitter, delivered at O’Reilly’s Twitter Boot Camp in June 2009. Here it is:
First of all, I love the notion of “create more value than you capture. As far as I can tell, O’Reilly first publically used this phrase in this post where he said:
At O’Reilly, we always say “Create more value than you capture.” All successful companies do this. Once they start capturing more value than they create, their market position erodes, and someone displaces them. It may take a while but it happens eventually.
Secondly, there are several great quotes in O’Reilly’s presentation. First there’s basically O’Reilly’s definition of karma as applied to social media:
The secret of social media is that it’s not about you, your product, or your story. It’s about how you can add value to the communities that happen to include you. If you want to make a positive impact, forget about what you can get out of social media, and start thinking about what you can contribute. Not surprisingly, the more value you can create for your community, the more value they will create for you.
It would seem kind of obvious, wouldn’t it. But …
I also found the following quotes from the presentation interesting – which emphasize the role attention as filtering and promoting that which you value:
- In social networks, you gain and bestow status through those you associate with
- A key function of a publishing brand is the bestowal of status by what you pay attention to
- If you only pay attention to yourself, you aren’t as valuable to your community
What I found interesting about the above bullet points is the emphasis on bringing value to the discussions your community is having. I think this is a very different focus from that of a traditional newspaper/publishing company, which has traditionally focused more on communicating to their communities what the publisher felt they should be talking about.
I think the need to foster discussion amongst their communities is a realization that media companies generally have had for some time. But it’s not traditionally been in their DNA, and it remains to be seen how well they navigate this transition.
Of course, the real purpose of the presentation was to provide tips for using Twitter, where Twitter is positioned as a tool for sharing and promoting ideas and within one’s communities.
O’Reilly provides some specific tips for using Twitter at the end of his presentation. But for a more complete intro to using Twitter, please check out The Twitter Book, a book O’Reilly co-authored with Sarah Milstein.
glenn
Twitter Parody – Flutter … funny
A Twitter parody video … Flutter.
Funny