The context for real time search

Vertical search used to be a hot topic in 2006-07, with search being applied to horizontal industries like travel, B2B (see this article for a big list of such search engines), products, health, people etc. Read this 2006 readwriteweb article for a good refresher. In 2009, many of these vertical search sites are still around and are in fact, prospering.

Now, with real time expanding the perimeters of search, will we see similar verticals being pop up? That was the question I kept asking myself after talking to a couple of entrepreneurs who want to build alternatives to Twitter search. They argued that real time search shouldn't be confine to what Twitter provides. There is room for different forms of search that build on real time conversations.

This sounds reasonable to me and is worth supporting to explore how real time search will evolve. Currently in the real time search arena, we have general search companies like Topsy, Collecta, Crowdeye as well as vertical search like Twitseeker (for people search), TwitterJobSearch (for job serach), PicFog (for image search) etc. There is no clear winner yet but it is informative to watch how these services evolve.

The big challenge to me is to think about the context of which real time matters and therefore how real time search matters. Emergence plays a big part. When you want to know emerging thoughts, patterns, trends etc, real time search becomes critical. Think about reactions to product launches, about the spread of virus, about unfolding of events like sports, weddings, disasters etc. All these are context where real time matters.

I am sure there is more context surrounding real time search. What are your views?

Read more »

There may not be a Google for real time search

Comscore released an interesting report about search ad coverage. One key point was that people were using more and more search terms in their queries:

An analysis of comScore data shows that search queries are actually getting longer and that as searchers become more experienced they are using more words per search query.

Based on ths report, John Battelle hypothsised that search query is becoming more and more like natural language, implying that searches are becoming conversations:

In short, our queries are getting closer to real conversation, real natural language, and Google's algorithms are having a harder time keeping up - matching advertiser demand to our increasingly complex queries.

Mark Cuban was pointing to the same path as he writes that search queries can now be framed as Twitter questions, with answers recommended by your social network.

If we assume that social Q&A or real time search will be a revolution in information retrival, will we see a Google equvialent? That seems to be the common perception and Twitter seems to be the most promising candidate.

However, I don't think there will be a monopoly in search as what we have seen in Google. The decentralised nature of conversations may lead to a loose network of recommenders and gurus. Anybody who has a query will be answered by a recommender who in turn will reference a recognised guru. It will a complex web of answers and questions, with the best recommenders and gurus rising to the top.

If that happens, we will not see a Google in real time search. That might be a good thing.

Read more »

Adding value to aggregation

Continuing from my previous post on democratizing aggregation, I want to elaborate about the value beyond bringing all the information together. That is only the first step. In fact, it is one of the least value adding aspect of aggregation. The important thing is what you do with the information you have aggregated. Here are three ways that you can add value to any aggregation service:
Search and coordination
Aggregation is mainly about data but search transforms data into information. If you add in the ability to coordinate actions, such that deals or actions can be closed much more effectively (food aggregator allows you to book restaurants etc ), then aggregation becomes a much strong value proposition.
This is why scraper models for jobs, like Indeed, create economic value: they transform "data" into information - in fact, they do so at the moment you do a search. Conversely, this is why Craigslist is getting devalued - it's more data, and less information (ie, I can personalize it less/receive less stuff that matches my preferences, expectations, etc, than elsewhere).
Aggregating context
A bit of context goes a long way. Aggregating context is just as important as the information itself. Outside.In for example aggregates posts from local bloggers and filters them according to locations. An aggregator will need to think about how to add context that is useful to the users they are serving.

Allowing the cream to rise to the top
The power of StockTwits is its understanding of this principle. We can see this in their announcement of a premium blog network. That is a brilliant strategy. Surfacing the best of the community and then giving them tools to let them do what they do is a powerful demonstration of edge competency. It is equivalent to a music label picking up promising artists, except with a much more effective cost structure and software. This also points to a new path to profitability for community powered models. Every aggregator, from food to fashion to health, should seriously employing this in their community.

Related articles by Zemanta

• Organising conversations (ac-idealog.blogspot.com)
• 5 small ideas for music related startups (ac-idealog.blogspot.com)

Read more »

Search, tags and follow

Organising or structuring information is a key value enabler in this age of abundance. Be it conversations, tweets, data etc, organization provides the key to efficient attention allocation. Currently, I have seen 3 scalable forms of organisations.

Search
The most commonly used form of organisation, search instantly organises information around your stated interest. However, as information increases, the cost of search is going up. Not only that, search itself is limiting in discovery and doesn't fit into our way of uncovering stuff.

Tags
Tags is different from search as it is more discovery based. Interfaces like tag clouds encourages explorations. Now, with the use of tags on Twitter becoming more and more common, we might see innovations in tags related forms of organisation. We are not done with tagging yet.

Follow
Social filters are the latest in information organisation. The breakthrough for me was the invention of 'follow' by Twitter. Before this, we can already bookmark or subscribe to people so that they can act as filters for information. This process however is cumbersome. 'Follow' provides an elegant way for people to easily select their social filters and create their personalised streams of information. What's more, the information tends to be diverse due to the human aspect of such organisation.

What is interesting is when these models are mixed and match. One example is searching through the #tags in Twitter search. That is a combination of tags and search. What other models are possible?

1) Tagging the people you follow and then searching through the people you tag to find relevant answers on particular topics

2) Follow tags from the people you are following to explore the areas they have interest in

3) Follow searches by the people you are following and bundle them with tags

4) Follow people who are searching the same terms or using the same tags


What other combinations do you see? More importantly, what other ways are there to help us organise information?

Read more »

Are we done with tagging?

My interest in tagging started way back when Umair Haque was talking about them and how transformative tagging would be.

Instead, people tag things they prefer, and ignore things they think don't prefer. In this way, preferences get implicitly aggregated, and then coordinated. That is, if 400 people tag bubblegen with technology, you might also reasonably suspect a significant proportion of them think it was cool enough to tag, which goes on to coordinate your preferences about bubblegen.

One immediate area when tagging can exert powerful influence is in the area of search. Tags represents an alternate way to find stuff.

At some point, tagging becomes more efficient than search. This point is reached when the number of Google clicks is greater than the number of related tag clicks (or similar proxy). When this happens, tagging becomes a perfect substitute for search.

Creating an alternate search paradigm was also the primary reason why Fred Wilson invested in Delicious. However, that vision was never realised. Delicious was sold to Yahoo and tagging became left by the sideway. I wonder if Delicious had continued, will it have reinvented search, like what Twitter is threatening now? It had the potential. For example, if you look at for tags, it is actually very similar to how the @ function works in Twitter. That in iself is quite a remarkable way of sharing.

Now, the only person I heard rallying about tagging is Marc Canter, who have said that:


We’ve certainly (collectively) dropped the ball on tagging

So, my big question is: are we done with tagging? Does our current services already provide what tagging has to offer or is that unexplored opportunity left by the early acqusition of Delicious?

Read more »

Interfaces that enhance serendipity

John Hagel had a great tweet:

maps are powerful interfaces focusing on relationships, not objects - applies to people and ideas, not just places - enhances serendipity

This inspired me to think about interfaces. Serendipity doesn't happen in a vacuum. It requires a medium that conveys the flow of information, the diversity of views and the connections to the edge. The interface, in fact, is a vital point that potentially enhances or limits the extend of discovery. It is the difference between enhanced serendipity and random browsing the Web.

The serendipitous service of the moment is Twitter. Through the 'river of news' style of presenting thoughts, links, conversations etc, one can easily discover stuff that are outside their normal web of interest. The value of discovery, however, depends greatly on the people you follow. If you pick the right ones like John Hagel above, you will find your perspectives being extended in ways you have not thought before. The challenge is, of course, to pick those people and that is why we will recommendation services like ReTweetRank and Mr Tweet becoming more important.

One other interface I can think of that enhances serendipity is the calender view from events companies like upcoming.com. They enhance discovery of events, and in the process, makes physical connections between people. Such interfaces are great for maximising discovery of time based items.

The common thread running through the two interfaces is the diverse source of information and how they are being compressed for easy scanning.
Specifically:

-sources of information are diverse
-information is stripped down to its bare minimum
-display is compact with little wastage of space
-format of display is in time format, either zoom in as in minutes updates to twitter or zoom out as in monthly displays in calender

Using these principle ( I am sure there are many more), we can design for better interfaces that enhances serendipity. Conversely, interfaces that do not do so are limited in discovery. Consider the disqus interface below: (i) information is not compresses with too much space that hinders scanning, (ii) no way to discover comments made on other blogs by the same commentor.




















In summary, interfaces are a great place to uncover new ideas that may form the basis of different products/services. What other great interfaces have you seen that maximises serendipity?

Related articles by Zemanta

• Steve Woodruff: The Strategic Serendipity of Social Media (mpdailyfix.com)
• MicroPlaza: A Personalized Twitter Memetracker (readwriteweb.com)

Read more »

Opportunities in enhancing discovery (serendipity)

I have find that Singapore is always a generation late when it comes to innovation, especially in the Web area. Hence, I am always 'listening in' to conversations around the Web, especially to people who are exploring the deep fundamental drivers that are shaping our economy and society.

In one of these listening sessions, I picked up on the concept of serendipity enhancement (hat tip to Ethan Bauley). It is different and yet may occupy the same importance as search. While search is powerful when the intention is clear, serendipity enhances social connections as it leads people to connect in unexpected ways. In a economy where social capital is growing in importance, such enhancement can bring tremendous value.

How then can you enhance serendipity? I have no strict answers but here a few thoughts that I am working on:

Serendipity is about flow: It is about maximising your exposure to the flow of connections and information, on your own terms of participation. Think about river of news but add in the ability of people to connect via the social objects being shared and discussed.

Such flow will come from services that help people expose their thoughts and work to where the flow is. Aggreagtors are the first manifestations of creating such values as they bring exposure to your writings. However, the bigger opportunity is in services which are able to move your information around and plug you into relevant flows. I called it SwitchBoard 2.0.



Serendipity is about edge connections: It is commonly known that the greatest value from your network typically comes from the weakest link. Serendipity increases your chance to connect to somebody that is outside your traditional network and hence, potentially bringing greater value to you.

This is why I lament here that current social networks are not meant to enhance social capital. What is needed is a way to let us, as individuals connect to the edge. To do this, we need to understand how social capital is built which then gives us answers on how to better a recommendation service. This is a key challenge and one that is still wide open.



Serendipity is about diversity: Discovering news things (and though that, new connections) needs a certain degree of randomness, and consequently a certain degree of noise. What many current services have done is to pair down the noise though specialised verticals. This has dramatically reduced diversity.

Aggregating across verticals to produce diversity is what we need to reproduce the serendipity that is found in our physical lives. That is one powerful benefit that services such as FriendFeed has introduce.

However, beyond activities, aggregating different perspectives on life, strategies, knowledge, values et al has the potential to really open up innovations. That is a difficult problem and one that I have yet to see attempts being made.


There are many more things to understand about serendipity and the opportunities it represent. This is my attempt to plug into the conversations that are happening. Let's see where serendipity will lead me to.

Related articles by Zemanta

• Exploring new concepts (ac-idealog.blogspot.com)
• Twinfluence - Twitter Influence Analyzer [del.icio.us] (twinfluence.com)

Read more »

Idea generation #53: Social Q&A services on your phone

I found this fascinating quote today:

The reason I am blogging about this, other than to get all of you to go download the app, is because Live Poker is the first first application to leverage Facebook Connect to allow iPhone users to play with their real friends and access social information such as real player photos. Just think about that for a second. We've now got apps on open mobile phones connecting with apps on open social nets. It's only a matter of time until we see more of these live mobile apps running on iPhone, Blackberry, Android, Windows Mobile, Symbian, and connecting to Facebook, MySpace, and other social nets.A VC, Nov 2008

The ability to tap on your social graph with your mobile has immense potential. One area is taking the concept of Aardvark and applying to the mobile phone. Aardvark is Yahoo answer + twitter. It analyzes the question you post and distributes them to your social graph according to who is in the best position to answer it.

Applying Aardvark to mobile makes a lot of sense. Very often, you need answers when you are NOT at your desk. Your phone is your best bet to get a quick answer. If you can leverage off your social graph to get the answer, even better. The app shoud be able to not only assess who has the best domain knowledge to answer your question, but who is currently as well. Hence integration with status information is also important.

I think integration your social graph into mobile apps is worth more thinking about. What are your thoughts.

Read more »

Startup Idea #87: Feeds as the next search

I normally don't try to link to popular articles or blog articles to maintain some degree of freshness for any readers that might come across my blog. However, this post by Fred Wilson's on the "feedization of the web UI" is too compelling for me not to mention it.

Feeds is always something I am mindful of. It is, to me, part of the next generation solution to the attention problem and this makes it an area full of potential. As pointed out by Fred, feeds represent an alternative or complement to search and it is interesting to note what happens when the Facebook generation uses more feeds than search.

Feeds however brings with it a new set of problems and that is in the diminishing returns when the feeds volume increases. To me, the solution is not better filters. Rather, it is to acknowledge that the front page of any news feeds is actually an attention market. As such, we need to design and develop 'market currencies' to make any news feed system effective.

Taking the current examples, Techmeme's currencies are the links, Digg's are the votes, Del.icio.us's are the tags. These currencies carry with them a 'price' and it is this price that will become a good mechanism to surface the most relevant news. Problem is that these currencies are easy to game and hence the appearance of link farms et al. Hence, for the innovators in the feeds space, it is worthwhile to spend some time thinking about how to improve the current feeds sustem using 'market currencies'.

Beyond market currencies, next post will suggest other startup ideas that can be done in the feeds space.

Read more »

Startup Idea #80: More sports business 2.0

I blogged in 2007 about how new markets such as tickets, sports, virtual worlds et al will bring about the need for a new Bloomberg, Fastsearch, Monitor101 (which unfortunately has closed down) et al.

It is good to see this happening in 2008 with the announcement of Ticket Stumbler, a search engine to find the best deals on the various secondary ticket markets. This is very smart as every new market needs a robust and powerful search to help users find the best deal. So what else can be we do here, specifically in the tickets market?

I think there are still lots of opportunities to aggregate all these market data and build something useful on it.

How about a Farecast to help predict the probability of a ticket price coming up or down? As more data becomes available, it is now possible to find relationships that can help in modeling the predictive model.

Alternatively, there is also opporunity to develop a algorithmic trading platform for people to trade tickets. While there are currently people trading tickets online, to my knowledge, there is no trading tools built to serve their needs. We can further expand on this idea by layering in the social element where traders can share their trading history. This is the Covestor model. The payoff for such a trading tool can be big.

Finally, how about a Seatguru.com to let users know where exactly will they be sitting. If 3D models can be made for every stadium with users reviewing their seating experiences, a powerful tool can be created. If a search can be built on top of this, where users search by the best view, rather than by price et al, it can be a killer application.

See the first post on sports 2.0.

Read more »

Startup idea #52: Comparison service for medical tourism

After reading a few good articles on current state of healthcare (see post here), and reading this story from Fast Company, it suddenly all made sense and I can see what a startup with little resources can do to create value for the healthcare sector.

The main issue threading across these articles is that healthcare is not going to the people who need it due to the games that insurance companies play. One solution, as pointed out by the Fast Company story is to seek cheaper treatment in other countries. I think there lies the opportunity for a startup to contribute to the solution for healthcare problem.

One idea is to be the Jellyfish.com + Tripadvisor.com for the medical tourists. The site will allow comparisons of different factors such as price, service quality, airfare costs et al for different types of medical services. It should also allow users to review recreational activities that can be enjoyed by the patients after their operations. The quality of information about the different medical services and the ease of the interface will be key success factors.

I will leave you with this paragraph from the Fast Company story that spells out the opportunity:

Medical tourism hubs will become different stops on the same assembly line: Brazil and South Africa for plastic surgery; Mexico and Hungary for dentistry; Costa Rica for a little of both; and Southeast Asia for the bodywork of heart surgery, organ transplants, and orthopedics.

Read more »

Idea generation #18: Live search

Another great idea by Buzzmachine on live search. It is a difficult and yet interesting problem to think about. Something that the page rank algorithm cannot solve easily. One possible way, as suggested by Buzzmachine, is to use Twitter. Possible...

Read more »

Startup Idea#11: Reconstructor for lacking content in search log analysis

Updated: 22 Nov 2007
Completely dropped the previous idea as it is not interesting enough on a strategic and economic level. The replacement idea is this: to have a last.fm type of application to automatically that plugs the gaps as revealed by search log analysis.
(See blog post here for how useful search log analysis can be)

Current search log analysis tells you what is missing in your content, what is the content needed to fulfill the long tail of search queries et al. The greater value is when the application can immediately suggest sources of content to plug the gap. This form of reconstruction has result in tremenduous value creation


Dropped Idea
This idea links up the growing popularity of virtual worlds (see cyworld, club penginue, world of warcraft) with real world chores. A simple virtual world can be created where children gain points and powers, not through accomplishing virtual tasks, but completing tasks in the real world.

Parents can assign tasks to the children in the areas of household chores, learning activities or physical exercises, who can complete them in exchange for points. These points can then be redeemed by the children in the game for points or powers in the virtual game. Parents will prepay the points in the game.

Alternatively, physical world rewards can be used to replacement the virtual worlds. The site can be tied to ecommence sites where children can preselect the gifts they want to redeem. The site then takes a commission for every item redeemed.

As a product expansion, such an idea can be extended to adults. Wifes can set up items to be completed for husbands who will receive gifts for finishing the tasks

Read more »

Startup idea #2: Search applications

This idea was inspired by an Business 2.0 article I read a few months backon search applications (read article here). While the article highlights the opportunites in improving search without coming up with a better algorithm, it tiggers something related in my mind.

There are now hundreds of search engines as well as search functions with all sites including ebay and amazon. What applications can be built on top of these results queries? In my mind, there may be a market for a new Nielson. What Nielson has done for traditional media, this new company should provide in the new media space. How is your product doing in Ebay? What prices are being transacted? Did it get positive mentions in Digg? Was the product site heavily tagged in De.Lic.Ous? How many sites have mentioned your product as determined by Google?

These are all questions waiting to be answered. It needs only somebody with a hell of engineering skill to link it all up andprofit from it.

Read more »