Abstract:

The process of extracting useful knowledge from large datasets has become
one of the most pressing problems in today's society. The problem spans
entire sectors, from scientists to intelligence analysts and web users, all
of whom are constantly struggling to keep up with the larger and larger
amounts of content published every day. With this much data, it is often
easy to miss the big picture.

In this paper, we investigate methods for automatically connecting the dots
-- providing a structured, easy way to navigate within a new topic and
discover hidden connections. We focus on the news domain: given two news
articles, our system automatically finds a coherent chain linking them
together. For example, it can recover the chain of events starting with the
decline of home prices (January 2007), and ending with the ongoing
health-care debate.

We formalize the characteristics of a good chain and provide an efficient
algorithm (with theoretical guarantees) to connect two fixed endpoints.
We incorporate user feedback into our framework, allowing the stories to be
refined and personalized. Finally, we evaluate our algorithm over real news
data. Our user studies demonstrate the algorithm's effectiveness in helping
users understanding the news.

-------------------------------------------
This paper, joint with Carlos Guestrin, received best paper award in SIGKDD
2010.