by Fern Halper, Partner
In a recent newsletter – Patterns for success: Options for analyzing unstructured information – I briefly touched upon search technologies and stated that search is evolving and is converging with business intelligence. This month I want to elaborate on how companies might use this new functionality and the implications of the convergence between search, text analytics, and business intelligence.
Let me start off by stating that I am talking about how businesses can make use of enhanced search capabilities that combine search with text analytics and even business intelligence software. So, I am not talking about typical site search, rather I am referring to how business users can utilize the unstructured and structured information that resides both in their companies as well as out on the Internet to make better tactical and strategic decisions. This information might include business intelligence reports as well as the data underlying these reports, blogs, news feeds, or other information that resides in the company or on the Internet.
A few definitions are in order. Enterprise search refers to technology that indexes and retrieves information in structured and unstructured sources in both internal data sources and web sites for a company’s user base. Typically, this search is still goal-oriented – the users know what they are looking for. Enterprises use search for a variety of purposes including simple site search as well as finding basic answers to questions such as, “what is my authorization limit?” or “how do I order a new computer?” as well as more advanced analysis that leverages text analytics.
Text analytics is a set of techniques that can discover and extract unstructured text and transform it into structured information that can then be leveraged in various ways. Various extraction techniques exist including entity extraction (e.g. people, place financial amount), concept extraction (e.g. unhappy customer) or fact extraction (e.g. an event specification), to name a few. For example, concepts may be created using various linguistic and statistics based processes. Different vendors make use of different techniques and have created patents for these techniques.
Convergence is bringing these two technologies together and then searching using extraction indexes created by the text analytics. In other words, search results are filtered by concepts, entities, facts, or other extraction types and utilized to answer questions.
Consider the following example of what might be possible utilizing this approach. A manufacturing company is interested in gathering intelligence on why it is losing market share. In the past, it would have looked at numerous reports including sales reports, warranty reports, customer satisfaction surveys and analyst reports. It would have also had business analysts searching the Internet to find any relevant information. Now, leveraging this new approach, the organization might create concepts that it uses to search through all of its internal documents and business intelligence reports as well as any external news feeds and articles. These concepts might include “unhappy customer. They might also utilize various entities like “competitors making financial transactions.”
The concept itself may come from a taxonomy or ontology that has already been created by the organization or a third party. Or, it may have been created by a subject matter expert working with this information via a GUI that is part of a software package. This extracted information is sent to a search engine interface. The results might appear as they would with any search engine. The results might also be piped to a business intelligence product to produce plots of percent unhappy, happy, and neutral (no comment) customers. These plots might have been derived from the text of customer care centers or customer surveys and would have drill down capabilities to enable line of business users to explore why these customers are unhappy and do further analysis. Levering this kind of approach, companies can derive insights that were not possible before.
Hurwitz & Associates believes that the convergence of search, text analytics, and business intelligence will ultimately reshape what business intelligence is and how it is performed. Vendors we are aware of that are active in this area include Nstein, IxReveal, Inxight, Insightful, and IBM. While the market is still evolving – customization needs, computing power, and market awareness being some of the hold-ups – companies have experienced significant top and bottom line impact using these technologies. These impressive results definitely make it worthwhile for companies to look at investing in this technology in the near term.
Hurwitz & Associates is undertaking a major research study in the area of text analytics. The study, entitled Text Analytics – the Road to Understanding your Company’s Unstructured Information, examines topics including the text analytics market, the technology behind text analytics, vendor offerings, and the value of this technology. The report is targeted for publication in Q4’06. We invite any vendor in this market to participate in the study. Additionally, we are looking for input from end-users. If you are an end-user currently using text analytics technologies or considering using them, we invite you to participate, as well.
Contact Fern Halper at firstname.lastname@example.org for further information.