Thunderstone Blog: Customized Search Engine & Search Software

Most Americans know QVC by its shopping channel on television. QVC's web site, however, also is one of the most successful online retailers, bringing in a substantial portion of QVC's $4.4 billion of sales yearly.

Thunderstone's Texis software plays an important role at QVC in a variety of ways.

Unlike television sales, where each item is on sale for only a few minutes at a time, QVC's entire inventory is available all the time on its web site. The online database thus has hundreds of thousands of products, and requires a robust search engine to help users find the items that satisfy them.

QVC needed its search engine to do more than simply search, however. The company had many advanced requirements.

For example, its customers are price sensitive, so QVC wanted to enable them to select by price as well as by description. A user can enter "gold necklace" and "$100", and QVC's Texis system automatically finds products within +/- 25% of the user-entered price. QVC found Texis the best product for combining a numeric range query with a text search.

Texis's real-time functionality also is crucial to QVC.

We update our inventory in realtime to make sure sold-out products are not offered to the customer. This is especially important In high volume periods like 'Fashion Day' where certain colors and sizes sell out quickly and frequently. This is a key function Texis provides that no other search engine we've seen can do.
Frendy Glasser, QVC Director of Data Administration

In addition to serving search results, Texis powers the "drill-down" or catalog browsing features, which are based on product metadata. And when a user views the detailed description associated with an individual item, Texis serves up that content too.

Many other Texis features are in use. Products are ranked according to QVC's proprietary criteria. And results are retrieved from "partner" databases for products such as books, music, and sporting goods.

The Texis applications are used as well by QVC's order-entry representatives when customers call asking for a product. All in all, the Texis installation at QVC is handling millions of database queries daily. QVC runs Texis on IBM AIX hardware, with the "front end" user interface written in ASP running on Microsoft NT servers.

What's also important to us, besides the performance and advanced features, is the flexibility to change how we respond to queries as the business situation evolves. We like knowing that Texis can be programmed to respond to searches according to almost any logic we come up with. Before Texis, we had another search package that was a major product in the industry, but we were frustrated by programmability limitations, and by its proprietary interface. Texis, in additional to being a great search product, is easily customizable due to its industry-standard database features.

We also value Texis's consistency in response time regardless of volume, its scalability, and its availability. We rarely, and I mean rarely, have any downtime problems with the Thunderstone Texis engine.
Frendy Glasser, QVC Director of Data Administration

Litigation is a booming business in the United States. That's not good news to some, but it creates interesting technology challenges. Larger and more complex lawsuits entail larger and more complex document management needs.

Some of the largest U.S. law firms, such as Skadden Arps, rely on Texis to search and manage legal documents through an application developed by Thunderstone integrator nMatrix Inc. The specialty of nMatrix is managing the "discovery" phase of U.S. legal proceedings.

In discovery, the parties provide the court with documents that may contain evidence relevant to the case. In major cases, millions of documents may be collected. They will be in many different electronic formats, or on paper that must be scanned and converted to searchable text through OCR process. The lawyers involved must search through the documents looking for relevant information.

The nMatrix system, called DocuMatrix, is unique in providing both full-text searching as well as classification and tagging. At the beginning of the process, a document is imported as an unstructured text object. Lawyers can immediately search through the material using Texis's various advanced search techniques. The approximate pattern matching tools are especially important for searching through OCR'd material which typically contains many typographical errors.

But that's not all. After a document is reviewed, authorized users tag it with various structured information, such as the names of people mentioned, dates, subjects, and many other details. Each entry becomes immediately searchable itself. In addition, the entered data is used for browsing and sorting the documents.

Texis's database characteristics make all this easy to accomplish. An administrator may specify the needed data fields, which are different from case to case. For any field, the administrator may designate a list of values to select from; others fields, such as a description, may allow free text entry. Users may search across any combination of the original text and added data.

We studied other search engines and found Texis to be the best suited to these needs because of it's superior database integration. This system has grown to be quite complex, but Texis has accommodated all the new requirements. It is really very versatile.
Arthur Finkel, CTO of nMatrix

An example of the complexities involved is access control. Users may be restricted to viewing only certain documents or even certain fields. Also, nMatrix recently added a feature allowing users to define their own thesauri for each case. Searches will automatically search for all the specified synonyms of a term in the thesaurus.

nMatrix also has found that the system is applicable to needs beyond the discovery process. Its nSite! product is targeted at knowledge management applications for lawyers as well as bankers and other professions. This encompasses real-time notification of newly arrived data, insuring that users stay up-to-date about important business issues.

We're planning to expand our use of Texis in many ways. Perhaps what's most important about the role of Texis in our products, is the way it has adapted to new requirements. We provide features and functionality today that we hadn't even thought of several years ago when we started with Texis. The fact that it has kept up with our product evolution says a lot about its maturity and superior design.
Arthur Finkel, CTO of nMatrix

If you manage a web site, you probably included certain "key words" with the aim of influencing search engine rankings.  But how do you know that you used the right words?  According to Wordtracker.com, you probably didn't.

"You probably are competing with too many other sites that use the same words," says Mike Mindel, founder of Wordtracker, which is based in London.  "That dilutes your effort to differentiate your offering.  And web users probably are searching related terms that could describe your business, but that you neglected to incorporate into your site."

Wordtracker has built a highly successful business dedicated to identifying which keywords are both popular and have the least competition. One of the most technically sophisticated services of its kind, Wordtracker relies on Texis to perform many important functions.  

Wordtracker provides a highly automated service, as opposed to the manual analyses offered by many "optimization" or "placement" consultants.  It starts with collecting millions of actual search terms typed by users on major metacrawlers.  Wordtracker continually refreshes this database so that it reflects popular terms, which shift on an ongoing basis.

With this data loaded in its Texis database, Wordtracker performs several related functions for its subscribers.  It takes advantage of Texis's thesaurus and morpheme processing to identify as broad a range as possible of terms related to one's business that people search for.  It uses Texis's fuzzy searching to find common misspellings.  It provides counts of how many times each word has been searched in recent weeks.  And it calculates "keyword effectiveness" -- the relationship between a term's frequency as a search and its frequency of appearance on web sites (the higher the former, and the lower the latter, the better).

"Wordtracker helps find keyword combinations that bear any relation to your business or service - many of which you might never have considered," Mindel says. And for customers of pay-for-placement services such as Overture, Wordtracker is more thorough in suggesting terms to bid on than tools those services provide themselves, he says.

To accomplish its statistical operations, a relational database was essential.  But the Wordtracker database also needed to behave as a search engine, allowing users to "search the search terms."  The two kinds of functionality, relational and search, needed to work together to sort the results and calculate various values for each word retrieved.

"Before we found Texis, we were using another SQL database for many of these functions and were having a terrible time, because its search capabilities were so weak."  Mindel says.  "Texis solved many technical issues for us."

Mindel also was impressed with Texis's high performance reindexing capability.  "Texis builds a database and two search indexes on 42 million keywords in under an hour which is nice," he says.  "And you can load up a second database without impacting performance on the first, which I've never seen before.

"Much of what we do depends on Texis.  It was a very wise purchase," says Mindel.

You know about the biggest web bookseller.  But did you know that its sales represent less than 8 percent of the U.S. bookselling industry?  That there are hundreds of other bookstores now competing online?  And that many of them are powered by Thunderstone's Texis software?

A substantial group of these other bookstores are served by Texis customer Booksite.com, a company that provides a complete online solution for independent book dealers.

"Texis is one of the keys to our success," says Richard S. Harte, Booksite founder and president.  "To get started, we knew we needed a database, a search engine, and a versatile application server environment.  We found all that in one tightly integrated package with Texis.  It enables us to maintain one master catalog, but modify it for each store."

The master Booksite database holds some two million records of books in print. The customization capability allows individual booksellers to build on that collection in many ways. They each may designate a unique subset of items to sell, and they set their own prices. They can add their own specialized inventory, such as used books. There's even a module for offering non-book merchandise.

The Booksite system also offers a variety of marketing features. An affiliate program allows each bookstore to attract business from a partner network. Booksellers can modify database entries with promotional features such as recommendations or reviews, or flag them as "on sale," "in stock," "quick ship" and so forth. There are specialized applications such as those for college textbooks or Christian music.

And of course there's the robust search capability! The Texis search engine produces separate results corresponding to each site's unique inventory. Depending on how a site is customized, users may search on various field combinations such as author, title, subject category, publisher, and International Standard Book Number (ISBN).

All Booksite services are run from a central facility, so that each bookstore needn't devote resources to hosting or other technical issues.  "Texis's broad range of features, together with its ability to handle lots of simultaneous transactions, helps us keep costs low," Harte says.  "Our prices start at only $160 a month per store.  But for each customer, we can build on that to customize their database content and applications.  That's important to independent booksellers, who succeed by being different from the competition."

Booksite also uses Texis as the basis of its shopping cart feature.  Thunderstone customers are sometimes surprised that Texis can provide an e-commerce solution, as opposed to just a search engine.  But Texis encompasses the same relational database standard used by most commerce systems.  Texis is unique among databases in its ability to query the database content either as a search engine, or by the standard SQL language, or frequently, the two in combination!

Booksite's Texis-based system has been in place and growing for more than seven years, and now serves about 150 stores. It is, in fact, one of the earliest retail sites on the internet with a shopping cart feature.

What users comment on most is the speedy search, but what Booksite's managers appreciate most is the reliability and simplicity of administration. The database features make it easy to maintain data assembled from multiple sources; for example, book details incorporate data from five sources, each in a separate table. Yet, Harte says, "Booksite doesn't take a big team to maintain. Because of the unified Texis structure, it's actually a simple system from a technical point of view. It does a lot by taking advantage of Texis's built-in features."