Thunderstone Blog: Customized Search Engine & Search Software

When the U.S. General Services Administration looked to upgrade the searching capabilities on its GSA Auctions® website, it could have written a host-based search engine to run on the mainframe system. Instead, GSA opted to use "off-the-shelf" Thunderstone Search Appliances — enabling the implementation of an affordable search solution that reduces system load while providing sophisticated search features expected by today's savvy users.

The GSA Auctions® website (http://www.gsaauctions.gov) empowers people in the general public to bid electronically on excess and/or surplus Federal assets. It supports fully web-enabled auctions that permit registered participants to bid on a single item or multiple items (lots) within specified timeframes.

Auctioned items can include run-of-the-mill office equipment and furniture, as well as more exotic Federal assets such as scientific equipment, heavy machinery, airplanes, vessels, vehicles, etc. The website enables GSA to auction-off and dispose of a widely geographically dispersed inventory of products. Participants can bid on and purchase available assets, without worrying about the actual physical location of any particular item or buyer.

Interested individuals may browse products offered on the auction site, or they can choose to search for items and place bids. With flexible and robust search capabilities powered by the Thunderstone Search Appliance, GSA Auctions® takes advantage of Thunderstone's proven technological expertise in the simultaneous searching of both structured and unstructured data.

Prior to implementing Thunderstone's Appliance-based search solution, things shaped up very differently on the website — in terms of its data access and retrieval functionality. Back-ended by a COBOL application utilizing a cgi interface to the mainframe's web server, the site originally supported a basic full-text search that required parsing the complete database to search each active item.

It seemed appropriate to consider deploying new technology, because the GSA Auctions® site deserved a more feature-rich and less resource-intensive search tool.

Driving TCO Lower by Integrating Mainframe Systems with "Off-The-Shelf" Products

Thomas Schaefer serves as Systems Architect and consultant to the General Services Administration. He helps the GSA identify innovative ways to derive optimal value from its Unisys ClearPath Mainframe investment and to maximize the productive use of all related I.T. resources.

According to Schaefer, GSA could have written a host-based search engine to run on the system. But, they figured, "Why reinvent the wheel?" Instead, GSA rapidly deployed several off-the-shelf Thunderstone Search Appliances to affordably implement a state-of-the-art search solution with reduced load on the ClearPath system for each request.

"By using Thunderstone Search Appliances, GSA Auctions® has gained the rich search features users have come to expect from sites like Google. The broader point is that, while there is a move to consider total cost of ownership and migrate applications to the ClearPath environment, not every problem requires developing custom mainframe software. Some problems are better solved by integrating existing components, mainframe or otherwise, into composite systems," Schaefer said.

Thunderstone's DataLoad API, which allows data to be pushed into the Appliance, gets used a lot by the people who administer the GSA Auctions® site. And they appreciate the "plug-and-play" reliability of their Thunderstone Search Appliances -- because every added server brings with it an additional production cost. Who wants to maintain yet another server? "With Thunderstone," Schaefer said, "I haven't logged-on in eight months, and everything keeps running just fine."

GSA continues to work with Thunderstone to further enhance and refine its electronic auction offerings to the public, using the Thunderstone Search Appliance model as its standard. Schaefer explained, "We like the fact that we still have the sophistication of Texis, but in an appliance."

The current search solution for the GSAAuctions.gov site includes two load-balanced Thunderstone Search Appliances (Enterprise Edition) located in a Minnesota production environment. Each Appliance runs active-active. Two additional Thunderstone Search Appliances (one Enterprise Edition and one Small Business Edition) are installed at a GSA facility in Utah, for the purpose of ongoing new application development and testing.

Kevin J. Payne, Director of System Applications at GSA, recalled, "We considered several internal and external search alternatives. Tom [Schaefer] came up with the idea of using a search appliance, which basically meant using either Google or Thunderstone."

"Thunderstone demonstrated a real willingness to make changes that we wanted and to customize their standard appliances to meet our needs. The ability to customize was a big thing. Plus, we really like the way the search engine runs," Payne added.

Customization for the GSAAuctions.gov search enhancement project enabled GSA's Thunderstone Search Appliances to add as many as 50 additional data fields (well beyond the three definable fields that come with the standard Appliance) for auction-item searches based on bid amounts and different geographic locations.

Payne said the procurement process was fast and easy. And he expects other key GSA projects to also have their search capabilities powered by Thunderstone Search Appliances in the months ahead. As an example he cited the GovSales.gov (http://www.govsales.gov) website. Part of the Federal Asset Sales Presidential e-Government (E-GOV) initiative, it will very likely implement its own Thunderstone Search Appliance by Fall 2008.

"These are systems that bring in billions of dollars a year to the U.S. government," noted Payne.

Providing Multiple Ways of Searching Structured/Unstructured Data to Find Precise Results

The GSA Auctions® site offers a variety of ways to search for desired items:

• Global Search
  The upper right corner of every page on the GSA Auctions® website contains a global search box that can search the entire site for all assets available to bid on and purchase. Users may refine displayed global search results by entering another search term in the search box above their search results and then clicking the "Search Within Results" button.

• Category Search
  The home page has a category search box for searching only user-selected categories of items. Directly above the category list, the category search box begins with "All Categories" as the default selection from the "category dropdown" list. This list changes to reflect items available for selection as users browse the different product categories.

• State Search
  Choosing a state from the Browse States dropdown box on the site's main menu header will produce a list of items located in the chosen state and will allow a user to enter keywords for searching the full-text information associated with all the displayed items.

• Advanced Search
  Accessible via a hyperlink under the global search box, advanced search allows users to: Delimit keyword search types (all words, any words, exact match.) Specify a current winning bid range. Search within a state for an item.

The professionals who work at GSA sometimes tend to write in government-spec language that typical users do not use themselves. For instance, what GSA refers to as "vehicles" on their auction site does not correspond to terms that most people use when searching for vehicles. Rather, people look for "cars," "sedans," "autos," "automobiles," "SUVs," etc.

With built-in Metamorph concept-based searching capabilities, the Thunderstone Search Appliances saved GSA hundreds of thousands of dollars in development costs, According to Payne.

May 2008 - Archive

CONTENTS

• Questions on Search
• Thunderstone Search Appliance Version 6.3 Released
• Quote of the Month
• Tech Tips: SYSUTIL and VXINFO
• Subscription/Unsubscription and Contacts

QUESTIONS ON SEARCH

As Web 2.0 technologies become more prevalent what does that mean for the role of search in the enterprise?

Web 2.0 technologies provide users the ability to create their own tools for organizing and finding information. While this can be a challenge for IT staff to set rules and create a framework for the applications, the ability for users to easily create applications will be used, and the challenge will be to make it easy for that content to be reused throughout the entire enterprise, rather than be stuck in an individual's application.

Enterprise search will evolve to allow more interesting queries using more data from the organization, and be able to consume data from and produce results in more formats to be used throughout the organization, and in individual applications users create. For example a user may create an RSS feed that publishes documents containing statistics about cancer, that another user combines with other feeds to aid their research.

One of the key challenges is that it can be more difficult to index the content in a Web 2.0 application, as AJAX and other technologies mean a single URL can house the entire application, and there is no good URL to visit for a search result. That requires care in designing the application to allow a search to work against Web 2.0 content.

To get your questions on search answered please email .

Thunderstone Search Appliance Version 6.3 Released

With New SOAP API and Single Sign-On Authentication Helper for Windows Environments

Thunderstone Software LLC has released Version 6.3 of the Thunderstone Search Appliance. The newest Appliance software enhancements improve the flexibility and ease of use of the Appliance in enterprise systems.

The updates apply to both the Thunderstone Search Appliance and the Thunderstone Parametric Search Appliance, include:

• DataLoad SOAP API
• Search SOAP API
• Single Sign-On Authentication Helper for Windows Active Directory environments whether using password, smart card, or other authentication mechanism.

The SOAP API allows users to more easily access the Appliance programatically for the purpose of loading data and searching. It should integrate easily into any development environment that supports SOAP.

The DataLoad SOAP API can be used to create custom connectors to push data from content repositories into the search appliance.

The Search SOAP API allows developers to bring results into their applications to show or manipulate how they want.

The Single Sign-On Authentication Helper lets the Appliance make use of ActiveDirectory based authentication schemes for results authorization, hiding results a user is not allowed to see completely from view.

If you have configured your Thunderstone Search Appliance to update automatically you should already have these available on your appliance, otherwise you can update the appliance manually to get these features.

If you have any questions about how these work please contact our support staff at or +1 216 820 2200.

Quote of the Month

"Thunderstone's approach is somewhat unique in basing its technology on a (SQL-compliant) relational database and full-text searches (allowing advanced pattern matching more traditional indexes find hard to cope with). This, and its relative speed, may be an advantage when using Texis as a search foundation to build applications on."

The Enterprise Search Report 2008
CMS Watch

Tech Tips: SYSUTIL and VXINFO

If you are writing a script that may need to run on multiple platforms then the <sysutil> and <vxinfo> functions can be useful for making your scripts more portable.

<sysutil> provides operating system independent ways of working with files, so the same code will work on Windows and Unix variants.

The <vxinfo> call lets you find out about the Vortex environment you are running in, which database is currently being used, where the software was installed, and which version of Texis is being used amongst others.

Used together these can improve the portability of your scripts and avoid headaches if they need to run elsewhere.

Feedback, suggestions and questions are welcome to

With New SOAP API and Single Sign-On Authentication Helper for Windows Environments

CLEVELAND, OH - May 27, 2008 - Thunderstone Software LLC (http://www.thunderstone.com), a long-time industry R&D leader in providing extremely robust search solutions for businesses, NGOs, governments and educational institutions, today announced that it has released Version 6.3 of the Thunderstone Search Appliance.

The newest Appliance software enhancements, which apply to both the Thunderstone Search Appliance and the Thunderstone Parametric Search Appliance, include:

• DataLoad SOAP API
• Search SOAP API
• Single Sign-On Authentication Helper for Windows environments (e.g. ActiveDirectory/SmartCard)

The SOAP API allows users to more easily access the Appliance programatically for the purpose of loading data and searching. It should integrate easily into any development environment that supports SOAP.

The Single Sign-On Authentication Helper lets the Appliance make use of ActiveDirectory based authentication schemes for results authorization.

The Thunderstone Search Appliance is a plug-and-play device combining the simplicity of a hosted service with the security and performance of a local solution. Built on Thunderstone's advanced Texis software, the Appliance can handle more than 1,000 typical queries a minute -- providing excellent value without adding administrative overhead.

The Thunderstone Parametric Search Appliance combines the flexibility and power of Texis with the ease of use of an appliance. It provides an easy way to create applications that combine full-text and structured data without programming.

About Thunderstone
Thunderstone Software LLC (http://www.thunderstone.com) pioneered simultaneous searching of both structured and unstructured data with the Texis relational database optimized for full-text search. Since 1981 Thunderstone has continued to develop its global reputation as provider of the world's most powerful, scalable and flexible enterprise search solutions.

Sales contact: Fred Harmon

+1 216 820 2200 ext.105

Media contact: Peter Thusat

+1 216 820 2200 ext.118