Thunderstone Blog: Customized Search Engine & Search Software

Access to information is vital for any business or organization. For the healthcare industry, this access can have life-changing effects. Unfortunately for healthcare providers, researchers, and administrative staff, healthcare organizations don’t always have an effective search solution that allows health workers to efficiently access the information they need to complete crucial tasks or make important decisions.

How bad is the search problem in the healthcare industry? According to a TEDx Talk by Dr. Karl Kochendorfer, some healthcare professionals turn to Wikipedia for vital information instead of official sources because they have such a hard time finding what they need through their organization’s search setup. In addition, Kochendorfer cites that clinicians spend a third of their time simply looking for information.

It shouldn’t be hard for healthcare professionals to find the information they need, but there are many obstacles that stand in the way. High-performance search engine software and search appliances can be part of the solution.

Why Does the Healthcare Industry Need a Better Search Solution?

A massive amount of data without the proper search solution makes for an inefficient, ineffective setup for retrieving key information. Healthcare information takes many forms, all of which play an important role. These can include just about anything that can be documented and stored, such as:

• Patient information
• Procedural details
• Patient care policies
• Reference tools and medical literature
• Forms
• Benefits information

No matter what form these pieces of information take, a single health care organization can amass a massive amount of data that must be available to a wide range of staff. According to the Stanford Medicine 2017 Health Trends Report, “the sheer volume of health care data is growing at an astronomical rate: 153 exabytes (one exabyte = one billion gigabytes) were produced in 2013 and an estimated 2,314 exabytes will be produced in 2020.” Even the medical collection of one doctor can become massive, as the Jeghers Medical Index’s contained a collection of roughly five million physical pages amassed by Dr. Harold Jeghers.

Another issue is that these documents are typically stored in various repositories, such as:

• Content management systems
• Institutional document libraries
• Team sites
• Electronic medical record databases
• Intranets
• Email servers
• File systems

As healthcare organizations continue to gather more information, they need to find a way to store new documents with pre-existing files and other data. Given the wide variety of medical subjects and types of files, information is typically stored in separate data silos.

While this system organizes different types of information into logical groupings, it can also create a situation where individuals need to search individual portals to find the right information they need. For example, a silo that contains insurance carrier documentation would likely not allow professionals to parse through notes from doctors and other providers. This scenario can make searches take longer than it should, and that’s if these users are even able to find the information they need. According to ComputerWeekly “it is estimated that knowledge workers spend more than twice as much time recreating content than they spend creating new content” because they can’t find existing files.

That means that healthcare organizations need to invest in high-performance search engine software or hardware that can not only handle hundreds of thousands of files, but also give users the ability to quickly and easily find what they need regardless of where the information is stored.

How Enterprise Search Technology Benefits Healthcare Organizations

Access to information is crucial. A high-performance enterprise search solution gives you the ability to search internal and external information across an entire organization. Enterprise search is designed to parse through a vast array of information found on multiple repositories. This allows different types of users within that organization, such as medical practitioners, administrative staff, and others, to look for and find the information they need.

However, the same user can have different needs depending on the specific search. For example, a doctor searching for drug facts during a patient visit will want different results than one looking up medical literature during lab work. To combat this, custom filtering can be set up so that certain types of results show up depending on the device used, such as a desktop in a research laboratory or an office laptop. In addition, custom filtering can be added to the searches themselves so that users can improve the relevancy of their search results both before and after their initial query.

While enterprise search software allows user to quickly and easily search for information, the technology behind it is far more complex. Enterprise search technology actively organizes information found in multiple repositories through metatagging, taxonomies, and categorization. This process creates a searchable index from various repositories. This enables the search software to logically find information for future queries so that users receive relevant information both quickly and efficiently.

Part of the reason enterprise search software makes it easier for users to find what they need is the types of searches it allows them to do. A customizable thesaurus feature allows organizations to improve the effectiveness of terminology that is more common to their industry. This can help improve the relevance of results by tying in generalities, associated phrases, acronyms, spelling variations, or other appropriate word associations. For the healthcare industry, this can help clear up issues when professionals search for a brand name drug or a generic drug, procedure names, medical abbreviations, or any other healthcare-specific terminology.

Enterprise search can also improve the effectiveness of searches involving statistics or data ranges. If a healthcare professional had a specific question or needed to find a certain data point, enterprise search would be able to parse through relevant literature and studies to yield appropriate results.

Another issue is that some information should only be accessible to certain users. Whether it’s due to HIPAA, meaningful use, or any other reason, certain documents should only be findable by certain individuals. HIPAA in particular requires a search solution that protects patient record from the wrong people. For example, even a search snippet can accidentally reveal patient names, so it’s important to invest in a solution that can anonymize these snippets and keep files accessible only by approved users. Enterprise search can be designed to give and remove access to certain repositories to different classes of users, whether it’s an admin level user or a member of the public.

How Thunderstone Solves Search Issues for Healthcare Organizations

Of course, the success of an enterprise search solution is only as good as the technology is uses. Thunderstone offers a suite of search solutions It is the only search engine software available with the fully integrated structure of an SQL relational database (RDBMS) that intelligently queries and manages databases containing:

• Natural language text
• Standard data types
• Geographic information
• Images
• Video
• Audio
• Other payload data

In addition, Thunderstone offers search engine software solutions that can handle databases with tens of millions of large records and even hundreds of millions of records and can store and search text documents of unlimited size within standard database tables.

Another issue is that medical documents can contain specific language not found in typical files. Thunderstone search solutions have an English language vocabulary of up to 250,000 word and phrase concept associations for natural language queries, as well as excellent proximity control, fuzzy searches, true regular expression matching and written numerical value searches for even more refined, accurate searches.

In short, Thunderstone enterprise search technology gives healthcare organization the power to handle massive amounts of data and the flexibility to manage their needs. If you’re in need of a search solution that allows gives your organization the ability to access your data when your users need it, request a demo today to test our technology for yourself or contact us to talk to a Thunderstone expert and your organization’s search needs.

Not all websites search needs are the same. A custom solution may be needed to handle your searching needs. Thunderstone's TEXIS can provide you with a full text search engine that will give you the power you need to handle your internal searches.

What Exactly is TEXIS?

TEXIS is a fully-integrated full text search engine. That makes TEXIS a flexible option that works with numerous integrated applications, including:

• message profiling and handling
• image library management
• help-desk support
• online news retrieval
• business intelligence
• research libraries
• litigation support
• Internet retail operations

TEXIS is powerful search engine option that can handle 10,000 records and more with an English language vocabulary of 250,000 words, plus phrase concept associations for natural language queries. In fact, TEXIS is the core technology for our other products, as it encompasses database and search engine capabilities.

What Makes it Different from Other Search Engines?

The key for TEXIS is versatility and customization. TEXIS isn't limited to website or fileserver searches; it can be tailored to do even more, giving you an option that be customized to be as powerful as you need it to be. TEXIS is the only search engine with a Structured Query Language configuration. This provides you with added benefits and advantages when it comes to complicated search requirements.

Fully-Integrated Full Text Search Engines from Thunderstone

Some of our customers use TEXIS as a search engine, others need it as an e-commerce system. Our TEXIS technology can be customized to fit your needs, so don't be afraid to tell us what you need and we'll be happy to work with you.Contact us online or give us a call at 216-820-2200 for more information on how TEXIS from Thunderstone can benefit you.

Executive Summary:

• TEXIS is the only fully integrated SQL RDBMS optimized for full-text search.
• TEXIS has high-performance ability to intelligently query and manage databases containing natural language text, numeric values, standard data types, geographic information, images, video, audio and other payload data.
• TEXIS powers real-time applications with zero-latency data insertion, providing immediate search availability of key data without waiting for scheduled index updates.
• TEXIS efficiently sorts and groups search results by any field(s) in the data. It can quickly sort tens of thousands of hits or more.
• TEXIS, the innovative development platform behind Thunderstone's entire line of enterprise search products, lets users and developers incorporate their own unique knowledge and expertise into customized search solutions that easily integrate with other applications.

What makes Texis different from other search engines and databases?

Thunderstone's Texis is the only search engine developed from the ground up as a fully integrated SQL RDBMS optimized for full-text search, and it's the only relational database that can store and search text documents of unlimited size within standard database tables.

Used by hundreds of thousands of database application developers around the world, Structured Query Language provides many advantages for satisfying complicated search requirements. SQL also holds great promise as a reliable, well-defined path for implementing unanticipated new search functionality in the future. All other search engines offer a much narrower range of possibilities based on proprietary interfaces.

While typical search engines do a nice job of searching unstructured text and traditional databases have an impressive ability to handle queries on fielded or structured data, text searching and relational database management each rely upon radically different paradigms for organizing and retrieving information. They both developed and matured over decades as completely separate technologies, and they don't “marry” easily.

Thunderstone is the only company that has accomplished the true marriage of a full-text search engine and a powerful SQL relational database in a single platform. Addressing this challenge, the simultaneous searching of structured and unstructured data, remains one of Thunderstone's core competencies.

Deep in the heart of the Texis RDBMS resides Thunderstone's Metamorph, a concept-based natural language search engine utilizing advanced lexical set logic.

Metamorph has often been classified as a form of Artificial Intelligence, since its functions fall into the categories of knowledge acquisition, natural language processing and intelligent text retrieval. The software attempts in its own way to understand your search queries, to represent its understanding to the data in the files and to come up with relevant responses as retrieved portions of full-text information which best correspond to your submitted queries.

Metamorph's starting vocabulary has 250,000+ word connections, constructed in a dense web of associations and equivalences. Search parameters can be adjusted to dynamically dictate surface and deep inference. The program's responses can be controlled so that they are direct or abstract in relation to user queries. Proximity of concept can be fine tuned so as to qualify degree of relevance, providing matches which are sometimes concrete, sometimes abstract, as desired.

Metamorph allows for editing word sets. This means that you may select which associations you would like in connection to any search. You can create your own concept sets permanently for future use. You can fine tune the search to use associations of only a certain part of speech. You can enter all known spelling variations of any particular search word in the same way. You can generally customize the program to include your own nomenclature and vocabulary, making it increasingly intelligent the longer it is in use. When you want to control exactly what associations are made with any or all of the words or expressions in your searches, you can do so by editing the equivalence set associated with any word already known by the Equivalence File or by creating associations for a new or created word not yet known.

You can call up the ApproXimate Pattern Matcher (XPM) and tell it to look for a certain percentage of proximity to an entered string, finding misspelled names and typos. You can also look for numeric quantities entered as text, thanks to the Numeric Pattern Matcher (NPM) which recognizes that “four score and seven” is the same amount as “87.”

Metamorph allows users to search for intersections of sets of lexical items, while also performing prefix and suffix morpheme processing. Users can specify, right in their queries, the delimiters of choice: i.e., they can look for lexical intersections within a sentence, a paragraph, a page, a designated amount of text or some other defined textual unit such as a memo.

Texis, with Metamorph inside it, provides a modular set of tools to attack the formidable problem of how to get at and deal with a large volume of information when you don't really know precisely what you need or where to find it. Thunderstone's Texis gives you the power, speed and flexibility to rapidly implement a customized search solution that will accomplish your data access/retrieval objectives in the most dynamic, efficient and pragmatic way possible.

Thunderstone's Texis has a number of characteristics and built-in advantages that differentiate it from other search solutions:

1. A fully integrated SQL database management system (DBMS) that follows the relational database model, Texis is optimized for addressing the inclusion of unlimited quantities of narrative full text. It provides a method for managing and manipulating an organization's shared data, where intelligent text retrieval is harnessed as a qualifying action for selecting the desired information. Texis simultaneously provides full-text, fielded and Boolean searching of both structured and unstructured content.
2. Texis powers real-time applications with zero-latency data insertion, providing immediate search availability of new data without waiting for scheduled index updates. Unlike other search tools, Texis ensures that all information which has been added to any table can be searched immediately -- regardless of whether the table has been indexed and regardless of whether it has been suggested that an index be maintained on that table or not. Sequential table space scans and index-based scans are efficiently managed by Texis so that the database can always be searched in the most optimized manner with the most current information available to the user.
3. Texis enables searchers to sort and group query results by any field(s) in the data. And Texis can quickly sort tens of thousands of hits or more. Other search tools either bog down sorting more than a few hundred items or else their sorting features are much more limited than the capabilities of Texis.
4. Texis allows you to treat the concatenation of any number of text fields as a single “virtual field.” As a single field you can create an index on the fields, search the fields and perform any other operation allowable on a field.
5. Texis has high-performance ability to intelligently query and manage databases containing natural language text, numeric values, standard data types, geographic information, images, video, audio and other payload data. While Texis excels at purposeful manipulation of textual information, it also performs useful mathematical operations on your data. You can construct queries that combine calculated values with text search.
6. Texis lets you create an unlimited number of independent search collections -- each with their own unique data types, fields, attributes or parameters. It also empowers users to submit queries to multiple search engines and/or multiple collections and have the results displayed together or combined.
7. Texis gives developers prototype-friendly customization tools, extreme flexibility, rapid deployments and a feature-rich API. It supports multiple Search User Interfaces that offer specially-defined views of query input and results for different audience types or even for each unique individual. Thunderstone's Texis imposes no user interface requirements. Texis Web Script (Vortex) maintains “neutrality” with regard to whatever HTML markup (or JavaScript or other user interface technology) is employed for the user results presentation.

Which enterprise search applications require the robustness and flexibility of Texis?

Texis is the premier solution when large-scale, mission-critical and/or complex information retrieval challenges call for full-text searching tightly integrated with traditional structured database querying. Businesses, governments, NGOs and educational institutions use Texis in a wide range of applications such as online catalogs, auctions, classifieds, automated categorization, litigation support, intelligence collection/analysis, risk assessment, quality control, CRM, knowledge discovery, document and multimedia management, internet publishing, vertical portals, real-time message handling, web searching and many others.

Thunderstone's Texis provides the ideal development platform for rapidly deployable, custom-designed applications that require both unstructured and structured types of searching:

• Online catalogs contain unstructured text (product name, description, etc.) and structured content (style/size, price, in-stock availability, etc.) Users expect the ability to search by item description, to navigate by price range or to do both in combination.
• Knowledge management systems demand very efficient and secure enterprise-wide information retrieval across multiple repositories that serve different types of users, who all want dynamic, context-sensitive views of defined content (structured data) with the ability to refine results through full-text searching (unstructured data).
• A Thunderstone solution provider customer has deployed Texis in a "brute force" full-text search scenario for its DoD Intelligence Community customer, using Thunderstone's Texis to search the contents of a massive Oracle database in a counter-terrorism effort. Texis is being used as an adjunct to Oracle full-text search because of its ability to scale while still providing superior performance in both rate of ingestion as well as search. Thunderstone's Texis enables this customer to search across a 20 terabyte index, ingesting 70-80 million new records per hour and returning typical search results in < 10 seconds.
• A Fortune 20 customer is using Thunderstone's Texis as the search platform for what they describe as the "single largest knowledge management system currently deployed at any corporation in the world." The application encompasses knowledge, people and processes, and it is used globally within the organization to access more than 30 terabytes of data. Users access the application 20+ million times per day, retrieving and sharing information from across the global enterprise. The application is the most-used corporate I.T. resource after e-mail.

Thunderstone's Texis lets users and developers incorporate their own unique knowledge and expertise into customized search solutions that easily integrate with other applications. For additional information call +1 216 820 2200 or visit us online at http://www.thunderstone.com.

The METAMORPH and METAMORPH INVERTED indexes in Texis are used to improve the performance of text searches using full-text queries with LIKE, LIKEP, and the rest of the LIKE family. Often the query involves other values, which are used to either sort the results, or further restrict the results returned.

One example is in the Webinator application, which provides the option to sort the results by date. Historically, the way to improve the performance of the ORDER BY was to use an INVERTED INDEX. If you also wanted to do date range restriction, then you could add a regular INDEX as well.

The Metamorph compound index will provide better performance than the three indexes since all the data is available from a single index, and also requires less maintenance. For the query:

You could create the index as:

Which is the CREATE INDEX statement you will find in the Webinator dowalk script.

If there are several fields that you might use in the query or ORDER BY, then you can specify all of them as additional fields. The order of the fields does not matter, and the engine may use any combination of them. If in Webinator you also wanted to allow searches and sorts based on the Depth field, you could add Depth to the index:

Then, with the ability of Vortex to ignore parts of the query you could write a query:

That will allow efficient searching and ordering on any combination of Visited and Depth, as long as a query is specified for the LIKE.

The compound index can also be used for GROUP BY or other queries that can fully rely on the index data, e.g.:

Key facts

• In a full-text index (any of the variations of METAMORPH INDEX) the first field specified must be the full-text field, and will be indexed accordingly.
• The first field may be a virtual field, if you want to search across multiple database fields. In the above example we would search the Title, Description, Keywords, Meta and Body fields as if they were a single field.
• The full-text index will only be used if the full-text field is being queried with a full-text query. In the above example, if there was no LIKE clause, or it was dropped by Vortex because it matches $null, then the METAMORPH INDEX would not be used.
• The additional fields beyond the full-text field should be small, fixed size fields, most commonly dates and numbers.
• Using too many additional fields can negate the performance benefits of having the index. Care should be taken to ensure that only those fields actually used in queries are represented in the index.
• The total size of the additional fields should be small relative to the size of the record, and should not exceed a few hundred bytes per record.
• The total size of additional indexed data (number or rows multiplied by size per row) should be no larger than 25% of physical memory on the server.
• If you specify a VARCHAR(N) field as an additional field, you will get a warning message "Variable size warning". The index will still be created, and N bytes of the field will be indexed (where N is from the declaration of the field) for each row. If N is large, this will bloat the index, reducing performance.
• Updating fixed size fields, including the additional fields can be done without causing the index to go out of date and needing to be updated. Updating the full-text field, or any variable sized field (e.g. VARCHAR, BLOB, INDIRECT) will still cause the index to require an update.
• Parts of the where clause that use the compound should be grouped together with parentheses for maximum efficiency.

February 2011 - Archive

CONTENTS

• Texis Version 6 Released
• New Product Survey
• Tech Tip: Multicolor Query Highlighting
• Subscribe/Unsubscribe and Contacts

TEXIS VERSION 6 RELEASED

The newest version of Texis, Version 6, is now in full release. New features include:

• More intuitive searches, including Unicode support and accent insensitive searches.
• Improved XML/XSLT support, including a new API for building XML based applications more easily.
• HTTP/1.1 support including gzip compression to reduce crawl times and bandwidth utilization.
• Enhanced options for showing search results, including multiple snippets and highlighting using CSS styles.
• Many more... See a full list of Version 6 features here.

Call us today at 216-820-2200 to find out how the latest version of Texis can help you.

NEW PRODUCT SURVEY

Thunderstone Software is currently working on a new web-based email product. We'd like to hear about what you might be looking for in an email solution, from search to archiving to contact management and beyond. Please take a minute to fill out our brief survey and share your thoughts. Thanks!

TECH TIP: MULTICOLOR QUERY HIGHLIGHTING

Highlighting can be done differently with 3 <fmt> flags:

• %mI - generates CSS code in an inlinestyle attribute at every hit. The highlighting will then not be depend on any external resources.
  • <fmt "%mIH" $query $text> gives:
    ... average, the <span style="background:#ffff66;color:black;font-weight:bold;">airport</span> will see ...
• %mC - places unique cssclasses on each search hit. This allows you to apply your own customization by defining the class yourself.
  • <fmt "%mCH" $query $text> gives:
    ... average, the <span class="query queryset2">airport</span> will see ...
• %mb - the previously existing method that bolds the matches is still supported.
  • <fmt "%mbH" $query $text> gives:
    ... average, the <b>airport</b> will see ...

You can also highlight the query with itself to provide a legend of the highlighting styles as the "search results for" above: <fmt "%mIH" $query $query>

Feedback, suggestions and questions are welcome. Send your email to .