Creating a good way to search company websites might seem like a lot easier task than the job facing, say, Yahoo or Google – which have to cover the entire Internet. But, in fact, companies can often face more difficult search-related problems.

Because companies use myriad customized and specialized software programs, they often find that important content is not in the latest Web format – or any Web format at all – making it hard or impossible for search engines to find it. And in a digital world, where costumers expect the latest information at their fingertips, such shortcomings can translate into lost dollars. For instance, if you’re searching an automaker’s site for information about a braking system, it’s likely to be in an owner’s manual in PDF format – not an HTML page. It might also be in a Help section, or somewhere else. For many corporate search engines, finding the information on the first try is not a sure bet.

Solving this problem – while it’s not as glamorous as participating in the latest Web 2.0 startup – has turned into a burgeoning cottage industry. Baynote, an 18-month-old Cupertino, CA company, is one of the latest in a rash of businesses now applying technology to improve corporate Web searching. Baynote’s solution, a Content Guidance System, which was announced yesterday, gives companies a piece of code, equivalent to a Web cookie, to place on their pages and in their search engine. It’s a small packet with a lot to do: tracking, among other things, what a site’s visitors search, how they move their mouse, what results they click on, and how long they spend looking at pages.

The system then feeds all this information back into an analytics engine hosted by Baynote. This engine then does something clever: it analyzes the data to decide when a search succeeds, and also builds a database of successful (and unsuccessful) searches. Over time, then, people who search on companies that use Baynote are given a list of documents that others have viewed when searching on the same term – along with how many people found it useful.

This technique builds on the concept of tagging, says Baynote CEO Jack Jia, in which people mark Web pages with publicly viewable labels that anyone else can use to search pages. The concept, which has become a cornerstone of popular websites such as Flickr and Delicious (both now owned by Yahoo), is drawing interest from businesses since the search terms are defined by a user community, and thus reflect popular opinion about content, rather than relying on either company librarians or computer-driven algorithms.

The problem with tagging, Jia says, is that most people simply don’t do it. And when users do, they rarely go back to update those tags. Baynote’s approach is to treat the search terms as tags, then allow others to see what searches have been successful and for how many people. Its Content Guidance System is currently being used by eight customers, including LSI Logic, Interwoven, and Semiconductor Equipment and Materials International (SEMI), a semiconductor industry association. It’s offered as a hosted Web application (with subscriptions ranging from $95 to $950 a month).

SEMI, which represents more than 2,100 manufacturers of semiconductor-related equipment, has used Baynote for about six weeks, as an enhancement to Alta Vista, the search engine it uses on its website. Jonathan Davis, SEMI vice president of marketing and communications, says the difference is noticeable: people who searched on “Semicon West,” for example, an event that SEMI organizes, used to get an abstract from the 2004 Semicon West as the first link, perhaps because it has the phrase “Semicon West” in it more frequently. Since introducing Baynote, though, the same search pulls up not only this year’s page, but also the registration page, since it’s currently the most sought-after one.

Baynote offers its clients a variety of ways to present information, including guides to what’s popular and what terms others have used after their first search attempt. For its corporate customers, it also generates reports that provide information on common search terms that always fail and where visitors with similar interests go when they’re on the company’s site. Some of Baynote’s technology is its own, but it’s partnered with IBM for full-text search and with Omniture for Web analytics (Omniture and Baynote share a venture investor in Hummer Winblad).

Baynote enters a market with plenty of other options, though. Companies such as Endeca Technologies, Fast Search & Transfer ASA, and Mondosoft A/S have developed methods to improve corporate search, both on company websites and internal corporate Web portals. But Dan Keldsen, an analyst with the Delphi Group in Boston, says these firms tend to focus on either enhancing the navigation aspect or the search aspect, rather than on both. In addition, he says, Baynote’s application “is clever – the search gets smarter as people use it. It’s a fairly novel approach, and there is an indirect kind of wisdom-of-crowds effect.”

Jia points out that Baynote is as much a search-engine enhancer as a pure search tool. He also says there’s plenty of room to improve business searching, citing a study by the Pew Internet & American Life Project showing that 83 percent of all general searches end in failure, and another from Atlas Research that says more than half of Internet users decide not to carry out common actions, such as filling out a registration form, downloading a white paper, or making a purchase, if it takes three clicks or more to find what they want.

While those studies were based largely on consumer search engines, if they’re at all similar to what businesses experience, this new cottage industry may thrive.