So, what is PDL, and how does it achieve semantic understanding while regex and string search do not? Let’s take a look at an example to find out.

First of all, PDL does not just search for the literal form of the word in the query: instead, it automatically extends its search to all morphological forms of the word. For example, when searching for the word “company” in financial news articles, the PDL query will not only find “company”, but also “companies,” the plural form. This feature often comes in handy, especially when the search involves a verb. Suppose that you are interested in extracting what the CEOs said. With regex or other substring search, you will need to list all possible verb forms such as “say,” “saying,” “says,” and “said.” With PDL, simply entering “say” in the query will automatically fetch all possible verb forms. This behavior can also be turned off by enclosing the word in the form function, which will then restrict the search to the literal form of the word, such as in the example below.

Another notable feature of the PDL language is its capability for users to tailor the scope of their searches using a range of built-in functions. Returning to the previous example, you may not wish to confine your search exclusively to the specific verb “say,” but rather include other synonymous verbs like “tell” or “mention.” Achieving this is straightforward with PDL – users can invoke the synonym function with the verb “say,” as demonstrated in (a) below. As the subsequent results table (b) illustrates, the captured text now includes various speech verbs such as “tell,” “emphasize,” and “claim,” in addition to the word “say,” capturing them in all possible verb forms. For additional flexibility, the user can also create and modify synonym dictionaries.

The PDL language offers various modes of information extraction, including proximity search (e.g., finding words A and B within a sentence, or within a 3-words range), syntactic relation (e.g., finding word A that is the subject or object of B), semantic relation (e.g., finding words that are synonyms/antonyms to word A), access to dictionaries and ontologies, and more. This language is expressive enough to capture complex patterns, and yet relatively easy to use, having a syntax that closely resembles English. Having access to this versatile query language significantly enhances the power and quality of text mining operations.

In conclusion, PDL is a powerful and versatile query language that enables users to extract meaningful information from text with greater efficiency and accuracy than competing methods like regex or string search. Its ability to understand and capture morphological forms, synonyms, and other complex patterns makes it an indispensable tool for solving text mining tasks that require semantic understanding. By leveraging the capabilities of PDL, users can enhance their information extraction processes and gain valuable insights from their data, making it a true Swiss army knife of information extraction.

The post Query languages—the Swiss army knife of information extraction appeared first on Megaputer Intelligence.

As a services company, EDS people are the key element of the company. Whereas software, hardware, and systems companies offer value mainly through products, services companies like EDS serve clients through people. The knowledge, skill, and experience of employees are the lifeblood of the company. As such, all corporate goals have important workforce components, such as change management and continuous re-skilling. In addition, maintaining the values of the company is an ongoing workforce priority.

Central to maintaining an effective workforce, is employee feedback. EDS leaders use a variety of forums to engage in constant dialog with employees. One mechanism for feedback is the EDS employee survey program.

The EDS survey program includes an annual survey, the Voice of the Employee (VOE) survey, in which all employees world-wide are invited to participate. Additional surveys are implemented three to four times a year. Called Pulse Surveys, these additional surveys are sent to stratified samples of employees across the company.

Historically, employees were providing responses to scaled questions on surveys, which delivered quantitative results along several variables such as leadership, communication, and the work environment. This data was analyzed and interpreted using traditional tools.

Since 2003, open-ended questions have been added to employee surveys. Text generated from open-ended questions offers a broader, richer source of information than do scaled questions alone.

Unstructured text data, of course, presents new challenges as well. Initially, text comments from open ended questions were processed manually. Teams of employees reviewed the text then coded each record into one or more topics. Inter-rater reliability was maintained by having several employees code a portion of the text, then comparing the results. After the coding process, summaries and examples were created and reported to EDS senior leaders for follow-up action.

While the value of text comments for improving organizational effectiveness was unquestioned, two problems persisted. First, the process was slow. Whereas survey reports of quantitative survey data were available a few weeks after a survey closed, text comment reports lagged by months. Moreover, the process at times involved more than a dozen employees. From a productivity standpoint, the slow, manual process could not be sustained.

Second, the manual coding process was difficult to adjust to varying situations. For example, if a topic became too broad to be meaningful (e.g. “unit-level communication”), a principled subdivision was difficult to achieve. At the other extreme, a single, widely-distributed comment (e.g. “many emails contain redundant information”) was difficult to link to higher-level topics.

By 2005, EDS survey analysts set three high-level goals for analyzing text comments. First, cycle time must be reduced to allow text and quantitative data to be reported at the same time. Second, the resource requirement must reduce from dozens of employees to one person. Third, the new approach must be scalable to a wide variety of projects.

To view the full document, please download the PDF: Full Nomination (PDF)

The post Megaputer nominated for DM Review’s 2006 World Class Solution Award in the category of business intelligence appeared first on Megaputer Intelligence.