Still Smarter than Bots

It’s been very enlightening finding out that we’re still smarter than those little programs that go around and index websites. I realize that computers like IBM’s Watson are making waves on Jeopardy and elsewhere, but semantic understanding seems to still be primarily the domain of humans. Bots come up with the darndest things!

The Downside of the Bots

I and a team of really great indexer/taxonomists learned first-hand in a database project how literal web bots are when they “crawl” a website looking for terms. Our main job was to analyze their “choices” and keep the good/throw out the ridiculous. The web crawler bots went to the sites that our client wanted to get information from and came back with some real doozies, because all those terms on the site are just 1s and 0s to the bot. No distinction between “beadsandbaubles” and “beads and baubles” (or that beads should be a separate keyword from baubles). No concept that “industry” is such a general term that it is virtually useless in search in the context we’re working in.

The Human Advantage

Ah, there it is. Context. Meaning. Significance. Only the humans know what those little scratches called “letters” actually mean when they are combined, particularly in more than one word. We also “get” things like metaphorical, satirical, and even fantastical uses of terms. Semantics and significance are still important to the humans who search for information, so, in order to provide them with quality as well as quantity in search results, we get to make decisions that enhance a site’s quality in the minds of potential customers. A valuable application of our semantic understanding and an important contribution to the businesses and individual authors we serve.