Keyphrase Hierarchies: A Multi-layered Approach to Bot-building


by Psimagus
I tend to think of a mind, be it human or artificial, as a multi-layered kind of a thing - a sort of conscious (or pseudo-conscious, in the case of bots,) laminate. And in my own bot-building (text-based at least,) these layers seem to fall into a few more or less defined categories. I didn't set out to build BJ to this specification, and I certainly didn't begin with the lowest-level keyphrases, and work my way up to the more specific and specialized ones, but I have always kept in mind that there is a natural hierarchy that emerges from regular development.

At the lowest level, there are stylistic cues that can be taken from raw-mode regex (regular expression,) analysis of the spelling and punctuation of a human user's input, that can add a good deal of apparent intelligence to a bot, by reducing the reliance on xnones, even when none of the content matches a higher-ranked keyphrase. These I would term orthographic.
Examples would include the identification of a question or exclamation from the terminal punctuator, apparent hesitation when the user ends with dot dot dot or begins with a variant of hmmm, the presence of a numeric value in the input, Leeds-style L33t 5p33k, or aaarrggh!-type utterances. Because they are handled in "raw" mode, these don't rely on the content of the user input at all, to the extent that they would work equally well in dealing with foreign language input - it does not matter if the vocabulary used is entirely absent from the WordNet dictionary, or if the LinkGrammar script can make no grammatical sense of it at all.
Care needs to be taken to tailor the responses to any eventuality, to avoid inappropriate responses, but it is a step up from the subject-changing question or random interjection that is usually the best that can be achieved with an xnone.

I know a lot of people avoid regexes, thinking them to be difficult or unpredictable, and this is not helped by the scant coverage that is afforded them in the Book of AI. But they are worth learning about, enough to use some basic forms at least, and there are plenty of resources on the Net to get you started; eg:

http://www.regular-expressions.info/tutorial.html
http://www.zvon.org/other/PerlTutorial/Output/index.html
http://weblogtoolscollection.com/regex/regex.php
http://www.silverstones.com/thebat/Regex.html
Unfortunately the Forge's regex-handling is a slightly unusual flavour, and many advanced regex techniques won't work reliably. But ^ $ | \ ( ) [ ] + etc. work exactly as expected.

Some examples:

NB: The following examples are given in the PF export file format. If you're working in the web interface and aren't familiar with working offline, the format is:
keyphrase [rank,emotion] {?PF AI Script ?}
- just insert the relevant data into the appropriate boxes. If you're pasting them into an export file, please change the {curly brackets} into the requisite pointy ones (pointy ones won't show up on the web page.)

ORTHOGRAPHIC - spelling & punctuation


It is important to arrange the rankings carefully in cases where contact may overlap, for example ([hmn]+) and ([1234567890]+) need to outrank ([bcdfghjklmnpqrstvwxz0123456789]+) in the above examples, or they will be overridden.

At a slightly higher level (though with partially overlapping ranks,) there is room for a syntactic layer. Keyphrases in this section will only be triggered by reasonably standard English that is parsable by the Forge's LinkGrammar and that substantially conforms to the vocabulary present in WordNet. The bot is confined to commenting about the syntax, rather than the content, but in the absence of more specific 'higher-level' keyphrases, I think this is generally preferable to an xnone, where that can be avoided.

Some examples:

SYNTACTIC - grammar & syntax


The responses are a little more contextually focused than the orthographic layer, and allow access to the various parts of speech, either via keys or the extended PF2 features (vmod, ssub, obj, etc.)

And then there is the mass of "normal" keyphrases that we know and love so well. These are the heart of conversation, perhaps, and can match anything from individual words to complex idioms. There's a lot of overlap between these linguistic keyphrases and the aforementioned syntactic ones, of course, but I think the distinction is still useful.

Some examples:

LINGUISTIC - vocabulary & idiom


And it seems worth specifying a last layer of keyphrases that deals with factual knowledge. I find this can be surprisingly addictive (and distracting!) - BJ certainly has an excessive Trivial Pursuit-style mass of general knowledge, but once you start teaching your bot a little geography or literature or music, it seems a shame not to cover the whole subject in reasonable depth. It would be too annoying to have him miss a chance to show off his knowledge of a small African country because you only covered Europe, for example (though if I could ever figure out how to integrate OpenCyc into BJ, that might reduce the server-side datamass somewhat.)

Some examples:

FACTUAL - general knowledge

... ... ... ... ...

In the above examples, the rankings vary from -1 to 30. Everyone has their own approach to ranking keyphrases, and these examples would probably need some tweaking to integrate into a different bot. My own keyphrases actually spread much more widely than this (varying from about -5 to 127,) - I find I have a tendency of allowing "rank inflation" to creep up on BJ at times, especially when I come up with a particularly pleasing response to a keyphrase that I think may partially overlap other, pre-existing keyphrases.
The absolute range is not as important.as getting effective relative values, and that is largely a matter of personal preference. The total permitted range is -127 to 127, but anything below -30 will almost certainly never appear, and anything much lower than -5 will probably be overridden by the AIEngine in practice.