Thoughts on Aritificial Intelligence

I've always had issues with the Turing Test, not because I disagree with equating computed intelligence with the ability to imitate intelligence, but simply because it equates intelligence with speaking skills. Obviously intelligence has numerous ways of expressing itself other than through speech. Apart from there being no reason for this particular expression of intelligence to be superior to other expressions of intelligence (provided it is a sufficiently powerful intelligence), the Turing Test is actually impeding the progress of artificial intelligence by diverting attention to natural language processing rather than pure artificial intelligence.

The fact is that a sufficiently intelligent organism will learn to communicate with us in whatever way is most appropriate for it, even if it is not in English, just like a dog would. Communication is a consequence of intelligence rather than the other way round. So what I'm saying is that we should focus on developing pure artificial intelligence rather than chat bots, which will then be able to learn to communicate with us in whatever form is most efficient rather than using English. It may be the case than pure artificial intelligence is easier to achieve than programming a convincing chat bot using the methods we are using right now.

If human imitation is what is being sought, than perhaps it would be more practical to imitate something simpler than conversation. An example would be playing a game. Any video game player can vouch for the differences between playing against a computer and playing against another player. Even simple path finding can give away that you are playing against a computer rather than a real human, or even the ability to adapt to strategies used by the player rather than having a fixed exploitable pattern.

But even this is limiting the expression of intelligence, which the computer would ideally be free to express its intelligence in any way. So then how can intelligence be measured without being limited by a particular expression? Intelligence is the ability to discover patterns which can be used to increase the probability of reaching a goal. The process of discovery is usually called "learning". If we are to accept this definition, then I shall make a case against the Turing Test by showing how a plant can be tested for intelligence.

I have lately been dabbling in some indoor gardening by placing a potato in a pot and watering it. Let's illustrate this in a picture:



I have noticed that if you place a light source to one side of the plant, after some time the plant will bend itself to face the light, like so:



And if the light is then moved, the plant will bend appropriately:



This seems like pretty nifty behaviour, but it is not intelligence for it is only responding to a fixed stimulus, in other words, there are no patterns being taken advantage of. But now suppose that we set a revolving light that orbits around the plant which goes around slowly enough for the plant to bend towards it synchronously with the light revolutions. In the absence of intelligence, the plant will simply keep on bending towards the light forever, but what if we come back to check on the plant a few months later and find this:



The plant, rather than bending towards the light, responds to the pattern of the revolving light by adapting itself so that there are leaves at every side of the plant so that at any point in time, there are some of the leaves which are facing the light. Should we ignore biological details on whether or not the extra leaves do more good than harm (maybe the leaves take less energy to maintain than the constant bending), this would be evidence that the plant is intelligent.

Now, if even a plant can be considered intelligent, which is as far away from holding a conversation with a human as an organism can be (I think), think of how unfair the Turing Test is on computers, which requires them to master the human language so much that it fools people into thinking that it is a human. Alan Turing came up with a test for human imitation, and the Turing Test is a perfect test for this, but to claim that this is the best test of intelligence is silly.

I will intentionally not propose a specific test myself, because a specific test would be quantitative and formal in nature, and intelligence is not that. Intelligence can be surprising and we should accept it as that. I would much rather take a qualitative approach to intelligence, where someone proposes a pattern that is being learnt by a computer and researchers investigate its validity.

Operant Conditioning

In behaviour psychology, in order to change the behaviour of a subject, you can use one of 4 different methods. These are categorized into reinforcements (rewards) and punishments and further into positive and negative. It is important to note that positive and negative have nothing to do with how they are perceived, only with whether something is added or removed to the subject.

Method	Description	Example
Positive reinforcement	This is when a pleasurable experience is given to the subject in exchange for increasing the desired behaviour	Getting a good mark for studying or giving a dog a treat for doing a trick.
Negative reinforcement	This is when an unpleasant experience is removed from the subject in exchange for increasing the desired behaviour	Removing some of a class's homework for being quiet or removing a dog's chain for not pulling it.
Positive punishment	This is when an unpleasant experience is given to the subject in exchange for decreasing the undesired behaviour	Getting a report for bad behaviour or giving a dog a scolding for not obeying.
Negative punishment	This is when a pleasurable experience is removed from the subject in exchange for decreasing the undesired behaviour	Removing a class's break time for being disruptive or removing a dog's toy for making noise.

Notice that these need not be manually applied by someone. It could be something like eating too many sweets giving you a stomach ache, which would be positive punishment.

Boolean Algebra and Probability

Boolean Algebra is a field of algebra which formalizes the operations performed in logic, that is, on propositional statements (statements that are either true or false). This requires operations such as "AND", "OR" and "NOT" in order to compose complex statements from simpler ones, such as "it is raining" and "the road is wet" being composed into something like "NOT(it is raining) AND the road is wet". These operators work in the following way:

The statement "Math is awesome AND One Direction suck" is true because each of the simpler statements being joined together is true. If at least one of the simpler statements is false, then the whole thing becomes false.

The nice thing about this algebra is that it can be naturally extended from normal numeric algebra by representing a true statement with 1 and a false statement with 0. So then we can turn the statement "Math is awesome AND One Direction suck" into "1 AND 1" which is equal to "1". Why does it equal 1? Because for "X AND Y" to be true, both X and Y have be true individually. For example the statement "Math is awesome AND One Direction make great music" is false because one of those substatements is false. With the OR operator, as long as at least one of the substatements is true, the whole thing becomes true. For example "Math is awesome OR One Direction make great music" is true because one of the substatements is true. The NOT operator takes one substatement and inverts it, that is, if it was true then it becomes false and if it was false then it becomes true. For example "NOT Math is awesome" is false and "NOT One Direction make great music" is true. Numerically speaking, the following list summarizes these rules:

0 AND 0 = 0
0 AND 1 = 0
1 AND 0 = 0
1 AND 1 = 1

0 OR 0 = 0
0 OR 1 = 1
1 OR 0 = 1
1 OR 1 = 1

NOT 0 = 1
NOT 1 = 0

Can the be some normal arithmetic equations which will give us these outputs?

NOT is easy. What arithmetic equation returns 1 when given 0 and returns 0 when given 1?
NOT X = 1 - X
1 - 0 gives 1 and 1 - 1 gives 0.

AND is also easy. Just multiply the two operands together.
X AND Y = X × Y

OR is a little more tricky. This is because there is not single operation that gives the same outputs as OR. However we can easily derive an equation by using DeMorgan's laws which express an OR using AND and NOT only. The statement "in order to apply for the job you need a degree or five year's experience" can be expressed as "do not apply for the job if you do not have a degree and have less than five year's experience". Formally this is written as

X OR Y = NOT(NOT X AND NOT Y)

which numerically becomes

X OR Y = 1 - ((1 - X)(1 - Y))
X OR Y = 1 - (1 - Y - X + XY)
X OR Y = 1 - 1 + Y + X - XY
X OR Y = Y + X - XY

This makes sense. The OR operator is the addition of the two substatements minus one in case they are both true. This will return 1 whenever at least one of the two substatements is a 1.

But it also makes sense on a higher scale. If you think about it, these rules are the exact same rules for probability. Probability is a generalization of Boolean algebra. Rather than just considering truth and falsity like Boolean algebra does, probability considers values between those two cases, such that 1 is certainty, 0 is impossibility and all the numbers in between are a degree of uncertainty, where the closer the number is to 1 the more certain the statement is.

In fact AND and NOT are defined in exactly the same way in probability. But isn't OR defined simply as X + Y? That is in fact only the case when X and Y are events which are independent, that is, cannot happen together. In this case X AND Y is always zero and hence can be dropped from OR's equation. The equation we have derived is useful for dependent events as well such as asking for the probability of "two fair dice resulting in an even number and a number greater than 3". In this case you can't just add the probability of a die being even (3/6) and the probability of a die being greater than 3 (3/6) as otherwise you get 3/6 + 3/6 = 1 which means that you are certain that this will happen. The problem is that it is possible for the two substatements to be both true, that is, a die being both even and greater than 3 (namely 4 and 6). To fix this just subtract the probability of this special case occurring (2/6) which gives 3/6 + 3/6 - 2/6 = 2/3.

Let's illustrate it with a Venn diagram:


The Venn diagram above is showing two events as a red and blue circle. They have some overlap between them which is coloured purple. If the red circle represented the chance of a die being even and the blue circle represented the chance of a die being greater than 3, then the purple overlap would be the numbers 4 and 6. Now we want to find the probability of the red OR the blue circle occurring. If we just add their areas together we end up with the purple overlap being added twice since it is common to both circles, but we just want to add it once. So we subtract it once after adding both circles, which will result in having just one purple overlap. This is what is happening when you use the equation X + Y - XY.

Don't edit batch files during runtime

So it turns out that Windows batch files running under cmd (at least in Windows 7) are not preloaded into memory before runtime. Instead each line is loaded from disk right before it is executed. What this means is that if you edit the batch file during its runtime, you'll end up modifying the behaviour of the batch file being run.

Try this:

echo started
timeout 10
echo continued
echo ended
pause

This will make the batch file show "started", then wait for 10 seconds and finally show "continued" and "ended". To run it just paste the above code into Notepad and save it as "something.bat", then just double click the saved file.

Now open the batch file with Notepad again and run the file at the same time. As the batch file is waiting for 10 seconds, delete the "echo continued" line and save (before the 10 seconds are up). When the batch file continues, it will not show "continued" but will immediately jump to "ended".

Strange behaviour since this sort of thing does not happen with Python for example. I noticed this when I was trying to run multiple versions of the same batch file by just changing something in the file and running a new instance of it. Be careful of this.

Random Indexing for word space context vectors

Last time I had described what a term document matrix is and how to make it's use faster using Latent Semantic Analysis. Today I shall be explaining how to use Random Indexing as an alternative to LSA. You can read more about it here.

Unlike LSA, random indexing is a very easy and very fast online process. It is online in the sense that you don't need to first completely construct the co-occurrence matrix in order to use it, but can instead use it as you are counting co-occurrences. It also works just as well as LSA. Interested yet?

First of all, how do you construct a co-occurrence matrix? You start off with a matrix full of zeros with columns for words in context w1, w2, w3, and w4 and rows for neighbouring words n1, n2, n3, and n4:

   n1 n2 n3 n4
w1 0  0  0  0
w2 0  0  0  0
w3 0  0  0  0
w4 0  0  0  0

You go through the corpus and you encounter word w2 and near it is word n4. You record this encounter by incrementing the matrix element for w2-n4:

   n1 n2 n3 n4
w1 0  0  0  0
w2 0  0  0  1
w3 0  0  0  0
w4 0  0  0  0

You do this for every neighbouring word next to every word in context and finally you have your co-occurrence matrix.

Now this very same process can be viewed in a different way. Instead of incrementing single numbers, it can be viewed as vector addition. Forget the names given to the columns:

w1 0  0  0  0
w2 0  0  0  0
w3 0  0  0  0
w4 0  0  0  0

Instead assign a vector to each neighbour word where each vector consists of just zeros and a single 1. No two vectors can have a 1 in the same position. So these could be the vectors assigned to our 4 neighbour words:

n1: (1 0 0 0)
n2: (0 1 0 0)
n3: (0 0 1 0)
n4: (0 0 0 1)

Now when you encounter neighbour word n4 next to word in context w2, you just take w2's row, which is a vector, and add it to n4's vector:

w2: (0 0 0 0) +
n4: (0 0 0 1)
    =========
    (0 0 0 1)

Which gives you the updated matrix:

w1 0  0  0  0
w2 0  0  0  1
w3 0  0  0  0
w4 0  0  0  0

Do this for every neighbour next to every word in context and you have your co-occurrence matrix.

All random indexing does is change the neighbour words' vectors. Instead of the vectors being as long as the number of neighbours, the vectors are shortened to a fraction of what they are supposed to be. Consequently the number of columns in the matrix will also be reduced. "But how can you put a 1 in a different position for every neighbour word?" I hear you ask. You don't. What you do is you still keep the vectors mostly zero, but now you randomly put a few 1s and -1s in it:

w1 0  0  0
w2 0  0  0
w3 0  0  0
w4 0  0  0

n1: ( 1  0 -1)
n2: ( 0  1 -1)
n3: ( 0 -1  1)
n4: (-1  1  0)

If we encounter word in context w2 with neighbour word n4, then the matrix will become:

w1  0  0  0
w2 -1  1  0
w3  0  0  0
w4  0  0  0

If word in context w2 also has word n2 as a neighbour, then the matrix will become:

w1  0  0  0
w2 -1  2 -1
w3  0  0  0
w4  0  0  0

Now it will not be possible to know how many times each neighbour word co-occurs with each word in context, but that's OK. What matters is that each time you encounter a neighbour word, it leaves its mark on the word in context's row and that the more often it is encountered, the more its mark is pronounced. This means that you can still find similar words by comparing their rows together. The difference is that the rows are shorter and hence faster to compare.

Notice the online nature of this process. At any point in the corpus traversal, the matrix is always in its compressed form and there are no complex transformations that need to be made on it. You can ever add more words to or remove words from the matrix, without any further processing or extra data. You just have to find a compromise between how many columns to remove from the matrix and the individuality of the rows. The smaller the rows, the less of an individual mark each neighbour word will leave after adding its vector.

Summary of research paper "Unsupervised Lexical Substitution with a Word Space Model" by Dario Pucci, Marco Baroni, Franco Cutugno, and Alessandro Lenci

This is a summary of the research paper http://www.evalita.it/sites/evalita.fbk.eu/files/proceedings2009/Lexical%20Substitution/LS_UNINA_UNIPI_UNITN.pdf. This paper describes a way to automatically choose a word which can substitute another word in a sentence with minimal resources.

Overview

The system described in this paper was used for the EVALITA 2009 where a manually constructed evaluation set in Italian was provided.

In order to determine how well a word can replace another in a context, two types of vectors are used: a contextualized vector and an out-of-context vector. The contextualized vector is a composition of out-of-context vectors.

First, out-of-context vectors are collected from the corpus for each word using the usual method of counting how often each word co-occurs with the word being represented by the vector. The counts are then weighted to give less importance to words that co-occur with many words.

In order to find a substitutable word for a target word in a sentence, the out-of-context vectors of each content word in the sentence are collected and their weights are summed together. This is the contextualized vector. The word with the out-of-context vector that is most similar to this contextualized vector is chosen as the best word to substitute.

Nitty gritty

A lemmatized and part of speech tagged 2 billion token Italian corpus was constructed from which to extract the out-of-context vectors. The 20,000 most frequent nouns, verbs, adjectives and adverbs were considered as content words. Out-of-context vectors for these content words were extracted by counting how often they appear with other content words in the same sentence. The counts were weighted using log likelihood ratios and the co-occurrence matrix of counts was compressed to half its size using random indexing.

To create a contextualized context vector, the context sentence is lemmatized and part of speech tagged. The sentence is then split into a left and right side with respect to the target word and each side is filtered of all non-content words. The words in the sentence sub-parts that are within a set number of content words from the target word are used to construct the contextualized sentence. Their out-of-context vectors are normalized, multiplied by "1/(4*d)" where "d" is the distance of the word in the sentence sub-part from the target word (the closest content word has a distance of 1), and summed together.

Finally, cosine measure is used to find the most substitutable words by measuring the distance of the contextualized context vector to the out-of-context vectors of all the words that have the same part of speech tag as the target word.

Evaluation

The system was evaluated by using an evaluation set which consisted of sentences with a target word and suggested substitutable words by human annotators. Each suggestion included the number of annotators that suggested it. When comparing the best suggested substitutable word by the system with the most popular suggested word by the annotators, the system matched 10.84% of the evaluation set (that included a most popular suggested word).

Proof that there is only one prime number preceding a square number and any other power number (or "What is the use of factoring polynomials?")

Students are made to study how to factor polynomials but are never given an interesting task that involves using polynomial factorization. Here is a use that I find interesting.

Let's say we have the polynomial "x^2 - 1". This is a difference of two squares, which means that you can easily factorize it into "(x+1)(x-1)". But what does this factorization mean?

It means that any number which precedes a square number can always be factored into two numbers: one more than the number that was squared and one less than the number that was squared. For example, 15 precedes the square of 4, that is "4^2 - 1", so we can immediately conclude that "4+1" (5) and "4-1" (3) are factors of 15. Another example is 99 which precedes the square of 10, that is "10^2 - 1", so we can immediately conclude that "10+1" (11) and "10-1" (9) are factors of 99.

OK so maybe this is interesting, but can we use this for something more interesting?

Are there any numbers preceding a square number that are prime numbers? If there is such a number, than it's factors must be 1 and itself. So we are looking for a number of the form "x^2 - 1" such that it can only be factored into 1 and the number itself (which is prime). We said that any number of the form "x^2 - 1" can always be factored into two numbers, "x+1" and "x-1". So if this number, "(x+1)(x-1)", can only be factorized into 1 and itself, then surely one of these two factors must be 1. Which of these factors can be equal to 1? The "x+1" or the "x-1"? Since "x" is a positive number (the number that is squared), it can only be the "x-1" factor.

Will the "x-1" factor equalling 1 automatically make the other factor, "x+1", a prime number? No, because the "x+1" factor could itself be factorizable by other numbers. The fact that "x-1" equals 1 means that "(x+1)(x-1)" could be prime. But if "x-1" is not equal to 1 then, given that "x+1" cannot be equal to 1, "(x+1)(x-1)" is surely factorizable by "x-1", making it not a prime.

In order for "x-1" to equal 1, "x" must be 2. In this case, the number preceding a square number is "2^2 - 1" which is 3. Notice that the other factor, "x+1" would also equal 3 when "x" is 2 which makes sense since we said that one of the factors must be 1 and the other must be the number itself. If 3 a prime number? Yes. So when "x" is 2, "x-1" is 1 and "x+1" is a prime number. Therefore 3 is a prime number which precedes a square number.

Are there any other prime numbers which precede a square number? No, because if "x-1" is anything else but 1, then neither of the two factors will be 1, making the number factorizable into two numbers which are not 1 and itself, hence not being a prime number. This means that 2 is the only number "x" can be, which means that there are no other numbers which precede a square number that are prime.

It's interesting to also note that using the exact same method, one can proof that 7 is the only number preceding a cube number that is prime. Use this website to find the factors of numbers preceding other powers of "x".

It seems that there can only one number preceding an nth power of "x" that can be prime, for every power of "n". Can we prove this?

What will the factors of "x^n - 1" for any "n" be? Regardless of "n", will "x-1" always be a factor of "x^n - 1"? That is, will 1 always be a root of "x^n - 1"? Yes, because "1^n - 1" will always be 0, for all "n". So we can be sure that for "x^n - 1", regardless of "n", "x-1" will always be a factor. This means that "x^n - 1" will always be factored into "(x-1)(...)" where the "..." is some other polynomial factor. Since "x-1" is equal to 1 only when "x" equals 2, then "2^n - 1" is the only number preceding the power number "x^n" that can be a prime number. Of course it could be that the other number could be factored into other numbers hence making "(x-1)(...)" not a prime number when "x" is 2, but if "x" is not 2 then "(x-1)(...)" cannot be a prime number because "x-1" will be a factor.

So, regardless of what "n" is, there can only be one number which precedes "x^n" that can be prime, and that is "2^n - 1", since "x^n - 1" can always be factored into "(x-1)" and some other factor that cannot be equalled to 1. The other factor might not be prime, but if it is then it is the only one preceding "x^n" for all "x". Such prime numbers are called Mersenne primes.

Is this useful? This might be useful for testing whether a number is a prime number by checking if the number precedes a number raised to some power, x^n. If it does then it can only be a prime number if it is a number of the form of "2^n - 1", otherwise it will be always be factorizable by "x-1", regardless of what "x" or "n" are.