Bench Talk

(Source: imagIN.gr photography/Shutterstock.com)

After more than a century of research on Artificial Intelligence (AI), the field has recently become both popular and enormously important. In particular, Pattern Recognition and Machine Learning have been revolutionized through Deep Learning (DL), a relatively new moniker for Artificial Neural Networks (NNs) that learn from experience. DL is now heavily used in industry and daily life. Image and speech recognition on your smartphone, and automatic translation from one language to another, are just two examples of DL in action.

Many people in the Anglosphere assume that DL is a creation of the Anglosphere nations. However, DL was, in fact, invented where English is not an official language. Let us first zoom back and have a look at AI history in the broader context of computing history.

One of the earliest mechanical computing machines was the Antikythera Mechanism, built in Greece in the first-century BC. Running with 37 gears of various sizes, it was used to predict astronomical events (Figure 1).

Figure 1: The Antikythera Mechanism was built in Green in first-century BC. The device consisted of 37 gears of various sizes. It was used to predict astronomical events. (Source: DU ZHI XING/Shutterstock.com)

The sophistication of the Antikythera mechanism was not surpassed until 1,600 years later when Peter Henlein of Nürnberg began building miniaturized pocket watches in 1505. Like the Antikythera mechanism, however, Henlein’s machines were not general machines calculating results from user-given inputs. They simply used gear ratios to divide time. Watches divide the numbers of seconds by 60 to get minutes, and minutes by 60 to get hours.

In 1623, however, Wilhelm Schickard in Tübingen constructed the first automatic calculator for basic arithmetic. This was soon followed by Blaise Pascal's Pascaline in 1640, and Gottfried Wilhelm Leibniz' step reckoner in 1670, the first machine to perform all four fundamental arithmetic operations of addition, subtraction, multiplication, and division. In 1703, Leibniz published his Explanation of Binary Mathematics, the approach to binary computing that is now used by virtually all modern computers.

Mathematical analysis and data science also continued to develop. Around 1800, Carl Friedrich Gauss and Adrien-Marie Legendre developed the least squares method of pattern recognition through linear regression (now sometimes called "shallow learning"). Gauss famously used such techniques to rediscover the asteroid Ceres by analyzing data points of previous observations, then using various tricks to adjust the parameters of a predictor to correctly predict the new location of Ceres.

The first practical program-controlled machines appeared at about this time in France: automated looms programmed by punch cards. Around 1800, Joseph Marie Jacquard and colleagues thus became the first practical programmers.

In 1837, Charles Babbage of England designed a more general program-controlled machine called the Analytical Engine. Nobody was able to build it, perhaps because it was still based on the cumbersome decimal system instead of Leibniz’ binary arithmetics. However, in 1991, at least a specimen of his less general Difference Engine No. 2 was shown to work.

At the beginning of the 20^th century, progress toward intelligent machines accelerated dramatically. Here are major milestones related to the development of AI since 1900:

• In 1914, Spaniard Leonardo Torres y Quevedo built the first chess-playing machine, using electro-magnetic components. It could play out king-rook endgames from any position without human intervention. Back then, chess was considered an intelligent activity.
• In 1931, Austrian Kurt Gödel became the founder of AI theory, and of theoretical computer science in general, when he introduced the first universal coding language that was based on integers. He used it to describe general computational theorem provers and to identify the fundamental limitations of mathematics, computation, and AI. Much of the later work in AI and expert systems during the 1960s and ‘70s applied Gödel’s approach to theorem proving and deduction.
• In 1935, American mathematician Alonzo Church published an extension of Gödel's 1931 results, solving the Entscheidungsproblem or decision problem, introducing an alternative universal language called lambda calculus. This is the basis of the popular programming language LISP. Alan Turing in the U.K. reformulated that result in 1936, using yet another equally powerful theoretical construct, now called the Turing machine (Figure 2). He also suggested a subjective AI test.

Figure 2: Alan Turing in the U.K. reformulated the popular programming language LISP in 1936, using theoretical construct called the Turing machine. (Source: EQRoy/Shutterstock.com)

• Between 1935 and 1941, Konrad Zuse built the first practical, working program-controlled computer, the Z3. In the 1940s, he also devised the first high-level programming language, and used it to write the first general chess program. In 1950, Zuse delivered the world’s first commercial computer, the Z4, several months before the first UNIVAC.
• Although the name "AI" was coined by John McCarthy at the Dartmouth Conference of 1956, the topic was addressed five years earlier at the famous conference on computers and human thought in Paris ("Les Machines à Calculer et la Pensee Humaine”). Herbert Bruderer rightly calls it the first conference on AI. During that conference, in which hundreds of world experts participated, Norbert Wiener played a game of chess against Torres y Quevedo’s famous chess machine mentioned earlier.
• In the late 1950s, Frank Rosenblatt developed perceptrons and simple learning algorithms for "shallow neural nets." These were actually variants of old linear regressors introduced by Gauss and Legendre around 1800. Rosenblatt later also thought about deeper nets but did not get very far.
• In 1965, Alexey Ivakhnenko and Valentin Lapa, two Ukrainians published the first work on a learning algorithm for deep multilayer perceptrons with an arbitrary number of layers. If there is a "father of deep learning" in feedforward networks, it is Ivakhnenko. His nets were deep even by post-2000 standards (up to eight layers). And like today's deep NNs, they learned to create internal representations of incoming data that are hierarchical and distributed. In recent decades, deep learning has become very important. It is a specialized branch of AI somewhat related to the human brain that contains about 100 billion neurons, each connected to 10,000 other neurons. Some are input neurons that feed the other neurons with data (sound, vision, tactile, pain, hunger). Others are output neurons that control muscles. Most neurons are hidden in between, where thinking takes place. Your brain learns by changing the strengths or weights of the connections, which determine how strongly neurons influence each other and encode all your lifelong experiences. Today’s DL artificial neural networks (NNs) are inspired by this and learn better than previous methods.
• In 1969, Marvin Minsky and Seymour Papert's famous 1969 book “Perceptrons: an introduction to computational geometry” about the limitations of shallow learning, discussed the problem that had in fact been solved four years earlier by Alexey Ivakhnenko and Valentin Lapa. It has been said that Minsky's book slowed NN-related research, but that is not the case, or certainly not for research happening outside the US. In subsequent decades, many researchers, especially in Eastern Europe, built on the work of Ivakhnenko and others. Even in the 2000s, people were still using his highly cited method for training deep nets.

So much for the history up to 1970. AI History Part II will take a closer look at what has happened since then.