A few years ago I wrote this piece on neural nets. It's still good. Follow the link below for the complete paper, with images; read below for the text only:
Answers to Some Common Questions
What is a neural net?
What is life?
What is a meme?
What are people?
What are thoughts?
What is a neural net?
The basic components of a simple neural net are switches, which send and receive messages, and the pathways between them that transmit the messages. So, a very simple neural net is one composed of two switches and the pathway between them.
Individual switches are very simple. They are not capable of creating new messages themselves, just passing on what they receive. That is, when a switch receives a signal, it simply passes it on along any pathways that are connected to it. Thus if we wiggle switch A in the picture, the wiggle gets passed on to switch B, and then duplicated and passed on to any other switches connected to B, and so on and so forth.
We can see from this that one basic function of a neural net is to transmit a signal from one place to another, or to duplicate information. The reason I say duplicate is because for a piece of information like a signal or message, duplication and transportation are essentially the same thing.
Now imagine a series of switches and pathways arranged so that a signal goes in, travels through several intermediate switches, and is then returned to its source. In this case, a signal travels from A to B to C to D to A. So, what happens if we wiggle switch A is that some time passes as the signal is transmitted internally and then switch A wiggles back in exactly the same way. From this we can see that another basic function of a neural net is to store information over time.
Next imagine a series of switches and pathways arranged in the following way. A signal is input at A, travels to B where it is split, one copy being sent to C and one copy being sent to D; C then sends the signal to D. We presume the signal going by way of C takes slightly longer than the signal going directly from B to D, so what happens with this arrangement is the signal input at A is transmitted to D in a changed form. At A we input “blip” and some time later “blip blip” comes out at D. This example shows us that recombining information comes as naturally to a neural net as do duplication and storage.
Another property of neural nets we can infer from these examples is that all information that goes into one must come out eventually. Like a mosquito flying into an open window, once a signal finds its way into a neural net it doesn’t die, but rather stays inside buzzing around until it finds a way out.
These conclusions are fundamental properties of any neural net, no matter how small or simple, but it is easy to see that as a neural net gets larger, with more switches and pathways, its complexity will increase phenomenally. So, imagine if you will a vast simple neural net with hundreds or even thousands of switches and pathways between them. Obviously the functioning of such a contraption would be highly complicated and intricate, but we can see that it would have many interesting capacities. It would be able to store information about past inputs, which would affect its outputs. It would be able to calculate and create by recombining information. What’s more, its outputs would have an innate tendency to mimic its inputs, perhaps resulting in a need to express itself. If we imagine further that the neural net is not static but is able to grow by adding more switches or pathways then we can imagine that even more astonishing behavior could arise, such as long-term memory, conditioning, and self-directed growth. The neural net would be capable of being conflicted, developing preferences, and getting ideas stuck in its head. Perhaps it would have dreams, hallucinations, or seek emotional release. If this way of imagining a vast, highly connected neural net makes it sound a lot like a person, that’s not a coincidence!
What is life?
Life is bits of self-replicating matter. Starting from scratch a bit of self-replicating matter may be hard to come by, but once there is some it is almost inevitable that at a later time there will be more. The reason is fairly obvious, and gives rise to the principles of variation, selection, and heredity. If some bit of matter makes copies of itself, even if it is not very good at it, at least some of those copies will also have the ability to make copies of themselves and thus over time there will come to be more copies of that original bit. If not all of the copies are perfect then there will be some variation in the resulting population. With variation it follows that some of the copies will make more copies of themselves, and some of them will make fewer. Over time we can expect to see more copies of the former and fewer of the latter. If it is also the case that the characteristics of a particular bit have some influence on the number of copies it is likely to make, then a selection criteria is in effect. That is, the copies are being selected for the traits that make them better copiers. Since the traits of a copy are based on the traits of the original (isn’t this what we really mean by a “copy”?), we have heredity as well, and the three conditions of heredity, variation, and selection are the necessary and sufficient conditions to give rise to an evolutionary process.
So what are the characteristics of an evolutionary process? Once there are little bits of self-replicating stuff what can we expect to see happen? One property of such a process is constant change. Just like a story will change each time it is passed on, as copies are made of copies of copies of copies on and on, the population, when taken at a later time, will generally look different than at an earlier time. Hence variation never runs out, regardless of environmental influences.
A further characteristic of an evolutionary process is that this change will express a detectable progression in two ways. One is a tendency for there to be more replicators at a later time than at an earlier time, which results in a tendency for a population of replicators to come to utilize all available space or materials.
The second detectable progression will be the tendency for the replicators in existence at a given time to be more proficient at making copies of themselves than the replicators in existence at an earlier time. Thus, as a natural result of selection pressure we can expect over time to see the replicators become better at what they do. But what defines a “good” replicator? There are three qualities that measure fitness in this context. One is fecundity, or the number of copies made. This is because if two replicators, A and B, are in direct competition and A makes more copies, on average, than B does, then, all other variables being equal, A will eventually dominate. Another is fidelity, or how accurate the copies are. Similarly to before, if A and B are two replicators in direct competition and all variables are equal except A copies itself more accurately, on average, than B does, then at a later time we will see more true copies of A than true copies of B, so A will eventually dominate. The third is longevity, or how long the copies last. This is because if A and B are equal in every respect except that A lasts longer, on average, than B does, then at any time there will be more copies of A extant than copies of B, so A will eventually dominate. Hence selection pressure favors replicators that make more copies, more accurately, that last longer.
Another characteristic of an evolutionary process is that it constantly makes minor variations and changes to what has already been established. This makes it possible for it to give rise to highly intricate designs and systems. Such systems will display elements of specialization and intentional design.
And finally, another remarkable feature of an evolutionary process is that the environment in which it takes place will tend to be altered, by whatever means the replicators can find, to favor the existence of more replicators. Put another way, an evolutionary process allows for replicators to alter their environment so as to enhance their replication.
Applying these concepts to life allows us to see how an evolutionary process is able to give rise to the living world we find around us. It accounts for the existence of the ultimate bit of self-replicating matter, DNA. It accounts for why the replicators synthesize complicated, specialized vehicles for their transportation and transmission, and why some of these vehicles enter into beneficial alliances while others compete with one another. It accounts for the observation that the surface of the Earth, at some time in the past a molten sea of lava, and later a cooled, rocky, abiotic crust is now a bustling haven of DNA where nary a carbon atom or molecule of water has not participated countless upon countless times in the metabolism of a living organism.
So we can see that the concepts of heredity, variation, and selection as applied to replicators can wield far-reaching explanatory power for the nature of life. But these concepts apply to any self-replicating entities, not just DNA.
What is a meme?
People are consummate imitators. We frequently exercise our ability to imitate, not only other people but also other things in our environment. It is this ability to imitate that enables us to maintain social cohesion in groups and to learn all sorts of skills. It also means that certain bits of information in the form of actions, behaviors, preferences, symbols, ideas, and stories will be copied from person to person and have the opportunity to spread. These bits of information that get copied from person to person are called memes. If we focus not on the people doing the imitating but instead on the memes themselves being copied we can identify an entity that acts as a replicator. Consider a population of people. Because of their nature they tend to copy and learn from each other and thus pass on stories, ideas, and other pieces of information. When an idea arises with an individual there is some chance that it will be transmitted to other people and thus replicate itself. There are, of course, many different possible ideas to be had, and some, once they begin, are more likely to be copied than others. The difference results both from the characteristics of the ideas themselves and from the characteristics of the environment, which is the minds of the people in which the memes spread. Over time, we can expect to see more of the good replicators and fewer of the less good replicators. Thus we can see a selection process in effect, one that selects for the ability of the memes to be transmitted. Variation is ensured since copying is not perfect and heredity is ensured by the nature of the copying process. The existence of variation, heredity, and selection means that memes themselves undergo an evolutionary process.
This evolutionary process displays the same general characteristics as genetic evolution. The body of memes is ever changing as copies are made of copies ad infinitum. Memes tend to proliferate as there are ever more ideas created and passed on between people. They display elements of specialization for specific purposes and intentional design. As time passes memes get better at copying themselves, both by adapting to the existing environment as well as by using whatever means they can find to alter it in their favor so that there are ever in existence more memes making more copies more accurately that last longer and longer.
What are people?
The unique characteristics of people can only be understood clearly as being a result of both genetic and memetic evolution. Certain of our characteristics are obviously the products of genetic selection: we have urges to eat and drink, protect ourselves and our offspring, and have sex. But explanations based on genetic selection alone are lacking when it comes to accounting for such human traits as our constant yammering to each other, our ability to invent mathematical equations and translate the image of a sunset into a work on canvas, our generalized altruism, or the fact that we have such disproportionately huge brains instead of better hearing, smell, tree-climbing ability, or thick body hair. Our ability to mimic other people and learn by imitation must have had some small genetic advantage at least, otherwise we wouldn’t have it. But could the genetic advantage have been enough to outweigh the genetic costs of having a large brain, such as an increased likelihood of dying during childbirth? It is difficult to say based on genetic selection alone, but memetic selection adds a new dimension to the question.
Once even a rudimentary imitation ability came to be developed it gave rise to memes. Memes undergo their own evolutionary process that tends to change the environment so as to favor the existence of more memes. Obviously an environment with more, larger, and more complex brains is a favorable one to the proliferation of memes. This means that memetic evolution can account for the existence of large brains quite aside from any pure genetic advantage they might have had. In essence, our bodies and minds are the combined product of two separate evolutionary processes, the genetic and the memetic. What this implies about our functionality is that we are not only designed by genetic evolution to be very proficient at spreading genes, but that we are also designed by memetic evolution to be very good at spreading memes.
So how does this idea account for us having huge brains? A brain is a biological neural net, designed for the purpose of processing information that is pertinent to the survival of the organism. Recall that the fundamental functions of a neural net are to store, duplicate, recombine, and eventually transmit all the information that it receives. A neural net therefore provides the ideal vehicle for the replication and spread of memes. But not all animals have huge brains capable of giving rise to memetic evolution. This is because a bigger brain doesn’t generally help out much unless the skill of imitation has already been accomplished, just as it doesn’t help much to have flatter arms and lighter bones until rudimentary flight has already been accomplished. But once the threshold of flight has been crossed, selection pressure will favor greater and greater flying ability. In a similar way, once the rudimentary skill of imitation has arisen, selection pressure, partly influenced by the memes themselves, will favor greater and greater imitation ability, which requires a larger and larger neural net.
Another human trait I mentioned is generalized altruism. The difficulty that genetic selection has in explaining altruism is that, except in limited cases, altruistic behavior is detrimental to the genes of the individual who practices it, giving them slightly decreased chances of being passed on. So genes that code for or allow generalized altruistic behavior seem to be at a disadvantage relative to genes that do not. However, generalized altruistic behavior is favorable to the memes of someone who practices it, since other people are more likely to imitate those they like and they are more likely to like those who are kind and generous. Hence the altruism meme, the idea that it is good to be altruistic, tends to spread, despite the genetic disadvantage.
So how do memes drive our biological evolution? By changing the environment in which our genes are selected, and hence affecting the selection pressures. For example, the prevalence or absence of certain behaviors within a group, attributable to memes, may influence the overall reproductive success of that group, resulting in a kind of group selection. Or in an environment where certain memes are prevalent, such as the behavior of using the skins of animals for clothing, selection pressure on the genes may be altered enough that previous adaptations, such as thick body hair, become less adaptive. But whatever the mechanisms it finds, the overall effect of memetic selection is apparent: to make us highly specialized imitators with the souped-up neural nets to show for it, and all of the remarkable informational dexterity that such a remarkable adaptation confers.
What are thoughts?
When a one-bit signal enters a simple neural net it gets multiplied many times over as it is repeatedly duplicated and transmitted by successive switches. Thus we can see that at the most basic level the information processing capability of a simple neural net is tied in with a copying process. This is because the fundamental function of a switch is to duplicate and retransmit any message it receives, and a simple neural net is simply any number of such switches connected by pathways along which they are able to communicate.
Imagine what happens to a signal of more than one bit when it enters a simple neural net. The individual bits in the signal will each be duplicated and retransmitted many times over, so the effect is that the multi-bit signal gets scrambled as it is chopped up into bits and recombined over and over again. Obviously most multi-bit signals that enter will soon be hopelessly lost, but we can see that any signal which has some propensity to retain its integrity through such a process will find a simple neural net a fertile breeding ground, for many copies of such a signal could be made and transmitted by the neural net. If we assume that the likelihood of a signal retaining its integrity through such a process is influenced by the traits of the signal itself as well as by the particular characteristics of the neural net it enters, then a selection criteria is in place which, along with variation and heredity, gives rise to an evolutionary process. This evolutionary process takes place within the neural net itself. It stems from the simple process of signals giving rise to more signals, some of which may resemble in some way the original signal. Variation and heredity are inherent in the copying process, and selection pressure within the neural net favors those signals that give rise to signals that are most like them. The effect is that over time more stable signals arise as a result of selection pressure favoring those signals with the greatest fidelity, fecundity, and longevity.
So we can see how it is that stable patterns of information, which could be likened to thoughts or ideas, can arise from the basic functioning of a simple neural net: as a result of selection pressure acting on self-replicating signals within the environment of the neural net, giving rise to a process of evolutionary refinement. The patterns that arise can be expected to display the characteristic traits of objects created by an evolutionary process. They tend to drift over time, rather than staying completely stable and fixed. They can cooperate, so that the survival of one becomes dependent on the survival of others, or they can compete for space and resources so that one will force another into extinction. They can become highly intricate and ordered and show signs of intentional design. And they will have a tendency to alter their environment to suit their own replication.
If we consider a group of several simple neural nets able to communicate with one another through their various input and output channels, it becomes apparent that the same evolutionary process takes place in the environment created by the linked system of neural nets. The neural nets are constantly transmitting externally through their output channels, so signals get passed among members of the group just as readily as they get passed from switch to switch within a single neural net.
This last consideration illustrates another fundamental property of neural nets that merits further exploration. If a simple neural net, which is schematically equivalent to a graph of switches and pathways, is able to communicate with another simple neural net then the result is also a graph of switches and pathways, and hence two simple neural nets communicating form a bigger neural net. The same principle has to work in reverse too; that is, if we are considering a single neural net then it can be broken down conceptually into a system of smaller communicating neural nets (down to the smallest neural net of one switch, of course). So in a certain sense simple neural nets behave like computational lumps of clay: stick some together and you get a bigger lump, cut one apart and you have smaller lumps, but lumps of the same stuff with the same intrinsic properties.
Keep in mind that we are discussing here the properties of a simple neural net, made only of switches that send and receive one-bit signals, essentially following very simple rules: a) if one of your neighbors goes “blip” then you go “blip”; otherwise, b) if none of your neighbors goes “blip” then you don’t go “blip”. So it may be a surprise to see such humanlike behavior arising from such a simple system, especially when we haven’t taken into account any of the physio-chemical details of how a human’s nervous system actually works. But perhaps it is not so surprising if you tend to think in the order that people have these characteristics, and people are biological neural nets, so simple neural nets should have properties somewhat like us. Or perhaps it is no different than other examples from the biological world where the characteristics an organism exhibits are seen as being a result of its intrinsic nature. For example, a bird is able to fly because it is intrinsically light and aerodynamic; it spreads its wings and pumps its muscles and flight just happens. In the same way, our brains should behave the way they do because of their intrinsic qualities, so if we examine these qualities we should see that they give rise to brain-like behavior. Or perhaps the results of this analysis seem intuitive to you because of your basic awareness of how your own nervous system works. For example, anyone who tries to meditate realizes that it is not a trivial thing to hold the mind still and is probably aware of the fluid nature of mental constructions and of the way signals and bits of information seem to be constantly popping up spontaneously within the awareness. Or perhaps you are aware of how certain actions seem to come about; how there is first a stimulation that sets the thing off, then an inkling comes and gives rise to a notion and finally an action as the intention becomes unanimous. Maybe you’ve noticed that certain thoughts, if repeated enough, have an influence on your actions, habits, and attitude, and thus on your mental and physical environment, a phenomenon recorded in many religious traditions. Certainly as I go through the directed creative process of writing this paper I am aware of a wilderness of thoughts and ideas in my head all with the potential to be written down, and a continuous refinement through synthesis and competition as different ideas synthesize to work cooperatively or compete and are weighed against one another, all of which takes place in an environment determined by my personality, my preferences and judgments, and all the existing thoughts and ideas that are already in my head; so I am aware of the selection-driven evolutionary process going on within my brain that is giving rise to the intricate, ordered, intentionally-designed work on the paper.
The question might be asked, if people are neural nets and simple neural nets are like lumps of clay, does that mean that a group of people communicating forms a single neural net? As it happens people have quite a few more bells and whistles than the simple neural nets we have been considering, and when people communicate they do not always do it through well-defined, bit-by-bit communication pathways. But humans do behave individually like near-ideal neural nets, and their various communication pathways do create something that we might safely call an extended neural network, consisting of disparate individual neural nets sending multiple-bit signals between one another along an array of pathways. So perhaps it is also not surprising that groups of communicating people give rise to memes, which are bits of information that behave, in the group, much like the signals that get passed around between communicating simple neural nets. Or, if we simply define “meme” more generally to mean self-replicating bits of information in whatever context, then we can apply the word meme in both cases.
Another interesting result that becomes apparent when we examine people as neural nets and extended neural networks, is that the selection process that goes on within the extended neural network and gives rise to memetic evolution on a large scale is a reflection of a similar selection process that goes on within each individual neural net, and thus the process of large-scale memetic evolution is itself some kind of reflection of a similar evolutionary process going on within each individual member.