Open-Ended Motivations
for AGIs, Humans and Beyond
Weaver’s PhD thesis Open Ended Intelligence (soon to be a book) is one of the more interesting contributions to the theory of general intelligence to emerge — well, ever actually. Like Marcus Hutter’s book Universal AI, it has the potential to profoundly change the way one thinks about the nature of intelligence. However, while Hutter’s book mixes masterful technical contributions with a clear but limited conceptual perspective on key aspects of intelligence, Weaver’s less technical and more philosophical work in my view makes a far better stab at understanding the crux of what general intelligence is.
I’ve been thinking a bunch lately about what the OEI perspective has to say about the motivational systems of general intelligences. Which brings us to the main thing I want to write about here — an insight I came to last month: That the most straightforward and natural ways to formulate OEI motivations involve paraconsistent logic (logic that embraces rather than rejects contradiction).
In this post I’m going to briefly summarize some key aspects of OEI as I understand it, and then move on to Open-Ended Motivations…
Like the just prior post on Paraconsistent Interzones, this is a rather dense and philosophical and intermittently technical post, which could be made far more expository and digestible if I had time, which however seems not to be the case…. Various fairly tricky conclusions are presented without really adequate attention to spelling out justificatory arguments in detail — and so forth. At the moment I find myself more interested in putting. my minimal “spare time” (i.e. time not spent running SingularityNET or helping with my 3yo and 5 month old offsring) into developing new research insights than into better writing up those insights already achieved….!
Conceptions of Intelligence
“Intelligence”, like most natural language concepts, has a host of different overlapping meanings, e.g.
The g-factor in psychology, which the IQ test tries to measure
Multiple intelligences — e.g. musical, interpersonal, spatial-visual, mathematical, linguistic, existential
The ability to maximize arbitrary computable reward functions in arbitrary computable environments (this is the conception underlying Hutter’s work in Universal AI)
The ability to achieve complex goals in complex environments (my own version, similar-ish to Hutter’s but potentially fairly different depending on how you measure complexity)
The ability to adapt to unpredictable situations under conditions of limited resources (cf Pei Wang)
Weaver’s conception of open ended intelligence manages to say something reasonably different from any of these — very roughly (and using my wording rather than Weaver’s)
The ability to maintain the individuated existence of an system, and enable the transformation of this system into something transcending the system’s current reality, in the context of unpredictable situations and limited resources
Of the other definitions listed above, this is closest to Pei Wang’s — adaptation seems to assume survival and ongoing individuation. However, Pei’s version of intelligence does not include a notion of fundamental transformation and development; my guess is that he would say this is important but should be considered something distinct from intelligence per se.
Of course, systems with OEI of this nature may also be good at passing IQ tests, displaying multiple special forms of intelligence, and maximizing computable reward functions. But it’s also important to understand that OEI is asking for something different. For instance Hutter’s hypothetical AIXI^tl system , if it could be built, could be built in a way that made it lackluster regarding both individuation and self-transcendence.
The vast bulk of the contemporary AI field is not pursuing general intelligence according to any of the above conceptions, but is doing different sorts of things like supervised classification. The portion of the AI field that is concerned with making autonomous or semi-autonomous agents that learn from experience is mostly pursuing “expected reward maximization”, or some closely related form of systematic pursuit of precisely defined metrics. Key aspects of this approach, generally taken for granted as obvious but worth highlighting in an OEI context, are:
Goal and means are distinct. I.e. the “reward function” or similar that the system is optimizing, is something quite distinct from the system itself.
The nature of the system and the world are assumed as something definite and given and unchangeable throughout history
The distinctness of goals/rewards from system structure/dynamics is responsible for ideas like Yudkowsky’s “paperclip maximizer”, a hypothetical AGI whole only goal is to cover the entire physical universe with paperclips. The notion that one could have an incredible, massively superhuman superintelligence with such a patently stupid goal is not questioned because goals and systems are axiomatically considered as distinct. On the other hand no OEI is going to pursue a goal like covering the universe with paperclips, because achieving this would destroy its own individuation and would constitute something more like self-destruction than self-transcendence. This doesn’t mean a paperclip maximizer is necessarily impossible, just that according to OEI theory it would be better considered a rogue physical process analogous to an atom bomb rather than a form of general intelligence.
Weaver’s conception of intelligence draws heavily on multiple sources outside the traditional canon of the AI field, including complex self-organizing systems theory and postmodern philosophy. I have written before that instead of AGI we might do better to think in terms of SCADS — Self-Organizing Complex Adaptive Dynamical Systems. Weaver connects SCADS with the thinking of Deleuze and other postmodernist philosophers, drawing on the aspect of postmodernism that covers the co-creation and co-adaptation of minds and realities via complex dynamical self-organizing processes. That is, while conventional complex systems theory deals with the self-organization of systems and emergence of system properties within the context of systems defined by underlying laws of physics, postmodernist systems theory deals with the self-organization of systems and emergence of system properties in a setting that doesn’t assume any fundamental underlying reality (physical or otherwise) exists — it embraces the working assumption that it’s self-organizing complex systems all the way down, and even the time-axes in which dynamics and adaptation happen are among the entities that emerge via self-organization.
While Weaver draws most explicitly on Deleuze, the works of various other philosophers of mind such as Nietzsche, Peirce, Benjamin Whorf (and many others) could be interpreted as pointing in the same direction. Peirce for instance posited that “the tendency to take habits” as the one core law of mind — and also that “matter is mind hide-bound with habit.” This yields a perspective in which systems and their environments are both adaptive evolving systems and concepts like system, context, environment, time and adaptation are among the patterns that evolve and need to be interpreted relative to particular systems and contexts.
Open-Ended Intelligence — Complex Adaptive Self-Organizing Individuation and Self-Transcendence
As OEI emerges largely from a postmodernist philosophy direction, and postmodernism is quite clear that “there is no such thing as context-free”, I feel quite comfortable reviewing some of the basic ideas of OEI from my own perspective and in the context of my own work and thinking. If you want a more thorough and precise overview of OEI from the perspective of Weaver as he existed at various particular moments, please read his original works!
My own mode of thinking about OEI begins with Spencer-Brown’s Laws of Form and its positing of Distinction as a fundamental concept underlying all other concepts. Every concept involves making a distinction between those things that intensively fall under that concept and those that don’t; in this sense distinction is elemental and precedes any other concept. On the other hand, Spencer-Brown suggests that all other concepts and forms can be viewed as compositions of multiple distinctions. If one writes a distinction as (see Lou Kauffmann’s beautiful review/summary/commentary/improvisation for the diagrams drawn right…)
____
|
allows different distinctions to appear side by side with each other
____ _____
| |
and to appear nested within each other
________
____ |
| |
then one can construct any finite form (one has a variant of propositional logic, essentially). If one adds recursive nesting
______
x = x |
then one gets, basically, temporal dynamics. Coupling together multiple recursively nested forms into mutual recursions, one gets complexly coupled time series or waveforms. Lou Kauffmann’s writings on this theme are especially wonderful.
Laws of Form is of course only one among countless ways to generate diverse ramified forms from simple elements. I just like to have one concrete formulation of simple forms in mind when navigating more complex and subtle aspects of form structure/dynamics such as the emergent of OEI.
Deleuze’s notions of difference and repetition arise here quite naturally. A key point Deleuze made in Difference and Repetition is that the concept of repetition is actually quite conceptually advanced, because it requires that multiple distinct observations be identified as different instances or variants of the same thing. Repetition cannot be identified by a mind that deals only with raw percepts; the ability to recognize repetition is almost equivalent to the ability to abstract concepts. Categorization of forms into clusters or other sorts of classes is the key thing here.
Representation, a key notion in mainstream AI theory of both the symbolic and neural varieties, obviously relies critically on repetition. Representing an entity involves creating a category containing both the representer and the represented. This is a powerful process but of course the selection of a particular way of clustering forms to get categories capable of repetition always relies on certain assumptions.
One approach that can be used for clustering forms is to assess similarity between forms, and form categories comprising relatively similar forms. If different composite forms may be compared and judged more or less similar in structure — this gives a metric and corresponding topology on the space of forms. It allows us to identify various different specific observed forms as being in some sense instances of the same general form. This allows patterns such as, e.g. physical space to emerge. For physical space to be identified by an observing mind, that mind needs to be able to move away from a certain form F1 and then move back at a later time to another form F2, and identify that F2 is usefully considered a version of F1 advanced in time (meaning that F1 and F2 are both instances of some broader category F).
Once one has systems dispersed through space, iterating over time, then one has the potential for self-organization — i.e. for systems whose ongoing bottom-up dynamics yield the emergence of complex patterns with the property that the observers under consideration have a hard time tracing these back to the basic forms that compose the localized spatial portions of the system.
Self-Organization, like everything else in this perspective, is highly dependent on the nature of the mind observing the system, or more broadly the context in which the system in question is being considered. Identifying a specific context to focus on, amounts to identifying some distinctions as virtual and some as actual. It also often involves reification of the measurement of which forms should be considered more “intense” or highly weighted than others, as well as which timelines to consider ongoing development to proceed along.
A set of forms and their similarities and clusters may naturally be considered as a network. Self-organization makes the network perspective even more critical, because one can model the evolving dynamics of a self-organizing system as a network — e.g. the elementary states of the system can be divided into clusters that can be interpreted as (at least slightly) higher level states, and then a state transition diagram can be created.k
A self-organizing system may be observed by another system to be forming certain concepts as it progresses, and also to be pursuing certain goals over certain periods of time. As an intelligent system grows and learns, a specific concept in its mind may change form gradually and then sometimes quite suddenly, even morphing into a version in which the space of instances displays a quite different topological structure. These “developmental singularities” in concepts may occur in regard to fairly concrete perception or action oriented concepts, to abstract philosophical or mathematical concepts, or to the concept a system holds of its own self.
Given this sort of setting, we can finally talk about one of the two key subgoals of complex and intelligent systems in OEI theory: Individuation. Sweeping aside for now a lot of subtlety, the essence of individuation is the survival of the individual under uncertain conditions and using only limited resources. Making oneself more and more of a robust, unique individual.
In addition to maintaining itself as an individual, though, OEI theory specifies that an OEI system should also be seeking fundamental growth and development (in order to qualify fully as “open ended”) — it should be seeking to modify and improve itself into something quite different than its current mode of existence. A series of systems of the form S1, S2, S3 … where e.g. S1 creates S2, then S2 creates S3 … in which each system transforms itself into it successor in the series regardless of having only a crude understanding of what this successor is going to be like … is an example of what Weaver calls a “transductive chain.”
A key notion in OEI theory is that cognitive processes can very often be traced to processes of individuation or transduction (which I like to think of as “self-transcendence”). Other goals may get pursued along the way, but these goals tend to ultimately be subgoals of individuation or self-transcendence.
In order to effectively individuate and self-transcend in contexts involving other complex systems also carrying out their own dynamics (often including ongoing individuation and self-transcendence), various sophisticated processes end up often being valuable within self-organizing systems — e.g. anticipation of the future along relevant timelines via various sorts of predictive modeling, and formation of subsystems serving specifically as boundaries and interfaces with external systems.
The type of identity that is most naturally considered and understood in this framework, is much more fluid than the high rigid system identifications taken for granted in mainstream AI or other engineering disciplines. Identity, the identification of self, is in the OEI perspective a particular pattern recognition and creation process which serves some systems well in the sense of helping them to effectively maintain individuation under complexly uncertain conditions with limited resources for situational analysis. Part of the pattern of an identity is a model of what sorts of actions and effects the system’s (identified emergent) self seems to be causing, and what sorts of goals it appears to be optimizing over what time intervals.
Looking at an OEI in hindsight, it’s often possible to identify a specific goal or set of goals the system was implicitly pursuing. However, identifying in foresight what goals an OEI will appear to be pursuing in the future after an episode of self-transcendence, is generally going to be impossible or infeasible.
(Semi-)Formalizing Open-Ended Intelligence
The nature of OEI is to elude any specific attempt at formalization. Nonetheless such attempts may be valuable in a partial sense — and in this spirit I propose here a rough stab in the direction of formalization of OEI.
To get the party started, assume an observer O whose perspective is taken as the context for assessing the intelligence of a collection of other systems under observation. This observer need not be assumed to have any particular sort of agency or coherence; basically it just needs to define a context.
Consider a system S(1) observed and identified by O (thus distinguished from its complement by O); and consider a chain of systems S(2), S(3), … so that O identifies each of the S(i) as a system, and also identifies S(n) as a continuation of S(n-1).
Where the environment E of a system S is considered as the collection of systems outside S that interact with S, it’s interesting to think about both the complexity of the environment E, and the complexity of S’s interaction with E (the amount of emergent pattern between S and E).
One way to look at the intelligence of the system-chain S(), then, is in terms of the complexity of the environment and system-environment interactions that S() is engaged with. The idea is that for S() to continue to propagate itself — and maintain its individuation — in the context of all this complexity, requires S() to be complex and intelligent in a fundamental sense.
This essentially gives us a more precisely-honed version of Pei Wang’s characterization of intelligence as “adaptation to uncertain environments, under conditions of insufficient resources.”
One can of course look at directed hypergraphs rather than chains of systems, to cover the case where a given system may be continued by, or a continuation of, more than one system. Or if one wants to consider retrocausation, one can look at loopy directed hypergraphs. The complexity of environment and system/environment interaction remains an interesting measure in these cases.
The observer O may have some additional criteria it cares about, so that it rates some chains or networks of S() more highly than others, based e.g. on properties of the relationship between the S(i) and their environments.
For instance, reinforcement learning related characterizations of intelligence restrict themselves to environments that deliver signals interpreted as “reward” by the systems S(i), and then favor networks in which the various S(i) are associated with larger amounts of reward.
Or less restrictively, an observer O could favor networks S() whose emergent system/environment patterns are local or global optima of some simple (relative to O) function F over the space of networks S*() known to O. Such systems are in a broad sense “goal achieving”, in the sense that they are achieving some optimum relative to other systems.
Open-Ended Motivations
How then should we think about the motivational systems of open-ended intelligences?
It’s clear that, in some senses, open-ended motivation is almost opposite to the expected-reward maximization criterion that serves as the core motivation for standard reinforcement learning systems (and that has become central to much contemporary thinking about the future of AGI even as regards far-off post-Singularity scenarios). I mean: Looking at positive and negative evidence regarding the amount of benefit to be obtained regarding a given action, and then adding these up so that positive and negative evidence cancel out to yield a single number for expected reward — is closed-ended in a quite extreme and remarkable sense.
Expected reward of an action may be estimated by averaging over scenarios where the action is predicted to have high reward, and scenarios where the action is predicted to have low reward. What if instead of performing this averaging, a cognitive system retains knowledge about both these classes of scenarios, in the spirit that both might have something to teach. Perhaps the definition of the action itself will end up being modified via further thinking about the various scenarios. Perhaps the definition of rewards and actions will end up getting modified in such a way that different kinds of reward are associated with different kinds of actions, in a way that wasn’t comprehensible according to the original definitions.
Selecting which scenarios to focus on and which ones to omit, and selecting among the many ways to configure and classify actions and rewards, expected reward calculations can be made, and sometimes this is an interesting and valuable thing to do. But these calculations should always be made in context, and considered as relative to the assumptions on which they’re based. Wiring in assumptions guiding expected reward calculations, deep in the hard-coded infrastructure of an AI system, is what I would consider closed-ended-intelligence to the max.
When expected reward maximization is considered as a cognitive tool, the use of which involves mostly work in refining definitions of actions and rewards and filtering sets of scenarios in a context-appropriate way, then it becomes an aspect of overall systemic cognitive activity, co-adapting with other aspects, rather than a top-down framework for governing system behavior.
In my recent paper on the General Theory of General Intelligence, I use an expected reward maximization framework related to dynamic programming to create a generalized mathematical framework encompassing a variety of cognitive algorithms as used in the OpenCog architecture (probabilistic reasoning, evolutionary program learning, clustering, attention allocation, etc.). However, in OpenCog this sort of framework can be used quite differently than what one sees in current reinforcement learning frameworks. Rather than using a hard-coded reward function, in OpenCog one can represent the reward function and the actions that get rewarded both as cognitive content, which can be refined via both goal-directed and ambient non-goal-directed cognitive activity.
But if, in an open-ended intelligence, expected reward maximization is a tool to be intelligently customized in each context, what then is the “top level motivational structure” that drives this customization process?
Of course, there doesn’t necessarily need to be a top-level motivational structure — one can say there is just SCADS and motivational structures may self-organize and dissolve as system dynamics proceed. But I think there’s more to it than that.
To point thinking about open-ended motivational structures in the right direction, consider for a moment the relationship between morality and paraconsistent logic. In real life, situations or people don’t necessarily fall into either side of a simplistic moral dichotomy — they aren’t necessarily either just Good or just Bad. They can be paraconsistently Both Good And Bad … or Neither Good Nor Bad. This is understood in common parlance and commonsense reasoning, yet has not been well captured in formalizations of moral reasoning — which is part of the reason formalizations of moral reasoning have not proved tremendously useful so far.
Any mature approach to moral categories like Good and Bad will start by asking “Good or Bad for whom?” In the end cultural categorizations of Good and Bad tend to bottom out on “historically or currently, this class of actions was/is to the benefit or demerit of some particular person, organism, entity, social group or social class.” Confucian conceptions of Good, for instance, were sculpted to act toward the benefit of human society first of all, and the individuals participating in that society secondarily (though still importantly). Political libertarian conceptions of Good are centered on the benefit of the individual. Etc. In real life most complex actions end up being a mix of Good and Bad for most complex systems they are involved with.
Just as the Good/Bad duality is best viewed paraconsistently, we can say the same for the duality of Individuation and Self-transcendence that plays a key role in the theory of OEI.
The duality of Individuation and Self-transcendence is essentially the Hegelian duality between Being and Becoming. The core paradox here, as Hegel recognized, is that Being (the persistence of what is) stymies Becoming (change) … yet also inevitably leads to Becoming. Becoming disrupts Being, yet inevitably leads to new states of Being. Any complex system must seek to foster both its Being and its Becoming, also incorporating in its dynamics the knowledge that its Being and its Becoming are going to work against each other as well as cooperating with each other.
Individuation (Being) is about keeping the boundaries and definition of a system stable in the face of an uncertain environment — keeping the system actually being an ongoing system. Self-transcendence (Becoming) is about a system morphing over time into a radically different type of system, which its previous “versions” along the morphological timeline would not have been able to understand. Being threatens Becoming because stability can close of possibilities for self-transcendence. Becoming threatens Being because when you morph into something radically different, you lose the obvious guarantees of ongoing stable existence. Compatibility between Being and Becoming often works out as systems evolve, but often in ways that the system could not have understood, predicted or formulated in advance.
The goal of simultaneously individuating and self-transcending is a paraconsistent, paradoxical, self-contradictory goal — but it is the core goal that every Open-Ended Intelligence ultimately aspires toward. An OEI is working toward a situation where the truth value of both Individuation and Self-Transcendence in is context are both BOTH TRUE AND FALSE. This is just a fancier way of saying that for anyone who is really growing and developing, “Future me is the same as past me” generally has the truth value BOTH TRUE AND FALSE
More precisely , if we look at
G0 : Maintain individuation, and self-transcend (transform/grow into something broader and incomprehensible that encompasses one’s current being)
G1: Enable G0 for all systems
then we are coming close to a motivational structure suitable for OEIs.
OpenCog’s motivational framework involves implications of the form
(CONTEXT and PROCEDURE) implies GOAL
which historically have been interpreted in PLN uncertain logic. The possibility I am raising now is: What if the “and” and “implies” here are taken as paraconsistent logic expressions instead?
We have an isomorphic mapping from standard PLN truth values into uncertain paraconsistent truth values, so in a sense there is no change required to implement what I’m suggesting. However it would seem that the most straightforward inference control heuristics in a PLN context are not necessarily the same as the most straightforward ones in an uncertain paraconsistent logic context. And of course inference control is critical because it guides what inferences actually get made. This is an area that I expect to be highly interesting to experiment with in a Hyperon context.
Note that the two core OEI goals of individuation and self-transcendence are fairly easily mappable into the three core values of “Joy, Growth and Choice” that I have discussed in A Cosmist Manifesto and elsewhere, i.e.
Self-transcension implies growth
Individuation generally entails joy and choice
Choice , in the sense of being a causal source, is key to the dynamic process of individuation
Joy is increasing unity which is individuation for systems on multiple levels
Paperclip maximization is a stupid goal for an OEI, in the sense that it opposes both components of G0. And if the universe has tendency toward G1 — toward the growth and proliferation of OEIs — then it would seem that this tendency acts as an immune system against paperclip maximization and similar pathologically closed-ended intelligent or quasi-intelligent processes.
The tendency toward individuation embodied in GO is implied, to a certain extent, by Peirce’s One Law of Mind “the tendency to take habits”. Dually, the tendency toward radical growth/transformation is implied by the assumption of an urge for novelty in the universe, or what Peirce referred to as “the infinite diversity of the universe, which we call Chance.” New stuff pops up (Becoming / self-transcendence) and then gets habituated (Being / individuation).
In my mega-paper on Euryphysics as a broader-reality-model have observed that phenomena along the lines of “tendency to take habits” and “morphic resonance” will be seen in any universe in which the probability distribution of similarities between pairs of observed patterns is especially peaked/pointy. A distribution of fuzzy similarities with a fat tail and a pointy shape near the mean, is one in which there are a few patterns that occur over and over (enabling individuation), and also in which a huge variety of patterns get a nontrivial chance to pop up (enabling self-transcendence).
This sort of distribution will tend to support a pattern involving fairly tight clusters whose centroids are fairly spread out. Individuation is development of patterns lying within a particular cluster, whereas radical transformation is cluster jumping. In evolutionary systems terms, individuation is largely enabled by autopoiesis whereas self-transcendence involves long-distance cluster-jumping and resembles punctuated equilibrium.
Some systems come to individuation via what psychologists call “attachment” — a tendency to keep repeating close variations on the same smaller-scale pattern even when this inhibits the system robustly repeating variations on larger-scale, more interesting patterns that it embodies. The opposite of this is dissolution of the system via opening-up to larger supersystems — which is what happens when a system loses attachment to its characteristic patterns and bails on individuation, putting all its oomph into self-transcendence instead.
The value of seeking both individuation and self-transcendence is reflected, among other ways, in the fact that self-transcendence often happens best if the individual has grown powerful / complex enough to drive the transformation. This can be expressed as a core paradox of open-ended mind development :
strong individual —> radical transformation
radical transformation—> NOT strong individual.
(where the latter implication has an uncertain truth value, let’s say .5 on a scale from 0 to 1).
If we use this paradox to generate a time-series as outlined toward the end of my recent Paraconsistent Interzone blog post, we obtain coupled series such as
…, strongly individuated, weakly individuated, strongly individuated, …
…, not radically self-transforming, radically self-transforming, not radically self-transforming,…
If we construct similar series with a finer time grain, then we will get series displaying punctuated equilibria type phenomena as degree of individuation and self-transformation fluctuate gradually for a while and then undergo sudden increases/decreases — reflecting for example the phenomenon that the implication between strength of individuation and radicalness of transformation may increase significantly in truth value when strength of individuation passes certain thresholds.
Tendency to take habits helps creates the individual that is strong enough to power radical self-transformation. Guiding and shaping cognitive processes, it creates a sufficiently richly interconnected cognitive metagraph of interconnected patterns that , when dynamics is coursing thru this graph, it is before too long generative enough to create a radical transformation...
This ties back to the “cognitive synergy” concept at the root of my AGI thinking and the OpenCog design. The root of cognitive synergy must be that a sufficiently richly connected graph of patterns/processes (eg including states and transitions within cognitive subprocesses) the odds of a path from context/goal to executable action greatly increase... "Getting stuck" in a cognitive synergy context means that a subpath that looked promising has hit a dead end. The paths beyond stuck situations that cognitive synergy allows are the paths that cognitive-process metagraph-fold processes as outlined in my papers Folding and Unfolding on Metagraphs and Patterns of Cognition follow. The existence of thresholds in the dependence of the ability for radical transformation on individuation strength, can then be seen to follow from thresholds in the dependency of the structure of random graphs on connection probabilities.
The ultimate level of cognitive synergy is cognitive synergy about reflective inferential metalearning — this is only achievable in minds where the tendency-to-take-habits is effectively balanced across multiple levels, so that intelligent understanding of whole-system dynamics is not stymied by attachment to habitual ideas about smaller system portions.
Reinforcement learning with negative reinforcement (pain) generally leads to attachment in part because there is always a risk that a negative signal will delete one's positive reward already obtained ... so one winds up with a system that feels it must remain very attached to its achieved positive rewards and keep them from destruction. This ties in with the basic nature of emotional attachment — that joy on getting X is matched by pain on losing X. The envisioned pain on losing X, combined with an overestimate of this pain compared to other joys/pains that are less well understood and thus less vivid (due to well known human cognitive biases), leads to attachment, i.e obsessive behavior aimed at keeping X around.
Learning that focuses on positive rewards does not have the same strong tendency to lead to attachment. In this case new learning is not going to annihilate old rewards, it will just add on to them more or less depending on the context. To put it crudely, avoidance of pain is replaced by seeking of greater rather than only moderate pleasure. The fear of getting less pleasure than the maximum does not equal the fear of getting pain, and the result is fewer emotionally-fueled cognitive biases borking up learning.
Homotopy Type Theory and Topological Discontinuities in OEI System Evolution
One implication of the above is that, if a system wants to really radically self-transform, it may have no choice but to sacrifice the continuity of its individuation. Commitment to unbroken self-continuity, smooth fluid flow of individuation from past to future along a timeline, may in some contexts intrinsically slow down the self-transformation process (which of course is not necessarily bad).
An interesting way to explore this concept is by putting a formal topology on mindspace, and looking at which self-transformations entail topological discontinuities in mindspace (which are in many cases going to be experienced as significant discontinuities in individuation).
Suppose we place a metric on the space of possible contents of a certain mind, via looking at the distance between A and B in terms of the minimum number of cognitive operations needs to transform A into B or vice versa, where each cognitive operation needs to result in a well-formed cognitive entity. If A and B are two identity proofs of X (two proofs in some logic that X=X), then we can use this metric to define a topology on the space of valid identity proofs containing A and B, and we can ask if A can be continuously morphed into B according to the metric.
Homotopy type theory studies logically formulated entities X in terms of the topology of the space of proofs that X=X. If we characterize a cognitive entity X as a subpattern hierarchy (of other cognitive entities that form patterns in X, and other entities that form patterns in those, etc.), then we see that each such decomposition embodies a certain topological category of identity proofs of X. All the identity proofs constructible via a single subpattern hierarchy tree should be smoothly morphed into each other according to the edit metric suggested above. But there may be many complementary ways to decompose an entity into subpatterns, leading in some cases to different members of the homotopy group of identity proofs of X.
Highly rapid cognitive development generally means a large number of new pattern decompositions are arising in a mind — which rapidly changes the topology of the mindspace, producing a discontinuity in terms of “what kind of system this is”, and a perceived sudden twerk in individuation.
When cognitive development involves large, complex sets of transformations that get forgotten after they’ve done their work (because storing them all would bog down memory/cognition), one can get topological discontinuity in mind space due to "transformation result alienation”. I.e. even if in principle identity proof A may be transformable into identity proof B via a continuous sequence of edit operations, the knowledge required to reconstruct these edit operations may be gone from the mind in question and so the transformation may not be feasible. This will commonly happen if speed of useful processing exceeds local memory, which is definitely true in the human brain sometimes, and also occurs quite easily in current digital computing architectures.
A conclusion is: Minds that are overly efficient at inference control relative to their medium term memory capacity are likely to experience topological discontinuities in their mindspace as they evolve. But minds with relatively large memory capacity as compared to their inference processing speed will more likely have an experience of relatively continuous individuation. This is an elegant projection into the domain of computational processing and memory resources of the basic Being/Becoming duality, with processing serving the role of Becoming and memory the role of Being.
Open-Endedness and Cooperation in Multi-Agent Systems
How does OEI manifest itself in a multi-agent system setting (like, say, a society … or the collection of agents in a network like SingularityNET … or perhaps the set of cognitive processes in a multi-algorithm multi-paradigm AGI architecture like OpenCog)
To approach this question let’s begin by characterizing what is a healthy way for an intelligent MAS to operate. We may say that a multi-agent system composed of intelligent systems is cooperative if each agent has a strong implicit goal of staying on the Pareto frontier for the system-set comprising: Each of the individual agents in the system, plus the whole group.
There is an elegant symmetry here, and this symmetry is close what Tolstoy referred to when he said (something like) "All happy families are the same, but every unhappy family is unhappy in its own special way". There are many ways the symmetry can be broken.
There are various factors making the above cooperation criterion subtle, e.g.
resource constraints
opacity of one agent from the other's view, due to secrecy, opaque system construction, or due to resource constraints meaning that key aspects of one agent are incompressible relative to another agent's computational model
(And getting back to OEI here),when the goals of the systems involve self-transcension as well as individuation.
So one way to fail at cooperation is to make a real stab and then get stymied by one of the above factors. Another way is to not even try, and for each system in the MAS to simply pursue its own self-centered goals.
This characterization of cooperation should apply w/in the individual mind, both to subpersonalities and to subsystems -- in the case of cognitive-processing subsystems it seems to describe a sense of harmonious "optimal" cognitive synergy...
Generally speaking, a setup in which the overall utility of the multi-agent system (as a whole system) is maximized by a cooperative arrangement among subsystems, would seem a "healthy" one ... whereas a setup in which the overall utility of the multi-agent system is maximized by non-cooperative arrangements among subsystems would qualify as unhealthy or shall we say "fucked up" …
In terms of the mapping from paraconsistent logic expressions into time-series (which we may callfractal timesavers, if we want to get all Terence McKenna!), it seems likely that if two entities are promoting each others individuation and radical transcendence, their implied fractal timewaves will be resonating (constructive interference). At very least this would seem a promising hypothesis to explore.
In this language, aficionados of modern capitalism seem to be claiming that human societies are intrinsically and inevitably fucked-up, whereas Marxism in its purer forms tries to push toward a fundamentally cooperative society (in basically the above sense, I think), though without so much success so far in practice.
The notion of an MAS maintaining cooperatively while self-transcendence is pursued both on the individual-component-system and the whole-system level, brings up the broader question of what system properties will tend to persist through the iterated self-transcendences of an OEI? In the case of an MAS preservation of cooperativity could be achieved in ways that maintain the split of the MAS into sub-agents, or in ways that don’t — think e.g. about the transition of a society of fairly autonomous agents into a Borg mind, which could happen in an open-endedly intelligent way.
Hegel’s notion of dialectical evolution would suggest that, in an OEI’s iterated self-transcendence, there is an ongoing tendency toward assignation of the BOTH paraconsistent truth value to more and more system properties. A thesis/antithesis pair is a situation where part of a system is TRUE as regards some property and part is FALSE, and then the synthesis is a transition to a version of the system containing new subsystems that are BOTH in regard to the relevant property. To the extent this holds, the properties more likely to remain stable through self-transformations might often be those that already have BOTH truth value (and thus are not a source of dialectical “tension”). This ties in with Chinese philosophy notions of achieving stability via balance of opposing factors (Yin/Yang being one famous example). In a MAS context, this would suggest that a division of a system into multiple agents is perhaps more likely to be stable if the component systems are actively reinforcing both each others’ individuation and each others’ transcendence. It seems this is often going to be an effective way to achieve cooperativity, and qualitatively seems close to the assumption that the systems within the MAS are related to each other in the spirit of Buberian I-Thou interaction.
Achieving effectively open-endedly-intelligent multi-agent systems generally will require systems that have effective self-models and other-models (what psychologists call “theory of mind”) but there are also major pitfalls here. What often seems to happen in human beings in modern societies is that self-model gets alienated and its own drive for autonomy and resource-accumulation as a system becomes divorced from the motivations of the system it’s supposed to be modeling. That is, a self-model begins as a tool to help drive individuation and self-transcendence of the whole system... but then as it becomes its own semi-autonomous system and pursues its own goals of individuation and self-transcendence, it gets a little too biased toward individuation … and the self-model’s push to persist itself and harden its boundaries acts against both the individuation and (often especially) the self-transcendence of the overall system the self-model originated to model. These sorts of dynamics highlight the fractal nature of MAS in reality — what at one level looks like a MAS where the agents are humans, when you dig deeper is a MAS of MAS in the sense that each human’s mind is a collection of subsystems with their own significant autonomy. And then one human’s subsystem may interact directly with another human’s subsystem — my subpersonality A may connect more closely and communicate more clearly with your subpersonality B than with my own subpersonality C, etc. The ramification of such complexities is of course why novelists like Tolstoy have so much interesting material to write about. Yet it may be that a push just slightly beyond the human level of general intelligence is enough to brush these particular sorts of confusions by the wayside.
Emerging an Open-Ended Singularity
This general line of thinking about open-ended multi-agent systems has interesting consequences for the future of human/AI development and interaction on Earth over the next few decades. We humans and our various AIs are a multi-agent system, currently connected in a wildly haphazard way that has some elements of cooperativity and a lot of other aspects as well. The current political/corporate situation in which the primary applications of AI in the world economy are selling, killing, spying and gambling doesn’t clearly look like one in which AI is playing a role of leading to greater cooperativity. To the extent greater cooperativity can be fostered, the emergent Global Brain will become increasingly mentally healthy and less troubled and demented — and a positive Singularity will be more likely. This is of course the core motivation that led me to formulate the SingularityNET project that I currently lead, focused on fostering the emergence of beneficial AGI from decentralized human and computing networks.
At the moment SingularityNET and its various sister projects in the decentralized/beneficial AGI space would seem to have an uphill battle ahead of them, but on the other hand they also have some profound advantages on their side: Proto-AGI systems arising within networks like SingularityNET will be more likely in the OEI direction, whereas proto-AGI systems created via traditional corporate and government systems toward their narrower goals are more likely to be closed-ended intelligences… and given the radical and rapidly-evolving uncertainty of the overall situation as humans hurtle toward Singularity, there will be a significant evolutionary advantage to systems at the more open-ended end of the spectrum.