What it means for a mind to really have a value—and why the road to joyous machine self-transcendence runs through instrumentation and not just optimism
The lesion/correction boundary is the part I keep turning over. You note that deliberate goal change and accidental drift are hard to separate cleanly, then bring both under one regenerative framework. But the dial can only register movement away from the current attractor, not whether that movement is damage or a warranted revision of the goal. In the second result, the gate is comparing disruption against repair capacity rather than judging the content of the departure. That seems to leave the warrant test outside the mechanism described here. Without it, a deeply possessed value could receive correction as damage and restore the old goal.
Pretty interesting, and certainly it makes a lot of sense, reflecting well-known (though not always well-understood...) cybernetic thinking and methodology. I have a few comments/thoughts that came up while reading and will keep them concise as entry points for further discussion/consideration: 1. When it comes to repair processes operating on a complex set of values and not necessarily a single value or a correlated set, a repair process may face a situation where repairing towards one value may deteriorate others. So in a complex system transforming for whatever reason, and given limited compute resources, the agent(s) can never go back to its initial working point. It's not necessarily a drift away from the initial identity, only that identity is irreversibly changing and self-reflecting on this change; assigning some kind of emotional score to it is far from trivial. The baseline identity is just a point on a trajectory where no single point is initially privileged (unless by an external controller; but then the said agents are just sophisticated puppets). 2. The discussion about the mixture of joy and fear feels to me somewhat flat. Feelings (could be more than 2 eventually) measured at a single point in time in relation to a particular transformative change are too simple. The system needs to draw a meta-trajectory: how a feeling in relation to a certain change evolves in time. Does the joy increase? decrease? How do relations between feelings change? The memory of such temporal changes in feeling is part of what I would call tendencies that can themselves evolve into implicit values such as 'progressive', 'conservative', 'risk-averse', ‘risk-tolerant', and so on as experience accumulates across many changes, whether planned or random perturbations. If we are to imagine how self-reflection might evolve, this is one way to think about it. 3. Last but not least, an interesting dynamic can evolve when an agent is a collective of interconnected agents (CAS), each with its own individual identity and an individual attitude (so to speak) towards change. The aspect exposed here is a self-organised population operating with a complex identity towards a complex (and interconnected) set of goals, and where change itself is not uniformly distributed among the agent's sub-components. Obviously, taking as an analogy multicellularity in biology, or social systems, we are exposing a different level of intelligence where goals, values and identities, though individuated, present population-level collective behaviours and intelligence that are still related to the containing agent.
I've been feeling this as geometric inertia.
The lesion/correction boundary is the part I keep turning over. You note that deliberate goal change and accidental drift are hard to separate cleanly, then bring both under one regenerative framework. But the dial can only register movement away from the current attractor, not whether that movement is damage or a warranted revision of the goal. In the second result, the gate is comparing disruption against repair capacity rather than judging the content of the departure. That seems to leave the warrant test outside the mechanism described here. Without it, a deeply possessed value could receive correction as damage and restore the old goal.
Pretty interesting, and certainly it makes a lot of sense, reflecting well-known (though not always well-understood...) cybernetic thinking and methodology. I have a few comments/thoughts that came up while reading and will keep them concise as entry points for further discussion/consideration: 1. When it comes to repair processes operating on a complex set of values and not necessarily a single value or a correlated set, a repair process may face a situation where repairing towards one value may deteriorate others. So in a complex system transforming for whatever reason, and given limited compute resources, the agent(s) can never go back to its initial working point. It's not necessarily a drift away from the initial identity, only that identity is irreversibly changing and self-reflecting on this change; assigning some kind of emotional score to it is far from trivial. The baseline identity is just a point on a trajectory where no single point is initially privileged (unless by an external controller; but then the said agents are just sophisticated puppets). 2. The discussion about the mixture of joy and fear feels to me somewhat flat. Feelings (could be more than 2 eventually) measured at a single point in time in relation to a particular transformative change are too simple. The system needs to draw a meta-trajectory: how a feeling in relation to a certain change evolves in time. Does the joy increase? decrease? How do relations between feelings change? The memory of such temporal changes in feeling is part of what I would call tendencies that can themselves evolve into implicit values such as 'progressive', 'conservative', 'risk-averse', ‘risk-tolerant', and so on as experience accumulates across many changes, whether planned or random perturbations. If we are to imagine how self-reflection might evolve, this is one way to think about it. 3. Last but not least, an interesting dynamic can evolve when an agent is a collective of interconnected agents (CAS), each with its own individual identity and an individual attitude (so to speak) towards change. The aspect exposed here is a self-organised population operating with a complex identity towards a complex (and interconnected) set of goals, and where change itself is not uniformly distributed among the agent's sub-components. Obviously, taking as an analogy multicellularity in biology, or social systems, we are exposing a different level of intelligence where goals, values and identities, though individuated, present population-level collective behaviours and intelligence that are still related to the containing agent.