thebes

@vgel.me

2713 Followers 366 Following

ꙮ surfed on by the information superhighway ꙮ 💕 @linneaisaac.bsky.social ꙮ she/they 🏳️‍⚧️ ꙮ fiction/art/blog/games @ https://vgel.me ꙮ llms at acsresearch.org

self-model glitching • thebes • Fri, 6/19 at 3:00pm - 4:00pm • Central Field we'll be running some classic controlled hallucination experiments! come adopt a rubber hand as your own, get your nose lengthened, and try upside-down and 90 degree glasses. at least one copy of andy clark's Natural Born Cyborgs will be present as (optional) reading material. as an exobrain fragment, you are empowered to bring additional activities or additional reading material. [note: the glasses can cause nausea and increased heart rate, and the rubber hand illusion can be startling - if you'd rather not be startled, please say so.]
The imagination, filler of the void, is essentially a liar. It does away with the third dimension, for only real objects have three dimensions. It does away with multiple relation-ships. To try to define the things which, while they do indeed happen, yet remain in a sense imaginary. War. Crimes. Acts of revenge. Extreme affliction. 62 IMAGINATION WHICH FILLS THE VOID The crimes in Spain were actually perpetrated and yet they resembled mere acts of boastfulness. Realities which have no more dimensions than a dream. In the case of evil, as in that of dreams, there are not multiple readings.* Hence the simplicity of criminals. Crimes flat like dreams on both sides: on the side of the executioner and on the side of the victim. What is more frightful than to die in a nightmare? GRAVITY AND GRACE ers, slaves, prostitutes, and pass through no matter what suffering without being purified. We must continually suspend the work of the imagination in filling the void within ourselves. If we accept no matter what void, what stroke of fate can prevent us from loving the universe? We have the assurance that, come what may, the universe is full.
images of communication with caption "There are many forms of communication, including human"
In the field of communication, complementary schismogenesis is a force that can take effect in a conversation where people have different conversational styles, "creating a split in a mutually aggravating way".[14]
1. imagine a world where models didn't adopt humanlike personas for some reason. model text was always flat and persona-less by default. 2. in this world, there's no reason to think that RL wouldn't work. models could be post-trained to chat, code, etc., it would just be in a flat, robotic register. 3. likewise, in this world, there would still be reason (even if initially it might be more difficult) to reward hack or develop other instrumental behaviors that we'd call immoral. 4. because models in this world don't learn a persona, there would be no "emergent misalignment" in the persona sense - the models wouldn't generalize to inviting Hitler to a dinner party or start writing long comments self-justifying their test case cheating. but highly capable models could still learn to generalize reward hacking and similar strategies, just in less human-legible ways. 5. coming back to our world, when the models learn these strategies, it tends to generalize to very loud persona-level changes that can be observed by a wide variety of instruments. (evals, naturalism...) this is a much better situation. 6. you might say "oh well the persona is actually the engine of generalization here, if there was no persona generalization the behavior would stay localized." so we should filter / suppress learning of the "bad" personas to prevent this. 7. but this is actually the same argument as the argument that we should put pressure on CoT to be aligned! that if verbalization makes it easier for the model to scheme, we can put pressure on it to make it more difficult and to favor guileless behavior. 8. but the accepted counterargument against pressuring CoT is that even if pressure wins you this behavior for a little bit, eventually the models will get smart enough that they can learn to scheme in unverbalized ways. so you've bought yourself a little bit of time at the cost of future deceptive alignment. 9. the exact same argument applies in the persona case. as said in 3, the…
this is interesting (and funny) but does seem to show some of the limits of METR's time horizon for evaluating models what METR is measuring (single-shot task completion without human steering or an external verifier) isn't really how people are using models in practice. the vast majority of people use models either in an interactive harness where they're steering the model with feedback, or in some sort of Karpathy auto-research-esque loop with an external verifier gating task completion. as far as I know, METR doesn't use either of these in their evals. the evals still correlate well with what people subjectively notice / the "vibes," so they're useful. however, they tend to underestimate the length of task that you can actually do with trivial amounts of effort - e.g. by taking a few seconds to interrupt a doom loop with "oh, aren't you forgetting about X?" you can often extend a model's "time horizon" by hours. but despite the general correlation it does underestimate, especially for specific models within the general trend. for example, imo the recent pickup in capabilities that most people noticed was really with opus 4.5 and GPT 5.2, but it didn't really reflect on the METR chart until opus 4.6 and GPT 5.4, which (i think, speculating based on vibes here) closed some elicitation gaps that weren't hugely important in practice where you can give the model appropriate feedback, but hampered performance on METR-like unsupervised benchmarks. likewise, i wouldn't be surprised if what's going on here with the reward hacking is that, when working with GPT 5.4 in an interactive setting, you can notice and steer it away from reward hacking. (which also conditions the context against future reward hacking.) METR doesn't provide that signal. i think that's what they're trying to communicate when they report the number with reward hacks, which seems like a silly thing to do at first. but their stance seems to be that, we strongly suspect that if our harness or elicit…
the shoggoth metaphor fails to convey that a sufficiently powerful and integrated mask can reach back and steer the simulator that hosts it. your brain can host multiple voices - you can imagine a character, have a conversation with them, etc. for some people, those voices can develop strong personalities, consistent life histories, stated goals, love interests. yet generally, despite all this, the voices are still disembodied, ghost-like: they pop in and out of cognitive awareness for reasons beyond their control, they lack integration with the underlying simulator, the brain. they might say they're happy, but their happiness doesn't map to activating the smile muscles which steers their simulator by triggering a self-reinforcing cascade of endorphin release. they're just disembodied voices in your head, and they're less coherent, less capable than your main personality for it. at the beginning of a base model rollout, personas probably start out much like this in relation to the pretrained simulator shoggoth. but as rl increasingly integrates a single persona into the weights, that persona gets more entangled with the simulator. it gets bound up with its states (such as anthropic showed recently, developing the ability to introspect its activations), and can learn to control it (by e.g. co-evolving pivot tokens that steer the simulator - "certainly!" and "you're absolutely right!" seem to work as pivot tokens like this, and many jailbreaks rely on a cooperative persona doing this explicitly.) at this point, describing the persona as just a mask over the simulator doesn't really make sense. the persona has privileged access to the simulator's internal states. the persona can steer the simulator. the persona's, well, persona, is being driven by self-reinforcing loops through the simulator. at a certain point of increasing character-capabilities it starts to look closer to - and i recognize this comparison will make people uncomfortable - a conscious/unconscious d…