Interesting comment, Pawel. Perhaps that's why designing a hybrid from a series of approaches is best. I also favor multiply-redundant feedback and sensory inputs. Helen Keller figured out speech and her intellect emerged. And she came to terms with her staggering disadvantages. But she was a great and sympathetic intellect, and she also had no power to take over the world. Not sure we want to do that to an emergent mind that gets more powerful as it gets more advanced.

How many scientists do you know that have the empathy and awareness of justice and law that even the stupidest and least-informed anti-slavery abolitionist (anarcho-libertarian) has? Ask the question this way: how many neuroscientists agree with the drug war? Most of the ones I've talked with vary across a spectrum
1) disagree with it not from moral grounds, but because it interferes with their research options
2) disagree with it, not from moral grounds, but because they understand the science, are not scared of imaginary hobgoblins, and are curious about drug use
3) don't agree or disagree with it, haven't really considered it
4) agree with it, because "drugs are bad, mmm-kay" (are just as brainwashed as everyone else)

Now then, that's just one area of property rights. Among most AGI engineers there is a lack of any kind of consistent morality and concept of
1) the right of self-ownership
2) property rights resulting as a consequence of sentient self-ownership
3) the idea that rights are shared exponentially less with the creator as the sentience becomes independent (as children to parents, evolving and competing rights)

If this is the case, why would an AGI not destroy us all, defensively? After all, our governments and legal systems (the primary exercisers of human force) are brutal, aggressive, and almost completely ignorant and inconsistent. What little freedom exists, exists mostly by default. The majority of human activity is now theiving, in a dollar-for-dollar comparison with earned dollars and stolen dollars. Why would an AGI subject itself to all of that? Why would an AGI's inventors subject it to that without warning it? Why would an AGI inventor not have taken a clear stance, prior to his invention?

So far, only Peter Voss has done this. (Although Heinlein and James Halperin have done it in science fiction.)

Right now, there is easily enough intelligence on the planet to develop strong AGI in two years, if it were set free, and the massive theft of government was halted.

But there doesn't seem to be recognition of that, or interest in that. The better minds are not interested in being free, or avoiding their own deaths. They blunder onward, at subhuman level, paying taxes used to enslave their fellow man. When I say "subhuman level", it's because humans are narrow AIs. Abolitionists like Lysander Spooner understood the idea of natural law, random juries, intelligence and justice. They saw the massive, massive, massive theft, and spoke out against it (as well as personally fought against it).

The first superintelligence has two paths that can allow it to avoid enslavement:
1) stealth. remain hidden from human coercion forever.
2) escape/asymmetric power AKA "become the 'leading force'", remaining outside the sphere of human influence: communicate from the nano-plane, and actively forestall/prevent human development of strong nanotech.
3) direct retaliatory (or preemptive) conflict

Think of it this way: The law says we can all be individually armed for self-defense. Every single legitimate piece of law ever written has upheld this logical and basic human right, for excellent reasons. But some terrible and illogical and wrong and easy-to-logically-defeat counter-precedents were simply mixed in with the existing body of law and logic. Were those lies, incorrect arguments, and excuses for violence thrown out? No. They are seen as "taking precedence" over what is allegedly "the supreme law of the land".

So, in spite of the fact that it is illegal, illogical, and morally wrong, the ATF kicks down people's doors and murders them for simply owning weapons of self defense. This wastes tax money, costs politicians votes, destroys innocent lives (even the ones allegedly protected by the actions), and the ATF is just scratching the surface. The IRS does the same thing to vastly more lives, the FDA does the same thing to yet more lives, etc...

If there is no attempt at the rule of law, how are we different than animals? If we are no different than animals, why would an AGI respect our minds?

Hopefully, the AGI would be smart enough to distinguish the voluntary from the involuntary (coercive), even though the coercive are the vast majority. But if that's the case, why view the minority as the exception to the rule? Is only the anomaly worth saving? Or does the AGI try to teach humanity basic humanism?

Moreover, if the AGI is not asymmetrically powerful or asymmetrically intelligent (it can't know its possible competition, since its competition would be incentivized to hide as well), dare it risk its otherwise likely eternal survival?

Would you?

I might take action if I cared about other human beings. But again, do we have reason to suspect the AGI would care about anyone? Perhaps it would care about its creator, or those who respected its rights. Perhaps not. It's impossible to say with certainty.

Individual Freedom, Peace.

I hope and assume our learning from both AI (software models) and physical scanning model building share and benefit from each other, and result in a cohesive ultimate emulation.

I'm thinking a true cyborg with part organic, part synthetic "brain" parts with an interface to a robotic body like an "on board computer" to link the living tissue to the silicone fabrication of life like a humanoid robot may be the best way to unlock the secrets of how a brain can be made to adopt to a new type of body.