This is a new area of moral philosophy that only became non-hypothetical when we have the processing and memory to achieve strong-narrow superintelligence around 2015.
I don't mean this in a strictly negative way, but in terms of the core principle, which can be summarized as:
New technologies create problems that cannot be predicted.
The classic example is the "blackening of London" due to pollution beginning in the 1660's. (See: Air Pollution in Industrializing England, Brimblecombe, 1978)
By contrast, the old-school Luddites got it wrong about the Jaquard looms.
Not hypothetical, and one can argue supportably that the present social dysfunction is a largely a function of automated algorithmic decisionmaking (See: Facebook, Youtube, et al.)
Key topics here involve voluntary human obsolescence—offloading not only repetitive tasks, but offloading competency and responsibility (militarized drones as an example.)
- If strong Artificial General Intelligence is ever achieved, the question of personhood will become non-hypothetical
There was already an attempt to list an algorithm as an inventor on a patent, which was rejected because the algorithm is not a "natural person." (Thaler v. Hershfeld, 09/02/21)
This is a legal question but deeply philosophical. What does it mean to be a person?
I'm calling to change the unfortunate name of this thought experiment to something more consistent with the mythology of AI. (Phillip K. Dick, here understood as a narrative philosopher, wrote about the difference between xenoglossia and glossolalia, which comment on Searle, in regard to Ancient Greek specifically. [See VALIS trilogy.] Dick is a major narrative philosopher along with Asimov, Lem, and recently, Rajaniemi. Dick and Asimov have probably had more influence that Searle in the public understanding of AI. They use mythology of AI to explore social concepts in the manner of Plato.)
There are more examples, so others should answer as well if I've missed anything.