A few days ago, I posted this question that was later closed for being off-topic. I don't mean to beat a dead horse, but I genuinely don't think the question broke any rules.

Question. Is my argument valid? Are there any significant holes or logical fallacies? Has any form of this argument been made before?

Here's why I don't think this question was off-topic:

• This help article lists the following topics as off-topic: career path recommendations, general programming questions, implementation questions unrelated to theoretical topics, and questions seeking pre-trained models. My question does not fall into any of these categories.

• This help article describes the type of questions that should be avoided. In particular, it states that subjective questions should usually be avoided, but are sometimes okay. But even if my question is to be classified as subjective, it fulfills all the criteria for an allowable subjective question.

• The comments mentioned that my post was "not a question, but a discussion point, and "this could lead to discussions because some of your assumptions may not be correct."

Though I did ask a specific question, I understand that it could have lead to discussion. But doesn't every question lead to some degree of discussion? I don't see how my question, which was focused on a specific argument against AGI, would lead to more discussion than open-ended questions like this and this.

Also, if someone finds incorrect assumptions in my question, they could have posted those as an answer. My question was asking whether or not I made any incorrect assumptions or other logical missteps.

• I meant for my question to be like the proof-verification questions on Math Stack Exchange. I didn't mean for it to be some kind of ongoing debate or discussion. I was looking for answers of the form, "This argument is flawed because _____."

Question. Why was my question marked off-topic? Is there a specific rule/guideline that I broke?

• I'm sorry to hear that. Unfortunately your post is no longer accessible, so I can't see the original message. Aug 25 at 13:41
• Hmm, the post was automatically deleted by Community a few days ago :/. Is there any way to recover it? Aug 29 at 1:00
• I don't know about that. You could try searching for that, or trying something like Internet Archive Aug 29 at 1:21
• @AndreGoulart I reposted Frank's question below. Thanks for contributing!
– DukeZhou Mod
Aug 31 at 1:12

I reviewed the question, which I like very much, but here's why it was closed:

It's more of a thesis that gets around to the question. In the previous incarnation of this stack, we were allowing it. But it becomes too easy to abuse, and so the community felt it was better not to allow.

I don't see this question as that, but I think it would be more suitable if you addressed a single claim per question. I want to see more of these questions, so I hope you'll give the subject another shot.

• In short, it was ON-TOPIC. But it seems more like a debate than an objective question. Is that so? @Frank, I liked your post. If you can reformulate a specific and objective question - or split into several shorter questions - that will be great. Aug 31 at 1:51
• @AndreGoulart I would very much like to get these topics reposted, and think 4 or 5 questions is better than one, and better for the stack. (We could link the related questions in first comment or subscript at the end of the posts.
– DukeZhou Mod
Aug 31 at 1:55

Repost of Question in question:

The following is an argument for why I don't think artificial general intelligence (AGI) is technologically feasible with machine learning methods. There are likely many flaws in my argument, but I do find the overall idea to be compelling.

Question. Is my argument valid? Are there any significant holes or logical fallacies? Has any form of this argument been made before?

TL;DR. Training an AGI would require resources (computational power and data) comparable to all the resources that nature has invested into the evolution of human beings. Given our current abilities, this does not seem feasible.

Claim 1. Most of human knowledge is encoded into DNA.

Consider our knowledge of language. Why is it that the language model GPT-3 needed to be trained on hundreds of billions of words over thousands of GPUs to develop language skills comparable to what a human can develop in just several years? The answer is DNA. Humans already have language skills genetically hard-wired into them when they are born, which allows a human baby to learn language significantly faster than a computer can (this idea is similar to the [purely speculative] idea of a language acquisition device). More importantly, there is a huge difference in the amount of time and energy it takes a computer to learn language compared to a human. This huge difference indicates that most of an adult human's language knowledge is not learned within their short lifetime, but rather encoded through their DNA. A similar argument can be made for other aspects of human knowledge.

So if most of human knowledge comes from DNA, how is this knowledge obtained? I argue that the answer lies in evolution.

Claim 2. Natural selection can be viewed as a machine learning system.

To generate the DNA of an intelligent animal, let's suppose we treat this animal as a machine learning model. The parameters of this model are segments of DNA, and the optimization objective is to maximize the likelihood of an animal's survival. Natural selection trains this model via the following process:

1. Start with $$n$$ animals.

2. Let $$\mathcal{L}$$ be a function that computes the likelihood of an animal surviving. Apply this function to each of our $$n$$ animals to determine whether they survive. Suppose at the end, we have $$n'$$ animals remaining.

3. Let $$c$$ be the number of children each animal gives birth to, on average. Using the $$n'$$ available animals, mix and match their genetic codes and add random mutations to create $$cn'$$ new animals.

4. Repeat steps 2 and 3 with $$n := cn'$$.

By definition, this process describes a machine learning algorithm. For instance, consider the following analogous method of training a neural network:

1. Start with $$n$$ neural networks with randomized weights.

2. Let $$\mathcal{L}$$ be the loss function of the neural network. Pick the $$n'$$ neural networks with the smallest loss.

3. Let $$c$$ be some constant. Using the $$n'$$ available neural networks, mix and match their weights and make random adjustments to create $$cn'$$ new neural networks.

4. Repeat steps 2 and 3 with $$n := cn'$$.

Of course, this is arguably much worse than backpropagation and gradient descent. I return to this point at the end of my argument.

Claim 3. Nature has invested significant computational power in “training” humans.

We might think of nature as a large simulation. Every worldly event, whether it be the wind or the rain or the inner workings of a plant, requires “computational power” to execute. Furthermore, humans have taken hundreds of millions of years to train, and each year, natural selection has probably processed billions of animals (this number is just a wild guess, but I would say it's very conservative) that are relevant to the evolution of humans. Let's say it takes a modern computer, on average, one day to simulate the life of one such animal. Combining these estimates, the total amount of computational time $$T$$ required to “train” a human is given by:

\begin{align} T &= (\text{# of years}) \cdot (\text{animals processed per year}) \cdot (1 \ \text{day}) \\ &= 10^8 \cdot 10^{9} \cdot (1 \ \text{day}) \\ &= 3 \cdot 10^{14} \text{years}. \end{align}

That's a lot of time!

Claim 4. Nature has invested significant data in “training” humans.

The argument here is similar to the previous claim: everything natural event (“whether it be the wind or the rain or the inner workings of a plant”) is a piece of data that may have been used to “train” human beings. The sum of natural events that have been relevant to the training of humans has been large, so the amount of data that has been invested into the training of humans has also been large.

Argument. From the above four claims, we may deduce three plausible scenarios in regard to AGI:

1. AGI will not be developed using machine learning methods.

2. AGI will be developed with machine learning methods that are many, many, MANY orders of magnitude more efficient than natural selection (in regards to data efficiency and computational efficiency). (See claims 3 and 4 for an idea of what “many, many, MANY” means.)

3. AGI will be developed using computers that are many, many, MANY orders of magnitude more powerful than current computers.

Scenario 3 seems unlikely given the failure of Moore's law. Scenario 2 sounds more reasonable, but given how AI has developed in the past few decades, I would still say scenario 2 is rather unlikely. Currently, we don't have machine learning methods that perform significantly better than neural networks (+ variants like RNNs and CNNs) and gradient descent (+ variations like Adam). While these methods are undoubtedly better than the procedure of natural selection I described above, I don't think they are “many, many, MANY orders of magnitude” more efficient, especially considering how inefficient the optimization of deep neural networks is. Therefore, scenario 1 is most likely to happen, at least in the near future.