I like the idea of an AI that can modify its own code so it can get exponentially smarter, but I have some criticisms to this idea:

First, most AI programs we know today are not smarter or dumber due to their programming, but rather due to their size, training data quality, training compute thrown at them, whether they are RLHF’d (read “censored”) or not, and even their context size. Several of these factors have little to do with code quality. So it is not enough for an AI program to alter its own code unless it can improve on these dimensions (e.g., AI needs to re-train itself of better data). But these dimensions training a huge neural network is such an expensive, time-consuming, and tricky task that no one person can handle.

Second, rewriting code is not the problem, rethinking about AI paradigms and algorithms is the key. Current AI systems can do the former pretty good, but they fall short in the latter because "rethinking" requires creativity, curiosity, and intuitive thinking which current AIs lack.

Third, there's no clear measure of "smartness" of a system. An AI system that thrives to make itself smarter must somehow know if it's going in the right direction. A lot of times, incremental improvements might not result in noticeable changes in AI benchmarks, so the AI can't basically know if the changes were in the right direction or not (in other words, gradient-descent style of improvement doesn't apply here because we (1) don't know how to measure gradients of "smartness", and (2) even if we rely on AI benchmarks as a proxy for AI smartness, improving part of the code doesn't necessarily lead to noticeable jumps in our AI smartness measures (we could be measuring noise!).

Forth, the idea of self-improving AI would make more sense if our AI systems were symbolic or deterministic programs.

This whole idea of self-improving AI sounds like claims of unlimited energy or perpetual engine designs: somewhere something is consuming energy and you're not creating enough to compensate for it. Similarly, an AI can't get smarter on its own because you can't extract intelligence out of thin air.