I think it’s more likely a compound sigmoid (don’t Google that). LLMs are composed of distinct technologies working together. As we’ve reached the inflection point of the scaling for one, we’ve pivoted implementations to get back on track. Notably, context windows are no longer an issue. But the most recent pivot came just this week, allowing for a huge jump in performance. There are more promising stepping stones coming into view. Is the exponential curve just a series of sigmoids stacked too close together? In any case, the article’s correct - just adding more compute to the same exact implementation hasn’t enabled scaling exponentially.

Ditto, I was about to start waxing poetic about my bard.

I did some source digging to hopefully best address your observations. Science journalism (even when internal and likely done in concert with the authors) is fundamentally a game of telephone. But looking at the source papers:

They say it in an incredibly formal way, but they do seem to come to the conclusion that the LLM develops understanding. The paper makes that case within an incredibly narrow context, but it does include:

We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. We argue that the observed semantic content cannot be fully attributed to a retrieval-like process, and instead requires the LM to perform some degree of generalization over the semantics. More broadly, we see programs and their precise formal semantics as a promising direction for working toward a deeper understanding of the behavior of LMs, such as whether or how LMs acquire and use semantic representations of the underlying domain more generally.

With it now clear that the generalized case is not shown: the specific type of understanding that they have shown is non-trivial.

Conclusion: This paper presents empirical evidence that LMs of code can acquire the formal semantics of programs from next token prediction.

A foundational topic in the theory of programming languages, formal semantics (Winskel, 1993) is the study of how to formally specify the meaning of programs.

From Winskel: The Formal Semantics of Programming Languages provides the basic mathematical techniques necessary for those who are beginning a study of the semantics and logics of programming languages. These techniques will allow students to invent, formalize, and justify rules with which to reason about a variety of programming languages.

Also notable but unrelated: Jin and Rinard’s paper was supported, in part, by grants from the U.S. Defense Advanced Research Projects Agency (DARPA).

/stares in smart glasses

As long as no one messes with their open source contributions… (ditto for MS)

To the one person who upvoted this: We should be friends.

On the other hand, the human participant scored 67 percent, while GPT-3.5 scored 50 percent, and ELIZA, which was pre-programmed with responses and didn’t have an LLM to power it, was judged to be human just 22 percent of the time.

54% - 67% is the current gap, not 54 to 100.

Thank you, I seldom see my own thoughts laid out so clearly. As a practitioner of the Dark Arts (marketing), this union of commerce and art is a foul bargain. I think it’s time the two had some time apart to work on themselves.

Until they can distribute the training load of large models to consumer graphics cards (and do something like SETI@Home) it does seem like the benefit of distributed training isn’t enough to overcome the friction.

Wikipedia got where it is today by providing accurate information. Google results have always been full of inaccurate information. Sorting through the links for respectable sources just became second nature, then we learned to scroll past ads to start sorting through links. The real issue with misinformation from an AI is that people treat it like it should be some infallible Oracle - a point of view only half-discouraged by marketing with a few warnings about hallucinations. LLMs are amazing, they’re just not infallible. Just like you’d check a Wikipedia source if it seemed suspect, you shouldn’t trust LLM outputs uncritically. /shrug

Oh I have an opinion, I just meant to keep track of everyone’s for the sake of conversation.

I’ve heard every possible combination of thoughts on A.I. We need like a 6-dimensional alignment chart.

To be clear, if they can make living dinosaurs, they totally should. Like, don’t make an amusement park out of it, but if we can bring dinosaurs back and choose not to, that’s gotta be some kinda unethical.

deleted by creator

But for real, if you want to know the state of AI, go to Hugging Face.

Fuck OpenAI’s attempts at regulatory capture. But A.I. is amazing. Fuck humans.