The unveiling of Meta’s new Lama 4 AI over the weekend sparked a wave of anticipation in the tech community. The first round of challenges set for it, including the coding vibe check with the bouncing ball, were cleared with aplomb. However, a slight hiccup in the handling of collisions was noticed, prompting further scrutiny. Was this a mere anomaly, or could it indicate a deeper issue?
This video is from Two Minute Papers.
To understand the implications of this, we need to delve into the workings of Lama 4. It is one of two new AI models, Scout and Maverick, available for public use. These models are smaller siblings of Behemoth, a more complex model that is still in the training phase. Behemoth serves a dual role—it is not only a more advanced model in its own right but is also used to teach the smaller networks.
A remarkable feature of Lama 4 is its ability to recall large amounts of data almost perfectly, as demonstrated by the Deep Seek test. When fed a substantial amount of information, it returned nearly all of it correctly. This capability could be compared to giving the AI a massive textbook and then asking it questions about the content.
However, when compared to its predecessor, Lama 4 seemed to fare worse. This was surprising, considering that Lama 4 has a context length of 10 million tokens, nearly 80 times more data than DeepC can handle. This implies that it can process approximately 10 hours of video and answer questions based on the content, a feat unmatched by other AI systems.
Despite its impressive context length, Lama 4 does have limitations—it may occasionally forget information. This, amusingly, makes it seem more human-like. Its capabilities and limitations make it a useful tool for handling large code bases and making changes, even when it’s not the best at coding.
Another notable feature of Lama 4 is its mixture of expert model. It works like a committee of specialized AIs, meaning that the whole neural network doesn’t need to be active at the same time. This makes it highly efficient, as it can run quickly even on high-end machines with a bit of quantization.
However, it’s not all smooth sailing for Lama 4. Independent studies stress-testing the context memory have raised concerns. Additionally, it does not come under an MIT license, which potential users should be aware of.
In conclusion, Lama 4 represents a significant leap in AI technology, particularly in handling large context projects. While it may not dominate all fronts, its unique abilities and innovations, such as the almost infinite text handling, set it apart. Like most technologies, it has its strengths and weaknesses. But the fact that we now have access to such high-performing, cost-effective AI models is a testament to the rapid advancements in the field. The future of AI looks promising, with free and open models paving the way for more open science.
