A new kind of AI model from a ChatGPT inventor is thrilling developers

Tokyo-based startup Sakana AI, led by prominent researchers from Google and a co-inventor of the foundational transformer architecture, has released a new type of large language model (LLM) built using a novel, resource-efficient technique. The company successfully created a superior Japanese language model by merging two existing open-source models, bypassing the enormous computational costs associated with training such systems from the ground up.

A New Paradigm in Model Creation

Sakana AI's method is called an “evolutionary model merge.” The process begins by combining the parameters of two parent models to generate a large population of derivative “child” models. These new models are then evaluated against performance benchmarks. The best-performing versions are selected to become the parents for the subsequent generation, and the cycle repeats. This approach, inspired by natural selection, iteratively refines the models, enhancing their strengths while pruning their weaknesses without requiring extensive retraining.

Challenging Industry Norms

The development of cutting-edge foundation models has been dominated by a handful of tech giants capable of investing billions in the massive computing power required for training. Sakana AI's methodology presents a significant departure from this capital-intensive standard. By effectively building on the work of the open-source community, the evolutionary technique dramatically lowers the barrier to entry for creating powerful, specialized generative models.

Market and Industry Impact

This innovation could democratize access to advanced model development. Smaller companies and research labs could adopt similar techniques to create customized, high-performing models for niche applications at a fraction of the traditional cost. The successful demonstration, where the merged model outperformed both its parent models on Japanese language tasks, validates the approach as a viable alternative to the industry's brute-force scaling strategies. This signals a potential shift towards more efficient and collaborative model-building ecosystems.

Future Applications

Looking ahead, Sakana AI plans to apply its evolutionary merging technique beyond language. The company aims to use the same principles to develop new models for other domains, including computer vision and image generation. This strategy suggests a broader ambition to establish a new, more sustainable paradigm for creating a wide range of sophisticated generative systems.