A new technique called “Skeleton-of-Thought” (SoT) shows promise for significantly speeding up text generation from large language models like ChatGPT, according to a new technical paper from researchers at Tsinghua University.
The paper explains that today’s most advanced AI chatbots and assistants use large neural network models to generate text, but do so sequentially – outputting words one-by-one. This sequential process bottlenecks the speed, taking many seconds to generate a lengthy response.
The researchers propose a new approach, SoT, which guides the AI to first output a concise skeleton or outline for the response, then generates the details for each part of the skeleton in parallel. By dividing up the work, multiple GPUs or processors can generate the full response much faster than the standard sequential method.
Experiments showed SoT accelerated response generation from a variety of large language models by 1.8x to 2.4x. Speedups were greatest on common question types like open-ended knowledge, common sense, and speculative questions. The technique was less effective for technical step-by-step questions requiring sequential reasoning.
The researchers believe parallel text generation could make AI assistants feel more responsive. Today’s chatbots have noticeable lag when formulating long answers, while SoT could provide an answer framework instantly, then fill it in rapidly. This might enable smoother conversational experiences.
Beyond speed, SoT’s skeleton-first approach could even improve relevance, as the AI focuses on covering distinct key points upfront. The parallel generation stage also adds diversity, as each part is created independently.
While promising, the technique needs refinement before being production-ready. Prompts must be improved to make the skeletons and parallel generation more robust across question types. Models could also be fine-tuned to better follow SoT’s multi-stage approach.
Nonetheless, being able to leverage parallel processing is a pivotal advancement for scaling up AI assistants. With SoT, chatbots could maintain swift and natural conversations as their knowledge grows into the trillions of parameters. The researchers believe the data-centric technique points the way towards more human-like thinking and writing from AI.