Follow us on Google News
Supermicro has launched a new artificial intelligence infrastructure solution developed with IBM and NVIDIA, claiming that the technology can help large AI models begin responding up to 56 times faster when processing lengthy prompts.
Announced on November 5, 2026, the system uses a method called key-value caching, or KV caching, to store information that AI models have already processed. This allows the models to reuse previous calculations instead of repeatedly performing the same work.
The companies say the technology could reduce delays, improve the use of expensive graphics processing units (GPUs) and allow data centres to serve more AI requests without increasing computing resources at the same rate.
The solution combines Supermicro’s Petascale storage servers, IBM Storage Scale software, NVIDIA’s Dynamo inference framework and NVIDIA Spectrum-X Ethernet networking.
Joint testing showed substantial performance improvements, although the results were obtained under specified benchmark conditions and have not been independently verified across commercial deployments.
How The Technology Reduces Repeated AI Calculations
When someone asks an AI chatbot to examine a lengthy document, the system must process the supplied information before generating an answer.
This initial stage, called prefill, involves converting the input into tokens and performing calculations to understand the context.
AI models store intermediate information from these calculations in what is known as a key-value cache.
For example, an AI assistant analysing a large company report may need to refer to the same document while answering several follow-up questions.
Without an available cache, the system may have to process much of that information again, consuming computing power and delaying its response.
The problem becomes more significant when applications handle large documents, software code, images and extended conversations.
Conventional systems generally keep frequently accessed information within GPU memory. However, this memory is limited and expensive.
When the information exceeds available capacity, systems may discard cached data or move it to slower storage.
Supermicro, IBM and NVIDIA have developed an approach that allows previously calculated information to be stored outside the GPU and retrieved when required.
The companies say this can reduce unnecessary calculations while allowing multiple AI systems to access shared information.
The technology is particularly relevant to AI agents, which may repeatedly examine documents and use software tools while completing complex tasks.
Tests Show Major Improvements In Response Times
The companies evaluated the system using a large language model containing 120 billion parameters.
Parameters are the internal values an AI model learns during training and uses to process information.
During one test involving a prompt containing approximately 130,000 tokens, the system reportedly delivered a 56-fold improvement in time-to-first-token.
Time-to-first-token measures how long an AI system takes to begin generating its response after receiving a request.
In the documented benchmark, processing the long prompt without the shared caching system took approximately 32.14 seconds before the first token appeared.
With the cache enabled, the time fell to approximately 0.57 seconds.
The companies also examined what happened as prompt length increased from 30,000 to 130,000 tokens.
With caching enabled, the response-start delay increased by only 0.377 seconds. Without it, the delay increased by 29.55 seconds.
These results suggest that reusing previously processed information could substantially reduce waiting times in applications that repeatedly work with lengthy material.
However, the 56-fold improvement does not mean that every AI response will be generated 56 times faster.
The measurement concerns the delay before output begins, and the advantage depends on the availability of reusable cached information.
Higher Throughput Could Reduce Pressure On GPUs
The companies also tested whether the system could handle multiple requests simultaneously.
Under their benchmark conditions, processing throughput increased from 0.19 to 4.26 requests per second.
That represents an approximately 22-fold improvement.
The test involved processing 200 requests, with the caching system reducing the total time required to complete the workload by approximately 95%.
A separate evaluation introduced heavy competing network traffic to examine how the system performed when other applications were using the infrastructure.
Under those conditions, the solution maintained an 18-fold throughput advantage over the benchmark configuration that repeatedly calculated the required information.
The companies say these improvements could help businesses make better use of existing GPUs.
Such efficiency has become increasingly important as AI infrastructure grows more expensive and data centres face constraints involving electricity, cooling and hardware availability.
However, the announcement does not establish a universal reduction in electricity consumption or operating costs.
Those benefits would depend on the workloads, hardware configuration, storage requirements and frequency with which cached information can be reused.
Three Technology Companies Combine Storage And Networking
The new platform combines technologies from three major infrastructure companies, each responsible for a different part of the system.
Supermicro supplies high-performance Petascale storage servers powered by AMD EPYC 9005-series processors.
IBM Storage Scale manages shared storage and allows multiple computing systems to retrieve previously calculated AI information.
NVIDIA Dynamo coordinates distributed AI inference, while its Spectrum-X networking technology supports communication between the servers.
Together, these components create a storage layer that can operate beyond the memory available on individual GPUs.
The system is designed to support large-scale applications, including coding assistants, document analysis, retrieval-augmented generation and AI agents.
Retrieval-augmented generation allows AI systems to draw on external documents or databases while preparing responses.
The companies say the solution has undergone testing involving high request volumes and competing network traffic.
Supermicro is also offering integration and deployment services intended to help businesses install the system more quickly.
The companies have not disclosed the price of a complete deployment or named customers using the newly announced solution in production.
For businesses building AI services, the development highlights an increasingly important part of AI infrastructure: reducing the cost of operating models after they have been trained.
As organisations deploy AI applications for everyday work, the ability to reuse previously processed information could become as important as improvements in the underlying models.
The commercial impact will depend on whether the reported benchmark advantages can be reproduced across different workloads and operating environments.
What This Means For You:
Faster AI infrastructure could eventually reduce waiting times for document analysis, coding assistants and complex AI services. Businesses considering the technology should examine real-world performance, costs and data-security requirements rather than relying solely on headline benchmark results.
