The adoption of generative artificial intelligence is accelerating rapidly, but moving from limited experimentation to production deployment is far from simple. A recent survey conducted by MIT Technology Review Insights in August 2024 highlighted the challenges and decisions organizations face during the transition to practical use of these tools. With 250 executives from a wide range of industries interviewed, significant insights emerge into the current landscape and the operational difficulties companies are experiencing.
The Adoption of Generative AI Systems. Rapid Growth
Generative artificial intelligence has exploded following the introduction of ChatGPT in November 2022, with companies worldwide beginning to explore large language models (LLMs) to solve complex and labor-intensive problems. Beyond the ability to solve complex technical issues, generative AI also offers the potential to automate repetitive processes and handle unstructured data, such as videos and PDF documents. These capabilities have attracted massive interest from companies eager to leverage these technologies to gain competitive advantages and improve operational efficiency.
According to the survey conducted by MIT Technology Review Insights, the estimated productivity gains from adopting generative AI could have an impact comparable to major historical innovations, such as the internet, robotic automation, and the steam engine. Projections suggest an impact on global GDP ranging from just under a trillion dollars to as much as $4.4 trillion annually. This broad range of estimates reflects the variability in the ways AI is implemented and the ability of companies to adapt their operating models.
The survey also highlighted that while 79% of companies planned to adopt generative AI projects within the next year, only 5% had managed to put actual use cases into production by May 2024. This delay is attributed to operational difficulties and the need to overcome challenges related to output quality, integration into existing systems, and high inference and training costs.
In addition to technical difficulties, trust in the effectiveness of applications also emerged as an issue. Two-thirds of the business leaders interviewed stated that they felt ambivalent or dissatisfied with the progress made so far, citing the complexity and cost of production deployment as the primary reasons. Many companies are therefore trying to build a solid technology stack that can support various foundational models, advanced integration tools, and orchestration solutions to facilitate the large-scale adoption of generative AI.
Operational Challenges and Implementation Complexity
Among the main challenges reported by business leaders, the quality of AI output is a concern for 72% of respondents. Additionally, 62% reported difficulties in integrating with existing infrastructures, while 58% cited high costs for both inference and model training. Latency is another crucial issue: 56% of companies struggle to reduce response times, particularly in high-interaction, real-time usage scenarios.
Another frequently cited problem concerns the management of context by generative models. Harrison Chase, co-founder and CEO of LangChain, emphasized that one of the biggest challenges is providing the right context to the model, especially when linking the results of an LLM to a specific dataset. An effective “orchestration layer” is needed to convey the appropriate context and ensure that responses are relevant and accurate. However, providing greater context to models often implies increased costs, making it crucial to find a balance between response quality and economic efficiency.
Training and inference costs are among the most significant challenges: about 58% of companies reported that the costs of running models are still too high, especially for applications requiring high scalability. The cost per token, as highlighted by Rowan Trollope of Redis, is a key parameter for optimizing model efficiency: reducing the cost per token can make large-scale inference more affordable, allowing companies to derive greater value from AI deployment.
The difficulty of quantifying return on investment (ROI) also represents a barrier to adoption. According to the survey, 48% of companies are trying to use key performance indicators (KPIs) to evaluate their AI projects, while 38% have developed specific frameworks to assess the impact of generative AI. However, the lack of standardized methods and the inherent complexity of calculating ROI slow down decision-making processes. Many organizations hesitate to invest further in AI without clear evidence of the value generated.
Scalability is another significant challenge. While 51% of companies mentioned difficulties in keeping up with growing demand and ensuring systems can handle an increasing number of users, latency becomes a closely related problem. Each new system component adds latency, which can compromise the user experience. Latency is particularly critical in real-time generative AI applications, such as voice interfaces, where even a few milliseconds of delay can negatively impact interaction.
Composite AI Systems: A Possible Solution
To address these challenges, many companies are exploring composite artificial intelligence systems or “compound AI,” which combine different models and AI technologies to manage complex tasks more efficiently. According to the survey, 54% of companies already use AI agents, and another 29% plan to do so in the future. Composite systems can reduce costs by segmenting work among cheaper models at certain stages of the process while simultaneously improving overall performance.
A central aspect of creating composite AI systems is the use of multi-step chains, which allows a complex task to be divided into a series of simpler, specialized steps. According to the survey, 50% of companies have already implemented multi-step chains in their generative AI applications, and an additional 18% plan to do so. This approach enables the use of specialized models for individual phases of the process, reducing costs and improving overall system efficiency.
Another key element is semantic routing, which allows user requests to be directed to the most appropriate tool, which could be another AI model or even a human intervention. This type of routing optimizes the use of available resources, avoiding overloading costly models for tasks that can be handled more economically.
The adoption of components such as Retrieval-Augmented Generation (RAG) is an example of the composite approach in action. RAG combines a model’s generative capability with searches through corporate databases or documents, improving the relevance and quality of responses. Currently, 38% of companies use this technique, and another 29% plan to implement it.
Another technological element supporting composite systems is the use of semantic caches and vector databases. Semantic caches, adopted by 32% of companies, help group and store responses to similar requests to reduce inference costs. Vector databases, adopted by 37% of companies, are essential for storing and searching complex representations of data and questions in vector format, thereby optimizing the ability to retrieve information.
Strategies for Building an Adaptable AI Stack
To build an adaptable AI stack, companies must address various strategic and operational challenges. A fundamental step is choosing the model or models on which to base applications: 67% of companies have opted for third-party closed models, such as those offered by OpenAI, while 42% have chosen open-source models on the cloud. The preference for open source is growing, partly because it allows greater flexibility and control over costs and security.
The adoption of on-premises open-source models is an option considered by 41% of the companies surveyed: 17% already use them, while another 24% plan to implement them in the future. This solution can offer companies greater control over data and reduced licensing costs associated with commercial models.
Another crucial element for building an adaptable AI stack is integrating existing technologies with new AI solutions. According to the survey, using standardized APIs, such as those offered by Redis and LangChain, allows companies to easily exchange and replace components within their stack, ensuring greater operational flexibility. Standard APIs help overcome differences between model parameters and facilitate the integration of new tools.
Another important consideration is latency management in composite systems, especially for applications requiring real-time responses. To overcome this issue, it is essential to adopt a high-speed data platform capable of supporting models that respond quickly, minimizing user wait times. Rowan Trollope of Redis emphasized that “latency is the new downtime”: response speed becomes a determining factor for large-scale adoption and the success of generative AI applications.
The complexity and costs of implementation remain significant barriers, but the adoption of technologies such as semantic caches and vector databases offers an opportunity to improve overall efficiency. These tools can reduce the load on more expensive models, increasing the speed and relevance of responses and are already adopted by a substantial portion of the surveyed companies.
In conclusion, building an adaptable AI stack requires balancing cost efficiency, latency management, and operational flexibility.
Why Enterprise AI Projects Fail and How to Reverse the Trend
Despite the transformative potential of artificial intelligence, a significant percentage of enterprise AI projects fail to achieve the desired outcomes. Recent estimates indicate that over 80% of enterprise AI projects fail, a rate twice that of traditional IT projects. This high failure rate reflects the difficulties many companies encounter in transitioning from experimentation to truly functional and production-ready projects.
According to the study “The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed” by James Ryseff, Brandon De Bruhl, and Sydne J. Newberry, five main causes of AI project failure were identified through interviews with 65 data scientists and engineers in the industry. The most frequently cited cause is the inability of corporate leadership to properly define project goals. A lack of strategic vision, combined with a poor understanding of AI’s actual capabilities, often leads to initiatives that fail to meet expectations.
Another significant cause relates to data quality. Without accurate, complete, and relevant data, even the best AI models fail to provide reliable results. The lack of adequate data was cited by more than half of the respondents as one of the main reasons for failure. This issue is often accompanied by a lack of appreciation for data engineering, which is seen as a less valuable activity compared to model development.
Other factors contributing to failure include a lack of adequate infrastructure to support the development team, operational errors by team members, and the intrinsic limitations of AI technology capabilities. Respondents also highlighted how inadequate leadership involvement in operational details and technical decisions contributes to poor alignment between business goals and the AI solutions developed.
To reverse this trend, organizations must adopt a more holistic and strategic approach. It is crucial that corporate leadership is actively involved in the process, ensuring that project goals are clear and realistic. Leaders must work closely with the development team to translate these goals into concrete and achievable technical requirements.
Moreover, investing in solid infrastructure and competent machine learning (ML) engineers is essential to overcome data quality issues and ensure proper model implementation. A clear understanding of AI’s real capabilities and limitations, combined with a long-term commitment, can help turn AI projects from failed experiments into tangible successes that bring real value to organizations.
Conclusions
Generative artificial intelligence represents a technological shift with extraordinary potential, but its full exploitation requires a profound rethinking of business and operational strategies. Companies today face a dual challenge: on the one hand, seizing the opportunities offered by this innovation to improve efficiency and generate value; on the other, overcoming technical, economic, and cultural barriers that hinder large-scale implementation. This duality reveals a crucial point: the adoption of generative AI is not a simple technological evolution, but a catalyst for systemic change.
A key aspect is the need to rethink the corporate digital infrastructure. The traditional approach, characterized by monolithic and static systems, is no longer adequate to support technologies that require adaptability, scalability, and optimal resource utilization. The emergence of composite and modular solutions, such as multi-step chains, semantic routing, and the use of vector databases, marks the transition to a vision where flexibility is the core of efficiency. Companies must learn to segment processes, distribute loads, and optimize costs, turning complexity into an opportunity to gain competitive advantages.
Another fundamental lesson concerns the relationship between innovation and ROI. The difficulties in measuring the economic impact of generative AI are not merely a technical obstacle, but a symptom of a broader gap: the inability of many companies to recognize and value the intangible benefits of innovation. Decision-making speed, large-scale personalization, and improved user experience are not easily quantifiable, but they can determine success in increasingly competitive markets. Companies that develop innovative methods for measuring the value generated by generative AI will have a decisive advantage.
Trust emerges as another strategic pillar. The fact that many business leaders declare themselves ambivalent or dissatisfied with the progress in implementing generative AI underscores a cultural as well as technological problem. Building trust cannot be limited to output quality: it must include transparency on costs, predictability of outcomes, and an ethical approach to data use. In an era where corporate reputation is increasingly linked to responsible technology management, the adoption of open-source models and on-premises solutions is not only a technical choice but also a statement of intent.
Finally, speed becomes a critical factor. In a scenario where latency is “the new downtime,” as stated by Rowan Trollope, the ability to respond quickly to user needs is not just a technical issue, but a factor that directly influences the perceived value by customers.
Investing in infrastructure that reduces latency and increases operational resilience is not a cost but a strategic investment that can differentiate companies in saturated markets.
In summary, adopting generative AI is much more than a technological issue: it is a strategic challenge that requires new skills, new metrics, and a holistic vision of innovation.
Comments