Stratio Business Semantic Data Layer delivers 99% answer accuracy for LLMs

Generative AI is now ready for enterprise decisioning following benchmark test

Click here to watch the on demand webinar “Stratio Business Semantic Data Layer delivers 99% answer accuracy for LLMs” and check how Stratio was able to achieve these results. Hosted by Román Martín González, Chief Technology Officer at Stratio BD.  Spanish version available here.

The results, produced for the first time by linking ChatGPT analysis of independent datasets with Stratio BD’s Business Semantic Data Layer, confirm that generative tools can now deliver reliable results from data analysis and interrogation. This means that businesses can rely on gen AI analysis to produce accurate answers, allowing employees with any level of technical expertise to ask data-related questions in their natural language, and receive answers with a high level of accuracy, immediately.

The test conditions mirror a benchmark test previously conducted by data.world two months ago, which resulted in LLMs delivering three times greater efficiencies when connected to a knowledge graph. This test was subsequently validated in December 2023 by dbt labs, which saw the accuracy rate hit 69%. In this latest replication of the test, Stratio BD found that ChatGPT 4 was able to accurately answer data queries ran on large amounts of data, up to 99% of the time.

Background

As readily-available Large Language Model (LLM) platforms like ChatGPT, Bard AI, Mistral, and Bing AI become more prevalent, business leaders are increasingly exploring the potential of Generative AI (Gen AI) technologies to transform their business operations. For instance, according to one study,  78% of organizations plan to invest in AI platforms over the next 12 months. 

However, business owners need reassurance they’re getting the most out of what this tech has to offer. LLMs, while capable of generating masses of human-quality text in milliseconds, may be unable to process technical data properly, giving rise to inaccurate responses (also known as ‘hallucinations’) and erroneous decision-making.
To avoid making wrong decisions, business users expect accurate results when using LLMs and Generative AI. Stratio Generative AI Data Fabric is able to deliver 99% answer accuracy for LLMs on the basis of a benchmark test in accordance with the methodology described in this document.

Benchmark Execution

The following is a summary of the results obtained for each of the scenarios described below:

To understand the results shown in the table, you need to take into consideration that:

• Low Schema corresponds to a subset of 12 tables from the standardized dataset
• High Schema corresponds to a subset of 28 tables from the standardized dataset

During the benchmark execution, three distinct types of scenarios were conducted:

1. Generation of SQL Queries on Technical Schemas with Generative AI:

• Context: Database’s DDL
• This involves the generation of SQL queries leveraging Generative AI, with the database’s Data Definition Language (DDL) as the contextual framework.

2. Generation of SPARQL Queries on Knowledge Graph with Generative AI:

• Context: Knowledge Graph
• This scenario focuses on generating SPARQL queries using Generative AI, with the knowledge graph serving as the contextual basis.

3. Generation of Stratio Virtualized SQL Queries on Stratio Business Knowledge Graph with Generative AI:

• Context: Stratio Business Knowledge Graph (Business Data Layer, Business Glossary, Governance Attributes)
• This scenario entails the utilization of Stratio’s Chain for SQL query generation, specifically on the Stratio Business Knowledge Graph. The contextual elements include the Business Data Layer, Business Glossary, and Governance Attributes.

Conclusions

The results obtained during the benchmark execution validate the following graph:

And we obtained the following conclusions:

1. The use of a semantic layer enhances the precision of obtained responses by 40% compared to a technical data layer.
2. Leveraging Stratio’s Business Semantic Layer improves responses by 80% over the technical layer and 40% over the semantic layer.
3. Implementation of Stratio’s SQL generation chain ensures accurate results in both Low and High complexity schemas, significantly mitigating indeterminacy and hallucinations associated with LLMs.
4. The achieved precision with Stratio Generative Data Fabric validates the feasibility of the use case wherein business users with limited technical knowledge can derive real-time insights from enterprise data.
5. Furthermore, Stratio offers the advantage of not requiring data movement from its source, as access is facilitated through the Virtualizer.

Stratio Generative AI Data Fabric

Stratio product which provides 99% accuracy is available now. This improvement in accuracy is thanks to three pillars:

1. Creation of the Semantic Layer. Establishing a layer that semantically interprets and represents the underlying data more effectively for querying and analysis. With this Semantic Layer, the improvement observed in the benchmark is a 37.5% increase in accuracy. This enhancement stems from the fact that connecting the LLM to a semantic layer enhances its understanding of the data model compared to connecting it to the technical layer. Consequently, it is capable of accurately responding to a greater number of queries.
2. Creation of the Business Semantic Layer: This involves the enrichment of the Semantic Layer with business knowledge and business terms that are created and linked to it. This information is then provided to the LLM when it is requested to respond to a business-specific query.
3. Stratio’s Chain Gen-AI for Natural Language Query Generation: Employing the Retrieval Augmented Generation pattern, this component integrates essential business knowledge into LLMs, enabling them to accurately respond to user queries formulated in natural language. As observed in the Benchmark execution, the use of Stratio’s Business Semantic Layer and Stratio’s Chain Gen-AI results in an improvement of 81.71% in Accuracy, achieving accuracies exceeding 95% in both Low Schema and High Schema.

What can we learn from this?

All you need to improve accuracy is a business semantic data layer. It is created with the help of actionable data governance in four simple steps:

1. Automatic discovery of the technical data model on which natural language queries are to be made.
2. Creation of the Semantic Layer through an ontology.
3. Mapping of the Semantic Layer with the technical schema.
4. Association of business knowledge to the Semantic Layer to create the Business Semantic Layer.

Giving business explanation to the different terms and making this resulting knowledge available to generative AI is the key. No betas, no waiting lists, no moving of the data from its storage location (like with many other providers). 

Already available within Stratio Generative AI Data Fabric product and used by clients in all industries, including banking, manufacturing, and retail.

By linking ChatGPT to our Semantic Business Data Layer solution, we can deliver highly accurate answers, without errors or hallucinations, and have solved the largest problem associated with LLMs. Businesses across every industry now have a viable way to significantly bolster productivity by automating otherwise lengthy tasks and queries and interrogating their business data quickly and effectively.”

Román Martín González, Chief Technology Officer at Stratio BD

Full results are available on the GitHub.  

For further analysis Román Martín González, Chief Technology Officer at Stratio BD has hosted an open webinar in English and Spanish . Click here to obtain the English version. Spanish version available here.