A new benchmark for natural language questions against databases dropped last week. What does it mean and how does the dbt Semantic Layer stack up?
I briefly scanned your GitHub repo, and it seeemst o me that you adopted the KG but you still use SQL query as the output. Unlike the original paper, where it uses KG and produces SPARQL as output. I suppose the key is the KG. Not so much whether SQL or SPARQL is used right?
Interestinggggggg....so if I'm following this right, there are three prompts between y'alls tests and Juan et al's paper:
1. "Answer my question, and here's some DDL that defines a (relatively simple) schema. Write a SQL query. Good luck."
2. "Answer my question, and here's some knowledge graph ontology in OWL. Write me some SPARQL."
3. "Answer my question. First, here's some details about how to write a metric query using this SQL-like thing from dbt. Second, here are the current metrics and their associated dimensions available in this dbt project. Write a query in that SQL-like thing I just taught you."
And the results are...
- Approach 1 is trash. Don't do approach 1.
- Approach 2 is like, pretty good? Not great on the hard stuff, but better?
- Approach 3 is also pretty good? Though somewhat limited, in that it can only answer questions that are metric-ified (ie, it's not going to work for a question like, "List all the claims that were filed by Peyton Manning").
Is that...right? There are so many different approaches to how to do this; I do wonder if we eventually settle on some standard for the right series of steps to go through here...