# Semantic Equivalence Analysis: LLM Query vs Ground-Truth Query
## Query Comparison
### LLM-Generated Query
```sparql
PREFIX sc: <http://schema.org/>
PREFIX secchain: <https://w3id.org/secure-chain/>
SELECT DISTINCT ?version
WHERE {
?software sc:name ?softwareName .
FILTER(?softwareName = "openssl" || ?softwareName = "OpenSSL")
?software secchain:hasSoftwareVersion ?sv .
?sv secchain:versionName ?version .
FILTER NOT EXISTS {
?sv secchain:vulnerableTo ?vuln .
?vuln sc:identifier "CVE-2023-3817"
}
}
ORDER BY ?version
```
### Ground-Truth Query
```sparql
PREFIX schema: <http://schema.org/>
PREFIX sc: <https://w3id.org/secure-chain/>
SELECT DISTINCT ?version
WHERE {
?lib schema:name "openssl" .
?lib sc:hasSoftwareVersion ?version .
?version a sc:SoftwareVersion .
FILTER NOT EXISTS {
?version sc:vulnerableTo ?vulnerability .
?vulnerability schema:identifier "CVE-2023-3817" .
}
}
```
---
## Semantic Equivalence Verdict: **NOT EQUIVALENT** ❌
---
## Detailed Analysis
### 1. Triple Pattern Differences
#### Pattern 1: Software Name Matching
**LLM Query:**
```sparql
?software sc:name ?softwareName .
FILTER(?softwareName = "openssl" || ?softwareName = "OpenSSL")
```
**Ground Truth:**
```sparql
?lib schema:name "openssl" .
```
**Difference:**
- **Namespace:** LLM uses `sc:name` vs Ground Truth uses `schema:name`
- **Matching Logic:** LLM uses OR filter (case-flexible) vs Ground Truth uses exact match
- **Risk:** May match different entities if `sc:name` and `schema:name` are not equivalent properties
#### Pattern 2: Version Retrieval
**LLM Query:**
```sparql
?software secchain:hasSoftwareVersion ?sv .
?sv secchain:versionName ?version .
```
**Ground Truth:**
```sparql
?lib sc:hasSoftwareVersion ?version .
?version a sc:SoftwareVersion .
```
**Critical Differences:**
- **Variable Binding:**
- LLM binds `?sv` to SoftwareVersion resource, then extracts `?version` as literal value
- Ground Truth binds `?version` directly to SoftwareVersion resource (URI)
- **Type Constraint:**
- LLM: NO explicit type constraint
- Ground Truth: `?version a sc:SoftwareVersion` (EXPLICIT type constraint)
- **Return Value:**
- LLM returns literal string from `versionName` property
- Ground Truth returns URI of SoftwareVersion resource
#### Pattern 3: Vulnerability Filtering
**LLM Query:**
```sparql
FILTER NOT EXISTS {
?sv secchain:vulnerableTo ?vuln .
?vuln sc:identifier "CVE-2023-3817"
}
```
**Ground Truth:**
```sparql
FILTER NOT EXISTS {
?version sc:vulnerableTo ?vulnerability .
?vulnerability schema:identifier "CVE-2023-3817" .
}
```
**Differences:**
- **Subject:** LLM filters on `?sv` (intermediate variable) vs Ground Truth on `?version` (result variable)
- **Namespace:** LLM uses `sc:identifier` vs Ground Truth uses `schema:identifier`
- **Semantic Consistency:** Both apply NOT EXISTS filter, but on different graph patterns
---
## 2. Risk Assessment
### High-Risk Differences
#### A. Missing Type Constraint ⚠️ **CRITICAL**
**Issue:** LLM query lacks `?version a sc:SoftwareVersion`
**Risk:**
- May match entities that have a `versionName` property but aren't proper SoftwareVersion instances
- Could include incomplete or malformed version nodes
- **Empirically confirmed:** LLM included 2 vulnerable versions (1.1.1g, 1.1.1l) that ground truth filtered out
**Likelihood of False Positives:** HIGH
**Likelihood of False Negatives:** LOW
#### B. Namespace Inconsistency ⚠️ **HIGH**
**Issue:** LLM uses mixed namespaces inconsistently
- Uses `sc:name` instead of `schema:name`
- Uses `sc:identifier` instead of `schema:identifier`
**Risk:**
- If `sc:name` ≠ `schema:name`, could match different software entities
- If `sc:identifier` ≠ `schema:identifier`, could fail to filter vulnerabilities correctly
- **Namespace aliasing matters:** RDF distinguishes `<http://schema.org/>` from `<https://w3id.org/secure-chain/>`
**Likelihood of Divergence:** MEDIUM-HIGH (depends on graph structure)
#### C. Different Return Types ⚠️ **MEDIUM**
**Issue:**
- LLM returns literal values (strings from `versionName`)
- Ground Truth returns URIs (SoftwareVersion resources)
**Risk:**
- String representation differences (URL encoding: `%3D` vs `=`)
- Cannot directly join/compare results with other URI-based queries
- May cause integration issues in downstream processing
**Empirically Confirmed:** 2 false negatives due to URL encoding differences
### Medium-Risk Differences
#### D. Software Name Matching Breadth
**Issue:** LLM matches both "openssl" and "OpenSSL"
**Risk:**
- Could match multiple distinct software entities
- Graph may contain both lowercase and capitalized entries representing different packages
- **Potential for false positives** if "OpenSSL" entity is distinct from "openssl"
**Likelihood:** LOW-MEDIUM (depends on data quality)
---
## 3. Edge Cases Where Results Could Diverge
### Edge Case 1: Namespace Property Divergence
**Scenario:** Graph contains versions where `sc:name` ≠ `schema:name`
**Example:**
```turtle
:software1 schema:name "openssl" ;
sc:name "OpenSSL_Project" .
```
**Result:**
- Ground Truth: Matches `:software1`
- LLM: Matches `:software1` (if "OpenSSL_Project" doesn't match filter, would miss it)
### Edge Case 2: Type-Incomplete Version Nodes
**Scenario:** Version node exists but lacks explicit `rdf:type sc:SoftwareVersion`
**Example:**
```turtle
:version123 secchain:versionName "1.1.1x" ;
secchain:vulnerableTo :cve_xyz .
# Missing: :version123 a sc:SoftwareVersion .
```
**Result:**
- Ground Truth: EXCLUDES (requires type assertion)
- LLM: INCLUDES (no type constraint)
**Impact:** LLM may include malformed or incomplete version records
### Edge Case 3: Identifier Property Namespace
**Scenario:** Vulnerabilities use inconsistent identifier properties
**Example:**
```turtle
:vuln1 schema:identifier "CVE-2023-3817" .
:vuln2 sc:identifier "CVE-2023-9999" .
```
**Result:**
- Ground Truth: Only filters by `schema:identifier`
- LLM: Only filters by `sc:identifier`
- If CVE-2023-3817 is stored under different namespace, filtering fails
### Edge Case 4: Multiple Software Entities
**Scenario:** Graph contains both "openssl" and "OpenSSL" as distinct entities
**Example:**
```turtle
:openssl_lib schema:name "openssl" .
:openssl_rust schema:name "OpenSSL" . # Rust crate wrapper
```
**Result:**
- Ground Truth: Matches only `:openssl_lib`
- LLM: Matches BOTH `:openssl_lib` and `:openssl_rust`
**Impact:** LLM may include versions from unrelated packages
### Edge Case 5: Version String Encoding
**Scenario:** Version strings contain special characters stored with different encodings
**Example:**
```turtle
:v1 secchain:versionName "1.1.1k\",override=True" .
# URI: https://...?q=openssl+1.1.1k%22%2Coverride%3DTrue
```
**Result:**
- Ground Truth: Returns URI with URL encoding
- LLM: Returns literal string with decoded characters
- **Empirically confirmed:** 2 false negatives due to this issue
### Edge Case 6: Vulnerability Association Level
**Scenario:** Vulnerability linked at different granularity levels
**Example:**
```turtle
# Direct association (what both queries expect):
:version_1.1.1g secchain:vulnerableTo :cve_2023_3817 .
# Indirect association (through software):
:openssl secchain:vulnerableTo :cve_2023_3817 .
```
**Result:**
- Both queries only check direct version-to-vulnerability links
- Would both miss software-level vulnerability associations
---
## 4. Semantic Transformation Analysis
### What Would Make Them Equivalent?
To make the LLM query semantically equivalent, it would need:
1. **Add type constraint:**
```sparql
?sv a secchain:SoftwareVersion .
```
2. **Use consistent namespace for properties:**
```sparql
?software schema:name ?softwareName . # Not sc:name
?vuln schema:identifier "CVE-2023-3817" . # Not sc:identifier
```
3. **Match exact software name:**
```sparql
FILTER(?softwareName = "openssl") # Remove || ?softwareName = "OpenSSL"
```
4. **Return URI instead of literal (if needed):**
```sparql
SELECT DISTINCT ?sv # Instead of ?version from versionName
```
### Modified LLM Query (Equivalent Version)
```sparql
PREFIX schema: <http://schema.org/>
PREFIX sc: <https://w3id.org/secure-chain/>
SELECT DISTINCT ?sv
WHERE {
?software schema:name "openssl" .
?software sc:hasSoftwareVersion ?sv .
?sv a sc:SoftwareVersion .
FILTER NOT EXISTS {
?sv sc:vulnerableTo ?vuln .
?vuln schema:identifier "CVE-2023-3817" .
}
}
ORDER BY ?sv
```
---
## 5. Summary Table
| Aspect | LLM Query | Ground Truth | Impact |
|--------|-----------|--------------|--------|
| Software name namespace | `sc:name` | `schema:name` | HIGH - May match different entities |
| Name matching | OR filter (2 values) | Exact match | MEDIUM - Broader scope |
| Version retrieval | Via `versionName` property | Direct URI | HIGH - Different return types |
| Type constraint | ❌ Missing | ✅ Present | **CRITICAL** - False positives |
| Vulnerability namespace | `sc:identifier` | `schema:identifier` | HIGH - May miss vulnerabilities |
| Return value | Literal string | URI | MEDIUM - Encoding differences |
---
## 6. Conclusion
### Verdict: **NOT SEMANTICALLY EQUIVALENT**
**Certainty Level:** HIGH (empirically confirmed with actual results showing divergence)
**Key Reasons:**
1. **Critical:** Missing type constraint allows false positives (confirmed: 2 vulnerable versions included)
2. **High:** Namespace inconsistencies may cause different entity matching
3. **Medium:** Different return types cause string encoding mismatches
4. **Low:** Broader name matching may include unintended packages
**Empirical Evidence:**
- 5 false positives (2 confirmed vulnerable, 2 encoding differences, 1 unverified)
- 2 false negatives (both due to encoding differences)
- 98.52% F1 score indicates queries are **similar but not equivalent**
**Risk Assessment:**
- **False Positive Risk:** HIGH (empirically confirmed)
- **False Negative Risk:** LOW-MEDIUM (only encoding issues observed)
- **Production Use:** NOT RECOMMENDED without modifications
The queries will produce different results on the same dataset due to structural and constraint differences, particularly the missing type constraint in the LLM query.
