Rilevera is an enterprise-grade security platform used by Fortune 100 security teams to write, test, document, and deploy security detections at scale. Its platform addresses a difficult detection engineering environment where 70% of detections remain unvalidated, 85% of alerts contribute to alert fatigue, and organizations face an average 45% coverage gap in defense evasion techniques.
Problem: Existing MITRE ATT&CK classification needed higher accuracy and efficiency
Rilevera’s existing two-stage classification system was already supporting production users, but the architecture needed stronger accuracy, more consistent classification, and lower inference overhead. Detection rules range from simple patterns to context-dependent and ambiguous scenarios. Some rules also map to multiple MITRE ATT&CK tactics or techniques. This creates classification challenges when the system relies on fixed prompt structures, limited examples, or confidence scores that do not clearly reflect the reliability of the prediction.
The system also needed stronger visibility into classification accuracy, latency, token consumption, and confidence-score distribution. Without a structured validation dataset and monitoring layer, improving prompt performance or identifying recurring failure patterns becomes harder. Rilevera also wanted to prepare for a future shift from LLM-only classification toward ML-based or hybrid classification. This required collecting labeled data from real classifications, user corrections, validation results, and edge cases without disrupting existing integrations.
Solution: AI-powered MITRE ATT&CK classification on AWS
GoML enhanced Rilevera’s existing two-stage MITRE ATT&CK classification system to improve classification accuracy, consistency, confidence scoring, and inference efficiency. The implementation aligns with GoML’s AI Matic platform, adding optimized few-shot prompts, context-window management, multi-pattern recognition, calibrated confidence scoring, edge-case handling, and structured evaluation while preserving compatibility with Rilevera’s existing integrations.
MITRE ATT&CK classification engine optimization
GoML improved Rilevera’s existing two-stage classification workflow to make MITRE ATT&CK classification more consistent across different detection-rule patterns. The work focused on prompt quality, context usage, baseline measurement, and multi-technique classification.
• Refined few-shot prompts with curated examples across simple, complex, and ambiguous detection rules.
• Optimized context windows to retain relevant rule information while reducing unnecessary token usage.
• Added multi-pattern recognition for rules associated with multiple MITRE ATT&CK tactics or techniques.
• Established baseline metrics for accuracy, latency, token consumption, confidence distribution, and classification failure modes.
MITRE ATT&CK confidence scoring and human review
GoML strengthened confidence scoring so Rilevera could better assess the reliability of each MITRE ATT&CK classification. The workflow also separates high-confidence classifications from cases that require analyst validation.
• Introduced multi-factor confidence scoring using LLM response characteristics and pattern-matching strength.
• Calibrated confidence scores against human-validated examples and ground-truth classifications.
• Defined confidence thresholds for automatic classification and human review.
• Flagged ambiguous and multi-technique detection rules for manual validation.
• Captured user corrections as validated examples for future system refinement.
MITRE ATT&CK classification performance optimization
GoML optimized the two-stage classification architecture to reduce repeated processing and improve inference efficiency. The performance layer also gives Rilevera better visibility into latency, throughput, and token consumption.
• Added intelligent caching for frequently classified detection-rule patterns.
• Improved batch-processing workflows where multiple rules require classification.
• Profiled the classification pipeline to identify latency bottlenecks.
• Added response-time and token-usage monitoring.
• Included production-scale load testing to validate performance improvements.
MITRE ATT&CK data collection and ML preparation
GoML created a structured data pipeline so production MITRE ATT&CK classifications contribute to a future training dataset. The approach prepares Rilevera for a later transition from an LLM-only workflow to ML-based or hybrid classification.
• Captures classification inputs, outputs, confidence scores, user interactions, and corrections.
• Stores validation results and identified edge cases in an ML-ready schema.
• Tracks the MITRE ATT&CK framework version associated with each classification.
• Produces labeled examples that can support future model training and evaluation.
• Preserves ambiguous and multi-technique examples for later model improvement.
MITRE ATT&CK framework adaptability
GoML designed the classification workflow to remain manageable as the MITRE ATT&CK framework changes. Version tracking and update mechanisms help Rilevera maintain consistency between new classifications and historical records.
• Tracks MITRE ATT&CK framework versions across classification data.
• Detects framework updates and changes in tactics or techniques.
• Supports streamlined workflows for updating classification logic after framework revisions.
• Maintains backward compatibility with classifications created under earlier framework versions.
MITRE ATT&CK classification testing and monitoring
GoML structured testing around a broader validation dataset to measure how the enhanced system performs across different detection-rule types. Monitoring continues after deployment to track classification quality and operational performance.
• Tests the complete two-stage MITRE ATT&CK classification pipeline.
• Validates classifications across simple, moderate, complex, and ambiguous rules.
• Checks confidence-score correlation against validated classification accuracy.
• Verifies existing API contracts and integration compatibility.
• Tracks accuracy, inference latency, token usage, and system performance through monitoring dashboards.
MITRE ATT&CK classification architecture on AWS
GoML retained Amazon Bedrock as the intelligence layer while strengthening the surrounding architecture for classification, data capture, and monitoring. The AWS stack supports the existing Rilevera application while preparing the system for future ML-based classification.
• Amazon Bedrock and Claude 4 support MITRE ATT&CK tactic and technique classification.
• Amazon API Gateway supports access to classification services.
• AWS Lambda and Amazon EC2 support processing and compute workloads.
• Amazon S3 and PostgreSQL support classification data, validation records, and ML-ready datasets.
• Python supports backend classification logic, while React supports integration with Rilevera’s application layer.
AI Matic delivery
The Rilevera implementation maps most closely to GoML’s Agentic AI blueprint because the project enhances an existing two-stage agent system responsible for MITRE ATT&CK classification.
The delivery approach combines agent prompt optimization, evaluation, confidence scoring, caching, monitoring, human escalation, and data capture while preserving Rilevera’s existing API contracts.
The AWS stack includes Amazon Bedrock and Claude 4 for the classification intelligence layer, AWS Lambda or Amazon EC2 for compute, Amazon API Gateway for service access, Amazon S3 and PostgreSQL for data, and Python for application development.
AI Matic delivery metrics
- Average TTFV: 15 days
- Average person-days saved: 9 days
Impact
- 100% structured capture of MITRE ATT&CK classification data for future ML training.
- 100% backward compatibility targeted with existing Rilevera APIs and integrations.
- Up to 2 weeks planned to adapt the system to major MITRE ATT&CK framework updates.
- 3–6 months of historical detection data used for validation and ML preparation.
- 12-week delivery plan covering optimization, testing, UAT, and deployment.
- Two-stage classification system enhanced with prompt optimization, confidence scoring, caching, and monitoring.
About
Before Gen AI and after Gen AI
“With Rilevera’s MITRE ATT&CK classification system, we focused on making each classification more measurable, efficient, and useful for security teams while creating the labeled-data foundation required for the next phase of ML-based detection engineering.”
Prashanna Rao, Head of Engineering, GoML
Key takeaways for MITRE ATT&CK classification
Common challenges
- Detection rules often contain ambiguous or multi-technique behavior.
- Confidence scores need validation before security teams use them for prioritization.
- Large prompts increase token usage and inference latency.
- MITRE ATT&CK framework revisions create version-management requirements.
- Future ML classification depends on high-quality labeled examples collected during production use.
Practical guidance
- Establish accuracy, latency, token, and confidence baselines before changing the classifier.
- Use representative few-shot examples across different detection-rule patterns.
- Calibrate confidence scores against human-validated ground truth.
- Route uncertain classifications to human review.
- Capture corrections and edge cases in an ML-ready data structure.
- Track MITRE ATT&CK framework versions alongside every classification.
- Keep evaluation and monitoring in place as prompts and framework versions change.
Ready to improve MITRE ATT&CK classification
Partner with GoML to build accurate, measurable, and ML-ready MITRE ATT&CK classification systems on AWS with the AI Matic platform.
