AI/ML 
Software Development 
Reactive QA finds bugs. Predictive QA stops them from happening. VM.PL builds custom ML systems that read your bug tracker, Git history, and test execution logs to predict which modules will fail before the next build – then rank your test suite accordingly. Drops into Azure Pipelines as a YAML gate. Leaves your existing test infrastructure untouched. First system in production in 6-10 weeks.
Here is the cost math. A bug caught during development runs about $100 to fix. The same bug in production: $1,500 to $100,000, depending on severity and domain (industry research citing IBM internal data from the 1980s, replicated by CISQ and Capers Jones across thousands of projects). The CISQ “Cost of Poor Software Quality in the US: A 2022 Report” puts the annual tab for US software defects at $2.41 trillion. Predictive QA changes where on that curve you catch the problem. The model does not replace QA engineers – it tells them where to look.
Reactive QA vs. predictive QA: what actually changes in production
Full regression on every commit takes 8-16 hours. AI-driven predictive test selection cuts those cycles by 60-80% while keeping defect detection above 95% (Google Engineering Productivity Research). One documented case: a regression suite that took 8 hours now runs in 40 minutes after AI-driven test selection was introduced – the difference between same-day and next-day feedback on every sprint (Element5 COO, Ranger case study, 2024). On a pipeline with 4-8 commits per day, that compounds across every engineer.
The market numbers match the engineering pressure. Global software testing hit $49.4 billion in 2025 (Grand View Research 2026). AI-powered testing specifically is growing from $9.32 billion in 2025 to $11.99 billion in 2026 at a 26.88% CAGR (Mordor Intelligence). The growth is not enthusiasm – it is structural pain. Flaky tests went from affecting 10% of teams in 2022 to 26% by 2025, a 160% jump in three years, as CI pipeline complexity grew 23% over the same period (Bitrise Mobile Insights, 10M+ builds). Google puts its own flakiness rate at about 16% of all tests; for a 50-engineer organization, that translates to $180K-$270K per year in lost engineering time before you count compute and deployment delays (Katalon / Bitrise 2025).
The 2025 World Quality Report (Capgemini / OpenText / Sogeti, 17th edition) found that 89% of organizations are piloting or deploying GenAI-augmented QA – but only 15% have reached enterprise-scale deployment. The gap has a clear cause: integration complexity (64% of respondents cite it), data privacy risks (67%), and a skills shortage (50%). Predictive QA built on ML classification against historical defect data does not have the hallucination problem that blocks GenAI adoption. The model outputs a binary defect-probability score – not generated text that might be wrong. The question it answers is concrete: which modules are most likely to fail in the next build? That question has an answer in your existing data.
| Criterion | Reactive QA (traditional) | Predictive QA with AI (VM.PL) |
| Goal | Find bugs after they occur | Predict defect hotspots before the next build |
| Test prioritization | All tests weighted equally, or manually ranked | ML ranks tests by defect probability – highest-risk first after every commit |
| Feedback speed | Full suite: 8-16 hours per cycle | Prioritized subset: 45-90 minutes on the same codebase |
| QA team focus | Covers all modules equally | Focused on high-risk areas; low-risk modules tested on schedule |
| Defect detection | Reactive – after the code change, sometimes after release | Proactive – risk score in the PR before merge |
| Flaky test handling | Treated as a test infrastructure problem | ML flags modules where test reliability is degrading – treated as a defect signal, not background noise |
| Technical debt visibility | Invisible until something breaks | Quantified: cyclomatic complexity + code churn = a risk score that climbs across releases |
| Shift-left ROI | Production defect: 15-100× cost of a dev-phase fix (CISQ / Capers Jones / NIST) | Mature shift-left programs cut production defects 60-90%, total cost of quality 40-60% (Total Shift Left, 200+ enterprise engagements) |
58% of enterprises waste over 30% of testing resources on redundant or low-priority tests; delayed feedback loops cost mid-sized companies $2.3M per year in rework (Capgemini World Quality Report 2022). The ML prioritization signal attacks that waste directly. Unlike self-healing automation or generative test generation, predictive QA does not require replacing your test framework. It layers on top of what you have.
How the ML model predicts software defects: methods and features
Three data sources, all of which you already have: your bug tracker (Jira / Azure DevOps), Git history, and test execution logs. The model calculates a defect probability score per code module after every commit, visible in the PR dashboard before the code merges.
XGBoost consistently outperforms simpler classifiers – logistic regression, naïve Bayes, SVM – on software defect prediction across heterogeneous feature sets (IET Software 2025; Scientific Reports / Nature 2025). CNN-LSTM hybrids reach 88-92% accuracy on complex temporal defect patterns (IJACSA 2024; Wiley IET Software 2025). Which method fits depends on how much historical data you have.
| ML method | What it predicts | Data requirement | Published accuracy |
| Classification (XGBoost) | Defect probability per module after each commit – driven by change frequency, historical defect rate, dependency depth, developer count | Bug tracker + Git history, 6+ months; ~500+ labeled defect records | AUC-ROC 0.84-0.95; F1 0.82-0.90 (Scientific Reports / Nature 2025; IEEE ICICCS-2023) |
| Time-series ML (LSTM / GRU) | When do defects emerge after specific change types? Release-cycle risk and freeze-period planning; bug fix time forecasting | Time-stamped test results, 12+ months; global cross-project models outperform per-project on sparse data (Frontiers in Big Data 2025) | CNN-LSTM hybrid: 88-92% accuracy (IJACSA 2024); LSTM + GRU: competitive on weekly bug-resolution forecasting (Frontiers in Big Data 2025) |
| Clustering (K-Means / DBSCAN) | Recurring failure families across releases – structurally similar defects that share a root cause | Defect descriptions + code diff data; works from 3+ releases | Cuts duplicate defect investigation effort by 30-50% (VM.PL project data) |
| Anomaly detection (Isolation Forest) | Defect patterns outside the training distribution – novel regressions the model has not seen before, including flakiness pattern shifts | Stable baseline test-run data from a clean release | Flags distribution shifts that precede novel failure modes; calibrated FP rate set per domain |
The five features that drive defect risk – all pulled automatically:
- Cyclomatic complexity (McCabe). The strongest single predictor on NASA datasets (Scientific Reports / Nature 2025). Pulled from SonarQube-compatible static analysis. AI-augmented models using complexity features cut false positive rates by ~15% versus complexity-only baselines (ResearchGate 2024). The correlation has held across studies from Basili et al. (1996) through current XGBoost-SHAP work (IEEE 2024).
- Code churn. The top predictor on AEEEM and industrial embedded software datasets (Scientific Reports / Nature 2025). High churn + low test coverage is the strongest combined signal for defect-prone modules. Pulled from the Azure DevOps Git API: commit count, lines changed, and number of developers touching the module in the last n releases.
- Dependency coupling (fan-in / fan-out). CK metrics – coupling between objects, depth of inheritance, response for a class – correlate significantly with defect count in object-oriented codebases (Basili et al., replicated in cross-company studies). Modules with many callers and many outbound dependencies fail disproportionately when changed. The dependency graph is built automatically via static code analysis.
- Test coverage gap. Low unit test coverage raises the risk score at equal change frequency, because changes there go unverified. Worth noting: a test suite with 90% line coverage and a 30% mutation score is significantly weaker than one with 70% line coverage and a 75% mutation score (Autonoma 2026). Coverage data comes from Azure Pipelines coverage reports, normalized automatically against Vitest 4 output.
- Developer concurrency. “Number of developers touching the module” ranked in the top 10 features across NASA, AEEEM, ReLink, and SoftLab datasets (Scientific Reports / Nature 2025). Modules that multiple engineers modify at the same time fail more often – merge conflicts and coordination gaps are measurable in the Git metadata.
Feature selection matters as much as algorithm choice. A carefully selected feature subset improves classification accuracy by 26% over unfiltered full feature sets, tested across 75 open-source projects with 1.2M+ lines of code (ResearchGate benchmarking study). In the Definition Phase, VM.PL uses permutation importance and SHAP-based feature selection to find the 5-10 features that carry the most signal for your specific codebase. ISTQB CT-AI v2.0 (released April 2026) explicitly lists data quality and feature transparency as requirements for AI-based testing systems – SHAP feature selection addresses both.
When your own data is thin: if you have fewer than 6 months of defect logs or fewer than 300 labeled records, VM.PL trains a cross-project model using similar codebases. Cross-project models typically reach AUC 0.75-0.81 (Feature-Oriented Defect Prediction study; Nature 2025). That is lower than within-project performance (0.85-0.95), but well above random and enough to drive meaningful test prioritization. The Discovery Phase determines which approach fits, and the Architecture Blueprint documents the expected accuracy range upfront.
Integrating predictive QA into your CI/CD pipeline
Your existing test suite stays intact. VM.PL adds the Predictive QA gate as a YAML step in Azure Pipelines. After every commit, the ML service scores code change context and returns a ranked test list. High-risk tests run immediately. Low-risk tests go to scheduled runs or pre-release sweeps. Nothing in your pipeline configuration changes except that one YAML step. The gate toggles off via a pipeline variable in under 60 seconds – no rebuild, no risk.
The ML service itself (Python 3.11+ / FastAPI) is fully decoupled from the CI/CD runtime. It talks to the pipeline orchestrator via REST API. Model updates, threshold changes, and rollbacks to a previous model version happen without touching pipeline YAML or application code.
Where Azure Pipelines’ built-in TIA stops and the ML layer starts. Azure Pipelines’ Test Impact Analysis selects tests by code coverage map: which tests exercise the changed files. That is static. The VM.PL ML gate adds the second dimension: given several tests that all cover a changed module, which ones have historically been most predictive of defects there? TIA narrows scope; the ML model sets priority within that scope. The combination – TIA for scope, ML for priority – is how VM.PL deploys by default.
Flaky tests add a layer that TIA alone cannot address. A module that was low-risk but starts producing inconsistent test results shows up in the ML risk score before it shows up as a failed build. The Probabilistic Flakiness Score (per-test stability over the long run, not just the last build) is tracked as a separate input signal and stored in pgvector/PostgreSQL 16.
| Component | Implementation | Role in the pipeline |
| Azure DevOps / Azure Pipelines | Predictive QA gate as YAML step – no changes to existing config. Gate calls the ML service REST API after every commit. | Triggers ML risk scoring; routes high-risk tests to immediate execution, low-risk to deferred runs |
| ML service (Python 3.11+ · FastAPI) | Azure Container Apps – scalable, no single point of failure. Endpoints: /predict, /health, /model-version. | Core prediction engine; integrable with GitHub Actions, Jenkins, or any CI/CD via REST API. Reference implementations for all three delivered in Delivery Phase. |
| Azure Application Insights | Monitors ML decisions, accuracy metrics, FP rate, AUC-ROC drift, flakiness shifts. | Full audit trail per ML decision; drift alerts; ISO 26262 / EU AI Act compliance log |
| SHAP dashboard (Next.js 16 · shadcn/ui) | Integrated into Azure DevOps PR view via Next.js API route. Per-module risk score + SHAP breakdown in PR comment. | Engineer sees why a module was flagged before approving the merge – not after a 12-hour build |
| Data pipeline | Azure Blob Storage · pgvector/PostgreSQL 16 · Drizzle ORM 0.45 · asyncpg | Historical test logs, bug tracker exports, Git metadata; vector similarity search for cross-release patterns |
| Testing | Vitest 4 (Next.js frontend unit + smoke) · Python pytest (ML service) · Azure Pipelines CI/CD | ML service unit tests, /predict endpoint smoke tests, accuracy regression suite run in the same pipeline |
| Rollback / disable | Pipeline variable toggle – no redeployment | Gate off in <60 seconds; existing QA runs unmodified |
| Flaky test tracking | Probabilistic Flakiness Score per test · pgvector/PostgreSQL 16 | Replaces binary pass/fail flakiness detection; modules with rising flakiness trend get elevated risk scores |
Engineers at a US SaaS client told us the most valuable change was not faster pipelines but a shift in developer behavior: they started thinking about defect risk at the commit stage, before the review. A PR annotated with “module risk: 0.87 – change frequency 12× above average” prompts different questions than a PR with no context. That conversation moves defect-prevention earlier in the process, which is the entire point.
Precision, recall, and threshold calibration
Every Predictive QA model produces two error types: False Positives (clean modules flagged as risky) and False Negatives (defective modules the model misses). Which error costs more is a business call – it depends on your domain, your test budget, and your release pressure. The threshold that controls this trade-off is set jointly with your QA team in the Definition Phase and lives as a pipeline variable. You can adjust it without redeploying anything.
Why accuracy is the wrong metric. Defective modules are typically fewer than 20% of a codebase. A model that predicts “no defect” for every single module still scores 80%+ accuracy. It also catches nothing. F1-score – the harmonic mean of Precision and Recall – is the correct optimization metric in defect prediction because it penalizes both error types and does not reward a model for ignoring the rare class. This is the metric VM.PL reports in the Architecture Blueprint and tracks live in Azure Application Insights.
| Metric | What it measures | VM.PL target | When it matters most |
| Precision | Share of flagged modules that actually contain defects. High Precision = fewer wasted test runs. | Standard software: ≥70%; high test-per-module cost: ≥80% | When running flagged tests is expensive – long execution time, cloud cost, complex environment setup |
| Recall | Share of defective modules the model catches. High Recall = fewer production escapes. | Standard software: ≥80%; safety-critical (ISO 26262 ASIL C/D, IEC 62443): ≥90% | When a production defect is expensive – regulated environments, revenue-critical systems, customer-facing APIs |
| F1-score | Harmonic mean of Precision and Recall. Primary optimization target when defective modules are <20% of the codebase – which is almost always the case. | Primary metric in the Architecture Blueprint; tracked continuously in Azure Application Insights | Every SDP context – accuracy is misleading on imbalanced datasets |
| AUC-ROC | Overall model quality independent of threshold. 1.0 = perfect; 0.5 = random. | Production floor: ≥0.80; XGBoost + SHAP on NASA/ISBSG: 0.91-0.95 (IEEE ICICCS-2023) | Model selection and re-training acceptance gate – no model goes live below this floor |
| False Positive rate | Fraction of clean modules flagged. Direct measure of wasted QA capacity. | Monitored continuously; drift above the agreed budget triggers re-training alert within 48h | When QA team capacity is the constraint – FP overload erodes trust in the whole system |
How threshold calibration works in practice. In the Definition Phase, VM.PL runs the trained model on held-out release data at multiple threshold values and plots the Precision-Recall curve. The QA lead and engineering lead pick the operating point based on three inputs: test execution cost per module, historical cost of a production defect in your domain, and release time pressure. For ASIL C/D automotive modules, Recall ≥90% is non-negotiable – a False Negative is an escaped safety defect. For a low-stakes internal tool where QA capacity is tight, Precision ≥80% saves budget. Same model, different threshold, different engineering contract.
SHAP explainability: turning a risk score into a QA decision
A risk score without an explanation is not useful. QA engineers need to know why a module is high-risk – not just that it is – to direct the test effort, brief the developer, and defend the prioritization call to a release manager. VM.PL ships SHAP (SHapley Additive exPlanations) as a standard deliverable in every Predictive QA build, not an optional add-on.
ISTQB CT-AI v2.0 (released April 21, 2026) – the current industry standard for AI-based testing – explicitly requires transparency, explainability, and human oversight. It lists explainability as a core quality characteristic alongside fairness, robustness, and safety. SHAP satisfies this requirement directly: every model decision produces a documented, feature-level rationale.
XGBoost + SHAP achieves AUC-ROC 0.91-0.95 and F1 0.82-0.90 on NASA and ISBSG datasets (IEEE ICICCS-2023; PLOS ONE 2024). The underlying math – Shapley values from cooperative game theory, computing each feature’s average marginal contribution across all possible feature orderings – is model-agnostic and well-understood. VM.PL delivers three levels:
- Global feature importance. Across all predictions: which features drive defect risk most in your codebase? Typically change frequency (last 7 days), dependency depth, days since last test, historical defect rate, developer count. Documented in the Architecture Blueprint and updated after every re-training run. Useful for engineering leadership making refactoring investment decisions.
- Per-module local explanation. For every flagged module: “Score 0.87 – change frequency last 7 days: 12× above average; dependency depth: 8 (critical); historical defect rate: 34%.” QA engineers see what to test and why before they approve the merge, not after a failure.
- Audit trail. Every ML decision is logged with its SHAP breakdown in Azure Application Insights – fully traceable for ISO 26262 audits, EU AI Act Article 13 transparency obligations, and internal reviews. No undocumented model decision reaches production.
The workflow difference SHAP makes. When SHAP shows dependency depth as the dominant driver on a module, the QA engineer targets integration testing – not unit testing. When developer count spikes on a module that four engineers are actively modifying, the QA lead flags it for a merge freeze. When change frequency dominates a recently refactored module, the engineer runs regression against the changed interface. Without SHAP, those calls happen on intuition. With it, they happen from data.
Drift detection and keeping the model current
A model trained on last year’s defect patterns does not know about this year’s new framework, new team, or new architecture. Three mechanisms prevent model decay.
- Population Stability Index (PSI) monitoring. PSI – the standard drift metric in production ML, originating in financial credit modeling – compares the distribution of model inputs (change frequency, dependency depth, defect rate, developer count) against the training baseline. Under 0.10: stable, no action. Between 0.10 and 0.20: investigate. Above 0.20: significant shift, re-train (Fiddler AI; Towards Data Science; AI Code Invest – consistent across sources). VM.PL runs PSI as a daily Azure Application Insights job with configurable alert thresholds and rolling window.
- Re-training triggers – time-based and event-based. Time-based: automatic re-training every n releases (typically 3-6 or quarterly). Event-based: PSI > 0.20 triggers re-training within 48 hours. New training data is the most recent completed releases. The model learns continuously from current patterns rather than drifting on stale data.
- Model versioning and rollback. Every re-training produces a new version with its own accuracy report (AUC-ROC, F1, Precision, Recall on held-out data). Deployment runs through Azure Pipelines: if the new version performs worse than its predecessor on the validation set, the previous version restores automatically within minutes. No model enters production without clearing the agreed accuracy gate. Version history is maintained for audit – every production decision links to a specific model version and its accuracy report.
Minimum re-training data. Typically the last 3-6 releases or 6 months of defect logs, whichever gives more labeled samples. Teams on 2-week sprints may hit threshold in 3 months; enterprise codebases on quarterly releases may need 12+. When within-project data is thin, a cross-project model (AUC 0.75-0.81) covers the gap until enough local data accumulates. The Discovery Phase defines this threshold for your specific cadence.
Flakiness drift. A module that was low-risk but starts generating inconsistent test results has changed in a way the binary pass/fail view cannot describe. VM.PL tracks the Probabilistic Flakiness Score – per-test stability over the long run rather than a single build – as a separate model input. When flakiness drift and code churn both rise on the same module, the risk score reflects both signals. This catches instability that would otherwise look like a test infrastructure problem.
Predictive QA across the testing pyramid
The ML model does not target one test level. It prioritizes across all of them – with different signals at each layer of the pyramid.
| Test level | How predictive QA applies | Data basis | Shift-left effect | Documented result |
| Unit tests | ML ranks unit test execution by defect probability for the changed function. Only tests for modified and dependent functions run per commit. | Git diff + historical unit test failure rate per function | Risk score in the PR before merge – developer knows which functions need more tests before the reviewer sees the code | 60-80% CI runtime reduction while maintaining >95% defect detection (Google Engineering Productivity Research) |
| Integration tests | ML identifies which interface changes have historically caused integration failures. API version diff and service dependency graph add context for microservice boundaries. | API change log + integration failure history; service dependency graph from Azure DevOps | Integration failures caught before the build reaches the integration environment – hours earlier in the feedback loop | XGBoost AUC 0.84 on code churn + API change features for integration failure prediction (ResearchGate 2024) |
| Regression tests | Test Impact Analysis extended with ML defect history: ML selects tests affected by the code change AND correlated with the module’s defect patterns. Full regression only pre-release. Azure Pipelines TIA handles scope; the ML model sets priority within that scope. | Coverage map (Vitest 4 / Azure Pipelines) + failure history + defect correlation data | 60-80% regression cycle time reduction; 8 hours → 40 minutes (Element5 COO / Ranger 2024); full pre-release coverage maintained | Google Engineering Productivity Research: selective execution holds >95% defect detection rate |
| Mutation testing | ML identifies modules with weak test suites (low mutation score) and targets mutation testing tools – Pitest for Java, Stryker for JS/TS – on those modules only. No more blanket runs. | Mutation score history across releases | Mutation testing budget concentrated where it matters; stable, well-covered modules excluded from expensive blanket runs | TDD+M: 64.9% vs 37.5% mutation coverage versus standard TDD; mutation score is a stronger quality signal than line coverage alone (Software Quality Journal 2020; Autonoma 2026) |
Technical debt, made visible. A module with rising cyclomatic complexity, increasing churn, and declining test coverage gets a climbing risk score across releases. That trajectory – available in the Azure Application Insights dashboard – gives engineering leadership a quantified argument for refactoring prioritization. Not “this code feels fragile” but “this module’s defect risk score has climbed 340% over the last six releases and is now the highest in the codebase.” One is an opinion. The other funds a sprint.
Regulatory standards in predictive QA builds
In regulated industries, the ML-based testing tool is itself subject to the standard. It needs to be qualified, documented, and supervised. VM.PL maps applicable standards in the Definition Phase and builds the compliance architecture before training starts – not after.
ISO 26262 tool qualification is tiered. For ASIL A/B, a TÜV SÜD certification of the testing tool is typically enough evidence. For ASIL C/D, the tool must be qualified on the target system hardware – which means additional validation effort against the physical platform (LDRA, Parasoft tool qualification guidance). VM.PL designs the Predictive QA system to stay in the “T1” classification where possible: the ML model supports test prioritization and evidence generation, but a certified QA engineer retains final approval. The ML output does not directly gate a safety-critical production decision.
| Standard | Requirement for predictive QA | VM.PL implementation |
| ISTQB CT-AI v2.0 (April 21, 2026) | Current version explicitly requires transparency, explainability, and human oversight. Explainability is a core quality characteristic alongside fairness, robustness, and safety. | SHAP explainability as standard · engineer override in dashboard · override feedback feeds re-training · non-determinism documented per CT-AI v2.0 |
| ISO 29119 (Software Testing) | Test planning, design, and results must be reproducible and auditable. | Automated test reports · full test history in Azure Application Insights · audit trail per ML decision (decision + SHAP + model version) |
| EU AI Act (High Risk, effective Aug 2025) | Transparent (Art. 13), human oversight (Art. 14), quality management (Art. 17), full documentation. | SHAP report (Art. 13) · Human-in-the-Loop release gate (Art. 14) · model version log · risk classification in Definition Phase · technical documentation per Art. 53 |
| ISO 26262 / ASIL (Automotive) | Full traceability from requirement to test result; human approval for safety-critical functions; tool qualification required (TÜV SÜD for ASIL A/B; on-target validation for ASIL C/D per Part 8 Clause 11). | Requirement-to-test traceability matrix · ASIL-aligned process · no autonomous ML output without human sign-off · tool qualification package for ASIL C/D (add-on to Delivery Phase) |
| IEC 62443-4-1 (Industrial Cybersecurity) | Risk-prioritized security testing; verification documented; SDL requirements. | Risk-score-driven security test prioritization · coverage documentation in Azure Application Insights · SDL alignment in Architecture Blueprint |
| DO-178C / DAL (Avionics) | Objectives-based assurance; MC/DC coverage for DAL A; tool qualification per DO-330. | Predictive QA handles test selection and evidence generation; MC/DC verification stays with qualified tools (DO-330 qualification kit in Architecture Blueprint); TÜV SÜD or Bureau Veritas cert available as add-on |
| GDPR / CCPA (test data) | Test environment personal data must be pseudonymized or synthetic; no ML training on personal data without legal basis. | Synthetic test data generation · pseudonymization before ML ingestion · on-premise option for regulated corpora · DPA with VM.PL as standard contract item |
Predictive QA in safety-critical contexts – automotive ADAS testing, medical device software, avionics – may be classified as high-risk under EU AI Act Annex III. VM.PL classifies the risk in the Definition Phase and designs the system to satisfy Articles 13, 14, and 17 from the architecture stage. For standard business software (SaaS, enterprise applications), the system typically falls outside the high-risk category and needs only general transparency obligations.
The 4D methodology for predictive QA projects
The Discovery Phase validates the data before anything gets built. We have seen the same failure mode enough times to describe it precisely: a team starts model training without verifying that defect logs are consistently labeled, that test execution history is complete, or that Git history has module-level granularity. The model trains on noise. The risk scores correlate with nothing. The project gets shelved. The Discovery Phase is exactly what it sounds like – it finds this before any budget is committed.
| Phase | Activities | Deliverable | Duration | Go/No-Go gate |
| Discovery | Defect log analysis (Jira / Azure DevOps) · data quality and volume assessment · CI/CD audit (Azure Pipelines / GitHub Actions / Jenkins) · Git history depth and granularity check · test coverage data availability · use case scoping · regulatory standards mapping · cross-project model feasibility if local data is sparse | Concept Document (Go / No-Go) | 1 week | Go: 6+ months defect logs, ~500+ labeled records, module-level Git history, CI/CD API accessible, at least one coverage report format available. Sparse data: cross-project model feasibility assessed. |
| Definition | ML architecture selection (XGBoost / LSTM / ensemble) · feature engineering plan · SHAP feature selection (permutation importance) · Azure Pipelines YAML gate · GitHub Actions / Jenkins reference implementations if needed · Precision-Recall curve + threshold calibration · SHAP dashboard spec (Next.js 16 + shadcn/ui) · PSI drift detection design (thresholds + rolling window) · re-training trigger design · Probabilistic Flakiness Score design · regulatory compliance architecture | Architecture Blueprint | 1 week | Approved by QA lead and engineering lead; regulatory mapping signed off; Precision-Recall operating point agreed contractually |
| Delivery | Feature extraction pipeline (Git API + Azure DevOps API + Vitest 4 coverage + bug tracker export) · model training on historical defect data · SHAP feature selection · threshold calibration on held-out release data · Azure Pipelines YAML gate integration · SHAP dashboard build (Next.js 16 + shadcn/ui + Azure DevOps PR) · Probabilistic Flakiness Score integration · formal accuracy acceptance gate | Production Predictive QA system | 4-8 weeks | AUC-ROC ≥0.80 (or agreed cross-project floor), F1 ≥ agreed target, FP rate within QA budget – all on held-out release data. No go-live without passing this gate. |
| Direction | Model monitoring (Azure Application Insights · daily PSI) · re-training triggers (time-based + PSI > 0.20 event-based) · model version control and rollback · threshold recalibration as codebase evolves · flakiness drift monitoring · infrastructure via Azure Bicep IaC (Container Apps, PostgreSQL Flexible Server, Key Vault) | Ongoing roadmap | Ongoing | Monthly accuracy report; PSI drift alert triggers re-training review within 48h; AUC-ROC below floor triggers immediate escalation |
Timeline and cost. First production system: 6-10 weeks from kickoff. Mid-market SaaS team (20-50 engineers, Azure DevOps): $30K-80K build, $5K-15K/yr to operate. Regulated enterprise (ASIL C/D or DO-178C compliance documentation): $80K-200K. Payback: typically 3-6 months. A 50-engineer team eliminating $400K+ in flaky-test waste and reducing production defects by 25-40% generally recoups the build cost within 4-8 weeks of go-live.
Why US and UK engineering teams work with VM.PL on predictive QA
VM.PL Software House (founded 2004, Wrocław, Poland) has shipped 475+ projects for US, UK, and Western European clients. 150+ engineers, NPS 71. Poland’s IT services market reached $19.5 billion in 2025 (PAIH ICT Sector Report), with over 400 universities and 22 ranked in QS World University Rankings 2025. English-first communication, 6-9 hours ahead of US East Coast, 1-4 hours of daily overlap.
Five differences from off-the-shelf QA tools:
- The model trains on your defect history, not an industry average. A generic pre-trained predictor applied to your codebase behaves like cross-project transfer learning: AUC-ROC typically 0.75-0.81. A within-project model trained on your own data: 0.85-0.95. VM.PL builds for your codebase, your team’s patterns, your domain.
- Experience in AI-Assisted Software Quality and Security: VM.PL used Claude Code to modernize an existing Building Management System (BMS). As part of the project, 87 security vulnerabilities were identified and resolved. AI supported the analysis of the existing system and preparations for its modernization, while experienced engineers remained responsible for critical architectural decisions and testing. The project reduced the total workload by 260 working hours. Reference: AI-Assisted BMS Modernization and Security Testing.
- SHAP explainability is included, not sold separately. Every risk score ships with a feature-level explanation in the PR dashboard. This satisfies ISTQB CT-AI v2.0 (April 2026) transparency requirements. QA engineers can challenge the score; that challenge goes back into re-training. A score nobody understands is a score everyone ignores.
- Azure-native but not Azure-locked. Azure DevOps, Azure Pipelines, Azure Application Insights, Azure Container Apps – the full stack for Microsoft environments. The ML service REST API is CI/CD-agnostic; GitHub Actions and Jenkins reference implementations ship in the Delivery Phase.
- Production systems, not pilots. Every engagement targets a live system with PSI drift monitoring, model versioning, a formal accuracy gate, and regulatory compliance architecture. The accuracy targets – AUC-ROC, F1, Precision, Recall, FP rate – are agreed contractually in the Definition Phase. Nothing goes live below the agreed floor.
- Full ownership, no licensing tail. Source code, model weights, feature extraction pipelines, SHAP dashboard, and Azure Pipelines configuration belong to you. No ongoing VM.PL licensing fees. You can hand the system to an internal ML team or another vendor without rebuilding.