“The Systems Science Institute at IBM has reported that the cost to fix an error after product release was four to five times as much as one uncovered during design, and up to 100 times more than one identified in the maintenance phase”- iSixSigma magazine
Series Context
- Part 1 - Introduction and Overview
- Part 2 - Business Impact Metrics
- Part 3 - Culture Metrics
- Part 4 - Quality Metrics (You are here)
- Part 5 - Speed Metrics
- Part 6 - Conclusion and Implementation
Introduction
Given the central nature of the “build quality in” mindset to Lean and Agile, my early drafts of the metrics dashboard devoted 3 full categories to quality:
- Technical Health
- Quality
- Deployment Health
I was fortunate enough to both get some great review feedback from +Dean Leffingwell on the drafts and spend some time at a whiteboard brainstorming with him. He dwelt on the fact that I had “too many metrics and too many quadrants” :) As we brainstormed, we came to two conclusions. Firstly, the 3 concepts listed above were just different perspectives on quality – and secondly, we could separate my individual metrics into “the basics everyone should have” and “the advanced things people should have but might take time to incorporate”. The result is the set of basic and advanced definitions below.
One might question the incorporation of highly technical metrics in an executive dashboard, however there are three very good reasons to do so:
- If our technical practices are not improving, no amount of process improvement will deliver sustainable change.
- If our teams are taking lots of shortcuts to deliver value fast, there is no sustainability to the results being achieved and we will wind up “doing fragile not agile”.
- If the executives don’t care, the teams are unlikely to.
On the “traditional quality measurement” front, my core focus is “are we finding defects early or late”? Thus, I look to both evaluate the timing of our validation activities and the level of quality issues escaping the early life-cycle. For deployment health, all inspiration was sourced from DevOps materials and as we re-structured the overall model it became apparent that many of these measures really belonged in the “Speed” quadrant – all that remained in the quality quadrant was clarity on production incidents.
Basic Definitions
Basic Metrics Rationale
Unit Test Coverage %
As I regularly inform participants in the training room, "if you do not aggressively pursue automated testing your agile implementation will fail!" It is impossible to sustainably employ an iterative and incremental approach to software development without it.
Static analysis tools will not tell you the quality of the unit tests or the meaningfulness of the coverage, but simply having coverage will give the developers confidence to refactor - the key to increasing maintainability. It should also increase the ratio of first fix resolution, giving confidence that defects can be resolved fast and minor enhancements made without causing unintended side effects.
Further, even if automated functional tests are still on the to-do list, testers who can read unit tests will be able to more effectively adopt risk-based manual testing and thus reduce manual test effort.
Mean Time Between Green Builds (mins)
Note that many ARTs will implement multiple CI cycles – local ones executing on branches and a central master cycle on the mainline. Whilst branch-level CI cycles might be of interest at the team level, the only one we are interested in at the ART level is the master on the mainline.
Red CI builds are of course an indicator of poor developer quality practices (failure to locally validate code prior to check-in), and most believe the full CI cycle should occur in under 10 minutes to provide an adequate level of timely feedback to the developers, but failure on either of these fronts will naturally extend the time between green builds, so they need not be discretely measured on the dashboard.
Mean Time to Recover from Red build (mins)
Two things will cause this metric to trend in the wrong direction. One is lack of the Andon mindset (its someone else's fault, or even worse its always red, just ignore it). The second is failure to regularly commit, resulting in complex change-sets and difficult debugging. The second is easily identified through the mean time between Green Builds, so the metric enables measurement of the establishment of the Andon mindset among developers.
Late Phase Defects #
The identification and resolution of defects during the execution of a story is evidence of good team quality practices, and should be excluded from any strategic treatment of defect trends. However, defects identified in functionality associated with a story after its acceptance or in late-phase (integration, performance, security, UAT etc) testing are indicators of a failure to "build quality in".
Whilst many teams do not formally log defects identified during story development, where this is done there will be a need for classification in the defect management system to separate late phase defects for reporting purposes.
Validation Capacity %
Great agile means a story is accepted once it is in production. Good agile means it is accepted once it is ready for production. For most enterprises in the early years of their agile adoption, this seems like a fairy-tale - the DevOps definition of "Unicorns" such as Amazon and Netflix resonates strongly!
The reality is for some time there will be testing and packaging activities which get batched up and executed late in development. Typical examples include:
- User Acceptance Testing - of course, the Product Owner as the embedded customer is meant to do this in good agile but for many they are neither sufficiently knowledgeable nor sufficiently empowered.
- Integration Testing - in theory redundant if the team is practicing good full-stack continuous integration. But for all too many, environment constraints prohibit this and lead to extensive use of stubs until late phase.
- Performance Testing - for many organisations, the performance test environments are congested, hard to book, and take days if not weeks to configure for a performance test run.
- Penetration Testing - a highly specialised job with many organisations possessing a handful of skilled penetration testers spread across thousands of developers.
- Release Documentation
- Mandated Enterprise Level Integration and Deployment preparation cycles for all changes impacting strategic technology assets.
Given that the backlog "represents the collection of all the things a team needs to do" , all of these activities should appear in backlogs, estimated and prioritized to occur in the appropriate iterations. It is a simple matter to introduce a categorization to the backlog management tool to flag these items as hardening activities.
Average Severity 1 and 2 Incidents per Deploy
High severity incidents associated with deployments are a critical quality indicator. Measurement is generally fairly trivial with the appropriate flagging in incident management systems. However, some debate may exist as to whether an incident is associated with a deployment or simply the exposition of a preexisting condition. An organisation will need to agree on clear classification standards in order to produce meaningful measures.
Advanced Definitions
Advanced Metrics Rationale
Duplication %
Duplicate code is bad code. Its simple. One line of duplicated business logic is a time-bomb waiting to explode. If this number is trending down, its an indicator developers are starting to refactor, the use of error-prone copy/paste techniques is falling and the maintainability of the source code is going up. Its potentially debatable whether one measures duplicate blocks or duplicate lines, but given the amount of logic possible to embed in a single line of code I prefer the straight up measurement of duplicated lines.
Average Cyclomatic Complexity
Cyclomatic complexity is used to measure the complexity of a program by analyzing the number of linearly independent paths through a program's code. More complexity leads to more difficulty in maintaining or extending functionality and greater reliance on documentation to understand intent. It can be measured at multiple levels, however from a dash-boarding perspective my interest is in function or method level complexity.
Average Branch Age at Merge (days)
This metric may take a little more work to capture, but it is well worth the effort. The modern ideal is of course not to branch at all (branching by abstraction), however the technical sophistication required by developers to achieve this takes some time to achieve.
Code living in a branch is code that has not been integrated, and thus code that carries risk. The longer the code lives in a branch, the more effort it takes to merge it back into the mainline and the greater the chance that the merge process will create high levels of late-phase defects.
![]() |
Whiteboard spotted at Pivotal Labs by @testobsessed |
Fault Feedback Ratio (FFR) %
When it comes to defects, we are interested in not just when we find them but how we respond to them. In his book "Quality Software Management vol 2: First-Order Measurement, Gerry Weinberg introduced me to the concept (along with many other fascinating quality metrics). Our goal is to determine what happens when we address a defect. Do we resolve it completely? Do we introduce other new defects in resolving the first one? A rising FFR value can indicate poor communication between testers and developers, hacked-in fixes, and deterioration in the maintainability of the application among other things. According to +Johanna Rothman in this article (), a value of <= 10% is a good sign.
Measuring it should be trivial with appropriate classifications of defect sources and resolution verification activities in the defect management system.
Average Open Defects #
When it comes to open defects, one needs to make a number of local decisions. Firstly, what severity are we interested in? Restricting it to high severity defects can hide all kinds of quality risk, but at the same time many low severity defects tend to be more matters of interpretation and often represent minor enhancement requests masquerading as defects.
Further, we need to determine whether we are interested in the open count at the end of the PI or the average throughout the PI. A Lean focus on building quality in leads me to be more interested in our every-day quality position rather than what we've cleaned up in our end-of-PI rush.
Conclusion
More than for any other quadrant, I wrestled to find a set of quality metrics small enough not to be overwhelming yet comprehensive enough to provide meaningful insight. At the team level, I would expect significantly more static code analysis metrics (such as “Code Smells”, “Comment Density” and “Afferent Coupling” ) to be hugely valuable. Kelley Horton of Net Objectives suggested a Defect Density measure based on “# of production defects per 100 story points released”, and “% capacity allocated to technical debt reduction”. For further inspiration, I can recommend nothing so much as the “Quality Software Management” series by +Gerald Weinberg.
“You should name a variable with the same care with which you name a first-born child” – Robert C. Martin, Clean Code
Great Article. Thank you for sharing! Really an awesome post for every one.
ReplyDeleteProject Centers in Chennai
JavaScript Training in Chennai
Final Year Project Domains for IT
JavaScript Training in Chennai
The blog you shared is very good. I expect more information from you like this blog. Thankyou.
ReplyDeletePython Training in bangalore
Python Course in Bangalore
Angularjs course Bangalore
Angularjs Training in Bangalore
Web Designing Course in bangalore
Web Development courses in bangalore
Salesforce Course in Bangalore
salesforce training in bangalore
Big Data Training in Bangalore
Hadoop Training in Bangalore
Mobile app development company in mumbai
ReplyDeleteGreat post and informative blog.it was awesome to read, thanks for sharing this great content to my vision.
ReplyDeleteGood discussion.
Android Training in Tambaram
Android Training in Anna Nagar
Android Training in Chennai
Android Training in Velachery
Android Training in OMR
Android Training in Porur
Android Training in T Nagar
Android App Development Course in Chennai
python training in bangalore | python online training
ReplyDeleteaws training in bangalore | aws online training
artificial intelligence training in bangalore | artificial intelligence online training
machine learning training in bangalore | machine learning online training
blockchain training in bangalore | blockchain online training
uipath training in bangalore | uipath online training
The data that you provided in the blog is informative and effective.I am happy to visit and read useful articles here. I hope you continue to do the sharing through the post to the reader. Read more about
ReplyDeleteselenium training in chennai
selenium training in chennai
selenium online training in chennai
selenium training in bangalore
selenium training in hyderabad
selenium training in coimbatore
selenium online training
I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon
ReplyDeleteSuch a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.Java training in Chennai
Java Online training in Chennai
Java Course in Chennai
Best JAVA Training Institutes in Chennai
Java training in Bangalore
Java training in Hyderabad
Java Training in Coimbatore
Java Training
Java Online Training
I am really very happy to find this particular site. I just wanted to say thank you for this huge read!!
ReplyDeleteangular js training in chennai
angular training in chennai
angular js online training in chennai
angular js training in bangalore
angular js training in hyderabad
angular js training in coimbatore
angular js training
angular js online training
I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you.Your post is just outstanding! thanks for such a post,its really going great and great work.You have provided great knowledge
ReplyDeleteAzure Training in Chennai
Azure Training in Bangalore
Azure Training in Hyderabad
Azure Training in Pune
Azure Training | microsoft azure certification | Azure Online Training Course
Azure Online Training
Great site and a great topic as well I really get amazed to read this.This is incredible,I feel really happy to have seen your webpage.It was very interesting and meaningful.I gained many unknown information, the way you have clearly explained is really fantastic.keep posting such useful information.
ReplyDeleteFull Stack Training in Chennai | Certification | Online Training Course
Full Stack Training in Bangalore | Certification | Online Training Course
Full Stack Training in Hyderabad | Certification | Online Training Course
Full Stack Developer Training in Chennai | Mean Stack Developer Training in Chennai
Full Stack Training
Full Stack Online Training
Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here.
ReplyDeleteDevOps Training in Chennai
DevOps Online Training in Chennai
DevOps Training in Bangalore
DevOps Training in Hyderabad
DevOps Training in Coimbatore
DevOps Training
DevOps Online Training
The data that you provided in the blog is informative and effective.I am happy to visit and read useful articles here. I hope you continue to do the sharing .
ReplyDeleteacte reviews
acte velachery reviews
acte tambaram reviews
acte anna nagar reviews
acte porur reviews
acte omr reviews
acte chennai reviews
acte student reviews
i am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
ReplyDeleteIELTS Coaching in chennai
German Classes in Chennai
GRE Coaching Classes in Chennai
TOEFL Coaching in Chennai
spoken english classes in chennai | Communication training
Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
ReplyDeleteAWS Course in Bangalore
AWS Course in Hyderabad
AWS Course in Coimbatore
AWS Course
AWS Certification Course
AWS Certification Training
AWS Online Training
AWS Training
ReplyDeleteI was really impressed to see this blog, it was very interesting and it is very useful for all.
list to string python
types of data structure in python
polymorphism in object oriented programming
how to use numpy in python
python interview questions and answers for freshers
types of data structure in python
This Blog is very useful and informative.
ReplyDeletedata science training in noida
Mua vé máy bay Aivivu, tham khảo
ReplyDeletevé máy bay đi Mỹ bao nhiêu tiền
chuyến bay từ mỹ về việt nam 2021
vé máy bay từ nhật về việt nam
giá vé máy bay từ Vancouver về việt nam
Thanks for sharing such a nice info.I hope you will share more information like this. please keep on sharing!
ReplyDeletePython Training In Bangalore
Artificial Intelligence Training In Bangalore
Data Science Training In Bangalore
Machine Learning Training In Bangalore
AWS Training In Bangalore
IoT Training In Bangalore
Adobe Experience Manager (AEM) Training In Bangalore
"Very Nice Blog!!!
ReplyDeletePlease have a look about "
data science courses aurangabad
Thank a lot. You have done excellent job. I enjoyed your blog . Nice efforts
ReplyDeleteData Science Certification in Hyderabad
I am sure it will help many people. Keep up the good work. It's very compelling and I enjoyed browsing the entire blog.
ReplyDeleteBusiness Analytics Course in Bangalore
This comment has been removed by the author.
ReplyDeleteExcellent Blog! I would like to thank you for the efforts you have made in writing this post. Gained lots of knowledge.
ReplyDeleteData Analytics Course
What an incredible message this is. Truly one of the best posts I have ever seen in my life. Wow, keep it up.
ReplyDeleteAI Courses in Bangalore
I wanted to leave a little comment to support you and wish you the best of luck. We wish you the best of luck in all of your blogging endeavors.
ReplyDeleteArtificial Intelligence Training in Bangalore
The Extraordinary blog went amazed by the content that they have developed in a very descriptive manner. This type of content surely ensures the participants explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.
ReplyDeleteMachine Learning Course in Bangalore
I need to thank you for this very good read and i have bookmarked to check out new things from your post. Thank you very much for sharing such a useful article and will definitely saved and revisit your site.
ReplyDeleteData Science Course
Thanks Your post is so cool and this is an extraordinary moving article and If it's not too much trouble share more like that.
ReplyDeleteDigital Marketing Course in Hyderabad
Really impressed! Everything is a very open and very clear clarification of the issues. It contains true facts. Your website is very valuable. Thanks for sharing.
ReplyDeleteDigital Marketing Training in Bangalore
I wanted to leave a little comment to support you and wish you the best of luck. We wish you the best of luck in all of your blogging endeavors.
ReplyDeleteData Science Training in Bangalore
ReplyDeleteWhat an incredible message this is. Truly one of the best posts I have ever seen in my life. Wow, keep it up.
AI Courses in Bangalore
It is a very helpful and very informative blog. I really learned a lot from it thanks for sharing.
ReplyDeleteData Analytics Course
I am really enjoying reading your well written articles. I am looking forward to reading new articles. Keep up the good work.
ReplyDeleteData Science Courses in Bangalore
Nice post. This is a great article and am pretty much pleased with your good work. Very helpful information. Thank you.
ReplyDeleteBest Data Science Courses
MPPSC Coaching in IndoreGet Free MPPSC Notes and MPPSC Study Material for preparation of MPPSC Exam.>
ReplyDeleteI really enjoyed reading this post and keep up the good work and let me know when you can post more articles or where I can find out more on the topic.
ReplyDeleteData Science Online Course
Cloud Computing has been the stage for emerging technologies and an increase in career opportunities for both fresher and professional.
ReplyDeleteCloud Computing Course Fees in Bangalore
I am always searching online for articles that can help me and you made some good points in Features also. Keep working, great job
ReplyDeleteData Science Training
ReplyDeleteVery informative blog! There is so much information here that can help thank you for sharing.
Data Science Syllabus
I like to view your web site which is very useful and excellent resource and truly adored reading your posting. Thank you!
ReplyDeleteData Science Course in Gurgaon
Well done for this excellent article. and really enjoyed reading this article today it might be one of the best articles I have read so far and please keep this work of the same quality.
ReplyDeleteData Analytics Course in Noida
Really impressed! Information shared was very helpful Your website is very valuable. Thanks for sharing..
ReplyDeleteBusiness Analytics Course in Bangalore
I have read your article, it is very informative and useful to me, I admire the valuable information you offer in your articles. Thanks for posting it ...
ReplyDeleteBusiness Analytics Course in Patna
Thanks for sharing such nice info. I hope you will share more information like this. please keep on sharing!
ReplyDeletePython Training In Bangalore | Python Online Training
Artificial Intelligence Training In Bangalore | Artificial Intelligence Online Training
Data Science Training In Bangalore | Data Science Online Training
Machine Learning Training In Bangalore | Machine Learning Online Training
AWS Training In Bangalore | AWS Online Training
IoT Training In Bangalore | IoT Online Training
Blockchain Training in Bangalore| BlockchainOnline Training
Driver Toolkit License Key Generator Download i'm altogether enjoyed for this blog. Its an informative difficulty. It maintain me absolutely plenty to resolve a few troubles. Its possibility are correspondingly outstanding and in upheaval fashion therefore speedy.
ReplyDeleteWifi Password Hacker i was surfing net and thankfully came throughout this website and found intensely exciting stuff here. Its in endeavor of truth amusing to settlement. I enjoyed loads. thank you for sharing this super aspire.
ReplyDelete360DigiTMG delivers the best Data Analytics course with placements. Top industry trainers, LMS access, world-class curriculum, mock interviews, and guidance sessions for resume preparation.Data Analytics Course in Faridabad
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteMake your loved ones' holiday even better with these merry Christmas greetings. “Wishing you and those that you love dearly a very happy .Christmas Wishes 2022
ReplyDelete