 |
The Great Offshore CMM Hoax
By
Richard Perrin PMP
|
CMM Side Bar
Level 1 – the Initial Level. This is where most businesses start. Their projects are typified by intense heroic effort, consistently running in fire-drill mode, working project teams to burnout and frequently running over budget by working 12+ hour days to meet deadlines.
Level 2 – The Repeatable Level – all project processes - Requirements Management, Project Planning, Project Tracking and Oversight, Software Quality Assurance, Software Configuration Management and Vendor Management are brought under project management control. The software managers for a project track software costs, schedules, and functionality; problems in meeting commitments are identified when they arise. The project's process is under the effective control of a project management system, following realistic plans based on the performance of previous projects.
Level 3 – The Defined Level. All Level 3 projects can be summarized as standard and consistent because both software engineering and management activities are stable and repeatable. This process capability is based on a common, organization-wide understanding of the activities, roles, and responsibilities in a defined software process. Studies by the SEI have shown that any company that has reached a CMM Level 3 can typically show a 400-800% ROI from implementing CMM.
Level 4- The Managed Level. Organizations functioning at this level can be summarized as predictable because the process is measured and operates within measurable limits. This level of process capability allows an organization to predict trends in process and product quality within the quantitative bounds of these limits.
Level 5 – The Optimized Level. Level 5 organizations analyze defects to determine their causes. Software processes are evaluated to prevent known types of defects from recurring, and lessons learned are disseminated to other projects. In other words, defects are frequently found and addressed before your customer knows there was a problem at all. A CMM Level 5 company operates with the goal of producing defect free software, the first time, every time. By the time they have achieved CMM Level 5 the ROI results keep getting better.
|
|
The offshore flight of IT jobs to India and other Third World countries has got Stateside IT workers in a panic. With another million IT jobs predicted to move offshore by 2007, displaced US IT workers are starting to push back, write their congressmen and other elected officials, quite correctly pointing out that if Corporate America guts the middle class, who will be left to purchase the pricey goods and services being turned out by Fortune 1000 Companies? However this observation is falling on deaf ears in favor of short term gain at the expense of long term security. After all, not only do the companies in India do the job for 1/5 of the cost in the US but they are claiming that they do it at a level of quality unimaginable for most US companies. Utilizing the ratings established by the Software Engineering Institute’s five level assessment model – the Capability Maturity Model or CMM (see “CMM Side Bar”) - many IT shops in India are claiming they are ‘certified’ at CMM Level 4 or Level 5, the highest rating possible. This is the hype, what’s the reality?
In looking at the CMM claims made by offshore companies my research has uncovered the following:
- Many offshore companies claim to be CMM certified. Impossible. Here is what the SEI says about “CMM certification”, (reprinted from http://seir.sei.cmu.edu/pml/):
- The terms “SEI certified” and “CMM® certification” are simply incorrect since there is no such thing.
- The SEI does not certify organizations
- The SEI only licenses and authorizes lead appraisers to conduct appraisals
- The SEI nor any other organization is a "certifying authority" of the results from an appraisal
- The SEI does not confirm the accuracy of the maturity levels reported in published listings and has no intention of doing so
- The main intended goal and purpose of the models and appraisal methods developed by the SEI is for self improvement. The outcome, which is entirely dependent on the organization that follows these practices, is to raise the level of quality of the products developed with a better ability to predict the time and budget needed to develop the product. The goal focuses less on a perceived business advantage and more towards the ability to reliably develop products in a repeatable fashion with continual improvement versus doing the same in a chaotic state.
- Maintaining a certain maturity status is a continuous process. Therefore once a certain level is reached, appraisals are still necessary to know if the maturity is being maintained over time.
Stateside businesses that have implemented offshore ‘solutions’ have found that members of offshore firms have cheated on certification ‘tests’ or pushed back project risks on the American teams to deal with the fallout. I spoke to several QA resources from a large US Insurance company that discovered a group of Indian programmers from a well known offshore CMM Level 4 company cheating on a written Java programming test. “The test results were filled out by 75 people identically, right down to the commas, periods, hyphens and question marks. It turns out they had taken the test home for the weekend and performed a ‘group answer session’. When we questioned one of the programmers about this his response was, “I cheated, so what? Everyone does’.” Another well known CMM 5 I/T company in India refused to honor specific terms of its contract with a privately held Fortune 300 American Engineering firm. As one resource told me, “These guys refused to be on the hook for delivering anything”. When the American team told the Indian company that they were in breach of the contract, the Indian company responded that they would be ‘changing the contract’. The offshore view of customer service is different – in the US if customer is dissatisfied with an outcome, the vendor will ask what they can do to fix the problem. With many offshore companies in India, their attitude is: “You gave us your specs and we built what you asked for. If it is not what you wanted, that is your problem”. An associate working a major offshore deal for a Fortune 300 company has encountered this attitude on several occasions from CMM level 4 and level 5 companies in India. Other customer service issues include:
Communication. You’re not just dealing with a different culture you’re dealing with a culture that has a very different understanding of what your requirements actually mean. I spoke with an engineer at a US Fortune 30 Company that stated when they first tried the offshore approach the results they experienced were unpredictable and varied – all this from an advertised CMM level 5 company in India. He said, “In short we asked for a banana and got an apple. Then we asked for a banana again and instead we got a strawberry. We literally had to open up an offshore office in India and hire the programmers directly into our organization before we achieved any control over the result. They could not, or would not understand our requirements. You’d think that after drawing a picture somebody would get it…” Quality. You’d think that a CMM level 4 or 5 company would offer clear ideas on how to improve a client’s software process by significantly decreasing software defects and increasing productivity for software development projects (See “CMM Side Bar”). Here’s the reality as one associate observed at a major US pharmaceutical company at which she is a Project Manager: “The offshore CMM 5 development resources just stated – ‘Just tell us what to do and we’ll do it’. You know, that’s not why we hired them. They’re the CMM 5 Company – don’t they have some idea on what needs improvement? That’s why we hired them in the first place. If we have to tell them what to do, we could just as easily do the work ourselves…” Security. Offshore Security is for all intensive purposes, nonexistent. Think about it. What articles have you read about India’s locked-down, hack-proof IT security infrastructure? (Or Lithuania’s or the Ukraine’s or China’s?) Think your data is going to be safe? What would your customers think if their IRA account numbers, social security numbers or credit card numbers were being processed offshore? In India where the average per capita income (according to the World Bank) is $470 per year, $18K a year buys you a house with servants… Offer any offshore I/T resource $10 K and you can get practically anything you want – I’ve been told this by stateside Indian I/T resources more than once who laugh whenever I broach the subject. Anything, including your precious data is for sale.
So let’s recap briefly. India, the center of the new I/T boom, has produced more CMM level 5 assessed companies than the rest of the world combined. Of the published number of CMM 5 assessed companies (http://seir.sei.cmu.edu/pml/index.asp), as of 4-17-2003, 75 of 116 CMM Level 5 companies are from India. If you buy that, there is a bridge in Brooklyn I’d like to sell you. Quite a bizarre paradox when one considers that the CMM processes and concepts that were developed and funded at Carnegie-Mellon University in the US and funded 100% by the US DOD are now largely being imported from a third world country…
I would be very interested in hearing from any company that can show me un-doctored, un-spun, verifiable evidence that offshoring software development led to the creation of software for their business that was designed and built as they imagined it would be, bug free, requiring no rework, no retest and ready for prime time, the first time and better than anything that was ever built in the US, verified by a non-Indian IT stateside auditor.
I bet I could count them on less than one hand and I doubt you can prove me wrong.
It appears that as more U.S. businesses perceive I/T as a commodity (I call this the “I/T Beer Goggles” effect) offshoring to a CMM 4/5 company is just an added bonus. However, if your company understands that you actually do get what you pay for, you might want to make sure that you’re getting the advertised CMM/CMMI high quality for that low offshore price, because if you are not, your savings will quickly evaporate because of:
A vastly different work ethic Miscommunications Misunderstood requirements Potential multibillion dollar class actions lawsuits from millions of customers whose supposedly ‘secure’ SSN, IRA account and credit card data has been compromised offshore
In other words, your projects may not be delivered to the specifications anywhere close to what you are expecting (See items 1 through 6 above for a refresher). Outside of the horrifying security issues (which are the subject of a separate white paper) you have a right to ask any company claiming a CMM/CMMI assessment level of 2 or greater:
- What was your last published assessment level?
- When did that occur? If the assessment is over 2 years old, you have the right to insist that they get reassessed for that level. After two years the assessment is out of date according to the SEI.
- You have a right to ask for and receive a copy of the company’s assessment documenting the strong areas and the areas still needing improvement.
- You have a right to know:
- Who performed the lead assessment?
- Who was on the assessment team?
- Ask to directly question any of the assessment team about the assessment.
- You have a right to question the vendor on specifics of their improvement processes regarding their customer interactions:
- What were the productivity improvements?
- By what percentage did the defect level drop?
- How much money has this saved you or how much have your earnings increased?
- Can your customers verify this? Ask for a customer list and do some of your own checking.
- Ask the vendor what specific improvements have been made internally to their own product or processes as a result of their CMM/CMMI activities. You can demand specifics: metrics and written verification.
Any vendor that refuses to comply with these requests should be immediately suspect as a “CMM assessed” organization.
And in case you think this is all fluff, here’s the clincher: I recently talked to an account executive from a consulting firm that maintains an offshore CMM level 3 I/T shop in Lithuania who confided that no customer of theirs has ever asked any of the above questions regarding the firm’s CMM assessment level. Do yourself a favor and get a clue.
“Caveat emptor” continues to be the rule in evaluating any company that claims to be CMM/CMMI assessed at level 2 and beyond. Don’t just “trust the number” - prove the number first before you send your services and your data offshore, or you will find you have bought yourself a long-term headache for some short-term relief.
=============================================================================
Having trouble with your CMM implementation or wondering if the process is working for you? EvolutionTen experts can help you evaluate your offshore or onshore vendor’s claims of CMM assessment and verify your vendor’s actual result. Give us a call and ask about our “CMM Capability” assessment. 708-848-8195.
© 2003 Richard Perrin. All rights reserved.
|
