Validation vs. Verification

In software development, validation and verification are key elements to a successful project, but often the terms are mis-understood. Per the CMMI guidelines, validation is used to demonstrate that the product will meet it’s intended use, and verification demonstrates that the product matches the specified requirements. Validation ensures that you "built the right thing" and verification ensure that you "built it right".

Verification is largely the domain of the development and QA teams and includes activities such as peer reviews, inspection, and testing. Whereas work that the requirements team does is a big input to their work, the requirements team itself is generally not involved much in verification activities. As you might expect, though, the requirements team is very much involved in validation activities.

In my next post, I'll further explain validation and provide some best practices.

There isn’t just one way

On a recent international trip, our team of requirements engineers had some amusing adventures. I’m going to share the story simply because it’s fun, and then I’m going to actually tie it into requirements, so stick with me on this one!

Our team consisted of resources that we’ll call RE1, RE2, etc. Now, RE1, RE2, RE5, RE6, RE7 and RE8 depart location A. They stop over in location B on the way to the location C in foreign country and pick up RE 3 and RE4.

After arriving in foreign country location C the full team heads to a hotel by bus. RE1 leaves a coat on the bus to the hotel in location C. RE1 tells someone at the hotel of the problem, but unfortunately has to leave the city to take a train to location D, 4 hours away the next day. But....3 days later is he back in location C. And, lo and behold, so is his coat! What are the odds? In a kind country like Japan, very good apparently. But wait, it gets better...

The team headed back to location C for a relaxing weekend (where the coat was found!). However on Sunday, the team prepared to head to the customer’s site in location D. At the train station, RE1 puts luggage on a train in location C. RE1 gets off the train to buy a Coke. 2 minutes later, the train leaves. RE1 is not on the train. Good news - it's not the train RE1 is supposed to be on! Bad news – RE1’s luggage is on it and we have no idea where RE1’s luggage is traveling to. RE1 runs off in the foreign train station to find help. The rest of the team boards the correct train and departs for location E (which will then connect to location D). Except, RE2 volunteers to stay behind in location C to make sure RE1 has money and a train pass to get to location D.

Paths diverging:
Path A: RE7 leaves 3 hours early to go to location D to go skiing.
Path B: The team including RE3 through RE8 travel on train to mid-route city location E. However, 30 minutes in, their train stops due to 90mph winds at random location F. The team cannot get off the train in location F because none of them know the language to understand when the train is re-boarding for departure. 4 hours later, the train moves on again from location F to location E. The team is very happy to make it to location E, fearing they’d have to return to location C for the night.
Path C: RE2 determines RE1 is ok and decides to leave location C to catch the next train to location E. From location C, RE1 travels to a ski resort in location G to find luggage in a train station. RE1 takes taxi to the correct station in location G, to actually find the luggage. From location G, RE1 finds a train to return to an original-route mid-route city location F. RE1 boards train in location F to get back on the path to mid-route city location E en route to location D.

Paths converging:
The team of RE3 through RE8 gets off of the train in location E to board another train to location D. While in the bathroom RE6 hears a familiar voice outside. RE6 rushes outside and finds out that it is RE1!
RE1 somehow manages to meet up with the team of RE3 through RE8 in location E to catch the next train to location D. Amazingly, this time RE1’s luggage is along for the ride.

But where is the 1 helpful RE2 who stayed behind? A train behind we hoped, arriving at location E in time to catch the last train of the night to location D. Unfortunately, the trains in location C going to location E did not show up at the platform. RE1 waited as the hours went by. The trains were canceled due to the high winds the rest of the team experienced. RE2 was stuck in location C. RE2 could not leave the platform because again, RE2 couldn’t understand the announcements if the train did show up. RE2 waited for 4 hours in freezing temperatures before giving up and going to the hotel in location C.

RE7 just managed to get caught in the winds minutes away from location E and although RE7 left 3 hours earlier, arrived too late to go skiing.

By noon on Monday, we were all back in location D, including RE2.

So what is the moral of the story? There are so many you can pick from, but RE1 was a fan of this one: “Do not assume you are not taking the fast path that something is necessarily wrong!” So actually, just because it seems like someone is taking you way out of the way to get from point C to point D, reality may strike and you will learn that going through point G is just as fast as the direct path.

Now applying this to requirements…back in location D on that same Monday, I was running some requirements sessions to gather use cases and business rules on a particular topic. My group was in a groove by midday, we had an approach on how we would get from blank slate to requirements on the topics. However, a co-worker wanders out of his session and into ours and starts to ask some questions in the middle of our conversation. He asks what the goal of our current topic is and starts to make suggestions on how to approach it.

Often you should respond with “Great! Someone new to identify things we might have missed?” I am always welcoming of new ideas, especially around an approach. But in this case, I was truly surprised by how disruptive it was to our flow.

Frankly, his ideas were great. Arguably they may have been more efficient even. However, the reality is that our approach was working for our SMEs and to change it was going to slow us down.

The real moral of the story – just because you take two different paths does not mean you will end up in two different places. There is a lot of requirements experience out there and we should all welcome the opportunity to look at different “paths” to get to the right requirements and just apply the best practices that work for the given situation.

Software Engineering Process Improvement

Seilevel spoke at the Dallas ASEE (Association for Software Engineering Excellence) Software Engineering Process Improvement Workshop on Saturday February 24, 2007! Tony Chen, co-founder and Principal Analyst at Seilevel presented “Beyond the System Shall – A Journey from Good to Great Requirements”.

Mike Konrad, chairman of the CMMI Configuration Control Board, presented “CMMI Adoption Successes and Challenges”, Jill Brooks, Raytheon NCS, presented “Implementing SPC for Peer Reviews”, and Eric Wulfekammer, 7-Eleven, presented “Stage Gates Process Control Points”.

Workshop attendees posed some interesting questions about CMMI and requirements and I would like to share them with you:

-As we climb levels of Process Improvement, how do you know when you have the right amount of process?
-What kind of predictions are there for the worldwide deployment of TSP (Team Software Process) and PSP (Personal Software Process) and People CMM (People Capability Maturity Model)?
-Where and how do you get started with PI (Process Improvement)? People CMM?

Some requirements questions were:

-What was found as the root cause in analysis of project issues?
-Was communication the issue?

You can answer these questions, discuss your answers to these questions and pose new questions here on the Seilevel Message Board.

Requirements Team Building

How do you build a team that is spread throughout various locations? First we have to define the word team. There are several definitions of the word team. This is the one that most closely applies to the requirements world. "a number of persons associated in some joint action": a team of advisers. This definition suggests that geography is not important to the team concept. However, experience suggests that a team that is segmented in different locations will have less cohesiveness and create communication challenges among other things. How do we gather, define and disseminate requirements on a project that will be developed overseas for instance. Well, the common ways in which to handle this issue is to use email, conference calls and travel.
Let us assume that there are no language barriers, only distance barriers in this scenario. How do we effectively deliver requirements and coordinate with the developers who are not local? Teams thrive on having a common theme. This could be a common color, fight song or method of operation. We could use face paints and mascots, but that may not always be practical in a business environment. We could create a song for each project that all of the members have to sing together at the start of each meeting, but not everyone will want to participate. Uniforms are often a good way to create the feeling of a team.
I would say that the method of creating the "team look and feel" can change based on the project and the team members. It should be something that the majority of the members agree is fun. This can be decided during the kickoff meeting.
The point is not simply to bond, but to create a sense of responsibility to the group and the overall success of the project. Product Managers and Analysts are often individuals and introverts. Creating a cohesive team will help to tame the alpha personalities and include the omega personalities that exists in every group. We want to create an "Army of One", to use military marketing campaign phrase.
Try to use this concept on a new project and compare the results with a similar project that did not tackle this issue from the beginning. I’m guessing that there will be mixed results, but that is still better than the status quo.

What, Not How – Beyond Requirements

As Requirements Engineers we all know the importance of requirements that specify what instead of how. We’ve read books about it, we’ve studied the difference. We’ve argued about it on the message board. However, we’ve always done this with respect to the requirements specifications themselves. What about the process we use for getting to good requirements?

Many Requirements Engineers have very different ways of eliciting requirements. Some ask questions a certain way. Some hold meetings a certain way. Some divide and conquer. Some put everyone in the room together and then let them “fight it out”. There are thousands of ways we can get to the root of a requirement – the “what” of the requirement. However, there is a tendency when training requirements expertise to want to explain, step by step, exactly HOW to get to good requirements.

I’d like to ask us all to take a dose of our own medicine. Perhaps we have missed the mark. Perhaps when training someone to write good requirements, we should not attempt to tell them how to gather good requirements but instead we tell them what good requirements look like. We see this in almost any requirements training book. Good requirements are clear, concise, and testable. Then we see lots of examples of bad requirements with lengthy explanations of why they are not clear, why they are not concise, or why they are not testable. The assumption we draw is “there is one correct way to gather requirements to ensure they are good at the end.” This is a bad assumption and the one I want to challenge today.

Rather than examine the process for how we get to good requirements and focus our efforts on tweaking our process, let’s study our results, the requirements themselves, and discuss what about our results we should adjust to make the requirements better. As we get a clearer and clearer picture of what we want the end result to be, we will naturally develop better and better methods for getting there. The most remarkable part is we may all have a different process. From a development team perspective or a business interest perspective, no one really cares HOW we get the requirements documented. Everyone is most concerned with what they say and whether the end product meets the needs of the business.

My advice? Stop trying to figure out how you elicit good requirements. Start looking at the requirements, determining if they are good, and discussing what would make them better. Eventually, as an individual, you may find a particular process that works well for you when trying to elicit requirements. However, your process may vary greatly from your co-workers process. As long as you both end up with good requirements, does it really matter?

Requirements are a Waste of Time

“Requirements are a waste of time.” I’ve had experience with people who have this attitude. Seilevel’s message board has threads in which people are asking “How do I convince others that requirements are worth doing?” I think a lot of us have come up against this.

Most product managers can site the chaos report and the strong tie between requirements deficiencies and failed projects. Being analysts, we can give them facts supporting the value of requirement: identify problems when it is cheapest to fix them, synch the team on a common vision, build the right thing the first time, and etc. Yet, many people persist “Requirements are a waste of time”. Why?

Usually, they’ve had a bad experience. Or, even worse, multiple bad experiences, such as:

• The requirements were poorly gathered; users weren’t heard and didn’t get what they asked for.
• The requirements were poorly documented; users were promised more than they received.
• There was a huge requirements effort, and nothing ever came of it.
• The requirements were perfect, but the implementation did not match the requirements.

But, we’ve given them our facts—why don’t they understand that “Requirements are good?” Because, in many instances, we are now dealing with emotion, in addition to logic. I’ve read a statistic that people remember almost 100% of their negative experiences but only about 20% of their positive experiences. I’m not sure if this is an absolute, but I do think it is generally true. (And, the parents I've talked to are sure it's true.)

As a result of the bad experience, the person may have been personally impacted. Maybe they feel they wasted a colossal amount of time. Maybe they were reprimanded by their boss. Maybe they were even fired because of a failed project. Maybe they wasted a lot of money, and they hate wasting money.

It’s important to realize that when we are interacting with such a person, we may be dealing with all of their internal, perhaps bio-chemical and subconscious reactions to their bad experience. How do we deal with this? Acknowledge it when possible. It’s amazing how identifying a problem can be a huge first step in overcoming it. Incremental success can also be great. Break the requirements effort down, show success early and often. Positive experiences will eventually trump negative ones.

In the end, when you succeed, celebrate to solidify the good feeling.

College Curriculum for Requirements Analysts

On Day 3 of RE’06, I attended a practice talk called “Requirements Engineering: An Industrial Perspective”. Brian Berenbach spoke to the success of engineering projects in the early 1900’s as compared to now. He spent much of the talk focused on the reason the projects are not as successful anymore, suggesting it is related to the resources we are hiring to do requirements.

The overall argument from Brian was that people graduating from school today are not well trained in engineering. His contention was that the first programmers were engineers and scientists by training. As children, those engineers played with model trains, boats, and electromechanical systems. Today’s children are not playing with such things to influence their mindset, driving them to study complex engineering.

Then he tore apart a current day Computer Science curriculum. He argued that Computer Scientists are not trained to build complex systems because the curriculum is obsolete. The course outline he used as demonstration reflected that CS grads do not take compiler and database classes now.

Finally, he referenced some thoughts by Phil Greenspun, to say that when students come out of these programs, they are not good at turning vague goals into concrete specs, lack experience with UI and testing, and are not dedicated to completing something of value. I just finished reading through the work Phil put together.

And in reading Phil’s points, some of his issues are that graduates of CS programs have not done much open-ended project work, haven’t done much group work and don’t have experience to make sense of large commercial tools and libraries.

I have to admit, I disagreed with many of Brian and Phil’s points. Now it is possible that this talk caused a reaction in me because I have a Computer Scientist background, and I just got defensive. But in truth, I do feel like my education was very valuable to what I do today in requirements.

Phil actually starts his argument with the point that US employers are shipping jobs overseas because they are dissatisfied with the US educated IT workforce. I immediately was skeptical given this premise. In my experience, the primary reason employers are moving to offshore resources is simply to save money, and it is barely (if at all) related to having better resources offshore.

Looking at the CS course curriculum, I was truly surprised by their points. The curriculum Brian showed during the talk was from a school I was not familiar with. I studied Computer Science at Purdue, where the curriculum did include compilers and databases, as well as a project to build an end-to-end compiler. I’m also skeptical that schools really aren’t teaching this way now! We did projects in teams throughout my time there, and my understanding is that such group work is becoming an even greater part of college courses. In addition, we were assigned open ended projects, such as one to work in a team, to design and build a cryptography-related application and to be able to explain the value of the application. Now, possibly these are only present in elective courses that not all students have to take, but then it’s on the employers’ backs to look for graduates that did.

Sure, there are things that the students will not learn in school - perhaps around interviewing users, or using industry tools, etc. But frankly, we can teach these things very quickly to employees if they have the right talents. Can everyone facilitate requirements meetings coming out of school? I will argue there is no way anyone out of school could do this without first being trained by the company they work for.

I fundamentally believe in the value of hiring smart people with a capability to do analysis work. If they can do this, then we can teach them the skills to do the actual job by pairing them up with experienced employees. We have had great success with hiring a strong group of product managers from a diversity of backgrounds including biochemistry, economics, English and psychology education, mathematics and engineering. Obviously most of these people as students did not have experience in interviewing users, yet they are fully capable now because they had the strong problem-solving foundation on which to build the requirements skill set.

When HOW the Requirement is Written Rather than What the Requirements Says Affects Whether the Requirement is in Scope

Have you ever had a project where you’ve thought, “If I had known the requirements were going to be used this way, I would have written them differently to make this process easier.” In a consulting environment this can happen a lot. Every customer is likely to have a slightly different way of consuming the requirements for a project. This situation reminds me that understanding how your documents will be consumed should drive how your documents are written. Let me give an example. The following is an incomplete list of requirements, with their development estimates, related to using an ATM:

Estimates for Functional Requirements (in hours):
10 hrs FR_01 System shall read user name, account number, and bank ID from ATM card.
5 hrs FR_02 System shall accept 4 digit number input from user.
3 hrs FR_03 System shall never display 4 digit number.
5 hrs FR_04 System shall not store 4 digit number.
20 hrs FR_05 System shall contact bank indicated by ATM card, provide account number and 4 digit number, and ask bank indicated whether a transaction may occur.
30 hrs FR_06 System shall support withdrawing funds from checking account.
0 hrs FR_07 System shall support withdrawing funds from savings account.
5 hrs FR_08 System shall support checking for balance.
5 hrs FR_09 System shall support depositing funds.

Now let’s say the development manager and product manager are asked to cut “all unnecessary requirements”. Look at requirement FR_04. Does this get cut? The product manager has asked that the system NOT store the 4 digit pin. The development team has indicated that NOT storing the 4 digit pin will require 5 hours of work. Does this mean that if the system DID store the PIN it would require no work?

Let’s look at requirements FR_06 and FR_07. FR_06 has a value of 30 hours and FR_07 has a value of 0 hrs. This could mean that once the work to withdraw from checking is complete, the system can withdraw from savings for no additional work or it could mean the development manager has estimated the tiny extra bit of additional work to withdraw from savings as part of the withdraw from checking requirement. What if the product manager knows that withdraw from savings is an optional feature and could be cut? With the above estimates she gains nothing for cutting this feature.

How might a product manager write these requirements differently if she knows how they will be consumed by development? The Product Manager needs to write the requirements in a way that allows for easier in/out scope discussions based on the estimates returned. An example:

FR_01 System shall accept and validate the PIN.
Must Have:
a) System shall read the account number and bank ID from the ATM card.
b) System shall accept a 4 digit number indicated by the user.
c) System shall not display the number.

Optional:
a) System shall not store the number

FR_02 System shall support feature to withdraw funds from user account.
Must Have:
a) System shall support contacting user’s bank.
b) System shall indicate amount to be withdrawn.
c) System shall indicate the checking account.

Optional:
a) System shall indicate the savings account.

The above approach allows the developer to offer an estimate for the must have requirements and an “additional” estimate for the optional requirements. This will make scope discussions easier for both the developer and the Product Manager. It will also help communicate the business needs to the developer without having to review the exact priority of each requirement.

The Importance of Requirements

The title of this post isn’t what it appears. This isn’t going to be a post about how managing requirements is a non-negotiable element of successful software development projects. Although, I am of the mind that there is still plenty of writing, talking and doing to be done around that topic.

No, this post is about a different aspect of the importance of requirements. It is about the importance of each single line-item requirement that is captured during a project. The background on this is that the question was raised (here) about whether requirements should be deleted from the repository if they are deemed to be out of scope or otherwise not relevant to the project.

My response is “No.” And here’s why.

Software requirements efforts are rife with information. Key responsibilities of the Requirements Manager are to capture, document, categorize, organize, evaluate, disseminate, facilitate review and manage change for that information. All of that information. Once a requirement is expressed it is in someone’s mind as being “a part of” the project. Whether it comes from a vision or scope document that was part of the preliminary project formation steps, one-on-one end-user interviews, or even a comment some executive makes in an offhanded way in a meeting that might not even have been specifically about requirements, the requirement is out there and for successful projects must be managed.

Now this is not to say that a Requirements Manager should be responsible for being in on every single communication exchange that occurs around the project and capture all of the data. This isn’t reasonable. But it still doesn’t dismiss the fact that requirements come from unconventional and informal sources in addition to the planned sources and exercises. So anyway, I mentioned these examples of how requirements are expressed to illustrate that every requirement has a source. And my experience has shown that the source of that requirement is INVESTED in that requirement. I have heard “pride of ownership” applied to this dynamic, with the implication that this somehow inappropriately involves personal ego, and that may be the case, but I also think it involves people simply doing their jobs by representing the interests of the group to which they report or the tasks that they perform.

So given that I know my source is invested in their requirements. I want to be able to show them that same requirement at the end of the project and be able to tell them the story of what happened to it. If it made it into the final product, I want to be able to show them the final requirements artifact that was delivered to development and testing. And I want to be able to show them where in the final app it was realized and show them the cases that were applied to test its functionality.

If the requirement did not make it into the final app, I still want to be able to show that invested source when it’s status changed from “In” to “Out” and be able to provide them some context for the reasons. I want to be able to answer the question was this just an abandoned requirement or one that just didn’t make it into the current release but will be considered for a future release. I want to be able to tell the source what reasoning was used to take it out of the current release. I should have that reasoning documented with the requirement so I don’t have to remember it.

And to be able to report any of these things to my requirement source, I have to have that requirement and all of its attributes documented, not deleted.

Peeling the Onion

A common piece of advice given to teams starting out to document requirements is “Find the actors and use cases”. Sounds like reasonably good advice, but how do we find the actors? Well, actors are stakeholders, a subclass of the stakeholder class if you will. Not every stakeholder will have direct interaction with the product but a stakeholder by definition has an interest of some kind in the outcome of the project. Actors are stakeholders so one way to find the actors is to identify and examine the stakeholders.

In fact, the importance of identifying stakeholders goes well beyond just finding actors. Stakeholders not only have an interest in the outcome of the project, and thereby provide “project requirements”, but they can also impact the product by providing business rules. Whether business rules are just requirements or something fundamentally different and whether they should be documented together or separately is a topic for another blog. Either way, it is critically important to identify and document the business rules, and therefore we must identify and examine the stakeholders, even the non-actor stakeholders.

Here’s an example: In the health care industry it is often important for systems to support the adherence to Medicare reimbursement rules. In most cases Medicare will not interact directly with the product. If we only identify actors who interact directly with the product, we may initially miss some important requirements (or rules).

So how do we find and identify the stakeholders? One common and useful technique is the ‘Onion Diagram’. This diagram is produced by first drawing a circle in the center of the page to represent the product. As an aside, the line around the product may be indistinct at the start of the requirements process and one of the goals of requirements analysis is to solidify the product boundary. A second circle is drawn around the product circle to identify “the system”. The product operates within the system and most of the directly interacting stakeholders (actors) will be found within the system circle. An additional circle is drawn to represent “the containing system”. Most, but not all, of the non-actor stakeholders will be found within this circle. Anything outside of the containing system is “the wider environment”. Identifying the stakeholders in the wider environment, such as government agencies, is often critical to the success of a project.

I personally find it easier to think of the containing system circle as “the organization” representing the legal entity which owns the system and is probably funding the development of the product. Not all systems are owned by legal entities but for products being developed by commercial enterprises the organization is probably a good boundary line to draw. The goal of course is not to draw a precise representation of systems but to identify stakeholders. The value of this diagram is that it provides a framework for thinking about where the stakeholders reside and what their interests are in the outcome of the project.

Like most tools of this nature, it is not necessary to follow a set of exact rules to gain the benefits. The diagram could be drawn with zero or many systems circles to represent the fact that the product is the system, or that there are systems within systems. There is also nothing wrong with starting with the outermost circle and working your way in to the product circle. What we are really drawing is a sort of extended context diagram. For those of us (like me) who are more comfortable with squares than circles the diagram can be drawn as a nested set of boxes. Other products within the system, and systems within the organization, can be drawn more easily this way, as in a more traditional system context diagram.

The important thing is to identify the stakeholders at each ‘layer’ of the onion. Document their interest in the project and their interaction, if any, with the product. The subset of stakeholders with direct interaction with the product should contain all of the actors. Some of the stakeholder groups may be at a higher level of abstraction than what is needed for use cases so look for subclasses within the groups.

See the references below for more information and some helpful templates.


References:

Scenario Plus: www.scenarioplus.org.uk
Volere: www.volere.co.uk

Product Democracy

One of the biggest challenges that our clients have is that they find it difficult to develop systems that truly meet the users' needs. In the typical Fortune 500 corporation, "the business" consists of all the people who do the job of making money. This could be manufacturing, distribution or sales. IT is the group that develops systems that in some way are meant to help "the business" either to make more money or to save money. In this corporate world, "the business" is the customer. In most organizations, "the business" behaves as true customers. They express needs and IT has the responsibility to deliver systems that meets those needs. What we have found is that many times IT does a poor job of understanding the needs and "the business" refuses to accept the developed system. This manifests either as extreme user dissatisfaction with the system or even outright refusal to use the system. These are two critical problems that waste tremendous amounts of time and money. We all want to get the users more involved, but it is incredibly difficult to do so when you have a large distributed user base. You certainly can't interview everyone.

You may have heard of a field called Prediction Markets or Predictive Markets. A few years ago some researchers discovered that when people put real money on the outcome of events and enough people placed bets, the group was often times better at predicting outcomes than experts. You can find some background on Prediction Markets at Wikipedia. It wasn't long before Prediction Markets were used to aid in software development. Both Microsoft and Google use variants of prediction markets to predict launch dates and to prioritize features of products.

One company that has fully embraced direct communication with their user community is Salesforce.com. Salesforce has created what they call their "IdeaExchange." In the IdeaExchange, users can discuss features, recommend features and then vote on the features that other people have recommended. In addition, when new releases come out, users can vote and make comments about their satisfaction with the releases. This becomes real time feedback to the developer population to create a much tighter community and to create releases that are much more in tune with what the users actually want. These systems have been around for a few years now, but we haven't yet seen significant adoption in our clients.

While these tools and methods are no substitute for strong Product Management, they can provide a significant aid to Product Managers in prioritizing features and getting a measure of how well their systems met the needs of the business.