Seilevel’s Requirements Management Tool Selection

Over the spring, we took on a new project to identify a best of breed requirements management tool to use when our customers are not using sufficient tools. Since the thing we do is requirements….well, you won’t find it too surprising to hear that we defined requirements for a requirements management tool!

We successfully applied our vendor selection process to this project. The product management team identified actors, use cases, data types, functional and non-functional requirements. We ultimately used about 150 requirements in our selection process. To identify a vendor list, we used our knowledge of tools, our customer’s tool experiences and the INCOSE Requirements Management Tools Survey. A co-worker had previously evaluated the INCOSE survey list and eliminated many of the vendors. We did a slightly more in depth review including RequisitePro, Contour, Doors, Caliber, CaseComplete, FeaturePlan, CaseComplete and FeaturePlan. We narrowed it further to Doors, Caliber and RequisitePro based on feature richness.

I’ll jump to the punch line and tell you we selected Borland’s Caliber RM as our tool of choice and are currently piloting it on our first project! Borland has some other related tools we hope to make use of - DefineIT to help visually model use cases and Together for creating UML diagrams.

So, let me explain a little about how we came to that decision. I certainly don’t want to indicate the other tools on the market don’t have a lot of capabilities. In our analysis of features, I scored each of the vendors’ capabilities against our requirements on a 3 point scale. I prioritized each requirement on a 3 point scale. And I took the product of priority and capability scores and summed those across the requirements. Actually, the total scores of all 3 tools were very comparable! What I found was that where one tool was strong, another might be weak, but they made up for it in another set of requirements. So a couple highlights based on my opinion in looking at them:

• Caliber is best out of the box for collaboration


• Doors is better at imports


• Caliber does not handle working on requirements while disconnected


• Tables and images were handled better in Caliber and Doors than in ReqPro


• I preferred the traceability mechanism in Caliber


• You can map screenshots to workflows or use cases in DefineIT (links to Caliber)


• Bulk edit is easiest to do in Caliber, much easier than Doors

So in the end, the functionality alone did not give me a clear winner, so this is where my non-functional requirements became critical to the decision. To put it simply:

• ReqPro was by far the worst


• Doors felt archaic


• Caliber was just an inviting look and feel and obvious on how to use it

ReqPro was eliminated quickly at this stage. Some of this is obviously personal preference. But for example, in Caliber, I instinctively would right click to perform some actions and the menu I expected was there. It sounds simple, but I did not have that experience in Doors and it frustrated me. And actually one of my favorite things, the Requirements Grid in Caliber is slick! It was easy to do just about anything I could think to do in the grid as far as viewing, filtering and mass updating requirements.

The next thing I looked at is extensibility of the remaining contenders. Doors has an extension language, DXL, that allows you to do quite a bit. But Caliber has an API that allows you to do even more. The Doors solution means you have to learn a scripting language, but once you do it’s pretty easy to use. However, you are limited to the commands of the language. The Caliber solution means you have to have someone who can write code, however you can do far more. So I expect extending Caliber will be more challenging up front, but you have more possibilities.

I also headed out to talk to some of our customers about their experiences with the tools. Because we are in consulting, we have to pay attention to what our customers are using, to a degree. I have found a lot of people use Doors, so I couldn’t entirely ignore that. However, just because it’s popular does not mean it’s the best tool out there. I will say though I heard fantastic reviews of Doors from one of our customers. They loved the tool and they loved the team from Telelogic. Another customer of ours talked to me about RequistePro. Let’s just say I could not get him to get me excited about the tool. He himself was interested in what I found in this search process. And, we have no customers that are using Caliber, so I couldn’t compare feedback here. I did remember some data from a former trusted co-worker though, who’s preference was definitely for Caliber a few years ago after her own analysis.

Working with the teams proved to be enjoyable all around. Doors was very responsive. As we have gotten into the final selection with Borland, they too have been incredibly helpful. In both cases, I was able to work with technical people who understood the tool. They were all very patient with me as I asked them a lot of questions around how to execute scenarios in their respective tools.

So in the end, as I mentioned, we ended up selecting Caliber as our recommended tool. I would say the heaviest weighing factor for me was the look and feel and ease of use of the tool.

Now I will say there are a lot of newcomers to the market in requirements tools (or old timers coming back) and we’ll be keeping our eye on those to see how they fit into the mix. It does look like a couple of them are changing a lot faster than the big named tools are changing, but they have a long way to go on some of the critical features.

And for now, we are really excited to pilot Caliber on our first project!

Are you listening?

Requirements elicitation is fundamentally about communication. We gather requirements from people who know what is needed or can help us figure it out. There are other ways to find requirements but elicitation is the most effective way in most cases and is almost certainly a significant part of the requirements gathering effort in almost every case. I can not think of an example of a project I worked on that did not involve interaction and communication with people at some point to gather and validate the requirements. So it is probably a good idea to think about ways to improve our communication skills.

Listening may be the communication skill that is easiest to improve and for requirements analysts it may also be the most important communication skill. For some of us it is an area where we may actually be sabotaging our efforts to gather requirements. Improvements in our listening skills can result in big payoffs in the form of better requirements, fewer missed requirements, and overall a more efficient and effective requirements elicitation process.

For any number of reasons, such as you’re too busy, you’re distracted, or you’re not really interested, it is easy to fall into a pattern of hearing without really listening. Hearing is just the physical process that happens when sound waves reach your ears. Listening is much more and requires the active participation of the listener in the communication process.

Active listening happens when you are genuinely interested in what is being said and you give the speaker feedback that lets them know you are receiving and understanding their message. The feedback should include non-verbal clues, eye contact, and nodding, etc. as well as verbal confirmation, restating or paraphrasing what they have said and asking appropriate questions for follow-up.

Think about a recent conversation you had with a stakeholder on a project. Were you really listening? Unless we really focus our attention on listening, it is easy to tune out after the speaker has said a few words. Were you listening and confirming or were you thinking about what you would say next or something else entirely?

Here are some tips on how to be a better ‘active’ listener:

• First, stop talking! Let the other person have the floor. Let them know you are paying attention to what they have to say.
• Give non-verbal clues. Most importantly, make eye contact. Nod your head when you agree. (or maybe even if you don't)
  
• Listen without judging or criticizing. Even if you disagree, let them express their point of view.
• Restate or paraphrase what the speaker says. This confirms your understanding and gives them a chance to clarify if necessary.
• Don’t interrupt, but when appropriate, ask good questions.
• Practice your note taking so you can record the conversation without distracting or interrupting the speaker.
  

Active listening is not easy. It takes focus and practice. But, the results are definitely worth the effort.

A Hard Truth

A business owner and I were prioritizing requirements for a product release. Among them were the performance requirements. He did not want to give them high priority, fearing untold hours would be spent on performance. He needs a breadth of the features delivered. My fear was that no consideration would be given to performance if we didn’t prioritize it.




After sleeping on it, I was able to express my position in better terms. I agreed that at this point in our project, tweaking performance is not a priority. If we want 3 second responses but are getting 5 second responses, we can live with that for now; other features are definitely more important.



However, if performance is so poor as to render a feature unusable (5 minutes instead of 3 seconds), then the hard truth is that we have not received the feature. It may have a beautiful user interface. It may produce 100% accurate results. But, if it cannot produce them in an acceptable time, then we didn’t get the feature. We need to include the performance requirements to make it clear we consider them part of the feature.



Stated in these terms, the business owner agreed. But, concerned, asked “What if the development team needs a month just to provide barley adequate performance?” My response which received a nod of agreement “Don’t you need to know that?”



It’s a hard truth that we don’t like to face. Sometimes there are simply minimum standards of acceptability, and we need to state them. And, be willing to pay for them.

The Value of Peer Reviews

You’re near the end of a long requirements phase, you’ve been working 60+ hours a week, and the project manager is pressuring you to deliver a final deliverable. So you do a quick spell check and proof read, and then toss it over for final review to business, development, test, etc. Hang on a second, though - there’s a problem with this picture. You missed giving the deliverable to a peer and asking for feedback. Peer reviews are an excellent technique for increasing the quality of your deliverables. In this post I’ll explain why peer reviews are so important and give you some ideas on how to do them successfully.

Why do peer reviews? After all, the PM and development are anxiously waiting for your document, so why hold things up? Well, for starters - requirement errors are very costly to fix - from 40 to 100 times more expensive to correct once implemented than if they were detected and fixed during the requirements process. So the return on investment (ROI) in doing peer reviews can be quite high. In fact, a study showed that as much as 10 hours of labor can be avoided for every hour spent inspecting requirements and other software deliverables(1). In addition:

• Turning over inferior work to business stakeholders, development, test, and other groups can reflect poorly on you and may cause a loss of credibility.
• Reviews are critical for encouraging continuous improvement, and can assist both the person conducting the review, as well as the person whose work is being reviewed. Reviews supply a mechanism for sharing approaches and provide critical feedback.

So how should you go about doing peer reviews? Karl Wieger’s Software Requirements, 2nd Edition book has an excellent chapter on doing peer reviews and inspections. Here are some ideas to consider:

• Provide as much advance notice to the person doing the review as possible - don’t drop a 100 page document into someone’s inbox and ask for feedback by the end of the day.
• Likewise, make sure you give the reviewer plenty of time. We’ve had debate in the past on the message board about how long to allocate, so you can go there to get different opinions. I won’t give a hard and fast rule - it depends on many things such as the complexity/density of the requirements, the familiarity of the reviewer with the domain space, etc.
• Do a basic check of the document before sending it out for a peer review. Make sure you do a spell check, clean up formatting, and make sure there are no gaps (such as "TBD") present, unless the document is a work in progress.
• The reviewer should utilize a standard checklist to use when conducting the review (e.g. is the requirement testable? is the requirement understandable?)
• The reviewer should focus on major defects/problems. While identifying minor problems such as spelling/grammar is helpful, it is not as important as identifying major issues such as unclear requirements, structural problems with the document, incomplete requirements, etc.
• Metrics should be collected from the review process - how many pages reviewed, how long was spent, how many problems were identified, etc.

1) Grady, Robert B., and Tom Van Slack. 1994. "Key Lessons in Achieving Widespread Inspection Use." IEEE Software 19(5):15-17.

Using questionnaires to gather requirements

When working with a large user population (hundreds or thousands) you will typically want to sample the population to do in depth interviews. But how do you know that the information that you gather is representative of the population as a whole? One way to get this information is to do surveys. In another blog post "product democracy" I described one innovative way that salesforce.com surveyed their users. However, a more traditional way to survey users is to use an old fashioned (or electronic) questionnaire. When would you use a questionnaire/survey to gather requirements?

• You need to survey a large population


• You need confidential answers


• You need quantifiable statistics


• Your users are geographically distributed


• Your users are extremely busy


• You have an unknown user group

In all of these cases it would be appropriate to use a survey to gather requirements. For a quick read on how to write survey questions try this link http://www.statpac.com/surveys/question-qualities.htm

For a more in depth discussion, I highly recommend the following book: Mail and Internet Surveys by Don Dillman

Requirements in time critical projects

I read another interesting article, Requirements Engineering for Time-to-Market Projects (requires IEEE access) by Christopher McPhee and Dr. Armin Eberlein.



This article talks about what techniques work best for critical Time-to-Market (TTM) projects. We’ve all had projects that were “urgent”, in fact most projects we work on are, so I thought this would be an interesting read. I’ll share some of the highlights with you. This article suggests that standard requirements engineering practices for large-scale organizations is ineffective when applied to TTM projects.



First of all, to increase speed in the process, there are only a few things you can do. Either do less things, do them faster, or do them concurrently. So they looked at techniques against these choices. Based on these criteria and looking at what would produce quality requirements, they found the top 4 techniques were:

• JAD


• Modeling


• Scenarios


• Prototyping

It’s funny to think about JAD as being fast, but I suppose if you set it up correctly and execute it well, it can be fast relative to talking to people individually. That said, it’s so rare to see JAD done well. I completely agree with the listing of modeling, scenarios and prototyping (see another post on the value in prototypes).

Just below those on the list are some more interesting techniques:

• Requirements re-use


• Checklists

I am looking for ways we can reuse requirements. This makes sense both from the perspective of projects in the same organization likely have similar requirements they could share (ex – non-functional requirements are often consistent) and at a functional level, many functions are reused across applications (ex – login), so likely have similar requirements. The checklist idea is a personal favorite. I live by them in most areas of my life. They take the thinking out of the process.

We are all familiar with the common attributes of good requirements – unambiguous, complete, concise, etc. In TTM projects, it may feel natural to shortcut the unambiguous attribute, but it turns out, developers really think that this one is more important in TTM projects. So they look for techniques that will lead to unambiguous requirements.

What was interesting is that they surveyed development teams about the usefulness of the various techniques on TTM projects. The top techniques were:

• Prioritization


• Change management


• Scenarios, use cases
  

What I found very interesting was that the top TTM techniques varied in an interesting way from non-TTM projects. So to start, change management is near the top on both types of projects. But requirements prioritization wasn’t even on the non-TTM top 10 list. And requirements reviews dropped significantly in usefulness from 3 on non-TTM projects to 9 on TTM projects. This last one is pretty interesting. I think it implies that you are moving so fast, you are probably getting requirements, building them and iterating on them through things like prototyping, more so than getting them right up front.

Anyway, it was something interesting to think about. Nothing terribly earth shattering though.

Agile Development

Someone asked me the other day about agile development. Should we do it? Can we reduce the time we spend on requirements if we are doing agile development? Is it only good for some kinds of projects?

If you're reading this blog there is a good chance you know about Agile and have an opinion about it. Here's your chance to join in the fun by commenting on this post.

I jotted down these notes to share with my friend who asked the questions.

• What is it?
– There are a number of ‘agile’ methods with some similarities and differences. (Extreme Programming, Scrum, etc.)
– These methods all use an iterative and incremental process to develop multiple releases of a software product.
– Iterations can be measured in days, weeks, or months.
– Not all releases are necessarily ‘production’ releases but even ‘internal’ releases are working software and not prototypes or ‘throw-away’.
– Business people are actively involved, providing requirements and reviewing release results.

• What Agile is NOT?
– Agile is NOT a way to get software without requirements.
– Agile is NOT a way to get software without testing.
– Agile is NOT a way to get software without planning.
– Agile is NOT a development methodology you can adopt successfully overnight.

• Some challenges to be aware of:
– As iterations become shorter, it becomes more challenging to deliver complete features with meaningful business value in each iteration.
– The first iteration may be especially difficult as it may include a lot of up-front costs and not many features.
– Developing and managing the overall plan for a project with many iterations can actually be more difficult than for a project with a single iteration.
– Integration can be a significant challenge for larger projects with lots of iterations and builds.
– Without careful attention to project planning, management, and control there is a tendency for agile projects to deteriorate into chaos.

• Agility is a good thing (done correctly).
– Becoming agile is a journey, not an overnight transition.
– A good way to begin the journey is to shorten the iteration cycle time for your software projects. Look for opportunities to make the iterations smaller and shorter.
– Insist on active business involvement in providing (and prioritizing) requirements and reviewing software releases.
– Do the right level of requirements documentation for the project size, complexity, and risk. Don’t look at agile as a way to get software without requirements.
– Do the right level of planning, testing, and change control for the project size, complexity, and risk.
– The Seilevel way of documenting requirements works well with iterative and agile development methods.

A Strawman Workflow for Requirements Change Management

One of things we are asked to do on almost every project is provide guidance on defining the workflow for managing the changes that inevitably occur to the requirements that the team has worked so hard to capture and document. After doing this enough times, there are certain actors and activities that are pretty standard to the process.

The Activity Diagram included here is a lightweight starting point for defining your Change Management (CM) workflow. You have worked very hard to get a set of requirements worth moving forward with into full-on development activities, the project deserves to have a well-defined CM process that is understood and followed by the team.



A few things about the diagram (which may be a little easier to view here):

• It implies that you have some tool into which Change Requests (CRs) can be entered and tracked. The diagram implies that the tool can automatically notify a project team member when a new CR is created. Having a tool is a necessity, automated email is nice-ity. If you have a tool designed specifically to support a CM process than you’ll have a leg up on automating the management of your CRs, but it can be done with a spreadsheet or document application. Be prepared for a lot of tedious manual work though.

• A Change Manager Role is identified in the diagram. This does not have to be, and for smaller projects will most assuredly not be, a separate full-time resource, but will most likely be a role that is owned by a resource that already owns one or more roles defined for the project. I have seen the QA Lead, Requirements Lead and the PM fulfill this role.

• The Change Control Board (CCB) is a critical part of the CM process. One of the biggest potential bottlenecks to managing change on projects is the ability to expeditiously evaluate and dispatch the Change Requests that are logged. The CCB must be comprised of individuals who can “rule” on the action to take on a CR. Your project may need a separate set of steps for generating, compiling and evaluating CR estimates, but this one is designed to be lightweight, and the assumption is that the CCB is comprised of individuals able to make both the estimates and decisions about how to the act on those estimates.

• The assumption here is that the person managing the requirements will become the CR owner of the assigned CR because the resolution of a CR is most likely going to involve an update to the requirements.

• The ‘Notify SME of CR’ step needs a little explanation as well. This could be a single email exchange with a single SME or it could involve setting up multiple meetings with a group of SMEs, or something else entirely. The idea is that while this activity looks fairly innocuous it could very well be a multi-step process in and of itself--the point being that the time and resources required to execute this step should be well understood, especially when you consider the repeated iterations that are often part of this step in the process.

• The ‘Update Project Models as Appropriate’ activity is also high-level with the understanding that this step will need to be detailed more to account for the particulars of your project—the intent is to hold a place for the activities that are triggered by an update to a requirement. So the intention here is to provide a starting point for a CM process, not define a one-size fits all solution. The final assumption here is that you will need to modify this template to fit the specifics of your project. With that in mind, it is also important to remember that the CM process that is defined at the beginning of the project will evolve as the project progresses and the process is executed in real-time in the midst of the project activities.