Requirements for Batch Processes

A batch process is used to do an automated manipulation of data, often for a bulk transfers of data. They execute transactions automatically without real time user input data.

You would typically implement a batch process if you needed to schedule the update or manipulation of data for a time in which the systems and data involved were less “busy”.

As an example, we might need to update changed product data every day, from one system to another. It may be important to do this update when there are not a lot of users trying to hit an application that uses that product data because it could slow down system response time for the users. In this case we would want to use a batch process. In the implementation, the changed product data would be extracted from the main system and imported into another system nightly. Then there may also be manipulations such as upon importing, such as concatenating the productName and productDescription fields into the productDescription field in the new repository.

There was a post on the messageboard recently about what models to use to capture batch process requirements. Use cases do not work well, and while process flows might sometimes work, the model choice really depends on the complexity of the transactions.

When I think about something like a batch process to load data, I start with a set of questions. To answer each of those questions, I employ the appropriate model:

Use data models, such as data dictionaries, ERDs, state tables:

• What data needs to be manipulated?


• What data needs to be batched?


• What are the attributes on the data to be batched?


• What is the desired end state of the data after running the process? (This might be a state of the data or it might be an output format.)


• What manipulations are executed on the data as part of processing? (If the manipulations get complicated, then a process flow diagram might help describe those steps.)

Use system models, such as context diagrams:

• Where does the data come from?


• Where does the data need to go?


• What other systems have data that is used in the manipulation?

Use people or system models, such as actor lists or context diagrams:

• What or who triggers the batch process to run?

Use non-functional models to identify non-functional requirements:

• When does the process execution need to be completed by?


• How frequent does the process need to run?
  

In capturing the requirements for batch processes, you need to start with a set of questions that have to be answered to gather the basic data, and then apply the known models to answer those questions.

Torpedoing Projects

I saw an interesting talk at INCOSE 2007 called Damn the Torpedoes! Lessons from Underwater Warfare. Terje Fossnes, from the Norwegian Defence Procurement Division – Submarines, talked about five cases from history of torpedo failures and went through the core causes behind the those. It was an interesting look at non-software projects in history (and a unique topic) that clearly failed, and to see common issues we still face today.

Here is a summary of the points he made about common issues across those failures:
1. Introducing unstable technology

New technology was introduced too early. They adopted the technology early to
gain a competitive advantage. If it works, that is great, but if not, the
results can be disastrous.

2. Lack of product verification

Engineers tend to believe in their own abilities and shortcut their testing.
There is executive pressure to deliver quickly and cheaply. Consequently,
testing may have been cut short. Though it is much more common to see extensive
testing today, it seems the lesson has been learned.

3. Lack of stakeholder involvement

In the cases of torpedo failures, very few stakeholders were involved. Those
that were involved played roles in planning, testing and accepting results.
However, they did not know of assumptions that were being embedded in the design process.

4. Lack of cooperation and information sharing

The projects with failures demonstrated a lack of collaboration. The crews
reported failures, but were later blamed for incompetence. They were not told
about workarounds that were implemented, so they did their own thing to fix
them, and it turned out worse. And sometimes engineers warn of design issues and
just are not believed.

5. Maintenance and training issues

Errors were introduced in maintenance. They ignored wear on the systems. They
skipped training.

6. Shortcutting the process

It is important to explicitly tailor the process if the activities do not add
value. However, in these projects, the tailoring was typically by accident or
because they were out of time.

All of these are interesting, and we’ve probably all seen them in different forms. The interesting ones to me are 3 and 6, as they relate to the requirements efforts failing – either in not using the right stakeholders or shortcutting the process, specifically the requirements process. These are definitely still problems today.

As an interesting side-note, he shared with us that it is not commonly known that torpedoes actually pass 5m below the ship, instead of impacting it directly. It’s the force of the ship coming down and hitting the water that breaks it. Some of my coworkers challenged this, so no promises that it’s actually correct!

Reminder—Handling Missed Commitments

I occasionally run across common situations and write a little reminder about dealing with them. Most of us have been in the situation of making a commitment and then not being able to fulfill it. Maybe we underestimated the effort to fulfill the commitment, maybe an emergency arose, or maybe we just made a mistake and forgot. Here are some quick reminders on what to do when you find yourself in this situation:

• Let the person to whom you have made the commitment know as soon as possible that you will not be able to meet it. This gives them the maximum possibility of meeting their needs elsewhere. It also reinforces that you respect their needs, even if you can’t meet them.


• If possible, provide alternatives to meeting the commitment. Perhaps a coworker can help? Maybe the need fulfilled by the commitment can be met in another way?


• Let the person to whom you have made the commitment know if you can meet it at a later date. Many deadlines can shift.


• Let the person to whom you have made the commitment know what you can do. If you can meet 75% of the commitment, there may be high value in that.


• If you can partially meet the commitment, involve the person to whom the commitment was made in prioritizing the tasks you can complete. Sometimes 50% of the tasks have 90% of the value, and meeting those 50% will be a great help.


• Apologize. This acknowledges the impact your missed commitment had on the person to whom you made the commitment. Missing a commitment can damage a relationship. An apology is a great way to begin to mend it.

Final reminder—most of us realize part of being human is making mistakes. We don’t judge others on whether or not they make an occasional mistake, but we do judge them on how they handle their mistakes.

Requirements engineering for software versus systems

At INCOSE 2007, I attended a panel discussion “Requirements Engineering for Software vs. Systems in General?". The moderator was H. Kaindl and the panelists included R. Griego, M. Hause, C. Hood and M. Mannion. This content is all paraphrased or interpreted from things the panelists said.

So to start, a common response from the panelists was that there is no real difference between requirements for systems and software, at least not a significant one. In fact, one of the panelists responded to being invited to the panel by saying “there is no difference, so this panel won’t last long.”

In reality, when we look at requirements engineering for systems and software, the outputs are similar – requirements. The construction processes are similar. The management issues are similar. The tool support needs are similar. This may be an oversimplification, but in general I agree with the point.

One thought is that many software people tend to avoid the systems world, thinking of it as “messy”. But if you think about it, software really is just a component of systems. As one panelist said, it’s like the central nervous system. In looking at the different levels of requirements we can see the relationship between software and systems requirements is a bit more obvious.

• Customer requirements


• System requirements


• System design requirements


• Sub-system requirements


• Sub-system design requirements


• Implementation

One panelist took this a step further to look at the different types of requirements to deomonstrate more commonalities:

• Functions – effects achieved by some entity


• Behavior – state change over time, function could be an abstraction of behavior


• Constraints – restrictions or limitations


• Structure – arrangement or relationship of elements in a system, physical or logical

This last one, requirements that describe structure are not as common in software as systems, but the rest are all common in both. In addition, large complex systems have software, hardware and human actors. You can still write use cases to define the functional requirements for composite systems.

There are differences between systems people and software people, with the languages they speak and their cultures. So a real issue that comes out of this is that the communication really breaks down. This is not to say that communication does not breakdown within software projects of course! But it’s a different type of issue with large system projects. People skip from the customer requirements into sub-system requirements. They assume something is going into software or hardware and no one really owns the system level to know for sure that it’s all covered and the dependencies between components are known.

In the end, this is an easy thing to fix – ensure someone is accountable at the system-level for making this communication happen. And use tools to trace between requirements of the sub-systems, up to system-level business objectives.

So this may be simplifying it a bit, but it really seems like the tools and techniques we apply in the software world also apply to the systems world – perhaps at a different scope, with varying degrees of complexity.

Back to School

The first day of school brings back great memories for me. Going shopping for a Superman backpack, trying out the new lunchbox (how will the thermos, two desserts and a note from mom all fit in there?), and getting ready to see old friends and make new ones. It's a time of excitement and of nervousness. Will I like my teacher? Will I have a lot of homework? How will I ever read all those books on the bookshelf?!? But it's also a time of opportunity.

Starting on a new requirements project can be a lot like going back to school. You get to meet new people, discover new areas and challenges in your organization, and maybe even get a new lunchbox! But perhaps most importantly, you get the chance to learn new things about gathering requirements.

Wait ... "learn new things about gathering requirements?" You already know how to gather requirements, right? Shouldn't that sentence read "show off your skills with requirements gathering" or "teach others about requirements?" Well, it certainly could be either or both of those. Hopefully you're able to educate the rest of the team about good requirements engineering practices during every project, and hopefully the team will be impressed with your RE prowess. However, I think the opportunity for each of us to learn more about what we do, while we do it, is even more important.

If you've been working in RE for a while, you may have a typical pattern or approach to your work, and that approach probably works well. That said, each of us can always learn something to help improve our work. Maybe a particular model did or didn't work especially well on this project. What made this project different? What can you take away from this experience to improve both the current project and your approach for the next one?

In addition to this "on the job learning," new projects provide great opportunities to find new sources of information outside the project. What new blogs, journals, discussion groups, or books are available on RE? What local professional associations meet to discuss software development topics?

The start of a new project only provides a good opportunity to learn – it's up to you to take advantage of that opportunity. Use the excitement and energy a new project brings to help you discover new things about RE. Those discoveries will improve both your skill set and the project you're working on, and that new Superman backpack will certainly get the conversations flowing!

The State Machine Diagram

One of the challenges faced by requirements analysts is the need to communicate the complex behavior of systems in an understandable yet rigorous and verifiable way. A significant amount of the effort in the requirements process is devoted to translating the users’ needs and goals from ‘the language of the user’ into an actionable and unambiguous specification suitable for use as input to the design and development process. This can be a special challenge for agile teams working through relatively rapid iterations where not a lot of time is devoted to model building before coding begins.


The state machine diagram is a UML artifact that works well for this purpose. Put simply, the state machine diagram captures information about the states an object can go through during its lifecycle. Some objects have relatively simple lifecycles and for these objects the state machine diagram may be overkill. However, be careful about dismissing any object from consideration. The lifecycles of many seemingly simple objects are more complex than they may at first appear. One of the benefits of using the state machine model is that it can help illuminate requirements related to events and state transitions that may otherwise be missed.


A state machine diagram can be drawn for any object but for the purpose of documenting requirements it is often useful to look at the product as a whole or at some component of the product or of the user interface as the object whose lifecycle we are modeling. A state machine diagram can be drawn for a use case along with, or instead of, an activity diagram.


The theory behind finite state machines says that an object has a finite set of possible states it can be in. An object in a particular state can respond to certain events by making a transition to another state. During the transition an action can take place. In state theory an action is an atomic (uninterruptible) operation. A state can have an entry action and an exit action so the sequence of events for a transition would be: 1) Exit Action (from the source state). 2) Transition Action. 3) Entry Action (for the target state). According to the theory, if we have a complete list of possible states and a complete list of possible events we can develop a complete model of the lifecycle of the object. In practice, some judgment may be required to determine the appropriate level of detail depending on the purpose of the model.



To get started, think about the initial and final states of the object. Keep in mind that an object may not have a relevant final state and may simply return to its initial state or to an idle state at the completion of a cycle. Identify the states an object will go through during its lifecycle, the transitions between states, and the entry, exit, and transition actions. For complex object lifecycles it may be necessary to model substates and composite states. A substate is simply a state that is nested within another state, the composite state. A composite state can contain any number of substates and substates can be nested to any level.


As you refine the model it may be helpful to create a state transition table. The table is typically created as a matrix and for each state, shows every other state and whether a valid transition exists. Check to make sure each state, other than the initial state and final state, has at least one transition into and one out of the state. For agile and iterative processes the state machine diagrams and state transition tables can be built up and refined over time and can provide a stable model across iterations.


At completion, the model should contain a full set of the relevant states an object can exist in during its lifecycle, the events the object will react to, the transitions the object will undergo, and the actions that will occur at each entry, exit, or transition. The state machine diagram can be used to validate and enhance the user stories and use cases and can also help communicate requirements in an unambiguous way to the development team.

Fun with CRC cards

Use cases have become an indispensable tool for requirements elicitation and documentation. They have the advantage of being relatively easy to write and, if written well, they are relatively easy to read and understand. This advantage comes in part from the fact that they are created using ‘natural’ language text. They can and should be written in ‘the language of the user’. Each use case tells a story about how an actor will achieve a goal using the functionality available in the system. For many agile and iterative software development processes some form of use case or user story is the primary requirements artifact.


Many teams using use cases eventually discover two disadvantages: 1) natural language text unfortunately allows a great deal of ambiguity, and 2) reading and reviewing any non-trivial use case has the potential to become tedious. The combination of these two disadvantages can mean missed or incorrect requirements. A design and implementation based on a requirements specification with missing or incorrect requirements will almost certainly result in the delivery of ‘bad’ software from the users’ perspective and frustration for everyone involved in the project.


So, is there a cure for the tedium and frustration?


One way to reduce the tedium is to reduce the size and complexity of use cases, possibly even limiting the use case documentation to the user story or use case narrative. This can help reduce the tedium but it can also allow for increased ambiguity. The ambiguity will have to be resolved at some point, either through additional work in design or additional iterations through the requirements discovery process.


The Class-Responsibility-Collaboration Card is a tool originally conceived as way to teach object oriented principles that has gained acceptance as a useful modeling technique. Responsibility-Based Modeling and Responsibility-Driven Design both build on the concept that software components have responsibilities and collaborate to produce results of value.


CRC cards can be used with use cases and user stories to help eliminate ambiguity and reduce tedium and frustration. With a little creativity they may even make the process fun!


A use case can be thought of as a collaboration. The objects in the system will collaborate to produce the result of value. Each object plays a part in the collaboration. Each CRC card represents one object in the system. At a conceptual level the focus should be on the business objects, in other words, things that exist in the real world. It may make sense to introduce a user interface object and a controller, but avoid the temptation to introduce design level objects.


Getting to the point where you can start using CRC cards of course requires writing use cases or user stories and identifying the objects involved. A common technique for finding objects is to go through the use case identifying the nouns in the narrative text. This is not a bad technique, but don’t rely on it exclusively. Take some time to think about the business objects involved in the use case. If you can think of objects that are not represented by nouns in the narrative it may make sense to revise your narrative to include the new objects.


For each class of objects you identify, create a CRC card. Name the class and list the things the objects will do and the things the objects will know about. Also list the other objects the objects in this class will collaborate with. CRC cards have traditionally been created using index cards. The small size of the cards helps emphasize the importance of limiting the responsibilities of any one class. The cards are cheap and easy to lay out on a table and move around. Large post-it notes or self-stick flip chart paper can also be used and can be hung on a wall during facilitated sessions.


Now comes the fun part. Once you feel you have a pretty complete set of CRC cards for a use case, assemble a group of subject matter experts (SMEs). Even better, have the SMEs help with the identification of objects and creation of the starting set of cards. Pass out the cards so that each person in the room has a card or so that the card for each object is held by a person or small group. Ideally, the person holding the card should have some domain knowledge for the object represented by the card.


Have the group ‘act out’ the use case. You may find you need a card for the actor and, as noted above, you may need a card for the user interface and a use case ‘controller’. Walk through the use case steps. At each step, identify the objects and the interactions they are involved in. Is data being provided as input or output? Is some process taking place to alter the state of an object? Are any objects or interactions missing? Document everything and work through each alternative course of events. At each step of the use case, each requirement must be supported by some object. Keep the discussion focused on business objects at the conceptual level. Avoid the temptation to create design level objects that do not represent things that exist in the real world.


CRC cards can help add rigor to the process without making it tedious. The cards’ ease of use makes them ideal for agile modeling sessions but they work well with any technique that includes user stories or use cases. The idea of role playing may not work for all groups so be aware of the culture you are working in. Getting people up out of their seats, moving about the room, and interacting can help keep everyone engaged. Once a team gets used to doing this as a regular part of the process they are unlikely to want to go back to the old, tedious, and error prone ways.

Reminder: Our Goal is to Produce Software

I’ve been in situations where I’ve seen process used, underutilized, misused, and abused all on the same project. The appropriate amount of process varies widely based on the needs of the project; it can be impacted by things such as the size of the team, the skills of the team, and the cost of an error. How do you determine the right level of process for your project?



Here’s my quick reminder: Process should exist to support the production of good software. If a process is a cost-effective method of producing good software, use it. If it is not, abandon it.