RE'08 Follow-up: Teaching About Unknown Software Requirements

At the REET08 workshop, Ray Barnes presented a paper “Teaching the Unknown and the Unknowable in Requirements Engineering Education” written by himself, Donald Gause, and Eileen Way.

In their paper, they talk about the need to accept that the "unknown" and "unknowable" exist in requirements.

Confusing? That is - "You don't know what you don't know."

And while it is challenging for a requirements engineer to deal with, it is still a very necessary skill. Their paper is specifically focused on teaching students about the uncertainty and how to work with it. They have the students do “real world” problems and give them specific techniques to deal with the uncertainty. The students seemed to experience the expected frustrations of not having all the answers in these problems.

I thought it was interesting that when they taught a specific technique to the students (through lecture), the students still didn’t apply it, even amidst their frustrations. The paper gives an example of teaching students about Toulmin’s model of logical argumentation multiple times. Finally the instructor helped 1 student use it, and later that student offered to present it to the class. It wasn’t until this last step - the student explaining the concept and value - that the rest of the students really got it.

That’s probably pretty typical of learning in general, both in that they have to experience to understand it and peers can be very influential in teaching. In fact, one of our favorite training games is called Each Teach from Thiagi, where we split the class up and lecture them on different topics, then ask them to teach the topics to one another. We have had outstanding success in students’ perceived understanding and actual understanding of the topics.

I found it to be an interesting talk and had some questions that came out of it:

1. How does unknowable relate to the untestable in requirements? There was an INCOSE paper that talked about the idea some requirements cannot be written in a testable format, so you can do risk analysis on these to mitigate the issue. Similarly, you might identify the areas that seem most unknown in your requirements and focus more time on those specifically. Or weight your risks and priorities accordingly.


2. How do you document the unknown or unknowable? If you have an area that you know is still fully undiscovered or in some way vague, it should still be tracked. Perhaps in an RM tool, or even in an issue tracking list as things to be further discovered. And certainly a risk list, as per point 1 above. Ideally you would document as much as you do know and flag it in some way for both follow up and just as a warning.


3. Do other industries have this issue with the unknown? Surely it’s not just systems and software, so what do other industries do to deal with the unknown? Probably a lot around risk analysis, but this is just a guess.


4. In teaching something like this topic of uncertainty – both in awareness and the skill to navigate it - how much of it is something that just has to be experienced in order to learn it? There isn’t always a black and white answer to how to approach the unknown. My thought is it’s probably 95% experience, with 5% teaching them helpful tools to try out.


5. When doing the real world exercises, can you write scripts for the instructors to use in role playing on how to be vague? I haven’t really tried this formally, but it sounds fun. And hard! In fact, related to the unknown problem, how can you predict what the students will ask in order to have pre-scripted responses?


6. I think one of the most important things in an experiential exercise is to debrief afterwards. I’m not sure if they did this here, but it wouldn’t surprise me. You put students in what is probably a frustrating exercise, and afterwards you need to talk about the experience, let them explain what was hard, and have them explain what they learned.