back
|19 Jul 2017|Klaus Leopold

Using Models to Forecast – An Interview with Troy Magennis

0
Troy Magennis and Klaus Leopold at LKCE13

Klaus: You use models to simulate different future scenarios. How did you come across this, and what is the best way to use models and simulations?

Troy: I have an electrotechnical education and am familiar with simulating electrical circuits. It felt like a good fit with IT. Simulations – especially Monte Carlo simulations – are used when it is too complex, or impractical from a time perspective, to formulate a mathematical function to solve a problem. Simulations calculate the result quickly and allow for experimentation, based on models, to measure which actions have the largest impact. Typically, Monte Carlo simulations are used when there are many interactive inputs in interrelated processes. Let’s take, for example, the weather forecast, which you also mentioned: A small rise in ocean temperature causes changes in temperature that
change the probability of rain in an area. These inputs reinforce each other, and some even have repercussions on earlier inputs. It is a complex system. We would never expect perfection from the weather forecast because of these relationships, but for some reason, we expect absolute predictability for IT projects, although they exhibit similar cascading complexity.
Klaus: Why not simply use estimations?
Troy: I established early on that every forecast and estimation are distorted if the effects from events like defect rates, additional requirements or uncertainties in effort are not simulated. Traditional estimations are based on the correlation between effort and time, but in complex systems, the work can also come to a standstill or be blocked for any number of reasons. The effort for a piece of work has absolutely no correlation to the calendar weeks that have passed. It is an unsuitable relation, as if I would say: A flight delay due to a technical defect has something to do with me sitting in the airplane. We always assume that the actual working time denotes a large portion of the delivery time. This is simply not true for most software companies and software development processes, if you also include not only the sheer development time, but also the time for testing, release and waiting time in the backlog. And these times should be taken into account, because from the customer perspective, work is not completed when development is completed, but rather when they can use the product free from failures.
Klaus: Models and simulations – that sounds very complex for many at first. In your view, is this approach even worth it and in which context can it pay off?
Troy: Most of the time it’s worth it. You have to do something, though, to achieve the results. If the number of employees is fixed and the current process cannot be changed even a little, it probably doesn’t make sense. Whatever the case may be, even simple models can help quantify, in dollars or euros, the effects and benefits of process changes. It was important for me to be able to answer questions and understand the effects when, for example, the defect rate is cut in half. Which employee skills in which situation bring the most benefit? Are defects or missing test environments a bigger problem? Simple models help answer these questions, and that goes beyond traditional estimation. Being able to see the answers in a monetary amount helps the argument as to why an investment in the change is valuable. It helps to make well-informed and carefully considered decisions.
Klaus: What could such a model look like?
Troy: Sometimes a simple spreadsheet can show the effects and benefits before and after a process. Early on, I realized I needed a more detailed analysis to find the answers to my questions. In my case, I wanted to model every step of a work item as it travels through the process of a specific team. So, I developed a simple XML language (SimML), that allows the work and process to be written in text form. It is typically 25 to 50 lines and is a mix of work, historical data and definitions of the team process. When I have a question about the costs of defects, for instance, the model specifies the current defect rate. Then I make a forecast for the delivery date and costs, cut the defects in half and observe the differences in the date and costs. The content of the model and its complexity emerge from the questioning. I cannot emphasize enough how important it is to think about the questions before developing a model. Only when you know the questions, should you then build the simplest model to be able to answer these questions.
 
Troy Magennis has been working in software development since 1994. He has held various positions, from Quality Assurance up to Vice President of a multinational company, and several years ago, started his own business, Focused Objective. Now he concentrates on what really interests him: Forecasting and its applicable simulations – especially Monte Carlo simulations. In this area, he advises companies like Walmart, Microsoft, Skype or Siemens.
 
Troy and I join forces and we are doing an “Improving Kanban – Metrics Special” training class this winter. The classes are certified as KMP II by the Lean Kanban University. Join us on 4+5 Dec in Vienna or 6+7 Dec in Dusseldorf. Sign-up now, limited seats!


This interview is part of my book Practical Kanban. Get it on Leanpub now!

No comments
This conversation lacks your voice:
Your E-Mail Address will not be published.