QOTW: Have you seen good assumptions go bad? And what can you do about it?

Replies to This Discussion

Permalink Reply by Anon on January 16, 2009 at 4:20am

Neil, I was working on a multi-million dollar project, providing fixed bid service to a Financial major. The project was won on an open bid, we presented the RFP response, explained our whole proposal to the customer, including risks and assumptions. We both thought that we were on the same page, except that things started falling one after the other. Many of our assumptions went out of the window. I want to point only one example out here:
a) One of the requirements was to migrate data from the existing system to new system that we were building.
Our broad assumption was that the customer would provide us all the information (DBs, tables, SQLs) related to the existing system and would clarify any questions and doubts we may have about the system. In addition we would require one knowledgeable business person who can help clarify any doubts that we may have.

In reality when we got the DB structure and started looking at it closely we found that the underlying tables did not follow any rules of Relational Database. There were no relationship between tables and data integrity was questionable.

The person who created the database/older system was no longer with the company.

We went back and forth several times and in the end had to look at every line of the old Stored procedures to figure out the relationships in the existing database and then work our way towards migrating the database.

Instead of a few weeks this took months with significant overrun on budget and manpower!

We were to put it mildly s********!

The mitigation is that the devil is in details. Ask as much questions as one can think of upfront and tie the assumptions down to impact on the development and the associated costs. Explain the assumptions and don't forget to ask the customer if they do understand what the assumptions means in their own words. Ask the customer what assumptions they think we would have missed out covering. People in general are nice and would want to help you out.

Thanks

• ▶ Reply

Permalink Reply by Chris on January 16, 2009 at 11:05am

In my opinion every good assumptions are good in some circumstances, that’s pretty obvious. They can be good or bad in some point of view at the beginning. The level of goodness of assumptions depends on made analysis. The poorer analysis, the bigger risk that assumption are bad. Assumptions – by definition – contains some kind of risk. Thus if I do assumption, I have to think of risk management. Another aspect of assumptions, they can go bad if circumstances change. So I try to examine whether my analysis are good and environment doesn’t change. Being aware of risk I try to make some mitigations, obviously more work in risk management if impacts on my work within given assumptions are bigger. Shortly that’s theory.

The worse thing when the good assumption appear bad – no matter what reason - in late stage of my project . Then I’m so determined to finish work, completely irrational state, Engagement etc don’t allow me to stop, to think and to take into consideration new position. Sometimes it’s so hard to throw away so beautiful piece of code. In such cases it is good to have good friend, who stays away and who put me straight. Then I try to stop and puzzle if I didn’t make it before.

• ▶ Reply

Permalink Reply by Peter Degen-Portnoy on January 16, 2009 at 1:49pm

In the many years I spent consulting and contracting, it was standard practice to document ones assumptions during the programs definition stage -- where business requirements were gathered, goals were set and budgets forecast. The format that I used as documentation was a risk assessment and mitigation plan -- RAMP.

Everything that my team and I could think of in the initial discovery and proposal stage that could impact the program and projects was documented. I would spend time with each team member and ask them, "What is keeping you up at night?" or "Let's look at this sub-system -- where can it screw up?" Their answers became the risks and hidden within the risks were the assumptions.

For example, the risk was that the embedded system's custom development environment would be insufficient for our application, which would require us to change the development environment or switch to a new embedded system. Hidden in that are the assumptions that there was a communication library in this embedded system that was sufficient for what we were doing, that the call stack size was configurable, that the compiler supported call by reference of parameters, etc.

After digging into each of these risks and their underlying assumptions, each one was quantified in term of how likely it was to impact the project, how severe the impact to the project would be, and how the impact would be identified, how the risk could be mitigated and how expensive the risk mitigation action would be. Finally, a cost/benefit statement could be made for each risk and underlying assumption.

Yes, it was a lot of work and it helped to steer around many of the large rocks that one could identify at a programs onset. However, one can only plan for that which one anticipates and even documenting assumptions (and reviewing them with the customer) was no guarantee that the customer took the RAMP seriously. I wished I could have done a monthly update of the RAMP -- it was the best document of them all. However, there never was any time for such updates. The program to which I am referring had industry driven deadlines, very high visibility, significantly insufficient time, and huge (1000%+) requirements churn.

The final chapter is this: in this program, a deeper assumption went unnoticed. We had architected separate sub-systems that communicated via remote procedure calls (RPC). It was an architecture that had been proven in a number of similar control systems. However, as the system was assembled and moved from the development environment to the production environment in the embedded system, it moved inexorably (due to system limitations and hardware production decisions) to a single monolithic system. We needed to pull the sub-systems into a single executable. There was no longer any need for a library for RPC. Never did we consider our base assumption that a fundamental architecture decision would be rendered completely inappropriate. We embedded an engineer at the customers site for a very lengthly engagement to pull out all the RPC code.

Here is my take away point: even under the best circumstances, changes drive a product or program onto the rocks, even if one identifies underlying assumptions and sets up processes to handle them. The changes can come from customers, internal forces, the industry, and even more surprising things, like global political events. The best thing a product manager can do is help his/her organization manage the inevitable slew of changes that impact products and programs.

• ▶ Reply

Permalink Reply by Jeff Celander on January 16, 2009 at 4:48pm

I have worked on commercial software for 30+ years. Here are some dumb assumptions:

- Software that used used 2 byte years (remember y2k)
- Assumptions that everyone in the world uses USA formated dates, times, and calendars.
- Assumptions that everyone in the world uses single byte english EBCDIC character sets.
- I have seen fixed window (screen) sizes that don't fit on a users display device.
- Making assumptions that all users have workstations that can handle 400,000,000 returned data base rows for display with no way of limiting/filtering the number of rows returned.
- Developers that assumed that noone would stumble on a hidden option that displays some salty langauge or inappropriate graphics.
- Making the assumption that all of your product documentation that you ship has no errors only to find out your tech support toll free number is a phone sex line.

These are just a small sampling of bad assumptions (made by large and small software companies) I have seen over the years.

• ▶ Reply

Permalink Reply by David Locke on January 16, 2009 at 6:53pm

The executives approved a change in their target market. They wanted to move up to the CxO sale. They had gone through the Fortune 400, 1000, 2000 cascade, so where could They go next. They were selling enterprise software. They were leaving money on the table. It was cool and right and wonderful to do the CxO-level selling. They assumed that the CxO sale was the right thing to do. They did not test this out. They did not wait until the next quarter. They jumped to it right in the middle of a quarter that had been forecast on the old sales strategy.

The sales funnel slowed down. They missed the quarter. The stock stalled. They no longer exist.

Was it a good assumption? All I know is that it led to a cascade of assumptions, an avalache of error.

Being a network is the way to mitigate against assumptions. If there is no one right way, then there are many. Each way will be built on assumptions at some point. It's not the assumption that is the error. The error is reliance on a single approach. If you simulaneously use a collection of approaches, then the failure of one will not cause all of them to fail. They could move a team of sales reps to the CxO sale, and leave the rest of their sales teams pursuing their current prospects. They could have met their quarterly projection.

That company had a sales culture, one of those mano-el-mano sales cultures. They dominated the company, so marketing was thin. They always made the push that made the quarter. They took a thin approach to getting things done.

Resilency provides indifference to assumptions.

• ▶ Reply

Permalink Reply by Steve Jones on January 18, 2009 at 5:23pm

We have to make assumptions to build for the future, but we should always take our assumptions with a grain of salt. Expect that they will miss the target and we need to change things.

I've seen a few big assumptions go bad, and I'll share a few large and small. First, in 94/95, Microsoft dismissed the Internet. They pushed their MSN system with the introduction of Windows 95 on 24 floppy disks, assuming that the internet would work like AOL, with a few, large disconnected systems. They changed on a dime in early 95, swifting the company to work with an open Internet and it was amazing to watch.

In the late 90s, early 21st century, they assumed that no software company would ever really grow large enough to challenge them. They assumed they'd be able to buy any company with a successful product and all their cash. Google surprised them to a large extent, and it was really amazing. Bad business and software assumptions.

In my career, I worked for an import company at one point. We constantly were working with a shifting inventory of wood and metal products from overseas and the manager of that division was worried about matching inventory to sales. So he had my build a report that looked back at history to try and "predict" what inventory we would need 2 or 3 months out.

I spent months on the report, constantly tweaking our model, and always having issues matching things up with demand. It really required a more complex set of algorithms than we could build at that time, or knew how to build, but also it really depended on someone knowing the market. Knowing what the conditions of growth were around the world, government changes, shipping issues, etc. Not to mention changes in suppliers.

The assumptions that this could be easily done by computer, enabling "cheaper", less experienced people to manage inventory was really wrong. The original report, done in a few days, was a good starting point and what was eventually used by the inventory people to place their orders, but with changes made based on their experiences. Computers, and software, cannot do everything. You often do need to remember that human intelligence really matters.

• ▶ Reply