Doing Software Estimation within constraints of Hofstadter’s law and Parkinson’s law

There are two popular adages in software development that people use when talking about software estimates.

Hofstadter’s law states that it always takes longer than you expect, even when you take into account Hofstadter’s Law

Parkinson’s law states that work expands so as to fill the time available for its completion.

Hofstadter’s law suggests that you will always underestimate even if you add a buffer. On the other hand, Parkinson’s law suggests that if you give more time to a task it will take more time to do it. In short, we are doomed either way. So, let’s not do any estimates. #noestimates FTW! The reality is that all customers will ask you for estimates and timelines before they award you with a project. 

Estimates are required for:

• Making a business case for the new project; 
• Knowing when the project will be delivered;
• Allocating money or teams of people for some amount of time;
• Working backwards from the end date.

One of the least favorite parts of my current role is to provide estimates for new software development project bids. In this post, I will not be talking about sprint estimates or release/milestone estimates. Instead, I will talk about product development projects. These usually involve uncertainty, ambiguity, and customers not sharing/knowing complete details. Examples of such projects include a new mobile banking application, omni channel customer onboarding insurance app, a back office customer 360 degree platform, oracle to postgresql migration (involves stored procedures as well), reengineering a mainframe based pricing engine to modern real-time pricing engine, and many others. As a side note, I also actively participate in their architecture and solutioning.

As you probably understand, these are not small projects. Each project could be between six months to a year long or sometimes even more. The bigger the project, the bigger are its chances of going over budget and over schedule. In this post, I will share what I have learnt about estimates over the last couple of years while doing the same for many such projects. 

These days MVP is the most abused word. Out of the cost, scope, and timeline, cost and timeline are fixed and scope keeps on evolving in the name of Agile. The story about the misuse of MVP and Agile will be shared in a future post. 

My Estimation Process

Coming up with correct estimates is hard. There are usually known unknowns and a few unknown unknown when you are estimating a new project for a new customer. Guarding your organization against these unknowns should always be in your mind when doing the estimates. You need to come forward with an estimate such that you are awarded the project and you can successfully deliver it at profile. You don’t want to overestimate so that you do not lose the project. At the same time, you don’t want to underestimate the project so that you end up losing money.

At a high level, my estimation process involves the following steps:

• Understand the problem statement.
• Break the problem statement into functional items and build an understanding about each item.
• Calling out Non-functional Requirements and the impact that they will have on solution and estimates.
• Estimating each functional item. I estimate items in the number of person days. 
• Adding effort for testing and other automation activities. These are done at the percentage level of the total dev effort. There are a few more things that get added to the project like project management, Agile PO and SM, etc, but I am not going to cover them in this post.

The above process is not unique. Anyone who has done estimates for long would have come up with a similar process. The important point in my view is not the process in itself. It is the experience that you gain by doing estimates multiple times and building an intuition on the kind of effort required to do a certain task. There is always a catch since I might not be the guy who will develop the software eventually. I might under-estimate the work. I try to cover that up by my software development experience, adding uncertainty factor, and applying lessons learnt doing similar previous projects. At the end of the day, estimates have to be realistic so that you can win the project. We don’t want to fall for Parkinson law.

Now, let’s cover each of the steps in the process in more detail.

Understand the problem statement

I know this might seem obvious. I will shamefully admit that I have given estimates in the past without understanding the problem statement. The main reason for that is not having an inhouse domain expert with whom you can spend time to understand the problem statement in depth. Most customers when they are looking to outsource projects expect that you understand the domain. Our organization has spent considerable resources building domain knowledge in certain domains that has really helped technical folks like me to come up with estimates. Customers these days expect a partnership model where they also want to learn from an IT service organization on how to better solve a problem based on the organization’s experience in building similar products. 

Things to do at this step:

• Make sure the functional/domain expert joins all customer meetings. If the domain expert and product owner are two different people then make sure they both join the calls.
• Ask customers to share a requirement document. Spoiler Alert: Most customers don’t have a requirement document. At best they will give you an excel sheet with one line requirements.
• Set the expectations to the domain expert and PO that they must expand on the customer provided excel sheet with details provided during customer interactions.
• Ask for UX mockups or wireframes or just plain drawings on pen and paper. I can’t stress this enough. It becomes much easier to estimate if you have a visual representation of a problem.
• If the business problem involves a complex workflow, then ask your functional expert to create a flowchart or something similar.
• Understand from the customer which features they consider are game changers and which features are showstoppers. I learnt this from How to build great products blog by defmacro.
  • A game changer: People will want to buy your product because of this feature. 
  • A showstopper: People won’t buy your product if you’re missing this feature, but adding it won’t generate demand.
• Try to capture high level NFRs during the conversation. It again comes by asking the right questions and with experience.
• Understand which platforms (web, mobile) the application has to support. For the web, which browsers and their versions and for mobile, which operating systems and their versions. 

The amount of details a customer provides you at this time determines how clear they are about the problem statement. If a customer lacks details, I suggest keeping your buffer high as there will be more surprises down the road. Some customers are accommodating and they will consider change requests, but there are others with whom you will have difficult and ugly conversations. 

Break the problem statement into functional items

As per the process we have, I get this list from our functional/domain expert. This usually is an improvised version of the requirement list received from the customer. There are a few times where we have to create the list. The important point that matters is that you have an excel sheet with items and item details listed on it. 

Things to do at this step:

• Make sure you have all the artifacts in a single folder. This includes an excel sheet with functional stories, UX wireframes, or any other document shared by the customer.
• Have a walk through of the item list with your functional/domain expert. You can clear out any doubts you might have.
• Document functional assumptions. You will have to share both functional and technical assumptions with the estimates. 
• Get a sense of complexity for each item. 

Calling out Non-functional Requirements

This is often overlooked. NFRs can impact estimates in a big way. A simple NFR like configurability could mean that all your frontend needs to be driven by metadata coming from the backend. This involves creating multiple components and ensuring performance is not impacted when screen metadata is delivered from backend services. 

In this step, you have to create a list of top 3-4 NFRs (Non-functional requirements) that are essential to the success of this application. These need to be discussed with customers so that there is no gap in understanding.

Things to do in this step:

• Expand each NFR so that you really understand what is expected. For example, for performance NFR you should document the number of concurrent users, response time, amount of data and its growth over time.
• Coming up with a list of tasks related to NFR and estimating effort for each task. For example, for performance NFR, you need to prepare test data, write performance test scripts, provision an environment, and finally the team has to spend time to improve performance. 
• Thinking about the solution at a high level. This involves architecture style, API style, and tech stack.
• Document technical assumptions. This includes supported devices, platforms, and using customer existing CI/CD processes and tools, etc.

Estimating each functional item

Now that you have the functional understanding, you need to come up with the tasks that you will do to finish this functional item. To get a better understanding, I list down the following:

• Understand if it involves both backend and frontend work
• How many screens you might need to create
• Complexity of each screen and number of components on each screen
• New APIs you might need to create
• Business and data level validations
• At a high level what will be the business logic
• Database schema changes required
• Any API integrations you might need to do
• Calling out failure scenarios
• Unit/Integration testing effort

Let’s take an example. Suppose that we have to estimate a functionality that involves customer registration using the mobile number. This will involve an OTP verification process. I know there are three screens to build: one for entering a mobile number, second for OTP entry, and third for successful registration. We can list down the number of components required. We know we will have a customer registration API. Also, we have to talk to the OTP service. We will need to create a customer in the database. We will add effort for failure scenarios like OTP invalid or not received in 30 seconds or OTP service not available. Just listing down these things gives us enough idea on how much effort will be required.

For estimates I use the fibonacci series – 1,2,3,5,8, 13, 21. I don’t go above 21.

There are a few other techniques to do software estimation.

• Evidence based scheduling
• PERT – Link
• Three-point estimation – Link
• Probabilistic forecasting – Link

In this post, I will cover how I do software estimates. Below is the screenshot of the estimates template sheet.

The interesting thing that you will note is the use of the uncertainty factor. This is something I learnt from Jacob Kaplan-Moss post on software estimation. Earlier, I used to use the PERT estimation technique, which talks about having three estimates: most pessimistic (P) case, most optimistic (O) case, most likely (M) case. I liked Jacob’s approach to software estimation better. So from the last couple of months, I have started using it. I add uncertainty on the item level: so, if the uncertainty factor value was 2, then Total PD will become 22. If you want, you can add uncertainty at each technical development effort as well. I personally don’t go beyond 2 for the uncertainty factor. Also, I only add uncertainty to items that I am completely unsure of. Typically, these are less than 30% of the items.

Things to keep in mind when estimating a functional item is that it is more than coding time. So, keep the following in mind.

• Coding time
• Handing failure scenarios
• Rework time
• Cross cutting concerns or reusable functionalities you will have to build
• Integrations you will need to do
• Learning time

Adding effort for testing and other automation activities

As I said in the previous point, software development is more than coding. To come up with the actual estimate we need to include many more things that are sometimes implicit. These include:

• QA time
• Waiting time
• Meetings time
• Deployment and support
• Dependencies
• Team productivity and holidays
• Documentation
• Learning time

I estimate three items: QA, DevOps, Misc. I estimate them in percentage of total dev effort. For example, if the total dev effort is 1000 PD, then I keep QA as 250 PD(25% of dev), 250 for DevOps (25 % of dev), and 200 for Misc (20% of dev effort). In total, this becomes 1700 PD. 

So, what about buffer 

I don’t add a buffer since I have already considered that as part of the uncertainty factor. If the delivery or project management team needs to add, they can add on top of it. The problem with buffer is that you don’t know what is the right value for it. In my view, each organization has to come up with the right buffer value to minimize risk and have some wiggle room for price negotiation.

Conclusion

In this post, I covered my process of doing software estimation. Software estimations are hard to get right. To improve your estimates you have to relook at your past estimates and understand the delta between your estimates and actual effort. Also, you need to consider it takes time to create an effective team. They will not be productive on day 1. You will need to keep the first month or so for the team to reach an optimum level of productivity. Keep in mind that it will never be 100%. The best you can target is 60%. This is one way that projects start out behind schedule from the very beginning, if the schedule assumes steady-state productivity right from the start.