Pre-requisites: Agile Definition
The Agile Manifesto has four statements that condense a lot of wisdom into few words. These statements can guide the culture change in the organization.
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
Here we will learn about the twelve principles behind Agile manifesto¶
Over the years I have had the opportunity to coach executives, PMs and teams. At each organization the agile transformation journey taught me a lot about what works and what doesn’t work. The examples used here are based on my experiences and conversations with fellow agile practitioners. Names, places and specifics have been changed to maintain anonymity. Any resemblance to actual persons or actual events is purely coincidental.
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software¶
Agile methodolgy recommends breaking down large requirements into thin vertical slices and get it done. By delivering early the team can get feedback from the customer and make changes to better align with the needs of the customer. Sometimes the customer has just a general idea of what they need, by having access to the product increment it helps the customer refine what their goals are or make changes if the market conditions have changed.
I was on an Agile team that developed internal tools used in critical business functions. Within three cycles the team completed a high priority feature and delivered it to the customer. This was the first feature delivery as an Agile team and the team was proud of the accomplishment.
A couple of days later we heard from the customers (business users) that there was a recent company directive that all internal tools that handle sensitive data must meet certain compliance requirements. The customer would not be able to use this feature and wanted the compliance requirements to be addressed immediately.
We re-prioritized the work planned for the cycle as it was felt that this was valuable to the customers. The team worked together to review the compliance requirements and understand exactly what was needed. Not all compliance requirements were applicable for this feature, the backlog was updated with the must have items and the team completed the work within the cycle. The customers were able to use the tool and commended the team for the quick turnaround.
Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage¶
Agile teams take advantage of every opportunity to better align with customer needs. The incremental planning and just-in-time backlog building approach makes it easier to adapt to changing priorities.
My team was working on a feature that customized the experience for each user. Depending on the user role a workflow would be presented that was tailored to that role. This would enhance the customer experience and they would be able to get their work done faster. Personas were created based on interviews with the end users and workflow documented. This feature required a major redesign of the product and would take several cycles to complete. We had started implementation of this feature and built a barebones workflow for one role. This was presented to the customer and the customer was satisfied with the implementation. The customer informed us that this would be a big differentiator for the product and the end users were excited about this feature.
A few weeks went by and we had completed barebones workflow for all the roles. At this time we heard from the customer that the priorities had changed and that role-based workflow was no longer important and would probably be removed as a requirement. They had been acquired by another company and the priority now was interoperability with the systems of the company that acquired them.
We started discussing the requirements for interoperability with our technical contact (customer) while the the contract was being re-negotiated. The legacy systems that this product was to interoperate with required files to be ftp’ed to a particular location in the specified format. It was a one-way data transfer mechanism. The team figured out that they could use simple transformation classes to convert the data into the specified format and upload the files. The team wrote an independent service that collected data from our product and generated files ready to be uploaded. We made sure our customer account manager had this information while discussing the contract. Since this would require just a litte more effort to do integration testing, we offered to build this service free of charge. A week later we learned that the customer was delighted that we had worked to understand and build a solution for interoperability. Their technical person had informed them that the solution would meet the needs. They were relieved that interoperability was much simpler than they anticipated and thanked us for figuring it out quickly. The contract was revised to include this new service and workflow requirement was retained.
After we delivered the product we heard back from the customer that the end users were happy with the workflow feature and they had seen additional benefits such as training new staff was easier with the workflow based solution where only the information relevant to the role was presented.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale¶
Delivering software frequently and early is a skill Agile teams have to build. Shorter cycle means more opportunities to deliver value and align with the customer goals. A shorter timeframe requires the team to become comfortable with breaking up work into slices that can be delivered within a cycle and have a build/release pipeline that supports faster delivery. The team should adjust the timeframe based on ability rather than because it looks better on paper.
Since we have already coverd how shorter timescale helps, let’s look at how teams arrive at the right duration for the development cycle.
This happened several years ago, the Agile transition was well underway and the teams were improving on delivering frequently. One of the teams decided to shorten their development cycle from 4 weeks to 2 weeks. Shortly after several teams decided to move to 2 week sprints. My team was on a 4 week cycle, we were asked by leadership in one of the review cycles why we can’t go faster like the other teams. Later i discussed with the team and the team felt that the 4 week pace was best suited for our team. I suggested that we try it for a cycle and if it does not work we can always swtich back. The team agreed and we adopted 2 week cycle, at the end of the cycle we met to discuss if this worked for the team. The team had been able to deliver only a few stories to production, this was due to the nature of work that fit better in a 4 week cycle. We switched back to 4 week sprints and informed leadership about our learnings.
A few weeks later I learned that a few of the other teams who were on 2 week cycle were not happy. They were finding it difficult to break down work items to that level and had work items that were moved from one cycle to next. The metrics indicated that their velocity had fallen since moving to the 2 week cycle. They were thinking of switching back to 4 week sprints. There were other teams for whom the 2 week sprint worked well and they were happy with the shorter cycle.
The lesson here is that shorter time scales are good but if they are too short then the teams will struggle to maintain a sustainable pace. Teams should not be pressured to adopt shorter and shorter duration, instead encourage them to find the shortest cycle that produces the best outcome.
Business people and developers must work together daily throughout the project¶
For organizations tranistioning to Agile, making business people available to the team daily is very challenging. There are organizations who do this right, however may organizations ignore this and teams are left with struggling to get clarity on requirements or receiving requirements at the right time. Sometimes it is seen as a waste of resources to allocate a business person or product owner for each team.
I have to admit that it has been tough to get leadership buy-in for this simple idea: Teams need access to a business person to be most effective. There is a lot of talk about customer collaboration but some organizations just don’t get that the team needs access to a product owner in order to get clarity on requirements, prioritize work and provide feedback on delivered software. Delay in getting the right information to the engineers could result in missed goals, failure to meet requirements or low value items delivered before high value items.
At a company i worked we had a team of brilliant engineers but the team was struggling to get work done and the PM on the team was frustrated because the team had to wait days or weeks to get a response from the business person. The team was new to the domain and needed a business person to explain the specifics of that domain. The team escalated to leadership regarding the need for a product owner to work with the team.
There were only a couple of people with deep knowledge of the domain and they were overseas. They already had responsibilities that kept them occupied full-time. The product owner was trying to help out as much as possible. The team was asked to consolidate all requests and send only one email per day since the product owner was distracted by too many emails from different members of the team.
The team was unhappy and morale was quite low. The PM shared this with me and I suggested the team get together to discuss the options available to us. My intention was to get the team to come up with an answer and I would nudge the conversation along by asking questions. In the meeting I let the team members brainstorm and asked the team to use concrete examples from past cycles where they had failed to deliver. It seemed to me from the conversation that the team was accepting work items in a development cycle even though the requirements were not clear. This meant that the leadership saw the team was working on backlog items but not delivering working software. I asked questions to confirm my understanding. As the team continued to discuss, it became clear that they were aware of this but did not want to seem like they were being difficult. I asked the team to identify action items based on what the team could control. The team converged on a single action item: Only accept work items for a development cycle that were clear and had a proper acceptance criteria.
Over the next few cycles the team worked on and delivered all work items that were clear. For the items the team rejected, they added details on why it was rejected and questions for the product owner. This meant that the workload was lower for the team and the team invested this time in improving their development pipeline. The leadership understood where the bottleneck was and worked out a way to get the product owner to fly-in and sit with the team. The product owner’s other responsibilites were transitioned to a different person.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done¶
When you have a group of individuals that take ownership of work they do and have an Agile mindset, they will generate tremendous value. A team centric approach will keep the best team togther and provide them the support needed.
I have seen organizations move individuals in/out of teams casually. Every time a high performing team is dispersed, the company loses value. In my experience, I have come across many instances of teams broken up just as they had stabilized and were working well together to deliver outstanding value. This is where the right leadership can have a big impact.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation¶
When working on complex and challenging problems there is no better way to move faster than face to face communication. Email, chat, documents, wiki are slow and less effective communication channels that drag down team velocity.
Working software is the primary measure of progress¶
In traditional approach the progress is measured in phases, for example: requirements phase, design phase, development phase, integration testing phase and deployment phase. The customer sees no value until all phases are complete.
Agile teams deliver early and frequently. The progress is measured in terms of work that is done. The customer received working software from early in the project and in increments that add more value.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely¶
Agile organizations value individuals and work to build a healthy work environment. There are many ways to promote sustainable development including
- Create a safe working environment
- Decision making should be done by those who have the best information
- Use agile metrics to learn, don’t use it for performance appraisal
- Simplify communication paths
- Change hiring practices to focus on growth mindset
- Appropriate rest & recharge cycles
- Investing in tools and training
- Rewards & recognition
Continuous attention to technical excellence and good design enhances agility¶
Most engineers i talk to agree 100% with this principle, however they also point out that they are often asked to take shortcuts and discouraged from pursuing technical excellence. Engineers want to invest time in creating a good design and doing the best possible work.
From several years of talking with engineers, PMs and leadership i have come to a conclusion that “technical excellence” has different meaning for each group. Moreover the ROI calculation is done differently by each group.
The technical choices i made when developing this website impacted my ability to regularly add/update content. When i started working on this site i decided to use a templating engine because i did not want to build each page by hand and reuse as much as possible. The first version worked and i was quite happy.
Later as I started to write about Agile, I was frequently distracted by having to know which files to update and the mix of “code” with content. I wanted to focus on writing, it should be just as easy to write for the site as it would be to write a text file.
I had used Markdown before and found it to be a match for my style. I discovered Mkdocs and liked the way it generated a static site from Markdown. Before this change i had stopped writing and for many months the content was not updated. Now I enjoy writing and am looking forward to regularly adding content on this site.
A while ago i was leading a program that was saved by the engineers who persevered in finding a solution that satisfied the customer needs within the constraints imposed by leadership. The program objective was to build a customized product for a large customer based on an existing product. The existing product was web based and was in use by several small customers. The challenge for the team was to build a highly scalable solution that was cost-effective and robust. The senior engineers provided recommendation on using a cloud provider that allowed demand based scaling. The high level estimates put the completion date beyond the completion data that was promised to the customer and the cost would exceed the program budget.
I met with a few of the engineers to understand their perspective. The engineers pointed out that the project requirements stated that the solution should scale based on demand. They had evaluated several offerings and found one that allowed them to dynamically scale up and scale down rapidly depending on the traffic that was hitting the servers. The engineers were recomending the best solution to satisfy the customer and be cost effective.
I met with leadership and found that they had tried to find ways to increase budget but it was not possible at this time and it was absolutely critical to get it done within the timeframe. Leadership was willing to make some adjustments within tolerance but wanted the engineers to find creative ways to get it done within budget.
I worked with the PMs to find a solution. We looked at hybrid (cloud + on-premises) models. However, despite our best efforts there did not appear to be viable option.
The engineers and PMs got together to brainstorm with one ground rule - All ideas should be considered regardless of how wild it is. Everyone started pitching ideas and we wrote them on post-its and stuck them on the wall. There were several insights that emerged from the discussion, the most important was regarding the design of the application. The team had estimated effort based on the design of an existing application. That application was web based and almost all processing was done on the server. Based on the metrics from that application it was estimated that two minutes of customer interaction would generate over 200 requests to the server. One of the engineers suggested using a thick client instead to reduce the requests to server. The impact to customers was discussed. Would they agree to installing thick clients? What kind of machines the customers typically use and what software would it need to enable the thick client to work? There were still a lot of questions but a quick back-of-the-envelope calculation showed that server requests would reduce by over 60% and that could make a big difference to the cost of hosting with a cloud provider. In additon some of the processing could be done client side and make the experience snappier for the customers. Everyone agreed that this idea was worth pursuing.
After several weeks of extraordinary effort on the part of everyone involved we had the answer: Yes, thick client would satisfy the customer needs as well as enable the team to deliver within budget and time. The advantages included (1) the UI would be much easier to build for a single operating system than it was for a multitude of browsers with compatibilty hacks. (2) the server load would drastically reduce enabling each server to handle 10x - 20x the number of clients (3) a non-trival amount of information lookup functionality would be available when offline.
Simplicity–the art of maximizing the amount of work not done–is essential¶
This is my favorite principle. It has helped me get unstuck in many situations.
About a decade ago, i got an opportunity to work with a team that was piloting Agile. This team had architects, business analysts, senior developers and testers. I was eager to put into practice all of my learnings and prepared a roadmap of how i would transition the team to Agile. The team had completed Agile 101 training so they were atleast familiar with Agile terms. Early on the first day, the engineering manager (my new boss) said he wanted to meet. He started the conversation expressing how excited he was that the team was piloting Agile and then explained what the team did. He mentioned that the team was running behind by several months on features to be delivered and was under a lot of pressure to speed up the work. He hoped that with the Agile transition the team would be able to make up for lost time and be ready with all the features in 3 months for integration phase. I meekly nodded my head and said i would try my best. When i left the meeting, i felt anxious and uncomfortable. The real world was not going to be like i imagined. The roadmap i had prepared, which included a couple of cycles of low pressure work to encourage learning and do team building, was useless. All my plans about how i would take charge on the first day and start the Agile transition in a spectacular fashion went up in smoke.
It was early around 8:00 AM in the morning and the team would be in by 9:00 AM. I went back to my desk to start over from scratch, i had a litte under an hour to come up with a plan. I couldn’t think of anything so i started the browser and started searching for resources. Agile pinciples was among the search results and i started reading the principles to refresh my memory. I read this principle “Simplicity - the art of maximizing the amount of work not done - is essential” and it suddenly hit me - i don’t need to come up with a plan, i don’t need a roadmap, i don’t need the answers to all my questions. I just had to introduce my self to the team and get to know the team, that was going to be my task for the day. The day went well and instead of me telling the team what the plan was for Agile transition, i listened to the team. I learned a lot about the team, how it operates, the challenges and what they thought about the Agile tranistion that was imposed on the team.
Looking back, i am thankful the engineering manager for being very upfront and honest about his expectations. I developed an appreciation for the power of listening and that the Agile principles are not just words i have to memorize, they can be applied to solve real problems.
The best architectures, requirements, and designs emerge from self-organizing teams¶
Self-organizing is often undervalued by teams new to Agile. This is often because self-organizing requires change in culture. A healthy working environment where people feel safe to learn from mistakes, empowers teams to take ownership and results in delivering the grestest value.
A few years ago, one of the teams in my org was discussing the limitations of a critical internal tool they were maintaining, the original developers had left and over the years a lot of “band-aid” fixes were put in place to keep the system from completely breaking down. One day during standup one of the developers was sharing how he had to spend the whole night fixing an issue that would have been easy to fix if the system was designed properly. After the standup the other developers stayed behind and started dicussing ways to make this better since each of them had at one point or another felt the pain. By afternoon that day a consensus had emerged that they would make improvements to the system. Over the next few days the team members continued the dicussion and made a detailed plan. They informed the PM about the plan and wanted to include the requirement in the backkog. The PM tried to convince the leadership that this would lead to huge time savings in the future and would improve team morale, the request was denied as leadership felt that it would cause disruption in the currently planned work and result in delays in fixing issues. The team was informed and was asked to postpone the redesign/refactoring to a later time.
A month later the PM informed the leadership that the team had completed the redesign and throughly tested it. The team would like to deploy this to production. The team had done this by working late nights and weekends, not letting this impact their planned work. Unfortunately the leadership did not see this as a positive and pointed out that the team could have instead completed more of the planned work from the backlog. The team was disappointed and the energy level went way down.
I met the PM at a conference and during conversation he shared the story. I recognized that the leadership was exhibiting a fixed mindset in this case. I asked how confident the team was that this tool would meet the customer needs and how thoroughly had they tested the new tool. The team had built automated tests for the most common scenarios and manually tested all known edge cases, they had a rollback plan in case of any major issues. More importantly they felt they could fix issues within minutes since the design was drastically simplified and every team member knew the code well. They had also contacted the users and made sure the users were comfortable in using the new system ( the ui changes were kept to a minimum ).
So I asked him if their customers ( internal tool users ) had given the ok then what was preventing them from deploying to production? The PM thought about this and after a while said he would bring this up with the team.
Later i heard back from the PM that the new tool was deployed and had exceeded expectations. The team had considered all the facts and were ready to face the consequences if things went wrong. They went ahead with the push to production and after the deployment collected metrics on how the system performed. Armed with this data they approached leadership and explained the whole situation. The leadership had heard informally from some users of the tool that it was performing much better and they were happy with the improvements. Now they knew the reason for the positive feedback and commended the team on their ability to execute.
Self-organizing teams choose how to best accomplish their work and take bold actions to make it happen.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly¶
A team that does not learn & adapt is not Agile. When a team reflects regularly on how to become more effective, a world of possibilities open up.
A few years ago, i was having a conversation with an executive and the topic of agile mindset came up. He wanted to know the one question to ask a team that will reveal if the team has an Agile mindset. My answer was “Ask the team about tasks in the current cycle based on inspect & adapt feedback”.
A few years ago, I was mentoring someone new to Agile PM role. One day i got a call from this person seeking an urgent solution to a problem. The team velocity was going down and feedback from the team was that it was due to slow compile time since the code base had grown large. The team wanted to refactor the code to be more modular and thought that would solve the slow compilation problem. However there was a deadlilne they had to meet and could not afford to spend a majority of the current development cycle to refactor the code. The problem was that the date could not be changed and with the slow compile time the team cannot deliver by that date.
To get more clarity i asked what was the reason given for not moving the date? I learned that it was because another team had a dependency on the feature and they had to complete a high-value feature in the next cycle. I inquired about the feature that the other team was working on. He said he would find out and call me back after he had the information. I suggested that after he find out he should meet with his team and discuss the findings before giving me a call.
Later that evening, i got a call from a very relieved PM. The team had figured out a solution. Turns out that the other team was working on an feature and needed the API details from this team in order to complete the coding. The team members discussed and decided to build a mock service with the proper API signature. This would allow the other team to complete integration with the mock API and complete their feature. The team was happy that they could unblock the other team and do it easily. He said the team also decided that they will reach out to other teams who have similar dependency on API signature and determine if they should build a mock service so the other team doesn’t have to wait for the full service to be built to start integrating. The team learned and adjusted behavior to become more effective.
Next topic: Agile Manifesto