ShapeUp and managing a data science project

Introduction

What is the best way to manage data science projects? It’s a question that I’ve personally grappled with for years. If you have participated in data science projects for a while, you have also likely tried a few different approaches. Arguably, the most common approach, like other software projects, is to shoe-horn them into Sprints. In my experience, this rarely works well.

I think the main reason for this challenge is that the outcomes of many data science tasks are uncertain. This is particularly true for early phase work (the proof-of-concept projects), or when trying to explore open-ended problems where the literature is vast and consensus on the best method is scant or non-existent. Furthermore, uncertain outcomes are compounded by poor quality, or unavailable, datasets. Taken together, these problems make running a successful data science project difficult: it may go on longer than it should, it may go in the wrong direction, it may be stopped dead in its tracks at any stage, or it may be prevented from getting started altogether.

Moreover, apart from the above challenges, it is often difficult for business stakeholders to understand the data science process. Therefore, it is essential to establish an effective way for data science teams to communicate with business stakeholders, not only to ensure the direction is correct, but also to ensure expectations are accurate.

Over the last few years, myself and others in the Adarga team, have sought to answer the initial question: what is the best way to manage data science projects. Finding an answer to this required some experimentation, but I think we found an answer that works for us. Like in other places I had worked previously, we started off working in Sprints. Later, we tried Kanban, which provided a marked improvement over Sprints. Specifically, it allowed us to pursue longer-term research projects without artificially squeezing work into two-week blocks. However, there were still some troubling niggles Kanban didn’t solve alone.

Remaining challenges

As time went by, the team encountered limitations with our approach.  For simplicity, I will list them:

• Product management and leadership did not have good visibility into ongoing research, and the status of packages of work. 
• Estimating when open-ended research tasks would be complete is extremely difficult. 
• Without a specified delivery date for tasks, it’s difficult to communicate to other stakeholders when a given piece of research will be finished. 
• Without an explicit timeframe, it is difficult to make appropriate trade-offs in time and there may be a tendency towards perfectionism. 
• As tasks unfold, new requirements may be discovered, meaning the data scientist must switch from research mode to requirement gathering mode.
• Understanding of task priority can be fuzzy. With a continuous backlog there can be an expectation that we are working on all ideas at once. 

There are few opportunities for retrospectives.

At this time, it was fortuitous that we stumbled across the ShapeUp methodology, developed by the team at Basecamp. I was introduced to their management methods through an excellent podcast they did (which you can listen to here). ShapeUp was designed with full-stack web development in mind, rather than Data Science. However, with a few tweaks we adapted it to our needs. I'll say up front that we are not using it exactly as envisaged. For example, for day-to-day work, we kept some Agile processes including stand-ups, backlog refinements and retrospectives. But, in short, Shapeup helped us achieve the following:

• It provided an effective mechanism for the leadership and product management teams to influence and direct the work that we’re doing and ensured it was relevant to our users.
• Make the absolute best use of data scientist's time. More person-hours spent on applied research; less time spent understanding what the task is.
• A clear time frame to enable scientists to balance detailed research and prompt delivery. 
• Improved the visibility of the Data Science team’s work across the organisation.

In a little more detail, these problems were solved by following a three-step process of Shaping, Pitching and Building.

Shaping

Shaping is just a way of describing what work we want to do. A shaped project may contain one to several pages of information. The general properties of a shaped project are (from the ShapeUp ):

Property 1: It’s rough

Work in the shaping stage is rough. Everyone can tell by looking at it that it’s unfinished. They can see the open spaces where their contributions will go. Work that’s too fine, too early commits everyone to the wrong details. Designers and programmers need room to apply their own judgement and expertise when they roll up their sleeves and discover all the real trade-offs that emerge. 

Property 2: It’s solved

Despite being rough and unfinished, shaped work has been thought through. All the main elements of the solution are there at the macro level and they connect together. The work isn’t specified down to individual tasks, but the overall solution is spelled out. While surprises might still happen and icebergs could still emerge, there is clear direction showing what to do. Any open questions or rabbit holes we could see up front have been removed to reduce the project’s risk.

Property 3: It’s bounded

Lastly, shaped work indicates what not to do. It tells the team where to stop. There’s a specific appetite—the amount of time the team is allowed to spend on the project. Completing the project within that fixed amount of time requires limiting the scope and leaving specific things out. Taken together, the roughness leaves room for the team to resolve all the details, while the solution and boundaries act like guard rails. They reduce risk and channel the team’s efforts, making sure they don’t build too much, wander around, or get stuck.

Our interpretation of these principles led us to create an outline of what we expect to put in a shaped project document. These include:

• A clear statement of the business value of the project. 
• A clear specification of what the DS task is (e.g. what do we want to predict/optimise/find? What is the hypothesis we want to validate? What kind of API do we want to provide?). 
• A minimal lit review, enough for completing the project. This often requires the person who is writing the Shaped project to be aware of the wider literature to be able to suggest recommended reading.
• A clear statement of the most basic desired outcome of the task, and priorities for where effort should be spent beyond this basic outcome. 
• A timescale which will enable the team to meet the minimum specification and provide headroom to stretch further. ShapeUp uses two lengths: 2, or 6 weeks. So far, ours have ranged between 3 to 6 weeks.
• Prioritisation of the characteristics of the task, for example: do we care more about precision or recall? Are there particular misclassifications we must absolutely avoid? Does it need to have a certain throughput? 
• Clear boundaries of what is outside the scope of the project, identification of rabbit holes. 
• Assumptions.
• Risks.
• Dependencies.
• Data sources.
• Clear identification of a deliverable, for example: a component, a trained model with some experimental results, or a write-up of research. 

Pitching

The goal of pitching is to secure buy-in from the product team for the project, and to prioritise which of the available projects we want to work on. This is achieved in a pitch meeting, held at least once a quarter. Based on our recent experience the buy-in process is expedited when product owners have contributed to the pitch during its writing. For example, data science members write the technical aspects, and the product team write the User Stories and Business Value. Lastly, the elevator pitch we include at the very top (like a research paper abstract) is written in collaboration. This ensures everyone is on the same page.

The final decision makers, who can give the green light to the project, are the Heads of Product and Data Science. During the meeting it may be helpful to deliver a short ‘pitch’ describing the business value of the project and an outline of what it will achieve. Once we have settled on the projects, these become part of our objectives for the quarter.

Bear in mind this is only the second or third quarter we have tried Shaping and Pitching but to date preparing projects in this fashion has helped bring clarity and focus to our work.

Building

Ensuring tasks are completed on time comes down to giving the team the space to do their work. For teams working on projects, it is important to ensure that nothing else is added to their plate while they’re in this phase. Doing this effectively might require that there are some members of the data science team who are available for urgent tasks that crop up while projects are ongoing. 

Additionally, under ShapeUp, projects are not automatically extended if they overrun. There are a couple of reasons for this: to help avoid the sunk-cost fallacy, to force us to re-shape the project if we want to extend, and to provide some structure to our product/data science roadmaps.

As an aside, and a reminder of how important it is not to allow context switching, this paragraph from the ShapeUp book is very insightful:

When people ask for “just a few hours” or “just one day,” don’t be fooled. Momentum and progress are second-order things, like growth or acceleration. You can’t describe them with one point. You need an uninterrupted curve of points. When you pull someone away for one day to fix a bug or help a different team, you don’t just lose a day. You lose the momentum they built up and the time it will take to gain it back. Losing the wrong hour can kill a day. Losing a day can kill a week.

As for the day to day running of the project, such as concerns about tools / techniques for breaking down tasks and tracking them (Trello, Jira sub tickets, etc), these are left to the individual teams to decide. To help guide project outcome quality, we also developed a Data Science Definition of Done for acceptance of completed work (experimental, tools, production), and these criteria need to be met at the end of the project. 

Finally, the end of project provides an opportunity for retrospectives.

How is this different from Sprint?

In our implementation there are similarities due to adopted processes – however, the key differences are:

• Inside the sprint, the team is effectively writing their own ‘tickets’ (or trello cards, or whatever) rather than having a wider team + PM do it before the sprint starts. The reason for this is that the uncertainty inherent in data science (research work in general) means it can be difficult to plan out all tasks up front. Rather than an unconditional list of tasks, data science can be like having a conditional set of tasks (IF this THEN that). Therefore, the data science team members write better tickets during the project because they’re closer to the problem and are figuring out what needs doing day-to-day.
• The PM’s role, at least at Adarga, is to work with the Data Science Team Lead to provide the big picture direction. As we stated above, they contribute to the shaped projects by providing the User Stories and the Business motivation. They also provide feedback on project outcomes and support the productionisation of successful prototypes.
• The planning/derisking is conducted at a higher project level, rather than at an individual ticket level. We think this is more important for DS tasks, where the risks are systemic and existential! Risks in engineering projects can be well captured and mitigated at the ticket level for a lot of day-to-day tasks.
• The periods are longer which means we can tackle meatier projects.

Other benefits

To wrap up, some other benefits that our adopted approach has brought about, include:

• Improved morale and focus in the data science team.
• Improved awareness of data science work in the company, since the Elevator Pitch makes it easier for anyone to quickly grasp the “what” and the “why” of the project.
• Improved collaboration and transparency between data science and product owners.
• The ability to use Gantt charts effectively, because work is in time bounded boxes.

In conclusion, this is probably not the end of our journey of finding the best way to manage data science projects. However, the lessons learned from trying a different approach have been extremely valuable, and so far, the results are exactly what we were hoping for.