The story of how to quickly check your team’s practices while having a lot of fun.
How do firefighters assess their readiness for an emergency situation? How do they ensure that everyone knows what to do in case of a fire?
The best way to check this is to organise maneuvers similar to a real life situation.
Firefighters train regularly to check and improve their procedures. They are inducting new members, they are testing new equipment, and they are building trust to each other.
Nothing can teach you more than real practice.
Like firefighters, data scientists need to perform as a team. This means introducing ways of working to new joiners, making use of the best tools and techniques they have at their disposal, and knowing each other’s skills and personalities.
In this article we would like to describe how we organised an internal Kaggle-like competition to test and assess our data science procedures.
Our small data science competition
Recently, we organised a machine learning competition at Ocado Technology to check how well equipped our data science teams were to solve real problems under pressure. We invited data scientists from our five offices (i.e. Kraków, Wrocław, Sofia, Barcelona, and Hatfield) to our headquarters in the UK. We ordered some pizzas and started a hackday.
We formed into teams and adopted only one rule: if two data scientists are working on the same team in real life, they cannot work with each other during the hackday. We wanted to encourage people to get to know each other.
We decided to use a Kaggle style competition. For people who are not familiar with Kaggle, it’s a competition where business problems, data, and evaluation metrics are defined by the organisers. Participants then have to build ‘only’ the corresponding machine learning models.
The goal of the competition was to predict the total time-at-door for Ocado delivery vans.
We wanted to know how much time it would take to deliver groceries to a particular address. Ocado is using these delivery times to plan the van routes with more certainty and thereby open up more one-hour time slots for customers to choose from.
Similar to Kaggle, we prepared some baselines and a leaderboard where we showed the best solutions; this gave participants additional motivation to build something better than everyone else. At the end of the competition, the teams presented their findings and models. We learned a lot about our data, our practices, and ourselves.
It was a great day so we wrapped things up with the customary pint in the pub.
Five lessons learned after the Kaggle competition at Ocado
You can easily apply these lessons in your data science team or data department:
1. Hackdays are a great chance to socialise
There’s nothing like a competition to get people from different offices working together and therefore getting to know each other. People can learn about their strengths and weaknesses. We found problem-solving to be a great team-building exercise. After the event, we created a survey which confirmed that people indeed had a lot of fun.
2. Machine learning models are only the tip of the iceberg
As an organiser, you need to choose the problem wisely: it cannot be too difficult to solve in one day but still should be challenging. You have to define the evaluation metrics, gather data, split it into training and test sets, write down the rules etc. During his presentation at NIPS 2016, Ben Hamner (CTO of Kaggle) confirmed that his employees invest hundreds of hours in properly setting up the competitions behind the scenes. In all data science projects, only 5-10% of the time is spent on modeling.
3. Data science is all about iterations
During the competition, some teams over-complicated their models: they tried to check too many things at the same time and overestimated what is feasible to do during one day. At the end of the day, only working models really matter (all teams had plenty of ideas on what they would have liked to check but ran out of time).
It works pretty similarly in real life. We’ve written about this here as well.
Practice and contests like this can show your team the benefit of iterative work.
4. Domain knowledge can make all the difference
Rather than trying a more complicated model, it’s better to first invest energy into understanding the metrics, analysing data, checking the distribution and outliers. The team that won the competition used their knowledge about Ocado’s business to improve their model. In real life, very often domain knowledge is essential.
5. Improve your engineering practices
Python and R are two of the most popular programming languages for data scientists.
To work effectively, you need to know your tools very well, including programming languages and frameworks. If you want to rapidly check hypotheses or add new variables, you cannot be blocked by technologies.
This hackday showed us that we need to work harder on unifying our technology stack and adjusting the induction process to ensure that everyone can easily get the data, make the analysis or model and share their results with the rest of the team.
As we’ve seen, a one day hackday event can provide a very useful health check for your team. You can check how people organise their work, what tools they are using and how they are working to solve problems. But hackdays can be beneficial not only for data science or engineering teams; management teams can use them to decide training budgets, investment in tools and technology, or for forming new teams. We therefore strongly encourage you to involve your managers or team leaders in these events as much as possible.
Try to conduct similar competitions in your company. We assure you that you will learn much more than you’re expecting while having a lot of fun.
Lukas Innig, Marcin Druzkowski
Marcin Druzkowski May 18th, 2017
Posted In: Blog
data science, hackathon, machine learning, office life, teamwork
Recently I took the difficult personal decision to move from the business domain where I had been for over six years since graduation – Supply Chain – and into the technology, systems and completely different domain of Payroll.
Everyone around me has been hugely supportive in this self-learning exercise and my new team leader put it best: throughout their career and all the way until retirement, a successful individual will go from being a small fish in a big pond to being a big fish in a small pond. It’s then you need to reset and get in a bigger pond again and this is the only way you can continue to grow.
Introducing yourself and your own style to a new team can be difficult – as can learning the nuances of others and integrating into what is already a performant group. One of my new colleagues – Felipe – introduced me to the Steam game called “Keep talking and nobody explodes”, a collaborative puzzle game aimed at rewarding good inter-team communication skills. We thought we would give it a go!
The game features a bomb, obviously. But importantly only one person is allowed to see it. Each of the sections of the bomb is a different type of puzzle; for instance the bottom left module in this image is a vertical wire puzzle where some wires need to be cut and others definitely not cut to succeed.
The rest of your group gets the manual! For a simple puzzle it gives explanations such as:
If there is a single yellow wire, cut the third wire. Otherwise…
More complex puzzles may require you to remember previous steps you have undertaken as well as build up a virtual image of where you are currently based on what the defuser is telling you. We opted not to use any pens or paper to aid us since it would have removed a lot of the confusion that good group communication and building consensus can actually resolve.
Communication is the number one key to this game. The defuser needs to constantly and articulately express what is on their screen. The group with the manual must gather consensus on the right course of action through speaking but also listen keenly to the information they are getting from the defuser and each other.
Get one of these things wrong and you get…
Dan, Felipe and Laszlo posing after a successful defuse
No matter how good you are at this game eventually there will be explosions. Learning from failure is a key part of this game and seeing behaviour like blaming each other or getting frustrated would be an instant indicator of underlying problems.
Fortunately for us we saw none of that, and the session was filled with shared ‘oooooooooohs’ as we became unsure, claps when we finally solved a complex challenge, and admissions of ‘my bad, guys!’ when anyone inadvertently made the wrong call.
A circle… with a line through it…. with a T on top!
Laszlo pondering the names of the colloquially termed BT, Trident, right dotted C and smiley face icons
One of the puzzles involved four random symbols and the defuser has to select them in the correct order. Unless you’re a pro at greek alphabet knowing your lambdas from your phis from your dotted lunate sigmas – there is going to be an element of ‘I have like a circle with a line through it with a big T on top rotated at 45 degrees’. Oh you mean a washing line on a manhole cover!
This is a hidden gem of an opportunity where the team quickly build common language to describe complex situations whether you see a washing line in that icon or not. In technology, understanding what someone means when they say ‘visitor pattern’ is easy – you can read a book on it! Understanding what your fellow colleague precisely means when they say visitor pattern – in reference to the unique way in which it was implemented to solve a problem three months ago – that is something you’re going to have to learn by reading clues from those around you. This vital cross-knowledge is part and parcel of the ‘forming, storming, norming, performing’ cycle of team behaviour and practising the skills relevant to getting to ‘performing’ fast is vital.
Too big a room
Dan, Laszlo, and Dan trying to help Paul defuse the bomb
We conducted this experiment over 1.5 hours. For the first hour it was five people in a room designed for 12+. We thought this was perfect, each of us spread out the paperwork for our manuals and got down to picking up the relevant bit of the manual at the relevant time and working collaboratively to solve it. Quite often a leader would emerge – someone who had read a little faster or found the page a little earlier or perhaps just understood how to read a Venn diagram a bit quicker than the rest. There was a bit of consensus gathering but mostly we went with the first answer to get to the defuser.
In the last half hour we were in a room really designed for six. People couldn’t spread out as much and at one point three of us were gathered around a single set of manuals with much more consensus being driven and more people being able to keep up with where we were on a challenge.
It is very hard to not be heard or indeed to be timid with your ideas when you are all in an appropriately confined space. I contrast this with the often vast meeting rooms that we use for Sprint retrospectives or the likes that require everyone to turn their heads constantly 60+ degrees to see if people are in agreement or not and whether everyone has had their chance to get their points across and will have to consider more often in future to book much smaller rooms wherever possible.
Breaking the ice
By the end of today’s session it was very easy to feel an integrated part of the social structure that makes up a team. I had the opportunity to see a great deal of situations in which people could have gotten easily frustrated with one another but actually all were willing to correct or be corrected by their colleagues. Work is often as much a professional as social grouping of like minded individuals and I find I am already building my appreciation for the talents and articulate nature of those around me.
As team building events go, a meeting room for an hour or so, an £11 game on Steam, and a few black and white printouts is a pretty cheap and surprisingly successful way of integrating a new member into the team dynamic.
Daniel Stoner May 10th, 2017
Posted In: Blog
collaboration, games, office life, puzzles, Steam, teamwork
In my team we conduct a retrospective every two weeks. On each occasion, we try to keep things fresh and stop the same format from killing our ability to spot and discuss a growing problem.
We were inspired by a paper on rotational cleaning and how regularly changing your cleaning process avoids bacteria strains resistant to your standard approach becoming common place. Although our usage is very different, the same theory can apply.
We begin each retrospective with a game to help mix things up, keep us energised, and make sure we’re alert and learning. Here are some you can try yourself.
The game: Trolling your team on the requirements
Dan wants a new car. Dan wants you to draw him his new car. All that he knows is he is looking forward to it and it’s going to satisfy all his dreams.
Take a whiteboard marker and design Dan’s new car for him.
Watch as the team make assumptions, draw incorrect conclusions, and only infrequently think to involve the original requester in the process. Maybe fast feedback and awesomeness come intuitively to you; I’ve a feeling for a lot of us this is something we have to remember to work at, hence this game!
How did the car end up? Was it exactly what the requester wanted? I suspect not, so highlight where assumptions were made and why they were incorrect. Ask the team what policies they could try out to make the next project a success.
For example, vague questions get vague answers. Try out a policy of deciding in the team exactly what is important to understand about the project, then be sure you get the answers you need to continue. A vague answer shouldn’t lead a team to decide (or assume) for themselves.
Did anyone ask if the requester liked the design before it was finished? Involve the requester in the design process.
Repeat the process. Now Dan wants a new house. Go!
The moral of the game:
We can all get a bit comfortable and complacent in our given domains, but it is important to remember that making assumptions about what a business user might want leads to a risk: the risk that the thing we build isn’t the right thing.
In my team, we found it did not come naturally to have an approach that restricted room for failure, so we decided it was a good idea for us to make feedback opportunities part of our standard process.
Definitely not the car of my dreams, despite the heart logo on the door which my team was sure I would love
Not far off the modernist home I would love to have – but what is with that fountain I didn’t ask for?
The game: Trolling your team on the estimates
Whiteboard markers out again. The task is to draw up to five cogs such that the cogs’ spokes multiply together to make X.
There should be a phase for planning and estimating how long the task will take, and then someone has to complete the exercise while you time them.
These cogs would make 8 * 6 * 6 = 288
We played for various values of X: 128, 34, 71, 260, 437
So how did your team do at estimating the time it would take? Try to discuss how people did in each instance. For example:
- What factors did the team take into account for their time estimate?
- For the first attempt, how much of that estimate was safety margin given no prior experience drawing cogs?
- Did they allow less time if they thought the number factored into a simpler equation?
- Does the number of factors really make a big difference to the time it takes to draw the cog?
We found it interesting to time how long was spent in the deliberation + estimation part of the task as well. Surprisingly, for a 37 second implementation task, it could take three minutes of deliberation before anyone even picked up a pen.
The moral of the story:
Estimation is tough! And estimates via triangulation aren’t always sensible. For instance, if people consider X=71 to be in the same class of problem as X=437 then they will scale up the time estimate and fail with disastrous effects. When we drew 437 spokes on a cog it became hard to keep track of how many we’d got up to, and the drawing had to start afresh at one point. Eventually fatigue set in and this cost further vital time that had not been factored in or considered.
As our team continued we found that, for numbers which were simple to factor, the team got more and more accurate at estimating. For more complex numbers we seemed to get worse. Often, this was because a big chunk of the work for more complex numbers was drawing a cog with many spokes (a cog with 437 spokes, for instance, because we couldn’t think of any way to factor that into smaller numbers). Yet even realising this we continued flip-flopping between vastly under and overestimating this type of task.
Any other ideas for retro games?
We have built up a good portfolio of fun games to play, to fill everything from 30 seconds to 30 minutes, but we would love to improve this collection. Share your ideas with us on Facebook, Google+ or Twitter.
Daniel Stoner, Senior Software Engineer
Daniel Stoner May 2nd, 2017
Posted In: Blog
games, office life, product ownership, retrospectives, teamwork
Leave a Comment
Productivity and team size
What is the ideal team size? Many suggest seven people – give or take one. Small teams in my experience are more efficient and need less management oversight. Recently our Business Planning Systems (BPS) team reached 10 people. We started to experience a declining trend in productivity, so we had to rethink how we were structured.
But it was not just team size
Our productivity decline was not just limited to team size. We were also struggling with having applied an emerging design model, new requirement process, and a lot of new team starters.
What could we do?
I did not want to reduce the number of team members, we have so much work to do! I also didn’t believe the solution was to switch to a new Agile framework and I didn‘t want to split our product backlog. This left a few options to experiment with:
- One product backlog with two sprint backlogs and two teams
- One product backlog with one sprint backlog and two teams
- One product backlog with one sprint backlog and two or more self forming teams.
The BPS team lead had pondered on this for a while and together the decision was made to go with Option 3: one product backlog with one sprint backlog and two or more self forming teams. We ran the experiment for four months at the start our winter roadmap.
The difference with Option 3 was the team could form as many teams as they felt necessary to work off the one sprint backlog. For example, two teams equally split, two teams formed of six and four people, or three teams formed of two, four and four people. This also meant a change for the BPS team lead; he remains team lead but delegates more accountability to the team.
- Team is empowered to take action, without team lead and PO involvement.
- Cross skilling of team members increased. One sprint you might be working with Fred and Steve and the next with Larry and Barry.
- It was far easier to get new starters up to speed in smaller groups.
- We retained one weekly entire-team ‘grooming’, so everyone was familiar with the business requirements and backlog.
What we monitored
- Would teams always form into the same group to work on the same type of issue, thus impacting team dynamics and cross skilling?
- Velocity of work coming out of the team.
- Number and type of issues identified in retrospectives.
- Team happiness.
- Delivery to business and their satisfaction with outputs.
How did the model work?
- Velocity of work continued as normal.
- Commit-to-sprint backlog stayed as normal.
- At sprint planning Phase 2 the team discussed stories and how to structure itself to best service those stories. For example, three stories involve UI so group those, three other stories focus on refactoring, security and support so group those.
Now the team could decide who wanted to work in which group. It is the team’s responsibility to make sure there’s a) enough work for the number of people in that group b) enough people to work on the stories in that group.
What else to address declining productivity?
I mentioned it was not just team size affecting productivity, but also applying an emerging design model. By not focusing on future architecture for complex features, we were spending considerable time refactoring and make compromises. We decided to trial an opt-out-of-sprint option.
- A team member could choose to opt out for a sprint or some of the sprint, to focus on an issue.
- We would reduce our expected sprint velocity for the upcoming sprint, based on the amount of time the person would need.
- At the end of the time allocated the team member would present their findings to the entire team, and we would then make a number of decisions i.e. work on the issue now; work on it later; the solution didn’t work so look into an alternative next time etc.
- There is no restriction on how often a team member can take the opt-out-of-sprint option. In the last four months only two people have chosen to do this and both times it was very valuable.
What was the end result?
Success! This is now how we work. The team loved the new approach and productivity improved. Disclaimer, to my knowledge the team did not break out into dance moves like the below image, but I like to think they did…
To wrap up, we gained the benefits of working in small teams but also of moving as a collective towards the one overarching product vision. Job well done team!
Ali Major, Product Owner
Ali Major May 6th, 2016
Posted In: Blog
Agile, backlog, management, product owner, productivity, sprint, team size, teamwork
Communication is important. Not just for a Product Owner, but the whole team. Especially if it’s not a collocated one.
Here is an excellent test for your team: a fun exercise that will uncover communication issues that nobody was aware of.
- Time needed: 45 – 60 minutes (30 minutes for the exercise + time for discussion).
- Number of people needed: 4-6.
- Divide the team to 3 groups.
- Groups A and C go to different rooms.
- Group A gets a picture only they are allowed to see (a childlike drawing works well).
- Group C gets a pen and a sheet of paper.
- Group B, called ‘Runners’, stays with group A for now. Their task is to communicate between Group A and Group C.
- Communication exercise-01
How it works:
- Group A and Runners discuss what’s on the picture. Runners can ask any questions they want, but they can’t see the picture and they can’t take notes. Group A can see the picture at all times.
- Runners move to Group C and convey the information.
- Group C draws a picture based on what they are told. They can ask additional questions about anything.
- Runners come back to Group A to ask for more details.
- Steps 2,3,4 are repeated as many times as needed until the time is up.
- The person facilitating the exercise should spend some time with each of the groups, silently observing.
- At the end, ask the teams how well they think they performed.
- Show both pictures to everyone and discuss. What went well? What could be improved? Any actions?
Some things to pay attention to:
- Does the team start with the big picture or the small details?
- Does the team optimise the way they work during the exercise?
- Is the team aware of time and how fast they are progressing?
- Is the team focussing on the most important elements?
- What assumptions are made?
How my team performed:
Drawn by the team
What we observed:
- The team jumped straight into details. Only 10 minutes into the exercise did they discuss the proportions of some key elements.
- Runners split after the first exchange to save time. They didn’t, however, communicate between themselves, which meant they asked the same questions twice. Surprisingly, they sometimes got different answers to the same questions, which required extra communication.
- Group A also split, so they could deal with Runners separately, which resulted in one member not knowing if the horse was going to be present on the team picture at all at the end of the exercise.
- The team didn’t plan the execution for the time given. This caused rushing at the end and asking for extra time to complete.
- Some key elements were described in detail, while others were very basic.
- An assumption was made that characters on the picture sit at a table. It was quite quickly rectified though.
- The overall result was good, but the team didn’t think so.
- The team enjoyed working together and want to repeat the exercise in a few months to see if they can perform better.
- The team is proud of their work.
It was definitely great fun and a good learning experience for the team. Try it yourself.
For more ideas about communications in non-collocated teams, see Ali’s posts, and check out my own blog for more product ownership ideas.
Thanks to my colleague, Lukasz Kucharski, who introduced me to this exercise!
Anna Miedzianowska, Product Owner
Anna Miedzianowska January 26th, 2016
Posted In: Blog
case study, communication, effective organisation, management, product owner, team building, teamwork
My team designs tools for our delivery drivers. We were pretty confident we knew the challenges they faced and the solutions they needed. But what if we were wrong?
Here’s how we made sure our products actually make their lives better!
We started our research by shadowing users. All team members spent a whole day working with the people who are using our product – internal users – our delivery drivers. It wasn’t training; we were actually helping the drivers deliver shopping to real customers.
We learnt a lot that day: about the users themselves; about the delivery process and how it is really handled by the system we provide; and a bit about the customers. We felt like experts for a while, even though it was very limited qualitative research.
We decided to put our expertise to the test and invited an experienced user to a workshop with us…
Our first task was to reverse our roles (kind of).
So, this is your job:
We asked Mike-the-user to pretend it was his first day on the job, even though he’s really spent the last 10 years doing it. All he was supposed to know was that the role involves delivering goods to customers.
Next, we asked the dev team to pretend to be his line managers and explain the job to him, step by step. Mike was a curious student, so he was allowed as many detailed questions as possible.
It started well, but very soon we got corrected: nope, that’s not how it works…
We were missing details, got some procedures mixed up and couldn’t answer some of the questions, even about our own system! We also discovered how our system can be misleading if things go wrong.
We are not on the same page
We now know that even by shadowing users, we can still misunderstand things. Especially if things go well during the day, we can’t see the whole range of issues our users face. Also, we are more likely to believe that any problems that occurred during our experience are happening more often, when in fact, they might be statistically rare.
So we are not on the same page. And we will never be. But that’s ok, we are very different people and have different skill sets – that’s why we have different jobs.
What we should do is work towards minimising this gap by educating ourselves about the users, and also educating our users about our job when possible.
- The meeting is more engaging for everyone
- We actively think about the process (as opposed to passively listening about their job)
- Misunderstandings are spotted quickly
- We build a relationship with users
Try it yourself – explain to your users what their job is. You will discover a lot of surprises.
Anna Miedzianowska, Product Owner
Read more about product ownership on my blog https://productownerblog.wordpress.com/
Anna Miedzianowska January 8th, 2016
Posted In: Blog
product owner, team building, teamwork, testing, user research, workshop
In this blog I’ll show you how to create simple, easy-to-follow roadmaps for the whole team to buy into (and stick to!).
My roadmap philosophy is ‘plan to achieve specified deliverables over a defined time period. This plan is dynamic, and goal and data driven (measurable)’.
A few key points:
- My roadmap is NOT completed in isolation but involves other Data Department Product Owners (POs), the Business and my development teams.
- The Data Department roadmaps span a three month period (we call them seasonal roadmaps).
- My roadmaps must take into account: legacy, new development, and support across multiple stakeholders.
- It’s dynamic, kept up to date, and not hidden away.
Step 1: Time to think
In the month prior to the start of the new roadmap, I ponder what is it we should be working on to meet our Data Department, clients and Ocado Technology requirements. I create a draft and run it by my development team lead and the Business representative/s. This is all high level and, importantly, not the specifics of how we will do the work.
Step 2: Stop pondering, draw circles
After we, the Data Department POs, finish pondering, we schedule a circle session (I love this exercise). We each explain what we think we should be working on and jot this on a board in one or two words.
Where we have the same theme – for example, monitoring – we circle this with a coloured marker. Each colour represents one of our teams. We also discuss dependencies and mark these accordingly. At the end of the exercise we can gauge:
- Range of work proposed and if we are trying to cover too much
- Common themes e.g. we have lots of circles around monitoring
- How many of these themes are dependent on other teams
Step 3: Finalise and socialise
Next is to tweak and socialise the draft roadmaps with the development teams and relevant stakeholders.
During the finalisation process I do four more very important things:
- A retrospective on how we performed against our previous roadmap, and what lessons-learned we should concentrate on during this new roadmap.
- Story Mapping to break into epic and stories, the ‘how’ of achieving the goals and features (the Post-it note company must love the take up of Agile…).
Once I have prioritised the themes, the team then rank the epics and stories into order for working on. Then we estimate time based on a comparison to other epics.
I disagree that this is a non agile approach. They are very rough estimations, done quickly. We will in the future groom the epics and stories to a Ready state when we get closer to needing to work on them. The estimation provides us with an idea of what we reasonably think we can commit to on this roadmap, for example we are asking the question can we do story C within the three months taking into account story A and B?
Lastly we jot down story acceptance criterias AND we play some games throughout the process. I suggest Pictionary as it lets you know who not to work on your UI!
Sometimes I am in the difficult position where I know one of our key features is lacking detail to adequately break into epics and stories. And if it’s not clearly defined, we shouldn’t work on it, right?
In reality, sometimes you just need to get on with it, so below are approaches I’ve taken in the past to mitigate the issue. Remember to always keep the stakeholders in the loop, and the team must agree.
- Add feature to roadmap and allocate a percentage of time during the three months. We postpone breaking the requirement into epics and stories until I have the missing information. Then we storymap, estimate what we think we can fit in, and update the roadmap because (repeat after me) the roadmap is dynamic.
- Add to roadmap with target metrics but agree between all stakeholders that this may not be achieved. As more information comes to light we update the roadmap.
- Leave it off the roadmap, with stakeholder agreement to add it to the next roadmap instead.
A dynamic roadmap should not mean mission-creep
By ‘dynamic’ I mean it stays relevant. It is a living document and kept up to date. I use the roadmap as a tool to push back, but I don’t become inflexible. The highest priority tasks can always be done.
My roadmap means I can identify the opportunity cost (e.g. feature X won’t be completed), and I can measure what changed and what was impacted during and at the end of the three months. It is a balancing act.
What’s crucial is communication, communication and, one more time, communication. I make a call based on discussions with senior managers, the business and the development team. It is also imperative that once the roadmap is updated it is socialised again.
Only three months? What about the long term vision?
Focusing on longer terms goals, but delivery in small chunks with metrics to measure against, works best for us. It allows flexibility where technology or goals are shifting.
I know many disagree that a roadmap should be less than six or even twelve months. It’s important that my roadmap goals are linked to the Data Department’s long term roadmap or the long term Business requirements. Both my products and the Data Department have far reaching vision statements, so we know if we are heading down the wrong path.
What my roadmap looks like
I use Roman Pichler’s Go Template. It is simple and doesn’t come with preconceptions like a Gantt chart does.
I was so proud when, for the first time, I could cut and paste directly from the Data Department long term goals roadmap into my seasonal roadmap! It seems for many POs their roadmaps don’t have a direct correlation with the next level above, or the level above that etc.
Roman Pichler source on roadmaps: http://www.romanpichler.com/blog/working-go-product-roadmap/
To wrap up
A roadmap is a great tool, but you need to find what style and process works for your team. Keep it dynamic and refer back to it each sprint planning.
Measure your success and enjoy crossing out deliverables as they are done.
Ali Major, Product Owner
Read more Product Owner advice from Ali
Ali Major January 5th, 2016
Posted In: Blog
data, organisation, planning, product owner, strategy, teamwork
We all know that what developers want to do with their time is write code, but in Logistics Planning we took a whole week away from that. So why? And was it really worth it?
I began working for Ocado Technology in October. In this short time we have moved from basic principles to build a big-picture overview of our product that the team understands and buys into, and which adequately meets the complex business needs. This is how we went about it.
Our journey started with a set of a challenges which included missing domain knowledge, new starters to the company, and the fact that the Logistics Planning was performed in a very large set of a complex Excel document.
We had to overcome these challenges in order to build a high level overview of our product and proceed with development effectively.
Additionally we did not want to replicate the existing Excel documents but rather provide a software product allowing users to perform the end to end actions. And just to make things more complicated, the development team is located in Krakow, potentially a real practical barrier when the business has its head office in Hatfield.
In order to build a mutual understanding of the challenges and to bring the development closer to the business, we organised a set of workshops with key users and the development team. Our goal was to create an environment where our combined brain power could be used to drive the product development, instead of relying on a single point of contact.
To start the workshop, we had to define the relevant group of people responsible for planning deliveries. During the workshop, Operations and Resource Planners explained how customer goods are delivered and planned. The business team highlighted the most important challenges and the frustrations during their day to day activities. Additionally the development team was able to ask questions to gain a better understanding of the problem domain.
Each workshop was finalised with a summary session, where the development team and I built a picture of the challenges. This included the processes that they are currently experiencing while using our systems, and the boundaries of what the system can do. When the set of workshop sessions were finalised, each development team member went for a half day visit at a CFC (our massive warehouses) to spend time with the planners, observe the way they work and experience their challenges.
The cycle of workshops took us a week, after which the whole team had enough confidence and domain knowledge to proceed with defining the Product Scope and Roadmap. The approach we took was to define a scope of work and concentrate on the end to end user actions and behaviour.
Scope of Work
We kicked off our work on the scope with an aim to define and list all features and actions we thought our product should cover. We had a cycle of a sessions where the whole team thought of the functionality and reasons why we should (or should not) implement certain behaviours. The Product Scope brainstorm sessions focused on providing a product which would solve end users problems in line with defined product vision.
The whole activity took us three days. The outcome was: a set of updated mockups, a list of the features we want to provide, and a list of additional discussions and workshops we would like to have with the end users.
At this point the list (product scope) looked quite scary, so the next step for us was to define the strategy for our journey from the place we were, to the place we wanted to be. In other words we wanted to build a roadmap with backlogs which would allow us to build a great product.
To drive the evolution of our product, we needed to look at the prioritisation of features according to their value to end users. The key outcome of this workshop was to align our priorities and meet the needs of our end users. This process enabled us to divide the entire product journey for Logistics Planning into multiple phases.
Each phase would be finalised with feedback sessions with our stakeholders, therefore planning required us to define the goal of each phase, and the user experience we want to bring.
The whole set of roadmaps activities allowed us to build a clear vision for every phase, and proceed with the first.
We believe that the time was well spent. By fully emerging the entire development team in the problem domain, we have not only accelerated everyone’s understanding but have also evolved and matured the product vision.
Similarly, the scoping session allowed us to look at feature development with a focus on the product as a whole, bringing consistency and coherency to our product from a logical perspective.
Is this anti-agile? We believe the opposite: by having a strong vision and structure to our work, we believe that we will receive more structured and constructive feedback. Having goals and a clear strategy on how to achieve them will allow us to react to change and unforeseen challenges in a more meaningful way.
The reality is that we have software engineers, not simply ‘coders’; what we really love doing is writing valuable code. Sometimes the best way to do that is to take a significant break from the keyboard!
Lukasz Kucharski December 16th, 2015
Posted In: Blog
Logistics Planning, product ownership, roadmap, strategy, teamwork, workshop
You know when you get your internet shopping delivered and it doesn’t have a key product, and even worse, it has been substituted for an item that you will never use? Well that is what can happen when Supply Chain Systems go wrong. Do you know why Ocado has industry leading levels of order accuracy (99.3% of all orders in the last year we delivered as expected)? Our Supply Chain Systems don’t go wrong.
We have complex forecasting algorithms that accurately predict the demand of over 40,000 products across three warehouses for the next 28 days. The algorithms can predict seasonality, promotion uplift and can also predict how much a product will sell before we have even sold a single item. You might not know what you want for dinner next week, but we do.
Our Ordering Systems are placing just in time orders for products that cost over £2 million in total on any one day. We place almost 2,000 purchase orders a day to cover the demand of two retailers.
If we’re not predicting the future with our forecasting engines or solving complex assignment problems with our ordering logic, we are writing real time Availability logic to inform shoppers what is in and out of stock in milliseconds. This precision engineering is what we do every day and, if we get it wrong, lives* are literally on the line.
So how do we do it. The advantage Ocado has over store-based competitors is its data. We can draw on a significant amount of data to accurately predict what demand we will see over the next month. That data goes through several stages to clean up and remove outliers. Once the data is processed and determined to be valid, we can draw conclusions from this. We then use linear regression to project into the future and place our purchase orders based on this predicted demand.
I think the proudest achievement that I have been part of is being able to launch Morrisons.com in under 9 months. This involved the re-engineering of core parts of our systems, keeping Ocado.com growing and functional while adding a whole new business onto our processes and workflows.
The first thing I would say about my team is that we are all great software engineers, every single one of us. How we achieve greatness is varied and no two people get there the same way. Some are quiet, some are loud. Some are into gaming, some are into Disney theme songs (ok that last one is just me…). We are together but not the same, as the advert goes.
Looking around at them all now, I see two pairs of engineers solving problems together. Each is asking and answering questions posed by the other person (or even themselves). One engineer is ‘in the zone’: headphones in, look of concentration on his face, slight rocking of his head when he gets to a guitar solo… and I imagine is currently smashing out some awesome code that will inevitably get rewritten 15 minutes later. Two other engineers are engaged in a debate about the correct use of the word ‘literally’. This could go on for a while…
This is what is happening now. Give it an hour and we might all be in a room, debating the direction we should take on a product we are building, or listening to someone’s experience with a new technology. Give it two hours and half the team will be in the kitchen boosting their caffeine levels, the other half will be debating the correct use of the word ‘literally’. Come on guys, we literally just talked about this!
Does this sound like the type of team you want to work in? If so, apply through this role and hopefully we’ll see you soon:
Senior Software Engineer – Scala/Java
Will Peck, Team Leader Supply Chain Systems
*Christmas turkeys don’t grow on trees.
Will Peck December 1st, 2015
Posted In: Blog
career, forecasting, jobs, Morrisons, ordering systems, prediction, Supply Chain, teamwork
Next Page »
One of our largest areas recently undertook a revolutionary self-selecting restructuring exercise, the goal being a complete split between teams working on two different propositions. The overall aim of this split is to achieve greater alignment, autonomy and purpose.
Following the principle that a collaborative and distributed approach is often the best way to solve a complex optimisation problem, we negotiated moves amongst the five new teams subject to a set of constraints around team size and experience levels.
The plan going into the event was shared well ahead of time to allow people to get their questions in (and added to the FAQ section) and the concepts clear going into the day. Alongside this, we ran multiple townhall sessions where people could air their concerns and ask their questions openly. There was a certain amount of ad-libbing and practical adjustments on the day, but on a whole it unfolded according to the plan.
Fuelled by 18 pizzas, we completed three exhausting rounds of moves and peer voting. At the end of each round, we (everyone, including POs) voted on the viability of each team. From this we measured two scores: an intra-team score, and an inter-team score. This lead to a few interesting dynamics, for example one of the teams gave themselves a high intra-team score, but scored low on the inter-team vote. They then gave a pitch justifying their viability as a team, and were able to dramatically increase their inter-team score in the next round.
The first round was deliberately suboptimal, hence dramatically lower scores. This was to encourage a large amount of movement in the following rounds.
Essential to finding a viable solution was an appreciation from all of the ‘greater good’ of Ocado Technology. On the day, some people made some really big compromises in order to serve the greater good and allow us to form balanced teams that are all capable of smashing out quality software.
After the final round of voting we then took a quick anonymous happiness reading by each dropping a green, yellow or red lego piece into a box. Although they were not perfect, we were extremely pleased the results, considering that our original goals was “at least 50% happy”.
We will be keeping a close eye on the impact of the shuffle-up by measuring the things that matter most to us: throughput and team happiness.
In terms of the solution itself: I am delighted. Every team has a reasonable level of experience whilst a healthy number of people have chosen to change domain. It is a vastly better result than I could have hoped for had we chosen a top-down approach.
The very next morning we did a big-bang desk move:
James Lohr, IT Head
James Lohr November 26th, 2015
Posted In: Blog
collaboration, culture, office life, team building, teamwork