Alex Howard Whitaker

Welcome to the Ocado Technology Webinars, where you can hear from the people building the ground-breaking, game-changing technology that powers Ocado, the world’s largest online-only grocery retailer.

In this webinar Alex Howard Whitaker, cloud services engineer at Ocado Technology, talks about the challenges and opportunities involved in adopting a cloud-first strategy using Amazon AWS.

Key Takeaways

  • When Ocado Technology decided to adopt a public cloud solution, the main challenges revolved around automation, security and microservices deployment
  • Ocado Technology used an Agile approach to cloud adoption
  • Constantly building for failure helped mitigate security and performance issues
  • The cloud adoptions strategy was shaped by a set of well-defined best practices and guidelines
  • The growing list of technology stacks made it clear that a managed services-focused platform provided the best solution for the team’s specific needs
  • Creating an environment where the cloud infrastructure and cloud development teams worked together ultimately improved the overall platform and the tools around it
  • Ocado adopted Amazon AWS for operational services and the Google Cloud Platform for data analytical services
  • Having a separate AWS configuration for each customer made data segregation easier to manage
  • APIs are a great way to implement access control for applications

00:46: Starting from scratch means there are many choices and there is no right or wrong answer

01:08: Ocado Technology wanted to use AWS for existing systems and had to take an Agile approach to its implementation

01:39: Adopting a public cloud solution brought questions around security and performance

02:28: The development team looked at various cloud success stories, including Netflix

02:49: In order to ensure consistent adoption, the cloud teams created a set of best practices and guidelines

03:36: Ocado chose to use managed services offered by a service provider wherever possible to accelerate development without any downtime

04:08: The systems created to manage the cloud were also hosted in exactly the same way as the cloud applications themselves

04:34: The cloud teams evaluated both AWS and GCP and found the former better suited for front-end, operational services while the latter more focused on back-end, data analytical systems

06:08: Amazon AWS provides a myriad of services and there was a lot to learn about their individual characteristics

06:28: The first AWS implementation was relatively simple and straightforward

06:59: The second attempt joined the network hub account with a VPN back end

07:50: The third configuration aimed to decentralize various end points to improve access speed

08:44: The cloud team learned a lot from deploying live applications into the AWS configurations, particularly around data segregation, service limits and throttling

11:53: In the fourth version of the AWS implementation, the Ocado Technology team used the information gained from live deployment to create a more flexible configuration that could scale easily

12:19: The new architecture was based on microservices that used APIs to ensure abstraction of resources

13:00: Using access control and tagging to create better permissions for AWS applications

15:40: The architecture of our deployment process included an app registry, cloud provisioning, AMI build automation, cloud formation scripts and more

17:25: Using AWS Elastic Beanstalk, Ocado Technology deployed 250+ applications over a choice of stacks (Java, Python, NodeJS) and servers

17:52: Concluding remarks

More about our webinars

You can keep up to date with the webinars by subscribing to our YouTube channel. This article provides clickable links that take you directly to the highlighted part of the video clip.

January 9th, 2017

Posted In: Blog

Tags: , , , , , , ,

Office kitchen

When we kicked off our Wrocław office we had only around 20 people, almost all developers, plus the head of the office. One day, the head came to us saying the office needed a development manager. Someone to help run the organisation, move us in the right directions, overcome obstacles, and inspire people. He asked each and every of us to help with the recruitment. We were going to recruit our manager.

Where to begin

The first thing that came to my mind after getting this breaking news was, ‘cool’! I, a software engineer, would have a real impact on who my manager would be and consequently what my everyday work would look like. I felt the power! But then I realised that with great power comes great responsibility…

So I started thinking about how I imagined my ideal manager. What’s important for me as a developer? And how should I measure whether a potential candidate meets the criteria?

Obviously, we were not left alone with our thoughts. We scheduled an internal meeting to discuss what we wanted to achieve and how our recruitment process would look. Our mission was to find a person who would be able to create and support happy teams in a highly effective organisation.

A development manager is neither project manager nor architect for us. We don’t require deep domain and technical knowledge. Instead, we felt the person needed to fit well in our working environment and help move us another step in our direction of travel. Sounds good, doesn’t it? But how to define a happy team or a highly effective organisation?

Some details about the way we work

The company is distributed between multiple locations (Hatfield, Kraków, Wrocław, Sofia, Barcelona). We work in small agile teams. Some of them do Scrum, some Kanban, and the rest… ‘Scrum-ban’. A few teams work as squads, without formally assigned leaders. All of us have great autonomy  —  this is one of our core values. Development managers support teams and help them grow. They look for potential improvements and directions in which we want to develop as organisation.

One of the few constants in our company is a change. We like to experiment: not only with technology but with Agile and management practices as well.

What makes organisations effective?

For me, the two main ingredients of effective organisation are: effective individuals, and good cooperation between them.

Good cooperation requires even better communication. Free and direct communication across the whole organisation (between all levels) is now a mainstream trend in all kinds of companies. It’s easy to say but harder to practice. Creating a respectful environment in which people are not afraid of voicing their opinions freely, their voices are heard and listened to, and there is trust in good intentions, is not a trivial task.

And what makes teams happy?

Sometimes I find it difficult just to define what makes me, as an individual, happy – and there are six of us in my team. One thing I am confident of is that a team cannot be happy without each and every of its members feeling happy.

Management 3.0 gives us some universal tricks when it comes to making people happy. Here is the list of twelve steps to happiness:

https://management30.com/wp-content/uploads/2015/06/12-Steps-to-Happiness-v1.00-Poster-color1.pdf

Looks easy, doesn’t it? But is the list complete? Does it work for everyone?

To sum up what a development manager means to us

The role is more about helping others than managing resources. It’s about listening and observing instead of talking. And most importantly, about making people learn what they need to be happy and effective and how to achieve it. Helping them by giving tools, knowledge, tips, and the freedom to use them how they like, instead of enforcing ready solutions.

The recruitment process

At this point, you should have a better feeling how this role is defined in our company. Let me shed more light on how the team played a part in the process of recruiting our manager, what we learnt from it, and how it impacted our further work.

Every candidate was scheduled a team interview – a 1.5 hour meeting with developers in our company. The aims were to:

  • Decide if the candidate would fit well in our environment and if we wanted to work with them
  • Give the candidate an opportunity to meet their potential colleagues, learn about our working environment, and decide if working with us looked like fun

At the beginning of the meeting, the candidate was asked to give us a presentation about developing great teams. This was just an entry point to the further discussion.

The course of the meeting was driven both by us and the candidate themselves, and usually differed slightly with every session. Our idea was to make it as natural and interactive as possible.

Most questions were based on the presentation, however, we had a few prepared in advance (in any interview we always want to know what a candidate expects from the role, what’s their vision, and what value they can bring to the organisation). We were also interested in their approach to handling conflicts and difficult situations, development of people, projects, teams and organisation.

We also made sure that there was time for questions from the candidate. It’s important to bear in mind that the team interview was more focussed on relations with our future coworker than on checking their knowledge.

How well did it work?

The first time we applied this process was at the end of last year, and the first successful candidate joined us in March.

Since then, we’ve found it very beneficial to get candidates exposed to the broader group of people, to the whole team with whom they’ll be working. Every person is looking from a slightly different perspective, which helps us making more informed decisions, and it gives candidates a broader insight into the company. It also helps to establish a better relationship with our future colleague from the very beginning.

We’ve found it especially important in the case of leaders and product owners. The first day our new development manager came to work I felt I knew him already, which made the conversation very natural.

He had very similar feelings. Wojtek – our first Development Manager in Wroclaw – shared his thoughts about the newly created process:

“During the team interview you have the possibility to meet the people you will be working with – hear about challenges in their daily work, what they like/don’t like. It allows you to make more informed decisions, not only compared to the recruiter or interviewer but also the website. You just feel the company culture. And believe me, it is much more important to feel this culture inside the team, during the team interview, than hear about it from a manager. It’s more impressive when (as in my case) you first observe the culture during the team interview, then hear about it from office head. You realise that you’ve just observed in practise what they are talking about. You know that it’s not a culture on paper but a living one. Last but not least, after a team interview, you’ve got ‘buy in’ from the team. You know that they accept you as their new colleague and manager and are open to work with you.”

Having software engineers involved in the decision process gives us a real impact on building the organisation, helps us to identify with results and take responsibility for them. However, it creates also some challenges, like putting aside personal objections to accept a group decision. The process is also time consuming and engages a lot of people.

Should you help recruit your next manager?

Introducing the team interview into the recruitment process turned out to be a very successful experiment, and I’d recommend you try it.

We’ve carried on with this approach for development manager positions, and have extended it for product owners as this role requires a lot of communication and coworking with the teams. Maybe one day we will do it for other positions as well…

Even if you are not interested in being a manager, it’s still good to remember ‘Management is too important to leave to the managers’ (Jurgen Appelo). Everyone has an impact on, and is responsible for, the environment around them. So be the change you are looking for!

Dorota Wojtalow, Software Engineer

July 21st, 2016

Posted In: Blog

Tags: , , , , ,

Team size notes

Productivity and team size

What is the ideal team size? Many suggest seven people – give or take one. Small teams in my experience are more efficient and need less management oversight. Recently our Business Planning Systems (BPS) team reached 10 people. We started to experience a declining trend in productivity, so we had to rethink how we were structured.

But it was not just team size

Our productivity decline was not just limited to team size. We were also struggling with having applied an emerging design model, new requirement process, and a lot of new team starters.

What could we do?

I did not want to reduce the number of team members, we have so much work to do! I also didn’t believe the solution was to switch to a new Agile framework and I didn‘t want to split our product backlog. This left a few options to experiment with:

  1. One product backlog with two sprint backlogs and two teams
  2. One product backlog with one sprint backlog and two teams
  3. One product backlog with one sprint backlog and two or more self forming teams.

The BPS team lead had pondered on this for a while and together the decision was made to go with Option 3: one product backlog with one sprint backlog and two or more self forming teams. We ran the experiment for four months at the start our winter roadmap.

The difference with Option 3 was the team could form as many teams as they felt necessary to work off the one sprint backlog. For example, two teams equally split, two teams formed of six and four people, or three teams formed of two, four and four people. This also meant a change for the BPS team lead; he remains team lead but delegates more accountability to the team.

Benefits

  • Team is empowered to take action, without team lead and PO involvement.
  • Cross skilling of team members increased. One sprint you might be working with Fred and Steve and the next with Larry and Barry.
  • It was far easier to get new starters up to speed in smaller groups.
  • We retained one weekly entire-team ‘grooming’, so everyone was familiar with the business requirements and backlog.

What we monitored

  • Would teams always form into the same group to work on the same type of issue, thus impacting team dynamics and cross skilling?
  • Velocity of work coming out of the team.
  • Number and type of issues identified in retrospectives.
  • Team happiness.
  • Delivery to business and their satisfaction with outputs.

How did the model work?

  • Velocity of work continued as normal.
  • Commit-to-sprint backlog stayed as normal.
  • At sprint planning Phase 2 the team discussed stories and how to structure itself to best service those stories. For example, three stories involve UI so group those, three other stories focus on refactoring, security and support so group those.

Now the team could decide who wanted to work in which group. It is the team’s responsibility to make sure there’s a) enough work for the number of people in that group b) enough people to work on the stories in that group.

What else to address declining productivity?

I mentioned it was not just team size affecting productivity, but also applying an emerging design model. By not focusing on future architecture for complex features, we were spending considerable time refactoring and make compromises. We decided to trial an opt-out-of-sprint option.

  • A team member could choose to opt out for a sprint or some of the sprint, to focus on an issue.
  • We would reduce our expected sprint velocity for the upcoming sprint, based on the amount of time the person would need.
  • At the end of the time allocated the team member would present their findings to the entire team, and we would then make a number of decisions i.e. work on the issue now; work on it later; the solution didn’t work so look into an alternative next time etc.
  • There is no restriction on how often a team member can take the opt-out-of-sprint option. In the last four months only two people have chosen to do this and both times it was very valuable.

What was the end result?

Success! This is now how we work. The team loved the new approach and productivity improved. Disclaimer, to my knowledge the team did not break out into dance moves like the below image, but I like to think they did…

Dancers

To wrap up, we gained the benefits of working in small teams but also of moving as a collective towards the one overarching product vision. Job well done team!

Ali Major, Product Owner

May 6th, 2016

Posted In: Blog

Tags: , , , , , , ,

Stand-up meeting

In October, my area within Ocado Technology undertook a radical transformation. Over the course of a single day we carried out an area-wide, self-selecting restructuring exercise. The team I now work in consists of six team members. None of us had worked together before. Although most of the other new teams decided to take on a fairly traditional Scrum approach to their process, we decided that such a radical reorganisation would be a good opportunity to introduce some experimental new process ideas and ways of working. We incorporated some of our favourite Agile/Lean development techniques and spent the next three months adapting the process to fit our team. Here’s a taste of some of the things we did.

We chose Kanban.

We decided to use a variation of the Arrow Kanban board, which is itself a variation on the traditional Kanban method. Where Kanban usually has “Work In Progress” (WIP) limits for each stage of the process, we simply have a “Story in Progress” (SIP) limit. While we were a new team (learning the systems and trying to get on top of in-office support) we set our SIP limit to two stories, which provided plenty of work for two pairs of developers. We’re now more familiar with things, so we’ve given ourselves room to take on a third story, but we only do this if there isn’t anything else which can be progressed or parallelised on the other stories.

Kanban board

Our take on Tomas Rybing’s “arrow kanban” board

Kanban is working great for us. We maintain two work queues; a “prioritised” queue and a “random” queue. Our Product Owner (PO) can move the user stories around between these queues, changing priorities whenever she likes. This means that we can be more flexible when it comes to avoiding taking on stories which we know will become blocked, we can respond to changing business needs or deadlines very easily and we can prioritise work which would otherwise cause us to block another team. We think that this is a huge benefit over sprint-based methods of working, where it’s hard to estimate how much work to commit to and – even with a one-week sprint – you are left with very little room to act in an agile way when priorities change.

We decided we didn’t like meetings. Well, some of them.

Most of the team were in agreement that we prefer writing code to holding endless meetings. Spending a few hours every two weeks in sprint planning sessions, estimation sessions and doing backlog grooming didn’t seem like our idea of fun… and the value we thought we would get from those meetings was questionable. Maintaining a small work queue and taking on stories whenever we are ready helps us avoid these lengthy planning meetings.

As a result of Kanban and a lack of meetings, we don’t estimate our stories, either. I’ve found that estimates are almost always wrong, even when only considered in a relative manner, so I’m skeptical that there is any value in estimating stories on a routine basis. Additionally, this removes the pressure upon the team to ensure that all of the stories estimated to be finished during a given iteration are complete by the end of the iteration – something which I’ve noticed often results in teams either speeding up or slowing down towards the end of the sprint or, worse, cutting corners such as leaving out testing in order to mark the story as “done”.

This doesn’t mean we don’t have meetings. We still talk about stories and break stories down into tasks, but we do this when we’re about to start work on the story so that we aren’t as affected by requirements changing between planning and coding. It means that we can dedicate more time to discussing complex stories, rather than bundling the conversation into a meeting with several other breakdowns also required. Very importantly, we don’t require the whole team to be involved in story breakdown – just the team members who have an interest in that story. Other team members are brought up to speed during stand-ups, or when we switch pair-programming duos; something we try to do as often as possible, but perhaps not as often as we might wish to.

Removing unnecessary meetings from our calendar also meant that we have been able to spend more time doing things with more business value, such as engaging more with our users in order to keep a tighter feedback loop.

We solve problems fast.

Our team has two stand-ups a day. Yes, two. Rather than giving individual status updates, we talk through the stories we have on the board (remember – a maximum of three!). We talk about the progress made, what the blockers are and what the next steps will be. Frequent stand-ups mean that blockers can be identified and discussed more quickly, and the process has become so well-oiled that we can finish each stand-up in around five minutes.

 

Despite ditching sprints, we still have retrospectives. In fact, we hold them weekly. This is probably more often than most of the scrum-based teams host retrospectives. Like our stand-ups, we’ve made these meetings as short as possible; they’re an hour. A different person hosts each week and, after discussing the things that went badly or well in the past week, we try to come out of every retro with some actionable items. If an action is specific, it goes on the board. If the action item is more culture or behaviour based, we make a meme about that aspect of our work and stick it onto our team board to remind us to take the retro discussion on-board. For example, on one occasion people disliked how our “let’s take this meeting offline” codeword, “Mango”, was being used to shut people up rather than genuinely move discussion out of a meeting (see the image).

Discussion points

Our “three things to discuss” meeting ideas canvas fills up

Sometimes, issues spring to mind which need team input but are perhaps too large to fall into the scope of a retrospective. This was most true when the team first had formed, and we wanted to make decisions about our release strategies, our code review policies, etc. To try and address these problems, we set up a section of our team whiteboard called “3 Things”. People could write down things they wanted to discuss in more depth and, once three things had been written on the board, we booked in an hour-long meeting to talk about those things. Now that we’ve spoken about many topics, we’ve started to notice that decisions can be made more quickly. We’re currently trialling discussing some of the smaller topics after our standups.

We measure ourselves. A lot.

Cumulative Flow Diagram

Our cumulative flow diagram (CFD) shows where stories get held up

Whenever our stories change state, we log it. From this data, we can track how much time stories spend in each state and we can measure our cycle time. I think we all agree that what we really want to be doing is writing code, rather than spending our time manually testing or working through a painful deployment process. By considering how much time we spend in each development phase, we can identify which types of stories get stuck in undesirable phases and work out how we can avoid this in future – usually by adding some more automation. We tend to look at the graphs of our data in our retrospectives so that we can look for problematic stories and follow our progress in improving the metrics we’re tracking. We find that our Cumulative Flow Diagram (CFD) is particularly useful at identifying bottlenecks where stories regularly get held-up.

Histogram

Histograms captured during a reto. Green blocks are the current week, orange outlines are the result from the previous week

Another thing we measure in our retrospectives is our Productivity, Collaboration and Happiness. “Our” is loosely defined here – individuals tend to factor in their perceptions of how well the team is doing, and also their own view. We cast a vote (between one and five) anonymously on post-it notes for each aspect and then draw the results up on the board. Once the results are up, we compare them with the results from the previous week and discuss the reasons for any variations. People seem to enjoy this activity. Despite the voting being anonymous, I’ve observed that people who give one of the three aspects a low score are almost always happy to identify themselves to the group and give specific reasons as to why they voted this way. I like to think that this is a sign of excellent trust within the team.

We don’t tell people how to manage their workload.

Despite the fact that I think our process is awesome (awesome enough to write a blog post), we’re hesitant to tell other people how to work. A team located near us has decided to try the switch from Scrum to Kanban. I’m delighted. Although they are doing things very differently from us, we’ve tried to give them some advice along the way.

It’s important to note that there’s no silver bullet when it comes to Agile or Lean software development. Many people think that “Agile” means “Scrum”, and that “Scrum” is a rigid process, but this couldn’t be further from the truth. What I have realised is that if you don’t enjoy your process then you probably won’t enjoy following that process. Don’t do something because I’ve said it works for my team. Don’t do something because it’s what all of the rest of the teams in your company do. Break the mould, and choose a process which works well for you.

I hope that this post gives you some confidence to pick and choose the parts of each Agile tool that your team finds the most useful, and ditch the parts that don’t provide enough value to your team.

Lawrence Weetman, Software Engineer

February 17th, 2016

Posted In: Blog

Tags: , , , , , , ,

Ali Major

As a Product Owner working with a team across two different countries, there are lots of problems to tackle to make sure projects run smoothly. In this series of blog posts, I will explore how my team and I have overcome issues.

For a bit of background info, I’m based in Hatfield and my colleagues in Krakow. We use Agile. In this first blog, I’ll explain how we work cohesively even though we’re miles apart.

Is the structure You + Team, or are you part of the team?

Let’s not beat around the bush, you should be part of the team, so don’t treat your team as anything less. And to be frank, you cannot be part of the team when trust has not been established and you have not invested in building relationships! You’ll just be the PO who is nominally part of the team because Scrum says you are…

Building a solid relationship takes time and I have tried many different ways with varying success. What works for one team does not always work for the other so re-evaluating this continually is very important.

So what were some of the things that have worked?

Communicate regularly

Communication is a potential hazard, so I made a deliberate effort to ensure that we communicate regularly i.e. every 1-2 days either on a work or social level.

To begin with I focused heavily on communication with the team’s leads. As time passed I began to find ways to engage more and more with the team as a whole. It is easier to just keep going back to the one person, but this approach hinders your effectiveness in the long run.

Why? Well you end up relying on that one person and at some point they go on leave; also information is double-handled as they relay everything back to the team. Finally it reduces overall interaction with the team, which you need to build a solid working relationship.

Show positive attitude and willingness to build a cohesive team

Don’t underestimate how much your mood and choice of words are perceived via email and hangout. If you are strictly business every time you talk to your team and seemed rushed or disinterested, then it is only fair the team reciprocates.

Regular visits to Krakow

After numerous visits of varying length, we came to the conclusion that it is best to not nick off as soon as your agenda is done. Spread your agenda over an extra 1-2 days and mix your time there with meeting your agenda and with normal day-to-day business.

Just being there to be asked a simple question, to go for lunch with the team, to contribute to Agile ceremonies… ahh nothing beats the luxury of walking over to a teammate to ask a question once in a while! Also it challenges the perception that you are just flying in and out for your needs, and shows that you value time with the team.

Games

I love games and I play them when I am in meetings in Krakow as well as from Hatfield on hangouts. Taboo, Pictionary, Charades, trivia, true & false, riddles…

Why? Games remove the perception ‘we only communicate about work content’. It can help a team to learn more about each other i.e. ‘I come from Australia so – true or false, dear reader – I have been bitten by a brown snake?’ Games makes meetings more fun and they can relieve tensions during heated and/or vigorous discussions.

Ali's team in Krakow

Agile ceremonies

I attend all the prescribed scrum ceremonies with the team bar the daily stand-up. The reason for not attending the daily stand-up is that, with daily interaction and the ability to track progress of tasks moving on the JIRA board, I have enough information on what is going on and any impediments. Also it’s easier for the teams to do their stand-up in Polish and this is meant to be a quick meeting.

Should I learn Polish?

English is the first language of our organisation. For my Polish to be at the same standard as the team’s English I would need many years of study. If you know me and have heard my attempt at Polish (or any language) you would probably come to the conclusion that I shouldn’t push it!…

I firmly believe making an attempt is appreciated, but in my situation investing time in developing my technical knowledge is time better spent.

Sometimes in our meetings it is easier for the team to discuss technical details in Polish. I would encourage this as being acceptable, but that this is not the norm and they need to re-summarise in English what the discussion was about.

April 17th, 2015

Posted In: Blog

Tags: , , , , , ,

Scroll Up