Originally published April 4 on Medium

As a Technical Team lead specialising in Networks and Telephony for British Red Cross, our team had many plans in place at the start of 2020.

A project to source a replacement Telephony solution to replace our legacy hardware was getting off the ground, we were ready to move to a new internet provider, plans were in place to move out of our physical data centres into the Cloud, a new Director of IT had just joined. It was to be a busy year.

Like just about everyone else, all bets were off come March and the true nature of Covid-19 was revealed.

Almost overnight we were faced with the temporary closure of over 200 sites from which we provided key services to those most in need. My team is dispersed across the country and already working from home. For us the initial shock was much less severe in terms of our ability to work remotely. Our job was to support those that were less familiar with working remotely or unable to do so.

Our on-premises telephony equipment was not built to scale for remote users and struggled under the weight of so many changes and demands. The support requests came in thick and fast from users desperate to continue to deliver services in the way they needed to from their home. This was complicated further by the BRC being a major responder to the crisis — maintaining and scaling existing services (e.g. supporting the NHS through helping people home from hospital to reduce pressure on the system, providing support at home, education first aid and resilience) and developing new offers such as a COVID support line, food delivery and the Hardship fund. It seemed that every day was an endless array of video calls and meetings with the phrase “Zoom Fatigue” being coined probably within the first fortnight.

There were many new headaches including users logging in remotely to contact centres. Others with a needing VPN access with the added complication of enabling softphone communication on a legacy system built around hardware at physical locations. Residential broadband connections collapsing under the weight of contention from properties around them with no way for our team to diagnose issues — was it us or their local connection ?

The team was constantly bombarded by system alerts from our network provider and our monitoring systems, but with our sites closed we had no eyes on site or hands to help us reboot equipment that became unreachable.

When large amounts of error messages come through on a daily basis, it’s a bit like The Boy Who Cried Wolf. It’s hard to separate out meaningful faults and after a while the alerts themselves are useless and generally ignored. Identifying and prioritising those faults relating to key frontline workers was our focus.

We had some fantastic support from 3rd parties. Olive Communications moved mountains to help us deliver managed telephony at incredibly short notice to allow us to deploy a contact centre using new technology, trained users and got us online delivering emergency National and Volunteer support lines to help those in crisis whatever way we could.

Over the following couple of months as the surge of urgent requests abated and we started to focus on the larger pieces we had planned for 2020.

The migration of an internet provider is no small task, new firewalls were installed and had to be configured and commissioned. Many unsociable hours were put in by the team over months to install, configure, test, re-configure, re-test. Old firewalls decommissioned and traffic rerouted through our new provider.

Although this work was extremely arduous, what I took away from it most was the prodigious amount of time and effort put in by not only our engineers but also by those across the Directorate who tested with us and verified the work in the wee small hours. Many 20+ hour days were shared. The firewalls were migrated successfully after yet another all night slog or two, but the support never wavered.

Two of our largest locations, Moorfields House and Paisley were also completely gutted and re-provisioned as part of a strategic redesign of our working spaces (Our People Our Places project). This involved socially distanced network engineers, thousands of metres of cabling, dozens of network devices and Wi-Fi access points deployed ready for people to return to work when possible.

The telephony project is not yet complete, as the requirements are now so different from where we were at the end of 2019 that they have to be redrawn entirely, but we are proceeding with what we can, as and when we have the opportunity. This piece of work will now stretch well into 2021 with the team now carrying over some of the covid-induced problems from legacy systems with it.

As we continued to navigate through what was left of 2020 and take over into 2021 all of the work and projects that were delayed or put on hold by the pandemic, the majority of us were and are still home working. Who knows what the network will look like in 18 months, very different from today in scope and delivery for certain. The details of what it will look like will keep us busy for some time.

We talked at the end of 2019 about planning for future change and we are back there again — further forward for sure and with a more defined idea of what a lot of it will look like but there as ever, is still much more to be done.

Carl Thompson
Technical Team Leader at the British Red Cross