How we Built a Remote Working Development Environment in 8 Days for a Leading Broadcaster

by Jay Dalu-Chandu

Client Profile

  • Company: FTSE100
  • Industry: Media & Broadcasting
  • Country: Europe & America
  • Employees: 31,000+

Business Needs

  • Tight 2 month project deadline to integrate to deliver content in the US
  • 60 developers were working on site in Portugal but the first Covid-19 lockdown was imminent
  • Developers were using lots of ancillary and peripheral equipment on site (TVs, Apple TV, Roku, Playstation, iPad, iPhone etc)
  • Developers should not use a less secure home network
  • Developers need access to a secure company development environment from home

Project timeline (8 days total)

1. Initial process

2 days | Team of 4

  • Brainstorming all possible solutions
  • Noting any shortcomings or difficulties for each solution
  • Creating a prioritised list of solutions
  • Marking each solution for effort and security risk
  • Prioritising 2 solutions to be explored in parallel

2. Deep dive

2 days | 2 teams of 2

Solution 1 - Hotspot Investigation

The corporate laptops were very locked down, with strong security measures

  • End user teams did not want any corporate restrictions to be removed or make changes to software or desktop builds
  • Developers were using Cisco AnyConnect VPN software to establish a secure connection to the business, which has it's own additional security restrictions, and doesn't allow use of a hotspot

Solution 2 - Home Broadband Router Investigation

  • Buying & configuring a consumer grade TP Link router for use in the developer's home
  • Simulating and running up Cisco Any Connect VPN client to remotely access any resources and assets they need access to
  • There was no time to consider multiple VPN connections or terminating information on different equipment / locations in the business because of the huge volume of firewalls and security rules in place at the business
"We came to the realisation that within the timeframe, all we needed to do was to simulate the VPN connection without dealing with all the restrictions in place on the corporate laptops"

3. Build

3 days | Team of 4

  1. We bought a TP link home broadband router from Currys for about £50
  2. We researched products that could simulate Cisco AnyConnect and found Open Connect, but this could not be run on the TP Link device firmware.
  3. Open Connect needed a Linux based operating system or similar so we researched what network operating systems can run on a TP Link and found Open WRT an open source based network operating software
  4. We removed TP link firmware and installed OpenWRT and Open Connect along with a number of other networking packages
  5. Test, test, test...
  6. By day 3 we had a rough working solution with all relevant software packages, and tools that would let us simulate a Cisco AnyConnect VPN sessions, and connect to the clients Cisco VPN Gateways

4. Tactical Solution (1 day)

1 day | Team of 4

  • We refined and templated the solution
  • We document the full process from out of the box, to firing up, to establishing a connection
  • We handed off the solution and documentation to an internal business team
  • We remained on hand to help with troubleshooting post launch
  • Over time we continued to develop & refine the solution eventually building an image with a small web UI for the developers to provision and setup the device

Next steps

  1. Operational acceptance - We discussed how to improve it to a point it’ll be operationally accepted by the business or whether it’s a solution someone could purchase or request via relevant channels at the business
  2. Compliance friendliness - Original solution was tactical to fit the time constraints and it didn’t tick all the boxes for compliance, governance, security, operational acceptance criteria and more
  3. Security concerns - In the company office you can have set top boxes or TVs with no registration process without any concerns about security, but in the home, the device could end up anywhere, or get shipped to the wrong address so there were lots of security concerns around allowing effectively a guest full access to the business network. This is why we were keen to implement a solution that was more strategic and addressed the wider security concerns.
  4. A more strategic solution - We worked with the operational, developers and security teams to detail requirements, and document various possible solution options
  5. Proof of concept - We completed 3 proof of concepts with vendors (Cisco Meraki, VMWare Velocloud, Fortinet) that would meet the developer, operational and security requirements

Successes

  1. Speed of delivery - Just 8 days to turn around a working scalable solution
  2. Huge increase in users - from the original 60 developers to over 700 just 9 months later
  3. Improved build and deployment - a refined, more robust image that can be deployed to multiple devices quickly over IP
  4. Plug & play - We simplified the process even further since first launch
  5. Lockdown friendly - This software has proven vital to the success of the integration project, but has also proved invaluable due to the Covid-19 lockdowns in Europe
  6. New feature - developers can now do low level diagnostic and packet captures (something they have to do quite often) much more easily - they can confirm and capture all of those packets, and put them in the cloud, for instance, to troubleshoot their diagnostics, which was very difficult for them previously
  7. Better monitoring - we worked with the security team to build better visibility for the solution utilising SPLUNK and logging from the OpenWRT kernel syslog messages.
  8. Better security - we worked with the security team to include more controls such as separate accounts, registering of end devices, introduced posture checks, security dashboards on SPLUNK.