3 January 2022 How we Built a Remote Working Development Environment in 8 Days for a Leading Broadcaster How we turned around a secure remote working and scalable solution in just 8 days utilising TP Link home broadband routers to allow developers to continue working through the pandemic Jay Dalu-Chandu IT Networks Case Study Client ProfileCompany: FTSE100Industry: Media & BroadcastingCountry: Europe & AmericaEmployees: 31,000+Business NeedsTight 2 month project deadline to integrate to deliver content in the US60 developers were working on site in Portugal but the first Covid-19 lockdown was imminentDevelopers were using lots of ancillary and peripheral equipment on site (TVs, Apple TV, Roku, Playstation, iPad, iPhone etc)Developers should not use a less secure home networkDevelopers need access to a secure company development environment from homeProject timeline (8 days total)1. Initial process2 days | Team of 4Brainstorming all possible solutionsNoting any shortcomings or difficulties for each solutionCreating a prioritised list of solutionsMarking each solution for effort and security riskPrioritising 2 solutions to be explored in parallel2. Deep dive2 days | 2 teams of 2Solution 1 - Hotspot InvestigationThe corporate laptops were very locked down, with strong security measuresEnd user teams did not want any corporate restrictions to be removed or make changes to software or desktop buildsDevelopers were using Cisco AnyConnect VPN software to establish a secure connection to the business, which has it's own additional security restrictions, and doesn't allow use of a hotspotSolution 2 - Home Broadband Router InvestigationBuying & configuring a consumer grade TP Link router for use in the developer's homeSimulating and running up Cisco Any Connect VPN client to remotely access any resources and assets they need access toThere was no time to consider multiple VPN connections or terminating information on different equipment / locations in the business because of the huge volume of firewalls and security rules in place at the business"We came to the realisation that within the timeframe, all we needed to do was to simulate the VPN connection without dealing with all the restrictions in place on the corporate laptops"3. Build3 days | Team of 4We bought a TP link home broadband router from Currys for about £50We researched products that could simulate Cisco AnyConnect and found Open Connect, but this could not be run on the TP Link device firmware.Open Connect needed a Linux based operating system or similar so we researched what network operating systems can run on a TP Link and found Open WRT an open source based network operating softwareWe removed TP link firmware and installed OpenWRT and Open Connect along with a number of other networking packagesTest, test, test...By day 3 we had a rough working solution with all relevant software packages, and tools that would let us simulate a Cisco AnyConnect VPN sessions, and connect to the clients Cisco VPN Gateways4. Tactical Solution (1 day)1 day | Team of 4We refined and templated the solutionWe document the full process from out of the box, to firing up, to establishing a connectionWe handed off the solution and documentation to an internal business teamWe remained on hand to help with troubleshooting post launchOver time we continued to develop & refine the solution eventually building an image with a small web UI for the developers to provision and setup the deviceNext stepsOperational acceptance - We discussed how to improve it to a point it’ll be operationally accepted by the business or whether it’s a solution someone could purchase or request via relevant channels at the businessCompliance friendliness - Original solution was tactical to fit the time constraints and it didn’t tick all the boxes for compliance, governance, security, operational acceptance criteria and moreSecurity concerns - In the company office you can have set top boxes or TVs with no registration process without any concerns about security, but in the home, the device could end up anywhere, or get shipped to the wrong address so there were lots of security concerns around allowing effectively a guest full access to the business network. This is why we were keen to implement a solution that was more strategic and addressed the wider security concerns.A more strategic solution - We worked with the operational, developers and security teams to detail requirements, and document various possible solution optionsProof of concept - We completed 3 proof of concepts with vendors (Cisco Meraki, VMWare Velocloud, Fortinet) that would meet the developer, operational and security requirementsSuccessesSpeed of delivery - Just 8 days to turn around a working scalable solutionHuge increase in users - from the original 60 developers to over 700 just 9 months laterImproved build and deployment - a refined, more robust image that can be deployed to multiple devices quickly over IPPlug & play - We simplified the process even further since first launchLockdown friendly - This software has proven vital to the success of the integration project, but has also proved invaluable due to the Covid-19 lockdowns in EuropeNew feature - developers can now do low level diagnostic and packet captures (something they have to do quite often) much more easily - they can confirm and capture all of those packets, and put them in the cloud, for instance, to troubleshoot their diagnostics, which was very difficult for them previouslyBetter monitoring - we worked with the security team to build better visibility for the solution utilising SPLUNK and logging from the OpenWRT kernel syslog messages.Better security - we worked with the security team to include more controls such as separate accounts, registering of end devices, introduced posture checks, security dashboards on SPLUNK.