Summer 2020, COVID Lockdown. A retrospective on installing 2000 APs in under 2 days.
I remember being at the WLPC in Phoenix in February 2020. The coronavirus, which had just been given the name COVID-19 days earlier, had not yet gripped the North American continent and flipped everyone’s world upside down. We were all still very blissfully unaware of what the near future would bring. Nobody was wearing masks yet.
A mere couple of weeks later, that had all changed. The virus “was declared a global pandemic on 11 March 2020 by the World Health Organization” (ECDC), and on March 12, the decision to start remote learning for all classes at the college where I work was made. At the time, it was a temporary measure – after all, we were all hoping that this would blow over fairly quickly. With any luck, things would be back to normal by the summer, if not sooner. Spread of infection in the area was still being tracked in the singles and tens of digits. By isolating and wearing masks we’d be able to contain the disease from tearing thru society. Spoiler alert – that didn’t happen. A month later, remote learning was announced to continue for the summer term.
Fast-forward to the end of June 2020. With what now seems like a quaint reminiscence of the before-times, an announcement for a plan to return to on-campus learning was made. It started with the quote “March 2020 seems like a lifetime ago.” This was in June, only 3 months after the start of it all. Reading this in hindsight, the use of this phrase at this point in time really underscores just how uncertain everything was and how much had already changed about daily life. Nobody had any idea what was happening, what would happen, or for how long it would last. For context, the first vaccines were made available in the US only in December 2020.
I am not writing an opinion on the successes and failures of remote learning. But even in 2020, it was understood that giving students as much “normal” as possible, while still preventing spread of infection, was the best path forward. When the decision to permit a return to campus was made, it included the caveat that “the majority of undergraduate instruction … will be held remotely.” What this meant for IT was that we needed to seriously overhaul the Wi-Fi network in the dorms. Students would be taking classes online from their rooms.
Zoom requires a solid 4Mbps for online meetings with multiple participants. Industry best practices call for less than 150ms of latency and 50% channel utilization. The legacy 2.4GHz “design” of a Wi-Fi network that I had inherited was already sub-par before even considering the need to do real-time voice/video. APs were mounted sideways in hallways, every-other or every-second room had a wallplate AP slapped in at one point, and the wiring was Cat5e split-pair. (Note: I’m linking to this as a quick explanation of the wiring. I do not recommend, for many reasons, installing network cabling in this manner.) Because of this wire-side bottleneck, the radio interfaces of the APs were actually faster than their uplink! APs were never designed for this scenario – they don’t have deep buffers – and traffic was being dropped constantly. I had even limited the PHY speed of the radios as an attempt to mitigate this! But, even in buildings where the wiring was newer and up-to-spec, and APs deployed in the correct locations, the existing Wi-Fi infrastructure had significant deficiencies and needed to be replaced.
By this time, we were a few months into installing Mist access points on campus. Following a very successful proof-of-concept installation, we had expanded the Mist footprint and had the new product working well in a number of academic, administrative, and residential buildings. We were developing plans to refresh the rest of the campus. I had been asking Mist to build a wallplate/hospitality style AP since that first proof-of-concept install in 2018, with the promise that we’d order the first 2000 of them that they could ship us. In August of 2020, they delivered – the Mist AP12 was publicly announced as a product. We received the first truckload of 2000 units just a week later on August 19.
We were told to expect students to arrive on campus starting September 8. Okay, great, that’s a solid 2 weeks to get everything installed. However, prior to their arrival, cleaning crews were tasked with sanitizing every surface in every room. Nobody was permitted to enter the dorm buildings after August 22, when this process was to begin. Once rooms were declared to be clean the doors were sealed with tape, only to be opened by the student assigned to each room.
What that meant was that we had only 2 days to replace all the Wi-Fi in all the dorms!
We knew this tight schedule was coming. We had been testing a prototype of the AP12 all summer, and working to optimize our installation process. Mist’s amazing hardware designers (Hi Josh and John!) were incorporating our feedback, and similarly working hard to improve the product before it shipped. We had worked with them to make some tweaks to the mounting bracket, ironed-out some requirements for PoE and the pass-thru ports, and otherwise kicked the tires on what is still my favorite Mist product.
We had also hired every low-voltage contractor in the area to work all summer to fix the split-pair wiring issues, install new patch panels and jacks on both sides of the cable, and re-certify every run with their cable testers. That summer we also installed Juniper EX3400 switches in these 37 buildings – thanks to Junos’ built-in automation capabilities and our use of Netbox as a source-of-intent, my colleague Jason was able to replace all of the switches in our dorm buildings at a speed of nearly one building per day!
We had also reached out to the entire college IT department, as well as a few other groups, for assistance. Like the students, the staff had all been sitting at home for a few months and were beginning to be afflicted by the simultaneous pandemic of cabin fever. Many of my colleagues wanted to do whatever they could to be on campus and help out in any way possible – we all wanted to make the fall term of 2020 as successful as it could be! We had enlisted the help of nearly 4 dozen volunteers from across campus. Even several Mist employees were willing to travel and help install APs during this two-day timeframe! (Sadly, we couldn’t take them up on that offer, as the campus policy at the time was to only permit individuals from the immediately surrounding area to be on-site – we were still trying to limit the spread of the virus, after all.)
I had come up with designs for where the APs needed to be installed, and printed out clear, easy-to-read floorplans showing where in each room an AP was to be installed. I mocked up various scenarios (single vs. double gang junction box, surface vs. flush mount, etc.) of installing the AP12 in my home, distilling the process down to a few easy steps. I recorded and narrated a video showing the entire process and typed up a one-page instruction sheet as a reference to follow during the installation. We organized the volunteers into 9 teams, totaling 55 people. Each team also included an appointed leader – an engineer, technician, or manager from the networking department who would be more able to troubleshoot or make decisions on the fly. That person also carried a 2-way radio for communication with the other teams and myself should any trouble present itself. (We had removed all of the existing APs from the buildings in advance, so there was no Wi-Fi network for folks to use while on-site.) They also had been loaned the master keys for their assigned buildings, so no door being locked would be able to impede our progress. (Anyone who’s ever done an AP-on-a-stick survey is smiling right now.)
The delivery truck arrived on the 19th. We were there, waiting eagerly to break down the pallets, unbox the APs, and prep the installation kits. We had purchased several dozen screwdrivers (including the special T8 security torx ones) and tool bags – one for each installer. We prepped boxes, one for each floor of each building, with the right number of AP12s, and a few extra 1-foot patch cords, just in case. Paperwork for each floor was placed on the top of the box, and everything went right back out the loading dock – this time to be delivered to the various buildings on campus that we were upgrading. The team doing this prep work was home by dinner time.
I should note here that with Mist, we no longer have a time-consuming high-touch process of provisioning new equipment. If you’ve ever installed enterprise Wi-Fi before, you are no doubt familiar with the process of unboxing new APs at your desk, connecting them to a switch, booting them up, downloading firmware, rebooting, assigning profiles/templates/groups, and printing out unique labels to apply to each and every AP. Cathartic though as this may be, it takes measurable hours of your time. With Mist, you can setup your provisioning processes such that it does not matter which specific AP gets installed where, or what firmware they have out of the box. AP identification is beaconed out via Bluetooth and is visible in the Mist Installer App on your smartphone – you don’t need labels. Installers can grab any AP of the correct model, connect it, and click it into the mounting bracket. You don’t need to have them match serial numbers in tiny font to a list in a spreadsheet. The provisioning of Mist APs, including any firmware updates and radio settings, can happen automatically!
Another of my colleagues, Felix, and I had written a script that leverages the Mist API to perform this provisioning automatically. (This is all now included in Mist in their Auto Provisioning feature, but at the time, we had to write the functionality ourselves – such is a pitfall of being a very early adopter.) Still, Mist’s API is so robust and well-designed that we were able to use it to build custom features, like this, that were tailored to our needs. In this case, our need was to have zero-touch detection, naming, and provisioning of these 2000 APs. This installer script connects to the Mist Websocket and REST APIs. Whenever a new AP comes online in our organization, our script receives a message on the websocket indicating this event and correlates it with our inventory. We fetch the serial number, MAC Address, and – most importantly – the LLDP neighbor information of every AP that boots up. This LLDP information tells us the name of the network switch and the port to which the new AP is connected to. We leverage our Netbox database, and are able to know exactly what building and room number this new AP is located in. The script generates the correct name for the AP (which includes the building code and room number), assigns the AP to the correct site in Mist, applies any firmware updates to match what our current production version is, adjust radio parameters (if any deviations from our standard have been applied to the building), and configures the pass-thru ports. An engineer or other skilled staff member no longer needs to have hands-on-keyboard for APs to be fully configured!
That is how we were able to leverage this team of volunteers to fully upgrade these 37 dorm buildings so quickly. All they had to do was walk from one room to the next, connecting a patch cord, tightening a few screws on a bracket, and clicking an AP into place. I had told them to make sure there was a little green light on the bottom of each AP – to make sure it had powered on properly before moving on to the next room. This entire process took this troupe of deputy installers less than 5 minutes per AP. It was impressive!
Since we had conservatively estimated for 20 minutes per AP, we planned to split the volunteers’ time into two days worth of work. Some volunteers were scheduled for the first day, while others were scheduled for the second day. We expected to be close to 50% done after 8 hours of work on Day 1. We were much farther along. By lunch time that first day, we had installed close to 1000 APs successfully. There were very few problems to deal with. The APs were easy to work with, the mounting bracket was uncomplicated, and the API scripting we had done was reliable!
One problem that we encountered really sticks out in my memory, though. I had told our installers to check for that little green light before they moved onto the next room. I distinctly remember fielding a panicked call over the 2-way radio – several of the lights were not turning green, they were blue! I had to stifle a laugh. A green light on a Mist AP means the AP is online and ready, whereas a blue light means it is providing Wi-Fi service to a connected client device. The APs had booted up, been provisioned, and were serving clients before the installers could pack up their screwdrivers! The AP that the worried installer had just mounted on the wall and was asking about was providing Wi-Fi to their own smartphone!
Because our progress was so rapid, we actually called it quits early the first day. We had decided that it wasn’t “fair” to the volunteers who were scheduled for the second day to have nothing to do. They were eagerly looking forward to coming back to campus, even if only for one day, and doing their part to help. We paused the work after successfully installing 1200 APs in 6 hours the first day.
We showed up again the morning of the second day with a new batch of volunteer installers, and intentionally slowed down our pace. We had completed the entire project by lunch time the second day. We spent a few hours after lunch cleaning up the warehouse and spot-checking a few locations that had been upgraded.
We had installed 2000 APs in 37 buildings with 55 people in under 12 hours of work.
This is a feat we would have never even attempted using any other vendor’s product. Mist’s rock-solid platform, dedicated product team, and fantastic API-first architecture is what made this possible.
I am pleased to say that the Fall 2020 term was successful. The college didn’t have to close down, students were able to virtually attend classes and socialize from in their dorm rooms without technical issues, and being on campus allowed access to laboratory and research materials that would have been impossible to use remotely. I don’t recall receiving a single trouble ticket from students living in the dorms that term regarding Wi-Fi service. Months of planning had culminated in these 2 days of concerted effort, and it had paid off.
