CCDC Journey - Zero to Nationals

Jul 25, 2024

My 2023-2024 CCDC experience

New Beginnings: Making the Team

When I first learned about what kind of opportunities were out there in the field of cybersecurity in 2022, it sounded like something I might want to do for a living. To learn more about it, I joined the Cyber@UCI discord, but was immediately intimidated by the barrier to entry and decided to just focus on my classes. In May 2023, after getting through some of the tougher classes at UCI, I noticed that I had a lot of free time in my hands. This is when I found and got to work on the Google Cybersecurity Certificate, a super beginner-friendly course for someone with absolutely no experience in IT to learn about the field. It is a super high breadth, low depth course which goes over the foundations of cybersecurity and what you will need to learn in order to excel in the field. With a good amount of CS coursework under my belt, I found a couple of the modules way too easy, but really enjoyed learning about the different aspects of the field.

The course was a nice passive learning resource (reading + videos), but it was lacking in hands-on practice. This is when I discovered the Collegiate Cyber Defense Competition (CCDC). During the first CCDC informational meeting I went to, we learned about the competition and what it would take to make the team. I also learned that there is another competition called the Collegiate Penetration Testing Competition (CPTC). Both competitions sounded super fun, so I really wanted to compete. As someone with no hands-on cybersecurity experience at the time, I wasn’t too confident in making either of the teams and competing. However, the upcoming school year would be my senior year at UCI, and not making the team would mean I would never compete at all.

I woke up one day and noticed that there was a CPTC meeting in progress. At this point, I didn’t even know that training had already started a couple weeks before. The CPTC captain said that I would probably be very lost if I were to show up to the ongoing meeting, but I was welcome to join anyways. Since it was a Saturday, there were no busses running, and I had to walk 30 minutes to get to the lab. When I showed up, everyone was hacking away at an “Easy” rated Hack The Box machine. I had no idea what was going on, but I stuck with it and tried to learn as much as I could. After that day, I was hooked onto Hack The Box, and completed all of the Starting Point machines in 2 weeks. From then on, I would show up every week (virtually after summer started) to hack with the group of people interested in competing, which eventually just became 4 unfortunately. Even more unfortunate, we would later find out that the club doesn’t have the funds for us to go compete 😔

Over the summer, along with CPTC training, I followed a training curriculum developed by our CCDC captain to learn hands-on fundamentals surrounding the competition and the cybersecurity field in general. The curriculum covered all of the basics from setting up VMs, Linux and Windows fundamentals, networking, red teaming, and blue teaming. After in person meetings started again in Fall, I learned that I was one of only very few people who completed the summer curriculum all the way through. This, in conjunction with the countless hours I’ve put in already, made me very confident that I would be part of the CCDC team. Another event that made me even more confident was the side quest that came up.

Here are the resources that helped me the most as a beginner (in no particular order):

Side Quest: Hivestorm

Hivestorm is a find-and-fix style competition which is very similar to a part of CyberPatriot, a competition for high and middle schoolers. Teams of 4 compete on virtual machines (VMs) with investigative Forensics Questions and security holes for us to find and mitigate. Lucky for us, we had someone who won at CyberPatriot Nationals three years in a row before coming to UCI. He and our CCDC captain would take care of Linux VMs, which left the Windows VMs to me and the person in charge of Windows in the previous CDCC season. Since neither of the two of us had any CyberPatriot experience, we (me especially) had a lot of work to do if we wanted to be useful during the competition. I had just finished learning the fundamentals of Windows from HTB Academy, and I would have less than a month to practice for Hivestorm. Thankfully, there are many publicly available CyberPatriot practice images with associated walkthroughs that were extremely helpful in my preparation. Drilling through multiple practice images helped me build a strong foundation in Windows knowledge. During the competition itself, the 3 non National Champions pulled our own weight for the most part and let our 3x CyberPatriot National Champion go crazy. He even brought his mechanical keyboard to our lab so we could really hear him carrying us to victory. We won first place by a large margin, which was a great confidence boost to start off the year.

Here are the practice images I enjoyed and learned the most from:

First Thrill: Invitationals 1

A few weeks after Hivestorm, it was time for the first invitationals hosted by WRCCDC. Invitational rounds act as practice competitions with no stakes for teams to give new competitors some experience, test scripts, and more. As for UCI, the goal was to give new competitors a chance to see what it’s like competing. The 3 people with previous CCDC experience sat out, and those interested in trying out for the main roster rotated in and out. Teams from other regions are also allowed to participate, so we competed against a wide variety of schools.

The competition itself was chill for the most part since red team didn’t do anything destructive. Me and the other Windows people just followed a checklist from a previous year and tried not to nuke anything. When I was working on the Domain Controller, I managed to lock myself out by disconnecting after changing the Administrator password with a random one (genius, I know). Now in most cases, this wouldn’t be very good, but since there was barely any Red Team activity and no services went down, we decided to leave the DC alone and just work on other machines, a certified CCDC moment. Overall, I had a very good experience learning about how the competition flows and actively working on something at all times.

The scores really don’t matter for this one, but here they are if you’re interested:

Forging Unity: Invitationals 2

With Invitationals 2 coming up, our club finally held tryouts for the main roster. The second invitational round is a bit different from the first because it is the last official practice before qualifiers, so most teams opt to compete with their main competitors of the season. Although this isn’t the case for ALL teams since there are no real stakes, the competition gives us a rough idea of where we stand in comparison to other teams.

I won’t go into the specifics of our tryouts process, but here is a peek at our team composition:
  • 1 Windows (me)
  • 1 Networking
  • 1 Hands-off Captain
  • 2 Corporate
  • 3 Linux

After we got our team packets, I started sweating because I was unfamiliar with a few of the protocols I would need to deal with on the Windows side, and there was a legacy Windows box which I had almost no experience with. The night before the competition, instead of being a normal person and getting a good night’s rest, I set up 2 VMs with services using the unfamiliar protocols, one being a legacy Windows box. Since the team packet is mostly vague, I wasn’t able to get perfect practice, but I got enough familiarity with the protocols to get some peace of mind and sleep... Oh wait, it’s time to head to the lab and get ready.

When the competition started, I felt way more adrenaline than the first invitationals, which was great for counteracting my lack of sleep. This time, I had my own playbook ready and I was much more aware of what I was actually doing. At this point in the year, our networking guy was still partially Windows, so I had a lot of moments where I was too overwhelmed to solo Windows and had to get some help from him. As for the Red Team, they were more active this time around but still didn’t make too many aggressive actions. However, we were pretty good at Red Teaming ourselves this competition, and had some trouble fixing services we took down on our own. On the bright side, we had good teamwork and communication, and I was really looking forward to competing with this team for the rest of the season.

Redemption: Qualifiers

Fast forward from December to February, it was time for Qualifiers. Only the top 8 teams from this round qualify to go to Regionals, so it’s a huge step up from Invitationals. In the previous year, the UCI team did not qualify due to a number of reasons I will not go into. Also, we were not happy with our services score from invitationals, so this competition would be a redemption round for us in two ways.

Preparation

There was a teaser sent out by WRCCDC back in December, which looked like this:

The competition was pasta themed, so they thought it would be funny to make the Domain Controller of the network fully Italian and color it like marinara pasta. Since we got this teaser pretty early on, I was able to learn and get some practice starting with a foreign Windows install and fixing up the machine so that it’s usable.

Leading up to the competition, we learned that there would be a Windows IOT machine on the network, something I’ve never even heard of. When I searched up “Windows IOT exploits”, the first thing I saw was an Remote Code Execution (RCE) exploit as SYSTEM, so I thought this would definitely be something Red Team would try. A part of CCDC is preparing defenses for public exploits, so I prepared a one-liner that would exploit the machine, get SYSTEM, and automatically patch the vulnerability. The nice part about Qualifiers as compared to Regionals/Nationals is that you get to use your own computers to compete. This meant that I could have the one-liner ready to go the moment the competition started.

The Competition

Once the competition started, I immediately fired off my one-liner to patch the IOT machine and started RDP’ing into the Domain Controller. I notice that the exploit failed, and noted that it’s probably a different version of Windows IOT than what I was expecting. I start running minute-zero scripts while also going into the settings of the DC to change the language to English. To my surprise, the button to install the English language pack was non-existent. Only a couple of minutes into the competition, I learned that I will have to compete with an Italian DC for 8 hours straight.

Fortunate for me, I did most of my work in PowerShell, which was in English and actually usable. I was able to get a WinRM session to all of the Windows machines except for the Windows IOT, so for that I decided to just use SSH which was pre-installed for us. I was able to go through my playbook pretty smoothly, but one thing that was bothering me was the web interface on the Windows IOT box. When I first accessed it, I saw a login form and thought it was the web portal for managing the device I learned about beforehand, so I logged in with known credentials. Upon “logging in”, you are greeted with some weird interface with the default credentials in plaintext. However, I later found out that the login form is fake, and there is no authentication taking place. You could edit the plaintext credentials and hit “save”, but it would randomly revert to the original. Since we already rotated credentials throughout the network, I stopped looking into this because there were more important things to take care of.

The rest of the competition was pretty chaotic, with many injects coming in and a couple services going down and back up. There was one inject that I could not figure out because I couldn’t navigate the Italian DC to set something up, which was very frustrating. Our networking guy did help me use google translate to clutch up a few times though, which was pretty fun.

Towards the end of the comp, there was an incident that I will never forget. At this point, we had been competing for almost 7 hours, and had all of our services up. While being in the middle of working on an inject, I saw a message pop up on the DC to go read a Red Teamer’s blog. Whenever you see a Red Teamer talking to you, it probably means they are confident that you won’t beat them, and in this case, that was 100% the case. Shortly after the message, I see all of the Windows services go down on the scoreboard. Scenarios like these are the most intense and difficult parts of CCDC, and I’m glad I was able to experience it (not everyone gets the chance). It’s really hard to balance between working on getting services back up and fighting Red Team at the same time. I first started by finding and terminating the Red Teamer’s SSH session, only to be greeted with a new message since they had other ways back into the box. After some quick investigation, it was clear that the Red Teamer had taken down the DNS server. In most CCDC and probably all real world environments, DNS is a very important service that needs to be available at all times. The DNS server being down meant domain names could not be resolved, which took every other service down with it. I tried to start the DNS service that had been stopped, but was getting “Access Denied” errors. This was because the Red Teamer messed with the ACLs associated with the DNS service, so my user did not have permissions to interact with it at all (even as an Administrator). Because none of us could figure out how to fix it, we decided to revert the DC and try to patch it up as much as possible before the Red Teamer could regain control. This was a lost cause though, because the Red Teamer came right back in and took down DNS again. We tried to fix the issue, but due to our inexperience with Windows ACLs, the services remained red for the last hour of the competition.

Results

After getting destroyed by Red Team at the end, I was not feeling too well right after the competition. As for our overall performance though, it was a huge step up from Invitationals 2, so I was pretty confident that we were in the top 3. My teammates had mixed feelings, some believing that UCI’s team would be knocked out at Qualifiers for a second year in a row. The qualifying teams were announced in a video premiere on YouTube, so me and the team watched it together. It was very exciting to find out that we had placed 3rd place out of 22 teams and would be moving on to the final WRCCDC 2024 competition! We scored only 36 points less than Cal Poly Pomona’s team, last year’s National CCDC Runner-up. They were my favorite team to interact with throughout the year since they were a super talented team that did not hesitate to share their knowledge. We were feeling good about our performance, but also knew that we had a lot of work to do if we were going to make it to Nationals.

Boss Battle: Regionals

To move on to Nationals, we would need to place 1st at Regionals, OR place 2nd at Regionals and win 1st place at the NCCDC Wildcard round, where 2nd place teams from every region compete for the last spot at Nationals. Now to explain why this section is called “Boss Battle”: We were up against Stanford, 1st place at last year’s WRCCDC and 1st place at last year’s NCCDC, and Cal Poly Pomona, 2nd place at last year’s WRCCDC and 2nd place at last year’s NCCDC. Also, we only had one person on our team who had been to Regionals in a previous season, and 5 people completely new to CCDC. Now this may sound doomed for us, but what we had that they probably didn’t was MASSIVE room for improvement. We made so many mistakes at Qualifiers which we knew how to avoid now. Also, we didn’t use a very useful tool for Qualifiers because it traumatized the team in the previous year, but we could learn to use this with some good practice. We prepared rigorously for 2 months, and we were ready for the battle.

Day 1

WRCCDC is hosted at Coastline College, which is pretty close to Irvine, so we opted to drive ourselves instead of staying at the hotel everyone else stayed at. Despite one of our teammates oversleeping, we had time to grab a nice brunch and made it to the opening ceremony on time. All other schools were welcomed by name, but the speaker forgot our school’s name so we had to welcome ourselves. The organizers announced that the AWS portion of the network was not ready, and that we would only have to worry about on-prem servers for day 1. Once the competition started, we entered a very hot room and started competing. We had to ask one of the organizers to give us extra air conditioning to survive the heat.

We started by provisioning laptops for each team member to use. We noticed one of the Windows Servers with no scored services had a nice monitor, keyboard, and mouse to use, so our captain assigned that one to me. I really enjoyed working on the server rather than a laptop like everyone else. This decision ended up having good and bad consequences which I will get to. I started by configuring the firewall to make sure red team cannot access my server in any way. Then I installed useful tools and set up my file system for ease of use. At the same time, I had to remote into the Windows Servers with scored services and do minute-zero things such as changing Administrator passwords.

What I wasn’t expecting was that the server I was working on would be the target of many injects and Orange Team requests, which included exposing remote management services for the Orange Team to use. This would mean that the server with all of my work on it, important passwords and more, would become targetable by Red Team, which was very scary. Being on the server for these injects/requests made them much easier to complete, but I had to take extra precautions to ensure security.

Overall, day 1 went pretty smoothly. We only saw a little bit of Red Team activity, and were able to shut them out of the network pretty well. Towards the end of the day, I looked for Red Team artifacts to try and clear out any persistence they were planning to use on day 2, but was not able to find any. Once we were done competing for the day, we went home wondering how much persistence Red Team had on our network. We were expecting a lot more Red Team activity the next day because we were told to expect this from our club alumni. At this point, all we could do was sleep to find out the next day.

Day 2

For day 2, we decided to eat at the same brunch place and headed over to Coastline for the in-brief. The organizers announced our relative standings in the different scoring categories (services, injects, etc.). Based on these standings, we guessed that we were most likely 3rd place at the start of day 2. We would have to step up our game if we wanted to make it to Nationals. The organizers also announced that the AWS portion of the network would go live an hour or two after the competition starts for day 2. This was exciting news for one of our Linux guys, who trained super hard for AWS and was waiting for his moment to shine.

Once we started competing again, it was pretty quiet in the room. There were a couple injects to work on, but no Red Team activity, which was a pleasant surprise. Also, the release of the AWS portion was delayed again. Once the AWS portion finally went live, things got much more interesting. Our AWS guy had a great plan for rapid password changes and executed it flawlessly. While he was doing his thing, I was met with a critical incident on the Windows Server I had been working on for the 2nd day in a row now. Although I had a job to do on the AWS network, I had to reprioritize and fight Red Team for the time being.

I was alerted by Windows Defender that an executable was just quarantined. Upon examining the details, I learned that it was a Sliver implant, which is an executable that would call back to the Red Team’s C2 server if not caught. Because I was very familiar with the server I was working on, I knew that their only entry point was through the remote management services I had enabled for Orange Team. I immediately turned those off and then reconfigured the Firewall they had tampered with, and I was able to feel 100% safe within a few minutes.

As for my job on the AWS side, I just had to harden one Domain Controller which was linked to the SSO service our team had to manage. I actually struggled quite a bit getting an RDP session to this machine because I did not know the full domain name, which was unique to each team, and my server thought I was logging in to the local domain, resulting in an authentication failure. It took me a while to blindly guess that SSH was enabled, which I used to learn the NETBIOS name of the domain to get myself logged in. While hardening the DC, I took down the AWS services twice because of how it was interacting with SSO, but me and the AWS guy were able to fix it in a timely manner. After this, we were chilling for the rest of the competition.

Results

The awards ceremony was the next day, so we had to get through an entire night of not knowing our placements. Once we got to the ceremony, we were welcomed with some pretty good catered food, and then some announcements. There were 4 awards given out for best in each point category, and our team won Best in Customer Service. This was all thanks to our corporate team member who did an amazing job taking care of Orange Team requests. Now, for the moment of truth... 2nd Place!

We were very proud of how far we’ve come, and it was a great feeling to get recognized for all of the hard work we put in. However, we couldn’t celebrate for long, because we had another competition coming up in 2 days: Wildcard.

2nd Chance: Wildcard

As mentioned before, NCCDC hosts a Wildcard event to give 2nd place teams in each region a chance to compete for a spot at Nationals. Since Western Regionals were held the latest out of all regions, we had the least amount of time to prepare for this: a mere 2 days, compared to other teams having weeks. The Wildcard event’s main component is CyberPatriot-styled, so our Hivestorm experience, and of course, our 3x CyberPatriot National Champion’s years of wisdom, would be invaluable.

We spent as much time as we could after the WRCCDC awards ceremony, the whole day after, and the time before the competition started the next day to prepare. With more machines to secure, a separate CTF component, and 8 team members competing instead of 4, we had to prepare a bit differently for this competition than Hivestorm. Despite the differences and lack of time, we were able to make very good use of our 2 days and felt ready to win.

As the Windows Lead, I worked on 2 Windows VMs for a majority of the competition, trying to get as many points as I could in the relatively short amount of time to compete. There was one machine that was much harder than the one I worked on in Hivestorm, and one that was much easier. I followed a very familiar process to score us a good amount of points, and used some knowledge from CCDC training to find some extra hidden points. Everyone did great, and we were able to secure ourselves a spot at Nationals!

The Finale: Nationals

Making it to Nationals was something I didn’t even consider when I started out less than a year before this moment, but here I was, surrounded by my amazing team ready to fly over to San Antonio, Texas for NCCDC 2024. As for preparation, we didn’t do much. This was due to a combination of there only being two weeks, being burnt out, and many people telling us WRCCDC was WAY harder than NCCDC and us having nothing to worry about. So we went in with very little Nationals-specific preparation and just tried our best to adapt.

Day 1

Similar to Regionals, we started by provisioning laptops. This time, there weren’t any servers with nice peripherals like from Regionals, so I was assigned one of the nicer Windows laptops. Thankfully, we are allowed to bring in some of our own peripherals for Nationals, so I brought my own mouse to avoid using a trackpad. Once we got started, we were annoyed by how restrictive the internet proxy was at Nationals. We had to take extra steps to get around certificate issues, and were only allowed to visit whitelisted websites. Even though we knew about this in advance, we were not expecting it to block us from every other link we clicked on while doing research.

As for the Windows Servers I had to manage, there were more than double than the number from Regionals. This meant making use of Windows Remote Management would be essential, allowing me to run commands on 10+ servers at a time. The sheer number of servers I had to manage became overwhelming because it was hard to enumerate all of the machines thoroughly. Most of the technologies deployed were familiar to me, but there were many more databases than what I was used to, which added on to the already hard enumeration task.

There was one inject that came in very early into the competition involving DNS that took away a lot of my time (and my sanity). It involved making a serious change to the active DNS server which was being used by scored services and our own laptops. This inject was especially frustrating because even after we got most of it figured out, and all of our tests were passing, pushing the changes resulted in DNS breaking completely. After over an hour of debugging, we eventually had to give up the inject completely, meaning no points for the inject, over an hour of precious time gone, and a lot of demotivation. The time lost was even worse since the first few hours of competition are most important for getting a good idea of the network and developing defenses against Red Team. While I was working on that inject, Red Team was probably establishing persistence in 8 more different ways. We will never find out how this competition would’ve went for us without that DNS inject, and this will probably haunt me for at least a few more years.

The rest of day 1 went mostly smoothly. There was one service that kept going down and back up randomly on the Windows side that took me a while to figure out, but it was a simple authentication issue that I overlooked. As for Red Team artifacts, I saw a lot of suspicious processes, system logs, and network connections towards the end of the day. I really wish I had more time to look into these, but the competition ended shortly after. What made NCCDC’s Red Team different from others was persistent malware that continuously took down defenses to get themselves back in. I did a lot of planning that night to deal with Red Team the next day.

Day 2

Day 2 for me was all about fighting Red Team. I started making aggressive changes to the Windows machines in an attempt to get rid of Red Team completely, and they reacted by making aggressive changes to make my green services red. Because our Linux subteam did such a good job, all 3 of the Red Teamers assigned to attack our team were focused only on Windows. I tried so many different things to get them out and remove their malware, but their foothold on our systems were too strong, and I could not get rid of them. At one point, I realized that they had code periodically taking down my firewalls. To counter this, I wrote a short PowerShell script which used an infinite loop to turn the firewall on and remove their custom rules. The Red Team countered this by writing an infinite loop of their own to turn my firewall off. When I was trying to change domain user passwords, I noticed that they had deleted some of my tools I had saved on disk, so I had to redownload them from GitHub. Later, they even started RDP’ing into the machine I was working on as the same user, effectively kicking me out, so I created another admin user and locked the account they were using. After all of this chaos, we decided to revert most of the machines with services down, one at a time, and harden them as fast as possible. This was time consuming, but it worked, and I was eventually able to get all of the services back up.

After all of this chaos, we spent the rest of the competition in peace, trying to solve some of the challenges given as a chance to gain extra points. The only information we got for these challenges was the name of the challenge and a light on a custom scoreboard to tell us if we solved it or not. We did a lot of speculating on what they were asking of us, but weren’t able to figure any of them out. Once the competition ended, I remember leaving that room feeling defeated and full of regret for not preparing harder.

Results

We went to the award ceremony to watch the top 3 teams get awarded. Congratulations again to UCF, CPP, and UT Austin! 🎉

As for us, we got an email a few weeks later informing us of getting 4th place, then another one with a detailed scorecard.

Although we did not place top 3, we had a lot of fun and learned so much. I would not have asked for a better team to compete with for my CCDC Journey. Special thanks to everyone who supported our team throughout the year with encouragement, insights, and hardware. Thank you to all CCDC organizers as well, who keep the competition fresh and interesting every year.