My job is computer security. My job, among other things, is to think like a bad guy and then prevent security breaches and/or catch them soon after they have begun executing their “kill chain”. Most people, even many very smart people, do not have the capacity to think like a bad guy. I have a real life story to illustrate.

Just because this is computer security don’t think this isn’t relevant to current events of a vital importance to the entire nation. I’ll tie all together before the end.

Please do not assume this happened at the company I work for. I have contacts with many other people in the security industry. We often share stories. Sometimes this story sharing is to warn others of how clever the bad guys are and how they succeeded or almost succeeded. Other times stories are shared about how mind bogglingly stupid and numerous some of the mistakes were in the implementation of a computer network system.

This story is about how stupid and numerous the mistakes were.

The type of business and other potentially identifying aspects of the story have been changed to protect the guilty. But the critical aspects of the story are true.

The company penetration testers were asked to test a tool used by customer facing employees. This tool allowed employees to assist the customers with their business with the company. It gave the employees access to personal information about the customer. The personal information access was required for the employee to do their job. The tool had been “released to production” months before the penetration testers (and apparently or other security professionals) took a look at things.

A simplified view of the tool architecture looked something like this:

Database Servers A & B are the only servers applicable to the Customer Assist Tool. The other Database Servers are for other web applications unrelated to the Customer Assist Tool.

Everything from the Load Balancer up were Internet facing. It wasn’t originally designed that way. Originally everything seen in this diagram was inside the corporate network. But because of COVID they had “reasons” and they changed the design so employees working from home could easily access the Customer Assist Tool.

The Internet facing Customer Assist Tool required a company network username and password. The Load Balancer did not. The Load Balancer accepted connections from anyone on the Internet. The Database Servers did not require any security tokens or login. Anything coming from the Load Balancer was considered valid.

The penetration testers didn’t bother trying to do a brute force attack on the login to the Customer Assist tool. They connected directly to the Internet facing Load Balancer and sent queries to the Database Servers. If they knew just a tiny bit of unique public information about the customers, say an email address, phone number, street address, or Social Security Number, they could then get access to extremely personal information from the database.

The penetration testers sounded the ALL HANDS ON DECK alarm. The incident response people (IR) showed up.

The software developers (SDs) of the system were brought into the virtual room and told this is a really big problem. Except for biologically required breaks you’re not leaving the room until this is fixed.

SDs: “We don’t see why this is such a big deal. Someone would have to know the URL for the load balancer. And the only people that might know it are the users of the tool. And we don’t think very many, if any of them are smart enough to figure it out.”

IRs: <blink><blink> “The penetration testers figured it out. And the bad guys out there do this sort of stuff all the time. It’s how they make their money. I’m not going to waste our time explaining this to you. Fix the problem. NOW!”

The IRs then asked how far the logs go back, “You do have logs, right?” The software developers assured the IRs they had logs. The logs went back 90 days. There probably were a few days of missing traffic between when the system was released to production and the oldest log files but most of it was there.

IRs: “Okay, good. We can find out if there was actually any customer information lost.”
SDs: “Oh. You want logs for that? We just log activity at the Customer Assist Tool Web Application. The penetration testers, and any bad guy activity, won’t be in those logs.”
IRs: “Okay…. are there ANY log on the database servers?”

The SDs go looking and find there are generic web logs available that go back to the beginning of the release to production. The IRs looked at the logs for a few seconds and realized the IP addresses of all the requests are of the Load Balancer. There is no indication of the origin of the request. Requests from the Customer Assist Tool are indistinguishable from a request from anywhere else on the Internet.

What about load balancer logs? Maybe. But they don’t go back very far. And if they do exist, all the data is intermixed with the other web applications and other Database Servers.

Within a few hours the SDs have a fix.

IRs: “Tell me about your fix.”

SDs: “The login credentials of the employee used to login to the Customer Assist Tool are passed to the Database Server which validates the credentials before responding.”

IRs: “Okay, we should improve upon that, but maybe that will be good enough that we don’t have to shut down the application until a permanent fix is in place. But that’s a question for our VPs to discuss. Oh, by the way, how many employees do you have authorized to use this tool?”

SDs: “Uhhh… all company employees can use this tool.”

IRs: <blink><blink> “Everyone in the company? Really?” <IRs go to the tool and verify they have access>

SDs: “Yes. If someone improperly used the tool to gain access to customer information when they weren’t supposed to they could be caught and could lose their job. Therefore the customer information is safe from misuse.

IRs: <some facepalm><others bang their heads against the wall> “This is a large company. There are thousands of employees. Anyone on the Internet can find valid company credentials in five minutes or less. We disable hundreds of accounts per week as we find credentials on the web ourselves.”

SDs: <blink><blink>

The story goes on but the important part is that the SDs, not stupid people, made a ton of errors. These errors started with not getting a security professional in the room when they changed the design. The errors compounded dramatically from there.

They had a world view much different than the bad guys and the security professionals.Things which could not even be imagined by the SDs were child’s play to the penetration testers and the IRs.

Now to tie this to current events. Our recent election.

Several courts reviewing the lawsuits claiming foul play have concluded the election was fair and honest.or, at least, there was insufficient evidence of widespread fraud to change the results.

As seen in the story above there are failures modes which not only allow unauthorized access/fraud but make it impossible to determine if such access/fraud occurred. Furthermore, unless someone is experienced in thinking like a bad guy they can honestly believe everything is “fair and honest” and be completely, totally, catastrophically, wrong.

I trust the courts to know their profession. I don’t trust them with security issues. I trust them to accurately asses the integrity of our election far less than the SDs could accurately asses the security of their system. The system they designed and built.

The legal professionals of the court did not design or build the election system. They did not evaluate the security after the (supposedly) COVID inspired changes were made from the viewpoint of a security professional. The original election security features had evolved over hundreds of years and thousands of people poking at it, finding faults, and attempting to prevent future fraud and errors. In the span of a few months a few people made changes which did not go through nearly as rigorous review as the pre COVID system.

I don’t know with a 100% guarantee that sufficient fraud occurred to change the election results. I do know, with 100% certainty, that many people were highly motivated to commit fraud. I do know, with 100% certainly, that some fraud occurred. I’m nearly certain the system in use has issues which make it impossible to detect fraud after the fact.

The bottom line to this is that anyone who says the election was fair and honest because the courts say it was is either lying or placing their trust in a body of people that don’t know anywhere enough about security to make that call.