A security story

My job is computer security. My job, among other things, is to think like a bad guy and then prevent security breaches and/or catch them soon after they have begun executing their “kill chain”. Most people, even many very smart people, do not have the capacity to think like a bad guy. I have a real life story to illustrate.

Just because this is computer security don’t think this isn’t relevant to current events of a vital importance to the entire nation. I’ll tie all together before the end.

Please do not assume this happened at the company I work for. I have contacts with many other people in the security industry. We often share stories. Sometimes this story sharing is to warn others of how clever the bad guys are and how they succeeded or almost succeeded. Other times stories are shared about how mind bogglingly stupid and numerous some of the mistakes were in the implementation of a computer network system.

This story is about how stupid and numerous the mistakes were.

The type of business and other potentially identifying aspects of the story have been changed to protect the guilty. But the critical aspects of the story are true.

The company penetration testers were asked to test a tool used by customer facing employees. This tool allowed employees to assist the customers with their business with the company. It gave the employees access to personal information about the customer. The personal information access was required for the employee to do their job. The tool had been “released to production” months before the penetration testers (and apparently or other security professionals) took a look at things.

A simplified view of the tool architecture looked something like this:image

Database Servers A & B are the only servers applicable to the Customer Assist Tool. The other Database Servers are for other web applications unrelated to the Customer Assist Tool.

Everything from the Load Balancer up were Internet facing. It wasn’t originally designed that way. Originally everything seen in this diagram was inside the corporate network. But because of COVID they had “reasons” and they changed the design so employees working from home could easily access the Customer Assist Tool.

The Internet facing Customer Assist Tool required a company network username and password. The Load Balancer did not. The Load Balancer accepted connections from anyone on the Internet. The Database Servers did not require any security tokens or login. Anything coming from the Load Balancer was considered valid.

The penetration testers didn’t bother trying to do a brute force attack on the login to the Customer Assist tool. They connected directly to the Internet facing Load Balancer and sent queries to the Database Servers. If they knew just a tiny bit of unique public information about the customers, say an email address, phone number, street address, or Social Security Number, they could then get access to extremely personal information from the database.

The penetration testers sounded the ALL HANDS ON DECK alarm. The incident response people (IR) showed up.

The software developers (SDs) of the system were brought into the virtual room and told this is a really big problem. Except for biologically required breaks you’re not leaving the room until this is fixed.

SDs: “We don’t see why this is such a big deal. Someone would have to know the URL for the load balancer. And the only people that might know it are the users of the tool. And we don’t think very many, if any of them are smart enough to figure it out.”

IRs: <blink><blink> “The penetration testers figured it out. And the bad guys out there do this sort of stuff all the time. It’s how they make their money. I’m not going to waste our time explaining this to you. Fix the problem. NOW!”

The IRs then asked how far the logs go back, “You do have logs, right?” The software developers assured the IRs they had logs. The logs went back 90 days. There probably were a few days of missing traffic between when the system was released to production and the oldest log files but most of it was there.

IRs: “Okay, good. We can find out if there was actually any customer information lost.”
SDs: “Oh. You want logs for that? We just log activity at the Customer Assist Tool Web Application. The penetration testers, and any bad guy activity, won’t be in those logs.”
IRs: “Okay…. are there ANY log on the database servers?”

The SDs go looking and find there are generic web logs available that go back to the beginning of the release to production. The IRs looked at the logs for a few seconds and realized the IP addresses of all the requests are of the Load Balancer. There is no indication of the origin of the request. Requests from the Customer Assist Tool are indistinguishable from a request from anywhere else on the Internet.

What about load balancer logs? Maybe. But they don’t go back very far. And if they do exist, all the data is intermixed with the other web applications and other Database Servers.

Within a few hours the SDs have a fix.

IRs: “Tell me about your fix.”

SDs: “The login credentials of the employee used to login to the Customer Assist Tool are passed to the Database Server which validates the credentials before responding.”

IRs: “Okay, we should improve upon that, but maybe that will be good enough that we don’t have to shut down the application until a permanent fix is in place. But that’s a question for our VPs to discuss. Oh, by the way, how many employees do you have authorized to use this tool?”

SDs: “Uhhh… all company employees can use this tool.”

IRs: <blink><blink> “Everyone in the company? Really?” <IRs go to the tool and verify they have access>

SDs: “Yes. If someone improperly used the tool to gain access to customer information when they weren’t supposed to they could be caught and could lose their job. Therefore the customer information is safe from misuse.

IRs: <some facepalm><others bang their heads against the wall> “This is a large company. There are thousands of employees. Anyone on the Internet can find valid company credentials in five minutes or less. We disable hundreds of accounts per week as we find credentials on the web ourselves.”

SDs: <blink><blink>

The story goes on but the important part is that the SDs, not stupid people, made a ton of errors. These errors started with not getting a security professional in the room when they changed the design. The errors compounded dramatically from there.

They had a world view much different than the bad guys and the security professionals.Things which could not even be imagined by the SDs were child’s play to the penetration testers and the IRs.

Now to tie this to current events. Our recent election.

Several courts reviewing the lawsuits claiming foul play have concluded the election was fair and honest.or, at least, there was insufficient evidence of widespread fraud to change the results.

As seen in the story above there are failures modes which not only allow unauthorized access/fraud but make it impossible to determine if such access/fraud occurred. Furthermore, unless someone is experienced in thinking like a bad guy they can honestly believe everything is “fair and honest” and be completely, totally, catastrophically, wrong.

I trust the courts to know their profession. I don’t trust them with security issues. I trust them to accurately asses the integrity of our election far less than the SDs could accurately asses the security of their system. The system they designed and built.

The legal professionals of the court did not design or build the election system. They did not evaluate the security after the (supposedly) COVID inspired changes were made from the viewpoint of a security professional. The original election security features had evolved over hundreds of years and thousands of people poking at it, finding faults, and attempting to prevent future fraud and errors. In the span of a few months a few people made changes which did not go through nearly as rigorous review as the pre COVID system.

I don’t know with a 100% guarantee that sufficient fraud occurred to change the election results. I do know, with 100% certainty, that many people were highly motivated to commit fraud. I do know, with 100% certainly, that some fraud occurred. I’m nearly certain the system in use has issues which make it impossible to detect fraud after the fact.

The bottom line to this is that anyone who says the election was fair and honest because the courts say it was is either lying or placing their trust in a body of people that don’t know anywhere enough about security to make that call.


33 thoughts on “A security story

  1. >> I do know, with 100% certainty, that many people were highly motivated to commit fraud. I do know, with 100% certainly, that some fraud occurred.

    Both of those points have been true of every election we’ve ever held.

    >>The bottom line to this is that anyone who says the election was fair and honest because the courts say it was is either lying or placing their trust in a body of people that don’t know anywhere enough about security to make that call.

    The courts don’t make a determination of “fair” or “honest,” they rule on violations of law. There have now been 54 cases brought to the courts, where both experts and “experts” have had the opportunity to demonstrate how fraud was committed that changed the outcome of the election. These experts were brought to the court by, AFAIK, the best lawyers the Republican party could find. And in not one case did the court hear an expert say something that indicated to them that laws had been broken or that fraud changed the outcome of the election.

    So although you’re correct that there’s the *possibility* of fraud significant enough to change the outcome of the election, I conclude from the 54 at bats and 54 strikeouts by the Republicans (who, by the way, were facing Republican judges, appointed by Trump himself in many cases) that the *probability* is very low.

    And if you really want to get into “fair” and “honest,” you should also be addressing the actions of the Republicans to throw the election: gerrymandering, voter disenfranchisement, voter roll purging, removing voting stations, hamstringing the USPS to try and make mail-in voting impractical…and on and on. The list is long. It’s not like the Republicans are pure as the driven snow on this issue.

    • I’ve never claimed the Republicans are or were “pure”. If they violated the law they should be prosecuted. Just as any other person should. The gerrymandering and other things you bring up may need addressing. But to the best of my knowledge none of those violate the law and are not applicable to the current disputes before the court. The gerrymandering, in particular, does not affect the presidential election (with the possible exception of (is it?) Nebraska and Maine).

      Many of the cases were thrown out because of standing (in the legal sense) or timing issues. This leaves the fraud issues unexplored in those cases.

      In many cases if proper procedures are not followed it would be impossible to show fraud occurred even if it were massive. Should the plaintiffs have to prove fraud occurred when the defendants eliminated the possibility of such proof? Do you really want that to be the law of the land?

      • >>In many cases if proper procedures are not followed it would be impossible to show fraud occurred even if it were massive. Should the plaintiffs have to prove fraud occurred when the defendants eliminated the possibility of such proof? Do you really want that to be the law of the land?

        Well, for one thing I don’t accept your premise that there’s potential for “massive” fraud that is also undetectable. I don’t know as much about software security as you do, but I know there are humans involved, which is always a weak point. So even assuming the system were perfectly opaque and data was completely inaccessible, someone would have had to commit the crime (unless you’re positing Skynet…I assume you’re not). And there’s the “detectable” part of the story. Humans are a weak link, and have yet to invent “the perfect crime.”

        So yes, plaintiffs should have to show why they believe fraud occurred based on actual evidence. Otherwise we’re left with a situation where anyone could say “I have no evidence of fraud, but the system is so opaque that you can’t prove there *isn’t* fraud, therefore we should throw out the election results.” At which point the state assemblies get to pick, which is one of the tactics Republicans have hoped to use (but only in those states where they control the assembly).

        To me, all of this just points to the continued need to create a fully transparent election system. But to get to “fair” and “honest,” as you assert to be your standards, that would also require that the precursor to the election system — voter enfranchisement — be fully fair and honest as well. That would eliminate the games Republicans have been playing over the past two decades trying to disenfranchise Democrats.

        • I don’t know or am claiming that this happened, but here is a scenario which could result in massive undetectable fraud due to sloppy procedures:

          1) Voter registration does not require you register in person with a strong ID verification (registration over Internet or by mail).
          2) The physical addresses for the registration are listed as apartment buildings owned/managed by trusted party operatives with many vacancies. Each vacant apartment gets five nonexistent voters.
          3) Absentee ballots are sent to everyone or all the fraudsters request absentee ballots.
          4) The owner/manager collects the ballots and turns them over to the crew which fills out and returns the ballots.
          5) Signatures are not verified, or the machine which does the verification is part of the fraud, or the signatures were all machine generated originally and duplicated during voting.

          A dozen or two people could easily generate thousands of votes and most of the people would not even have to know each other.

          In Minnesota, replace the apartment building with ballot purchases.

          So, how do you detect this after the fact without verifying the signatures or someone talking?

          • But there are lots of scenarios like this. If all we need to turn over an election is to come up with possible scenario, we’ll never have another election again.

            “Possible” and “actually happened” are very different. I’m happy to have the courts focusing on the latter.

        • Apparently, you are defining Democrats to be illegal voters and felons. You may actually be right about that but it is probably not the look you want.
          The Democrat talking points, of course, are about minority voters and the impossibility of minority voters navigating the giant barrier of needing a picture ID. The same picture ID that they use to visit a doctor’s office, enter a government building, buy a gun, cash a check, and many more everyday activities. So their point has nothing to do with actually empowering voters and everything to do with enabling vote fraud. As for real disenfranchisement, that occurs when a fraudulent vote cancels a legal one.

        • Ferdinand Marcos had elections, too. Enough things have changed about procedure alone in the past year, that if Trump were leading the entire process from registration to certification in Congress would be examined with a thoroughness that would make the treatment by today’s Judicial Branch seem absolutely slap-dash.

      • Democrats are impure.

        Republicans are impure.

        Green and Libertarians are impure.

        Every political party without exception is impure, because a party’s purpose is to consolidate power to its members and only incidentally and most accidently are they responsive to the democratic process.

        Individuals acting alone, convinced of the impossibility of getting caught (frequently erroneously), also cannot be trusted. All those people getting caught for double voting, and voting their dead relatives ballots, are sufficient proof of that.

        The electoral process must be designed with the assumption that individuals and groups will be trying to compromise the system. Frankly, we’d be better off if the electoral workers were 100% prison inmates trying to earn a month off their sentence. At least in that mindset, the system would be designed from the beginning to be operated on a zero trust system.

        We cannot afford to have “trust” in our elections. Everything must be checked, verified, accounted, audited, reconstructed and interrogated. Every deviation must be unavoidably detected. Only by having no trust can we have faith in the electoral process.

        • Trust but Verify. Good advice when we deal with strangers. As the national government and the Executive Branch get stronger and more intrusive every year, the need for this philosophy becomes more apparent. No one should be exempt merely because they say, “But I’m a good person! I’m for the people!”

          • Trust but verify?

            No longer. I used to say verify, then trust

            Now, not even that.

            Verify, and verify again.

            Never trust.

            The Real Kurt

    • “And in not one case did the court hear an expert say something that indicated to them that laws had been broken…”

      “Expert.” In multiple Georgia cases, it was brought up that the Secretary of State signed an agreement to specifically violate state election law, in many ways but I’ll focus on the signature verification. The law states that the ballot signature must be matched to the voter registration signature. The agreement allowed them to match the ballot signature to the ballot REQUEST signature. And it made it harder to reject an apparently invalid signature, in direct contradiction of the law.

      Additionally, poll observers and poll workers testified that in several precincts, NO signature verification was being done. That clearly violates state law.

      There was testimony to thousands of people “living” in postal facilities voting. Documented; I spotted three in my own small town and verified them myself. State law requires the address used to be the person’s actual residence.

      There was testimony to thousands of people voting who are listed in the SS database as deceased. State law does not allow dead people vote.

      There was testimony and video evidence that poll observers did not have the ability to observe required by state law.

      I think this one made it into a suit, but I’m not positive; state law requires counties to maintain chain of custody records –ballot transfer record — for the ballots that were put in the drop-boxes. At least one metro Atlanta county claims they don’t have any. That violates state law.

      Note that I’m not getting into the statistical expert testimony regarding Dominion systems.

      • >> That clearly violates state law.

        If that’s true, then why didn’t the republican judge listen to the best republican lawyers available and agree with you?

        It’s really hard to believe this is all a conspiracy when the folks running both sides are all on the same team….

          • As with so many of these stories, the fact that the courts didn’t find in favor of the Republicans inclines me to believe either a) it’s not true, or b) there’s some other detail we’re not hearing that invalidates the legal argument.

            So when you talk about “who’s holding the leash” I have to point out again: these are Republican judges, in many cases appointed by Trump himself. If there’s any leash, it goes back to him.

          • Who’s holding the leash is often a useful question to ask. When discussing surprising turns in philosophy and voting patterns of Federal court judges, it is harder to discover. Federal court judges are appointed for life. You would think they would vote in support of the philosophy and goals of the person who appointed them to the post, but that seems to be the exception, not the rule. We have several examples of presumably conservative Supreme Court Justices who were appointed by Eisenhower or Nixon who then proceeded to vote with their leftist brother (and now sister) justices.

            As for why no effort to affect the certification process through the courts has proved successful, yes, there may be something about the facts that we are not privy to, but more likely the Supreme Court does not wish to address a “Political Queston”, which they deem “Non-Justiciable” and leave for the legislators to resolve. With the talk of “Packing the Court”, aka adding seats to the court, the Court is understandably chary of even taking a case so they can express the idea that it is a Non Justiciable Political Question.
            The answer to who holds the leash may be easily discovered by examining judicial culture. Judges have since the days of John Marshall and Marbury v Madison jealously guarded the independence of the Judicial Branch, and have long fostered the collegiality among judges. The friendship between Ginsberg and Scalia is only the most recent of such friendships crossing philosophical lines.

            As to whether the facts may be other than as we think we know them, consider the actions of the judge in the General Flynn case. Ordinarily when the prosecutor dismisses his case, the judge happily removes it from his crowded docket. He does not seek other counsel to find someone who will continue prosecution or find some way to compel the prosecutor to continue.
            But, as was said in the 1960’s, A doctor with Draft-Age sons could find a heart murmur in a statue. A legal equivalent is that California Chief Justice Rose Bird could always find reversible error. It’s exceedingly strange how affidavits signed under penalty of perjury could not get cases to be heard.

  2. Pingback: Election questions | The View From North Central Idaho

  3. Pingback: Election fraud stories | The View From North Central Idaho

  4. “sandworm” by greenberg. dominion and es&s can be hacked to take part of a vote and subtract from the intended recipient, and give it to another person, even while the machine gives a paper receipt showing the vote was not compromised.

    and controlled in real time.

    all thanks to the fact that most such systems are connected to computers.

  5. the only answer is to return to paper ballots, small precinct voting areas, and observers there 100% of the time. to witness identifying the voter, determining registration, and marking the voting attempt.

    it just takes participation by relatively honest people.

    • I’m not convinced that is “the only answer”. But I agree it would be far better than what we have now.

    • When I was a child, the voting precinct was only three blocks away. People walked there. Many of the people knew each other. Nobody had to drive and find a parking place. There was just the one precinct at that location. We didn’t need five different colors to sort people out. It made politics local. And personal.

  6. I recommend that when the two Georgia Senatorial elections are held in January the election officials in Georgia’s red communities and counties better be fully prepared to counteract and neutralize the fraud which will take place in the blue dominated communities.
    The election officials in any red community which does not have 100% turnout should be fired for political incompetence.

  7. I also work in computer security, and have seen this kind of thing more than once.

    You’re absolutely correct on all counts.

    Going back to paper balloting would be an enormous improvement – and that could be improved by a simple expedient that’s deployed in several countries – required dipping of a finger in ink before entering the polling booth.

    Ink on finger would likely have great social significance, and could potentially eliminate the need for voter registration for most people (probably not overseas military personnel, though).

    The Real Kurt

    • We have tens of millions of non-citizens in this country. Voter registration with real proof of citizenship and residency is still required.

      We seriously need to eliminate all absentee ballots except for the military. And even then, there needs to be proper procedures. Several states delay sending out military ballots, so that there is little possibility of their votes being received on time. Illinois is famous for this. They even mark the envelope “military ballot” so the Democrats can identify them and “misplace” them.

      • Registration not needed. Day-of showing of driver’s license, passport, etc. all that is needed.

        • Driver’s license? That doesn’t prove anything. In particular, it doesn’t prove you’re a citizen, or (in many states) even a legal alien.
          The real problem is that the Democrats are absolutely insistent that no identity or eligibility verification mechanism is acceptable, ever.
          The motivation for this insistence is entirely obvious, but rarely stated.

  8. >>It must be assumed that everyone, everywhere, even very remotely associated with anything to do with voting is a lying, cheating thief who cannot be trusted at any level, anytime, for anything, and treat them that way.

    Agreed. Which is why in a response to one of Joe’s posts a few weeks ago I sent links to some work by cryptography researcher Josh Benaloh, who (among many other people) has thought deeply about how to make a very transparent cryptography system that would make the entire process fully auditable and non-hackable. Note, however, that he doesn’t have solutions for some of the outside problems, e.g. establishing who should be able to vote in the first place.

    >> who is actually who and their citizenship status, and lots of purple ink to verify who actually voted. Which the Democrats will never allow, and Republicans probably aren’t enthusiastic about, either, if for no other reason than it drastically shortens the path to some sort of national ID

    A “national ID” isn’t inherently bad as a technology. It’s all a question of what laws you put around it for its use. Ultimately the reason the Internet is an open sewer is because of anonymity: anybody can say or do anything without accountability. I’m a big fan of accountability, and non-anonymity as a result (at least wrt the Internet and voting). You may fear what leftists will do with a national ID, but that doesn’t mean it’s a bad idea, it means we just have to fight the leftists (and the far right, for that matter) to make sure they can’t distort it to something corrupt. Identity and accountability are inherently tied together.

    • There are, obviously, tradeoffs. I understand your point and I have more than a little sympathy for that point of view.

      However, I’m of the strong opinion that anonymous speech is required for a free society. Dismissing the availability of anonymous speech with “it means we just have to fight the leftists (and the far right, for that matter) to make sure they can’t distort it to something corrupt” is shortsighted at best. It’s like telling someone they to need to fight back when they are attacked by a bully then tying one (or both) hands behind their back.

      The cancel culture in the US is a pretty conclusive proof of that. Read The Gulag Archipelago to be convinced beyond reasonable doubt.

      • >>The cancel culture in the US is a pretty conclusive proof of that.

        (sigh) Yes, unfortunately you’re all too correct on that front.

        I don’t have a good answer, other than to assert that in a rational society, anonymity is unnecessary because retribution isn’t legal or culturally supported.

        Sadly we’re a long way from being a rational society.

        • I’ve said this many times before, but I think it would be difficult to say it too often:

          It is irrational to believe people are rational.

  9. Now, we know why it has been promises, promises, and nothing. And, why the AGs statement on election fraud – move on, nothing to see there – was BS.

    “Bill Barr ‘not intimidated’ by Trump’s ‘deposed king ranting’ on Twitter as pressure mounts over Hunter Biden probe”

    • Am I off base? This kind of attitude seems almost a confession that removes all doubt from my mind that the fix was in – Trump would not allowed to be reelected. That helps to also explains all the other falling dominoes.

      And the deep state – it’s in plain view and under our noses. The party tag makes no difference. It is now in full control of our government.

Comments are closed.