That’s odd

I’ve frequently read that important discoveries and inventions more often start with “That’s odd.” than with “Eureka!”

And so it was with a discovery of mine a week ago today.

I write software tools for the cyber security team at a major corporation. The culture is somewhat freewheeling. In the first couple of days when I started work my boss told me something to the effect of “People create their own positions here.” After the first couple of months I would talk to him no more than once a month. Sometimes it would be far longer than that. I did, pretty much, whatever I wanted. At review time I would be told, “We really like what you are doing and keep it up.” My model was look at what people around me were doing and write tools to make their job easier, faster, and enable better results.

I sometimes would joke that my goal was to eliminate the jobs of the people around me by writing the software to replace them. In reality what I did just meant people could be far more productive. Cyber security is never ending and I don’t see an end in sight for a job in this field as long as we have computer networks and human nature is what it is.

Nearly everything I did was little web application which would do things like check IP addresses for being on black lists and geolocate them, pull data and reformat from sensors, and graph data on “dashboards” for management to look at. My background task was working on something much bigger. I would see patterns in some of the data I was pulling from sensors and would try to get someone to investigate what I thought was suspicious activity.

The investigators would look at it for a few seconds and tell me something to the effect of, “I can see anything here. I need to see A, B, and C as well in order to know if this is anything.” So, a week or two later, I would show them similar data with A, B, and C added to the set. Again they would look at it for a few seconds, not see what I was seeing, and tell me they needed X, Y, and Z as well.

This went on for some time. I was somewhat frustrated and annoyed but I was learning how they did their jobs and what data they needed from multiple sources to evaluate a potential threat. But tens of thousands of rows in a spreadsheet with dozens of columns still didn’t allow people to quickly see the patterns I believed I was seeing. About two years ago I had kind of an eureka moment and I came up with a much better way of viewing the data (patent idea submitted to our attorneys was made late last year).

I started writing the software and explained it to anyone who expressed the slightest bit of interest in what I was doing. I gave the software the name “Bird Dog”. It essence it’s hunting through the grass and brush searching for specific things of interest to the (cyber security threat) hunter. It then points them out and then, when given the command, flushes them into the “air” such that only the the blind could not see them.

Everyone that sees it thinks it’s awesome but as much as I try I’m the only one that uses it. Everyone likes the data it produces but they don’t use it themselves. I think I need to make it easier to use but that’s a different story.

Last weekend I was putting in extra hours working on Bird Dog because I had gone through a major rewrite and it was to the point where things were working again as features were reenabled and new features were showing up. It’s very exciting to see what things will show up in the data with the proper visualization.

One of the things I had occasionally done in the past was to run a set of our externally facing IP addresses against the lists of “high risk” IPs. I didn’t have a complete set of our IP addresses but I had gathered some from public sources and had somewhat automated the process. I still had to copy and paste the list into a web app, click a button, and download the .CSV file into Excel. It didn’t take long but I never found anything and didn’t do it very often.

After the rewrite Bird Dog had a new data source. The new data source included more of our externally facing IP addresses. Bird Dog would now have not just my hand crafted list of IPs but IPs from the firewalls and other sources that might not be on any easily available list. And Bird Dog automatically added the risk scores to every public IP it saw, not just the IP addresses which were not ours (a previous limitation).

Last week during my testing of the new Bird Dog code one of our IPs was given a risk score indicating it was considered “Malicious”. That’s odd. I have been doing those sort of checks for years and I had never seen that before. But, it was one of the new features of Bird Dog and I knew it was possible.

I pointed it out to my boss. He and I spent a few minutes on it. We tried to find out why it was considered high risk but the supplier of the risk score for that IP had a 404 error on the web page for that one IP.

Another investigator was assigned and we looked some more. We didn’t make much progress and could create a story matching all the data that it was a false positive and we didn’t need to worry about it. We were about to close the case and move on when the vendor who had supplied the risk scores showed up for a meeting.

One of the guys (who plans to attend Boomershoot this year, BTW) stopped by my desk and asked how things were going. We chatted about Boomershoot some and then I told him I was a little frustrated about the missing risk score “evidence” for the one IP address. They get their information from various sources and had provided a link to the original source which where I was getting the 404 error from. He pointed out his company had cached the web page and we could just click on a different link. It wasn’t obvious to either I or the other investigator and we both missed it.

Together the vendor and I looked at the cached web page. We quickly determined that as far as our network security was concerned it was certainly a false positive. But the data was something we couldn’t ignore.

When my boss, a former police officer, came back to the office I showed it to him and asked if it should be forwarded to the police for investigation. He asked me to write it up and forward it to him and he would forward it to Corporate Investigations who handles all interaction with law enforcement.

Within a couple hours the referral had been made. Later that day my boss wrote an email to our director (some details redacted, indicated by XXX, for various reasons):

Joe was working on his Bird Dog code and identified a XXX IP address labeled “High Risk” by XXX.  After additional analysis, Joe and Mike found the IP address was listed in a cached webpage where someone posted XXX links to suspected child pornography

This was immediately handed off to XXX and the appropriate LE referral was made to the National Center for Missing and Exploited Children.

Great work by the entire team to keep digging and hopefully contribute to protecting a vulnerable child!

Neither I nor anyone I know clicked on the links. We all know better than to do that. You never want to go there.

This is probably the best, for certain measures of best, find so far by Bird Dog. And it was totally inadvertent. Sometimes it’s the odd things that are the most important to follow up on.


14 thoughts on “That’s odd

  1. It’s also worth noting how important it can be to keep going down the rabbit hole until you get your answers. Most people don’t have the patience, the time, or both, to thoroughly follow up every oddity. So good judgment and discrimination is also important.

    Nicely done!

    (Another exciting phrase in the lab is of course “what the…!?”)

    • Well, you won’t actually find out what is IN the rabbit hole until you get in it. In “The cuckoo’s Egg”, Cliff Stoll was originally tasked with finding out why there was a 75 cent overage in the account, and by tracking that down he discovered hackers in West Germany were looking for things to sell to the Soviets.

      And it is as you say, the first thing a scientist says is “Why did it do THAT?” Second question is “Why? THEN the experiment is constructed to test the hypothesis. To leave all that out is to render our children stupid and unable to distinguish between a hypothesis (I.e. The world is flat or no one has been to the Moon) and a theory (I.e. Relativity, Evolution)

  2. Good job, Joe!
    Another friend of mine worked on a team whose specific mission is/was tracking down child porn. Unlike your situation, they had to review the media before referring the case to domestic or international law enforcement. They had a psychologist come in to counsel them for PTSD every few weeks, the job stress was so horrendous.

  3. Isaac Asimov, to the best of my knowledge, is the one who made the observation about “Eureka” vs. “that’s odd”. I remember reading it back in the ’70s, most likely in one of his collections of essays.

    I hope to hear soon on the release of your software – I’m in the information security trade now, and good tools are always welcomed.

    The Real Kurt

  4. A while back, I wrote a software (even though its not my real job) at the laser mine. Just a special purpose calculator for adjusting laser readings. It takes the place of writing readings on a form and tapping them into a calculator, then entering the results into the laser comm software. Entering the readings on the form (for the record) and the software are still needed, since I am not a real programmer, don’t have access to certain information, and wrote the little thing while being complained at for “not working”. I find it useful, none of the other techs use it. But, I can write the numbers in with my left hand and tickle the computer with my right, so I have that going for me. This particular product is playing footsie with end of product life, so fugit. .

  5. I read an article from another tech detective explaining how the US spy tech crews can install any kind of porn on your computer and completely hide the evidence they did it . Make it appear you are guilty and hide their tracks .

    • Having read about what the Israeli techies did with undocumented holes in the programs run by the Iranians, and how Sharyl Attkisson’s computer would turn on by itself, I don’t doubt what you say.

  6. Good work, Joe!

    Re: Visualization. Have you looked at Tableau? “Telling stories with big data sets” is their thing. May not be applicable, as in this story, it was all about one evil IP.

    • Yes. I haven’t actually used it but I did look at it pretty close and talked to a friend of mine from MS who works there now. Unless it has add some drastically new features it doesn’t come close to what I’m doing. If you want to have lunch sometime I’ll explain it.

Comments are closed.