I’ve frequently read that important discoveries and inventions more often start with “That’s odd.” than with “Eureka!”

And so it was with a discovery of mine a week ago today.

I write software tools for the cyber security team at a major corporation. The culture is somewhat freewheeling. In the first couple of days when I started work my boss told me something to the effect of “People create their own positions here.” After the first couple of months I would talk to him no more than once a month. Sometimes it would be far longer than that. I did, pretty much, whatever I wanted. At review time I would be told, “We really like what you are doing and keep it up.” My model was look at what people around me were doing and write tools to make their job easier, faster, and enable better results.

I sometimes would joke that my goal was to eliminate the jobs of the people around me by writing the software to replace them. In reality what I did just meant people could be far more productive. Cyber security is never ending and I don’t see an end in sight for a job in this field as long as we have computer networks and human nature is what it is.

Nearly everything I did was little web application which would do things like check IP addresses for being on black lists and geolocate them, pull data and reformat from sensors, and graph data on “dashboards” for management to look at. My background task was working on something much bigger. I would see patterns in some of the data I was pulling from sensors and would try to get someone to investigate what I thought was suspicious activity.

The investigators would look at it for a few seconds and tell me something to the effect of, “I can see anything here. I need to see A, B, and C as well in order to know if this is anything.” So, a week or two later, I would show them similar data with A, B, and C added to the set. Again they would look at it for a few seconds, not see what I was seeing, and tell me they needed X, Y, and Z as well.

This went on for some time. I was somewhat frustrated and annoyed but I was learning how they did their jobs and what data they needed from multiple sources to evaluate a potential threat. But tens of thousands of rows in a spreadsheet with dozens of columns still didn’t allow people to quickly see the patterns I believed I was seeing. About two years ago I had kind of an eureka moment and I came up with a much better way of viewing the data (patent idea submitted to our attorneys was made late last year).

I started writing the software and explained it to anyone who expressed the slightest bit of interest in what I was doing. I gave the software the name “Bird Dog”. It essence it’s hunting through the grass and brush searching for specific things of interest to the (cyber security threat) hunter. It then points them out and then, when given the command, flushes them into the “air” such that only the the blind could not see them.

Everyone that sees it thinks it’s awesome but as much as I try I’m the only one that uses it. Everyone likes the data it produces but they don’t use it themselves. I think I need to make it easier to use but that’s a different story.

Last weekend I was putting in extra hours working on Bird Dog because I had gone through a major rewrite and it was to the point where things were working again as features were reenabled and new features were showing up. It’s very exciting to see what things will show up in the data with the proper visualization.

One of the things I had occasionally done in the past was to run a set of our externally facing IP addresses against the lists of “high risk” IPs. I didn’t have a complete set of our IP addresses but I had gathered some from public sources and had somewhat automated the process. I still had to copy and paste the list into a web app, click a button, and download the .CSV file into Excel. It didn’t take long but I never found anything and didn’t do it very often.

After the rewrite Bird Dog had a new data source. The new data source included more of our externally facing IP addresses. Bird Dog would now have not just my hand crafted list of IPs but IPs from the firewalls and other sources that might not be on any easily available list. And Bird Dog automatically added the risk scores to every public IP it saw, not just the IP addresses which were not ours (a previous limitation).

Last week during my testing of the new Bird Dog code one of our IPs was given a risk score indicating it was considered “Malicious”. That’s odd. I have been doing those sort of checks for years and I had never seen that before. But, it was one of the new features of Bird Dog and I knew it was possible.

I pointed it out to my boss. He and I spent a few minutes on it. We tried to find out why it was considered high risk but the supplier of the risk score for that IP had a 404 error on the web page for that one IP.

Another investigator was assigned and we looked some more. We didn’t make much progress and could create a story matching all the data that it was a false positive and we didn’t need to worry about it. We were about to close the case and move and when the vendor who had supplied the risk score showed up for a meeting.

One of the guys (who plans to attend Boomershoot this year, BTW) stopped by my desk and asked how things were going. We chatted about Boomershoot some and then I told him I was a little frustrated about the missing risk score “evidence” for the one IP address. They get their information from various sources and had provided a link to the original source which where I was getting the 404 error from. He pointed out his company had cached the web page and we could just click on a different link. It wasn’t obvious to either I or the other investigator and we both missed it.

Together the vendor and I looked at the cached web page. We quickly determined that as far as our network security was concerned it was certainly a false positive. But the data was something we couldn’t ignore.

When my boss, a former police officer, came back to the office I showed it to him and asked if it should be forwarded to the police for investigation. He asked me to write it up and forward it to him and he would forward it to Corporate Investigations who handles all interaction with law enforcement.

Within a couple hours the referral had been made. Later that day my boss wrote an email to our director (some details redacted, indicated by XXX, for various reasons):

Joe was working on his Bird Dog code and identified a XXX IP address labeled “High Risk” by XXX.  After additional analysis, Joe and Mike found the IP address was listed in a cached webpage where someone posted XXX links to suspected child pornography

…

This was immediately handed off to XXX and the appropriate LE referral was made to the National Center for Missing and Exploited Children.

Great work by the entire team to keep digging and hopefully contribute to protecting a vulnerable child!

Neither I nor anyone I know clicked on the links. We all know better than to do that. You never want to go there.

This is probably the best, for certain measures of best, find so far by Bird Dog. And it was totally inadvertent. Sometimes it’s the odd things that are the most important to follow up on.