I was nearly finished with a 20 page paper (of sorts) on searching for bots in computer networks when I took a break and scanned the contents of my RSS feeds. This struck me as particularly timely and funny:

As I told my boss last week I was disappointed in the algorithms used in what is considered “state of the art” tools. I actually found a strong inverse correlation in the “scoring” of network traffic of highly suspicious traffic compared to clearly normal traffic. The higher scoring traffic should indicate high probability of the traffic being communication with a Command and Control Server (C2 Server) and lower scores with normal traffic. I easily found instances where just the opposite was true.

When I used synthesized data I could get the expected scoring results but real world data demands new detection algorithms. It looks to me like bot builders also do research. Existing algorithms appear to be essentially garbage.

All this awareness would make us liable. Without them its ignorance, if we hire them it becomes negligence and I prefer ignorance.

RyanSepe
February 28, 2020
Suggested caption to this cartoon:

[There is way too much truth in this.

Companies have finite resources. They have to prioritize their cyber security efforts. If something is documented as an active issue, or even a potential weakness, and they don’t address it in a timely manner they have legal liability issues to deal with as well as fixing the problem.

In the “big picture” view of things companies have a lot of motivation to “not put it in writing” until they have the resources to deal with it. On the other hand, if managers don’t show they have a backlog and are overworked they aren’t going to get the resources to fix things in a timely manner. I have more than a little sympathy for cyber security managers caught in this dilemma.

After illegal computer access incidents have been made public Barb sometimes tells me, “I wish they would just stop doing that!” I would be out of a job, but the world would be a better place. So much money is spent on security that from a big picture you see it as huge waste of human and even natural resources (millions of computers monitor and guard against intrusion as their sole purpose). Even when the criminals are caught (extremely rare) they will never have to pay for all the resources spent in finding them and bringing them to justice.

And, of course, it’s never going to happen. Some of these criminals do it for the “free” money. Others do it for the thrill. And some do because they are spies in search of information useful to their country. There are always going to be those type of people. The best we can do is find them, stop them, and prosecute them if we can build a case against them.—Joe]

I like pushing the limits in certain directions.

Recently I have been spending nearly every waking hour working on my Bird Dog software for work.* I’m dealing with information on billions of network connections. I extract the stuff of interest and present it in an way which makes it easier to find the wood slivers in the hay stack. After using all the algorithmic tricks available I started finding places to do more parallel processing.

It was with great satisfaction that I found that I pretty much continuously keep all eight logical processers at 100 percent when doing certain tasks:

Each one of those processors is over 1000 times more powerful than the single processer I had on my first personal computer. And just the Bird Dog executable would take up over 75% of the hard disk space on that computer. Never mind the O/S or the database software which wouldn’t fit on a dozen hard disks I was so proud of at the time I first purchased it. “I’ll never run out of room on this disk!”, I foolishly told myself.

I now routinely open up text files in Vim for review and/or editing that are 50 to 100 times larger than what that hard disk could contain.

I like living in the future.

* I received an email from the company patent team earlier this week. They told me they are pursuing a patent on Bird Dog. I think the existing invention disclosure is okay, but the next one will be AWESOME! I’m really excited about what is coming up next. It’s as if a decade or more of my life’s work is coming to a focus on this one thing. I’ll probably need a more powerful computer, or set of computers, though.

I’ve frequently read that important discoveries and inventions more often start with “That’s odd.” than with “Eureka!”

And so it was with a discovery of mine a week ago today.

I write software tools for the cyber security team at a major corporation. The culture is somewhat freewheeling. In the first couple of days when I started work my boss told me something to the effect of “People create their own positions here.” After the first couple of months I would talk to him no more than once a month. Sometimes it would be far longer than that. I did, pretty much, whatever I wanted. At review time I would be told, “We really like what you are doing and keep it up.” My model was look at what people around me were doing and write tools to make their job easier, faster, and enable better results.

I sometimes would joke that my goal was to eliminate the jobs of the people around me by writing the software to replace them. In reality what I did just meant people could be far more productive. Cyber security is never ending and I don’t see an end in sight for a job in this field as long as we have computer networks and human nature is what it is.

Nearly everything I did was little web application which would do things like check IP addresses for being on black lists and geolocate them, pull data and reformat from sensors, and graph data on “dashboards” for management to look at. My background task was working on something much bigger. I would see patterns in some of the data I was pulling from sensors and would try to get someone to investigate what I thought was suspicious activity.

The investigators would look at it for a few seconds and tell me something to the effect of, “I can see anything here. I need to see A, B, and C as well in order to know if this is anything.” So, a week or two later, I would show them similar data with A, B, and C added to the set. Again they would look at it for a few seconds, not see what I was seeing, and tell me they needed X, Y, and Z as well.

This went on for some time. I was somewhat frustrated and annoyed but I was learning how they did their jobs and what data they needed from multiple sources to evaluate a potential threat. But tens of thousands of rows in a spreadsheet with dozens of columns still didn’t allow people to quickly see the patterns I believed I was seeing. About two years ago I had kind of an eureka moment and I came up with a much better way of viewing the data (patent idea submitted to our attorneys was made late last year).

I started writing the software and explained it to anyone who expressed the slightest bit of interest in what I was doing. I gave the software the name “Bird Dog”. It essence it’s hunting through the grass and brush searching for specific things of interest to the (cyber security threat) hunter. It then points them out and then, when given the command, flushes them into the “air” such that only the the blind could not see them.

Everyone that sees it thinks it’s awesome but as much as I try I’m the only one that uses it. Everyone likes the data it produces but they don’t use it themselves. I think I need to make it easier to use but that’s a different story.

Last weekend I was putting in extra hours working on Bird Dog because I had gone through a major rewrite and it was to the point where things were working again as features were reenabled and new features were showing up. It’s very exciting to see what things will show up in the data with the proper visualization.

One of the things I had occasionally done in the past was to run a set of our externally facing IP addresses against the lists of “high risk” IPs. I didn’t have a complete set of our IP addresses but I had gathered some from public sources and had somewhat automated the process. I still had to copy and paste the list into a web app, click a button, and download the .CSV file into Excel. It didn’t take long but I never found anything and didn’t do it very often.

After the rewrite Bird Dog had a new data source. The new data source included more of our externally facing IP addresses. Bird Dog would now have not just my hand crafted list of IPs but IPs from the firewalls and other sources that might not be on any easily available list. And Bird Dog automatically added the risk scores to every public IP it saw, not just the IP addresses which were not ours (a previous limitation).

Last week during my testing of the new Bird Dog code one of our IPs was given a risk score indicating it was considered “Malicious”. That’s odd. I have been doing those sort of checks for years and I had never seen that before. But, it was one of the new features of Bird Dog and I knew it was possible.

I pointed it out to my boss. He and I spent a few minutes on it. We tried to find out why it was considered high risk but the supplier of the risk score for that IP had a 404 error on the web page for that one IP.

Another investigator was assigned and we looked some more. We didn’t make much progress and could create a story matching all the data that it was a false positive and we didn’t need to worry about it. We were about to close the case and move on when the vendor who had supplied the risk scores showed up for a meeting.

One of the guys (who plans to attend Boomershoot this year, BTW) stopped by my desk and asked how things were going. We chatted about Boomershoot some and then I told him I was a little frustrated about the missing risk score “evidence” for the one IP address. They get their information from various sources and had provided a link to the original source which where I was getting the 404 error from. He pointed out his company had cached the web page and we could just click on a different link. It wasn’t obvious to either I or the other investigator and we both missed it.

Together the vendor and I looked at the cached web page. We quickly determined that as far as our network security was concerned it was certainly a false positive. But the data was something we couldn’t ignore.

When my boss, a former police officer, came back to the office I showed it to him and asked if it should be forwarded to the police for investigation. He asked me to write it up and forward it to him and he would forward it to Corporate Investigations who handles all interaction with law enforcement.

Within a couple hours the referral had been made. Later that day my boss wrote an email to our director (some details redacted, indicated by XXX, for various reasons):

Joe was working on his Bird Dog code and identified a XXX IP address labeled “High Risk” by XXX.  After additional analysis, Joe and Mike found the IP address was listed in a cached webpage where someone posted XXX links to suspected child pornography

…

This was immediately handed off to XXX and the appropriate LE referral was made to the National Center for Missing and Exploited Children.

Great work by the entire team to keep digging and hopefully contribute to protecting a vulnerable child!

Neither I nor anyone I know clicked on the links. We all know better than to do that. You never want to go there.

This is probably the best, for certain measures of best, find so far by Bird Dog. And it was totally inadvertent. Sometimes it’s the odd things that are the most important to follow up on.