On The Wire

(Content not available)

Fortune 100 Company has serious Citrix issues affecting over 22,000 users regularly. Our Remote Diagnosis service was exactly what they needed to uncover the technical causes of the problems. Graphical annotation helped dozens of technologists and managers understand the problem and improved their ability to collaborate with their vendors to gain consensus toward mitigation. Very successful optimization of existing systems.

Our participation remotely on the trouble call bridge, working with their packet capture team to gather the appropriate packets enabled successful remediation of dozens of Citrix related problems. This company had excellent project management, packet capture and technical teams.  They needed an experienced skilled analyst to uncover the details so consensus could occur between teams and vendors.

After spending $600,000.00 on server CPU upgrade that did not improve response time, this company found the real problem they were trying to resolve.

We helped them diagnose the true cause of the slowdown.  It was the Java code in the client that was causing an 89 second slowdown in the application. Once pinpointed the client was able to show our annotated proof to the vendor responsible and the problem was fixed with a simple change.

Global 2000 Company receives 10x Application Throughput after diagnosing problem with Onsite Analysis. This organization received great reward for not one dollar of new equipment spending. Solved with a VLAN parameter change we identified, carefully implemented through change control and validated with monitoring to demonstrate return on investment.

This organization had excellent change control where the technologists would presented their change to other technologists in a presentation room allowing other organizations and technologists to understand and challenge the viability of the change before it was approved. The evidence we provided made easy work of bringing consensus with other groups and vendors.

Looking at a Packet Trace File is often the first step in uncovering the problem.  Like the TV Show "Yankee Workshop" just buying or having the same tools doesn't mean you can turn out beautiful results. No matter how smart, tools require skills that takes time and one must be expertly trained, and use the tools regularly so they become second nature.

The three "T's" are Tools to analyze, with knowledge of Theory, the second "T", on how protocols transact enables diagnosis.  In order to diagnose issues one must possess the Tools, Theory and the Time, the third "T" to be proactive. If missing any of the 3 "T's" our Packet Trace File Analysis services can bring paradigm shifts that overcome silence or finger pointing.

Datacenter consolidation creates performance problems for slow Oracle 11g Client queries.

An application's server infrastructure was moved from an international location into a central datacenter in the USA. The application uses data sources on a MS-SQL database, but weekly the application requires an extraction from an Oracle database.

The weekly Oracle data extraction original physical server environment took 45 minutes to complete in the international location on native physical servers. After moving to a USA virtualized datacenter the Oracle data extraction took over 18 hours and crashed before completion.

The part of the application, the weekly data extraction apparently was not tested before the application cutover. Now that the cutover has occurred the situation has become a high severity issue affecting business.

The organization has a highly effective cross organizational project management team that gets involved when high severity business impact exists. The project managers bring in all the responsible parties onto a conference bridge managing across technical and management silos.

The global services group has central analysis tools, in this case NetScout's Sniffer InfiniStream. The servers in the former international environment do not have infrastructure embedded packet capture capability so we have been using MS-NetMon on the platform as it is included by Microsoft with the server and easier to allow its use from a logistic and security perspective.

The packet capture traces illustrate slowness not on the part of the Oracle server where the data is being extracted from, but in the requesting Oracle Client running on a 2008 Server with MS-SQL 2008 R2 (or R1). The Oracle Client (running on the new virtual MS 2008 SQL Server in the datacenter) sends a request to the Oracle Linux server also in the datacenter. The reply is very fast with about 10,000 bytes. After receiving the 10,000 bytes the Clients sits for roughly 150 milliseconds after fully receiving the previously requested data before it sends the SQLnet "get next row" request. It is this delay before spawning the get next that causes the slowness for many thousands of the same "get next row" requests.

The problem was found and analyzed in the new virtualized datacenter production environment and replicated / confirmed in the new virtualized datacenter development environment.

In Oracle and third party support forums many have reported the same difference in Oracle Client vs SQL plus / Microsoft SQL Management Suite performance.

Learned that the part of the Oracle Client 11g being used is the quite legacy OLEdb still being included and that Oracle does not support its client in any virtual environment other than its own! Therefore the only option is to use native physical machines...

Pentagon 911 Lessons Learned - IT Disaster Recovery Plan

Bill Alderson discusses the technical and IT Best Practice lessons learned from the events subsequent to the Pentagon 911 disaster. Several key IT Disaster Recovery Plan lessons can be learned from this video.

Transcript for this Video

Good afternoon. This is Bill Alderson coming to you from beautiful Austin, Texas. I hope everyone is doing well today. I’m going to have a short presentation for you. It’ll be under 30 minutes in respect of your time.

We’re going to talk a little bit about the Lessons Learned 10 Years Later, from the Pentagon 9/11 disaster. I was privileged to be one of those that got called. We all remember the event and probably where we were, what we were doing, who we were with, how we responded. Our world was kind of devastated for a while. A couple of days later I got a call from a Pentagon General asking if we wouldn’t come in and bare a hand in helping them recover communications. It was obviously a very proud time in my life to be prepared and to be asked to serve in that capacity.

We responded and came in and helped out as best we could. We analyzed a lot of various problems. We took a look at various communication circuits, network management systems, security infrastructure, pretty much the whole gamut of IT. We analyzed, assessed, looked at, and prepared a 100-page report for them so that they’d know the results of our analysis and then how to proceed with that information.

My organization serves large-scale environments, the Global 2000, military, government. In the last 10 years, we’ve gone into Iraq a half a dozen times and Afghanistan and helped diagnose problems across the entire war environment, and then back to the Pentagon to analyze other things. That’s the nature of our service. It’s high visibility, large-scale critical problem type environments.

In the last 20 years of diagnosing critical problems down to the technical root cause, we’ve learned a few other things. We’ve learned that it’s not just the technical problem that we want to avoid, but the processes which lead to the ability of those technical problems to exist. Essentially, our root cause is changed from technical root cause back into the best practices that obviate those problems. We look back and find what processes were omitted or not in effect.

Typically I’ve got a CIO whose enterprise was just melting down and we came in, we diagnosed the problems, mitigated the problem, and of course they have to ask, “Well what can we do to prevent?” Not only in the last 20 years have we been diagnosing critical problems, but we’ve been analyzing what kinds of procedures, processes, and systems allowed those critical problems to exist. That’s kind of the way that we’re moving forward with the new Apalytics Corporation, to help people mitigate and identify those things that they can do that will avoid crisis and avoid problems.

I hope you came to learn a little bit and to just see my perspective and that sort of thing. Now, what we’ve done is we’ve kind of taken all of these lessons learned and we call this Technical Systemization. There are pillars to technical systemization. We’re going to go through some of those pillars today because they directly relate to the events of 9/11 and pretty much any other type of disaster. Taking the lessons learned and applying them.

Now, Technical Systemization is very much an implementation. ITIL is a model. ISO has models. We have protocol models. We have the OSI model. Well, the OSI model is just that, it is a model. It is not an implementation. I want to make and draw a distinction between that of a model, like ITIL is a model, the OSI model is a model, but an actual implementation, such as IPX, TCPIP, or Apple Talk. Those are true implementations. They look different from the model because they are applied to your exacting problem and your exacting environment in such a way as to have peace to make things truly work.

By changing the name of your organizations to model, to be the same as, for instance the ITIL model, is not necessarily implementing the technical systemization that will allow that to work. That’s part of our message. Each institution needs to have their own Technical Systemization, their implementation of ITIL. ITIL is not a protocol like TCPIP. TCPIP is an implementation of a model, but not the model itself. That’s not the implementation.

One of the things that we’ve learned is that there are a bunch of silos out there. The Pentagon has silos. A lot of them have demarks for who runs the network, desktop, server, platform, network management, and security. All of these are one of many silos. What we call Business Technology Integration is the ability to do cross-silo collaboration, whether it is system design, metrics and monitoring, or network documentation.

Let’s just take network documentation for a moment. As I travel around and look at various places I walk and invariably I see diagrams on the wall. Sadly, most of them are labeled 2008, 2007, 2010 and they are typically dated and that sort of thing. I don’t know why they’d still be on the wall. Nevertheless, it’s probably because they haven’t been updated. It’s an interesting thing. If you take all your business silos, desktop silo, network silo, application and platform silos and with each one of them build a network diagram or a system diagram, if you were to take those and lay them horizontally from the client to the server and the application, could you put those things together and say, “Okay, here’s the desktop, here’s the network, here’s the WAN, here’s the security infrastructure, here’s the platform, and here’s the application and all the dependencies”? Could you put your finger on the diagram on the left and give a troubleshooter and end-to-end diagram? I don’t think so.

Case in point, when we arrived at the Pentagon, we came into a room and we started, “Okay, first thing, we need to know a little bit of the lay of the land.” There was kind of a gulp and they said, “We have a great network documentation system, however, it was one of the servers impacted in the disaster.” They didn’t have anything printed out because it was online for ready reference. Don’t forget, sometimes those systems go down. It’s probably a good idea to have the recent version printed out somewhere so that you can refer to it.

We started with a blank white board and started diagramming. Who had to do that diagramming? Well, the key technologists that had a lot of other fires burning. We came in to help. Because these key technologists had to help understand so that we could bare a hand and start helping them so you’d have more people helping, they were out of their key duties. It’s incumbent to make certain that your key technologists are documenting the systems and that you are training everyone so that everybody knows the key elements of the system.

We’re going to talk a little bit about that, but that’s one of the examples of how organizational silos need cross technology collaboration teams to pull together system documentation that’s congruent. It gives you an end-to-end picture in network monitoring metrics and that sort of thing.

Back to The Art of War, Sun Tzu and The Art of War is one of the books that are studied by most military officers at one time or another. The bottom line to this is ‘Know Yourself’ and ‘Know Your Enemy’. That’s a fundamental thing. If you know yourself, you can react to things. In the area of knowing yourself, we need to develop end-to-end cross technology, cross-silo documentation.

We have literally hundreds of people who are willing participants in optimizing our network systems and applications for the end users, but I find secretly that some of the people who come to the table a little bit later, the people who’ve come in the last few months, they’re not going to come and tell you, “Hey, I’m really impotent because I don’t know where anything is. I don’t have a diagram. You don’t have a diagram.” They’re not going to come tell you this. It is one of the biggest lost production areas that your organization has. You have hundreds of people who are willing participants, but yet are impotent to participate because there is very little true end-to-end cross technology, cross-silo documentation.

When there’s a trouble ticket they can, first of all, identify the client, put their finger on the client network. Then they can start moving to the right and into the network and see the network infrastructure that they’re dependent upon. Then they can move their finger to the right and see that firewalls, and load balancers, and WAN optimizers that they’re dependent upon for that system to work. Then they can move over to the right and look at the data center, the data center distribution, into the virtual platform, into the virtual image that they’re running, then into the virtual switch, then on into that process that they’re connecting to, then back into the application, and then the application having secondary responsibilities or geared capabilities. Now you have the ability for a help desk person to kind of understand your infrastructure and very rapidly assimilate and at least triage where the problem may or may not be.

Then you can see what- Network Management. You can say, “Test Point One. Test Point Two. Test Point Three. Test Point Four.” You can visualize your infrastructure. Probably the number one thing that I would stress is that you must know yourself. If you don’t have that, it’s costing you orders of magnitude more than you think and a lot more than it would cost you to put these systems in place.

Now, once you have that cross technology, cross-silo documentation and everyone can meaningfully participate, then you make sure you train everyone on the design constraints, your architecture, and certify them on your system architecture, your abilities, your capabilities, your constraints of your architecture so that they’re not trying to go in and modify to make it look like where ever they came from previously. You want your architecture to be pure and to work systematically with all your various components. It’s important to educate everyone on that part.

Now, one of the things, and an example of something that we’ve done inside large scale environments, is to help people to develop scalable cross location, cross domain data collection capabilities. Today, you have so many firewalls, load balancers, locations, different domains, different security levels, etc, that you can’t just go out and discover your network and then map it. It’s very difficult. If someone can, usually they can do it in one area, but not in another area.

Our philosophy is you pull all that information from a variety of sources into a single repository across the organization. That becomes what we call, Rapid Network Rendering Database. That database becomes a repository and you create a SOA, over to the right there you’ll see a SOA, so that other corporate systems can come in so that you can export to your network management systems to make certain that you’re monitoring all of the various dependencies of applications and systems.

Over on the left, when data goes into that RNR database, one of the things that we do is we make sure that it has the correct endpoints on it. When those systems have the correct endpoint, when those objects, your network management objects, your router port objects, your switchboard objects, have ends placed on them, you can do reconnection of those objects. When they are simply an object and don’t have two endpoints, it’s really difficult. Just populating a database without those endpoints connected on both sides isn’t going to do you any good. You’re not going to be able to export them.

We’ve designed a way of, when that stuff comes in, we put the endpoints on them and then we connect the points and populate a Visio diagram so that you can see the details of your network infrastructure port by port logically in a visual way to get the VLANS and all the virtual machine information so that a troubleshooter can truly see the environment that they are trying to troubleshoot. That’s just one particular way that we like to see things done.

Now, the second thing is know your enemy. Your enemy is a Root Cause of the problems that you have. You want to perform formal root cause analysis whenever possible. If your helpdesk solution is to always reimage, reboot, or reload, that’s problematic. They should take a moment. They should take just the opportunity for a moment to make sure . . . I’m not saying leave the problem go forever, but I’m saying resist the temptation to reboot a server or even an end user workstation until you’ve had the opportunity to collect a little bit of data. That data doesn’t have to be analyzed right there at that moment, but can be put together as a collection of information, trace files, etc. to then do analysis retrospectively. A lot of folks have some of those capabilities, but I’m saying that you need to involve your helpdesk in capturing those sort of things so that the recurring problems are able to be diagnosed and then obviated.

Hey, the best solution to any problem is not to have it in the first place. Unfortunately, few are motivated with such things. Management needs to know how many trouble tickets you’ve closed and how many problems you’ve solved. Well, wouldn’t it be great to say, “Because we were so proactive, we obviated the problems”? Of course, at that point you lose your justification for the tools, systems, and people. Truly, that should be the objective of any senior management person, to obviate and have all your technologists actually there, but kind of like the Maytag repairman without a lot to do, but when the problem occurs they’re there to help.

The other part of it is the ITIL problem management analysis, going about finding out your statistical number problems that you have. At the Pentagon, when we arrived there, they were getting right about 100,000 SNMP traps a day. Well since then, they’ve started a filtering mechanism. It was so overwhelming, 100,000 a day, that they were not able to respond to the ones that were the most critical and severe. They learned that, “Okay, if you’re going to have 100,000 SNMP traps a day, you’d probably better figure out which ones are your most material, which ones are your biggest ones and how to go about taking a look at that.”

Now, the third thing that I want to talk about in Problem and Change Management is that I go into organizations and I find something that’s amiss and all we have to do is change a few parameters or repurpose a couple of existing systems or capabilities. Invariably, they look at me and they say, “Well, we have no way of promulgating this optimization. We have no way to put this into change control and to have a fix. We either have to have an emergency crisis that’s burning down the building or we have to have a multimillion dollar procurement in order to promulgate and have a set of steps in which we can do change.” My recommendation is to make sure that you have a way to implement free changes that are not procurement or catastrophic kind of response changes. There are many free ways of improving things.

I was at a site a few weeks ago. We went in and their entire Internet access was very slow. We went in and diagnosed the exact cause of the problem. We tested. We implemented, proved that it mitigated. They looked at me and they said, “We have no way of implementing this change to thousands of our users.” They have no way of linking it unless they’ve made a three or four million dollar procurement to any ability to change their infrastructure. Anything that would be cost free has no mechanism by which you can implement and repurpose your systems with just a few process changes. I’d recommend that you think about that. Make sure you can do things that are free.

The other thing is, is there a Master Plan? Now, after the Pentagon disaster, these guys were prepared. They had master plans. They had in-the-event-of plans. They had future plans that mapped the way forward. They took our documents, other things that we learned, and a fresh realization of the threats and vulnerabilities and they integrated that in rapidly. They took that forward. Congress actually allocated a half a billion dollars to beef up the vulnerabilities of the Pentagon. That wasn’t because they, Congress, wanted to spend money. It was because the people who managed the Pentagon network infrastructure had an articulated strategy and they knew where their threats and vulnerabilities were. They had it articulated in a well-articulated plan that allowed them to move that forward and to get champions within Congress and within their leadership in order to accomplish these feats.

If you don’t have a Master Plan, what you’re doing today, your ‘is’ environment, and then you have a disaster, what’s going to happen is you’re probably just going to rebuild in the same way that you were before. That’s, of course, going to be a few generations behind. In the way that we talk about it today, if you’ve seen those insurance commercials, it’s like, “Man, if your car is totaled we’re not going to give you the same 1974 Pinto station wagon. By golly, you’re going to get a 1975 Pinto station wagon in replace of your old car.” It’s the same thing here. If you have a well-articulated plan and there’s an emergency, a problem, or an unfortunate situation and you have to rebuild part or your entire infrastructure, if you have this plan, it’s an excellent place to start. It’s a good thing to do.

The next thing is Decision Support Metrics. Well, remember all those systems that were given 100,000 SNMP traps a day? Which ones are important? Which ones aren’t? Well, those were all SNMP traps of yesteryear, the red-green, not working or working. Today, it’s more about performance. What we call Decision Support Metrics is developing a collaborative cross silo management capability validating your end user application capabilities with signatures so that your monitoring systems should be validating any potential changes. Talk about change control. If your monitoring system is monitoring all of the vital signs of your key mission critical applications and you make a change it’s going to tell you, boom, immediately, that you have a problem affecting one of your end user applications or capabilities. That takes a lot of care and feeding.

The secondary thing about good Decision Support Metrics is that if you know right where your users are slow and things that you obviously can’t optimize without a project… You have a whole list of projects that are out there that you would like to spend on. If you spent in every area: storage, network, and every part of your infrastructure- your budget would be depleted rapidly. In today’s day and age, you use Decision Support Metrics to identify your bottlenecks so that your portfolio spending can be pin point accurate. You can take what you learn from those metrics and you can apply them to your budget allocation, your resource allocation and you know that you are going to get a return on investment that is worthy of the money that you’re spending.

Today’s CIOs have an incredible problem. They’re asked to upgrade in a forklift fashion across the board. It’s cost prohibitive. You can’t do it. Even more so today, we need to have Decision Support Metrics that are helping us prioritize exactly where we’re going to spend the money and quantify the return on investment by indentifying end user signatures, by identifying the dependencies and the process flow of the application, especially when you’re starting to talk about cloud. That gives me a whole other thing to talk about and we’ll do that another day. Now we have dependencies and the vulnerabilities of moving to cloud, what about cloud, and then cloud to cloud. How are you going to analyze those problems? We’ll talk about that another day.

Okay, so performance indicators. You’ve got NetFlow to go back and forth for rate and flow information to find out if certain applications are working so you have network flows. You have device status. That’s your SNMP. What’s going on with your components? Do you know that your executives and your business leaders do not care about the status of a router, a switch, or something? What do they care about? They care about the capabilities and the abilities of their people.

In the war environment, commanders don’t care that this link is down or that link is down. What they want to know is, “Can we carry on certain command functions?” We have to translate our performance indicators from what we’re comfortable with, which are router ports and switch ports and links, into to what our executives and our end users care about. Where am I impacted? Who is impacted? What applications and what capabilities do I have or not have to run my business or my command on?

Knowing what that path is between the client and the server, and the response time at the server, and then the latency going back and forth, and the ability to capture those transactions across your network, is very important.

In talking about multi-points and multi-tier I’d like to just mention that when you have good Decision Support Metrics, you can see that the process, in other words the CPU, of your web tier or the CPU of your app tier, or your SQL tier or mainframe tier, or your security authentication is consuming most of your response time. By having that multipoint, multi-tier transaction analysis, the ability to know exactly what is slowing down your end user transaction that allows you to spend in the right areas, diagnose in the right areas, and serve your end users well.

Now, back to the Technical Systemization, you can go on our website. We’ve got this model on our website of our Technical Systemization. There are thousands of things that you need to do. There are hundreds of procedures and processes, but these six things we believe. If you do these and you get these built into your culture of your organization, that people can remember them, they can focus on them. It’s not too big. Saying, “We have to implement all of ITIL,” is overbearing. “All the ISO standards”- overbearing. When you say, “Hey, our organization is going to implement these six fundamental things,” and they become the pillars of your ability to operate… These are the things that we believe are important.

Just a little bit about the company. Some of you know me and have worked with me before. Those who don’t, I look forward to that potential opportunity. We are dedicated to Crisis Avoidance. We want to help you eliminate the problem through the integration of Best Practices in your organization. If you have a problem, our Assessment Services are here to provide you immediate response, Technical Services to help you resolve problems, mitigate issues, and then to perform assessments to find out where you might be vulnerable. If you’re in a proactive state, if you’re looking at spending new money on infrastructure or Decision Support Metrics, we can come in and help you assess and help you build those things out. Our Cornerstone Services will come in and help you with an embedded analyst so that they can help you promulgate these types of virtual teams into those various areas and improve and benefit your entire infrastructure.

In summary, documentation, Documentation, DOCUMENTATION. Maintain a highly skilled technical staff. You may have a lot of vacancies. The prevailing thought is that those vacancies are more valuable than the average employee. In other words, it’s better to have a vacancy than it is someone who is of average. If you have average people, you need to build them into highly skilled technical people so that they can materially… Part of that is the CIO’s job to make sure that they are empowered. Have that Master Plan, employee problem management, and change control, dynamics, Decision Support Metrics that drive your portfolio spending, Business Technology Integration to make sure you have a collaborative environment.

Then those virtual teams that help you. Virtual teams are implemented without any reorganization. If your organization is ineffective, you apply virtual teams to address certain problems and pick the right people from each one of those silos and you can change your organization and improve things without even having to reorganize. Then finally, be an end user advocate. Make those end users your priority.

I’m Bill Alderson. It’s been nice meeting with you today. I respect your time and energy. I appreciate your coming along today. If you have any questions or comments please email me at [email protected].

I appreciate you stopping by today and we look forward to talking to you in the future.

I’m out, Bill Alderson.

Pentagon 911 Lessons Learned - IT Disaster Recovery Plan Video

Enterprise Architecture IT Disaster Recovery Plan Steps to APM Under Pressure

This session takes ideas from many IT disaster critical problem resolution IT consulting engagements and puts them together into 30 minutes on how to avoid IT Crisis. This presentation identifies IT best practices that can be used when creating an IT Disaster Recovery Plan, particularly if the disaster affects application performance. Your staff will appreciate the references on how to troubleshoot network problems during an IT disaster.

Enterprise Architecture defines a multi-site, multi-vendor network with multiple computer operating systems integrated by the benefiting institution, supporting multi-tier mission critical applications, having its own monitoring and support organizations. Compare your internal processes to the set of best practices known to obviate crises for inclusion in an IT Disaster Recovery Plan

Transcript for this Video

Good afternoon. This is Bill Alderson. I appreciate you joining us this afternoon for just about 30 minutes or so, as we talk about six things that you can do. In many, many years of performing critical problem resolution, going in when nobody else was able to solve it, and high stakes situation, high complexity, lots of people waiting for you to fix the problem type of thing, my mind always goes to the end of the engagement when everybody stands around and says, “Oh, that was a pretty simple solution.” Typically, there are multiple things that go wrong in a crisis or a critical problem. When you go back and reconstruct or do the anatomy of a critical problem, you always find that the root cause is some technical problem, of course, typically or configuration related or something of that nature.

After we're done doing the critical problem resolution, everybody says, “Well, why did that occur? How did that problem get us to this debilitating point? How can we prevent it in the future?” That's been my guiding light to help people, yes, go in and help them when they do have a problem. We do that. If I can obviate problems through best practices, the things that you can do to avoid a problem, that's much better than having a problem that you have to resolve. I think we'd all agree.

Today I'm going to talk a little bit about six of the things that you can do. Of course, there are hundreds of things that we can do, literally hundreds. There are best practices, ITIL, standards of various types, certifications of various types, and all sorts of things that we can do, but we can't remember all of them all the time, every single day. I'm going to bring to you six of the things that you can do to avoid crisis, to avoid critical problems, and to avoid problems in general. That’s what we're going to talk about.

This is our website here on the top left. You'll see we've got a variety of services. We've taken the Jefferson Memorial and we've used it and the pillars in the Jefferson Memorial to illustrate our services and the steps to illustrate other services and that sort of thing. If you have a chance some time, you ought to hit that “Start Here” button on the front of our Apalytics web page and go through and just listen to see how the web page works and that sort of thing, and to look at our collaboration systemization model. It's pretty cool.

Then over on the right we've got CIO Tech and we’ve got SynSynAck.com. Syn Syn Ack, of course, is the TCP three-way handshake and we talk about performance analysis and that sort of thing. Then CIO Tech, we talk about things that are of interest to the CIOs. That's a little bit about us.

Little bit about me, I started as a communications engineer at Lockheed Missiles and Space Company. If you read Steve Jobs' book, not his book, but the book by Walter Isaacson that just came out, which I highly recommend because it really does a good job of explaining how Steve Jobs and his company, his problems, the adversities and that sort of thing drove this guy to build some really excellent products that the market really loved. In that book, he talks about how Lockheed, Fairchild and the other companies in the Silicon Valley, the defense money, and the defense contracting that was in there actually spawned the high tech industry and chip building and that sort of thing. Of course, I can remember when we did multi-layer boards over there and wire wrapping. That was the beginning of printed circuits. Then they just kept getting smaller and smaller.

I joined Network General in the mid '80s, and we started working on the Sniffer. That's when I really got into protocol analysis and critical problem resolution. It was in '86, '87. Subsequent to that, I started Pine Mountain Group. Some of you may know me from that. I created the Sniffer Training program. I licensed that to Network General and they trained thousands of people. I also crated the certified net analyst program and I trained about 50,000 people in 22 countries, of whom we certified over 3,000 people as certified network forensic professionals. During that time, and in my career, I've been able to provide service or training to 75% of the Fortune 100. I've been around in a lot of various environments.

I sold Pine Mountain Group to Net QoS, a performance management company, in 2005, where I was one of their technology officers. Then Net QoS sold out to CA Technologies in 2009. I hung out for just a little longer than the founders did. Then I decided that I had another company in me so I started Apalytics Corporation to help people solve problems and learn from those problems to provide and implement best practices so that we could obviate those problems.

Here we are. Six things we can do. Six things we can remember. I put this under technical systemization. Technical systemization involves things going all the way back to the IETF. The first RFCs were not about how technology was going to work but how we were going to collaborate to build the most important technical resource today that we have, which is the internet, interconnecting all of us and providing a platform for business and communities to develop across an electronic system. Technical systemization is how you're going to collaborate. How are you going to do all of these various things? I've taken what I believe to be the things that should be utmost on your list of things to do to avoid crisis. These are our suite of IT best practices.

It doesn't really matter which order they're in, but first up is Decision Support Metrics. People are out there buying millions’ and probably billions of dollars’ worth of network management tools, analyzers, all sorts of application agents and gizmos and gadgets aplenty, to basically try to figure out what's wrong or to be notified about when things are going wrong, and that sort of thing. I am finding that even though we've spent an inordinate amount of money and time and energy, that few people are really using the tools.

The proclaimed experts today are the people who install it and keep it updated. Well, that is a very important part. You have to install the equipment, keep it updated in the versions, and then you have to have users who can go in and connect and click and get the reports. I'm finding that there are very few people who are truly doing analysis on these very expensive tools. Few are really looking at their enterprise, their environment, their applications and the signatures of their applications, the dependencies and that sort of thing.

I call this Decision Support Metrics because those metrics developed by operating system details and monitoring tools and metrics, whether it be SNMP, NetFlow, response time management to performance management, agent tools that are inside the actual platforms, cloud based statistics, etc. If you're not really looking at what your dependencies are and tuning those systems to look at your specific applications and your dependencies, and I'm going to talk a little bit about problem management later, but integrating that with problem management, finding out what problems are causing you issues the most, it's a very important thing to take and use those systems and turn them into Decision Support Metrics.

Decision Support Metrics means that there's actionable information in those statistics. You can tune your pages and your systems to go out and say, “Okay, I've got this application that we really need to serve the application owners here. We need to really monitor this application. When people have problems with it or there's capacity issues, we have the statistics and a base line and we're watching each one of our major applications so that we know what the next step is going to be.” It's very important to do that.

Here are some of the things with Decision Support Metrics. First of all, you've got the desktop and then you've got the server, you've got all of the various components behind the server which is the applications, and the virtual platforms or the cloud platforms, the databases, the multi-tier capabilities. You have to watch all of that. You really need to architect around those sorts of things. Look at those application data flow and dependencies.

In order to be able to really monitor well, you have to know where your test points are. Back when I was at Lockheed, if there was a problem with a printed circuit card, our multilayer board, we had test points on the board. You could go to those test points and perform a particular measurement to see if the health of that system was up to par. You could measure it, adjust it and that sort of thing, but you had to go to those test points. Well, our large mission critical networks have test points in them if we take and document them and lay them out schematically so that you can see those test points. That’s how you know how to build your Decision Support Metric systems is because you have good documentation and you can see it.

It's difficult to do this today because there's so much compartmentalization. It's very difficult for the average technologist, who's in a silo, to actually go outside and look from desktop to server and all points in between to look at that application. There's been some slowness to train people who understand capabilities of the desktop, their servers, their network infrastructure, and their SAN infrastructure. When there are problems that are pacing or gating their system's performance, they can go and set up test points, or maybe already designed test points, to expertly monitor those various dependencies. That includes mission critical applications, your exchange servers, your mail systems, your internet. We could basically take a lot of our reactive people and repurpose them to looking at monitoring tools and monitoring systems that would eventually obviate problems, reduce the amount of reactive workload and improve the performance and reliability.

There's a problem with documentation. There's a problem with good Decision Support Metrics. It's a holistic problem that doesn't change overnight. It's a cultural issue that starts with best practices and an intention to improve as you go through. All of that's designed to help you recoup your monitoring tool investments. You've got a lot invested in those monitoring tools.

You always want to focus on that user performance and go in and do root cause diagnosis. That all ties together with another topic that I'm going talk about and that is Architecture Ownership. A lot of my customers are government contractors and large institutions that utilize outsourcing organizations or perhaps, you are an outsourcing organization, or perhaps you're a government entity and you have a lot of contractors and those contractors are changing. One of the things that I like to encourage you to do is to make sure that regardless of what's going on with your contracts and that sort of thing, that you maintain Architecture Ownership. That requires accurate technical documentation and to address that.

I'm going talk to you about that a little bit so that when you change contractors or a contract, you have a different outsourcing organization, you’re looking to outsource, or maybe you're looking to insource and change from outsourcing back to your own insourcing, you want to make sure that you don't have any limitations by indispensable contractors, indispensable employees because you own your architecture. You own your system. Now that doesn't mean you can't have a contract perform your Architecture Ownership functions, but it probably needs to be separate from your regular operations so that it's more of an auditing function.

In Architecture Ownership, what do we mean by that? There are two components of it. One is, to know yourself. That is to basically render your architecture into diagrams so that you can see your environment, all your technologists can see your environment and understand it, and your architects can work around pieces of paper.

Today, with our lights out data centers and that sort of thing, it's impossible to do this unless you have good documentation. Not only good documentation schematically and of the configuration, and that sort of thing, but also of your racks and that type of stuff because you don't have, and you don't want to have, people in your data centers. You really want all that work to be done either online or virtually or from multiple locations. In order to accomplish that, it's essential to have good system documentation.

What we believe that you need to do is pull all the router switch and server and configs out and basically put those on paper in a logical schematic type diagram. That documentation needs to span multiple security zones and be scalable. You really want a diagram that's not the server guy's diagram, network guy's diagram, storage area network guy's diagram, or desktop people's diagram. Whose is it? It’s not the application diagram. Whose is it? It is your organization's diagram cross silo. It's to put together an end to end diagram of all the pertinent detailed information, so you can put your finger on where an end user is connected, that's getting poor performance and you can move your finger through virtual circuits all the way through so that you can find the logical and the physical.

If you have a bad switch port somewhere, of course you've got redundant systems, right? Redundant systems can sometimes be a problem because you may have a bad fiber on one of those redundant connections but it keeps flipping back and forth. Therefore, you have an intermittency. It does work but when it works, it works in a degraded state. Then it flips back and forth between the two. It's important to be able to view your system all out so you can see everything.

Another part of Architecture Ownership is that I have noticed that there are literally sometimes hundreds of people who are in the IT organization who, if you really quiz them and ask them, don't have a clue of where they fit in the organization technically. It might be a server person, network person, desktop person, virtual person, the person who takes care of the SAN or the WAN or what have you. Those people need to understand, in the end. Once you have a really good document that depicts your environment, then you can take that document and you can train all of the people. “Here's our architecture. This is why we do it this way. Here's how it works. Here are the tools and systems that we use to manage it.” Then you bring all of these people out, who previously might have been hiding under their desks when there was a problem, because you've enabled them to now understand their environment. You have your network rendered so that you can see it and now you train people.

What we believe that you need to do is train people in your architecture, your systems, your monitoring tools, your capabilities, your organization, your trouble ticket system, so that all your technologists are banging on all eight cylinders. Instead of having five or six people in your entire organization that are really into it and understanding everything, you bring it up to several hundred through training. What we believe that you should do is have a certification program for your architecture.

First you document it. Then you train everyone on it and you test them and certify that they truly understand before you give them the keys to your active directory, before you give them the password for the routers and switches and platforms. They go through this orientation and certification on your particular architecture so that they are not trying to make it look like where they came from, but rather bearing the hand to help you continue to build your architecture and your system. Architecture Ownership is a very important component.

Problem in Change Management, of course, is essential to, one, diagnose root cause, but also to enable rapid continuous system performance optimization. Doing root cause analysis, of course, that's my area of expertise and that's where I get called in a great deal. We come in, we diagnose the problem and then we do root cause analysis. Usually it's had to escalate quite a bit before people finally say, “Hey, let's get somebody in here to help us solve this problem. We have smart people. We have capable people but they're busy doing other things. Let's bring somebody in to focus on this.” We come in and we help with that solution.

Well, one of the things that we do when we do health checks, is check out your trouble tickets and find out what's been pervasively causing problems. If we can basically prioritize those bad trouble tickets, we can then go mitigate the root cause of those problems and basically obviate future trouble tickets, which lowers your costs of your help desk and that sort of thing. Some organizations are so reactive that they just keep hiring more and more help desk people to be basically customer interface folks and that just keeps growing and growing so large and so wide that you can see the curvature of the earth in those cubicle areas where the help desk is. It makes a lot of sense to basically mitigate those particular problems through statistical analysis and then root cause analysis.

The first thing that you need to do is you need to do a problem statement. You need to know what your problems are. You need to reverse engineer your system documentation. Perform micro and macro analysis. Macro analysis is using your decision support tools to find out about where the problem is. Micro analysis is getting down to the packet level to diagnose the problems. That's another thing that you need in order to get the critical problem resolution and have a really good organization. You not only have to have the macro analysis, SNMP tools, NetFlow tools, performance management tools, but you also need the ability to go down to the micro level with packet-level analysis.

Another thing that is helpful, if you have a lot of applications and you're finding that you're having to go out and diagnose problems a lot with your applications and your applications are said to be slow, if you basically embed into your application development an early and frequent deep packet performance analysis and then subsequent design of the monitoring system to look at those application vital signs, that will yield incredible ROI for you.

It's our recommendation that you have regularly embedded deep packet inspection early in your development cycles and redevelopment, refreshing cycles. Rather than waiting and putting an application out there, spending $20 million on new hardware refresh, software refresh, and that sort of thing, and you go to roll it out, it's got so many SOAs and interfaces with other systems that when you turn it all on and you start using it, you find out that the end user won't use it because it's so abysmally slow because there was no performance analysis done in a micro fashion incrementally over the period of time. You need to look at doing application analysis early in your cycle.

Here are some of the things that you can do. Identify your dependencies, do micro and macro analysis, and do latency analysis on your web, your SAN, your cloud, your metal ware, your load balancers, firewalls. Find out where your problems are early on. That will help you with those Decision Support Metrics to be able to design metrics to watch those areas that you know in the future, as you scale, might become a problem.

Then Business Technology Integration. Like I said, a lot of technologists and technology managers are finding that they're in a silo, they've got operations and engineering for each different type of technology and it's hard to get an end to end view of the entire system. We've come up with some ways that we believe that you build virtual teams to basically have a representative of each silo for your Architecture Ownership, for your Problem Management, for each one of those. If you put them on a team together, you can end up having a collaborative design environment where all your silos are working together. You can have documentation that is end to end from the client all the way to the server, the application, and all the various systems in between. You can really get some traction in there. That is dependent upon a Master Plan.

A Master Plan is a prerequisite for collaborative design. You have to know what your objectives are. We like to focus you under Business Technology Integration and Master Plan Development so that you document the “is” and then help migrate toward that future.

This is our cross-silo collaboration model. It is essentially a way in which you visualize all of your various organizations and all of your various silos, cloud, your network, your desktop, all these different things, and these folks, security and platform. You basically build out an Architecture Ownership group, a Business Technology and Master Plan Development group, an Application Development Optimization. These are virtual teams. You don't have to change your logical organization or your budgeting silos. You just need to put this together so that you have the glue to help everybody understand one another’s environment in a team sort of way. Wrap all of that stuff up into your “is” Master Plan. Then you've got your “way ahead,” your “future” Master Plan so that you know where it is that you're headed.

These are the things that you can do: Architecture Ownership, Decision Support Metrics, Business Technology Integration, Master Plan Development, Cross-Silo Optimization, Problem and Change Management, and Application Development Optimization. Really good things that can be in your forefront. Just remember, take a look at that Jefferson Memorial and remember that each one of those pillars and each one of those steps are steps to your user's best interest.

If along the way, Apalytics can help you, Apalytics is dedicated to crisis avoidance through integration of best practices. Our TS is our Technical Services where we'll come in and help you with existing problems. Our AS is our Assessment Services where we can come in and help you identify the risks and then help you develop mitigation strategies. Then, finally, our cornerstone services which help integrate and obviate problems through the expert integration of best practices.

I appreciate you joining me today. We appreciate you spending your time with us. I hope it's been useful to you. If there's anything that I can do to help you or if you have any questions about this presentation, just go ahead and email me at [email protected]. I appreciate your help in building out our industry, helping us all grow and understand what we should be doing. Apalytics. Application Analytics Software and Services. We're at your service. We appreciate you and your joining us today. Take care.

APM Application Performance Monitoring Tools in the Theatre of War tells the story of how application performance management tools were deployed to analyze and report on key inteligence biometrics applications throughout US CENTCOM Area of Responsibility, and CONUS back to the FBI IAFIS and Biometric Fusion Centers.

Transcript for this Video

Hi and Good Afternoon. This is Bill Alderson, thank you for joining us. I’m going to do a presentation on Network Application Monitoring Tools in the Theater of War. This is a presentation that I did at the MilCIS conference last year in Australia. It was a lot of fun, going Down Under and talking to some of our military counterparts down there about network management- Had a good time! Anyway, appreciate you joining us today.

So a little bit about me, I consider myself an application and network performance analysis advocate. I know that end users are the most important folks in my career. I focus on working toward their benefit in everything that I do. I started out at Lockheed in Sunnydale, California, the heart of the Silicon Valley, and was there from 1978 to 1984. I was a communications analyst. I was a young guy that loved communications, and was playing with every kind of computer that was coming out at that time- IBM, Apple, others.

I was right in the middle of it. It was a lot of fun. A little later on, I got working with Network General and the Network General Sniffer, at the start-up. Then I took my experience with the Sniffer, and I founded Pine Mountain Group, and created Network General Sniffer Training, which Network General licensed from me, in perpetuity. Then Pine Mountain Group, we trained about 50,000 people in 22 countries and certified over 3,000 network forensic professionals.

I’ve done a lot of work with 75 of the Fortune 100, federal and state government agencies. I did a large event for Net World InterOp, called Network Forensics Day. I sold Pine Mountain Group to NetQOS in 2005, and continued on as was the Technology Consulting Officer. CA then acquired NetQoS, and I hung around as principle services architect on CA Application Performance Management services products. Then just this last year I founded Apalytics Corporation. I’ve got several customers at Apalytics. We did a sub-contract to U.S. CENTCOM, where we went and did analysis of pretty much the entire war area.

That’s what I’m talking about, and some of the things I learned through that experience, of looking through every network control center throughout CENTCOM’s AOR, in all those various countries. Then working with those systems and helping to upgrade and architect, document their systems so we could perform better network management, etc. I’m also consultant to the OSD-CIO, at the Pentagon in area communications. I also provide services to the Department of Justice. I’ve got a pretty wide variety of experience. The talk today, is about network management, but in order to understand network management, one must understand everything from end to end.

From the client, where you’ve got the user, the business transactions, the security access, the application interface, the operating system, the computing platform, the OSI model. The seven layer model that brings it down to where you can take all of your data across disparate networks and systems, and then have it arrive at the server and get error checked and that sort of thing.

We’re basically looking at the full spectrum of everything end to end, from client to server. Well, one of the things in the war area is that there’s a lot of different people, and a lot of different organizations, that are in charge of different packets. From where the war fighter puts the packet on the wire and until it gets to the server.

So consequently, in that traversing all of these different network management zones, and different countries, continents, going up to space and back, these packets are really mistreated. Everyone along the path has their own SLAs, their own network management systems, and they’re all different. By golly, if you go through and you talk to anybody, all the contractors and everybody involved, everybody did a wonderful job, and all met their SLAs. Everybody inside those silos is very happy- with one exception, the end user.

The end user is going to cross all these dissimilar systems, and when he’s got a problem, where do you go to capture the packet, if it doesn’t get there? Who do you blame? Who do you call up, who do you talk to? The end to end analysis that I’ve done from all those areas, back to CONUS for different applications and that sort of thing, basically said, “Hey man. If you’re an end user, you have no advocate. You have no centralized advocate.” It’s very, very difficult. You can call your local guy and he’ll do local things, but he always comes back and says, “Must be somewhere else.” Now your end user is in trouble. So those end users are usually the ones that contact me. Powerful, high visibility, high stakes end users, usually call me up and say, ‘Hey.’ That’s exactly what the military did. They called me up to have me help with a big application, and we went out and looked at all their network management and said, “Well, you don’t know where any of your packets are going. You need some NetFlow information.”

We implemented NetFlow across the entire AOR so we could see what applications were running and where. In particular- one biometric application, it was very interesting; it was awesome to analyze this application. I helped the programmers actually recode several parts of their application to improve performance and that sort of thing. It was a really great thing. So network management does not just involve the purchase of some network management tool then turning it on and installing it. It involves understanding the clients that are accessing the applications that are going through it. So you can fine tune it, nurture it. So you can find the signatures of the vital signs of those applications. When somebody tells me about network management and oh yeah, I’m a network management guy. Well, do you know applications? Do you know the issues on the client? Do you know the issues on the server?

Do you see the virtualization issues? Do you see all the man-in-the-middle, all the firewalls, all the WAN optimizers? Do you see all the load balancers and that sort of thing? Do you understand how those things come into practice? Do you understand the quality of service across that end to end network? “Well, I know everything there is to know about this network management package. I can install it, and I’ve installed it.” Yeah, but in order to install and use these systems, you must intrinsically understand what it is you’re trying to help the clients achieve; and how to measure the performance of those systems all along the way. You’ve got to document your systems.

I’m going to talk to you a bit about, after we get off this slide, a number of things. We’ve got a lot of slides to cover today. But we’re going to get there very quickly and give you a cursory overview. Here’s your basic client to server, so the client is separated from the server. We started doing that back in the late 80s. You took your client and your applications, you run it across various stacks, using HTTP or Java. You go across your transport network, you come up the other side, you go into your server. You pop back up your stack and into your respective upper layer system. Then you go into your applications and processes on your server.

You have to draw this out. You have to know where your connection points are. You have to know where you are in the universe. If you’re having a problem somewhere, you need to know exactly how you’re going to hook in, where you’re going to hook in, what kind of tool you’re going to use, so that you can deconstruct the problem.

Now, people going into the cloud and that’ll be another topic that we talk about. Cloud computing can be very problematic. Because, with the client and the server, you own all of the infrastructure; but when you put that server over in the cloud you have no access to packets, network management information, statistics, etc., other than what that vendor gives. It’s going to be a cloud all right. So basically, I’m trying to help you understand that when you go in to analyze your environment, you need to know from end-to-end.

Here’s a client talking to an app server or an HTTP server. The back end is talking SQL or some other type of middleware or back end process. Well you have to know where to go to connect your network management systems for packet capture, for metrics, for analysis. So that you can understand, and if anything goes wrong, which it will at some point or another, and you’ll need to get macro information about the use of the network, the use of the platform, the CPU, etc., of all the different systems. You need to see the entire system from end-to-end, and understand all the various components so that you can basically design the vital signs for the applications and systems. It’s not as simple as just being, okay, I’m monitoring layer three of the network. You have to be a little bit higher than that, you have to help your customers. That’s my definition of network management.

Now typically at one time or another, whether during implementation or later on, you’re going to have a problem. You’re going to be looking for that proverbial needle in a haystack. Here’s your needle in a haystack. It’s out there, it’s somewhere, let’s take a look where it might be. Hmm. It could be anywhere in Afghanistan, Iraq or some point in between, or over somewhere in the United States or down to Fort Watuka. Your packets are traversing all of these various environments. Where is it? What’s slowing it down, who’s holding it up? Who’s routing it. Who’s misrouting it? Who’s redirect it? Who’s changed it in its path because of a WAN optimizer or load balancer that you were unaware of? Those are your men-in-the-middle, and they’re sitting out there impacting your environment. So you need to know your environment.

Then every once and a while you’ve got to gather some packet traces. You’ve got to gather them from somewhere in between where the user is and their resultant service. So it could be anywhere. We usually go and gather packet traces. We gather packet traces based upon the macro information that we get from our network management system, to tell us and help us triangulate where the problem might be. That’s what we use network management for, and then we’ll go out and we will capture some traces and then get down to the bottom line. I’ll just tell you right here, and you can all argue with me, you can fight with me. If you were a network management person and if you are implementing large scale systems, and you have no ability to capture packets across your path- You are impotent. You can’t do it, you never will be able to do it. I’m telling you if you don’t have bottom line, root cause deep packet analysis skills and capabilities, and the ability to capture those packets. And you are impotent and you cannot do your job. Sooner or later you’re going to come up against a problem that absolutely requires deep packet captures. You design your network management systems to help you find the area, and then zero on it. Can you solve many problems with network management systems and capacity? Absolutely. Can you solve every one? No. It’s not a very pretty picture when you’re standing around with several million dollars’ worth of network management tools, giving you charts and graphs and all sorts of wonderful views of stuff, and you can’t solve the problem.

It’s because you don’t have deep packet capture capability. You don’t have the root cause analysis capabilities. That’s what I focused my whole career on, trained 50,000, and executed for 75 of the Fortune 100. To help them solve these types of problem. Then some people say, “I like packets so much I’m going to capture every single one and I’m going to store them forever.” Then I ask, “Okay. You’re going to store every packet everywhere? You’ve got unlimited amounts of money and resources and storage? Who’s going to analyze those packets? Who’s qualified within your organization? How are you going to filter in, and have the worst offending problems bubble up? What’s your strategy for using these systems?” I know a lot of folks have a lot of this sort of stuff up in their environment, and they have all these tools and capabilities, but nobody ever goes and looks at them. “Oh, but they’re there in case we need to do retrospectives.”

That’s good, and I’m not saying you shouldn’t have any of it, but I’m saying that you’d better make certain that if you’re going to buy this stuff, turn it on and start caching packets all over the world, that you at least ought to have somebody who’s around who can help you take a look at them and know when and why. So when you finally get down to the stack with the problem, now you go to work in deep packet analysis, to deconstruct in order to ultimately, boom, find the needle in the haystack. That’s what it’s all about, is getting to the bottom line. Definitive results. Performance Orientation. Getting the job done. No ifs, ands, wells or buts about the situation. In order to get all that done, we use ITIL , we use People and Processes, Paradigms, tools, systems, platforms. We use all of these various systems. It’s not just network management. It’s not just deep packet analysis; it’s not just application optimization. We have a whole portfolio of capabilities. As I’ll talk about in some future, I have a list of these things that I believe are the most important and we’ll talk about those at some point. Actually, on the 30th of the month we’re going to talk about that.

It requires a seamless integration of people, processes, knowledge and technology. It’s a well-rounded organization. I’m going to talk to you a little bit about what I call my network management servo-loop. Over on the left, you’ve got a website, let’s say. You’ve got an application, and you want it to perform. Let’s just say that you want it to perform in a way that allows that webpage to come up within 15 seconds. So what you do is over on the left hand side, you say I want the maximum possible performance to accomplish this business objective. I’m going to set my command, my desired response time, to be at 15 seconds. So that goes into the network management servo-loop. This is a feedback servo-loop mechanism. If you haven’t seen this before, you can look it up on the internet and see some others. But I have taken this and adapted this, because it’s a feedback loop, and it’s a servo-loop, it’s a closed loop system. Where you say, this is what I want and expect from the system. Then you have a resulting actual user experience that you compare to what it is that your objective is. You come up with a signal deviation.

At the top left there, you have what your desire is, 15 seconds. You then have your feedback signal, which is let’s say it takes 25 seconds. You have a signal deviation of ten seconds. You are not meeting your SLA or your desired performance by ten seconds. So what do you do? In most environments, executive management thinks all the technologists have it all figured out, and the technologists think that the executive management have it all figured out. I’ve got news for you. I’ve been working in this industry for 20 some odd years. Neither one of them really knows, nor have a systematic approach, until you get them into the servo-loop. Let’s get the executives in there. Let’s show them some metrics, show them some business metrics.

When you talk to an executive about their business metrics and how it coincides with the network and the application metrics, they start listening. You don’t lose them in presentations; you have their full, undivided attention, because you’re talking about deliverables for the business. I always like to get metrics, feedback sensors. Metric are a vital part of the business. Performance indicators. End user reports. Right? You get your executives involved, because why? Because executives control the policy, procedures, the resources, and the entire organization. They need to be involved in this. Okay?

You take a guy like Steve Jobs. He was involved in the process. He understood his company’s technology. Then you take another guy like John Scully, and you put him in there who’s selling sugar water for years. He’s a bottom line dollars guy, how much can we make? But in technology, you have to have someone involved in the technology. If your executives are not involved in the technology, it’s your job as the CIO, or the executive technologists to get them involved in a meaningful way. The way that you do that is you pop in some business metrics and compare them to some of your technical metrics, and you’ll bring them right in. Then they’ll get involved and help you solve the problems that you have by allocating resources, priorities, focus, policy and budget.

Then you’ve got the actions. The actions are carried out by the technologists and the technology folks. They go out and they apply. They’re the tinkerers; they’re the guy behind the curtain, changing this knob, changing that knob. Optimizing and installing and improving the environment in the system. After executive management puts the controls on there, the technologists apply those priorities. Then we should measure the actual user experience again. We measure that actual user experience, it comes back, Okay. Then little by little, between the executive function and the technology functions, we start to move the servo-loop toward the maximum possible performance. We keep tweaking it, but it does involve executives & management, and it does involve the technologists. If the executives are not involved, you won’t have the type of priority and prioritization and resources necessary.

Okay. Now let’s talk performance indicators. Well, these are the three main performance indicators that I advocate for. You’ve got network flows. Network flows are from the left, you’ve got clients and you’ve got an IP address, and you’ve got a socket. Those are the application sockets. If you know every packet that goes across your network, because every router in the network is reporting on these network flows. You don’t need another gismo to pop in there. Those routers perform the actual task of looking at all those PCP connections, from IP address on the left at the client, and IP address on the right at the server, and the application ports that they’re using between those two. Then they tell you and record the rate and flow volume information. I can tell you if an application is or isn’t working well, and how many people are using it. Why? Because I have rate and flow information, and it’s coming from my routers, and it’s going to a database, getting in the database, and I can go back and tell you what applications are on your network. That was very interesting in the war environment, because there were certain applications that were consuming large amounts of bandwidth and others that were consuming hardly any bandwidth. Some of them were being accused of consuming bandwidth that they weren’t. So when you really go out and get the facts of who’s talking to who through whom, that’s bottom line. That’s what net flow does. Who is talking to whom through whom?

Then you’ve got rate and volume information, you’ve got capacity planning, you’ve got all of those sorts of things available at your fingertips. Net flow and network flows is a very important thing. It tells you where those applications are running around the world, and who is running those applications by location. And, the volume and the rate can be compared, and tell you what type of performance they're getting. If you understand the theory, you understand the network management. So again, this is not about turning up a product. This is not about clicking and installing a product. This is not about buying it, installing it and having it there. It’s about interpreting it, it’s about looking at it, and it’s about architecting it. It’s about understanding the applications; it’s about understanding the vital signs. And it’s about trying to figure out what’s going wrong and where, and being able to triangulate that and being able to solve those typical problems.

Okay. So then you’ve got standard SNMP, that’s device status. Standard SNMP just tells you a router’s overloaded with CPU, or a circuit is overloaded or an interface is overloaded. It’s very good information. But what if you don’t even know the path that your packets are taking? You’ve got to know the path your packets are taking. If you know your path, then you can exploit device status, to find out what devices on your particular path are impacted by memory, by CPU, by other types of capacity limiting characteristics. The first thing is, you’ve got to know your path. Well, there are not very many tools out there or capabilities at layers two and three. You can give a layer three path, but it’s hard to get a layer two path as well.

You’ve basically got the network flows. You’ve got device status, and then third, response time. Response time monitors, there’s several different flavors. AuthNet has some really nice stuff. NetQoS, which was acquired by CA, has some nice stuff. There are a couple of other players in the field that I don’t have a lot of confidence in quite yet, but it’s growing. The performance marketplace is no longer about red, green, up, down. It’s about, ‘it’s slow’, ‘something’s wrong’, ‘we don’t know where, it’s working, but not very well’ and ‘it’s impacting our end users’. So you put these response time monitors, which basically capture the packets or listen to the packets as they go into the server. They timestamp it at the server, and they say this packet response time took 100 milliseconds.

You can put timers on these things and thresholds, so that if your response time is slow, you can automatically trigger a trace file. You don’t have to wait until 2:00 in the morning. You can actually have your response time monitor over monitoring a bank of servers, and when the response time gets slow it automatically triggers a packet capture, so your technologists can go back in the next day and do retrospective analysis on the packets that were slow. So do we care if everything is fast? Do we want to capture all those packets? No. What do we want? We want to trigger on the events that are poor, and that’s what these systems do. They record the response time, it’s an awesome system. It also gives you the network round trip, because the [inaudible 26:54] come back and it records that so you can see. One of the problems that we had was satellite delay and multiple router hops adding satellite delay. It’s about 700 milliseconds round trip across a satellite. Well, when I see a response time that says it’s 2.1 seconds, I know that that 2.1 seconds is because they’ve gone across three satellite hops at three times 700. You can figure out and start to surmise what’s going on, and then you can look at optimizing your routing.

These are your tools and your key issues. You’ve got your performance management system. You’re getting your performance metric, the feedback, and your business metrics. You’re getting your trouble tickets; you’re getting your information back. Some of the things that you need to look at are route analytics, to identify instability in routes, server response time, net flow base, and rate and volume information. It gives you hints on conditions limiting user experience. Also look at SMNP device status. Well, you’ve got to know the path in order to know what devices are in that path, and because you may have hundreds or thousands of different devices. Knowing what your particular path is and exploiting that path, to see if anything in that path is failing. Those are value added steps. Then of course packet capture at key locations, so you can get down to root cause analysis.

Here’s a network performance management architecture diagram. There’s a portfolio of tools. You’ve got Remedy Trouble Tickets; you’ve got Opsware, ArcSight. You’ve got a large number of different systems-Packet Design's route analytics, CA Application Performance Management NetQoS’ response time monitoring, and NetFlow tools, , OPNET's Network Diagramming capabilities. There’s a portfolio. It’s a Swiss army knife, all working together. You can’t use just one company or one suite of products. You can reduce that set as much as you possibly can. But you always want to make certain that you’ve got every best of breed and aspect of your system. This is where we went out and we installed systems at 15 different sites across Afghanistan, Iraq and areas in the United States so that we could have NetFlow and response time monitoring. This is how long it took us. It took us one year to install. This was pretty much unheard of, and the ability to get this system up and running and actually do analysis on major applications in fewer than 12 months, inside a war environment, was unheard of. We had an incredible team down at CENTCOM out there. The war fighters, man, they were working their buns off to try to help us get all of this stuff, so we could figure out what was wrong with this biometric application. This biometric application by the way takes fingerprints and iris scans. And it looks at latent prints of the guys who were building bombs and that sort of thing, those IEDs. That application was helping them put those guys on an alert list.

A war fighter would have a bad guy walk through, or people would be walking through their security station going into Fallujah or into Baghdad or whatever. And they would immediately pick up on the fact that thousands of people were coming through, and we just found the guy who had a latent print on an IED. This guy is now caught. That’s how the surge actually worked in my opinion, was because we had electronic means to zero in on who’s a bad guy and who’s in proximity and who’s not. This application and the analysis of and the fixing of this application really helped the effort over there. We found route changes, due to packet loss, slow server response time. We found problems with TCP offload engine. TCP offload engine is where the TCP stack is moved from the server, inside the server, using the server’s CPU and memory, off onto the network interface card. There was some problems, and I’ll show you a couple of slides here on some of the examples of this sort of stuff. Of course, this took many weeks and months of analysis after we had all this stuff put together. But we found all these problems and started mitigating them, and things improved. And that application I was telling you about was even more effective in performing.

Across the network there are qualities of service incongruities. Network and application issues requiring packet level capture, we analyzed a lot of stuff. Here’s a few screenshots of some of the stuff. Here’s some route change analytics going on. Here’s some satellite retransmission, took three and a half seconds to retransmit across many of those satellite circuits. This is a detailed analysis of that sort of stuff. Here’s a TCP offload engine recovery issue. These are all commercial, off the shelf stuff. In the U.S. or other parts of the world, these particular systems are not always found to be wanting in a low latency environment. But you put them in a satellite environment with high latency, and manifestations of severe problems start to show up. That wouldn’t show up otherwise. We found TCP offload engine problems. We also found WAN optimizers and load balancers and other things, doing weird and funny things. I called them our own man-in-the middle. I developed a method by which we could identify that man-in-the-middle, and this is some of the analysis associated with that.

Then processing analysis.- How long it takes? Over on the left, you see processing, processing, processing. That was before we optimized the application. Over on the right, the same transaction only took under two seconds instead of 18 seconds. You add those types of optimizations up and you’ve got a lot of optimization. Then there was data duplication. What does data duplication mean? Data duplication is when you have a problem with that TCP offload engine where you’re sending the same data multiple times. This is an example of where the data is being sent across the network. So not only did you have clogged pipes, not only did you have networks that were saturated, but now you’re sending multiple copies of the same data. Here is an example of that packet loss induced TCP offload problem, and the wasted bandwidth associated. You’ll see a red line and a green line. A red line was the wasted bandwidth; the green line is what was normally required. But in this particular environment, because of that TCP offload engine and the packet loss that was associated with it that triggered it or was the catalyst.

See, if you had no packet loss, a lot of these problems don’t manifest. When you do have packet loss you end up with three and a half second response times. You end up with data duplication. There are a lot of things that are exacerbated, because in a war environment you are not dealing with perfect commercial communication systems. So bottom line. Multi-tier identification of your environment; you have to know and document your network, and then you set your test points up. Your front end tier, your middleware tier, your SQL tiers; you have to know how to find and design your network management and your response time monitoring and NetFlow systems. In order to do that, you have to document very effectively. Then you can instrument your front end, your middleware tier, your back end tier, and your mainframe tiers. You can then take and instrument your systems.

This is basically showing test point two, where we pulled over to a super-agent to do response time monitoring, with the ability to capture those packets like I talked to you about, when response times get slow, it would automatically capture packets at that location. Pull it all together, what do you have? A user clicks on their screen on the top left. You see where it says ‘user click?’ Then it comes down. Over on the right, the different colors, red, green, blue, those represent processing times, network serialization, transport and switching queue times. Then, of course security authentication- How long does it take to authenticate the user to be able to access that data? If you take a gander up at the top left, he clicked, and on the bottom right the user finally gets the information. Did that take 15 seconds or 25 seconds or 71 seconds, what have you? Then basically take it with the next step, and you bring it down to the various tiers. The process, where is it? We fixed the process problem in the application, reducing the response time. This is an example of what some of those response time monitors look like over there. You’ve got your web tier monitor at the same time as your app tier, your SQL tier, your mainframe tier and we found the process at the app tier that was causing a lot of issues, and we helped resolve those.

I know I’m going just a couple minutes over today, but I just wanted to finish this out. The bottom line is, in order to find that needle in a haystack, in order to make certain that your users are productive; you have to design your systems not to find problems necessarily, but to nurture and build a system whereby you are obviating the problem. In other words, you are fixing the problem because you have such good intelligence on the information. In ‘The Art of War,’ the first thing that’s said, is know yourself. Know Yourself. Document. Know Yourself, Monitor Yourself. Know where you are, so that when you do have a problem, you can go over and you can find that needle in a haystack, using scientific, automated capabilities. The future of network-centric warfare is dependent upon having these types of capabilities.

Just buying a bunch of stuff and installing it is not what is needed. You have to have architects who are looking at this, who understand the big picture, and who can solve problems in all manners, from client to server and all points between. Anyway, if there’s anything we can do to help you; we know how to do it. And we’re at your service. It was good being with you today. This is Bill Alderson, signing off. Appreciate it.

Complexity of WAN Optimization, Application Acceleration devices makes analysis of problems and root cause analysis more complex. Yes, they are beneficial, but if you have an issue only the manufacturer has the ability to fix the problem. Sadly not much motivation exists for them to help you determine that the esoteric problems you have manifesting are related to the devices.

Transcript for this Video

The four major areas causing IT Complexity Acceleration.

1.) Technology 2.) IT Silo / Dept Growth 3.) Personnel Specialization 4.) Out of Sight - In the Cloud!

Hi, I'm Bill Alderson and welcome to a little session on IT Complexity Acceleration.

Thanks for stopping by today. I'd like to talk to you about the issues affecting IT Complexity and its Acceleration. First of all what do we have? We have technology. Every area of technology is getting more and more complex. Now sometimes it makes it better; less problems and that sort of thing. But there's a lot of new technology out there that's actually making things much more complex- Multi-tier applications SOA interfaces. That's interfaces to other people's data that you're bringing into your application.

You've got WAN optimizers you've placed in between your client and your serve; additional devices that are taking the data, compressing it, encrypting it, optimizing it for certain applications. For certain applications it's on, for some applications it's off. Sometimes it works well, sometimes it doesn't work well. But let me tell you when there's a problem there's a problem and it requires detailed analysis.

Why do you think that Riverbed, maker of WAN Optimization, Application Acceleration appliances bought all of the developers of Wireshark? - Because they had to! The complexity of the WAN Optimization, Application Acceleration technology is vastly accelerated beyond native client server architecture. Used to be you could see every router, every switch, and every device between the two. Now you put in these very quiet WAN optimizers. You also have other things like load balancers. Those load balancers go in between the servers and the clients. So when you do a TCP connection you connect up to the load balancer. The load balancer then sends another request on a separate TCP port, another OSI model and goes to the server.

So you're really not interacting from client to server. You are interacting from client to WAN optimizer, WAN optimizer to load balancer, from the load balancer to the server- vastly more complex. In addition, we have on the network interface cards and all of our devices; we now have TCP offload engines in order to be able to do more with the processor on the board and take away from the processor in your CPU of your hosts and of your clients. So the TCP offload engines are taking that and putting them in the offload. That's yet another set of complexities, another set of dynamics to where you're not truly communicating from the client to the server.

We're in a day and age where we're buying lots of monitoring tools and systems and equipment. We have a plethora of men in the middle. I call them "Our man in the middle", those load balancers, and those TCP offload engines. Then you have firewalls. Sometimes the firewalls are actually taking sessions from the client to the server and in between changing TCP options.

Well all of these things are good, but, if you don't have analysts who can really look at this data when something goes wrong, you can't really troubleshoot until you get the next round of what?- Forklift upgrading. That's what the vendors are counting on, typically. We're doing very little to optimize what we have today. Everything seems to be having to be attached to a new technology before we'll do anything. So there are many free options to optimize, to tweak, and to improve the relationship between the client and the server. Those load balancers, those WAN optimizers and all of these different things we can optimize if we take the time to do it. And we can get huge ROI, huge bang for the buck by using brain cells to analyze things instead of just budget to buy new stuff.

Your IT silos, your departments are growing, your budgets are greater, there's more complexity, there's more dependency. Because of those greater budgets we have more differentiated silos so we're diversifying our budgets even more. In between those silos are walls that go up. When there's a problem from client to server and you've got the desktop, the firewalls, the network, the load balancers, the WAN optimizers, the servers, you've got the applications and now when there's a problem- who do you go to?

There are very few generalists who know and understand your entire architecture. In fact that's the next topic here. We've got personnel specialization. All of our folks are becoming like doctors. They have more and more vertical specialization so when you have a problem that's from client to server and somewhere in between you have to assemble a huge team of people to basically address all the various issues associated with that technology. There are very few people who really understand from client to server and all points in between. You have very few generalists today and a lot of knowledge is hidden from them because the silos are defensively hiding information from one another so that no one can accuse them of a problem. So it's a growing divided support group problem because of personnel specialization.

And last, but definitely not least, an emerging growth trend is the move to the Cloud. What do we have? We've got remote data centers; we've got data centers that are being consolidated. We have fewer of them and they're further away from us. They're lights out. There's less knowledge, less intimacy and less interaction with the technology. It's a separated support group from who usually takes care of those things. It's a lights out operation. Everything has to be done virtually. And if it has to be done virtually, that means documentation must be at the very highest ability and highly collaborative; which is more than likely a problem, again.

So here are some things that will help you with this IT complexity acceleration. Number one: Take a look at Architecture Ownership. I have some pieces I talk about in my best practices about Architecture Ownership on our Apalytics’ website and Apalytics’ YouTube Channel. Number one in Architecture Ownership is ‘know yourself’. That means that you document your systems, you document your network, you document where those load balancers and WAN optimizers are. You document both sides of them so people can see how their packets are being treated. You document your net QOS, your quality of service metrics. Everything is documented so the technologist who's trying to look at an end-to-end communication session knows exactly what's affecting their packets and their systems.

After you have documented your system and it's very good and it's highly collaborative you need to train everybody and certify people on your architecture. Don't let anybody into your sight or don't let anybody have the keys to your servers and your routers and your switches and that sort of thing until they truly understand your architecture. That is only done by showing them the diagrams, showing them your architecture, showing them the packet flow, showing them the application flow so they are fully understanding of the client to server before they get in there and start tweaking things and changing things.

That change that you're having problems with, I know, you don't like change anymore because it imputes problems into your environment. I totally agree. But if you have people who understand all the pieces of the puzzle across all the business silos and understand end-to-end transactions you're going to be much better off in that capability. Architecture Ownership- take a look at my Best Practices on it.

Second: Decision Support Metrics. Decision Support Metrics allow you to find out if I upgrade that server, if I upgrade that database, if I upgrade that network connection, if I update that SAN, what is it going to do exactly for the response time of my mission critical application? Is it going to go from 25 seconds to 18 seconds? Is that what it's going to do? If you don’t know, then why are you spending millions of dollars on it if you don't have quantification? Maybe you don't have any Application Performance Management, APM, in your environment? Maybe you've bought all to those tools but they're actually not working? Maybe your help desk can't even use them? So you can buy tools and you can buy systems but if your technologists are not using them…. So they may be bought by one of your IT silos but not shared in the other silos so that they can gain the benefit of the understanding. Your tools should have an end-to-end cross technology, cross silo view to them.

The other thing is your silos. There are walls are growing around them. Business Technology Integration: the building of virtual groups so that you basically take a look at network documentation and metrics across the entire enterprise- not just within a silo. So right now if you go out to your silos they all have documentation but nobody has typically an end-to-end view of how the end user is affected. All these silos are being plotted and they're doing wonderful work and that sort of thing- but if you talk to the end user community and the business unit folks who are impacted by slowness and intermittent problem and that sort of thing you get a different picture, right? Okay, I know it because I hear it, I see it, I feel it, I talk to people, and I get called in to solve these problems.

You need virtual teams to deal with documentation, metrics and optimization of your environment. Now you need a Master Plan so I've got some stuff on Master Plan Development on the website- go take a look. And lastly, Application Optimization during development and operations: You need to take a look at your systems from end to end. Find out what your transactions are doing. Tweak your metrics so that they are pinpointed and getting signatures of your application performance.

Well that's enough for today about IT Complexity Acceleration. Join our communities. Let us know what you think about these videos and the information. Really these pieces of data are valuable if you take a look at them and you think about them and you integrate them into your mind and your thinking- you'll be a very successful IT Executive and Chief Information Officer.

I appreciate you stopping by. I'm Bill Alderson and my very best to you today.

IT Network Redundancy - Just Another Single Point of Failure (SPOF)? Trace by reverse engineering the Network logical packet flow to find out!

What is redundancy? Network redundancy attempts to reduce single point of failure (SPOF) by having two or more active-active, active-standby or hot standby components whenever possible. Most try for Active-active where both components are continuously active providing load balancing features and fail over to one another in the event of failure. The trouble is, sometimes two items are installed and operating, but when one fails, part or all of the service goes down. The reason is that despite having two components, logical dependencies having complex configuration require rigorous testing and high theory engineering. The result is that some just have double the dependencies at twice the cost producing a lower MTBF because of the multiple components - not to mention the support complexity. The key is rigid documentation and test labe case studies - resources most environments don't have access to. The video describes a visualization of a reverse engineered system tracing application packet flow showing that instead of one set of single points of failure... any of the redunanct componenets that might fail would bring all end user functions down.

Effective network documentation of network application flows and application characteristics uncover the truth about redundant components on your enterprise data center system architecture diagram. Wisely consider logical transaction and packet flow when preparing an IT disaster recovery plan - particularly when updating the plan for cloud spof disaster recovery and disaster recovery testing.

Network component redundancy requires the purchase of two of everything and makes the design and maintenance very complex. Are you getting what you think you paid for in both product and complexity costs? Sadly most redundant systems are either not operational because the redundant configuration engineering exceeds the staff's ability or it has flaws creating a worse set of dependencies than a single point of failure (SPOF) design would. By documenting your layer 2/3 environment well and superimposing application transaction steps thereon you can quickly see any flaws. The example is classic! Enjoy.

Complete Transcript:

Speaker: Here's another one. You bought two of everything because management said, “what if that thing goes down?” Some non-technical auditor came in and said, “what if that goes down, that's really important, you ought to have two of those.” So we bought into this and we started buying two of everything. Well, when I go into an enterprise and I start documenting their system to troubleshoot a problem, and I do my layer two, layer three diagrams depicting layer two and layer three analysis of where a packet goes, step by step through an infrastructure by VLAN and by component. This starts to expose our dependencies logically. Now you can see that you have two of everything from a physical standpoint, but logically, the way that we configure it and how packets actually flow through it, is an important thing to also diagram. So I've diagramed for you, a very complex environment, where we have an application server right here and we have a database server right here. The application server is what serves up to the end users, the user interface, and the database server is where the application server goes to get all the data from for those particular transactions.

So let's start out here with the yellow line, the yellow brick road let's call it. We go down the yellow brick road here. We come into a firewall, into our application environment, and we come in on port three. We go on port two. We come around here, we go up to VLAN II on switch one. VLAN II on switch one needs to communicate with the device on VLAN II of switch two. SSo it goes across the trunk, comes over here, gets into VLAN II on switch two, goes into the content server on switch two and into the contents switch which load balances between all of our application servers. It comes through here, goes out VLAN III on the other side of the contents server, comes up and over and back through into VLAN III on switch one. Which then connects up to our applications server called Prod DB or Prod App. So, here we are, we've just gone from the user which is out here, through our firewall, out through our firewall, through both of our switches, through our content switch, back and forth across the trunk a couple of times, headed out and now we've finally got to our server. So are we dependent upon, for the initial transaction, what? This firewall, this switch, these trunk circuits, and this switch. Am I not correct?

Well, to complete the transaction, the Prod App server has to do a sequel query on the database over here down at Prod DB. So consequently it then sends out that goes into VLAN 4 on switch two, comes back out, goes into firewall interface four, comes over to interface five, goes back out, goes into the switch and back out of the switch on VLAN 5, and then heads to Prod DB. Now at least from my vantage point here, I've traced this logic to show that for this transaction to be successful, we have to have not only switch one, switch two, firewall one and firewall two. If any of those components were to go down, which is the reason why we have two of them, our user would not be successful. So until we basically take a look at and diagram layer one, layer two transactions into our applications and through those complex switches, we can't really see what's going on. So this sort of a process of knowing where your layer two and layer three information is, is very important.

MTU defines Maximum Transmission Unit. Firewalls blocking ICMP disables IP's dynamic Path MTU Discovery causing fragmentation at VPN, L2TP tunnels impacting performance. Gateways are not able to respect native DF bit flags (Don't Fragment) because they are isolated on another OSI Model stack interface. The performance hit comes from what I call the "two for one blue light packet special" caused by the overhead of the tunnel header forcing two packets across the tunnel for each originating packet. That is why most VPN Client software adjust their MTU to 1300 so that does not occur - but what if you go through multiple tunnels? Then it becomes a manual hide and seek process to arrive at an MTU that works. There is of course much more to this than meets the eye so be careful in setting router and gateway MTU without fully thinking through the theory, analyzing the results with an analyzer and setting up network monitoring signature triggers to watch for problems in the future. The best way to approach the topic is to make it as automatic as possible through creatively setting your router MTU setting on the interface that goes to your gateway or Internet.

Transcript for this Video

MTU Maximum Transmission Unit is the size of the datalink layer payload of a packet. If it is not allowed to automatically perform path MTU discovery using ICMP Internet Control Message Protocol packets due to security filters on firewalls, then a logon and connect can be successful, but a data transfer will fail when a larger packet is sent. Or performance will suffer across high error prone links because a VPN tunnel will create two packets for every one packet to overcome the MTU mismatch exposing double the packets to packet loss. If one packet is lost both packets will have to be restransmitted by a higher layer protocol like TCP.

Complete Transcript: Here's another example of a problem that we've found that is kind of layer 3 related, VPN and L2TP tunnels. A VPN and a L2TP tunnel are there, and they're basically layer 7 services. Where it takes in a layer 3 packet, repackages it, and then puts it in another layer 3 environment. The don't fragment bit, which we use in IP at layer 3, to tell routers not to break up packets, that is not respected when it gets to a layer 7 device acting as a VPN concentrator or an L2TP tunnel. Those are actually layer 7 type devices which do tunneling for us or some other functioning. If you want to call it something other than layer 7, it's an application, so that's why I call it layer 7, you could call it layer 4. In any event, it's a tunnel going through your network.

When your packet leaves your station, it's 1,500 bytes at maximum, and that's called a maximum transmission unit, or MTU. That is the size of the payload at the datalink layer that the packet exiting your machine goes out. Now that 1,500 byte packet, as you can see here, travels out of the client, goes into your network, and it hits a tunnel device. Like let's say an L2TP tunnel. That L2PT tunnel then takes that 1,500 byte packet, and it has to put that 1,500 byte packet, but it still has than 1,500 byte MTU. It can't make a packet larger than 1,500 bytes. What it does is it takes your 1,500 byte packet, and it breaks it up into two packets, a large packet and then a smaller packet, and then puts the tunnel overhead on there, so that it can go across the network. At the other end of the tunnel, the tunnel then collapses that packet and lets go of your original packet pretty much the way you let it go outside of your particular machine. Again, we start out with a 1,500, we go in, we have a 1,400 and a 100, and then it comes back out the other side of the tunnel, and it comes back to 1,500.

What happens is, you're getting the 2-for-1 Blue Light Special here. You're getting two for one. You're getting two packets on the high packet loss, error prone internet, and consequently the effect of that is that you're going to lose more packets. And if one of those fragments, or those little segments are lost, you're going to have to re-transmit both of them, and those intervening device, being layer 3, are not performing end-to-end retransmission. So it will rely on your upper layers on the two ends, typically at layer 4 or layer 7, to retransmit those packets for you, and that can be very costly to your performance.

One of the things you can do, and by the way this happens from an internal network where you're sending a 1,500 byte MTU packet over to a VPN concentrator. That VPN concentrator is then having to put VPN overhead on top of that 1,500 byte, and it still has to live with the MTU on the next network of 1,500, so it has to break your packets up into two segments. When it does that, obviously, you still have this 2-for-1 overhead special. In some cases you can end up with packets that get lost, and then consequently your performance plummets when you need it most. When does your device send a 1,500 byte packet? It send a 1,500 byte packet when it's got a lot of data to send. When you're doing an FPT, when you're doing something that is a large data transfer. When you really need your network to work, that's when you start getting the 2-for-1 special. When you're doing that FTP download, or you're looking at a big website, or you're looking at a webcast. That's when these problems are going to occur to you.

In preparation for that, what we do is lower our MTU. There are several methods of lowering our MTU, so that when our packet, which is at maximum 1,500, reaches an L2TP tunnel or it reaches a VPN concentrator, it's already smaller. So that you get a 1-for-1 packet ratio. Most of your VPN clients today actually cut your MTU on all of your communications down to 1,300 bytes, so that when it gets to the VPN concentrator it will automatically not have to get the 2-for-1 Blue Light Special and expose your packets, two packets for every one packet for potential packet loss, on the internet.

There are several ways of accomplishing that. While all this stuff works automatically, and the MTU mismatches will get fixed, provided you are allowing the ICMP, internet control message protocol packets, to get from the device and go back to the client or back to the server who sent the packet. Unfortunately today we have firewalls, and our firewalls are blocking the necessary ICMP path discovery packets that normally would manage and make sure you don't get the 2-for-1 Blue Light Special on your own. But because of security constraints, we've turned off ICMP, and so consequently we have to do what is called end-to-end MTU management of our systems. It would behoove you to get an analyzer out and put it at these various locations and find out if you are, indeed, using ICMP destination unreachable path MTU discovery mechanisms effectively, or if, in fact, your firewalls are dropping those, and understand how this process is working in your particular environment.

So what I recommend is out here on these routers on the edge, if you do have and want this to be automatic, you can take your clients which are out here on your internal network, take your first router and set the MTU where you do have ICMP destination unreachable path MTU discovery packets where those are allowed inside your normal network, where they are not blocked by a firewall. And what we can do is lower the MTU on the inside link next to the firewall on the router, and then all this will be automatic. By lowering your path MTU on this particular router interface down to 1,300, essentially every packet that comes through there will automatically get its MTU lowered by path MTU discovery. Then your packets will flow through onto your VPN or onto your L2TP tunnels already smaller to go through that trip and get the expanded headers, OK?

This is one of the things you might want to look for and might want to understand in your environment so you know how things are working and if you were ever audited by a technical group, you would have the answer appropriate to the question for security purposes or what have you. You need to know about path MTU discovery and at every single point along the path what is going on with your MTU size. Thank you very much for listening, and we'll have more.

Oracle performance tuning -SQL and for that matter MS SQL tuning or any SQL server performance tuning can be analyzed by capturing TCP database transactions at the packet level. The Oracle Performance Tuning video is an example of the value of network analysis to optimize existing systems rather than await the forklift upgrade and its expense to improve performance. Andy hey - that forklift upgrade might not contain the silver bullet you are looking for to improve performance. There are many factors that can impact database and application performance - it might be a network or authentication issue slowing down your database. ...better to start database tuning with "brain cells" than capital budget. Transcript for this Video

Today, we're going to talk about Oracle Performance Tuning opportunities for application optimization. So you basically take and measure your performance. So let's just say you click and it comes back and it takes ten seconds. You then take a look at that transaction with an analyzer, etc., and you then calculate what it's doing and figure out, theoretically, how long that should take or experientially how long that should take.

Let's just say you come up with ten seconds as a measured performance, and you found that you could theoretically get this done in under a second. So there's nine seconds of what we call optimization opportunity.

Well, that's the first thing that you need to do is to determine if your optimization is viable, if you're getting a material return on your investments. So you take a look at measured performance, calculate theoretical performance, and arrive at your optimization opportunity.

After that, you want to take a look at some real information. Here is an actual Wireshark trace file, a SYN, SYNACK, ACK. So how long does it take to connect up across the network? Well, it takes four milliseconds to connect up to the application and back at the TCP layer. Then, it goes and it says, "Okay. Connect user Oracle," and he comes over here and he gets a, "Hey, resend this." That takes 73 milliseconds. Then he resends it and it comes back in a half a millisecond. So you can see right here that that's quite a bit of time. 73 milliseconds is a very long time. If you only do it once per transaction, you're only burning under 100 milliseconds.

However, this particular transaction performs about 86 queries, and each one of those queries starts out with a 73 millisecond connect request to the Oracle database. Well, that's going to take 86 times 73 milliseconds. So it's going to be pretty substantial in how much that costs you in the form of the transaction. That's an extra six seconds added to your transaction already.

So, if you could basically reduce that down, that would be optimal. However, let's just think about this. What if you only logged in one time and then performed 86 queries across a persistent connection? That would be a better way of architecting this application, and indeed, that is the suggested optimization.

So, in the next session, we're going to talk about to your CIO what this means in dollars and cents and how to justify your return on investment for something that you're looking at trying to optimize.

Now, Oracle performance tuning SQL optimization is free, right? It's just something that you have to do. Sadly, today most of your optimizations are in the form of large scale, forklift upgrades, and sadly, a lot of times we don't even look at opportunities to optimize the existing systems. Well, those opportunities do, in fact, exist, and in the time of tough economy and tight budgets, we're starting to go back and take a look at how can we optimize what we already own and improve our lifecycle cost and improve our applications. So that's what we're talking about today.

Watch Bill Alderson's video on Oracle performance optimization ROI justification based on detailed packet analysis findings. This video is for CIO's to help justify portfolio spending on database improvements... a technical companion Video is posted showing the case study packet trace explaining the technical analysis to find optimization opportunities

Transcript for this Video

This session is on Oracle database management to optimize performance and yield a significant ROI. First of all, a quick review from your technologist session. You measure the performance, you measure or calculate the theoretical performance, and arrive at an optimization opportunity. That's the delta. What that looks like is a trace file where we look at SYN, SYNACK, Oracle Connect. The Oracle connects. It takes 73 milliseconds. That 73 milliseconds happens for each and every SQL query, and what we want to do is knock out and use, not an implicit logon to the database for each and every query, but one logon and then several secondary queries. That will allow us to knock out that 73 milliseconds entirely practicing Oracle performance optimization.

So here's how you might justify this. First of all, you put together a spreadsheet of your measured performance and your theoretical performance, and you come up with your optimization opportunity. In this case, it's 72 milliseconds. So what you do is you've got your number of users at your call center, your underwriters, your branches, your managers, and your report time generation. Now this can be different for different companies, of course. This is basically looking at a financial institution and a large number of transactions that people are doing at different labor rates and that sort of thing.

So we basically take the micro-transactions and we calculate that out to find out how long it's taking. So if it's taking an extra six seconds per transaction, and that's because of this 72 millisecond logon, and we can knock that down to 72 milliseconds, then you can calculate out for this many users. I've just done some very simple equivalent person calculations. For instance, if you've got 3,000 people in the call center, you can probably drop 49 of those people just by this single optimization.

Now if you take into consideration all of the cost elements, you'd save $3 million in the first year of this optimization, just in the call center. So that's probably a good enough justification right there. But you add them all up, and you've got $4 million. Now, if the optimization costs $325,000 to rearchitect and logon once to your database and then have multiple subsequent queries, instead of 86 queries and 86 logons, you would end up achieving a cost savings in the first year of $3.6 million. Now, if you have five years of useful life, that would be a $20 million savings. If you calculate out what your recoup rate is at $333,000 per month, it would take you one month to pay off the investment of $325,000 in rearchitecting this application in a very simple manner and save yourself $20 million over the next 5 years. Not a bad day's work. Consider becoming a performance oracle!

Slow Domain Logon Analysis - Root Cause Identified - Solved!

Root Cause Identified - Application Performance Optimized are the words CIO's and End Users enjoy hearing. Network Slow, Application Slow - these trouble tickets take more time and are rarely definitively diagnosted to the true root cause.

[bitsontherun GF3m0Ujn]

Transcript for this Video

So the first thing I'm going to talk about is an application log on, that was taking a long time. As a matter of fact it was into the minutes, and sometimes the user would get disconnected and not even be able to log onto the application. Well the first thing we did was took a capture. We took a trace, everybody says, let's take a trace. Of course you take a trace, and you look at this thing, and it's like where do I start. It's a bunch of protocol goulash, and it's the same thing for me until I zero in on the TCP session or on the particular application or on the particular client or server, it's protocol goulash. So we have to zero in on these things.Now we found that there was a problem with slow domain logon, and so we kind of suspected that it might have something to do with domain controllers. So I Ioaded a que check agent on the server and on the workstation. Now Qcheck is a freebie from Ixia Communications and you can go out on the web and find cue check out there. You can download it, put it on your server, put it on your workstation, and then you can do a through put or a latency test to find out how it's performing at layer four. Well we did that and we found that it was giving us very poor performance with other tasks.

Now domain controller usually isn't doing file copies and those sort of things. It's basically doing security authentication. When we went to do through put test we found that it was very poor, for copying files or doing the que check. Now what we found when we went in and looked at that poor performance was, and if you take a here, a check sum error. You'll see here, TCP check sum error incorrect. I blew this up so you could see it. TCP check sum error, or check sum incorrect. Now what that means is, is that the TCP check sum calculated by the server was wrong when it went out on the wire.

Now it can have a good data link control. It can have a good IP header check sum, and a good data link layer check sum, but have a bad TCP check sum, and that's exactly what we found in this case. Now we wanted to find more occurrences of this and see how often this was occurring, but we still wanted to see all the other packets so we used Ethereal which is a free network analyzer public domain that you can find at Ethereal.com. By going in an doing a find, by a string, you can find in the decode, TCP check sum, and I just put TCP check. Then I click find, and then boom, I would find several TCP check sum errors. So TCP check sum incorrect, check sum incorrect, check sum incorrect, not all but many of these packets had bad TCP check sums.

I thought to myself, what could be possible causing that to occur? And then I went to the machines, server, network interface card. And I found that check sum off load was enabled for both transmit and receive to the network interface card. Now the reason why this has started to be popular is because Microsoft announced in 2001 that they did some test with network interface cards that did the offload of the TCP check sum, and hardware instead of in their operating system, and so by using the hardware nic card as the TCP check sum offload device system using it's processor. The server had much better performance, because it didn't have to check all these check sums and calculate them.

So everyone beat a path to this particular door. All network interface card manufacturers started performing TCP check sum offloading, in the actual nic hardware. Well what we found was any NIC that we found in a Windows 2000 machine. A Windows 2000 server, that had TCP check sum receive and transmit enabled, those were the perpetrators of the bad TCP check sums. And we exercised those machines by using the que check agent, so that we could see the machine truly with an offered load. And we did find, and corroborate, that other domain controllers suffered too. But we didn't find it on Windows 2003 server, we only found it on Windows 2000 servers, but we found it on pretty much every one that we looked for. So it was kind of my belief that perhaps we found a little problem here with Microsoft's operating system, in regards to how it allows in Windows 2000, the TCP check sum offload to the network interface card. Because we found multiple of these. Now after we turned the TCP check sum offload off, in both cases of an HP network interface card, and also a 3Com network interface card.

The performance went up, so it didn't matter what network interface card vendor it was. The performance went dramatically up, when we changed to turn TCP check sum offload off on Windows 2000 servers. And we calculated that out by doing a que check measurement and we got much better performance orders of magnitude, and an absence of TCP check sums. So we very meticulously change network interface card vendors. We turned TCP check sum load on and off, and every single time we found if you're using a Windows 2000 server with TCP check sum offload, at least in the case of Hewlit Packard and also 3Com network interface cards. That they occasionally cause TCP check sum miscalculations to occur from those devices.

So it might be a good idea for you to go out and check all your Windows 2000 active directory servers and find out if they have the same thing turned on, and you might want to see if you're getting TCP check sum errors. If you are, turn off TCP check sum offload, and I'll bet your problem goes away, and your authentication occurs faster and your users of these applications are much happier.

A new digital border wall for your network, datacenter or device? Compromise of the wrong database can be catastrophic to a nation, customers, the organization - and you!

United States "data sovereignty" can be better protected

United States "data sovereignty" can be better protected, shielding high value data from foreign states, and internal user threats alike. Hop Sphere Radius security algorithms create a perimeter by limiting the number of hops network packets traverse as they go through routers. As each packet passes, routers decrement a toll (hop count value) until hop count is exhausted and the packet discarded.  Packets are discarded by design to prevent packets looping infinitely. All TCP/IP device default settings have enough tollto circle the world multiple times, opening compromise to hackers outside the required communications sphere. Don't firewalls protect from such situations? Don't firewalls protect from such situations? Ask any compromised organization having spent huge budgets on every security firewall appliance known to man - not always. A digital border can be set on each device uniquely to its appropriate sphere. Safety can be magnified beyond firewalls by setting a digital border on the device itself, without dependence on a secondary device, providing additional individual security. Wouldn’t it be comforting to know the server containing critical data or the biomedical device maintaining a loved one in the hospital had an individual digital border so it cannot directly communicate to a hacker? Wouldn’t it be comforting to know the biomedical device maintaining you or a loved one in the hospital had an individual digital border so it cannot directly communicate beyond the hospital to a hacker, even if firewalls were misconfigured or failed? Every compromised organization had many firewalls, yet they were vulnerable. And if that isn’t enough, NSA’s own hacker toolkit, recently compromised, gave hackers new automated “point and shoot” tools to break into firewalls and servers. We desperately need powerful individual digital borders providing an additional layer of security. Can your most valued database open a direct tollway to China or Russia right now? Can your organization's most valued database open a direct tollway to China or Russia right now? Many can, because they allow software updates via the Internet - huge fail. Complaints about foreign state and hacking from beyond the United States are prevalent. Why are high stakes, high value devices allowed to have the ability to connect to distant hackers simply for the convenience of direct internet updates? Eighty percent of nuisance and real hackers are located outside the United States beyond the rule of law. Thousands of unsophisticated nuisance hackers and terrorists running "script kiddie" Internet downloadable hacker tools are consuming massive firewall processing and human resources, of which, 80% can be eliminated by changing Hop Sphere settings. What we're doing is not enough ... we are by our own reports losing the cyber battle. At the recent RSA Conference in San Francisco much discussion was about the "Reasons Why We’re Losing The Cyber Battle". Awaiting my turn to present at a military Cyber-Symposium, I listened as cyber commanders bemoan how international state sponsored hackers are compromising their highly secured systems. Digital borders stop foreign hackers ability to directly connect to U.S. systems. Rhetorically, do we need foreign governments accessing our military servers? high stakes experience ... escorted by heavily armed forces into the Pentagon following 911... I was reminded of being escorted by heavily armed forces into the Pentagon immediately following the 911 disaster for communications recovery. Our military hit at its heart, shocked, yet resolute to overcome. One can only imagine what it must have felt like to experience such a blow as a commander. And now it's a new war - digital cyber warfare.

I was challenged to find a solution - remembering years of deep packet security analysis starting in the early eighties "staring" at protocol analyzers at Lockheed in Sunnyvale and later working with Harry Saal and Len Shustek inventors of "The Sniffer" at the Network General startup (now NetScout). I considered having trained 50,000 professionals in Network Forensics having certified over 3,000 as Certified NetAnalysts across 27 countries, and later serving as Technology Officer at NetQoS/CA Technologies solving complex problems on the world stage.

I've been challenged by thousands of network forensic professionals that are 98% hyper critical technical thinkers. In training they play "stump the chump", an affectionate term for challenging an instructor's knowledge. Liken it to President Donald J. Trump's unabashed challenge to his opposition. Let's just say I've been "schooled" in network application and deep packet security theory and it continues with the emergence of Hop Sphere Radius Security. Grasp the TCP/IP theory... be open to this "not-so-new" solution Consider this “not-so-new” idea, grasp TCP/IP hop theory, without dismissing its ability to be rapidly deployed and unique device security benefits. After careful consideration, most wonder why we didn't change default Hop Sphere long ago - broadly. Saal remarked “this is a nice and clean approach to reducing the exposure surface, and well worth implementing broadly...” Network General founder Harry Saal remarked “this is a nice and clean approach to reducing the exposure surface, and well worth implementing broadly, not only in governmental circles” Safety can be magnified by setting a digital border on each device, network or data center individually, denying hacker direct connection despite firewall misconfiguration or failure Learn more about adding powerful new digital borders to prevent:

Database and Data Center Compromise - Keep packets inside data center even when firewalls fail. Foreign State Hacking - stop Russia, China and others from direct access, preventing lateral positions. Internal User Threats - limit general user access to high value databases. IoT Device Takeovers - protect cameras and IoT from faraway hackers. DNS Web Redirection - to Foreign Locations "Beyond the Rule of Law" Nuisance Deep Web "script kiddie attacks" - from consuming security device and human resources.

Click here to receive more information on Hop Sphere Radius Security Algorithms.

The Pentagon was locked down; our team was escorted by heavily armed forces outside the emergency perimeter, through security and past the office of the Secretary of Defense until we arrived in the basement entering the network control center where we would assist in communications recovery after the 911 Pentagon disaster. No need to fear an accurate definitive irrefutable diagnosis - unless you don't want the true root cause! Our five-member team of Forensic Network Security Analysts started solving communications problems caused by the damage. While the nation was in shock we felt honored to be called and recommended by both Meta and Gartner Group at the time.

The team was made up of the highest caliber network technology troubleshooters, field tested solving complex problems for Fortune 500 and government organizations, trained over 50,000 network technologists in 27 countries and certifying over 3,000 as Certified Network Forensic Security and Problem Resolution Specialists – Certified NetAnalysts.

Step by step we diagnosed communications problems within the Pentagon and between other critical sites. In this article, I will use the Six Thinking Hats Methodology/Model to describe part of our process. I will begin by describing the model and how it offers effective ways to deal with IT Problems.

Each of the Six Thinking Hats describe different ways of thinking about the same problem from differing perspectives, motivations and personalities. Every person has one or more of the six hats as their default personality and their angle/agenda that creates competing opinions. Six Thinking Hats methodology helps us appreciate varied perspectives and harness all the benefits of diverse thinking. Sometimes this is referred to as parallel thinking allowing problems to be solved without long “linear” meetings sidetracked by participants with opposing agendas and personalities. Essentially this way of addressing problems uses all participants, their default personalities and agendas in an organized manner. Here are the Six Thinking Hats:

White Hat – Facts, Neutral, Objective Information Red Hat – Emotions, Hunches, Intuition, Gut Feelings Black Hat – Critic, Analyst, Logical, Negative Yellow Hat – Sunshine, Optimism, Positive Green Hat – Creative, Growth, Possibilities, Ideas Blue Hat – Cool, Agenda, Process, Organize, Overview, Decision

Our first meeting was to gather information. That would be a “White Hat” function, starting with a “Blue Hat” organization agenda to establish priorities and create a process. The idea behind Six Thinking Hats is to harness each thinking style without allowing any one style to dominate. So imagine a meeting where all Six Thinking Hats are involved. The order of the Thinking Hats start with the Blue Hat to bring about order. But rather than look for a Blue Hat person per se, everyone in the meeting works to be the best Blue Hat possible. That means regardless your typical hat, we all play the role of a Blue Hat thinker as a group, and if there is any competition, it is to be the best “Blue Hat” thinker even though we may typically be a different Hat color by default.

We agreed that we would determine the facts and create a process to organize our efforts. That meeting used Blue Hat, White Hat and looked at Red Hat to check in with everyone as to how they felt about the process.

White Hat process looks at the facts. Where was the Pentagon damaged? What communications were impacted and what were the priority of troubleshooting efforts? The first fact was that the online network diagram system was damaged and diagrams had not been printed for disaster recovery, so that meant using giant white boards to diagram from collective memory to bring everyone up to speed. This was not a time to critique, but a time to use the best minds to bring everyone up to speed, making a priority list and start working the priorities.

The first meeting was a success, our team worked with dozens of others that had already been working around the clock, were tired and we provided skillful aid. The building was still burning and the smell of smoke everywhere.

As we all worked the troubleshooting priority list Six Thinking Hats allowed us to get intuitive hunches using Red Hat, rule out obvious dead ends using Black Hat and keep the process moving using Blue Hat.

As problems were resolved and systems started working through alternate circuits and backup systems we used Green Hat to discover lessons learned, how things could be done differently in the future and Yellow Hat to use those lessons learned in a positive manner to improve the future outlook at a difficult time.

Critical complex problems require deep packet analysis. Capturing packets is rarely easy in large network environments. We had to use network taps and network switches to gather packets. Looking at network traces is easy, reading and interpreting packet traces requires knowledge of deep theory, communications protocols, timing, applications and the ability to use analyzers skillfully. As an instrument rated pilot I often let right seat passengers hold the aileron controls as we take off. It makes it seem easy - like looking at packets - anyone can do it. Few technologists that capture and look at packets are able to actually solve a problem - that's hard and requires more than running software - like landing an airplane in the fog - it requires skills and knowledge few possess. Make sure your network analysts have tools and deep packet analysis training to solve problems before you let them "land your corporate jet" or diagnose your network security applications with the CEO on board.

At the Pentagon motivation was high at the time to accurately mitigate problems as nobody was on the hot seat for that disaster's responsibility in the IT management team. On the darker side, other high stakes troubleshooting endeavors have proven that some very powerful people do not wish some problems to be diagnosed accurately, desiring rather to hide the root cause of problems because of who manages, or what pet projects or favored vendors might be implicated. I have been asked more times than I care to admit to spin an otherwise irrefutable definitive diagnosis to protect its implication. If there is any reasonable doubt, it too is reported allowing them to formulate their own conclusions. No need to fear an accurate definitive irrefutable diagnosis - unless you don't want the true root cause!

In summary,

White Hat allowed us to use and collect neutral objective facts and information for the report. Green Hat allowed us to see how things could be done better in the future. Black Hat critical thinking enabled many negative aspects to be recognized and not repeat failures. Red Hat enabled hunches and gut feelings to be used to diagnose problems quickly. Blue Hat organization ideas allowed an orderly process to occur resulting in cool thinking that brought about productive troubleshooting and decisions. Yellow Hat optimism led us to see how fortunate we were that primary circuits were missed and how we could use the findings and mitigations to improve Pentagon survivability in the future.

Thinking Styles

Western thinking is based largely on Analysis, Judgement and Argument with strong ego involvement with the winner surviving criticism while maintaining adherents. This thinking comes from 1.) Socrates who points out wrong argumentatively, 2.) Plato who believed we see only shadows of the truth, as it is hiding, and 3.) Aristotle who’s thought promoted systematized inclusion/exclusion like “box” thinking.

Japanese style is largely White Hat, polite, mutual respect, face saving, listening without preformed ideas adding neutral information until things gradually become obvious.

Many other inputs to thinking styles exist, but the above illustrate our thinking is influenced by many things and we should embrace the variations and utilize alternate thought to solve our problems and reduce the time meetings consume.

Applying Six Thinking Hats

Before a meeting begins the basic order of Hats should be set, but the order can be changed and returned to as needed to maintain a dynamic process. Only one Hat should be used at a time with all participants “thinking in parallel”. The power of all thinkers using the same Hat is powerful and moving a group or yourself through the Hats presents productive conclusions and team consensus.

If people are critical together it decreases the “black cloud” feeling of one critical person opposing a Green or Yellow idea. It also helps non-critical people do a better job thinking critically and vice versa.

If a participant wishes to “show off”, or be in competition, they should participate well in the Hat color in use along with others opposed to being contrary or out of current Hat thinking.

While playing the game, which is essentially managed removal of ego some rules apply:

No idea attacks or personal antagonisms Black Hat covers devil's advocacy and risks together as a group so as to be working together to uncover risks reducing idea attacks and allowing everyone to participate in identifying risks. Don’t switch Hats in the middle of a “Hat” just because you thought of something. Emotions are expressible Participants remain who they are, but follow the rules, i.e. aggressive people are still aggressive, just staying within the “Hat”. Blue to start and Blue to finish, like bookends.

A typical meeting might be set as Blue, White, Black, Red, Green, Yellow, Blue. No need to use every color in each meeting. After heated discussion or outcomes advise going through Red to see how people feel about the results.

Six Thinking Hats Benefits

It works, you see results immediately Simple to learn, use and implement You can do it by yourself Modifies behavior without attacking Used at all levels Improves cross-technology interaction Reduces conflict Encourages cooperation Enhances thinking Supports change initiatives

Consider reading "Six Thinking Hats" Book by Edward de Bono

How Close To Maximum Theoretical Performance Are Your Applications?

What Would It Cost To Improve?

If Key Applications Ran At Near Theoretical Speed How Much Would It Save?

How Long Will It Take To Recoup Investment?

Apalytics' Executive NetAnalyst identifies how to get the answers to these questions.

IT Performance Validation benefits C-level decisions

TCP Selective Ack TCP Dup Ack TCP Previous Segment Lost

TCP Selective Acknowledgment creates performance complexities. Bill Alderson provides a short TCP tutorial on the function of selective acknowledgement troubleshooting. TCP checksum offload engines (TOE) and WAN optimization cause problems with selective ack causing data duplication.

Transcript for this Video

TCP Selective Ack is a TCP Option that reduces packet retransmissions of data already sent and received. If only one packet is lost, only one packet is retransmitted. When Selective Ack is not used lost data is sent from the first packet lost even if the node received subsequent packets. Selective Ack allows the recipient to notify the sender of what data "holes" are lost and only those holes are resent.

Transcript: Another problem that's brand new out there in the IT environment, and one that we're seeing more and more because we've got OSPF links that are being load bounced across what people believe to be equal path costs. Well, when you have truly equal paths, a wide area network rarely gives you equal paths in latency. So, you may have BGP or OSPF load balancing going on inside your infrastructure or inside the internet and you may have two parallel paths going to the same place that you're load balancing across. Where packet number one is getting there after packet number two because we've got what we call SKU delay or one path is faster than the other path. We're putting packets in a queue, where they're going one path one time, one path another time. What I'm seeing, because of that, is the following manifestation, and we're seeing lots of use of selective acknowledgements.

Now, a selective acknowledgement is an advanced TCP function wherein we show the sequence number here, that's the native sequence number, and we're sending out 448 bytes here of data. What we're finding is that down here that this packet, the smaller packet, the 448 byte packet, sent after the larger packet, the 1460 byte packet. Believe it or not, the second packet is arriving before the first packet. Consequently, what happens is TCP says, nope, I can't acknowledge the receipt of those 448 bytes, because I have not received the 1460. So, the native sequence number plus length equals acknowledgement is not going to work for us. And the reason is, that until it gets the 1460 and plus the 448, it's not going to acknowledge it appropriately. But we have a mechanism so that we don't have to retransmit everything that's been transmitted. We can say I've got this part of the transaction. I got a forward sequence number, but I didn't get the past sequence number and I'm missing and I have a hole in my TCP data.

I can tell you about it using a TCP option called selective acknowledgement- this something that is set up in the TCP SynSynAck options, when the session first begins. And you can go in and look at those packets and you'll see that these options are being set-up. Selective ACK is one of them that is very popular. It works quite well; however, you're going to see the manifestation of the receiving protocol stack, kind of complaining a lot that it's getting packets out of sequence. I kind of want to show you what that looks like. So, here we have 1460 plus 448, and here you that this is the same left edge of that data. So, this is the left edge. So, we didn't get this 1460. We didn't get 1601 plus 1460. We are missing, in fact, this packet right here. But it's telling me, although, I didn't ACK that packet, I am selectively acknowledgement. This stands for selective left edge of the data, which starts at 3281,which is 3281 and I've got 448 bytes there and so the right selected right edge of that data is up here, which would be the end.

So, if you take the 448, plus the 281, it's going to equal 729. Okay, it's all very simple math, but this is the way the TCP protocol stack works, for selective acknowledgements. Now, once the node does, indeed, get that packet, then we see the acknowledgement going up to 729 down here. It all recovers. But notice the chattiness of your session. You've got a recipient of a bunch of data who got information out of sequence. So, he sends one, two, three, four selective acknowledgements. I got this, but I didn't get the one before it. So, consequently, we need to be on the watch. We need to see if some of theses fancy features that we're turning on and all of these routers and gizmos and widgets, because we went to router school and they said you can turn on this load balancing option. Well, be careful that after you turn that thing on, that you go out and put a packet capturing analyzer before, and you put a packet capturing analyzer afterwards, to validate what you just did to your customers or your packets. This is happening a lot out there, because people are turning on these links or turning on these features and they're not turning on an analyzer to find out what was going on. So, that's another example of stuff that's happening out there in the field that you need to know about on a regular basis.

Every organization has a unique technical architecture “fingerprint,” characterized not only by the products it has chosen to deploy, but also by the organization’s business culture and its impact on the technical evolution of the enterprise.

Even organizations using identical collections of products are likely to have disparate technology fingerprints. These differences can arise because: network configurations require varying degrees of customization; the way a product is used can differ substantially from one organization to the next; and network-monitoring practices range from the obsessive to the negligent.

When it comes to the technology of his enterprise, the CIO must be both a geneticist and a sociologist. He must have a firm grasp of the “DNA” of his network, and he must understand the societal framework in which his technology is used. In addition to understanding the core components of his network, the CIO must understand the people and practices that put the "ridges" in his technology fingerprint. He must understand what his users need from his technology, what their emotional responses to his system are. And his understanding must not be limited to a snapshot of the present moment. His understanding must have an historical basis: to do his job properly, he must understand how his users' needs and responses to the technology have changed over time.

In my 25 years as a practicing Critical Problem Resolution specialist, I have diagnosed hundreds of debilitating network problems. When I arrive on-site, the root cause of a major network issue has rarely been identified, much less remediated. As we perform active and passive system analysis, facts begin to emerge. We compare what we have discovered to the theories that guide our work; the incongruities between observed fact and known theory help us to identify possible causes of the debilitating issue. We can then propose a mitigation plan, which we validate before implementation.

Our ability to rectify these debilitating problems is rooted in two primary factors. First, our efforts have uncovered new information integral to the resolution of the problem; and second, we interpreted that information in such a way that we could recommend a paradigm shift that would remedy the debilitating issue.

After such an arduous adventure, thinking clients often ask, “How can we keep this from happening again?” In the past 25 years, we’ve been asked that question a lot, and our response is always the same. We offer them the following set of best practices: the five pillars of Technical Systemization™:

1. Architecture Ownership 2. Business / Technology Integration 3. Problem / Change Management 4. Decision Support Metrics 5. Application Development Optimization

Our Technical Systemization™ methodology is fully supported and compatible with Internet, IT Architecture, ITIL, and ISO standards and principles. It works for organizations of all types and sizes – without altering your organizational structure. We assemble multidisciplinary, virtual teams that deliver training, create artifacts, and implement best practices across both technological and organizational lines, collaborating in a manner similar to the one used by our divers intelligence agencies as they span the demarks between governmental silos to execute coordinated responses to terrorist threats.

Over the coming weeks, I'll be writing quite a bit about the things the issues that are most relevant to your ongoing efforts as a senior IT Professional. Stay tuned, and please feel free to comment on our posts; there's a lot we can learn from each other.

Most cyber professionals don't have the opportunity to see an actual cyberattack play out. They understand concepts and they hear about different TTPs, but rarely does somebody have the opportunity to see an entire attack flow and how it affects systems in real time.

Watch and learn as a real cyberattack plays out in front of your eyes on the Cloud Range virtual cyber range.

A live instructor will demonstrate how cyber defenders detect and respond to an attack as it occurs in real time, while showing attendees how security tools play a part in the process. For this unique experience, participants won’t know what type of attack they will witness until it happens. Participants will have the opportunity to engage with peers and with the instructor to discuss the steps that the attackers appear to be taking, as well as the necessary defense methods as the attack unfolds.

This immersive learning experience allows participants to simply observe or to also verbally engage during the session. At the end, learners will be assessed on their understanding of the attacker’s tactics, and the steps that were taken by the instructor to detect and respond to the attack.

Debbie Gordon

As founder and CEO of Cloud Range, Debbie Gordon is a globally recognized entrepreneur leading a new category in cybersecurity. Cloud Range was founded on the premise of closing the cybersecurity skills gap by giving security teams the ability to gain real-life experience and practice defending against live cyberattacks in a protected customized dynamic environment. A consummate entrepreneur, Debbie began her career 25+ years ago in the technical education/certification space and has since built and sold several companies in eCommerce, IT asset management, and training.

Marcus Linder

Marcus Linder has over five years of incident response and cyber range development experience. A United States Navy veteran, he has led multiple cybersecurity engineering teams to develop and implement fully independent incident response training cycles.

Full Session Text

📍 My name is Debbie Gordon. I'm a founder and CEO of Cloud Range. We are a full service cyber range provider. We help security teams around the world. Be prepared for cyber attacks by immersing them in live simulations on a very protected environment.

For this upcoming session, I am very excited to have Marcus lender here with me. Virtually Marcus is one of our attack masters. So he's going to be leading an actual simulation on our range.

Just a couple of quick things, just to recap what we talked about this morning. There are a few reasons that cyber ranges important number one is that there's a huge skills gap and it's really an experienced shortage. So people are only as good as the experience that they have. So we need to immerse people in simulations in order to accelerate experience.

And the skill shortage is only getting worse. The number of attacks, the number of technologies and there's just not enough people to that have enough experience. Traditional education and certifications are not enough. One reason is that most of them are most programs are very.

Theoretical and don't have the ability until recently to have hands-on experience. And we're able to address that with live simulation. And then finally on the job training is really not an option. It happens, but it's not really a practical or dependable way to ensure that people are trained and have the right experience that they need to protect their, to protect companies.

We have seen an increase in utilization of cyber ranges. Cloud range was founded in 2018 and in just that short time, we've seen just a huge increase in adoption rate of the use of a cyber range. Because as I indicated earlier, it's like a flight simulator once it's available.

Nobody doesn't use it. You don't want to crash the plane. If you have the way, a way to practice in a safe environment. We're able to simulate and then accelerate, people's learning. As a result

traditionally for testing really a lot of red teaming, but what we've done is moved the the use cases of a cyber range to training the defenders. And so SOC and DFI, our teams are able to practice, detecting and responding to attacks. We also do red teaming and purple teams. As well as advanced tabletop exercises.

But what we're going to see today is an example of a live simulation of an attack. And I'm not actually going to say which one it is because I'm not sure what Marcus has pulled out of the hat today. But he's going to show you how how it happens and the environment that you'll be seeing is an actual enterprise environment on the range. It's a multi-segment network. There are live tools in it. So you'll see fully licensed tools. I'm not sure which ones you'll see today, but we have things like QRadar and Splunk and Palo Alto and checkpoint, and many others in the range so that people are able to train on using tools that they use in their live environment.

Usually we have about. Somewhere between eight and 10 people logged into an exercise at one time, it is a team exercise. So this is something that's typically scheduled and facilitated similar to a sports practice, because if you just say to a sports team, go on the field and practice without a coach it's probably not going to be that productive.

So people like Marcus part of cloud range who helped teams by facilitating the simulation exercises. With that. I am going to turn it over to Marcus Linder I'm hailing from central Illinois with cloud range. Thank you very much, Marcus.

Hello. Thank you, Debbie. As she said, I am Marcus Linder. I am a senior, attack master with cloud range, and I am going to run you through an attack. Now, the big thing to remember is that from a training defensive side, it is. Primarily about leading the team down a trail of breadcrumbs so that it engages their analytical brain.

When they have a real attack on their network, they're thinking in the way they should to remediate that attack. I'm going to be going through those breadcrumbs and those major points of this attack from a defensive side. We're going to start out and right as you open up into your workstation.

You see that. Workstation Windows 7 CTN2 to an employee called and said, that is acting weird. As a SOC analyst, I should probably investigate that, which I actually have it open right here. That's not good. So early understanding of what's going on is everything has been encrypted. But I don't necessarily want to take their word for it, but I'm going to start looking at files that have been recently modified.

And sorting by date modified today, then you can see that a bunch of files have been encrypted, but also because something had to launch this encryption attack, we find actually a PowerShell. Found in these, and obviously this is significantly close to the bottom because it encrypted quite a few files.

If we open that in notepad plus we can actually find that there are a couple of registries keys here for H key current user software, Microsoft UAD. So if we look at this registry entry, we'll go ahead and open up the registry editor and we go to H key current user software, Microsoft, and then you add, we can actually find a key right here, which. It's most likely the public key that our files were encrypted with. So that's good.

That's good information to know, but it doesn't actually help us decrypt anything because we need the private key that the attacker has. Now, if we look at components, we can actually see everything that was encrypted on the system, which is a lot. So if we So from this we need to think of some sort of initial access vector.

W what could that be? And one of the earliest and simplest things you can look at is Microsoft outlook. And if you look at Microsoft outlook, there is a message today. It says, hello, attached to the salary report for last year. Please review it. For executive branch only. I'm not going to click on it, but if you go to the location of that PowerShell script, which is C user 82 epi at a local temp, which is a very suspicious for a location.

Anyway if you look there there's actually a salaries dot doc X, but if we open that with no pad plus it's a bunch of. Gobbly goop, except there's a couple of strings in here. So this PowerShell that exe, which is not great, and a windows document and then we actually have this IP of where it's actually going out to and grabbing in images dot PHP with a password of, there is no cow level and pulling that down.

Oh, and there's that PowerShell script that we. And then it is going to execute PowerShell again and execute the PowerShell script. But now we have a potential IP to look for. So if we go back to our machine and open up our Palo Alto firewall,

And I've already thrown in the query for that IP, if it's still in here. It is. So if we look at that IP that we got from that salary dot doc, we can find that it's connecting to two machines within our network, the 110 and the 100 dot 11. The windows seven CT and two is the dot 11. So we need to look at the dot temp. Oh, it was encrypted as well. Wonderful. So if we

if we go ahead and look Regedit on this machine. And we go back to the components that were encrypted. We can actually find that the Z drive was also encrypted. So not our necessarily our local drive. If we look at the file Explorer, we see that the Z drive is actually a file share. And if we were to open this, we see that these were also encrypted, but these files are on our file server.

So we should probably go look at our files. On our file server, we can look into that file, share, see the secret documents and see that everything is also encrypted. However, our files share, this is something I know, but the teams have to discover for themselves along with all of this information.

Is that if you actually have the folder go to properties and go to previous versions, there's actually shadow copies of all these files. So without paying any ransom demand, without paying, without needing to decrypt anything, we can just go ahead and restore that entire folder to the last known good.

And so if we open it, we can actually see our good files and the encrypted files for side-by-side.

So once, so the file server is successfully remediate. Now, something that would happen right now that I'm not going to do in this demonstration is that we would act like the FBI seized some server from the known attacker. And they're giving a copy to you saying if you need to decrypt your state, here's a copy of that server.

So then they'd have to actually connect to the attacker server and try to find in there the decryption keys and the the decrypt script that the attacker would have on his box.

And so then we would have conversations with the teams about, okay, how would you. Prevent this from happening in the future. And they would say better firewalls firewall policies blocking this IP, which is also a good idea activating different group policies to stop local users from being able to run PowerShell commands theirs.

And then we have them do it. So then not only are they, do they know how the attack happens and where it goes through, but also the remediation process. Also what I found as the most valuable component to this is did this attack as I've gone through. Somewhat simple in design. However, the key is when they are looking at the file server and don't know what to do.

And then you say, oh how would you look at. Previous versions of the files, and then they would see that their shadow copies or if they can't find the initial attack vector. Okay. How would an attacker potentially get to this box with no previous connections? And normally they'd be, they wreck their brains and go maybe phishing and then they check the emails and then you can see that those that the attack came from a phishing email.

And then once we have all of the remediation actions done throughout the network, we ask them not just what physical things you could do, but Policy changes you can make, as in give, making sure users have education about potential spear, phishing attacks, or the dangers of certain types of malware and ransomware and things like that.

And so it's. I think it's a very valuable method to train people because it leads them down this bread trail. And when they get to a point where they're stuck, they have to engage that analytical brain and fight through to find the next breadcrumb.

Does anybody have any question?

Or did I possibly lose anybody anywhere,

Marcus? Yes. Here we go. Hey. Yeah. So questions. So in a comment right now, you talked about, and I just seeking clarification And your range. You made a comment. I think that you solicit an action from the user. As in I interpreted as a verbal response. So is there like a digital twin in your environment that the system and illustrator or network administrator actually gets to then in my speech this morning, I actually talked about friction.

So you're providing. The CIS admin would friction. So do they actually get to go implement those changes and the infrastructure to then see if it rectifies the breach?

Yes. And they not only do they get to implement the changes, but they also can see them happen. For instance, in the firewall, you can see these connections and you can instantly go into policies.

And create a policy to block that IP if you want it. And most of the time, since this is free range, not care and it's virtual. We don't care about you burning it down. If something catastrophic happens, normally there's no provisions other than the incident commander or whoever's running this particular exercise from the team perspective. If one of their analysts says, Hey, can I block that IP at the firewall? And they say, go ahead. At no point, do I say, oh no, you need to wait a second and make sure Nope. If you want to block it in. Normally in this instance, they, I know they've tried to isolate the boxes from the internal firewall.

So then they can't communicate out in further infect anything. All of that stuff is completely up to the teams and if they want to implement it, they implemented, I will ask a probing question like how could they have gotten on the box or something like that. And then if a team is very much struggling, then we can go to a much more coaching role of, okay if this box is infected and this box is infected.

Does this box have a spear phishing email on it either. And walk them through that analytical process a little bit more if they're struggling really badly, but most of the time it's, they run free. And then there's a couple proddings. If they get stuck somewhere and run down a rabbit hole for 45 minutes.

Thank you. Great response. One more question. So environment that range that you provide, is it a generic range or does the customer have the opportunity to export their environment? So that way they have a familiar environment. And the reason I say familiar is because, when we're defending our or your environment. I've always argued that you have to understand your environment. So while I was the chief engineer at CENTCOM, I could look at the ports and protocols of 4,000 applications. And I can tell you, based on IP address, what country they were coming from. And so when you block a, when you block something in and whether IDs IPS, router what's the impact to the network.

So what's the outcome, the mission outcome of the application. What did you just stop when you were denying the adversary? The point of the question is that a generic environment that you're doing the training or is it customized for the. For the customer. So they know exactly what the impact is to their environment.

So from a. Default perspective, it's a generic environment, but it's also simple enough that it's simple and complex. It's not too complex, but it's simple enough so that the any team that comes in here can understand it relatively quickly. And if they need reference, they can look at like the network map and just see, oh, is our server segment, our user segment, our SIEM segment.

But if a customer does want a customized environment, we will do that as well. We, a little bit more of a headache obviously recreating their environment, but we will do that if that is what the customer is adamant about.

Nope. Perfect. Thank you.

And I'll just add to that. It's really about the objectives of the exercise.

Fortunately, or unfortunately for a lot of the fortune 1000 companies that we work with even what may seem like a really great security team still needs to focus on even how to detect and investigate things and respond to them versus making assessments about the actual environment and where things came from.

So there are. We can replicate anything. They're just often diminishing returns on the time to do that. Based on the objectives of the exercise, but it is absolutely. But what you're seeing here is is a generic environment. We do, we have custom environments for OT networks in different segments.

But this, what you're looking at here is relatively generic, but it represents the different segments within an enterprise network.

So my question was about whether I'm really your customer base. Do you tend to find that this is about teams that are, just doing some early training and trying to prepare, is it people that are having. Fairly well trained teams, but want to be more advanced or is it people who, yeah, we got compromised.

And so now we're behind the eight ball already and we need you to come in and help. Who do you see most frequently as your customers?

I would say in my experience, it is the first two the the early team that needs some experience. We have some very early level exercises that let them start developing those processes, not just in remediating a attack, but also with working together and understanding how they communicate and things. And then we give full feedback of, Hey this team member, I was watching their screen. They found all of the breadcrumbs and everything important, but they didn't speak up. And so they we need to. Probably encouraged communication with this group.

There's also more senior teams that we have taken through some earlier levels and they've excelled. So then we give them harder scenarios and to that to really comes and it's something to the benefit of the generic network is. There's not a simple tool that just tells you exactly what's going on X, Y, and Z place.

And you become reliant on the tool. You actually have to analyze how it's working together. What is potentially trying to communicate with where your gaps in the knowledge or alerts that you'd be seeing are I had a senior team go through last week and I hadn't destroyed any scenario we put in front of them.

And I actually had to help them last week because they couldn't see a very hidden man in the middle attack. They could see the rogue box, but they weren't understanding what was happening from it. And so it's very much those first two, I would say the third is probably something we've seen. I haven't been on any calls where someone has says we're coming in because of this.

But I know many times teams that we've trained have come back. I know a team went through ransomware and then came back the next week and was like, oh, we got hit by ransomware two days ago and remediated it in two hours. And that, that just makes you super excited. But no, the it's mostly teams looking who have seen the value of this and are looking to increase knowledge, whether that's from an early intermediate or advanced perspective.

Did that answer your question?

I heard that there are some services that bounce around different IP address. So you cannot see the source how you found this.

I know actually I was having a conversation, but that it most of our stuff is statically routed in this. But we can throw on a DHCP router that then gives everything within a, sub-net just a random IP.

And then, so you have to discover that a little bit more. That's not a difficult change in something we could emulate on one of our networks quite easily.

How many different tools do you guys typically put into your cyber cloud? And can you give us, I know you've talked about some of these, but is there a typical number that an organization will have and use during a particular session?

I would say, I know we have we have different firewall licenses but we can only have one of those.

So we'll do one of those. We normally have a primary SIEM and then we have a. A secondary SIEM in most, I know we're working on getting a couple other licenses and there, so as far as number of tools, I would say normally three to four, if they if they request more, we can throw more on there.

Whether they want to keep the older, have a brand new, just now everything goes to. A Splunk server or something like that. We can set that up, but I would say normally three, sometimes four.

Hey Marcus, Dave wills again. So the companies that are coming in or the customers that are coming in and training are they training primarily IPv4 IPv6 or mixture?

It's mostly been IPv4, we haven't gotten a request for IPv6 yet. But. That's a couple of routing changes that shouldn't be bad or hard to do.

Yeah. I just wonder because as you think about it and maybe applies more so from an external perspective, a lot of people were in favor of natting because then, not only did it provide obfuscation, it limited the adversary's ability to, pinpoint your IP address and then launch an attack with IPv6. Essentially it's like unlimited IP addresses, which makes it a whole lot harder to determine where the attack coming from. And so then in the department of defense, IPv6 has been. Mandated since 2011. But as we look across industry pick VM-ware for example as you look across our product line, there's different levels of implementation of IPv6.

It's not standard as they go out and acquire, a product from, a small business. There's a lot of work to bring it up to a certain level. And so I'd just be interested. And as you guys make progress towards. Implementing IPv6 in the range. What you see as far as I'll say maturity and the customer's ability to respond appropriately,

I don't think it would be. I'm assuming that this range was just instantly transformed into an IPv6 range. I don't think it would cause a substantial issue because in our trainings, we ensure that there's always an. An instant reporter. So somebody logging and cataloging all the information, regardless if they have some communication that they always use for incidents that we encourage them to use and continue to communicate.

But we always ensure that there is somebody designated to that role. So if there is an IPv6 address it would take slightly more time to ensure that you wrote that down correctly. I don't think the relevance of IPv6 would change the outcome of the scenario too much, or with the teams that we know about, they would be able to just keep continuing and stride.

They'd be like, oh, that's IPv6. And that would, I think, would be their biggest comment.

Hey, I've got a question. Do you guys have the ability to allow incident responders to. Bring up a VM of their own, maybe running tools my TCP dump or Kali Linux or anything like that. Can they run that in the environment and kind of interacting? With those types of tools, as well as, the actual hardware devices.

Yes, cause we actually have, they're not on the network map obviously, but if somebody requests that we have Kali boxes in the environment and we have sysinternals on a hidden share in the environment. So then they can actually pull all of those tools down and running them on any box in the.

Okay yeah that's super useful. The difference between looking at the dashboard provided by vendors versus a real deep level SOC analysts would want to look at things at the packet level and kind of look at the cooler, viewpoint that they might have.

Okay. That's cool. Thank you.

You're welcome. And honestly, I, in that. Scenario. I mentioned where that advanced team was having trouble. They ended up pulling down Wireshark and throwing it on one of the individual user stations to be able to see that path HTTP traffic was getting repackaged. So it's, all of those tools are there and they're meant to be used.

Excellent. Thank you.

You're welcome.

Hey Marcus, this is Jim Wheat. I was just wondering what SIEMs you're currently running in your range.

So I know currently it is, we have Q radar in Splunk currently, and this one that's running right now. The the one we have, we also have an ArcSight and I believe we're getting a. SIEM. And and oh, we also have a security onion incidence on here as well.

Okay. Thank you.

You're welcome.

The, there are anything that can be can be virtualized, can be put into the range. Most of the tools that we have in there are ones that most of our customers, it's the usual suspects of tools in there. So as we add them as because customers request them and then they just become generally available.

Yes. Got one more question. Here

Will Smith with the question don't slap me.

So I have a question. So you started the, this cloud range in 2018. What are the biggest friction points or pain points that your customers or clients have come across that you have overcome? What would some of those be? Thanks.

Okay. So I would say the biggest hiccups we have that we overcame was the idea that we talked to a customer. You need to do training there. They're like, that's a great idea. Okay.

Give us training. And then that's where it would just, they'd come to us at the end of the year and be like, oh we paid for this training. And it's nobody ever came and got training. So I think that the most effective use is we now schedule and annotate. Who's going to be there when they're going to be there.

And. We already assigned which scenario we're going to run beforehand. Normally with the CISO or whoever's in charge is not going to be actually be in the exercise. And we'll say, what do you need to work on? How does that, what would help you succeed as a team? And so now we tell them you'll be here on Monday.

At 1:00 PM and we're going to do this scenario. So then the team shows up all into a zoom meeting and they start running. I make sure everybody's ready to go and we'd go in and do a scenario. And it ensures that those companies and customers get that training A, they're paying for, but B get the value out of the training of, okay.

We know that these eight people got trained last month in ransomware. And so now. Are confident in their ability to continue progressing as a team and not just, oh, we have this money for training, but no one's taking advantage of it, okay. So I would say that's the biggest struggle I've seen that we've overcame that way.

Yeah. And that just to add to that that goes to the I just didn't Eisenhower quadrant of urgent versus important that the last thing we want to do is have a customer spend a bunch of hard earned money with us and not take advantage of it. And so it has to be scheduled out. If I get a guarantee, if I'm.

If we didn't schedule these things out for them, that they would probably get to a year later and say, gosh, we never found time. This goes back to the, going to the gym. You don't, you have to make time. You don't just, nobody has extra time in their life. So training is one of those things that unless prioritized it doesn't get done because there's always something that's going to be more urgent.

So Marcus, you're absolutely right that, that's something that I think it's a good lesson learned by us and our customers that it does have to be planned out.

Glenn. Do you have any any thing that you think our customers have overcome besides the obvious Glenn Ewing's here in the audience? Chief revenue officer of cloud range.

Everyone glen Ewing CRO at cloud range, I think to build on. Marcus's response and answer your question a little bit more. I think another thing that customers appreciate is as Debbie mentioned, really that programmatic discipline that we bring to the process. And we've talked a lot about teams and, there are a lot of analogies we can use from a sports perspective.

But think of a coach that tells her team, Hey guys, we're going to have a game at 10 o'clock on Sunday. Have a great week practice on your own and show up, ready to play versus we're having game it's Sunday at 10:00 AM. You have the day off tomorrow to say, we have film at eight, you get taped at 12 you're on the field at one, and we're doing that.

And the entire week, who's going to be better prepared. That's, and I think customers appreciate that by the way, because they're all busy. And are the chief information security officers, many of whom that we do. I don't have time to think about that. And they don't want to think about it.

Hey, want you to bring that program to them? Bring a solution, not just describe the problem. And that's a lot of what we're doing in the market.

I could just add to what you just talked about, think about it. You have that team and you tell them all, yeah, go get training. Here's your training budget and they all get training with a different team, and then they're expected to come together and win. Yeah. Now you need to train with the people that you're going to be with, or at least a contingent of the same group of people that you're going to fight the battle with.

It's military training, let's go, everybody do something different. You need to train with the team that you're going to work on. The problem with.

Thank you bill. And one more thing I'll add going back to the organizational behavior and management of humans, part of this a lot of people in positions in security and leadership especially if they didn't come from the military they don't have a lot of training on managing people.

A lot of them came up through technology and they, that's what they're doing. And so when we're able to help them see things about their team and make observations and recommendations about how their team is working it is a huge weight off their shoulders because that's not something that.

They often have the time or energy to think about doing a lot of people want to become better managers and better leaders, but when things are moving so fast and security, I'm a SOC manager. Doesn't always have time to say, Hey, how am I going to make sure this person gets better leadership skills?

And that this person gets better, better communication skills. And so by doing these exercises, All of that stuff happens at once. And people like Marcus are attack masters. They're able to then provide feedback and say, Hey, I saw this person they're really good at this, but they can use some more work on this.

This person really shined on this. We recommend a little more work on this and that takes a huge burden off security leaders because people need to be focused on what they do best and sometimes observations and management of their team. From a non-technical standpoint is not what they're focusing on.

So this really helps them. They're too.

Marcus any anything left in the scenario that you wanted to show?

Oh not in the scenario, but I did want to run through a couple of sections of the debrief just to show what we do then after the scenario. And this is a very cut down portion. This is not the real thing. We've got three minutes.

I can be super fast. Oh, I got to share my screen. Okay. Fast fast.

Okay. So this is part of one of our debriefs, the we'd go through what the objectives were, what kind of skills are supposed to be gained and flexed during this incident? We tell you how the scenario is. We give it a difficulty level and in the estimated time of about seven to 10 people doing it and then we'd go through and say every single one of the breadcrumbs that we discussed and how.

You're potentially supposed to think about that after the fact. So once they've gone through the whole scenario, they then get reinforcement on what was done, how well they did it, what information they should have gotten from those pieces of information. So then they can know after what they not necessarily should have done, but how the thought process would have been best implemented to achieve.

The remediation of the attack. Okay. And now that's all Debbie.

Really appreciate it. And just a little bit also when we go through the exercises, these, we usually block off about 40, excuse me, four hours for these. The first three hours are usually the actual scenario. And then the last hour consists of. The debrief, which was an extended version of the slides.

The Marcus just went through really quickly. And then the participants do a few different types of feedback. So they go into our learning management system. They take a quick quiz that assesses their understanding of things that happened, even if they weren't the ones that were hands on keyboard for specific actions.

We're able to track that. And those can track back to the MITRE, excuse me, to the NIST framework. So we can look at different KSA codes that they're achieving. And then they also they also give some feedback about themselves. What did they learn? What do they want to get better at? Where were they challenged?

What types of things do they want to work on in their careers? That kind of information is then presented back to their leadership within about a week of the exercise and that stuff is gold. Because the leaders are able to see, what does this person say about themselves? Oh, I had no idea that they wanted to move into forensics or something else.

And so it gives them the ability to start thinking about career paths for their teams as well. And then also what exercises and scenarios we can work on with them in the future. So it really helps set out a plan for them.

All right. Marcus, thank you so much. Great job. And thanks everybody. Thank you everyone.

Unleash the Full Potential of Network Detection and Response

Consider the anatomy of a cyberattack. Look at gaps in security tools and cost of breach. Current challenges, gaps and different approaches to NDR. Scalable deep packet inspection requirements and considerations. Application of machine learning and encryption for the next generation Network Detection and Response.

Paul Barrett is Chief Technology Officer for NETSCOUT's Enterprise and Federal Businesses. Paul’s role encompasses NETSCOUT’s enterprise service assurance, cybersecurity and DDOS products.

Paul has over two decades of expertise in the analysis of network communications – areas of focus have included the human perception of voice and video transmission quality, the health of applications and services delivered over IP networks, and most recently cybersecurity.

Paul has more than 25 years of experience in the IT industry and has been actively involved in international standardization for much of that time. Between 2005 and 2016, Paul was appointed to a number of ITU-T study group vice-chair and working party chair positions; he was also the United Kingdom’s head of delegation for these study groups for much of this period. The ITU is the United Nations specialized agency for information and communication technologies. Paul is a named inventor on 20 patent applications.

Full Session Transcript

I'm Paul Barrett, I'm CTO for Netscout's enterprise and federal businesses. If you're not familiar with NetScout We have been in business for about 30 years and pretty much everything we do relate to analyzing traffic flows on networks.

And traditionally we were doing that for large enterprises, government agencies, communication service providers, to help them use deep packet inspection, to understand the health of the network, but also the health of the applications and services running on those networks. Now in 2015 Arbor networks joined NetScout and they, one of the market leaders in the area of DDOS, detection, and mitigation, and that really got us thinking, we said, isn't it about time?

We took deep packet inspection technology and started to think about how we can apply it at scale to solve. Some of the hardest problems in cybersecurity, particularly in the area of network detection and response. So what I'm going to do probably about 30 minutes to give time for some questions at the end just take a look at some of the big challenges in cybersecurity.

I want to focus on some specific gaps and then we can talk about different approaches. And then we can bring in scalable, deep packet inspection. And I actually want to go through some of the requirements and considerations that you have to think about if you really going to leverage the true benefit of this technology.

I also want to touch on machine learning and the relationship between deep packet inspection and successful machine learning. I want to talk, touch on encryption because people often say, Hey, deep packet inspection, what about encryption? So I think it's it's worthwhile addressing that.

And then we'll just finish off by talking a bit about how all of this can come together to deliver as advanced network detection and response. So let's start with that big picture. And Debbie in the previous session touched on this top on, but we just, none of us have got enough security staff and it's worse than that because we're overwhelming them with far too many events and possible incidents.

But then we've got to think about the data breaches. If anyone thinks they can eliminate data breaches, they are frankly kidding themselves. That's just not going to happen, but anything we can do to either reduce the number of data breaches, or if we accept that some are going to occur. If we can stop them from progressing too far to try and control the cost of a data breach.

That is a very valuable thing to be able to do, but the area I really want to focus on is the existing security tools and the gaps. So a typical large enterprise or government agency has something like 70 to 87. Sorry, cybersecurity tools deployed. Now the problem arises if you keep investing, but you keep buying tools that basically do the same thing.

You're not really moving the needle. So the question is, are there gaps in our existing toolset that need to be filled? Let's just start off with kind of the anatomy of a cyber attack. Now I'm not going to focus on cyber warfare or the actions of nation, state actors. I want to focus on really what is for most of us, our biggest problem, which is basically cyber criminals, people out there to make money.

Now, the way I like to think about this is for any crime, we always talk about there being three things. There being a motive and opportunity and a means. Now, in the case of a cyber attack, the motive is access to sensitive information. It could be things like credit card records that could be healthcare records.

And by the way, a healthcare record is far more valuable on the dark web. Then a list of credit card numbers. It could be intellectual property. It could be my future product plans, or quite honestly, it could be any kind of confidential information such as sensitive financial information. There lies the motive for the attacker.

So then we need to think about the opportunity which we normally think of as vulnerabilities. Now I'm going to focus on the attack surface, but the fact of the matter is. If we're going to deliver applications and services to our employees and customers, we're going to have to expose those applications and services to the Internet, one of the challenges we have today is we were all driving forward with our digital transformation initiatives. And that's great because the whole point of digital transformation is about business or mission agility. The ability to quickly stand up an application to address a new opportunity or respond to a threat or disruption.

But that has a consequence. It means that my tax surface is also changing very fast. And of course, perhaps the biggest vulnerability we have is that ultimately we have humans involved in all of this, and then we get to the means, or in this case, we tend to talk about threats. So if an attack has successfully infiltrated my network, they will have installed some form of malware inside my enterprise network.

And then of course, I've got attacks coming from the outside in I've got people launching DDoSs attacks. And once again, I've got those pesky humans, I've got insider threats. Now there's a fairly well-established concept that when you have the combination of threats and vulnerabilities, that's when risk arises, what do you use?

My other language when you have means that are able to be applied to an opportunity, I have risk, and of course that can be the risk of. Theft of information, it can be denial of access in the form of a DDoS attack or encryption of data. Ultimately within the scope we're talking about here, that's going to lead to some kind of extortion demand.

Now, cause we said you're never going to completely eliminate breaches. That's not a realistic goal. We just need to contain them as best we can. If I have small and occasional well-controlled breaches, I can probably survive that. But if I have. Many breaches or I have a small number of very large breaches that can start to damage the reputation of my business or my agency.

And, sadly we've all seen examples of where that kind of reputational damage has actually taken businesses completely out. And before I move on, I just want to mention the way that ransomware attacks have evolved and certainly how we've observed them to evolve in the last say, 18 months to two years now, the original form of ransomware attack, where somebody gets into your network, they find your data, they encrypt it, and then they send you an extortion demand saying we'll only give you the keys.

If you pay us this much, Bitcoin. But we've all got better at backing up our soft. So what a modern ransomware attack looks like is the attacker will, first of all, infiltrate your network and they will find that sensitive information and they will exfiltrate it. And only then when they have your information, will they actually contact you and say, Hey, I have your sensitive information.

Here's proof that I have it. And if you don't pay me what I want, I'm going to put that. Maybe on the internet, for example, they will also follow up with the threat of encryption or even actually performing cryption because even though you may have a really good backup, you're still going to experience a major outage.

If you've got to recover all of your systems from those backup. And at the interesting third leg to this stool that we've seen be added to these kinds of attacks is launching a DDoS attack. Unfortunately, if he wants to launch a DDoS attack today, you basically go on the dark web. You find a provider, you enter the details of the victim.

You, you basically choose how large you want the attack to be, which relates to how much you're willing to pay. And then you can launch a DDOS attack. And we call this triple extortion and it's all about just putting as much pressure on the victim as possible, turning the screw so that in the end they just throw their hands up and say, okay, how much do you need me to pay you? So let's start to think about what are the gaps in our existing tool sets. Now I'm going to focus on these four. I think they're really important. The first is advanced early warnings. So I need to be able to detect those intrusions as they happen in real time. That's really important if I'm going to get ahead of the data breach and stop it as soon as possible, but also want to be able to triage.

I want to understand who is affected and how bad. And then we get to that attack surface. I want to understand what my attack surface really looks like. And this is interesting because actually my experience, most enterprises don't really have accurate information about what is truly exposed to the internet.

We often find that they've are applications and services. The people didn't know were actually active and being actively connected to. And then we're back to this point about digital transformation. How is that attack surface changing how it is evolving and how do I stay on top of that? Now, so far, I've been giving examples around exposing applications and services to the internet.

But if I'm thinking about deploying zero trust policies, for example, in the form of network segmentation, I can apply the same concept to a network segment and think about what is the attack surface of that network segment. Now, these first two really fall under the broad category of vulnerability. Yeah, the second to contact tracing.

Now, clearly it's a concept that has very much come to the fore during the pandemic, but as a notion, as a concept, it actually goes back to the beginning of the last century. When people realized that when there was an outbreak or an epidemic, if they could identify patient zero, For a particular outbreak and then understand who did they interact with and who did those people interact with?

They could focus their energies. They could basically almost ring fence the incident. And as I say, focus all of the energy that they needed to control it and really. The same applies in cybersecurity. When I'm thinking about lateral movement, how an attacker comes into the network and then starts to seek elevated privileges and work their way towards that sensitive information.

If I can identify the entry point and then understand the lateral movement, once again, I can focus my energy and my resources. And lastly, I want to have the ability to go back in time. Now, oftentimes as we all know, breaches, aren't discovered until sometime after they actually occurred. So I have to be able to go back and say, okay, how did the attacker get hit?

What tactics and techniques did they use? Did they actually successfully take anything? Did they perform data analytical iteration? And I think really importantly, how can I stop that particular attack vector or those particular techniques from being used again in the future? And you can think of these two on the right as being more related to threat detection.

As I said, there's a typical, large enterprise has perhaps 70 cybersecurity tools, but I just want to focus on these four classes of tool for a moment. Now, the first is your security information and event management your SIM and. Pretty much any organization has a SIM and most organizations, certainly large organizations are deploying a security orchestration automation and response system.

This is where your playbooks live. So for a particular type of incident, I've already thought about how I want to address that incident, how I want to respond, and hopefully I can do that. Automated. Yeah, I've got my endpoint detection and response. This is where I'm installing agents on computing hosts.

Now I would say that all three of these are absolutely necessary as part of a complete cybersecurity portfolio, but they're necessary, but not necessarily sufficient because it's really important. We understand and have visibility of what's happening on the network. An attacker almost never enters the network at the location that they ultimately want to be because they're either coming in through, they found a vulnerability through the internet, or possibly more likely they've got a foothold through something like a phishing attack or a spear phishing attack.

The point is they don't instantly land on the database with all the goodies they've got to perform lateral movement. They've got to elevate those privileges and move from host to host until they actually get to where the data. So I believe it's really important to have powerful network detection response, and really reap the full benefit of this technology.

If we're going to complete the picture and have a comprehensive portfolio, now it may come as no surprise to you that, I'm going to propose that deep packet inspection and an in particular, highly scalable, deep packet inspection has a very important role to play. So I thought I would just take a bit of a step back here and pause and say if I truly want to deliver DPI scale, what are the kind of considerations?

What are the requirements I have to address now? The first one is that you need to work. Hundreds of protocols. Now, if you sit down and look at everything going on in your network, chances are 80% of the traffic belongs to one of, let's say 10 common protocols. But if you're serious about protecting all of your mission, critical applications and services, they're going to be using lots of other protocols.

So you've got to have packet pauses that work with all of these protocols. You need to generate compact, but highly actionable metadata. Now I know it's a bit of a mouthful, but I'm going to expand on that in a moment. But really what we're saying here is I can't store packets forever. So I've got to decide which pieces of information am I going to use to generate metrics and which pieces of information am I actually going to keep in their original form.

And we can talk more about that in the most. You potentially need to scale to hundreds and thousands of monitoring locations and aligned to that is the requirement operating all multi-cloud locations. All large enterprises are now saying, I'm not sure that the idea that everything's going to move to public cloud is still totally valid in my experience.

Most organizations are trying to decide for each application and service is the best location, public cloud. Is it private cloud or is it a CoLocation. And we will see in a moment, we ended up with these very complex environments and of course you need to work at whatever the latest networking speed is. So let's start with that.

Highly actionable and the metadata. So what I drawn here is a representation of the different types of information I can pull from the network and the width of each of these segments represents that the sort of the quantity of information that we need to store so starting at the bottom as the packets themselves.

And there are times when information in packets is just really useful. I can generate all of the metadata I want, but if I can drill down and actually look at what was the original content of a packet that gives me the answer that I need, the point is, in a typical system, I can probably keep packets for a few days.

Realistically, there are techniques I can use. Maybe only keep the important parts of the packets. And I discard some of the payloads that might help me increase the retention time of my packet. And I can even get sophisticated and say, what if I see in a suspicious event, maybe I keep put those packets into it into a separate store, but ultimately we need to be looking for generating metadata.

Now the next level. For me, this is the most precious part of deep packet inspection. This is where we're basically using protocols specific pausing of packets. So if it's a decrypted HTTPS packet, I can pull out the URL. I can pull out the user agent. If it's DNS, I can see what was the query type?

What was the domain? What were the IP addresses that came back? If it's SMTP, for example, I can see the name of an email attachment. I can look at the certificate and a TLS handshake. So this is very rich and very actionable. Above that it's also often useful to just be able to know who was talking to whom on what protocol of what time and where, for understanding things like lateral movement, that's really valuable information.

And then at the top we have the opportunity to generate metrics kind of key performance indicators. If you like. And here I can summarize the behavior of each of my servers, of my clients, of the applications sometimes. An application begins. The misbehave could be an indicator that the application has a problem, but it's also oftentimes an indicator that there's an attack starting or an attack is in progress.

So it's really helpful to be able to look at performance and areas as well. The other thing I can do with this kind of summary information is I can use it to get my initial indication that something anomalous is happening. I can start to triage that problem, understand the extent of it who potentially is impacted.

Then I can start to troubleshoot. Then when I have the context that I need, I can start to drill down through these increasingly rich layers of information either to the transaction information or sometimes all the way down to the packet contents themselves. I also mentioned multicloud. So this is very stylized, but regionally, really, these are all the components of a typical multicloud environment.

So I have my data center, my co-lo, my public cloud, my SAS services. And I've probably got some remote sites. In terms of where do I put packet visibility? Typically people will seek to put a physical appliance at the edge of their data center, because that gives a really important location because I get to see what my applications, what my attack surface looks like to the outside world.

I might also instrument a bit deeper into the data center, perhaps in front of mission critical applications or in front of my shared enabling servers. We're finding a lot of customers are deploying in colos now. And that's because you've got these really important peering connections with your public cloud provider, but also increasingly with your SAS providers.

So you'll have direct peer and connections with say office 365 or Salesforce or. And of course, mobile, large campus. It makes sense that the deploy an appliance, but there are locations where I can't go and install a network tab. Public cloud is the obvious one. But there are techniques and methods whereby the public cloud providers will give you a copy of the packet.

In their networks. And if they won't let you do that, there are other techniques such as deploying inline and getting access to packets that way. But you're going to need a virtual probe because essentially your packet analysis is going to be running as a workload in that cloud. But if you take that same concept of a fully virtualized software agent, I can use it in the colo.

I can also use it in my data center to dig deeper into the application stack to start to get visibility of east west traffic. And of course, I can also use that approach for locations where it's just not economically viable, or it doesn't make sense to deploy physical application server, but I probably can deploy some kind of virtualize network function.

So if we take this kind of approach of using the appropriate form factor for our deep packet inspection engine, then we can cover all of these multicloud environment.

Now I put a throw away line in the slide with the requirements you've talked about working at the latest networking speeds. I just want to give people an exercise and you don't have to do it right now, we throw around numbers like 10 gigabits per second, without really thinking about it.

You just take a moment at some point to write out 10 billion. That's. 10 zeros after it. And start to think about increasingly large examples of numbers until you get to 10 billion. That's how many bits per second you need to process. If you're going to perform deep packet inspection on a 10 gigabit per second link on that, how many bits you would a process every second?

Now, just for fun. I came up with this number. I said let's say you've got a 40 gig link. It's occupied on average at 10 gig in each direction. How many bits of information is that in a year? And I came up with this number. I'm not entirely sure how you say it. It's 630 million billion.

Billion million. It's a big number and I backed into it because it also happens to be the approximate or the estimated width of our galaxy, the Milky way in miles. It's an enormous number. And yet that represents the number of bits passing at a single network monitoring location. Yes, that might put some people off and say, oh, I an earth.

Am I ever going to process that much information? But think about it for a moment. Our core networking and switching infrastructure is now operating at a hundred gig and even 400 gig. We already have network equipment that operates at the speed. We have firewalls, web application, firewalls, and other security tools.

So actually delivering deep packet inspection for the purposes of cybersecurity is entirely possible. And there are a number of companies who have invested in this technology over the years, and it basically solved all the problems in this area and been able to keep up with the ever-increasing network speeds.

So putting all of this together, we're starting to think about how do I apply deep packet inspection to network detection and response. So maybe we can start top left. I need that affordable visibility everywhere. So think back to the multi-cloud diagram, I've got to make sure I've got different probe form factors so that I can put a probe everywhere I need.

Now I want early detection, but if I'm processing every packet in real time, then I'm in a perfect position to spot anomolies occurring on the network and to alert somebody as soon as they happen. And I get attack surface visibility from that as well. And this is really important. I get insight into how the attack surface is actually being used.

What connections are being made, right now and not relying on somebody to run a scan every week. When anything quite frankly can happen between one scan and another, if somebody set up a rogue service or server that is being connected to, as soon as I see a connection to that, I can detect that and I can alert.

And then we get into what I've written the separation of the signal from the noise. And this is really about that generation of metadata, deciding which parts of the packets am I going to convert into metrics? Which parts am I going to keep? Maybe it's something like a URL or the results of a DNS query in which can I choose to save, because if I'm going to have retrospective analysis, I have to basically be able to keep data for long periods.

And that means data reduction. So there's this balance here about, I've got to decide, which is the information and I'm going to keep, and which can I actually safely get rid of now the way I like to think of generating this kind of metadata is who building a record. You're building a history of everything that happened on the network, but the great thing is you're adding to it in real time.

You're constantly adding, Hey, what's the latest thing that's happening on my network, but at the same time, because I have a historical record, I can go back and investigate incidents in the past. And the last point about consistent data really is just saying that I want to be receiving the same metadata, regardless of whether it's a large appliance, a small appliance or a virtual appliance that affordable visibility everywhere needs to be providing me with consistent information.

So people often ask about machine learning. And I want to make a statement that in my mind, network detection and response is not synonymous with machine learning or artificial intelligence, there's network detection and response. I like to always spell it out. It's a hugely powerful technology. And when you drive it with deep packet inspection, There's a whole host of analytic.

You can do that. Do not require AI or ML, but that said, it's not a technology we should dismiss because there are some very powerful use cases for machine learning. So I'm just going to work through this and we can think about how deep packet inspection relates to machine learning. Now, I'm going to assume that we have some operational challenge, whether it's a network operations challenge or a security operations challenge.

And I want to operate at massive scale. I want to have digital transformation. I want to be agile. The only way I can achieve. Is through automation. I think we can all agree on that. And this is the point at which I go, maybe I can introduce a bit of machine learning to apply an extra level of automation, an extra level of detection, of classification of prediction.

Now, if you've ever spent time working with machine learning, You really have to use high quality data, both if it's a supervised algorithm and you actually, you've got a training set really important that you have a high quality training set that covers all of your areas of anticipated operation. But even for an unsupervised approach, you've got to have high quality data.

It's like the old phrase, I think it was from the 1970s or eighties in the early days of computing of garbage in garbage out. But it really is as simple as that I can have the best analytics in the world, but if I'm not feeding those analytics with high quality data, I'm probably going to be in trouble.

And of course, deep packet inspection. I certainly passionately believe I've been in this area for over 25 years, that the information you get from the network from the packet really does constitute one of the highest quality sources of data you can have in our industry. We have this phrase, we can talk about packets don't lie, but they really don't because you are looking at the actual transactions, the interactions between our computing hosts.

Now, this is another point I always like to make with this slide, which is. People I've heard people say Hey, I've eliminated human error because I've automated everything. And I'd like to say to people, yeah. But hang on. Most of that automation is probably software, isn't it? And who wrote the software?

At some point there's a human involved. All you've done with automation is provided the opportunity to take those mistakes and errors and to deploy them at massive scale and massive speed. So we cannot take it for granted. But our automation is doing what we think it is. And the same is true of machines.

Yeah, I've spent quite a lot of my career around machine learning and I will guarantee you this, it will take you by surprise. There will always be situations where it ends up doing something you just didn't anticipate. So there is also for both automation and machine learning, a very important requirement to have some form of independent visibility in terms of what's happening.

In my environment, whether it's just the operation of my applications or whether, for example, making sure that my saw has actually done what I thought it was going to do. And again, this is a great application for deep packet inspection, because you can passively take a copy of what's happening on the network and perform independent analytics.

Okay. Often people say, wow, deep packet inspection, and that's great, but isn't everything encrypted now. I wanted to talk about that. The other development there's been in the area of encryption is the emergence of something called perfect forward secrecy (PFS). Where there is no longer a kind of master private key that you can share with your decryption devices.

Now, TLS 1.3 pretty much has the perfect forward secrecy on by default. And you can use it with TLS 1.2 by enabling an ephemeral Diffie-Hellman Forward Secrecy cipher suite. So what do we do in this situation? My first answer to people. Don't panic. We've actually been dealing with this situation for a long time, particularly in highly regulated industries, such as healthcare and financial services and the trickiest to identify those aggregation locations, because the way I'm going to gain authorized visibility into the traffic is I'm going to take the encrypted feed.

I'm going to terminate the encryption layer of it. I'm going to perform my analysis and then I'm going to reencrypt the traffic, as it goes on towards its eventual destination. If we think about what I've shown here, I showed a data center runner, a public clouds zone, and I've got some customers, employees connecting either over an enterprise network or via the internet.

So if I'm interested in connections, Coming from my customers or employees coming into the applications and services that I'm hosting I can use. What's called the reverse proxy, and that can perform that termination and re-encryption we spoke about. And if I'm interested in connections being made from inside my own enterprise to the outside world, it could be a business to business application.

For example, I could deploy a forward proxy, which uses essentially the same approach. But there are locations inside my data center and public cloud where I can also do this. I may want to do it in my security stack because I'm going to have a number of tools that want to be able to look at the raw packet content.

I might stick it in front of my shared enabling services. Now there are situations where I might want to actually decrypt traffic deeper into my application stack at the moment, what we're looking for, there is an, a passive approach to decryption. And I'll talk about that a little bit more in my next slide.

So I got asked this question a few months ago, someone said, Hey, Paul, what do you think enterprise encryption should look like in the near future? Now the first answer's kind of obvious, but it still needs to be said. It has to be using best-in-class encryption. It has to be secure. We need to minimize the attack surface of our environment.

Another observation is it needs to be quantum safe. I think quantum computing is a lot closer than many of us realize. Now quantum computing is really good solving hard mathematical problems. And what was encryption rely on the fact that we can't solve half difficult mathematical problems. And so we have some really smart people who are figuring out which parts of our encryption algorithms are safe from a quantum computing attack.

And which ones are vulnerable with the idea that the next generation of encryption will only use quantum safe algorithm. Now, why am I mentioning this? Here's the problem we have. We have all these super smart people worrying about quantum safe encryption and we have a whole bunch of super smart people worrying about privacy on the public internet and quite rightly so what we're lacking is people thinking about how do I design in visibility into the infrastructure of my data centers and my public cloud.

So that I can have sufficient visibility of what's happening on the network for my cybersecurity needs, for my service assurance needs. And also I've got to be thinking about compliance. We need to be really deliberate about this. Now, if I do it right, I can also potentially achieve that passive decryption that we spoke about.

And I was involved in the definition of a standard called enterprise transport security, which introduces the notion of using all of the TLS 1.3 stack. But with an externally managed key, we've called it managed Diffie-Hellman. We presented this at a number of conferences, including ones organized by NIST.

And in the next couple of months, working with our partners at few weeks, we're actually going to be demonstrating how, if you use the, a managed architecture, you can produce a very secure framework that say leverage is all the best parts of TLS. 1.3, but retains these all important components of visibility.

Okay, so just a couple of slides left. No, in my view, cybersecurity product can operate in isolation. It's never an island, think back to those sort of four quadrants I showed earlier. So I'm just going to briefly mention. One of our products on this cyber investigator. This is an advanced NDR tool that's based on scalable DPI.

I just want to talk through two workflows. So the first one is a northbound workflow. We've got our packets, we're applying scalable, deep packet inspection, generating our metadata, and then we can perform analysis and we can identify events or even incidents and send that up to our SIM. But I also need to acknowledge that I've got all these other sources of data in my network that are being fed into the SIM, a large number of logs, but other sources of machine data, any of those can identify a potential incident.

But as long as I have some network context, that incident can be passed over to the NDR tool and then we can drill down and investigate it properly. Think about that pyramid. I could got the starting point at the top of that pyramid. I've got the context. Now I can drill down into the conversational information.

I can drill down into that transactional information. And if it's an event that occurred fairly recently, I can even go back and look at those. Those are all important packets. So let's just say, this is our approach to that. And we very much see it as a component of the overall ecosystem. So just to wrap up starting at the bottom of this, we're doing everything from the network because packets don't lie.

We're looking at applying scalable, deep packet inspection, and we need to be able to do it, but almost unlimited scale across all the different parts of the multi-cloud environment. And if I do that, I get my advanced early warning. I get. Visibility and understanding of how the attack surface is changing.

I've got understanding of my vulnerabilities and then on the right, I've got my ability to understand lateral movement with that contact tracing analogy through my metadata record. The ability to go back in time and look at a particular instance. That's my understanding of my threats. And as we spoke about at the beginning, if I have threats, acting on vulnerabilities, I have risk.

If I can get a good understanding of those vulnerabilities and our good understanding of those threats, actually in a position to reduce my risk. So just to finish up I started off talking about the big picture challenges. We spoke about some of the gaps that we certainly see in the existing tool sets.

We had a little bit of a, almost a retrospective and say what do I actually need to consider if I want to deliver the packet inspection at scale. We had those requirements because consideration spoke a bit about machine learning and. It doesn't have to be a part of NDR, but it is nevertheless, a very powerful source of analytics and you need to drive it with the best quality data.

I spoke a bit about encryption. I regard it as an entirely navigable problem. We have lots of customers who are handling it today, and there's plenty of products out there that can help you with it. But at the same time, we do need to be keeping an eye on the future and thinking about what our next generation architectures are going to look like.

And then we finished off by talking how all of this can come together to deliver the benefits of advanced network detection and response. So I'm going to stop sharing and having a look in the Q and a. So here's a question we can maybe just take a couple of questions. How can you collect packets in the public cloud if you don't have access to the network?

I touched on this earlier. Nearly all of the major cloud providers, AWS and Google cloud. And I think Oracle have just announced, actually provide a mechanism or means of receiving a copy or a mirror of the packets on the network. But even in the environments that don't, so for example, at the moment as you are, it doesn't have a direct packet mirror.

I can use techniques such as deploying myself as a virtualized network function. So effectively going in line, just like I would have fire wall or really any other network function. Basically provide a single hop router, take a copy of the packet. And in terms of route resiliency there's lots of ways of approaching that is active standby architectures, or you can use a load balance.

I, and I've got one other question here which I thought I'd take if deep packet inspection is so powerful, why not more security products using it? That's a great question. As you saw through this presentation, making DPI work in a practical and scalable way. It's a big engineering challenge.

And a lot of people have given up, but as I've mentioned, there are a small number of vendors such as ourselves. Who've been doing this for a long time and it persevered. And as a result, we are able to overcome these challenges, deliver the technology and continue to keep up with the ever increasing network speed. So those were the, I think the only questions I had, let me just check whether we had any more. It doesn't look like we had, so in that case I thank you very much for your attention and I'm going to close this session.

Visit ExtraHop.com for more information.

Defend the “When” with Network Intelligence

Perhaps the oldest cliché in cybersecurity is “it’s not if, but when you get compromised.” Cliché, yes, but also true. So, why then, are we all focused on defending the “if”? In this presentation, Bryan Lares will explore why the cybersecurity industry has it wrong and make the case for why we should defend the “when.”

Bryan will examine what opportunities exist for defenders when they focus their attention inside the perimeter and tap into the rich world of network intelligence. Bryan will discuss practical applications for packet-based insights and behavioral analysis: from fundamentals like asset inventory and other hygiene, to decryption of encrypted network protocols, and applying behavioral analysis to detect and continuously monitor attacker lateral movement and prevent breaches.

Bryan Lares is one of the newest members of the ExtraHop leadership team, serving as Vice President of Product. His team drives the global product strategy, roadmap and execution for ExtraHop's groundbreaking Network Detection & Response platform Reveal(x).

Bryan has over 20 years of experience in enterprise technology and has built and scaled multiple platforms in the endpoint, network and cloud security markets. He has held product and strategy leadership roles at hyper-scale technology companies like Microsoft and Dell Technologies and growth stage unicorns like IronNet and SparkCognition. In addition, Bryan was also a member of the Digital and Technology Strategy practice at global management consulting firm Strategy& (Booz & Company).

Bryan holds a Bachelor of Science from Purdue University, an MBA from the University of Texas and lives with his family in Austin.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

Short Clip Script:

What we're here talking about is what we like to call the intelligence gap. So that is the time in between intrusion and breach. But why is it difficult to monitor what's going on between intrusion and breach during that dwell time? One is 70% of devices are unmanaged two, at least 72% of attacks involve evasion and 67% of traffic as we know is encrypted. And there was some great discussion about that over the last hour or so. So it makes it really difficult to monitor what we call the mid game, which is between intrusion and breach.

Full Session Script:

It's good to be here in person. I don't know about most of you, but this is my first in-person conference in quite a while. It is fantastic to be here. For those of you that don't know me, my name's Bryan Lares. I've been in the Austin tech community for the last 15 plus years.

My journey is I've always been. In to data and in different analytics and different sides of it. Like a lot of you I started out building large scale data platforms and data warehouses, and started out as a data engineer. I progressed, I went to business school right down the street here at UT and then progressed to helping companies grow and differentiate through data. About 10 years ago I used what I learned to determine that cybersecurity is a fantastic industry. If you're trying to get the most out of data, it's a very data-driven industry and always say, signal noise is really important.

So I've spent the last 10 years of my life building analytics driven products in the cybersecurity space. I've been, as many of you that live here in Austin, I'm sure a similar path. I spent some time with the wonderful people at Dell technologies, and I guess my claim to fame there, if anyone works at Dell was I helped build the endpoint security business combination of acquisitions and some internal IP. And that was a fantastic experience. I was also co-founder of the cyber security business at spark cognition founded by Amir Hussein here in Austin. Amir's one of the preeminent entrepreneurs here in Austin and myself and a guy named Gerald Cappleman, who now is the head of engineering there, we built that product platform together. And now I am the head of product for ExtraHop networks. If you're not familiar with ExtraHop it's, first of all, a fantastic company and a fantastic set of people it was founded by two gentlemen Jessie and Raja, who are experts in deep packet inspection and network analysis, it was actually founded as a network performance management tool. And over the last five years, they have built out the market leading cloud enabled cloud scaled, a network detection and response platform.

So that is a little bit about me. I am the V P of product at ExtraHop, and I am really excited to be here to talk to you. I live right down the street. I love the Austin cyber community. So bill, thank you for putting this on and I hope this continues to grow. And thanks for giving back to the Austin cyber community.

All right. I'm going to talk to you about two things here today. One I am going to talk to you about some of the challenges with what happens in between intrusion and breach and why is that important? And then two, I'll teach the, how of, how do we monitor and gain visibility into into that activity?

Many cyber attackers really have the advantage. I think the overused trope is that an attacker has to be right once, a defender has to be a right every single time, but what's really the impact of that. These stats hit me the other day as we were putting together this presentation.

The average cybersecurity spend for large enterprise is between two and 5 million and then mid, 500 to 2 million and small is 500,000. In general, those are less spending than the average costs of a data breach at 4.2 million a year. And that price has gone up 10% in the last year.

So on average companies spend, or it costs you more to get breached than you spend on your entire cybersecurity budget in one year. But what's the chances of you have actually been breached? The latest surveys is that 41% of enterprises were breached in 2021. And the average dwell time of those breaches was about 21 days.

So the average dwell time is 21 days. The takeaway is that one you're spending less on cyber security than it's actually going to cost you to get breached. Two, there's at least a 41% chance that you're going to get breached this year.

So probably in the next two years, you will be breached. And number three, once you are breached, it's going to be about 21 days on average of dwell time, before that breach is identified. So why is it? It's the classic defender's dilemma. As I mentioned earlier, that the attackers have to be right once the defenders have to be right every single time across your entire attack surface.

That is a really difficult position to be in, and it's a difficult position to be in from both a prevention standpoint and detection standpoint. What we're here talking about is what we like to call the intelligence gap. So that is the time in between intrusion and breach. But why is it difficult to monitor what's going on between intrusion and breach during that dwell time? One is 70% of devices are unmanaged two, at least 72% of attacks involve evasion and 67% of traffic as we know is encrypted. And there was some great discussion about that over the last hour or so. So it makes it really difficult to monitor what we call the mid game, which is between intrusion and breach.

But, what I found was really interesting was the spending and if you look at the MITRE attack matrix, which all of us spend a lot of time in It really is very focused on post-compromised only 12% of the tactics and techniques in MITRE are pre intrusion.

And then the rest of it, 88% of those techniques are post-compromised. So we spend a lot of time thinking about how do we prevent intrusions, where a vast majority of the tactics and techniques used by attackers are post intrusion. The other interesting stat was that 75% of our cyber security investments are actually for intrusion prevention, which by the way is very important.

End point defense IDS, firewalls. That's all very important and it needs to be part of your security posture. I find it very interesting that only 25% of investments are for that post intrusion compromise. So we're spending, if you connect that to the previous slide we're spending 75% of our investment on only 12% of those MITRE tactics and techniques.

So the scale is just, it's pretty far off. And this really goes to the law of diminishing returns from an investment standpoint of how many, if you're, when you're thinking every day about how you're going to spend your next cybersecurity dollar, how are you going to allocate your resources?

You've got to think about. If I invested enough in that post-compromised detection capabilities versus all the dollars I'm putting in prevention, because it has to be a balanced between the two.

What are we talking about here? How do you defend the mid game as we're talking about? We believe one of the most important parts of defending the mid game is to gain visibility into that network traffic, especially the network traffic that's going east-west within your environment.

As we talked about earlier, the previous speaker talked about the importance of deep packet inspection. And I completely agree with that a hundred percent. One the network doesn't lie two it's really important to be able to do behavioral analytics on that network traffic, to be able to understand and gain visibility into that post compromise activity.

Once the intrusion has happened and once you're in post compromise the script can be flipped, you're actually talking about monitoring that network behavior and the attacker then has to be right multiple times versus the defender being right multiple times.

Cause as they go through that attack progression, there's a number of behaviors that can be monitored to look for. Different tactics and techniques across the enterprise to be able to determine that. So everything from recon to privilege escalation, to lateral movement, to data exfiltration. I was looking at this earlier and I found it really interesting.

I was looking through the latest Mandiant and Manny does a fantastic report every year on incidents. And they actually calculate how often tactics and techniques are used based on the MITRE framework in incidents. So it's a fantastic report that you guys should look through. When you get a chance.

But the data I found interesting is if you think about close post compromise, 16% of incidents. Attackers gain persistence through scheduled tasks and jobs, which that behavior can be monitored at the network level. 29% of incidents attackers use process injection to escalate privileges, and that can be monitored at the network level.

In 32% of incidents, attackers performed internal recon through systems files, directories, and that can be monitored post-compromise. I'll pick one more. 37% of incidents, attackers use application layer protocols, web, DNS, to communicate with C2. And in 14% of those incidents, the attackers use encrypted channels for that communication.

So what did we learn from that? We learned that almost every single compromise, is post compromise. There's a number of high value behaviors that can be monitored at the network level to detect breach and to detect intrusion. Two, we learned that a lot of that activity is encrypted, so decryption is important and it's really important to have the right telemetry . automate that monitoring for you.

Let's go with a example from an attack and go ahead and build a slide out. A lot of people are familiar with colonial pipeline. It was very important attack from the last several years. And it was attacked from dark side. So during the colonial pipeline attack after initial intrusion, the attacker, behavior was missed by their firewall, by their CASBY by their IDS, by their NOC, by their DLP, EDR, and SIM.

Once that initial intrusion happened, that wasn't prevented, it ended up costing $153 million for colonial. And as there was five days of downtime. There was a significant ransomware payment and it was a huge business impact to colonial. But what if colonial had that visibility into the post-compromise intrusion?

One of our customers, a major, a large leading home improvement company. I'll let you guess who that might be was also attacked by dark side, but once the intrusion happened, there was behaviors identified within the SMB protocol. To identify data staging activity, to identify a enumeration activity from bloodhound and to identify some of the ransomware activity.

And so that was identified after post intrusion, but before the breach started to occur. So remediation was able to take place. So in this particular example, there was no downtime. There was no ransomware and there was less than 20,000 files encrypted. So it's really important.

This kind of puts a point on how important it is to really invest in that mid game. In addition to the investment you're making on the prevention side.

What does it take to have differentiated prevention or differentiated visibility into the mid game? I'd say one thing I, and I love that we set this up and talked about it earlier is protocol. There's a wide range. If you look at all the network detection capabilities you have in your stack and in your environment, there's a wide range of different protocol fluency within those tools. And everything from minor fluency, which could be visibility into NetFlow activity to mid range, which is packet payload, matching and advanced fluency, which can be everything from decryption to TCP, real assembly, to full visibility and PCAP capture.

If you're trying to build that full level of visibility, it's not just one or two protocols. It's everything that includes Microsoft protocols, file transfer protocols, database protocols, decryption protocols, and L seven visibility protocols, everything from SMB to LDAP, to these different databases, to FTP.

These are the protocols that when you're thinking about lateral movement, when you're thinking about east-west activity, these are the protocols that these hackers are using. These are the protocols you have to monitor post-compromise. And by the way, you should be looking into your defenses as you're evaluating new vendors, as your going through and assessing your own set of defenses.

You've asked the tough questions, ask how many protocols are monitored as coming detectors. They have built for protocols. That's really the key to detecting that mid game. So why else is, from a detection standpoint, why is protocol fluency important? I'll give you three as a person that's dedicated his life to data science and the analytics.

I'll give you three reasons. One, it allows broader section coverage. If you look at if you look at the MITRE attack, it's your, the level of tactics and techniques post compromise that you can actually gain visibility into significantly increases as you increase the number of protocols that you're monitoring.

Two, it enables detections from a significantly higher fluency. So you're able to reduce false positives and increase your signal to noise ratio. So if you think about when you're building machine learning models for detection capabilities, Data is king and being able to build those features on a wide range of protocols and a wide variety of metadata and a wide variety of traffic data that increases the fidelity of your detectors.

So it's important to really ask the question of how deep that protocol coverage is and how much you're actually able to monitor both from a detection and the depth of investigation perspective.

Second thing that argue is really important from the visibility into the mid game is decryption. So we mentioned earlier 80 and 90% of network traffic is encrypted 70% of malware campaigns use encryption. If you're using a tool to try to monitor the mid game that does not have decryption capabilities, it's really when you're trying to monitor encrypted traffic, there's two approaches.

There's one, one is to decrypt that traffic and look at the decrypted traffic. And the second is encrypted traffic analysis. By the way, we use both approaches. But the problem with encrypted traffic analysis, which is a valuable tool, but the problem with it is it's it reminds me of what I do when I visit my in-laws or my are my wife's family in India.

So my wife personal story. My wife's family moves from India to the U S about 40 years ago. So I'm really blessed when I go to visit India, we Mumbai is where their family is. We don't stay in hotels. We get to stay with the family. And they're all from that area. So they speak Marathi was, which is the native dialect or native language from Marathi. And and obviously I don't even know my wife does. So when we're there, when we're in their homes, sharing their homes, sharing, their food, having great family dinners and catching up they're all speaking. Marathi that actually to me is similar to trying to analyze encrypted traffic without decryption, because I can, I've gotten good enough that I can catch things here and there , I know, enough words to be dangerous. I can tell based on body language, you know what they're trying to say, but you know what. If my wife's aunt, says something really mean about me, but smiles and nods, I'll be like great. This is awesome. So it's a very similar approach.

Just you can't get the same level of fidelity, the same level of visibility without truly decrypting. And that's why it's really important to do both.

The last point about is what it takes to really defend the mid game is high fidelity, advance detection, analytics. A couple of key pieces here. One is I would ask questions. If I was looking at a solution to put in my environment, I would ask where is that analysis being done?

Actually I'd ask two things first. I'd ask what types of detection capabilities do you have? If it's because there is a lot of different ways to do detection on network traffic. There's obviously classic IDs looking for known indicators. There's heuristics, there's statistical analysis. There's supervised machine learning.

There's unsupervised machine learning. There's deep learning. There's IOC based detection. So there's a lot. And by the way, there's no. Just like at the end point, just like another it's you, there is different solutions and different approaches for different types of attacks. So it's really important. That there's a wide variety of detection capabilities is the first thing I would ask.

And then when you're thinking about your behavioral analytics, that machine learning and the supervised unsupervised, deep learning. I would look at a couple of things. One I would look at is it cloud-based is a cloud scale because a lot of, it's fantastic to have it's to have machine learning, learning, and behavioral analytics, in your data center, on the sensor, close to the network, it's better than nothing.

But the problem with that is, is that the attack surface and attacks are constantly evolving. And if you have that machine learning in the cloud, Then you're constantly able to evolve that machine and tune that machine learning to the constantly shifting attack surface two you're also able to correlate attacks with other customers and other users.

At ExtraHop we have a data science platform that uses threat craft technology to basically correlate and look at attacks across our entire customer base. So thousands of customers we're looking at we're anonymously, looking at those attacks, quarterly correlating, those attacks, and constantly improving our machine learning models.

If you're not doing, if you're only using on-prem machine learning, then you're not getting the advantage of that type of those type of analytics. This is a print nightmare attack. And this is shows the benefits of encrypted traffic visibility. So imagine a successful attack of print nightmare compromise. A compromise endpoint is able to connect through SMB to the vulnerable server and gain communication back by the way, almost any network. Any network tool that has visibility into east west lateral movement is able to detect that, both the client and the server are going to require a mediation.

So at that point, you're just identifying work that needs to be done for the security team, because you've already been breached and the damage has already been done.

But here's where it gets tricky. What about a failed attack? And what about a failed attack where you don't have a wide set of visibility into into protocols? When that attack fails, You really get no visibility into the traffic flow and you're not aware that the end point is compromise basically the attackers free to traverse the network to try to attack other servers, but if you have advanced protocol fluancy, With decryption, you're able to actually identify that attack, whether it was successful or whether it failed. So you're reducing and you're able to stop that attack before actually becomes a breach. So your dwell time has reduced and you can initiate remediation before that server gets compromised and you can quarantine that end point to help accelerate your remediation or response. The two things I wanted to talk to you about at the beginning, one is this really important, not just to invest in prevention capabilities, but to post-compromised detection.

It's the other half of the coin. And by the way, it is the underinvested in part of that coin. Two, I would say post-compromised, not all post-compromised detection at the network level is created equal. It's really important to high of have high protocol fluency.

It's really important to have decryption capability and it's really important to have cloud scale detection and machine. Those are the keys to being able to thwart attacks, to reduce that time to dwell and to really take your defenses to the next level. Thank you very much.

Five Ways Attackers Leave Ransomware Vulnerable to Detection

Ransomware has evolved and time may not be on your side. Once inside, adversaries are acting faster, moving laterally to accumulate enough data to ensure that you will pay—but intruders following the ransomware playbook have needs that cause them to risk detection. Every time they take a step, you have an opportunity to regain the advantage—if you know what to look for.

Intrusions are a terrifying thing to consider, but they don’t spell doom: Visibility and response inside the perimeter are your best hope to prevent crippling damage from the ransomware menace. In this demo-filled session, we’ll show you how to spot indicators that leave modern ransomware exposed, giving you the opportunity to shut down the extortionists before they do real damage:

Enumerating targets in your environment Moving laterally toward your valuables Escalating domain privileges inside encrypted channels Phoning home over noisy DNS Staging data exfiltration for second-phase extortion

Highlight Clip Text

What's interesting. And there's not much as a defender you can do because everything has to be right . But once the intruder lands into the environment things start to change. Very little flexibility comes into play. And a poly-com speaker all of a sudden is doing FTP file transfers.

Huh? That's weird. Or your desktop. Yeah. It may do things like UPNP kind of things of scanning the environment, trying to figure out what it does when you plug it in. But all of a sudden it's got a PS exec to a domain controller. Huh? That's weird. I didn't predict it to do that.

Full Session Text

Don't we all love to talk about ransomware and what's the conversation we have as security people, if they would just stop clicking that bait, if we could just get them to stop clicking that bait. And then it's if we would just stop outsourcing so much, we could get control of it. And we have all of these ideas of ransomware.

But a lot of it feels pretty helpless. It feels like there's not a whole bunch that we could do about it is the problem. What I want to talk about is ransomware, and how do we get here. , not so much from the attack perspective is, but it's been around for such a long time.

And yet we can't seem to slow it down. We can't seem to lower its impact in some way. Let's go step back a little bit. Let's go back in time to where it all started. And I put 2016 year because these are pivotal change periods that sort of happened inside a ransomware.

In previous to 2016, most of ransomware wasn't even a nuisance yet. Because it was primarily a consumer problem. It was a very automated Internet spraying strategies. And, I love the story of that. Chicago, Russian grandma got her grandchildren's pictures were encrypted on her computer and the Russian guys were over there telling her how to go get a bit coin.

To pay them about 500 bucks. And it's interesting how the grandson is tracking Bitcoin value too. And the conventional wisdom at that time, that everything was, Hey, you got to have backup. You got to have backup and you should put some EDR, let's go spend a little bit money past the antivirus, put EDR onto that end system.

That was the conventional wisdom. Then we moved to 2018 in ransomware pivoted again. It became a business problem. They wanted a bigger pay day? This 500 bucks stuff. And it's such a pain in the butt to try to teach the consumers how to go get a Bitcoin that they decide I'm going to go after businesses.

Because I can get a better pay day. And we see that the average ransomware payments starts to increase the $6,700 at that particular time. And the attacks become spirit. Conventional wisdom? When I met you convention with the analyst, all of the media and all your buddies are S saying these kinds of things, and it was, Hey, we've got to get MFA in place.

Let's get MFA in place. Let's do a whole bunch of Phishing training, we've got to get these people to stop clicking this bait. We come back to 2021? It ransomware pivots again. The attack gets much bigger? The average ransomware payout. This isn't the demand. This is the payout.

After it's been negotiated by, would it feel go our our consultant from the accounting firm that will help you negotiate that thing down? Becomes $170,000. The thing that's changed that you need to recognize is that what happened was when they want this kind of a payment, they've got to create a lot more damage.

The strategy change from an automated Internet attack into a land and pivot, it's got to get into the infrastructure in order to be able to get you to pay enough, because it has to be a denial of service at that particular point. Al. What's the conventional wisdom that you're reading about zero trust.

Let's go get zero trust. We've got to get that perimeter much, much more secure? Let's more backups? Cyber insurance. This is the conventional wisdom. It's wise things that we all have to do, but the point here is that it hasn't slowed down. We're, perhaps we're looking, we're focusing a little bit in the wrong places and you can see that this is true because.

I think about those conversations that we have, I guess over the virtual cooler, because we don't actually go into the water cooler and have conversations anymore, but it's what do people say? There's this myth about ransomware is, as I said, if I could just get them to stop clicking that bait.

Myth number one is. You got to stop it before it starts. Let's look at a couple of interesting statistics. There's this company. No. Before that produces this annual report there, if they're a Phishing training company really interesting report, I just loved their report.

6.6 million users, 23,000 companies, Phishing training for one solid year, 4.8% of the people still click. Go figure. But they talk about that as a success because it used to be 38% of people would click. But after a full year, they got it down to 4.8%. That's an important number to consider.

. The other thing. All , . Phishing's the problem. . But what about vulnerabilities? What about all those other attack strategies that are out there about how do I stop them from the beginning? Pen testing, we all, because if we're, if we have compliance, we bring pen testers in once or twice a year.

This company positive technologies, who's a well-kn pen testing company. 93% of every engagement ends in an interim. Everyone. Because if they can't get in, you're not going to hire them back anyway. And what's interesting about that also is that no social engineering, and pentesters have to follow rules. They got a time constraint. Don't touch this, don't touch this. Cause I k you'll break that. There's all those. I think, what does that mean? What does that mean for us? Our over-focus on the one bookend, the ransomware, which is the initial access side of it is to a certain degree feudal.

Is it's we all kinda k this. We have a language in our industry about perimeters walls. I think about the movie Planet Z years ago when that movie that came out and who knew that zombies could climb on top of each other and go over the the Israeli wall. But anyway but that's one of the things , the.

Portion of it when we, when I have conversations with customers and what not. And you're probably thinking this too as well. You k what? I got backups, and I got cyber insurance. I'm resilient because I have cyber insurance in place. But the thing that we need to think about is we talked about that the ransomware demand is 107 or the ransomware payment is 170,000.

But the damage is 1.8 million on average? It's after the fact? Your mediation is temporary. The backup is critical. No question about the cyber insurance is critical from a risk calculation perspective was, but it's a temporary or it's a partial relief.

And that's where a lot of the business people missed. Is we got backups that just gives you an opportunity to negotiate the ransom portion of it. But you're just after the fact waiting for the ramifications of ransomware. The thing that I want you to think about it, This is what's happened to ransomware.

Ransomware has evolved. It is a land and pivot movement today? The one that we care about as businesses, because we can, all of us can within one to five assets being compromised over the Internet. And we as ITP, we were good at shaking fingers at people. ? Shame on you. I told you don't click on that.

Stop going to those websites. We love blaming everybody else. And but the thing that I want you to think about is there's something different about ransomware. We have to think about what is the motivation and what is it that they're trying to accomplish. . We start on the beginning of the initial.

. Many different tactics to be able to use inside Azure. And you have to be ready for these. However, the thing that I think from looking at those statistics that we looked at is that where your battle where's that where you skirmish did you put all your money all your time into that portion of it that you are inevitably doomed to fail?

And I'm going to go a little bit more into this, and I'm going to show you some math. We're going to do some math, and do some statistical analysis. And then we talked about on the side, the other bookend? Let me get the backups in place. Let me get the ransomware protection and all that.

But this is the thing is the middle is we're missing it? This is where most of the, this is where the damage actually happened. This is the land and pivot. They've got to get to a critical, massive damage that is going to make you. It's all game theory . The thing that's interesting about ransomware is they don't really care what it is, .

It's the quality isn't so important. Yeah. There's the double ransom. They want to steal it. They want to sell it and all that stuff, but the primary is denial of service. It's quantity versus quality is the motivation behind ransomware. And when you think about what that means, Also is that they have a time constraint?

Ransomware on average is less than well, it's less than four days . Of a dwell time. And that means they are in a rush. Because they don't want to get discovered and get, get stopped, but mostly they just want to get to the end game, which is pay me. We, what is that in ransomware, you can treat it as a three-part playbook. And we k this is true because we've seen the, like the Conti ransomware playbooks, we've seen all of these postmortems that we see the behavior inside of it. And you see a very repeated pattern associated with ransomware, which is pretty similar to most land and pivot attacks, except it does it much more noisy.

And it does it super fast. What's happening in the middle that we call the mid game, the ransom we're mid game, just to give it a sort of a label is there's these five things that are happening inside there. And you just, you sorta k this because an intruder lands blind into the environment.

What they got to figure it, they got to do intruder stuff. I gotta do intruder stuff. I've got to figure out where the heck am I what's around me. And I'm going to land on something easy. But I got to move towards something valuable, like we're showing over on this side.

That's what I've got to get to in order to make you pay. We think about these five kinds of things that are happening inside of this this type of a workload. . All . Let's go take a look at a couple of case studies? That is a true is this thing true.

Let's go first, take a look at. , content is good because we've read through their playbooks. And they've done so much destruction inside of the environments that we k these things, but this is a post-mortem from Diefer reports. This was an actual incident responders report of all the things that, the content that, this particular one on.

It was a phishing attack. G no surprise there. 4.8% people are going to click. They're spraying phishing attacks against here. And then on day one, we see them pull out all these tools, because why they got to figure out what the heck is wrong. Th these are just natural things that they must do.

Here's all these scanning discovery, enumeration tools, the things that are naturally built into windows and to, into all of the ecosystem. And the other thing you also see, that's very common with ransomware because they're in a wickedly fast environment. They go after active directory because active directory commonly used and all of our infrastructures and active directory ask you to question. Sure. I'll tell you where the other domain controllers are. You betcha. And it just, they both, I don't k. You ever run bloodhound non-SEC on your network or something like that. I just look at that.

God, that's amazing. That's amazing. This thing is going to tell me all this stuff? Here's all these types of things that are running inside here. Then they too, we've seen that it's starting to move laterally and it's using RDP and it's figured out what other things that it can use inside of .

Day three to me is the most fascinating. . No action. You k what the attackers, they take their kids to the playground. They got to go take them out to the park and run with the dogs and all that stuff. And maybe they have meetings meeting day or something like that. And then we see all these actions until we get today five we're encryption, it started.

And then the delivery of the note starts. , the point is that these skill sets on the attacker side have specialists. You were not attacking one guy. There's all these teams that are just great at phishing they're they're code writers looking for exploits, they k how to do brute force, then there's. The specialists that k how to move laterally through environments, the whole tool chains there. When we think about hobo strike, they use the Coldwell strike. If things like that? I These powerful tools that are constantly being used.

And do we want, do you think that the, these guys are using the same people to do the extortion cycle? That's a specialization, that's game theory. They're doing all this specialization, but on our side, we think. I've been an IT guy I'll defend all of it. And of course, what really happens.

We get bumped out. As soon as the note gets delivered legal financial consultants start jumping in per incident responders, parachute into and figure out root cause things like that. . All . There's one example. Just keep that in mind. Here's another. , this is against Darktrace, I'm sorry, against a dark side.

And remember dark side was Conti was colonial and just a ton of other attacks that they've done inside of it. And this particular one was one of our customers. Here, this is a north American retail company. And what happens is that this, they get this initial access from exploiting some virtual desktop, and then , from inside the network, we're able to start firing alerts? That's a, Hey, there's this unusual interaction happening, or these things had never done this before, and it's starting to read data and transfer data and it just looks weird. I think, because this is one of the things that machine learning can tell you that your policies let me come back to that. But anyway, here's some, we can see that some some SMB file reads are being done, but the point here is the prediction portion of it is that it didn't do it before. Why is it doing it ? Why is it using these particular sets of tools? Here's an example of where we're able to see this action of the mid game coming into play, and we can stop that.

When we go back to that three part portion of it is the first part. Is mostly security. Hygiene is probably your best hope. And then at the end, did you just is the recovery portion of it, but the goal is stop it before the encryption starts. And what I'm trying to get at here is that we need to think about how do we do this.

I thought I'd try to visualize what does this look like from an attacker's perspective? . The there's a shin Kaizen in Japan. I don't k if anybody's ever written on these things there. They're fantastic. They're always on time. You just walk into it exactly one time and they take off. But this is what it looks like from an attacker's perspective is as a defendant, I have to have everything ready before I let the train take off. But as an attacker, I can, I, I can find any opening inside of it to be able to penetrate the particular perimeter defenses, and then on the what's.

What's interesting. And there's not much as a defender you can do because everything has to be right . But once the intruder lands into the environment things start to change. Very little flexibility comes into play. And a poly-com speaker. All of a sudden is doing FTP file transfers.

Huh? That's weird. Or your desktop. Yeah. It may do things like UPNP kind of things of scanning the environment, trying to figure out what it does when you plug it in. But all of a sudden it's got a PS exec to a domain controller. Huh? That's weird. I didn't predict it to do that.

There's all these things that happen inside of here that makes. Different the game changes. What we call that in the industry is one, we call it the defender's dilemma. Has anybody heard that term before? No, this is a, this is, it's not urban legend. It's true. Defender's dilemma, Rand corporation wrote a book about it back in 2015, I think is when it sorta took on.

And what it says is that. Out on the perimeter, you are at a disadvantage. The attacker has the upper hand. They control the cadence. They control where they're going to attack how they're going to attack. How frequently all of those things they're under control. You have to have everything ready before the attack?

You don't get to adjust during your time? It all has to be ready, but the narrative that's less understood in the industry is that there's another. The other dilemma is the intruders dilemma. And if you think about yourself as an attacker, or if you've ever done any hacking or whatnot, I land it's dark.

I got to start shining flashlights all over the place to try to figure out where the heck I am. And I'm always on the wrong place. Because hopefully you're not putting your domain controller out on the Internet. And or whatever, that would be those valuable resource. Are not directly accessible.

There's this whole dilemma that comes into play. And in reality, isn't and Rob Joyce, who I don't k what his job in the holidays, but I think he's doing something he's some consulting work, but Rob Joyce used to be in charge of the NSA and he talks about that. The worst nightmare for attackers is people that are watching the inside of it because these, all these trip points come into place.

I think of it visually like gauntlets. All . The question becomes, can I prove it . Let's prove it. All . I built this little calculator. All . The first one, the top half of the calculator is about the defenders. . What is the probability? Because in the world of intrusions, it's really about the number of opportunities that you have.

We remember, we talked about the phishing attack? We said 4.8%? 4.8% of the people will click. The question becomes how many phishing emails do I need to send you before I get one of your people to click? If I only send 100 emails, that's it just 100. I have a 99 point 26% chance that you're going to click.

, let's go make this a thousand? Because surely your business is more than a thousand? It's the 9, 9 9 9 9 9 9 9. It's rounding up to a hundred thousand? The question becomes, is this what you're going to battle? Or is this where you do security hygiene and skirmish? The next one is the intruders dilemma is what is the probability that you're going to be able to spot the intruder, moving laterally through your entire. If you can't see it, the probability is doughnuts zero, zero. You are not going to catch it. The question becomes how many?

In our example, with the content thing, this is, there was actually 23. Individual non-repeating things that were in that five day period. But the reality is they use those tools over and over. You don't do 85 once you do 85 commands, tons and tons of times. But let's again, use 23%.

The question becomes what is the probability that you can catch it? Let's just put an arbitrary number and say that let's say that you have tooling in place. This kind of behavior. All . Let's put a, let's put 20, 20% in here. Hi. Oops. Why is this not working?

What's going on here? . . That's too many. Yeah. 2%. If we just plug in a number like that, I think that I do some sort of a weighted analysis to figure out that, Hey, I actually have some good tooling in place and I can catch these kinds of things. I put a number into it. It says that, Hey, I've got a really high probability. I can do this.

Catch it. The point being is that let's fight where we have a better chance. , however boy, you say Don, that's a bunch of bull. You're you want me to feel comfortable with intruders in the mist? Hey, you k what? They're already there. Tough . If they're already there.

What do we do about it? How do you do this? Gartner coined this term, that's called a SOC visibility triad? This is detection and response. This is the core detection and response is, they talk about the need for. There's three sources of data in order to be able to have an effective response system.

And that has to do with one that we all probably have in place, which is EDR or something like that, where we need end point data. We need end point data. And then we use that. The other one is SIM . We get logged data. That's. The high-level telemetry information about sort of everything that's happening inside the environment.

And then the third one, which is less used, which is the network. . You need the network data, there's things inside of it. If you combine these, you end up with the sock visibility triad that gives you this opportunity. All . Here's the bad news. . Let's think about what is it intended for?

And what is the data it's looking at? . EDR is, looks at host files? It's looking at registry, it's looking at processes. It's goal is don't let this particular. Get compromised. SIM logs. What is it really for how many you don't sit is I don't k. What's the word? SIM is like SIM as like a crappy boss.

You feed them all the information. You, it takes all the credit. I found that. It was something else that did all the work, it collects, it's the real thing about SIM is it's just reporting. And it's compliance, that's really what it is, but it's the aggregation of all the other tools that give it the intelligence or the action for it to be able to do it.

And then NDR is a network portion of it. What are the objectives here ? The, my proposition here is. And the mid game as we define those five particular things that happened inside the mid game of target enumeration, lateral movement, command and control, data staging I missed one domain, escalation.

The, that those are the things that are happening on the network? It's that lateral stuff that's happening on the network. And these things are traditional security controls. And sin just can't catch it. . And this is the problem. This is why this is why we're in the state that we're in is we're looking at it in the wrong way.

. Let me give you, try to visualize what I mean by this. .

All if I am in the detection and response, I need to see everything . We call visibility. I sort of hate that word visibility, cause it doesn't mean anything to a certain degree, but whatever let's use it because but it's just visibility. What? I need to see everything on the inside, everything on the outside.

And if I can get all of it, if I can really get that. I'm in great shape. I should be able to see this little bitty attacker over here, over here. What happens with network data is you get to see everything on the outside, but you can't see anything on the inside. We get that .

EDR. It's flips it around the complete opposite direction. You can see everything on the inside, but you can't see anything on the outside. Here's the dirty little secret about EDR. . Is this what everybody thinks? It looks like, but it actually looks like this? Is there so many blind spots?

I ask people this, I throw this many CSOs and it later they look at me say, that's wrong. You don't, we have 90% coverage on our end points. And then I, and then we talk about it and I said, but what about your third party? What about all that IOT? What about the OT? What about the server?

What about the Linux machines? What about this one? Oh yeah, because I, for my particular responsibility of all of these endpoints, Yeah, but this is really what it looks like. And what we see from looking at real networks, it's about 60% of the end points don't have anything on them. .

And but that's where the threat comes in. This is really what it actually does look like as far as from there's too many blind spots associated with , the last one I want to show is SIM this is what SIM looks like. It's hazy, it's logs. And there's this it's only noisy.

There's very little actionable stuff inside of it. I, anyway, I just, something for you to consider from this perspective wise is that this is what would it looks like? We go back down to the, we'll go back down to the those five things we talked about. Attackers have to land, they got to start discovering, figure out what are they enumerating for?

What are they going to actually exploit? How are they going to get there? Bloodhound. I just, it just baffles me. And then they've got to start moving laterally. And what does that mean? Moving laterally? That's using all of the live off the land kind of tooling that's out there. And you're you, can't very well blacklist, any of the Microsoft protocols that are inside of your environment or anything like that.

You've gotta be able to figure out which is. Which is not real . Which means you have to start inspecting those kinds of things inside of there. Then, as I said, is domain escalation. I, And I specifically say domain escalation rather than saying privilege escalation is because we all have windows in our darn infrastructure.

And it just so easy or there's so much tooling out there to be able to exploit. Domain controllers. That's the other portion, but then of course, command and control because they got to pull down new tools, talk, get orders on what are they going to do before they move it?

And then the third port, the last one is data staging? Is it, you have to stage the data before you. And you're always trying to pull it. They're always trying to pull it out. There's this, all of these landed areas, there's all these clues. And I don't k if it's breadcrumbs, I would say they're rocks.

It's easy, giant Boulder sitting out there punching you in the face. This thing is really noisy. It's running 80 fine. It's like, why? Oh, I don't k. Whatever. And then this is, these are the things that are happening on network. Things to consider that's not to say that these are not the tools because they absolutely have a critical portion to play inside of your infrastructure.

But when you keep on over reliant, like we've talked about before, over-reliance on perimeter defenses, over-reliance on backups? Is here is your over-reliance on your traditional security controls, trying to solve a problem that hasn't even slowed down a little bit. It's only gotten worse. It hasn't gotten it.

And people are saying I've got an EDR program. I'm good. Or the SIM. We're looking at the wrong kinds of things that are what the attackers are doing during these particular cycles. Extra hop. We obviously do network. We're a little biased from that perspective wise, but this is the truth?

This it's complicated network is always been a difficult and complicated area for security controls. But what we see is because of machine learning, cloud scale computing and things like that. It makes it very possible? It's you have a difficult time writing policies for things that you don't own.

And the thing that machine learning gives you is this ability to. That's weird. I didn't expect it to do that. It never did that before. And those are those telltale signs that there is intruder and miss. And anyway, that's that's the material. We, there's a online demo.

You can try it yourself, but these are things to consider as you as you think about what your next set of strategies going to be for being able to mitigate against. Ransomware. Thank you.

Thank you, Don. How many questions do we have here? Yes, sir.

In the yard. the, the thing with ransomware has gotten so sophisticated that it seeps in one of the first things it does is it goes, and it messes up your backups. You think you're protected and you go and you try to use the backups and the backup. Worthless. They don't really do anything for you.

I'll repeat it real quick. NDR. How does it apply to the fact that ransomware gets in completely destroys and messes up your backups? When you think you're going to go and be able to restore, you're not able to do that. And I just wondered, how does you k, what you offer help with that particular problem or does it.

Yeah we, those backups become so critical, but as an attacker, trying to get to a backup, I hope your backups are not sitting on the Internet, is they've got across quite, some perimeter domains and whatnot. What we, the thing about. Then what I said about Rob Joyce and the network the portion of it that's really fascinating about the network is especially for security people is you don't have to own that database.

But if you're, if your network monitoring system is to understand database also the language inside the database to see what the unusual behavior that's happening. And why this particular thing had never made these kinds of queries. There wasn't another dark side, one way. One of our one of our, one of our customers in line with what you're saying was that they saw this end point, which was a laptop actually was pulling down.

And it wasn't a lot that it was, I think it wasn't IP phone, but it was some other IOT device. And, but that's what you're going to land on. Are you going to land on something easy ? You've got to move towards that database. That's the opportunity for you to be able to catch it. .

Is because intruder things tend to do we kinda like to think that we'll, it gets it's inside. The noise is hard for me to distinguish the good from the bad actions, but you there's actually. Intruders have to do intruder things because otherwise they wouldn't be an intruder. When they're moving, the opportunity is what I said was the data staging is, as you can see these things going after data, and you can see the entropy patterns associated with how things normally read and write to data versus how, when you're encrypted.

The spot, the sporadicness of how that repetition happens inside of it. There's all these telltale signs of it. . And that's the thing that I want you to think about is. Intruders have to do intruder things and that's your opportunity to be able to catch it. If you can spot it and then through what we have through automated responses, what we call break glass responses and things like that, where we'll talk to knack or to an EDR endpoint and say, dear, there's something weird going here, raise a flag.

Isolated? That's the actions you have to take, but then there's automated ways to be able to do that, which I would consider more like break class kind of strategies rather than thinking automated response, because nobody really does automated responses.

Then we have more questions who had a question here.

You did, you're going to make a question up. We'll work. We can't wait to hear her.

I have noticed a technology and techniques race in the development of ransomware techniques from 2016 to the present. What in the world could be next? If they already have gained this sophistication of being able to get in.

Get people to click lateral movement and extortion. Where are the push points? Is it going to be in the getting in? Is it going to be in the lateral movement or is it going to be extortion or is there something that they, that nobody's thought of that they're going to pursue? I just wonder what's next in this world.

There's as, as far as sophistication, I would what's interesting about this restaurant, where is such a big problem, it's crippling for us as a business. It's a, it's an existential threat? That's the real word that's commonly used. It's existential?

Because it puts you out of business. And it has put many, some businesses out of business and the thing is. W we consider ransomware and advanced threat today because of the fact that it's moved into a land and pivot strategy, but it's probably the least sophisticated of advanced threats?

The ones that are really killers are supplied. Because it breaks all your trust models. It's zero is a zero days because I k what's going to happen. And the third one is insiders? That's those are the killer. When in reality, ransomware is actually one of the easier ones to catch and we see a lot of our customers catching them because the motivation.

Hammers, sledgehammer motivations. They're moving so fast that it's, they're super noisy. And the problem is we as defenders, we're not accepting that we are ignoring it and saying I got my perimeter. I got MFA in place, and we're not filling. The holes that exist that are the opportunity for us to catch it .

It is. Are there some, we, these other things that we talked about at our more sophisticated advanced threats as they adopt these things? Start moving slower, start office gating, turning off EDR, which we see them doing things like this today. Th it, they become more and more difficult just as those three that I mentioned before.

Supply chain. Zero day insiders? That's if you can stop one of those, you can stop ransomware. There's no question about that.

Follow up questions, anyone? Yes. A up question.

I see then an opportunity. And see if you agree that in any new field you're going to have criminals, so different types break into the field. And what we have is that we have a new field. Stealing liquor, drugs, robbing banks, and other things. They had to start with small safes that locks the vaults and then cards, the taction and the whole bit.

I see it moving forward. What that means is that the ransom or gangs youth are merging within the sophisticated, slower and filter. Activities. So would you see a consolidation of that and maybe moving the time horizon out? if we do succeed in keeping the sledge hammers out, do you see the time horizon increasing so that they will be harder to detect and they'll be able to be more patient in in succeeding?

I think that's a good, that's a good assessment. And we have to keep on upping our game because it is five versus five continuously. We, when we think about the the SolarWinds breach how did we figure it out? We had to see that movement on the. There was no other way.

And to a certain degree on a supply chain, what other options do you have? Because they just broke your whole trust model. Is, I didn't k. I had no idea. I've got a thing. Yeah, we need to be much more sophisticated about it. And this is, hopefully you'll take away what I said about the dilemmas.

Take that, give that a little bit of thought? Is that, are you thinking about it? Why are you battling. Math is math? That's all there is to it. Anyway, thank you very much.

Let's hear it for Don shin from ExtraHop corporation. Thank you, Don. Excellent brilliant presentation.

Your insight from many years of watching all of this and then putting it into this type of presentation is excellent. Don't you think? Yeah. Yeah. Thank you so much. Appreciate it. Don Shen from extra hop corporation.

Nick Leghorn, Director of Application Security, The New York Times

Nick's Session 1: Writing Policies That Aren’t Miserable for Everyone Involved

Every organization has policies, and for good reason. Consistent decision making, standardized approaches, and clearly defined roles and responsibilities makes sure that everyone understands how to interact with different teams to get things done. Policies are good but almost everything about the way we commonly do them is an awful experience, from the team that has to draft this monstrosity of legalese, to the long and drawn out approval process where arguments about commas last long into the night, to the poor engineers who need to bring it to an oracle to decipher it so they can do their jobs.

Rather than sticking with the status quo, Indeed is trying a new approach to policy documentation that aims to reduce all of these pain points and make policies a legitimately useful document for everyone in the business, from upper management to legal and even engineering teams, all while maintaining compliance with applicable laws and regulatory frameworks.

Nick Leghorn runs the Security Governance, Risk Management, and Compliance team within Indeed. In this talk he will go over the high flying inspiration for their approach to policy documentation, walk through how to craft a BYOD policy in five minutes flat, and talk about some success stories where their document format helped increase productivity and reduce demand on the compliance team.

Nick's Session 2: Building a Security Team that Never Says “No”

Hear Nick and 30+ Cyber Leaders May 10-11, 2022 Online or In-Person

Nick Leghorn is the Director of Application Security at the New York Times. After graduating from Penn State University with a degree in Security and Risk Analysis, his first job was working for the U.S. Department of Homeland Security quantifying terrorism risks and identifying mitigations to provide the best risk reduction for each dollar spent.

Nick has spent his career working for a number of large companies, including Rackspace Hosting, Shoretel, Mitel, and Indeed, improving the security of both the infrastructure itself as well as the processes within the company.

Check out more on Disaster.Stream Podcast!

Session by Charlene Deaver-Vazquez of FISMACS.com

Title: New Mathematical Models for Forecasting Cyber Attacks

Probability theory gives us the ability to quantify risk and forecast events in a predictable way. These methods have been used in epidemiology, seismology, finance, and even space and nuclear safety analysis. Today a new algorithm gives us the ability to model the phenomenon of one event increasing the likelihood of more such events, referred to as "self-excitement". We witness this when a restaurant seats the first customers in the front window because they intuitively know it increases the likelihood of more customers. We also see it when a tweet goes viral. Applying these models to the cyber realm gives us an unprecedented opportunity to peer into the future with the goals of improving our current decisions and developing mitigations before the risk is upon us.

Charlene Deaver-Vazquez

Charlene has worked in the IT field for over 30 years, and as a Subject Matter Expert for over 12. She created the Probabilistic Risk Model for Cyber Framework (P-RMOD4Cyber). She authored the book "Ensure Your Business Success with Risk-Informed Decisions: the easy way to quantify risk". For the last several years she has been providing analytical services for the Nuclear Regulatory Commission.

Austin Cyber Show Session: The Art of Cyberwarfare Author Jon DiMaggio

Session Title: The Art of Cyberwarfare Insights from the Author Jon DiMaggio

Join us as Jon share insights on understanding and analyzing cyber attacks by advanced attackers, such as nation states.

Cyber attacks are no longer the domain of petty criminals. Today, companies find themselves targeted by sophisticated nation state attackers armed with the resources to craft scarily effective campaigns. This book is a detailed guide to understanding the major players in these cyber wars, the techniques they use, and the process of analyzing their advanced attacks. Whether you’re an individual researcher or part of a team within a Security Operations Center (SoC), you’ll learn to approach, track, and attribute attacks to these advanced actors.

The first part of the book is an overview of actual cyber attacks conducted by nation-state actors and other advanced organizations. It explores the geopolitical context in which the attacks took place, the patterns found in the attackers’ techniques, and the supporting evidence analysts used to attribute such attacks. Dive into the mechanisms of:

North Korea’s series of cyber attacks against financial institutions, which resulted in billions of dollars stolen The world of targeted ransomware attacks, which have leveraged nation state tactics to cripple entire corporate enterprises with ransomware Recent cyber attacks aimed at disrupting or influencing national elections globally

The Art of Cyberwarfare: An Investigator's Guide to Espionage, Ransomware, and Organized Cybercrime

The book’s second part walks through how defenders can track and attribute future attacks. You’ll be provided with the tools, methods, and analytical guidance required to dissect and research each stage of an attack campaign. Here, Jon DiMaggio demonstrates some of the real techniques he has employed to uncover crucial information about the 2021 Colonial Pipeline attacks, among many other advanced threats. He now offers his experience to train the next generation of expert analysts.

About the Author

Jon DiMaggio is the chief security strategist at Analyst1 and has over 15 years of experience hunting, researching, and writing about advanced cyber threats. As a specialist in enterprise ransomware attacks and nation-state intrusions, including the world’s first ransomware cartel and the infamous Black Vine cyberespionage group, he has exposed the criminal organizations behind major ransomware attacks, aided law enforcement agencies in federal indictments of nation-state attacks, and discussed his work with The New York Times, Bloomberg, Fox, CNN, Reuters, and Wired. You can find Jon speaking about his research at conferences such as RSA and Blackhat.

Austin Cyber Show Session by Gary Hayslip Author CISO Executive Primer

Title: Cliff notes from a CISO Author's Perspective

Join us as Gary shares insights from a CISO Perspective

The CISO Desk Reference Guide Executive Primer is written primarily for the CISO's colleagues. The primary perspective of this book is one of expectation. What are the expectations the CEO should have for their CISO? What support should the CFO expect to provide the organization's CISO in support of their mission? What are the expectations the CISO will place on their colleagues to help make the organization more resilient? What kind of support should a CISO expect from the board? As important, what expectations should the entire leadership team, including the board, place on the CISO in terms of communications, teaching, expertise, risk assessment, metrics, meeting regulatory requirements, and preparing the organization to detect, respond to, and recover from cyber incidents?

Gary is an experienced Global CISO with repeated success delivering innovative security programs to safeguard billion-dollar enterprises at every touchpoint. Intensely focused on driving continuous improvement that maximizes security program efficiency and minimizes costs. Demonstrated ability to collaborate at all levels to champion new ideas, gain buy-in, and build consensus. I bring this wealth of information technology, security leadership, and risk management experience to my role as the CISO, for SoftBank Investment Advisers – The Vision Fund & Vision Fund II and SoftBank Group International – The LATAM Fund, The Opportunity Fund, and The Tech Fund. My previous executive roles include multiple CISO, CIO, Deputy Director of IT, and Chief Privacy Officer roles for the U.S. Navy (Active Duty), the U.S. Navy (Federal Government employee), the City of San Diego California, and Webroot Software.

Proven cybersecurity professional demonstrating an exceptional record of communication and public speaking skills; I am adept at presenting multifaceted security & risk concepts to audiences of varying knowledge levels. Gary established a reputation as a highly-skilled communicator, author, and keynote speaker. He co-authored the CISO Desk Reference Guide: A Practical Guide for CISOs – Volumes 1 & 2 series and I authored The Essential Guide to Cybersecurity for SMBs and Developing your Cybersecurity Career Path. All of these books are considered among the leading resources for enabling CISOs to expand their leadership and business expertise. Gary sits on four security & technology advisory boards and write for Forbes Technology Council. He is an active member of the cyber community with memberships in the professional organizations ISC2, ISSA, ISACA, and Infragard. Gary holds multiple professional certifications, including CISSP, CISA, and CRISC, and have earned a BS in information systems management from the University of Maryland University College and an MBA from San Diego State University.

Colonel (COL) David Wills, U.S. Army, retired 2021,as the Deputy Director for Command, Control, Communications, and Computers (C4) Systems, U.S. Strategic Command (USSTRATCOM). As the Deputy Director, C4 Systems, COL Wills is responsible for providing and assuring global-integrated C4 systems capabilities for the full spectrum of assigned missions. The USSTRATCOM mission requires a range of C4 systems to support national-level decision making and to conduct nuclear, global strike, space, and cyberspace operations.

Past assignments have taken COL Wills and his family across the globe during both peace and war. He is a 1991 graduate of the United States Military Academy (USMA) at West Point. He served in several operational and joint assignments. As a company grade officer, he served as HAWK Fire Direction Center, Stinger Man-Portable Air Defense System (MANPADS), Chaparral, Avenger Platoon Leader and Battalion Adjutant (S-1) with the 3rd Battalion 1st Air Defense Artillery (ADA) Regiment and 2nd Battalion 2nd ADA Regiment at Fort Hood, Texas. He served as the Assistant Battalion Operations Officer (S-3) with 4th Battalion 7th ADA Regiment, Fort Lewis, Washington. He also served as the Assistant S-3 and Delta Battery Commander 2d Battalion, 1st ADA Regiment Patriot at Fort Bliss, Texas. Following Battery Command, he transitioned from Air Defense Artillery to Signal as a Functional Area (FA) 53 Systems Automation Officer and served as Battle Command Battle Laboratories Automation Chief, Fort Leavenworth, Kansas. As a field grade officer, he has served as the Deputy to the V Corps Automation Management Officer, Heidelberg, Germany. COL Wills also served in three Joint positions: Chief Network Engineer and Branch Chief of the Network Engineering Branch, J6 Current Operations and Plans Division, United States Central Command (USCENTCOM); Enterprise Services Branch Action Officer, Joint/Combined Information Environment Division, Deputy Directorate for C4/Cyber, Joint Staff; and Pentagon Information Technology Services Division Chief, Deputy Directorate for Information Technology Services, Joint Staff J6. He has deployed throughout his career beginning as a Lieutenant in support of Operation Sea Signal, Guantanamo Bay, Cuba in 1994; Operation Southern Watch, Dhahran, Saudi Arabia in 1996; and Operation Iraqi Freedom assigned to the Multi-National Corps Iraq (MNC-I) in 2003 and Combined Joint Task Force (CJTF)-7 in 2007.

COL Wills’ awards and decorations include the Defense Superior Service Medal, the Bronze Star, the Defense Meritorious Service Medal with Oak Leaf Cluster, the Meritorious Service Medal with two Oak Leaf Clusters, the Joint Meritorious Unit Award with seven Oak Leaf Clusters, the Army Superior Unit Award, the Coast Guard Special Operations Service Ribbon and the Parachutist Badge.

In addition to his Masters’ Degree in Strategic Studies from the United States Army War College (USAWC), he holds Masters’ Degrees in Human Resources and in Computer Resource Information Management from Webster University, as well as, a Bachelor of Science Degree in Computer Science from USMA. Currently, he is pursuing a Chief Information Officer (CIO) Certificate and Masters’ Degree from the National Defense University. His Military education includes USAWC, Command and General Staff Officers Course, the Combined Arms and Services Staff School and the Systems Automations Management Officers Course.

The Austin Cyber Show is excited to showcase authors who bring their expertise to the Cybersecurity industry through their publications. Join this free Cybersecurity conference via zoom or in person.

Gary Hayslip, author of Executive Primer, The Executive's Guide to Security Programs presents Tuesday, May 10 at 11am CT, with the session "Cliff notes from a CISO author's Perspective."

Jon DiMaggio, Author of The Art of Cyberwarfare will join the Austin Cyber Show to share insights from his book and his experience in the Cybersecurity industry. Join his session on Wednesday, May 11 at 11am CT to hear from Jon first hand.

Extra credit! While the authors of 13. Ransomwared aren’t joining us for the show, this book features Bill Alderson’s company HOPZERO as a security solution. Bill will share his expertise in sessions throughout the show.

Join us at the Austin Cyber Show at no cost via zoom or in person at Concordia University, May 9-12, for these and many other exciting sessions to build your capacity to defend your company from cyber threats.

As your lovely mother walks out the door on this mother’s day, she backs out of the driveway to leave, only for you to find her wallet has fallen out of her purse and into the street. Grateful to be the one who has found it, you immediately imagine the worst case scenario, and decide to take a closer look to be sure she wouldn’t be able to be taken advantage of if someone else found it next time.

As you open the wallet you find the following contents, you are horrified to open the zipper pocket and see her password reminder post-it. Wondering what her level of vulnerability is, you are now drawn to see what a hacker may be able to accomplish if someone nefarious had found her wallet!

Use the items below found in the wallet and see what vulnerabilities you can find!

Did you make it to the congratulations page? If so we would love to hear from you. Share about your experience on social media by sharing a cyber tip you might give your mom to be sure she never gets hacked, and tag @packetman007 on Twitter, @austincyber on Facebook or Bill Alderson on LinkedIn.

Our community event at the Austin Cyber Show is great for folks like your mom, your kids, or your colleagues who could brush up on their cyber-know-how. for May 9 or Catch the recordings on our website AustinCyber.Show.

CISO: Day in the Life of CISO's Sameer Sait & Mustapha Kebbeh

In this session, two seasoned CISO's will talk about their experiences in leading and managing information security organizations. First they will discuss the day-to-day activities in the life of a CISO, followed by the more strategic and business-facing activities required to be successful in the role and finally delve into how external events impact both their strategic and tactical objectives.

Sameer Sait, Former CISO, Amazon/WFM

CISO Entrepreneur Investor Advisor

Sameer Sait is a seasoned information security leader with years of experience leading global, cross-functional teams. He recently exited the Information Security function for Amazon.com's largest subsidiary - Whole Foods Market. His global organization is responsible for IT governance/risk/compliance, security operations, threat intelligence, identity/access management, application and architecture security, eDiscovery & digital forensics. Prior to this, Sameer served as the VP, CISO at two Fortune 500 firms: Arrow Electronics and MassMutual.

Sameer is an advisor to a number of security startups and an active angel investor. His educational background includes a master’s degree from Carnegie-Mellon University and a bachelor’s degree from the University of Arizona.

Mustapha Kebbeh, CISO Brinks

As Chief Information Security Officer for a global enterprise, I head vision and strategy for all IT cybersecurity, enterprise risk management, and IT enterprise architecture impacting 75,000 employees around the world. My goal is to protect information data and technologies from actual and potential threats, both internal and external to the organization.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

Data versus Measurement: Why Your Current Metrics Aren’t Enough

It’s time to get away from just tracking click rates and completion times, and focus on building a comprehensive program that measures the full picture of human behavior in your organization. While most security leaders focus on tracking a set of KPIs, without context and correlation, these data points simply are not enough to move the needle on reducing cybersecurity risk. In order to meaningfully improve your organization’s security posture, it’s time to focus on measuring the right metrics in the context in which they matter.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

The Big Disruption in Cybersecurity

Security awareness & training programs and content has improved dramatically in recent years, yet security and risk leaders still cannot quantify the decrease in risk or increase in vigilant behaviors their programs are enabling. And with 85% of breaches being caused by human behaviors now is the time to make your employees your best defense against threats and incidents. The move to human risk management is the future and security professionals can take the first steps now.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

See it all in 55 seconds. Then pick the sessions of interest from the list below. All sessions are available in the Austin Cyber Show section above.

Session Directory

• Writing Cyber Policies that Aren't Miserable for Everyone
• The Art of Cyberwarfare Insights from the Author Jon DiMaggio
• New Mathematical Models for Forecasting Cyber Attacks
• Cyber Book - New Releases: Authors Featured at Austin Cyber Show
• Managing Tactical US Military Networks: HOPE is Not a Plan
• Cliff Notes from a CISO Author's Perspective
• Anatomy of a Cyber Attack
• Defend the "When" with Network Intelligence
• Unleash the Full Potential of Network Detection and Response NDR
• Mother's Day Lost Wallet Challenge
• Day / Week / Month in the Life of a CISO
• IAM: The New Security Perimeter
• The Big Disruption in Cybersecurity
• Data Verses Measurement: Metrics are Not Enough
• Assigning Cost to Security Risk
• Networked Medical Device Connectivity Security
• Social Engineering Cyber Attacks
• Cyber Incident Response Case Study
• Mystery Solved: Deep Packet Security Analysis
• Constantly Changing Cybercrime
• Navigating Global Public Web Server Privacy Laws
• Five Ways Attackers Leave Ransomware Vulnerable to Detection
• Reclaiming Your Identity
• Cybersecurity Over the Horizon: The Future of Cybersecurity
• What Drives Security Convergence?
• Remaining Ransom Aware: What to Do When One Gets Past the Goalie
• Building a Security Team that Never Says No
• Building a Modern SOC Requires Attack Surface Validation
• Security Leaders Use Ranges & Simulation to Improve Security Posture
• Double Espresso Morning Cyber Brief to Start the Day
• Breach Defense Certificate
• Exfiltration Prevention Certificate
• IETF's New "TCP" QUIC Certificate

Social Engineering Attacks: Why do we fall for them and what we can do about it.

By Dr. Ian Wilkinson

Would you believe that 98% of all cybercrime or cyber-attacks within the past year have been the result of social engineering? Seismic data loss attributed to some form of social engineering attack has been an epidemic-level threat that is currently plaguing thousands of people, both personally and professionally, at home and in the workforce. Phishing, a highly effective method that enables threat actors to deceive users and steal important data normally through unsolicited email attempts, was the cause of more than 240k successful cyberattacks in 2021. Addressing social engineering-induced cyber-attacks is important to information technology (IT) security managers to minimize organizational risks and effectively safeguard data from associated security breaches.

However, most of us think we practice vigilance and caution when exploring our connected world. We use protective tools and measures, processes, and procedures, yet the numbers do not lie. Social engineering is a method used by threat actors – employing their tactics, techniques, and procedures to trick us into providing sensitive information or information that can be used against us, yielding the desired outcome that benefits a threat actor. S So why do we keep falling for social engineering attacks and when are we most vulnerable? Well, it has everything to do with how we perceive the world around us.

Most of our behavior can be mapped to a theory expressed by a professor of psychology at the Yale University School of Management. The basis of Dr. Victor Vroom’s Expectancy Theory suggests that most of our behavior is a product of and motivated by anticipated results or consequences. Vroom (1964) proposed that a person behaves in a certain way based on the expected result of the chosen behavior. Knowing ourselves, our connected environment, and tailoring our expectations in favor of vigilance can be the key to reducing our vulnerability to social engineering attacks.

About Ian Wilkinson

Ian Wilkinson CTO|MA-ITM|PMP|ITIL|CISSP|GCIH|GCIA|TS-SCI - United States Army

Ian Wilkinson CEO Cyber Ballet LLC

Ian’s professional experience reflects 25 years of creative leadership and visionary capacities in complex, competitive, and highly regulated government, and commercial information technology industries. His leadership has been sought by domestic and international partners—lead teams which managed ERP systems, cybersecurity, systems integration, vendor relationships, network infrastructure, and organizational support. As Founder and CEO of Cyber Ballet has been assisting organizations in developing their information technology (IT) strategy and infrastructure, facilitate IT training, and provide innovative solutions that take advantage of opportunities for growth.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

Networked medical device connectivity. What does this mean and why does it matter? Life sustaining devices such as patient monitors, infusion pump systems, ventilators, and a host of biomedical equipment are typically connected via hospital and healthcare facility Ethernet or Wi-Fi. The fact that they are connected to the network and potentially the Internet creates an inviting attack surface for attackers. Peel will expertly guide you through design & threat modelling considerations of medical device security, including the shared responsibility model, and the introduction of the new IEEE 11073 interoperability standard for devices.

About C. Emma Peel

Peel is a dynamic professional speaker highlighting her expertise in information security and compliance. She enjoys sharing insights through popular and relevant articles tackling the challenges and complexities of post-pandemic Cyber Security. Noted by her peers as possessing a vast wealth of knowledge, not just in Cyber-security but in many other areas of IT and possessing a true passion for technology. Currently Director Security & Compliance, Draeger Medical, following eight years as Global Information Security Officer & Risk Manager at a G2000 pharmaceutical and life sciences organization. Previously worked as a Principal Technical Consultant where she designed technical architectures for biomedical and energy organizations. Holds a BS in Computer Science and Mathematics from the University of South Alabama.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

Assigning Cost to Security Risk.

The importance of communicating cyber risk in financial terms to the board is growing day by day. Many have long considered this necessary given these individuals' primary responsibility is to move the company in a better financial direction and often have zero understanding of technology or security. Almost everything that they do revolves around the cost, and the trade-offs and the ROI of various investments. In this session, Jeff Moore will give a peak into his world, where assigning cost to security risk is the primary objective, allowing officers and board members to perform their due diligence. Understanding a company's financial exposure to cyber risk and assigning cost to security is a fairly new phenomena and part of the convergence between business leadership and technology that is moving at a very fast pace

About Jeffrey

Chief Product Security Officer, Draeger Medical, Jeffrey is a respected cybersecurity leader with over 20 years of management and technical experience. He is a visionary responsible for developing strategies to reduce risk and address compliance by enhancing IT security policies and procedures, security operations, and technical capabilities. When considering a strategy, Jeff places importance on the sustainability for the business while being adaptable to emerging cyber security needs. Jeff moved from being Global Head of Cyber-Security at a Global 2000 pharmaceutical and life sciences organization, previously working at Adidas AG, Bindview, and Peregrine Systems, having started his security career at the European Space Agency (ESOC)

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

Reclaiming Your Identity

Learn how to claim your identity before criminals do, or reclaim your identity after criminals have used your stolen "personally identifiable information" (pii) to open fraudulent new accounts in your name.

FrozenPii.com is a free public service that makes it easy for you to start the process of claiming/reclaiming your identity. Frozen Pii gives you the information and links to the national credit reporting agencies (NCRAs) - Experian, Equifax, TransUnion - to: 1) get your free credit reports, by law, 2) correct your credit reports, by law, and 3) freeze your credit reports for free, by law.

Frozen Pii also provides you with information and links to:

Several other credit reporting agencies (CRAs) to protect against new-account fraud for utilities, TV services, cell phone services and bank accounts, The Social Security Administration (SSA) to protect your SSA benefits, including SSA annuities, Medicare and Medicaid, The Internal Revenue Service (IRS) to protect your income tax refunds, The non-profit Identity Theft Resource Center (ITRC) for free ID theft victim help with expert advisers by telephone or real-time chat, The Federal Trade Commission (FTC) for online identity theft victim help through IdentityTheft.gov, and The FTC for online help for internet scams.

Tom O'Malley

Frozen Pii, LLC

Frozen Pii was founded by Tom O'Malley, a retired state (1980-1985) and federal prosecutor (1986-2019) who specialized in the investigation and prosecution of cybercrime, including computer hacking, identity theft and identity fraud (2009-2019). He also is a data breach victim.

In 2015, Tom and 21.5 million other data breach victims had their classified personally identifiable information (pii) stolen from the U.S. Office of Personnel Management. The OPM data breach prompted him to build Frozen Pii to help his family, friends and colleagues prevent their stolen digital identities from being used by criminals to get fraudulent new-account credit cards and loans.

In 2017, Tom and 145 million other data breach victims had their personally identifiable information stolen from Equifax, one of three major nationwide credit reporting agencies.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

Protecting customer's data privacy has become a new source of stress for any company with a web presence. Compounding that, international trade requires compliance with international privacy laws regarding the collection, storing, and use of personal information to avoid fines that can add up to millions of dollars.

In this presentation we will compare privacy regulations from the US and from other -- European Union (GDPR), Switzerland (FDPA), Japan (APPI), Brazil (LGDP) -- countries and their impact on online business. We will also touch on how to balance these laws and on how to turn such compliance into benefit to their business.

About Kim Green

Kim Green (Cisco CCNA R&S, CCNA Sec; CompTIA A+, CySA+; ecfirst CCSA; Microsoft MCSA; PaloAlto ACE, PCEC) is an IT Professional passionate about community building, networking, and security. She has presented to local chapters of ISSA and OWASP, North Carolina PBL, and been a guest speaker at Triangle InfoSeCon. Kim is also published in Academy and Raleigh ISSA.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

The nature of cybercrime is constantly changing and becoming more difficult to defend. This session will evaluate some of these changes in the nature of the internet and the new attack vectors you should monitor. Be aware that we seeing a dramatic increase in cybercrime as Russia is behind much of cybercrime and they are using new methods to attack the Ukraine and will be learning what new attacks are most successful.

There has been $1 trillion invested in cybercrime in the past 5 years and yet the facts show that cybercrime has increased in frequency and severity every year for the past twenty years. We will discuss why $1 trillion in funding has not only not stopped cybercrime but why it has not ever even resulted in a reduction in cybercrime.

I have worked on some of the most significant breaches in the past ten years and seen the sophistication of the attacks transform year after year. It is reported that the losses to cybercrime exceed $6 trillion. Think about the R&D budgets $6 trillion will fund. One successful government breach he worked on resulted in shutting down an organization that was spending $100 million per year in R&D and the math reveals that some criminal organizations are spending far more.

About Jack Blount

While Jack spent most of his 40-year career in the technology industry he says the four years he spent as CIO for the federal government is what really exposed to him the sophistication of cyber criminals and nation states involved in cybercrime. While much of the world blames Russia for being the world leader of cybercrime Jack realized that probably 80% of cybercrime is actually developed and executed by the Chinese. What he repeated found working the CIA, the military and other government organizations is that the Chinese use source code found on the dark web from Russia that the Chinese use to make Chinese attacks appear as Russian in origin. This practice is still being used today.

While Mr. Blount has officially retired to spend more time with is grandkids and horses he is still very connected both in government and in the industry and is active as a consultant on cybercrime, AI, Quantum computing, and robotics.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11.

Mystery Solved: Deep Packet Security Analysis

TLS version and cipher suite are the two most critical parameters negotiated during the TLS Handshake. Agreeing on a less secure option can leave a host open to vulnerabilities. This session uses Wireshark to show how the options are negotiated in the packet.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

In this session Brian Clinkenbeard, one of the prolific and beloved security gurus of the industry, will look at exposing failures in incident response through case study. Brian is an interesting character, as you will learn immediately. His brain works much faster than his lips. He will entertain and challenge you. And, if you listen carefully, you will truly be educated on something significant in security.

Incident Response: Exposing the Cause and other Cascade Failures

Incident response can be painful when internal operations are not staffed, trained, or prepared for a proper response. These types of responses not only expose flaws and vulnerabilities specific to the incident being investigated, but often end up highlighting other vulnerabilities as well.

This presentation explores how an incident response itself not only exposed the root causes, vulnerabilities, failed processes, and procedures, which, in the end, look exactly like the incident being investigated. Here, a suspected internal threat is ultimately found guilty only of incompetence to secure themselves, or the organization.

About Brian

Virtual CISO and Inventor of new and innovative approaches and methods for cyber security. Cyber Sec Asset in CONSTANT TRAINING - Dev Guru. As a technology executive familiar with multi-tenant product architecture, competitive analysis, technology vision expertise and e-commerce expert, Clinkenbeard suggests "I knock on the sky and listen to the sound." Your technology should be a peripheral to you, not vice-verse.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

We welcome attendees to examine in an interactive session the latest cybersecurity trends and their implications for organizations facing new and emerging cyber risks and threats.

Cybersecurity has always been a never-ending race, but the rate of change is accelerating. Companies are continuing to invest in technology to run their businesses. Now, they are layering more systems into their IT networks to support remote work, enhance the customer experience, and generate value, all of which creates potential new vulnerabilities.

At the same time, adversaries—no longer limited to individual actors—include highly sophisticated organizations that leverage integrated tools and capabilities with artificial intelligence and machine learning. The scope of the threat is growing, and no organization is immune. Small and midsize enterprises, municipalities, and state and federal governments face such risks along with large companies. Even today’s most sophisticated cyber controls, no matter how effective, will soon be obsolete.

In this environment, leadership must answer key questions: “Are we prepared for accelerated digitization in the next three to five years?” and, more specifically, “Are we looking far enough forward to understand how today’s technology investments will have cybersecurity implications in the future?”

Daniel Wallance

Daniel Wallance is an Associate Partner with McKinsey & Co. He focuses on cybersecurity for global banks and financial services institutions across such topics as technical assessments, strategy and cyber program development. When not living in a virtual world of Zoom and Microsoft Teams, you can find Daniel in New York or traveling around the world. He is a member of Business Executives for National Security and holds a master’s degree in systems engineering from MIT.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

Remaining Ransom Aware

Ransomware, among the most intimidating threats on today’s security horizon, is an issue best approached from a “not if but when” perspective. It’s no longer enough to bolster your defenses. Today’s CISO should know what to do when one gets past the goalie.

We’ll discuss:

Defensive practices for those hoping to avoid a ransomware attack Practical steps for those who find themselves in a ransomware situation The pros and cons you’ll need to weigh when choosing whether to pay

Todays CISO should know what to do when one gets past the goalie

Mustapha Kebbeh Brinks Chief Information Security Officer CISO

As Chief Information Security Officer for a global enterprise, I head vision and strategy for all IT cybersecurity, enterprise risk management, and IT enterprise architecture impacting 75,000 employees around the world. My goal is to protect information data and technologies from actual and potential threats, both internal and external to the organization.

My strength involves continually fortifying corporate security through standardization, training, and innovative technology to empower employees on all levels. Through my efforts, I have helped Brinks, IBM, Vodaphone, and CompuCom safeguard their IT infrastructure and mitigate risk from malicious hackers, human error, and general security gaps.

Key factors include bringing in the right talent, building a culture of security-minded personnel, and getting everyone involved. It’s also critical to think about the business from more than a security viewpoint by selecting tools and technology to drive revenue and business growth as well as system protection.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day

This session will report on some observations about how the degree of convergence of physical and cyber security is evolving in larger organizations from a number of industries as they seek to optimize security outcomes.

Herbert Mattord, Ph.D., CISM, CISSP completed 26 years of IT industry experience before joining the faculty at Kennesaw State University in 2002. He was the Manager of Corporate Information Technology Security at Georgia-Pacific Corporation, where much of his practical knowledge in information security was acquired. He is on the faculty at Kennesaw State University with the rank of Professor where he teaches Information Security and Cybersecurity. He serves as the Director of Education and Outreach for the Institute for Cybersecurity Workforce Development (cyberinstitute.kennesaw.edu). He is the co-author of several books published by Course Technology and an active researcher in information security management topics.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day.

Sam Harris, Stratascale

Building a Modern SOC Requires Continuous Attack Surface Validation:

The Modern SOC requires continuous validation of the attack surface to understand what threats exist, how those threats change every day, and the impacts they may cause.

Protecting your enterprise from breaches and bad guys in the cloud era means staying several steps ahead. If your Security Operations Center (SOC) fails to keep up with changing technologies and tactics, your risks multiply significantly.

The enterprise SOC assumes responsibility for centralized, consolidated cybersecurity ranging from incident prevention through detection and response. Tasks typically focus on security monitoring as well as device management, threat intelligence, incident response, and training.

Enterprises have now added Attack Surface to their SOCs and are expanding that Attack Surface as fast rate forcing the SOC to know a wide range of technology, attacks, and prioritization strategies. In order for those SOCs to be successful they’ll need to ensure the data they receive about the Attack Surface is continuously validated to avoid falling into the rabbit hole.

Sam Harris

Security Professional specializing in Vulnerability, Attack Surface, and Risk Management.

Responsible for the vision and development of service offerings to assist organizations in securing their Attack Surface. Fortune 500 Security Professional who assisted in developing an automated Vulnerability and Risk Management program for multi-national sites.

Practice Lead and Service Operations Manager of Security Operations for Managed Security Services provider.

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day. Attendees will walk away with new insight and leadership lessons learned to defend against ransomware, phishing, and data exfiltration attacks. Five Cyber-By-Fire Skill Certificates are available to earn at the event and via Zoom Events online for 30 days afterward.

Building a Security Team that Never Says “No”

One of the most prevalent perceptions of any security team is that they just always say “no,” that they will be a blocker to progress and slow things down. This leads to teams circumventing the security process, lack of support from leadership, and an inability to implement any good new security measures. It’s a story that we all have seen time and again throughout the industry.

As more companies are starting to understand, having a strong security culture doesn’t necessarily mean having to always be a blocker. Instead, implementing a security program with the mindset of never saying no to any request or new idea can enable the security team to effectively implement security controls and improve the company’s security posture at a rapid pace in collaboration with the business instead of fighting against it. And doing it all even with a smaller security team.

Learn about the concepts and frameworks that companies like Indeed have established to enable this remarkable change, the mindset that is needed within the security team to make them successful, and how avoiding a single word can lead to dramatic changes in perception.

The alternative is:

Nick Leghorn is currently the Director of Application Security at the New York Times, and previously built the Security GRC team within Indeed from the ground up using these same principles. Nick has spent his career working for a number of large companies with complicated environments including Rackspace Hosting, Shoretel, Mitel, and Indeed improving the security of both the infrastructure itself as well as the processes within the company.

About Nick

Nick Leghorn is the Director of Application Security at the New York Times. After graduating from Penn State University with a degree in Security and Risk Analysis, his first job was working for the U.S. Department of Homeland Security quantifying terrorism risks and identifying mitigations to provide the best risk reduction for each dollar spent.

Nick has spent his career working for a number of large companies, including Rackspace Hosting, Shoretel, Mitel, and Indeed, improving the security of both the infrastructure itself as well as the processes within the company.

Nick's other Session: Writing Cyber Policies that Aren't Miserable for Everyone

http://austincyber.show/writing-cyber-policies-that-arent-miserable-for-everyone/

Join US and Canada cyber community members at the inaugural Austin Cyber Show Conference at Concordia University Texas, May 10-11. During the two-day cyber defense conference, participants can engage in discussions with peer leaders and industry experts on the cyber risks and challenges that businesses, leaders, developers, educators, and students face each day. Attendees will walk away with new insight and leadership lessons learned to defend against ransomware, phishing, and data exfiltration attacks. Five Cyber-By-Fire Skill Certificates are available to earn at the event and via Zoom Events online for 30 days afterward.

Would you board a plane if you knew the pilot got all As in flight school? Probably yes, but what if you learned that the pilot got all "A"s , but this was their first time actually flying a plane, and that they never went through any flight simulation training? You’d likely turn around and walk back to the terminal.

Just as surgeons must complete a residency, and pilots must undergo flight simulation, cyber defenders should be held to the same standard.

Near 50 years ago Internet Protocol Pioneers imagined a network that would interconnect people around the world. Little by little the Internet Engineering Task Force IETF though working groups worked to build what we now call the Internet.

A new set of network protocols have been developed that improve device to device security and performance. It's named QUIC. During these Certificate sessions the story will be told how TCP/IP literally replaced all company proprietary protocols from IBM, Xerox, Apple, Novell and Microsoft proprietary communications. TCP/IP replaced company protocols and now the new QUIC Protocol developed by the same IETF organization is replacing the TCP features with many improved benefits.

Three to ten or in some cases up to100 times faster applications result, in fact, enabling some applications to communicate globally for the first time - and it has only just begun.

Learn how it works, why it is faster and more secure for device to device communications, but offers larger organization networks some security challenges. Make no mistake device to device data is more secure, it's just when used on large networks it may allow some security or denial of service problems on networks for firewalls, gateways and load balancers. Security firewall vendors have yet to fully support the new protocol quite yet.

But due to its compelling speed, server and online service providers have started using it, and due to its performance and rapid mobile session movement support it has become a compelling reason for adoption. Development options for Microsoft, Apple and Google devices has made QUIC the go-to solution for faster web servers and mobile applications.

From a weekend experiment in 2012 it become an IETF working group and QUIC was ratified as a standard IETF Transport Protocol Summer of 2021. In the words of a popular load balancer vendor, it seems "QUIC is eating the Internet".

QUIC's success is a good thing, but many enterprise networks don't yet have it managed to mitigate some of QUIC's network firewall issues, as most large firewall vendors, for now, recommend turning QUIC off by applying firewall deny filters. Despite these warnings, QUIC continues to gain adoption due to its compelling features.

That is why every Desktop, Server, Network, Application Developer and Security professional needs to understand how it works, why it is faster and how to manage its use more securely through improved monitoring, analysis and special settings to help mitigate some of the problems networks may face now that all Web Browsers, Servers and popular Social Media Apps on most all operating systems are QUIC enabled.

Collaborate and hear from protocol experts and Internet Protocol pioneers as they demonstrate live QUIC analysis benchmarks to help technology practitioners better understand and manage the evolution of QUIC.

Sessions will provide essential understanding and testing for the new QUIC Protocol Certificate is part of the offering both In-person and Online May 10-11 2022 on the beautiful Concordia University Campus in Northwest Austin. Group company and Professional Organization discounts are offered, with educational Scholarships available.

The fundamentals of Exfiltration prevention start with Data Loss Prevention DLP. Below is a diagram of various DLP methods designed to prevent data from leaving an organization by various methods. Users often mistakenly send sensitive data in email text or attachments. DLP email scanning discovers and sequesters sensitive data before it is sent outside a safe boundary.

We created the diagram below to step through various scenarios explaining how various aspects of the system work step by step. Key to the conversation is what DLP protects and what it does not protect and why.

Encryption is key to data security, but there are ways attackers use encryption to evade detection. This is a significant issue for different DLP capabilities. Decryption is often employed in DLP scenarios, unless the encryption private certificate is not owned by the organization, as with most SaaS and external web applications.

These are essential concepts application developers, systems and network administrators need to understand to better protect an organization's vital data.

Breach Defense is one of five Certificates offered.

Breaking down the anatomy of a massive breach starts with a diagram describing attacker steps taken to avoid detection. Several lessons have been learned that help organizations defend against supply chain breaches.

The criminals exfiltrated data to a Microsoft cloud server using a DNS name that was designed not to create suspicion, but the DNS name server name was suspicious.

In the materials are additional diagrams suggesting by color code what parties could be considered at fault.

Many simple best practices will be examined that would have stopped the breach at various points. Once beyond the simple fundamental errors, more sophisticated attack methods were successful at completing the breach.

May 10-11 2022 Session by James Azar CyberHub Podcast

Title: Double Espresso Morning Cyber Brief to Start the Day

Join us as James gets AustinCyber.Show off to a Double Espresso start May 10, 2020

James Podcast has become a Hub of the Infosec Community. Joint him every day as he brings to our attention the latest in current cyber information events and patches that are essential.

A daily podcast with host and CISO James Azar delivering the latest in risk, impact, and mitigation for cybersecurity practitioners globally. The show is live at 9 AM EST Monday through Thursday and tune in each Friday for a special Tech Corner with some of the industry's brightest minds to discuss the latest challenges, innovations, and technologies.

James mission is to provide substantive and quality content that’s more than headlines or sales pitches. We want to be a valuable source to assist those cybersecurity practitioners in their mission to keep their organizations secure.

James Azar CISO

Driven Security minded and business oriented security professional James works, leads, and is dedicated to the security and business mission. In his experience, James has served as CTO, CIO, and CISO but his true passion lays at intersection of Security and Business where innovation and out of box thinking are needed to drive security growth within an organization.

James is the host of the fastest-growing cybersecurity podcast in the country CyberHub Podcast, CISO Talk, and a new and noteworthy privacy podcast called Goodbye Privacy as well as the Other Side of Cyber on ClubHouse.

James is a global public speaker having spoken at events like CyberTech Israel, RSA, Data Connectors, FutureCon and has been published in Fox, OANN, AJC, ABC, NBC and James also writes for Substack.

James enjoys multiple espressos a day, chasing down a steak with Whiskey, Bourbon, or scotch.