On The Wire

(Content not available)

Fortune 100 Company has serious Citrix issues affecting over 22,000 users regularly. Our Remote Diagnosis service was exactly what they needed to uncover the technical causes of the problems. Graphical annotation helped dozens of technologists and managers understand the problem and improved their ability to collaborate with their vendors to gain consensus toward mitigation. Very successful optimization of existing systems.

Our participation remotely on the trouble call bridge, working with their packet capture team to gather the appropriate packets enabled successful remediation of dozens of Citrix related problems. This company had excellent project management, packet capture and technical teams.  They needed an experienced skilled analyst to uncover the details so consensus could occur between teams and vendors.

After spending $600,000.00 on server CPU upgrade that did not improve response time, this company found the real problem they were trying to resolve.

We helped them diagnose the true cause of the slowdown.  It was the Java code in the client that was causing an 89 second slowdown in the application. Once pinpointed the client was able to show our annotated proof to the vendor responsible and the problem was fixed with a simple change.

Global 2000 Company receives 10x Application Throughput after diagnosing problem with Onsite Analysis. This organization received great reward for not one dollar of new equipment spending. Solved with a VLAN parameter change we identified, carefully implemented through change control and validated with monitoring to demonstrate return on investment.

This organization had excellent change control where the technologists would presented their change to other technologists in a presentation room allowing other organizations and technologists to understand and challenge the viability of the change before it was approved. The evidence we provided made easy work of bringing consensus with other groups and vendors.

Looking at a Packet Trace File is often the first step in uncovering the problem.  Like the TV Show "Yankee Workshop" just buying or having the same tools doesn't mean you can turn out beautiful results. No matter how smart, tools require skills that takes time and one must be expertly trained, and use the tools regularly so they become second nature.

The three "T's" are Tools to analyze, with knowledge of Theory, the second "T", on how protocols transact enables diagnosis.  In order to diagnose issues one must possess the Tools, Theory and the Time, the third "T" to be proactive. If missing any of the 3 "T's" our Packet Trace File Analysis services can bring paradigm shifts that overcome silence or finger pointing.

Datacenter consolidation creates performance problems for slow Oracle 11g Client queries.

An application's server infrastructure was moved from an international location into a central datacenter in the USA. The application uses data sources on a MS-SQL database, but weekly the application requires an extraction from an Oracle database.

The weekly Oracle data extraction original physical server environment took 45 minutes to complete in the international location on native physical servers. After moving to a USA virtualized datacenter the Oracle data extraction took over 18 hours and crashed before completion.

The part of the application, the weekly data extraction apparently was not tested before the application cutover. Now that the cutover has occurred the situation has become a high severity issue affecting business.

The organization has a highly effective cross organizational project management team that gets involved when high severity business impact exists. The project managers bring in all the responsible parties onto a conference bridge managing across technical and management silos.

The global services group has central analysis tools, in this case NetScout's Sniffer InfiniStream. The servers in the former international environment do not have infrastructure embedded packet capture capability so we have been using MS-NetMon on the platform as it is included by Microsoft with the server and easier to allow its use from a logistic and security perspective.

The packet capture traces illustrate slowness not on the part of the Oracle server where the data is being extracted from, but in the requesting Oracle Client running on a 2008 Server with MS-SQL 2008 R2 (or R1). The Oracle Client (running on the new virtual MS 2008 SQL Server in the datacenter) sends a request to the Oracle Linux server also in the datacenter. The reply is very fast with about 10,000 bytes. After receiving the 10,000 bytes the Clients sits for roughly 150 milliseconds after fully receiving the previously requested data before it sends the SQLnet "get next row" request. It is this delay before spawning the get next that causes the slowness for many thousands of the same "get next row" requests.

The problem was found and analyzed in the new virtualized datacenter production environment and replicated / confirmed in the new virtualized datacenter development environment.

In Oracle and third party support forums many have reported the same difference in Oracle Client vs SQL plus / Microsoft SQL Management Suite performance.

Learned that the part of the Oracle Client 11g being used is the quite legacy OLEdb still being included and that Oracle does not support its client in any virtual environment other than its own! Therefore the only option is to use native physical machines...

Pentagon 911 Lessons Learned - IT Disaster Recovery Plan

Bill Alderson discusses the technical and IT Best Practice lessons learned from the events subsequent to the Pentagon 911 disaster. Several key IT Disaster Recovery Plan lessons can be learned from this video.

Transcript for this Video

Good afternoon. This is Bill Alderson coming to you from beautiful Austin, Texas. I hope everyone is doing well today. I’m going to have a short presentation for you. It’ll be under 30 minutes in respect of your time.

We’re going to talk a little bit about the Lessons Learned 10 Years Later, from the Pentagon 9/11 disaster. I was privileged to be one of those that got called. We all remember the event and probably where we were, what we were doing, who we were with, how we responded. Our world was kind of devastated for a while. A couple of days later I got a call from a Pentagon General asking if we wouldn’t come in and bare a hand in helping them recover communications. It was obviously a very proud time in my life to be prepared and to be asked to serve in that capacity.

We responded and came in and helped out as best we could. We analyzed a lot of various problems. We took a look at various communication circuits, network management systems, security infrastructure, pretty much the whole gamut of IT. We analyzed, assessed, looked at, and prepared a 100-page report for them so that they’d know the results of our analysis and then how to proceed with that information.

My organization serves large-scale environments, the Global 2000, military, government. In the last 10 years, we’ve gone into Iraq a half a dozen times and Afghanistan and helped diagnose problems across the entire war environment, and then back to the Pentagon to analyze other things. That’s the nature of our service. It’s high visibility, large-scale critical problem type environments.

In the last 20 years of diagnosing critical problems down to the technical root cause, we’ve learned a few other things. We’ve learned that it’s not just the technical problem that we want to avoid, but the processes which lead to the ability of those technical problems to exist. Essentially, our root cause is changed from technical root cause back into the best practices that obviate those problems. We look back and find what processes were omitted or not in effect.

Typically I’ve got a CIO whose enterprise was just melting down and we came in, we diagnosed the problems, mitigated the problem, and of course they have to ask, “Well what can we do to prevent?” Not only in the last 20 years have we been diagnosing critical problems, but we’ve been analyzing what kinds of procedures, processes, and systems allowed those critical problems to exist. That’s kind of the way that we’re moving forward with the new Apalytics Corporation, to help people mitigate and identify those things that they can do that will avoid crisis and avoid problems.

I hope you came to learn a little bit and to just see my perspective and that sort of thing. Now, what we’ve done is we’ve kind of taken all of these lessons learned and we call this Technical Systemization. There are pillars to technical systemization. We’re going to go through some of those pillars today because they directly relate to the events of 9/11 and pretty much any other type of disaster. Taking the lessons learned and applying them.

Now, Technical Systemization is very much an implementation. ITIL is a model. ISO has models. We have protocol models. We have the OSI model. Well, the OSI model is just that, it is a model. It is not an implementation. I want to make and draw a distinction between that of a model, like ITIL is a model, the OSI model is a model, but an actual implementation, such as IPX, TCPIP, or Apple Talk. Those are true implementations. They look different from the model because they are applied to your exacting problem and your exacting environment in such a way as to have peace to make things truly work.

By changing the name of your organizations to model, to be the same as, for instance the ITIL model, is not necessarily implementing the technical systemization that will allow that to work. That’s part of our message. Each institution needs to have their own Technical Systemization, their implementation of ITIL. ITIL is not a protocol like TCPIP. TCPIP is an implementation of a model, but not the model itself. That’s not the implementation.

One of the things that we’ve learned is that there are a bunch of silos out there. The Pentagon has silos. A lot of them have demarks for who runs the network, desktop, server, platform, network management, and security. All of these are one of many silos. What we call Business Technology Integration is the ability to do cross-silo collaboration, whether it is system design, metrics and monitoring, or network documentation.

Let’s just take network documentation for a moment. As I travel around and look at various places I walk and invariably I see diagrams on the wall. Sadly, most of them are labeled 2008, 2007, 2010 and they are typically dated and that sort of thing. I don’t know why they’d still be on the wall. Nevertheless, it’s probably because they haven’t been updated. It’s an interesting thing. If you take all your business silos, desktop silo, network silo, application and platform silos and with each one of them build a network diagram or a system diagram, if you were to take those and lay them horizontally from the client to the server and the application, could you put those things together and say, “Okay, here’s the desktop, here’s the network, here’s the WAN, here’s the security infrastructure, here’s the platform, and here’s the application and all the dependencies”? Could you put your finger on the diagram on the left and give a troubleshooter and end-to-end diagram? I don’t think so.

Case in point, when we arrived at the Pentagon, we came into a room and we started, “Okay, first thing, we need to know a little bit of the lay of the land.” There was kind of a gulp and they said, “We have a great network documentation system, however, it was one of the servers impacted in the disaster.” They didn’t have anything printed out because it was online for ready reference. Don’t forget, sometimes those systems go down. It’s probably a good idea to have the recent version printed out somewhere so that you can refer to it.

We started with a blank white board and started diagramming. Who had to do that diagramming? Well, the key technologists that had a lot of other fires burning. We came in to help. Because these key technologists had to help understand so that we could bare a hand and start helping them so you’d have more people helping, they were out of their key duties. It’s incumbent to make certain that your key technologists are documenting the systems and that you are training everyone so that everybody knows the key elements of the system.

We’re going to talk a little bit about that, but that’s one of the examples of how organizational silos need cross technology collaboration teams to pull together system documentation that’s congruent. It gives you an end-to-end picture in network monitoring metrics and that sort of thing.

Back to The Art of War, Sun Tzu and The Art of War is one of the books that are studied by most military officers at one time or another. The bottom line to this is ‘Know Yourself’ and ‘Know Your Enemy’. That’s a fundamental thing. If you know yourself, you can react to things. In the area of knowing yourself, we need to develop end-to-end cross technology, cross-silo documentation.

We have literally hundreds of people who are willing participants in optimizing our network systems and applications for the end users, but I find secretly that some of the people who come to the table a little bit later, the people who’ve come in the last few months, they’re not going to come and tell you, “Hey, I’m really impotent because I don’t know where anything is. I don’t have a diagram. You don’t have a diagram.” They’re not going to come tell you this. It is one of the biggest lost production areas that your organization has. You have hundreds of people who are willing participants, but yet are impotent to participate because there is very little true end-to-end cross technology, cross-silo documentation.

When there’s a trouble ticket they can, first of all, identify the client, put their finger on the client network. Then they can start moving to the right and into the network and see the network infrastructure that they’re dependent upon. Then they can move their finger to the right and see that firewalls, and load balancers, and WAN optimizers that they’re dependent upon for that system to work. Then they can move over to the right and look at the data center, the data center distribution, into the virtual platform, into the virtual image that they’re running, then into the virtual switch, then on into that process that they’re connecting to, then back into the application, and then the application having secondary responsibilities or geared capabilities. Now you have the ability for a help desk person to kind of understand your infrastructure and very rapidly assimilate and at least triage where the problem may or may not be.

Then you can see what- Network Management. You can say, “Test Point One. Test Point Two. Test Point Three. Test Point Four.” You can visualize your infrastructure. Probably the number one thing that I would stress is that you must know yourself. If you don’t have that, it’s costing you orders of magnitude more than you think and a lot more than it would cost you to put these systems in place.

Now, once you have that cross technology, cross-silo documentation and everyone can meaningfully participate, then you make sure you train everyone on the design constraints, your architecture, and certify them on your system architecture, your abilities, your capabilities, your constraints of your architecture so that they’re not trying to go in and modify to make it look like where ever they came from previously. You want your architecture to be pure and to work systematically with all your various components. It’s important to educate everyone on that part.

Now, one of the things, and an example of something that we’ve done inside large scale environments, is to help people to develop scalable cross location, cross domain data collection capabilities. Today, you have so many firewalls, load balancers, locations, different domains, different security levels, etc, that you can’t just go out and discover your network and then map it. It’s very difficult. If someone can, usually they can do it in one area, but not in another area.

Our philosophy is you pull all that information from a variety of sources into a single repository across the organization. That becomes what we call, Rapid Network Rendering Database. That database becomes a repository and you create a SOA, over to the right there you’ll see a SOA, so that other corporate systems can come in so that you can export to your network management systems to make certain that you’re monitoring all of the various dependencies of applications and systems.

Over on the left, when data goes into that RNR database, one of the things that we do is we make sure that it has the correct endpoints on it. When those systems have the correct endpoint, when those objects, your network management objects, your router port objects, your switchboard objects, have ends placed on them, you can do reconnection of those objects. When they are simply an object and don’t have two endpoints, it’s really difficult. Just populating a database without those endpoints connected on both sides isn’t going to do you any good. You’re not going to be able to export them.

We’ve designed a way of, when that stuff comes in, we put the endpoints on them and then we connect the points and populate a Visio diagram so that you can see the details of your network infrastructure port by port logically in a visual way to get the VLANS and all the virtual machine information so that a troubleshooter can truly see the environment that they are trying to troubleshoot. That’s just one particular way that we like to see things done.

Now, the second thing is know your enemy. Your enemy is a Root Cause of the problems that you have. You want to perform formal root cause analysis whenever possible. If your helpdesk solution is to always reimage, reboot, or reload, that’s problematic. They should take a moment. They should take just the opportunity for a moment to make sure . . . I’m not saying leave the problem go forever, but I’m saying resist the temptation to reboot a server or even an end user workstation until you’ve had the opportunity to collect a little bit of data. That data doesn’t have to be analyzed right there at that moment, but can be put together as a collection of information, trace files, etc. to then do analysis retrospectively. A lot of folks have some of those capabilities, but I’m saying that you need to involve your helpdesk in capturing those sort of things so that the recurring problems are able to be diagnosed and then obviated.

Hey, the best solution to any problem is not to have it in the first place. Unfortunately, few are motivated with such things. Management needs to know how many trouble tickets you’ve closed and how many problems you’ve solved. Well, wouldn’t it be great to say, “Because we were so proactive, we obviated the problems”? Of course, at that point you lose your justification for the tools, systems, and people. Truly, that should be the objective of any senior management person, to obviate and have all your technologists actually there, but kind of like the Maytag repairman without a lot to do, but when the problem occurs they’re there to help.

The other part of it is the ITIL problem management analysis, going about finding out your statistical number problems that you have. At the Pentagon, when we arrived there, they were getting right about 100,000 SNMP traps a day. Well since then, they’ve started a filtering mechanism. It was so overwhelming, 100,000 a day, that they were not able to respond to the ones that were the most critical and severe. They learned that, “Okay, if you’re going to have 100,000 SNMP traps a day, you’d probably better figure out which ones are your most material, which ones are your biggest ones and how to go about taking a look at that.”

Now, the third thing that I want to talk about in Problem and Change Management is that I go into organizations and I find something that’s amiss and all we have to do is change a few parameters or repurpose a couple of existing systems or capabilities. Invariably, they look at me and they say, “Well, we have no way of promulgating this optimization. We have no way to put this into change control and to have a fix. We either have to have an emergency crisis that’s burning down the building or we have to have a multimillion dollar procurement in order to promulgate and have a set of steps in which we can do change.” My recommendation is to make sure that you have a way to implement free changes that are not procurement or catastrophic kind of response changes. There are many free ways of improving things.

I was at a site a few weeks ago. We went in and their entire Internet access was very slow. We went in and diagnosed the exact cause of the problem. We tested. We implemented, proved that it mitigated. They looked at me and they said, “We have no way of implementing this change to thousands of our users.” They have no way of linking it unless they’ve made a three or four million dollar procurement to any ability to change their infrastructure. Anything that would be cost free has no mechanism by which you can implement and repurpose your systems with just a few process changes. I’d recommend that you think about that. Make sure you can do things that are free.

The other thing is, is there a Master Plan? Now, after the Pentagon disaster, these guys were prepared. They had master plans. They had in-the-event-of plans. They had future plans that mapped the way forward. They took our documents, other things that we learned, and a fresh realization of the threats and vulnerabilities and they integrated that in rapidly. They took that forward. Congress actually allocated a half a billion dollars to beef up the vulnerabilities of the Pentagon. That wasn’t because they, Congress, wanted to spend money. It was because the people who managed the Pentagon network infrastructure had an articulated strategy and they knew where their threats and vulnerabilities were. They had it articulated in a well-articulated plan that allowed them to move that forward and to get champions within Congress and within their leadership in order to accomplish these feats.

If you don’t have a Master Plan, what you’re doing today, your ‘is’ environment, and then you have a disaster, what’s going to happen is you’re probably just going to rebuild in the same way that you were before. That’s, of course, going to be a few generations behind. In the way that we talk about it today, if you’ve seen those insurance commercials, it’s like, “Man, if your car is totaled we’re not going to give you the same 1974 Pinto station wagon. By golly, you’re going to get a 1975 Pinto station wagon in replace of your old car.” It’s the same thing here. If you have a well-articulated plan and there’s an emergency, a problem, or an unfortunate situation and you have to rebuild part or your entire infrastructure, if you have this plan, it’s an excellent place to start. It’s a good thing to do.

The next thing is Decision Support Metrics. Well, remember all those systems that were given 100,000 SNMP traps a day? Which ones are important? Which ones aren’t? Well, those were all SNMP traps of yesteryear, the red-green, not working or working. Today, it’s more about performance. What we call Decision Support Metrics is developing a collaborative cross silo management capability validating your end user application capabilities with signatures so that your monitoring systems should be validating any potential changes. Talk about change control. If your monitoring system is monitoring all of the vital signs of your key mission critical applications and you make a change it’s going to tell you, boom, immediately, that you have a problem affecting one of your end user applications or capabilities. That takes a lot of care and feeding.

The secondary thing about good Decision Support Metrics is that if you know right where your users are slow and things that you obviously can’t optimize without a project… You have a whole list of projects that are out there that you would like to spend on. If you spent in every area: storage, network, and every part of your infrastructure- your budget would be depleted rapidly. In today’s day and age, you use Decision Support Metrics to identify your bottlenecks so that your portfolio spending can be pin point accurate. You can take what you learn from those metrics and you can apply them to your budget allocation, your resource allocation and you know that you are going to get a return on investment that is worthy of the money that you’re spending.

Today’s CIOs have an incredible problem. They’re asked to upgrade in a forklift fashion across the board. It’s cost prohibitive. You can’t do it. Even more so today, we need to have Decision Support Metrics that are helping us prioritize exactly where we’re going to spend the money and quantify the return on investment by indentifying end user signatures, by identifying the dependencies and the process flow of the application, especially when you’re starting to talk about cloud. That gives me a whole other thing to talk about and we’ll do that another day. Now we have dependencies and the vulnerabilities of moving to cloud, what about cloud, and then cloud to cloud. How are you going to analyze those problems? We’ll talk about that another day.

Okay, so performance indicators. You’ve got NetFlow to go back and forth for rate and flow information to find out if certain applications are working so you have network flows. You have device status. That’s your SNMP. What’s going on with your components? Do you know that your executives and your business leaders do not care about the status of a router, a switch, or something? What do they care about? They care about the capabilities and the abilities of their people.

In the war environment, commanders don’t care that this link is down or that link is down. What they want to know is, “Can we carry on certain command functions?” We have to translate our performance indicators from what we’re comfortable with, which are router ports and switch ports and links, into to what our executives and our end users care about. Where am I impacted? Who is impacted? What applications and what capabilities do I have or not have to run my business or my command on?

Knowing what that path is between the client and the server, and the response time at the server, and then the latency going back and forth, and the ability to capture those transactions across your network, is very important.

In talking about multi-points and multi-tier I’d like to just mention that when you have good Decision Support Metrics, you can see that the process, in other words the CPU, of your web tier or the CPU of your app tier, or your SQL tier or mainframe tier, or your security authentication is consuming most of your response time. By having that multipoint, multi-tier transaction analysis, the ability to know exactly what is slowing down your end user transaction that allows you to spend in the right areas, diagnose in the right areas, and serve your end users well.

Now, back to the Technical Systemization, you can go on our website. We’ve got this model on our website of our Technical Systemization. There are thousands of things that you need to do. There are hundreds of procedures and processes, but these six things we believe. If you do these and you get these built into your culture of your organization, that people can remember them, they can focus on them. It’s not too big. Saying, “We have to implement all of ITIL,” is overbearing. “All the ISO standards”- overbearing. When you say, “Hey, our organization is going to implement these six fundamental things,” and they become the pillars of your ability to operate… These are the things that we believe are important.

Just a little bit about the company. Some of you know me and have worked with me before. Those who don’t, I look forward to that potential opportunity. We are dedicated to Crisis Avoidance. We want to help you eliminate the problem through the integration of Best Practices in your organization. If you have a problem, our Assessment Services are here to provide you immediate response, Technical Services to help you resolve problems, mitigate issues, and then to perform assessments to find out where you might be vulnerable. If you’re in a proactive state, if you’re looking at spending new money on infrastructure or Decision Support Metrics, we can come in and help you assess and help you build those things out. Our Cornerstone Services will come in and help you with an embedded analyst so that they can help you promulgate these types of virtual teams into those various areas and improve and benefit your entire infrastructure.

In summary, documentation, Documentation, DOCUMENTATION. Maintain a highly skilled technical staff. You may have a lot of vacancies. The prevailing thought is that those vacancies are more valuable than the average employee. In other words, it’s better to have a vacancy than it is someone who is of average. If you have average people, you need to build them into highly skilled technical people so that they can materially… Part of that is the CIO’s job to make sure that they are empowered. Have that Master Plan, employee problem management, and change control, dynamics, Decision Support Metrics that drive your portfolio spending, Business Technology Integration to make sure you have a collaborative environment.

Then those virtual teams that help you. Virtual teams are implemented without any reorganization. If your organization is ineffective, you apply virtual teams to address certain problems and pick the right people from each one of those silos and you can change your organization and improve things without even having to reorganize. Then finally, be an end user advocate. Make those end users your priority.

I’m Bill Alderson. It’s been nice meeting with you today. I respect your time and energy. I appreciate your coming along today. If you have any questions or comments please email me at [email protected].

I appreciate you stopping by today and we look forward to talking to you in the future.

I’m out, Bill Alderson.

Enterprise Architecture IT Disaster Recovery Plan Steps to APM Under Pressure

This session takes ideas from many IT disaster critical problem resolution IT consulting engagements and puts them together into 30 minutes on how to avoid IT Crisis. This presentation identifies IT best practices that can be used when creating an IT Disaster Recovery Plan, particularly if the disaster affects application performance. Your staff will appreciate the references on how to troubleshoot network problems during an IT disaster.

Enterprise Architecture defines a multi-site, multi-vendor network with multiple computer operating systems integrated by the benefiting institution, supporting multi-tier mission critical applications, having its own monitoring and support organizations. Compare your internal processes to the set of best practices known to obviate crises for inclusion in an IT Disaster Recovery Plan

Transcript for this Video

Good afternoon. This is Bill Alderson. I appreciate you joining us this afternoon for just about 30 minutes or so, as we talk about six things that you can do. In many, many years of performing critical problem resolution, going in when nobody else was able to solve it, and high stakes situation, high complexity, lots of people waiting for you to fix the problem type of thing, my mind always goes to the end of the engagement when everybody stands around and says, “Oh, that was a pretty simple solution.” Typically, there are multiple things that go wrong in a crisis or a critical problem. When you go back and reconstruct or do the anatomy of a critical problem, you always find that the root cause is some technical problem, of course, typically or configuration related or something of that nature.

After we're done doing the critical problem resolution, everybody says, “Well, why did that occur? How did that problem get us to this debilitating point? How can we prevent it in the future?” That's been my guiding light to help people, yes, go in and help them when they do have a problem. We do that. If I can obviate problems through best practices, the things that you can do to avoid a problem, that's much better than having a problem that you have to resolve. I think we'd all agree.

Today I'm going to talk a little bit about six of the things that you can do. Of course, there are hundreds of things that we can do, literally hundreds. There are best practices, ITIL, standards of various types, certifications of various types, and all sorts of things that we can do, but we can't remember all of them all the time, every single day. I'm going to bring to you six of the things that you can do to avoid crisis, to avoid critical problems, and to avoid problems in general. That’s what we're going to talk about.

This is our website here on the top left. You'll see we've got a variety of services. We've taken the Jefferson Memorial and we've used it and the pillars in the Jefferson Memorial to illustrate our services and the steps to illustrate other services and that sort of thing. If you have a chance some time, you ought to hit that “Start Here” button on the front of our Apalytics web page and go through and just listen to see how the web page works and that sort of thing, and to look at our collaboration systemization model. It's pretty cool.

Then over on the right we've got CIO Tech and we’ve got SynSynAck.com. Syn Syn Ack, of course, is the TCP three-way handshake and we talk about performance analysis and that sort of thing. Then CIO Tech, we talk about things that are of interest to the CIOs. That's a little bit about us.

Little bit about me, I started as a communications engineer at Lockheed Missiles and Space Company. If you read Steve Jobs' book, not his book, but the book by Walter Isaacson that just came out, which I highly recommend because it really does a good job of explaining how Steve Jobs and his company, his problems, the adversities and that sort of thing drove this guy to build some really excellent products that the market really loved. In that book, he talks about how Lockheed, Fairchild and the other companies in the Silicon Valley, the defense money, and the defense contracting that was in there actually spawned the high tech industry and chip building and that sort of thing. Of course, I can remember when we did multi-layer boards over there and wire wrapping. That was the beginning of printed circuits. Then they just kept getting smaller and smaller.

I joined Network General in the mid '80s, and we started working on the Sniffer. That's when I really got into protocol analysis and critical problem resolution. It was in '86, '87. Subsequent to that, I started Pine Mountain Group. Some of you may know me from that. I created the Sniffer Training program. I licensed that to Network General and they trained thousands of people. I also crated the certified net analyst program and I trained about 50,000 people in 22 countries, of whom we certified over 3,000 people as certified network forensic professionals. During that time, and in my career, I've been able to provide service or training to 75% of the Fortune 100. I've been around in a lot of various environments.

I sold Pine Mountain Group to Net QoS, a performance management company, in 2005, where I was one of their technology officers. Then Net QoS sold out to CA Technologies in 2009. I hung out for just a little longer than the founders did. Then I decided that I had another company in me so I started Apalytics Corporation to help people solve problems and learn from those problems to provide and implement best practices so that we could obviate those problems.

Here we are. Six things we can do. Six things we can remember. I put this under technical systemization. Technical systemization involves things going all the way back to the IETF. The first RFCs were not about how technology was going to work but how we were going to collaborate to build the most important technical resource today that we have, which is the internet, interconnecting all of us and providing a platform for business and communities to develop across an electronic system. Technical systemization is how you're going to collaborate. How are you going to do all of these various things? I've taken what I believe to be the things that should be utmost on your list of things to do to avoid crisis. These are our suite of IT best practices.

It doesn't really matter which order they're in, but first up is Decision Support Metrics. People are out there buying millions’ and probably billions of dollars’ worth of network management tools, analyzers, all sorts of application agents and gizmos and gadgets aplenty, to basically try to figure out what's wrong or to be notified about when things are going wrong, and that sort of thing. I am finding that even though we've spent an inordinate amount of money and time and energy, that few people are really using the tools.

The proclaimed experts today are the people who install it and keep it updated. Well, that is a very important part. You have to install the equipment, keep it updated in the versions, and then you have to have users who can go in and connect and click and get the reports. I'm finding that there are very few people who are truly doing analysis on these very expensive tools. Few are really looking at their enterprise, their environment, their applications and the signatures of their applications, the dependencies and that sort of thing.

I call this Decision Support Metrics because those metrics developed by operating system details and monitoring tools and metrics, whether it be SNMP, NetFlow, response time management to performance management, agent tools that are inside the actual platforms, cloud based statistics, etc. If you're not really looking at what your dependencies are and tuning those systems to look at your specific applications and your dependencies, and I'm going to talk a little bit about problem management later, but integrating that with problem management, finding out what problems are causing you issues the most, it's a very important thing to take and use those systems and turn them into Decision Support Metrics.

Decision Support Metrics means that there's actionable information in those statistics. You can tune your pages and your systems to go out and say, “Okay, I've got this application that we really need to serve the application owners here. We need to really monitor this application. When people have problems with it or there's capacity issues, we have the statistics and a base line and we're watching each one of our major applications so that we know what the next step is going to be.” It's very important to do that.

Here are some of the things with Decision Support Metrics. First of all, you've got the desktop and then you've got the server, you've got all of the various components behind the server which is the applications, and the virtual platforms or the cloud platforms, the databases, the multi-tier capabilities. You have to watch all of that. You really need to architect around those sorts of things. Look at those application data flow and dependencies.

In order to be able to really monitor well, you have to know where your test points are. Back when I was at Lockheed, if there was a problem with a printed circuit card, our multilayer board, we had test points on the board. You could go to those test points and perform a particular measurement to see if the health of that system was up to par. You could measure it, adjust it and that sort of thing, but you had to go to those test points. Well, our large mission critical networks have test points in them if we take and document them and lay them out schematically so that you can see those test points. That’s how you know how to build your Decision Support Metric systems is because you have good documentation and you can see it.

It's difficult to do this today because there's so much compartmentalization. It's very difficult for the average technologist, who's in a silo, to actually go outside and look from desktop to server and all points in between to look at that application. There's been some slowness to train people who understand capabilities of the desktop, their servers, their network infrastructure, and their SAN infrastructure. When there are problems that are pacing or gating their system's performance, they can go and set up test points, or maybe already designed test points, to expertly monitor those various dependencies. That includes mission critical applications, your exchange servers, your mail systems, your internet. We could basically take a lot of our reactive people and repurpose them to looking at monitoring tools and monitoring systems that would eventually obviate problems, reduce the amount of reactive workload and improve the performance and reliability.

There's a problem with documentation. There's a problem with good Decision Support Metrics. It's a holistic problem that doesn't change overnight. It's a cultural issue that starts with best practices and an intention to improve as you go through. All of that's designed to help you recoup your monitoring tool investments. You've got a lot invested in those monitoring tools.

You always want to focus on that user performance and go in and do root cause diagnosis. That all ties together with another topic that I'm going talk about and that is Architecture Ownership. A lot of my customers are government contractors and large institutions that utilize outsourcing organizations or perhaps, you are an outsourcing organization, or perhaps you're a government entity and you have a lot of contractors and those contractors are changing. One of the things that I like to encourage you to do is to make sure that regardless of what's going on with your contracts and that sort of thing, that you maintain Architecture Ownership. That requires accurate technical documentation and to address that.

I'm going talk to you about that a little bit so that when you change contractors or a contract, you have a different outsourcing organization, you’re looking to outsource, or maybe you're looking to insource and change from outsourcing back to your own insourcing, you want to make sure that you don't have any limitations by indispensable contractors, indispensable employees because you own your architecture. You own your system. Now that doesn't mean you can't have a contract perform your Architecture Ownership functions, but it probably needs to be separate from your regular operations so that it's more of an auditing function.

In Architecture Ownership, what do we mean by that? There are two components of it. One is, to know yourself. That is to basically render your architecture into diagrams so that you can see your environment, all your technologists can see your environment and understand it, and your architects can work around pieces of paper.

Today, with our lights out data centers and that sort of thing, it's impossible to do this unless you have good documentation. Not only good documentation schematically and of the configuration, and that sort of thing, but also of your racks and that type of stuff because you don't have, and you don't want to have, people in your data centers. You really want all that work to be done either online or virtually or from multiple locations. In order to accomplish that, it's essential to have good system documentation.

What we believe that you need to do is pull all the router switch and server and configs out and basically put those on paper in a logical schematic type diagram. That documentation needs to span multiple security zones and be scalable. You really want a diagram that's not the server guy's diagram, network guy's diagram, storage area network guy's diagram, or desktop people's diagram. Whose is it? It’s not the application diagram. Whose is it? It is your organization's diagram cross silo. It's to put together an end to end diagram of all the pertinent detailed information, so you can put your finger on where an end user is connected, that's getting poor performance and you can move your finger through virtual circuits all the way through so that you can find the logical and the physical.

If you have a bad switch port somewhere, of course you've got redundant systems, right? Redundant systems can sometimes be a problem because you may have a bad fiber on one of those redundant connections but it keeps flipping back and forth. Therefore, you have an intermittency. It does work but when it works, it works in a degraded state. Then it flips back and forth between the two. It's important to be able to view your system all out so you can see everything.

Another part of Architecture Ownership is that I have noticed that there are literally sometimes hundreds of people who are in the IT organization who, if you really quiz them and ask them, don't have a clue of where they fit in the organization technically. It might be a server person, network person, desktop person, virtual person, the person who takes care of the SAN or the WAN or what have you. Those people need to understand, in the end. Once you have a really good document that depicts your environment, then you can take that document and you can train all of the people. “Here's our architecture. This is why we do it this way. Here's how it works. Here are the tools and systems that we use to manage it.” Then you bring all of these people out, who previously might have been hiding under their desks when there was a problem, because you've enabled them to now understand their environment. You have your network rendered so that you can see it and now you train people.

What we believe that you need to do is train people in your architecture, your systems, your monitoring tools, your capabilities, your organization, your trouble ticket system, so that all your technologists are banging on all eight cylinders. Instead of having five or six people in your entire organization that are really into it and understanding everything, you bring it up to several hundred through training. What we believe that you should do is have a certification program for your architecture.

First you document it. Then you train everyone on it and you test them and certify that they truly understand before you give them the keys to your active directory, before you give them the password for the routers and switches and platforms. They go through this orientation and certification on your particular architecture so that they are not trying to make it look like where they came from, but rather bearing the hand to help you continue to build your architecture and your system. Architecture Ownership is a very important component.

Problem in Change Management, of course, is essential to, one, diagnose root cause, but also to enable rapid continuous system performance optimization. Doing root cause analysis, of course, that's my area of expertise and that's where I get called in a great deal. We come in, we diagnose the problem and then we do root cause analysis. Usually it's had to escalate quite a bit before people finally say, “Hey, let's get somebody in here to help us solve this problem. We have smart people. We have capable people but they're busy doing other things. Let's bring somebody in to focus on this.” We come in and we help with that solution.

Well, one of the things that we do when we do health checks, is check out your trouble tickets and find out what's been pervasively causing problems. If we can basically prioritize those bad trouble tickets, we can then go mitigate the root cause of those problems and basically obviate future trouble tickets, which lowers your costs of your help desk and that sort of thing. Some organizations are so reactive that they just keep hiring more and more help desk people to be basically customer interface folks and that just keeps growing and growing so large and so wide that you can see the curvature of the earth in those cubicle areas where the help desk is. It makes a lot of sense to basically mitigate those particular problems through statistical analysis and then root cause analysis.

The first thing that you need to do is you need to do a problem statement. You need to know what your problems are. You need to reverse engineer your system documentation. Perform micro and macro analysis. Macro analysis is using your decision support tools to find out about where the problem is. Micro analysis is getting down to the packet level to diagnose the problems. That's another thing that you need in order to get the critical problem resolution and have a really good organization. You not only have to have the macro analysis, SNMP tools, NetFlow tools, performance management tools, but you also need the ability to go down to the micro level with packet-level analysis.

Another thing that is helpful, if you have a lot of applications and you're finding that you're having to go out and diagnose problems a lot with your applications and your applications are said to be slow, if you basically embed into your application development an early and frequent deep packet performance analysis and then subsequent design of the monitoring system to look at those application vital signs, that will yield incredible ROI for you.

It's our recommendation that you have regularly embedded deep packet inspection early in your development cycles and redevelopment, refreshing cycles. Rather than waiting and putting an application out there, spending $20 million on new hardware refresh, software refresh, and that sort of thing, and you go to roll it out, it's got so many SOAs and interfaces with other systems that when you turn it all on and you start using it, you find out that the end user won't use it because it's so abysmally slow because there was no performance analysis done in a micro fashion incrementally over the period of time. You need to look at doing application analysis early in your cycle.

Here are some of the things that you can do. Identify your dependencies, do micro and macro analysis, and do latency analysis on your web, your SAN, your cloud, your metal ware, your load balancers, firewalls. Find out where your problems are early on. That will help you with those Decision Support Metrics to be able to design metrics to watch those areas that you know in the future, as you scale, might become a problem.

Then Business Technology Integration. Like I said, a lot of technologists and technology managers are finding that they're in a silo, they've got operations and engineering for each different type of technology and it's hard to get an end to end view of the entire system. We've come up with some ways that we believe that you build virtual teams to basically have a representative of each silo for your Architecture Ownership, for your Problem Management, for each one of those. If you put them on a team together, you can end up having a collaborative design environment where all your silos are working together. You can have documentation that is end to end from the client all the way to the server, the application, and all the various systems in between. You can really get some traction in there. That is dependent upon a Master Plan.

A Master Plan is a prerequisite for collaborative design. You have to know what your objectives are. We like to focus you under Business Technology Integration and Master Plan Development so that you document the “is” and then help migrate toward that future.

This is our cross-silo collaboration model. It is essentially a way in which you visualize all of your various organizations and all of your various silos, cloud, your network, your desktop, all these different things, and these folks, security and platform. You basically build out an Architecture Ownership group, a Business Technology and Master Plan Development group, an Application Development Optimization. These are virtual teams. You don't have to change your logical organization or your budgeting silos. You just need to put this together so that you have the glue to help everybody understand one another’s environment in a team sort of way. Wrap all of that stuff up into your “is” Master Plan. Then you've got your “way ahead,” your “future” Master Plan so that you know where it is that you're headed.

These are the things that you can do: Architecture Ownership, Decision Support Metrics, Business Technology Integration, Master Plan Development, Cross-Silo Optimization, Problem and Change Management, and Application Development Optimization. Really good things that can be in your forefront. Just remember, take a look at that Jefferson Memorial and remember that each one of those pillars and each one of those steps are steps to your user's best interest.

If along the way, Apalytics can help you, Apalytics is dedicated to crisis avoidance through integration of best practices. Our TS is our Technical Services where we'll come in and help you with existing problems. Our AS is our Assessment Services where we can come in and help you identify the risks and then help you develop mitigation strategies. Then, finally, our cornerstone services which help integrate and obviate problems through the expert integration of best practices.

I appreciate you joining me today. We appreciate you spending your time with us. I hope it's been useful to you. If there's anything that I can do to help you or if you have any questions about this presentation, just go ahead and email me at [email protected]. I appreciate your help in building out our industry, helping us all grow and understand what we should be doing. Apalytics. Application Analytics Software and Services. We're at your service. We appreciate you and your joining us today. Take care.

APM Application Performance Monitoring Tools in the Theatre of War tells the story of how application performance management tools were deployed to analyze and report on key inteligence biometrics applications throughout US CENTCOM Area of Responsibility, and CONUS back to the FBI IAFIS and Biometric Fusion Centers.

Transcript for this Video

Hi and Good Afternoon. This is Bill Alderson, thank you for joining us. I’m going to do a presentation on Network Application Monitoring Tools in the Theater of War. This is a presentation that I did at the MilCIS conference last year in Australia. It was a lot of fun, going Down Under and talking to some of our military counterparts down there about network management- Had a good time! Anyway, appreciate you joining us today.

So a little bit about me, I consider myself an application and network performance analysis advocate. I know that end users are the most important folks in my career. I focus on working toward their benefit in everything that I do. I started out at Lockheed in Sunnydale, California, the heart of the Silicon Valley, and was there from 1978 to 1984. I was a communications analyst. I was a young guy that loved communications, and was playing with every kind of computer that was coming out at that time- IBM, Apple, others.

I was right in the middle of it. It was a lot of fun. A little later on, I got working with Network General and the Network General Sniffer, at the start-up. Then I took my experience with the Sniffer, and I founded Pine Mountain Group, and created Network General Sniffer Training, which Network General licensed from me, in perpetuity. Then Pine Mountain Group, we trained about 50,000 people in 22 countries and certified over 3,000 network forensic professionals.

I’ve done a lot of work with 75 of the Fortune 100, federal and state government agencies. I did a large event for Net World InterOp, called Network Forensics Day. I sold Pine Mountain Group to NetQOS in 2005, and continued on as was the Technology Consulting Officer. CA then acquired NetQoS, and I hung around as principle services architect on CA Application Performance Management services products. Then just this last year I founded Apalytics Corporation. I’ve got several customers at Apalytics. We did a sub-contract to U.S. CENTCOM, where we went and did analysis of pretty much the entire war area.

That’s what I’m talking about, and some of the things I learned through that experience, of looking through every network control center throughout CENTCOM’s AOR, in all those various countries. Then working with those systems and helping to upgrade and architect, document their systems so we could perform better network management, etc. I’m also consultant to the OSD-CIO, at the Pentagon in area communications. I also provide services to the Department of Justice. I’ve got a pretty wide variety of experience. The talk today, is about network management, but in order to understand network management, one must understand everything from end to end.

From the client, where you’ve got the user, the business transactions, the security access, the application interface, the operating system, the computing platform, the OSI model. The seven layer model that brings it down to where you can take all of your data across disparate networks and systems, and then have it arrive at the server and get error checked and that sort of thing.

We’re basically looking at the full spectrum of everything end to end, from client to server. Well, one of the things in the war area is that there’s a lot of different people, and a lot of different organizations, that are in charge of different packets. From where the war fighter puts the packet on the wire and until it gets to the server.

So consequently, in that traversing all of these different network management zones, and different countries, continents, going up to space and back, these packets are really mistreated. Everyone along the path has their own SLAs, their own network management systems, and they’re all different. By golly, if you go through and you talk to anybody, all the contractors and everybody involved, everybody did a wonderful job, and all met their SLAs. Everybody inside those silos is very happy- with one exception, the end user.

The end user is going to cross all these dissimilar systems, and when he’s got a problem, where do you go to capture the packet, if it doesn’t get there? Who do you blame? Who do you call up, who do you talk to? The end to end analysis that I’ve done from all those areas, back to CONUS for different applications and that sort of thing, basically said, “Hey man. If you’re an end user, you have no advocate. You have no centralized advocate.” It’s very, very difficult. You can call your local guy and he’ll do local things, but he always comes back and says, “Must be somewhere else.” Now your end user is in trouble. So those end users are usually the ones that contact me. Powerful, high visibility, high stakes end users, usually call me up and say, ‘Hey.’ That’s exactly what the military did. They called me up to have me help with a big application, and we went out and looked at all their network management and said, “Well, you don’t know where any of your packets are going. You need some NetFlow information.”

We implemented NetFlow across the entire AOR so we could see what applications were running and where. In particular- one biometric application, it was very interesting; it was awesome to analyze this application. I helped the programmers actually recode several parts of their application to improve performance and that sort of thing. It was a really great thing. So network management does not just involve the purchase of some network management tool then turning it on and installing it. It involves understanding the clients that are accessing the applications that are going through it. So you can fine tune it, nurture it. So you can find the signatures of the vital signs of those applications. When somebody tells me about network management and oh yeah, I’m a network management guy. Well, do you know applications? Do you know the issues on the client? Do you know the issues on the server?

Do you see the virtualization issues? Do you see all the man-in-the-middle, all the firewalls, all the WAN optimizers? Do you see all the load balancers and that sort of thing? Do you understand how those things come into practice? Do you understand the quality of service across that end to end network? “Well, I know everything there is to know about this network management package. I can install it, and I’ve installed it.” Yeah, but in order to install and use these systems, you must intrinsically understand what it is you’re trying to help the clients achieve; and how to measure the performance of those systems all along the way. You’ve got to document your systems.

I’m going to talk to you a bit about, after we get off this slide, a number of things. We’ve got a lot of slides to cover today. But we’re going to get there very quickly and give you a cursory overview. Here’s your basic client to server, so the client is separated from the server. We started doing that back in the late 80s. You took your client and your applications, you run it across various stacks, using HTTP or Java. You go across your transport network, you come up the other side, you go into your server. You pop back up your stack and into your respective upper layer system. Then you go into your applications and processes on your server.

You have to draw this out. You have to know where your connection points are. You have to know where you are in the universe. If you’re having a problem somewhere, you need to know exactly how you’re going to hook in, where you’re going to hook in, what kind of tool you’re going to use, so that you can deconstruct the problem.

Now, people going into the cloud and that’ll be another topic that we talk about. Cloud computing can be very problematic. Because, with the client and the server, you own all of the infrastructure; but when you put that server over in the cloud you have no access to packets, network management information, statistics, etc., other than what that vendor gives. It’s going to be a cloud all right. So basically, I’m trying to help you understand that when you go in to analyze your environment, you need to know from end-to-end.

Here’s a client talking to an app server or an HTTP server. The back end is talking SQL or some other type of middleware or back end process. Well you have to know where to go to connect your network management systems for packet capture, for metrics, for analysis. So that you can understand, and if anything goes wrong, which it will at some point or another, and you’ll need to get macro information about the use of the network, the use of the platform, the CPU, etc., of all the different systems. You need to see the entire system from end-to-end, and understand all the various components so that you can basically design the vital signs for the applications and systems. It’s not as simple as just being, okay, I’m monitoring layer three of the network. You have to be a little bit higher than that, you have to help your customers. That’s my definition of network management.

Now typically at one time or another, whether during implementation or later on, you’re going to have a problem. You’re going to be looking for that proverbial needle in a haystack. Here’s your needle in a haystack. It’s out there, it’s somewhere, let’s take a look where it might be. Hmm. It could be anywhere in Afghanistan, Iraq or some point in between, or over somewhere in the United States or down to Fort Watuka. Your packets are traversing all of these various environments. Where is it? What’s slowing it down, who’s holding it up? Who’s routing it. Who’s misrouting it? Who’s redirect it? Who’s changed it in its path because of a WAN optimizer or load balancer that you were unaware of? Those are your men-in-the-middle, and they’re sitting out there impacting your environment. So you need to know your environment.

Then every once and a while you’ve got to gather some packet traces. You’ve got to gather them from somewhere in between where the user is and their resultant service. So it could be anywhere. We usually go and gather packet traces. We gather packet traces based upon the macro information that we get from our network management system, to tell us and help us triangulate where the problem might be. That’s what we use network management for, and then we’ll go out and we will capture some traces and then get down to the bottom line. I’ll just tell you right here, and you can all argue with me, you can fight with me. If you were a network management person and if you are implementing large scale systems, and you have no ability to capture packets across your path- You are impotent. You can’t do it, you never will be able to do it. I’m telling you if you don’t have bottom line, root cause deep packet analysis skills and capabilities, and the ability to capture those packets. And you are impotent and you cannot do your job. Sooner or later you’re going to come up against a problem that absolutely requires deep packet captures. You design your network management systems to help you find the area, and then zero on it. Can you solve many problems with network management systems and capacity? Absolutely. Can you solve every one? No. It’s not a very pretty picture when you’re standing around with several million dollars’ worth of network management tools, giving you charts and graphs and all sorts of wonderful views of stuff, and you can’t solve the problem.

It’s because you don’t have deep packet capture capability. You don’t have the root cause analysis capabilities. That’s what I focused my whole career on, trained 50,000, and executed for 75 of the Fortune 100. To help them solve these types of problem. Then some people say, “I like packets so much I’m going to capture every single one and I’m going to store them forever.” Then I ask, “Okay. You’re going to store every packet everywhere? You’ve got unlimited amounts of money and resources and storage? Who’s going to analyze those packets? Who’s qualified within your organization? How are you going to filter in, and have the worst offending problems bubble up? What’s your strategy for using these systems?” I know a lot of folks have a lot of this sort of stuff up in their environment, and they have all these tools and capabilities, but nobody ever goes and looks at them. “Oh, but they’re there in case we need to do retrospectives.”

That’s good, and I’m not saying you shouldn’t have any of it, but I’m saying that you’d better make certain that if you’re going to buy this stuff, turn it on and start caching packets all over the world, that you at least ought to have somebody who’s around who can help you take a look at them and know when and why. So when you finally get down to the stack with the problem, now you go to work in deep packet analysis, to deconstruct in order to ultimately, boom, find the needle in the haystack. That’s what it’s all about, is getting to the bottom line. Definitive results. Performance Orientation. Getting the job done. No ifs, ands, wells or buts about the situation. In order to get all that done, we use ITIL , we use People and Processes, Paradigms, tools, systems, platforms. We use all of these various systems. It’s not just network management. It’s not just deep packet analysis; it’s not just application optimization. We have a whole portfolio of capabilities. As I’ll talk about in some future, I have a list of these things that I believe are the most important and we’ll talk about those at some point. Actually, on the 30th of the month we’re going to talk about that.

It requires a seamless integration of people, processes, knowledge and technology. It’s a well-rounded organization. I’m going to talk to you a little bit about what I call my network management servo-loop. Over on the left, you’ve got a website, let’s say. You’ve got an application, and you want it to perform. Let’s just say that you want it to perform in a way that allows that webpage to come up within 15 seconds. So what you do is over on the left hand side, you say I want the maximum possible performance to accomplish this business objective. I’m going to set my command, my desired response time, to be at 15 seconds. So that goes into the network management servo-loop. This is a feedback servo-loop mechanism. If you haven’t seen this before, you can look it up on the internet and see some others. But I have taken this and adapted this, because it’s a feedback loop, and it’s a servo-loop, it’s a closed loop system. Where you say, this is what I want and expect from the system. Then you have a resulting actual user experience that you compare to what it is that your objective is. You come up with a signal deviation.

At the top left there, you have what your desire is, 15 seconds. You then have your feedback signal, which is let’s say it takes 25 seconds. You have a signal deviation of ten seconds. You are not meeting your SLA or your desired performance by ten seconds. So what do you do? In most environments, executive management thinks all the technologists have it all figured out, and the technologists think that the executive management have it all figured out. I’ve got news for you. I’ve been working in this industry for 20 some odd years. Neither one of them really knows, nor have a systematic approach, until you get them into the servo-loop. Let’s get the executives in there. Let’s show them some metrics, show them some business metrics.

When you talk to an executive about their business metrics and how it coincides with the network and the application metrics, they start listening. You don’t lose them in presentations; you have their full, undivided attention, because you’re talking about deliverables for the business. I always like to get metrics, feedback sensors. Metric are a vital part of the business. Performance indicators. End user reports. Right? You get your executives involved, because why? Because executives control the policy, procedures, the resources, and the entire organization. They need to be involved in this. Okay?

You take a guy like Steve Jobs. He was involved in the process. He understood his company’s technology. Then you take another guy like John Scully, and you put him in there who’s selling sugar water for years. He’s a bottom line dollars guy, how much can we make? But in technology, you have to have someone involved in the technology. If your executives are not involved in the technology, it’s your job as the CIO, or the executive technologists to get them involved in a meaningful way. The way that you do that is you pop in some business metrics and compare them to some of your technical metrics, and you’ll bring them right in. Then they’ll get involved and help you solve the problems that you have by allocating resources, priorities, focus, policy and budget.

Then you’ve got the actions. The actions are carried out by the technologists and the technology folks. They go out and they apply. They’re the tinkerers; they’re the guy behind the curtain, changing this knob, changing that knob. Optimizing and installing and improving the environment in the system. After executive management puts the controls on there, the technologists apply those priorities. Then we should measure the actual user experience again. We measure that actual user experience, it comes back, Okay. Then little by little, between the executive function and the technology functions, we start to move the servo-loop toward the maximum possible performance. We keep tweaking it, but it does involve executives & management, and it does involve the technologists. If the executives are not involved, you won’t have the type of priority and prioritization and resources necessary.

Okay. Now let’s talk performance indicators. Well, these are the three main performance indicators that I advocate for. You’ve got network flows. Network flows are from the left, you’ve got clients and you’ve got an IP address, and you’ve got a socket. Those are the application sockets. If you know every packet that goes across your network, because every router in the network is reporting on these network flows. You don’t need another gismo to pop in there. Those routers perform the actual task of looking at all those PCP connections, from IP address on the left at the client, and IP address on the right at the server, and the application ports that they’re using between those two. Then they tell you and record the rate and flow volume information. I can tell you if an application is or isn’t working well, and how many people are using it. Why? Because I have rate and flow information, and it’s coming from my routers, and it’s going to a database, getting in the database, and I can go back and tell you what applications are on your network. That was very interesting in the war environment, because there were certain applications that were consuming large amounts of bandwidth and others that were consuming hardly any bandwidth. Some of them were being accused of consuming bandwidth that they weren’t. So when you really go out and get the facts of who’s talking to who through whom, that’s bottom line. That’s what net flow does. Who is talking to whom through whom?

Then you’ve got rate and volume information, you’ve got capacity planning, you’ve got all of those sorts of things available at your fingertips. Net flow and network flows is a very important thing. It tells you where those applications are running around the world, and who is running those applications by location. And, the volume and the rate can be compared, and tell you what type of performance they're getting. If you understand the theory, you understand the network management. So again, this is not about turning up a product. This is not about clicking and installing a product. This is not about buying it, installing it and having it there. It’s about interpreting it, it’s about looking at it, and it’s about architecting it. It’s about understanding the applications; it’s about understanding the vital signs. And it’s about trying to figure out what’s going wrong and where, and being able to triangulate that and being able to solve those typical problems.

Okay. So then you’ve got standard SNMP, that’s device status. Standard SNMP just tells you a router’s overloaded with CPU, or a circuit is overloaded or an interface is overloaded. It’s very good information. But what if you don’t even know the path that your packets are taking? You’ve got to know the path your packets are taking. If you know your path, then you can exploit device status, to find out what devices on your particular path are impacted by memory, by CPU, by other types of capacity limiting characteristics. The first thing is, you’ve got to know your path. Well, there are not very many tools out there or capabilities at layers two and three. You can give a layer three path, but it’s hard to get a layer two path as well.

You’ve basically got the network flows. You’ve got device status, and then third, response time. Response time monitors, there’s several different flavors. AuthNet has some really nice stuff. NetQoS, which was acquired by CA, has some nice stuff. There are a couple of other players in the field that I don’t have a lot of confidence in quite yet, but it’s growing. The performance marketplace is no longer about red, green, up, down. It’s about, ‘it’s slow’, ‘something’s wrong’, ‘we don’t know where, it’s working, but not very well’ and ‘it’s impacting our end users’. So you put these response time monitors, which basically capture the packets or listen to the packets as they go into the server. They timestamp it at the server, and they say this packet response time took 100 milliseconds.

You can put timers on these things and thresholds, so that if your response time is slow, you can automatically trigger a trace file. You don’t have to wait until 2:00 in the morning. You can actually have your response time monitor over monitoring a bank of servers, and when the response time gets slow it automatically triggers a packet capture, so your technologists can go back in the next day and do retrospective analysis on the packets that were slow. So do we care if everything is fast? Do we want to capture all those packets? No. What do we want? We want to trigger on the events that are poor, and that’s what these systems do. They record the response time, it’s an awesome system. It also gives you the network round trip, because the [inaudible 26:54] come back and it records that so you can see. One of the problems that we had was satellite delay and multiple router hops adding satellite delay. It’s about 700 milliseconds round trip across a satellite. Well, when I see a response time that says it’s 2.1 seconds, I know that that 2.1 seconds is because they’ve gone across three satellite hops at three times 700. You can figure out and start to surmise what’s going on, and then you can look at optimizing your routing.

These are your tools and your key issues. You’ve got your performance management system. You’re getting your performance metric, the feedback, and your business metrics. You’re getting your trouble tickets; you’re getting your information back. Some of the things that you need to look at are route analytics, to identify instability in routes, server response time, net flow base, and rate and volume information. It gives you hints on conditions limiting user experience. Also look at SMNP device status. Well, you’ve got to know the path in order to know what devices are in that path, and because you may have hundreds or thousands of different devices. Knowing what your particular path is and exploiting that path, to see if anything in that path is failing. Those are value added steps. Then of course packet capture at key locations, so you can get down to root cause analysis.

Here’s a network performance management architecture diagram. There’s a portfolio of tools. You’ve got Remedy Trouble Tickets; you’ve got Opsware, ArcSight. You’ve got a large number of different systems-Packet Design's route analytics, CA Application Performance Management NetQoS’ response time monitoring, and NetFlow tools, , OPNET's Network Diagramming capabilities. There’s a portfolio. It’s a Swiss army knife, all working together. You can’t use just one company or one suite of products. You can reduce that set as much as you possibly can. But you always want to make certain that you’ve got every best of breed and aspect of your system. This is where we went out and we installed systems at 15 different sites across Afghanistan, Iraq and areas in the United States so that we could have NetFlow and response time monitoring. This is how long it took us. It took us one year to install. This was pretty much unheard of, and the ability to get this system up and running and actually do analysis on major applications in fewer than 12 months, inside a war environment, was unheard of. We had an incredible team down at CENTCOM out there. The war fighters, man, they were working their buns off to try to help us get all of this stuff, so we could figure out what was wrong with this biometric application. This biometric application by the way takes fingerprints and iris scans. And it looks at latent prints of the guys who were building bombs and that sort of thing, those IEDs. That application was helping them put those guys on an alert list.

A war fighter would have a bad guy walk through, or people would be walking through their security station going into Fallujah or into Baghdad or whatever. And they would immediately pick up on the fact that thousands of people were coming through, and we just found the guy who had a latent print on an IED. This guy is now caught. That’s how the surge actually worked in my opinion, was because we had electronic means to zero in on who’s a bad guy and who’s in proximity and who’s not. This application and the analysis of and the fixing of this application really helped the effort over there. We found route changes, due to packet loss, slow server response time. We found problems with TCP offload engine. TCP offload engine is where the TCP stack is moved from the server, inside the server, using the server’s CPU and memory, off onto the network interface card. There was some problems, and I’ll show you a couple of slides here on some of the examples of this sort of stuff. Of course, this took many weeks and months of analysis after we had all this stuff put together. But we found all these problems and started mitigating them, and things improved. And that application I was telling you about was even more effective in performing.

Across the network there are qualities of service incongruities. Network and application issues requiring packet level capture, we analyzed a lot of stuff. Here’s a few screenshots of some of the stuff. Here’s some route change analytics going on. Here’s some satellite retransmission, took three and a half seconds to retransmit across many of those satellite circuits. This is a detailed analysis of that sort of stuff. Here’s a TCP offload engine recovery issue. These are all commercial, off the shelf stuff. In the U.S. or other parts of the world, these particular systems are not always found to be wanting in a low latency environment. But you put them in a satellite environment with high latency, and manifestations of severe problems start to show up. That wouldn’t show up otherwise. We found TCP offload engine problems. We also found WAN optimizers and load balancers and other things, doing weird and funny things. I called them our own man-in-the middle. I developed a method by which we could identify that man-in-the-middle, and this is some of the analysis associated with that.

Then processing analysis.- How long it takes? Over on the left, you see processing, processing, processing. That was before we optimized the application. Over on the right, the same transaction only took under two seconds instead of 18 seconds. You add those types of optimizations up and you’ve got a lot of optimization. Then there was data duplication. What does data duplication mean? Data duplication is when you have a problem with that TCP offload engine where you’re sending the same data multiple times. This is an example of where the data is being sent across the network. So not only did you have clogged pipes, not only did you have networks that were saturated, but now you’re sending multiple copies of the same data. Here is an example of that packet loss induced TCP offload problem, and the wasted bandwidth associated. You’ll see a red line and a green line. A red line was the wasted bandwidth; the green line is what was normally required. But in this particular environment, because of that TCP offload engine and the packet loss that was associated with it that triggered it or was the catalyst.

See, if you had no packet loss, a lot of these problems don’t manifest. When you do have packet loss you end up with three and a half second response times. You end up with data duplication. There are a lot of things that are exacerbated, because in a war environment you are not dealing with perfect commercial communication systems. So bottom line. Multi-tier identification of your environment; you have to know and document your network, and then you set your test points up. Your front end tier, your middleware tier, your SQL tiers; you have to know how to find and design your network management and your response time monitoring and NetFlow systems. In order to do that, you have to document very effectively. Then you can instrument your front end, your middleware tier, your back end tier, and your mainframe tiers. You can then take and instrument your systems.

This is basically showing test point two, where we pulled over to a super-agent to do response time monitoring, with the ability to capture those packets like I talked to you about, when response times get slow, it would automatically capture packets at that location. Pull it all together, what do you have? A user clicks on their screen on the top left. You see where it says ‘user click?’ Then it comes down. Over on the right, the different colors, red, green, blue, those represent processing times, network serialization, transport and switching queue times. Then, of course security authentication- How long does it take to authenticate the user to be able to access that data? If you take a gander up at the top left, he clicked, and on the bottom right the user finally gets the information. Did that take 15 seconds or 25 seconds or 71 seconds, what have you? Then basically take it with the next step, and you bring it down to the various tiers. The process, where is it? We fixed the process problem in the application, reducing the response time. This is an example of what some of those response time monitors look like over there. You’ve got your web tier monitor at the same time as your app tier, your SQL tier, your mainframe tier and we found the process at the app tier that was causing a lot of issues, and we helped resolve those.

I know I’m going just a couple minutes over today, but I just wanted to finish this out. The bottom line is, in order to find that needle in a haystack, in order to make certain that your users are productive; you have to design your systems not to find problems necessarily, but to nurture and build a system whereby you are obviating the problem. In other words, you are fixing the problem because you have such good intelligence on the information. In ‘The Art of War,’ the first thing that’s said, is know yourself. Know Yourself. Document. Know Yourself, Monitor Yourself. Know where you are, so that when you do have a problem, you can go over and you can find that needle in a haystack, using scientific, automated capabilities. The future of network-centric warfare is dependent upon having these types of capabilities.

Just buying a bunch of stuff and installing it is not what is needed. You have to have architects who are looking at this, who understand the big picture, and who can solve problems in all manners, from client to server and all points between. Anyway, if there’s anything we can do to help you; we know how to do it. And we’re at your service. It was good being with you today. This is Bill Alderson, signing off. Appreciate it.

Complexity of WAN Optimization, Application Acceleration devices makes analysis of problems and root cause analysis more complex. Yes, they are beneficial, but if you have an issue only the manufacturer has the ability to fix the problem. Sadly not much motivation exists for them to help you determine that the esoteric problems you have manifesting are related to the devices.

Transcript for this Video

The four major areas causing IT Complexity Acceleration.

1.) Technology 2.) IT Silo / Dept Growth 3.) Personnel Specialization 4.) Out of Sight - In the Cloud!

Hi, I'm Bill Alderson and welcome to a little session on IT Complexity Acceleration.

Thanks for stopping by today. I'd like to talk to you about the issues affecting IT Complexity and its Acceleration. First of all what do we have? We have technology. Every area of technology is getting more and more complex. Now sometimes it makes it better; less problems and that sort of thing. But there's a lot of new technology out there that's actually making things much more complex- Multi-tier applications SOA interfaces. That's interfaces to other people's data that you're bringing into your application.

You've got WAN optimizers you've placed in between your client and your serve; additional devices that are taking the data, compressing it, encrypting it, optimizing it for certain applications. For certain applications it's on, for some applications it's off. Sometimes it works well, sometimes it doesn't work well. But let me tell you when there's a problem there's a problem and it requires detailed analysis.

Why do you think that Riverbed, maker of WAN Optimization, Application Acceleration appliances bought all of the developers of Wireshark? - Because they had to! The complexity of the WAN Optimization, Application Acceleration technology is vastly accelerated beyond native client server architecture. Used to be you could see every router, every switch, and every device between the two. Now you put in these very quiet WAN optimizers. You also have other things like load balancers. Those load balancers go in between the servers and the clients. So when you do a TCP connection you connect up to the load balancer. The load balancer then sends another request on a separate TCP port, another OSI model and goes to the server.

So you're really not interacting from client to server. You are interacting from client to WAN optimizer, WAN optimizer to load balancer, from the load balancer to the server- vastly more complex. In addition, we have on the network interface cards and all of our devices; we now have TCP offload engines in order to be able to do more with the processor on the board and take away from the processor in your CPU of your hosts and of your clients. So the TCP offload engines are taking that and putting them in the offload. That's yet another set of complexities, another set of dynamics to where you're not truly communicating from the client to the server.

We're in a day and age where we're buying lots of monitoring tools and systems and equipment. We have a plethora of men in the middle. I call them "Our man in the middle", those load balancers, and those TCP offload engines. Then you have firewalls. Sometimes the firewalls are actually taking sessions from the client to the server and in between changing TCP options.

Well all of these things are good, but, if you don't have analysts who can really look at this data when something goes wrong, you can't really troubleshoot until you get the next round of what?- Forklift upgrading. That's what the vendors are counting on, typically. We're doing very little to optimize what we have today. Everything seems to be having to be attached to a new technology before we'll do anything. So there are many free options to optimize, to tweak, and to improve the relationship between the client and the server. Those load balancers, those WAN optimizers and all of these different things we can optimize if we take the time to do it. And we can get huge ROI, huge bang for the buck by using brain cells to analyze things instead of just budget to buy new stuff.

Your IT silos, your departments are growing, your budgets are greater, there's more complexity, there's more dependency. Because of those greater budgets we have more differentiated silos so we're diversifying our budgets even more. In between those silos are walls that go up. When there's a problem from client to server and you've got the desktop, the firewalls, the network, the load balancers, the WAN optimizers, the servers, you've got the applications and now when there's a problem- who do you go to?

There are very few generalists who know and understand your entire architecture. In fact that's the next topic here. We've got personnel specialization. All of our folks are becoming like doctors. They have more and more vertical specialization so when you have a problem that's from client to server and somewhere in between you have to assemble a huge team of people to basically address all the various issues associated with that technology. There are very few people who really understand from client to server and all points in between. You have very few generalists today and a lot of knowledge is hidden from them because the silos are defensively hiding information from one another so that no one can accuse them of a problem. So it's a growing divided support group problem because of personnel specialization.

And last, but definitely not least, an emerging growth trend is the move to the Cloud. What do we have? We've got remote data centers; we've got data centers that are being consolidated. We have fewer of them and they're further away from us. They're lights out. There's less knowledge, less intimacy and less interaction with the technology. It's a separated support group from who usually takes care of those things. It's a lights out operation. Everything has to be done virtually. And if it has to be done virtually, that means documentation must be at the very highest ability and highly collaborative; which is more than likely a problem, again.

So here are some things that will help you with this IT complexity acceleration. Number one: Take a look at Architecture Ownership. I have some pieces I talk about in my best practices about Architecture Ownership on our Apalytics’ website and Apalytics’ YouTube Channel. Number one in Architecture Ownership is ‘know yourself’. That means that you document your systems, you document your network, you document where those load balancers and WAN optimizers are. You document both sides of them so people can see how their packets are being treated. You document your net QOS, your quality of service metrics. Everything is documented so the technologist who's trying to look at an end-to-end communication session knows exactly what's affecting their packets and their systems.

After you have documented your system and it's very good and it's highly collaborative you need to train everybody and certify people on your architecture. Don't let anybody into your sight or don't let anybody have the keys to your servers and your routers and your switches and that sort of thing until they truly understand your architecture. That is only done by showing them the diagrams, showing them your architecture, showing them the packet flow, showing them the application flow so they are fully understanding of the client to server before they get in there and start tweaking things and changing things.

That change that you're having problems with, I know, you don't like change anymore because it imputes problems into your environment. I totally agree. But if you have people who understand all the pieces of the puzzle across all the business silos and understand end-to-end transactions you're going to be much better off in that capability. Architecture Ownership- take a look at my Best Practices on it.

Second: Decision Support Metrics. Decision Support Metrics allow you to find out if I upgrade that server, if I upgrade that database, if I upgrade that network connection, if I update that SAN, what is it going to do exactly for the response time of my mission critical application? Is it going to go from 25 seconds to 18 seconds? Is that what it's going to do? If you don’t know, then why are you spending millions of dollars on it if you don't have quantification? Maybe you don't have any Application Performance Management, APM, in your environment? Maybe you've bought all to those tools but they're actually not working? Maybe your help desk can't even use them? So you can buy tools and you can buy systems but if your technologists are not using them…. So they may be bought by one of your IT silos but not shared in the other silos so that they can gain the benefit of the understanding. Your tools should have an end-to-end cross technology, cross silo view to them.

The other thing is your silos. There are walls are growing around them. Business Technology Integration: the building of virtual groups so that you basically take a look at network documentation and metrics across the entire enterprise- not just within a silo. So right now if you go out to your silos they all have documentation but nobody has typically an end-to-end view of how the end user is affected. All these silos are being plotted and they're doing wonderful work and that sort of thing- but if you talk to the end user community and the business unit folks who are impacted by slowness and intermittent problem and that sort of thing you get a different picture, right? Okay, I know it because I hear it, I see it, I feel it, I talk to people, and I get called in to solve these problems.

You need virtual teams to deal with documentation, metrics and optimization of your environment. Now you need a Master Plan so I've got some stuff on Master Plan Development on the website- go take a look. And lastly, Application Optimization during development and operations: You need to take a look at your systems from end to end. Find out what your transactions are doing. Tweak your metrics so that they are pinpointed and getting signatures of your application performance.

Well that's enough for today about IT Complexity Acceleration. Join our communities. Let us know what you think about these videos and the information. Really these pieces of data are valuable if you take a look at them and you think about them and you integrate them into your mind and your thinking- you'll be a very successful IT Executive and Chief Information Officer.

I appreciate you stopping by. I'm Bill Alderson and my very best to you today.

IT Network Redundancy - Just Another Single Point of Failure (SPOF)? Trace by reverse engineering the Network logical packet flow to find out!

What is redundancy? Network redundancy attempts to reduce single point of failure (SPOF) by having two or more active-active, active-standby or hot standby components whenever possible. Most try for Active-active where both components are continuously active providing load balancing features and fail over to one another in the event of failure. The trouble is, sometimes two items are installed and operating, but when one fails, part or all of the service goes down. The reason is that despite having two components, logical dependencies having complex configuration require rigorous testing and high theory engineering. The result is that some just have double the dependencies at twice the cost producing a lower MTBF because of the multiple components - not to mention the support complexity. The key is rigid documentation and test labe case studies - resources most environments don't have access to. The video describes a visualization of a reverse engineered system tracing application packet flow showing that instead of one set of single points of failure... any of the redunanct componenets that might fail would bring all end user functions down.

Effective network documentation of network application flows and application characteristics uncover the truth about redundant components on your enterprise data center system architecture diagram. Wisely consider logical transaction and packet flow when preparing an IT disaster recovery plan - particularly when updating the plan for cloud spof disaster recovery and disaster recovery testing.

Network component redundancy requires the purchase of two of everything and makes the design and maintenance very complex. Are you getting what you think you paid for in both product and complexity costs? Sadly most redundant systems are either not operational because the redundant configuration engineering exceeds the staff's ability or it has flaws creating a worse set of dependencies than a single point of failure (SPOF) design would. By documenting your layer 2/3 environment well and superimposing application transaction steps thereon you can quickly see any flaws. The example is classic! Enjoy.

Complete Transcript:

Speaker: Here's another one. You bought two of everything because management said, “what if that thing goes down?” Some non-technical auditor came in and said, “what if that goes down, that's really important, you ought to have two of those.” So we bought into this and we started buying two of everything. Well, when I go into an enterprise and I start documenting their system to troubleshoot a problem, and I do my layer two, layer three diagrams depicting layer two and layer three analysis of where a packet goes, step by step through an infrastructure by VLAN and by component. This starts to expose our dependencies logically. Now you can see that you have two of everything from a physical standpoint, but logically, the way that we configure it and how packets actually flow through it, is an important thing to also diagram. So I've diagramed for you, a very complex environment, where we have an application server right here and we have a database server right here. The application server is what serves up to the end users, the user interface, and the database server is where the application server goes to get all the data from for those particular transactions.

So let's start out here with the yellow line, the yellow brick road let's call it. We go down the yellow brick road here. We come into a firewall, into our application environment, and we come in on port three. We go on port two. We come around here, we go up to VLAN II on switch one. VLAN II on switch one needs to communicate with the device on VLAN II of switch two. SSo it goes across the trunk, comes over here, gets into VLAN II on switch two, goes into the content server on switch two and into the contents switch which load balances between all of our application servers. It comes through here, goes out VLAN III on the other side of the contents server, comes up and over and back through into VLAN III on switch one. Which then connects up to our applications server called Prod DB or Prod App. So, here we are, we've just gone from the user which is out here, through our firewall, out through our firewall, through both of our switches, through our content switch, back and forth across the trunk a couple of times, headed out and now we've finally got to our server. So are we dependent upon, for the initial transaction, what? This firewall, this switch, these trunk circuits, and this switch. Am I not correct?

Well, to complete the transaction, the Prod App server has to do a sequel query on the database over here down at Prod DB. So consequently it then sends out that goes into VLAN 4 on switch two, comes back out, goes into firewall interface four, comes over to interface five, goes back out, goes into the switch and back out of the switch on VLAN 5, and then heads to Prod DB. Now at least from my vantage point here, I've traced this logic to show that for this transaction to be successful, we have to have not only switch one, switch two, firewall one and firewall two. If any of those components were to go down, which is the reason why we have two of them, our user would not be successful. So until we basically take a look at and diagram layer one, layer two transactions into our applications and through those complex switches, we can't really see what's going on. So this sort of a process of knowing where your layer two and layer three information is, is very important.

MTU defines Maximum Transmission Unit. Firewalls blocking ICMP disables IP's dynamic Path MTU Discovery causing fragmentation at VPN, L2TP tunnels impacting performance. Gateways are not able to respect native DF bit flags (Don't Fragment) because they are isolated on another OSI Model stack interface. The performance hit comes from what I call the "two for one blue light packet special" caused by the overhead of the tunnel header forcing two packets across the tunnel for each originating packet. That is why most VPN Client software adjust their MTU to 1300 so that does not occur - but what if you go through multiple tunnels? Then it becomes a manual hide and seek process to arrive at an MTU that works. There is of course much more to this than meets the eye so be careful in setting router and gateway MTU without fully thinking through the theory, analyzing the results with an analyzer and setting up network monitoring signature triggers to watch for problems in the future. The best way to approach the topic is to make it as automatic as possible through creatively setting your router MTU setting on the interface that goes to your gateway or Internet.

Transcript for this Video

MTU Maximum Transmission Unit is the size of the datalink layer payload of a packet. If it is not allowed to automatically perform path MTU discovery using ICMP Internet Control Message Protocol packets due to security filters on firewalls, then a logon and connect can be successful, but a data transfer will fail when a larger packet is sent. Or performance will suffer across high error prone links because a VPN tunnel will create two packets for every one packet to overcome the MTU mismatch exposing double the packets to packet loss. If one packet is lost both packets will have to be restransmitted by a higher layer protocol like TCP.

Complete Transcript: Here's another example of a problem that we've found that is kind of layer 3 related, VPN and L2TP tunnels. A VPN and a L2TP tunnel are there, and they're basically layer 7 services. Where it takes in a layer 3 packet, repackages it, and then puts it in another layer 3 environment. The don't fragment bit, which we use in IP at layer 3, to tell routers not to break up packets, that is not respected when it gets to a layer 7 device acting as a VPN concentrator or an L2TP tunnel. Those are actually layer 7 type devices which do tunneling for us or some other functioning. If you want to call it something other than layer 7, it's an application, so that's why I call it layer 7, you could call it layer 4. In any event, it's a tunnel going through your network.

When your packet leaves your station, it's 1,500 bytes at maximum, and that's called a maximum transmission unit, or MTU. That is the size of the payload at the datalink layer that the packet exiting your machine goes out. Now that 1,500 byte packet, as you can see here, travels out of the client, goes into your network, and it hits a tunnel device. Like let's say an L2TP tunnel. That L2PT tunnel then takes that 1,500 byte packet, and it has to put that 1,500 byte packet, but it still has than 1,500 byte MTU. It can't make a packet larger than 1,500 bytes. What it does is it takes your 1,500 byte packet, and it breaks it up into two packets, a large packet and then a smaller packet, and then puts the tunnel overhead on there, so that it can go across the network. At the other end of the tunnel, the tunnel then collapses that packet and lets go of your original packet pretty much the way you let it go outside of your particular machine. Again, we start out with a 1,500, we go in, we have a 1,400 and a 100, and then it comes back out the other side of the tunnel, and it comes back to 1,500.

What happens is, you're getting the 2-for-1 Blue Light Special here. You're getting two for one. You're getting two packets on the high packet loss, error prone internet, and consequently the effect of that is that you're going to lose more packets. And if one of those fragments, or those little segments are lost, you're going to have to re-transmit both of them, and those intervening device, being layer 3, are not performing end-to-end retransmission. So it will rely on your upper layers on the two ends, typically at layer 4 or layer 7, to retransmit those packets for you, and that can be very costly to your performance.

One of the things you can do, and by the way this happens from an internal network where you're sending a 1,500 byte MTU packet over to a VPN concentrator. That VPN concentrator is then having to put VPN overhead on top of that 1,500 byte, and it still has to live with the MTU on the next network of 1,500, so it has to break your packets up into two segments. When it does that, obviously, you still have this 2-for-1 overhead special. In some cases you can end up with packets that get lost, and then consequently your performance plummets when you need it most. When does your device send a 1,500 byte packet? It send a 1,500 byte packet when it's got a lot of data to send. When you're doing an FPT, when you're doing something that is a large data transfer. When you really need your network to work, that's when you start getting the 2-for-1 special. When you're doing that FTP download, or you're looking at a big website, or you're looking at a webcast. That's when these problems are going to occur to you.

In preparation for that, what we do is lower our MTU. There are several methods of lowering our MTU, so that when our packet, which is at maximum 1,500, reaches an L2TP tunnel or it reaches a VPN concentrator, it's already smaller. So that you get a 1-for-1 packet ratio. Most of your VPN clients today actually cut your MTU on all of your communications down to 1,300 bytes, so that when it gets to the VPN concentrator it will automatically not have to get the 2-for-1 Blue Light Special and expose your packets, two packets for every one packet for potential packet loss, on the internet.

There are several ways of accomplishing that. While all this stuff works automatically, and the MTU mismatches will get fixed, provided you are allowing the ICMP, internet control message protocol packets, to get from the device and go back to the client or back to the server who sent the packet. Unfortunately today we have firewalls, and our firewalls are blocking the necessary ICMP path discovery packets that normally would manage and make sure you don't get the 2-for-1 Blue Light Special on your own. But because of security constraints, we've turned off ICMP, and so consequently we have to do what is called end-to-end MTU management of our systems. It would behoove you to get an analyzer out and put it at these various locations and find out if you are, indeed, using ICMP destination unreachable path MTU discovery mechanisms effectively, or if, in fact, your firewalls are dropping those, and understand how this process is working in your particular environment.

So what I recommend is out here on these routers on the edge, if you do have and want this to be automatic, you can take your clients which are out here on your internal network, take your first router and set the MTU where you do have ICMP destination unreachable path MTU discovery packets where those are allowed inside your normal network, where they are not blocked by a firewall. And what we can do is lower the MTU on the inside link next to the firewall on the router, and then all this will be automatic. By lowering your path MTU on this particular router interface down to 1,300, essentially every packet that comes through there will automatically get its MTU lowered by path MTU discovery. Then your packets will flow through onto your VPN or onto your L2TP tunnels already smaller to go through that trip and get the expanded headers, OK?

This is one of the things you might want to look for and might want to understand in your environment so you know how things are working and if you were ever audited by a technical group, you would have the answer appropriate to the question for security purposes or what have you. You need to know about path MTU discovery and at every single point along the path what is going on with your MTU size. Thank you very much for listening, and we'll have more.

Oracle performance tuning -SQL and for that matter MS SQL tuning or any SQL server performance tuning can be analyzed by capturing TCP database transactions at the packet level. The Oracle Performance Tuning video is an example of the value of network analysis to optimize existing systems rather than await the forklift upgrade and its expense to improve performance. Andy hey - that forklift upgrade might not contain the silver bullet you are looking for to improve performance. There are many factors that can impact database and application performance - it might be a network or authentication issue slowing down your database. ...better to start database tuning with "brain cells" than capital budget. Transcript for this Video

Today, we're going to talk about Oracle Performance Tuning opportunities for application optimization. So you basically take and measure your performance. So let's just say you click and it comes back and it takes ten seconds. You then take a look at that transaction with an analyzer, etc., and you then calculate what it's doing and figure out, theoretically, how long that should take or experientially how long that should take.

Let's just say you come up with ten seconds as a measured performance, and you found that you could theoretically get this done in under a second. So there's nine seconds of what we call optimization opportunity.

Well, that's the first thing that you need to do is to determine if your optimization is viable, if you're getting a material return on your investments. So you take a look at measured performance, calculate theoretical performance, and arrive at your optimization opportunity.

After that, you want to take a look at some real information. Here is an actual Wireshark trace file, a SYN, SYNACK, ACK. So how long does it take to connect up across the network? Well, it takes four milliseconds to connect up to the application and back at the TCP layer. Then, it goes and it says, "Okay. Connect user Oracle," and he comes over here and he gets a, "Hey, resend this." That takes 73 milliseconds. Then he resends it and it comes back in a half a millisecond. So you can see right here that that's quite a bit of time. 73 milliseconds is a very long time. If you only do it once per transaction, you're only burning under 100 milliseconds.

However, this particular transaction performs about 86 queries, and each one of those queries starts out with a 73 millisecond connect request to the Oracle database. Well, that's going to take 86 times 73 milliseconds. So it's going to be pretty substantial in how much that costs you in the form of the transaction. That's an extra six seconds added to your transaction already.

So, if you could basically reduce that down, that would be optimal. However, let's just think about this. What if you only logged in one time and then performed 86 queries across a persistent connection? That would be a better way of architecting this application, and indeed, that is the suggested optimization.

So, in the next session, we're going to talk about to your CIO what this means in dollars and cents and how to justify your return on investment for something that you're looking at trying to optimize.

Now, Oracle performance tuning SQL optimization is free, right? It's just something that you have to do. Sadly, today most of your optimizations are in the form of large scale, forklift upgrades, and sadly, a lot of times we don't even look at opportunities to optimize the existing systems. Well, those opportunities do, in fact, exist, and in the time of tough economy and tight budgets, we're starting to go back and take a look at how can we optimize what we already own and improve our lifecycle cost and improve our applications. So that's what we're talking about today.

Watch Bill Alderson's video on Oracle performance optimization ROI justification based on detailed packet analysis findings. This video is for CIO's to help justify portfolio spending on database improvements... a technical companion Video is posted showing the case study packet trace explaining the technical analysis to find optimization opportunities

Transcript for this Video

This session is on Oracle database management to optimize performance and yield a significant ROI. First of all, a quick review from your technologist session. You measure the performance, you measure or calculate the theoretical performance, and arrive at an optimization opportunity. That's the delta. What that looks like is a trace file where we look at SYN, SYNACK, Oracle Connect. The Oracle connects. It takes 73 milliseconds. That 73 milliseconds happens for each and every SQL query, and what we want to do is knock out and use, not an implicit logon to the database for each and every query, but one logon and then several secondary queries. That will allow us to knock out that 73 milliseconds entirely practicing Oracle performance optimization.

So here's how you might justify this. First of all, you put together a spreadsheet of your measured performance and your theoretical performance, and you come up with your optimization opportunity. In this case, it's 72 milliseconds. So what you do is you've got your number of users at your call center, your underwriters, your branches, your managers, and your report time generation. Now this can be different for different companies, of course. This is basically looking at a financial institution and a large number of transactions that people are doing at different labor rates and that sort of thing.

So we basically take the micro-transactions and we calculate that out to find out how long it's taking. So if it's taking an extra six seconds per transaction, and that's because of this 72 millisecond logon, and we can knock that down to 72 milliseconds, then you can calculate out for this many users. I've just done some very simple equivalent person calculations. For instance, if you've got 3,000 people in the call center, you can probably drop 49 of those people just by this single optimization.

Now if you take into consideration all of the cost elements, you'd save $3 million in the first year of this optimization, just in the call center. So that's probably a good enough justification right there. But you add them all up, and you've got $4 million. Now, if the optimization costs $325,000 to rearchitect and logon once to your database and then have multiple subsequent queries, instead of 86 queries and 86 logons, you would end up achieving a cost savings in the first year of $3.6 million. Now, if you have five years of useful life, that would be a $20 million savings. If you calculate out what your recoup rate is at $333,000 per month, it would take you one month to pay off the investment of $325,000 in rearchitecting this application in a very simple manner and save yourself $20 million over the next 5 years. Not a bad day's work. Consider becoming a performance oracle!

Slow Domain Logon Analysis - Root Cause Identified - Solved!

Root Cause Identified - Application Performance Optimized are the words CIO's and End Users enjoy hearing. Network Slow, Application Slow - these trouble tickets take more time and are rarely definitively diagnosted to the true root cause.

[bitsontherun GF3m0Ujn]

Transcript for this Video

So the first thing I'm going to talk about is an application log on, that was taking a long time. As a matter of fact it was into the minutes, and sometimes the user would get disconnected and not even be able to log onto the application. Well the first thing we did was took a capture. We took a trace, everybody says, let's take a trace. Of course you take a trace, and you look at this thing, and it's like where do I start. It's a bunch of protocol goulash, and it's the same thing for me until I zero in on the TCP session or on the particular application or on the particular client or server, it's protocol goulash. So we have to zero in on these things.Now we found that there was a problem with slow domain logon, and so we kind of suspected that it might have something to do with domain controllers. So I Ioaded a que check agent on the server and on the workstation. Now Qcheck is a freebie from Ixia Communications and you can go out on the web and find cue check out there. You can download it, put it on your server, put it on your workstation, and then you can do a through put or a latency test to find out how it's performing at layer four. Well we did that and we found that it was giving us very poor performance with other tasks.

Now domain controller usually isn't doing file copies and those sort of things. It's basically doing security authentication. When we went to do through put test we found that it was very poor, for copying files or doing the que check. Now what we found when we went in and looked at that poor performance was, and if you take a here, a check sum error. You'll see here, TCP check sum error incorrect. I blew this up so you could see it. TCP check sum error, or check sum incorrect. Now what that means is, is that the TCP check sum calculated by the server was wrong when it went out on the wire.

Now it can have a good data link control. It can have a good IP header check sum, and a good data link layer check sum, but have a bad TCP check sum, and that's exactly what we found in this case. Now we wanted to find more occurrences of this and see how often this was occurring, but we still wanted to see all the other packets so we used Ethereal which is a free network analyzer public domain that you can find at Ethereal.com. By going in an doing a find, by a string, you can find in the decode, TCP check sum, and I just put TCP check. Then I click find, and then boom, I would find several TCP check sum errors. So TCP check sum incorrect, check sum incorrect, check sum incorrect, not all but many of these packets had bad TCP check sums.

I thought to myself, what could be possible causing that to occur? And then I went to the machines, server, network interface card. And I found that check sum off load was enabled for both transmit and receive to the network interface card. Now the reason why this has started to be popular is because Microsoft announced in 2001 that they did some test with network interface cards that did the offload of the TCP check sum, and hardware instead of in their operating system, and so by using the hardware nic card as the TCP check sum offload device system using it's processor. The server had much better performance, because it didn't have to check all these check sums and calculate them.

So everyone beat a path to this particular door. All network interface card manufacturers started performing TCP check sum offloading, in the actual nic hardware. Well what we found was any NIC that we found in a Windows 2000 machine. A Windows 2000 server, that had TCP check sum receive and transmit enabled, those were the perpetrators of the bad TCP check sums. And we exercised those machines by using the que check agent, so that we could see the machine truly with an offered load. And we did find, and corroborate, that other domain controllers suffered too. But we didn't find it on Windows 2003 server, we only found it on Windows 2000 servers, but we found it on pretty much every one that we looked for. So it was kind of my belief that perhaps we found a little problem here with Microsoft's operating system, in regards to how it allows in Windows 2000, the TCP check sum offload to the network interface card. Because we found multiple of these. Now after we turned the TCP check sum offload off, in both cases of an HP network interface card, and also a 3Com network interface card.

The performance went up, so it didn't matter what network interface card vendor it was. The performance went dramatically up, when we changed to turn TCP check sum offload off on Windows 2000 servers. And we calculated that out by doing a que check measurement and we got much better performance orders of magnitude, and an absence of TCP check sums. So we very meticulously change network interface card vendors. We turned TCP check sum load on and off, and every single time we found if you're using a Windows 2000 server with TCP check sum offload, at least in the case of Hewlit Packard and also 3Com network interface cards. That they occasionally cause TCP check sum miscalculations to occur from those devices.

So it might be a good idea for you to go out and check all your Windows 2000 active directory servers and find out if they have the same thing turned on, and you might want to see if you're getting TCP check sum errors. If you are, turn off TCP check sum offload, and I'll bet your problem goes away, and your authentication occurs faster and your users of these applications are much happier.