Networks Networks and Servers

Showing posts with label Networks. Show all posts

How to Setup a Virtualization Lab (III)

Failover Cluster Networking

The first step in the setup of a failover cluster is the creation of an AD domain because all the cluster nodes have to belong to the same domain. But before doing so, I changed the networks settings again in order to adjust them for this purpose.

LAB-DC:IP: 192.168.1.10
Gateway: 192.168.1.1 (Physical Router)
DNS: 127.0.0.1
Alternate DNS: 192.168.1.1

LAB-NODE1:
IP: 192.168.1.11
Gateway: 192.168.1.1
DNS: 192.168.1.10 (DC)
Alternate DNS: 192.168.1.1 (Physical Router)

LAB-NODE2:IP: 192.168.1.12
Gateway: 192.168.1.1
DNS: 192.168.1.10
Alternate DNS: 192.168.1.1

LAB-NODE3:
IP: 192.168.1.13
Gateway: 192.168.1.1
DNS: 192.168.1.10
Alternate DNS: 192.168.1.1

LAB-STORAGE:IP: 192.168.1.14
Gateway: 192.168.1.1
DNS: 192.168.1.10
Alternate DNS: 192.168.1.1

Therefore, I created a domain comprised of 5 machines; a DC and two member servers as Hyper-V VMs, a member server as a VMware VM and another member server as a VirtualBox VM.

So far I have demonstrated the possibility of integrating in the same logical infrastructure virtualized servers running on different platforms using different virtualization techniques; in this case we have VMs running in a Type 1 hypervisor (Hyper-V) and in two distinct Type 2 hypervisors (VMware Workstation and VirtualBox).

The option to create a network with static IP addresses is as valid as the alternative of using DHCP. Later on I plan to explore the several options provided by the cluster networking in Windows 2008 but for the time being I kept my network in a simple and basic configuration in order to proceed with the lab installation.

How to Setup a Virtualization Lab (II)

As mentioned at the end of my previous article, the installation of my lab continued with the creation of virtual machines on the desktop computer. But this time I used VMware and VirtualBox to explore the possibility of using a set of virtualized servers across different and competing virtualization technologies.

I insisted on the network configuration details because that is the basis of all the work ahead; a single virtual machine may be important but I want to show how they can work together and therefore the correct network configuration of paramount importance.

Import a Virtual Machine into VMware

I started by installing a VM on VMware Workstation. Better yet, I took advantage of what was previously done and used the generalized .vhd file I left behind! Since VMware does not directly support the use of .vhd files, I had to convert the file from the format used by Hyper-V (Virtual Hard Disk, i.e., .vhd) to the format used by VMware (Virtual Machine Disk, i.e., .vmdk).

The VMware vCenter Converter Standalone utility is a free application which can be obtained directly from VMware’s official site but doesn’t solve the problem as it doesn’t support this type of conversion, although it can convert from other formats and even directly from servers running Hyper-V. But what interested me was to use the work already done and so I resorted to the WinImage tool.

The process was very simple:

I selected the appropriate option from the Disk menu and select the proper source file;

How to Setup a Virtualization Lab (I)

Now that I have concluded a general overview of most of the theory related to High Availability and Virtualization it is time to start testing some of those concepts and see them in action.

My goal for the next posts is to produce a series of tutorials showing how anyone can easily install a handful of virtual machines and be able to explore the wonderful possibilities provided by this technology. I will be using an old laptop powered by a Turion 64 X2 CPU with a 250 Gb SSD HD and 4 Gb of RAM combined with a desktop running Windows 7 Ultimate on a Athlon 64 X2 4800+ with 4 Gb of RAM and lots a free disk space scattered through 3 SATA hard drives.

Virtual Machines Creation

I will not go through the details of OS installation because I am assuming the ones reading these tutorials are way passed that.

I started by installing a fresh copy of Windows Server 2008 R2 SP1 Standard on a secondary partition in my laptop. Once I was done with the installation of all the available updates from Windows Update and with OS activation, I was ready to add the Hyper-V role in order to be able to install the virtual machines. To do this I just went into Server Manager/Roles, started the Add Roles Wizard, selected Hyper-V and followed the procedures. Nothing special so far, right?

Note: All the pictures are clickable and will open a larger version in a separate window.

Network Virtualization

When we think of network virtualization, we always think of VLANs but there is much more to network virtualization than just VLANs. Network virtualization is when all of the separate resources of a network are combined, allowing the administrator to share them out amongst the users of the network. Thus, it is a method of combining the available resources in a network by splitting up the available bandwidth into channels, each of which is independent from the others, and each of which can be assigned (or reassigned) to a particular server or device in real time. This allows each user to access all of the network resources from their computer either they are files and folders on the computer, printers or hard drives etc.

The theory behind network virtualization is to take many of the traditional client/server based services and put them "on the network". Certain vendors advertise virtualization and networking as a vehicle for additional services and not just as a way to aggregate and allocate network resources. For example, it's common practice for routers and switches to support security, storage, voice over IP (VoIP), mobility and application delivery.

One network vendor actually has a working card that is inserted into a router. On that card is a fully-functioning Linux server that has a connection to the backbone of the router. On that Linux server, you can install applications like packet sniffers, VoIP, security applications, and many more.
Network virtualization provides an abstraction layer that decouples physical network devices from operating systems, applications and services delivered over the network allowing them to run on a single server or for desktops to run as virtual machines in secure data centers, creating a more agile and efficient infrastructure. This streamlined approach makes the life of the network administrator much easier, and it makes the system seem much less complicated to the human eye than it really is.

Network virtualization is a versatile technology. It allows you to combine multiple networks into a single logical network, parcel a single network into multiple logical networks and even create software-only networks between virtual machines (VMs) on a physical server. Virtual networking typically starts with virtual network software, which is placed outside a virtual server (external) or inside a virtual server, depending on the size and type of the virtualization platform.

Redundant Protocols

If you read the previous posts, your network already has redundant links and now you must decide how packets on the network will select their paths and avoid loops. This isn't a new problem; redundant paths have been addressed by protocols like Spanning Tree Protocol (STP) at Layer 2 and routing protocols like Open Shortest Path First (OSPF) at Layer 3. But these protocols can take 40 seconds or more to resolve and converge and this is unacceptable for critical networks, especially those with real-time applications like VoIP and video.

STP is a link management protocol that provides path redundancy while preventing undesirable loops in the network. For an Ethernet network to function properly, only one active path can exist between two stations. This protocol should be used in situations where you want redundant links, but not loops. Redundant links are as important as backups in the case of a failover in a network. A failure of your primary router activates the backup links so that users can continue to use the network. Without STP on the bridges and switches, such a failure can result in a loop.

To provide path redundancy, STP defines a tree that spans all switches in an extended network and forces certain redundant data paths into a standby (blocked) state. If one network segment in the STP becomes unreachable, or if STP costs change, the spanning-tree algorithm reconfigures the spanning-tree topology and reestablishes the link by activating the standby path.

An upgraded version of STP called RSTP (Rapid Spanning Tree 802.1w) cuts the convergence time of STP to about one second. One disadvantage to RSTP (and STP) is that only one of the redundant links can be active at a time in an "active standby" configuration another is that STP when changes the active path to another router, so the gateway addresses of the clients must change as well. To avoid these problems, you must run Virtual Router Redundancy Protocol (VRRP) along with STP and RSTP on your routers, which emulates one virtual router address for the core routers and takes about three seconds to fail over.

The advantage of using VRRP is that you gain a higher availability for the default path without requiring configuration of dynamic routing or router discovery protocols on every end host. VRRP routers viewed as a "redundancy group" share the responsibility for forwarding packets as if they "owned" the IP address corresponding to the default gateway configured on the hosts. One of the VRRP routers acts as the master and others as backups; if the master router fails, a backup router becomes the new master. In this way, router redundancy is always provided, allowing traffic on the LAN to be routed without relying on a single router.

But because VRRP and RSTP work independently, it's possible VRRP will designate one router as master and RSTP would determine the path to the backup router as the preferred path. Worst case, this means if the backup VRRP router receives traffic, it will immediately forward it to the master router for processing, adding a router hop.

Redundant devices

Today’s networks are high-tech and most times high speed. Common to most Wide Area Network (WAN) designs is the need for a backup to take over in case of any type of failure to your main link. A simple scenario would be if you had a single T1 connection from your core site to each remote office or branch office you connect with. What if that link went down? How would you continue your operations if it did?

Adding redundancy is the most common way to increase your uptime. First, make sure there's redundancy within your core router; redundant CPU cards, power supplies and fans usually can be added to chassis-based routers and switches, and some router and switch vendors have equipment with dual backplanes. With redundant CPU cards, you can force a failover to one card while you upgrade the second one, instead of having to bring the whole router down for the upgrade.

The goal of redundant topologies is to eliminate network downtime caused by a single point of failure. All networks need redundancy for enhanced reliability and this is achieved through reliable equipment and network designs that are tolerant to failures and faults and networks should be designed to reconverge rapidly so that the fault is bypassed.

Network redundancy is a simple concept to understand. If you have a single point of failure and it fails you, then you have nothing to rely on. If you put in a secondary (or tertiary) method of access, then when the main connection goes down, you will have a way to connect to resources and keep the business operational.

The critical point is that highly reliable network equipment is expensive because it is designed not to break and this typically includes things like dual power supplies, watchdog processors and redundant disk systems.

A highly available system may be built out of less expensive network products but these components may lack the redundant power supplies or other features of high-reliability equipment, and therefore, they may fail more often than the more expensive equipment. However, if the overall network design takes into account the fact that equipment may fail, then end users will still be able to access the network even if something goes wrong.

High Availability – Solutions

Companies are under increased pressure to keep their systems up and running and make data continuously available. Being now held to a higher standard for application and data availability, the trick is to design server and storage systems that are highly available and almost bullet-proof against unplanned downtime. In order to achieve the highest levels of availability a company has to implement a complete solution that will address all the possible points of failure. But what are the available options if you want to design a high availability solution?

You can see in the chart the main solutions for the three areas to be addressed; storage, services and networks:

In the following posts I will explain these solutions in further detail. Keep reading, ok?

High Availability – Measurement (II)

Reliability metrics

Failure rate

Reliability can be quantified as MTBF (Mean Time Between Failures) for a repairable product and as MTTF (Mean Time To Failure) for a non-repairable product.

According to the theory behind the statistics of confidence intervals, the statistical average becomes the true average as the number of samples increase. So, a power supply with an MTBF of 50,000 hours does not mean that the power supply should last for an average of 50,000 hours because the MTBF of 50,000 hours, or 1 year for 1 module, becomes 50,000/2 for two modules and 50,000/4 for four modules. It is only when all the parts fail with the same failure mode that MTBF converges to MTTF.

If the MTBF is known, one can calculate the failure rate (l) as the inverse of the MTBF. The formula for l is:

Once a MTBF is calculated, what is the probability that any one particular module will be operational at time equal to the MTBF? For electronic components we have the following equation:

But when t = MTBF

This tells us that the probability that any one particular module will survive to its calculated MTBF is only 36.8%, i.e., there is 63.2% probability that a single device will break before the MTBF!

Bathtub curve

Over many years, and across a wide variety of mechanical and electronic components and systems, people have calculated empirical population failure rates as units age over time and repeatedly obtained a graph such as shown below. Because of the shape of this failure rate curve, it has become widely known as the "Bathtub" curve.

This curve (in blue) is widely used in reliability engineering as describing a particular form of the hazard function which comprises three parts:

The first part is a decreasing failure rate, known as early failures.
The second part is a constant failure rate, known as random failures.
The third part is an increasing failure rate, known as wear-out failures.

The name is derived from the cross-sectional shape of a bathtub and the curve is generated by mapping the rate of early failures when first introduced, the rate of random failures with constant failure rate during its "useful life", and finally the rate of "wear out" failures as the product exceeds its design lifetime.

Reliability examples

Example: Suppose 10 devices are tested for 500 hours. During the test 2 failures occur.
The estimate of the MTBF is:

Whereas for MTTF is:

Another example: A router has an MTBF of 100,000 hours; what is the annual reliability? Annual reliability is the reliability for one year or 8,760 hours.

This means that the probability of no failure in one year is 91.6%; or, 91.6% of all units will survive one year.

High Availability - Measurement (I)

Measuring Availability

The need for availability is governed by the business objectives and its measurement’s primary goal is:

To provide an availability baseline (maintain it);
To help identify where to improve the systems;
To monitor and control improvement projects;

As technology has evolved over the years, most systems can achieve 90% availability with little built-in HA redundancy and it requires more discipline in the IT department rather than any hardware or software to achieve this goal. But to achieve more than 90% of system availability, some special considerations need to be made and we will look into that in the following posts.

It is important to recognize that numbers like these can be difficult to achieve, since time is needed to recover from outages. The length of recovery time correlates with the following factors:

Complexity of the system: The more complicated the system, the longer it takes to restart it. Hence, outages that require system shutdown and restart can dramatically affect your ability to meet a challenging availability target. For example, applications running on a large server can take up to an hour just to restart when the system has been shut down normally, longer still, if the system was terminated abnormally and data files must be recovered.
Severity of the problem: Usually, the greater the severity of the problem, the more time is needed to fully resolve the problem, including restoring lost data or work done.
Availability of support personnel: Let's say that the outage occurs after office hours. A support person who is called in after hours could easily take an hour or two simply to arrive to diagnose the problem. You must allow for this possibility.
Other factors: Many other factors can prevent the immediate resolution of an outage. Sometimes an application may have an extended outage simply because the system can't be put offline while applications are running. Other cases may involve the lack of replacement hardware by the system supplier, or even lack of support staff.

Availability Metrics

Mean Time to Repair (MTTR)
Impacted User Minutes (IUM)
Defects per Million (DPM)
MTBF (Mean Time Between Failure)
Performance (e.g. latency, drops)

Availability is usually expressed as a percentage of uptime in a given year. The following table shows the downtime that will be allowed for a particular percentage of availability, presuming that the system is required to operate continuously. Service level agreements often refer to monthly downtime or availability in order to calculate service credits to match monthly billing cycles. The following table shows the translation from a given availability percentage to the corresponding amount of time a system would be unavailable per year, month, or week.

High Availability - Objectives

The main objective in designing any High Availability (HA) and Disaster Recovery (DR) strategy is Business Continuity. Each business has its own level of tolerance for system failures and outages and depending upon that tolerance, a suitable strategy can be planned and implemented. If a business can accept 90% system availability, then there is no need to build any HA infrastructure.

Although HA solutions are frequently discussed in business environments, these considerations apply to any type of organization-defense, educational or non-profit-where HA is required. When it comes to organizations other than businesses, it is not very easy to calculate the costs associated with downtime. HA and DR systems in these organizations become more of a requirement from the service standpoint than from the perspective of cost associated with downtime.

Availability of the systems should be seen from the end user's perspective. Any time a user cannot connect to the system is considered as downtime but this does not necessarily mean the main computer system is going down because, in many cases, a poorly performing system is also considered an Unavailable System.

So, High Availability does not really mean to build redundancy into only one system or database or application, but it is a combination of redundancies being built into all areas of the process. For every business or organization, the database plays an important role; everything is built around the database, hence, most of the efforts for High Availability are concerned with making the database "Highly Available."
More specifically, a high availability architecture should have the following traits:

Tolerate failures such that processing continues with minimal or no interruption;
Be transparent to (or tolerant of) system, data, or application changes;
Provide built-in preventative measures;
Provide proactive monitoring and fast detection of failures;
Provide fast recoverability;
Automate detection and recovery operations;
Protect the data so that there is minimal or no data loss;
Implement the operational best practices to manage your environment;
Achieve the goals set in SLAs (for example, the RTO and the RPO) for the lowest possible total cost of ownership.

System designers often build reliability into their platforms by building in correction mechanisms for latent faults that concern them. These faults, when correctable, do not produce errors or failures since they are part of the design margins built into the system. They should still be monitored to measure their occurrence relative to designers anticipated frequency, since excessive occurrence of some correctable faults is often an indicator of a more catastrophic underlining latent fault.

Reliability, recoverability, timely error detection, and continuous operations are primary characteristics of a highly available solution.

High Availability - Terminology (II)

Planned outage/downtime

Planned outages include maintenance, offline backups and upgrades. These can often be scheduled outside periods when high availability is required.

Unplanned outage/downtime

While planned outages are a necessary evil, an unplanned outage can be a nightmare for a business. Depending on the business in question and the duration of the downtime, an unplanned outage can result in such overwhelming losses that the business is forced to close. Regardless of the nature, outages are something that businesses usually do not tolerate. There is always pressure on IT to eliminate unplanned downtime totally and drastically reduce, if not eliminate, planned downtime.

Note that an application or computer system does not have to be totally down for an outage to occur. It is possible that the performance of an application degrades to such a degree that it is unusable. As far as the business or end user is concerned, this application is down, although it is available.

Unplanned Shutdowns have various causes. The main reasons can be categorized into:

Hardware Failures – Failure of main systems components such as CPUs and memory; or peripherals such as disks, disk controllers, network cards; or auxiliary equipment such as power modules and fans; or network equipment such as switches, hubs, cables, etc., can be the causes of hardware failures.
Software Failures – The possibilities of failure of software mostly depends upon the type of software used. One of the main causes for software failure is applying a patch. Sometimes, if a patch does not match the type of implementation, then the application software may start to behave in a strange way, bringing down the application and reversing the changes, if possible. Sometimes, an upgrade may also cause a problem. The main problem with upgrades will be performance related or the misbehaving of any third party products, which depend upon those upgrades.
Human Errors – An accidental action of any user can cause a major failure on the system. Deleting a necessary file, dropping the data or a table, updating the database with a wrong value, etc., are a few examples

System

A system is composed of a collection of interacting components. A component may itself be a system, or it may be just a singular component. Components are the result of system decomposition chiefly motivated to aid in the partitioning of complex systems for either technical, or very often, for organizational or business reasons. Decomposition of systems into components is a recursive exercise. All components are typically delineated by the careful specification of their inputs and outputs and a component that is not decomposed further is called an atomic component.

Service

A system provides one or more services to its consumers. A service is the output of a system that meets the specification for which the system was devised, or which agrees with what system users has perceived the correct values to be.

Failure

A failure in a system occurs when the consumer (human or non-human) of a service is affected by the fact that the system has not delivered the expected service. Failures are incorrect results with respect to a specification or unexpected behavior perceived by the consumer or user of a service. The cause of a failure is said to be a fault.

Mean time to failure (MTTF)

Hardware reliability can be predicted by statistically analyzing historical data. The longer a component operates, the more it is likely to fail due to aging. The mean time to failure of a component is just that: a statistical forecast to measure the average time between failures with the modeling assumption that the failed system is not repaired. The greater the MTTF of a component, the less likely it is to fail. You can use the MTTF of a component (if it is known) as useful information for establishing preventative maintenance procedures and its value for an overall system can be improved by carefully selecting your hardware and software.

MTTF is the number of total hours of service of all devices divided by the number of devices.

Definition of High Availability

High Availability (HA) is the ability of a system to perform its function continuously (without interruption) for a significantly longer period of time than the reliabilities of its individual components would suggest. HA, then, is a trade-off between the cost of downtime and the cost of the protective measures that are available to avoid or reduce downtime.

The term High Availability, when applied to computer systems, means that the application or service in question is available all the time, regardless of time of day, location and other factors that can influence the availability of such an application.

In general, HA it is the ability to continue a service for extremely long durations without any interruptions. Hence, HA is a system design approach and associated service implementation that ensures a prearranged level of operational performance will be met during a contractual measurement period.

HA systems should protect companies from two possible failures: system failures and site failures. Though true HA solutions should guard against both system and site failures, usually HA systems are regarded as protection from system failure, while site failure is typically protected by a Disaster Recovery (DR) system.