TSHOOT: Troubleshooting & Maintaining Cisco (Part1 of Part2)

Plan maintenance for complex networks by outlining strategies and steps covered in this chapter for effective reliability and uptime.

Explore network maintenance processes and procedures, including installation and configuration of new devices, backups, and troubleshooting. Monitor performance, capacity planning, apply software upgrades, and maintain documentation to support reliable services.

Explore Cisco IOS CLI management, including show and debug commands and embedded event manager, accessible via serial console or Telnet/SSH, with GUI tools for backups and logging.

Shows configuring a Cisco device to use NTP with time server at 10.1.23, sets Pacific Standard Time with DST, and enables timestamps with time zone and millisecond precision for logs.

Explore implementing backup and restore services using FCP, with examples of copying using FGP with a specified user name and password, and storing those credentials in a configuration.

Learn to implement backup and restore services using the configure replace command, navigate privilege exec and global configuration modes, and roll back changes.

Create a physical and logical topology in a simulator using three routers, two multilayer switches, a layer-2 switch, PCs, and a server; implement redundant links with spanning tree.

Compare shoot-from-the-hip troubleshooting with structured methods used in Cisco maintenance, emphasizing analyze and eliminate, gathering information, and forming and testing hypotheses to avoid unverified solutions.

Use show ip route to inspect the routing table. The spot the difference example shows how missing a static default route, a gateway of last resort, blocks packets.

Verify and define the problem after a report, use a troubleshooting ticket system, gather information, assign the issue, and decide to solve or escalate to tier 2 or 3.

Define the problem, establish a plan, identify targets across one machine or the entire network, assemble toolkits and access, then analyze information and escalate if needed.

Analyze assumptions, identify potential causes, and eliminate them to refine the hypothesis in the troubleshooting process.

Test hypotheses to find a solution, and assess its impact and urgency. Apply a rollback plan if the fix creates further problems; if unresolved, return to gather more information.

Document accurately to underpin effective troubleshooting by collecting per-link and device performance statistics via snmp and netflow, wireshark measurements, and ip sla latency and jitter.

Communicate clearly across all phases of structure troubleshooting to verify and clarify the reported problem, gather and analyze information, validate, escalate if needed, and test a hypothesis to solve it.

Apply change control to implement network changes, verify outcomes, and rollback if needed; document, back up configurations, and communicate changes to reduce outages and meet business requirements.

Explore troubleshooting fundamentals: gather information, analyze symptoms, form and test hypotheses, and apply top-down, bottom-up, or divide-and-conquer methods using the OSI model and tracer tools.

Create a network baseline and measure basic performance using show commands and NetFlow statistics, while IP SLA features monitor operations and guide troubleshooting and escalation across tiers.

Utilize the show IAP route command with a prefix and the longer prefixes keyword to limit output. Display addresses for a branch office or data center using the summary address.

Filter Cisco IOS show outputs using the pipe operator with include, exclude, or begin. Apply case-sensitive regular expressions to IP input, switch interfaces, and running-config lines.

Redirect ios show command output to a tftp server and to a file named show tech.txt. Repeated redirects overwrite the file; the t option displays on the console.

Test network connectivity with Cisco IOS commands by performing source-address pings and interpreting ICMP echo replies to determine if routers along the path can return replies.

Explore how Cisco IOS debug commands collect real-time packet details, including source, destination, and next-hop forwarding, while noting that some packets fast-switched or CTF-switched may not display.

Observe real-time rip debug data on Cisco IOS to see rip version 2 updates, one-hop routes, and how destinations beyond 15 hops become unreachable, with multicast 224.0.0.9 versus broadcast 255.255.255.255.

Filter Cisco IOS debug output using the debug condition command to target rip activity on an interface, and view or unset conditions with show debug condition or no debug condition.

Diagnose hardware issues with Cisco IOS commands, exploring show controllers, show platform, show inventory, and show dialog, plus diagnostics and the time domain Reflektor meter for upcoming labs.

Learn to configure switch port analyzers using span and monitor sessions to copy all traffic from Fa0/7 to Fa0/8, verify with show monitor, and understand frame types and ingress restrictions.

Use the router’s net flow cache to view active flows by issuing show IP cache flow; filtering options let you limit output to specific source or destination IPs for troubleshooting.

Enable network event notification to support proactive network management, using SNMP and syslog to trigger alerts, and leverage Cisco IOS Embedded Event Manager to define actions in response to events.

Enable a network event notification using an EEM applet that triggers on the configure terminal pattern and logs 'configuration mode was entered' and 'change control policies apply authorized access only'.

Explore how to collect and filter routing information with the show ip route command, view the routing table, and understand administrative distance and directly connected versus dynamic routes.

Review chapter 3 highlights how to use troubleshooting tools and applications effectively, detailing practical methods for diagnosing and resolving problems.

Illustrates how a lan switch determines if host b is on subnet using ip and subnet mask, checks the arp cache for host b’s mac, and transmits the ethernet frame.

Process a broadcast ARP request; host B recognizes it is for itself, sends a unicast ARP reply to host A, and updates its ARP table with the MAC mapping.

Switches forward the reply only to host A using their MAC address table, and learn host B's MAC address and interface in the LAN MAC address table.

Diagnose and resolve switched-network failures by tracing layer 1 and layer 2 issues, from faulty cables and unseated NICs to VLAN/trunk mismatches and misconfigurations.

Understand spanning tree operation: root bridge selection by least path cost, and root port selection with the lowest upstream bridge ID or port ID using the sender's ID for ties.

Identify how non-root, non-designated ports enter blocking, while designated ports move to listening and learning, and finally forward in normal operation, enabling information frames to be forwarded from the switches.

Analyze spanning tree on port 88, a 10 gigabit ethernet link, identify root port and designated port, confirm root bridge, and note bpdu activity with 0 path cost during convergence.

Explore spanning tree failures and how type 1 and type 2 faults affect forwarding and blocking, despite STP reliability, causing partial connectivity loss or bridging loops and broadcast forms.

Demonstrates how a mismatch between VLANs allowed on the port channel and on the physical interfaces can cause spanning-tree failures.

Explore inter-vlan routing and multilayer switching, comparing routers and multilayer switches as they use OSPF or EIGRP routing protocols or static routes and maintain routing tables.

compare multilayer switches and routers, noting that switches connect homogeneous networks with wire-speed ethernet packet switching, while routers link diverse networks using multi-purpose hardware and upgradeable features.

Examine sample data plane and control plane commands for routers and multilayer switches, noting that control plane troubleshooting is similar across devices while data plane troubleshooting differs.

Cisco express forwarding builds the FIB and adjacency table from control plane data, like routing and ARP cache, to ensure accurate packet forwarding; verify alignment when problems occur.

Explore layer 2 switching within VLANs and trunked traffic, including MAC address tables, then configure multilayer switching with SVIs and the IP routing command to enable inter-VLAN routing.

Routed interfaces depend on the connected subnet, so down interfaces remove routes; SVIs are nonphysical and stay up while at least one VLAN port remains in spanning tree forwarding state.

Examine the command output for interface Fa0/0 and the workstation ARP cache, revealing the virtual IP and MAC addresses of the HSRP group provided by both routers.

Verify HSRP operation through practical troubleshooting steps, focusing on standby behavior, failover readiness, and network maintenance in the Cisco troubleshooting and maintaining course.

Compare HSRP with VRRP and GLBP to understand alternatives for gateway redundancy. The lecture highlights virtual IP usage, preemption behavior, and hello timer values across protocols.

Demonstrates connectivity testing with ping between T1, T2, and T3, confirming 100 percent replies, then uses show spanning tree commands on a switch to diagnose vlan and interface issues.

Review essential layer 2 troubleshooting commands, including show mac address-table, show vlan, show interfaces trunk, show interfaces switchport, trace route, and spanning-tree commands for vlan and interface.

Identify three common etherchannel problems, focusing on port inconsistency within the channel and across opposite-side ports, and address uneven traffic distribution among channel bundle members.

Examine how multi-layer switches and routers share routing responsibilities, using OSPF or static routes to populate routing tables and determine reachability, while performing packet switching from layer 2 headers.

Differentiate SVI from a router port: a router port isn't tied to a VLAN and may not run spanning-tree or dynamic routing, and its status affects the connected subnet's route.

Explore routing concepts and data structures by tracing a packet from host A to host B through IP and MAC addresses, encapsulation, and protocol interactions like ARP, DHCP, and ICMP.

Verify routing functions with IOS commands by inspecting the arp cache using show ip arp, refreshing entries with clear ip arp, and mapping frame-relay PVC/DLCI with show frame-relay map.

Verify routing with iOS commands by clearing the IPv4 CEF and FIB tables and rebuilding them with APOC generation increments, noting APOC availability in iOS 12.4 revision 20.

Explore interior gateway protocol concepts, including neighbor relationships, learning paths, routing updates and redistribution, directly connected routes, best-path selection by metric, equal-cost load balancing, and administrative distance.

Learn to monitor EIGRP and IGMP using show commands: list interfaces participating in EIGRP processing, display IGMP neighbors, and inspect the EIGRP topology table to select specific prefixes.

Explain why only one topology table entry exists and determine if a neighbor relationship was never established or if the route was not exchanged, using hello packets to discover neighbors.

Diagnose why a serial interface isn't added to the EIGRP topology by ensuring the interface is up, its IP matches the configured network statement, and it is not passive.

Learn how OSPF uses shortest path first to select intra area, inter area, and external routes, with routing information flooded by area border routers.

Explore OSPF data structures, including the interface table for OSPF-enabled interfaces, passive interfaces, the neighbor table tracking active neighbors via hello messages, dead-time, and adjacency state.

Analyze area databases by counting type 1, type 2, and type 3 entries for areas 0, 1, and 2 across routers A to E.

Verify OSPF neighbor relationships on the link by ensuring interfaces are activated on both sides. Detect parameter mismatches via debug ip ospf output to understand information flow within an area.

Monitor OSPF effectively by using show commands to view router ID, areas, and spf run counts; inspect interfaces, neighbors, database, and spf statistics for routing stability.

Monitor OSPF in real time using debugging commands to observe routing changes and OSPF packet headers; verify hello exchanges and neighbor establishment, and diagnose mismatches in timers or area numbers.

Identify a misconfigured ospf network statement that matches the correct ip address, then replace it or use interface config mode to set ip ospf process-number area.

Apply the correct ospf configuration in this troubleshooting example to resolve the issues highlighted in the previous slides.

Review the show iap route profile output to gauge routing table stability via 5-second sampling intervals and prefix add and forwarding path changes.

Analyze why the OSPF to EIGRP redistribution process advertises only one route when two equal-cost paths lead to the same destination, and how this affects routing.

On router B, IGMP selects a route learned from s.r.o 1 and installs it as an external route with administrative distance 170 in the IP routing table.

Learn how BGP processes and advertises routes from internal neighbors, using the network statement and redistribution from static or other IGPs, with prefixes in the IP routing table.

Explore how BGP selects the best path from neighbors, advertises routes with administrative distances (eBGP 20, iBGP 200), and applies split-horizon, access lists, prefix lists, and route maps for filtering.

Explore the BGP neighbor table on a router, listing configured peers with their autonomous system numbers, internal or external status, session state, capabilities, uptime, and prefixes exchanged.

Configure BGP neighbors manually by IP and host number, initiate a TCAP session to exchange open messages for AS numbers and address families, and reset peering if AS numbers mismatch.

Identify causes for BGP peering failure, including lack of IP connectivity and non-direct paths, mismatched source and neighbor IPs, and AS number mismatches in open messages.

Examine a BGP troubleshooting example where the 192.168.224.0/19 prefix learned via BGP is installed in the routing table as the best internal path.

Troubleshoot BGP neighbor configuration with an ISP, identify why the BGP neighbor address fails, and verify correct IP addressing to ensure successful peering.

Diagnose a bgp troubleshooting scenario for the 192.168.2.0/24 network advertised by isp one to route to isp one, not through isp two, and configure a neighbor relationship with isp one.

Explore troubleshooting various dynamic routing protocols such as eigrp, ospf, and bgp, using trace route, ping, and show commands to verify connectivity, adjacencies, and routing table paths.

Review chapter 5 troubleshooting commands: use show ip route, show ip route [network/mask], and show ip cef to verify routing and forwarding, arp, and frame relay mappings.

Identify how routing protocols handle reception of routing information from neighbors, manage route data structures, perform route injection or redistribution, and select and install routes while transmitting information to neighbors.

Learn how EIGRP uses three data structures—the interface, neighbor, and topology tables—and view them with show ip eigrp interfaces, show ip eigrp neighbors, and show ip eigrp topology.

Identify how OSPF uses four main data structures: the interface table, the neighbor table, the link state database, and the routing information base to determine reachable networks and path costs.

Enable ip route profile in global config to monitor route stability; it tracks routing table changes over five seconds and shows results with show ip route profile.

Explore common BGP session failures, including IP connectivity issues, non-direct peering, and AS number or source IP mismatches that prevent neighbor establishment.

Learn to gather bgp data with show ip bgp summary and show ip bgp neighbors, then observe exchanges using debug ip bgp and debug ip bgp updates with targeted filters.

Explore troubleshooting addressing services in Cisco networks, and learn practical techniques to identify and resolve addressing issues.

Describe common NAT and PAT issues, explain dynamic host control protocol basics, and identify IPv6 routing internal issues.

Review how nat and pat affect nat sensitive protocols such as ipsec, icmp, and sip, and examine nat traversal methods, vpn considerations, and application-aware handling for voice, video, and dns.

Emphasizes using diagrams for NAT/PAT configuration, showing how IP NAT inside/outside interfaces, NAT pools, and routing reachability interact and require advertising translated addresses.

Apply conditional debug to isolate NAT/PAT issues by defining a debug condition for an interface. Keep outputs focused by using show debug condition and removing the condition when finished.

Explore a nat/pat troubleshooting example showing fast ethernet 0/0 as the inside interface and s0/1 0/8 0/1 as outside, with dynamic and static nat, using show ip nat translation to view translations.

Examine a NAT static translation entry in the NAT translation table and diagnose routing issues caused by no route back or overlap with a dynamic address pool.

Apply NAT/PAT troubleshooting by using a special iOS debugging tool to reveal filters, trace path from intermediate routers, and diagnose a failed TCAP session caused by a remote reset.

Explore NAT/PAT troubleshooting with an incorrect inbound access-list scenario, showing how a port translation mismatch and packet denial arise on a serial interface.

DHP overview explains how a client obtains IP configuration and DNS details from a server via broadcast discovery across subnets, with routers forwarding using IP helper-address.

Demystifies the DHCP client–server exchange, showing how a client broadcasts a discover from 0.0.0.0, receives offers, selects one with a request, and receives an ack with configuration parameters.

Troubleshoot dhcp snooping to secure address allocation and avoid blocked transactions. Address improper trust boundary configuration, missing dhcp snooping on lans, and misconfigured rate limits that degrade performance.

Diagnose DHCP issues by locating servers and clients, verifying relay agents and pool sizes, and checking ACLs, firewalls, and DHCP snooping on correct interfaces.

Check the DHCP server statistics and verify the address pool with show ip pool; confirm 254 addresses are available and none allocated, indicating correct server and client configurations.

Port 67 is now active, illustrating a DHCP troubleshooting step, and Cisco IOS replaces show IP socket with show UDP and show socket commands in release 12.4(11)T.

Explore how a router acting as a dhcp server can cause ip address duplication, and learn to verify dhcp pool settings and timers using running-config and dhcp pool commands.

Identify and verify excluded IPs from the DHCP dynamic pool using a shell command to prevent static addresses from devices like servers and printers from causing IP conflicts.

Reintegrate the 10.1.100 address into the DHCP pool and exclude the static range 10.1.0.1 to 10.1.1.20 to ensure unique IP assignments; then renew leases to resolve past address conflicts.

Identify and diagnose a DHCP relay agent issue by evaluating client misconfigurations, relay agent or server faults, and network filtering or security barriers.

Compare IPv6 and IPv4 to reveal similarities in command syntax and differences that affect troubleshooting; IPv6 uses no broadcasts, ICMPv6 multicast neighbors, and longer subnets like /96 or /128.

Learn how IPv6 debug commands support troubleshooting to identify symptoms and isolate problems, including neighbor discovery, duplicate addresses, and auto configuration issues, and understand ICMPv6's role in IPv6 failure conditions.

Determine the interface status using the show ipv6 interface commands, confirm the interface is up at both layers, and verify a global unicast and link-local address.

Verify the fast ethernet interface configuration to ensure it uses stateless autoconfig to automatically obtain the IPv6 prefix and other information.

Examine IPv6 auto configuration address and its valid and preferred lifetimes, based on ICMPv6 router information, and confirm connectivity with remote pings.

Analyze IPv6 troubleshooting through bi-directional redistribution between RIP IPv6 routers, configured with show ipv6 protocols, to advertise Ethernet and loopback networks; note the use of a 15 metric to influence route selection.

Verify IPv6 RIP configuration on the router using the IPv6 RIP command to obtain a snapshot of enabled interfaces and check for no interfaces enabled on the frame relay link.

Troubleshoot IPv6 routing by enabling the routing process on the serial interface and using debug IPv6 routing to observe missing routes added to the routing table.

Explore IPv6 troubleshooting example 3, diagnosing OSPFv3 configuration errors in a backbone area 0 with stub areas 1 and 2, amid a power outage.

Apply good troubleshooting by pinging multiple addresses or interfaces to verify reachability; the fast Ethernet 00 interface on router 1 succeeds while the same test from R2 fails.

Verify IPv6 OSPF configuration on the interfaces using the show ipv6 ospf interface command, ensuring OSPFv3 is enabled in area 0 on both interfaces.

Troubleshoot edge and border routers and tunnel endpoints, and review a tunnel command output that shows IPv6 ping success and encapsulation/decapsulation messages.

ipv6 tunnel is configured and operational, but ospf fails across it; static routes restore connectivity, while bgp is not used in this scenario for dynamic routing across the tunnel.

Master troubleshooting techniques and address services as outlined in chapter 06 review of the tshoot: troubleshooting and maintaining Cisco course.

Analyze NAT sensitive protocols and exclusions like IPsec and ICMP, and master essential troubleshooting commands: show ip nat translations, show ip nat statistics, and debug ip nat.

Explore how enabling router interfaces supports udp broadcast for six protocols, including dns (53), time service (37), netbios name service (137), netbios data service (138), tacacs (49), and dhcp/bootp (67–68).

Explore essential Cisco IOS IP version 6 troubleshooting commands, including debugs and show IP version 6 interface, routers, route, and protocols.

Describe and troubleshoot network application services, and identify performance issues on Catalyst switches and routers.

Cisco designed NetFlow, now in its ninth version on the standards track to become industry-wide standard, creating a net flow cache for active flows to enable traffic accounting and security.

Netflow overview defines a flow as a unidirectional packet stream between a source and destination, matching seven fields such as source IP, destination IP, and input interface, including tos/dscp.

practice IP SLA overview by simulating a source device sending a generated packet to a responder destination, which replies with timestamp information to compute performance metrics using UDP.

Cisco auto qos autodiscovery unfolds in two phases: discovery gathers baseline traffic classes and volumes using Embarq, then configuration applies qos policies based on collected templates.

Explore how the monitor keyword was removed from show ip sla commands in Cisco IOS release 12.4, with the deprecated monitors replaced by show ip sla statistics for troubleshooting.

Review IP SLA monitoring statistics showing a return code of OK, with one success and no failures.

Troubleshoot auto qos by ensuring the interface uses the correct encapsulation, switching from bp to hdlc, and enable auto qos to restore connectivity.

Analyze auto qos multi-link fragmentation on low bandwidth interfaces and configure bandwidth for a serial interface, illustrating why a 200 kbps setting indicates a slow link.

Identify switch performance issues by examining packet loss with the show interface command, which reports input and output packets, error counters, and per-interface statistics (unicast, multicast, broadcast, bytes).

Explore show interface commands and the parameter descriptions they reveal, with guidance on possible causes and issues for each parameter. For deeper details, refer to Cisco's book.

Analyze switch forwarding hardware on an access switch with 24 fast ports and a 1 gbps uplink, totaling 2.4 gbps, and explain congestion and frame drops when all ports transmit.

Identify causes of high switch cpu load on layer 3 switches from command tasks, routing updates, and snmp polling, including show tech-support, show running-config, and copy running-config startup-config; adjust polling.

Examine dhcp issues arising from misconfiguration, such as limiting dhcp messages per second on an interface, which can degrade switch performance and stem from poor baselining and dhcp snooping tuning.

Identify DHCP vulnerabilities and abuse methods, including starvation attacks and rogue servers that redirect traffic, and mitigate with DHCP snooping, port security, and rate limits.

Explore common HSRP issues that cause instability and performance degradation, including duplicate standby IP addresses, pre-staged changes, timer misconfigurations, lapping links, missing peers, and related switch error messages.

Diagnose switch performance issues caused by excessive broadcasts to resolve connectivity, IP address assignment, and slow download problems using a path troubleshooting technique.

Troubleshoot switch port performance by checking speed, duplex, and utilization; verify the port operates at full duplex 1000 mbps with near-zero utilization and interfaces up.

Identify switch performance issues by analyzing the show process, noting excessive load at 90 percent over five seconds, 94 percent over one minute, and 2 percent over five minutes.

Use the show process command to identify which task or process consumes nearly half the switch's resources and assess its impact on performance.

Use bottom-up approach to determine if layer 3 or 4 security policies block traffic, starting at PC, verifying switch interface with show interfaces, and clear counters to address intermittent connectivity.

Examine the VLAN 10 access list and its large output, similar to VLAN 10 underscore out, to determine if it degrades performance and prevents user connections.

Identify how network background tasks, IP background tasks, and TCP timer processes contribute to high router CPU load and how to diagnose them.

Explore how CEF status affects switching paths: if CEF is enabled globally, it may not be enabled on an interface.

Enable cef and use the show ip cef command to inspect the fbi table and fcf status, noting next hop and the gateway of last resort as default route.

Identify why some packets are processed even with CEF enabled due to an incomplete adjacency table or main processor handling. Gather information with the show CEF not C-f switched command.

Analyze utilization and load from packet processing using the show processes command and pipe filtering. The example shows no problems related to packet processing, marking step 2.

Analyze packet forwarding by identifying the switching mode used by the router and the interfaces involved, and use the show IPCA command to verify if C-f is enabled.

Continue analyzing step 4 by using the show IAP interface command to verify switching types, noting global CEF is enabled and all interfaces support CEF switching.

We examine the IAB entry for routing information in Step 5, using example 10.11.1.1, and confirm the adjacent entry shows interface FastEthernet 0/1 with the next-hop address.

Verify the ARP cache entry and MAC address information on the router in step 7 to conclude there are no switch-related problems.

Diagnose and resolve router memory issues by understanding how fragmentation prevents finding a usable memory block in Cisco IOS, and how packet memory buffers incoming and outgoing packets.

Troubleshoot router memory issues in Cisco networks, part 2, and learn practical techniques for diagnosing and maintaining memory performance.

Identify security-related causes of router memory issues, such as worms or viruses, especially when there are no recent changes; mitigate by adding an access list to drop malicious traffic.

Identify how BGP memory use affects router stability by examining processes: BGP input/output, the BGP router, and the BGP scanner, including memory for the BGP RIB, IP RIB, and prefixes.

Identify and monitor memory availability for routes and line cards on a router, using show dialog outputs to compare memory across card types and detect BGP memory shortages.

Review the chapter on troubleshooting network performance issues and note the key topics discussed in this chapter.

Explore Chapter 7's application services, including network classification, scalability, networking, acceleration, wide area network acceleration, and optimization, plus the four-step cycle: baseline traffic, optimize, measure, adjust and verify, deploy.

Explore net flow concepts and how they enable traffic accounting, billing, planning, security, and monitoring, focusing on unidirectional flows defined by source and destination IPs, ports, protocol, and input interface.

Ambar enables baselining and traffic classification by recognizing applications, including web-based protocols and UDP port assignments, through protocol discovery; SLB provides server load balancing with a virtual server farm.

Apply a three-step process to troubleshoot performance: assess technical nature, isolate to a device or component, and diagnose at the component level, noting routing protocol updates and SNMP polling.

Identify memory allocation failures by symptoms like low memory messages and no output from show commands, and note causes such as insufficient memory, memory leaks, security problems, or buffer leaks.