Great to hear that the problem has been resolved. It’s often stressful when you’ve got a production network not working and you’re trying to troubleshoot. Now that everything is OK, you have the time to go over issues calmly and cooly to learn for the future!
Anyway, now that I’ve said that, here goes:
Troubleshooting multicast packet loss should not be different than troubleshooting packet loss in general. Multicast doesn’t inherently add any particular characteristics to packet loss that would not be present under other circumstances. Both the Mstat and the Mtrace utilities are useful for troubleshooting multicast routing failures and to examine the behavior of a multicast topology including statistics and choice of path, however, they don’t deal with all types of packet loss. Only packet loss that may be due to faulty multicast routing.
Unless your packet loss is due exclusively to faulty multicast routing, packet loss is typically due to other issues independent of multicast, that have to do with congestion, corrupted packets, MTU issues, or misconfigurations in ACLs route maps, or other policies.
Indeed, Wireshark is not so helpful here because of the reasons you give, but also because of the fact that you need to know where to collect packets on the network. You have to first zero in on the problem and then choose what packets to capture.
However, for voice applications including protocols such as SIP, RTP etc, Wireshark can be VERY helpful because it has a great telephony utility where you can follow a whole conversation and see the SIP exchanges and the voice packets of a particular conversation. It analyzes in detail these exchanges In such a case, you can examine the quality of the voice conversation. All of these tools are found under the “Telephony” menu option in the application.
I have a question regards to how a receiver can join a mcast group during register suppression window.
Lets assume a publisher is trying to register the source with RP and currently no active recipients are there.
So RP sends a register STOP message to source and a register suppression timer starts.
During this period, if an active receiver sends a PIM join message for that group to RP, how long the receiver need to wait to receive the MAST stream? Does it need to wait till the end of register suppression timer window ? Does Source sits idle til the timer expiry before it cant start sending the stream?
This is an excellent question. If a subscriber sends a join message to its local router, this will be forwarded as expected to the RP. If a PIM Register-Stop message had previously been sent to the first-hop router (the router connected to the multicast source), the RP will typically wait for the first-hop router to send a new PIM Register message. This is assuming a strict adherence to the “rules” of this operation. This means that the host wanting to join will have to wait for this timer to elapse so that the first-hop router will start sending multicast traffic to the RP once again.
However, as you have suggested in your post, this seems inefficient. So I tried labbing this up, and found that if a join message reaches the RP during that timer, the RP will automatically send a join message to the first hop router, informing it that it wants to receive traffic. The first hop router in turn will ignore its timer and will begin sending multicast traffic.
This is highly vendor and platform specific, and it may be that some IOS versions or other vendors don’t do this while others do. It is likely that most modern implementations do this to avoid the inefficiencies involved.
i work in a low latency trading environment where multicast is heavily used.I was giving a presentation on PIM JOIN and REGISTER and one of the dev guys asked this question and i was clueless for some time.I will get back to him with this explanation
Thanks again
Bejoy
I am practicing configuring PIM parse-mode on multi layer switches and had a few questions regarding this type of implementation. I am using Cisco modeling labs.
Assume we were using a single multi-layer core switch (Core-1), and 2 access layer switches with a multicast source, and a few multicast receivers. The multicast Source and the first multicast receiver are connected to the first access switch, on the same VLAN, lets call this VLAN 5. The second multicast receiver is connected to the second access switch on a separate VLAN, lets call it VLAN 10. The multicast source is sending multicast packets with a destination IP of 239.0.0.8 that both receivers would like to receive.
Assume the default gateway for VLAN 5 is an SVI of 192.168.5.254/24 on the core switch, and an SVI of 192.168.10.254/24 for VLAN 10’s gateway.
I am using two routers as my receivers, I turned off routing on them, configured an igmp join group for 239.0.0.8 on their interfaces, and assigned them a default gateway so that they can communicate with other subnets.
Heres my config for the core:
Core-1:
ip multicast-routing
int vlan 5
ip address 192.168.5.254 255.255.255.0
ip pim sparse-mode
int vlan 10
ip address 192.168.10.254 255.255.255.0
ip pim sparse-mode
ip pim rp-address 192.168.10.254
The configuration on the access switches are just access ports on each respective VLAN for the Multicast source and receivers, and the uplinks are configured properly to allow frames from the proper VLANS across to the core. IGMP Snooping is also configured for each VLAN in the access layer switches.
I am not able to route multicast traffic with this current configuration, and I dont even see what I was expecting to see when I do sh ip mroute on the core, but at layer 2 multicast traffic flows within the same VLAN fine, and I see the receivers respond when I use a ping to 239.0.0.8 on my source. It may just be the cisco modeling labs itself, but my questions to verify my work are the following:
Do i have to do anything on the access layer switches for this to work? My thoughts are that as long as they each have their mrouter port from receiving PIM hellos or igmp membership queries, they are able to forward multicast traffic to the core.
Does all multicast traffic regardless of the igmp snooping mechanism get forwarded towards the mrouter port?
What are you using to determine if you are able to route multicast traffic? Also, what do you actually see as the output of the show ip mroute command?
Multicast will flow just fine in Layer 2 switches, even if you don’t configure the mrouter port, it will simply flood all multicast traffic as if it was broadcast.
Now having said that, your configuration seems fine at first glance. Here are the responses to your questions:
In a typical scenario, you don’t need to do anything on the access layer switches for multicast routing to work. They should be able to forward multicast traffic to the core switch as long as they have their mrouter port from receiving PIM hellos or IGMP membership queries.
In a typical IGMP Snooping setup, all multicast traffic is forwarded towards the mrouter port, assuming it is configured. IGMP Snooping is a mechanism to constrain the flooding of multicast traffic by dynamically configuring Layer 2 interfaces so that multicast traffic is forwarded to only those interfaces associated with IP multicast devices. Therefore, when a switch receives the IGMP membership report from a host for a particular multicast group, it adds that host’s port number to the associated multicast table entry. It then forwards future frames for that multicast group directly to that host by forwarding the frames out the port connecting to that host.
As you suggested as well, one thing to note is that CML may not fully support all multicast routing features and functionalities, so the problem may be due to CML’s limitations.
Let us know how you get along, and if we can be of further help!
@lagapidis
im somewhat confused about packet encap level
i think i have good grasp about the overall process igmp + pim but i’d like to focus on the pim register msg :
S1 forwards mcast traffic to 239.1.1.1 grp , its received by the DR, R1 who sends a PIM register msg to the RP and this is the phase i’d like to do a pause… 1st the DR previously had to discover the RP (static, auto-rp (sparse-dense or auto-rp listener) or bsr) so when the DR already knows how to reach the RP it can send the PIM Register msg to this RP … and for what i know and read on the correspondent lessons, this msg is unicast (https://www.cloudshark.org/captures/611435f7d00d)
Src: 192.168.12.1 (DR), Dst: 2.2.2.2 (RP)
So in a nutshell the encap here is : ICMP → IPv4 (SRC 192.168.1.1 DST 239.1.1.1) → PIM (if we expand it we can see the type is a register (1) , and flags where we can see “0100 … = IP Version: IPv4 (4)” as one of its flags , does it mean PIM is encapsulating the previous IPv4 mcast packt ( SRC 192.168.1.1 DST 239.1.1.1) ? → IPv4 unicast (Src: 192.168.12.1 (DR), Dst: 2.2.2.2 (RP) → L2 Ethernet → L1
This was for Register (1) , Register-stop (2) is basically the same except as the lesson mentioned is does not contain any payload (ICMP for this example) so Flags is hex Flags: 0x40000000 and the Null-Register Bit is 1 (yes)
EDIT : I also have a question about Tunnel0 and Tunnel1. Tunnel0 is only on non-RP Routers for forwarding this kind of message ? which kind of tunnel is this ? Tunnel0 and Tunnel1 is in RPs for both, encap and decap … but i think im missing something here.
Now the interesting thing is that I have been going through some of the PIM RFCs, and looking at the header structure of the PIM register message (which is type 1 as seen in the Type field). I haven’t been able to find any information about the PIM Options fields! The flags are not defined, and they don’t seem to exist in the RFC. If anyone has any info about this, please feel free to post!
I personally believe that the flags in the PIM message indicate the IP version of the original multicast packet, so it means that the PIM is encapsulating an IPv4 multicast packet.
The Register-Stop message, as you mentioned, doesn’t contain any payload. It’s sent by the RP to the DR to signal that it should stop sending Register messages for a particular source/group pair. This happens when the RP has learned about the source via the normal multicast routing process.
As for your question about Tunnel0 and Tunnel1, these are commonly used in PIM implementations to handle PIM Register messages. Tunnel0 is typically used on the DR to encapsulate the original multicast packet within the PIM Register message, while Tunnel1 is used on the RP to decapsulate the original packet from the PIM Register message.
I have a scenario that requires configuring multicast with multiple vlans using only L3 switches. I am using a Cisco C9300X-24Y-A fiber switch as the core. The access or leaf switches are all Cisco C9200L-48P-4X switches. The C9300X is the FHR and LHR and no traffic needs outside of these L3 switches. My solution is this:
Enable ip routing and ip multicast routing on the Cisco C9300X
Create the required VLANs and SVIs for the multicast traffic on the C9300X.
create vlan number
name vlan name
interface vlan number* ip address ip address* subnet mask
Enable PIM sparse mode on each of the SVIs ip pim sparse-mode
Configure an IP address on each switch in an Management VLAN with a default route configured on the C9200L switches pointing to the C9300X switch.
Create and configure trunk links from the C9300X to the C9200L with the required VLAN
Enable IGMP Snooping per vlan or per port on each C9200L switch.
Configure each access port on the C9200L to be in its respective VLAN.
Your solution seems to be on point and should work for your scenario. I assume that the 9200 switches are only functioning at Layer 2, correct? Only the 9300 is operating at layer 3, and that’s why it is the FHR and LHR, right?
Assuming that is the case, I would only suggest that after enabling PIM sparse mode on each of the SVIs, you should also designate the 9300 as the rendezvous point (RP). This is necessary for PIM sparse mode to work correctly. You can use the ‘ip pim rp-address’ command to do this. Otherwise, it all looks good! Let us know how your implementation goes.
Thank you for the reply. So, yes the 9300 is L3 and the 9200s are L2. My follow up question is that if I designate the 9300 as the RP, since this network is small enough I can manually configure the RP on each switch, but I would have to create a route to the RP?
Yes, you are correct. If you are designating the 9300 as the RP, you can manually configure the RP on each switch. That way, because your network is small, you can avoid using AutoRP or BSR which automatically pick and advertise the RP. This will simplify your config.
As for your question about creating a route to the RP, yes, you have to make sure that the IP address that is used as the RP is reachable from all devices. Just make sure that the routing protocol or the static routing you configure allows all devices to be able to route to that address.
This network is configured to support a virtual video routing appliance. The appliance, called a Director, resides in VLAN 5 and uses a multicast discovery mechanism at address 239.255.255.250 UDP port 3702. This appliance is connected to the Cisco 9300 mixed stack comprised of a 9300-24Y (fiber) and 9300-24P (copper). The issue is that the appliance only discovers the end points residing in it’s own VLAN, and not in any other vlan.
I have ip routing and ip multicast-routing enabled on all the switches
PIM is configured on all the SVI interfaces
a management vlan, VLAN 2 is configured for ssh and routing
a static default route on the access switches points back to the core switch over VLAN 2
ip igmp snooping vlan (vlan) immediate-leave is configured on all the VLANs
ip pim rp-address 2.2.2.2 is configured on the core
an ip route on the access switches points to the RP address on the core
trunks are configured from access switch to the core allowing vlans 1-11
I am not certain what I am missing, but this should be pretty straight forward!!!
After looking over your configs, I don’t see a clearcut problem that may be causing your issue. However, some troubleshooting steps you can take to determine the issue include the following:
IGMP snooping: IGMP snooping can prevent multicast traffic from being forwarded to all ports in a switch VLAN and can limit the forwarding of multicast traffic by dynamically configuring Layer 2 interfaces so that multicast traffic is forwarded to only those interfaces associated with IP multicast devices. You could try disabling IGMP snooping on all VLANs and see if that resolves the issue.
Check Multicast Routing Table and Memberships: Use the following commands to check that your multicast routes and IGMP memberships are as expected:
show ip mroute
show ip igmp groups
Confirm RP Reachability from All Access Switches: Since the RP address is static, ensure all devices have clear paths to the RP (2.2.2.2). Verify that the static route pointing to the RP works correctly and there are no other routes or access control lists that could interfere.
Try these out and let us know your results so that we can help you out further…
In this particular case, the specific (192.168.1.1, 239.4.4.4) multicast route is pruned because R2 no longer requires that traffic. It has no hosts that request it. The multicast traffic destined for H3 is handled by R4 and no longer needs to pass through the RP which is R2. In order for you to see the P flag, R4 must send a prune message to R2. In order for that to happen, R4 must make the switch from the RPT to the SPT. To troubleshoot your topology, make sure this takes place on R4 for the multicast group that H3 is registered to.
This result is not due to the ip pim spt-threshold infinity command. This command will not affect the pruning on R3.
The purpose of the ip pim spt-threshold infinity command is to prevent the switching over from the RPT to the SPT. This is useful when the SPT and the RPT are the same from the point of view of the router in question. This is the case for R3, so this command is only needed on R3.
When configuring Multicast, I often see the Source-Based tree called the Shortest-Path tree. It’s basically the tree where the multicast sender (or the FHR connected to it) is the root of the tree and it involves the shortest/best path to reach a particular destination.
My question is, is every source-based tree also a shortest-path tree? In Sparse-Mode, between the FHRs and the RP, assuming that a multicast stream is active, a source-based tree is built which is used to forward the flow to the RP.
Is this also a shortest path tree? Because technically, it doesn’t go straight to the destination LHR but to the RP instead so the path between the FHR and the LHR isn’t the shortest or the most direct.
First of all, let’s clarify some terminology: Source-Based Tree (SBT) and Shortest-Path Tree (SPT) are two terms that refer to the same thing, just from slightly different angles. SBT emphasizes that the tree is routed at the source. The SPT emphasizes that the path taken is the shortest route based on the unicast routing table from the source to each receiver. In any case, it is indeed the same thing.
Yes because they are the same thing.
Well not quite. The initial path that multicast traffic takes in sparse mode is indeed from the FHR to the RP and then to the receiver. This is not necessarily part of the SBT, and indeed in most cases it is not. Eventally, the last hop router (LHR) may decide to switch to the SBT assuming they require a more optimal path. This will take place depending upon the ip pim spt-threshold.
No, because between the FHR and the RP, we are actually joining the Root Path Tree (RPT), or the shared distribution tree. The RP is the root of this tree, and it decides where to forward multicast traffic once it reaches the root. This is a temporary control plane mechanism until the threshold kicks in. Does that make sense?
