Analyze traffic and uncover Hyper-V networking problems has never been easier
Networking problems frequently challenge administrators. Introducing a virtualized switch to the mix adds another layer of complexity and multiple failure points. We can use the popular Wireshark tool to analyze traffic and uncover problems.
Requirements for Success with Wireshark
First, you need the software. You can download Wireshark from Wireshark.org. The site includes substantial information and links to more. Due to the extensive depth of the tool, the value that you get from Wireshark depends directly on how well you’ve learned it. Ideally, you’d go through a guided course and practice on training captures. I understand that you might have more immediate needs. This article illustrates enough to get you started but expect to invest time in training and practice.
Second, you need a working knowledge of Ethernet frame structure. You do not need anything near expert level, but you won’t get far if you can’t make sense of what Wireshark reveals. We have an article series on basic networking that can get you started.
Remote Captures in Wireshark
Wireshark can capture information on remote systems. However, it includes more hints than details. I could not find any directions that I felt comfortable sharing. Fortunately, you have alternatives.
Wireshark will run on Windows Server. Because it relies on the Qt library for its graphical interface, you can run the entire program on a Core mode installation by manually starting “C:\Program Files\Wireshark\Wireshark.exe”. I have no objection to running Wireshark on a server. However, I do not like RDP or similar remote connections to servers. These technologies present a significant attack surface for malware and intruders. Use at your own risk.
During the Wireshark install, you can also select the TShark program, which gives you command-line access to captures. TShark works inside a PowerShell Remote session. That means that you can install TShark on a system that you want to capture “remotely”, output its capture to disk, and then import it into a management system. I will not spend much time on TShark in this article, but I will get you started.
TShark Fundamentals
First, install at least the TShark portion of Wireshark on the target server. That might require a remote desktop connection as Wireshark has no official support for remote or scripted installation. However, running “Wireshark-Win64-<VER>.exe /s” at a command prompt, (or via a script, or possibly even a remote session), should install the software with default options.
Second, open a remote PowerShell session to the server using credentials with administrative privileges on the target:
Connect-PSSession -ComputerName <SERVERNAME>
Alternatively, you can supply credentials at the point of entry:
Connect-PSSession -ComputerName <SERVERNAME> -Credential (Get-Credential)
Once you have your remote session, run Get-NetAdapter to retrieve a list of adapters on the remote server:
Locate the adapter(s) that host the Hyper-V virtual switch on the server and note the value(s) for ifIndex. In my case, I want interfaces 4 and 10. With that knowledge, initiate TShark. Tell it which interfaces to include in the capture and where to write an output file with the -i and -w switches, respectively. That looks something like this:
& ‘C:\Program Files\Wireshark\tshark.exe’ -i 4 -i 10 -w C:\Users\esadmin\Documents\cap.pcapng
You do need the leading ampersand. If you use tab completion for assistance in entering the path to TShark, PowerShell will insert it automatically.
Upon pressing [Enter], the capture starts and writes to the file. Most importantly, you need to know that pressing [CTRL]+[C] stops the capture. Because we did not specify a capture limit, it will run until we either cancel it or the remote system runs out of disk space. Less importantly, the TShark program does not generate all its console output in a way that PowerShell remote sessions can process. You will see some things that look like error messages and other things will not appear at all. Just remember how to start and stop the capture and you will get the expected capture file.
TShark allows you to restrict captures with limits and filters. I will leave learning about that to you. Start with tshark.exe –help. The instructions above will generate a capture file that, at worst, has more data than you want. Once you have that file, you can transfer it to your management workstation and use Wireshark to operate on it.
A Warning about Wireshark and Resources
Wireshark will write to capture files, but it defaults to keeping captured packets in memory unless told otherwise. When possible, only run captures for the time needed to gather the data relevant to the problem you want to solve. Take care to set limits on long-running captures to ensure that you do not consume all host memory or disk space. Remember that a full disk will cause any VMs on that disk to pause. Also, remember that Hyper-V prioritizes processes in the management operating system, so it will squeeze virtual machines as needed to provide CPU and memory resources to Wireshark.
Set capture limits from Wireshark’s main interface by clicking the Capture menu item on the menu bar and then clicking Options.
The Input tab allows you to select the adapters to watch and to define capture filters. The Output tab gives you options for writing to files. You can set finite capture limits on the Options tab that apply whether writing to memory or disk, along with some handy quality-of-life settings.
While we frequently want to capture all data so that we don’t miss environmental problems, you can greatly reduce capture size with capture filters. Unlike display filters, capture filters tell Wireshark to discard information without storing it. Use these cautiously; if you inadvertently throw out interesting frames, you’ll have to perform additional captures.
Finally, remember that 10GB and faster interfaces can already generate heavy CPU loads. Using Wireshark to capture and decipher frames costs that much more. Few systems drive their networking capabilities anywhere near their maximums but remain mindful.
Traffic Must Pass a Physical Adapter for Wireshark to Capture It
With the current way that the Hyper-V virtual switch projects into the management operating system, Wireshark cannot bind directly to it. Instead, we attach it to one or more physical adapters. This means that, at the management operating system level, Wireshark cannot intercept any traffic that never leaves the VMBus.
The VMBus limitation primarily impacts internal and private virtual switches. Without a physical adapter, you have few options. If you have an unused physical adapter, you could temporarily bind the virtual switch to it with Set-VMSwitch. If your host uses the older LBFO technology, you can add a team NIC in another VLAN and bind your virtual switch to that. Even with these alternatives, you will still miss anything that does not cross the bound adapter.
However, this should only present a problem in edge cases. Wireshark and TShark can operate just as well inside a virtual machine as they can in the management operating system. Wireshark does not distinguish between virtual and physical adapters. Set it to watch the virtual adapters involved in your communications chain, and you’ll see the traffic. If you can’t install either product inside a given virtual machine, you still have Hyper-V’s port mirroring feature.
Capturing All Virtual Switch Traffic
When you don’t know exactly what you’re looking for, which applies well when you don’t have much experience with network captures, just get everything. When you first open Wireshark, it will present all network adapters that it can operate with. Find the physical adapters that host your virtual switch and highlight them:
Remember that choosing anything that says “vEthernet” in its name binds to that virtual adapter, not the virtual switch. For switch monitoring, you must choose the physical adapter(s).
You can either right-click your selection and click Start Capture or you can click the blue shark icon at the left of Wireshark’s icon menu. If you made a mistake in adapter selection or just want to change it after the capture has started, select Options from the Capture menu:
Once the capture starts. you should see a rapidly scrolling screen like the one below. If you’re working on a problem, reproduce it while the trace runs.
Once the trace has captured enough information, click the red square button on the toolbar to end it. Regardless of your intentions, I recommend saving the file. It’s better to have a capture file that you don’t need than the opposite.
You can scan through the capture to look for anything that seems out of place or just to acclimate yourself to a network capture. If you’ve never used Wireshark before, the topmost pane shows a list of captured frames with some basic information about each. The middle pane tries to break the selected frame down into its individual components. Click on the triangle icon to the left of any item to drill down further. Wireshark uses “dissectors” to interpret frame components. Anything that it doesn’t recognize goes into the generic “Data” portion. The third pane shows a binary dump of the frame. If you click any part of that, the dissector pane will shift focus to that location.
Listings such as this allow you to peruse the activity crossing your virtual switch. You can investigate whatever interests you.
Exercise 1: Capturing Virtual Switch Traffic by Port
Tracing traffic by port can help you locate breaks in communication. It helps you to discover if messages that you expect to arrive on a virtual machine ever make it to the virtual switch at all. You can ensure that servers on virtual machines respond to clients as expected. You can watch for traffic coming from unexpected (potentially malicious) sources.
In my example exercise, I want to verify that my “primary” domain controller properly receives and responds to authentication traffic. For the most basic trace, I can set a display filter on a previously captured file or on an active trace with this format: tcp.port == 389:
For thoroughness, I want to look at all traffic that a domain controller would utilize for authentication traffic. I can filter to multiple ports like this: tcp.port == 88 or tcp.port == 389 or tcp.port == 636 or tcp.port == 3268 or tcp.port == 3269
Pressing [Enter] or the white arrow with the blue background at the end of the filter field will update the display to show only frames that match the filter:
Scanning the filtered view, I see frames that it clearly identifies as LDAP and others that it marks only as TCP. When Wireshark cannot identify a frame, look to the Info column. In the third row of the screenshot, we see that it has marked the frame as [ACK]. That tells us that the frame contains an acknowledgement of a previously received frame.
If I want to find out what the frame acknowledged, I can right-click on the line item, hover over Conversation Filter, and choose one of the offered items. In this case, I don’t want to miss anything, so I choose Ethernet as the least specific filter:
In response, Wireshark pares down the display to only the items that belong to that particular “conversation”. Also, notice that it updated the display filter:
I know that 192.168.5.1 belongs to the domain controller of interest. I also know that 192.168.5.2 belongs to my “secondary” domain controller. Therefore, before I even performed any of these tasks, I could have guessed that these frames carry requests or updates that keep domain information synchronized. To confirm, I select the first frame in the conversation in the top pane. In the second pane, I find the Lightweight Directory Access Protocol section that indicates a dissector has come into play. In the bottom frame, I locate the highlighted information (remember that this matches whatever I selected in the middle frame):
The frame appears to have something to do with DNS settings. I look at the same portion of the second frame:
We already know that the third and final frame in the conversation is an ACK. So, we can surmise that 192.168.5.2 asked 192.168.5.1 about SVDC02 as a DNS server, got a NO_OBJECT result, and acknowledged receipt of the result. It appears that I may have some DNS troubleshooting to do.
However, I was interested in authentication traffic. We learned that the tracked conversation dealt with DNS servers. I can return to my previous filtered view by clicking the drop-down arrow at the end of the filter line and choosing the filter that I want to see again:
Exercise 2: Including or Excluding Virtual Switch Traffic by IP Address
To continue with the scenario set up in exercise 1, I still want to see all the authentication traffic to my “primary” domain controller, but I want to exclude anything between it and my “secondary” domain controller. The simplest display filter looks like this: ip.addr != 192.168.5.2. If I wanted to only see traffic on that IP, then I could use double equals (==) or eq instead of!=.
Of course, I don’t want non-authentication traffic. So, let’s modify the filter to ip.addr != 192.168.5.2 and (tcp.port == 88 or tcp.port == 389 or tcp.port == 636 or tcp.port == 3268 or tcp.port == 3269). Pay attention to the usage of parentheses. This grouping tells Wireshark that we want traffic where no frame includes IP address 192.168.5.2 but does contain any of the TCP ports inside the parentheses:
The remaining list tells us multiple things:
- No non-domain controller except 192.168.10.1 talked to the domain controller during the capture (were we expecting traffic from someone else?)
- We see the beginning of a conversation between the domain controller and 192.168.10.1 (indicated by the SYN packets)
- 192.168.10.1 performed a bind and SASL operation
- All traffic was on port 389
- We see the end of a conversation (indicated by the RST, ACK packet followed by a FIN, ACK packet)
While not captured in the screenshot, the Info contents provide enough preview information for me to understand what the SASL conversation was about. However, I can click on the individual frames and use the other two panes to get a deeper look at the traffic.
Exercise 3: Determine the Physical Adapter(s) Used by a Virtual Machine
The Hyper-V virtual switch makes its own decisions when placing traffic on the members of a switch-embedded team. If you use the Hyper-V Port load balancing algorithm, it will affinitize each virtual adapter’s incoming traffic to a physical adapter. While it can dynamically change affinities in response to events, each virtual adapter will always receive on exactly one physical adapter. If you use the Dynamic load balancing algorithm instead, then Hyper-V can exploit Ethernet and TCP/IP characteristics to distribute physical adapter use down at the conversation level.
If you want to view its decisions in action, Wireshark can help. Get a capture of traffic on your switch’s physical adapters. Select any frame in the top pane. In the middle pane, expand the Frame group at the top, then the Interface item. Look at the Interface description field:
Skip around in a generic capture and look at the ways that it uses physical adapters. Notice how it freely distributes multicast and broadcast traffic as it sees fit. Notice how it picks an adapter for any given individual unicast conversation and keeps it there.
We will expand on this subject in the next two exercises.
Exercise 4: Determine the MAC Addresses Used by a Virtual Machine
This exercise may seem pointless because you can use PowerShell or the various graphical tools to find the MAC assigned to a Hyper-V virtual network adapter. Bear with me though, as you may see things that you don’t expect.
This exercise begins similarly to exercise 3. Pick a frame from the top pane and look in the center pane. The second section, after Frame, is Ethernet. It shows the MAC addresses involved in the frame, which probably aligns with what you see in your tools:
Then again, it might not:
In fact, even though it includes the IP address of a virtual machine (192.168.127.3, visible in the third row), neither the source nor destination MAC belong to a Microsoft virtual adapter. For this reason, I counsel against filtering Hyper-V virtual switch traffic by any MAC owned by a virtual adapter unless you’re doing something like validating MAC address spoofing.
How did this happen? Short answer: Hyper-V silently utilizes the MAC addresses of physical adapters when load balancing traffic from a single virtual adapter. If that seems strange, understand that physical switches do the same thing. Knowing the MAC address that Hyper-V assigned to a virtual adapter does not guarantee that the virtual switch will only use that MAC in conversations involving that adapter. The only Ethernet segment that absolutely must have the correct MAC for an adapter is its direct switch connection. In Hyper-V’s case, that connection only exists on VMBus which, as we discussed earlier, cannot be seen in Wireshark. If you want a longer explanation, I wrote an article that talks about how this very thing can cause problems when using a dynamic-mode Hyper-V virtual switch in conjunction with load balancers.
You can see the MAC-to-adapter matching by comparing the MAC to the interface ID or description (as shown in exercise 3). You can use this information to filter a virtual machine’s traffic by adapter as shown in the next exercise.
Exercise 5: Find Traffic for a Virtual Machine that Uses a Specific Virtual Adapter
We’ll combine what we learned in the previous two exercises to answer a specific question: how do I filter the traffic from a specific virtual adapter that crosses a specific physical adapter? In case you skipped the previous sections, this question only makes sense when your Hyper-V virtual switch involves a physical adapter team.
The part of the virtual machine that does not change is its IP address, so I will filter by that first. Next, I will have Wireshark look at the frame object. As you type the filter, it will make suggestions. I begin my filter with ip.addr == 192.168.127.3 and frame.. Note that this is an incomplete query, and it includes a period at the end of frame:
You can see that Wireshark makes suggestions to help us out. The subcomponent of frame that interests us is the interface, so start typing that to shorten the suggestion list:
If you recall the Wireshark-assigned interface ID from previous exercises, then you can select the interface_id subcomponent and that number. I like repeatable, memorable things, so I will use the interface_description with the name that I gave the adapter in Windows: ip.addr == 192.168.127.3 and frame.interface_description == PTL. You do not need quotes around the name:
My display now contains traffic for that virtual machine that uses the designated physical adapter, even though none of it includes the virtual machine’s “correct” MAC address:
Expect to see many frames marked “TCP Spurious Retransmission” on the physical adapter(s) that substitute their own MAC in place of the virtual adapter’s. Network load balancing does not come free.
Expand on these Lessons
This article only scratched the surface of Wireshark’s capabilities. Most importantly, it empowers you to see below the layer 3 and higher pieces that the virtual adapters deliver into the guest operating systems. You can now see the data that enters and leaves your virtual switch and use that knowledge to find the truth behind those vague “it must be something wrong with the network” excuses.
Source :
https://www.altaro.com/hyper-v/wireshark-hyper-v-networking/
Print
