0

Snooping on .NET EventPipes – lowleveldesign.org

 3 years ago
source link: https://lowleveldesign.org/2021/01/20/snooping-on-net-eventpipes/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

lowleveldesign.org

Software tracing, debugging, and security

While playing with EventPipes, I wanted to better understand the Diagnostic IPC Protocol. This protocol is used to transfer diagnostic data between the .NET runtime and a diagnostic client, such as, for example, dotnet-trace. When a .NET process starts, the runtime creates the diagnostic endpoint. On Windows, the endpoint is a named pipe, and on Unix, it’s a Unix domain socket created in the temp files folder. The endpoint name begins with a ‘dotnet-diagnostic-’ string and then contains the process ID to make it unique. The name also includes a timestamp and a ‘-socket’ suffix on Unix. Valid example names are dotnet-diagnostic-2675 on Windows and dotnet-diagnostic-2675-2489049-socket on Unix. When you type the ps subcommand in any of the CLI diagnostics tools (for example, dotnet-counters ps), the tool internally lists the endpoints matching the pattern I just described. So, essentially, the following commands are a good approximation to this logic:

# Linux
$ ls /tmp/dotnet-diagnostic-*
/tmp/dotnet-diagnostic-213-11057-socket /tmp/dotnet-diagnostic-2675-2489049-socket
# Windows
PS me> [System.IO.Directory]::GetFiles("\\.\pipe\", "dotnet-diagnostic-*")
\\.\pipe\dotnet-diagnostic-9272
\\.\pipe\dotnet-diagnostic-13372

The code for the .NET process listing is in the ProcessStatus.cs file. After extracting the process ID from the endpoint name, the diagnostics tool creates a Process class instance to retrieve the process name for printing. Armed with this knowledge, let’s try to intercept the communication between the tracer and the tracee.

Neither named pipes nor Unix domain sockets provide an API to do that easily. I started looking for the interceptors for either the kernel or user mode. I found a few interesting projects (for example, NpEtw), but I also discovered that configuring them would take me lots of time. I then stumbled upon a post using socat to proxy the Unix domain socket traffic. I wondered if I could write a proxy too.

Writing an EventPipes sniffer

The only problem was how to convince the .NET CLI tools to use my proxy. I did some tests, and on Linux, it’s enough to create a Unix domain socket with the same process ID but with the timestamp set to, for example, 1.

Let’s take as an example a .NET process with ID equal to 2675. Its diagnostic endpoint is represented by the /tmp/dotnet-diagnostic-2675-2489049-socket file. In my proxy, I am creating a Unix domain socket with a path /tmp/dotnet-diagnostic-2675-1-socket. The file system will list it first, and dotnet-trace (or any other tool) will pick it up as the endpoint for the process with ID 2675:

The code to create the proxy socket looks as follows:

private static async Task StartProxyUnix(int pid, CancellationToken ct)
{
var tmp = Path.GetTempPath();
var snoopedEndpointPath = Directory.GetFiles(tmp, $"dotnet-diagnostic-{pid}-*-socket").First();
var snoopingEndpointPath = Path.Combine(tmp, $"dotnet-diagnostic-{pid}-1-socket");
File.Delete(snoopingEndpointPath);
var endpoint = new UnixDomainSocketEndPoint(snoopingEndpointPath);
using var listenSocket = new Socket(endpoint.AddressFamily, SocketType.Stream, ProtocolType.Unspecified);
listenSocket.Bind(endpoint);
using var r = ct.Register(() => listenSocket.Close());
try
{
var id = 1;
while (!ct.IsCancellationRequested)
{
listenSocket.Listen();
if (ct.IsCancellationRequested)
{
return;
}
var socket = await listenSocket.AcceptAsync();
Console.WriteLine($"[{id}]: s1 connected");
// random remote socket
var senderSocket = new Socket(AddressFamily.Unix, SocketType.Stream, ProtocolType.Unspecified);
await senderSocket.ConnectAsync(new UnixDomainSocketEndPoint(snoopedEndpointPath));
Console.WriteLine($"[{id}]: s2 connected");
_ = SniffData(new NetworkStream(socket, true), new NetworkStream(senderSocket, true), id, ct);
id += 1;
}
}
catch (SocketException)
{
/* cancelled listen */
Console.WriteLine($"Stopped ({snoopingEndpointPath})");
}
finally
{
File.Delete(snoopingEndpointPath);
}
}

On Windows, it’s more complicated as there is no timestamp in the name. Thus, I decided to create a fake diagnostics endpoint that will look like an endpoint for a regular .NET process but, in reality, will be a proxy. Remember that CLI tools also call the Process.GetProcessById method, so the PID in my endpoint name must point to a valid process accessible to the current user. The process must be native, so the diagnostic endpoint name is not already taken. I picked explorer.exe 😊, and to record EventPipes traffic, I will use explorer as the target process in .NET CLI tools, as on the image below:

And the code for creating my proxy named pipe looks as follows:

private static async Task StartProxyWindows(int pid, CancellationToken ct)
{
var targetPipeName = $"dotnet-diagnostic-{pid}";
var explorer = Process.GetProcessesByName("explorer").First();
var pipeName = $"dotnet-diagnostic-{explorer.Id}";
try
{
var id = 1;
while (!ct.IsCancellationRequested)
{
var listener = new NamedPipeServerStream(pipeName, PipeDirection.InOut, 10, PipeTransmissionMode.Byte,
PipeOptions.Asynchronous, 0, 0);
await listener.WaitForConnectionAsync(ct);
Console.WriteLine($"[{id}]: s1 connected");
if (ct.IsCancellationRequested)
{
return;
}
var sender = new NamedPipeClientStream(".", targetPipeName, PipeDirection.InOut, PipeOptions.Asynchronous);
await sender.ConnectAsync();
Console.WriteLine($"[{id}]: s2 connected");
_ = SniffData(listener, sender, id, ct);
id += 1;
}
}
catch (TaskCanceledException)
{
Console.WriteLine($"Stopped ({pipeName})");
}
}

The fake diagnostic endpoint would work on Linux too, but the timestamp is less confusing. And we can always use our proxy to send some funny trace messages to our colleagues 🤐.

What’s left in our implementation is the forwarding code:

static async Task Main(string[] args)
{
if (args.Length != 1 || !int.TryParse(args[0], out var pid))
{
Console.WriteLine("Usage: epsnoop <pid>");
return;
}
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (o, ev) => { ev.Cancel = true; cts.Cancel(); };
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
await StartProxyWindows(pid, cts.Token);
}
else
{
await StartProxyUnix(pid, cts.Token);
}
}
private static async Task SniffData(Stream s1, Stream s2, int id, CancellationToken ct)
{
var outstream = File.Create($"eventpipes.{id}.data");
try
{
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
var tasks = new List<Task>() {
Forward(s1, s2, outstream, $"{id}: s1 -> s2", cts.Token),
Forward(s2, s1, outstream, $"{id}: s2 -> s1", cts.Token)
};
var t = await Task.WhenAny(tasks);
var ind = tasks.IndexOf(t);
Console.WriteLine($"[{id}]: s{ind + 1} disconnected");
tasks.RemoveAt(ind);
cts.Cancel();
await Task.WhenAny(tasks);
Console.WriteLine($"[{id}]: s{1 - ind + 1} disconnected");
}
catch (TaskCanceledException) { }
finally
{
outstream.Close();
s1.Dispose();
s2.Dispose();
}
}
private static async Task Forward(Stream sin, Stream sout, Stream snoop, string id, CancellationToken ct)
{
var buffer = new byte[1024];
while (true)
{
var read = await sin.ReadAsync(buffer, 0, buffer.Length, ct);
if (read == 0)
{
break;
}
Console.WriteLine($"[{id}] read: {read}");
snoop.Write(buffer, 0, read);
await sout.WriteAsync(buffer, 0, read, ct);
}
}

I’m saving the recorded traffic to the eventpipes.{stream-id}.data file in the current directory. The code of the application is also in the epsnoop folder in my diagnostics-tools repository.

Analyzing the EventPipes traffic

I also started working on the 010 Editor template. At the moment, it only understands IPC messages, but, later, I would like to add parsers for some of the diagnostic sequences as well (feel free to create a PR if you work on them too!). The template is in the debug-recipes repository, and on a screenshot below, you can see the initial bytes sent by the dotnet-counters monitor command:


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK