How to Detect the MOVEit Breach with OpenTelemetry

This blog will explore the story of the MOVEit breach, diving deep into the .NET Framework and wrangling with its obscurities, extending OpenTelemetry (OTel), and ultimately…a story of perseverance. Sort of.

What Happened

On May 31, Progress Software publicly disclosed that there were in-the-wild exploits of a critical CVE in their MOVEit Transfer program. MOVEit Transfer is a file-sharing application written in .NET Framework. You can think of it like an enterprise Dropbox, with both cloud and on-prem versions. If your team is anything like ours, the breach was the first time you’ve heard about MOVEit, but it turned out to be a big deal: Installed in thousands of companies, and handled sensitive data all over the world.

This specific CVE was every security professional’s nightmare scenario: An RCE from just knowing the server URL. This gave the attackers complete access to the server (which, reminder, is running on-prem) empowering them to with it as they wished, including taking all of the files stored inside. The attackers scanned the internet for public MOVEit instances and hacked into a lot of servers. A lot.

Progress did an admirable job at handling this situation - notifying customers on time, updating them on how they were responding, and in general being on top of things. Nobody wants to be in that situation, and when it unfortunately happens, it shows the true nature of a company’s security culture.

After Progress’ disclosure, the hacking group behind the breaches (Cl0p) pretty much issued an internet-wide ransom notice:

How They Got Access

CVE-2023-34362 is incredibly cool and complex, and there have been several great writeups on it (link). From our point of view, the attack was comprised of 4 separate vulnerabilities chained together, and looked as follows:

On the one hand, this attack chain isn’t new: It’s an SQLi and a deserialize leading to a webshell, which we’ve seen dozens of over the years. What’s special about the MOVEit attack was how difficult it was to exploit these primitives: It required a high level of sophistication. Reading into the exploit, we understood why security teams weren’t able to pick up on it for so long.

Fortunately, at Miggo, we’re focusing on application-level attacks and uncovering how applications are exploited in the wild to ideally prevent attacks like ever happening. So we rolled up our sleeves and asked ourselves: Could we have stopped the MOVEit breach from affecting our customers? If so, how? What do we need to offer to make sure teams are prepared?

Miggo and OTel: The Needed Combo 

First, it’s important to understand how Miggo is deployed: with an agent, agentless, or a hybrid model. For Windows-hosted .NET applications, our agent is based on the great OpenTelemetry project. If you’re not familiar with OTel, you can think of it like the Dynatrace, Datadog, or NewRelic’s tracing, but as an open-source and standardized version- an in-application monitor that creates events on application lifecycle events, correlating between them.

Behind the scenes, tracing looks like this:

‍

In the above image, each line is a span, denoting when certain things happen, like an HTTP request, a database query, or an internal computation. It contains tags with metadata, like who initiated this HTTP request, or what port they connected to. This chains together into a larger tree called a trace. Most companies use traces for observability, figuring out performance bottlenecks, or debugging.

Observability is great, and the world needs more of it. At Miggo we’ve taken observability and added a layer of security to it: We use these traces to analyze anomalies. Miggo’s traces can originate from our sensors, OpenTelemetry, or from APMs, we don’t mind. Since we base our sensors on OpenTelemetry, we gain the advantage of all the great work OpenTelemetry does for instrumentation and standardization while wearing our security goggles.

There’s more to our product, like agentless deployment methods, eBPF, and all sorts of exciting things, but you didn’t come to read about those aspects of Miggo, you came to read about how we could’ve detected and responded to MOVEit.

MOVEit is written in .NET Framework version 4.7.2, which is supported by OpenTelemetry, and thus supported by Miggo. We know from our experience in other runtimes and languages that not all deployments are cut from the same cloth, but it couldn’t be very difficult, now could it?

Spoiler (read in Morgan Freeman’s voice): It was.
‍

MOVEit Breach with a Sprinkle of OTel

We wanted to get a running version of the MOVEit exploit, to understand what we were seeing to ensure we could accurately detect, run a gap analysis, and respond.

We spun up a non-public Windows VM in our cloud provider and install a vulnerable version of MOVEit. As a result, we ended up with a shiny instance of MOVEit and a local MySQL database:

We continued our adventure by installing the out-of-the-box, vanilla OpenTelemetry auto-instrumentation .NET sensor on MOVEit, and browsing through the application. Taking Ghost (a CMS in node) as an example, we expected to see traces that look something like:

An HTTP request, followed up by a bunch of internal actions. 

Firing up MOVEit:

This? Only one span? No database calls, no calls to external services, nothing? It looks like MOVEit is mostly a single file, human.aspx. But maybe things will pick up - after all, it was just a few ordinary requests. We took an exploit implementation, ran it, and got a webshell! Let’s take another look at the traces, shall we?

What we’re seeing here is the request triggering one of the most important pieces of the puzzle, the SQL injection. As you can see, you can’t see the SQL injection. Or even realize that anything  is going on here. That’s not great.

Not all was lost! As mentioned above, Miggo’s sensor is based on OpenTelemetry and extends it. So we installed our sensor, and now we could see two different parts of the vulnerability:

1. The header containing the SQLi payload

2. The request to the third party from vanilla OTel’s instrumentations (localhost:8000 was our malicious auth server):

Distilling all of the above, here’s where we’re at (alongside a sneaky spoiler of where we’ll be by the end of this article):

This is on par with our previous experiences - most observability solutions don’t have the security angle in mind, our sensors bring that to the table, but it has some gaps and could use some love.

The rest of the article is going to be spent doing two things:

1. Gaining visibility into the SQL injection and RCE
2. …by way of adventures in .NET bytecode, infrastructure, and Windows hijinx
   ‍

The Search for SQLi

We started out covering the gaps with the SQL injection. It’s a key component to the vulnerability, and SQL client instrumentations are just something we like. It was surprising that there were no MySQL driver calls - we remembered seeing that MySQL was supported in the .NET OTel sensor. Let’s check that out:

Recent MySql.Data versions are instrumented out-of-the-box, we loved seeing that. What version does MOVEit use?

This version’s supported, so things should just work. But they didn’t - we weren’t seeing any SQL queries. So, we needed to look at the MySql.Data implementation:

namespace MySql.Data
{
#if NET5_0_OR_GREATER
  static class MySQLActivitySource
  {
    // ...

That nice #if there practically means “don’t run on .NET framework”. And MOVEit runs on .NET framework. Yikes. The MySQLActivitySource shown above is the connecting layer between MySql.Data and OpenTelemetry.

But let’s back up for a second, what’s even an ActivitySource?

An ActivitySource is like a pub/sub channel for Activities, which are spans in the observability world. They have an associated bag of key/value pairs, they have durations, and they can be nested arbitrarily. Sounds like spans and traces!

MySql.Data creates activities when executing SQL queries. A main source for traces in OpenTelemetry is hooking up with ActivitySources generated by several built-in libraries like ASP.Net, and apparently 3rd party libraries are part of the fun as well.

But as we can see above, MySql.Data generates these activities only in .NET 5 and above. Since we’re in .NET framework, no activities will be generated, so no traces.

Maybe we can take a look at the previous implementation of the MySql.Data instrumentation inside OpenTelemetry? Nope, the previous code was based on certain extension points, which aren’t functional in newer MySql.Data versions. That sucks.

Other instrumentations that we’ve done in .NET have relied on libraries providing hook points, like the ASP.NET instrumentation hooks into the request lifecycle. But relying on 3rd-parties cooperating with instrumentations isn’t the only way to go about instrumenting. In other runtimes, there are mechanisms for wrapping functions with our own logic. In node and python, that’s good ol’ monkey patching. But in Java, the prominent method is bytecode instrumentation, and we thought this technique might be applicable in .NET too.

What’s bytecode instrumentation? C# is a high-level language. The compiler translates that into bytecode (IL, Intermediate Language), which is run by the .NET runtime (known as the CLR, Common Language Runtime), translating it into machine code:

And this bytecode is available at runtime:

using System;
                    
public class Program
{
    public static void Main()
    {
        var method = typeof(Program).GetMethod("Four");
        var body = method.GetMethodBody();
        foreach (byte b in body.GetILAsByteArray())
        {
             Console.Write($"{b:X2} ");
        }
    }
    public static int Four()
    {
        return 4;
    }
}

Gives us:

00 1A 0A 2B 00 06 2A

That certainly is a bunch of bytes, and they’re very special bytes, because they return 4:

0x00	nop
0x1A	ldc.i4.4
0x0A	stloc.0
0x2B	br.s 0x00
0x06	ldloc.0
0x2A	ret

If you don’t believe us, follow along at home by manually disassembling with Wikipedia’s help.

Not only is the bytecode available for reading at runtime, but it’s also possible to modify a method’s bytecode, and even fabricate classes out of thin air! In Java, there’s an amazing package called Byte Buddy which gives high-level functionality for searching for methods and wrapping them in your logic. Most importantly, it allows hooking into when a method is called and when it returns. If there’s a nice way of doing this in .NET, we’re set for a win - we can wrap MySql.Data’s important functions and make traces out of them.

We embarked on a side quest to try and figure it out, prowling through the .NET OTel sources, and ended up finding this little subdirectory, containing nice names like Kafka, MongoDB, Wcf, and more.

For example, this beautiful ASP.NET instrumentation:

[InstrumentMethod(
   // ...
)]
public static class HttpModuleIntegration
{
    // ...
    internal static CallTargetState OnMethodBegin<TTarget, TCollection, TFunc>(...)
    {
        // ...

That sure looks like bytecode instrumentation - searching for a library, looking for symbols, and decorating methods. We didn’t understand what was going on, but it felt like a slam dunk! Wondering how we could have missed that, we started looking around for .NET OTel documentation about doing bytecode instrumentation, but couldn’t find any. What are the conventions here? How do we declare another one? What’s its favorite soup?

Later on, we discovered that this mechanism is built upon amazing prior work by Datadog’s .NET team, and they created this documentation that’s just out of this world.
‍

Adventures in Creating Our First Bytecode Instrumentation

Developer experience. We all know it’s important. Windows development was something that we just hadn’t done in a while. Since MOVEit is in .NET framework, we had to run a Windows machine, which means a local VM or cloud VM or heavens forbid a physical machine.

We started like good developers by creating a small test scenario: A tiny application using MySql.Data to connect to a database and execute a prepared statement:

using var connection = new MySqlConnection(connectionString)
connection.Open();

var command = new MySqlCommand("SELECT 1 as one", connection);
using var reader = command.ExecuteReader();
reader.Read();
Console.WriteLine(reader["one"]);

When instrumenting, we’re after the connection to the database, and the query execution. Our first attempt looked something like this:

[InstrumentMethod(
    assemblyName: "MySql.Data", // ①
    typeName: "MySql.Data.MySqlClient.MySqlConnection", // ②
    methodName: ".ctor", // ③
    returnTypeName: ClrNames.Void,
    parameterTypeNames: new[] { ClrNames.String },
    minimumVersion: "5.0.0",
    maximumVersion: "8.7.0",
    integrationName: "MySqlData",
    type: InstrumentationType.Trace)]
public static class MySqlDataIntegration
{
    internal static CallTargetReturn OnMethodEnd<TTarget>(
        TTarget instance, Exception exception, in CallTargetState state)
    {
        File.WriteAllText(
            "C:\\Windows\\Temp\\instro-mysqldata.txt",
            "hola! from instros sqlmydata"
        ); // ④
        return CallTargetReturn.GetDefault();
    }
}

What’s going on here? Whenever the MySqlConnection constructor is called, we want to execute some code. In this case, because we had absolutely no idea when this was going to run, we opted to write to a file. Still, the best way to debug code, don’t @ me.

In 1. We specify what dll we want to instrument. In C# terms, that’s the assembly name. Easy peasy, “What is MySql.Data”.

In 2. We specify the class’ fully qualified type name. It’s possible to grab that with type.FullName, for example:

using System.Net.Mail;
// ...
Console.WriteLine(typeof(SmtpClient).FullName); // System.Net.Mail.SmtpClient

3. Is where we say which method we want to override, with .ctor being a fancy name for a constructor, and finally

4. Is our grand debugging mechanism.

Putting it all together, we’re asking to instrument the constructor of the MySql.Data.MySqlClient.MySqlConnection found in MySql.Data assembly.

How we got to this code was a bit of a journey and took a few iterations. We started out by looking at other instrumentations and following by example, both the OTel and Datadog ones. Along the way we’ve heavily relied on auto-instrumentation loader’s debug logs, found in C:\ProgramData\OpenTelemetry .NET AutoInstrumentation\logs, and from there poking at the source code to see what went wrong. Honestly, we were very impressed at the level of debug logs available, and how helpful they were in resolving issues (like a mismatched type name).

This code isn’t standalone - it needs to be injected by the OpenTelemetry auto-instrumentation loader. That means recompiling the auto-instrumentation library, which took like 7 minutes before we figured out how to shave it down by cutting unnecessary build phases. After several failed attempts, the first success was majestic:

Success! We managed to wrap a constructor call without the application having to do any code changes whatsoever. Great start to the journey, now all we had to do was replicate it: See how to grab the connection details to enrich the span, and how to instrument the query execution.

While trying to make the dev loop shorter, we committed a cardinal sin: We tried to only overwrite specific files. Running the OpenTelemetry installation takes a couple of minutes. So our dev loop was:

1. Fiddle around with code
2. Compile for a few minutes
3. Run the installer for a few minutes
4. Everything fails
5. GOTO 1

Looking at 3, we thought that once it’s been run once, why do we even need the installer? Let’s just overwrite the dlls that the instrumentations reside in. Nothing could go wrong, could it?

Except…OTel writes its dlls to Program Files. As part of the lazy build, we copy the OpenTelemetry.AutoInstrumentation.dll (the dll containing bytecode instrumentations) over to the installation directory. Loaded the application again, and nothing happened. It just kept the old logic. We wanted to know which version of the dll is loaded, to make sure we’re not hallucinating. Now’s a good time to introduce a noble protagonist to the story: dnSpy.

dnSpy is a .NET decompiler and debugger, and it has saved our skin countless times. Once an application is running in IIS, we can attach dnSpy to the IIS service (w3wc), look at the loaded modules, and look at what code it runs:

The image above is of an arbitrary MOVEit class, but we can do that with any loaded module. That’s how we were able to see that our code wasn’t updated: We could replace everything with a Console.WriteLine, and it’ll persist. But why wouldn’t it? Looking at the modules window:

As the path might suggest, the dll is located in the GAC.

The GAC (Global Assembly Cache) is a special place in Windows where shared .NET libraries are stored. If multiple applications need to use the same library, instead of each app having its own copy, the library can be placed in the GAC. This way, all applications can use that single shared version, saving space and ensuring consistency. It's like a central storage spot for commonly used .NET components, making it easy for different programs to share the same code.

So our code wasn’t loaded from the installation directory, it was loaded from the GAC. Okay, let’s just delete the GAC entry - after all, we don’t want no cache, right?

WRONG. Of course an assembly has to have a GAC entry! Because if it doesn’t, obviously the dll can’t be found:

FileNotFoundException: Could not load file or assembly 'OpenTelemetry.AutoInstrumentation, 
Version=1.0.0.0, Culture=neutral, PublicKeyToken=c0db600a13f60b51' 
or one of its dependencies. The system cannot find the file specified.

Lessons learned: If we wish to override dlls, we have to make sure to also update them in the GAC. It’s possible to either copy files directly, use gacutil, or do it from powershell:

[System.Reflection.Assembly]::Load(
  "System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
) | Out-Null

SQL Instrumentation: Now For Real

That was quite a detour, wasn’t it? Let’s go back to our real goal: Bytecode instrumentation for MySql.Data to grab database connections and queries, to figure out whether an SQL injection is happening.

Now that we have a dev loop for writing and testing code, we got to the interesting part of every instrumentation: pinpointing the key parts in the function flow where interesting data is available, and building spans from there. This can be approached, like any research question, in two main ways:

1. Black box: Put a breakpoint somewhere and start hitting “step-into”
2. White box: Read a lot of code and see where execution hits a narrow waist

In this case, we started out with the opportunistic black box: From our small program, we started stepping into the MySql.Data code to see where it led us. As a reminder, here’s the program:

using var connection = new MySqlConnection(connectionString)
connection.Open();

var command = new MySqlCommand("SELECT 1 as one", connection);
using var reader = command.ExecuteReader();
reader.Read();
Console.WriteLine(reader["one"]);

We set a breakpoint just before the call to ExecuteReader and started stepping in.

A note about debugging IIS: When dlls are inserted into the GAC they get optimized. That means when attaching to an existing process and stepping through execution, some variables won’t be available, or some lines can’t be stepped into, lovely things like that which just make your day.

There’s a little-known trick to make your day better. The behaviors above are a result of the .NET JIT optimizing away some of the code. When dnSpy is the one running the process, it’s ok because dnSpy saw the unoptimized version, but dnSpy is not the one running IIS. dnSpy can’t see the unoptimized code, so it doesn’t know how to map it back!

Luckily, there’s a way to nicely ask the JIT to make our lives nicer:

[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0

This improves the debugging experience, especially in cases where previously the only alternative was stepping over the CLR bytecode. Fun times!

Back to our breakpoint before ExecuteReader. A few more step-intos, and we get to the function we’d want to instrument:

This function is a natural “narrow waist”, which all the high-level functions to execute queries end up with. Here’s a first stab at doing it:

[InstrumentMethod(
    assemblyName: "MySql.Data",
    typeName: "MySql.Data.MySqlClient.MySqlCommand",
    methodName: "ExecuteReaderAsync",
    returnTypeName: "System.Threading.Tasks.Task`1[MySql.Data.MySqlClient.MySqlDataReader]", // ①
    parameterTypeNames: new[] {
        "System.Data.CommandBehavior",
         ClrNames.Bool,
         ClrNames.CancellationToken
    }, // ②
    minimumVersion: "5.0.0",
    maximumVersion: "8.7.0",
    integrationName: "MySqlData",
    type: InstrumentationType.Trace)]
public static class MySqlCommandExecuteReaderAsyncIntegration
{
    private static readonly IOtelLogger Log = OtelLogging.GetLogger();
    private static readonly ActivitySource Source =
        new("OpenTelemetry.AutoInstrumentation.MySqlData");

    internal static CallTargetState OnMethodBegin<
        TTarget,
        TCommandBehaviour
    >(
        TTarget instance,
        TCommandBehaviour commandBehaviour,
        bool execAsync,
        CancellationToken cancellationToken
    )
    {

        if (instance == null)
        {
            return CallTargetState.GetDefault();
        }

        var activity = Source.StartActivity("mysql.Execute", ActivityKind.Client); // ③
        if (activity == null)
        {
            return CallTargetState.GetDefault();
        }

        return new CallTargetState(activity);
    }

    internal static CallTargetReturn<TReturn> OnMethodEnd<TTarget, TReturn>(
        TTarget instance,
        TReturn returnValue,
        Exception exception,
        in CallTargetState state
    )
    {
        var ret = new CallTargetReturn<TReturn>(returnValue);

        using var activity = state.Activity;

        if (activity is null || activity.IsStopped || instance is null)
        {
            return ret;
        }

        var mysqlCommandType = instance.GetType();
        // ④
        var commandText = mysqlCommandType
            .GetProperty("CommandText", BindingFlags.Public | BindingFlags.Instance)
            ?.GetValue(instance);

        activity.SetTag("db.statement", commandText);

        return ret;
    }
}

Here's what's happening:

1. This odd 1` and [...] syntax is how generics are represented. See the table under the Remarks here
2. This is the function’s full signature
3. Is how traces get initialized
4. Is our instrumentation’s actual implementation - grabbing the CommandText of the SqlCommand

Putting it all together, we get the below trace:

Hello there SQL statement! Miggo’s backend has rules for detecting and alerting queries containing SQL injections. So after all of this, we have the first major part of the vulnerability in the bag! Revisiting our table:

The webshell that was found in the wild (human2.aspx) was a wrapper for extracting data from the MOVEit database. So now that we have SQL instrumentation, we can figure out what queries the webshell executed, and determine its full blast radius.

Process Execution

From our experience, once we get the first couple of instrumentations out of the way, the rest will flow. The vulnerability itself is a deserialization vulnerability, which means it needs to call into another mechanism to execute code. A common source for constructing payloads is ysoserial.

So we did what we like doing in other platforms: Instrument the standard library functions enabling process execution, using the same mechanism as the SQL injection instrumentation:

With this out of the way, we’ve got every part of the vulnerability mapped and visible!
‍

In Closing

We started out in humble beginnings, not knowing all too much about how MOVEit works, how the exploit behaves, and without even knowing that it’s possible to do bytecode instrumentation in .NET. This was a meaningful (and fun) project to work on. During the course of this research, we were able to deeply understand the intricate details of CVE-2023-34362, break it into its constituent parts, and ensure that Miggo will be able to respond to such a vulnerability in the future.

For us, the journey was fun and challenging, and we learned a lot. Thank you for tagging along, and see you next time in our grand adventures!