Evading Anti-Virus Detection with Metasploit

This week I attended a free, technical webinar hosted by David Maloney, a Senior Software Engineer on Rapid7's Metasploit team, where he is responsible for development of core features for the commercial Metasploit editions. The webinar was about evading anti-virus detection and covered topics including:

• Signatures, heuristics, and sandboxes
• Single and staged payloads
• Executable templates
• Common misconceptions about encoding payloads
• Dynamically creating executable templates

After Kaspersky Lab broke news of the "Red October" espionage malware package last week, I thought this would be an interesting topic to learn more about. In the post, Kaspersky is quoted saying, "the attackers managed to stay in the game for over 5 years and evade detection of most antivirus products while continuing to exfiltrate what must be hundreds of terabytes by now."

Separating Exploits and Payloads

Vocabulary in the world of penetration testing may not be familiar to everyone, so let's go over a few terms you may see.

• Vulnerability: A bug or design flaw in software that can be exploited to allow unintended behavior
• Exploit: Software which takes advantage of a vulnerability allowing arbitrary execution of an attacker's code
• Payload: Code delivered to a victim by an exploit
• Signature: Set of rules or pattern match against code
• Sandbox: Protected segments in OS, where code can be run safely

Metasploit by design, separates the payload from the exploit. Payloads can come in two types. A single-stage payload includes all code intended for use in the attack. A staged payload has a small initial exploit which then connects back to a server using shell commands to download subsequent payloads. This is an important distinction because many anti-virus products have signatures for common first-stage exploits, but not for the much wider universe of secondary payloads. By building first-stage exploits that can evade detection, additional payloads can be installed and remain resident without detection.

A Unique Exploit for Every Target

To have unique initial exploits that will not have anti-virus signatures, Metasploit Pro includes tools to bundle exploits inside otherwise randomly generated executables. These tools create C code which assign variables in random orders and with random values. Functions are created at random to manipulate and perform calculations on these variables. The functions are then called randomly, building a random call tree, making it very difficult to develop a signature because the execution flow and memory maps are all random.

Of course, eventually, we want the random calculations to stop and the exploit to execute so a payload can be downloaded and executed. Amazingly, one of the key ways to hide the payload from the anti-virus is simply to wait to decode the encoded (obfuscated) exploit until after the anti-virus has completed its scan of the executable. Anti-virus vendors are keenly aware that their products hurt end user performance and so the amount of time which they can sandbox and scan an executable is limited. If the initial payload's random functions take a sufficient time, then the anti-virus releases the file from the sandbox. This delay is configurable and is very effective, allowing the exploit to be decoded and executed without detection.

The Next Generation of Exploits

It's been 8 months since these randomization generators were released with Metasploit Pro and anti-virus companies are starting to catch up. Still, only 8 of the 44 scanners used at VirusTotal detected one of these exploits bundled with randomized code. The next generation of generators are designed to avoid using shell code entirely, further reducing anti-virus products' ability to detect malicious behavior. Instead of shell code, system calls are starting to be used directly, pulling payloads directly into memory. Since anti-virus depends heavily on scanning writes to the file system, this also reduces the exploits surface area. PowerShell version 2.0 seems to be the vehicle of choice for these next generation of exploits and thus far has gone completely unnoticed by anti-virus vendors (according to David anyway).

Additional Resources

• Evading Anti-virus Detection with Metasploit: video, slides
• The Odd Couple: Metasploit and Antivirus Solutions
• Why Encoding Does not Matter and How Metasploit Generates EXE’s
• Facts and myths about antivirus evasion with Metasploit
• Using Metasm To Avoid Antivirus Detection (Ghost Writing ASM)

Use Metasploit to Verify Rails is Secured from CVE-2013-0156

On January 8th, 2013 Aaron Patterson announced a major security vulnerability on the Rails security mailing list, affecting all releases of the Ruby on Rails framework. This vulnerability allows an unskilled attacker to execute commands remotely on any unpatched Rails web server. Unsurprisingly, it's getting a lot of attention; Ars Technica estimates more than 200,000 sites may be vulnerable. With all the hype, it's important to separate the facts from the fiction and use the attacker's own tools to verify your site is secure.

Within 36 hours of the announcement of CVE-2013-0156, the developers at Rapid7 released a metasploit exploit module. Metasploit lowers the barriers to entry for attackers, making the whole process a point and click affair with a slick web GUI. Fortunately, the Rails security team has provided many easy to implement mitigation options. But, how do *know* you've really closed the vulnerability, particularly to the most automated and unskilled attacks? No better way than to try and exploit yourself.

It's best to scan your unpatched site first so you can be certain the scan is working as expected and you don't end up with a false positive that you've eliminated the vulnerability. Here is the quick and dirty introduction to running Metasploit, and executing a scan:

UPDATE: I've changed the Metasploit instructions here a bit to include setting the VHOST option. Teammate Steph Skardal was using these instructions and together we found that without a VHOST set, the RHOSTS are resolved to an IP address. It's worth checking your Rails logs to verify a request is being received and processed. If you don't see anything there, check your nginx or Apache (or whatever) access logs for any possible 301 redirects.

1 2 3 4 5 6 7 8 9 10 11 12 13 14

git clone git://github.com/rapid7/metasploit-framework.git cd metasploit-framework ./msfconsole use auxiliary/scanner/http/rails_xml_yaml_scanner set RHOSTS mycompany.com set VHOST app.mycompany.com set RPORT 80 set URIPATH /rails_app set VERBOSE true show options run [+] mycompany.com:80 is likely vulnerable due to a 500 reply for invalid YAML [*] Scanned 1 of 1 hosts (100% complete) [*] Auxiliary module execution completed

If you don't get back a "likely vulnerable" message, it's probably because you're still running Ruby 1.8.7. As of this writing the metasploit module exploit states:

The technique used by this module requires the target to be running a fairly version of Ruby 1.9 (since 2011 or so). Applications using Ruby 1.8 may still be exploitable using the init_with() method, but this has not been demonstrated.

It's only a matter of time before Ruby 1.8 becomes supported, but this does give some folks a bit more time. Now let's review the mitgation strategies provided by the announcement to show you just how easy it can be to secure yourself.

Disabling XML Entirely

The nature of the vulnerability is in parsing XML in request parameters. If you don't parse XML, you should disable XML parsing entirely by placing one of the following snippets inside an application initializer.

1 2 3 4

# Rails 3.2, 3.1 and 3.0 ActionDispatch::ParamsParser::DEFAULT_PARSERS.delete(Mime::XML) # Rails 2.3 ActionController::Base.param_parsers.delete(Mime::XML)

Removing YAML and Symbol support from the XML parser

I couldn't say it better than Aaron than myself, so I'll give it to you straight from the announcement:

If your application must continue to parse XML you must disable the YAML and Symbol type conversion from the Rails XML parser. You should place one of the following code snippets in an application initializer to ensure your application isn't vulnerable. You should also consider greatly reducing the value of REXML::Document.entity_expansion_limit to limit the risk of entity explosion attacks.

The entity_expansion_limit recommendation is not strictly part of CVE-2013-0156, but should be implemented as well to limit your exposure to entity explosion attacks.

To disable the YAML and Symbol type conversions for the Rails XML parser add these lines to an initializer:

1 2 3 4 5 6 7

#Rails 3.2, 3.1, 3.0 ActiveSupport::XmlMini::PARSING.delete("symbol") ActiveSupport::XmlMini::PARSING.delete("yaml")   #Rails 2.3 ActiveSupport::CoreExtensions::Hash::Conversions::XML_PARSING.delete('symbol') ActiveSupport::CoreExtensions::Hash::Conversions::XML_PARSING.delete('yaml')

Removing YAML Parameter Parsing

While it's *much* less common to parse YAML in request params than XML, Rails does support this, but not by default (except in version 1.1.0!). There is no fix for YAML params injection, so it must be disabled . The methods for doing this differ in Rails among rails versions.

1 2 3 4 5 6 7 8 9 10 11 12 13

# Rails 2.x: find and remove all instances ActionController::Base.param_parsers[Mime::YAML] = :yaml   # Rails 3.x: add to initializer ActionDispatch::ParamsParser::DEFAULT_PARSERS.delete(Mime::YAML)   # Rails 3.2, 3.1, 3.0: add to initializer ActiveSupport::XmlMini::PARSING.delete("symbol") ActiveSupport::XmlMini::PARSING.delete("yaml")   # Rails 2.3: add to initializer ActiveSupport::CoreExtensions::Hash::Conversions::XML_PARSING.delete('symbol') ActiveSupport::CoreExtensions::Hash::Conversions::XML_PARSING.delete('yaml')

Go Check your Site!

As you can see, with just a few lines of code, any site can manage their exposure to this risk. I strongly urge you to read the security announcement and avoid the hype. Then go patch your site!

Case Sensitive MySQL Searches

MySQL's support for case sensitive search is explained somewhat opaquely in the aptly titled Case Sensitivity in String Searches documentation. In short, it explains that by default, MySQL won't treat strings as case sensitive when executing a statement such as:

SELECT first_name FROM contacts WHERE first_name REGEXP '^[a-z]';

This simple search to look for contacts whose first name starts with a lower case letter, will return *all* contacts because in the default character set used by MySQL (latin1), upper and lower case letters share the same collation. The documentation for both MySQL and PostgreSQL have lengthy discussions on the topic.

Enough with the backstory, how do I perform case sensitive searches?!

The docs say to convert the string representation to a binary one. This allows "comparisons [to] use the numeric values of the bytes in the operands". Let's see it in action:

SELECT first_name FROM contacts WHERE BINARY(first_name) REGEXP '^[a-z]';

There are other strategies available, such as changing the character set being used for comparisons with the COLLATE function. This would likely work better for cases where you had many columns to compare.

SELECT first_name FROM contacts WHERE first_name REGEXP '^[a-z]' COLLATE latin1_bin;

You can even go so far as to have MySQL switch character sets and collations. But you do have to do this for each database, each table, and each column you need to convert. Not terribly fun.

Do this now on all your production Rails app servers:

1	ps ux | grep Rails

The first column in the results of that command show which user runs your Rails and Passenger processes. If this is a privileged user (sudoer, or worse yet password-less sudoer), then this article is for you.

Assumptions Check

There are several different strategies for modifying which user your Rails app runs as. By default the owner of config/environment.rb is the user which Passenger will run your application as. For some, simply changing the ownership of this file is sufficient, but in some cases, we may want to force Passenger to always use a particular user.

This article assumes you are running nginx compiled with Passenger support and that you have configured an unprivileged user named rails-app. This configuration has been tested with nginx version 0.7.67 and Passenger version 2.2.15. (Dated I know, but now that you can't find the docs for these old versions, this article is extra helpful.)

Modifying nginx.conf

The changes required in nginx are very straight forward.

1 2 3 4 5 6

# Added in the main, top-level section user rails-app;   # Added in the appropriate http section among your other Passenger related options passenger_user_switching off; passenger_default_user rails-app;

The first directive tells nginx to run it's worker processes as the rails-app user. It's not completely clear to me why this was required, but failing to include this resulted in the following error. Bonus points to any one who can help me understand this one.

1	[error] 1085#0: *1 connect() to unix:/tmp/passenger.1064/master/helper_server.sock failed (111: Connection refused) while connecting to upstream, client: XXX, server: XXX, request: "GET XXX HTTP/1.0", upstream: "passenger://unix:/tmp/passenger.1064/master/helper_server.sock:", host: "XXX"

The second directive, passenger_user_switching off, tells Passenger to ignore the ownership of config/environment.rb and instead use the user specified in the passenger_default_user directive. Pretty straight forward!

Log File Permissions Gotcha

Presumably you're not storing your production log files in your apps log directory, but instead in /var/log/app_name and using logrotate to archive and compress your logs nightly. Make sure you update the configuration of logrotate to create the new log files with the appropriate user. Additionally, make sure you change the ownership of the current log file so that Passenger can write your applications logs!

Automated VM cloning with PowerCLI

Most small businesses cannot afford the high performance storage area networks (SANs) that make traditional redundancy options such high availability and fault tolerance possible. Despite this, the APIs available to administrators of virtualized infrastructure using direct attached storage (DAS) make it possible to recreate many of the benefits of high availability.

High Availability on SAN vs DAS

A single server failure in a virtualized environment can mean many applications and services can become unavailable simultaneously; for small organizations, this can be particularly damaging. High availability with SANs minimize the downtime of applications and services when a host fails by keeping virtual machine (VM) storage off the host and on the SAN. VMs on a failed host can then be automatically restarted on hosts with excess capacity. This of course requires SAN infrastructure to be highly redundant, adding to the already expensive and complex nature of SANs.

Alternatively, direct attached storage (DAS) is very cost effective, performant, and well understood. By using software to automate the snapshot and cloning of VMs via traditional gigabit Ethernet from host to host, we can create a "poor man's" high availability system.

It's important for administrators to understand that there is a very real window of data loss that can range from hours to days depending on the number of systems backed up and hardware in use. However, for many small businesses who may not have trustworthy backups, automated cloning is an excellent step forward.

Automated cloning with VMWare's PowerCLI

Although End Point is primarily an open source shop, my introduction virtualization was with VMWare. For automation and scripting, PowerCLI, the PowerShell based command line interface for vSphere, is the platform on which we will build. The process is as follows:

• A scheduled task executes the backup script.
• Delete all old backups to free space.
• Read CSV of VMs to be backed up and the target host and datastore.
• For each VM, snapshot and clone to destination.
• Collect data on cloning failures and email report.

I have created a public GitHub repository for the code and called it powercli_cloner.

Currently, it's fairly customized around the needs of the particular client it was implemented for, so there is much room for generalization and improvement. One area of improvement is immediately obvious: only delete a backup after successfully replacing it. Also, the script must be run as a Windows user with administrator vSphere privileges, as the scripts assumes pass-through authentication is in place. This is probably best for keeping credentials out of plain text. The script should be run during non-peak hours, especially if you have I/O intensive workloads.

Hopefully this tool can provide opportunities to develop backup and disaster recovery procedures that are flexible, cost-effective, and simple. I'd welcome pull requests and other suggestions for improvement.

Changing Passenger's Nginx Timeouts

It may frighten you to know that there are applications which take longer than Passenger's default timeout of 10 minutes. Well, it's true. And yes, those application owners know they have bigger fish to fry. But when a customer needs that report run *today* being able to lengthen a timeout is a welcomed stopgap.

Tracing the timeout

There are many different layers at which a timeout can occur, although these may not be immediately obvious to your users. Typically they receive a 504 and an ugly "Gateway Time-out" message from Nginx. Review the Nginx error logs both at the reverse proxy and application server, you might see a message like this:

upstream timed out (110: Connection timed out) while reading response header from upstream

If you're seeing this message on the reverse proxy, the solution is fairly straight forward. Update the proxy_read_timeout setting in your nginx.conf and restart. However, it's more likely you've already tried that and found it ineffective. If you expand your reading of the Nginx error you might notice another clue.

upstream timed out (110: Connection timed out) while reading response header from upstream, 
upstream: "passenger://unix:/tmp/passenger.3940/master/helper_server.sock:"

This is the kind of error message you'd see on the Nginx application server when a Passenger process takes longer than the default timeout of 10 minutes. If you're seeing this message, it'd be wise to review the Rails logs to get a sense for how long this process actually takes to complete so you can make a sane adjustment to the timeout. Additionally, it's good to see what task is actually taking so long so you can offload the job into the background eventually.

Changing nginx-passenger module's timeout

If you're unable to address the slow Rails process problem and must extend the length of the time out, you'll need to modify the Passenger gem's Nginx configuration. Start by locating the Passenger gem's Nginx config with locate nginx/Configuration.c and edit the following lines:

ngx_conf_merge_msec_value(conf->upstream.read_timeout,
                              prev->upstream.read_timeout, 60000);

Replace the 60000 value with your desired timeout in milliseconds. Then run sudo passenger-install-nginx-module to recompile nginx and restart.

Improving Error Pages

Another lesson worth addressing here is that Nginx error pages are ugly and unhelpful. Even if you have a Rails plugin like exception_notification installed, these kind of Nginx errors will be missed, unless you use the error_page directive. In other applications I've setup explicit routes to test exception_notification properly sends an email by creating a controller action that simple raises an error. Using Nginx's error_page directive, you can call an exception controller action and pass useful information along to yourself as well as present the user with a consistent error experience.

Simple Example of Dependency Injection with Rails

Today I came across a great opportunity to illustrate dependency injection in a simple context. I had a Rails partial that was duplicated across two subclasses. The partial was responsible for displaying options to create a new record from the data of the current record. It also offered two types of copy, shallow and deep. The shallow copy used a button to POST data, while the deep copy offered a form with some additional options. The only difference between the partials was the path to post data to. Let's see this in code.

1 2 3 4 5 6 7 8 9 10 11 12 13 14

#app/views/fun_event/_copy_options.html.erb button_to(t("create_and_edit_shallow_copy"), fun_event_path(:from_event => @event.id, :return => true), :    id => "shallow_copy_btn")   form_tag(fun_event_path(:return => true)) do   #form code end     #app/views/boring_event/_copy_options.html.erb button_to(t("create_and_edit_shallow_copy"), boring_event_path(:from_event => @event.id, :return => true), :    id => "shallow_copy_btn")   form_tag(boring_event_path(:return => true)) do   #form code end

The first, failed iteration

To remove the duplication, I passed in a path option into the partial, replacing specific references with the generic.

1 2 3 4 5 6 7 8 9 10 11 12 13

#app/views/fun_events/copy.html.erb <%= render :partial => "events/copy_options", :event_path => fun_event_path %>     #app/views/boring_events/copy.html.erb <%= render :partial => "events/copy_options, :event_path => boring_event_path %>   #app/views/events/_copy_options.html.erb button_to(t("create_and_edit_shallow_copy"), event_path(:from_event => @event.id, :return => true), :    id => "shallow_copy_btn")   form_tag(event_path(:return => true)) do   #form code end

Can you guess where this led?

undefined method `event_path' for ActionView::Base:0xd6acf18

Dude! Inject the dependency!

Obviously the event_path variable I was passing was a string, not a method. I needed the method so I could pass in the appropriate arguments to construct the URL I needed. Had there not been two different calls to the routes, I would likely have just passed in the string needed in each context. But in this case, I was forced to think outside the box. Here's what I ended up with.

1 2 3 4 5 6 7 8 9 10 11 12 13

#app/views/fun_events/copy.html.erb <%= render :partial => "events/copy_options", :event_path => method(:fun_event_path) %>     #app/views/boring_events/copy.html.erb <%= render :partial => "events/copy_options, :event_path => method(:boring_event_path) %>   #app/views/events/_copy_options.html.erb button_to(t("create_and_edit_shallow_copy"), event_path.call(:from_event => @event.id, :return => true), :    id => "shallow_copy_btn")   form_tag(event_path.call(:return => true)) do   #form code end

The changes are really quite subtle, but we use Object's method method to pass the reference to the method we want to call, and simply pass in the arguments when needed. Mind == Blown