Peering Inside Snow Leopard Security

From the beginning, Apple made it clear that Mac OS X 10.6 Snow Leopard was focused on improving the performance of the operating system and providing developers with new tools for harnessing the power of modern hardware and multiprocessor systems. The included security-related changes are no different, and for the most part are completely invisible to the user.

These changes provide new tools to assist programmers in producing more secure applications and harden the core operating system, which will result in a safer computing experience for most Mac users.

Despite these improvements, Apple missed a major opportunity to include a key operating system feature that could nearly wipe out a entire category of attack.

Securing Memory and the Power of 64-Bit Security — The changes I describe here are fairly subtle and technical, so those of you who don’t care about things like stack versus heap memory might want to skip to the next section.

The most significant single improvement is that the operating system has now been compiled with stack memory protection by default. Essentially, this places what’s known as “canaries” in stack memory: fixed values in fixed locations that are pushed around if an attacker uses a buffer overflow attack, enabling the operating system or program to detect attacks.

(Buffer overflows happen when input values to a program – something as simple as a URL entered in the location field of a browser – is larger than expected. The data that overflows can be used to crash software or a system, or gain privileged access.)

Stack memory protection makes an entire class of buffer overflow attacks much more difficult to exploit, even when a software vulnerability is present. Developers will need to enable it for their own applications, but by default, Apple uses this feature everywhere it can to limit attacks.

A second overarching improvement is the migration to 64-bit applications and components throughout Snow Leopard. While Apple touts 64-bit support largely for its speed boosts and to enable memory-hungry applications to address more memory, the Intel CPU architecture offers substantial hardware security capabilities that generally aren’t available in 32-bit environments.

While Snow Leopard includes both 32-bit and 64-bit kernels, it’s possible for a 64-bit capable Mac to boot with the 64-bit kernel only under Mac OS X Server 10.6 – Snow Leopard Server. When running Snow Leopard Server, the 2008 and 2009 Xserves and Mac Pros boot the 64-bit kernel by default; the 2008 and 2009 iMacs are capable of booting the 64-bit kernel in Snow Leopard Server, but must be placed in that mode by holding down the 6 and 4 keys at startup. Apple posted a Knowledge Base article with details about accessing the 64-bit kernel in Snow Leopard Server, but it has offered nothing yet for the regular version of Snow Leopard.

Wherever possible, Apple appears to try to use new security technologies for Snow Leopard on 32-bit systems, but most of the real security advantages are possible only when running 64-bit software on 64-bit hardware.

One of the key areas in which this prevents trouble is in the heap memory, where Apple uses a combination of technologies: one leverages 64-bit hardware, while the others are software enhancements. (The heap is a pool of free memory that applications can dynamically use on a temporary basis, as opposed to the more-structured and static stack memory.)

When programming an application, heap memory locations that should only accept data can be marked as non-executable, and this will be enforced by a 64-bit processor (similar hardware protection has been used since Mac OS X 10.4 Tiger for stack memory on 32-bit processors). This, again, increases the difficulty of heap-based memory exploits, which are a common form of attack.

Apple further hardened the heap through use of stronger heap checksums to detect when someone has tried to modify a portion of memory. Combined with a related feature that terminates processes if it detects double null bytes where they shouldn’t be, this approach doesn’t eliminate all heap based memory attacks, but makes the life of the attacker much more difficult.

A final advantage of the move to 64-bit code is that applications now move data around more securely, skipping the stack completely. Function arguments are passed via registers, which, again, complicates the life of an attacker trying to attack your Mac using memory corruption techniques.

Sandboxing and Safari Enhancements — Sandboxing is the process of restricting what kinds of activities an application can perform. For example, you can sandbox an application so it can read files, but not write them, or restrict it from accessing the network. It’s a great way to limit the damage if an attacker is able to exploit an application on your Mac, since they’ll be stuck in the sandbox.

Apple provides sandboxing services that any developer can use, and has slowly been increasing the number of Apple applications that implement sandboxing through software updates. Apple continued this trend with Snow Leopard, sandboxing a number of new applications and features. One example is the x264 codec for handling H.264 video, which will make it harder for attackers to build malicious video files designed to corrupt your video player and allow them to exploit your Mac (a not-uncommon attack vector).

There’s been discussion over increased sandboxing in Safari, but that’s not quite how Apple improved browser security and stability. Instead of trying to sandbox browser plug-ins within Safari, Snow Leopard now runs them as separate processes. That way if a plug-in crashes, it doesn’t crash your entire browser. (In fact, Apple told us that browser plug-ins are the number one cause of crashes in Mac OS X, so making them independent processes should increase general reliability as well.)

Running browser plug-ins as separate processes improves security more than just increasing sandboxing, because Safari includes support for a legacy requirement that allows the use of a somewhat less-secure version of a common programming function called malloc that’s important for memory management. These plug-ins now run using the more-secure version of malloc used by the rest of Snow Leopard. By separating plug-ins into separate processes, developers potentially have more opportunities to add sandboxing to their plug-ins.

Some WebKit-based plug-ins still run within the main Safari process, but most of the major plug-ins have migrated to this new architecture, improving security and stability.

A New Firewall Setting — With Leopard, Apple introduced a new firewall capable of restricting inbound access to specific applications, not just network ports and protocols (see “Leopard Firewall Takes One Step Forward, Three Steps Back,” 2007-11-05). In Snow Leopard, Apple implemented a minor default usability change some users will want to disable.

You can find the settings in the Security system preference pane in the Firewall view, which now provides just a Start or Stop button. If you click the Advanced button, the settings are nearly identical to those in Leopard – but with one minor change: a new checkbox allows signed software to receive incoming connections automatically.

This setting allows applications signed by a valid certificate authority – the same authorities that sign Web pages for secure SSL/TLS sessions – to receive incoming connections with no additional steps. Previously, you would have had to add the application or approve a firewall exception when Leopard noted the application trying to set up the incoming connection.

This bypass was likely included to reduce the number of dialog boxes users need to click when installing software from known companies. Don’t worry: even if a program is allowed by default you can still manually change the setting to block access.

Users who want more control over their security should disable this setting, since anyone willing to pay the money can purchase a code signing certificate. Just because a program is signed doesn’t necessarily mean you want it to accept incoming connections.

New Malicious Software Protection — Back in Mac OS X 10.4 Tiger, Apple introduced a new File Quarantine feature. Enhanced again in Leopard, it displays a warning the first time you run any file that was downloaded using common Internet programs like Mail, Safari, and iChat.

In Snow Leopard, File Quarantine now checks these programs to see if they contain certain malicious software, which is sometimes hidden in downloads to trick users into installing it. If the file is infected, you’ll see a new warning that explicitly warns you the file is dangerous.

Right now, according to reports on the Internet, the feature only checks for two known types of malicious software. Apple has stated that additional checks can be added using the normal Software Update approach if needed.

This doesn’t mean that Apple has added antivirus software to your Mac. Using File Quarantine will protect you from running a few specific instances of malware downloaded using standard programs, but won’t catch other malicious files, such as any transferred via USB drive. The feature also won’t necessarily protect you if an attacker exploits your Mac, such as through a Web browser vulnerability, and then uses that toehold to install additional malicious software. And it can’t remove the infection from compromised files.

The initial version offers Apple a great capability to push out protection to users in case a larger infection starts to propagate. (For detailed information on how the enhanced File Quarantine feature works, we recommend Dan Moren’s excellent article at Macworld.

A Missed Opportunity — One major disappointment in the midst of all these security enhancements is that Apple did not improve the Library Randomization feature introduced in Leopard. Also known as ASLR, and found in Windows Vista and Windows 7, it’s a powerful operating system security technology that nearly eliminates the memory-based attacks we’ve spent so much time talking about.

Library Randomization picks different memory locations for key operating system components each time the system starts up. Even if an attacker exploits a vulnerability on your system, it is nearly impossible for them to tie into the operating system and actually do anything malicious (or otherwise) because they can’t rely on where the hook can be found.

Library Randomization in Leopard and Snow Leopard does shift around some important pieces of the operating system, but leaves the memory location of one key component static across all Macs (dyld, the dynamic loader). With dyld in place, an attacker has a roadmap to continue their exploitation and potentially take over your system.

Randomizing the location of dyld is no small task, but Apple had a perfect opportunity to make the change with Snow Leopard, since so many other important parts of the operating system were being updated. Combined with the 64-bit enhancements, it would make memory exploitation of any type extremely difficult and provide years of worry-free Mac computing.

Continually Improving Security — Snow Leopard also includes a few other small changes. Users concerned with privacy can disable location services in the Security preference pane (in the General view, check Disable Location Services). As on the iPhone and iPod touch, location services allow your current coordinates – derived via Wi-Fi signals as well as future GPS hardware – to be used by system components and third-party software. Date & Time, for instance, now uses Wi-Fi signal snapshots to set your time zone automatically.

Apple also increasingly phased out the use of the setuid function in the operating system, which reduces security by running processes under administrative or other user accounts.

It’s important to remember that Apple has gradually been enhancing security, sometimes with major enhancements, through Software Update long before the release of Snow Leopard. Sandboxing, increased stack memory protection, reducing use of setuid, adding anti-phishing to Safari, and a series of other changes have found their way onto our Macs outside of major operating system version updates.

Overall, Snow Leopard is more secure than Leopard, although Mac users on 32-bit processors won’t see all the benefits.

Still, I am extremely disappointed that Apple failed to complete Library Randomization. Microsoft has experienced significant real-world security benefits with their adoption of ASLR, and had Apple taken this step they would have practically eliminated memory-based attacks like buffer overflows.

Although most of the security enhancements in Snow Leopard are hidden deep within the operating system, they should provide practical benefits to all Mac users on 64-bit architectures. While the only true test of security is how effective it is in the real world, on paper it looks like life is now at least a little harder for any potential Mac attackers.