Making kinit Less Sticky

planet sipb, posts 1 Response »
Jun 132010

I upgraded my laptop to Snow Leopard yesterday, and one thing I’m still reeling from is the changes to Kerberos. I’m not usually one to fault developers for wanting to move forward at the cost of compatibility, especially for rarely used features, so when I found that Apple made substantial changes to the user-facing side of Kerberos, I started updating my scripts and configuration to catch up.

I still have a few more bugs to track down, so i you know how to solve either of these, let me know:

Automatically renewing tickets.
/System/Library/LaunchAgents/com.apple.Kerberos.renew.plist appears to be a launchd job to do this, but I can’t figure out what’s supposed to trigger it.
Triggering code on ticket acquisition/renewal.
On older versions of OS X, you could set the libdefaults.login_logout_notification option in /Library/Preferences/edu.mit.Kerberos and cause Kerberos to call into a bundle in /Library/Kerberos Plug-Ins, but that doesn’t seem to work on Snow Leopard – the Login and Logout Notification API appears to be gone

In the mean time, after two hours of source diving later, I have solved one of my major bugs with Kerberos: de-stickifying options passed to kinit.

kinit seems to have gotten a lot of TLC in Snow Leopard—it appears to have been substantially re-written to take advantage of the Kerberos Identity Management API, which looks to be an attempt to genericize everything that made Kerberos on OS X special (multiple credential caches, system Keychain integration, etc.).

Unfortunately, one of the features it gained through this API was the ability to remember Kerberos ticket settings. In Leopard, if you changed ticket parameters in the “New Tickets” window of Kerberos.app (such as the duration of the tickets or the flags on them), those changes would be remembered the next time you used Kerberos.app to get tickets. But if you changed ticket parameters by passing flags to kinit, they would only work for one invocation.

But now, if I pass a flag like -l1m (get tickets that last one minute), that sticks across kinit invocations:

fanty:~ evan$ kinit -l1m -r5m broder
fanty:~ evan$ klist
[...]
Valid Starting     Expires            Service Principal
06/13/10 00:43:14  06/13/10 00:44:13  krbtgt/ATHENA.MIT.EDU@ATHENA.MIT.EDU
	renew until 06/13/10 00:48:13
fanty:~ evan$ kdestroy
fanty:~ evan$ kinit broder
fanty:~ evan$ klist
[...]
Valid Starting     Expires            Service Principal
06/13/10 00:43:26  06/13/10 00:44:26  krbtgt/ATHENA.MIT.EDU@ATHENA.MIT.EDU
	renew until 06/13/10 00:48:26

I found this undesirable, because when I pass flags to kinit, I want them to be for that invocation only, and any other time, I want my “defaults” – i.e. tickets that last as long as possible.

It turns out that the KIM API implementation on OS X takes backends into the standard OS X preferences API, and writes its settings to ~/Library/Preferences/edu.mit.Kerberos.IdentityManagement.plist. And in particular, it checks for the RememberCredentialAttributes to determine whether or not to store the preferences that are passed in. So just run

defaults write edu.mit.Kerberos.IdentityManagement RememberCredentialAttributes -bool false

to disable this feature. (Replace false with true if you want to undo the change)

If you want to follow the maze of twisty passages yourself, you can grab KerberosLibraries-81.46.1.tar.gz from Apple Open Source. Here’s the relevant call chain (except for kinit, which is in KerberosClients/kinit/Sources, all paths are relative to KerberosFramework/Kerberos5/Sources):

  • main (kinit.c:591)
  • kim_ccache_create_new (kim/lib/kim_ccache.c:214)
  • kim_ccache_create_new_with_password (kim/lib/kim_ccache.c:234)
  • kim_credential_create_new_with_password (kim/lib/kim_credential.c:411)
  • kim_credential_remember_prefs (kim/lib/kim_credential.c:224)
  • kim_preferences_create (kim/lib/kim_preferences.c:620)
  • kim_preferences_read (kim/lib/kim_preferences.c:433)
  • kim_os_preferences_get_boolean_for_key (kim/lib/mac/kim_os_preferences.c:412)
  • kim_os_preferences_copy_value (kim/lib/mac/kim_os_preferences.c:205)
  • kim_os_preferences_copy_value_for_file (kim/lib/mac/kim_os_preferences.c:142)

Around here the linearity of the call chain starts to break down, but kim_os_preferences_get_boolean_for_key gets called with kim_preference_key_remember_options, which gets passed to kim_os_preferences_cfstring_for_key in kim_os_preferences_copy_value, returning CFSTR ("RememberCredentialAttributes")

Eventually CFPreferencesCopyValue gets called asking for the “RememberCredentialAttributes” key in “edu.mit.Kerberos.IdentityManagement”, checking current-host/current-user, any-host/current-user, current-host/any-user, and any-host/any-user configuration settings, in that order.

Posted by evan at 1:13 am Tagged with: , , ,

No Need for Virtualization

posts No Responses »
May 012009

Today Duncan Keefe, Senior Manager in Apple’s Information Systems and Technology, presented on campus about how Apple’s IT department functions.

Somewhat understandably, there was a lot of sales pitch for Mac OS X Server in there incorporated in the talk, not to mention a lot of how Apple’s IT department is as awesome as the rest of Apple (which certainly seemed to be true based on the numbers we saw today). But some of the discussion on how to effectively communicate with your userbase would have been interesting for anyone who works in support, and there were a couple of interesting technical tidbits in there as well, and one in particular that still has me excited.

For example, did you know that Apple’s IT infrastructure is 71% based on open-source solutions? While I know as well as anyone that a lot of pieces of OS X itself are open source, they’re making use of a lot of enterprise-grade systems like SAP, which I thought would offset that number more.

Or another interesting fact: after migrating large parts of their infrastructure to Mac OS X Server and Xserves instead of Solaris or AIX or other systems, the sysadmin to server ration for OS X servers is 1:276. To compare that to the organization I know, SIPB has about 30 or 40 servers in its machine room, and there are about 20 people who have access to the machine room, not counting people without physical access who maintain servers on XVM – or people who just don’t have physical access. Now granted, Apple’s number is for maintaining a network with completely homogenous hardware and operating systems, and our tiny farm of servers probably runs more services per server than theirs, but the idea of a single person being able to run the entire SIPB machine room is…stunning.

But the truly interesting thing that Duncan mentioned was in response to someone’s question about virtualization. He responded that Apple currently doesn’t use virtualization for their IT infrastructure. Instead, they developed an in-house app that allows them to dynamically shuffle services around their servers based on the resources those servers need. Apparently their average server utilization is 60%.

And that is the dream of the cloud – by providing an environment large enough to contain your entire enterprise, you can smooth out what would otherwise be debilitating spikes in individual services. And in particular, this is the perfect answer to server virtualization.

If you look at your average, non-virtualized, single-purpose server, it’s probably at about 10% resource utilization, which makes it hard to justify buying a new server for each individual application. Virtualization is often touted as the solution to this problem – you run a bunch of single-purpose virtual machines on a single physical host. You can take advantages of features like guest migration to balance load dynamically. But if a service doesn’t need to exist in a completely independent instance of the operating system, you’re probably losing on the operating system overhead, in terms of disk space and RAM and probably processor usage as well. I’m willing to guess that the cost could be as much as 5% or 10%, which matters when you have hundreds of systems.

By dynamically shuffling applications without the extra overhead of full OS virtualization, you can take take advantages of the economies of scale without that overhead. Which is just awesome. And the 60% average utilization? Also amazing. It’s just about the perfect number: high enough that your servers aren’t twiddling their thumbs, but low enough that any one server should be able to handle a sudden spike.

I’ve been kind of excited about this idea all day, although I can’t really think of a scenario that I could apply the concept to. MIT’s infrastructure is too heterogenous in terms of both hardware and operating system to benefit, and all of the servers I maintain for SIPB are too specialized, or too heavily used already to benefit, or are run services that are un-migratable – or some combination thereof.

But it’s fun to think about what a system like that would take to implement – you’d have to be sure to never assume that a service lived on a fixed IP address. How often do you re-balance services? Unfortunately, I was a little too busy dragging my jaw across the floor to actually ask any interesting questions while Duncan was there.

Anyway, that was my exciting tech story for today.

Posted by evan at 7:57 pm Tagged with: , , ,
© 2012 No Name Blog Suffusion theme by Sayontan Sinha