BlueChips: Hassle-Free Apartment Finances

March 9th, 2010 @ 9:15 am UTC by evan

I live in a really awesome apartment. I’m living with really awesome people. And we tend to err on the side of awesome when it comes to buying stuff for the apartment. Specifically, we’re big fans of communalism – we do communal groceries, communal furniture, whatever.

But all four of us are paying for stuff for all four of us, it does make keeping track of money a little tricky. The traditional solution is for everybody to stuff their receipts into a drawer, and every month you all sit down and slowly work your way through the receipts.

Each of us owes you $50 for your grocery run, and you owe me $20 for my grocery run.

It’s a painful process, and the difficulty scales super-linearly as you add more people. Even with four, it would be pretty bad.

Now, it turns out that programmers hate tedious tasks like that, so there’s a long history amongst my friends of programmatically solving this in various ways. When we moved into this apartment, I figured I’d try my hand at it, and BlueChips is the result.

Since we set it up, BlueChips has been used by us and by other roommate setups to manage their expenses. We use it for tracking everything – rent, utilities, groceries, furniture, when all of us go out for dinner…

BlueChips has a very simple data model. There are users. A user can move money in two ways: expenditures and transfers. In an expenditure, one person spends money on behalf of a bunch of people. As a result of the expenditure, each of those people owes the spender some amount of money. BlueChips lets different people owe different amounts as the result of a single expenditure. For example, when we pay rent, each of us pays a different percentage, and BlueChips can follow that.

BlueChips’ biggest feature, though, is its ability to calculate the transfers necessary to settle the books. When it makes this calculation it also does something we call “pushing transfers”. Let’s say Larry owes Moe $1, and Moe owes Curly $1. BlueChips can “push” the $1 through, and will tell you that, to settle the books, Larry should give Curly $1.

If you’re still confused, or just want to see what the app looks like, I have a demo instance running at http://demo.bluechi.ps.

The software’s been around for a year, and it’s been open-source for most of that time, but I’ve never quite gotten around to putting a finishing coat of polish on it and getting it into a form that other people can use it.

When I lived with Scott Torborg and some other friends over the summer, we used BlueChips again for handling finances. Scott decided to put some of that polishing effort into BlueChips, and I have him to thank for all of the styling, excellent test coverage, and the iPhone interface, along with innumerable other tweaks.

I finally coded up the last big feature that BlueChips was missing: the ability to add new users without directly interacting with the database.

And so today I’m pleased to announce that I’ve tagged and released a version 1.0.0 of BlueChips.

BUT WAIT! THERE’S MORE!

For those of you MIT folks, I’ve worked with the scripts.mit.edu team to provide a Scripts autoinstaller. To install BlueChips, you can run the following commands from any Athena workstation:

dr-wily:~ broder$ add blue-sun
dr-wily:~ broder$ scripts-bluechips

Please remember that this is not a Scripts-managed autoinstaller. If you run into any problems, like it says, please let me know at bluechips@mit.edu.

And if you find BlueChips to be missing a feature you want, please feel free to write it yourself! In general, I don’t expect to have a lot of time going forward for new feature development, but I’m more than willing to review contributions from others. It’s my hope that the community can pick up my slack and keep BlueChips moving forward.

How to Teach Unix?

March 1st, 2010 @ 1:44 pm UTC by evan

As part of my involvement with SIPB, one of the biggest problems we run into is getting people started. As much as we emphatically insist that you don’t need to know anything about computers coming in (just be interested in them), it’s hard to implement that in practice.

One area that I think we do a particularly bad job of spinning people up on is how to use Unix-like environments. We’re a very Linux-heavy organization, and without some amount of *nix (and, in particular, *nix command line) comfort, it’s hard to figure out where to start.

When I’ve tried to teach people this sort of thing in the past, one thing I’ve always found is that you can’t use a system you don’t understand. You might be able to apply formulas to it (i.e. you might know “ls” or “blanche“), but without understanding the system, you can’t do things like building awesome complicated pipelines of 12 different commands, or whatever. So in the last 6 months or so, whenever I’m trying to teach somebody something, I take the time to teach it to them from the ground up. But I still didn’t have a good answer for teaching Unix.

I realized last night that I really learned how to think about Unix in 6.033, when we read the Unix Paper. In particular, sections 3, 5, and 6 are a pretty concise explanation of open, read, write, pipe, fork, exec, wait, exit, not to mention how input/output redirection, file descriptors, and shell fork+exec loops work.

And, modulo some slight naming changes, all of the information still applies to modern Linux. Not bad for a paper that’s 36 years old!

In any case, I’ve decided that pointing people at those three sections of that paper is my new answer for how to go from a formulaic understanding of Unix to actually being able to work with it. But it’s still only a start.

When did Unix click for you, and what actually did it? How do you help it click for other people? What other good beginner material is there, not just for Unix but other technical topics as well?

Debian Vcs-* Fields: Abusing the Google Juice

March 1st, 2010 @ 2:08 am UTC by evan

This is a blatant abuse of any Google juice this blog has picked up.

Every time I look for the definition of Vcs-Browser, Vcs-bzr, Vcs-cvs, Vcs-git, Vcs-hg, Vcs-hg, and Vcs-svn, it takes me forever to track it down.

In particular, searching for debian vcs-svn doesn’t actually find what I’m looking for. It took me running through a series of blog posts linking to forum posts linking to online list archives linking to Debian bugs to finally get the hint.

Now, for everybody’s sake, the information is not in Debian Policy, but rather in the Debian Developer’s Reference. Specifically, section 6.2.5.

Hopefully Google will pick this up and make searching for those fields easier. But even if it doesn’t, at least I know that I have a blog post with the information.

A Comment on Commenting

February 28th, 2010 @ 12:50 am UTC by evan

After Kevin’s post on commenting, I realized that I tend to be really bad about following through with blog comment conversations.

Kevin pointed out that he’s more likely to take the discussion to zephyr, the mostly-MIT-internal chat server. In fact, Nelson started the Iron Blogger event as a way to combat the fact that we tend to have all our interesting discussions on zephyr, instead of with the rest of the world. So blogging openly but replacing “commenting” with zephyr really defeats a lot of the point.

I know that for me the biggest reason I like having discussions on zephyr is because it’s easy to have a discussion. I don’t have to go seek out replies to my commentary – they show up automatically.

On the other hand, I read blogs through an RSS reader. I don’t tend to visit sites directly. And certainly I don’t go back through a blog’s history looking for replies to my replies. This means that it’s far too easy to make a comment and never look at the comment site again.

To try and combat this, at least for my blog, I’ve installed the “Subscribe to Comments” plugin. It was really easy – the plugin automatically adds the subscription checkbox to the comments form, although I decided to move it to put it above the comment textarea.

I’d encourage the rest of you to do the same – let’s bring the discussion, as well as the blogs, out of the MIT bubble.

Making Xen Suck Less: Part 3

February 21st, 2010 @ 11:23 am UTC by evan

Paravirtualized Clocks

In theory, Xen dom0’s are supposed to forcibly sync their system clock to the domU’s. In practice, due to some incompatibility between either Ubuntu’s version of the dom0 or domU patches, that doesn’t work, even though the feature is enabled, which leads to clock drift and occasionally weird clock lockup bugs.

The easiest way to fix this is to disable the Xen clock syncing entirely, and rely on the standard Linux clock mechanism. You can do that by adding these two lines just before exit 0 in /etc/rc.local:

echo '1' > /proc/sys/xen/independent_wallclock
echo 'jiffies' > /sys/devices/system/clocksource/clocksource0/current_clocksource

You’ll want to be sure to run NTP or some other service to keep your clock in sync.

Epic Softfail

February 11th, 2010 @ 2:26 pm UTC by evan

Somebody complained today that an e-mail I sent got caught in MIT’s spam filters, so I took a look at the message to see if I could figure out why it went through for me.

(For the record, using just the headers at your disposal to figure out why spam filtering is doing something strange is always a futile endeavor)

I didn’t figure out what was going on, but then I noticed this in the headers:

Received: by 10.102.218.17 with SMTP id q17cs56702mug;
        Thu, 11 Feb 2010 09:42:54 -0800 (PST)
Received: by 10.224.59.28 with SMTP id j28mr118230qah.109.1265910085825;
        Thu, 11 Feb 2010 09:41:25 -0800 (PST)
Return-Path:

Received: from dmz-mailsec-scanner-4.mit.edu (DMZ-MAILSEC-SCANNER-4.MIT.EDU [18.9.25.15])
        by mx.google.com with ESMTP id 17si5760822qyk.35.2010.02.11.09.41.25;
        Thu, 11 Feb 2010 09:41:25 -0800 (PST)
Received-SPF: softfail (google.com: domain of transitioning prvs=165867b240=uptrack@ksplice.com does not designate 18.9.25.15 as permitted sender) client-ip=18.9.25.15;
Authentication-Results: mx.google.com; spf=softfail (google.com: domain of transitioning prvs=165867b240=uptrack@ksplice.com does not designate 18.9.25.15 as permitted sender) smtp.mail=prvs=165867b240=uptrack@ksplice.com
Received: from mailhub-dmz-1.mit.edu (MAILHUB-DMZ-1.MIT.EDU [18.9.21.41])
	by dmz-mailsec-scanner-4.mit.edu (Symantec Brightmail Gateway) with SMTP id A4.AB.13801.441447B4; Thu, 11 Feb 2010 12:41:24 -0500 (EST)
Received: from dmz-mailsec-scanner-1.mit.edu (DMZ-MAILSEC-SCANNER-1.MIT.EDU [18.9.25.12])
	by mailhub-dmz-1.mit.edu (8.13.8/8.9.2) with ESMTP id o1BHdVsa009188
	for ; Thu, 11 Feb 2010 12:41:23 -0500
X-AuditID: 1209190f-b7bbfae0000035e9-cf-4b7441445cb2
Received: from mail-qy0-f202.google.com (mail-qy0-f202.google.com [209.85.221.202])
	by dmz-mailsec-scanner-1.mit.edu (Symantec Brightmail Gateway) with SMTP id 6B.AF.10714.241447B4; Thu, 11 Feb 2010 12:41:22 -0500 (EST)
Received: by qyk40 with SMTP id 40so1272989qyk.14
        for ; Thu, 11 Feb 2010 09:41:22 -0800 (PST)
Received: by 10.229.130.205 with SMTP id u13mr94912qcs.47.1265910082497;
        Thu, 11 Feb 2010 09:41:22 -0800 (PST)
Received: from ksplice.com ([64.27.0.149])
        by mx.google.com with ESMTPS id 20sm1621806qyk.9.2010.02.11.09.41.20
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 11 Feb 2010 09:41:21 -0800 (PST)

A lot of spew, of course, but the interesting lines are the two SPF softfails near the top: “domain of transitioning prvs=165867b240=uptrack@ksplice.com does not designate 18.9.25.15 as permitted sender

It took me a little while to figure out what was going on – I know that Ksplice sends its e-mails through Gmail, and I know that ksplice.com’s SPF record includes the Gmail mail servers.

But that e-mail went to a list at MIT, which then expanded to my MIT e-mail address, which then forwards to my GAFYD address. That SPF validation was performed by Gmail when it received the e-mail from MIT’s mail servers, and MIT’s mail servers aren’t authorized to send mail from ksplice.com.

I’m not sure how this is avoidable for this sort of mail forwarding – MIT’s mail servers could just as easily be spammers pretending to forward mail from ksplice.com. Maybe the solution is some way for me to tell Gmail that MIT is authorized to send my mail to me. Either way, it’s just more proof that SPF doesn’t work.

Update: Anders points out that this can be solved with “Sender Rewriting Scheme“, which basically just changes the envelope on the message to something that contains an obfuscated form of the original e-mail, but whose domain is that of the forwarder.

Making Xen Suck Less: Part 2

February 10th, 2010 @ 9:32 pm UTC by evan

Now for part two in my ongoing series on making Xen suck less. Last time we looked at making networking work for hardware virtualized machines. Networking for paravirtualized VMs does work out of the box, but this hint might help if you’re running into performance problems.

Paravirtualized Networking Performance

If you’re running a web server or some other server that’s sending large files (or sometimes small files), you may find that your VM seems to hang inexplicably on those transfers.

For some reason, the paravirtualized Xen networking drivers advertise that they support on-board TCP segmentation. In fact, they seem to pass the packets onto the wire un-segmented, which frequently will cause the packets to be dropped for going over the MTU.

If you’re using xen-create-image, there’s a commented out line in /etc/network/interfaces that runs ethtool -K eth0 tx off. That’s close to the right issue. You actually want to add a line to your /etc/network/interfaces so that it looks something like this:

auto eth0
iface eth0 inet static
 address 18.181.0.80
 gateway 18.181.0.1
 netmask 255.255.0.0

 post-up ethtool -K eth0 tso off

Stay tuned for more hints, including how to deal with clock issues and magic sysrqs. I’ll also be pulling walkthroughs together on converting paravirtualized to hardware virtualized VMs, and how to upgrade older Ubuntu releases to more recent ones safely.

Making Xen Suck Less: Part 1

February 7th, 2010 @ 12:55 pm UTC by evan

Right now, all of my Xen dom0’s run Ubuntu Hardy with Xen 3.3 from hardy-backports. Before we even talk about making Xen work, that statement bears some looking at.

I use Xen for a variety of reasons. Some are historical – the Invirt Project was built on top of Xen, and migrating away from Xen to a solution like KVM or VMWare would require working with users that are running paravirtualized operating systems. Some are circumstantial – I still have hardware that doesn’t support hardware virtualization.

My reasons for using Ubuntu are far less logical – I know how to use it, and don’t want to learn anything else if I don’t have to.

And I use Xen 3.3 because it’s way more stable than Xen 3.2.

In any case, if you find yourself using Xen 3.3 on Ubuntu Hardy as a dom0, there are a lot of tricks I’ve picked up for making it work better. Over the next few weeks, I’ll be working my way through them. I’ll be tagging them all with xen-tips for easy retrieval later.

As a disclaimer, I have no idea if these problems have been fixed in later versions of Xen or Linux, or if they’re specific to the Xen and/or kernel shipped by Ubuntu. For me, there’s a lot of value in getting all of my software from my distribution, so these instructions are designed to help do that.

HVM Networking

I have no idea whose fault this is, but HVM networking just doesn’t seem to work out of the box. qemu-dm, which emulates the VM’s devices, hooks the VM to a tap net device, while Xen sets up networking for a vifN.0 device. As far as I can tell, the intent was to connect the tap and vif devices, but nothing does.

For Invirt, we worked around this by writing a wrapper script around qemu-dm to make sure everything was setup correctly. If you want to use this script, you can drop qemu-dm-invirt in /usr/sbin and qemu-ifup in /etc/xen. (You’ll probably want to replace vif-invirtroute in qemu-ifup with vif-bridge or vif-route or whatever networking script you’re using).

/usr/lib/xen/bin/qemu-dm is hard-coded to run /etc/xen/qemu-ifup, if it exists. Without the qemu-dm-invirt wrapper, though, qemu-ifup doesn’t have any access to the domain ID for the domain it’s setting up. qemu-ifup then sets up and triggers the normal Xen networking script, which repeats the same setup it did for the vifN.0 interface.

Then, in your Xen config file, be sure to set device_model = '/usr/sbin/qemu-dm-invirt'.

Complex Systems and Simple Failures

January 25th, 2010 @ 8:45 am UTC by evan

Complex systems do not always fail for complex reasons. They quite often fail for the absolutely dumbest possible reasons.

- nelhage

Like say, for instance, when you’re debugging why one of your servers takes 7 seconds to do what the other server can do in less than 1.

root@black-mesa:~# time lvcreate -n test -L 1G xenvg
 Logical volume "test" created

real    0m0.309s
user    0m0.000s
sys     0m0.008s
root@torchwood-institute:~# time lvcreate -n test -L 1G xenvg
 Logical volume "test" created

real    0m7.282s
user    0m6.396s
sys     0m0.312s

Sometimes it turns out to just be that the directory it’s logging to takes 7 seconds to list:

root@black-mesa:~# time ls -a /etc/lvm/archive/ >/dev/null

real    0m0.005s
user    0m0.000s
sys     0m0.004s

root@torchwood-institute:~# time ls -a /etc/lvm/archive/ >/dev/null

real    0m7.007s
user    0m6.644s
sys     0m0.364s

And occasionally, that’s not just because your disk is failing or you’re running into caching issues. Occasionally, it’s just because that directory somehow has hundreds of thousands of files in it:

root@torchwood-institute:~# ls -a /etc/lvm/archive | wc -l
301369

And very, very rarely, if the gods are smiling on you, deleting those hundreds of thousands of files causes things to work again.

root@torchwood-institute:~# find /etc/lvm/archive -name '.lvm_torchwood-institute.mit.edu_*' -delete
root@torchwood-institute:~# time ls -a /etc/lvm/archive >/dev/null

real	0m0.015s
user	0m0.000s
sys	0m0.012s
root@torchwood-institute:~# time lvcreate -n test -L 1G xenvg
  Logical volume "test" created

real	0m0.341s
user	0m0.000s
sys	0m0.016s
root@torchwood-institute:~# time lvremove -f /dev/xenvg/test
  Logical volume "test" successfully removed

real	0m0.226s
user	0m0.004s
sys	0m0.012s

It kind of sucks to spend a week on and off, talking with developers, trying to figure out what’s going on. But it’s also really nice when it turns out to be something I can just fix myself.

ssh-keygen randomart

January 18th, 2010 @ 2:35 pm UTC by evan

So, at some point in the recent past (I don’t think I noticed when), ssh-keygen started displaying a “randomart” representation of keys that it generates:

fanty:~ evan$ ssh-keygen -f test -C evan
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in test.
Your public key has been saved in test.pub.
The key fingerprint is:
20:e9:b0:5b:5a:2b:ad:e8:4d:e4:b3:a0:32:49:2d:97 evan
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|     .           |
|  . o .          |
|   + . .         |
|  o.=   S        |
| ooE .           |
|.o*+o            |
|=.+oo            |
|=o.o             |
+-----------------+

Does anybody know why or what it’s supposed to mean?