postgres

I often run my ops like I take care of data; a bit overzealously. Case in point, when setting up a new database, I like to throw on a metric for database size, which gets turned into both a graph for trending, but also an alert on database size. Everyone is always on board with trending database size in a graph, but the alert is one people tend to question. This is not entirely without justification.

On a new database, with no data or activity, deciding when to alert is pretty fuzzy. When we set up a new client within our managed hosting service, I usually just toss up an arbitrary number, like 2GB or something. The idea isn't that a 2GB database is a problem, it's that when we cross 2GB, we should probably take a look at the trending graph and do a projection. Depending on how things look, we'll bump up the threshold on the alert to a new level, based on when we think we might want to look at things again. For example, in this graph we take a month long sample, and then project it out for three months. We can then set a new threshold somewhere along that line.



While this is good for capacity planning, there's more that can be gained from this process. The act of alerting forces us to pay attention. And if we get notices before our expectations, we go back in and re-evaluate the data patterns. Of course, some times people will question this. Getting a notice that your database has passed 4GB can seem pointless when you have 100+ GB of free space on your disks. And besides, isn't that what free space monitors are for?

Here is a graph of another of our clients database growth. Their data size is not particularly large (don't confuse scalability with size; it doesn't take a large database to have scalability issues), but what's important is that we kept getting notices that the size was growing, and when talking with the developers, no one thought it should be growing at nearly this rate. Eventually we were able to track down the problem to purging job that had gone awry. Once that was fixed, the growth pattern leveled off completely (and the database size returned to the tiny amount that was expected!)

There has been a lot of chatter the past week about Apple replacing MySQL with Postgres in the new OSX Lion Server [U.S. | England | New Zealand ]. Most of it seems to tie things back to Oracle's new stewardship over the MySQL project, a lot of that stemming from what I would say is FUD from the EnterpriseDB folks, regarding doom and gloom about the way Oracle might handle the project in the future. Not that the FUD is entirely unwarrented; While Oracle has done a pretty decent job with MySQL so far, looking at what Oracle has done to projects like Open Solaris certainly would make one queasy. And yes, we've seen an uptick in people asking for help with Oracle/MySQL to Postgres migrations since the acquisition of Sun. That said, I have an alternative theory. Maybe they just like it better?

OK, I am just trying to set the record straight. People are still confused thinking I might be going to PGWest, but I'm not. I know where the confusion comes from; on the PG West website, there is a picture of me in the banner graphic; which makes people think I am going to PG West. This is not unreasonable, it's just untrue. For what it's worth, I did ask Joshua to remove my picture when people first started asking me if I was going, and he said he would, but that was well over a month ago. I do think he will take it down, but in the mean time, I figure I should at least put some effort into clarifying things myself. So, to be clear, I will not be going to PG West this year. Also, to be clear, it's not that I have anything against PG West per se. I've gone to multiple PG West cons in the past, and I suspect I'll probably go to more in the future. It's just that this year, I've got something better to go to. That something is Surge.

What is Surge?

Surge is the premeire conference on internet scalability. Now, I have to disclose, I am affiliated with the conference, but this conference really stands on it's own merits, no question. Now in it's second year, Surge packs an incredible lineup of people leading large scale operations on the net. Reading through the speakers list, I see companies like Yahoo, Wikia, Message Systems, Varnish, MyYearbook, Percona, Etsy. If you are trying to grow at scale, you can learn a lot from this crowd.

Yeah, but I'm a DBA

So most people going to PGWest are probably DBA's, or at least work closely with Postgres, so it makes sense for them to go to PGWest; I get that. But here is why you may not want to. The thing about Surge is that, while it isn't a database conference per se, a fair amount of the content does revolve around managing data. Let's face it, if you are running a website at scale, chances are you have to deal with large amounts of data. Whether it's massive data on disk, or dealing with massive throughput of data, or trying to figure out how to visulaize all that data, Surge has it covered. And what I find most intriguing is that because Surge is not focused on any particular technology, you get to see both problems and solutions from different angles, which I think helps to learn even more. Of course, you don't have to take my word for it; scan the speakers list, check out the talk profiles, and see if there isn't something there looks awesome.

See you in September

In any case, record set straight. You know where I'll be, I hope to see you then. Oh, and in case you need incentive, early bird pricing is still in effect until the end of July. Get on it!

I think I'm just about dug out from last weeks NYC trip for PGEast. I have to say I had a good time at the conference, although I was kept busy enough not to be able to get to see nearly as many presentations as I would have liked. One thing I did walk away with was a renewed appreciation for the projects we get to work on at OmniTI. I angled most of my talks toward PostgreSQL 9.0, going in thinking that a majority of people would have probably made the move already. Apparently that isn't the case, which in retrospect seems obvious; a majority of our customers still run some 8.x version of Postgres, as we work to vet applications and find the right windows to make their upgrades happen. Still, we are fortunate to have some clients who are very aggressive when it comes to server versioning, and we're lucky enough to get to run a healthy amount of Postgres 9, in some cases with fairly complex distributed server arrangements.

In any case, as per the title of this post, slides for my talks are now up on slide share, feel free to peruse, and if you have any questions please let me know.

Earlier today my colleague Depesz posted a nice write up showing one of the use cases for the new 9.1 feature, "Writable CTE's". It certainly shows one handy thing that this feature will enable, but it's not the only one. Here's a quick little bit of SQL I have been playing with for some time that re-implements the infamous "UPSERT" command (a long time sticking point for people trying to make MySQL apps more cross-database compatible) into Postgres.



Now, to be fair, this bit of SQL does have a race condition (think two people trying to insert the same actor at the same time), so it doesn't really solve all of your problems, but if you are looking for a quick hack, it might just do the trick. Also don't be afraid to play with it; this was like 2 minutes of thought and making sure the syntax worked; you could certainly try turning it around or coming up with other variants. That's actually one of the coolest things about this feature; waiting to see what use cases people come up with for it.

If you spend anytime either giving or receiving database tuning advice, perhaps the most common phrase you will hear is "it depends". Of course that is made in reference to a whole lot of things, but one of the things it depends on is read vs write load. While that's easy to point out, it's not always that obvious as to how best to measure read/write load for a system.

Once again someone has brought up the idea of having a hints system in Postgres, which means once again we're all subjected to watching people trod out the same tired, faulty, and even self-contradictory reasons to try to justify the idea that Postgres doesn't have, need, or even want a hinting system. As frail as the arguments might be, people are so entrenched in their positions now that even having a discussion on the topic is difficult. And in fact, there are really two discussion going on here; one is whether Postgres should have any type of hinting system, and the other a more specific discussion on if we should have a query hinting system.