zillablog - postgres

LISA 2009 Wrap-up

nospam@example.com (Robert Treat) — Mon, 09 Nov 2009 00:06:00 -0500

While a good portion of the Postgres community was making their way to France for PGDay Europe, fellow BWPUG member Greg Smith and I were manning the home-front in Baltimore at the 2009 Large Installation and Systems Administration (aka LISA) conference, held this year in Baltimore, MD. The two of us took to the exhibition floor to man a booth for the PostgreSQL project, a two-day stint that gave us plenty of face time with the LISA attendees. For me it had been three years since my last LISA conference (at my other local city, Washington, D.C.) so I was curious to see how things had changed since then. Some thoughts/notes I took while working the show floor:

1) There were a lot of Postgres users at the show. A lot of happy Postgres users. Compared to 3 years ago when we ran into just a few, more than half the people who stopped at the booth were already using Postgres.

2) No one asked me "so why should I use Postgres instead of MySQL". Which is not to say the topic of MySQL didn't come up, but the above question is by far the #1 question I normally hear working community booths (even got it at OSCon this summer), so to not get anyone asking was quite a surprise in retrospect. I think this is probably due to two factors; first that Postgres advocacy has been working hard to make the case for Postgres and clarify the differences between the two projects, and second that we've gotten a lot of converts over the past three years so there's much more knowledge about Postgres these days. A couple people showed there was still work to do; some glossed over differences between the projects, and one person even thought Postgres was the commercial version of MySQL; so the job of Postgres advocacy goes on.

3) So where did they come from? Many of the people who told us they were happy Postgres users also mentioned previous database systems they had worked on. These aren't formal numbers, but I'd say the breakdown was close to 55% MySQL, 35% Oracle, and 5% Sybase and 5% MSSQL. Again rough numbers, but that seems about right. As the LISA crowd is heavy on system administrators, the complaints were mostly that MySQL was a pain to keep running (regular corruption issues and similar problems), and Oracle just couldn't justify its cost.

4) One person I spoke told me about a problem they had with setting up authentication. They run a university where they initially setup authentication for students via LDAP which they thought was pretty nice. They then ran into a problem because the students had to write scripts for classes, which required them to hard-code in their ldap passwords in the scripts, which were easily read by other students. They ended up solving the problem by configuring the apache server to run files as the script owner rather than the more standard "nobody" user, which allowed them to prevent others from seeing individual scripts. This isn't the first person I've run into with this type of problem; I'd love to see more people blogging on topics like this.

5) Several people asked about the business model behind Postgres. Many people get stuck in the idea that every piece of open source project has a single corporate backer/owner. I've been a big proponent of highlighting both the strength of the Postgres community and the nature of being a true Open Source project; so for me these are great questions to get to talk about, but it's something we should make sure other folks volunteering for booth duty are prepared to answer.

Finally, I want to say a big thanks to the folks running LISA and to the crowd at large. In a conference thin on DBA's, we still managed to get a number of donations which will help with further advocacy efforts. I guess system admins are into solid database software too.

Amazon Offers New RDS (aka MySQL) Service and New Database Related Virtual Machines

nospam@example.com (Robert Treat) — Tue, 27 Oct 2009 07:12:36 -0400

Amazon Web Services has announced a new service it is touting as Amazon Relational Database Services, designed to operate the operational management side of running a relational database. To be specific, the service is built around MySQL, and as the announcement reads

"Amazon RDS provides a fully featured MySQL database, so the code, applications, and tools that you use today with your existing MySQL databases work in Amazon RDS without modification. The service automatically handles common database administration tasks, such as setup and provisioning, patch management, and backup."

It is certainly an interesting offering for folks running MySQL, especially if you are managing you're own MySQL instances in Amazon's cloud infrastructure already. I didn't see anywhere where it listed the storage engines that would be available with the offering, which would be the first blocker for moving to such a service (I'm guessing that it will offer both InnoDB and MyISAM, but it doesn't say)

There are also some questions I have about how its back-up system works. It mentioned several times that backups can be done "automatically", and that you can use file system snapshots to restore your database to "any point in time" once deployed on their service. I'm a little skeptical about that, as filesystem snapshots don't necessarily just work (tm) when it comes to database backups, and MySQL backups are easy enough to get wrong in general, but it's certainly testable and would be a nice approach to solving the problem if it works.

The other thing worth noting is that the service doesn't offer replicated slaves, yet. From the Amazon RDS site, one of the new services they plan to offer "soon":

"High Availability Offering — For developers and business who want additional resilience beyond the automated backups provided by Amazon RDS at no additional charge. With the high availability offer, developers and business can easily and cost-effectively provision synchronously replicated DB Instances in multiple availability zones (AZ’s), to protect against failure within a single location."

Well, that doesn't sound like MySQL replicated slaves anyway, so running multiple services might still be a manual exercise. It's actually an important detail in my book; while Amazon is talking up the ability to scale up the new RDS service, MySQL is probably the worst of the 5 major databases (Oracle, DB2, MS SQL, MySQL, Postgres) for scaling up a database instances; being designed far better for scaling out; so any tools to help with this operation are key factors to the new service for me.

Speaking of scaling up, tucked away in the overall Amazon RDS announcement is also the announcement of new higher class EC2 instances, designed with running databases in mind.

* Double Extra Large: 34.2 GB memory, 13 ECU (4 virtual cores with 3.25 ECU each), 850 GB storage, 64-bit platform
* Quadruple Extra Large: 68.4 GB memory, 26 ECU (8 virtual cores with 3.25 ECU each), 1690 GB storage, 64-bit platform

Perhaps I'm just biased by the number of large scale instances we work with, but 32GB seems about the baseline of where I'd want to start out with for my database servers, so these new instances look promising. These EC2 instances aren't tied to the RDS service, you can run Oracle, Postgres, or whatever on them. I'd still like to see this scale up more (if folks running Postgres could go from the current "large" instance up to a 32 core, 256/512GB machine without having to get new hardware... the software could handle that and there would be no additional licensing... well that would be pretty compelling).

Anyway, Amazon has made a pretty big move into the database space with these announcements. I'm kind of curious what impact this might have on Microsoft's Azure service actually. Anyway, I'd encourage you to check out the new Amazon RDS site, and the new EC2 instance information (they've lowered some prices btw).

Denish looks at RubyRep

nospam@example.com (Robert Treat) — Thu, 20 Aug 2009 11:23:42 -0400

The OmniTI database team is noted for managing both large and high volume systems, but also for doing this in heterogeneous database environments. Accomplishing that is not always easy, often requiring custom solutions, so we try to keep our ears to the ground and investigate new tools as they come along. When RubyRep was first announced, I put it on the back-burner: a multi-master replication system that can work on both mysql and postgres? Sounds pretty pie in the sky to me. Luckily Denish (one of the DBA's on our team) isn't as cynical as me, so he took it for a test drive. I have to admit that so far it looks good, enough so that I prodded him to write down some notes on it, to which we've now gotten them up on-line. If you're into database replication solutions at all, I'd encourage you to take a look at RubyRep as well.

First Issue of Open Source Database Magazine Is Out

nospam@example.com (Robert Treat) — Wed, 22 Jul 2009 11:08:13 -0400

Happy to see the first issue of the new Open Source Database Magazine has been released.

For those that aren't aware, Open Source Database Magazine is a re-incarnation of the old MySQL Magazine. The open source database ecosystem has grown a lot of the last year, with the rising popularity of newish systems like Drizzle and MariaDB, the continued growth of the PostgreSQL community, the revival of old concepts like CouchDB, and the really ground breaking stuff like HadoopDB.

So, check out the website, check out my article, and let the OSDBzine folks know what new stuff in Open Source Database you'd like to see more about.

Three Cheers for the search_path!!!

nospam@example.com (Robert Treat) — Thu, 09 Jul 2009 09:51:00 -0400

Lately the search_path feature has been getting a bum wrap. People deriding it over security concerns, usability issues, and down right lack of usefulness. Well my friends, I say don't listen to the haters! The search_path feature is a really handy tool that you can use in a lot of interesting ways. I'm not just talking about making your database look like schema's don't exist (a primary argument for the feature way back in 7.3), but enabling functionality that would otherwise be somewhat of a pain. So I give you three cheers for the search_path:

Continue reading "Three Cheers for the search_path!!!"

Got Team?

nospam@example.com (Robert Treat) — Tue, 16 Jun 2009 15:03:00 -0400

This blog post is just a test posting for some new functionality just added to Planet PostgreSQL. Since it revolves around teams, I thought I'd do some self promotion, or rather, team building promotion. You see, we're still looking for DBA's here at OmniTI. Our database management practice continues to grow, and we need smart people to come work on some of the biggest systems and highest transaction loads you can find on Postgres. We'll also make you work with things like Oracle, MySQL, Sqlite, and who knows what else, just enough to make you question your assumptions. It's not for the faint of heart, but it's certainly interesting. Sound good? See the website for details, and who knows, maybe you'll end up on Planet Postgres as part of "Team OmniTI"!

The Asynchronous Services Analogy

nospam@example.com (Robert Treat) — Thu, 11 Jun 2009 23:52:02 -0400

Today I had a chance to sit through a sneak preview of Theo Schlossnagle's new talk Scalable Internet Architectures, to be delivered next week at Velocity 2009 (Dev sessions are an underrated side benefit of working at OmniTI). As always Theo packs a lot of good information into his talks; I could probably do blog entries on half a dozen ideas I jotted down; but I wanted to highlight something that he mentioned with regards to scaling websites via asynchronous services.

Continue reading "The Asynchronous Services Analogy"

The First Rule of Postgres Club

nospam@example.com (Robert Treat) — Fri, 05 Jun 2009 00:04:00 -0400

I had to chuckle when reading the comments of Alan Snelsons blog entry talking about PostgreSQL's project stability in the face of recent MySQL unsteadiness. Of course it shouldn't be a surprise that mentioning Postgres around MySQL people you might get some type of negative reaction. And vice versa of course. And with other db communities too, not just these two. But I digress. However what I really liked about the comments was this quote:

"PostgreSQL vs. MySQL is a non-issue for most people because the decision is already made. I don’t know what’s the point in bringing that up every-time, it shines a bad light on the postgres community."

The beauty here is that I found this blog post via Planet MySQL, not Planet Postgres. Further I've never heard of Alan Snelson; the post's author; he hasn't spoken at a Postgres event, and isn't syndicated on Planet Postgres. As far as I know, he just some guy who uses MySQL a lot, and now is looking at Postgres. Apparently he likes it and wants to spread the word. And of course people who don't like hearing about Postgres use this as a mark against the Postgres community, even when the Postgres project has nothing to do with it. You know software is no good when people who use it want to tell you how awesome it is!

In truth, that's how a lot of Postgres advocacy works these days. When you see people posting on slashdot, most of the time it's not from people working directly with the project. For those of us who have been using Postgres for a long time, we've long since got the message that MySQL users don't want to hear about Postgres. When I sat in the MariaDB feature request round table at the Percona Performance conference, listening to people ask for feature after feature that Postgres already has; partial indexes, index fillfactor, online ddl, extensible data types, dtrace probes, PITR, the list goes on; I didn't wave the Postgres banner. The only time I spoke about Postgres was when Monty had questions about Postgres' implementation. Some people asked me why I wasn't more vocal, why didn't I tell everyone about all the wonderful things that Postgres can do? What can I say... it's the first rule of Postgres club.

Guide to Postgres 8.4 video now available

nospam@example.com (Robert Treat) — Wed, 03 Jun 2009 23:39:00 -0400

For those looking for more information on the "Guide to Postgres 8.4" talk I have been giving recently, I thought I'd put up a few pointers to the information now available. First, there is now video available online from the presentation I gave at PGCon. It contains audio, video, and the slide stack to go with it. For some reason the audio feed doesn't work on my MacBook, though it does seem fine from my Linux box, so if you can get audio, it should be ok. I also noticed that some of the slides were cut in odd ways, so if you want the full slide stack from the talk, thats available on the PGCon website talk page. There's also a slightly longer slide stack from a 3 hour version of this talk I gave at PGEast which I made available at slideshare. There was supposed to be video of that session made available, but I haven't seen it yet, once it's up, I'll let you know. Hope this helps, and good luck with Postgres 8.4.

Reflections on PGCon 2009

nospam@example.com (Robert Treat) — Mon, 25 May 2009 11:01:01 -0400

Well, another PGCon has come and gone. I always find it ironic how you struggle to cross the finish line, exhausted after such a long week, and yet really it goes by so quickly; one certainly doesn't have time to reflect while at the conference. Actually my Monday was a perfact foreshadow of the week to come; get to the airport early to make a conference call, only to realize I forgot my boarding pass and then scambling to make the flight. As normal I landed in Ottawa completely unprepared for actually getting from the airport to campus, but was once again bailed out, running into Stephen Frost, whose flight had gotten bumped so he ended up on my plane, and so I hitched a ride with him into the city. My Tuesday would have similar ups and downs, spending a lot of hours on work projects, catching some of the tutorials for the day, some disorganized dinner plans, and then finally breaking bread with some of the Etsy crew with whom I've been working with recently.

Continue reading "Reflections on PGCon 2009"