solaris

Last week's "announcement" of the death of OpenSolaris has steered a lot of questions my way about where people should go, and/or where OmniTI will go, now that OpenSolaris future looks non-existent. As one of the more open users of Solaris related technology, and running some beefy loads on top of it, it makes sense that people would be curious as to what we might be doing next. I would start with saying that as a company, we don't have an official policy on this yet, and probably won't. We evaluate each situation on a customer by customer basis, so what follows here is more my personal feelings on what people should do at this current point in time.

The one thing I have noticed from the people I have already spoken with is that there seem to be two major camps, an over simplification to be sure, but I break this down into the free software camp (those motivated by a desire to remain on open source, and/or support, free software as a primary driver of technology decisions), and those more interested in the technology than the ideals behind it. Depending on where you fall into that spectrum, you have different options available to you, and will likely reach very different conclusions.

Too Soon?

The first thing I have said to everyone is that it is honestly too soon to make any moves. Oracle is notorious for being poor communicators, and at this point I don't think we've seen enough official communication to really know what's going to happen. This doesn't mean you can't start planning though! We've been looking at some of the available options since before the Oracle/Sun merger was closed, so it doesn't hurt to start evaluating the options out there. However there's no need to rush in to things; it is possible that the announcement of OpenSolaris's death might be premature. I personally don't believe Solaris can't survive based on the model we've just seen laid out; there are too many people learning the gnu tool chain who won't be willing to invest big money into a tool that is hard for them to use. They need a low cost / free option for people to familiarize themselves on (and all the better if it installs gnu tools by default). There's an outside chance Oracle might come to this conclusion, which would give new life to OpenSolaris.

A more likely alternative to that theory is that some other group might pick up OpenSolaris maintenance and start pushing it forward. Certainly not an easy task, but there are already several different distribution of OpenSolaris available, so the userland level management has the resources, we mostly would need to figure out how to handle the more core technologies that have been maintained by Sun. I think this might also be possible, as there are numerous companies already heavily invested in OpenSolaris technology, and there are Solaris internals hackers looking to move out of Oracle, it's not an impossible leap to think we might see something worked out. And if Oracle continues to make technology available via the CDDL (which most of the current signs seem to indicate), this could work out. I would say that this might not resemble the OpenSolaris as it is now, but could definitely be an option for current users who'd like to remain on the OpenSolaris platform.

Other Options?

Of course, you might not want to put all your eggs in that basket. So what other options do we have? Well, that mostly depends on what you're getting out of OpenSolaris now, and what you want out of your OS going forward. For many people, I suspect that Solaris 11 Express might be a suitable replacement, especially for those running mixed OpenSolaris / Solaris environments. Migrating up to full Solaris 11 will also cover most of your technology needs, so depending on pricing I suspect people may find that a cheaper alternative to migrating to a new platform. Of course, if you want to stick with a free software solution, this won't really be an option.

FreeBSD seems to be the most obvious alternative platform. If you're currently taking advantage of dtrace, zfs, and zones, FreeBSD gives you options to cover all three. It won't be the same; the dtrace and zfs implementations are pretty close aiui, but for zones you'll probably have to use either Jails or OpenVS, neither of which am I a fan of. I think you'd also find a larger overlap in system utilities (tar, find, grep, etc..) between FreeBSD and Solaris, so for people (and scripts) making the transition, this might be an easier move. The big question here is probably hardware support; if you can't get FreeBSD running on your hardware, that's likely to be a show stopper, unless you can work out a new hardware purchase in the transition

So, if you don't want to go closed Solaris, and FreeBSD isn't an option, that probably leaves you on Linux. People sometimes think I don't like Linux; I'm actually very comfortable on it. My first "unix" was Linux, and we run some extremely demanding systems on Linux and it has performed well in those cases. However if you're trying to do deep introspection, systemtap is a poor man's dtrace. And if you are relying on zfs, you'll have a hard time finding a suitable replacement amongst the current Linux options. Personally I am most comfortable on ext3, but I tend to give up on file system snapshots, which is a painful submission if you have to make it. XFS is probably the next most common option, and generally I've no bones about using it if you want to avoid ext3. Of the three "advanced" replacements; ext4, btrfs, and zfs on linux; I think ext4 is probably your best bet, but only because zfs is too new for any serious database systems, and if you are moving off OpenSolaris to get away from Oracle, "butter" seems like an odd choice.

And so...

I think it's wise to keep things in perspective. There are some cases where you want to be a technology leader (we've been running Postgres 9 for months on some systems), but generally speaking when it comes to picking the operating system and filesystem for your database, it's best to tread lightly. Now is a fine time to start evaluating your options; at least figure out what features are critical to your enterprise that you'll need to replace (and don't just think about database, you might be relying on crossbow for something, or who knows what else). We'll certainly be watching the current options available, and I suspect diversifying a little, over the next 6 months, as we wait for the picture to clear up where we can. We're not in a hurry (after all, we do have the source code of what we're running now), and I don't see much reason for others to be either.

After a brief hiatus, BWPUG is back with an all new meeting for June!

This months speaker will be Greg Smith, who reprises his talk from BSDCan, presenting on "PostgreSQL on FreeBSD". The talk discusses some of the technical and business hurdles in deploying database on the FreeBSD architecture, and touches on topics like what former users of OpenSolaris might be looking for in a new OS.

When: June 9th, 6:30PM.
Where: 7070 Samuel Morse Dr, Columbia, MD, 21046.
Host: OmniTI

As always we'll have time for networking and likely hit one of the local restaurants after meeting, hope to see you there.

I'm starting to see more and more chatter about a report from MarketWatch about IBM's apparent move to purchase Sun Microsystems; a lot of it centered around people's $favorite_open_source_project and how the ramifications of the deal might affect things in the long term. Personally I see a lot of downsides for open source as a whole if this deal goes through, based on the contrast of Sun's tight control of many of todays most popular open source offerings, and IBM's more general hands off approach.

Before I go too far, let me disclose that I submitted a talk for CommunityOne designed to cover some of the tools we've open-sourced that combine Postgres with various parts of Solaris (like dtrace and zfs), which was turned down.

Now, I'm willing to accept that my talk didn't fit the theme of the event or that Sun didn't find me to be a compelling speaker, or maybe they would just rather forget about Postgres in general; it happens, no big deal. But what I do have to take issue with is roster of talks that is chock full of Sun employees at an event called "Community One". Looking at the released list of sessions, I get 24 of the 31 talks being given by Sun employees; that's over 75%! If you want your conference to be about community, you need to have some actual community members involved.

I guess I shouldn't be surprised by this from a company that thought it could buy itself an open source database community, but last years line up wasn't nearly this Sun heavy. Not to mention the recent flack from the MySQL conference being so weighted toward MySQL employees. Is anyone at Sun paying attention? I suspect they have met the internet, but they seem to be two steps back on meeting the open source community.

So, not sure I will be attending CommunityOne this year. It is just a train ride up north, so there is still an outside chance, but that lack of community in this community conference is uninspiring (I suppose if I want community inspiration, I should be looking at Open Source Bridge), but I'd love to hear from others if they are going.

I had mentioned to a few people our TB+ disaster recovery scheme at the PG-East conference last week, with hopes that we would be doing a full on recovery test in early April. Lucky for us, we've been able to do a rough run through, so I wanted to report some results. First, a quick recap of why most of the common [http://www.postgresql.org/docs/current/interactive/backup.html backup solutions] suck for our needs:




pg_dump is pretty much a joke with 1TB+ of data, and especially on our system which has constant data churn, and enough mutating schema to make getting a consistent snapshot unworkable.


pitr would be nice for failover, but it isn't a real disaster recovery system. The key problems are issue with either corrupted xlogs making thier way to the slave, or data corruption issues getting propogated into your "backup". If you don't have a static snapshot, you can hose yourself in some un-fun ways.


slony (or bucardo, or other replication systems) also suffers from the issue of data corruption getting propogated onto your slaves, with no method to get back to a legitimate copy of your work. Again, this is fine when trying to solve failover, but not always the right answer for backups.





So, what we need is to make a copy of the database, and stick that some place safe and secure, so in case something goes horribly wrong (for **really** scary versions of horribly), we can get back to data that we know is good. The basic scheme goes like this:

Need a working PostgreSQL setup with archive_mode = on. We're using 8.3.1 on Solaris 10 / u4, all on top of [http://en.wikipedia.org/wiki/ZFS ZFS].


Issue a pg_start_backup() in postgres


Do a zfs snapshot of each filesystem (all tablespaces, and be sure to include all of $PGDATA, including all xlogs)


Issue a pg_stop_backup() in postgres


Ship the snapshots over to a back up machine (ideally somewhere remote, possibly on tape)


* Wait 3 days for the last step to finish [[image /xzilla/templates/default/img/emoticons/smile.png alt=":-)" style="display: inline; vertical-align: bottom;" class="emoticon" /]]

Once you have a the full snapshot, the idea is you can lay this onto your server, fire up postgres, and it will start as if coming up from crash recovery. The key to this process is having some form of archive recovery running so that you can run the snapshot commands between the start and stop backup commands, thereby capturing all the xlog files (and it is critical that you have all of them). The actual start-up process is un-climactic:







Exciting eh? We're currently going through some more testing, but all results so far have been positive (tables look right, seems easy to access everything, etc...), certainly we'll post more if we uncover something. For those looking at the details of the above, it didn't actually take 3 days to send the data to the other machine, but it is upwards of 24 hours, and can take longer when pushing to tape or similar setups; which becomes really important when you want to pull the data back out from the backups. Also it's worth noting you could probably do this with something other than zfs, but zfs certainly makes it nice and easy to script the whole thing and ship it around.

This morning when I walked into the office, Theo asked "Did you hear the news?". When I didn't immediately recognize what he was talking about, he knew I hadn't. At [http://www.omniti.com/ OmniTI], we are in an interesting position, as we use both PostgreSQL and MySQL, and we do a lot of it on top of [http://www.lethargy.org/~jesus/archives/77-Choosing-Solaris-10-over-Linux.html Solaris]; so news that [http://mysql.com/news-and-events/sun-to-acquire-mysql.html Sun was buying MySQL] certainly has the potential to impact us. Personally I don't think it will cause much change for us, but I thought I'd share some thoughts and ideas on the topic after talking with a number of people around these various communities today.




The first question on peoples mind seems to be "Why would Sun by MySQL? Haven't they been pushing PostgreSQL lately?". (Yeah, that's really 2 questions). Last weekend I saw [http://imdb.com/title/tt0472062/ Charlie Wilson's War], and I now have an analogy for how Sun has approached PostgreSQL. Basically the scene looks like this:

Me: So, what is Sun's strategy for PostgreSQL?

Berkus: Well, strictly speaking we don't have one, but we're working on it.

Me: Whose "we"?

Berkus: Me and three other guys.

(My apologies to Mr. Berkus, those are my words in his mouth, not his).



Seriously though, Sun really has not invested a whole lot into PostgreSQL relative to the size of thier company, or the size of the investment they are making in MySQL. What they have invested I think has been a really nice boon for the PostgreSQL community, and I'd like to think they have been rewarded on some level as well, but realistically, there hasn't been that much capital invested. As [http://people.planetpostgresql.org/greg/ Greg] put it on IRC, for one billion dollars you could take the top 100 Postgres developers and pay them each a million dollars a year for 10 years!



Now, just because they haven't invested the pinkies touch in PostgreSQL, that doesn't explain why they would buy MySQL. I don't have any insider info, but my guess is that it is all about control. Companies don't like to invest a lot of money into something that can't control, and PostgreSQL is like hearding cats; you can't control development, can't control release, can't control any of it; you have to be a team player, and multi-billion dollar companies don't like being team players, especially when they are the junior members. The other side of the coin I'd guess is that buying MySQL is like buying Facebook, maybe the business isn't really worth that much, but you get a huge install base and some level of revenue from that, so being able to point to that collection of people tends to comfort some shareholders.



So, having thought about the why, this of course gets people to wondering what is going to happen in the future. At this point I don't think we really know, details of the arrangement, and more importantly how MySQL people will be integrated into Sun (I don't think it is going to be at arms length as people think) will make a lot of difference. Rather than decide on any one facet, let me speculate on them all [[image /xzilla/templates/default/img/emoticons/smile.png alt=":-)" style="display: inline; vertical-align: bottom;" class="emoticon" /]]



This will be horrible for MySQL: Sun being Sun, non-Solaris platforms and non-Java programming languages will become second class citizens, MySQL support will get gobbled up inside the Sun machine, and, being constrained by a lack of solid OLTP engines not owned by [http://www.oracle.com/innodb/index.html direct] [http://www.dbms2.com/2007/12/21/ibm-acquires-soliddb/ competitors], they will re-position MySQL as an entry level database somewhere next to Derby, filling out a line-up where Postgres is pushed for mid-level business, and Oracle stays atop for large enterprises. This isn't even a worst case scenario; imagine that Sun engineering decides that the MySQL code base just isn't salvageable, and sets a path for MySQL 7 to be rewritten in Java (hey, it happened to [http://en.wikipedia.org/wiki/ArsDigita_Community_System ACS]). Given this stress, people being walking away, and Monty is out the door in 6 months time. OK, yeah thats all pretty wild, and not very likely, but you never know... [[image /xzilla/templates/default/img/emoticons/tongue.png alt=":-P" style="display: inline; vertical-align: bottom;" class="emoticon" /]]



This will be horrible for PostgreSQL: Stick with the staff of 4, or stick with the Billion. Basically the end game is Sun drops support of PostgreSQL, and maybe carries it further by promoting MySQL in the other open source products it controls; making the uphill battle that much steeper. This might also cause strain between some of the PostgreSQL related companies that have agreements with Sun (think Greenplum and Thumper).



This will be great for MySQL: MySQL has a lot of problems, many of them centered around thier management team. Depending on how things are aligned, Sun might be able to straighten out a lot of them. (One example, maybe Sun drops the whole dual license thing and makes MySQL a pure GPL product, that would be cool). Sun also has a lot of smart people, so perhaps they can straighten out many of the technical short-comings that MySQL has, and maybe they can even help push Falcon to a point where it is ready for production use.



This will be great for PostgreSQL: With Sun now in charge of a suite of database systems (Derby, Postgres, MySQL), and also in charge of a large development staff that specializes in the database field, they start promote a true Open Source Database environment. This would mean things like having MySQL developers work along side the PostgreSQL project, or taking some of MySQL's gui tools and turning them into database independent tools that work on all three database systems. They might also provide support for working groups trying to get the various open source databases to work better together; they might even turn the MySQL users conference into an Open Source Database conference.



Yeah, probably some of those things will happen, but most probably wont; I will make this prediction though... this change, like **all** changes MySQL does, will piss off some small number of thier users who swear off the database and switch to PostgreSQL. That much I am sure of, the rest, who knows?

I've been following reports of the 8.2.3 statistics bug for a few weeks now, mainly with intrest/concern as we run some 8.2.3 databases at [http://www.omniti.com/home OmniTI]. The reports all seemed reasonable (see emails [http://archives.postgresql.org/pgsql-hackers/2007-02/msg01751.php here], [http://archives.postgresql.org/pgsql-hackers/2007-02/msg01902.php here], and [http://archives.postgresql.org/pgsql-hackers/2007-03/msg00068.php here]), but it seemed odd because I haven't experienced such pains myself. I was kind of chalking it up to our 8.2.3 instances just not being heavily taxed (they can't all be TB databases [[image /xzilla/templates/default/img/emoticons/wink.png alt=";-)" style="display: inline; vertical-align: bottom;" class="emoticon" /]] ), but with some people [http://archives.postgresql.org/pgsql-hackers/2007-03/msg01529.php swearing off the 8.2 branch] until this gets fixed, I thought it might be worth a closer look to make sure of what was going on. I don't have answers per say, but I do have some information... I ran a few tests on our machines that run Solaris and PostgreSQL (compiled with Sun Studio, not GCC):





Of course, this is Solaris, so I poked [http://www.lethargy.org/~jesus/ Theo] for some dtrace-fu, and (in the end) got the following results:



The first couple times I did this with a lower number of calls (1000 range) the numbers didnt seem so bad, but the ratio looked sketchy so I kicked it up a notch to get a better reproduction and sure enough, they start to look at lot like those reports on -hackers. This makes more sense as the bug in question isn't something that should be able to be optimized away.



So what does this mean? I dunno... if you do a fair amount of testing to make sure your servers can handle the load you're going to be putting on them, I don't see much reason to avoid 8.2.3 (Again, our boxes are humming along fine). Even if you think this will cause you a performance issue, testing your schema and application against any of the 8.2.x branch is a good way to spend time while you wait for 8.2.4 (the fix is in CVS now, so we should see a released fix soon), and in many cases resolving those types of fixes are generally more time consuming anyway.