zillablog

Database Scalability Patterns - OSCon 2010

2010-07-23T19:42:06Z

Howdy folks,

slides are up for my talk, "Database Scalability Patterns", which I gave this week at OSCon 2010. You can get them from the OSCon page, from slideshare, or just watch it below

Database Scalability Patterns

View more presentations from Robert Treat.

I hardly gnu, you?

2010-07-17T17:14:58Z

This morning I was browsing some twitter, when I came across this tweet:

"Everyone who bitches about the GPL ... I'd like them to remove all the GPL software from their computers, and see how they get on."

Now, this is all apart of some brouhaha over Wordpress and it's templates or something. I personally don't care about any of that (I've long moved past Wordpress, as I find them to be bad open source citizens), but it did get me to thinking, as a person who does think the BSD license is a better open source license than the GPL, just how much of an inconveneince would this be? I certainly do rely on a bunch of gnu software that I'd just as soon not live without, like my playstation, or our dvr, and probably our router too. However I don't really have a choice in these things, and to be fair I don't think there is anything perticularly special about linux that makes it better for a dvr than you could do with BSD.

But let's consider things where I do have a choice, where most software developers have a choice, which is in the tools we work with and things we work on. This is pretty broard, so let's just consider the classic LAMP stack that most people work on. Apache is of course, available under the Apache license, so it's already in the clear. While I like Linux, I've long replaced it in my life with a mix of OSX, *Solaris, and *BSD, for computers where I get to choose the OS. Yes, our eeepc does run Linux, but I could probably switch that to some BSD system if needed. Likewise MySQL has always had the backseat next to Postgres or Oracle, and even the NoSQL fanboys have plenty of non-gpl options (Couch, Hadoop, Cassandra, Voldemort, etc...) to pick from. And finally, whether your P is Perl, PHP, or Python, all of those languages are available under non-GPL licenses. So, I guess the "LAMP" stack could go on.

Surely there must be some things though right? I started to think about other tools I work with regularly, like X and vim, and while I'm sure there are some tools that might be gpl, certainly many are not, and I'd guess between Solaris and BSD, I could make a GPL-free stack that I'd be comfortable working on, with software that is already available. This isn't to say I wouldn't miss anything. I love my recursive grep, find the -P argument for xargs amazing, and find BSD tar just crippling to work with. None of those would be as bad as losing screen, which is a must have for any serious server work. Of course there are alternatives for all of these (not that I know anyone who uses tmux), and I'd bet some of these features could be easily re-implemented in a BSD version if needed.

I think where this really get's you is in the software that has been built upon these base tools, perticularly in the area of PHP software development. A lot of people over that last 10 years have produced GPL lciense software, like Drupal, PHP-Nuke, phpBB, phpMyAdmin, and more. I think most people didn't really think about there license choices back then (perhaps not now either); "if it's good enough for Linus". It's too bad, I know I'd much rather license phpPgAdmin under the BSD than GPL, but I think we're pretty much stuck at this point. I have noticed some newer projects (Habari for instance) have chosen non-GPL licenses; I don't know if that is a trend or anything, but it wouldn't hurt if it was.

BWPUG July Meeting 2010-07-14: PostgreSQL Performance Farm

2010-07-13T15:29:00Z

A reminder note from Stephan Frost:

Hey all!

This month BWPUG returns to DC/Falls Church!

When: July 14th, 6:30PM (this Wednesday!)
Where: 3150 Fairview Park Dr, Falls Church, VA
Host: Noblis, Inc. and the
Noblis Innovation and Collaboration Center (NICC)

It's basically at 495 and 50 in Northern Virginia. We'll be discussing
the work and progress made so far on the PostgreSQL Performance Farm
scripts done by Scott (an intern working for Noblis over the summer
from Virginia Tech).

If you're planning to attend, please RSVP to sfrost@noblis.org. The
host/facility we'll be meeting at does require a government issued
photo ID (eg: driver's license or passport).

Hope to see you there!

Thanks,

Stephen

BWPUG June Meeting 2010-06-09: PostgreSQL on FreeBSD

2010-06-03T18:54:00Z

After a brief hiatus, BWPUG is back with an all new meeting for June!

This months speaker will be Greg Smith, who reprises his talk from BSDCan, presenting on "PostgreSQL on FreeBSD". The talk discusses some of the technical and business hurdles in deploying database on the FreeBSD architecture, and touches on topics like what former users of OpenSolaris might be looking for in a new OS.

When: June 9th, 6:30PM.
Where: 7070 Samuel Morse Dr, Columbia, MD, 21046.
Host: OmniTI

As always we'll have time for networking and likely hit one of the local restaurants after meeting, hope to see you there.

PGCon 2010 slides are up

2010-06-08T01:05:32Z

They've actually been up for a few days now, but since I haven't mentioned it, I guess no one noticed

This includes slides for both the "Know More Waiting / Postgres 9" talk, and my "Check Please! / Postgres Monitoring". I've uploaded the slides to the conference webstie for download, but I think it needs to get a site cache rebuild or something, in the mean time you can view the slides on my slideshare page.

As always if you have any questions, please feel free to contact me.

ossdb-snapshot, lvm database snapshot tool

2008-06-03T22:28:00Z

Just tossing this out there on the chance it might be useful for others. This script, dubbed ossdb-snapshot, is based on the mysql-snapshot tool from Jeremy Cole. The idea is to simplify management of lvm snapshots for purposes of creating postgresql snapshot style backups (I've left the mysql pieces in place in case anyone wants to make it a truely cross-db app), either for backup or pitr setup. Quick example after the jump.

In this system, the lvm setup on the master looks something like this:



[root@luigi ~]# lvdisplay

  --- Logical volume ---

  LV Name                /dev/pipedbvg2/pipedbdata

  VG Name                pipedbvg1

  LV Write Access        read/write

  LV Status              available

  LV Size                185.00 GB

  Current LE             47360

  Block device           253:0



  --- Logical volume ---

  LV Name                /dev/pipedbvg2xl/pipedbxlog

  VG Name                pipedbvg1xl

  LV Write Access        read/write

  LV Status              available

  LV Size                27.00 GB

  Current LE             6912

  Block device           253:1





Basically we want to grab a copy of postgresql using the start/stop backup commands, mounting our lvm snapshot at /pipedb/pitr so we can copy it to another box. The commands look something like this:



[root@luigi ~]# ./ossdb-snapshot --dbtype pgsql -u postgres -p password -c --origin-lv pipedbdata --origin-vg pipedbvg2 --snapshot-name pgpitr --snapshot-size 20GB --mountpoint /pipedb/pitr



[root@luigi ~]# lvdisplay

  --- Logical volume ---

  LV Name                /dev/pipedbvg2/pipedbdata

  VG Name                pipedbvg2

  LV Write Access        read/write

  LV snapshot status     source of

                         /dev/pipedbvg2/pgpitr [active]

  LV Status              available

  LV Size                185.00 GB

  Current LE             47360

  Block device           253:0



  --- Logical volume ---

  LV Name                /dev/pipedbvg2/pgpitr

  VG Name                pipedbvg2

  LV Write Access        read/write

  LV snapshot status     active destination for /dev/pipedbvg2/pipedbdata

  LV Status              available

  LV Size                185.00 GB

  Current LE             47360

  COW-table size         20.00 GB

  COW-table LE           5120

  Allocated to snapshot  0.00%

  Snapshot chunk size    8.00 KB

  Block device           253:2



  --- Logical volume ---

  LV Name                /dev/pipedbvg2xl/pipedbxlog

  VG Name                pipedbvg2xl

  LV Write Access        read/write

  LV Status              available

  LV Size                27.00 GB

  Current LE             6912

  Block device           253:1



[root@luigi pitr]# rsync -a --progress --delete --exclude='postmaster.pid' --exclude='postgresql.conf' --perms /pipedb/pitr/ mario:/pipedb/data/



[root@luigi ~]# ./ossdb-snapshot -d --origin-vg pipedbvg2 --snapshot-name pgpitr





The above works fine in my setup for doing pitr setups, if you have a different setup or need, you might need to adjust things accordingly.



phpPgAdmin 4.2.3 Released
2010-04-13T15:05:43Z

                phpPgAdmin 4.2.3 Released

-----------------------

4-13/2010



The phpPgAdmin Team is happy to announce a new bugfix release for phpPgAdmin. Version 4.2.3 fixes several long standing bugs and fixes some PHP 5 compatability issues in the 4.2.x branch. All users of phpPgAdmin are encouraged to upgrade to this new version. 





Download

--------

To download right now, visit:

http://phppgadmin.sourceforge.net/?page=download

(RPMs available soon)





Demo

----

To give the fully-functional demo a try, visit:

http://phppgadmin.kattare.com/phppgadmin4/





Deprecation Warning

-------------------

Note, this may likely be the last version of phpPgAdmin released on the 4.2.x  branch. We are currently planning to release phpPgAdmin 5.0 this summer, which will support PHP 5+ only, and Postgres 7.3+. Legacy users should continue to use 4.2.3. 





Regards,

The phpPgAdmin Team

 
            


Is lastval() evil?
2006-04-01T05:41:00Z

                Got bit by a bug in our application today. The code in question inserts into the site table, calls lastval(), then takes that value and inserts into host, calls lastval() again, then inserts into application table. (We're building relationship trees).  The problem? We added a new trigger to the site table, to have it do an insert into another table to handle some advanced messaging.  Unfortunatly this table had a sequence on it...  




What ended up happening was we were adding sites fine for a couple hours, but messaging was broken. By the time we tracked that down, we got messaging working (almost) but then site insertion started tossing errors. Fooy!  We chase a number of red herrings but finally realized that our lastval() calls were generating imroper sequence numbers! So we are now replacing all calls to lastval() with a curval() and the appropriate sequence name. 




Anyone else been bitten by that?  I actually told my app developers we could run into this problem in the future when we first decided to use lastval(), but we wanted to use some new 8.1 goodness and also thought this was a good way to hide the sequence machinations from our timid junior app developer, plus I didn't figure it  would happen so quick... we never even made it through one release cycle!  So, I am hereby declaring lastval() as evil, or at least a problem waiting to happen... beware the lastval()!!






 
            


watch for momentary monitoring
2010-04-10T16:50:00Z

                One of the things I preach about a lot is good monitoring of your database servers; having tools in place to tell you both what good looks like and when things go bad is critical for large scale success. But sometimes you just need to monitor a momentary process, where setting up a check in your normal monitoring software is overkill. In these cases one tool that can help out is the watch command. 



Case in point, the other day I needed to back up a fairly large partitioned table (about 1.3TB on disk). The plan? A quick little script to pg_dump each of the partitions (about 325). Feed the script through xargs -P so I don't swamp the box, but I get some concurrency out of things. And of course, I planned to run the whole thing in screen session. But dumping this much data will take some time, so how to check on the progress?



When working on databases, one of the most natural things to me is to whip up some SQL to see what going on inside my database. Then you pipe that through watch, and you have some quick and simple monitoring. This example happens to be on postgres, but you could do it with any database's command line program. 

 
Continue reading "watch for momentary monitoring"