zillablog - postgres

Database Scalability Patterns - OSCon 2010

nospam@example.com (Robert Treat) — Fri, 23 Jul 2010 15:42:06 -0400

Howdy folks,

slides are up for my talk, "Database Scalability Patterns", which I gave this week at OSCon 2010. You can get them from the OSCon page, from slideshare, or just watch it below

Database Scalability Patterns

View more presentations from Robert Treat.

BWPUG July Meeting 2010-07-14: PostgreSQL Performance Farm

nospam@example.com (Robert Treat) — Tue, 13 Jul 2010 11:29:00 -0400

A reminder note from Stephan Frost:

Hey all!

This month BWPUG returns to DC/Falls Church!

When: July 14th, 6:30PM (this Wednesday!)
Where: 3150 Fairview Park Dr, Falls Church, VA
Host: Noblis, Inc. and the
Noblis Innovation and Collaboration Center (NICC)

It's basically at 495 and 50 in Northern Virginia. We'll be discussing
the work and progress made so far on the PostgreSQL Performance Farm
scripts done by Scott (an intern working for Noblis over the summer
from Virginia Tech).

If you're planning to attend, please RSVP to sfrost@noblis.org. The
host/facility we'll be meeting at does require a government issued
photo ID (eg: driver's license or passport).

Hope to see you there!

Thanks,

Stephen

BWPUG June Meeting 2010-06-09: PostgreSQL on FreeBSD

nospam@example.com (Robert Treat) — Thu, 03 Jun 2010 14:54:00 -0400

After a brief hiatus, BWPUG is back with an all new meeting for June!

This months speaker will be Greg Smith, who reprises his talk from BSDCan, presenting on "PostgreSQL on FreeBSD". The talk discusses some of the technical and business hurdles in deploying database on the FreeBSD architecture, and touches on topics like what former users of OpenSolaris might be looking for in a new OS.

When: June 9th, 6:30PM.
Where: 7070 Samuel Morse Dr, Columbia, MD, 21046.
Host: OmniTI

As always we'll have time for networking and likely hit one of the local restaurants after meeting, hope to see you there.

PGCon 2010 slides are up

nospam@example.com (Robert Treat) — Mon, 07 Jun 2010 21:05:32 -0400

They've actually been up for a few days now, but since I haven't mentioned it, I guess no one noticed

This includes slides for both the "Know More Waiting / Postgres 9" talk, and my "Check Please! / Postgres Monitoring". I've uploaded the slides to the conference webstie for download, but I think it needs to get a site cache rebuild or something, in the mean time you can view the slides on my slideshare page.

As always if you have any questions, please feel free to contact me.

ossdb-snapshot, lvm database snapshot tool

nospam@example.com (Robert Treat) — Tue, 03 Jun 2008 18:28:00 -0400

Just tossing this out there on the chance it might be useful for others. This script, dubbed ossdb-snapshot, is based on the mysql-snapshot tool from Jeremy Cole. The idea is to simplify management of lvm snapshots for purposes of creating postgresql snapshot style backups (I've left the mysql pieces in place in case anyone wants to make it a truely cross-db app), either for backup or pitr setup. Quick example after the jump.

In this system, the lvm setup on the master looks something like this:



[root@luigi ~]# lvdisplay

  --- Logical volume ---

  LV Name                /dev/pipedbvg2/pipedbdata

  VG Name                pipedbvg1

  LV Write Access        read/write

  LV Status              available

  LV Size                185.00 GB

  Current LE             47360

  Block device           253:0



  --- Logical volume ---

  LV Name                /dev/pipedbvg2xl/pipedbxlog

  VG Name                pipedbvg1xl

  LV Write Access        read/write

  LV Status              available

  LV Size                27.00 GB

  Current LE             6912

  Block device           253:1





Basically we want to grab a copy of postgresql using the start/stop backup commands, mounting our lvm snapshot at /pipedb/pitr so we can copy it to another box. The commands look something like this:



[root@luigi ~]# ./ossdb-snapshot --dbtype pgsql -u postgres -p password -c --origin-lv pipedbdata --origin-vg pipedbvg2 --snapshot-name pgpitr --snapshot-size 20GB --mountpoint /pipedb/pitr



[root@luigi ~]# lvdisplay

  --- Logical volume ---

  LV Name                /dev/pipedbvg2/pipedbdata

  VG Name                pipedbvg2

  LV Write Access        read/write

  LV snapshot status     source of

                         /dev/pipedbvg2/pgpitr [active]

  LV Status              available

  LV Size                185.00 GB

  Current LE             47360

  Block device           253:0



  --- Logical volume ---

  LV Name                /dev/pipedbvg2/pgpitr

  VG Name                pipedbvg2

  LV Write Access        read/write

  LV snapshot status     active destination for /dev/pipedbvg2/pipedbdata

  LV Status              available

  LV Size                185.00 GB

  Current LE             47360

  COW-table size         20.00 GB

  COW-table LE           5120

  Allocated to snapshot  0.00%

  Snapshot chunk size    8.00 KB

  Block device           253:2



  --- Logical volume ---

  LV Name                /dev/pipedbvg2xl/pipedbxlog

  VG Name                pipedbvg2xl

  LV Write Access        read/write

  LV Status              available

  LV Size                27.00 GB

  Current LE             6912

  Block device           253:1



[root@luigi pitr]# rsync -a --progress --delete --exclude='postmaster.pid' --exclude='postgresql.conf' --perms /pipedb/pitr/ mario:/pipedb/data/



[root@luigi ~]# ./ossdb-snapshot -d --origin-vg pipedbvg2 --snapshot-name pgpitr





The above works fine in my setup for doing pitr setups, if you have a different setup or need, you might need to adjust things accordingly.



phpPgAdmin 4.2.3 Released
nospam@example.com (Robert Treat) — Tue, 13 Apr 2010 11:05:43 -0400
phpPgAdmin 4.2.3 Released

-----------------------

4-13/2010



The phpPgAdmin Team is happy to announce a new bugfix release for phpPgAdmin. Version 4.2.3 fixes several long standing bugs and fixes some PHP 5 compatability issues in the 4.2.x branch. All users of phpPgAdmin are encouraged to upgrade to this new version. 





Download

--------

To download right now, visit:

http://phppgadmin.sourceforge.net/?page=download

(RPMs available soon)





Demo

----

To give the fully-functional demo a try, visit:

http://phppgadmin.kattare.com/phppgadmin4/





Deprecation Warning

-------------------

Note, this may likely be the last version of phpPgAdmin released on the 4.2.x  branch. We are currently planning to release phpPgAdmin 5.0 this summer, which will support PHP 5+ only, and Postgres 7.3+. Legacy users should continue to use 4.2.3. 





Regards,

The phpPgAdmin Team



Is lastval() evil?
nospam@example.com (Robert Treat) — Sat, 01 Apr 2006 00:41:00 -0500
Got bit by a bug in our application today. The code in question inserts into the site table, calls lastval(), then takes that value and inserts into host, calls lastval() again, then inserts into application table. (We're building relationship trees).  The problem? We added a new trigger to the site table, to have it do an insert into another table to handle some advanced messaging.  Unfortunatly this table had a sequence on it...  




What ended up happening was we were adding sites fine for a couple hours, but messaging was broken. By the time we tracked that down, we got messaging working (almost) but then site insertion started tossing errors. Fooy!  We chase a number of red herrings but finally realized that our lastval() calls were generating imroper sequence numbers! So we are now replacing all calls to lastval() with a curval() and the appropriate sequence name. 




Anyone else been bitten by that?  I actually told my app developers we could run into this problem in the future when we first decided to use lastval(), but we wanted to use some new 8.1 goodness and also thought this was a good way to hide the sequence machinations from our timid junior app developer, plus I didn't figure it  would happen so quick... we never even made it through one release cycle!  So, I am hereby declaring lastval() as evil, or at least a problem waiting to happen... beware the lastval()!!








watch for momentary monitoring
nospam@example.com (Robert Treat) — Sat, 10 Apr 2010 12:50:00 -0400
One of the things I preach about a lot is good monitoring of your database servers; having tools in place to tell you both what good looks like and when things go bad is critical for large scale success. But sometimes you just need to monitor a momentary process, where setting up a check in your normal monitoring software is overkill. In these cases one tool that can help out is the watch command. 



Case in point, the other day I needed to back up a fairly large partitioned table (about 1.3TB on disk). The plan? A quick little script to pg_dump each of the partitions (about 325). Feed the script through xargs -P so I don't swamp the box, but I get some concurrency out of things. And of course, I planned to run the whole thing in screen session. But dumping this much data will take some time, so how to check on the progress?



When working on databases, one of the most natural things to me is to whip up some SQL to see what going on inside my database. Then you pipe that through watch, and you have some quick and simple monitoring. This example happens to be on postgres, but you could do it with any database's command line program. 

 
Continue reading "watch for momentary monitoring"