zillablog - postgres

Actually, the Relational Model doesn't scale

nospam@example.com (Robert Treat) — Mon, 08 Mar 2010 11:20:56 -0500

Before all my fellow DBAs' heads explode, let me just say that I am a relational guy. I like the relational model, think it's the best tool for the job, and think every programmer (not just DBA's) should aspire to be as familiar with it as they are with AJAX, MVC, or whatever other technology pattern you think is important. I'll even take that a step further; I think the NoSQL movement is mostly a re-hash of failed technologies from the last century. Object and document databases had their run in the market (some might say "they had their time"), and they were pretty thoroughly beaten by the RDBMS; that some people have reinvented that wheel doesn't change the game.

That said, I find the recent comments from Jeff Davis on the relational model and scalability to be overlooking some things. The state of computing tasks has changed over the past two decades, and what we know about computer engineering has also changed. Working on highly scalable systems like we do at OmniTI, you can't escape some of the inherent problems that you face when working in these types of environments. As much as I'd like the answer to every problem to be "just use an RDBMS", Brewer's CAP theorem just isn't something you can ignore.

When most people think about the relational model, they think of it in terms of parent-child relationships between tables. Without getting too deep in the details of it, I think it's pretty fair to say that Primary Keys and Foreign Keys are very large part of any relational implementation, and that pretty much all RDBMS strive to allow you to add these constraints to your model; it's what helps keep the data consistent. But there's the rub. CAP theorem points out that as we strive for tighter and tighter consistency, we are pulling away from availability, and sacrificing partition tolerance. Two theoretical systems that run smack dab into each other in the real world. This isn't really something new; if you have ever de-normalized, dropped a foreign key, or split data across multiple nodes, you've run into this before.

Now, where CAP theorem falls on it's face (imho) is that it also ignores another holy trinity of software development; Cheap, Fast, and Good. The size of your problem is dictated by the resources you have available; if you can afford decent tools (and let's be clear, decent is not your web dev throwing up MySQL on an EC2 instance) it is quite likely that the stressors of the relational model will never impact you in a way that most CAP folks are worried about. This is also one of the places the NoSQL movement fails; by throwing the baby out with the bath water. Giving up your data integrity before you have scalability issues is a form of premature optimization. The trick, as Theo would say, is having the experience to know when such optimizations are and aren't premature.

So what's the take away? I like to say that you use the relational model because it is best, and you use something else because it is necessary. Most SQL implementations can scale very well, and they should be your first choice when starting a new project. But we also can't pretend that there aren't inherent problems as these systems grow larger; let's understand the trade-offs and engineer appropriately.

OmniTI is heading to PGEast 2010

nospam@example.com (Robert Treat) — Tue, 02 Mar 2010 09:09:00 -0500

PGEast is the premiere Postgres conference held inside the U.S. each year, and this years conference, in Philadelphia, is now less than a month away. The organization and formatting have evolved a little from previous years, but one things still continues; a very strong presentation line up. We at OmniTI are very happy to be among that group of people, with four talks in this years conference lineup;

Know More Waiting, A Guide To PostgreSQL 9.0 by Robert Treat (hey, that's me), will give an overview of the upcoming PostgreSQL 9.0 release. While we're still a few months ahead of release, but we have a pretty good idea of what's coming in the next release, and this talk will help you start planning for how you will be able to take advantage of the new features coming our way.

PostgreSQL, meet AMQP, by Theo Schlossnagle, looks at pg_amqp, a "contrib" style module for Postgres that provides transaction style message queuing from inside of Postgres, using the AMQP standard.

Yet Another Replication Tool : RubyRep by Denish Patel, will delve into one of the newer Postgres replication solutions on the block. RubyRep is design for dead simple installation and setup, while still delivering advanced features like data comparing, synchronization between servers, and even master-master replication options.

Database Scalability Patterns by Robert Treat (me again), takes a look at the common patterns around scaling your database solution, and looks at some of the different options available to people scaling with Postgres.

But wait, there's more! While we at OmniTI are definitely excited to be participating in PGEast this year, there are a number of other good talks and speakers, including Magnus Hagander, Jeff Davis, Baron Schwartz, and many others. For full talk details, check out the conference talks page; I hope you'll journey out and say hi, it should be a pretty good time.

BWPUG March 10th, Falls Church take two.

nospam@example.com (Robert Treat) — Mon, 22 Feb 2010 21:02:39 -0500

Barring a repeat of last months snowmageddon / snowpocalypse, we're going to take another stab at heading down to Falls Church, Va, for the March BWPUG meeting. If you haven't felt like trucking out to Columbia, then please try to make this one. Depending on response we may be able to do this more often.

When: March 10th, 6:30PM.
Where: 3150 Fairview Park Dr, Falls Church, VA
Host: Noblis, Inc. and the
Noblis Innovation and Collaboration Center (NICC)

It's basically at 495 and 50 in Northern Virginia. We'll be discussing the upcoming PostgreSQL 9.0 release, including a preview of my upcoming talk at PG East.

If you're planning to attend, please RSVP to Stephen Frost via sfrost at noblis.org. The host/facility we'll be meeting at does require a government issued photo ID (eg: driver's license or passport).

Hope to see you there!

Re-inventing Greg's method to prevent re-inventing

nospam@example.com (Robert Treat) — Tue, 05 Sep 2006 11:52:00 -0400

A few weeks ago, Greg posted a very nice example of using pl/perl to ensure email validity inside your database. Certainly the principals of the post all make sense; make use of a database function to do your validity checking and save yourself from having to re-implement that logic in various applications that interact with your database. Of course, with PostgreSQL's wide array of procedural language support, there's more than one way to do it.

So the way I've decided to re-implement this is by using pl/php. As its name implies, pl/php is a PHP based procedural language that can be use for most tasks where other procedural languages can be used inside of a PostgreSQL database. It is developed primarily by Command Prompt, Inc. and is available as open source software.

Now, the thing that was so nice about Greg's example was that he didn't just write a little regular expression to do validation, rather he used a well developed CPAN module to do his dirty work. When I started to re-implement this in pl/php, I found no end of one off hacks for doing this in PHP, there are actually several posted right in the manual, but I wanted something that would be more robust. After a bit of digging, I came across Pear:Validate. As it's name implies, Pear:Validate is a package design to give robust, reusable validation routines from inside PHP. In my case it was just what I needed.



CREATE OR REPLACE FUNCTION valid_email(text)

RETURNS boolean

IMMUTABLE

LANGUAGE plphpu

AS $$

    require_once 'Validate.php';



    $validate = new Validate();



    return $validate->email("$args[0]",array("check_domain"=>false,"use_rfc822"=>true)) ? 1 : 0;

$$;

Thats it! In keeping with the spirit of things, I've installed this into our pagila sample database, and then tested out a few email addresses:



pagila=# SELECT valid_email('xzilla@users.sourceforge.net');

 valid_email

-----------

 t

(1 row)

pagila=# SELECT valid_email ('Robert Treat <xzilla@ users . sf . net>');

 valid_email

-----------

 t

(1 row)

pagila=# SELECT valid_email('www.brighterlamp.org');

 valid_email

-----------

 f

(1 row)

Yep, PEAR:Validate was able to handle the whitespace and leading text of the second example just fine. And just like any other function, we can also go ahead and use this function as part of a domain and incorporate that change into our database tables as needed.



pagila=# CREATE DOMAIN validemail AS text

pagila-# NOT NULL

pagila-# CHECK ( valid_email(VALUE) );

CREATE DOMAIN

pagila=#

pagila=# ALTER TABLE customer ALTER email TYPE validemail ;

ALTER TABLE

As you can see, it is very easy to incorporate this change into an existing system, and sharp readers will realize that we just verified the validity of all of the emails in our database as well! For those that want to see what an error looks like, we can show you that too:



pagila=# INSERT INTO customer (store_id, first_name, last_name, email, address_id, active)

pagila-# VALUES (2,'Gregis','Mijiro','gtsm123@aol',40,1);

ERROR:  value for domain validemail violates check constraint "validemail_check"

And wa-la! Friendly error message, centralized logic, and easy to update with a simple change to the valid_email function. While pl/php may not be as polished as something like pl/perl, you can still get a lot of gooey goodness out of it for those who are inclined to lean toward PHP.

Note: I'll be at php/db|works in Toronto next week doing a talk on pl/php, showing this and a couple of other neat tricks. If you're planning to attend, you've just gotten a sneak peek. If you're not planning to attend, it's not too late to change your mind.