Thursday, April 30, 2009

All fields become optional, all relationships many-to-many

Some wisdom on database app aging:
  1. All Fields Become Optional: As your dataset grows, exceptions creep in. There’s not enough research time to fill in all your company profiles, there’s one guy in Guam when you expected everyone to be in a U.S. state, there’s data missing from the page you’re scraping, you have to pull updates from a new source...
  2. All Relationships Become Many-to-Many: Some guy works in DC but lives in Virginia, so he needs two Locations. A new type of incoming email needs to be shoveled out to different feeds. A state has both a primary and a caucus. Someone eventually realizes categories never really were mutually exclusive...
Important to remember as we try to build a data commons that will last a long, long time.

No comments: