database question - how to organize

database question - how to organize

Postby darter » Thu Mar 14, 2013 6:59 pm

Suppose I had a website where each user could post the best song they heard that day and and make a comment. (Just as a silly example). Assume we expect expect 1000 users with unique user names. So, I might expect that there would be one record created possibly each day for each user. Would I just just save an individual record with the three fields and the database would gradually just grow that way. So after 365 days we could expect there would be 365,000 records. Suppose a user wants to see all of his records. It seems like the search has to look over all those records and all those user names. This seems really wasteful. What am I supposed to thinking about? Or maybe databases can just do this without any problem. I'm using postgresql and I'm just starting to think that soon the problem would get huge. With that size of data should I start to think about carefully organizing everything? Or is a typical database search so quick that I needn't even worry about a 'slow' search?
darter
 
Posts: 7
Joined: Sun Feb 10, 2013 5:19 am

Re: database question - how to organize

Postby setrofim » Thu Mar 14, 2013 11:11 pm

Don't worry about it. 365,000 entries isn't that much. For a typical request, network latency will be greater than the time to do a query. You can always add indexes for the columns you're querying on. As with most performance questions the answer is -- first implement your system, load test it and see if there are any issues before thinking about performance optimization.
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm

Re: database question - how to organize

Postby Yoriz » Thu Mar 14, 2013 11:39 pm

You could page the results using limit and offset, so your grabbing chunks of data rather then everything in one big chunk.
New Users, Read This
Join the #python-forum IRC channel on irc.freenode.net!
Image
User avatar
Yoriz
 
Posts: 1164
Joined: Fri Feb 08, 2013 1:35 am
Location: UK

Re: database question - how to organize

Postby darter » Fri Mar 22, 2013 3:02 pm

Thanks for the guidance. It looks like my concern is addressed by indexing something (as mentioned) - perhaps just one field and that orders the data by that field so searching is then quicker. I read a little here:

http://www.programmerinterview.com/inde ... -an-index/

Also people talk about having 1TB of data as though it is not all that big for a database.
darter
 
Posts: 7
Joined: Sun Feb 10, 2013 5:19 am

Re: database question - how to organize

Postby setrofim » Fri Mar 22, 2013 3:23 pm

darter wrote:Also people talk about having 1TB of data as though it is not all that big for a database.

We live in the age of Big Data. 1TB is peanuts :).
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm


Return to Web Development

Who is online

Users browsing this forum: No registered users and 0 guests