PDA

View Full Version : Terabytes and terabytes of data



JuliasKils
05-14-2004, 05:41 AM
After going through the trouble of building my own machine I'm now looking to take some specs into building the ridiculously powerful monster computer and with it I need space. A lot of space. I'm thinking around 4-5 terabytes. I was told it is possible for me to install some sort of controller and have it connected to a box that holds hard drives. Is this possible? Can anyone give me some serious details on how to do so? The comp I would be putting it on has dual (2x) Intel 3.2 processors, about 4 gigs of ECC ram, the max of everything available right now even dual 19" LCD monitors. I'm sure it could handle the load but how do I set up that many hard drives externally all connected through my box? Thanks.

Yawgm0th
05-14-2004, 05:51 AM
Building a server or something?
You're going to want a bunch of SCSI drives and a few controllers for them. You'll spend thousands, maybe tens of thousands on these if you want terabytes of data. You'll have some expensive HDDs and some expensive controllers, along with a RAID setup (meaning you need to but at least twice as many HDDs)..
I won't go into details unless there isn't anyone here that has actually used SCSI. I could probably work out a good setup based on knowledge alone, but it's better to trust someone with experience.

JuliasKils
05-14-2004, 06:04 AM
Yup, building a server.. basically it is for sharing some heavy files between my team across the world (video editing, 3dsmax models, got some big projects going on). My bandwidth can handle it I've got about 5mb upstream.

I knew about the SCSI drives but not sure what the controllers look like nor what 'box' I'm supposed to buy to plug these up or if this RAID setup is software (I'm assuming it is) or hardware.

Thanks for the quick reply though, can you or anyone hook me up with some new egg selections for the brand of drives and whatever these controllers look like and where to purchase this RAID thing :lol: I learn fast just need a good footing.

JuliasKils
05-14-2004, 06:21 AM
Ahh.. alright, so only the SCSI drives are capable with these RAID cards/controllers. And these controllers allow me to plug up multiple hard drives. Gotcha. Now.. what type of RAID card are we talkin about? Like this?

http://www.newegg.com/app/ViewProductDesc.asp?description=16-115-002&depa=0

Or this?

http://www.newegg.com/app/ViewProductDesc.asp?description=16-116-023&depa=0

I'm assuming the second one as it allows 2.4 terabytes of data/8 drives on a single card, which means I would only need 2 of them to achieve the desired results. The only thing that worries me is that the max PCI slots on a motherboard is.. what 4? Seems so on the motherboard I'm getting, which means I would probably only have space for 1 card. Am I absolutely limited to those 4 slots or is it possible for me to get perhaps like an external PCI card or some such? Throwing at out ideas here as I'm trying to learn the jist of things.

Yawgm0th
05-14-2004, 06:38 AM
Niether; those are for ATA and Serial ATA.

Here's a list of what you would use:
http://www.newegg.com/app/searchProductResult.asp?submit=advance

Here's a list of hard drives you would use:
http://www.newegg.com/app/viewProduct.asp?DEPA=0&order=PRICE&submit=property&catalog=14&mfrcode=0&propertycodevalue=3348,%200,%200,%202960,%200,%200&keywords=&minprice=&maxprice=&description=+&Order=votes

I would suggest you go here and setup an account for more help:
http://beefy.nucleardays.com/forums/index.php?referrerid=12

Just like TweakTown forums, except all of the Tweaktown regulars went there, on top of having a bunch of nice features that make everything better. I would have pointed you there earlier, but I figured it would be best if we didn't have thirty people helping you with your computer. For this, I think there would be some more experienced members for you SCSI problem.

However, if you don't watn the hassle, I'm always here (and there )for more help. :D

By the way, you know I finished that guide, right?

JuliasKils
05-14-2004, 07:16 AM
Yup, been following the guide for the past 20mins, thanks again for giving us all a hand, it is definitely appreciated.

JuliasKils
05-14-2004, 07:17 AM
Ehh.. that list doesn't go anywhere
http://www.newegg.com/app/searchProductResult.asp?submit=advance


...Holy crap. $500 drives?
http://www.newegg.com/app/ViewProductDesc.asp?description=22-111-130&depa=0

That would mean even with 8 I would only have a little over a terabyte once the OS is installed. And thats $4k right there. Well.. expensive, but not too too bad I guess. If I can get a second SCSI card thing I'll go ahead after it.

Thanks for the link to the new forum, I'll signup once I finish these guides. I still didn't get a chance to pick up on the other comp, the one with the overheating processor but someone brought me a brand fresh new comp yet to be touched with a gig of ram and 2.2 intel so I'm just setting this one up till I get the nerve to go back to the other.

Yawgm0th
05-14-2004, 07:42 AM
I'm about to post the complete document in HTML on my site so people don't have to download a zip folder. Should be much easier. Plus I actually ran it through spellcheck.

You could also ask about the computer at that site. I wouldn't be suprised if it's faster than the one you just got (it probably is if the guy didn't build it himself).

For the cards, go to Newegg and click Advanced Search. Double-click Hard drive\RAID controllers and search for scsi. That should come up with everything you need.

Planatoid
05-14-2004, 08:56 PM
What about getting a few SATA Raid controllers and several 250gb harddrives? Since you are doing heavy graphics work you could get say two 74gb 10,000rpm SATA harddrives and use the others for storage. It's not like the harddrives are going to get much of a workout sending files over the internet. There are several PCI Raid controllers that allow up to 8 drives. Pop in 2 of those and a lot of 250gb harddrives and you're on your way to terabyte lane.

Just my 2 cents.

<Zack

JuliasKils
05-15-2004, 12:48 AM
Is there a big difference between SATA and SCSI? Yeah, it is just for storage and passing files nothing too extreme. I would prefer a bit cheaper as long as it holds the data and isn't flakey.

Yawgm0th
05-15-2004, 09:03 AM
You could get much faster and more reliable drives with SCSI. However, you could get as much space for much cheaper with Serial ATA. It's really up to your budget. Either way, you'll be spending thousands of dollars, but a SATA setup should be a few thousand or so cheaper.

JuliasKils
05-15-2004, 09:24 AM
:lol: a few thousand or so cheaper works for me but the problem I have is setting this up. Do I just buy the card and the hard drives? Do I need anything else? Software? Are the hard drives connected like mine are currently just as letter drives or must they be access through a command prompt? Can I use both SCSI and S-ATA or just one or the other?

Yawgm0th
05-15-2004, 09:56 AM
It shouldn't be too hard to setup. If you put Windows on a SATA or SCSI drive in RAID you will have to load some drivers during the installation process. They should come with whatever RAID card you get, but if not you should be able to download them and put them on a floppy. You can use both, no problem. There will be different speeds and different amounts of space involved, but you can certainly use both without problems.

Are you putting this in that server your building? You're going to want to make sure the motherbaord has a good five or six PCI-X slots for the cards.
For example, if you build a dual-opteron computer (unless your going to buy pre-built, which can be plausible with osmething this expensive), you might want this:
http://www.newegg.com/app/ViewProductDesc.asp?description=13-151-121&depa=0
The fact that it has onboard SATA/RAID and SCSI, along with 4 PCI-X slots make it great for having tons of storage attached. The dual-Gigabit ethernet is also great for transferring any big files (which obviously you will have) over a network or the Internet.

For SATA/RAID cards. which you'll need, you can go to advanced search at Newegg and search for Hard Drive/RAID Controllers with SATA as the search term.

I pulled out a few Google links so you can read up on RAID. RAID 1 will be faster and less expensive, but you might consider RAID 5. Note that you will basically have to buy twice as many drives as you are going to use for RAID, but it will be worth it. You can get big SATA drive for around a $100, so I wouldn't worry about price. It would cost exponentially more if you went with all SCSI.

http://www.webopedia.com/TERM/R/RAID.html
http://www.acnc.com/04_01_04.html
http://www.uni-mainz.de/~neuffer/scsi/what_is_raid.html

JuliasKils
05-15-2004, 02:23 PM
The article:
http://www.uni-mainz.de/~neuffer/scsi/what_is_raid.html

Is a damn good read, I had no idea RAID functioned like that! Able to salvage data even if 1 drive dies thats incredible. Didn't know they read like 1 single drive.. explains everything.

So I guess the whole placing hard drives in a box and connecting them singly was just a big fairy tale. Only leaves one.. area of questions..


Do I need to install OS on all of them for them to function (probably a stupid question) or just the single drive I'm using, with the drivers of course.

And how the heck do I hook them all up? The hard drives I mean. Don't they require IDE cables? Lets say for instance I used something like this.
http://www.newegg.com/app/ViewProductDesc.asp?description=16-103-142&depa=0

Doesn't say anywhere about how many drives it supports. The question is.. where do I plug what into? Certainly I must be missing something in the equation. Once I install the RAID card and the drivers.. where do I plug the drives into? Pretty sure I don't plug 4 IDE cables into the motherboard or some such.

Yawgm0th
05-15-2004, 03:11 PM
I'll e-mail you with some pictures and such for good explanations of some things...

But you don't need to install the OS on all of them. I suggest you just use the on-baord SCSI or SATA with that motherbaord or whatever one you get (should have one, the other, or both on-board) for your OS. After you install the drive on it, you can just put in the cards and drives and install them.

bigjackusa
05-16-2004, 04:25 AM
There really are storage boxes hooked up by a wire.http://compnetworking.about.com/library/weekly/aa070101a.htm

JuliasKils
05-16-2004, 04:43 AM
Unfortunately that link doesn't work. I would really like to see what the storage box looks like and which to buy.

bigjackusa
05-16-2004, 05:31 AM
I just tried the link and it worked for me. Here is a link to a specific product as well (the other was an article on NAS [network attached storege])
http://www.lacie.com/products/product.htm?id=10123

JuliasKils
05-16-2004, 06:00 AM
How strange the link works for you but not me. Hm.
..So you actually put the hard drives IN these things? :lol: Or do they come with the hard drives? Probably the ladder..

I think I get the jist of it all now, thanks alot Yawgm0th for the email, extremely informative! Time for me to get back to work on the other hot running box. Thanks again the both of you, should have 2TB of storage in no time!