Please report all spam threads, posts and suspicious members. We receive spam notifications and will take immediate action!
Page 1 of 3 123 LastLast
Results 1 to 10 of 21

Thread: Are we using the right benchmarks for our reviews?




  1. #1
    Join Date
    Mar 2002
    Location
    Austin, TX
    Posts
    36

    Default

    I don't know what Cam thinks of me posting this thread, but I'm going to on the thought that we could use more community input on our testing.

    I'm reaching out to you guys(and gal). I'm sure everyone at TweakTown would love it if there were suggestions for possibly more benchmarks, removal of others, or a combo of both. Below is a list I compiled of the benchmarks TT uses for different types of reviews. From the list, please give your thoughts on the reliability of these benchmarks and ideas on what new tests could be included. This goes for overclocking and more.

    One more thing before I get the list out. Does anyone have any thoughts on the base test systems used on TT? In our reviews do we have enough comparisons (ie. 5950 vs. 9800XT, etc.) so that you can decide which board performs best? Do we provide enough information to help you decide if a product is worth your money? What questions do we leave unanswered?

    THE LIST:

    VIDEO CARDS:
    3DMark03(heading out the door)
    Aquamark3
    Halo
    Unreal Tournament 2003(When '04 comes, should we switch, use both, ideas?)
    Quake3
    Code Creatures
    Comanche4
    Jedi Knight 2

    - For video cards, we currently use the above. One of the main concerns is whether or not we should continue supporting FutureMark benchmarks. There have been an almost intolerable amount of discrepancies with mostly the 3DMark benchmark and we would like to know how our readers feel about this.

    CPU/CHIPSET:
    SANDRA2004
    PCMark2002(2004?)
    3DMark2001SE/2003
    Comanche4
    Jedi Knight 2
    Quake 3
    UT2003

    - We use a currently almost identical setup for testing Processors/Chipsets at TweakTown. Is this adequate? Would you like to see some more consideration in our benchmarking? Perhaps we could run a Pi Benchmark, memory latency tools, or something? Let us know!

    STORAGE:
    SANDRA2004
    Media Encoding
    Transfer Rate/DVD Decryption

    - Perhaps we should grab a copy of HDTach? Are our storage reviews lacking enough performance details? Should we have sound testing as well? Would you like audio clips available to you to hear for yourself how loud or quiet a storage device is?

    That's all I've got for now, back to the SATA article. Please think through what I've said above and give us some feedback. We want the community to recognize that we need your input to improve our site and make everyone's voice heard at the hardware manufacturers...Thanks everyone for any comments.
    Thomas Simmons
    Just some guy

  2. #2
    Join Date
    Nov 2001
    Location
    Taipei, Taiwan
    Posts
    4,308

    Default

    Anyone? :)
    Cameron "Mr.Tweak" Wilmot
    Managing Director
    Tweak Town Pty Ltd

  3. #3
    Beefy Guest

    Default

    For CPU tests, you might also want to consider something along the encoding / decoding lines. Use a program to encode an MP3 from a WAV file, or convert VOB to DIVX or something like that.

  4. #4
    Join Date
    Sep 2002
    Posts
    1,636

    Default

    how about trying to run some DC programs?
    but make sure to use the same workunit to compare all the systems
    :2cents:

  5. #5
    Join Date
    Mar 2002
    Location
    Austin, TX
    Posts
    36

    Default

    Any preference to the distributed computing program used? Perhaps someone would like to give us a yay or nay on FutureMark benchmarks (ie. 3DMark, PCMark, SYSMark). It'd be nice if there were some suggestions for 3d testing benchmarks we could use.
    Thomas Simmons
    Just some guy

  6. #6
    Beefy Guest

    Default

    Any DC should be fine, but I'd either go with SETI or F@H.

    Aside from that, the testing is probably good enough to give an idea of how the products perform. In 6 months you might want to review it all again though. :)

    Actually, putting in a new DX9 game could be a bonus.

  7. #7
    Join Date
    Dec 2001
    Posts
    494

    Default

    should use mm, united devices and distributed.net for dc benchmarks. hd tach would be really helpful. also try benching with max payne 2, s.t.a.l.k.e.r, xiii, need for speed underground, splinter cell, ffxi, prince of persia, tron 2.0, microsoft flight simulator, and a few newers ones?

  8. #8
    Join Date
    Jul 2002
    Location
    Canberra, Australia
    Posts
    1,158

    Default

    Quote Originally Posted by mistert
    VIDEO CARDS:
    3DMark03(heading out the door)
    Aquamark3
    Halo
    Unreal Tournament 2003(When '04 comes, should we switch, use both, ideas?)
    Quake3
    Code Creatures
    Comanche4
    Jedi Knight 2
    The following are my suggestions (with reasons underneath each)

    Aquamark3
    - Recent benchmark, seems OK

    Unreal Tournament 2003 (then 2004 when it arrives)
    - The Unreal engine is the basis for many games and is also a good indication of gaming performance

    Code Creatures
    - Ageing benchmark, but still seems powerful enough to use.

    Jedi Academy
    - Use instead of Jedi Outcast because it's a newer version of the Outcast engine, especiall with the Dynamic Glow option which is very intensive on ATi cards.

    3DMark03 (but not 3DMark01)
    - Yes I know the controversy very well, but the latest version of 3DMark03 has removed a lot of the "optimizations". As long as Futuremark continues to try to remove such cheating, 3DMark03 is OK to include as long it is in a benchmark suite and not the sole indicator.

    RTHDribl 1.2
    - This is one of my favorite demos and is a genuine, graphics-card only stress test/benchmark. It only works on PS2.0 or higher compatible cards, so it's a true DX9 test.

    Not sure about using Prince of Persia or NFS Underground as these are console ports and can run on even low end cards with little stress. A recent full PC game like Call of Duty, or even a BF1942 benchmark (due to its popularity) would be much more popular and indicative of a graphics card's practical performance.

  9. Default

    Its all about the games!

    Although I like seein 3dmark and aquamark scores on different cards, they are almost completely useless. There is no one that is going to be using the Exact same system at the exact same ambient temperatures with the exact same amount of thermal paste etc. etc., so, synthetic benchmarks are pretty much useless to me. We all know how GFX cards stack up, ATI for example goes 9600Pro->9600XT->9700Pro->9800->9800Pro->9800XT.
    (I only used card that are retail avaliable)

    Most everyone can recite that from memory and knows that the benchmarks will be higher as you keep going up the list.

    But when it comes to certain games, thats where it all counts. Certain game engines might run better/worse with different settings on certain cards, and at least If im using a barton 2500 and you're using a barton 2500, and same memory/HD speed (very possible to happen) the FPS's will most likely be the same, as other factors wont affect too much. So if im only getting 80 FPS in RTCW on my ti4200, and I see you getting 150 or so with a different card, I can see a real world gain in performance.

    So, basically without rambling, I'm saying if you just stopped using synthetic benchmarks (I'd keep Sandra for CPU/Memory just because Intel vs AMD is kinda confusing) I wouldnt miss it, I can always head over to Tom's or Anand and get all the synthetic benchamrk scores I need, you could get a little niche here by spending all the synthetic bench money on new games.

    Game suggestions:

    RTCW (most widely played Q3 engine)
    NFSU: Underground (Dx9, and VERY well coded engine)
    Jedi Academy (What Persian Said)
    Max Payne 2: (Prop Engine, Great game, Substitute for 3d2001, both use same engine)
    UT2k4 (When it comes out, I'm sure a ton of games will use its engine)
    HL2 and Doom3 of course in the future.

    PlanetSide or some other MMORPG.
    NFL2k4 (just trying to include all genres)
    Halo (Most Graphic Intensive DX9)

    Thats all I can think of now, other than that, make sure you always throw in some other benchs from different products on the same system. I really dont think a review is helpful unless it is at least a little bit of a roundup, as most people who dont own a game will know what the Average FPS is with what products.

    Hope that helps. Peace.
    The FatGuy's Rig:

    <small>AMD 1600+ XP O/C'd to 1560 mhz // Thermaltake Volcano 7+ // Aopen AK77 Pro K266(A) Chipset O/C'd to 147 mhz // 256MB Elixir PC2100 DDR CAS2 // Visiontek GeForce4 Ti4200 O/C'd to 310/580 w/ Thermaltake HSU // SoundBlaster Audigy X-Gamer // WD205BA 20GB 7200 // Sony 52x/24x/52x // Linksys 10/100 NIC // ThermalTake 480w Butterfly PSU // Silver Windowed Aluminum Mid-Tower Case (Modded) // Vantec Nexus Fan Controller// Thermaltake 80mm Blower // 3xEvercool 80mm Fans // 2xBlue Cold Cathodes // 2xSound activated Blue Cold Cathodes // Dell 17" Monitor // Microsoft Trackball // Creative 5.1 Speakers</small>

    Come Crunch with TweakTown

    <img src="http://www.statgfx.com/statgfx/folding/?&username=FatGuy3&border=0,0,0&custom =0,0,255&label=0,0,0&header=255,0,0&st ats=0,0,255&trans=yes&template=fah_origina l&.jpg" alt="www.Statgfx.com" />

  10. #10
    Join Date
    Nov 2001
    Location
    New England Highlands, Australia
    Posts
    21,907

    Default

    How about Science Mark for CPU's and chipsets/mobo's? :?:

Page 1 of 3 123 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •