Trying to make this run faster

Hi Everyone:

First, I'd like to say that I'm new here and I hope to find the help I need.


I wrote a c program that reads a GIGANTIC file (~16GB) line by line using fscanf in a while loop as show below

Code: while(fscanf(file, "%d %d %d", &inst_type, &in_delay, &sp_type ) != EOF)
{

// here I do simple processing with the variables read from the file
} I'm supposed to output a result every million lines; the problem that this thing takes A LONG LONG TIME. It's been weeks so far and it's not done (running on a quad-core opteron).

Anyways, if I don't read the file and instead generate those variables randomly, the program runs very quickly (few minutes to finish).


I tried to output something(say result y) every 100 thousand lines and see how fast it's going and I noticed the following.

The first 10 y's came out quickly (they followed one another in an almost fixed interval of a one second or two), but then the y's started to come out slower and slower. It was like becoming exponentially SLOWER.


I don't know what the problem is. Is it the way I'm reading the file? Is there a buffer somewhere that's getting huge and making things dog slow? I really don't know.


Does anyone have any idea about this?


Thanks


P.S.
I also tried to profile (using gprof ) the program, but did not have luck getting it to work. There was all kids of errors that gprof generated and I'm no mood getting busy with those.

Source: C Board

Whats's Hot

	C Programming Language (2nd Edition) Book (Prentice Hall PTR) List Price: $60.33 Price: $43.21 You Save: $17.12 (28%)
	C Programming: A Modern Approach, 2nd Edition Book (W. W. Norton & Company) Price: $91.18
	C Programming for the Absolute Beginner Book (Course Technology PTR) List Price: $29.99 Price: $17.67 You Save: $12.32 (41%) Persuade: NEW ISBN13: 9781598634808 <a title='Condition Guide' href='/content/Condition_and_S hipping_Guide.htm' target='_blank'>Click here to outlook our Condition Guide and Shipping Prices</a>
	Programming in C (3rd Edition) Book (Sams) List Price: $49.99 Price: $31.49 You Save: $18.50 (37%) ISBN13: 9780672326660 Train: NEW Notes: Discredit New from Publisher. No Remainder Mark.
	Absolute Beginner's Guide to C (2nd Edition) Book (Sams) List Price: $34.99 Price: $23.09 You Save: $11.90 (34%)
	C Primer Plus (5th Edition) Book (Sams) List Price: $54.99 Price: $34.64 You Save: $20.35 (37%)
	Practical C Programming, 3rd Edition Book (O'Reilly Media) List Price: $34.95 Price: $23.07 You Save: $11.88 (34%)
	Expert C Programming Book (Prentice Hall PTR) List Price: $42.99 Price: $31.86 You Save: $11.13 (26%)
	C All-in-One Desk Reference for Dummies Book (Wiley Publishing, Inc.) List Price: $34.99 Price: $23.09 You Save: $11.90 (34%) ISBN13: 9780764570698 Circumstances: USED - ACCEPTABLE Notes:
	C Programming - A Beginner's Course Book (CreateSpace) List Price: $24.95 Price: $24.95

C Programming News

It's a Big Yes for THE BIG 'C' PR Newswire (press release) - Jan 08, 2010 Washington PostIt's a Big Yes for THE BIG 'C'PR Newswire (seethe release)It produces and distributes programming in every genre, including series, telefilms, theatric releases and family entertainment around the world and for Laura Linney and "The Big C" go to series for Showtimeall 184 advice articles » C-SPAN Radio app highlighted in Apple iTunes Store Radio Business Report - Jan 08, 2010 C-Extent Radio app highlighted in Apple iTunes StoreRadio Partnership ReportThis includes key Congressional hearings and speeches from subject and world leaders, C-SPAN's day after day Washington Journal call-in program, and other news and and more » More 'Potter' magic? Enter our Media Prognostication Contest USA Today - Jan 06, 2010 More 'Fritter away' magic? Enter our Media Prognostication ContestUSA Today wire programming will be: a) The launch of Hasbro and Discovery's new means for kids; b) The planned launch of the Oprah Winfrey Network in 2011; c) and more » Get the facts: Practice management software Chiropractic Economics - Jan 09, 2010 Get the facts: Repetition management softwareChiropractic EconomicsNET programmers are working with a “lounging robe” that makes programming easier, but adds an additional layer of computer customs… for everything. Writing on the wall points to Apple tablet, but Jobs must prove worth buying TMCnet - Jan 09, 2010 Book on the wall points to Apple tablet, but Jobs must affirm worth buyingTMCnetStill others position it as a companion sift to use while watching television _ the tablet would deliver bumf related to the program airing on the and more »

Ultimate C/C++ Programming - Tutorial 1

***Download Links Here*** + Visual Studio***: ___________________ _____________________ READ THIS: All Rude Comments Will Be Blocked and ...