Pagescraper randomly quits?, Anyone experience this?

You should be able to use a text editor to insert this line for each PageScraper meter:

But with so many meters, I don't know what will happen. I guess it's worth a shot though.

Hi uziq,

I did what you told me - added the lines to all the entries. The file became really large, really fast. Anyways, I kept resetting Samurize to keep the file size now. The one I got is 80 or so MB. Here's the bottom page of the file before it quit.

I don't see anything wrong here. Samurize just quit out of nowhere again like usual and at random.

I may have spotted a potential problem. Disable the log file and try this update. If it still crashes, enable the log and post it again as you did before.

This post has been edited by uziq on Jul 5 2009, 06:14 PM

Hi uziq,

I just wanted to say that I just saw your post and am trying out your new plugin now. I made two configs -

stisev_debug.ini
stisev.ini

I went back to stisev.ini and am testing out your updated plugin.

Two questions:

1) May I ask where in the log the problem you found was?
2) Why does Pagescraper take such CPU time? I realize this is a really stupid and ridiculous question, but it takes about 3-5% of CPU non-stop (each second). I guess my thought is that if all of my configs are configured to scrape every 15minutes, why does it take a continuous amount of CPU. It should scrape (take a lot of CPU at once) and then not do anything for a long time, right?

UPDATE: lol, a couple seconds after I posted this, it crashed

I'm back to debug config now.

I guess I'll have to mess with the code a bit more.

1. A few meters called InternetOpen or InternetOpenUrl just before the crash.
2. How do you know it's PageScraper and not Samurize? Once PageScraper has downloaded/scraped/filtered/output data, it enters a wait state and uses 0% CPU until the next refresh (determined in Adv Options).

uziq,
Crashed again. The log is 50MB in size (pretty small IMO). I'm going to upload it now and provide you a link. (see below)

1. Ah, thank you.
2. I think you're referring to the CPU % question I posted earlier (and that you're not asking me why I think it's PageScrapper crashing and not Samurize. I answered both in a & b.
a) When I remove the pagescraper config and leave everything else (time and the system temp stuff), it doesn't take nearly the CPU time -- about 1.5% every few 7 seconds or so! OR when I remove the pagescaper plugin but keep my current config, the CPU thing disappears
b) I'm -fairly- certain it's pagescraper crashing, since I ran multiple tests to confirm what was crashing and found it always crashed when Pagescraper ran, but never did without it.



New question
4)



Here's the end of the code for the NEW Pagescrapper_crash2_log.txt

uziq:
Here's multiple links to the (same) file = PageScraper_Log_crash2_log.txt

LINK: http://rapidshare.com/files/215485582/Page...h2_log.rar.html

LINK TO ENTIRE SAMURIZE FOLDER (virus/trojan free, confirmed) .RARed +passworded (same password as other file).

http://rapidshare.com/files/215489705/Samurize.rar.html

MD5: F279FB6DECD6333C05901E2C8BAEF082

Password has been PMed to you.

.TXT log is 49.1 MB (51,485,735 bytes)
.RAR-ed fle is 2.32 MB (2,433,024 bytes)

Only 2.32MB to download . See your PM box for password.

This post has been edited by stisev on Mar 30 2009, 04:06 PM

uziq,
I was wrong about one thing.

I removed ALL of the plugins, but kept the config the way it is, and Samurize still takes 3-5% CPU. Strange. Very strange. I'm going to tinker with the CPU stuff before reporting back to you, but my most major concern is the crashing

Regarding the CPU taking time, I changed ALL the entries to either scrape every 10 minutes or 100++ minutes and I still get the CPU 3-5%.

The pagescraping log is showing a TON of this. Please see the log for details. The log is flooded with this stuff and I believe this is what is causing the CPU to be used. I don't understand why it is there though. Just as you said, once it's scraped and Pagescraper returns the value, it should just wait like a good little plugin (hehe) for whatever value it's been set at (in this case 10minutes)

This post has been edited by stisev on Mar 30 2009, 05:08 PM

Hi uziq,
Sorry to take up so much of your time, but here's a log of my Samurize config for 9 minutes.

Technically, it should do all it should be doing at the beginning (scraping on load) and then waiting for 10 minutes, correct?

Well, here's my Samurize log for 9 minutes (precisely). As you can see, Pagescraper is doing a bunch of stuff after initial scrape.

After 15:23:27.759, it goes into 600 second (10min) wait mode, BUT Pagescraper keeps sending these "GetWebpage - exit" and "GetWebpage - entry" signals until period 15:33:25.056 where one entry starts before the others and the rest start at 15:33:34.181.

****This is the cause for the 3-6% CPU use that I'm associating with Pagescraper.****

Here is the FULL 10 minute log from beginning of 1 10minute cycle to the end and to the beginning of another.

The file is named "Pagescraper_TIME_TEST_log.txt" (RARed)
There is no password protection on this file.
http://rapidshare.com/files/215518425/Page...ST_log.rar.html

I'm guessing the 'high' CPU is due to so many meters saving to the log file. I'd expect the 'non-debug' config to use less CPU.

I don't see any obvious problem in those log files (except for INTERNETSTATUS_TURNSTRINTOINDEX, which I'll need to work on).

The GetWebpage entry..exit pairs are expected since the refresh rate of your meters is 1000ms. GetWebpage simply returns the most recently scraped data that PageScraper's worker thread produced. This thread downloads/scrapes/etc and then sleeps for the time specified in Adv Options.

uziq,
No, the debug version takes only -slightly- more. The regular non-debug takes about the same amount of CPU which is 3-5% every second it's open. That is what's so baffling for me. I don't understand what pagescraper (note that this CPU taking is NOT there when pagescraper is not enabled, but a curious thing is that it does still happen even if pagescraper config is present, but the plugin is not. :-\

Then Samurize itself must be responsible for the CPU usage.

Don't forget that it will always cost something to update and draw the config to the screen. The more meters there are, or the larger the config, the higher the CPU will be.

Hang on... lol this is bizarre

This post has been edited by stisev on Mar 30 2009, 06:56 PM