Planet Replay Forum Index Planet Replay
The Destination for ReplayTV Owners and TV Enthusiasts
Back to the home page
 
 FAQFAQ   SearchSearch   7 days of topics7 Days  30 days of topics30 Days  MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

replaytv guide won't refresh while 100% CPU usage [w/PATCH]

 
Post new topic   Reply to topic    Planet Replay Forum Index -> IVSmagic
View previous topic :: View next topic  
Author Message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Sun Jan 27, 2008 4:43 pm    Post subject: replaytv guide won't refresh while 100% CPU usage [w/PATCH] Reply with quote

I just noticed that one of my ReplayTV's hasn't had it's guide updated since Jan 6. The other one, and the local guide, are up-to-date. When I click 'refresh' to manually refresh it, I see a 'php-cgi.exe' process for the request, but then it starts using 100% of the CPU and after a 2-3 minutes, it goes away - but the guide is not updated. Refreshing the other replay takes only 10 seconds or so.

I've tried deleting the replay and re-adding it, but I'm still having this problem. I tried optimizing and rebuilding the database, and neither helped.

There's nothing interesting in any of the logs. I can see the 'replay guide update of *jlv* succeeded' in the status log for my manual refreshes of the working replay, but no message at all for the refreshes of the replay it isn't updating. The last successful update message was in the scheduler log -- for Jan 6. There's nothing in the error log.

I am able to download the guide from this ReplayTV just fine from DVA and the Poopli Updater -- it's just IVSm that is having the problem.

Any ideas?

Quote:
UPDATE: Feb-24-2009:
The patch for this is included in a post below.
Since it's not available elsewhere, I've made a fixed version of refresh_replay_guide.php available for download.
Save the downloaded file in the include directory of IVSmagic, which defaults to C:\Program Files\IVSmagic\IVSm\include



Last edited by jlv on Tue Feb 24, 2009 10:08 am; edited 3 times in total
Back to top
View user's profile Send private message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Sun Jan 27, 2008 5:29 pm    Post subject: Reply with quote

I did just notice this in the Apache error log --

[Sun Jan 27 20:12:10 2008] [error] [client 10.0.0.2] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed

That's just basically saying that Apache aborted the request because it took too long. The IVSm-httpd.conf has the stock Apache timeout of 300 secs. The first of these in the log is dated Jan 6, of course.

I did finish receiving a show on that unit around the 5th, and the scheduler logs indicate the last successful guide update was at 9:07am on the 6th, so I don't think that show could have (somehow) been the cause.
Back to top
View user's profile Send private message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Sun Jan 27, 2008 7:36 pm    Post subject: Reply with quote

I take that back -- I had a received show complete inbetween those two updates. And when I look at the guide data IVSm has inserted into the database, it ends just before that show.

Looking at the guide snapshot with the command line tools shows me that the show header has bogus-looking values for actors, guests, and directors. With that in hand, I started debugging refresh_replay_guide.php and found that it is going into an infinite loop at line 293, which is where it is trying to look for the title+episode+description. Changing that line from:

for ($_offset=148; $_offset < ($_offset + ($_title_length +1) + ($_episode_length +1) + ($_description_length +1)); $_offset++)

to:

for ($_offset=148; $_offset < ($_offset + ($_title_length +1) + ($_episode_length +1) + ($_description_length +1)) && $_offset < 512; $_offset++)

eliminates the infinite loop, by bounding the offset to the size of the show structure. This at least means that my replay is now getting it's guide updated again!
Back to top
View user's profile Send private message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Sun Jan 27, 2008 9:26 pm    Post subject: A better fix Reply with quote

After debugging this further, I see that this loop is there because someone was trying to work around a bug in splitting apart the description block of a FixedProgramRecord. The description block has two optional headers: one of 4 bytes if the has_parts bit (0x40) is set in the ProgramFlag, and one of 8 bytes if the is_movie bit (0x20) is set in the ProgramFlag.

The purpose of this 'for' that was becoming an infinite loop for me looks to try to guess the offset into the description block. I didn't understand why that was there, when the two bits of the ProgramFlag were already split out. However, in debugging this, I found one show whose ProgramFlag was 0x80000020. is_movie should be true, and has_parts should be false. However, the $part_flag variable set at line 280 was TRUE. I found that every time the high bit was set in ProgramFlag, the values of these two variables would be wrong.

I suspect that there is some weirdness with $_program_flags having the high-bit set -- perhaps signed/unsigned. I know that when I included this debug code:
Code:

debug("program_flags = ".  ascii2hex(substr($_show_data, 120, 4)).
          " = ".$_program_flags);
debug("movie  ". ($movie_flag==0x20 ? "Y":"N") .
           " =". ($_program_flags&0x20));
debug("parts  ". ($parts_flag==0x40 ? "Y":"N") .
           " =". ($_program_flags&0x40));

it would show this:
Code:
program_flags = 80000020 = 2147483680
movie Y =32
parts Y =64
parts should NOT be true.

So, it appears that PHP is doing the wrong thing for logical operations on (unsigned int) numbers with the high-bit set.

Since this code doesn't use the bits in the high byte of ProgramFlag, I changed it to only extract the lower 3 bytes of it. And since the bit tests now work correctly, I removed the broken loop and replaced it with direct adjustments on the offset into the description block. This is my final patch:

Code:
--- refresh_replay_guide.php.orig   2007-05-12 09:25:40.650497600 -0400
+++ refresh_replay_guide.php   2008-01-28 00:02:34.698753600 -0500
@@ -224,7 +224,14 @@
     // debug("EVT: " . $_evt_size);
     // debug("MPG: " . $_mpg_size);
     
-    $_program_flags=ascii48bit2dec(substr($_show_data, 120, 4));
+    //$_program_flags=ascii48bit2dec(substr($_show_data, 120, 4));
+    // When the 0x80000000 bit is set, then bit tests like these fail!
+    //       ($_program_flags & 0x00000020);
+    //       ($_program_flags & 0x00000040);
+    // I.e., one show had the value 0x80000020 -- yet both of the above
+    // returned true!!   Possible signed integer problem?
+    // Since this logic doesn't use the flag bits of high byte, just skip them
+    $_program_flags=ascii48bit2dec(substr($_show_data, 121, 3));
     $_event_time=ascii48bit2dec(substr($_show_data, 124, 4));   
   
     if (!$_event_time) $_event_time=$_show_id;
@@ -279,6 +286,9 @@
     $movie_flag=$_program_flags & 0x00000020;
     $parts_flag=$_program_flags & 0x00000040;
     $ppv_flag=$_program_flags & 0x00000080;
+
+    if ($parts_flag) $_offset += 4;
+    if ($movie_flag) $_offset += 8;
           
     if (!$movie_flag)
     {
@@ -290,12 +300,6 @@
       else if ($_program_flags & 0x00010000) $tv_ratings="TV-Y7";
     }       
       
-    for ($_offset=148; $_offset < ($_offset + ($_title_length +1) + ($_episode_length +1) + ($_description_length +1)); $_offset++)
-      if ((substr($_show_data, ($_offset + $_title_length), 1) == chr(0))
-      and (substr($_show_data, ($_offset + ($_title_length +1) + $_episode_length), 1) == chr(0))
-      and (substr($_show_data, ($_offset + ($_title_length +1) + ($_episode_length +1) + $_description_length), 1) == chr(0)))
-        break;
-       
     // debug(str_repeat("-", 50));
     // debug($_offset);
     
Back to top
View user's profile Send private message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Sun Jan 27, 2008 9:27 pm    Post subject: Reply with quote

And I'm glad IVSm is open source now. Whew. Good night!
Back to top
View user's profile Send private message
hdonzis
Planet Overlord


Joined: 05 Jan 2005
Posts: 9021
Location: San Antonio, TX

PostPosted: Sun Jan 27, 2008 10:39 pm    Post subject: Re: A better fix Reply with quote

jlv wrote:
However, in debugging this, I found one show whose ProgramFlag was 0x80000020. is_movie should be true, and has_parts should be false. However, the $part_flag variable set at line 280 was TRUE. I found that every time the high bit was set in ProgramFlag, the values of these two variables would be wrong.

I suspect that there is some weirdness with $_program_flags having the high-bit set -- perhaps signed/unsigned. I know that when I included this debug code:
Code:

debug("program_flags = ".  ascii2hex(substr($_show_data, 120, 4)).
          " = ".$_program_flags);
debug("movie  ". ($movie_flag==0x20 ? "Y":"N") .
           " =". ($_program_flags&0x20));
debug("parts  ". ($parts_flag==0x40 ? "Y":"N") .
           " =". ($_program_flags&0x40));

it would show this:
Code:
program_flags = 80000020 = 2147483680
movie Y =32
parts Y =64
parts should NOT be true.

So, it appears that PHP is doing the wrong thing for logical operations on (unsigned int) numbers with the high-bit set.


It's probably using signed integers for its logical operations. Which means that it two's complements the signed integer, 0x80000020, to 0x7FFFFFE0, which then has both the parts bit and movie bit on. While this isn't very common, it makes things like anding -1 with 1 come out correctly. Basically it performs logical operations on the absolute value of the signed integer so as not to take the signed value into account (that logical operations come out the same whether the interger is positive or negative)...

Good thing that IVSmagic doesn't care about the movie subratings!

What I find very eerie about your findings is that extract_rtv, which is very old, had very similar code in it (in C) for skipping over the movie info and parts info:

Code:
               /* Parse the description info out of this file */
                  while((offset<length) && (*(int *)(buffer+offset)==0))
                     offset+=4;


Which I corrected in a similar way that you did well over a year ago (Aug 2006):

Code:
                        if(flags & has_parts)
                           offset+=sizeof parts;
                        if(flags & is_movie)
                           offset+=sizeof movie;


Henry
_________________
Here's my Poop (I know that's the old Poopli, but I like it that way!)
Back to top
View user's profile Send private message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Mon Jan 28, 2008 1:50 pm    Post subject: ascii48bit2dec() is the fault Reply with quote

I wondered if PHP was messing up, so I tried this:
Code:
$php -r '$x = 0x80000020; print $x."\n".($x & 0x20). "\n" .($x & 0x40). "\n";'
2147483680
32
0
That obviously works... so I figured it must be how it parses the int with ascii48bit2dec(). I extract that function from common.php and added this:
Code:
$x = ascii48bit2dec("\200\000\000\040");
print $x."\n";
print ($x&0x20)."\n";
print ($x&0x40)."\n";
$x = $x + 0;
print $x."\n";
print ($x&0x20)."\n";
print ($x&0x40)."\n";
and when I run that it becomes clear:
Code:
$ php Test.php
2147483680
32
64                          BAD
2147483680
32
0

ascii48bit2dec() returns the result of bcadd(), which is a string. When that string is used in the logical operation, it's being converted to a signed integer -- but it's value is capped at MAX_INT (2^31-1).
Code:
$x = '2147483680';
print $x." => ".($x&0xffffffff)."\n";
$x = 2147483680;
print $x." => ".($x&0xffffffff)."\n";

gets

2147483680 => 2147483647                 BAD
2147483680 => -2147483616


A fix for ascii48bit2dec() ends up being to coerce it's return value into an integer directly. This seems to work:
Code:
--- common.php  Mon Jan 28 15:45:10 2008
+++ common.php.fix      Mon Jan 28 16:20:26 2008
@@ -2466,3 +2466,3 @@
 
-  return $addant;
+  return $addant + 0;
 }
and that even works when the function is parsing an 8-byte value (like the size of the mpeg file).
Back to top
View user's profile Send private message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Mon Jan 28, 2008 3:04 pm    Post subject: Reply with quote

After really looking at what ascii48bit2dec() does, and how it's used, I'd consider replacing it:
Code:
function ascii48bit2dec($bytestring)
{
  foreach (unpack("C*", $bytestring) as $byte) {
    $result = bcadd( bcmul($result, 256), $byte);
  }
  return $result + 0;           // for numbers > 2^31, this will return a float
}
Back to top
View user's profile Send private message
hdonzis
Planet Overlord


Joined: 05 Jan 2005
Posts: 9021
Location: San Antonio, TX

PostPosted: Mon Jan 28, 2008 3:04 pm    Post subject: Reply with quote

Well, without your printing logically anding the string value it wouldn't have been clear that it was turning that into a positive all ones value (such at all bits would have been true and you would have seen a whole bunch more things in IVSmagic happen). When you print logically anding the numeric value, it shows as I expect as a negative of the two-complement value...

That'll teach someone to use string as numerics instead of integers! Maybe a string value is defaulted fitting in a 32 bit value (like atoi()) such that it is only allowed to be 2147483647 to -2147483648 on the logical operations. But, on the arithmetic operation, it allowed it to fit into a larger size integer...

That's good that you were able to figure out the solution without losing any bits of information!

Henry
_________________
Here's my Poop (I know that's the old Poopli, but I like it that way!)
Back to top
View user's profile Send private message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Mon Jan 28, 2008 3:08 pm    Post subject: Reply with quote

YIKES!

There's another copy of the infinite loop inside common.php at line 355.

And that code implements the class 'replay_5000_guide_parser', which is a duplicate of the show parsing code from refresh_replay_guide.php.php. I wonder why that class isn't being used here?
Back to top
View user's profile Send private message
jlv
Replay fan
Replay fan


Joined: 27 Jan 2008
Posts: 51
Location: MA

PostPosted: Tue Feb 24, 2009 9:27 am    Post subject: Egads... Reply with quote

I just came here to the forums to see if anyone else was having the problem I'm seeing -- that guide for one of my Replay's won't refresh, and that IVSmagic uses 100% CPU while trying to refresh it.

I'm glad to see someone not only had the problem, but also posted the fix.

It's sad that it was me, and that (1) I'd forgotten about doing this and (2) I'd lost the fix.

Sometime back in December when I was installing WiRNS, I did an 'update' on IVSmagic. I don't know why I did. What the update managed to do was replace the fixed code with the release (buggy) version. At the time, I only thought it deleted piles of debugging code I'd put in various places, so I didn't think twice of it. I didn't remember fixing any code.*

I just checked IVSMagic out of the CVS repository and will get this fix checked in, so this doesn't happen to me (or anyone else) again.

(Ah, I gotta get write access...)


* (I have a memory-loss problem that impacts the establishment of medium- and long-term memories. Needless to say, I've had this problem for as long as I can remember...
Seriously, it causes me no end of frustration, because if I don't work on something for 2-3 days, I totally forget about it.)
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Planet Replay Forum Index -> IVSmagic All times are GMT - 8 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Planet Replay topic RSS feed 


Powered by phpBB © 2001, 2005 phpBB Group