Planet Replay

[hdonzis]

In case anyone's interested in working on this MyReplayTV interface, here is the documentation on the protocol. Unfortunately, if there are any questions, without MyReplayTV operating, it would be very difficult to obtain answers..

Anyway, WiRNS uses the getxact protocol for scheduling shows on 4Ks and simply passes through the putxact messages. In order to determine the Replay's show scheduling, it would be necessary to parse th putxact messages. Since MyReplayTV did it, it can't be all that hard...

Henry
_________________
Here's my Poop

[jonwz]

Hi Ian, thanks for joining the discussion. I've also got freesco working on vmware to get my replaytv dial off of my landline (as I'm thinking of going strictly to voip).

I have two 3xxx replaytv's at home and I'd love to get the functionality that you described in your server notes:

"Finally the "MyReplayTV" link shows a listing of all your units. (Obviously, the list will be empty until such time as you've made a successful connection.) Clicking the link for each unit will display a list of the ReplayChannels and recorded programs on that unit (akin to the ReplayGuide)".

I'm guessing it's been years since you migrated to WiRNS, but do you remember if that function was working well at the time?

Maybe a naive question, but did you ever think of running your scripts (and apache and mysql I guess) on top of freesco?

Please don't get me wrong, WiRNS sounds like great work, but I try to keep things as simple as possible. As I already need freesco (and vmware to allow a winmodem to answer the calls through my dual ported ATA) I'd first like to try adding in your code to freesco, so I can get the display multiple replay guide function.
_________________
Own three upgraded 3xxx's -

[ijprest]

Looking back at my scripts (which are mirrored on the darkwoods site), it looks as if I had started to reverse-engineer the MyReplayTV stuff, but I honestly don't remember how well it worked.

If you're interested in looking at the code, the relevant scripts are /localhost/myreplaytv.pl (the web interface), and /rns/cgi-bin/2.0/getxact.pl & putxact.pl (communication with the Replay). Since getxact.pl is blank, it doesn't appear as if I ever got to the point where I was able to do remote scheduling, but it does look like I was able to display a list of recorded shows.

(It's possible that I have a newer version of the scripts than those mirrored on darkwoods, but I lost a lot of stuff in the Great RAID-Array Crash of '06, so it'll take some digging on my part to see if I've still got it.)

...

As for the second issue... I think you'll have a hard time running any of my old stuff on Freesco. It's a single-floppy distribution, so I suspect you'd have a hard time installing Perl & MySQL. Perhaps you should look into other small Linux distrubutions ("Damn Small Linux" comes to mind, though I've never used it).

It might require a few modifications, but you could get away without using MySQL. If I were doing it again, I'd probably substitute SQLite in place of MySQL. Much smaller and easier to administer. Search for DBD::SQLite.

[jonwz]

Thanks for the reply.

FREESCO is very expandable and the author "Lightening" gives, well, lightening support.

It starts out floppy based, but you can quickly move it to hard drive, and then there are all sorts of addin packages - like mysql,perl, apache.... You can see what I mean by looking at one of the mirrors of the add-ins at http://www.freescosoft.com/

Let me know if I need more than the three packages I mentioned.

My biggest concern might be the custom proxy package
needing to be recompiled?

I haven't coded in years, but maybe this will be a good learning opportunity.

If you can think of any more tips other than your instructions at
http://www.darkwoods.com/mirrors/replaytv/websites/rns_prest.htm
Please let me know. I see a sentence "It assumes you've already completed all of the previous steps", and at this point, I'm not sure what those would be....

Thanks again, Jon
_________________
Own three upgraded 3xxx's -

[ijprest]

Oh, cool. Adding the packages to Freesco looks promising.

Freesco doesn't come with a compiler, but a quick check turned up this:
http://freescofaq.hopto.org/faq/11_17_en.html. Following those directions should allow you to recompile micro_proxy.

Unfortunately, the darkwoods archive doesn't appear to have mirrored all my instructions. Fortunately, however, the Wayback machine seems to have captured the whole site. Check out this archive from 2005: http://web.archive.org/web/20050308232401/http://www.prest.ca/RNS/

The important bit will the the "How to set up Transproxy and a modified micro_proxy to log traffic" page. These directions probably won't work out of the box on Freesco, so be prepared to do some fiddling.

[jonwz]

Ian, thanks for the update. I've asked over in the FREESCO forum if one of the experts would compile the micro_proxy.
_________________
Own three upgraded 3xxx's -

[jonwz]

Hi again Ian. Looking through the instruction pages, I see reference to transproxy for tracing the flows to the RNS server. If I'm not going to trace, can that set of instructions be omitted?

The nice folks over at FREESCO compiled micro_proxy but I might be wearing out my welcome there if I go back for another one.

Please let me know - thanks, Jon
_________________
Own three upgraded 3xxx's -

[ijprest]

Transproxy was required to transparently proxy the web traffic from the Replay to the micro_proxy process.

I honestly don't remember why I couldn't have just hooked it up directly, but I'm certainly no Linux/networking guru, so I'd be happy to be proved wrong.

But perhaps there's another (built-in?) way to do transparent proxying in Freesco?

[jonwz]

Ian, let me put my question a different way. It seems you went through a progression of first capturing the flow between replaytv and server, and then replacing the server. At this point, possibly all people like me need to do is install your scripts, populate the mysql database, and update the FREESCO DNS to point the RNS server name locally (to the system running your scripts)?

Maybe no proxy functions are required? FREESCO does allow redefining the port if it needs to be changed from the default of 80 if needed by a replaytv server.

Henry, in our discussions I got the impression that WiRNS doesn't need proxy function, just DNS redirection. I understand your focus has been Replaytv versions 4 and beyond, but maybe the thought process applies here?
_________________
Own three upgraded 3xxx's -

[hdonzis]

[ijprest]

[delver]

This forum has been so useful, but I've hit a wall with WiRNS and my Showstopper.

I am dumping cable for free OTA digital (49 channels in my area). My 5040 remaps and controls my ChannelMaster 7000 converter box beautifully, using the instructions at http://planetreplay.com/phpBB2/viewtopic.php?t=15155.

I also set up my Showstopper to connect via a Freesco box according to directions at http://www.livinitnaturalstate.com/rtv/FreescoRTV/README.TXT . I then tried telling Freesco that my DNS server was my WiRNS box. When I try to grab a WiRNS channel lineup it looks like a successful connect for a while, followed by a "network receive error" and it tells me to try again (infinite loop with retries). If I point Freesco's DNS back to OpenDNS the Replay TV lineups come in okay but I lose the channel remapping I could get with WiRNS. Is there any way to configure WiRNS 2.0 to just proxy my Showstopper's http requests and remap the channels in the process? Or do I have to set up my own proxy server and to do the remapping myself? Would your old scripts work for this purpose (and where can I find them)?

In the meantime, I can use the DirectTV lineup and remove all the channels that are not OTA. If you send only two digits to the CM7000 it tunes to the first subchannel available (07 becomes 07.1). But I would like to have all the subchannels available to my Showstopper like I have remapped for my 5040.

BTW, I called ReplayTV's tech support and they said they had no intention of modifying the OTA listings to accommodate the digital transition. I suggested they add "digital OTA" to the cable or satellite listings and they said something like "can't be done."

[hdonzis]

[delver]

Thanks for the suggestion. I grep'd my logs for references to my Freesco IP and then worked from there.

Here is an excerpt from the wirns.log when ShowStopper was getting data (trying to change to a WiRNS lineup). Note that Freesco caches the DNS addresses once it knows them:

[4/21/2009 14:06:18] [DNS] Returning <WiRNS IP> for production.replaytv.net to <Freesco IP>
[4/21/2009 14:06:19] [DNS] Returning <WiRNS IP> for ntp-production.replaytv.net to <Freesco IP>
[4/21/2009 14:06:19] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 14:06:20] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 14:06:20] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 14:06:20] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 14:06:20] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 14:06:29] [PLUGIN] ZipcodeProvider: Proxying request.
[4/21/2009 14:08:22] Hijacking headend request, because we serve it locally.
[4/21/2009 14:08:25] Hijacking headend request, because we serve it locally.
[4/21/2009 14:08:28] Hijacking headend request, because we serve it locally.

...this is followed by 5040 related stuff, and then what looks like another attempt to get a WiRNS lineup on my ShowStopper at 16:10...
[4/21/2009 15:18:09] Updating ReplayGuide/ToDo information
[4/21/2009 15:18:09] Stopping ToDo Timer
[4/21/2009 15:18:09] Processing ReplayGuide Information.
[4/21/2009 15:18:09] Processing local shows.
[4/21/2009 15:18:09] Processing local shows done.
[4/21/2009 15:18:10] Refreshing Replays.
[4/21/2009 15:18:10] Refreshing RecordingGuide for: <my 5040>(<my 5040 IP>)
[4/21/2009 15:18:14] Parsed 168/168 entries.
[4/21/2009 15:18:14] Refreshing RecordingGuide for: <my WiRNS box>(<WiRNS IP>)
[4/21/2009 15:18:14] Parsed 0/0 entries.
[4/21/2009 15:18:14] Refreshing Replays done.
[4/21/2009 15:18:14] Refreshing ReplayGuide information.
[4/21/2009 15:18:15] Added 186 ReplayGuide shows for <my 5040>
[4/21/2009 15:18:15] Added 0 ReplayGuide shows for <my WiRNS box>
[4/21/2009 15:18:16] Added 0 ReplayGuide shows for all
[4/21/2009 15:18:16] Refreshing ReplayGuide information done.
[4/21/2009 15:18:16] Checking for shows to hide from Poopli.
[4/21/2009 15:18:16] Checking for shows to hide from Poopli done.
[4/21/2009 15:18:16] Updating ReplayGuide flags.
[4/21/2009 15:18:16] Updating ReplayGuide flags done.
[4/21/2009 15:18:18] Checking Manual Recordings.
[4/21/2009 15:18:18] Processing ToDo Information.
[4/21/2009 15:18:18] Building ToDo List for: <my 5040>
[4/21/2009 15:18:37] Added 88 ToDo entries for <my 5040> in 19.24768 seconds.
[4/21/2009 15:18:37] Determining conflicts...
[4/21/2009 15:18:37] Completed conflict determination in 0.4406336 seconds.
[4/21/2009 15:18:37] Done.
[4/21/2009 15:18:37] Configured to refresh ToDo and Replay Guide every 240 minutes.
[4/21/2009 15:18:37] Update of ReplayGuide/ToDo information done
[4/21/2009 16:10:59] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 16:11:00] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 16:11:00] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 16:11:00] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 16:11:00] [NTP] Proxying request to ntp-production.replaytv.net
[4/21/2009 16:11:02] [PLUGIN] ZipcodeProvider: Proxying request.
[4/21/2009 16:12:59] Hijacking headend request, because we serve it locally.
[4/21/2009 16:13:02] Hijacking headend request, because we serve it locally.
[4/21/2009 16:13:04] Hijacking headend request, because we serve it locally.

Excerpt from wirns.debug.log for the same day -- these errors probably occurred when my Showstopper was connecting, but it's interesting that they don't happen every time the Showstopper connects:

[4/21/2009 04:27:25] [PLUGIN] ChannelGuideProvider Exception: Index was outside the bounds of the array. at WiRNS.CGProvider.HandleMessage(String message, Byte[] data, Socket handler) in c:\CVS\WiRNS2\ChannelGuideProvider\CGProvider.cs:line 92
[4/21/2009 16:13:07] [PLUGIN] ChannelGuideProvider Exception: Index was outside the bounds of the array. at WiRNS.CGProvider.HandleMessage(String message, Byte[] data, Socket handler) in c:\CVS\WiRNS2\ChannelGuideProvider\CGProvider.cs:line 92

Any ideas? I looked at the source code for the exception and it's puzzling to me, but I don't know my away around the WiRNS code yet.

[hdonzis]

[delver]

I had WiRNS upgrade itself to the latest version and repeated my experiment. No more exceptions were thrown (debug file is empty) but I'm still getting the same frustrating results.

I pointed Freesco's DNS back to my WiRNS box. Then I cleared the program guide on my ShowStopper twice (to get rid of current and backup versions). Tried changing the zip code to 95503 and only the WiRNS lineups for my own zipcode appear (none for Eugene). No luck getting one of those WiRNS lineups to load, however. Tried changing back to my own California zip code. It's always the same result when I try to get a lineup through WiRNS, no matter the zipcode: It completes "Getting information about channels" and then suddenly disconnects with this message.

Connection Failed.
The Hard Disk Recorder is unable to connect to the ReplayTV Service to get channel guide program information.
Network receive error.
Try connecting again...

One time a few days ago the nightly connect error message said the server sent incomplete data, if that's any help.

Is anyone having success serving guide data to a ShowStopper with WiRNS 2.0? I've read examples of people using an older version to serve guide data from DataDirect to their ShowStopper, but haven't heard of anyone proxying through WiRNS to the ReplayTV server to get a modified lineup via WiRNS.

[hdonzis]

I don't know if you read Ian's post, but it sounds like he has the 2.0 WiRNS working with his ShowStopper...

I'm not sure what you mean by "proxying through WiRNS to the ReplayTV server to get a modified lineup via WiRNS". Since you posted that you have it working with your 5040, I assumed you already had WiRNS configured with a lineup to have it download guide data, not proxy...

However, you say that when you try a different ZIP code you only get the WiRNS lineups. Are you saying that when you tried 95503 you only got lineups which started with "WiRNS:" and none of the normal lineups? Is this the same on your 5040 that you only have "WiRNS:" lineups available to choose from?

Anyway, since Zap2It got rid of their free software service, it has been replaced by SchedulesDirect. If you were to use the SchedulesDirect.org service, then WiRNS should work the same as it did with Zap2It. However, since you have the 5040 as well, then that should allow you to configure WiRNS to use the normal ZIP code guide data instead and serve that to your ShowStopper as well...

Henry
_________________
Here's my Poop

[delver]

It was Ian's post that convinced me to try this. Maybe I should contact him to see how he is doing it.

First, when I have my ShowStopper go through WiRNS and put in an outside zipcode, the only lineups I see are the "WiRNS: " lineups for my own zipcode. That is NOT the behavior the 5040 gets -- it will list the local lineups for the new zipcode and include the WiRNS lineups for my own zipcode.

Maybe you can help me to understand just what WiRNS is doing. There is a checkbox to "proxy" guide data to a networked ReplayTV. My 5040 seems to get the correct lineup info whether or not this box is checked. Since the HTTP requests from the Replay units has been worked out, I assumed that proxying meant WiRNS hitchhiked on the daily updates to populate its own database of channels and shows. Can WiRNS also pretend to be a Replay unit and get the info on its own? -- without intercepting communication with an authorized unit (the 5500 series serial number I see displayed in WiRNS config might be the key to this). I'm puzzled by the Proxy box in Replay configuration and the Proxy column on the WiRNS home page. Can you tell me what this really is about?

There is no way to "register" a ShowStopper with WiRNS 2.0 (I naively put in the serial number the first time I set this up and found that ShowStopper was not in the list of Replay types -- just 4000 and 5000 series machines). It would be nice to add ShowStopper/2000/3000 to the list of Replay types in WiRNS.

This leads me to three questions.

1. Is the protocol for serving data to the Showstopper the same as later models? Could it be the Showstopper is expecting something it isn't getting from WiRNS?
2. Is WiRNS just serving up data from its own database of channels, shows and Replay units? How does it respond to requests from a machine that is not in its own database of Replays?
3. Is there any chance that this is the fault of my Showstopper unit? I upgraded to a 160GB drive (it uses the max 137GB) large cluster option, of course. And this unit is loaded to the hilt with recorded shows, mostly from PBS so the commercial skip feature isn't needed. If someone else is having success, I might try to put in a fresh hard drive to see how it does, if I can still find an IDE drive that will work with the unit. Of course, if this is the problem, why can I still get proper updates by configuring the Freesco DNS normally?

[hdonzis]

[hdonzis]

I posted a reply right when PlanetReplay went down last night, so I'll try again

[delver]

Thanks for going to the trouble to make some changes to WiRNS. I had WiRNS update itself and tried a few things. Here's what I've learned so far:

1. Unchecking "Serve Guide Data to Replays" does not get me a "proxying" log entry. The setup connection still dies after the channel update with a "Network Receive Error." The log entries look the same.

2. I still find that outside zipcodes only list the "WiRNS: " lineups for my own zipcode -- not that I'm bothered by that.

3. It's really true that WiRNS won't serve data to a machine that's not in its database. I came up with an interesting hack. I added my ShowStopper serial number to WiRNS using the Freesco box IP address, and called it a 4040 unit. Then I placed a <serial number>.skp file in the WiRNS directory. The idea was to get the SS into the database but keep WiRNS from pestering it like another networked unit. This hack produced an interesting result -- it made it past the channel list update, got well into the channel guide data, and then quit with an "out of memory" error. That was using an outside zipcode and trying to make it load the WiRNS: OTA lineup for my zipcode.

I've cleared the guide data again and unplugged the machine. I'll try another setup connection with my original zipcode tomorrow or the day after and let you know if that worked. I will probably reenable the "Serve Guide Data to Replays" option, since it is my goal to get modified lineups.

One small problem: the ShowStopper serial number includes a dash (PV-HS000...) and I don't know if the dash is included in the guide server login request, so I had to put both versions of the serial number into WiRNS with two separate "skp" files. Does anyone have the old "proxy.pl" file mentioned in the Twiki that lets you log the exchange between a Replay and the guide data server?

[hdonzis]

Did you try the updated code with "Serve Guide Data to Replays" checked? That was what I was trying to get working for you...

That's very strange that whe you uncheck "Serve Guide Data to Replays" that you don't get the "proxying" log entry. Everything I "see" indicates it should at least log that entry...

The RNS spec that Ian wrote says that the serial number is eighteen characters, if that helps you figure out the file name to create...

I'm thinking at this point that if I had a WireShark of WiRNS "talking" to the ShowStopper and DNNA that I could see more of what is going on, both for the ZIP code problem and the net connect problem. If you could take a WireShark snapshot of changing your ZIP code on your ShowStopper and send that to me, I could at least see if I can figure out where the problem is. In addition if you could take a second WireShark snapshot of your ShowStopper net connecting, then hopefully I can figure out what is going on...

[delver]

Some of the mystery is solved. WireShark is an awesome tool.

I finally decided to do a zipcode change on both my 5040 and my ShowStopper. Then I compared to http exchange that took place with both.

First, I noticed that my 5040 gets all the appropriate channel lineups (headends) plus the WiRNS lineups via WiRNS but the ShowStopper only gets the WiRNS lineups. Looking at the WireShark logs, I see that getzipcode2.pl gets passed along to the ReplayTV server, than passed back to the 5040 with the WiRNS lineups added. But, when WiRNS deals with the ShowStopper, it just passes back the WiRNS lineups and never queries the "mother ship" ReplayTV server. I can't figure out why because the format of the getzipcode2.pl request is identical for both machines.

Second, I noticed that the format of the getcg2.pl request is completely different between the two machines. My 5040 sends a request in this format:
GET /cgi-bin/2.0/getcg2.pl?A20090520C10760Z0_10600Z0_10672Z0_...HTTP/1.1

But the ShowStopper sends the request in this format:
GET /cgi-bin/2.0/getcg2.pl?@20090503:10760,0;10672,0;10641,0;10479,0;...; HTTP/1.1

The 5040 format is documented in the Twiki. The ShowStopper format is documented in a PDF file by Ian on how to set up your own RNS server for a ShowStopper.

It seems that WiRNS is not fluent in the ShowStopper dialect for this request. The poor ShowStopper keeps asking and keeps getting ignored by WiRNS. Eventually it gives up, and I think that's where the "Network Receive Error" comes from.

Here's the big question. Would it be easy to modify WiRNS so it can handle either form of the getcg2.pl request? And can it be modified to handle the getzipcode request consistently, regardless of the machine doing the asking? Or am I better off just modifying a tool like Privoxy to intercept the traffic from my Showstopper and remap the channels that way?

Since quite a few people are using the Freesco technique with their ShowStoppers, it would be nice to add 2xxx/3xxx/SS to the list of Replay types in WiRNS. Then it would know not to query those machines for fancy networked box features but could still handle the guide data requests correctly for them and give us the options of channel remapping, "no phone numbers," "no zones," etc.

[hdonzis]

[delver]

Here's what I've done so far:

-Asked WiRNS to update itself. The ChannelGuideProvider.dll (2.0.3402.388) is dated 4/24/09. That is unchanged from before.
-Renamed my <serialnumber>.skp file to <serialnumber>.rtv.
-Deleted the ShowStopper from the list of Replays in WiRNS.
-Made sure Serve Guide Data is checked (but also tried it unchecked).

Still no joy. The list of headends from getzipcode seems to be correct, then the getheadend requests get proxied up but not passed back down to the ShowStopper and it logs out with a Network Receive error. In fact, the logout consistently displays result=c1000505, if you happen to know what that code means. I tried forcing a netconnect, too, and the getcg2 requests still go unanswered for the ShowStopper.

Did Ian get his 3000 model to work with the newer WiRNS?

[hdonzis]

[delver]

There were channel listings but they were strange. I suspect they were left over from when I had the ShowStopper listed as a Replay in the WiRNS database. I went ahead and cleared the guide data twice, then tried again.

The Global setup only passed back the WiRNS lineups and the logout result code was 0. (I noticed that the mother ship responded to getzipcode with "401 authorization required," and then WiRNS passed back just the WiRNS lineups to the ShowStopper.) Then I chose a WiRNS lineup and it reconnected with a Local setup. This time the following happened:

Showstopper to WiRNS: GET getzipcode2.pl...
WiRNS to RNS: GET / HTTP/1.0
RNS: 401 authorization required
WiRNS to RNS: GET getzipcode2.pl
RNS to WiRNS: <zipcode info>
WiRNS to ShowStopper: <zipcode info + WiRNS lineups>

Then came the individual headend requests:
SS to WiRNS: GET getheadend.pl?CAxxxxx...
WiRNS to RNS: GET getheadend.pl?CAxxxxx...
RNS to WiRNS: <headend info>
--nothing relayed back to the ShowStopper, so it asks two more times for this headend, then moves to the next one, where the pattern repeats--

The strange exception is the antenna headend -- that one WiRNS actually passed back to the ShowStopper!! Of course, WiRNS also replies with the WiRNS headends immediately when asked for them.

We never get to the getcg2 requests because the ShowStopper decides to logout with error code c1000505 at this point.

There are several quirky behaviors here that should help you debug this. Why the "authorization required" response on SetupGlobal? Why the blank GET followed by the correct zipcode request during SetupLocal? Why does the Antenna lineup get passed to the ShowStopper but none of the other headends?

[hdonzis]

[hdonzis]

[delver]

Sorry, I was using a bit of shorthand. The getheadend requests do have headend=CAxxxxx&country=US after the question mark.

Regarding the SetupGlobal transaction, I should have looked closer. It starts out looking similar but then takes a weird turn:

Showstopper to WiRNS: GET getzipcode2.pl...
WiRNS to RNS: GET / HTTP/1.0
RNS: 401 authorization required
WiRNS to RNS: GET getzipcode2.pl
--then instantly (14 milliseconds later), without waiting for a reply from RNS (and I never see it come in later):
WiRNS to ShowStopper: <WiRNS lineups ONLY>
--then Showstopper queries WiRNS for the four WiRNS headends and logs out successfully.