The Farrellel Universe

I have created this entry to allow us to have a discussion about a MythTV listing replacement.

I had hoped to install MythTV this summer, but no listings could be a deal breaker. I hope we can form some solution to keep this from happening.

I like the idea of a centralized system to parse screen scrapings and serve nice neat XML files to whoever requests them, including MythTV installations. The full HTML download and scrape algorithm would only have to be run once in a while, saving the bandwidth/processing of having everyone downloading the same full HTML file and parsing it locally every time. Additionally, patches to correct scraping errors only have to be applied to a single server.

Here are some random musings about problems we might encounter, and possible solutions:

• bandwidth requirements: datacenters and big fat fiber lines are not cheap, and don't scale too well to completely free web services. I see two ways to mitigate these issues: 1) we can include hashes of each of the XML files, and instruct clients to compare hashes before downloading a new XML file (local caching, basically), which would reduce overall bandwidth usage, and 2) we can have a central server scraping the screen, generating XML files, and then "load balancing" requests out to volunteer servers who keep caches of the XML files and the hashes. If a user of the service has the bandwidth, and would like to contribute, they could install a small app and also become a cache server.
• Litigation: This could be a big issue. I'm not sure what the laws say surrounding the issue of screen scraping, but I imagine the scraping itself falls under fair use just like recording the radio to tape. As for the listings themselves, they may be copyrighted, and the companies providing them (Yahoo, Comcast, whoever) probably won't hesitate to file suit to get us to stop scraping (we'd be affecting ad revenue, after all). I don't really have a solution for this, but it's definitely worth discussion.

I am willing to help out with this project, including hosting, programming, designing, evangelizing, whatever. I'm in the middle of building my own embedded MythTV system right now, and I would love to contribute. This seems like a good way to do that.

BitTorrent.

Scalability - solved.
Litigation - partially avoided.

I am interested in helping out with the project. I have setup a MythTV box in the past and know how important it is to have reliable TV listings. I have no programming skills, but can admin Windows/Linux machines fairly well. I have a dedicated fiber connection and might be able to provide a dedicated server if the situation seems right. Please let me know if I can help.

I'll help out with this in any way. Aero, I figure a torrent server can stay in seed-only mode and do the scraping and shaping. The client can have a change-aware torrent client behind it and a variety of widget front ends to start off with.

This is an idea that I proposed on the MythDev list and believe it's the best way to proceed. Scripts could run from different systems and update a central database. This database would be what all the MythTV users download; specifying what portions they wanted.

The scripts should scan multiple guide sites, removing dependence on any single one, plus suppport manual guide info entry from the community to fill in any gaps and further distance the efforts from perceived IP and copyright problems.

The main concerns about all this were the dependency on a single server for the MythTV users (which COULD be eliminated by using P2P networks to grab files based on date ranges).

I REALLY believe this is the way to go. A community site where every channel in the world, as well as internet channels and pod casts, are all in one site database for browsing and fetching.

Does http://schedulesdirect.org/ negate the need for us to develop this?

Not really, since this is based on someone else's feed and registrations are closed.

I'm a windows developer with in-depth scraping experience, willing to help.

What about myth tv? I am trying to get it installed and hope it has a guide too.