Monday, 1 September 2014

More remote control decoding

Astro have now retired their old decoders, which I spent some time a few years ago trying to control from my PC. It seems at least someone has found a use for that info, but from reading the thread, what they actually wanted was the remote codes for the latest generation Samsung decoders that Astro replaced all the old decoders with.

In the public interest then, here is an updated LIRC config, and additionally a kernel rc configuration file for the Samsung Astro Byond decoders. Be warned: the kernel rc configuration is using standard key codes, including KEY_POWER, which may cause your PC to shutdown in many default configurations of GNU/Linux desktops - where all available input devices are grabbed by X. Changing to KEY_POWER2 might help with that particular problem - but unless you configure X to ignore the remote control, you'll still see side effects from other keys.

1. The kernel configuration file (/etc/rc_keymaps/AstroByond).

#table AstroByond, type: RC6
0x80562700 KEY_0
0x80562701 KEY_1
0x80562702 KEY_2
0x80562703 KEY_3
0x80562704 KEY_4
0x80562705 KEY_5
0x80562706 KEY_6
0x80562707 KEY_7
0x80562708 KEY_8
0x80562709 KEY_9
0x8056270C KEY_POWER
0x8056270F KEY_INFO
0x80562720 KEY_CHANNELUP
0x80562721 KEY_CHANNELDOWN
0x8056274B KEY_SUBTITLE
0x80562754 KEY_HOME
0x80562758 KEY_UP
0x80562759 KEY_DOWN
0x8056275A KEY_LEFT
0x8056275B KEY_RIGHT
0x8056275C KEY_OK
0x8056276D KEY_RED
0x8056276E KEY_GREEN
0x8056276F KEY_YELLOW
0x80562770 KEY_BLUE
0x80562783 KEY_SHOP
0x80562784 KEY_FAVORITES
0x805627A9 KEY_BACK
0x805627CC KEY_TV
0x805627D8 KEY_SCREEN
0x805627F2 KEY_RADIO
0x805627F5 KEY_BUTTONCONFIG
0x805627FB KEY_DOT

2. A traditional lirc configuration file (/usr/local/share/lirc/remotes/AstroByond.conf). Note that the codes seem to be 2s complements of the values above, with an additional prefix. I'm not sure whether this is due to a difference in the way LIRC interprets codes compared with the kernel drivers, or if it is a result of misdetection of other parameters. LIRC's auto-detection is a bit off anyway, as it failed to detect the toggle_bit_mask, and I had to first detect as raw codes, then analyze the resulting file. I think spotted a pattern of alternation between 85xx and 8Dxx for the lower 16 bits, so figured that the toggle_bit_mask was 0x8000. After setting that manually, the remote became very responsive in irw. I did similar manual patching of bit 15 in the ir-keytable results above, but it doesn't seem to make a difference there, just looks cleaner if the keys all have the same prefix.

#
# this config file was automatically generated
# using lirc-0.9.0(emulation) on Mon Sep  1 22:10:16 2014
#
# contributed by Jason Rumney
#
# brand:                       Astro Byond
# model no. of remote control: URC931000-01R00
# devices being controlled by this remote: Samsung Astro Byond Decoder
#

begin remote

  name  ASTROBYOND
  bits           37
  flags RC6|CONST_LENGTH
  eps            30
  aeps          100

  header       2709   757
  one           476   394
  zero          476   394
  gap          105172
  toggle_bit_mask 0x0000008000
  rc6_mask    0x100000000

      begin codes
          KEY_POWER                0x037FA958F3
          KEY_CHANNELUP            0x037FA958DF
          KEY_CHANNELDOWN          0x037FA958DE
          KEY_0                    0x037FA958FF
          KEY_1                    0x037FA958FE
          KEY_2                    0x037FA958FD
          KEY_3                    0x037FA958FC
          KEY_4                    0x037FA958FB
          KEY_5                    0x037FA958FA
          KEY_6                    0x037FA958F9
          KEY_7                    0x037FA958F8
          KEY_8                    0x037FA958F7
          KEY_9                    0x037FA958F6
          KEY_DOT                  0x037FA95804
          KEY_OK                   0x037FA958A3
          KEY_UP                   0x037FA958A7
          KEY_DOWN                 0x037FA958A6
          KEY_LEFT                 0x037FA958A5
          KEY_RIGHT                0x037FA958A4
          KEY_TV                   0x037FA95833
          KEY_RADIO                0x037FA9580D
          KEY_HOME                 0x037FA958AB
          KEY_BACK                 0x037FA95856
          KEY_RED                  0x037FA95892
          KEY_GREEN                0x037FA95891
          KEY_YELLOW               0x037FA95890
          KEY_BLUE                 0x037FA9588F
          KEY_INFO                 0x037FA958F0
          KEY_FAVORITES            0x037FA9587B
          KEY_SUBTITLE             0x037FA958B4
          KEY_SHOP                 0x037FA9587C
          KEY_SCREEN               0x037FA95827
          KEY_BUTTONCONFIG         0x037FA9580A
    end codes

end remote

Sunday, 24 August 2014

Android Studio - a frustrating experience in usability

Lately Eclipse has become excessively slow and unstable. So I decided its time to take a look at Android Studio.

I downloaded, and installed with the default settings - mistake number one, as I found out later. Startup took forever, just like Eclipse. I created a new project, choosing the Settings Template, as I know I'm going to need a Settings page for this app, which also took forever. When it eventually finished, there was an error about gradle failing to sync...so basic operations like editing and compiling will not work. WTF! So about half an hour of poking around and searching the net later, I find that my proxy got set when I installed, but today I'm working from home. Apparently some of the other slowness is attributable to that too, as it seems much more responsive when I have disabled the proxy.

Advice #1: When designing software, use the system proxy settings, don't introduce your own. People move between networks, and they shouldn't have to reconfigure a dozen separate places when they do.

So now I have a project, and gradle seems to be happy for now. I'm going to be using websockets, so I add in autobahn, and its dependencies jackson-core and jackson-mapper. But before adding any code that actually uses them, maybe I should make sure the basic boilerplate that was generated compiles and runs.

Compiling goes OK. So now to run on my emulator. I am targeting a specific device here, so I don't really want to run it on the Android-L emulator that was installed by default, so time for some more downloading. Start SDK manager from the Tools menu. A dialog comes up, shows a progress bar goes across the screen showing me that SDK manager is taking a long time to start, 10 seconds later it disappears. Nothing happens. Start checking the paths in config - is there something missing. 5 minutes later, the SDK manager appears on the screen. Check RAM and CPU usage for an explanation - 5GB free, nothing taking more than 4% CPU. Kill SDK Manager, and check again while it is starting. The same. So its not a CPU or memory problem. 5 minutes, maybe a network timeout. Maybe its getting its proxy settings from yet another place that I haven't yet discovered and is waiting for the timeout.

Advice #2: Pop up a window, or at least a splash screen as soon as you can, to let users know that your program is starting. And don't do network stuff on the main thread. Preferably put a message somewhere saying what you are doing when something might take a long time, so users know what to troubleshoot.

Once SDK Manager has started, I select the SDK I need to download, and start the process. It fails. Why? Because it does not have permission to write temporary files. Since Vista, Windows has had this policy that programs have no business writing into the Program Files directory.  This stops viruses and trojans from overwriting perfectly good programs. It's a good thing, and Linux and Solaris have similar policies.  Temporary files go in the system TEMP directory (/tmp, /var/tmp, %USERPROFILE%\Temporary Files, whatever). Files that need to be updated by a program go into a directory for that purpose (/var/cache/..., %USERPROFILE%\Application Data\..., whereever).  This isn't an unusual or unreasonable requirement.  But applications like Android Studio, and Eclipse, choose to ignore it, which causes headaches to users on all platforms.

Advice #3: Follow platform conventions about where to put your files. It's understandable in 2005, when Vista has just come out, that some software has not had time to mend the error of its ways. In 2014, it is inexcusable, especially for cross platform software that also runs on platforms that have had the convention of separating programs and data since at least the 1980's.

To resolve this, I managed to set the SDK to point to my old copy from the Eclipse days. I think I'd already gone through this problem in the early days with Eclipse, so I had an SDK already in a separate directory. The latest SDK in there is Kit Kat, not Android L, but the bonus was that it already had the even older SDK I need for my emulator. A bit of patching of project files to set the compiler and target to android-19 instead of android-L, and I'm back at the point I was before with the older SDK installation. I could have just updated the SDK now, but I don't fancy waiting around for the 500MB download to complete.

So now to launch the emulator. The custom device I created long ago is now visible, but it isn't working. Eventually I ended up deleting it and recreating. It still didn't work. Unknown to me, the fact that I've installed the Intel HAXM acceleration through SDK manager has made the Intel target visible (which I chose, as it is supposed to be faster, and I don't really care about ARM vs Intel differences for this Java only project), but the HAXM accelerator is not actually installed, only downloaded by the SDK manager. Some messing around at the command-line and a web search later, I find this info, and manually install the HAXM accelerator. But this isn't the only thing stopping my emulator from starting. Another web search, and I find that Hardware Graphics acceleration is only available in Android 4.3 and later, something that wasn't at all obvious when I checked the box. In both cases, there was no obvious error message. The emulator just failed to start, and only by going to the command line and trying the command where I could see the output was I able to diagnose the problem.

Advice #4: When you install things, install them properly. Or at least give a prominent warning to the user when you find it is not properly installed, telling them what manual steps they need to do. And if you have options that only work in certain situations, say so, right there next to the checkbox, so users don't waste their time selecting options that won't work. When users don't heed your warnings, don't just fail silently, tell them what went wrong so they can quickly fix it.

What should have been a five minute session getting the initial boilerplate for my program set up and working turned out to be a four hour mission. I'm tired, so I'm putting it away for now. When I come back, I'll probably be doing most of the work in Emacs to avoid the hassle. Development tools should not be like this. When you have a download that tops 2GB when you add up all the parts, at least some of that code bloat should be going into usability, especially for that first setup experience. I've never used IDEA before, and to be fair, most of the problems seem to be with the Android specific portions, but still, I don't think I'd consider paying for the commercial version IDEA based on my experience today. It may not be fair, but that is how association with a mess like this affects peoples' perceptions. And Android Studio is still a beta product you say? It's been in beta for over a year now. It's had plenty of time to improve, and like every Google product by the time it comes out of beta, it either won't be relevant any more, or you'll have missed the boat if you aren't already using it.

Thursday, 11 April 2013

Astro xmltv listings

For quite a while now, I've been pulling TV listings from the Astro website. At first this involved scraping some HTML and the scripts I had were pretty ugly and worked only for the channels I wanted. Then they added some RSS feeds, which made the scraping a bit easier and less prone to breaking every time they updated the look of the website (which was often). For a couple of years now, the RSS feed has been replaced with a JSON API, and I had a pretty good grabber written for that, but never got around to sharing it. Recently they switched to a dedicated server for the JSON API, and gave it an overhaul, so my script stopped working. Now that I've updated it to the new API, its a good time to share.

#!/usr/bin/perl -w

eval 'exec /usr/bin/perl -w -S $0 ${1+"$@"}'
    if 0; # not running under some shell

=pod

=head1 NAME

tv_grab_my_astro - Grab TV listings for Malaysia using Astro JSON feeds.


=head1 SYNOPSIS

tv_grab_my_astro --help

tv_grab_my_astro --configure [--config-file FILE] [--gui OPTION]

tv_grab_my_astro [--config-file FILE] 
           [--days N] [--offset N]
           [--output FILE] [--quiet] [--debug]

tv_grab_my_astro --list-channels [--config-file FILE]
           [--output FILE] [--quiet] [--debug]
                 
                
=head1 DESCRIPTION

Output TV and listings in XMLTV format for channels available on
Malaysia's Astro subscription satellite service.

First you must run B<tv_grab_my_astro --configure> to choose which stations
you want to receive.

Then running B<tv_grab_my_astro> with no arguments will get a listings for
the stations you chose for five days including today.

=head1 OPTIONS

B<--configure> Prompt for which stations to download and write the
configuration file.

B<--config-file FILE> Set the name of the configuration file, the
default is B<~/.xmltv/tv_grab_my_astro.conf>.  This is the file written by
B<--configure> and read when grabbing.

B<--gui OPTION> Use this option to enable a graphical interface to be used.
OPTION may be 'Tk', or left blank for the best available choice.
Additional allowed values of OPTION are 'Term' for normal terminal output
(default) and 'TermNoProgressBar' to disable the use of Term::ProgressBar.

B<--output FILE> When grabbing, write output to FILE rather than
standard output.

B<--days N> When grabbing, grab N days rather than 5.

B<--offset N> Start grabbing at today + N days.  N may be negative.

B<--quiet> Suppress the progress-bar normally shown on standard error.

B<--debug> Provide more information on progress to stderr to help in
debugging.

B<--list-channels>    Output a list of all channels that data is available
                      for. The list is in xmltv-format.

B<--version> Show the version of the grabber.

B<--help> Print a help message and exit.

=head1 ERROR HANDLING

If the grabber fails to download data for some channel on a specific day, 
it will print an errormessage to STDERR and then continue with the other
channels and days. The grabber will exit with a status code of 1 to indicate 
that the data is incomplete. 

=head1 ENVIRONMENT VARIABLES

The environment variable HOME can be set to change where configuration
files are stored. All configuration is stored in $HOME/.xmltv/. On Windows,
it might be necessary to set HOME to a path without spaces in it.

=head1 SUPPORTED CHANNELS

For information on supported channels, see http://www.astro.com.my/

=head1 AUTHOR

Jason Rumney, jasonr -at- gnu -dot- org. This documentation and some code
copied from tv_grab_se_swedb by Mattias Holmlund, mattias -at-
holmlund -dot- se, which was in turn partially copied from tv_grab_uk
by Ed Avis, ed -at- membled -dot- com. Overall code based on
documentation at http://wiki.xmltv.org/index.php/HowtoWriteAGrabber

=head1 BUGS

=cut
use strict;
use XMLTV;
use XMLTV::ProgressBar;
use XMLTV::Options qw/ParseOptions/;
use XMLTV::Configure::Writer;

use JSON;
use DateTime;
use Date::Parse;
use Date::Format;
use File::Path;
use File::Basename;
use LWP::Simple qw($ua get);
sub t;

$ua->agent("xmltv/$XMLTV::VERSION");

use HTTP::Cache::Transparent;

# Although we use HTTP::Cache::Transparent, this undocumented --cache
# option for debugging is still useful, since it will always use a
# cached copy of a page, without contacting the server at all.
#
use XMLTV::Memoize; XMLTV::Memoize::check_argv('get');

my $default_root_url = 'http://api-epg.astro.com.my/api';
my $default_cachedir = get_default_cachedir();

my ($opt, $conf) = ParseOptions( {
   grabber_name => "tv_grab_my_astro",
   capabilities => [qw/baseline manualconfig apiconfig/],
   stage_sub => \&config_stage,
   listchannels_sub => \&list_channels,
   version =*> "0.1",
   description => "Malaysia (www.astro.com.my)",
} );

#check config

if (not defined( $conf->{cachedir} )) {
    print STDERR "No cachedir defined in config file " .
	$opt->{'config-file'} . "\n" .
	"Please run the grabber with --configure.\n";
    exit 1;
}

if (not defined ( $conf->{'root-url'} )) {
    print STDERR "No root-url defined in config file " .
	$opt->{'config-file'} . "\n" .
	"Please run the grabber with --configure.\n";	
}

if (not defined ( $conf->{'channel'} )) {
    print STDERR "No channels selected in config file " .
	$opt->{'config-file'} . "\n" .
	"Please run the grabber with --configure.\n";
    exit 1;
}

# Astro webserver is slow to generate responses, with lots of little
# JSON requests, so initialise the caching to use a 12hr timeout.
init_cachedir( $conf->{cachedir}->[0] );
HTTP::Cache::Transparent::init( {
    BasePath => $conf->{cachedir}->[0],
    NoUpdate => 12*3600,
    Verbose => $opt->{debug},
    ApproveContent => sub { return $_[0]->is_success },
} );

# Get the actual data and print it to stdout.

my ($xmldecl, $channels) = load_channels( $conf->{'root-url'}->[0] );

my ($odoc, $root );
my $warnings = 0;

binmode STDOUT, ":utf8";

write_header( $xmldecl );

write_channel_list( $conf->{channel} );

my $now = DateTime->today();
t $now;
my $date = $now;
$date = $now->add_duration($DateTime::Duration->new( days => $opt->{offset}))
    if ( $opt->{offset} );

my $bar = undef;
$bar = new XMLTV::ProgressBar( {
    name => 'downloading listings',
    count => $opt->{days} * @{$conf->{channel}},
} ) if (not $opt->{quiet}) && (not $opt->{debug});

for ( my $i=0; $i < $opt->{days}; $i++ )
{
    t "Date: $date";
    foreach my $channel_id (@{$conf->{channel}})
    {
	# We already warned the user if the channel doesn't exist.
	if ( exists $channels->{$channel_id} )
	{
	    t "  $channel_id";
	    my ( $ch_name, $ch_code, $url, $ch_num, $ch_cat, $ch_astroid ) = @{$channels->{$channel_id}};
	    print_data( $url, $channel_id, $date, $ch_astroid, $ch_cat )
		or warning( "Failed to download data for $channel_id on " .
			    UnixDate( $date, "%Y-%m-%d" ) . "." );
	}
	$bar->update() if defined( $bar );
    }
    $date = $date->add_duration(DateTime::Duration->new(days => 1));
}

$bar->finish() if defined( $bar );

write_footer();

# Signal that something went wrong if there are warnings.
exit(1) if $warnings;

t "Exiting without warnings.";
exit(0);

sub t
{
    my( $message ) = @_;
    print STDERR $message . "\n" if $opt->{debug};
}

sub warning
{
    my( $message ) = @_;
    print STDERR $message . "\n";
    $warnings++;
}

sub list_channels
{
    my ( $conf, $opt ) = @_;

    ( $xmldecl, $channels ) = load_channels( $conf->{'root-url'}->[0] );

    my $result="";
    my $fh = new IO::Scalar \$result;
    my $oldfh = select( $fh );

    write_header( $xmldecl );
    write_channel_list ( [sort keys %{$channels}] );
    write_footer();
    select( $oldfh );
    $fh->close();

    return $result;
}

sub config_stage
{
    my( $stage, $conf ) = @_;

    die "Unknown stage $stage" if $stage ne "start";

    my $result;
    my $writer = new XMLTV::Configure::Writer( OUTPUT => \$result,
					       encoding => 'utf-8' );
    $writer->start( { grabber => 'tv_grab_my_astro' } );
    $writer->write_string( {
	id => 'root-url',
	title => [ [ 'Root URL for grabbing data', 'en' ] ],
	description => [
	    [ 'The file at this URL describes which channels are available ' .
	      'and where data can be found for them. ', 'en' ] ],
	default => $default_root_url,
    } );
    $writer->write_string( {
	id => 'cachedir',
	title => [ [ 'Directory to store the cache in', 'en' ] ],
	description => [
	    [ 'tv_grab_my_astro uses a cache with files that it has already ' .
	      'downloaded.  Please specify where the cache shall be stored. ',
	      'en' ] ],
	default => $default_cachedir,
    } );

    $writer->end( 'select-channels' );
    return $result;
}

sub get_default_cachedir
{
    my $winhome = $ENV{HOMEDRIVE} . $ENV{HOMEPATH}
    if defined( $ENV{HOMEDRIVE} )
	and defined( $ENV{HOMEPATH} );

    my $home = $ENV{HOME} || $winhome || ".";
    return "$home/.xmltv/cache";
}

sub init_cachedir
{
    my ( $path ) = @_;
    if ( not -d $path )
    {
	mkpath( $path ) or die "Failed to create cache-directory $path: $@";
    }
}

sub load_channels
{
    my ($baseurl) = @_;
    my %channels;
    my $xmldecl = "<\?xml version='1.0' encoding='utf-8'\?>\n";
    
    my $full_url = $baseurl . "/pack?format=jsonp";

    my $download = get( $full_url );
    my $json = new JSON;
    my $chan_data = $json->allow_nonref->utf8->relaxed->decode($download);
    $chan_data = $chan_data->{'services'};

    foreach my $ch (@{$chan_data})
	{
	    my $ch_id = $ch->{'service_id'};
	    my $ch_name = $ch->{'title'};
	    my $ch_code = $ch->{'service_key'};
	    my $ch_num = $ch->{'channel_number'};
	    my $ch_cat = $ch->{'channel_category'} || 'unknown';
	    my $ch_uri = "$ch_num.$ch_cat.astro.com.my";
	    $ch_uri =~ s/ /_/g;
	    $ch_name = xml_escape($ch_name);
	    $ch_uri = lc($ch_uri);
	    
	    $channels{$ch_uri} = [
		$ch_name, $ch_code, $baseurl, $ch_num, $ch_cat, $ch_id ];
	}
    # TODO: Parse from astro website

    return ($xmldecl, \%channels);
}

sub print_data
{
    my( $url, $channel_id, $date, $astroid, $category ) = @_;
    my $start_date = $date->ymd('-') . 'T00:00';
    my $end_date = $date->ymd('-') . 'T23:59';
    my $full_url = "$url/guide/start/$start_date/end/$end_date/channels/$astroid?format=jsonp";
    my $download = get( $full_url );
    my $json = new JSON;

    my $schedule = $json->allow_nonref->utf8->relaxed->decode($download);
    if ($schedule && ref($schedule) eq "HASH")
	{
	    $schedule = $schedule->{'guides'};
	    foreach my $programme (@{$schedule})
		{
		    my $event_id = $programme->{'event_id'};
		    if ($event_id != 0)
			{
			    my $prog_id = $programme->{'program_id'};
			    my $title = $programme->{'name'};
			    my $episode_title = $programme->{''};
			    my $episode_id = $programme->{'group_order_num'};
			    my $desc = $programme->{'description'};
			    my $startTime = $programme->{'display_datetime_utc'};
			    my $stopTime = $programme->{'display_datetime_end_utc'};
			    my $start = str2time($startTime);
			    my $stop = str2time($stopTime);
			    $start = time2str('%Y%m%d%H%M%S %z', $start);
			    $stop = time2str('%Y%m%d%H%M%S %z', $stop);
			    
			    # XML escaping of fields that are likely to contain problem chars.
			    $title = xml_escape($title);
			    print "  <programme channel=\"$channel_id\" start=\"$start\" stop=\"$stop\">\n"
				. "    <title>$title</title>\n";
			    if ($episode_title)
				{
				    $episode_title = xml_escape($episode_title);
				    print "    <sub-title>$episode_title</sub-title>\n";
				}
			    if ($desc)
				{
				    $desc = xml_escape($desc);
				    print "    <desc>$desc</desc>\n";
				}
			    print "    <category>$category</category>\n";
			    print "    <url>http://api-epg.astro.com.my/api/event/$event_id?format=html</url>\n";
			    if ($episode_id)
				{
				    print "    <episode-num system=\"onscreen\">$episode_id</episode-num>\n";
				}
			    print "  </programme>\n";
			}
		}
	}
    if ($@) {
	# Exception occurred above.  Add a comment to the xmltv file
	# about the error, and allow the details for other channels to
	# keep being processed.
	print "<!-- Error parsing channel listings for "
	    . $channel_id . "\n  from URL: " . $full_url
	    . "\n  Message: " . xml_escape($@)
	    . "\n-->";
    }

    return 1;
}

sub write_header
{
    my ( $xmldecl ) = @_;

    print $xmldecl;
    print '<!DOCTYPE tv SYSTEM "xmltv.dtd">' . "\n";
    print '<tv>' . "\n";
}

sub write_channel_list
{
    my ( $channel_list ) = @_;

    # Write list of channels.
    t 'Writing list of channels.';

    foreach my $channel_id (@{$channel_list})
    {
	if ( not exists $channels->{$channel_id} )
	{
	    print STDERR "Unknown channel $channel_id." .
		" See http://www.astro.com.my/" .
		" for a list of available channels or run" .
		" tv_grab_my_astro --configure to reconfigure.";
	    next;
	}

	my ( $ch_name, $ch_code, $namespace, $ch_num, $ch_cat, $ch_astroid ) = @{$channels->{$channel_id}};
	print "  <channel id=\"$channel_id\">\n"
	    . "    <display-name>$ch_name</display-name>\n"
	    . "    <display-name>$ch_num</display-name>\n"
	    . "    <icon src=\"http://www.astro.com.my/channels/images/channellogo/$ch_num.png\"/>\n"
	    . "    <url>http://api-epg.astro.com.my/api/service/$ch_astroid?format=html</url>\n"
	    . "  </channel>\n";
    }
}

sub write_footer
{
    print "</tv>\n";
}

sub xml_escape()
{
    my ($string) = @_;
    if ($string)
	{
	    $string =~ s/&/&amp;/g;
	    $string =~ s/</&lt;/g;
	    $string =~ s/>/&gt;/g;
	    $string =~ s/"/&#34;/g;
	    # Some of the feeds come in as ISO8859-1 and I can't figure out how
	    # to get them encoded properly as UTF-8 in perl, so convert all non-ASCII
	    # to character references.
	    $string =~ s/([^\x{20}-\x{7E}])/'&#' . ord($1) . ';'/gse;
	}
    return $string;
}
    
### Setup indentation in Emacs
## Local Variables:
## perl-indent-level: 4
## perl-continued-statement-offset: 4
## perl-continued-brace-offset: 0
## perl-brace-offset: -4
## perl-brace-imaginary-offset: 0
## perl-label-offset: -2
## cperl-indent-level: 4
## cperl-brace-offset: 0
## cperl-continued-brace-offset: 0
## cperl-label-offset: -2
## cperl-extra-newline-before-brace: t
## cperl-merge-trailing-else: nil
## cperl-continued-statement-offset: 2
## End:

Wednesday, 28 July 2010

Remote Control

I eventually gave up on the idea of trying to prod the serial port on my Astro STB and bought a cheap IR transceiver from some guy in Hong Kong via ebay. My first attempts at learning the remote codes were frustrating - irrecord complained about something being wrong after incorrectly guessing the settings, and when I finally got some codes learned, I found that a lot of the buttons had the same code, so 2, 6 and 8 were all being detected as KEY_2. Eventually I started trying to manually configure the basic settings based on some of the standard protocols, and eventually hit on RCMM-32, which learned all the codes and can reliably detect them when I press the buttons on the remote.



So now the codes are learned, along with the codes for the extra buttons for controlling a Panasonic DVD player on the bottom of my TV remote, which have now found a use giving basic control of Freevo in case the wireless keyboard, bluetooth equipped mobile phone or one of many wifi equipped gadgets is not close at hand.



Next step is to go the other way and send the codes out to the STB. The transceiver doesn't seem to be sending anything out at all. I still haven't narrowed down the problem, it may be a faulty IR LED, or an incompatibility between the transceiver (MCEUSB gen 1) and the software driving it (LIRC 0.86). Now that I have the remote codes, it might also be a good time to go back to trying the serial link, but that'll have to wait for another night. For now, I'll just post up the Astro remote codes, in case anyone else is struggling to get them working with LIRC.




# brand: ASTRO
# model no. of remote control:
# devices being controlled by this remote:
# Astro Satellite Decoder STB (Malaysia) [Philips DSR4201/68]

begin remote

name ASTRO
bits 8
flags RCMM|CONST_LENGTH
eps 2
aeps 100

header 417 278
three 167 778
two 167 611
one 167 444
zero 167 278
ptrail 167
pre_data_bits 24
pre_data 0x225027
gap 99817
toggle_bit_mask 0x0

begin codes
KEY_0 0x00
KEY_1 0x01
KEY_2 0x02
KEY_3 0x03
KEY_4 0x04
KEY_5 0x05
KEY_6 0x06
KEY_7 0x07
KEY_8 0x08
KEY_9 0x09
KEY_RED 0x6D
KEY_GREEN 0x6E
KEY_YELLOW 0x6F
KEY_BLUE 0x70
KEY_KPASTERISK 0xF6
KEY_TEXT 0x3C
KEY_MENU 0x54
KEY_SHOP 0xAA
KEY_EXIT 0x83
KEY_BACK 0xA9
KEY_EPG 0xCC
KEY_INFO 0x0F
KEY_UP 0x58
KEY_DOWN 0x59
KEY_LEFT 0x5A
KEY_RIGHT 0x5B
KEY_OK 0x5C
KEY_VOLUMEUP 0x10
KEY_VOLUMEDOWN 0x11
KEY_CHANNELUP 0x20
KEY_CHANNELDOWN 0x21
KEY_AUDIO 0x4E
KEY_SUBTITLE 0x4B
KEY_MUTE 0x0D
KEY_MAIL 0xF3
KEY_FAVORITES 0x84
KEY_HELP 0x81
KEY_POWER 0x0C
end codes

end remote


Wednesday, 17 March 2010

Controlling my STB

My latest experiment is to see if I can control my Astro decoder via the serial port on the back, so I can have Freevo record from different channels (currently I'm mostly recording the kids shows from NHK that are on in the middle of the night, so we have to remember to leave the decoder on channel 963 each evening). There is of course no documentation about the serial port anywhere on the net, but a few US set top boxes have been hacked, so there is some hope.

What I know after an hour of poking around:

Manufacturer: Philips
Model Number: DSR4201/68
Serial port: 115200 8N1 (diagnosed by observing the output)

Firmware seems to be developed in France, as the only thing I have got out of the decoder so far is some debug messages in French as I shut down. Thomson also supply decoders to Astro, so maybe they are providing the firmware for both their own and Philips boxes?

Here is the output I see:

TE : c0c9877c : Ln : b40 > ini_mem
TE : c0c9877c : Ln : b40 > : free_mem
TE : c0c9877c : Ln : b40 > gmhw_init
TE : c0c9877c : Ln : b40 > appel a la fonction Get_Dynamic_mem, retour 0
TE : c0c9877c : Ln : b40 > Valeur de la taille totale dispo pour Partition Privee =2363736
TE : c0c9877c : Ln : b40 > Adresse de la memoire Dynamique 0xc0dbeea8
TE : c0c9877c : Ln : b40 > ok

No hits on any of that on Google unfortunately, so I'm on my own.

Monday, 22 September 2008

Atoms Moved

I've now successfully imported a 700 post blog into draft.blogger.com


There were a few more quirks of blogger.com along the way. Having next and previous links in the Atom feed causes blogger to reject the entire import file, so I ended up putting them in as comments. There were also some old posts that didn't have an author (perhaps from some beta version of Roller, and blogger rejected those due to the empty name field.

Finally, I had an old comment from the blog's early days that was not UTF-8 encoded in the database (it doesn't appear correctly in Roller either), so I had to reencode that before blogger.com would accept it for import.


Here is the final export template for Apache Roller 4.0.



#set($pager = $model.getWeblogEntriesPager())
#set($map = $pager.getEntries())
<?xml version="1.0" encoding='utf-8'?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:thr="http://purl.org/syndication/thread/1.0">
<id>$model.weblog.id</id>
<title>$utils.escapeXML($model.weblog.name)</title>
<subtitle>$utils.escapeXML($model.weblog.description)</subtitle>
<updated>$utils.formatIso8601Date($model.weblog.lastModified)</updated>
#if ($pager.nextLink)
<!-- link rel="next" type="application/atom+xml" href="$pager.nextLink"/ -->
#end
#if ($pager.prevLink)
<!-- link rel="previous" type="application/atom+xml" href="$pager.prevLink"/ -->
#end
#foreach( $day in $map.keySet())
#set($entries = $map.get($day))
#foreach( $entry in $entries )
<entry>
<id>$entry.id</id>
<title>$utils.escapeXML($entry.title)</title>

<author>
#if ("$entry.creator.fullName" != "")
<name>$entry.creator.fullName</name>
#end
</author>

<published>$utils.formatIso8601Date($entry.pubTime)</published>
<updated>$utils.formatIso8601Date($entry.updateTime)</updated>

<content type="html"><![CDATA[$entry.text]]></content>
<category scheme="http://schemas.google.com/g/2005#kind"
term="http://schemas.google.com/blogger/2008/kind#post"/>
<link rel="self" type="application/atom+xml" href="http://example.com/$entry.id" />
</entry>
## use Atom threading extensions for comment annotation
#foreach( $comment in $entry.comments )
<entry>
<id>$comment.id</id>
<title>$utils.escapeXML($utils.truncate($utils.removeHTML($comment.content), 40, 50, "..."))</title>

<author>
#if ("$comment.name" != "")
<name>$utils.escapeXML($comment.name)</name>
#end
#if ("$comment.url" != "")
<uri>$utils.escapeXML($comment.url)</uri>
#end
#if ("$comment.email" != "")
<email>$comment.email</email>
#end
</author>
<published>$utils.formatIso8601Date($comment.postTime)</published>
<updated>$utils.formatIso8601Date($comment.postTime)</updated>
<content>$utils.escapeXML($utils.removeHTML($comment.content))></content>
<thr:in-reply-to ref="$entry.id" type="application/atom+xml" href="$entry.permalink"/>
<category scheme="http://schemas.google.com/g/2005#kind"
term="http://schemas.google.com/blogger/2008/kind#comment"/>
<link rel="self" type="application/atom+xml" href="http://example.com/$entry.id" />
</entry>
#end
#end
#end
</feed>

Sunday, 14 September 2008

Moving Atoms

Having now found an apartment in Malaysia, I'm now faced with the logistics of moving. One problem I'm facing is that for the last 5 years I've run my own server, which now supports several blogs, a forum and my mobile phone's OTA backup. If I knew it was just for a week or two, I could take it offline temporarily, but from what I can tell, internet connectivity in Malaysia is probably not going to be reliable enough to keep all this online for a reasonable proportion of the time, and unless I pay serious money I'll be stuck with the hassle of a dynamic IP address. So I'm looking to migrate at least the blogs and forum off to other services. I've just finished migrating the posts and comments from my first blog (there are still hard links back to my server that need fixing), which was no easy task, as although draft.blogger.com supports Atom based import, it is very particular about some things, with very little documentation and completely useless error handling - it just seems to stop processing as soon as it finds something it doesn't like, and if nothing has been imported yet you get a general meaningless error message, but if one or more posts were successfully imported, it just silently fails to import the posts starting with where it failed. So here is what I found.


In addition to needing to be a valid Atom 1.0 feed, each entry needs a unique self referencing link: <link rel="self" type="application/atom+xml" href="post-id"/> The href does not have to be real, just unique, so I used example.com in my export from roller. The import process also does not accept empty tags for the post or comment author's email, name or uri (according to the rnc schema, only email cannot be empty).


The following is the template I used to export posts and comments from Apache Roller 4.0. It is based on an earlier export template for JRoller by Damien Bonvillain that output Atom 0.3, with updates for Roller 4.0, Atom 1.0 and blogger.com's undocumented quirks. The exported content still needs some post processing; removing or filling in empty author child tags, checking the truncation of comment titles or misformed content has not broken anything, and replacing relative references (which shouldn't be there in the first place), but generally it works for short blogs. There seems to be a hard-coded limit in getRecentWeblogEntries of 100 posts, so it needs rework to use a pager and an external script to fetch all the pages.


As with the original, paste the contents below into a new roller template, then use the template to access your blog. If you have too many posts for blogger.com or the export script to handle, you could try using a pager in the export template, or break up the export file manually after extracting everything.




#set($entries = $model.weblog.getRecentWeblogEntries('', 100))
<?xml version="1.0" encoding='utf-8'?>
<feed xmlns="http://www.w3.org/2005/Atom" thr="http://purl.org/syndication/thread/1.0">
<id>$model.weblog.id</id>
<title>$utils.escapeXML($model.weblog.name)</title>
<subtitle>$utils.escapeXML($model.weblog.description)</subtitle>
<updated>$utils.formatIso8601Date($model.weblog.lastModified)</updated>

#foreach( $entry in $entries )
<entry>
<id>$entry.id</id>
<title>$utils.escapeXML($entry.title)</title>

<author>
<name>$entry.creator.fullName</name>
</author>

<published>$utils.formatIso8601Date($entry.pubTime)</published>
<updated>$utils.formatIso8601Date($entry.updateTime)</updated>

<content type="html"><![CDATA[$entry.text]]></content>
<category scheme="http://schemas.google.com/g/2005#kind"
term="http://schemas.google.com/blogger/2008/kind#post"/>
<link rel="self" type="application/atom+xml" href="http://example.com/$entry.id"/>
</entry>
## use Atom threading extensions for comment annotation
#foreach( $comment in $entry.comments )
<entry>
<id>$comment.id</id>
<title>$utils.escapeXML($utils.truncate($utils.removeHTML($comment.content), 40, 50, "..."))</title>

<author>
<name>$utils.escapeXML($comment.name)</name>
<uri>$utils.escapeXML($comment.url)</uri>
<email>$comment.email</email>
</author>
<published>$utils.formatIso8601Date($comment.postTime)</published>
<updated>$utils.formatIso8601Date($comment.postTime)</updated>
<content>$utils.escapeXML($utils.removeHTML($comment.content))></content>
<thr:in-reply-to ref="$entry.id" type="application/atom+xml" href="$entry.permalink">
<category scheme="http://schemas.google.com/g/2005#kind"
term="http://schemas.google.com/blogger/2008/kind#comment"/>
<link rel="self" type="application/atom+xml" href="http://example.com/$entry.id"/>
</entry>
#end
#end
</feed>