Smoking toooooo much PHP

After a little wait (~ 3 weeks), having asked ASUS to provide me with the source code for the drivers to the DMB-TH Digital TV receiver, I finally got two tarballs of the source code. One step closer to an open source friendly solution to watch Digital TV in China and Hong Kong.

It took a little persuasion to convince the technical support team at ASUS that I was entitled to the source as it was based on GPL code, but once they understood, and had asked their supervisor/ R&D department about it. It was just a matter of waiting for the R&D dept to get time to do it.

I was very impressed with ASUS in general - their tech response times for product support where pretty amazing (usually answering within a few hours). If you have the option of buying ASUS, I can highly recommend their email tech support. Friendly, positive, and responsive.

The driver modifications are all to drivers/media/dvb/*, and consist mostly of 2 new files dealing with the demodulator, and a rather unfortunate hack at db3000mc.c - Basically replacing the core of that file with the routines for the Legend chip. Rather than creating a new file (which looks like it would have been a far better idea).

Anyway, I'm going to try building these changes against a recent kernel, then see if I can sort out the db3000mc mess. If anyone want's to help out, drop me an email and I can send you the files.

After quite a bit googling and guesswork, I finally got my portable to connect via my Motorola A1200  to peoples.net (unlimited GRPS). So here's the settings for reference (as most of the google results are not quite accurate).

for starters: /etc/bluetooth/rfcomm.conf
(you need to find out your bluetooth id - plenty of results if you google "bt grps howto's" )

rfcomm0 {
        bind yes;
        device ***device id here **;
        channel 5;
        comment "Alans phone";
}

you need to configure pairing - I think this is the one that worked
/etc/bluetooth/hcid.conf

options {
        autoinit yes;
        security user;
        pairing multi;
        passkey "**MAKE UP A PASSKEY ***";
}

And now for the magical stuff.. (the 0.0.0.0:10.0.0.1 was the magic bit that solved all the connection problems)
/etc/ppp/peers/peoples

debug
connect  "/usr/sbin/chat -v -f /etc/chatscripts/people-connect"
user "ptc"
password "ptc"
/dev/rfcomm0 115200
show-password
noipdefault
usepeerdns
defaultroute
noauth
0.0.0.0:10.0.0.1
ipcp-accept-local
ipcp-accept-remote
debug
kdebug 7
#nodeflate
novj
noccp

#remove if you want pon to run in the background
modem -detach

the chat script
/etc/chatscripts/peoples-connect

TIMEOUT 35
ECHO    ON
ABORT   '\nBUSY\r'
ABORT   '\nERROR\r'
ABORT   '\nNO ANSWER\r'
ABORT   '\nNO CARRIER\r'
ABORT   '\nNO DIALTONE\r'
ABORT   '\nRINGING\r\n\r\nRINGING\r'
'' ATZ
OK 'ATM1L1'
OK 'AT&f+cgdcont=1,"IP","peoples.net","",0,0'
OK 'ATDT*99***1#'
CONNECT ""

This is disconnect (not really used it, but might help...)
/etc/chatscripts/people-disconnect

SAY 'Starting GPRS disconnect script\n'
"" \K
"" +++ATH0

And hopefully when you do

#pon peoples

you get an internet connection from anywhere in Hong Kong..

Over the last few weeks, apart from a huge workload, I've been busy moving house - now living in the clouds 52 floors up..

Anyway, one of the important projects to get done for myself, was setting up mythtv on this server (my main web server doubles up as my PVR/video jukebox.)

Just some of the highlights of setting up mythtv

• Radeon 9600SE All in wonder Card - TV in did not end up working (well, the card's quite good as a graphics card anyway, and it came with a usb remote control) so it was not a total loss..
• Bought Haupage PVR500MCE - appears to work ok, although you have to be carefull with the TV arial cable to get good reception. (very nice card with 2 tuners built in.)
• No way to get TV listings for Hong Kong Free to air channels - see later......
• Display did not render chinese (which is what the channels guides come with) - ended up modifying the theme.xml / ui.xml in one of the theme's to use a chinese font -> that solves the schedule browser
• have to use chinese font for osd display. (to display the active watched program)
• Rewind / go back, and constant recording what you watch is really nice..

I had to write a small application to grab the listings for Hong Kong TV and output XMLTV format ->
see the svn files here http://www.akbkhome.com/svn/akpear/XML_Tv/
it's pretty simple to use, only requires the mbstring to be working. (usage instructions inside the php file.)

Unfortunatly, I still have quite a uphill battle to finish for the box, let alone thinking about all the other fun stuff I started recently, and have left in limbo due to moving house. Still todo on the box:

• Sort out Remote key mappings for the ATI remote -> it's pretty messed up for almost all the features (TV/Video watching/EPG etc)
• Autoripping of VCD/DVD's when inserted..
• How to map autoripped VCD/DVD's into imdb for the video library
• Add bigger disk ;)
  

The unfortunate thing about mod_proxy, is that it's a pain in the ass to set up correctly. Another one of the machines I work on had been using this to allow some remote access to a private box. Unfortunatly even though I had IP restricted access, I had obviously missed a setting somewhere. Looking at my logs the other day, I realized that apache was doing quite a trade in URLs...

Last time I misconfigured mod_proxy, my ISP had phoned me up to let me know there was spam eminating from my server, so I had re-configured mod_proxy and fixed the open proxy on that server. My guess is that the abusers of mod_proxy had concluded that such a direct attack (POST xx.xx.xx.xx:25 ... MAIL FROM ....) was rather self defeating, as ISP's tend just to block a server if it is shooting out lots of spam (or suddenly has high traffic outbound on port 25)

So I was supprised to see what was being requested from my open proxy. Captcha images!!!!!

Guessing from the referrer information, I think some guy in Bejing had come up with this idea:

- Set up an online game site, and make the users fill in a captcha to play the game.
- Except, the captcha actually comes from a free webmail provider, and the entered data enables them to set up new webmail accounts, and send out webmail.

I presume that doing that directly from their server caused them to get blocked pretty quickly from the webmail provider, so they just looked around for open proxies to solve the problem..

Quite smart in some respects, and I like the bit about on-line gaming to do this - lot's of kids like these on-line games, and dont think twice when they solve a captcha.. - Let's just hope that blog spammers dont get in on the act..

I had a nice phone call today from my ISP, they let me know that my outgoing port 25 had been auto-blocked due to excessive traffic.

Agh, an instant chkrootkit indicated nothing scary, so I started digging into the reasons behind this. I have to say my local ISP Netfront, has world class service. Gary, who I was dealing with, was very co-operative, opened up port 25 so we could track down the problem. To be honest, I was amazed at the service where they actually called me to let me know they where blocking my service. I'm sure there are many a broadband fixed IP user who would envy this kind of service.

After chkrootkit failed to find anything and Gary had opened the port, I ran

#netstat -n | grep 25
tcp        0      0 myip:33236         someip:25           TIME_WAIT

This indicated a few outgoing connections on port 25, so I asked one of my collegues who thankfully knows linux better than me;) for ideas to track down what process was doing this.

#fuser 33236/tcp
33236/tcp:             25045

This indicated the process id that was causing the connection, and now down to good old ps

#ps auxw | grep 25045
www-data    25045  ......  /usr/sbin/apache2 -k start -DSSL

Agh - Apache was causing port 25 connections. - My first thought was some horrific mistake I made with my PHP code, but a quick look through the apache log files indicated that my suspicions where not quite correct. the apache log file had rather a large number of these.. - from various ip addresses.

210.245.151.81 - - ... "POST http://202.81.252.1:25/ HTTP/1.1" 200 ..

At this point I started to suspect the reverse proxies on my server (especially as I had set it up again recently on another server and had to deal with the default config there)

My debian based install had a file in apache2/mods-available called proxy.conf, which I had not copied to apache2/mods-enabled. I had only copied the proxy.load file.

In this file, was the critical section.

        <Proxy *>
                Order deny,allow
                Deny from all
                #Allow from .your_domain.com
        </Proxy>

This blocks all access to the proxy, So after adding this file, to enable my old reverse proxies,  I had to add sections like this to open a few specific proxies

        <Proxy http://devel/>
                Order allow,deny
                Allow from all
        </Proxy>

The only problem I had was that my php5 server was running on port 81, and this config failed to allow access via the reverse proxy.

      <Proxy http://php5.akbkhome.com:81/>
		#this doesnt work?!!!
                Order allow,deny
                Allow from all
        </Proxy>

Anyway, at least I'm not a spammer anymore..

As things have been getting busier here, I've started partnering up with a friend to provide better service to my consulting clients. It already has quite a few benefits

• A nice desk in an office, use of all the usual office facilities.
• Other programmers to bounce ideas off of, (rather than just irc...)
• Possibility to offer bigger projects, or charge more ;)
• I can farm out alot of the re-installing, virus cleaning, PITA Windows jobs..
  

You can see more about my partners at their web site: www.qsolutions.hk.com, while mainly a MS/ASP/.NET shop, part of my involvement is to raise their skills and knowledge of Linux and PHP. Along with helping out on the linux based on-site maintenance. (and maybe remove the flash movie from their splash page.)

As part of this move, I really need to build a proper desktop workstation. The easy bit of which is deciding to go for 2 19" monitors, as having started using dual monitors, I would never return to single ones for developing. But the bigger problem comes with deciding the machine architecture.

Starting with the extensive range of motherboards, with weird and wonderfull built in hardware. I need to work out how compatible the one I chose will be. Then there is the other big choice, to 64bit or not... These are some random thoughts the matter

• Is there a huge difference of speed in the AMD64 (I know it's a bit faster, but is it huge enought to compensate for the other issues)?
• Is running AMD64 as a 32bit system going to be less painfull (no chroot enviroment for open office etc.)?, or does this just defy the point... (although the thought of firefox64 not running flash is quite attractive - visting www.hkgolden.com to see why..)
  
• I was concerned about having to upgrade to 2.6 kernels, but it looks like someone fixed davfs2 for 2.6 (which I critically depend on.)
• Is it worth trying out gentoo or are the debian-amd64 packages fast enough?, or is it going to be as annoying as freebsd?, long install times, no synaptic? (I guess it wont have gnome cvs build nightmares like freebsd.. - autoconf tools for gnome doesnt work correctly on it.) - does it solve the open office issues..
  
• how well sync'd is debian's amd64 port, the sparc64 install I've done seems a little outdated at times, compared with the intel boxes..
  

The motherboard choice is even more confusing, this gives you an idea of costs and availability of hardware in Hong Kong and I managed to find a debian motherboard compatibility list, although It looks like one of those 'trying to hit a moving target' tasks, as the list of available motherboards changes every month..

I suspect either an Abit or a Gigabyte might be the way to go, as they are pretty common here, but I'm totally confused about the motherboard graphics card integration, - It looks like the onboard graphics card is going to be useless for dual head dvi monitors. So I probably need a nVidia professional, but even that decission looks complex..

Such a pain the backside picking all this stuff, I kind of wish someone would do all the deciding for me, and deliver it at my door. But unfortunatly since I'm one of the better linux experts in Hong Kong, that doesnt leave the door open to finding anyone who could do this properly.... And if started offering this service (it's unfortunatly pretty easy for the competition to copy ...)

- mind you, if you do want a custom linux box designed and delivered in Hong Kong.. give me a call, and I'll quote a consulting fee to design it.. The hardwares probably less than HK$5K, so comparing that to a baddly spec'd dell box, theres still a saving of at least HK$7K in fees' for designing the bugger..

I had to laugh the otherday reading one of the Microsurf's blogs about how they suddenly had to deal with customer comments, and suprisingly enough, not many where positive. I think the guy was from the .Net team, and I considered he probably get's it easy, at least he's not on the IE team..

I gave up microcrap years ago, and only now use it for interoperability testing, where most of the time it fails, (XP included..) but I was supprised this week, when working with a new team here, when someone asked me why he should use linux rather than microsoft.

Youth and alot of Microsoft giveaways had ended up with the impression that this stuff fulfilled every need and desire. I have to admit I was a little shocked, I rarely find people who actually like what microsoft attempts to deliver. Most are resigned to the fact that it's a pile of crap but they are either to lazy to learn something new or, blindly believe that the next version will be better.

I started explaining the concept, that as a developer you have the ability to actually fix things with linux. Where he seemed to think that microsoft actually fixed things, especially as his old company had some developer/partner package.. But digging deeper it was pretty clear that solving problems just meant news about workarounds comes faster.. Actually fixing stuff still required something close to a miracle. (CSS fixes for IE came to mind)

But what really would have been a killer example was reading about Webdav support on XP, basically it's completely broken (there are even Knowledgebase articles saying so from 2003). Yet again re-inforcing my view of Microsoft as a company that may fix bugs in each release, but breaks just as many new things with each effort. So however many blogs you see coming out of microsurfs, the knowledge that they will completely ignore their customers (which thankfully I managed to remain from being), means that they still have a long while before anyone actually expects their software to work.

My father just retired, and rekindled his interest in doing Video conferencing (He's in the UK, I'm in HK). So after a bit of checking for linux compatible usb webcams, I settled on a Creative webcam Live!

Set up was not to bad, it uses a spca50x chip, which builds and compiles quite easily. The only oddity was that the spcatools that can be downloaded from the driver site didnt appear to work. However, gnomemeeting and streamer displayed the output perfectly.

I ran into huge trouble though with networking and video conferencing, I initially set up gnomemeeting behind my firewall (eg. the webserver with ipmasq on it.), but after a few experiments it became obvious that the whole relaying of h323 video connections was increadibly complex, and really not worth the hastle. In the end I just moved the camara to my firewall/webserver/comms server (eg. it runs irc/gaim).

I did at one point get video conferencing working within the house to a win98 box with netmeeting, but getting the video to actually display in netmeeting was very hit and miss.

In doing all this, I wandered off topic and started looking at alternatives, the two big ones being SIPs and Skype. It's pretty difficult to get an overview of SIP, but by the looks of things it looks like it's not much better than the current gatekeeper type of concept of gnomemeeting/netmeeting. The only thing going for it is that all the hardware manufacturers have been building routers and phones specifically for this protocol. (I think there's even a free SIPS router for linux). But in reality it looked like it had the same architecture as gatekeeper, which was the core to the fundimental problem with this point to point via a special gateway concept. The gateway was always an extra consideration that was a hastle to set up.

Skype, is something that I've been hearing more and more about, and from reading their technical FAQ, they seem to have spotted the above problems and thought of a half decent solution, basically by turning Skype clients that have direct internet access into relays (or quazi gatekeepers), along with using P2P type technology to do the connections (eg. doing originating connections from inside firewalls, to outside, rather than trying to cleverly route external connections to an inside machine).

The only shame about the Skype stuff is the propriatory protocol and client, which makes it highly seceptable to an open source implementation decimating their market, and the fact that smart people cant hack the client to add nice features like video conferencing.

But I'm still waiting to see how my gnomemeeting works with netmeeting, lets hope that my parents broadband is not firewalled to death by their ISP..

There were some interesting comments from my last post on Hyping by blog. Jackson Miller pointed out very bluntly, isnt that what blogs are for, and while he is partly correct, if a blogs are purely a advertising / news feed for a project, then they are not really blogs, but more project sites. What makes blogs interesting normally, is not that they publish release announcements, but that you get some insight into other things a developer may be doing, often unrelated to a project they are well known for. So I guess the conclusion was, if some one starts hinting you are Hyping and not blogging, perhaps you are..

I saw a few posts on Artima more recently, that included detailed analysis on RoR indicating alot of what I considered RoR to be. Ruby, while having interesting features, doesnt appear to have what could be called an elegant language construct, or a particularly huge following, which for me are part of the consideration on whether to invest time into experimenting with it, (which C#/ASP.net did justify, but produced similar returns). RoR, turns out to be little more that a clever combination of tools to write skeletons and some reasonable libraries, which while useful, really doesnt justify the excitement, but I gues it's an improvement on the ASP.NET, where the solution is not forced in your face so much, and alternatives are frowned apon (try googling for the equivilant of mysql_escape_string in .NET, and you will see what I mean)

Imap Continued.

Bincimap unfortunatly was unable to deliver the promise that it looked like it could. This week I got a call saying that Outlook express (or more like 'lookout express') users where having problems. It's pretty common knowledge that although outlook says it supports IMAP, it's implementation is buggy to the point of unusable. I know this from googling mailing lists and seeing the amount of kludges and workarounds that appear to have gone into IMAP servers, just to support this pile of crap.

Normally when you get problems with outlook and imap, you brush it off as intermittenant problems with a crap piece of software, and suggest they upgrade to a real email client (thunderbird, or evolution come to mind). But sometimes, company owners or important sales staff are not really that open to changing the ill gotten ways, so Outlook support has to be suffered.. (at least at an hourly rate!!!). So this time (after a few goes at modifying the settings on outlook) I decided to examine what was going on a little closer, including doing protocol dumps.

The key problems where that deleted messages (and ones that had been moved to another folder) would reappear as unread, new when you pressed the send/recieve.

To my amazement, outlook spawns new connections and does alot of imap operations concurrently, without a care in the world on how complex this may be to the server (eg. 3 connections all doing operations on the inbox folder). And menu operations often open new connections, and drag and drop operations dont. - It's all a bit like a beginners VB program, completely undesigned, and thrown together a few minutes after hello world worked.

I've given the protocol dumps to the bincimap developers, but over the weekend, I also discovered that my wife's palmphone, was unable to read email. I can pospone problems with companies a few days, but I better fix my wife's issues faster!. So after another marathon protocol dumping sessions, it became clear that bincimap was sending a little too much information for snappermail to understand. So I quickly switched over to dovecot imap.

I feel a bit disapointed here, us fickle users, jump from one ship to another so easily. I did get the chance to look at bincimap's souce, and it was very clean C++, and pretty well designed. And having given the author (Andreas Aardal Hanssen ,who was very responsive) a reasonably high quality set of bug reports, I didnt feel to bad deserting to another application.

Dovecot on debian proved amazingly simple to migrate to, the only change required after apt-getting was modifying /etc/dovecot/dovecot.conf and changing the line protocol = imaps

Other than that, restarting evolution, which should provide another good blog review, and I have now finally tested, used and configured all 5 major open source imap servers..

mbox must be about the worst designed format ever, this week in a small office I consult for, a few of the staff started complaining they couldn't open some of their mailboxes. It didnt take long to realize that the server was overloaded. 10 people, each had an inbox averaging 500Mb, and outlook checking email every 3 minutes, a few of them also doing a full scan of their imap folders checking for new mail, which range from 2->4Gb.

The poor server was suffering badly, so yet again, I investigated IMAP servers. I've tried cyrus, courier, uw-imap and while each has advantages cyrus and courier have tended to be a little annoying to set up, messing around with auth and protocol issues. uw-imap is the root cause of the above issues (although mbx format does help alot). I was interested to find bincimap (binc is not courier). The overview points out that it's a pure maildir backend mail server (which usually perform pretty well with a cache, and dont have interface issues with folders containing folders not being usable). It also appeared to be pretty simple to set up, although no examples for use with exim where obvious.

I tested the installation on my development box, and after a bit of hunting around and guesswork, I put together simple instructions for converting a exim4/uw-imap installation to exim4/bincimap. The only downside to the conversion was that I lost all my "important" flags from thunderbird. I was actually quite impressed that my own instructions where so amazingly simple.

mono/ASP.net and C#

My experiments with ASP.net and C# have been contining, highlights of this week where discovering that codebehind, without VS.NET is a real waste of time, the codebehind concept assumes that you want to have a compiled .dll. So if you are developing a web page with codebehind, your roundtrip testing becomes edit/build/(install)/test.. I may as well write it in C!!!, at least you can compile that on the fly (tcc a very nice small C compiler).

After playing around with various <% language="C#" src="..."> options, I eventualy came up with the kludge of doing quasi virtual includes

<!-- #include "lib.cs" -->

It's far from clean (it feels a bit like working with function libraries), but at least it works.

Next on my challenges was getting ASP.Net working with mysql, This is a challenge in it'self (I ended up copying the bytefx.dll into my web root's bin directory to get the import working). From what I gather, ByteFX who wrote one of the main mysql connection toolkits for .NET, (which looks like it is now owned by mysqlAB) is unfortunatly not very well documented. The example on the go-mono.org site works, but It did not take long to be reminded that if you want to work with C#, you have to think the Microsoft way. The one true solution, or the one true solution (both of which look good on the face of it, but are shit round the edges.)

Any good PHP programmer knows that sending raw data from a URL to the database is a security nightmare waiting to happen. So we have these wonderfull features like addslashes, and mysql_real_escape_string(). - normally hidden nicely in a DB abstaction layer. We also get to use bound parameters in some database backends that are designed that way. The .NET way is take parameters or give up.. - you cant escape strings, you must use parameters. (excuse my memory here - this example may need fixing)
eg.

mycmd.executeText = "select * from sometable where name=@name";

MysqlParameter param = new MysqlParameter("@name", MysqlType.VarChar);

param.Value = "some'test";

mycmd.Parameters.Add(param);

A couple of problems with the above code come to mind, (apart from it's suffers the .NET problem, adding as much noise as possible to a piece of code, trying hiding the purpose).

• MysqlParameter docs are difficult to find, and mostly in javadoc style format - which not exactly informative.
  
• @ is used by mysql for variables, so apparently the @ will get changed to ? later...
• Debugging what is actually going to the server is impossible! (as far as I could tell). I ended up turning debugging on , on the server, just to be sure what data was ending up at the server.

I can't say it's all bad, but it get very difficult to see the gem's between the rocks when you spend your time writing simple methods to solve common problems all the time.

Client ever asked you to turn a HTML table into a excel spreadsheet?

Given one hour to convert a complex piece of data retreival code to output to an excel file. The thought came to me, why not do it in javascript, based on the existing HTML. write a small piece of javascript that iterates through a HTML table, and posts a form with the data as a CSV to a 2 line PHP script.
This is the HTML

<form method="POST" action="quickexcel.php" onsubmit="return toExcel('data')">
<input type="hidden" id="exceldata" name="exceldata" value="">
<input type="submit" name="_submit" value="Download as Excel">
</form>
<table id="data"> ...... table with data ...... </table>

you can see the javascript here (it's pretty simple)
and the two line php file..
<?php

header('Content-type: application/vnd.ms-excel');
echo $_POST['exceldata'];