I was DDOSed and I didn't notice

For the past 15-20 years, I've been running a little website called Programming Basics that teaches programming to kids. It's an ancient, very amateur website that clearly shows its 20 year age too. It has amateur programmer art. It has JavaScript popovers and other archaic HTML stuff. It has PDFs of handouts that teachers can print out. It's main distinguishing characteristic is that it still works after nearly 20 years, and I haven't taken it down for some reason. It's very obscure, and it receives a very small amount of traffic.

And I think someone tried to DDOS it, but I'm not sure.

I host that website on Amazon Web Services (AWS), and when I looked at the bill a few weeks ago, I noticed that the charges seemed elevated. The per month charges have been pretty much the same for a decade, so I thought I must have accidentally left a cloud computer running or something. But after digging through my billing reports, the charges seemed to be caused by unusual web traffic to my Programming Basics website.

Digging into the general usage statistics provided by AWS, it seems I received 126,755,687 requests for the page "/" on February 28. The day before, I only received 209 requests for that page. The "/" page automatically redirects to the "/en/" page. So typically, I should receive a similar number of requests to the "/en/" page as to the "/" page or even a little more since bookmarks and search engines will usually go directly to the "/en/" page. Instead, I received 785,650 requests for the "/en/" page. That's incredibly high, but it's strange that while most of the clients didn't bother following the redirect, some did. The ones that didn't follow the redirect were obviously basic traffic generating bots that simply generate requests, but  why were some bots coded up differently to follow redirects? Accesses to other webpages on the website and accesses on other days seemed fairly reasonable. I wonder why that attack was only for the root page of the website, especially considering that the page was essentially blank? Wouldn't it have been better to access a page with a larger file size? Or a spread of different pages or even non-existent pages? I suppose it doesn't matter since the whole website is a static website anyway.

Digging into hourly usage statistics, it seems that almost all the requests happened in a one hour period:

28.05 GB was used right at 12:00 UTC time. The timing is a little odd. I suppose the attack was scheduled in advance to occur right at 12. But why was it so short? Did AWS recognize an attack was occurring and block it? Or did the attackers only purchase a small DDOS attack, so it couldn't be sustained? Or did the attackers realize that they attacked the wrong target or that it was pointless trying to attack a website hosted by Amazon and call it off?

I'm too lazy to download the gigabytes of logs and do a proper analysis, but when I took a look at one or two log files, it seems like the attack happened around 12:50 UTC and lasted only 3-4 minutes, so maybe it could have been manually triggered after all. But if it was manually triggered, maybe the attacker would manually visit the website to verify if the attack was working or not. If so, I could search through the logs, and maybe I could pick out the request that comes directly from the attacker's computer. Of course, maybe the botnet automatically monitors its own effectiveness. That might explain why some of the requests followed the redirect while others did not. The requests that followed the redirect were actually trying to verify the effectiveness of the attack by sending a normal request to the website and measure its response time.

AWS provides summary statistics of the country where requests came from. The accesses seem to be spread pretty widely geographically, so it really was a distributed botnet. Here's a breakdown of the top few countries where the accesses came from:

1. United States 27,235,432
2. Bulgaria 15,866,047
3. Turkey 6,927,133
4. France 6,166,693
5. Germany 6,141,788
6. Indonesia 6,115,375
7. Netherlands 5,347,080
8. Canada 5,156,815
9. Australia 4,458,130
10. China 3,511,069
11. India 3,093,340
12. Japan 3,039,478
13. Vietnam 2,406,960
14. Brazil 2,321,232
15. Russian Federation 1,859,764
16. Iran, Islamic Republic of 1,631,397
17. Colombia 1,510,684
18. United Kingdom 1,485,535
19. Korea, Republic of 1,323,381
20. Bangladesh 1,219,066
21. Spain 1,170,194
22. Thailand 1,081,853
23. Finland 1,052,344
24. Ecuador 921,248
25. Poland 867,562
26. Argentina 843,783
27. Ukraine 842,882
28. Mexico 838,305
29. Hungary 719,549
30. South Africa 669,794
31. Philippines 610,215
32. Kazakhstan 599,328
33. Italy 540,665
34. Luxembourg 532,379
35. Chile 512,108
36. Libya 486,914
37. Venezuela, Bolivarian Republic of 471,266
38. Singapore 455,583
39. Ireland 435,134
40. Peru 396,041
41. Latvia 357,309
42. Dominican Republic 353,845
43. Sri Lanka 314,126
44. Norway 298,881
45. Albania 268,222
46. Myanmar 235,063
47. ...

I just looked at a few log entries of requests, it seems like some clients only made a few requests while other clients would submit hundreds of requests, all within the span of a few seconds. Just randomly grabbing a few IPs from the logs and doing IP lookups, it looks like the requests seemed to be coming from compromised servers in various data centers. Just grabbing some random IPs, I can see Hurricane Electric, Heymman Servers, BelCloud, RK Telecom, Maxnet Telecom, Min Proxy Company--just a lot of servers from all over. I wonder if these are compromised servers or cloud servers bought using stolen credit cards. Is it possible to report these servers to the service providers as being compromised so that they can be taken down and fixed?

So overall, it does seem like I was the victim of a DDOS, but since everything is cloud-hosted, I didn't really notice at all. To be honest, the site is so off my radar, I don't think I would have noticed even if the DDOS really had taken down the website and made it inaccessible. Honestly, I can't fathom why someone would want to run a distributed denial of service against the website. It's a really insignificant site with little traffic and no commercial value, so there can't be any commercial reason to try to knock it off the Internet. I don't think any Internet scammers sent me any threats or extortion messages asking for money to avoid a DDOS. I even looked into my spam folders and didn't see anything there. Maybe I pissed someone off on the Internet and they decided to attack back by hiring a DDOS service, but I don't think I annoyed anyone on February 28th. Perhaps I annoyed someone before then, and they only scheduled the attack for later, but that seems to defeat the point of a DDOS if it just seems random and doesn't cause me to fear that hackers are out to get me. 

So a DDOS attack was made against my website. But it was over in three minutes. I don't know why it ended so quickly. I didn't even know about it until a week later. I don't know why I was attacked. It's all just very confusing and mysterious.

CorelDraw Graphics Suite 2019 Review

Since I'm originally from Ottawa, I've always used CorelDRAW for vector graphics. This actually works out well. Since I'm not an artist or designer, I rarely need to do any vector graphics work, so CorelDRAW has worked for me because it comes with a lot of functionality, I could make a one-time purchase of a perpetual license to the software, and I was occasionally able to get good deals when buying it.

I previously used CorelDraw X5, and it did what I needed it to do, but the menus didn't work quite right on Windows 10, so I was looking to upgrade if the upgrade price ever dropped to around $100-$150 or so, but the price never dropped that low, so I just kept using my old version. Especially since I now make my own vector graphics package, I rarely needed CorelDraw except for some occasional obscure feature. Unfortunately, Corel declared that 2019 would be the last year they would offer upgrade pricing on CorelDraw, so I decided to pick up a copy of CorelDraw 2019 since it would be my last chance to get an upgrade.

I have to say that I feel a little disappointed with CorelDraw 2019. CorelDraw has always been a buggy piece of software. But usually it's the new features that are buggy, but if you stick with the core vector graphics stuff then it works fine. Usually, the new features would be so buggy that they would be unusable, but Corel wouldn't bother fixing it until a later version, so you would just have to pay for an upgrade to fix those bugs and get working versions of the new features. Unfortunately, it seems like they rewrote the core user interface code in this version, so now the core vector graphics functionality is buggy. I suspect it might be related to the fact that they've rewritten stuff so that it works on the Mac (previously, CorelDraw was Windows only). This is annoying to me because CorelDraw 2019 is too buggy for basic vector graphics work, but it likely won't be fixed unless I buy an upgrade to a later version, but Corel isn't going to be selling upgrades any more. I'm sorely tempted to keep using CorelDraw X5. The bugs are just little annoying little things like the screen blanking out if you scroll the window using the scrollbar, requiring you to press ctrl-W to manually refresh the screen. Groups also no longer snaps properly to grids. If you try to move a group, CorelDraw will choose one of the objects of the group (I think it's the top one?), and snap that to the grid instead of aligning the group as a whole. This makes grids sort of useless to me. CorelDraw also doesn't let you snap to grids and snap to objects at the same time. It gets confused and tries snapping to objects, and will completely ignore any possible grid snapping you can do. If basic functionality like scrolling and snap to grid don't work, then how is anyone supposed to get any productive vector graphics work done with CorelDraw?

On top of that, CorelDraw feels slow and sluggish. To be fair, CorelDraw has always felt slow and sluggish, but if you keep using an old version, then after a few years, your computer gets fast enough that it feels snappy and usable. Still, I was hoping that Corel would have left well enough alone, and stopped meddling with the old code so that it would stay fast. That's not the case. It feels sluggish. After all these years, Corel still has not learned that responsiveness is one of those magic unspoken features that make a graphics package feel good to use. Even though Corel Photo-Paint has many more features than my old copy of Photoshop Elements, I still use Photoshop Elements as my primary paint program because it's just so much faster and responsive. CorelDraw 2019 also just stops and hangs for a couple of seconds sometimes. I think it might be that the saving code is now very slow for some reason. Since CorelDraw autosaves fairly often (due to its buggy nature), I think CorelDraw will just occasionally become unresponsive as its incredibly slow autosave happens.

In the end, I feel like I've wasted my money. I bought CorelDraw 2019 because it was the last upgrade version they would offer. But CorelDraw 2019 is really buggy and not very usable. These bugs likely won't be fixed until a later version of Corel, which there won't be any upgrade pricing available for. Every time I use CorelDraw 2019, I keep wanting to go back to using my old version of CorelDraw X5 instead, which I sometimes do. I think the verdict is that if you are in a rush to upgrade CorelDraw because it's a last upgrade version available, DON'T get CorelDraw 2019 because it's too slow and buggy. If you can find an upgrade to an older version of CorelDraw, that might be a better choice to buy actually. Otherwise, just stick to your old version.

Copying Hard Disks with Bootable Windows GPT Partitions Using Linux

This is just a placeholder blog post. I keep intending to do a proper blog post on this topic, but I never get around to it. Unfortunately, I always forget the steps I need to do when I'm in the middle of copying my hard disks and can't consult my notes, so I'm going to just put some placeholder notes here and flesh them out properly later.

Note: Copying hard disks is tricky. I disclaim all responsibility if you use these steps and you lose data or your BIOS becomes corrupted or whatever. This blog post mainly serves as notes to myself.

About GPT Partitions
I still don't fully understand how a modern UEFI system boots from GPT partitions. I think with GPT and UEFI, your hard disk contains multiple partitions. One of them is a FAT partition that's special because it contains some boot loader programs that contain the instructions needed to load an OS from one of the other partitions. That partition is called the EFI System Partition. The UEFI BIOS of a computer will start Windows in two ways

1. Usually, the BIOS stores the specific UUID label of the boot EFI System Partition and the name of the boot loader program on that partition to load. The BIOS can then quickly load the boot loader and then continue on to load the OS.
2. When you first install Windows, the BIOS doesn't have that information yet, so the BIOS is able to find the EFI System Partition on the hard disk itself, find the default Windows boot loader on that partition, and then run that to start Windows.

Why It's Tricky Copying Windows

Copying Windows GPT partitions is hard because

• Windows makes it hard to copy Windows partitions
• the bootloader program on the FAT partition has to be changed to load things from the new partition
• the BIOS has to be changed to know about the new bootloader

Steps for Doing the Copy

By default, Windows is configured with some settings that lets it keep the file system in an inconsistent state on shutdown (in order to have faster shutdowns and bootups), which is a bad time to copy it. I tried various methods for disabling that, but the only reliable approach seems to be to turn off hibernation entirely. You need to start a command prompt in Administrator mode. Then run

     powercfg -h off

If you're copying to a smaller hard disk, you might want to use Windows Disk Management (right-click "This PC", choose "Manage", then choose "Drive Management" under the "Storage" category on the left) to shrink your partitions in advance, but that usually never works, and Linux can shrink your partitions anyway.

Also make sure that you have a bootable version of a Windows rescue CD or the Windows installation media.

Now, you can start-up Linux to start copying your hard disk. I always use the GParted LiveCD to do this. GParted can be slow to start-up because it scans through all your hard disks very slowly to find all the partitions. I think it might also do some really slow thing with Windows partitions as well. That scanning step gets slower the bigger your hard disk is too. But I've found it to be pretty reliable. Use this to copy your partitions to the new hard disk.

msftres Partition

You may find that your hard disk has a msftres (Microsoft reserved) partition. GParted is unable to copy this partition. It is not necessary to copy the partition. It is just a partition that Microsoft reserved so that if you ever install Windows Bitlocker disk encryption, Microsoft can store the decryption code there. If you do want to keep this msftres partition, you can manually create one. First, copy all the partitions before the msftres partition. Then boot into the Windows installation CD (you did remember to make one, right?). Go into the Advanced Repair settings and get to the command prompt. Then use the "diskpart" program to create an msftres partition. I forget the exact steps. You can type stuff like "help," "help select disk," "help list partition," "help create partition msr" or something to get the exact commands you need. But you need to select the new hard disk, then use something like "create partition msr size=128" to create a 128MB msftres partition. Then, you can reboot into GParted to copy the rest of the partitions.

 

Update 2020-9-22:

> list disk

> select disk n

> list partition

> create partition msr size=128

GParted Fix-ups

GParted will make the copied partitions have the exact same IDs as the old partitions. This can be a problem if you keep both the new hard disk and the old hard disk in the same computer (e.g. when moving from a hard disk to an SSD, you might want to keep the old hard disk around in the same computer as a backup). It's not clear which boot partition the BIOS will use when starting up. And when Windows is loaded, the wrong one can potentially start up. And even if the right one starts up, the wrong partition might end up being mapped to the C drive. To be safe, if you intend on keeping both hard disks in your system, you should go in and change the UUIDs of all the partitions on either the new drive or the old drive. I'm not 100% sure about Windows recovery partitions. To have the recovery work properly on the new hard disk, it might be better to have it keep the same UUID, so then generating new UUIDs for the old hard disk may be better. But I could never get that recovery partition to work properly anyway, and I would rather have at least one proper working copy of my hard disk in case the copy goes bad, so maybe it's better to generate new UUIDs for the new hard disk.

GParted also often doesn't copy the partition labels and flags correctly. You can go in and set those manually so that they're the same on both disks. I've forgotten to do this before, and everything still seemed to work, so this might not be necessary.

Update UEFI BIOS with Location of Bootloader

Now that Windows is copied, you need to update the BIOS with which bootloader to use on start-up. If you didn't change the UUIDs of the newly copied partitions, then this might not be necessary since the existing BIOS entry for the location of the old bootloader should still work. You might need to swap hard drive cables to ensure that the new hard drive takes over the old drive number from the old hard drive. Or maybe not? 

If you did change the UUID of the boot partition, then this is definitely necessary. Open a Linux terminal. Become root by using "sudo bash". Then use the "efibootmgr" program to create the necessary entries. 

 

Update 2020-9-22:

> efibootmgr or efibootmgr -v 

to list boot entries

> efibootmgr -o i,j,k

to change boot order e.g. efibootmgr -o 2,1,3

(check man pages for other commands)

I can never figure out how to create new BIOS EFI entries using efibootmgr, so I sometimes try to use some other approach to create these new entries. Sometimes, if you start up your system with only your new hard disk, the BIOS won't find the default bootloader, so it will default to searching for the Windows bootloader itself, and it will then add an entry for it in the BIOS itself. Or you can try starting a Windows Recovery CD or installation DVD, going to the command-line repair tools, and trying to use "bcdedit", "bcdboot", or "bootrec /rebuildbcd" to do this. I'm not sure what these programs do, but I think one of them will create a new EFI BIOS entry for the bootloader partition.

Once you have an EFI entry in the BIOS for the bootloader, you can go back to using "efibootmgr" to rearrange the order of your boot entries so that it comes first, which is a lot easier than creating a new entry from scratch.

Update Bootloader with New Location of Windows Partition

Now the BIOS can find the bootloader to start loading Windows, but the bootloader may not point to the correct partition to actually start the OS. Again, you can try starting a Windows Recovery CD or installation DVD, going to the command-line repair tools, and try to use "bcdedit", "bcdboot", or "bootrec /rebuildbcd" to do this. Again, I'm not sure what these programs do. I don't actually know what the BCD is, and the Microsoft documentation is very vague on that fact. I think it refers mostly to the configuration files for the bootloader on the EFI FAT system partition, but I don't know. Sometimes, everything has gone bad, and you need to use "bcdedit" to start a completely new BCD store. In any case, after randomly running some combination of those programs, Windows will somehow fix itself and become bootable.

Checking Windows

When you do manage to successfully boot into Windows again, going into Drive Management to make sure that the correct drive is listed as your boot drive and that it is the C drive. You might also need to manually remove drive letters from some of your recovery drives and other drives. Don't forget to reenable hibernation by going to the command prompt as an administrator and using

     powercfg -h on

Fixing Up Your Linux Bootloader

I normally use Windows, but I keep a copy of Linux on my drives for occasional use. Normally, I just let Linux install a boot loader into the EFI system partition and add an entry to the BIOS. I change the boot order to normally boot to Windows, but I use the "boot from alternate drive" key on startup to show all the EFI boot entries, and then I manually choose the Linux one. 

After copying a Linux partition to a new hard disk, you have to reinstall the grub bootloader on the EFI system partition to point to the new Linux partition. Reinstalling grub is mostly impossible, so I find it easier to simply reinstall Linux over the old version (I keep my Linux data on a separate /home partition, so that it's safe to do that without losing data).

Transcoding Some Videos

One of my websites has some videos on it, and I usually just embed some YouTube videos there. You don't have to pay for hosting it, YouTube takes care of encoding the videos so that it can be used on multiple devices, and you can potentially get some views from people searching for stuff on YouTube. But recently, I've started to get concerned about embedding third-party widgets like that. It's a little unclear how compliant my website can be with its privacy and cookie policy if these third party widgets can change their own cookie and privacy policies at will.

So I looked into what's involved in hosting the videos myself. It turns out the hit from hosting the videos myself wouldn't be too bad. Since the videos were video slideshows, it turns out they actually compress really well. I played with different ffmpeg, and I found that I could drop from the 50MB files that my video program produced to 5MB files by using two passes, variable bit rate, and a large maximum duration between key frames.

Now the second problem. There are two main video formats on the web: webm and mp4. Apple owns patents on mp4, and they purposely refuse to support any video formats except mp4 on their devices so that anyone who wants to provide video content to Apple users must pay for Apple's patents. I couldn't just use ffmpeg to transcode my videos to mp4 format because it doesn't come with a proper patent license (licenses are needed to decode and encode the h.264 video and AAC audio). I tried scouring around the Internet for a properly licensed version of ffmpeg that I could buy, but I had no luck in this. I could have just purchased a whole new video program with its codec packs, but it's hard to tell whether the codecs that come with a video program would expose the tuning parameters I needed to get the small sizes I wanted.

In the end, I went with a cloud transcoder since they presumably purchase a patent license for their services. It turns out most of the cloud transcoding services have gone bankrupt, so there's only a few big ones left like Amazon Elastic Transcoder, Zencoder, and Telestream cloud. Initially, I was leaning towards Zencoder because they were pretty upfront about the fact that they let you set all the ffmpeg parameters yourself and they said they 2-pass encoding. But the system seemed sort of messy--you needed to copy your files into s3 and give them read rights to it. At that point, it seemed easier just to go with Amazon since I already had an account with them. At first, I couldn't get the Amazon stuff to start, but apparently, the Amazon Transcoding console won't start until you upload a video file to s3 first, which is a little bizarre, but whatever. In the end, Amazon actually exposed the parameters I needed to get my slideshow to compress well, and the final file sizes seemed to be competitive with what I was getting from 2-pass encoding with ffmpeg myself, so I suspect that Amazon must be enabling that but not saying it in their documentation. The web interface requires you to manually enter in the settings for every single file you want to transcode, which is a pain, but I only had 15 videos or so, so it wasn't too bad. It's possible to script the transcoding using their APIs, but I was too lazy to do that.

Actually serving mp4 videos on a website also requires a patent license (separate from the patent license for encoders and decoders). Fortunately, I was offering free educational Internet videos, and those are exempt from royalties.

It's sort of annoying that the technical aspects of putting a video up on my website only took me an hour or two to figure out, but the process of trying to figure out how to do so legally ended up taking two days. I really hate how Apple is doing everything possible to sabotage the web and extract maximum profit from it--they patent important parts of the HTML specification, they refuse to support formats that can be used without patents, they refuse to support new standards in their browsers if it makes them competitive with apps--it's just ridiculous sometimes.

LibreOffice vs. OpenOffice

If you're using Windows, use OpenOffice. If you're using Linux, use LibreOffice.

OpenOffice was stagnating a while ago, so I switched to LibreOffice. Both systems were a little sloppy back then, but LibreOffice seemed a little nicer. When OpenOffice was revived as Apache OpenOffice, I tried it out, but it seemed to have poor support for file formats and it couldn't import some of my old LibreOffice files (even though they're both supposedly using the same standard ODF file format), so I stayed with LibreOffice.

But LibreOffice has always been problematic for me on Windows with lots of features not working quite right for many years. I just tried OpenOffice right now, and it just "works." No rendering artifacts. No problems with PDF export. It just works. The download and installation experience isn't quite as polished as with LibreOffice, but it just works.

I think the LibreOffice developers are mostly Linux developers who work for Linux distributions, so it is just works better there and is better integrated with those distributions. OpenOffice is used in commercial Windows products, so the developers make sure it works properly on Windows.

Update (2018-3-10): Development on OpenOffice has mostly stopped, so I decided to try the latest LibreOffice 6. PDF export was still broken on Windows. Performance on Windows has now gotten so bad that it was painfully slow to just type up some text on some slides for a presentation. I went back to OpenOffice.

Getting Type 1 Postscript Fonts from pdflatex

For the past several years, I've been using pdflatex to generate PDFs from my LaTeX, and it usually works like a charm. All modern versions of pdflatex correctly use type 1 (vector) postscript fonts and not type 3 (bitmap) fonts.

Just today, I was generating a PDF for a camera-ready, but whenever I looked at the list of fonts in the document, I kept seeing type 3 fonts there. I couldn't figure out where those type 3 fonts were coming from. I initially thought they might be from some figures, but that ended up not being the case. After playing with my document for a while, I discovered that the problem was that I was using some non-English characters, which was causing problems for LaTeX. Apparently my default installation of LaTeX had type 1 fonts that only covered the standard English character set (plus some accents). When I used characters like French guillemets quotation marks, LaTeX would need to use type 3 fonts because those were the only versions of the fonts with those characters. Using \usepackage[T1]{fontenc} did nothing since my LaTeX installation simply didn't have the necessary fonts.

Fortunately, there is a package called cm-super that seemed to contain the necessary fonts. I couldn't figure out how to coax my Fedora TexLive distribution to download this package, so I tried to grab it in Windows using MiKTeX. Unfortunately, the package is fairly large, and MiKTeX kept killing my computer while trying to download it. The download would start quickly, but gradually slow down to a crawl while my cpu utilization would go upwards. I suspected MiKTeX was doing something silly like storing the file in memory, but in a linearly expanding buffer of some sort. My Thinkpad X61t has always had heat problems, and running with high cpu usage for such long periods kept causing my laptop to overheat and die before the download would finish. Fortunately, I was able to download the packages plus some index files directly from a mirror, and get MiKTeX to load the package locally instead of from the Internet. Some people also needed to do the usual MiKTeX oddness to get things to work right afterwards.

But everything seemed to work ok once I had cm-super installed. My PDFs contained only type 1 fonts. I really don't have a strong understanding of why cm-super isn't included in my default LaTeX installation. Are the fonts too big? Is the quality of the fonts not as good as the default ones? I have no idea. But it looked ok to me, so I'm going with it.

Upgrading from Windows Vista to Windows 7

I recently did an upgrade from Windows Vista to Windows 7, and I encountered a few difficulties with the install process, which I documented here.

Windows Mail is Missing

In Windows 7, they removed Windows Mail. Although Windows 7 can still run the Windows Mail program, the upgrade process will uninstall Windows Mail, which is somewhat annoying (I'm not sure where the Windows dev team started having the great idea of uninstalling functionality that was working fine before the upgrade, but I think the new Windows team has some weird priorities that previous Windows teams didn't seem to have).

During the upgrade process, Windows 7 will tell you that it will keep your old Windows Mail settings around, and it will be easy to install Windows Live Mail Desktop later. This is untrue. Windows Live Mail Desktop will automatically import your old Windows Mail settings only if it detects that you have Windows Mail. Of course, Windows 7 uninstalls Windows Mail, so Windows Live Mail Desktop will not automatically import your old settings. So if you use Windows Mail in Vista, you should install Windows Live Mail Desktop FIRST, have it pick up all your old settings, and then upgrade to Windows 7 SECOND.

If you didn't do this, then you have to manually re-enter your account settings, then import your old mail from C:\Users\[userid]\AppData\Local\Microsoft\Windows Mail, and then go into contacts to import your old contact list from Windows Contacts.

East Asian Handwriting Recognition

The whole reason I upgraded to Windows 7 was because the Chinese Handwriting Recognition was a little flaky. It didn't integrate properly with Internet Explorer and Google (it kept losing focus and changing its language), and it was hard to resize the input panel, etc (it, oddly, seemed to work fine in Firefox though). I was hoping that they finally fixed this problem in Windows 7. And they did. They removed East-Asian Handwriting Recognition support from Windows 7. Instead, they want me to shell out an extra $150 for Ultimate in order to get access to this feature, which they removed during my upgrade to Windows 7. To the product manager who earned a nice bonus for thinking of this way to increase Microsoft's revenue: "Thanks a lot, asshole." I remember a time when Microsoft was mean and dangerous, but at least they would try to do right for their customers. Nowadays, their main areas of innovation seem to be finding fiendish ways to increase revenue.

Fortunately, Windows 7 still includes limited East Asian handwriting recognition in the IMEPad (to be removed in Windows 8, no doubt). And it seems to work more reliably than in Windows XP or Windows Vista. The instructions for enabling the Chinese handwriting recognition in IMEPad are the same as for Vista.

No Going Back

I was so annoyed with losing all this functionality during the upgrade that I was going to revert my upgrade back to Windows Vista. Windows has a long history of allowing you to undo even really invasive Windows upgrades (e.g. you can uninstall Windows XP and revert back to Windows 95 after such an upgrade). But, apparently, they stopped providing this functionality ever since Sinofsky took over the Windows Vista team (it might also be related to Windows Activation). So I'm stuck with it. I'm starting to think this will be the last Windows upgrade that I will ever do though, so I'm not too annoyed.

Stitching Together Scanned Images

Recently, I was scanning a lot of legal sized documents, but my scanner could only scan A4 sized paper. So I had to scan each document in parts and then stitch them together afterwards.

There is lots of software available for stitching together photos into a panorama, but surprisingly, I had trouble finding software for stitching together flat images (or for creating a "linear panorama" as some people call it) even though this task would supposedly be a little bit easier than making panoramas. Of course, it's possible to manually load up the images into a paint program and manually align them and blend them, but I had to stitch together a lot of documents, so this was too time-consuming for me.

In the past, I've used software like Hugin, Panotools, or Autostitch for stitching together photos, but I could never quite get the various tutorials on how to handle linear panoramas to work quite right.

I own a copy of Photoshop Elements 6.0, which does contain algorithms for this sort of stitching (under File...New...Photomerge Panorama). Initially, I was quite pleased with the results, but after inspecting the final stitched images more carefully, I began to realize that the algorithm had lots of problems rotating images in order to get good alignment between them. If you only superficially inspect the final result, everything seems fine because Adobe seems to use some sort of fancy blending algorithm (something like Enblend) that hides a lot of these imperfections. Unfortunately, the blending algorithm can't hide the fact that because of poor alignment, lines that should be straight aren't. Even when you manually rotate the images to help the alignment algorithm, Photoshop Elements doesn't help you try to fine-tune the rotation or positioning afterwards, so it's hard to get things aligned quite right (I was hoping that if I could get things close to alignment, the algorithm would "snap" things to the best position). Adobe is always tweaking their algorithms, so it's possible that later versions of Photoshop Elements give better results though.

In the end, I actually found a research tool from Microsoft that provided excellent automated results. Even better, Microsoft's Image Composition Editor was very fast and very easy to use, so I highly recommend this program. Eventually, I hope to figure out what I was doing wrong with Hugin because it's always useful to have an alternative that supports a lot of manual tweaking, but so far the results of the Image Composition Editor have been so good that I've only needed to resort to alternate tools (i.e. Adobe Photoshop Elements) only once, which occurred when I was stitching four images together that had only a small amount of overlap.

Vista Service Pack Installation Problems

I recently installed Windows Vista SP2, and like Vista SP1, I seemed to have problems getting it installed properly. It would spend a huge amount of time installing itself, reboot, and then fail during its third stage, and then spend a lot of time reverting the install. Windows Update would show Code 80004005 in the error details field for why the update failed.

Fortunately, after browsing the web, I found a forum post (I no longer remember where it is), where people mentioned that the problem comes up with dual-boot machines. My machine has both Linux and Windows installed, and Vista apparently fails its TPM security checks or something like that if the GRUB boot manager starts up in-between the computer starting and Vista starting. Fortunately, I installed GRUB on my Linux partition instead of my MBR, so I could simply change my active/boot partition to boot directly into Windows. Afterwards, the service pack could apply itself, and then I could boot on a Linux live CD to reset the active/boot partition back to the Linux partition.

Personally, I find this sort of annoying. It would be nice if Windows wouldn't die on dual-boot machines or give users a prompt to allow then to ignore TPM problems during the boot process or, at the very least, give more useful error messages.

MTU Problems When Using Vista Internet Connection Sharing (ICS) with an XBox 360

So like other people, I don't really like the idea of paying $100 for an XBox 360 wifi adapter, so I'm using Internet Connection Sharing through my laptop to get my XBox connected to a wireless access point (yes, there are easier ways of doing this, but like usual, I'm doing something strange).

But when I did this, the XBox kept complaining that my MTU was set too low since it needed an MTU of 1364. This is a little silly since a proper IP stack should automatically fragment its packets to accomodate the MTU, but whatever.

I ran the command netsh interface ipv4 show interfaces to find out what the MTUs on my Vista laptop were, and they were all set to 1300. This seemed a little strange because 1300 is oddly low and also a nice round number that a human must have decided on. But I certainly hadn't set my MTU to 1300, so I'm not sure why it was set to that.

I spent a while browsing around, but finally I figured it out. It was that annoying Cisco VPN client on my laptop. I have no idea why they set the MTU so low (what sort of intermediate router would use 200 bytes of overhead per packet? It's silly to reserve so much), but the VPN Client comes with a Set MTU program that I was able to use to set everything back to normal, and then my XBox stopped complaining.

Chinese Handwriting Recognition in Windows XP

If you want to write Chinese characters in Windows XP, there are lots of options available like Google's IME, but a lot of these options aren't practical if you don't know any Mandarin. There are websites for inputting Chinese in Cantonese, but still it isn't ideal.

Apparently, the IME in Windows XP can be upgraded to support limited hand-writing recognition in Chinese, but Microsoft conveniently hides the IME upgrades on its websites. Fortunately, I found a forum, which describes how to download these upgrades from the MS Office website. Once I installed these upgrades, I could then use the IMEPad to write Chinese characters using a mouse or tablet. In fact, the IMEPad option was available in my plain version of Windows XP, but clicking on the IMEPad button did not result in anything happening.

Resolution Independence in Vista

I recently got a Thinkpad notebook that crams a 1400x1050 resolution onto a tiny screen. It's a great screen, but unfortunately everything ends up being a bit too small. If you have resolution independent applications, the increased resolution can be a real joy though because the text becomes so clear and easy on the eyes. Although previous versions of Windows had very limited support for resolution independence, things supposedly had improved quite a bit in Vista.

Unfortunately, things don't quite fit together that well. Just like earlier versions of Windows, you can increase the default font size in Vista. Unfortunately, this causes various glitches in things. For example, text no longer fits in the dialog boxes of Windows Media Player (I suspect that the Windows Media Player people must have written some custom dialog box code because I'm pretty sure the default dialog box code scaled things correctly even back during the Windows 95 days). And lots of fonts on web pages are specified in terms of pixel sizes, and their sizes don't get adjusted, so the text ends up being too small to see. Internet Explorer does have a zoom capability that redefines pixel sizes, but it also increases the size of the default font, which has already been increased, making things look too big.

Vista also has the option of simply scaling up all pixels to a larger size. Basically, Windows will fool legacy applications into thinking that the screen has a lower resolution, so that the applications render at a smaller size. This render is then scaled up to fill the larger resolution. Resolution independent applications, however, can render at the full resolution. Unfortunately, Adobe (Acrobat) Reader is considered a legacy application. Being able to read .pdf documents with crisper text is one of the main reasons I chose the high resolution screen, and you lose all those benefits if you render everything to a lower resolution and then scale it up. I imagine the same problem crops up in other applications as well. For example, Internet Explorer doesn't have strong support for vector drawing formats like SVG, so it's not possible to get high resolution graphics on web pages, so you have to live with scaled up versions of bitmaps instead.

Based on this experience, I imagine it will take another few years before it becomes practical to use Windows comfortably on a high resolution display. The new version of MacOS will supposedly have strong support for resolution independence, but I doubt that they've actually managed to solve the problem in a comprehensive manner. I think that doing resolution indepence right probably requires a larger rethinking of UI widgets and layout managers (basically, it will be the same technology needed to layout a single UI design on both tiny mobile device screens and large LCD monitors with needing lots of manual tweaking), but I haven't really heard much about research into that area by Apple, so I doubt they've hit the final solution yet.

3D Tools for the Casual 3D Artist

Every few months, I get the urge to learn how to do rudimentary 3d art, so I do a quick tour of 3d tools available for casual users, and I always come back disappointed. For casual users, it's important that tools be cheap, not require users to have artistic talent in order to create simple objects, and be easy to use.

Immediately, the big, mainstream 3d modelling packages can be ruled out because they're too expensive. Of course, it's possible to use illegal copies of these programs, but I prefer not to do that. Then there are the mid-tier tools, but I'm always hesitant to shell out several hundred dollars for little-known tools from little-known companies that may or may not be any good.

This pretty much leaves low-cost and open source tools. The tool with the most mindshare currently is Blender. Unfortunately, Blender is a good example of a software application where little or no attention has been paid to the user-interface. If you read the development docs, this was an intentional decision on the part of the developers. For example, up until a couple of months ago, Blender proudly touted the fact that it had no undo functionality (telling you to save often instead). Personally, I'm tempted to use terms like "amateur-hour" here, but honestly, if I were a professional 3d artist who had just shelled out thousands of dollars for this tool, I would probably be ok with learning all of the weird UI quirks and hotkeys needed to be productive with the tool. Since I'm not a professional 3d artist and the tool is free, it's probably unfair of me to apply my standards on the tool. I have made a few honest attempts at trying to learn Blender, but all the hotkeys made my head spin and it seemed to crash quite often for me. Plus, based on the tutorials, it seems like users need powerful 3d visualisation skills to make effective use of Blender--skills that I don't possess. There's no way I can think in terms of "take a sphere, extrude this section while rotating it like so, perform a lathe on that object, and then edit the mesh." At best, I can look at a 3d object and play with the vertices a bit to get the shape that I'm looking for.

Milkshape3d is another popular 3d tool, and I've been tempted on many occasions to shell out the cash to go buy it. Milkshape3D is nice because it does only a few small things, but it does those things reasonably well. Unfortunately, every time I'm about to mail away my money to Switzerland, I start to think about the tedium of clicking in multiple windows everytime I want to add a single point to the model, and how I always get confused when I have to keep looking at different windows to know where I am, and I decide to wait for something better. I'm probably being hopelessly optimistic because manipulating 3d geometry on a 2d screen with a 2d input system (the mouse) is, by definition, hopelessly complex. But still, when I've used scaled back versions of professional tools (e.g. GMAX), they always provide nice facilities for manipulating polygon vertices without much hassle, so maybe that's what I'm looking for.

Art of Illusion seems to be reasonably powerful, but I think its 3d graphics isn't hardware accelerated, so it always seems a little sluggish to me. Plus, I can never figure out what the icons mean and the tooltips take too long to show up, so I spend much of my time holding my mouse pointer over icons to figure out which one does what I want.

Recently, I tried Wings 3D, and I really liked it. It's possible to perform lots of complex operations with complete ease. Pretty much all of the UI is self-explanatory and mouseable. The main advantage of the UI is that since everything is explained on-screen, you only have to memorize five things to be productive--namely what G, L, Q, space bar, and the middle mouse button do. With everything else, all you need is a general idea of what you want to do, and you'll probably be able to find it in the menus somewhere. The only problem with Wings 3d is that it doesn't work with many graphics cards. I tried it on an ATI Rage card, and an Intel integrated Extreme Graphics, and the UIs ended up being too horribly mangled to be usable. Finally, on an ATI 7500, the UI still didn't quite render correctly, but it was enough to be usable, and it was a joy to use. I was tempted to dig into the code and see if I could find any workarounds for my problems, but Wings 3D is coded in Erlang, so...yeah.

So in conclusion, if you want to dip your toe into 3d graphics on the cheap, try Wings 3D, and if it doesn't work, give up. Of course, I may be the only person who finds xfig to be an easy-to-learn program that one can quickly become productive in, so perhaps my experiences may differ from that of others.

Creating Fedora Core 3 Disk Images for User Mode Linux

So, for the past couple of weeks I've been working on creating a Fedora Core 3 disk image for UML. In the end, the process ended up being quite straight-forward, but in a case of "a little bit of knowledge can be a dangerous thing," I kept screwing it up.

Pretty much, all you have to do is to create an ext2 loopback disk image as suggested in the UML docs, and then copy a running fedora core system (using cp -r -d --parents --preserve=all or whatever) into it (obviously, you should avoid copying the disk image into itself). That's pretty much it. Labeling the disk image as /1 doesn't seem to be enough to get UML to mount the disk image properly as its root file system, so you do have to modify the /etc/fstab file so that the root file system is mapped to /dev/ubda (for some reason, the virtual drive appears as /dev/ubda instead of /dev/ubd/0 like in other images) using an ext2 format (unless you compiled ext3 support into the kernel). It's probably a good idea to enable networking in the kernel as well by turning on these kernel options:

UML Network Devices/Virtual network device
UML Network Devices/TUN/TAP transport

Then, you can just boot up UML into the image you created. You don't have to change the Kudzu configuration, change the networking to use something other than eth0, or anything. It just works. Below are the things I tried to do, which ended up being completely unnecessary:

Although Fedora Core 3 intially boots with a small initrd RAM disk image as its root file system before switching to using the hard disk as its root, this initrd is only for loading additional kernel modules. Since User Mode Linux has most options compiled directly into the kernel, it is not necessary to use an initrd image with UML. Also, it is not necessary to create ubd devices in /dev since Fedora seems to do that automatically. And hostfs mounting /sbin or /bin doesn't work out too well. You can mostly get away with hostfs mounting /usr but you have to prevent xfs from starting up (simply remove /etc/init.d/xfs) because I think it tries writing stuff into /usr, which won't work because on the host-side, UML won't have sufficient permissions to write there. I also didn't bother properly installing uml-tools, meaning that UML couldn't use the port-mapper to bring up the virtual console xterms. Since Fedora Core 3 requires you to log-in through a virtual console, you need to get the port-mapper working properly (I had to modify UML to find the port-mapper in my home directory instead of its default location). Also, Virtual Console #7 and #8 won't appear or won't show anything, so don't think the boot-up has failed if you don't see anything there. And that pretty much summarizes many, many days of mistakes.

Running User Mode Linux from 2.6.10 Kernel

I don't really know anything about mucking around with the Linux kernel (I'm the type of guy whose favourite text editor is pico and who can't figure out how to use Debian meaning he has to use RedHat instead), but somehow I signed up to do a term project involving a bit of kernel hacking, so I guess I have to learn. Since kernel stuff seems a bit messy, I opted to work with User Mode Linux (UML).

Now, when one browses around through the main website, there's a lot of stuff about ancient versions of UML based on 2.4 kernels, there's a couple of mentions of 2.6 kernels, and tucked away somewhere, there's a brief aside that UML has been integrated into Linux kernels 2.6.9 and beyond. Seeing as everything else on the main website seemed a little dated, I thought that it would be best to use the UML that's integrated into the main Linux branch because it would be more likely to be maintained and tested there. In actuality, the UML code that's been integrated into the kernel doesn't work all that well, and it took me a few days of searching through various websites, wikis, forum discussions, etc. to figure that out. So here are the goods on how I got UML working on my Fedora Core 3 system.

Apparently, the UML in the current 2.6.10 vanilla kernel doesn't work. Instead, grab the 2.6.9 vanilla kernel from kernel.org or somewhere. Then, go to Paolo Giarrusso's UML site and grab the latest 2.6.9 patch (it's in his archives in the guest patches section). After you've unzipped and patched everything, simply taking the default configuration from

make menuconfig ARCH=um

isn't sufficient. One has to enable these options in the kernel

Character Devices/File Description Channel Support
Character Devices/Port Channel Support
Character Devices/tty Channel Support
Character Devices/xterm Channel Support
Block Devices/Virtual Block Device
File Systems/Pseudo File Systems/dev File System Support

Then, you can follow the normal instructions about compiling, stashing a root_fs file system somewhere, and then using the devfs=(no)mount option appropriately.

This is enough to get UML to bring up a prompt, but that's as deep as I've dug so far. I'm still weighing whether it would have been easier simply to have started with a Debian package of UML.

CorelDraw 11 Import/Export Woes

Being from Ottawa, I'm a CorelDraw user (Corel is an Ottawa-based company). I'm not a very good CorelDraw user (I just use it to draw simple diagrams), but I'm still a CorelDraw user. Just like other CorelDraw users, I have to live with both the good and the bad features of CorelDraw. It comes with a lot of features and applications for a good price, but you have to live with the ever expanding resource usage of each version, buggy features, an under-designed user interface, and a general lack of polish. Perhaps things will get better now that they're under new management.

I bought a discount version of CorelDraw 11 a few months ago, and I use it for mostly computer science type things: presentations and LaTeX diagrams. In order to use CorelDraw in this sort of configuration, you need to do a lot of bulk importing and exporting. I just thought I would blog my approach to making CorelDraw 11 do what I want in case the one other person who uses CorelDraw in this sort of configuration is interested :-).

For example, I usually draw all my LaTeX figures in one CorelDraw document and then use a bulk export macro to export each page as a different eps file that I can reference in LaTeX. CorelDraw 11 doesn't let you configure eps export options from a macro, so you have to export one eps file by hand first and configure the export options there, and then when you run the macro, it will just reuse the export options that you set before.

Also, pdflatex prefers to import vector graphics in the pdf format. Unfortunately, CorelDraw 11's pdf exporter doesn't support bounding boxes, so the exported pdf files do not import correctly. I found this program called eps2pdf (I use the one that comes with a nice Windows GUI because I can't figure out how to invoke the preferred eps to pdf conversion utility "epstopdf") that can take eps files exported from CorelDraw and converts them into pdf documents that can be imported into pdflatex. The program chokes if the Corel-exported eps files have text converted to curves or if the Corel-exported eps files are too large. It's possible to invoke eps2pdf from the command-line to convert all eps files in a single directory, which is useful.

Just recently, I've tried to import a lot of xfig-created eps files into CorelDraw 11. This sounds a little odd, because it doesn't make sense to create diagrams in xfig when you own a copy of CorelDraw. The reason I needed to do this was that I needed to generate a lot of complicated machine-generated Postscript diagrams. I don't understand Postscript, but xfig generates very clean, easy to understand Postscript output. So when I'm in this situation, I draw a couple of simple primitives in xfig, export it as eps, then write a program that follows the template generated by xfig in creating its own Postscript diagrams. Unfortunately, Corel's eps importer imports the text in xfig eps files as curves instead of as text. I managed to side-step this problem by using ps2ai.bat to convert the eps Postscript files into ai Postscript files. CorelDraw 11 then imports the text in these ai files without problem. Unfortunately, somewhere along this conversion process, extra outlines get added to objects. It's possible to simply delete these outlines by hand (especially easy if you use the ObjectManager view), or you can import the xfig eps files directly, copy all of the non-text objects, and then use these objects to replace the non-text objects in the imported ai files. There's probably an easier way to do this, but I'm still trying to figure it out.