Friday Afternoon Adobe Rant

I have to join the voices today that are complaining about Adobe. I'm building a multi-year 64 bit server to run our Firefox sessions for hundreds of people. It should take me 5 minutes to install the PDF and Flash plugins. But guess what, it's 2010 and they still don't support 64bit platforms. So now I have to try and wrap them to run in 32bit mode which is uncovering bugs when used with Firefox 4. The beta 64bit Flash player seemed to show some promise, but I can't deploy it because it's too old in terms of security exploits.

So Adobe, you have a big market share with Internet content. You need to step up to the plate and make this content easier to deliver. If you can't, the world is going to move past you quickly.

Note: Yup I know about the open source projects to replace these products. They aren't a fit for us at this time.

Accessing Remote Devices From Centralized Hosts

What is interesting to me whenever we look at software and hardware to solve problems is the needless level of complexity and cost. Let me explain. When you meet with vendors, their first inclination is to add lots of hardware to your infrastructure....and other Cities must just buy lots of it because they seem shocked that we question their solutions and want to keep it simple.

In this particular case, we are adding the ability for our Recreation department to put photos of citizens on their recreation cards. When you review cameras, you begin to hear, "Well you will just add a PC at each remote site to connect to the camera and process the photos". What?!? Huh?? It's very clear that this is unneeded and adds support and upgrade costs. I think sometimes these vendors fail to understand that if you deployed in their design, you would have 100 extra PCs on your network in no time.

So how do we fit in something like a remote camera into a thin client environment? I'm a strong proponent of centralized software and only having very minimal hardware in the field. I didn't want to get a usb webcam that needed drivers and software on the thin clients. Too many problems, and keeping up with ever changing hardware is very difficult. Very often we buy a few devices, and within a few months that same model isn't available anymore. Touching and QAing thin clients and updating them with drivers is not a good idea. I also didn't want the hosts to communicate via USB to remote sites. We always use stateless connections to keep things simple and stable.

So I honed in on the Cisco/Linksys WVC80N Internet camera. Ok, so some problems were solved. We can use the motion sensors, and our employees at the remote sites won't have to push any buttons. All they have to do is sit down the citizen and the photos are taken automatically. Very cool. The camera snaps the photos on motion, and then uploads them via FTP. Because some sites are on lower bandwidth lines, I decided not to send the photos back to our servers. Otherwise anytime there was movement photos would be uploading constantly and possibly slowing their performance. Ever see how many kids run around at recreation sites? :) I wanted to only send the photos when they are really needed by our host software.

It was a logical progression of thought then to house the photos at the remote site until they are used. But where? It turned out that we could store them on the thin clients very easily. We already use FTP to transfer files back and forth from USB sticks to the server to ensure that all file transfers and connections are stateless. So what I did was create a "fake" USB stick in /media that is really a RAMfs (tmpfs) with 20MB of space. This was to avoid having to have a USB stick plugged into the thin client to house the pictures. It was not desirable to store the photos on the flash drive of the thin client itself, these drives have a fixed amount of writes.

When someone moves in front of the camera, it shoots the pictures and puts them on the thin client located at that site via FTP. When we then need those photos from our host based software, we FTP then off the thin clients and back to the server. Clean, solid state and hardware independent. When this camera is discontinued, all we have to do is buy the newer model and drop it into the same location.

Total cost to take photos of our citizens, $110.

Projects Keeping Me Busy

Life at Largo never slows down. When you push new technology live, there is always more waiting in the wings. Here is what is going on:

Evolution 64 bit: As I have reported everyone has been moved to release 2.28 on SLED 11 SP1. Still blazingly fast even with hundreds of people working. I moved the bug-buddy binary out of the way and inserted a custom script that dumps backtraces to flat files and into a common directory. I can see how many people, what people and why people are crashing in real time. Looks like we are hovering around 20 crashes a day that dump BTs and a few more that crash at startup. Around 400 unique people check email a day, so most people are having a stable experience. BTs are being shipped to Novell and we hope to start getting some patches soon. Those patches should also land upstream and help everyone else too.

OpenOffice 64 bit: 3.2.1 is working well and stable. It's still running so fast that even under a user load you don't see the splash screen at all. On a multi-user system you never really have a "cold start" but the UI opens in about 2-3 seconds at which point you can immediately start typing. Nice. We have identified that extensions are causing the memory leak, and hopefully Sun/Oracle will be able to work on it soon. In the meantime we just bought more memory (cheap) to keep everyone out of the swap device.

Firefox 64bit: Pavel Janik finished testing our new 48 processor (8 cpu x 6 core) server to try and set some new OpenOffice compilation records. I won't spoil the surprise and you can read his blog when it comes out. I'm downloading OpenSuse 11.3 right now and we will move that server back inside the DMZ and prepare it for our employees. I'm going to begin testing 64bit Firefox 4 and see how it works. Adobe seems to have pulled back 64bit flash, so I'll be experimenting with the various plugins and see how they work. I'll check on the state of the various media players and see which one works. We used mplayer last time and it works great, but I always keep an eye on other solutions.

Debian Lenny Xserver Crash On Thin Clients: I finally packed up a bug report on the crashing Xserver issue. It only happens about every 2-3 weeks, with compiz enabled and happens when you close a child window. Bug report is here

Verizon USB 760 EVDO Modems + Laptop Thin Client: Verizon released a nasty modem, the USB 760. They thought it would be a good idea to put software and a modem into one USB stick, nice. Linux sees it as /dev/sr0 and tries to automount what it thinks is a CD/DVD drive. The workarounds are not pleasant, you have to tell udev to then immediately eject the drive and then attempt to fire up your dialer. I'm poking around at some ideas to make it not so clunky. Note to Verizon: Please don't make hardware devices like this anymore!

GDM + XDMCP: Halfline and I are trying to get synced and have some time to experiment with newer versions of GDM and find out why it won't let thin clients connect with XDMCP anymore. Hopefully that will happen in the coming days.

Update: 2 second cold starts on Firefox on the 48way server. Obviously no user load, but promising. :)

A Word About My Book

A few years ago I was asked to write a book and some of you have purchased it. Obviously one hopes that a book always meets the expectations of the reader, but there have been some negative reviews. Some of the goals while it was being written were not entirely matched to what people thought they were buying. Let me explain, and tell everyone what is in the book:

Obsolete Technology: When I was changing cubicles a few weeks ago all of the contents had to be packed and moved. I was shocked at the high number of books that were completely obsolete. I had a RedHat 6 book from 1999 and on inspection you find that nearly every technology has completely changed in 10 years. I wanted to create something that had a longer shelf life. In the time since the book was published, HP has already released two new thin client models. Had the book dug deeply into the internals of the model available at that time, it would have been just another technical book sitting around. So I wanted to include some topics that never change.

People Issues: The book deals a great bit about *people*. We as techies sometimes don't like to think about them or talk to them :) , but they are the biggest issue that you have to overcome when you move to thin clients and even more so when you move to Linux. Yeah, it saves lots of money, for sure...but if your administrative people aren't on board any project such as this is doomed. The book also deals with the fact that IT staff will have greatly changed duties post migration. You no longer have lots of people running around keeping client devices running and focus moves to R&D and centralized support. Instead of maintaining, you are improving and that sometimes requires different types of people.

Pictures And Pictures: All of you would be surprised at the number of people that think Linux is "character" and "command line". Many people think that if you move your email to Linux that you have an xterm window open with pine or something. During the editing process they asked me about including the screen shots, and I felt strongly that we needed to SHOW people applications like Firefox, Evolution, OpenOffice and Realplayer running graphically and similarly to how they look and work on Microsoft Windows. People also have this impression that if you have to get to Microsoft Windows applications from the Linux desktop that you have to log out in some manner and then connect to the various software packages. I felt it important to show that everything runs in a consolidated desktop, Windows and Linux apps right fine side by side.

Unique Site Requirements: Every deployment will be completely different, so it would be impossible to give exact instruction on how to setup and configure everything. So the book had to be about 'ideas'. I tried to describe exactly what was possible, and then leave it the reader to come up with a plan. I had no way of knowing the skills of the staff, the skills of the users, the money available and exactly what operating systems would be used. I also felt that more specific details would be available on my blog. I always try and blog about major milestones that we reach and hope that it's helpful.

There are good days and bad days on any computer system, but we have inched our way up to about 700 thin client workstations and things are still lightning fast. This design works, is stable and saves a lot of money.

Time Consuming Upgrades

Centralized computing makes it so difficult to upgrade hundreds of users to the latest release of the ever exploited Adobe Acrobat Reader. :)

browser:/u # ls -lad acro*
drwxr-xr-x 3 root root 4096 Jan 13 14:08 acrobat_9301
drwxr-xr-x 3 root root 4096 Feb 23 10:29 acrobat_9310
drwxr-xr-x 3 root root 4096 Apr 23 13:32 acrobat_9321
drwxr-xr-x 3 root root 4096 Jun 30 16:06 acrobat_9331
lrwxrwxrwx 1 root root 12 Apr 23 13:33 acrobat_9_live -> acrobat_9321
browser:/u # rm acrobat_9_live
browser:/u # ln -s acrobat_9331 acrobat_9_live

and then push the reader into Firefox

browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_sound/firefox/plugins/
browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_nosound/firefox/plugins/
browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_noflash/firefox/plugins/

All done. :)

64bit Evolution Is Now Live

After that small hiccup with running Evolution on 32bit Linux and the memory limitations, we are now live on 64bit SLED 11 and the difference in speed is striking. The per user memory requirements have grown, but it's a small price to pay (and memory is cheap!). The top screen below is 220 concurrent Evolution sessions. CPU is barely used at all. Disks are configured RAID 1 which has proven to be the best for Evolution (RAID 5 is much slower). When using the Groupwise backend, hitting Send/Receive the dialog only appears for a split second. The server is has 4 CPUs, quad core and I think we could easily get 500 concurrent users on here with no performance penalty. I'll have a few days of tuning and minor issues. Next on deck for me: getting gdm fixed to allow XDMCP connections again, and starting to work on that new monster server to run Firefox.