Lockdowns, Flash & Text Files

Work continues on getting GNOME 2.30 ready for our employees and making great progress. One of the benefits of this project is that we will be able to save some money and resources by reducing our server count. Right now we have two identical GNOME servers and users log into one or the other. Both servers can run our highest concurrent load (350 users), and are in place so that if we have a failure the other server is always ready to go. We have to have a backup of some type -- You can't have hundreds of people sitting around because of hardware problems. But the fact is that in all of these years, the hardware hasn't had a single failure.

Our goal on the next upgrade is to only have one server on a nice multi-core, multi-CPU server with 64bit Linux. We then will create a physical-to-virtual copy once a week and have it ready to go. If the physical machine fails, we will put the users on the VM server until the problem can be resolved. As I an designing the desktop, I'm going to write code so that if it's running on VMware certain features will disable so as to provide the best possible speed. An example will be compiz, when running virtually, that feature will disable and drop back into 2D. Another example might be to turn off thumbnailing and so on. Hopefully the VM server will never need to be used, but we will be ready and save a good amount of money by reducing one physical server replacement every 3-5 years.

Lockdowns

Very often you find a great piece of open source software to use, but it allows the users to tinker too much and change settings. Often these settings will cause the software to fail or work in a manner that we don't want. Such was the case with using gnubiff. It works great, but one bad click while tinkering and it will no longer connect to the post office. Or another issue is where users think it's wonderful to check for new email every 1 second; which does wonderful to your post office. :) So what I do is get everything working as expected and then create a template and replace the things that are specific to users with variables. In this case their account name and password are the only two things that need to be unique. The shot below shows how I then implement this design. Grab from their current .gnubiffrc the stuff I need, and then use sed and merge it into my template and give them a fresh working configuration file before the software launches.



Flash & Firefox
There are days when you feel that all of your bug reports and efforts will never advance and then suddenly you start getting updates that actually work. Such was the case yesterday when I installed Firefox beta 6 running 64bit with the just released preview 2 of 64bit Flash. Preview 1 and the earlier 10.1 64bit beta were really pretty bad on our configuration. Frames and sound were choppy, playback would block the browser from scrolling. I was actually getting better results with a wrapped 32bit Flash release, but that's so clunky to me and I'm trying to run 100% 64bit. I'm happy to report that preview 2 is working MUCH better now. Sound is strong and frames are better, even with remote display and remote sound. This gave me a chance to test the volume control. It's working great...it detects the sound card on the remote thin client from the GNOME server and has no idea it's running over the network. Excellent.

In the shot below, you can see the video playing along with a zoom of the volume slider. SiS is the sound card on the thin clients.




Text Files
One of the things that users loved on the last desktop release was the use of thumbnails for all of their documents. It really does aid in finding what you are seeking. I have an eye for consistency, and always hated that text files (.txt, .ics and .csv) have never really been appealing in Nautilus. So it turns out that 'convert' very nicely can change a text file into a picture. I created a new schema for text/plain and pointed it to 'convert' and thought it would work. It turns out that Nautilus sends %o without a PNG file extension. So convert has no clue what file type to convert into, and it didn't work. So I whipped up a quick script called text-thumbnailer. This script also takes care of the issue that if the file does not end with .txt, convert doesn't understand the file type. The script softlinks a .txt file in /tmp and generates a PNG and then passes it back to Nautilus as expected.

In the shot below, you can see the thumbnail schema, the text-thumbnail script, and a Nautilus window which contains the resulting thumbs. Scrolling through the folders now is much cleaner and consistent.



So along with various other projects, this development will continue. I'm getting positive feedback on the results and am looking forward to the coming weeks and ongoing improvements.

Beta Testers On GNOME 2.30, Making Progress

Lots of things keeping me busy, but making excellent progress on building a new server to run 64bit OpenSuse 11.3 and GNOME 2.30. Now that gdm is working with XDMCP, I was able to get the logins working. Halfline (thanks!) once again helped me with turning off the animations that were displaying when the authentication screen appears and also then as it goes away and the users wallpaper appears. This type of animation is too slow over remote display, and caused you to have to wait while the screen blinked multiple times. I'm sure it looks great on a local video card, but not suitable for here and not needed in a business/work environment.

Very nice to see enable_esd no longer required. PulseAudio is hooked right into GNOME and once I selected a sound theme with a login clip, it worked immediately. The pulseaudio volume slider bar is working great, even though the sound daemon is running remotely on the thin client. Excellent.

I was able to get beta testers logged in Wednesday and they are using this server fulltime. It's obviously not fully functional, but enough icons and applications are working to do their work.

Previously we were using 3 different techniques to connect to the various Microsoft applications around the City. Some where using Citrix, some were using RDP/Rdesktop and others were using an old product (UIS, Unix Integration Services) that has been discontinued for years. The other System Administrator has been working magic and working on standardizing everything around RDP. This makes it easier for him...just one connection package. It also allows us to fully deploy the new connection method that I had blogged about previously. This has some disadvantages, but is MUCH faster and provides much better sound.

The old technique had rdesktop running on the GNOME server itself. Connection would then be made to Windows. Windows would communicate with rdesktop via RDP, and then we used X11 and Pulse to provide the screen and sound. 100% centralized, stable, easy to upgrade and make changes (blue boxes below). The way we are deploying on the new server, the GNOME server signals the thin client to start a rdesktop session and that session then talks to Windows with RDP. The user presentation is probably at least twice was fast (green boxes below).

So as he continues to get applications ready for RDP, I'll add more and more icons and get them to the beta testers. This is providing the same integrated Linux/Windows desktop we had previously, but with lowered network usage and far better user experience.

Always have to give a screenshot, right? So here is a shot of the current status. MIME handlers are working great for the various file types. Application-browser is starting to fill in nicely with our various City applications. And I launched Internet Explorer to demonstrate how cleanly Windows apps integrate and work right off the standard GNOME icons.



Huh? You are probably wondering why we run Internet Explorer. 99% of all Internet usage is done on Firefox/Linux. But many other Governmental agencies (state and federal mostly) designed pages that are hard coded to only work with Internet Explorer. Our tax dollars at work. :)

Thanks For Ideas Guys

Yesterday when I posted the MIME bar prototype, I was given some ideas to make the dialog easier to use and fit better into current UI designs. I was mostly thinking about the desktop flow, but UI ideas are always welcomed. I whipped up a new version in Glade and connected the signals to the existing code and it's already working. It's not something that I could release because it's hard coded for our software packages, but might be a good idea for a GNOME project in the future. I guess in theory some more X-Categories could be added so it knows where to place these items. I'll be creating similar ones now for word processing files and for Pictures.

Shot below, I like that it has a similar appearance to the main-menu.



Update: A quick dupe of the screen and change of the thumbnails and the Picture and Audio/Video dialogs are starting to work. Thanks again Kevin for the awesome 'nailer' program for the videos.

MIME Ideas Continue, Fulltime GNOME 2.30

Many things are happening at once, but I have made some changes to the new MIME bars to test ideas with our users. I'm now logging on GNOME 2.30 (OpenSuse 11.3) fulltime from a thin client. I have gotten all basic applications (Evolution + Firefox + OpenOffice) running. I can start the other applications manually easily. After I installed another group of icons for our beta testers, I'll start getting feedback.

I have always wished that Nautilus gave you a hover mouse page preview larger than the thumbnails that currently appear. If you have hundreds of thousands or millions of documents many of them are very similar, especially those on letterhead. So experimentally I have created a tabbed dialog that appears when you double-click on a file. The "Options" tab allows you to select your next step. (shot below):



The Details tab shows you a big preview of the first page of the document, along with the owner of the document and the file size. Very few people discovered the "properties" found in the right-mouse of nautilus. They call all the time to find out the owner of files; so hopefully this will help. And believe it or not, but a lot of people don't understand K, MB and GB acronyms; so I'm testing the idea of displaying the file size fully expanded with lots of zeros. :)



I'm looking forward to getting feedback, and think these minor tweaks of the UI will make the system easier than ever for our employees.

Testing GNOME 2.30 & Updates

One of the benefits of starting to poke at a desktop upgrade to GNOME 2.30 is being able to take what I learned the last time and making improvements. Certain types of specialized launch scripts that were developed are being merged into a core library which will run when any icons are triggered. One such example is the gnome-panel, which still apparently treats a double-click as two single-clicks. I thought this was changed at some point, but it's still working in this manner. Users tend to double-click (or more!) on icons and are launching multiple occurrences of the same application. So I brought over and improved the code from the old desktop. Gnome-panel now detects that someone is repeatedly clicking and disregards them for a few seconds. The popup below displays:



The MIME bars that were deployed on the previous desktop release have worked well. When users double-click on files in the file manager, it asks them what step they want to perform. The previous bars had buttons that were stacked left to right, and on certain monitor resolutions we were running out of real estate. I also wanted to better categorize them into middle level functions (Edit/View, Save, Print & Email). The screen below is my first attempt and I'm installing the MIME types this very second.

This design might seem odd at first glance, but think of this situation: You get a PDF file and you know you aren't going to read it online and for sure are going to print. With a single click, it's queued and already finished. Users like having steps 1-2-3 that are consistent and simple.



I hope to have some early beta testers on this server by Thursday or Friday of this week.

Still poking at the slow printer dialog on SLED 11/Evolution to figure out what's happening. My attempts to hack out certain polling areas of the code were not successful. I'm also looking at f-spot for use in a multi-user environment, working on making it easier to pass files back and forth to USB sticks, beta testing OpenOffice 3.3 and checking into connectivity between OpenOffice and the Oracle database.

Gnome 2.30 & main-menu

It's good to see GNOME 2.30 running on our thin clients (below). My thanks to those that helped me get this working and squash a few bugs. I have been removing all of the software that is automatically installed on OpenSuse and won't be used by our users. Next up will be connecting our core group of applications (Firefox, Evolution, OpenOffice) so that I can get some beta testers logged in. I also have been pondering some updated MIME bars based on feedback and things we learned from the last upgrade. These bars will launch when you double-click on a file in the file manager and ask you what task you want performed. Everyone has gotten used to them and people really like the desktop to flow in this manner. Sure would be nice to be able to put these into Nautilus as a single click.



One very interesting usability issue that I found when reviewing how we currently have the main-menu grouped is that most of the regular users never figured out that they can type software applications into the Filter....even though the cursor focus is there when it opens and it's blinking waiting for you to type. :| The users also didn't really figure out that you can click on the Groups on the left edge to reduce the number of packages displayed. Instead, most of them are scrolling down the entire list of software and looking for what they want. I really need to ponder this more, but sure sounds like an improvement area in the design.



I'm also excited to see NX 4.0 is coming out soon, and will allow you to log into our desktop servers from an iPad. How cool is that?

Project Updates

Just back to the office after a long weekend and I thought that I would create an update on the various projects.

Recreation System
The developer and business analyst are making progress on new recreation system. Now that IT got a signoff on the receipts, the final design has been implemented. A screen was designed to allow them to add coupons on the far right side of the printout. Following our internal goal of never having users obtain a file manager, this step is performed inside the GUI itself. All available coupons display and the user just has to click on it and hit the arrow which loads it into the printout page. They are beginning the process now of implementing more robust accounting principles at this time.

(shot below, sorry Steve Perry isn't coming to Largo) :)



Evolution Print Dialog
We have had some problems with libgnomeprint which will require me to change the source. We have about 60 printers, all HP using JetDirect. They are always in various states of function; paper jams, paper outages, toner shortages, waiting on paper tray, etc. Some of them are turned off, and some of them have latency from the Internet. When you hit the print button in Evolution the dialog will very frequently gray out and then block Evolution which grays out too. After a few seconds the dialog returns and everything works. Jeff Steadfast happened to be on IRC and gave me a few areas to try to improve this situation. I'm going to comment out the call for it to poll the printers. We don't really care about their statuses and users will just send the print jobs and await their completion. Sounds like a nice gconf key idea to me for future releases.



Debian Crashes
Does anyone want to backport some packages for me? This bug is really causing us some grief. It's causing our thin clients to crash for 3D/Compiz users and some people have gone back to 2D because they are annoyed by losing the Xserver. The last instruction was to upgrade Mesa, which will require an extensive upgrade of all of the X packages too which would take a lot of QA work. If anyone has any ideas, feel free to hang them on the bug report.

Halfline XDMCP Hero
XDMCP really hasn't worked for a while now in the latest OpenSuse releases and Halfline cooked a patch which I was able to get compiled and it's working! It's been submitted to Suse/Novell and Vincent already classified the bug report (thanks!). The production "Desktop" server is running Gnome 2.16 so there are many new things for me to test over remote display since that release. I also see the wallpapers now transition/animate with a .xml file and I need to check performance over the network. I built Compiz 0.9.0 and it's crashing the Xserver a good amount of the time (See Debian Crashes) above. There is a lot to review before I can get beta testers on here; but it's very encouraging.

Yup, the final login screen will be a bit more professional. When I am hunting down configuration files, I always change them so radically to leave no doubt that I found the right settings.

Photo Taking Hardware

In a previous blog I had described the security camera that we selected for taking pictures of our citizens for recreation cards. We found a low sitting tripod for around $15, which pivots the camera at a 90 degree angle. The pictures are taken sideways and then corrected via the 'convert' command line utility in scripts to rotate back 90 degrees and appear correct on the IDs. These cameras are completely self reliant and don't need any attached hardware or PCs to run them. Total cost including tripod + camera should come in less than $150.

Recreation System, Milestone 1

Our IT/Integration division was able to go live with the first milestone of changes to the recreation system on a new server. I haven't been working on the coding on this project, but have been helping with some technical aspects including cameras, mockups, printing and ideas.

The old recreation system used older Motif widgets and was first built in the mid-1990s. We already had thin clients at that point. In the mid 90s the remote sites were in 8 bit color, 800x600 and only had limited bandwidth. The screens were built with those limitations in mind.
(shot below)




For the first milestone the front page was changed to a 'portal' which displays a lot more information. Motif was upgrade to 2.3 which gave us anti-aliased font. The theming system was upgraded as best as possible to allow GNOME colors to alter the look and feel of the software and match the rest of the applications. All of the code from the old system came over without modification; the screens just had to be touched in order to accommodate the new font metrics.

(shot below, no GUI nazis please :) )



The developers are now starting to alter the code of the screens and add functionality. The biggest new features being implemented are membership cards with photos and bar codes, and better accounting integration. If you are a citizen of Largo, these milestones will slowly increase the quality of your experience at our recreation sites. Be assured that all of this technology is being developed with costs and long term impact in mind.

Java & Sun/Oracle Blaze The Way :)

I never thought that I would such high marks to anything related to Java, but as it stands right now this is the only piece of the puzzle working natively and well on 64bit Firefox 4.0. I wanted to draw myself a picture of the pieces (below) so that I could work out in my head how this will all work. There are 4 main areas where I need to run embedded plugins based on our users requirements. 4 of the 5 plugins have caveats and are not deployed in the clean manner that I would like. Some of them are still only 32 bit, some of them don't natively use PULSE and some of them are missing features when built natively 64bit.

Green indicates satisfactory techniques, red indicates problems.



I guess for now we tell our parents and grandma to keep installing the 32bit flavors.

64bit Browser Work Continues

I was pulled off of doing Admin work for a few days to help with some technical projects and assisting in getting some pieces together for our Integration Division. I had previously noted that a system was developed for taking citizens pictures for our recreation cards. I wrote a small ksh and perl script to create the new cards. (shot below). Integration staff will make final changes and add some colors and move the boxes around slightly to fit the perforated paper; but it's all essentially working. The Panther development toolkit we use has no runtime licenses at all on Linux. So that along with the inexpensive cameras and using perl to print the cards has kept the costs to a minimum.



So now I'm back working on the 64bit 'browser' server which will run our Citywide Firefox sessions. I got mplayer compiled natively and it's playing all of our media files correctly. Nice. Now I'll work on getting the multimedia plugins loaded into Firefox. We use the RealPlayer (Helix) to play our commission meetings, but sadly this too seems to be missing 64bit support. Will check for source code, maybe I can build it myself. I guess I have come to the realization that in the short term, I certainly won't have a nice clean 64bit implementation; I'll load as much as I can and upgrade as more packages are released in the future. It's all super fast, the users won't see much of a difference.

I tried loading Chrome as downloaded from Google and there were some dependency problems. I found a release from OpenSuse (11.3) and was able to get it installed natively. What is interesting is that I have found it to be very SLOOOOW. Firefox is very responsive on this server; cold starts in about 2-3 seconds and pages load quickly and spinning the mouse to scroll a page is crisp and works as expected...even over remote display. Chrome feels like page rendering is fighting mouse movement, and spinning the mouse wheel is slow and not very smooth. When you grab the scroll bars, they seem to be jerky and slow. Possibly no one is testing Chrome over remote display, but as it stands right now it's not even close to be usable in our environment.



Work will continue over the next few days with plugin testing and loading more software. I should be able to put on some beta testers in the coming week or so and begin testing printing and printers.

Friday Afternoon Adobe Rant

I have to join the voices today that are complaining about Adobe. I'm building a multi-year 64 bit server to run our Firefox sessions for hundreds of people. It should take me 5 minutes to install the PDF and Flash plugins. But guess what, it's 2010 and they still don't support 64bit platforms. So now I have to try and wrap them to run in 32bit mode which is uncovering bugs when used with Firefox 4. The beta 64bit Flash player seemed to show some promise, but I can't deploy it because it's too old in terms of security exploits.

So Adobe, you have a big market share with Internet content. You need to step up to the plate and make this content easier to deliver. If you can't, the world is going to move past you quickly.

Note: Yup I know about the open source projects to replace these products. They aren't a fit for us at this time.

Accessing Remote Devices From Centralized Hosts

What is interesting to me whenever we look at software and hardware to solve problems is the needless level of complexity and cost. Let me explain. When you meet with vendors, their first inclination is to add lots of hardware to your infrastructure....and other Cities must just buy lots of it because they seem shocked that we question their solutions and want to keep it simple.

In this particular case, we are adding the ability for our Recreation department to put photos of citizens on their recreation cards. When you review cameras, you begin to hear, "Well you will just add a PC at each remote site to connect to the camera and process the photos". What?!? Huh?? It's very clear that this is unneeded and adds support and upgrade costs. I think sometimes these vendors fail to understand that if you deployed in their design, you would have 100 extra PCs on your network in no time.

So how do we fit in something like a remote camera into a thin client environment? I'm a strong proponent of centralized software and only having very minimal hardware in the field. I didn't want to get a usb webcam that needed drivers and software on the thin clients. Too many problems, and keeping up with ever changing hardware is very difficult. Very often we buy a few devices, and within a few months that same model isn't available anymore. Touching and QAing thin clients and updating them with drivers is not a good idea. I also didn't want the hosts to communicate via USB to remote sites. We always use stateless connections to keep things simple and stable.

So I honed in on the Cisco/Linksys WVC80N Internet camera. Ok, so some problems were solved. We can use the motion sensors, and our employees at the remote sites won't have to push any buttons. All they have to do is sit down the citizen and the photos are taken automatically. Very cool. The camera snaps the photos on motion, and then uploads them via FTP. Because some sites are on lower bandwidth lines, I decided not to send the photos back to our servers. Otherwise anytime there was movement photos would be uploading constantly and possibly slowing their performance. Ever see how many kids run around at recreation sites? :) I wanted to only send the photos when they are really needed by our host software.

It was a logical progression of thought then to house the photos at the remote site until they are used. But where? It turned out that we could store them on the thin clients very easily. We already use FTP to transfer files back and forth from USB sticks to the server to ensure that all file transfers and connections are stateless. So what I did was create a "fake" USB stick in /media that is really a RAMfs (tmpfs) with 20MB of space. This was to avoid having to have a USB stick plugged into the thin client to house the pictures. It was not desirable to store the photos on the flash drive of the thin client itself, these drives have a fixed amount of writes.

When someone moves in front of the camera, it shoots the pictures and puts them on the thin client located at that site via FTP. When we then need those photos from our host based software, we FTP then off the thin clients and back to the server. Clean, solid state and hardware independent. When this camera is discontinued, all we have to do is buy the newer model and drop it into the same location.

Total cost to take photos of our citizens, $110.

Projects Keeping Me Busy

Life at Largo never slows down. When you push new technology live, there is always more waiting in the wings. Here is what is going on:

Evolution 64 bit: As I have reported everyone has been moved to release 2.28 on SLED 11 SP1. Still blazingly fast even with hundreds of people working. I moved the bug-buddy binary out of the way and inserted a custom script that dumps backtraces to flat files and into a common directory. I can see how many people, what people and why people are crashing in real time. Looks like we are hovering around 20 crashes a day that dump BTs and a few more that crash at startup. Around 400 unique people check email a day, so most people are having a stable experience. BTs are being shipped to Novell and we hope to start getting some patches soon. Those patches should also land upstream and help everyone else too.

OpenOffice 64 bit: 3.2.1 is working well and stable. It's still running so fast that even under a user load you don't see the splash screen at all. On a multi-user system you never really have a "cold start" but the UI opens in about 2-3 seconds at which point you can immediately start typing. Nice. We have identified that extensions are causing the memory leak, and hopefully Sun/Oracle will be able to work on it soon. In the meantime we just bought more memory (cheap) to keep everyone out of the swap device.

Firefox 64bit: Pavel Janik finished testing our new 48 processor (8 cpu x 6 core) server to try and set some new OpenOffice compilation records. I won't spoil the surprise and you can read his blog when it comes out. I'm downloading OpenSuse 11.3 right now and we will move that server back inside the DMZ and prepare it for our employees. I'm going to begin testing 64bit Firefox 4 and see how it works. Adobe seems to have pulled back 64bit flash, so I'll be experimenting with the various plugins and see how they work. I'll check on the state of the various media players and see which one works. We used mplayer last time and it works great, but I always keep an eye on other solutions.

Debian Lenny Xserver Crash On Thin Clients: I finally packed up a bug report on the crashing Xserver issue. It only happens about every 2-3 weeks, with compiz enabled and happens when you close a child window. Bug report is here

Verizon USB 760 EVDO Modems + Laptop Thin Client: Verizon released a nasty modem, the USB 760. They thought it would be a good idea to put software and a modem into one USB stick, nice. Linux sees it as /dev/sr0 and tries to automount what it thinks is a CD/DVD drive. The workarounds are not pleasant, you have to tell udev to then immediately eject the drive and then attempt to fire up your dialer. I'm poking around at some ideas to make it not so clunky. Note to Verizon: Please don't make hardware devices like this anymore!

GDM + XDMCP: Halfline and I are trying to get synced and have some time to experiment with newer versions of GDM and find out why it won't let thin clients connect with XDMCP anymore. Hopefully that will happen in the coming days.

Update: 2 second cold starts on Firefox on the 48way server. Obviously no user load, but promising. :)

A Word About My Book

A few years ago I was asked to write a book and some of you have purchased it. Obviously one hopes that a book always meets the expectations of the reader, but there have been some negative reviews. Some of the goals while it was being written were not entirely matched to what people thought they were buying. Let me explain, and tell everyone what is in the book:

Obsolete Technology: When I was changing cubicles a few weeks ago all of the contents had to be packed and moved. I was shocked at the high number of books that were completely obsolete. I had a RedHat 6 book from 1999 and on inspection you find that nearly every technology has completely changed in 10 years. I wanted to create something that had a longer shelf life. In the time since the book was published, HP has already released two new thin client models. Had the book dug deeply into the internals of the model available at that time, it would have been just another technical book sitting around. So I wanted to include some topics that never change.

People Issues: The book deals a great bit about *people*. We as techies sometimes don't like to think about them or talk to them :) , but they are the biggest issue that you have to overcome when you move to thin clients and even more so when you move to Linux. Yeah, it saves lots of money, for sure...but if your administrative people aren't on board any project such as this is doomed. The book also deals with the fact that IT staff will have greatly changed duties post migration. You no longer have lots of people running around keeping client devices running and focus moves to R&D and centralized support. Instead of maintaining, you are improving and that sometimes requires different types of people.

Pictures And Pictures: All of you would be surprised at the number of people that think Linux is "character" and "command line". Many people think that if you move your email to Linux that you have an xterm window open with pine or something. During the editing process they asked me about including the screen shots, and I felt strongly that we needed to SHOW people applications like Firefox, Evolution, OpenOffice and Realplayer running graphically and similarly to how they look and work on Microsoft Windows. People also have this impression that if you have to get to Microsoft Windows applications from the Linux desktop that you have to log out in some manner and then connect to the various software packages. I felt it important to show that everything runs in a consolidated desktop, Windows and Linux apps right fine side by side.

Unique Site Requirements: Every deployment will be completely different, so it would be impossible to give exact instruction on how to setup and configure everything. So the book had to be about 'ideas'. I tried to describe exactly what was possible, and then leave it the reader to come up with a plan. I had no way of knowing the skills of the staff, the skills of the users, the money available and exactly what operating systems would be used. I also felt that more specific details would be available on my blog. I always try and blog about major milestones that we reach and hope that it's helpful.

There are good days and bad days on any computer system, but we have inched our way up to about 700 thin client workstations and things are still lightning fast. This design works, is stable and saves a lot of money.

Time Consuming Upgrades

Centralized computing makes it so difficult to upgrade hundreds of users to the latest release of the ever exploited Adobe Acrobat Reader. :)

browser:/u # ls -lad acro*
drwxr-xr-x 3 root root 4096 Jan 13 14:08 acrobat_9301
drwxr-xr-x 3 root root 4096 Feb 23 10:29 acrobat_9310
drwxr-xr-x 3 root root 4096 Apr 23 13:32 acrobat_9321
drwxr-xr-x 3 root root 4096 Jun 30 16:06 acrobat_9331
lrwxrwxrwx 1 root root 12 Apr 23 13:33 acrobat_9_live -> acrobat_9321
browser:/u # rm acrobat_9_live
browser:/u # ln -s acrobat_9331 acrobat_9_live

and then push the reader into Firefox

browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_sound/firefox/plugins/
browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_nosound/firefox/plugins/
browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_noflash/firefox/plugins/

All done. :)

64bit Evolution Is Now Live

After that small hiccup with running Evolution on 32bit Linux and the memory limitations, we are now live on 64bit SLED 11 and the difference in speed is striking. The per user memory requirements have grown, but it's a small price to pay (and memory is cheap!). The top screen below is 220 concurrent Evolution sessions. CPU is barely used at all. Disks are configured RAID 1 which has proven to be the best for Evolution (RAID 5 is much slower). When using the Groupwise backend, hitting Send/Receive the dialog only appears for a split second. The server is has 4 CPUs, quad core and I think we could easily get 500 concurrent users on here with no performance penalty. I'll have a few days of tuning and minor issues. Next on deck for me: getting gdm fixed to allow XDMCP connections again, and starting to work on that new monster server to run Firefox.

OpenOffice, Evolution & Benchmarks

The week has been busy and it's only Tuesday. Here are updates on my current projects.

OpenOffice
Migration to 64bit Linux is complete and things are working pretty well. Even with 100 users, OOo opens in about 2-4 seconds, nice. I was able to confirm that indeed there is a nasty memory leak when using the Start Center and have submitted a bug report and valgrind information. The short term work around was just to install enough memory to cover the leak. Hopefully someone will be able to look at it soon.

Evolution
We migrated the rest of the users to SLED 11/SP1 and things appeared to be working well initially. We then started getting reports of Evolution crashing during peak loads. After some investigation it was determined that we are having the same problem that we did with OpenOffice -- having too many people to run the 32bit release. You basically run out of 'low' memory even though when viewing 'top' you perceive of no memory problem. When low memory is exhausted, it begins picking processes and zapping them with oom-killer. I have the servers configured to make them super easy to migrate, so we downloaded 64bit SLED 11 and it's already installed and working on another box. I'm testing Evolution and mime types, and all we have to do tomorrow is just backup /home for everyone and move it over and change IPs. Everyone then instantly is on a new server.

Benchmarks & New Server
The one area of our computing infrastructure that is growing the fastest is the use of Firefox/browsing. More than ever it's being used for data retrieval, videos and downloads. We also are purchasing some new software that might make heavy use of Java and we wanted to ensure the best possible performance; so using budgeted monies we ordered a new server.

The new server finally arrived this week. It's got 8 CPus each with 6 cores. I took a shot of 'top' below so you can see what it looks like. I had to pick 6 point font just to get the CPUs all to display. :)

One of the things that we do when these new servers arrive is hook up with our good friends working on OpenOffice and give them a chance to try and set a new compile world record. This is done before the server is placed into production and when they are finished we move it inside our firewalls and reload the operating system. So in the coming days, hopefully we will get some interesting information concerning just how fast this new server runs.

OpenOffice Gurus?

So this week we deployed OpenOffice on 64bit Linux for the first time. It's smoking fast and works great especially under user loads; however it's chewing up LOTS of memory. It seems to be using easily 3 times more memory. I have been using lsof to check the documents that the users have open and they are nothing out of the ordinary and some of them are just a few pages with no images.

Anyone have any thoughts on this matter? Could it be leaking this badly? This type of memory usage is not sustainable; so I guess that means I'll have to start looking for it by process of elimination. Possibly it's user settings of some type. Very odd.

(top shot below shows memory usage).

Evolution 2.28 & 64 Bit OpenOffice Going Live Starting Today

Evolution 2.28
Those of you that have met me and communicate with me know that I'm pretty fair in describing good & bad experiences in working with technology. The road upgrading from Evolution 2.6 on SLED 10 to something newer has not been very fun and took 1.5 years, which is WAY too long. There were a few times in recent months where we had meetings and discussed moving to completely new solutions. We are using the Groupwise backend with very heavy calendaring and had a minimum level of features that we wanted to get included and working. Sometimes I still get the feeling that Linux vendors are somewhat caught off guard when you *REALLY* run desktop software not on MS Windows. Evolution is by far the best solution to deploy on Linux over the Groupwise Java client (which is absolutely horrid), and the web interface. I just wish that more resources were allocated to its advancement.

With SP1 of SLED 11 enough upstream patches have been merged that our features are working well enough to deploy. It's not perfect and some bugs remain, but users can perform their calendaring functions including meeting resends and retracts. I will be moving over the users to the new server and then doing a bit of QA testing on their accounts to ensure that everything is working.



OpenOffice 3.2.1 On 64 Bit Linux
We had an issue last week whereby hardware is failing on our production OpenOffice server. As luck would have it, the replacement computer had already arrived and was sitting on the loading dock. I did a review of all of the packages that we needed and determined that everything was available in 64 bit versions. The server was quickly built and will be put into production this week. Openoffice on this server is FAST. The composer UI of Writer is appearing fully rendered in about 2-3 seconds, and running so quickly that the splash page doesn't even appear at all. Very nice, and thank you to everyone involved with making my job easier. :)

XDMCP & OpenSuse 11.3 Anyone?

I have been poking at getting indirect XDMCP working on OpenSuse 11.3 with gdm with no success. Has anyone gotten this working? I did see that recently gdm was rewritten and there were some complaints that not all features were brought over to the new version.

Firewall is off on the inside network, gdm.schemas was edited..which looks like it replaced the custom.conf file. I put the lines into custom.conf anyway just to be safe. Xaccess looks good for indirect connections (*), yast2 security has remote connections allowed.

If anyone got it working, step by step instructions are appreciated.

I hate to think that 1975 xdm is going to be greeting our users on shiny new OpenSuse 11.3. :)

Ideas Welcome, Low Level Mouse Events

My current project is related to Accessibility. Certain parts of the accessibility modules will help me and are being tested.

One requested idea is to make use of a 5 button mouse. By default the 5 button mouse works per the defaults: Thumb1 does a Back, and Thumb2 does a Forward.

The special need that we have is to eliminate the need to use the keyboard in combination with the Left mouse button. I'm trying to map it like this:

Thumb1 = Hold_Shift_L + Regular_Left_Mouse_Button + Release_Shift_L
Thumb2 = Hold_Control_L + Regular_Left_Mouse_Button + Release_Control_L

This would let them make non-continuous selections and shifted selections without the keyboard. It seems like imwheel is the closest software package for this purpose, but it seems incapable of sending a physical mouse click after another mouse click. Can anyone confirm this is true?

What I tried in the .imwheelrc file is this:

".*"
#,Thumb1,H|E|L|L|O
,Thumb1, Shift_L|Left|-Shift_L


The commented line works great, click Thumb1 and the string appears in the current window. You also can issue a Control-A to select all, works great. But when you attempt to simulate a mouse click, it doesn't seem to understand how to do that. The line below the commented line above to me should work, but doesn't work. I also tried Button8 instead of Left and that also does not work.

Anyone know of another program or technique that could provide this functionality?

Thanks!

Client Based RDP & GNOME Icon Launching

RDP Architecture Change

For many years we have integrated MS Windows applications into our GNOME desktop and delivered them to our thin clients. Various techniques have been used to do so, at first we used UIS (Unix Integration Services) which was wonderful but is no longer available. We used Citrix to deliver applications as well. Because the RDP protocol added high color support a few years ago, this now is becoming our primary delivery technique. It's always been our goal to run as little as possible on physical thin client hardware and keep as much on the server as possible. The more that you run at desktops, the greater the chance that you will have to touch them with updates. It also is harder to troubleshoot and resolve problems when it's offloaded from a server. So what we have done is run rdesktop on the server and this has worked great. Rdesktop talks with Windows and then the presentation is delivered to the thin clients with X. For most applications to date performance has been appropriate. But there are some things that are now making this design not work as well as we would like:

• More and more users are getting dual and triple screens to do wide screen work with lots of real estate. This is a lot of data to push over X.
• Certain CAD applications are becoming more sophisticated and are doing lots of movements and zoom that should deliver reasonable response times.
• More applications are starting to deliver sound and video from Windows.
• Products such as Aqua Connect allow you to deliver Mac OS X applications from servers which also require faster response times.

As seen in the diagram below, our current design kind of created a double hop. RDP screens and sound went to the server and then went to the thin client with X and pulse.

Rdesktop runs on the HP Thin clients fine, but obviously when you do so that then creates a more client/server environment and condition where if you want to make a change to that software package you are going to have to touch desktops. While rdesktop isn't changing heavily, an example of what might happen is the users finding a keysym problem after deployment that requires a change to the configuration files. When rdesktop is on the server, such a change takes 2 minutes.

The other problem is how to make an icon launched from the GNOME server initiate rdesktop running locally on the thin clients. The user account doesn't exist on the thin client and is simply running an X server and then using XDMCP to connect to the server.

What has been devised is a technique that seems like it will work well. The thin clients basically have 2 user accounts, 'root' and an account of 'user' with an ID of 1000. I set up an account called 'user' with ID of 1000 on both of the GNOME desktops and then have granted sudo permissions for all other users to have access to the rsh command as 'user'. When they click on the icon, it will sudo the rsh command and pass to the thin client instructions on all of the command line arguments to run and then initiate a local copy of rdesktop to display :0.

Performance is much faster running in this manner; X and PULSE are completely removed from the loop and the Windows servers now have a straight RDP hop right to the thin client as seen in the diagram below:


Full screen video works well with this design,sound is much less prone to breakups and applications are much faster. We continue to test this design, but it's showing promise.

GNOME Icon Launching

During the debugging stage of Evolution I have noticed certain user techniques which have been unknown to me and I have taken steps to improve the user experience. When a user clicks on an Evolution icon our launch script automatically kills any running sessions of that software and starts a new one. The reason it's designed this way is:

• Regular users cannot be expected to understand how to kill applications with a GUI or to look at process lists. People have no idea what this means or how to do it.
• Users understand reboots, and if they had a local computer would just reboot it. Obviously, they can't reboot the servers. Hundreds of other people would not be pleased. :)
• There are 15 hours of the day and weekends when someone from IT is not here to kill processes that are stuck or not responding.
• The regular GNOME dialog that comes up when an application is not responding very often does not halt all related processes. Yeah it might get evolution.bin, but it very likely will leave the data-server and alarm processes happy churning away.
• Very often users log off without closing Evolution correctly and sometimes just turn off their thin clients. This leaves processes behind on the server.
  

So halting all prior instances is desirable. They click on the icon, it cleans up the process list and deletes any temp files that might be hanging around and ensures 100% that they will get a running instance. Sounds great, right? Apparently not. Certain techniques that I saw when I was scoping the server were revealed.

Users Click On Icons Multiple Times To "Speed It Up".

I guess the theory is that an icon is like a water pump. The harder you pump, the faster and more likely you are to get water? Users have been double and triple clicking on the icon. Each click was then halting and restarting Evolution. Nice. So what they see is Evolution blinking multiple times until they stop clicking and then finally it opens.

I installed a small script that I called 'gun slinger' that counts the number of seconds since the last click of certain icons and ignores repeated and quick clicking. Here is the popup they see :)




Users Use The Shortcut Icon To Replace The Window List GUI

Instead of finding an already running process in their window list, they just click on the shortcut icon again. If they have something maximized over the top of Evolution, they have no idea how to bring it to the front and have just been clicking on the shortcut again.

I installed a bit of code in the launch script for Evolution that if it detects an already running session, it uses wmctrl to give it a window manager hint. It comes to the front, un-minimizes and flips the cube to the right side to try and get them to run the already running session. They then get this dialog:




All of these techniques are being logged to a file and I'm seeing how many people have been doing this all along. I'm hoping these two steps will eliminate abnormally shutting down Evolution and also reduce the likelyhood of damaged files and perception of instability.

User Presentation Available

Today I'm having a quick meeting with interested users as I begin to look at using OpenSuse 11.3 to replace 10.2 as our desktop server. I built some quick glade mockups to kick around some ideas. The MIME bars that you get when you double-click have worked well for us, so I'm looking at ways to make improvements. I also am going to write a simple application to allow them to make Compiz changes which falls in between ccsm and simple-ccsm. The slides are cryptic, but might be interesting reading.

The presentation is here

What I'm Working On...

I haven't posted in a while, guess it's time. The days seem to go quickly and blend together. I sit in a cubicle listening to XM Radio with headsets most of the time, but am making steady progress on some projects.

Evolution Deployment
Still not live yet on SLED 11 :( , but in the last 3 weeks have made some progress. Our beta testers have been patient so far, and I really hope we are nearing the end of this process. All features that we need are now merged. We had a nasty performance problem that took a while to resolve. It turned out that a user had birthdays entered on a blackberry, and Groupwise gladly synced and uploaded them. The problem is that it uploaded them for 100 years. Our version of evolution-data-server doesn't play nicely with a calendar that goes out that far into the future. We found it, cleaned the users data and things are better. Now are awaiting the code to make bug-buddy launch so that I can get some backtraces from the users.

Thin Client Updates
All for of our thin client devices are synced and work with the same code and provide identical features. Our support group has started updating the thin clients around the City with the new code and for the most part is a nice upgrade. The one issue that I am tracking is that there seems to be an X crasher when using 3D effects in Debian Lenny on the ATI driver. It happens to me about once every 2 weeks and you lose all of your work. I need to get a bug report packed and hope that someone is interested in working on a fix. It doesn't seem to happen to 2D users.

Firefox Server Upgrade
With the explosion of Internet and Firefox usage we are going to be building a new server to hold us for the next few years. We also got word that we are buying a major new application that uses Java in the browser. Java has always proven not pleasant to deploy, so hopefully this server will meet our needs and provide adequate performance. It's an eight processor, six core machine. 'top' is going to report 48 CPUs, that's going to be interesting for sure. :)

Next Generation Desktop
I brought down OpenSuse 11.3 beta and have started poking at it and am going to be building a server for alpha testing it as part of a plan to upgrade our desktop (login) servers. These servers provide GNOME and 3D. XDMCP seems kind of broken and untested in the current betas. I'm going to see if updates exist. Gnome-shell was a no-go over remote display, so looks like we will be using Compiz for another upgrade cycle. I did some testing of compiz and performance was excellent. I'm doing a kickoff presentation for our users next week and will be showing them screen shots and mockups of ideas that I have to improve our current design. Of course, as things are deployed I'll post them here too.

Thanks Pepp!

When you have an idea for a feature and do a bounty and finally get it back it's a wonderful thing.

Today we finally received just such a feature; Drop and drag from Evolution into PDF format. Being a Governmental agency, our users have to keep copies of hundreds of emails a day in project folders. Previously in order to keep them in PDF format, they would have to do them one at a time by "printing to PDF".

This feature will save us countless hours. Once it's been tested fully, I'll work with the Evo guys and get it merged into the community build too.

It's a good day.

Calling OpenSuse 10.3 Gurus

Yup, I now it's old. We have NFS server running on OpenSuse 10.3 and everything is working great. We now have a circumstance where we need detailed logging to this machine for a short period of time and the command line arguments for rpc.mountd don't seem to work as I expect them.

By appending --debug all (and optionally --foreground to run on the foreground) I was expecting to see lots of spewage and be able to monitor all NFS activity including reads, writes and deletes in realtime. But the only thing that is bring reported is when the initial mount comes from the client. All interaction with files creates no logging at all.

I thought maybe this was a syslog problem, and thus ran it in the foreground and expected to see this information displayed; yet I'm getting no logging at all.

Drop me a line in the comments area if I'm having a conceptual failure in this area. :) I also have been looking for a description of the debug levels (all, auth, call, general and parse)...yet no man pages seem to tell you exactly what they mean. I guess this might fall under 'read the source code'?

Happy weekend all.

Why We Develop With Panther

People are pretty passionate about their development tools, we found something that works for us based on our needs. Maybe if you are in the Government or Business segments, this information will be useful.

The development tools on Linux most commonly in use are great for writing 'desktop applications'. Want an email client? Use C. Want a note taking package? Use Mono. But those tools are not a good fit for our needs. We want to quickly connect to a database, drop and drag fields out, give the users a simple grid and entry system; and then give them quick reports. We always say that the type of software that we write boils down to "enter, list, sort and print". When we get a request, they usually want it finished quickly, and we don't want our staff debugging garbage collection, threads, compilation and fighting every widget interaction. Our custom software isn't as complicated as Firefox or Evolution, and does not need low level languages. We also want it to run on Linux, and carry no runtime licenses.

The other big issue for us is portability and server upgrades. When an operating system or hardware becomes obsolete, you don't want to spend a time messing with stable software trying to get it to work correctly on a new box. It's much easier to just tar it all up and move it over and it just *works*. Sometimes too you move to new operating systems. All of our software started on SCO Unix and then moved to Linux, and we didn't want to have to recompile and debug working applications during that transition. Lastly, when database upgrades and changes come we didn't want to have to touch the software yet again and fight library interactions.

So how did we obtain these goals? We are using a software package called Panther from Prolifics. The way that it works in a nutshell is that when you develop, you are writing GUI layout instructions and logic code into the 'client.lib'. This file itself does not execute and is highly portable. You then get a 'prorun' binary from Prolifics for whatever platform you wish to deploy on, and it converts your UI instructions into native widgets of the current operating system. It's super fast, with a small memory footprint and serves us well.

This image shows how software is deployed with Panther. You develop your screens on any platform, and a runtime engine on other operating systems instantly convert your screens to that platform.


I built a very simple (blingless!) screen to demonstrate. Here is a shot of the development environment running on Linux/X.




Once your screen is finished all of your instructions are stored in client.lib. You can then pass it to the prorun binary for various platforms unmodified. I do want to be clear that you do of course have to make considerations for the real estate of the various platforms. But Panther does a good job of adding scroll bars to screens when they don't fit. Making this work becomes kind of a art form, and is elegant.

Here is the screen running in Linux/X




If you unset DISPLAY, it instantly runs in character mode. It replicates the widgets as closely as possible, and on a workstation with a mouse, you are able to click into fields and activate the pulldown menus.



They have a prorun (jserver) process that can go into the cgi-bin directory of your web server and once called converts the client.lib into HTML. Your screens are instantly on the web, unmodified. In theory you could NFS mount the client.lib file and run the same code on MS WIndows, Linux/X, Linux/Character and on the web. Any changes you made instantly were deployed to all users.

Here is the screen running in the webserver.




Here is the same screen running on Microsoft Windows.



So for the last few months I have been working with our Integration staff in making Panther fit better into the GNOME desktop. Artwork is being upgraded and all hard coded colors are being removed. GNOME themes automatically set old school X resources as closely as possible to the right colors, which Panther picks up nicely. These steps along with anti-aliased fonts in newer versions of Motif have allowed us to deliver better looking software. The shots below demonstrate a deployed screen, with colors picked up from GNOME theme.








Linux is a wonderful runtime environment for running custom in-house software, and maybe this post has given you some ideas for your own organization.

Update: Receiving and uploaded Microsoft Windows screenshot.

18 Second Boot

I have been working in between things on the hp 4410t thin client laptop based on ideas from some of users and IT staff. One thing that was important to me was making it work better than our fat PCs that are used to remotely log into the City network. One part of that goal is making so that it's ready to work as quickly as possible after it's powered on. I stripped out all services not required and have gotten the machine fully booted and networked in 18 seconds. I might be able to shave off a few seconds yet, but I think it's getting pretty close to the fastest possible experience with this hardware. Not bad.



When I'm finished with everything, I'll post a flowchart showing the design flow of network connections and features. I'm working to make this as simple as possible and want to reach a goal of users just having to know to push a single button and everything is networked and ready for their login.

Items left for me tinker with: Backlight issues when moving back and forth to battery, More intelligent WiFi logic to understand whether it's in our buildings to reduce boot times, resolve issue of very buggy x-server-xorg-video-intel (2.9.1-2) on Debian squeeze/sid, remove unneeded packages to try and reclaim a bit more flash drive space and resolve some pulseaudio problems at boot time.

Miserable Admin Things

Some things make the life of an Admin absolutely miserable when you are trying to keep hundreds of users working with adequate performance. My updated /etc/hosts entry sure makes things work better. :)

192.168.99.99 apps.facebook.com

Various Updates

I haven't posted any new information for a while, but the City has been keeping me busy. Here are the things that I have been working in no particular order.

OpenOffice
I upgraded the whole City to OpenOffice 3.2 this morning. As I have mentioned previously upgrades are very simple when they are deployed from a centralized server. All I do is point the launch scripts to the new version and it immediately goes live. So far, no major support issues.

Firefox 3.6
We went live with 3.6 and so far everything is working well. Performance is good over remote display on multi-user servers. I have already started beta testing 3.7.

Evolution
Migration to the new Evolution has been slower going than I would have liked. We were running a developer build and have now synced to the official release. We found a few regressions and issues, and I sent a document which contains our show stoppers. Hopefully we can get this project finished soon.

Thin Client OS 2.0
I finished our internal "2.0" thin client operating system release and it's starting to be pushed out into widespread use. Another week or so, and I'll be comfortable deploying to our other 550 thin clients. HP released Debian Lenny, and I made all of the modifications required to better support our design goals. The major changelog items: + Increased USB stick supported files + Click sound when USB stick is inserted and removed + Upgraded NX client for remote sites + Graphical first boot which requests information about how thin client will be used and desired resolution and color depth + Merged Intel Xorg drivers + Support for 5745 thin client + Hard code 5735 thin clients to only work in our cyber cafes + Custom Largo splashy at boot + Support for SmartBoard white boards + Removed Xinerama and moved to xrandr. + Automatic shutdown in 3 hours of inactivity + Faster startups and shutdowns.

I'm very pleased with this release, all of our thin clients work with an identical Linux OS.

5745 Thin Client
Our loaner t5745 HP thin client is working *great*. I have started to move it around to regular users who are testing it with their own monitor hardware and their own techniques. Reports are that it's 10-15% faster than the 5725. Nice.

SmartBoard (Electronic White Board)
The new IT conference is finished, and we have deployed an electronic white board. The drivers and software works 100% on Linux, so I installed it in the 2.0 thin client release. That means anyone in the City that wants to use this technology just has to buy the right hardware. The picture below is the smart board mounted to a wall, running off a thin client. You can export these documents into PDF with just a few clicks.



4410t Thin Client Laptop
I have gotten some time to work on the back burner project of adding a thin client laptop into our infrastructure. I'll blog about this in more detail in the coming days. I downloaded some bleeding edge Intel drivers which are buggy, but making progress in the design and work flow. I basically want to make it so that you just turn on the device and within a few seconds you are able to log into our network. I'm experimenting with how best to drop from wired to wireless to EVDO automatically. I'm not reinventing the wheel, just trying to eliminate applets and prompts and just work for our needs and network. Lots to think about.

The picture below is my testing of an EVDO modem, testing ideas on how to handle the phone number and when exactly it should dial. Our number one support issue on fat laptops is networking.

HP t5745 Externals And Internals

Of course one is always tempted to take apart something that is working fine and see what makes it tick. I have been running the new HP t5745 thin client at my desk for about a week now and it's working really well. Performance still feels about 10% faster than the older model. The Intel Xorg drivers are stable and fast, and I haven't seen any blinking or video problems. Looks like this one is a winner.

The inside is very clean. Memory, Apacer flash module and CPU take up the most space.



The front is well designed; microphone, headphone and two USB ports easily available to the users.



The back is clean, and offers all the ports we need. PS2 ports are appreciated to keep the USB ports clear for other devices.



If you are considering a move to thin clients, this is the best model I have seen to date. With a 400 dollar price point, and anticipated 10 year duty cycle it's certainly attractive. Our 5725s are proving to be solid work horses with very few failures even after 3 years.

HP 5745 Thin Client Arrived

Yesterday we received our first HP 5745 thin client, and I'm pleased to say that it's working well so far. The onboard Intel video card supports 3D/Compiz and will not require us to add our own video cards; which will lower their price point by at least 75 dollars. Compiz is fluid and responsive and movement is crisp.

It was really perfect timing for the 5745 to arrive. I had just finished the Lenny update for the 5725s and 5735s and the build installed and booted with very few modifications. So for us, all three models will run the same operating system and have identical features. I just need to write a small amount of auto-detection code for first boot.

In the next few days, I'll take the new thin client apart and take photos of the internals. The new footprint and design is excellent; it's only 1.75 inches wide.

XrandR, No More Xinerama

This issue apparently is well documented on the Internet, but it was new to me. Quoted from Wikipedia:

XRandR 1.2

XRandR 1.2 only permits one virtual screen per display device. It is not possible to assign each monitor on a device to a different screen (sometimes called "Zaphod" mode), or to combine monitors from multiple devices into a single screen.^[3]

One practical limiting effect of this is that it is not possible to run a different WM on each monitor, since window managers are limited to one per screen.

These specific issues are resolved in XRandR 1.3^[4], but the Xorg Foundation has been criticised for removing useful, and widely used^{[citation needed]}, functionality before an effective substitute is available

(end of Wikipedia snip)

HP ports Debian Linux to our thin clients periodically, and I then take their builds and customize them for our needs. I was pretty excited to be able to deploy Lenny after it was finalized on our hardware. But it turns out that Xinerama support was dropped before xrandr (1.2) was feature complete. After lots of tinkering I was able to get dual screens working, but unfortunately it seems that VGA-0 is hard wired to be your left screen and DVI-0 is the right. When they are flipped, XDM authentication and the GNOME panel open on the right screen which doesn't work well at all. So the net result is that when we touch thin clients that are running two monitors, our support staff is going to have to move cables and move things around. We purposely put the DVI monitor on the left side because it offers a slightly better picture quality.

As was noted in the Wikipedia article, three monitors are not supported at all, so we are going to have to run Etch on users that need that many screens.

I guess the point I wanted to make is that I understand that things need to evolve and move ahead, but there are really people using features like Xinerama. I don't care if something replaces it, but please don't drop the old features until the new ones are working. It's embarrassing that our custom GUI when the thin clients boot the first time looks like this now (Three Monitors is disabled):