Running Hundreds Of Evolution Users

Over the last few months, we have had an increase in number of concurrent Evolution users, and also have had an increase in calendar and email usage. Evolution makes heavy use of cache when interacting with GroupWise and we were starting to have disk IO performance problems from 8am until about 10:00am. This is the heaviest load, when everyone is downloading heavily from the post office. The disk drives were getting so busy that the UI was starting to slow and performance was not acceptable. Interestingly, CPU usage was under 10% even with 250 people.

We had just ordered our annual allotment of servers. We use a trickle approach and move servers around based on changing requirements. Three new HP DL580G5s arrived . The older DL580s used 15K 3.5" Ultra320 SCSI 72GB drives. The new server uses 10K 2.5" SAS 146GB drives. Below are the specs, the difference in speed is amazing. We also had previously used RAID 5 which might have contributed to the bottleneck. We configured the new server and used RAID 1+0. A backup and restore of /home and IP number change the new server immediately went live with only a 15 minute interruption. The new server is so fast that when you hit [ Send/Receive] the dialog flashes and disappears so quickly it can barely be seen.

Disk performance is excellent even with 250 people:

iostat -x 10

Linux 2.6.16.54-0.2.3-bigsmp (oa3) 12/20/07
avg-cpu: %user %nice %system %iowait %steal %idle
0.97 0.00 0.24 0.01 0.00 98.81

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.00 71.90 0.00 22.30 0.00 1507.20 0.00 753.60 67.59 0.01 0.43 0.25 0.56

Previously, the drives were %util from 75-100%

Technical Specifications:

• Hard Drive Capacity : 72.8GB
• Generation: Ultra320
• Data Transfer Rate: 320 MB/sec
• Rotational Speed: 15,000 rpm
• Form Factor (Drive): 3.5-inch low profile
• Interface: Wide Ultra 320 SCSI; LVD
• Data Storage Device Type: SAS (Server Attached Storage) device
• Hard Drive Device Type: Hard drive for server/storage unit (Hot-plug)
• Height: 1 inch
• Pin Configuration: 80 pin Hot Swappable/ Pluggable
• Hotswap Tray: Included (Attached)

Technical Specifications:

• Hard Drive Capacity : 146GB
• Generation: SAS
• External Data Transfer Rate: 3.0 GB/sec
• Rotational Speed: 10,000 rpm
• Form Factor (Drive): 2.5-inch low profile
• Interface: SAS (Serial Attatched SCSI)
• Data Storage Device Type: SAS (Server Attached Storage) device
• Hard Drive Device Type: Hard drive for server/storage unit (Hot-plug)
• Height: 0.591"
• Width: 2.75"
• Hotswap Tray: Included (Attached)

Largo Desktop Video

I have recorded my first video to show everyone the new Compiz-Fusion desktop on our thin clients. The speeds that you are see are under normal loads, and exactly what the users experience. I'm just about finished setting up all of the icons and software launch scripts.

Notes:
- The blinking is not seen on the thin clients, movement is smooth
- I demonstrate the MIME bars I wrote to make it easier for them to process files
- Evolution is being moved to a new server next week that is much faster, you are seeing the speed of the current machine. Preview pane display is *much* faster on the new server.
- Video encoding for the web is new to me, so I'll do better next time. :)

Bliss

Bliss is your first 4-way Quad Core server. :)

DBera Is My Beagle Hero

We use Beagle extensively at Largo for searching historical documents. We have hundreds of thousands of documents going back 20 years. Each night I build a static index of each department, a static index of the city-wide shared directory and then the users personal directories. One of the requirements we have is to block users from searching departments of which they are not a member. We also had the requirement that we wanted users to be able to search individual static sources as well, to reduce the number of hits. The only way previously to do this was to have a zenity front end run before Beagle, and display all available indexes available to each user and then allow them to toggle the desired sources. It then passed this list to the beagle daemon and started the UI. Ok, it worked. But if you wanted to change your criteria you had to close Beagle and relaunch. Not very friendly.

So DBera cooked up a small fix that allows you to select from the sources via the UI itself. When a user runs Beagle now it figures out which sources they have access to search. It then starts the daemon with all of these sources and displays a dialog as below:



The UI then comes up and you are able to alter your search criteria on the fly. While all sources appear in the pulldown, only those connected with the beagle daemon will return results.



It's working perfectly for us. There is an ongoing conversation about how to integrate something like this into the main build. The patch right now is hard coded for our needs, but it sure seems like this would be helpful for other people. If you have comments or suggestions, add them to the feature request.

DBera, thanks for this step forward.

Friday Afternoons

Sometimes you do fun/stupid things on a Friday afternoon, especially after a week of troubleshooting and debugging.

I can now confirm that Stella works fine with Compiz-Fusion and Pulse Audio via remote display to thin clients. :) [shot below]

Have a great weekend all.

3D Desktop Release And Testing

The replacement desktop server for our thin clients is nearly ready, but unfortunately vacations and Christmas have pushed the go live date into January. That gives us time to finalize, test and do some training for volunteers.

The new desktop server is running OpenSuse 10.2. Users can pick running regular 2D GNOME or they can enable 3D (Compiz-Fusion) if they so wish. The login scripts check hardware at login and if a user is configured for 3D and logs in with an Xserver that doesn't support Compiz, it automatically drops back into regular metacity.

I created the following cheat sheet which I will be handing out in the class tomorrow. I always gain more respect for technical documentation writers when I try and create content for 'regular users'. Let's see what the users say tomorrow :)



I also have experimentally added avant-window-manager to the desktop which is enabled via icon for those that like to try new things. This provides a 3D panel. Because users like to delete things, the startup script automatically adds an icon to the far left of avant that looks like our City seal; this icon when clicked opens the application-browser. That way they never get themselves in a position that they cannot reach software applications. Further testing is ongoing, but seems like it's deployable and runs well over remote display.

Bug-Buddy-Largo

I know there have been some blogs about the future design of bug-buddy. The current design has not been effective: When a program crashes it's very likely that users have lost work or time and they usually aren't very happy. They normally never go through the process of submitting bug reports and just want to get back to the software. So lots of useful information by hundreds of people is being lost and not getting back to the developers.

Bug-buddy needs a way for Admins to dump the information to a directory and hide the GUI from the users completely. It's my job to review them and get them submitted. I was happy to find that the hooks into bug-buddy from a crashing application are not deep, and that I could slip in my own script called bug-buddy to perform a fully automated dump.

If you want to get dumps from your users, the following simple script might be useful:

more /opt/gnome/bin/bug-buddy
#!/bin/ksh
#
# $1: Program that crashed
# $2: Process ID
# $3: Version Number

/opt/gnome/bin/zenity --info --text "Currently dumping crash information, Evolution will restart when finished" &

pid=`echo $2 | awk -F "=" ' {print $2}'`
prg=`echo $1 | awk -F "=" ' {print $2}'`

stamp=`date +%Y%m%d%H%M%S%N`
outfile="$prg.$stamp.$USER.txt"

/usr/bin/gdb -p $pid --eval-command="thread apply all bt" --batch --nw > /home/largo/tmp/evolution_crashes/$outfile

Trust Your Business

I live very close to downtown Safety Harbor Florida. It has a very small town feel, and lots of people walk around at night and window shop and sit in the gazebo. One business had the idea of running electronic slides in their store front. Too bad they didn't know they could run Linux. :) (pics below).

The Desktop Hexagon Cubed?

I'm prepping the Compiz-Fusion machine for city wide deployment and working on feature requests that we had on the older GNOME desktop. Previously, our support staff would use x11vnc and vncviewer to support users that are having problems. We have had requests to allow anyone in the City to use this capability. I wrote a small zenity and python front end and now employees can grant access for viewing, and also remote control other users desktops. Months ago when I did testing, Beryl didn't work well over VNC. I brought down the latest version of x11vnc and it's working much better. One can remote control another 3D desktop while using your own. (Shot below). Now, does this give you a headache or what? :)

Evolution, Compiz, Thumbnails And Mplayer

I was on vacation last week, and got right back into my projects this week. I had movement on many fronts, most of it good.

Evolution: Now that all of the users are live, we have been shaking out some bugs/issues and possible regressions. We don't use the keyring, and the dropback code code seems buggy. It's asking users for passwords in a manner that is not repeatable. We are also getting some EDS crashers for those with heavy calendars and those that proxy often. SR's entered and working on reaching a better milestone.

Compiz: Interestingly, I'm finding the 3D desktop is really increasing my requests for beta testing accounts. Normally, users are very slow in moving to new technology and this seems to be exactly the opposite. Cool stuff. I experimented this week with the widgets layer and video plugins. When the widgets layer is enabled and you press F9, applications that you have configured to sit in that layer display. I wrote a very simple python/Glade UI to display our current local weather. I configured all Compiz sessions to put that new UI and a clock into the widgets layer. So now all users can press F9 and see the local weather. (shot below). It would be great if someone wrote something a little more polished :) These images are statically named on weather.com.



Thumbnails: Kevin DeKorte wrote some great programs that work with mplayer and has been a big help in making some changes. I am using his nailer program to generate thumbs for video clips in Nautilus. I had 2 real world issues that most thumbnailers forget: 1) Sometimes files that users have downloaded or saved are incomplete or damaged, the thumbnailers have to account for that fact. 2) Sometimes a thumbnailer will get a file and lock/spin and not be able to generate an image. They need to have a timeout and give up after a few seconds. I had to turn off the evince thumbnailer for this very reason. You can't have hundreds of stuck processes on a server.

Mplayer: A patch had been posted on the list to use pulseaudio with mplayer. The current SVN version is still looking for the older polyp libraries. This meant it failed to find the libraries and fell back and used ESD. While ESD works with our pulseaudio daemon, this is definitely cleaner and works great. I also found a patch to have mplayer work with the video Compiz plugin. However it seems not to work with remote display, and will require more investigation. The video plays all blue, similar to how XV looks when you move focus away from the window. I put back the XV build and will continue using that until I figure out the problem. I moved mplayer into production today and we shipped forms to the users for signatures to gain access.

Pulse Audio, mplayer and gnome-mplayer Moved To Beta Group

After doing some R&D on a laptop, I got pulse audio working as desired and sent it to HP. Their engineer built from source and compiled it for their thin clients and returned it to us a working build. This build has been moved out to about 20 people and up to this point the only sound they have gotten is standard GNOME login and logout clips.

After testing a few ideas, I have settled on using the mplayer engine, and the gnome-mplayer UI for our deployment. Many of the players have the controls running in a UI separate from the display canvas, which is just not deployable for 'regular users'. Users lose windows in their stack, and often don't know how to find them again. gnome-mplayer is wrapped around the display canvas. It's got only the most basic of controls and should be an excellent fit.

In the beginning of testing, I was concerned that remote display and Compiz would make playback too slow for deployment. However, it turns out that the business use content (quicktime, windows media, real) works just fine. Below I am attaching two shots. The second is the IronMan quicktime (way cool btw!)

Evolution Update And OpenOffice 2.3

I finished moving over the 750 users to Evolution on SLED and last week was spent tuning the server. We have been using the GroupWise backend for about 18 months, and many users had built up massive calendars which were being pushed down each morning and slowing down the server. We have decided against retaining calendar cache from session to session because it's proven impossible to explain to people that they are seeing calendar events that might not be accurate during the period that the post office syncs all of the new events. It's better to just delete cache and push down a fresh set of data; once they see events it's 100% known to be current. Using the environmental variable GETQM_TIME_INTERVAL I tuned their calendars to only reload every 15 minutes to reduce open SOAP connections. We also activated auto-purging of old calendar events in the post office. Things are running much better this week as we continue to tune, here is a 'top' with 210 concurrent users running Evolution; I think we could get another 100-200 concurrent on this machine if required.



OpenOffice 2.3 came out yesterday and I put it live for the whole City. So far we haven't had any support calls related to the new release. The biggest features I have noticed so far are the new icon artwork, centered composer window and brand new chart module. Thank you to all of the QA people that make these upgrades so painless for me. As I have mentioned before all I have to do is install the RPM and change 2 lines in the OpenOffice launch script to point to the new release in /opt. I keep the old versions and a rollback is just as easy should major problems be discovered.

Humble Book Announcement

A year ago I was approached to write a book about Thin Clients, and it's finally been released. I received my physical books yesterday and it's very rewarding to see a project like this come to completion.

I spent many hours thinking about ways to present this material. The problem is that every single reader will have completely different requirements, user counts, staff, software and money. It would have been easy to write 400 pages describing in great detail our exact thin clients, our exact distributions and software packages. But that wouldn't have been useful for most readers. There is also the issue of technology churn; thin client models change all the time, and operating systems are in a state of constant change. So I decided the book should be designed to offer *ideas*. So often we get phone calls from people that are thinking of moving to Linux and/or Thin Clients and they don't even know how to get started or what options are available. It also covers what is often the biggest challenge: the user community and those in charge of the company or organization.

I'm hopeful that I have found the right balance to help be the middle person to allow many more people to deploy the wonderful things that all of you have written.

The link for the book is here.

Mid-Week Vacation Day

I took a day off yesterday to avoid the weekend crowds and visit Busch Gardens Tampa. It's about a 30 minute drive from my house. The park wasn't crowded and I was able to get on most of the rides. I was most looking forward to riding Sheikra again since it got upgraded to having cars with no floors. Sheikra is one of only 4 Dive Coasters in the world. You are seated 8 across and 3 deep in stadium seating, cranked to the top of the hill and then for a few seconds are halted at the top looking straight down 200 feet.

Photos don't do justice to the size of this drop when you are standing next to it; or looking down from the top :)

Pulse Audio On HP Thin Clients

Yesterday we had a conference call with HP and worked out a plan to have Pulseaudio built and installed on our HP t5725 thin clients. This was always a phase 2 goal, and now that the thin clients are installed, this is my primary project. I have been impressed with Pulse so far. Sound on Linux is horribly inconsistent (Alsa, OSS, ESD and NAS anyone?) and difficult to deploy. Various applications support different sound libraries, and what Pulse does is support enough of them to allow most things to just work. We hope to deploy this sometime in October or November.

Here is the only change I had to make so far:
protocol-esound.c
74c74
< #define MAX_CACHE_SAMPLE_SIZE (1024000) ---
> #define MAX_CACHE_SAMPLE_SIZE (2048000)

Increasing this cache allowed the standard login.wav GNOME sound file to play over Esound. Some of these probably should be environmental variables or settings, so that they do not require a recompile.

Our biggest sound requirements have been resolved:
Esound handles GNOME login sounds and other Linux applications
OSS is handled with the padsp binary, this allows Citrix clients to play all of the sounds from Microsoft Windows apps.
PulseAudio is handling the shockwave plugin in Firefox, and also works natively with Xine to allow playback of Windows Media, Quicktime and Real content. For grins I'm also testing remote display of TV tuners and DVD playback.

Here is a simplified diagram of multiple servers and applications all redirecting sound to the thin client and it coming out the attached speakers.

Finally An Evolution Upgrade For Us

As of today I have checked in and approved the Evolution build that we will used to upgrade the whole City on September 4th. We have been running Suse 9.3 for several years on a build that I would give a letter grade of a "B". That was the first release that we used with the GroupWise postoffice. It had some threading problems and not all of the GroupWise features were enabled.

We entered the beta program for Evolution on SLED 10 in the fall of 2006, which then continued into the SLED SP1 beta. Focus on this release has been interaction with the GroupWise calendar system, proxy, free/busy, meetings and appointments. I was a bit disappointed that it took one year to complete, but that isn't the fault of the developers. My view is that Novell should add additional people to the project; there simply are not enough human resources. This is a huge piece of software for use by corporate and enterprise customers, and needs more love.

The new release is labeled 2.6, but contains a lot of code from later releases. The mail view supports the triple panes (shot below):



The calendar view contains the most amount of changes. The mini-calendar has moved to the left and memos are now listed on the right side. The proxy feature was picked by our users as the most important feature to get working, so I focused on debugging and testing that functionality.

You might notice all of the "On The Web" calendars. We have developed some scripts that automatically generate .ics files each night with information useful to our users. For instance, in our financial system we can enter leave requests. We generate departmental Ical files which show which employees are off which days and can be checked easily at a glance. Very cool. (Shot below).



If your organization has GroupWise and you are thinking of moving to Evolution, I would highly recommend testing this release on SLED 10 SP1. It's been tested and QA'd greatly with that backend.

Thanks to Harish, Chen, Varadhan, Srag, Flex, Sankar and everyone else for working on this project with me.

Compiz-Fusion Looking Great

Once again I synced with the latest version of Compiz-Fusion and things have never looked or worked better. My notes:
- Previously the transparent hexagon was too slow to deploy over remote display, especially when you had software packages on the 6 sides. It's much faster now. Spinning the hexagon barely shows up as a blip on the server, seems like it would scale just fine.
- It's so much faster now in fact that I was able to enable the new spinning gears plugin for testing. Eye candy for sure, but the user community loves that kind of stuff.
- The new cubecap plugin is awesome. It gives you a lot more control over the image, and correctly supports transparent PNGs. It also works correctly if you have more than 4 sides.

To everyone that is working on this project, Thank You.

Vacation In New York & New Jersey

I took off a week and used the time to do some traveling. There were a few things that I wanted to do in the New York and New Jersey area.

I went and saw a ballgame at Yankee Stadium before the new one opens. Lots of history, and it was nice to sit and have a dog and watch some baseball


I then made my way down to Atlantic City and pulled some slots and walked the boardwalk. I left with a few less dollars in my pocket. :)



The grand daddy part of the trip was riding Kingda Ka at Six Flags in Jackson New Jersey. It's the worlds tallest and fastest roller coaster. You go 0 to128MPH (206 kph) in 3.5 seconds and then are launched up 456 feet (139 meters). It's quite something and I rode it 3 times. The picture below does not do justice to the size of this coaster.



I also loved El Toro, a wooden roller coaster with a 76 degree drop. It goes over 70MPH (113 kph) and is excellent.

Re-Sync With Compiz Fusion

With a few tips from the IRC guys and some general poking, I was able to get Compiz-Fusion to run concurrently with Beryl on the same server, to our thin clients. Now that they have "un-forked", I need to jump in and help test. I have about 20 people using Beryl, and they will continue to do so until I have had a chance to do some tuning and QA work on Fusion. Some new plugins have been added since my last build in April (ages ago in Internet time!) and they are very welcomed. Expo is very slick, and allows you to move applications between viewports via thumbnails.

[ Shot: 1680x1050 x 16bit color giving me around 100FPS over our network. 1440x900 x 16bit is giving about 120FPS ]



[ Shot: Spinning hexagon, cool new reflection plugin underneath. ]

Changing libgnomecups For Multiple Evolution Users

A quad processor server with 16GB of memory is able to run 200-300 Evolution sessions very easily. No tuning or changes are required to Linux or Evolution, it just works. However, one has to modify libgnomecups to scale over about 20 concurrent users. We are prepping a SLED server for deployment, and I had to make this change once again. I've made it twice before on various older versions of libgnomecups.

If you have over 20 concurrent Evolution users and people print their email (which they do!), cupsd gets hammered and consumes a lot of resources. It appears that if you print once from Evolution, you continue to always request status information from cupsd repeatedly until you close the current session. Cupsd is unable to keep up with the requests fast enough and what happens is that the Evolution composer window gets horribly unresponsive. The letters appear in the text widget at about 1/5 the speed of what you can type.

Before the patch, here is what cupds looks like with around 40 concurrent users. Notice cupds consuming CPU time. At the time of the shot, no print jobs were in the queue.



I make the following change to the code and recompile:

diff gnome-cups-printer.c.orig gnome-cups-printer.c
38c38
< #define UPDATE_TIMEOUT 5000 --- > #define UPDATE_TIMEOUT 500000

diff gnome-cups-queue.c.orig gnome-cups-queue.c
16c16
< #define UPDATE_TIMEOUT 3000 --- > #define UPDATE_TIMEOUT 300000

I then replace the stock libraries and reboot and today with identical user load, cupsd barely shows and the slow typing issue is fixed.



The server is now ready for 200-300 concurrent users. I'm not sure why Evolution keeps polling cupsd after a print job has been submitted. This change makes no apparent changes to the print process. libgnomecups should be altered in a manner that scales better, or at a minimum allow an environmental variable to be read. That would eliminate the requirement for a patch and recompile.

Install Going Well, Back To R&D

IT staff has been installing the new HP thin clients, and we haven't had any serious technical issues. Normally something happens during the install process that was completely missed, but I think our extensive beta period and testing resolved most of the issues beforehand. We are getting very positive feedback on the huge increase in performance. I guess going from 33Mhz to 1Ghz does that! :) It's my perception that the old thin clients will end up on E-bay, possibly in groups of 100. I'll alert everyone when they are available.

So with the thin clients moving out, I'm starting to work on some R&D again:

Evolution Printouts
Evolution really needs to have a plugin system for easily creating custom printouts. As I have mentioned in previous posts, printouts are the world to end users. A plugin system would allow end users to design and contribute xml files or something back to the project. Really all we would need is the ability to set column position, font, font size, fields, draw boxes, sort, assign colors and have a control-break. I have started to write my own custom printouts using ksh and the perl module PDF::CreateSimple to eliminate duplicate entry that was being done around the City. In this case I wanted to create a printout of all City employees, so I load them all into the Address Book view and then do a SelectAll and right-mouse and select "Saves as VCard...". All employees are then exported and placed into a single .vcf file. My ksh script then loops through and formats the information and creates a PDF file. I generate a regular listing, and wallet cards as seen in the shots below:



Sync With Fusion
I have about 20 users on Beryl with excellent results and today tried to sync with Fusion. Fusion is the future and I sure would love to help debug and test. However, compiz isn't running very well on ATI 9250s these days so I had to go back to Beryl. I entered a bug report. Something changed in Compiz after the Beryl fork, and I'll have to wait for patches. I'm on the IRC all the time as "dave_largo", if anyone wants me to try some fixes, hit me up anytime.

550 More To Go

Trucks pulled up to Largo on Thursday and Friday with multiple pallets of HP 5725 thin clients. We pulled 10 out of the pile and tested them to ensure that nothing had changed since our beta units. Everything looks great. All city employees will soon have far better performance and the opportunity to run 3D applications and the 3D desktop.

BigSur of Tampa will be assembling them next week, and IT Support will be placing them around the City at the various facilities.

10 down, 550 more to go.

Printers And Printing

Everyone that knows me, knows that when I mention improvement areas I do so only to alert people about the things that make my job more difficult as an Administrator. It's always my goal for these projects to improve and market share to increase.

Last week I blogged about Cairo not working on 8 and 16 bit Xservers. We are still struggling with this issue as I do my best to implement new versions of software.

This has felt like a Monday as I struggle with printing. Techie people hate printers, and I am in that camp. We sometimes have a very hard time understanding why people won't just look at things on their computer screen. 99.9% of people have the opposite viewpoint. Unless software prints well, it's a failure. I cannot stress how much users love their printers and holding paper. Unfortunately, this important feature to users is the least developed and mature on Linux. We (and I group myself with all of you) just don't get it.

I have a brand new OpenSuse 10.2 server that I am trying to fully implement. Here is the current user experience with printing on this OS. I'm not picking on OpenSuse, because all of the flavors have the same basic problems:

1) I have blogged about this before. Here is how Acrobat looks to the users when selecting a printer. Note the white space in the widgets. Someone told me this was being fixed, but apparently that hasn't been built into new OS's. (shot of printer selection below) We have 60 printers and they have to scroll up through the widget to find their printer.


2) Sometimes people ask why I don't use Evince instead of Acrobat. Well, it's even worse. My bug report today indicates this is the new printer dialog, and now the report was pushed over to gtk+. First of all the printers don't display, and require users to enter their own printer commands (ack!). And then even if they enter the right command, it still doesn't work. The print job is stuck until evince is closed. (shot of new dialog below). And because of "internet time", even if a fix is made in the most recent library versions, I'm stuck with no fix for OpenSuse 10.2 because it's too 'old'.



3) So why not use eog for these tiff images right? Well firstly it won't load multi-page images and on OpenSuse 10.2 and SLED 10 it prints the first page as a solid black box. (shot of output of tiff black output). This is how images print from Evolution as well, it's not just a eog problem.


4) And the user experience is made worse by the fact that some operating systems make the default printer the default for all users. The printers don't "stick" and have to be reset each time they print...making them go through all of the bad dialogs mentioned above. (shot of yast2 setting a system wide printer for *all* users).


5) When you print a calendar from Evolution, it doesn't print the "Location" which is the second most useful piece of information. Many power users print their daily calendars and walk from meeting to meeting...but have hand write the location information on the printout.


I have been having some conversations with the Tomboy and Evolution developers about adding additional printer options and hope that all of the projects consider this in the design. If you want to earn the love of users, make them print well and with many options.

[ Update ] The evince developers showed me how to build evince using the old style printer dialog, and now that's working. Turns out that the cups backend is not enabled on the new printer dialog on OpenSuse 10.2. It's a compile time option and is set to file,lpr instead of file,cups.

When Major Problems Are Left Open

There are a lot of developers out there, and I certainly understand that everyone is busy. I'm sure that everyone perceives that their bugs are important, but in some cases certain issues should be blockers and resolved. It's always fun to work on new features, but I sure would trade those features for stability.

At some point GNOME started using Cairo...which fails to support 8 and 16 bit color depth on Citrix Metaframe for Unix Xservers:

Error: Cairo does not yet support the requested image format:
Depth: 16
Alpha mask: 0x00000000
Red mask: 0x00007c00
Green mask: 0x000003e0
Blue mask: 0x0000001f

Error: Cairo does not yet support the requested image format:
Depth: 8
Alpha mask: 0x00000000
Red mask: 0x00000000
Green mask: 0x00000000
Blue mask: 0x00000000

I entered bug reports April of 2006 and then again February of 2007 and here we sit. We are finally at a point where I can't keep older versions of operating systems running and *have* to upgrade. I have to check color depths when users launch applications in GNOME and for these 2 depths I have to use xmessage for the dialog because zenity too won't work. Not cool. Support is having to alter settings in devices and take support calls and in some cases we will have users that have a few applications that are down until our new thin clients arrive.

Can we get some love in these areas so that we get back to what worked fine 5 years ago?

[Update] Research and checking the bug reports have indicated a lot of people having these types of problems. This library should not have been pushed into GNOME until it was ready. These aren't minor issues!

It Doesn't Matter If I Can Use It

My blogs over the last year have been mostly technical, and *my* thoughts and usage of technology. But in a way, it doesn't matter if I can use it. Without users, there is no need to have System Administrators. It's my job to act as a buffer for the users, and try and monitor changes in technology that are fits for a business environment.

We have about 10 people using our new thin clients on a 3D desktop, and they are having success and enjoy the experience. So I wanted allocate one blog to talking about *real* people. These are users that do not care about technical issues, and just want to get their work done on a modern computer network.

The picture below is of Linda. The Executive Secretary for Management Services. When I walked up she was testing the new Chart2 module in OpenOffice and learning how it works. She is a heavy Evolution and OpenOffice user.



The picture below is Sandy, the Executive Secretary for the Fire Department. She heavily uses Evolution and OpenOffice. She is in the process of merging multiple documents into a massive Master Document so that an automatic table of contents can be generated. She also gains access to a few Microsoft Windows applications via Citrix.



The picture below is Jo-Ann, the Executive Secretary from Human Resources. She uses Evolution and OpenOffice very heavily as well.



These 3 people, along with the other testers have provided me with invaluable information and suggestions about little things that can be done to improve the desktop experience. So today, my hat is off to them.

Monday Morning Hexagon

We have a weekly publication called the Monday Morning Briefing that is pushed to desktops each Monday. When you log in after a new one has been published, it opens immediately in Acrobat Reader. Needless to say I'm not in favor of this intrusive window, but lost the battle to people higher in our City org chart. When people sign your paycheck, you do what they say. :)

I'm experimenting with a method to extract the first 2 pages of the document and putting it on the top and bottom of our 3D workspace hexagon. People would then see it when flipping between views and could flip it down to read the content. I built this prototype with OpenOffice. I created a custom paper size of 11" x 11" and then simply exported it to PNG format. It needs some work to make the fonts easier to read, but I kind of think it makes good use of that real estate.

Those handling the merge of Beryl and Compiz, please make sure the ability to put images on non-cubes is pulled from Beryl...this is a very useful feature.

We are up to 10 fulltime 3D users and things are going well. The biggest thing that people struggle with is the concept that increasing the number of workspaces up from "1" actually gives you multiple hexagons. The big order of 560 new thin clients is expected to arrive on 6/18. Things will be very busy around here deploying that much hardware....the first time in 10 years we have touched desktops.

(Shot below)

500 Thin Clients Available In June/July

I would never blog about something in order to make money. You get enough SPAM already. :) This is an issue of trying to find a good home for working hardware.

In the June/July time frame we (The City of Largo) will be retiring around 500 thin clients. They might be a perfect fit for some kind of academia need, maybe you want to put one into each dorm room for very basic access to the Internet or something? Or maybe you have a non-profit company that needs basic workstations?

They are 10 years old, and obviously not as fast as new thin clients.

Here is what I know will run on them. Keep in mind their age in terms of performance, but they do OK with this software:

* GNOME ( Older versions work better, the last few versions have gotten much slower, but work).
* Firefox ( 2.0 works fine, but you have to use Shock 7 because of endian bugs in 9)
* OpenOffice 2.2
* Evolution ( Older versions, like Suse 9.3 work better for performance. Whatever is slowing down GNOME is slowing Evolution as well).
* xterm for character applications works great.
* X3270 for IBM connectivity works perfectly well.
* GIMP works.

Each thin client will consist of:
1 - NCD Explora 451 thin client running at 33Mhz, X, XDMCP chooser, RISC based.
1 - Power supply
1 - Keyboard
1 - 3 button PS2 Mouse
( No monitors are included)

The terminals have 100Mb ethernet, 1 serial port, 1 parallel port. They have 2MB of video memory and support 1024x768/ 16bit color. We will create a small amount of documentation if they are purchased on getting them to boot and provide any books that we still have.

Unfortunately they cannot be shipped out of the US.

We will be putting them on our web site soon, and we are looking for something like a few dollars for each unit + staff time to pack them + shipping costs.

Leave me a comment if you would like to communicate further.

More Desktop Work And Files

I have been keeping very busy lately. I have been working closely with the Evolution developers to get the last of our Evolution/GroupWise issues resolved so that we can deploy on SLED 10 SP1. That deployment is getting closer and closer, and we are down to only a few showstoppers.

We put a few more people live on HP thin clients and they are helping test the 3D desktop. The Citywide deployment of new thin clients will happen in June/July but we continue to increase the testing groups and are getting good feedback. It's interesting to watch people go from being wide eyed at the new desktop bling, to it being such a normal part of their usage that it's hard to imagine it not be there. Mostly, the 3D desktop is helping with navigation between running software applications. Window management has always been difficult for people, and this seems to really help.

I have mentioned it before: In the last few years I have noticed that increasingly people are struggling with file management skills. I'm finding that people tend to find a file once, and then add shortcuts to their desktop. I know this is coming from using Microsoft Windows at home and in previous jobs. The drawback is that if you ask people to move a file or share it with another employee they are losing skills in directories/folders. 5-10 years ago people would mostly open a software application and then use that program to interact with files; and now they are working in reverse and using the file manager for navigation and using that to launch software.

On our beta desktop I have hooked up the "MIME Bars" that I mentioned previously, and they are working and people seem to be happy with the results. The dialog opens when you double-click on a file in Nautilus and when you interact with attached files in Evolution. This is hiding nearly all of the directory structures and should reduce our support calls greatly. As much as it pains me that we (all of us in the IT field) are losing this file management conceptual issue, it's more important to make it work for the entire Bell Curve of employees and help them be productive.

Each person has a Desktop, Pictures and Documents directory and all applications now attempt to place files into those areas. Shortcuts to those directories are added to their desktop automatically so we can give consistent instructions to all employees. (shot below).



Because of the size of our deployment, each major software application is running on it's own server, so I had to create NFS mounts between the major applications and our NAS server so that they all share files. GNOME wants to have the Desktop and Documents directories in $HOME which would prohibit the desktop from working correctly. In other words, let's say you are running Firefox and tell it to save a file into Desktop, the thumbnail would never show up on your desktop because it's sitting on the wrong server. Sure would be nice to have gconf keys for these items. :) But for now I created softlinks to the NAS as seen below.

Spending Taxpayer Dollars

One of the things that amazes me about other Governmental agencies is the huge disconnect they seem to have in regards to where the money is coming from to pay for their computers and infrastructure. People are upset about their property taxes going up, and then go to their jobs and install licensed software without review, and purchase client/server applications and hardware which everyone knows has the highest total cost of ownership. It's shocking beyond your dreams how much money other agencies are spending unnecessarily.

In my almost 15 years of working for Largo, I have always developed plans and tried to implement ideas that find the right balance between functionality and being respectful to those people that are paying the bills, the taxpayers.

Under the watchful and thoughtful eye of myself and our staff, we received approval last night for our new thin client purchase which will be deployed to our entire City. Our current NCD Explora 451 terminals will be retired after a 10 year duty cycle and replaced with a quantity of 560, HP 5725 thin clients. Our price was around $625 each, including upgrade to 1GB memory, 2GB flash drive, optical mouse, PCI expansion slot and ATI video card. Every attempt was made to purchase devices to have the longest possible duty cycle and lowest possible TCO. Duty cycle should fall somewhere between 7 and 10 years. If they last 10 years, that's $62.50 per desktop per year for hardware. What would shorten the duty cycle closer to 7 years would be something coming in technology that we cannot even envision at this time. All of you know how quickly things change.

HP was very excited about our experimental work with 3D desktops to thin clients, and made the required changes to their Debian Linux build to support ATI video cards which can be added to the PCI expansion slot. All employees will have fast and stable devices that support 3D desktops, and 3D software and we can deploy to them software on any operating systems that are the best fit.

Thank you to everyone that has emailed me with your thoughts as they have helped during this R&D process. Also, thank you to everyone that has made software changes to allow your software to run better on thin clients and via remote display. The user experience is improving all the time, and hopefully the taxpayers are happy too.

PS: As of May 1st, our dress code changed and my 15 years of wearing a tie to work has ended. If I seem to have an extra happy kick to my step, you will know why. :)

gnome-terminal Still Too Slow

I was on vacation for almost 3 weeks, and have started to poke again at my various projects.

Over the last 4-5 years, every time that we upgrade or test new versions of GNOME I have tried to slip in gnome-terminal to my beta testers as a replacement for good ol' xterm. gnome-terminal as everyone knows has a much better user interface and fits really nicely in with the rest of our applications. Emulation works fine, and it *should* be ready for primetime. However, as was the case in previous years it's just way too slow to deploy. I keep getting feedback about the 'slowness' of character applications on our network, and must say that I agree. We are using modern thin clients now with RENDER, and any scrolling or cursor movements is still painful. I certainly understand the difference in fonts between gnome-terminal and xterm, but if it cannot be used or deployed they don't have a lot of value. Maybe gnome-terminal needs an option to run old school fonts or rendering for situations such as this?

So, unfortunately for this next round of GNOME upgrade, we are moving ahead with xterm again. gnome-terminal remains my favorite application that I can't use. :)

Shot below is the type of application that I have to deploy along with the version that I tested.

Evolution 3.0

Lots of talk about GNOME Of The Future. So I took some time to think about Evolution and reviewed our support calls and came up with my desired Evolution 3.0. I do also feel strongly that we need to sit on a feature freeze for a point release or two and just work on the internals and bug fixes. I also believe that Evolution needs to move away from being a 'clone' and create it's own identity.

There also is a lot of talk about increasing GNOME market share. I really believe the biggest growth area could be in the business environment, and groupware is one of the most important pieces. Email/Calendaring, Web Surfing and Word Processing are the keys to that market segment.

Improvement Areas:
- Proxy into GroupWise creates a massive folder tree, that is too hard for users. For many if they have to scroll too far, they will never find it.
- People have trouble sometimes with window management, and using the stack and working on multiple messages at the same time.
- Searches are not retained.
- Tomboy notes should be better integrated into email messages and calendar events. It sure would be nice to have notes that you have taken and attach them to a meeting.
- People still talk about and miss the old Summary page from version 1. It was nice to get a high level view of various components.
- More thumbnails please in the attachment area.
- Hide things like .zip files from users and extract them into individual files with thumbs. They shouldn't have to know how to extract the files and where to save them.

So I created mockups. Harish, Chen, Varadhan if you could go ahead and have this done over the weekend, that would be great. :)

Daylight Savings And Then Conference Rooms

This week started out on a sour note with messing around with some daylight savings issues. Apparently some fully patched Suse operating systems still are not working correctly with Evolution syncing with GroupWise calendar data. I poked around and was able to get it working well enough to make it until April. They moved the daylight savings time back to save money and resources, right? :)

Solveig Haugland is here this week teaching OpenOffice classes. The user community has for many years posted paper monthly calendars on the conference rooms with future meetings. This data is helpful, but of course is obsolete the second a new meeting is booked in the room. I have had an idea in my head for a while to place a flat panel monitor in the hallway to perform the same function. So I wrote a small routine to take GroupWise resources and generate a screen showing the current meeting, meetings later in the day, and future bookings. In the case of our IT training room, this allows people walking up the hall to see future computer training opportunities. It's very simple, you just log into an account that has a muted wallpaper and hides the gnome-panel and it kicks off a process that reads the right .ics file. It then sorts and parses the data and then uses ghosd to display the information to the screen. It flashes and reloads every minute. It's kind of cool to watch the classes shift and move up as time progresses through the day.

Shot below of what people see in the hallway on the flat panel monitor:

MIME Bar Update

Thank you to everyone that responded to my blog about experimenting with creating MIME bars to help users with files. I have made some change based on feedback. Anyone hacking GNOME that wants to work in this area, feel free to hit me up on the IRC. I'm going to focus starting tomorrow on direct printing and being able to bypass starting a full application in order to print. Photos should be pretty easy, and I saw some information about printing OpenOffice documents on the command line. I'll start poking in that area.

Today I pushed out my first Acrobat Document document helper for email and Nautilus to beta testers. I reviewed our support calls and found that we get a lot of requests for help in extracting single pages from a PDF document and then inserting that page into OpenOffice. Often we will receive something like a 20 page PDF file, and they want to include only a few of those pages in a memo they are creating. So I created a button that does that very thing. It runs 'convert' and extracts the pages in PNG format and places them into their Pictures directory. Previously users were using GIMP, and had problems remembering the steps.

Here is the bar that comes up for application/pdf files:



Here is what happens when you select the option to extract the PDF into multiple PNG images. Nautilus creates thumbs of the resulting files.

MIME Tiles Testing

I am still working on our next-gen desktop server and have been thinking about problems areas, based on observations and reviewing the support desk calls for the last few years. I'm trying new ideas on the following improvement areas:

- Many users never right-mouse click, ever. They do not understand contextual menus, and their workflow is 1, then 2, then 3. This is similar to old character based software. This doesn't seem to be based on age of employees, it's just how they think and work on computers.

- Most users have no idea about file types, file extensions, where things are saved, and what to name files. They want to click and tell the computer *what* they want to do, versus understanding the steps to make that happen.

- Users butcher file badly, and for instance save documents called David.odt and david.odt in the same directory, which often leads to problems for them later.

- Users do not use option menus well, and program names in applications like Evolution are of no use. GIMP does what? EOG does what? They know OpenOffice, but mostly they want to see words like "Edit", "Save" and "View" versus program names.

- Users do not understand applications like gnome-screenshot. That program has been ineffective because once the dialog comes up, they don't know what to do with the window. Sometimes they want to save it, but they don't know where the file is being saved. Sometimes they want to email it out immediately; but they don't know how to do that with the current UI. And it saves in PNG format, which people not on Linux sometimes cannot open.

So what I am testing, (with positive feedback from our beta group) is MIME application tile bars that I wrote with python/Glade. The bars come up for Pictures, Documents, Acrobat, Video and Audio files with unique options for each. For instance the Picture Bar has replaced gnome-screenshot/PrintScrn, and also comes up when you double-click on photos in Nautilus and when you receive pictures in Evolution. A consistent interface is provided in all interactions with pictures. The tiles then tell them exactly what will happen to the picture, without further skills.

I know this makes advanced users cringe, but I haven't removed their ability to manaually save files and move them around. At the most, it's one more click for them.

Here are shots of what I am testing.

[ Bar launches when you double-click on photo from Nautilus ]
[ When attaching to Evolution, all formats are converted to .JPG. They can still manually drop-drag from nautilus if their skills are more advanced and they understand file formats ]


[ Same bar launches from Evolution on picture attachments]


[ Same bar comes up as drop in replacement for gnome-screenshot ]


[ Documents bar, opens when double-click on OOo and MS Office files ]