Compiz In 3D

Many thanks for "wodor" for updating the anaglyph plugin to compile with the latest version of Compiz. While certainly one could not sit and work at their desk all day with this plugin enabled, it shows that the community is thinking outside the box. The current plugin requires the old style red/blue glasses but I made the recommendation today to see if it can be changed to work with the newer polarized style.

Keep up the good work guys. Not everything can be deployed, but I'm enjoying the ride.

(shot below)

Largo Updates

Things have been busy for me in the new year, and I thought I would relay my current projects:

Evolution Debugging
I have been working with Sankar and Chen on improving Evolution on SLED 10 SP1. We have made some gains, but still have more work to reach my goal. I received -debug packages from Novell and we are now auto-generating crash dumps when users crash. The good news is that all of these patches should be landing in 2.21 as well and will help advance Evolution connectivity to GroupWise.

Kyle TX Visit
The City of Kyle TX did a site visit and sent us a really nice email message with their observations. I think that people in general have seen so little of the Linux desktop and Compiz that they are shocked when they see how far it's come in the last few years. We demonstrated license free authentication, desktop presentation, email, web surfing and groupware. We also demonstrated single click access to MS Windows applications. I'm very glad they took the time to visit and wish them the best of luck.

Testing NoMachine NX
We are evaluating NoMachine as a replacement for Metaframe For Unix. We use Citrix for our distant remote sites that are connected over lower bandwidth lines. We have always had to keep a Sparc machine around just for this function. So far, NoMachine is running well and they seem to have some things on their possible road map that will be very useful to us; PulseAudio support and compositing for Compiz.

New OpenOffice Server
Our current OpenOffice still runs well and has been in production since 2003. However, it needs an operating system refresh in order to allow me to build the current PulseAudio. Pulse will allow the users to hear sounds in their Impress documents and anywhere else where sounds are present.

Weather-Wallpaper
This is completely bling, but the users love it. I have added weather-wallpaper to the desktop server. It automatically generates wallpaper based on the weather in the Largo Florida area. By default it requires users to enter the NOAA code for our area. Weather-wallpaper creates a directory called .weather-wallpaper which contains your personal settings. So all I did was get it working correctly and put a tarball of this directory into /u2/local/skel which is extracted for the users immediately before it's launched. This eliminates completely them having to do anything, and restores it back to a working state if they tinker. (shot below)

Running Hundreds Of Evolution Users

Over the last few months, we have had an increase in number of concurrent Evolution users, and also have had an increase in calendar and email usage. Evolution makes heavy use of cache when interacting with GroupWise and we were starting to have disk IO performance problems from 8am until about 10:00am. This is the heaviest load, when everyone is downloading heavily from the post office. The disk drives were getting so busy that the UI was starting to slow and performance was not acceptable. Interestingly, CPU usage was under 10% even with 250 people.

We had just ordered our annual allotment of servers. We use a trickle approach and move servers around based on changing requirements. Three new HP DL580G5s arrived . The older DL580s used 15K 3.5" Ultra320 SCSI 72GB drives. The new server uses 10K 2.5" SAS 146GB drives. Below are the specs, the difference in speed is amazing. We also had previously used RAID 5 which might have contributed to the bottleneck. We configured the new server and used RAID 1+0. A backup and restore of /home and IP number change the new server immediately went live with only a 15 minute interruption. The new server is so fast that when you hit [ Send/Receive] the dialog flashes and disappears so quickly it can barely be seen.

Disk performance is excellent even with 250 people:

iostat -x 10

Linux 2.6.16.54-0.2.3-bigsmp (oa3) 12/20/07
avg-cpu: %user %nice %system %iowait %steal %idle
0.97 0.00 0.24 0.01 0.00 98.81

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.00 71.90 0.00 22.30 0.00 1507.20 0.00 753.60 67.59 0.01 0.43 0.25 0.56

Previously, the drives were %util from 75-100%

Technical Specifications:

• Hard Drive Capacity : 72.8GB
• Generation: Ultra320
• Data Transfer Rate: 320 MB/sec
• Rotational Speed: 15,000 rpm
• Form Factor (Drive): 3.5-inch low profile
• Interface: Wide Ultra 320 SCSI; LVD
• Data Storage Device Type: SAS (Server Attached Storage) device
• Hard Drive Device Type: Hard drive for server/storage unit (Hot-plug)
• Height: 1 inch
• Pin Configuration: 80 pin Hot Swappable/ Pluggable
• Hotswap Tray: Included (Attached)

Technical Specifications:

• Hard Drive Capacity : 146GB
• Generation: SAS
• External Data Transfer Rate: 3.0 GB/sec
• Rotational Speed: 10,000 rpm
• Form Factor (Drive): 2.5-inch low profile
• Interface: SAS (Serial Attatched SCSI)
• Data Storage Device Type: SAS (Server Attached Storage) device
• Hard Drive Device Type: Hard drive for server/storage unit (Hot-plug)
• Height: 0.591"
• Width: 2.75"
• Hotswap Tray: Included (Attached)

Largo Desktop Video

I have recorded my first video to show everyone the new Compiz-Fusion desktop on our thin clients. The speeds that you are see are under normal loads, and exactly what the users experience. I'm just about finished setting up all of the icons and software launch scripts.

Notes:
- The blinking is not seen on the thin clients, movement is smooth
- I demonstrate the MIME bars I wrote to make it easier for them to process files
- Evolution is being moved to a new server next week that is much faster, you are seeing the speed of the current machine. Preview pane display is *much* faster on the new server.
- Video encoding for the web is new to me, so I'll do better next time. :)

Bliss

Bliss is your first 4-way Quad Core server. :)

DBera Is My Beagle Hero

We use Beagle extensively at Largo for searching historical documents. We have hundreds of thousands of documents going back 20 years. Each night I build a static index of each department, a static index of the city-wide shared directory and then the users personal directories. One of the requirements we have is to block users from searching departments of which they are not a member. We also had the requirement that we wanted users to be able to search individual static sources as well, to reduce the number of hits. The only way previously to do this was to have a zenity front end run before Beagle, and display all available indexes available to each user and then allow them to toggle the desired sources. It then passed this list to the beagle daemon and started the UI. Ok, it worked. But if you wanted to change your criteria you had to close Beagle and relaunch. Not very friendly.

So DBera cooked up a small fix that allows you to select from the sources via the UI itself. When a user runs Beagle now it figures out which sources they have access to search. It then starts the daemon with all of these sources and displays a dialog as below:



The UI then comes up and you are able to alter your search criteria on the fly. While all sources appear in the pulldown, only those connected with the beagle daemon will return results.



It's working perfectly for us. There is an ongoing conversation about how to integrate something like this into the main build. The patch right now is hard coded for our needs, but it sure seems like this would be helpful for other people. If you have comments or suggestions, add them to the feature request.

DBera, thanks for this step forward.

Friday Afternoons

Sometimes you do fun/stupid things on a Friday afternoon, especially after a week of troubleshooting and debugging.

I can now confirm that Stella works fine with Compiz-Fusion and Pulse Audio via remote display to thin clients. :) [shot below]

Have a great weekend all.

3D Desktop Release And Testing

The replacement desktop server for our thin clients is nearly ready, but unfortunately vacations and Christmas have pushed the go live date into January. That gives us time to finalize, test and do some training for volunteers.

The new desktop server is running OpenSuse 10.2. Users can pick running regular 2D GNOME or they can enable 3D (Compiz-Fusion) if they so wish. The login scripts check hardware at login and if a user is configured for 3D and logs in with an Xserver that doesn't support Compiz, it automatically drops back into regular metacity.

I created the following cheat sheet which I will be handing out in the class tomorrow. I always gain more respect for technical documentation writers when I try and create content for 'regular users'. Let's see what the users say tomorrow :)



I also have experimentally added avant-window-manager to the desktop which is enabled via icon for those that like to try new things. This provides a 3D panel. Because users like to delete things, the startup script automatically adds an icon to the far left of avant that looks like our City seal; this icon when clicked opens the application-browser. That way they never get themselves in a position that they cannot reach software applications. Further testing is ongoing, but seems like it's deployable and runs well over remote display.

Bug-Buddy-Largo

I know there have been some blogs about the future design of bug-buddy. The current design has not been effective: When a program crashes it's very likely that users have lost work or time and they usually aren't very happy. They normally never go through the process of submitting bug reports and just want to get back to the software. So lots of useful information by hundreds of people is being lost and not getting back to the developers.

Bug-buddy needs a way for Admins to dump the information to a directory and hide the GUI from the users completely. It's my job to review them and get them submitted. I was happy to find that the hooks into bug-buddy from a crashing application are not deep, and that I could slip in my own script called bug-buddy to perform a fully automated dump.

If you want to get dumps from your users, the following simple script might be useful:

more /opt/gnome/bin/bug-buddy
#!/bin/ksh
#
# $1: Program that crashed
# $2: Process ID
# $3: Version Number

/opt/gnome/bin/zenity --info --text "Currently dumping crash information, Evolution will restart when finished" &

pid=`echo $2 | awk -F "=" ' {print $2}'`
prg=`echo $1 | awk -F "=" ' {print $2}'`

stamp=`date +%Y%m%d%H%M%S%N`
outfile="$prg.$stamp.$USER.txt"

/usr/bin/gdb -p $pid --eval-command="thread apply all bt" --batch --nw > /home/largo/tmp/evolution_crashes/$outfile

Trust Your Business

I live very close to downtown Safety Harbor Florida. It has a very small town feel, and lots of people walk around at night and window shop and sit in the gazebo. One business had the idea of running electronic slides in their store front. Too bad they didn't know they could run Linux. :) (pics below).

The Desktop Hexagon Cubed?

I'm prepping the Compiz-Fusion machine for city wide deployment and working on feature requests that we had on the older GNOME desktop. Previously, our support staff would use x11vnc and vncviewer to support users that are having problems. We have had requests to allow anyone in the City to use this capability. I wrote a small zenity and python front end and now employees can grant access for viewing, and also remote control other users desktops. Months ago when I did testing, Beryl didn't work well over VNC. I brought down the latest version of x11vnc and it's working much better. One can remote control another 3D desktop while using your own. (Shot below). Now, does this give you a headache or what? :)

Evolution, Compiz, Thumbnails And Mplayer

I was on vacation last week, and got right back into my projects this week. I had movement on many fronts, most of it good.

Evolution: Now that all of the users are live, we have been shaking out some bugs/issues and possible regressions. We don't use the keyring, and the dropback code code seems buggy. It's asking users for passwords in a manner that is not repeatable. We are also getting some EDS crashers for those with heavy calendars and those that proxy often. SR's entered and working on reaching a better milestone.

Compiz: Interestingly, I'm finding the 3D desktop is really increasing my requests for beta testing accounts. Normally, users are very slow in moving to new technology and this seems to be exactly the opposite. Cool stuff. I experimented this week with the widgets layer and video plugins. When the widgets layer is enabled and you press F9, applications that you have configured to sit in that layer display. I wrote a very simple python/Glade UI to display our current local weather. I configured all Compiz sessions to put that new UI and a clock into the widgets layer. So now all users can press F9 and see the local weather. (shot below). It would be great if someone wrote something a little more polished :) These images are statically named on weather.com.



Thumbnails: Kevin DeKorte wrote some great programs that work with mplayer and has been a big help in making some changes. I am using his nailer program to generate thumbs for video clips in Nautilus. I had 2 real world issues that most thumbnailers forget: 1) Sometimes files that users have downloaded or saved are incomplete or damaged, the thumbnailers have to account for that fact. 2) Sometimes a thumbnailer will get a file and lock/spin and not be able to generate an image. They need to have a timeout and give up after a few seconds. I had to turn off the evince thumbnailer for this very reason. You can't have hundreds of stuck processes on a server.

Mplayer: A patch had been posted on the list to use pulseaudio with mplayer. The current SVN version is still looking for the older polyp libraries. This meant it failed to find the libraries and fell back and used ESD. While ESD works with our pulseaudio daemon, this is definitely cleaner and works great. I also found a patch to have mplayer work with the video Compiz plugin. However it seems not to work with remote display, and will require more investigation. The video plays all blue, similar to how XV looks when you move focus away from the window. I put back the XV build and will continue using that until I figure out the problem. I moved mplayer into production today and we shipped forms to the users for signatures to gain access.

Pulse Audio, mplayer and gnome-mplayer Moved To Beta Group

After doing some R&D on a laptop, I got pulse audio working as desired and sent it to HP. Their engineer built from source and compiled it for their thin clients and returned it to us a working build. This build has been moved out to about 20 people and up to this point the only sound they have gotten is standard GNOME login and logout clips.

After testing a few ideas, I have settled on using the mplayer engine, and the gnome-mplayer UI for our deployment. Many of the players have the controls running in a UI separate from the display canvas, which is just not deployable for 'regular users'. Users lose windows in their stack, and often don't know how to find them again. gnome-mplayer is wrapped around the display canvas. It's got only the most basic of controls and should be an excellent fit.

In the beginning of testing, I was concerned that remote display and Compiz would make playback too slow for deployment. However, it turns out that the business use content (quicktime, windows media, real) works just fine. Below I am attaching two shots. The second is the IronMan quicktime (way cool btw!)

Evolution Update And OpenOffice 2.3

I finished moving over the 750 users to Evolution on SLED and last week was spent tuning the server. We have been using the GroupWise backend for about 18 months, and many users had built up massive calendars which were being pushed down each morning and slowing down the server. We have decided against retaining calendar cache from session to session because it's proven impossible to explain to people that they are seeing calendar events that might not be accurate during the period that the post office syncs all of the new events. It's better to just delete cache and push down a fresh set of data; once they see events it's 100% known to be current. Using the environmental variable GETQM_TIME_INTERVAL I tuned their calendars to only reload every 15 minutes to reduce open SOAP connections. We also activated auto-purging of old calendar events in the post office. Things are running much better this week as we continue to tune, here is a 'top' with 210 concurrent users running Evolution; I think we could get another 100-200 concurrent on this machine if required.



OpenOffice 2.3 came out yesterday and I put it live for the whole City. So far we haven't had any support calls related to the new release. The biggest features I have noticed so far are the new icon artwork, centered composer window and brand new chart module. Thank you to all of the QA people that make these upgrades so painless for me. As I have mentioned before all I have to do is install the RPM and change 2 lines in the OpenOffice launch script to point to the new release in /opt. I keep the old versions and a rollback is just as easy should major problems be discovered.

Humble Book Announcement

A year ago I was approached to write a book about Thin Clients, and it's finally been released. I received my physical books yesterday and it's very rewarding to see a project like this come to completion.

I spent many hours thinking about ways to present this material. The problem is that every single reader will have completely different requirements, user counts, staff, software and money. It would have been easy to write 400 pages describing in great detail our exact thin clients, our exact distributions and software packages. But that wouldn't have been useful for most readers. There is also the issue of technology churn; thin client models change all the time, and operating systems are in a state of constant change. So I decided the book should be designed to offer *ideas*. So often we get phone calls from people that are thinking of moving to Linux and/or Thin Clients and they don't even know how to get started or what options are available. It also covers what is often the biggest challenge: the user community and those in charge of the company or organization.

I'm hopeful that I have found the right balance to help be the middle person to allow many more people to deploy the wonderful things that all of you have written.

The link for the book is here.

Mid-Week Vacation Day

I took a day off yesterday to avoid the weekend crowds and visit Busch Gardens Tampa. It's about a 30 minute drive from my house. The park wasn't crowded and I was able to get on most of the rides. I was most looking forward to riding Sheikra again since it got upgraded to having cars with no floors. Sheikra is one of only 4 Dive Coasters in the world. You are seated 8 across and 3 deep in stadium seating, cranked to the top of the hill and then for a few seconds are halted at the top looking straight down 200 feet.

Photos don't do justice to the size of this drop when you are standing next to it; or looking down from the top :)

Pulse Audio On HP Thin Clients

Yesterday we had a conference call with HP and worked out a plan to have Pulseaudio built and installed on our HP t5725 thin clients. This was always a phase 2 goal, and now that the thin clients are installed, this is my primary project. I have been impressed with Pulse so far. Sound on Linux is horribly inconsistent (Alsa, OSS, ESD and NAS anyone?) and difficult to deploy. Various applications support different sound libraries, and what Pulse does is support enough of them to allow most things to just work. We hope to deploy this sometime in October or November.

Here is the only change I had to make so far:
protocol-esound.c
74c74
< #define MAX_CACHE_SAMPLE_SIZE (1024000) ---
> #define MAX_CACHE_SAMPLE_SIZE (2048000)

Increasing this cache allowed the standard login.wav GNOME sound file to play over Esound. Some of these probably should be environmental variables or settings, so that they do not require a recompile.

Our biggest sound requirements have been resolved:
Esound handles GNOME login sounds and other Linux applications
OSS is handled with the padsp binary, this allows Citrix clients to play all of the sounds from Microsoft Windows apps.
PulseAudio is handling the shockwave plugin in Firefox, and also works natively with Xine to allow playback of Windows Media, Quicktime and Real content. For grins I'm also testing remote display of TV tuners and DVD playback.

Here is a simplified diagram of multiple servers and applications all redirecting sound to the thin client and it coming out the attached speakers.

Finally An Evolution Upgrade For Us

As of today I have checked in and approved the Evolution build that we will used to upgrade the whole City on September 4th. We have been running Suse 9.3 for several years on a build that I would give a letter grade of a "B". That was the first release that we used with the GroupWise postoffice. It had some threading problems and not all of the GroupWise features were enabled.

We entered the beta program for Evolution on SLED 10 in the fall of 2006, which then continued into the SLED SP1 beta. Focus on this release has been interaction with the GroupWise calendar system, proxy, free/busy, meetings and appointments. I was a bit disappointed that it took one year to complete, but that isn't the fault of the developers. My view is that Novell should add additional people to the project; there simply are not enough human resources. This is a huge piece of software for use by corporate and enterprise customers, and needs more love.

The new release is labeled 2.6, but contains a lot of code from later releases. The mail view supports the triple panes (shot below):



The calendar view contains the most amount of changes. The mini-calendar has moved to the left and memos are now listed on the right side. The proxy feature was picked by our users as the most important feature to get working, so I focused on debugging and testing that functionality.

You might notice all of the "On The Web" calendars. We have developed some scripts that automatically generate .ics files each night with information useful to our users. For instance, in our financial system we can enter leave requests. We generate departmental Ical files which show which employees are off which days and can be checked easily at a glance. Very cool. (Shot below).



If your organization has GroupWise and you are thinking of moving to Evolution, I would highly recommend testing this release on SLED 10 SP1. It's been tested and QA'd greatly with that backend.

Thanks to Harish, Chen, Varadhan, Srag, Flex, Sankar and everyone else for working on this project with me.

Compiz-Fusion Looking Great

Once again I synced with the latest version of Compiz-Fusion and things have never looked or worked better. My notes:
- Previously the transparent hexagon was too slow to deploy over remote display, especially when you had software packages on the 6 sides. It's much faster now. Spinning the hexagon barely shows up as a blip on the server, seems like it would scale just fine.
- It's so much faster now in fact that I was able to enable the new spinning gears plugin for testing. Eye candy for sure, but the user community loves that kind of stuff.
- The new cubecap plugin is awesome. It gives you a lot more control over the image, and correctly supports transparent PNGs. It also works correctly if you have more than 4 sides.

To everyone that is working on this project, Thank You.

Vacation In New York & New Jersey

I took off a week and used the time to do some traveling. There were a few things that I wanted to do in the New York and New Jersey area.

I went and saw a ballgame at Yankee Stadium before the new one opens. Lots of history, and it was nice to sit and have a dog and watch some baseball


I then made my way down to Atlantic City and pulled some slots and walked the boardwalk. I left with a few less dollars in my pocket. :)



The grand daddy part of the trip was riding Kingda Ka at Six Flags in Jackson New Jersey. It's the worlds tallest and fastest roller coaster. You go 0 to128MPH (206 kph) in 3.5 seconds and then are launched up 456 feet (139 meters). It's quite something and I rode it 3 times. The picture below does not do justice to the size of this coaster.



I also loved El Toro, a wooden roller coaster with a 76 degree drop. It goes over 70MPH (113 kph) and is excellent.