Etch Becomes Lenny

HP recently released a new thin client operating system based on Debian Lenny for the soon to be released 5745 model. They support the older 5735 as well. While they don't support our 5725s officially, I felt confident that it would work.

When HP released Etch, I made our custom modifications all self contained and easily linked into /etc/rc2.d. So it was easier than ever to accept this upgrade and prep it for release to our employees. Newer X.org and Mesa libraries should make it easier to test gnome-shell and Compiz upgrades. After just a few hours, it's already running and stable. The 5725s worked fine; just needed to install the SiS driver. I should be able to deliver this to additional testers in the next few days. The UI and login screens haven't changed, so they shouldn't even notice a difference.

I am introducing one new feature, which is going through my QA process at this time. We have had a digital smart/white board for a number of years that has been under utilized because it never had a permanent home. IT is getting a new conference room, so it's finally going to get the wall mount it deserves. It works just like a white board with magic markers, except it's done electronically. One picks up 'fake' colored pens, and as you write the 'ink' appears on the screen through a projector. You can then save your pages into PDF or image format, clear the board with a click and start adding more notes. The company (Smart Tech) fully supports Linux, and I am at this time loading their software on our thin clients. This is one of the rare exceptions where the software is better suited to be running locally. So what I did was create a custom XDMCP connection that once activated waits 60 seconds before launching the dashboard type interface used by the board. This gives you enough time to get logged into the GNOME server so the software doesn't sit over your authentication screen. I'll take some pictures of the board in operation once everything is tested and the room is finished.


The shot below is the chooser which is available on thin clients connected to the Smart Boards. This is the first thing you see after the thin clients are booted.




The shot below is the software which connects to the Smart board products. One can use a mouse or virtual pens to create output.



Update: The Smart Board is working on Linux and the thin client, pictures below.

Lots Of QA Work And Testing

As December begins I have been reviewing my workload and the next 30 days is going to be busy with QA and software testing of several software packages. I have finished the next thin client OS upgrade and it's now in the hands of our Support division who will schedule it for deployment to our 550 devices.

My projects at this time, in no order:

Firefox 3.6 & Flash 10.1: Adobe labs released 10.1 and I have integrated it into Firefox 3.6 for testing. So far both products seem to be working. I'm also checking into the proxy and bandwidth issues for the Personas. Cool bling, but lots of traffic for hundreds of users to be hovering their pointers over the images and trying them out.

Evolution On SLED 11: We are waiting for integration of patches and official RPM to try and reach a milestone that will allow the rest of our users to be moved to this new version. About 190 people out of 750 are using the new version.

OpenOffice 3.2: It's getting pretty close and several features that users have requested are available in this new release. It also includes a feature in Impress that we bountied. I developed a new python/Glade GUI that pops open the first time you launch 3.2 announcing the new features.

Tomboy on SLED 11: We haven't upgraded Tomboy in a while, and I hope to soon install Tomboy on SLED 11 and get that deployed to our users to replace the current SLED 10 version.

Compiz & gnome-shell: Our current GNOME desktop servers are starting to get a bit old and I'm opening a new project to begin the process of moving to something newer. It probably will be OpenSuse 11.2. This would take many months, and probably would go live in the next 4-6 months. It looks like we will use compiz for one more iteration of the desktop and then explore gnome-shell on the next major upgrade around 2012. Gnome-shell is not yet optimized and tested for remote display and won't be ready in time. I'm going to check on the status of the C++ port and make a decision about which version to try and deploy.

ZeitGeist: I have promised to check out this technology for enterprise use, and will be doing that in the coming weeks. I have to consider how it integrates with Tracker, how to replace or compliment the features we have in Beagle, how to run this via cron jobs and not hammer drives during business hours. I also have to experiment with NFS mounting the data stores that Zeitgeist is mining, they aren't all on the same physical computers.

Thin Client DVD Support: I have been thinking about possibly loading a software DVD player on the thin clients for users that have permissions to mount DVDs. Increasingly they are receiving training and demo videos on DVD media. Lots to consider with this move, how to stop them from playing "_Dumb and Dumber_" ? :)

Lots to do, and always fun. My thanks to everyone that works on these technologies, you do the hard work!

DVD/CD Readers And Thin Clients

When we went live with our HP thin clients 2 years ago, one new feature we added was the ability to use USB sticks and transfer files. I know there are some tools for doing this already, but opted instead to use FTP. I felt that using a stateless connection would be more stable, and also allow us to use this feature when people are logging in with NX over low bandwidth networks. As long as the server can resolve the remote IP, it will find the FTP daemon running on the remote thin client. FTP already works fine over differing network speeds, and once the transaction is complete nothing is mounted back to the server. Very clean to me and has worked well.

After some conversations in our architecture meetings, we have agreed that CD/DVD readers are really no different than USB sticks; and in fact they probably can hold fewer files these days. So I made a last minute addition to our new thin client update to support these types of drives.

When you plug in a USB stick, it's appropriate to just mount it immediately. I then tuned the thin clients to always flush data every 10 seconds. So we tell people that if they place files on USB sticks, just wait 10 seconds and then they can remove the stick. udev sees the stick being removed and the mount is removed. This is working well, and avoids having to have some kind of "Safe To Remove" panel app that we know people don't use anyway. :)

Being that CD/DVD drives have to eject, I couldn't just mount the disk when it's inserted..or they wouldn't be able to eject the disk. The GNOME desktop is running on the server, so it too is not aware at all of these drives. So I wrote a very simple UI that pops up (no GUI Nazis please :) ), when a drive is powered on. This is running on the thin client itself. I tried to keep it to the bare minimum and so far it's working well.

When the USB CD/DVD is turned on, udev starts up the small UI. When they click on the Mount button, it generates a directory name based on the physical hardware and creates the mount. The whole idea of mounting things to usb0, usb1 and so on has not worked well because users never were able to figure out which directories were in use. So now they only see the drives that are mounted, and each has a descriptive name. In the shot below, you can see the directory mounted from the drive after it's mounted.




The users then double-click on the USB icon and up comes Nautilus which is FTP jailed to just USB devices on the thin client. Proftpd only displays files types which we have approved. To the user, this looks just like a regular file manager that they always use. In the shot below, pictures and videos are displayed from CD media.



The user then drops and drags the files to the desktop, and the FTP feature of Nautilus is activated. Thumbnail generates and the file is now stored on the server.



I'm sure there are other ways to do this, and this definitely falls under a WFM; but if you are deploying thin clients and wondering how to handle these devices...maybe this will be a good solution.

Groupwise 8 Upgrade + Client

The City uses Groupwise as our post office to house all of our email and calendar data. There were some mandatory calendaring requirements from our employees and GW best fits those needs. Last weekend, we upgraded from version 7 to 8. We are experiencing normal problems and awaiting some patches, but overall it went pretty well.

One of the things that comes up often in our contact with Novell is the issue of us using Evolution vs their Java client for Linux. We had tested the version 7 client a few years ago and it was not a good fit for us. So I checked out the new version with an open mind.

A lot of work has gone into making this release have feature parity with the Windows version, and they seem to be making progress. One of the things that the users really liked about Evolution 1 was the old Summary Screen, and Groupwise 8 has a similar feature (shot below).



Another nice feature is the ability to enter your own notes and comments on each email message as pictured below:



Overall from a feature standpoint, it seems robust and delivers in that regard.

So why don't we use it? The client runs on Java, and unfortunately it's S-L-O-W. To be fair, I installed the client on SLED 11 and it's running on the same server as Evolution 2.24. It flickers a lot at startup and on resizes, drag bars are painful to move, pulldown windows are slow. It's entirely possible that this product works perfectly on a local video card, but over remote display it's really pretty painful.

The other issue from version 7 that remains is that it consumes a lot of memory, and then continues to leak badly. When you are on a standalone computer, it might be ok for 1 or 2MB to leak at a pop, but when you have 300 users running, that suddenly becomes 300MB or 600MB for me!

I started the client and then opened top. Below you can see the client running along with the same email opened in evolution for my account.



After just a few minutes, it was already leaking (below).



I'm going to run it all day and see how bad it gets. At this point, there is no way this would scale and provide the performance that users expect in our environment.

It seems like from time to time there are discussions in Novell about Evolution vs Java, and where to use development resources. From our perspective and needs, Evolution is far better and we hope that Evolution continues to get the TLC it deserves.

HP 5735, Disappoints

I get email and know that some of you are thinking about thin clients in your organization. I have always tried to be honest and describe what works and what doesn't.

After a good amount of testing, we have been unable to get the HP 5735 to work well enough to put into production. They may work in other areas perfectly well, but our goal was to add ATI 9250 video cards to them and use them for the 3D desktop. Even though they have higher specs than the HP 5725, under no conditions have they ever run faster. 3D performance is very sluggish and certainly not fast enough for day to day use.

I know this isn't a perfect test, but the results seemed to correlate with Compiz "feel". It also could easily be replicated by anyone else. For my test, I booted an X server with just a xterm window and then connected to our remote Compiz servers and ran glxgears. The 5725 was the clear winner with 653 FPS. I also tested the onboard X1200 and then 2250 video cards.



We only purchased a few 5735s for testing, and will be moving them into our cyber cafes which don't use 3D and for that purpose they will work well.

We will continue to use the HP 5725 (which works great) until we have another device that performs as well.

gnome-shell Over Remote Display

It's been my pleasure to see gnome-shell mature over the last few months. I was in Boston last year when it started on the white boards, very cool stuff.

I have been wanting to experimentally get gnome-shell and Zeitgeist into the hands of 'regular users' for feedback. I built a Fedora Core 12 server, and got gnome-shell running. I then tested it over remote display. Clutter failed in my first test because it doesn't yet support 16 bit color depth. I bumped the thin client up to 24 bit color and tried again. I built a page on Live with my results and it's here

The visuals are not working correctly, but once it's up and running performance seems fine.
I don't have the expertise to fix this type of low level bug in clutter, but if anyone with the knowledge wants to find me on the IRC, I will gladly test patches and ideas.



We aren't the only people running thin clients, and one hates to not have this major feature available for enterprise customers that want to run GNOME 3.

Acrobat 9.2 Is Out

I normally don't blog about software updates. But from the Acrobat site when you attempt to download Reader, they are still pushing 9.1.2. That version along with 9.1.3 has lots of security alerts. After poking around for a while, I finally found that 9.2 is only available on their FTP site.

Yup, I know about Evince...but it's not an option for us at this time.

Evolution 2.24 Going Live, HP 5735 Work Continues

After a one year and not always pleasant process we have reached a milestone with Evolution on SLED 11 that has the features and stability with which to deploy. The shot below is my calendar with data loaded from webcals and from the Groupwise backend.

Note to Linux desktop vendors: Having an email client, word processor and browser are the key elements to success in creating more customers. Specialty apps can still run on Windows, but these core applications need to be robust and feature rich in order to entice people to use Linux as their desktop.

The good news is that these patches will be coming for all customers on SLED, and also have mostly been merged into the community build.



I've moved over about 120 users to the new version, and the top below shows the load with 80 concurrent. The box has some kick, :) , and will easily be able to handle our maximum load of 300 concurrent.




I also have been working with HP to get some better performance out of the 5735 thin client. It has specs better than the 5725, but for some reason is slower in speed. I received a new build to test and am working on installing that now. I'm sure we will figure out the issue and be able to start deploying the newer model around the City.

Detailed Specs: Thin Client Hardware

I finally had a few moments to start creating blog entries about our last thin client operating system upgrade. This blog will review the hardware that is currently in use here at the City.

As I have mentioned previously, our "live" production thin client is the HP 5725. We are taking the steps to allow some HP 5735s and HP 4410s to be integrated into our infrastructure. With up to a 10 year duty cycle, costs are significantly reduced with all of this hardware.

HP 5725
This device has been discontinued but is a fantastic thin client. We have hundreds of people on these and they have proven to be great workhorses. If pictures are not loading on your feed, go to my blog site.

Here is a picture of the inside of the 5725. Note that there are no moving parts, and that heat from the CPU is pulled away in an ingenious manner.



The onboard graphic card did not support 3D, so we purchased the optional PCI expansion daughter card and cage. We were then able to use the ATI 9250 card, which has a heat sink and has no moving parts as well.



A nice clean front, power button and 2 USB connections.



Lots of ports on the back, you never know what you will need in the future.



And lspci gives us some nice information about the internals:

00:00.0 Host bridge: Silicon Integrated Systems [SiS] 741/741GX/M741 Host (rev 03)
00:01.0 PCI bridge: Silicon Integrated Systems [SiS] SiS AGP Port (virtual PCI-to-PCI bridge)
00:02.0 ISA bridge: Silicon Integrated Systems [SiS] SiS963 [MuTIOL Media IO] (rev 25)
00:02.1 SMBus: Silicon Integrated Systems [SiS] SiS961/2 SMBus Controller
00:02.5 IDE interface: Silicon Integrated Systems [SiS] 5513 [IDE]
00:02.7 Multimedia audio controller: Silicon Integrated Systems [SiS] AC'97 Sound Controller (rev a0)
00:03.0 USB Controller: Silicon Integrated Systems [SiS] USB 1.1 Controller (rev 0f)
00:03.1 USB Controller: Silicon Integrated Systems [SiS] USB 1.1 Controller (rev 0f)
00:03.2 USB Controller: Silicon Integrated Systems [SiS] USB 2.0 Controller
00:0a.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 8d)
00:0b.0 VGA compatible controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (rev 01)
00:0b.1 Display controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (Secondary) (rev 01)

5735

The 5735 should be an excellent addition to our environment. I'm still trying to get a few technical issues resolved with HP and fully expect this to outperform the 5725 in every regard.

Once again the internals are solid state:



The 5735 needed an addon daughter card for our ATI 9250s, but also comes with a PCIE card as well. I ordered a PCIE based ATI FireMV 2250 for testing; but found that Etch didn't yet support it. That's on the back burner for the next update, very possibly we will get a boost in performance.



The 5735 made some improvements in the front. Users might want to easily plug in headphones? Yup! Microphone jack also provided (but not implemented by us at this time).



Lots of ports on the back, including a 4th video port and gigabit ethernet.



and lscpi reports the internals:

00:00.0 Host bridge: ATI Technologies Inc RS690 Host Bridge
00:01.0 PCI bridge: ATI Technologies Inc RS690 PCI to PCI Bridge (Internal gfx)
00:04.0 PCI bridge: ATI Technologies Inc Unknown device 7914
00:13.0 USB Controller: ATI Technologies Inc SB600 USB (OHCI0)
00:13.1 USB Controller: ATI Technologies Inc SB600 USB (OHCI1)
00:13.2 USB Controller: ATI Technologies Inc SB600 USB (OHCI2)
00:13.3 USB Controller: ATI Technologies Inc SB600 USB (OHCI3)
00:13.4 USB Controller: ATI Technologies Inc SB600 USB (OHCI4)
00:13.5 USB Controller: ATI Technologies Inc SB600 USB Controller (EHCI)
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 14)
00:14.1 IDE interface: ATI Technologies Inc SB600 IDE
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)
00:14.3 ISA bridge: ATI Technologies Inc SB600 PCI to LPC Bridge
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:05.0 VGA compatible controller: ATI Technologies Inc RS690M [Radeon X1200 Series]
02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit Ethernet PCI Express (rev 02)
03:06.0 VGA compatible controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (rev 01)
03:06.1 Display controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (Secondary) (rev 01)

4410t

A laptop footprint will help us greatly in two areas. There are people that want to be able to carry a device into conference rooms, and this should work nicely. The other area is that we have fat PCs currently in use that are used as thin clients most of the time. The users are booting Windows and then immediately starting NX with EVDO cards. We are testing the idea of using these devices for this purpose.

Looks like a regular laptop, but only costs $500. This device contains no moving parts and Debian Squeeze loaded perfectly, Intel video card is working well. Note to HP: the case has a slight fingerprint problem, lots of wipes needed during testing.



Ports on the side, this device was ordered with no DVD/CD reader, but it's available.



and lspci displays more details:

00:00.0 Host bridge: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub (rev 07)
00:02.0 VGA compatible controller: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (rev 07)
00:02.1 Display controller: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (rev 07)
00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 03)
00:1c.1 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 2 (rev 03)
00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 (rev 03)
00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 03)
00:1c.5 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 6 (rev 03)
00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 93)
00:1f.0 ISA bridge: Intel Corporation ICH9M LPC Interface Controller (rev 03)
00:1f.2 IDE interface: Intel Corporation ICH9M/M-E 2 port SATA IDE Controller (rev 03)
00:1f.5 IDE interface: Intel Corporation ICH9M/M-E 2 port SATA IDE Controller (rev 03)
02:00.0 Network controller: Broadcom Corporation Device 4353 (rev 01)
85:00.0 Ethernet controller: Marvell Technology Group Ltd. Device 436c (rev 10)

If anyone is considering a move in this direction, I hope this information is helpful.

Low Level Linux Idea

I don't know if anyone that works in these areas of Linux will see this, but here goes:

What would be really helpful to me is adding a few more ulimit commands to the shell. I'd love to be able to have control over the CPUs. I understand that this slightly crosses over into the area of the 'nice' command.

Here is a real world example: We run Firefox from a centralized server for our City employees. We also have a few cyber cafe terminals configured around the City, so that they can log into their personal email during breaks. I'd like to throttle those devices back so that their Firefox sessions only run on one CPU. If only one cyber cafe is in use, response would be fine. As more are used, they would only slow down each other..and not the rest of the City sessions.

Anything running once this setting was enabled would not even see the other CPUs and perceive that you are on a single CPU workstation. I can think of numerous places where this would really help me tune for machines that run hundreds of concurrent users.

Gotta Love the HP 4410t

I'm just about done with our thin client updated operating system, and am *very* impressed with the HP 4410t laptop form factor. It boots super quickly to the chooser, and the Intel Xorg drivers found in Debian Squeeze/Sid are excellent. 3D rendering is fast and responsive, video playback is fast and frame rates are high. I finally got Pulse installed and it found the audio hardware and playback is excellent.

My thanks for everyone that works on all of these software packages, you make my job a lot easier!

Shot below is two concurrent video and audio streams with cube turning. The 4410 was able to do this perfectly with no dropouts. This test was performed on the 1Gb ethernet port and everything is being pushed over the network.

OpenProj Performance Issue Fixed

We were having some slowness with OpenProj over remote display. In the area marked in black, scrolling was sluggish with certain types of charts.



So a bit of Google work and the issue was resolved. There is a java setting that seems to greatly increase performance to remote display. The setting is found in the run.conf file which is placed in $HOME/.openproj. Change the JAVA_OPTS line to disable pmoffscreen:

JAVA_OPTS="-Xms128m -Xmx768m -Dsun.java2d.pmoffscreen=false -Djava.util.prefs.userRoot=/home/drichard/.java/.userPrefs -Djava.util.prefs.systemRoot=/home/drichard/.openproj"

Detailed Thin Client Specs & Designs Coming

One of the things that people mentioned on Amazon concerning the thin client book that I wrote was they wanted to have detailed information concerning specific hardware. When I was writing the book, I considered this fact and had to weigh also the time that it takes to get a book to press (over a year). The computer world changes greatly in 6 months, let alone 12 or more months. Proof of that fact is that our HP 5725 thin clients from 2 years ago are already discontinued and the 5735 has completely different specs.

I find that this blog has been a great resource for *me*. I frequently look back at things that I wrote and that helps me refresh my memory on current projects. So over the next week or so I'm going to publish detailed blogs concerning my re-design of our current thin clients to newer operating systems. I'm also going to include information concerning changes made, because for the most part they occurred based on direct user feedback. If you bought that book and wanted more information, consider this a free Appendix. :)

The first few blogs are in my mind, and will be:

1) Thin client hardware, detailed specs of 5725, 5735 and 4410t (laptop) devices. I'll open them up and take pictures.

2) Operating system installs. The 5725 and 5735 have Debian Etch on them, but I had to do make a good amount of changes to make them work for our needs. I'll describe what I added and how I got 3D enabled. I'll also describe how I got common and consistent interfaces and code running on all of them. All 3 thin clients have identical functionality.

3) First Boot. I don't like plug-and-pray/generic Xorg.conf files because they sometimes give the users unexpected results. Have you ever tried to explain to users why their icons are different sizes periodically? On first boot, the thin clients ask appropriate questions, and precise Xorg.conf files are generated for their hardware. Users are not allowed to change hardware, so we are assured that an IT staff member will make selections.

4) Chooser. After a thin client boots, the users select from multiple computers servers. The old thin clients would start a second X instance on vt9 and the chooser would continue to run on vt7. This was the default behavior from the HP build. I didn't like this design and modified it. I'll describe how I got one Xserver to quit and another to start, and the technical benefits that we now gained. Hint: You can't kill cleanly a running Xserver from a script and expect the video to continue working!

5) USB Access. When a user plugs in a USB stick, it's only known to the local thin client. I did not want the server to attempt to mount these ports. The old OS used FTP to allow the users to gain access to their files. I made some big improvements in this design. I built on what was working, and made it much easier.

So that's it for now. I'm sure I'll have a few more topics. I spend a good amount of time researching issues online, and things that people post are *always* helpful to me. I'm always hopeful that I'm returning the favor.

Interesting Usage Stats

We have 2 servers that allow users to get GNOME desktops from our thin clients. I wrote a little script to display usage stats just for my information. Now that we have been live for a while, the data is probably settled to a plateau. Just north of 50% of users have continued using Compiz/3D effects. Around 1/4 of those that enable 3D effects also turn on Avant Window Navigator. I would be happy to share all of my observations at a Largo Hackfest (cough, cough), once the dates are scheduled! :)

The first server (A):

=====================
Total Users: 162
=====================
3D Users : 78
NX Users : 32
AWN Users : 19
=====================
Total On 3D Capable Hardware : 130
Percent Running 3D : 60 %

The second server (B):

=====================
Total Users: 118
=====================
3D Users : 41
NX Users : 32
AWN Users : 14
=====================
Total On 3D Capable Hardware : 86
Percent Running 3D : 47 %

Thin Client Laptop Making Progress

I continue to work on staging a common functionality build for the HP 5725, 5735 and 4410 thin clients. I'm making good progress, and have alpha testers on all of the hardware configurations and pushing them updates every few days. Xinerama is working well on 3 screens, and the people love having the real estate. We received a second 4410 laptop thin client for testing, so I have moved one to my desktop for fulltime usage. I'm tinkering with some battery notification ideas. The GNOME "desktop" is on the server, and your battery status is not known at all to the server. I think I am going to just cron a job to check for status on the command line and then pass them a simple zenity dialog when it gets low. Looks like these devices are going to run about 3 hours on battery, which should work great for us.

I downloaded the latest xorg intel packages for Sid and they are working well. No more Vesa, and I can put it into the native 1366x768 wide screen mode. Nice crisp clean fonts. The device detects if you are on the wired network and then makes use of XDM and provides 3D (shot below). Otherwise NX is used.

Up next for me is a cleaned up chooser GUI, notification dialog when network cord is not plugged into the desktop thin clients, pulseaudio cleanups, ftpd configuration for remote access to USB sticks. This update will offer the users a fresh set of features, and build on the success of our current deployment.

Thin Client Upgrade R&D Continues

I haven't blogged in a while, but have certainly been busy. I did take off a week for vacation and had a nice time with friends and family and visited Cedar Point in Ohio to ride some roller coasters.

My current R&D project continues- I'm getting a new Linux operating system upgrade ready for our 550 thin clients. There were a few features we wanted to add, and we also wanted to be able to support the new 5735 thin client that replaced our model. I took the HP released Debian Etch flash build, and have been adding in our customizations. Both old and new thin clients are running the same build, and have identical features. I have Xinerama working for 2 or 3 monitors in both NX and XDMCP. 1 monitor continues to support Compiz/3D desktops.

A few ideas that I had for the new 5735 model will have to wait for the next upgrade. The ATI X1200 on board cards are not supported in Etch, and seems to need the radeon-hd package. The 5735s ship with both PCI and a PCIE/x1 expansion slots. So we bought an ATI 2250/x1 for testing. This card too requires radeon-hd. So some back burner ideas for the future. :)

The latest news is that HP released some thin client laptops and we got a loaner for testing. I had to install Debian Squeeze/SID in order to light up all of the hardware including an Intel/4500 video card. I then wrote all of the custom scripts and code to support this hardware as well. All 3 thin clients now work identically from a common code base and it's just awesome. We are kicking around the idea of allowing some employees to use the laptop footprint instead of the desk model, which would allow them to carry it around the building and use it anywhere. I also have experimentally installed local copies of OpenOffice and Acrobat Reader to allow a (very) minimal amount of off-network work. If someone needs to do a presentation off-network, it will work fine. The details are still being kicked around, but it's looking promising. The thin client specs. With a price tag just over $500, they are pretty attractive.

All 3 devices boot to the same system chooser menu (below). The laptop is especially fast and gets here in about 10 seconds after pressing the power button.



I have scripts to automatically figure out if you are on the hard wired network in which case XCMCP login is started. I haven't gotten 3D working on this Intel driver yet, but that's my goal. If they are not plugged into the city network, or on the wireless card it automatically starts NX and they can get the same desktop. I have been walking around the City Hall building and it's all working really well.

Largo Hack Fest Wiki Page

Stormy has been so kind as to drop me a few lines concerning having a Largo Hack Fest. There are issues related to enterprise and thin client use of GNOME and Linux that I think would be interesting to all of you developers. I'm envisioning that we set up a network, and let you connect your laptops and software and then test them over remote display to thin clients and see how they work. Lot's of cool things happening with gnome-shell, but unfortunately I haven't had a free second to try and build it and test it remotely. I was also thinking that we could converse with neighboring City and County governments and have them directly describe issues they are having in thinking about moving to open source software. Lastly, we have around 800 'regular users' that can be used for focus group testing and bouncing ideas. Things that make sense only to technical people assure us of a nice 1% desktop share. :) We have to take strides to go after regular users, and enterprise usage. That's where the numbers will increase.

I know the Boston Summit is coming up soon, so we have to think about a time to hold the event. I have created a very basic Wiki page with my current ideas. You are very welcome to begin adding your own notes and ideas and let's try and get some dates locked in. It would be my pleasure to meet more of you face to face!

Starting To Prep HP 5735 and 5725 Upgrade

Now that the Firefox upgrade process is nearly complete and going live tomorrow, I have begun working on another long term R&D project. A few years ago we ordered 600, 5725 thin clients from HP and they have worked very well for us. That model was discontinued in favor of the 5735. We need to be able to deploy 5735s into our mix as new workstations are added around the City. The 5725's still are barely into their 10 year duty cycle and will work fine for many years to come.

The 5735 has a few nice options not found on the 5725:

- Slightly faster CPU
- Analog and Digital on-board video ports vs Analog port
- 1Gb network interface vs 100Mb
- X1250 integrated video card vs SiS

As anyone knows that has supported an enterprise, uniqueness of client devices increases support levels. Thankfully HP has released a build of Debian Etch that runs on both devices! I will be able to make on upgrade and push it live on this mixture of hardware. Very very cool. I'm tuning Etch to run in the manner that we want, and being mindful of my goal of making it run on both devices. My current functionality wish list includes the following changes:

- QA all upgraded packages, including new X drivers and pulse audio
- Experimental cron job to init 0 devices not being used for a few hours. This goes beyond any normal power save modes and completely turns them off. We have users that leave them on for weeks and months at a time.
- Bug fixes for some issues discovered after the first deployment.
- Support for 2, 3 and 4 concurrent monitors in 2D, along with continued 3D support with one monitor.

In the case of multi-head monitors, X documentation is limited and you kind of have to figure out things on your own. At this point, I have gotten 3 monitors running from a single thin client. I should have the 4th working soon (picture below).

I'm also going to take a peek at the X1250 on-board video card and see how it works. On the older thin clients, we upgraded to 9250s because of their limited capabilities. It will be interesting to see how they work.

Mesa To The Rescue

My thanks to the people that clarified for me where Qt and GL (Mesa) are used in Google Earth. The earth canvas itself is coming from GL. The first thing I did was try and install an updated version of Mesa built on OS 11. That made things even worse. I then installed the version that shipped with OS 11.1 and it's rock solid. No blinking, no font problems, no areas of the screen turning white.

That Mesa update, along with using newer Qt libraries has produced a very nice looking UI with anti-aliased fonts. Now all I need to do is QA the ability to launch Firefox and Evolution from within the software, and it's ready for deployment.



Update: Interestingly, using the trick to soft link in the new Qt libraries for anti-aliased fonts breaks printing and the edit >> copy image feature. I suspect that users will kind of need that. :) I have reverted back to the shipped Qt libraries and things are running well. I have not been able to figure out exactly what GE does when you click on the email button. It's supposed to pass the mailto: tag to Firefox, which it seems not to do. $BROWSER pointing to Firefox doesn't seem to help either. The File >> View In Google Maps option works fine and calls Firefox. Time to strace and see what it's doing.

[ Shot of GE with stock Qt libraries, sorry to see anti-aliasing gone]

Anti-Aliased Images In OpenOffice 3.1 Over NX

We just got a support call from a user at a remote site running NX at 20Mb. He reported that ever since OpenOffice 3.1 was installed, images are moving very slowing in Draw. 3.1 introduced anti-aliased images, so that was my first thought. I had him disable this new setting, and he said it immediately got faster. I just tested at 100mb, and it's doing the same thing. Slow scrolling and movement.



Thanks for all of the comments and ideas on the Qt/Google Earth 5 font rendering problems. Over the next few days, I'll be tinkering with libraries to see if I can make it work better.

Calling QT Gurus

I'm prepping Google Earth 5 for deployment, and found an issue that appears to be a bug/peculiarity in the QT libraries. When you first launch it, everything is working and nicely anti-aliased. But as you start navigating through the maps the fonts all blink and then turn into white blocks (shot below). If you go into preferences and pick another font, then it seems to repaint and fix itself. I haven't tinkered with QT stuff since we moved from KDE to GNOME, so if anyone has some ideas...drop a comment!

Firefox 3.0 Stable, Moving Live

Normally when I am staging an upgrade, I will use the latest release -- even if it's only in beta. I spent time off and on testing Firefox 3.1/3.5 and was never able to get it stable enough to deploy. It was crashing way too much, and the core problem seemed to be the interaction with the Flash 10 plugin. I know, I know....3.1/3.5 is still a beta release. But one hates to move out a version into production that is nearly obsolete.

Anyway, I have installed 3.0.10 and all of the plugins including Flash are extremely stable and working well. I'm getting fairly decent frame rates to the thin clients, and the pulse audio streams are working well. I also had good luck with the FoxTab 3D plugin and that too will be pushed live as part of the upgrade too.

I also have been spending time on Evolution 2.24/SLED 11. We have about 30 fulltime users on it, and it's slowly inching to the finish line. We have one more major feature to include (resend of a meeting from the Sent folder). Then comes bug hunting, crash reductions and then some valgrind/leak hunting.

Evolution Picture Gallery Under Development

During the 2008 UI Summit, one of the things that I mentioned is that once people get a file manager in software, you lose them. They don't understand navigation and files and directories (right Sebastian Faubel?). The two most popular features in OpenOffice believe it or not are the email button which automatically attaches the current document to Evolution, and also the picture gallery. When you click on the picture gallery button, clipart and photos are displayed and they can drop and drag them right into OOo Writer.

Pepp and Mbarnes have hacked on a patch to integrate that feature into Evolution. The beauty of the patch is that it consumes no real estate for those that don't want to use it. We currently have an icon on the desktop that opens Nautilus with emoticons and city logos; once this patch is merged I'll be able to move all of those pictures directly into Evolution.

OpenOffice allows you to add your own Galleries, so in theory the two could be synced for consistency.

New Impress Layouts Landed In Milestone 49

Our bountied OpenOffice change to add two new layout types landed in milestone 49. My thanks for Sun for accepting the patch and getting it merged so quickly. This will be a big help for our users.



Our other bounty, adding sentence case, is currently going through review and will require an OASIS file specification change. But it's getting attention, and moving.

Thanks Pepp!

Tuning For Evolution 2.24

Now that SLED 11 has been released and our server has arrived, I am in the process of prepping Evolution 2.24 for citywide deployment. The server has been built and configured, and today for the first time I started load testing. I am tuning for 300 concurrent, which Evo 2.6 on SLED 10 easily handles. With a user load of 21 people, I was getting the results below. Not good. Notification-daemon + dbus were chewing up tons of CPU.



With some tips in #evolution and poking around I made the following changes to the Evolution launch script:

/usr/bin/gconftool-2 --type Boolean --set /apps/evolution/eplugin/mail-notification/dbus-enabled False
/usr/bin/gconftool-2 --type Boolean --set /apps/evolution/eplugin/mail-notification/status-blink-icon False
/usr/bin/gconftool-2 --type Boolean --set /apps/evolution/eplugin/mail-notification/status-enabled False
/usr/bin/gconftool-2 --type list --list-type string --set /apps/evolution/eplugin/disabled "[org.gnome.evolution.sa_junk_plugin,org.gnome.evolution.mail_notification]"

This turns off all things blinking, dbus and calling the notification-daemon and now it's running much better. We already have a mail-notification popup applet running, and the alarms daemon will provide notification of meetings.

I will continue to tune over the coming weeks.

This is making me ponder how well newer versions of the GNOME desktop are going to scale on multi-user servers. Just a friendly reminder to people working on this code; keep those of us running hundreds of people in mind! :)

Update: Here is the same user load after the changes.

Hero Of The Week: Chen

No offense to my other heros, but the award this week goes to Chen. :) He just merged into Evolution the first set of patches to allow users to retract/resend meetings with the Groupwise backend. This is a HUGE thing for us, and we are very thankful.

PDFShuffler Is Wonderful

My thanks for the comment that pointed me to PDF Shuffler. What an awesome piece of software, and a big help for us. When I first tested it, it only supported 24 bit color. I submitted a bug report and within a few days it was already fixed to work in 16 bit as well.

You drop and drag PDF pages right into the UI, and all of the pages load as thumbnails. You can then shuffle them into any order you want, and you can delete individual pages with a right mouse click.

I have already taken some initial steps to possibly bounty out a few more features:

- Drop and drag of photos which insert as a PDF page
- Drop and drag of OpenDocument files which convert to PDF pages.
- Running total at the bottom to show total number of pages, and total file size of the resulting PDF.

OpenOffice 3.1 Stable, Thank You

On Monday night I installed and configured OpenOffice 3.1 for the whole City. Once again it was painless and I just had to make the change once. We didn't get ONE call about the upgrade and no bugs have been reported. I know that means that un-named people are QAing this behind the scenes. So my thanks to all of you.




The server is running as expected and user load rarely is going over 10%. The shot below is how it looks with 125 concurrent OpenOffice sessions running. I guess having a 4 CPU, Quad Core kind of helps. :)

Experimental GUI For Users

As a very "back burner" project I have been experimenting with some ideas to assist our users. It has not been successful to explain file types (pdf, odt, jpg), or concepts like megapixels to most of them.

Here is what they struggle with from their perspective:

1) Sending pictures to other people via email and not getting it rejected because their email system won't accept the file size. 10 megapixel? What's that? What is 2MB?

2) Having dissimilar files and wanting to put them together into one PDF. OpenOffice reads odt's great; but PDFs only kind of load into Draw, how do I merge that together? Ok, now I inserted my 10 megapixel picture into Openoffice and used the corner handles and have made it 1 inch by 1 inch. That makes it smaller right? Why after a photo resize is the document still so big?

3) I want to use licensed software like Adobe Acrobat Pro because it lets me merge pieces together, I don't care that it's expensive and I only want to use 1% of the features.

4) Our vendor shipped us a PDF and now I need to merge it with my own odt and create a new PDF.

I have been tinkering with an idea to create a simple GUI to let them just drop and drag files (OpenDocument, Pictures and PDFs) into 12 positions and then it makes certain sizing decisions automatically. It then allows them to build a merged PDF with one click. No file managers would be found, everything is drop and drag. Maybe this is an early prototype for a module that should be in OpenOffice? A document sorter screen for putting pieces together?

I don't get to hack all that often anymore and don't know python and GTK in the manner I wish I did. I built the screen in Glade-2, and am stuck at how to activate the Buttons to accept a drop and drag from Nautilus. I know this is super simple for someone that does this all the time. If you want to throw me a bit of code in the comments, it's appreciated. I want to detect the release of the file into the button area, and then load the file path that it sends into a field that I can use for processing. I'm going to then have it build a thumbnail of the first page of that piece, and then update the Total Pages and Total Size fields so they can see how big it is as they go.

I'll blog about this GUI further when I get it actually working and have tried it with users to see how it works.

PS: No GUI Nazis please :)

Google Earth Look Bad?

While waiting for our Fedora ISO to download, I have been tinkering with some small-ish projects. There have been some calls about the font rendering in Google Earth 4. I brought down version 5 for testing and found the fonts were still not anti-aliased and looked terrible on our GNOME desktop. This post explains that Google seems to ship static QT libraries that do not render well. The post explains how to move the shipped libraries out of the way and then softlink into the QT libraries which exist in /usr/lib. Google Earth now looks wonderful. I'm going to come up with settings that work well with remote display and Compiz and push this all out to users in the coming weeks.

Prepping For OpenOffice 3.1

There are always multiple projects moving ahead concurrently. OpenOffice 3.1 is almost done, so this afternoon I have been prepping the dialog the users will see the first time they launch the newest release. Users won't navigate to web pages, won't read the release notes, and won't read a long message describing everything that has changed. This GUI just gives them a short overview, and hopefully a few people will scroll down and see the new features. It only runs once, and then puts a dot file in $HOME.

We also encourage users to run OpenOffice at home, so we give them a checkbox to request media for the latest version. We will then burn them a copy of the software for Windows and Mac and interoffice it to them. A shot of the dialog is below.



On another note, this blog concerning OpenDocument support in SP2 of Microsoft Office 2007 is a huge win for all of us. As organizations begin to upgrade, we will not longer have to train users to store in .odt here, and then always email as .doc to outside vendors and contacts. My thanks to everyone that worked hard to pressure Microsoft to add this feature. This issue seems to have been driven mostly outside the United States; so my thanks to everyone around the world.

Dear Adobe, Please Stabilize Flash

I continue to test Firefox 3.1 with Flash 10 and have found it to *extremely* unstable:

Program received signal SIGSEGV, Segmentation fault.
0xae6b610d in ?? () from /u/firefox_3100b3/firefox/plugins/libflashplayer.so
(gdb)

Frame rates and sound both seem better than 9, but it crashes constantly during testing. Right now there is no way I could deploy the most recent release.

Adobe: It's great that flash is so widely used, people must like using it for content development. But with this much content on the web, you always need to provide a quality player too. I understand that many companies have made cuts in staff, and I would be happy to help debug it for you. Please think about deploying bugzilla for Acrobat Reader and the Flash Player. Please, help us help you!

PS: Already tested swfdec, it's no where near ready for enterprise deployment.

Update: Having some better luck with FF 3.5B4, not crashing as much as B3; but still crashing more than it should.

Update2: Lots of ideas and things to try from the comments. I'm now testing older builds of Flash 10 to see if they provide better stability (current is r22, I have r15 here too). I'm also going to try using Firefox 3.0.X builds and see if that works better too. I had been using the 3.1/3.5 builds with the anticipation that it would be released the same time as this new server goes live. The video card we have is ATI 9250. I'll test over other video cards, and also test with a new version of pulseD.

Update3: OpenSuse 11.1 seems to have some GTK issues that are making the testing process difficult for me. Current plan is to blow the server and load with Fedora Core 10 and see if that works better. OpenSuse/Stable seems to still have some file manager bugs, with the GUI constantly bouncing around the screen and resizing.

Flash On Flash With Pulse

I think that I have gotten Flash working with pulse now to our thin clients. Here are my work notes in case this helps anyone.

From our old system, this was what the daemon reported when playing Youtube videos:

I: client.c: created 27 "Native client (TCP/IP client from 128.222.233.233:53650)"
I: client.c: client 27 changed name from "Native client (TCP/IP client from 128.222.233.233:53650)" to "flashplayer"
I: module-volume-restore.c: Restoring volume for
I: module-volume-restore.c: Restoring sink for
I: sink-input.c: created 20 "playback" on oss_output.dsp with sample spec s16le 2ch 44100Hz
D: memblock.c: Memory block too large for pool: 1048576 > 16376
D: memblockq.c: memblockq requested: maxlength=132300, tlength=88200, base=4, prebuf=87318, minreq=882
D: memblockq.c: memblockq sanitized: maxlength=132300, tlength=88200, base=4, prebuf=87320, minreq=880

Here is the same page on OpenSuse 11.1

I: client.c: created 31 "Native client (TCP/IP client from 128.222.233.236:38065)"
I: client.c: client 31 changed name from "Native client (TCP/IP client from 128.222.233.236:38065)" to "flashplayer"
I: module-volume-restore.c: Restoring volume for
I: module-volume-restore.c: Restoring sink for
I: sink-input.c: created 22 "playback" on oss_output.dsp with sample spec s16le 2ch 44100Hz
D: memblock.c: Memory block too large for pool: 1048576 > 16376
D: memblockq.c: memblockq requested: maxlength=4194304, tlength=44100, base=4, prebuf=44100, minreq=8820
D: memblockq.c: memblockq sanitized: maxlength=4194304, tlength=44100, base=4, prebuf=44100, minreq=1
D: memblock.c: Pool full
D: memblock.c: Pool full
D: memblock.c: Pool full
D: memblock.c: Pool full
D: memblock.c: Pool full

The sound would play for a short while, and then get choppy. The browser would start to gray out and eventually freeze.

I then started looking around and found that many people were mentioning that libflashplayer was no longer needed and to use the Alsa-Pulse "device" instead to increase browser stability. I straced aplay and could see that it was looking for /etc/asound.conf which it was not finding. I looked in the directory and noticed that asound-pulse.conf is what was loaded instead. I soft-linked it into asound.conf and aplay -Dpulse started to work as expected. I then removed libflashplayer.so and started up Firefox and it's working *much* better now. The pulse daemon is no longer getting hammered with spewage.

Things like this really make you want to put in a little classic Queen. :)

Staging Firefox 3.1 Again

The last few weeks have certainly kept me busy. I have been prepping and testing Evolution 2.24 and OpenOffice 3.1 for deployment here at the City.

My next major project is to get the new Firefox server finished and deployed for the whole City. I got some excellent feedback from people with some parameters to tune to lower the memory footprint of Firefox. I also am testing some new code that extracts out certain settings from the users prefs.js file and then merges those changes back into a global template. This will ensure that everyone is always given a working session each time they click on the icon and can't configure it in a manner which "breaks" it (i.e, disconnecting from the proxy server or whatever).

I also have globally installed FoxTab (shot below) which gives Firefox the features in Safari 4 for displaying thumbnails in 3D of opened pages. No flame wars please - I don't know who did it first. :) But regardless, it's cool and is working well even over remote display.

I also updated to the latest version of Flash, and am now starting to poke at the issue of Firefox crashing the PULSE daemon on the thin clients. It appears that there are some settings where you can tune the priority of applications pushing sound to the daemon, and I think possibly the stock OpenSuse 11.1 settings are anticipating PULSE to be running locally on the same computer; and not being pushed over the network. I'm certain I'll find it.

Anyway, this seems like it's going to be a good upgrade and that people will like it.

Pierre-Eric Rocks 2: Evolution PDF Drop And Drag

Being a Governmental agency, everything that we do is about record retention and long term storage. Citizens can request email messages at any time and we have to provide them in a format that can opened without cost based software. In other words, if someone wants email from 10 years ago we can't just simply give them a binary backup of the Groupwise 4.1 post office and expect them to get it loaded on SCO Unix and working. For ease of retention, we keep email messages in PDF format in project folders. More so in Evolution 2.6, but even in Evolution 2.24 there are a few steps and a requirement that users know how to navigate to the right folders in order to write out PDFs.

In order to make this easier, I starting poking at some ideas. I noted that currently Evolution drops and drags messages in mbox format only. I had some IRC conversations on #evolution with mbarnes, mcrha, chen and xkahn and created a scope of work and placed it on BGO here. Pierre-Eric (pepp_) was interested in the project and we agreed on the scope of work and we allocated some bounty money. An initial patch has landed. Default behavior of course will be to continue to write out mbox files, but with a gconf key change Evolution will now write out PDFs into Nautilus from a drop and drag.

I was only able to fund the areas required for our needs. I'm hopeful that in the future this functionality is expanded and moved into the UI of the Preferences screen.

This one small change will save our users countless hours of time (and frustration) in the future.

Obligatory screenshot

Having Problems With Acrobat Reader 9.1?

If you have started using Acrobat Reader 9.1 on Linux, there is a nasty little bug that took me a while to locate. It's leaking files into /dev/shm that don't delete, and cause problems on the next startup. We were getting calls from people that when they clicked on Reader, their whole Xserver would freeze until the stuck process was removed. After running strace, I was able to see that it was attempting to use /dev/shm right before the lockup; and found the leaking files.

I placed these lines into the script that calls Reader and things are working as expected:

rm /dev/shm/sem*$USER 2> /dev/null

Linux Motif Development Toolkit Update

Some of my coworkers have been busy testing new features and updating artwork for our development tool for custom applications. To this day, nothing else allows you to rapidly build business applications and everyone is pretty happy with it. Motif added anti-aliased fonts recently, and with some Tango artwork the programs really look much nicer. GNOME themes automatically set old X resources, so the applications are now coming up in the same colors that users have selected. Not a lot of time invested, and huge gains already in appearance.

The toolkit we use is called Panther, and is from a company called Prolifics.