Dave Richards - City of Largo Work Blog

Losing Planet GNOME Feed

2012-01-27T09:07:00.002-05:00

Looks like I'll be losing my Planet GNOME feed in the next few weeks based on the new policy changes. I'll still be blogging at the same frequency, but you'll have to pick up the feed from Blogspot. I've greatly enjoyed the interaction with all of you and hope that it continues.

Full Steam Ahead

2012-01-26T13:25:00.005-05:00

Now that the tuning issues have been resolved, we have started to move more and more users over to the GNOME server. Just now we hit a new high of 108 concurrent users logged into one server.

And as promised let's see how the server is doing with the 108 users. (shots below). The answer is: excellent. Time to ramp up workstation upgrades and move more people over. There have been a few bumps in completing this project but it's been worth the effort. All City employees will be able to log in and get core software packages (Browser, authentication, word processing, image editing and more) without any license costs.

DBus Fixed + Support Portal Changes

2012-01-24T13:42:00.004-05:00

After a lengthy process, finally there was a solution in the issue of dbus not allowing more than about 95 users into our new GNOME server. Dbus needed to be patched (thanks Vincent!), and then we continued to hit the limit of 1024 files being opened by the user 'messagebus'. /etc/sysconfig/ulimit has an entry that is supposed to configure the number of files that a user can open at once. It turns out that PAM is performing the same function and was doing the blocking. So I made the change as seen below to /etc/security/limits.conf and it worked. I don't see any other roadblocks and we'll start moving people over to the new server in greater numbers starting next week. Good news!

And here is our support portal showing the 100 concurrent load, woohoo.

We met yesterday and finally made some decisions on the way that NX will be implemented for outside connections. The world has changed greatly in 5 years since the last install, and we are trying to find the right balance between security and functionality. I added a new tab to the portal to show "Logs" and we now can see people logging into the GNOME server is real time. The servers are full of information to alert us that people are having problems or technique issues; so I started to gather all of that information and get it into a new section called "Alerts". We'll be able to see people who tried to launch software that they do not have a license to run, crashes, password problems and problems with settings. This should allow us to be a bit more proactive and do more with less (tm).

Poor Mans GoogleEarth

2012-01-19T12:33:00.004-05:00

In my last blog, I mentioned that I had gone in and made changes to our thin client build to accommodate running NX sessions along with local RDP. We had another thin client project scheduled for 30-45 days in the future and because I was already in the code it was the best choice to just finish it and roll out all features at the same time.

Our county is doing a cool project to gather GPS information on the squad cars from all of the agencies and build a KML file for Google Earth. This will allow the various departments to see if fellow police officers are nearby. My coworkers did all of the front work and got this connectivity working back to the county. Our police department wanted to display this information on a TV in the 911 center. Instead of allocating a full PC with operating system to this function (a waste), we decided to just build a new kiosk mode on our thin clients. So I made the changes to the build. A chooser comes up, you click on a button and GoogleEarth runs on one of our big servers and remote displays back to the thin client. Easy and works great and saves time and money vs installing a thick client for this function.

The next logical step of their project is trying to get this information into the squad cars. There are many scaling and bandwidth problems with trying this. The hardware in rugged PCs is sub-optimal for this function. These devices are run 5 years, so many of them are very old and would come to a crawl with GoogleEarth installed. There also is the issue of sharing bandwidth of this function with the EVDO cards. The networking already is busy doing mission critical functions and cannot be bogged down with another major feature. Another idea would be to run the Google Earth software on a server with their already existing NX infrastructure, but the prospect of 25+ of these running concurrently was not pleasant.

So I had an idea that we are testing now, which was to create a poor mans Google Earth. This was so simple that it took me literally less than 1 hour to write, test and put on the GNOME menu. A cron job runs every five minutes and does a screen grab of the Google Earth session running in the 911 center and dumps it to an image file. I then wrote a quick Glade/Python UI that simply displays this picture and is on a 2 minute timer and refreshes the image. The memory footprint is very light, and everyone is sharing the same image file. So this should scale wonderfully and will barely produce any server load.

Here is the UI. Buttons are nice and big to accommodate their touch screens and allow them to touch with their fingers in a moving car. The image is in a scroll region to ensure it works correctly with their 800x600 and 1024x768 resolutions.

In the shot below, I have marked the memory footprint of Google Earth on the server. 25+ of these would be quite a load in memory and CPU. That's 25+ users all performing the same steps and retrieving the same information independently...not a good design. The other window shows the very simple code used to grab the screenshot from the 911 center. The script pings the device, if it's alive it grabs a shot, creates multiple scales and then moves the finished product into place. The Glade/UI then loads the image into the canvas.

Local RDP, NX 3.5 Installed

2012-01-17T07:45:00.003-05:00

After waiting many months for the release of NX 4 we reached a point where we could not wait any longer to use compression technology with the new GNOME desktop servers. We were really hoping that NX 4 would fall into place so that we could deploy iPads running host based software. So last week I removed the NX 4 beta and installed the released/stable NX 3.5 After remembering a few tweaks we did a number of years ago, it's all up and running. I removed the NX 4 client from our thin clients and reinstalled the older version as well. A few script changes and improvements and things were working. I should be able to get a test build to remote site users this week.

Our division got word very late in the implementation process of the new MS Windows based Recreation software that thin client workstation IDs were needed for the software to properly know which cash drawer to use. For Recreation sites running XDMCP, this was no problem and already implemented. I had offloaded RDP/Rdesktop sessions many thin client releases ago and they were already running on the local device. But we have one site that was using NX in full screen mode with no window manager. So when a local RDP session was started, there was no window manager to grab it and allow the user to move the two windows around. RDP would just sit over NX and obscure the view. It was clear that a new type of thin client build was required, one that connected to our servers with NX and that was running a local window manager. After a few white board drawings, I picked the path that seemed the best solution and made the necessary changes. As is seen in the shot below, when the user picks this option it automatically detects the size of the screen and NX is started consuming about 90% of the screen. The window manager then wraps this window and they are able to move it around nicely. When they click on this MS Windows app, the server sends a signal to the thin client who then launches Rdeskop locally and the window manager grabs this window as well. I put a small modal applet in the upper left hand corner to allow the user to disconnect this session and shut down the window manager. A bit more testing and this should be ready for beta testing; hopefully later this week.

DBus Root Cause, iPad Testing

2012-01-06T08:25:00.009-05:00

In my last blog I mentioned we were having dbus problems. Right around 96-98 users it would stop allowing additional people into the server and dbus would begin chewing lots of CPU. At first I thought it was hitting a limit and denying additional requests. It turns out that it was hitting a 1024 open file limit, which is clearly seen by going into /proc/process_number/fd and watching the number grow. When it hits 1024 the server goes bad. The ever wonderful Vincent Untz found the git entry in dbus that fixed this issue. These patches had not been merged into OpenSuse 11.4. He is going to generate a new RPM for us. I'll test and we'll try and get it into the update channels. The OpenSuse bug is here.

It's been a disappointment that NX 4 hasn't been far enough along to deploy iPads using NX technology. We have been testing other ideas and only found them to be good enough for IT staff, and they are not ready for consumption by regular users. The various techniques under testing:

VNC: Fast over Wifi; doesn't work over EVDO from iPad; resume not working well; no rotary of ports
RDP: We tested RDP/iPad -- Windows/NX Client -- GNOME/Linux; the hop in the middle produces lag, applications like Xournal are too slow.
Citrix: We are going to test Citrix/iPad - -Windows/NX Client -- GNOME/Linux to see how it works. This is not yet working.
NX4/Safari: The alpha release allows you to connect to GNOME with the Safari browser. It's way too slow on a tablet for regular use. We're hopeful a native client will fix this.

Because of increasing requests for this technology to move in beta testing, we are going to move ahead with an approach that is more "client/server". We will continue working on host based solutions as we move into the new year. Document records retention is paramount in Government, we need solutions that will comply with the law.

The tablet solution as it's shaping up now:

For email we have been looking over Groupwise 2012 and it's wonderful on the tablets. Using just Safari, one gets the UI below. Unlike the older Groupwise version, a single touch of the message list instantly opens the message in a preview pane.

The calendar also is designed well for the real estate of at tablet

Dragon Naturally Speaking works well, and allows you to easily email the text generated by your spoken word:

So being that the NX upgrades are not yet ready, how do we allow people to transfer and read their documents on tablets? There are many middle-ware type applications for this function. But we always have to consider cost and more importantly staff size. At a certain point your infrastructure is so complicated that you almost get to the point that no single can take a day of vacation. We found an application for the iPad that seems elegant in simplicity and are going to begin testing. It's called FileApp. This software is very suited for integration into a GNOME (and even mentions it!) desktop, and supports OpenDocument (OpenOffice/LibreOffice) files. It gives you a very simple interface that shows you all documents you have downloaded from your GNOME desktop, and a single tap of the UI starts an FTP server on the tablet. You are then able to gain access to it the desktop and transfer files. Once the connection is closed, your document list refreshes and you can view them "offline".

From FileApp tap the WiFi symbol and the drop down appears. When this is displaying, FTP is running on the tablet. The UI gives the end users their FTP address which is based on their IP address at the time.

From the GNOME desktop, simply type in the FTP address into Nautilus and it displays all of the documents stored on the tablet:

Once the drop window is removed,FTP closes and your document list refreshes and all available documents appear. You can view them by date or file type.

And further firming up the fact that OpenDocument files are not second class citizens, they open directly on the tablet and won't have to be converted to PDF before their viewing.

This software is moving through our Infrastructure group for testing and then will move to the IT Director. Once that's done, we'll move this to a limit number of testers.

Calling DBUS Experts....

2012-01-04T14:35:00.000-05:00

We're expanding our GNOME desktop project and dbus on OpenSuse 11.4 is kicking our butts. If anyone with knowledge of this code has any tips, they are gladly accepted as a comment.

What's happening is that right around 96 to 99 users the system dbus seems to no longer accept GDM connections and more users cannot log in. Very often during this threshold we also see dbus chewing lots of CPU as it seems to be receiving retries over and over again which all fail. /var/log/messages show gdm crashing and to my eyes it's happening when it's trying to talk to dbus. I have installed the debug symbols and should get better backtraces starting tomorrow. The shot below shows the crash

The documentation is kind of lacking in regards to the tunable parameters. One doesn't always know the default settings so as to know what should be increased. It's also not very clear which resource is failing. I set the following resources this morning and the issue happened again:

The technique that I used was to look at the source code and find where the defaults are set and then try and double them. There are more parameters, any ideas which ones might help? It would be wonderful if dbus-monitor showed you these types of failures, but it seems to only show you bus activity which really doesn't help. As soon as we drop below the 96 users, everything works correctly and users can log on and off with no problems. This leads me to strongly believe this is a parameter that is being reached.

Any tips? Drop a comment. Thanks!

Thank You For All You Do!

2011-12-16T09:30:00.000-05:00

I only have two work days until an extended Christmas vacation and when I return it will nearly be the New Year. I have received many very kind comments and email messages this last year, but to me the true heroes are those writing open source software. When we speak to other Government agencies we are always proud of the fact that authentication, GNOME desktop, email client, internet browsing, photo editing and document construction are all done with NO LICENSES OR COST. (( hardware cost aside). You guys all rock.

As is tradition, I enabled good ol' Xsnow for users to activate on their desktops for these last few weeks of December; which is especially fitting here in Florida.

Thanks again to all of your hard word, we have been able to deploy a desktop that looks like this:

GNOME Server Load, Project Updates

2011-12-14T14:42:00.003-05:00

The new GNOME server project has continued since the last blog. We had a slight setback in that the server was reporting ECC errors, which indicates failure of memory, but possibly caused by memory, backplane, motherboard or CPU. The server never failed or halted, but we held off adding more users until it can be resolved. We have been replacing pieces. Motherboard was swapped out yesterday and so far the error has not returned. When we get past a burn in period with no problems,we'll start adding more users.

So the always interesting issue of how well GNOME scales. One is always concerned that a certain feature or function of the desktop will chew up a lot of CPU and slow the performance of the other users. Many software applications are designed with the perception that they will be run from a stand alone computer with little regard for chewing CPU, disk or leaking memory. The shot below is 'top' running for 90 concurrent users. The server runs about .5% to 4% busy, and normally sits around 1-2%. This is excellent and should scale nicely. I would think we could get 300 users on the server with no problems.

I have continued testing Groupwise 2012 and its web interface in Firefox and then on iPads and the result so far have been quite good. I'll post a more thorough blog concerning the testing and results. I also have been teaching myself how to use the Groupwise server as an LDAP server to allow the iPad tablets to get complete address books of all City employees. It's an area that I have never done before, so I have been reading lots of examples and documentation. I'll figure it out. Groupwise 2012 also has a feature that allows you to publish your calendars to the Internet as ics/iCal files and I've been working on getting that working and tested. Cool stuff.

The City is buying new recreation software that runs a front end from Microsoft Windows. Using the local RDP/Rdesktop client from the thin clients it's working well. We had an unforeseen issue where the software requires the client (dhcp) names to be used as part of the cash register groups. In the past, all thin clients in the City were generically named and addressed because it never mattered. This meant we had to do a complete reorganize of IPs into logical groups. This change was implemented Tuesday morning and after a few hiccups is now complete.

I have been pondering again the prospects of a Largo Hackfest for 2012. It sure would be nice to meet all of you and show everyone how Linux can and is being used in the enterprise. Maybe we can make that work.

86 Users With Difficulties

2011-12-06T09:13:00.003-05:00

In the last few weeks we have been adding more and more users to the new GNOME desktop server and hit a tuning issue where it would stop allowing people right at 80 concurrent. My first inclination was that it was a tuning issue with GDM and I had a few IRC conversations with the ever friendly Halfline. I found some error messages in /var/log/messages and after Googling was able to find the cause: Dbus. There appears to be a bug in the code that counts the number of concurrent processes from the same user account. My reading indicates it's supposed to be 256, but others have reported it stopping far short.

So someone working on Hardy found a fix, which I copied and it works on OpenSuse 11.4 as well. You add a system-local.conf file to /etc/dbus-1 with the following XML parameter. After this change was made, we now have hit 86 users with excellent results. I'll blog in the coming days about the user loads, so far we're very pleased.

Easier Photo Management

2011-12-01T12:50:00.004-05:00

I have previously blogged about the fact that we implemented a simple photo management tool on our thin clients so that users could easily get pictures into software without having to understand file management. This also eliminates the need to give users full access to USB sticks when all they want to do is take a few photos and insert them into OpenOffice Draw for a flyer. It's been working well.

A conversation with an end user has produced a new feature. I added the ability to select the size of the image that is placed into the clipboard. Their options are 320x240, 640x480, 800x600 and 1024x768. All they have to do is click on the thumbnail and it's immediately in the clipboard, and then paste into the various products and all the work is done. The issues of file size and megapixels is not easily explained to end users and this keeps it very simple.

In the shot below, pictures were taken at 5MP and then I selected 320x240 and then pasted it into Evolution and for an email message, it's the perfect size. This keeps the original high quality off the server when only this smaller photo is required.

(( This feature and function is unrelated to those users that have full access to USB sticks, this is just the default UI that all City employees have available. ))

One interesting issue is the Glade screen opening in a mismatched theme. This is because the software is running physically on the thin client and the theme is not loaded on the local flash drive. A project for another day. ;)

Support Portal Updates, Groupwise 2012

2011-11-29T12:36:00.006-05:00

I took vacation days around Thanksgiving and am back in the office and once again working on projects. Over the last few weeks I have been poking at our support portal software end-of-day and adding a bit of code here and there. Yup, the theme is probably ugly ;) , but I'm trying to test various settings to see if any of them affect speed. In the past certain themes caused OpenOffice to perform more slowly, and I'm testing this concept against LibreOffice. Always nice for me though to see the old school mwm buttons.

In the portal, the user detail screen now allows you to set the record as a bookmark and also the UI has been cleaned up for setting a watchdog to alert you when the user logs either on or off the network. Very often someone in IT will have an action item to fix something for the end user that requires them to be logged out. Now we don't have to watch them manually.

The thin client detail screen now has the ability to record a movie (cyan below) of the user session using the wonderful vnc2swf utility. We can see their session, it records a swf file at the same time. These files are easily opened in browsers and can be sent to vendors to show them multiple steps. Similar to the user detail screen, you can configure a watchdog item for when a thin client is powered on. You can also set a thin client as a bookmark. These simple little things save lots of times and lots of scribbled pieces of paper.

I'm revisiting the clunky UI of the charts generated on the server summary screen. Right now it's just creating an image with perl. Simple, but not elegant. Sure would be nice if we had some charts in GTK that had hover tooltips. Maybe some Googling will find something that works better. You can also see that our two GNOME servers are now running 80+ concurrent users (circled in green). Loads are great, and capacities will be increased as we go through the month of December. The concurrent loads of the various software packages can be seen below as well. CPU loads are excellent.

Groupwise 2012 was released for beta testing as part of a technology preview. I built a new SLES 11 server on some old physical hardware and got the pieces working. Probably all of the design considerations being discussed warrant an entire blog update; but we are reviewing the concept of just using a web interface instead of Evolution to gain access to Groupwise. In the past this was not an option because the UI was a bit clunky and major features (such as free/busy) were missing. GW 2012 has moved in the direction of having the web interface be your primary login. Seems like maybe the days of having a client piece for email might be nearing an end. The shot below shows 2012 running. GW detects an iPad/tablet login and presents and interface designed for that footprint. I should be able to begin testing that aspect in the coming days. So far it's much nicer than the older interface.

On deck for me: continued testing of GW 2012, more testing of Evolution patches we just received, increasing the number of users logged into the new GNOME servers, implementing some user suggestions to make the thin clients interact better with digital photos, iPad testing, MoinMoin 2.0 beta testing.

New Desktop Progress Continues

2011-11-17T08:50:00.006-05:00

We have been adding more groups of users to the new GNOME desktop and have gotten loads around 80 concurrent and besides a few bumps things have been going well. We are going to run in the 80-90 concurrent range for another few weeks and then add more people after Thanksgiving.

I'm always fascinated with the support calls and UI questions that arise. Usually prior to deployment I get a feeling for places that are going to cause us problems, but sometimes I'm completely surprised by how others perceive software.

The one area that I knew would be troublesome is the GNOME keyring. That feeling proved itself true. We're using "mail-notification" which connects to Groupwise and sits in the notification area and alerts you of mail that has arrived. Works great and people love it. The launch scripts are fully ware of Groupwise and the IP and their user names and pre-fills it in for them. So all they need to do is enter their email passwords. The key ring opens, asks for the email password and then continues to the screen where it's requesting the keyring password. Users don't understand why it's asking for another password and then are annoyed by the keyring popping open each day. One can store the passwords in the default keyring, which is great...but the UI doesn't really tell you that fact. You have to NOT enter passwords; even though it's asking for passwords. You then have to accept a dialog indicating this storage to be unsafe. I fully understand what it's designed for, but we have found this whole process to be support intensive. I filed a bug report, and included a small mockup of an idea of how to make it easier in my view:

Some users are calling because they don't know how to lock the screen. The Quit docklet in Avant has this feature, but it's displayed in a secondary menu. This applet has been helpful to us because we have lots of users in 1024x768 so real estate is at a premium for them. A hover tooltip might help a bit, but unfortunately most users don't check nor use tooltips. I might just make a regular desktop icon and for those people with the space, they can pull it to their panel as a shortcut. This issue hasn't been too bad, only a few people called.

One of the more interesting issues is concerning calls we were getting about "Microsoft documents being empty and blank when opened". So we got the documents and they appeared to work fine with LibreOffice and OpenOffice. Further questioning found the issue: The gsf-thumbnailer is not able to create a thumbnail for Office documents as it can for OpenOffice files. So the users were double-clicking on them and seeing the preview window (below) and then seeing an empty page and never physically opening it in OpenOffice; because in their mind the document was empty. So what I'm going to do is change the code slightly so the default/blank panel will say something like, "No preview for this document, open to view contents".

It's great to continue advancing this project and be able to provide newer technology for our end users.

Excellent Scaling Numbers

2011-11-10T13:21:00.002-05:00

We have been slowly moving users to the new GNOME desktop and the results have been excellent. The server is a 4 processor, quad core HP and today we hit 75 concurrent users. CPU usage is under 1% and memory consumption is not bad at all. Unless we hit some as of yet unknown limitation, it looks like we could get a full load of 300 concurrent users pretty easily.

It's interesting that the highest amount of CPU is being chewed up the Citrix client (wfica) which seems to be very "talky". We are in the process of moving to 100% RDP as our connection technique to Windows, so in the coming months those will all go away.

Very promising start to the migration process.

Get It While It's Hot

2011-11-07T08:37:00.002-05:00

Seemingly unannounced a new version of Adobe Reader was released in the last few days and it's here.
We have been very disappointed that that the last release was in February. I've seen countless exploit bulletins about this software in the last 9 months, but we'll take it.

The shot below shows Adobe 9.4.6 along with a friendly reminder popup that users see alerting them that other software exists for reading PDFs. :)

Bi-annual disclaimer: Yup, I know that Evince does a great job with PDFs. However there are certain types of 3D content that can be inserted into a PDF that will not display in Evince; we therefore have to offer both software packages.

Desktop, Portal & NX 4

2011-11-04T13:18:00.005-04:00

I haven't blogged in a while, but things here have been busy. The new GNOME desktop has passed all tests and with the 3.0 kernel on has been performing like a champ. We had a meeting and decided to begin deployment and will be going live starting next week. I modified our thin client builds to no longer allow users to connect to the older GNOME server and that will be the official cutoff once it's pushed with FOG. A few of our departments are already basically live on the new desktop, and next week we'll make it official. Over the next few months, we'll push over more and more people until the old server is no longer used. It's nice to see this project moving to completion. When we get heavier user loads running, I'll post information about performance.

The back burner project of the support portal software continues. Now that we have our heads around the functionality we desire, I have been cleaning up the UI and making it more consistent with other software. The [Reports] tab (shot below) has been developed and is already saving us time. I have been trying to create reports that would take a LONG time to do manually. For instance, in the shot below the portal compares devices in the FOG server vs those devices configured to boot via DHCP. Through the years there have been device failures and sometimes the old entry is not removed. To manually check this on 500+ devices would take hours. It now happens in about a second. I have other reports in mind, and will continue advancing them a bit each week. UI needs work, but data is accurate.

I started revamping the UI and the results have been pleasing. The beauty of Glade is that these changes are cosmetic and very few coding changes were needed. Move widgets around, give them the same name and they just start working. In the [Users] tab (below), I moved all of the filters to the right side and cleaned up the results area. I started writing a new feature to alert us when users login or logoff the network. I also added the ability to see the last 3 people that you viewed in detail. You can now save users as a bookmark for later use. When going in and out of 800+ users, these features are wonderful. Lots of room for improvements, but making progress.

We received a new alpha build of NX 4 and I spent a few afternoons testing it fully and considering ways to get it deployed. The thin clients are all running the same universal build, so settings need to be configured to work in all types of logins and with rotating users and differing monitor resolutions. Making progress slowly and submitted to them a list of all issues that hinder deployment.

NX 4 still does not have a technique to deploy on iPads in the current build. It's coming, but not yet available. So we had an idea to get them up and running until we can deploy 100% NX. My coworker Brian set up a virtual Windows session to allow RDP connections from our tablets. The user profiles are then configured to automatically start NX and then connect to the GNOME server. This simulates how a native NX client would work, and at least will allow us to get some beta testers out there. In the shot below it's Ipad-->RDP-->Windows-->NX-->GNOME Desktop. Performance is good, and should get even better when the middle hop is removed.

Good days ahead for us as we move in lots of new technology, all of it as cost efficient and stable as possible. Happy Friday.

IE Deployed For Testing

2011-10-14T13:50:00.004-04:00

As I mentioned yesterday we are testing an idea to make it easier for those users that need IE to just click on links in Evolution and have it work. Initial code has been pushed to our Beta GNOME Desktop and is being reviewed. You have 10 seconds to request IE after clicking on a link via a small dialog in the lower right hand corner. In the shot below, I requested IE and it used the email link. FF always launches and is sitting behind.

So how best to get the URL to Windows from Linux? Here is the technique we used, low tech and simple...which means it will work. :) This snippet of code is from the python/glade UI. It takes the URL and writes it to a file in a temporary folder. Multi-user systems need file names that will never clobber one another, so we use $USER. We then simulate them clicking on IE from GNOME via the command line. This ensures that all licenses are checked and also that we log the launch of the software.

On the Windows side of things, the .bat file looks for the file written out from python. If it's there it uses it, otherwise it just opens IE on the users configured home page.

Now to send out email to the user and alert them of the change and await feedback.

Integrating IE Into GNOME & Firefox

2011-10-13T16:20:00.003-04:00

One of the things that has improved greatly over the last 5 years is the number of web sites that are now more compliant to standards. We used Netscape and Firefox on SCO Unix and then Linux through the years and there were always sites that just failed to work. In those cases we had to give people licenses for Internet Explorer on Windows. In this last upgrade to IE 8, we are down to 12 people out of 800 that need IE. WooHoo.

One of the sites that won't work is when our users have to do Webinars/Webex. The bad part has always been that the links come in email and are clicked. This of course opens Firefox on Linux. The users then had to copy and paste the link over to IE and away they would go. Sub-optimal obviously. So how best to handle this for those 12 users? Having a dialog appear to ask which browser you want to use is clunky, and very few sites really need IE anymore. So I'm developing a solution that promotes users using Firefox first and then dropping back to IE as a fallback. When they click on a link in email, it launches Firefox as always. Then based on $USER it knows if you have an IE license and gives you a secondary popup windows (circled in blue) for 10 seconds. If you do nothing, it goes away. From this dialog, you are able to select IE if needed and the URL is then going to be passed to Windows; and RDP will delivery the browser. This only happens if you click on a link in email, not when you just launch the Firefox icon.

I installed this rough code to get an idea of how the users will like the functionality. I'm pleased with the results and it seems like it will work well.

Software Usage Tracking

2011-09-30T14:37:00.003-04:00

Time has allowed me to put in some code to the new support portal for tracking software usage. We'll easily recoup this time in three areas. 1) the amount of time support spends looking for this information in the various license files 2) support is now one click away from being able to log into the server running the software without having to look through multiple sheets of information 3) we will be able to now begin the process of reducing software packages that are no longer being used.

In the shot below, the software packages are searched from /usr/share/applications and matches appear on the primary UI. When clicked the detail screen opens and displays a bar chart of the current month which indicates the total number of times users clicked on the icon. Below that it displays all of the users that are licensed to use this software. You can click on their name to pull up the user detail screen. At the bottom we have technical information concerning the location and name of the .desktop, the launch script and the remote server which runs this particular package. We also have stats on the total number of clicks for the day, month and year.

When you click on the Server IP button, it detects if the remote server is Windows or Linux. If it's Windows it opens a Rdesktop connection. If Linux, it opens a command line window.

If the system admins click on the launch script or .desktop button, it opens that file in gedit for review.

There are just a few more areas in my head to write, and then I'll spend time cleaning up the code and making the screens look nicer. The feedback from my coworkers has been positive, and it's great to hear that these few hours are freeing up their time by making things easier.

Happy weekend.

The iPad Villagers At The Gate

2011-09-28T13:59:00.004-04:00

The picture above is how I feel lately concerning the user community and using iPads. If you support end users, I'm sure you know what I mean. :) It should have been fairly painless to implement as a thin client, but unfortunately the NX 4 release is taking far longer than anticipated. We have not yet seen anything ready for deployment on tablets and because of the torches we are going to begin testing under another design. This should give the users the same presentation but it's a bit clunky, we are hopeful this a short term (very!) design.

Instead of the tablets connecting to the GNOME/Linux server directly with NX 4, we are going to add a VM instance of Microsoft Windows and use the RDP protocol. Users will use a RDP client, connect to Windows and once authenticated will immediately start the NX client on Windows and connect to GNOME and give them the new desktop. The diagram below shows how it will work. This is -very- nasty, but unfortunately we have to move *something* into general testing.

Another quick project came to me, the desire to make Beagle scan small subsets of our data and make it easier for end users to find just documents under a certain category (folder). So I hacked our Beagle front-end and added "Spotlight Searches" which simply reduce the number of documents returned in the query. Users cannot build complex queries on the command line, things need to be very simple and this meets that goal. If they want to scan just our City Policies for certain key words, it's just a few clicks away.

Desktop Home Stretch

2011-09-22T15:54:00.002-04:00

After the technical issues of the last few months, the GNOME desktop upgrade is finally on the home stretch. We're very excited about this deployment and the users seem happy with the results. Now that the kernel issues are resolved, the server is VERY fast. Even with 50 users, avant-window-navigator is fully rendered in about 3-4 seconds and ready for launching software. Several months ago I spent time firing all of the processes in certain orders and multi-tasking them to get the fastest possible response. I'm pleased with the results.

We're still waiting on the next NX release so that I can test this over lower bandwidth devices. Sadly that is taking longer than expected, but one can always be hopeful of a release date soon. I'm not expecting any problems with thin clients, the area that will need testing is tablets.

One of the biggest flaws that we have on the older GNOME desktop is clear and provable records of who is using what software. We have so many little MS Windows applications and it's difficult to even know how often they are *really* used. Our Windows apps and Linux apps each go through a common set of launch scripts so it was very simple for me to dump each desktop click to a CSV file (below). We now have an instant audit of all activities. This data can be reviewed in a spreadsheet, and also will very soon be available from our support portal UI.

[ Very simple data log of each users clicking activity]

As time allowed, I have started to hook up all of our flat files to the support portal and move the [ Software ] tab past being vaporware. When the portal is started, it reads all of the files in /usr/share/applications and finds all of those active on the desktop. It then reads in our license files and builds an array with this data. Now our support staff can search for programs, hover their mouse over the icon and see exactly what operating system and IP is being used. It also indicates the users that have licenses to run the various packages. (as seen below). This is a huge time saver for them. Everything is dynamic, and eliminates having to use another tracking software package for this purpose. Start up the portal, and it's reading live data immediately.

I built a quick UI to get my head around the information that I want to display when you click on the software detail button. Please, please...no UI critique. :) The goal in this first pass is to just work out the functionality and then I'll do cleanups later. Using the flat files described above, the UI will render a chart of the number of clicks of the software package in the current month (purple). You'll then be able to use the arrow buttons to navigate through the months and look for usage. It will display buttons for all users that have access in the middle section. The lower area will give more technical information concerning the app, including the .desktop and launch scripts used. It will allow you to click on buttons and edit those files with gedit. When you click on the IP address of the server which is used for the various packages, it will either telnet or rdesktop to that server for review and to make it easier to kill stuck processes. Right now, support has many sheets of paper with this information for 250 software applications running on around 25 servers. This will be a huge time saver. The lower section also will display additional usage information to assist in determining how heavily software is being used.

The last few days have been spent going through all of the icons on the production desktop, figuring out exactly what task they perform and then contacting users to see if they still use the software. I'm then upgrading the icons, testing them with Seamless RDP and updating the artwork. I hope to be finished with this process by Monday.

Everyone is sensing this project is nearing completion, and I think we'll have a deployment to be proud of.

And There Is Java Sound

2011-09-12T15:00:00.004-04:00

Sometimes one creates a post just as a way to document something that they do not do that often. In a few months when I have to do this again, it'll be here. That is the case today. :)

We are getting our first application that uses Java to deliver sound, and of course it has no way by default of delivering to the thin clients. After some Googling and poking around, the process is not terrible. I installed Java 1.6.27 into /usr/java and then got the open source version of Java in RPM format and extracted the two PULSEAUDIO files:

rpm2cpio java-1_6_0-openjdk-1.6.0.0_b17-7.3.x86_64.rpm | cpio -ivd ./usr/lib64/jvm/java-1.6.0-openjdk-1.6.0/jre/lib/amd64/libpulse-java.so

rpm2cpio java-1_6_0-openjdk-1.6.0.0_b17-7.3.x86_64.rpm | cpio -ivd ./usr/lib64/jvm/java-1.6.0-openjdk-1.6.0/jre/lib/ext/pulse-java.jar

I then extracted the sound.properties file as an example of how to add the lines for PULSE.

rpm2cpio java-1_6_0-openjdk-1.6.0.0_b17-7.3.x86_64.rpm | cpio -ivd ./usr/lib64/jvm/java-1.6.0-openjdk-1.6.0/jre/lib/sound.properties

You then hand move these files into the appropriate folders in the real Sun release, and merge in the PULSE lines into the sound.properties file. Restart your browser and Java sees PULSE and considers it a sound card. Very nice, we now have sound.

In the shot below, the vendor software is displaying the sound card and the clips play nicely.

Happier OpenSuse Days

2011-09-09T15:43:00.002-04:00

This week we scheduled, installed and rebooted OpenSuse 11.4 with the 3.0.4 kernel. We broke the infamous rule of technology and also changed from the kernel-desktop to desktop-default; which means now that it's working beautifully we are not 100% sure which of the two issues resolved the problem. If one had more hours, we'd prove or disprove both theories but at this point we are going to enjoy the success and just move ahead. We've used both kernels on other servers and both have worked fine. Previously on this new server disk performance was terrible after user loads increased and now it's absolutely blazing in speed. Even with 40+ users, avant-window-navigator is starting to open in about 3 seconds even over remote display. You can barely even notice that 40 concurrent people are working -- in other words it's working as I would expect from Linux. I have solicited our users for more beta testers and hope to get us in the 50-60 mark by next week.

Other updates:

No NX updates this week to test, so I'm going to re-install and tinker with the current beta to see how it works on our new Wifi network. Still have those performance ideas in my head, waiting for the new Linux client updates in order to test them.

A few Evolution patches to test were received from Novell. They were compiled for SLED 11 SP2 which is not yet released. So I built a quick server with old hardware and tested them. The patches were not entirely satisfying, but at least now we know their status. Emailed results back to the developers.

We have a requirement for our first Java applet to work with PULSE, which I have never done before. A quick Google indicates it's possible, possibly via the Alsa layer. Time to tinker and better understand how this all works. Right now the sound from Java is very nicely trying to play on the server in the computer room. :)

I spent a little time hacking on the new Support Portal software and adding features that are useful to me (and others). It's now much easier to search for users and display their departments and to search by departments. With many hundreds of employees, it's almost impossible to keep them all straight. Next up, I'm going to write some quick code to keep track of what's in /usr/share/applications/*.desktop and log which users are clicking on which applications and how often. All of our icons pass through two common shell scripts (one for Windows, one for Linux) and it's going to be very easy to just write out data when they are clicked. This will allow us to better understand applications that can be removed from the network because no one is using them anymore.

Started to look at upgrading MoinMoin from 1.8 to the 2.0 (beta) release. Lots of libraries that need to be upgraded, and will require an operating system upgrade as well. Downloaded OS 12.1 (beta as well) for review and testing. Technology churn really keeps us employed, right?

Happy Weekend.

OpenSuse + Support Portal Code Deployed

2011-09-02T09:52:00.003-04:00

The OpenSuse guys have been helpful in testing ideas to find the server bottleneck. I really don't stress over these things. Our older GNOME server is running fine and only those people that volunteered to beta test are seeing the performance issues mentioned in prior blogs. What's interesting is that even with the occasional freezes and slowdowns, they still prefer the new technology. All of the UIs are much nicer and easier and they really like avant-window-navigator. I'm sure we'll find it.

One of the things that is nice about hacking code is that you can picture ideas in your head and just write it and make it work. I love to see charts, stats and data. I also love having the computer monitor itself and report to us issues. Our Support staff has the same viewpoint. So I wrote the "Load" detail screen. The main portal displays in green the current user load for all of the big servers, and now when you click on it it gives you a chart with a much longer timeline. It then creates a front end to "top" and shows the top processes. The user name is a button, and you can click on it and the user detail screen which was already coded opens. It then hunts down all of their sessions along with information about their department. One can hone in on the sessions even further if desired and obtain information about the device they are using.

If any of the servers are over 10% busy for more than 10 minutes, a warning notify-send popup is generated and alerts all of us to review the issue.

The processes are GTK toggle buttons (marked in purple) and I'm going to allow our support staff to select processes and click the STOP button to shut them down. All of these features were available from the command line obviously, but this front end simplifies the whole thing greatly.

Up next for me: New NX 4 preview code (hopefully next week), testing the new WiFi with iPads, trying to get our Moin Wiki upgraded, trying to get 2 Evolution patches merged for SLED 11, installing and testing the 3.0 kernel on OpenSuse. But first, a three day weekend to recharge.

OpenSuse 11.4 Reload & Problem Is Back

2011-09-01T09:16:00.002-04:00

Tuesday we moved our OpenSuse 11.4 users to a VM copy of the new GNOME desktop and we wiped the physical machine completely. We pulled the drives from the RAID array and put them in a new order, and then re-added a new RAID (1+0) partition. Our customizations are always placed in /u and /u2 so a few well placed tarballs and tweaks and we had a brand new machine with the same functionality as the old one.

Very sadly, the same issues have come back. I appreciated all of the ideas and tips presented in the comments area in prior blogs. But I do have a bit more information and it's very odd. This server should run 200 users easily, but as we get over about 20-30 and especially at 40 we see performance get very poor. Here is the new information: If I vi /etc/services it blinks for a few seconds and then opens. If I copy /etc/services to /tmp/services and /home/services and vi those files...it opens immediately. So there is some kind of contention or lock on the /etc/ directory. This contention seems to be the core of the problem. So many services are constantly looking at those files, and they are somehow bottlenecked. If you have any ideas to assist, the bug report is here.

Networking is still sub-optimal; note the dropped RX packets.

eth0 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:72
inet addr:128.222.99.243 Bcast:128.222.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12191295 errors:0 dropped:73239 overruns:0 frame:0
TX packets:12731836 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5112551787 (4875.7 Mb) TX bytes:10726315924 (10229.4 Mb)
Interrupt:16 Memory:f8000000-f8012800

eth1 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:74
inet addr:172.23.1.235 Bcast:172.23.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:73278 errors:0 dropped:238 overruns:0 frame:0
TX packets:8009 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15639151 (14.9 Mb) TX bytes:827822 (808.4 Kb)
Interrupt:17 Memory:fa000000-fa012800

There are a few more steps that we can take, including installing the 3.0 kernel to see if that helps. We might have to start looking at other distributions if we don't see progress soon.

Other projects continue: Writing some code for the support portal, testing NX, looking at upgrading our Moin Wiki, WiFi upgrades. I also have been poking at some ideas to improve NX performance to our thin clients, will blog about that if it pans out.

Server Reload & NX Testing

2011-08-26T15:32:00.005-04:00

Server Reload

I have mentioned in previous posts some problems with our OpenSuse 11.4 server which is going to be used for running GNOME. It's been running slower than expected and is having some very odd disk and networking issues. The OpenSuse kernel guys have been great at reviewing all of our files and data, and right now everything looks like it should work fine...yet performance is not as expected. So we are going to rule out any chance that something failed during the upgrade from OpenSuse 11.3 to 11.4. As everyone that upgrades knows, sometimes what you get after an upgrade isn't the same as a fresh install. So next week we are going to make a Clonezilla backup of the server, and do a fresh install of OS 11.4 and then lay our customizations over the top. Our beta testers will be moved to a VM instance and won't have any down time. We'll know more next week. I keep reminding everyone that this is a normal part of upgrades and testing. Previous GNOME upgrades took time to debug and certify as well. The production machines are all working fine, so only those people that have volunteered to test are seeing these issues.

Testing NX To Anticipate Video Expansion

I've mentioned many times that we run old school X for those people on our fiber option network, which is the vast majority of users. One thing that is happening is that video training and video playback in general is exploding. We have noticed that certain streams (Flash, cough) seem to weigh heavily on our network from time to time. Many videos play fine, but at times higher quality (or frame rates?) cause the network to get more busy. The other issue we are seeing is that users expect a "video" to be a "TV" and are very often stretching the window as big as possible before there is frame loss or it gets too grainy. If we did not anticipate further growth in this area, we probably could continue to use our current design. But we are expecting this to continue to grow.

We use NX technology at our remote and low bandwidth sites. I have been testing it on the high speed network as a method to compress the video and video streams. I made a quick graphic below to demonstrate the difference.

The current design is to use X or XV and Pulse audio. With NX, the physical Xserver is on the system console itself and only a compressed stream is delivered to the thin clients. I made some tweaks to the configuration files and disabled XV and tested and results are not bad.

The shot below is a Flash video playing from youtube to a thin client over NX. This type of compression software is intelligent enough not to hammer your network. I had not tested this in several years and really it's working pretty well.

The one caveat with using NX/RDP/VNC type software is that sometimes it gets confused about screen changes and leaves artifacts. This is why it was never deployed on our high speed network. Remote X feels very similar to being on the console in terms of being crisp and repaints. In the shot below, while the video was playing I changed tabs in Firefox and the video section remains as an artifact. I had to take this shot with my camera because this is what the eye sees on the thin client. However, if you do a screenshot the artifact is not there. X thinks that area is gone, but the compression formulas don't always know to repaint all areas.

This testing was done with NX3.x, and we are awaiting the NX 4 Linux native client to see how it works. Pulse is now supported and will reduce our network overhead even more. We have a long way to go before we decide to make this type of change, but we are trying to say ahead of the curve with trends.

Happy Weekend.

Post Vacation Projects Continue

2011-08-19T15:12:00.004-04:00

One returns from vacation with a new sense of energy, so I am working again on the various issues and projects.

OpenSuse 11.4 Kernel Problem

I got a lot of nice ideas to try and get our OpenSuse 11.4 server running better, and sadly none of them have made major improvements. I finally feel like we have ruled out everything except the kernel and have opened a bug report with the kernel guys. None of the admin tools are showing why the machine is running poorly, but it feels as if the whole server is running from the swap device. Copies of this server moved to VM suffer the same results. Unless I can get some kind of movement in this area soon, I am going to have to consider other Linux distributions. A few well placed tarballs and a VM copy will allow me to do that quickly, but it's still not pleasant. Everything is in place and working, we just cannot scale over 35-40 users. This hardware should run 150+ easily.

Evolution Issues Continue

Still communicating with some of the Evolution developers in the hopes of nailing down some nagging issues. Email and calendar events flow and things are working, but it's not yet to a milestone that gives me comfort. Everyone suffers from low resources in the IT field these days it seems.

Firefox 6

It's out already and I have been testing it heavily with various pages and technologies. Another Flash release came out in early August and that is being given the Youtube stress test over remote display with PULSE. While it never will be as robust as a local video card, playback is not terrible and more and more users are playing this type of content. I have some ideas to improve how this works, and will blog about it in the future. Just not enough hours in the day.

NX Testing and Ideas

I was able to connect finally with a thin client to NX 4 and started testing it in preparation of it going live in the coming months. I also had a few product feature ideas that I emailed to them. While we are using NX right now only at remote sites with low bandwidth, we are watching the marketplace and the whole "cloud" thing. If we move anything off site, we'll need NX for compression.

Support Portal Project

One of the best parts about writing software is when people actually use it, and we recoup the time invested in coding. Merging lots of little functions into one UI is saving time for our support staff, and also I feel allows us to provide better service. When there are problems with printers for instance, we know about it within just a few minutes.

Once again I will mention, please no HIG comments. :) I have kind of come to the mindset that projects should not be over-engineered up front. I'm talking about the types of projects where you spend so much time in design and building it "correctly" that the end users don't see any code for months and months. I'm building ideas and testing them with real people. Some of the data is still stored in flat files which is clunky of course; but the end users never see that stuff. Why make them wait for you you to build and optimize a database when just experimenting? If I abandon an idea, only a short period of time was used. I'll clean as I progress and like the results.

I have started to write some code for the handling of printers. You can now select a department and all of the printers in that department appear. Each printer offers three options/buttons: the first button launches Firefox and just goes to the IP of the printer. This connects with the HP web admin page found on all of their printers. The second button will display the number of print jobs currently in queue (over all servers), and when you click on the button it takes you to the UI for the viewing and halting of print jobs. The third button (marked in green) is an idea I had to make use of the under-utilized "lpmove" command. Yup, you can move print jobs to other printers...and no one ever uses it. When you have 60 printers, you can barely remember them all, let alone where they are located. So using lpmove on the command line would require digging out some kind of paper document to remember where each printer is located. So I have started to build a simple screen that will allow us to enter in the closest printers to each printer, along with directions from one to another. That will allow our support staff to take the stuck jobs, and fire them to a printer that might be located 50 feet away from the original. You might be wondering: Why not just cancel it and just have the user resend it? The real world answer is that many users tend to print (and expect it works 100% of the time) and delete and empty trash before they see that it really worked. What I envision then is the end user would get an email message alerting them the print jobs were moved to a new printer, with directions on where it's located. It'll be interesting to see how this all works. Printers really continue to be a nightmare, anything I can do to reduce our time on them seems worthwhile.

I also wrote some of the initial code to allow for searches of our users by first, last or account name. All matching users appear in the form of buttons. Hovering your mouse over the buttons gives you further details. This continues my pet peeve about making people open a detail screen to see information that should be easily found on the primary UI. If you do click on the user button, a detail screen appears with much of the same information found in the tooltip. It then also scans the servers and looks for their logins and lights up a monitor when found. We allow users to log in multiple times, so it's possible they have 1-4 active logins at a time. Other detail information that will appear on this screen (not yet written) includes information about their print jobs, and then a listing of software applications they are allowed to run. Linux software is deployed to nearly everyone because it's mostly free, Windows software has licenses and must be checked prior to launching. We also are looking at tracking how often all software is used, so that we can review it periodically and remove those packages that are no longer needed. If you click on a user login monitor, it takes you to the thin client detail screen and gives you greater detail about the physical device they are using. This allows us to check software versions, color depth, type of connection and the like.

Happy Friday!

OpenSuse 11.4 Woes

2011-08-09T09:07:00.005-04:00

I'm leaving for a vacation tomorrow so I thought I'd post a blog concerning the status of our OpenSuse 11.4 GNOME server. We are in the home stretch of the deployment, but unfortunately the underlying Linux is giving us problems. I thought I'd relay the information and maybe someone has some ideas on getting this resolved. When I get back in just over a week, we'll start doing some major changes one at a time to resolve the issue. I'm not concerned about project success, but the best path is not yet clear.

All of the software is installed and for the most part GNOME and avant-window-navigator are working as expected. The hardware is beefy and should support at least 200 users+, but as soon as get to about 20 users - things really start slowing. We are seeing problems with the networking and disk performance. They may be related, but I haven't yet been able to prove it.

We have cloned the server and moved it VMWare and it's suffering the same problems. So I feel pretty confident it's not hardware. We are using this hardware elsewhere with no problems.

Disk IO Problem

For the first time we tried to deploy ext4, all of my Googling seems to indicate no major problems and no complaints about performance. I did not see any "multi user" issues reported. If we have a few (10ish) people on the server things work just fine. Once you get more, the disk becomes VERY slow. If I vi /etc/hosts (shot below), it just sits for 3-4 seconds with a blinking cursor before the file appears.

With a user load, if I run the passwd command it sits for5 seconds after entering a password before it completes. On other servers even with hundreds of people, all of this happens instantly. if I run yast2 and install software with a load, the whole server becomes very slow and non responsive. On other releases of OpenSuse we have never seen this before.

While this might seem a clear IO problem, what also enters my head is that we are having some networking problems too; and I'm wondering if the NFS mounts are having problems which is affecting local disk IO. Unfortunately I cannot dismount the NFS mounts to prove that idea, too many necessary pieces are on remote file systems.

Networking

It seems like others are having OpenSuse 11.4 dropped RX woes too. The first shot below is the desktop server, and note the packets that are being dropped. This number climbs constantly every few seconds.

But note on this shot below from OpenSuse 11.3 how it *should* behave. This browser server has over 100 people using Firefox 5 and you can see it's being hammered on the network side and there are almost no dropped packets. This reflects how our other Linux servers work.

I have noticed that certain NFS activities really slow the machine, when they are in Nautilus and generating thumbnails there is a noticeable slowdown for the other users. But it's not clear if this is "networking" or "disk io"

What's Next?

Aside from someone out there having intimate knowledge of OpenSuse 11.4 and a quick fix, we'll begin making drastic changes when I returned. The first thing I'll do is grab the experimental 3.X kernel that I see on some of the OpenSuse channels. I have one inclination that we are having a kernel/scheduling problem because it's affecting two subsystems. Currently I have upgraded to the latest stable release of kernel-desktop-2.6.37.6-0.5.1.x86_64. If that doesn't work, the next thing that I'm going to do is reformat the server with ext3 and rule out a file system problem. This is a nasty thing, and going to require lots of redoing work that is already complete. But what we have now cannot be moved into production.

One other idea I had was that somehow the -desktop kernel is not suited for multiple users, and I could try moving to the more vanilla version; but it sure seems like this change won't make disk and networking better.

If you work on OpenSuse and have any information, it's greatly appreciated. We are anxious to move this server into production.

Project Updates

2011-08-01T09:42:00.009-04:00

It's been a while since my last blog, but as always projects continue to evolve and mature.

Sandboxed/Jailed Firefox Sessions

My last few blogs were concerning us creating sandboxed/jailed Firefox sessions to run certain City applications and that has been fully deployed and working well. I was able to get Firefox 5 pushed live with no problems. Flash 11 was also released and that too has been deployed. I am currently testing Firefox 6 with Java 1.7 on the web and with our internal applications. As is the norm lately with upgrades, 1.7 fails to work with some of our web based software. Wasn't the point of a browser to make it easier to deploy software? :)

Networking Problem, GNOME Desktop

I have been fighting a networking problem on OpenSuse 11.4 where after a certain amount of people log into the server it starts to get odd lags and we see RX errors on the NIC.

RX packets:45731224 errors:0 dropped:307554 overruns:0 frame:0

When you issue a command such as "vi /etc/hosts" it sits and blinks for about 3 seconds before it displays. However, as users log off at night it suddenly begins working again. I'm going to update the kernel tonight and the next step will be to install another type of network card and see if the problem goes away. This unfortunately has prohibited us from adding more beta testers. Very odd.

Portal UI And FOG Thin Client Updates

My coworker Brian has been doing a great job in getting our FOG server (running in VM) configured and working. He has been testing various techniques to push out updates to our thin clients. Many combinations are being tested in terms of performance. He is looking for the sweet spot in how many current thin clients to update at the same time. Sometimes there are circumstances where doing 5 thin clients at a time twice is faster than doing 10 all at once. I should have more information on our final analysis in the coming weeks.

While he has been improving pushing updates to the thin clients, I have been spending time as it's available working on our "Support Portal". Having all of your user sessions on a server is usually pretty wonderful to support, but in many cases it relies on command line tools and tinkering by hand. The tools that come with the distros are more designed for configuration versus server monitoring. I have been trying to think of ways to have the servers do most of the work, all of the information is right there; but it shouldn't take a System Admin to review it and know how to process.

I know those of you that build screens all the time will want to post HIG and UI comments. Please, these are back burner ideas that are still being developed. Our support staff already seems pleased with the features and capabilities and it's barely even started. If we have to do more with less (tm), this is certainly one way to obtain that goal.

The server monitoring screen is giving us graphs that show total number of users, load and total print jobs. The later is probably our most support intensive. I hate having to go into child screens to get information, so the designs are trying to make use of tooltips as much as possible.

Here is live data coming from the servers, user loads are displayed. Hovering your mouse over the load graphs (blue) gives you a tooltip that shows you the user accounts names running that particular software/server.

The lower graph (cyan) turns read when print jobs appear to be stuck. Hovering your mouse over the graph shows you the user and printer that appears to be having problems.

If you do click on a print job graph, it brings up a UI that shows the details of the print jobs on that particular server. The blue area is a toggle button and let's you pick multiple at once. The print jobs will then be capable of being cancelled or moved to another printer. The red area is the name of the printer and when clicked initiates a Firefox session and goes to the IP of the printer. Those of you with HP printers know that they provide an administration console on port 80. The magenta area is the name of the user. Clicking on this area will being up a user detail screen; which is not yet written.

When print jobs are stuck, we now get notify-send popups alerting us of this fact. A similar popup appears when the server seems to be 10% busy after several samples.

The companion piece to the FOG thin client updates was to create a screen to find and maintain files which contain server side settings for the thin clients. Once they get an update, they attempt to download a flat file which contains their settings and then reboot into the right mode. From the thin clients tab you can enter 3 or 4 octets of the IP address and the results will appear. A wrench symbol appears on thin clients that have server side configuration files completed. If you hover your mouse over the thin client entry, it gives you detailed specs on the device. You don't have to open a child screen to view the data.

You can filter and look for thin clients running in many different modes. If someone asks us how many thin clients are still in 1024x768, for the first time we can easily obtain that information. We can also easily query and find out how many thin clients are using two monitors.

We can search for the "function" (purpose) of the thin clients. We are using the same hardware for different purposes around some the city. Some are full featured workstations and others are set into Kiosk modes. Others are configured for low bandwidth sites and use NX.

If you click on a thin client pushbutton, a child screen appears and returns all of the information about the device. We can see who is using it, how many monitors it has and whether it's using HDMI/VGA/DVI cables. We can reset it back to factory defaults, reboot it, we can request a remote control, and we can do a wake on lan and power it on remotely.

At this point enough is working in this code for us to finish configuring the 650+ thin clients around our sites. We hope to have that complete by September. The UI will progress and advance, and I promise will even start looking nicer. :)

We've Got Browsers

2011-07-14T09:33:00.002-04:00

I mentioned that I had a project assigned to try and get multiple browsers running to the same user at the same time so that we could run the release supported by the various vendors on our intranet. One doesn't want to allow older browsers and versions of Java to get out to the Internet. Behold, three versions running in one session:

In this case, a user can use the Oracle and SAP software and currently surf the Internet using Firefox 3, 4 and 5.

I had to think about the best way to make this work, so the current version always works normally and makes use of the users bookmarks and settings (obviously). The older versions run as users "firefox3" and "firefox4" and I made use of the -ProfileManager feature in Firefox to allow multiple users to run from the same user account. I then locked down prefs.js and localstore.rdf to settings I want to be used. For instance in order to make it look more like "software" I hid the URL and menu bars so they aren't tempted to try and surf with the sessions. The beauty of this design is that when Firefox 3 and 4 are no longer needed, I simply delete the user account and all of of these settings and infrastructure for that version is deleted.

If you are struggling with the same issue, this technique allows you to run all of your Firefox sessions from the same server. Our server for Firefox is a monster, and it would have been a shame to have to offload this functionality to another server or VM instance.

Many Projects Continue

2011-07-12T11:50:00.004-04:00

For those interested, many concurrent projects continue here at Largo.

FOG & Host Stored Configurations & Portal
We have continued working on implementing network updates using FOG and things are going very well. We have a group of about 40 devices configured to network boot and have been testing the wake-on-lan features and configuring FOG to do updates at specific times. Our network guy has been reviewing the logs to see how much these updates impact the network. I completed the UI to allow us to store host based settings for the thin clients (pic below). It's a very simple file that is downloaded at first boot of update and configures the thin client back to the settings we desire. In between things I have been hacking on the portal UI below too and slowly adding features and figuring out ways to make it easier for our support staff to perform repetitive and common tasks that are handled via the command line right now. I'll post a more detailed blog when more is finalized and working. As mentioned, this really is a back burner project.

Firefox 3, 4, 5, 6, 7 & 8

OK, I know we are only up to Firefox 5. :) But the new aggressive version upgrades and feature changes have presented a challenge for us. It's very easy to do upgrades here, it would only take me a few minutes to physically install the new release. The impact is the *software* on the backend. Many vendors are very slow in "supporting" browser and Java changes. One vendor in particular that currently owns Java has software that has to run Java that is 2-3 versions old. This whole issue is a challenge for us on a centralized computer, I can't even imagine the gray hair it's causing on a client/server network. I tend to be the type of person that tries to turn a problem into a strength; and I believe we have a solution. What we are going to do is keep a copy of older Firefox releases for specialized intranet applications and not allow these sessions to get out to the Internet (for security reasons obviously). They will be jailed inside our network. For instance we have one financial package that requires Firefox 3.6, and another that requires Firefox 4. What I am going to do is create user accounts "firefox3" and "firefox4" and when the user clicks on an icon for those apps it will "su" into that account, start the appropriate version of Firefox and then automatically connect to the software authentication screen. These sessions will not be able to "surf" to the Internet. When they click on the Firefox icon, it will launch the most recent release (5 in this case) and use their personal bookmarks and settings. I don't want these older versions of Firefox running as the end user and impacting their settings. The graphic below is what I sent out to our IT staff to illustrate how it will work. Some technical issues remain, but most of the scripting is already in my head.

Virtual GNOME Desktop Working
The other hurdle that we completed was getting the new virtual GNOME desktop server working automatically. The physical server can be copied once a week now. One issue we had was that when rebooted the server would try and use the same IP address as the live server. So a few lines of code at startup detect a VM instance and changes the address of the NICs automatically. So we had a few users log into this second server and they were very pleased that all of their settings and customizations were all the same. This virtual GNOME desktop will allow us to split the load of users and allow for a second host to be available to be available in the event of some kind of hardware failure.

NX iPad Testing Continues

2011-07-01T14:40:00.005-04:00

Now that the NX pre-release is working well enough for testing, I have been spending time asking questions and understanding how it works. My expectation was that there would be a place to configure your resolution just as you do in the NX clients. But the way it works is that you log in 800x600 and then use the desktop tools (xrandr front ends or whatever) to change your resolution on the fly. These tools are not exposed to our users, so I had to come up with a way to do this automatically. I'm sure when I have a few days to think about it, I'll have more ideas on how to properly get this deployed.

The code below was the hack-ish thing that I did to make it work for now. The xdpyinfo string returned by NX is different than our thin clients. Xsession checks for 800x600 and then does a secondary check to see if the NX string is returned. In that case it can then kick you into any resolution that you wish using xrandr -s x.

NX then automatically implements something similar to the -scale command of xrandr and fits whatever resolution you pick into the real estate of the tablet. From the code sample, you can see that I tested 1024x768 which gives you an almost 1:1 scale for the iPad. I then bumped it to 1440x900 and it automatically fits the entire tablet and is usable, but a bit grainy. Probably we would not bump it up that high nor give it the wrong aspect ration in this manner. Regardless, it was an interesting test. In the shot below I'm running software in 1440x900:

The speed of the presentation is still not ready for end users, they are impatient and it's hard to explain that tapping icons 50 times in a row doesn't make it faster. :) My tests were over 54Mb WiFi and it seems about 80-85% the speed that I would desire. Much better than early previews, but not ready for prime time.

Next up will be additional testing of NX over EVDO and also testing the webplayer from browsers over broadband connections. If this works well, it might mean we can stop having to give City employees client software to take home to log into the City network. I need to see better performance before that change could occur.

Happy long weekend!

PS: The iPad stand from Amazon is wonderful.

Lunch With A Side Of GNOME

2011-06-29T13:19:00.002-04:00

Sometimes your plans for the day are altered greatly because of external circumstances. Nomachine released the latest NX 4 preview last night. We have been very anxious for this technology in order to deploy iPad/tablets, so this was my primary project for today. Prior releases inched closer to our goals, but were *far* too slow to do any beta testing. VNC testing over EVDO was painful as well. At this time no native client is offered for the iPad, instead it works by just using the Safari browser and then connecting to a web server. X is started inside the browser and your desktop appears. Performance on Firefox/Linux/Wired is very snappy and fast. Safari/iPad/WiFi works fairly well as does Safari/iPad/EVDO. So for the first time ever, I was able to take an iPad to lunch with me and log into our new GNOME server. I present, lunch with a side of GNOME:

Lots of issues remain, but it's wonderful that we have progressed this far. Back to testing.

Gotta Love Glade + Python

2011-06-28T17:07:00.003-04:00

It's wonderful to have ideas in your head and be able to quickly get them prototyped and running. Using Glade and Python, I was able to connect the features that we need to make it easier to support our thin clients remotely. The software is detecting the number of monitors, resolution of the monitors, whether DVI or VGA is in use, color depth, Xserver size, uptime and ping response. It also allows us reboot the device, reset back to factory defaults, request remote control access and get a command line prompt from the Linux OS on the thin client for troubleshooting. (There are local processes that run and might have to be checked: Rdesktop and PulseAudio are two examples). I also have it snapping a screenshot and placing it over the top of the monitor image. I'm just using -gravity north from "convert" right now, and it's going to take some tinkering to get the various aspect ratios to fit correctly...but I'll leave that for a Friday afternoon project. :)

My coworker Brian had great luck with FOG today, wake on LAN is working and we configured all of the thin clients in our training room for PXE boot. Using FOG and the new server side storage of settings we were able to easily push an update to the thin clients, they rebooted and configured all of their settings and then booted right up to our XDMCP chooser. Very cool stuff, and we are looking forward to continued testing.

Doing More With Less, Improving User Experience

2011-06-27T15:08:00.004-04:00

Usually when there are improvement areas in IT, we already know about them and have them in our heads. The roadblock is almost always the number of hours in the day, or money. The use of computer technology is exploding here at the City, and unfortunately we are not really hiring enough people to cover these increases. So we are trying to find every possible way to reclaim staff time. Sometimes in doing so you actually improve the user experience as well, and that is the case with my current project.

When you have 600+ thin clients (or PCs) it's very difficult to know exactly how each is configured. There are also sometimes cases where not all updates have been applied. We also sometimes have circumstances where users request additional features that require a small change to their thin client. Steps are being take to allow all of these tasks to be performed remotely.

These ideas have moved out of my head and into the form of screens. Please, no UI nazi comments. :) I'm sure HIG is being broken, and that a few pixels here and there are wrong. Right now this is just a work in progress to move designs on paper to the real world. Some of this is coded and now working and being tested for functionality. The thin client detail screen offers as many remote features as possible. From this screen we can remote control the user, force all settings back to defaults, reboot their device, see the number/orientation/resolution of their monitors, see which cables (DVI/VGA) are in use and also for the first time store all of their settings on the server. Here is the current (rough) screen. I also plan on splitting a thumbnail of their current screens on the monitors images as well so that support has an instant preview of what the user is seeing. Looks like xrandr can return back a string that indicates the exact monitor model, and I'm checking to see how that works.

We currently configure many settings locally to ensure the best possible user experience. If their monitor won't rotate, that feature is disabled. The monitor is configured for optimal resolution only and does not perform any plug-and-pray type polling for these settings. If you have ever supported users with changing resolutions, you know how important this stability is to your mental health. :) The UI above stores these entries in a flat file sitting in a jailed area of our internal FTP server. When the thin clients are reset to factory defaults or updated with a new operating system they now try and download this file automatically. If the file is found, the configuration UI toggles the right buttons and reboots itself in 10 seconds. So no more human intervention is required. Once it works, it will now always work after upgrades. It's working well, and we are going to try and test this in our training room which contains 16 devices to evaluate a network load.

My coworker Brian is concurrently working on testing FOG to allow for PXE updates. In our initial deployment of these thin clients, PXE updates were tested with the HP provided Altiris and we were not happy with the results so it was never enabled. It's our hope that FOG will allow for easier updates and scale better. Once that is up and running, the technology described above will allow the thin clients to automatically go right back to the tried and true settings that the user was previously using. Getting everyone on the latest code will solve problems for sure and these centralized techniques will make this easier than ever.

There are a few other ideas in my head to increase efficiencies. Another that is being developed is software to make it easier to allow us to see how the servers are performing. One negative aspect of using tools like "top" is that if several of us are monitoring the same server we are all consuming resources obtaining the same data. So what I did was write a very quick background script that checks user counts, CPU usage and number of print jobs and then create charts just *once* and store it in a JPG image. When you go into this monitoring tool, all you are doing is simply loading in the JPG image every 5 minutes. 10 of us looking at this at once will be no problem and we are all sharing the same resources. I put in a bit of code so that if it detects a condition where there might be a problem the chart turns orange and then red to catch your eye. The first iteration of this test UI is below. Once the features are working as expected, the UI will be improved.

So things are busy for sure, but it's fun to be hacking again and the City will for sure easily recoup these hours in staff resources.

SQL To The Rescue

2011-06-20T15:31:00.003-04:00

As I continue to close up the last of the minor issues with the new GNOME desktop, I decided to quickly implement a dialog that appears after you log into the server. The print cost database is now loading information completely via cron and the numbers are astonishing. We have departmental printers and costs are high. I can't imagine the costs in Governmental agencies that have desktop printers. Be assured that we are doing everything possible to save tax payer dollars here at Largo. This kind of stuff drives me crazy!

For the first time, when users log in they are made aware of their print consumptions. As you can see, I don't print very often. :)

And the SQL to perform this query took just a few minutes:

We are still settling in on a value for the cost of a theoretical 100% coverage. The cost per page that they quote you is always for 5% coverage. So 2 cents a page @ 5% coverage is around 40 cents per fully printed page. That value is multiplied times the coverage rate and number of pages.

Fun With Aspect Ratios & Resolution

2011-06-16T17:01:00.003-04:00

I'm not sure what it says about a person if this seems fun, but today I worked on improving some of our internal scripts that allow our support staff to remote control end users. :) This was a project brought about because of a desire for increased tablet usage. One caveat of that process is dissimilar resolutions when using VNC. Some of the users have higher resolution and some work in portrait mode and vncviewer would open 1:1 in the resolution on the remote site. This created scroll bars that had to be navigated, and the user would move things around and not realize you couldn't see their whole screen.

So as part of the iPad remote control project, I came up with some additions to the ksh code that checks resolutions on both sides and then always ensures that their screen fully fits on yours. This code is used in both areas: support to remote control end users and also to remote control your desktop session from any device. These shots show the results:

Desktop user (1440x900) is remote controlling an end user who is in portrait mode (1024x1280):

iPad (1024x768) is remote controlling a portrait thin client (1024x1280):

iPad (1024x768) is remote controlling a landscape thin client (1440x900):

The code is really very simple, get the height and width of both devices and figure out the aspect ratios. Then use that to figure out if it's best to use the width or height as your baseline, subtract a few pixels for window manager and lower panel, and calculate the other value based on the aspect ratio.

Fun stuff, and useful for the rest of our staff. It's already live and I should get feedback tomorrow.

Testing Remote Control Ideas

2011-06-15T14:23:00.003-04:00

As the GNOME desktop project is getting closer to completion, I spent some time testing an idea that I have had for a while: Offer another option to remote control an existing login. On the current server, if you log in and already have a session running (from another thin client) it allows you to 1) exit without killing the other sesson or 2) Kill the other session and log in cleanly from the current device. We of course have talked about running NX or similar technology for all City thin clients which makes it easy to just steal a session. But old school X is still our preferred technology in all of our buildings with fiber optics. It has a different feel, is more crisp and your software looks like "software" and not a "picture". If you have used NX or VNC, you know exactly what I mean. When possible, we go for the best presentation.

So with this design, I wanted to offer a another option: 3) Remote control your existing session. This solves many use cases for us. You might want to gracefully shut down your software or you may want to quickly access your already running software.

I'm using our good friend x11vnc and it's working well. When you log in for a second time, it alerts you of that fact and then offers you the option to remote control your existing session. Once selected it initiates x11vnc on your original thin client and then starts the viewer on the second device. The second device can be another thin client....or an iPad. In the shot below, I logged into the thin client initially, and then attempted to log into the iPad; got the dialog and am doing a remote control.

What I'm looking at now is the best way to use the -scale option. The second login will very probably be in another monitor resolution. In the case of the shot above, original thin client was 1280x1024 and then I -scale'd it down to 1024x768 for the iPad. Lots of aspect ratio issues to consider.

A fun little useful project, and a good sign that the new server is almost finished.

GDM & Login Cleanups

2011-06-13T11:58:00.002-04:00

We are inching closer and closer to going live with the new GNOME desktop upgrade for our thin clients. With all of the complexities of hundreds of icons and the nuances of having hundreds of users, it really does take 6-12 months to prep a server for this type of deployment. Not only do you have to consider the GNOME server, but also all of the remote servers and software applications. We are also weighing upgrades and connection techniques (RDP vs Citrix vs UIS).

We are up to about 45 concurrent users that are testing the new server, and uncovered a nasty issue in GDM related to two users trying to authenticate at the same time. Halfline/Ray was awesome as usual to create a patch and it's already posted. I just now need to get it built on OpenSuse 11.4 and will test.

My focus last week was on optimizing the user experience in authenticating and then the time it takes before avant-window-navigator launches.

GDM Tuning
These are the changes I made to GDM to make it run faster, and produce a dialog as soon as possible after XDMCP request

I opened the .ui file of the password dialog box with glade-3 and made it a fixed size. Previously the "animation" process was kind of slow over remote display. It would display with no entry widget and then wiggle a bit and resize larger and then complete. My strings and logos won't be changing, so it was ok to test with my art/strings and size accordingly. This improves the user presentation and seems to make it a bit faster.
I turned off server side GDM wallpapers, hesitantly. I like the idea of keeping as much as possible on the server in case one wants to make changes in the future, but by turning this off I was able to speed up the process of getting the password dialog. I made a change to the thin client build that gives the Xserver a black wallpaper (-br), so the old school gray/white grids are gone. Users only see this for a few seconds at most each day, so comfort has increased in this change.
The /tmp/orbit-gdm directory basically never deletes any linc* files, and at one point we had a million temporary files (after many, many months of me not knowing about it). I now have a cron running that cleans these out each night with linc-cleanup-sockets. I'm sure that temp file placement is diminished when packed full of old files.
I removed several .desktop files in the gdm startup group that we didn't need. It was attempting to start another pulseaudio process, even though pulse is already running on the thin client. This was probably creating some nastiness and delays.

Xsession Tuning
I have hand selected some gconf settings that are loaded when users log in and found that they were taking a few seconds to load. On a home/personal computer forcing in settings would never be done, but in a production environment where people are working 24 hours of the day...I have to ensure a working session each and every time. These gconf settings remove any customizations that might cause failure. So what I did was break them into smaller groups and then fire them off into the background concurrently. This was able to shave off several seconds from the point that you enter your password and hit ENTER to the point the panel appears. I also reviewed every line of the script and removed any items that were unneeded or obsolete.

This morning I put all of these changes live, they had been tested on a VM clone of the server and worked fine. So far the results are promising. The GDM dialog appears almost immediately upon request and the panel appears in about 4 seconds after the password is accepted.

This week I'll continue to ensure that all icons are brought over from the older GNOME desktop, QA work, and also working on testing the VM copy that is being generated once a week. We are experimenting with techniques to have the VM server detect first boot after copy and transition to new IPs automatically.

Happy Friday & Updates

2011-06-03T16:38:00.002-04:00

I haven't blogged for a while but have certainly been busy. Things I worked on this week:

Print Job Data
The project to generate print job costs is finished and running nearly unattended. I have been QAing the files before the load into the sqlite3 database each morning, but should be able to cron that process in the next day or so. It'll just email me the exceptions if there are any. Printing costs are way too high, now comes the hard part of trying to break 20 year old habits and changing the culture.

GNOME Desktop
I'm adding more and more users and testing features and making tuning changes. There is some lingering NFS oddness still and I have some ideas of changes to make on the NFS remote server. I also found that /tmp/orbit-gdm was holding 1 million temp files, and now implementing a cron job with linc-cleanup-sockets running as gdm to remove the dead files. We are getting some odd delays at the point of XDMCP connect and the screen fully appearing, checking into that too. I have been updating artwork and icons and testing to make sure that the software packages are launching as expected. We are running about 40 concurrent users now and things are working well.

Password Protected OpenOffice Files
We have a City policy that if you password protect files, that you have to fill out a form and give it to your Director. It seems like that wasn't always happening so I was asked to write a script to find the files that were password protected. I found that unzip -l lists the .odt archive regardless of password protection...however the Thumbnails folder was missing in the case of a password. Makes sense, generate no thumbnail for a document that is locked..otherwise people would see the first page. So a scan found about 20 documents that were locked, not terrible.

iPad VNC Testing
NX still hasn't released version 4, so I started testing VNC over EVDO on the iPad. It's really really slow and not something that could be deployed. It only works in 8 bit color and that gives them a pretty messy experience...but even at 8bit color it's too slow. 16bit color took a full minute for the GDM wallpaper to even appear, and then GDM timed out and the connections drop. Completely not usable. I'm going to poke around with other ideas, and hope that a NX client is released at some point.

Happy Weekend!

And One More Thing, Resolutions

2011-05-19T16:32:00.004-04:00

Just wanted to pass along a little trick that I use when writing software. About half of our 600 monitors are still 1024x768. We don't replace monitors until they are dead as good custodians of taxpayer dollars, and LCD panels are lasting many years. Usually IT developers have wide screens and some of them have dual monitors. It's easy to forget about the rest of our users, and suddenly software is developed that doesn't fit on their screens. Nothing worse than having to re-engineer a beautiful UI after finding this out during beta testing.

So what I always do is use my handy wallpaper that has a black box drawn at 1024x768 for testing screens.

We Have Print Data

2011-05-19T15:59:00.005-04:00

After some short meetings with our Director and other staff, there seemed to be some excitement about analyzing our printing data. They gave me a list of some additional functionality and I had fun learning how to use some new widgets with Python/Glade and how to retrieve data from sqlite. I don't get the chance to write software very much anymore, and I still find it enjoyable after all of these years.

Here is the shot of the UI, simple and easy to use. The users select the data range, the departments and then the various functions they wish to query. The information is returned in three tabs on the right side which appear as "Department", "Printers" and "Users". One of the requested features was the ability to get the data as it appears in the Tree into OpenOffice Calc. The sqlite query is looping to fill the tree, and it was very simple to just dump the data concurrently out to a perl library which allows you to generate an identical .ods file. When the Tree displays, the file is finished and ready for use. I added an area labeled 'Files' which contains the spreadsheets. When clicked, the buttons use the normal MIME popups that we use on the rest of the desktop. The perl library does not generate the thumbnail file that normally appears in an .ods file, so the preview window comes up blank.

In the shot below, I selected OpenOffice - Calc opens the .ods file and will allow them to tinker with the numbers or copy and paste into other applications as needed. It took a bit of Googling to find the right technique to get the numbers over as integers and not as strings. But it's all working.

The primary UI gives them SQL SUMs of the various data, and I knew that people would be shocked at the number. So if you double-click on any of the Tree rows, it opens a child window which performs the same SQL statement except showing the detail records. It's very interesting to be able to see all print jobs by department, printer and user in chronological order. You can really get a better understand of how people are using/abusing the software.

I have a bit of work to do yet to merge in the log files coming over from the MS Windows spoolers which are in a different format. At that point management can review the data and consider ways to reduce costs with the use of other technologies such as portrait monitors, policies, training and tablets.

Paper & Printer Usage Under Review

2011-05-11T09:46:00.002-04:00

As part of the GNOME desktop upgrade, a project has been initiated that is near and dear to my heart -- printing. There is a strong culture in many people to fight computerization and continue to work with 10 or 20 year old techniques. The amount of paper and ink used is tremendous and our printing costs are finally coming to light.

CUPs dumps flat files into /var/log/cups/page_log which contains lots of useful information, but is in no way in a format that can be researched. This is especially true when you have multiple servers and operating systems. We have always known that printers are heavily used, because they are a big draw on IT support. However, we have never been able to prove an exact dollar amount nor provide any specifics about poor techniques being used around the City. So I wrote some quick ksh scripts that scan through the page_log each night at midnight and dump the information into a sqlite database. This is yielding very interesting information.

I have done some SQL statements by hand to begin to look at the data, but obviously this isn't going to be the way that the IT Director or analysts will work. So I am writing a quick Glade/Python screen for running the various searches and the current shot is below. This won't be used by end users and won't get much more bling & art. It will serve the purpose of allowing our staff to select a date range, select departments and then include/exclude various desktop functions. The ink coverage is estimated per print job and we will be able punch in the per page cost for HP printers and it will give an estimated cost of the range selected. Initial estimates are that this cost might be north of 150K a year. :\

If we can use the new desktop, portrait LCD monitors and iPads to change peoples work habits, I believe we can increase efficiencies and reduce costs.

A Good Detour Through Lake Worth Florida

2011-05-04T09:53:00.011-04:00

The week has been productive. One smaller project that I completed was resyncing avant-window-navigator to the latest versions and deploying this on our Beta desktop. This has a fix in that allows the shinyswitcher to work with VNC and NX connections. I'll be testing that fully this afternoon. This was the last patch required to allow iPads to log into the City and get a fully functional desktop. Cool stuff.

The other project was initiated with a conference call to our IT friends at Lake Worth, Florida. They are exploring ways to implement thin clients, Linux and open source software packages. We offered to send them a copy of our customized thin client build to assist in this process. A customized build has a serious flaw: It's a customized build. :) It was designed for us, and our networking designs were hooked into the scripts which have matured and changed through the years. We have a long term goal of redoing our IP scheme, so making the thin client build a bit more modularized was on our radar anyway. I grabbed some post it notes and went through the code and made a list of all server connections and features that other organizations might want to disable and then built some new UI screens that appear on first boot. The new design makes IP changes a snap, and allows for additional government agencies to piggyback off the project if they so desire. A few hours of design, coding and testing and it's working. There are still some additional areas that need work, but enough is done that we can now ship them a copy of the build and it will work on their network. The UI could probably be better, but it only displays for 30 seconds the first time a thin client is configured and never is seen again. I got feedback from our support division and these screens allow them to touch and update a thin client faster than ever.

On first boot (factory settings), splash page is generic. This provides a visual cue that no settings are configured and the build is currently generic.

The build detects no settings and comes up to a configuration screen (glade/python) that requests that you pick the desired City. (( If you are a Government agency and thinking about HP 5745 thin clients, email me and let's chat about adding you to the build ))

The Site Settings tab has all of the options and IP addresses of servers.

Select the monitor resolution. We hard code the values into xorg.conf to ensure a consistent user experience.

How many monitors does the thin client have?

Does the monitor physically rotate to landscape and portrait? If the monitor doesn't turn, why even allow them to do so?

What is the function of the thin client? Some are are used for desktops, and others are used for Kiosks (human resources, cyber cafe).

After saving the settings, the thin client is now configured for City of Largo.

And users in Lake Worth will get all of their customized artwork.

It's always exciting to hear about other agencies that are exploring ways to save money and as mentioned, these changes improved the quality of code and functionality at our site as well.

The Road To Seamless Windows

2011-04-27T14:33:00.002-04:00

While we do run a lot of open source and Linux software, of course there are still some applications that run in Microsoft Windows. I personally don't work on the Windows side of things, and it's handled by the other System Administrator.

The launch scripts to run these applications are mature and work well. In the last year we made some changes mentioned in previous blogs in regards to using Rdesktop running on the thin client itself, along with allocating some thin client RAM for video cache. All of which have made the applications run virtually the same speed as they would on the server.

The last technological feature that was always on the radar was running them in a seamless window. This is the opposite of how it worked previously; a container would open and the software was sitting inside of this space. This design wasn't terrible, but disallowed users from having all of the resizing capabilities that they wished. A new version of Rdesktop (1.7.0) was just released in the last week or so and the Windows administrator spent some time testing it once again. Previously there were some stability problems when running seamlessly, and it would sometimes crash. It never felt ready for daily and enterprise use.

He tested Rdesktop 1.7.0 and felt it was very stable and would work well within our deployment. The next obstacle was that the HP thin clients are running Debian Lenny and no backports of Rdesktop had been done and possibly never will take place. So we built a VM to run Debian Lenny with full development tools and libraries. This VM image will be kept for future projects that might arise. Rdesktop was compiled cleanly and then we moved it over to the thin client manually and installed the binary and libraries and everything worked. I spent some time this morning rewriting the rdesktop launch scripts on the thin clients to understand the desire for a seamless window and use the appropriate flags. Rdesktop is a bit twitchy about the order of the arguments and this had to be tested and QA'd.

So after less than a few hours of work, it's now ready for release to our beta desktop users. Once stress tested, this technology will be pushed to the rest of the City. In the shot below, you can see Internet Explorer running in a seamless window, tightly integrated into the Linux desktop. A single click of an icon in avant and you can be listening to Steve Perry in just a few seconds. ;)

FireFox 4 Scaling Well

2011-04-26T10:31:00.003-04:00

There is always a period of time after you requisition hardware that you are a little nervous about performance. Very happy to report that Firefox is scaling very well and we have jumped over the magic number of 100 concurrent users in FF.

The only open issue right now in regards to Firefox is that it still is having an occasional oddity in regards to NFS. From watching patterns, it seems like if you are downloading a big (1GB+) file and writing it over NFS, the server get's sluggish. It never comes to a crawl, but has a noticeable slowdown. I was able to make this work better by downgrading from NFS4 to NFS3. I'll see if I can figure it out and submit a proper bug report.

The shots below are 100 concurrent FF sessions. Not everyone is surfing at the same time, but it still shows the memory usage along with CPU for maybe 30ish active sessions. Very happy to see Flash not multi-threading. :) I believe this server could easily handle another 100 users, and can easily be upgraded in terms of RAM.

Ipad, Server & Desktop Updates

2011-04-22T17:08:00.011-04:00

A Word About UI Design And Artwork
I have always been a person that shows screenshots and mockups when blogging. I find that when I read other blogs and can't see a picture, I find myself wondering exactly what they are describing. From time to time I get comments about how the UI "sux" or that the artwork is "terrible". I wanted to make a few comments; I never take it personally, but what really helps me is to make your own mockups and post them for me to see. I'm not an expert, and sometimes people have made some excellent suggestions. The other bigger comment is that being a Government agency, if we had an artist doing all of our icons and if we spent hundreds of hours working on UI design - as a taxpayer that would be disturbing. If we had that many hours to work on software and art, I would consider us over staffed. I always try and reach a balance point where the UI works and is functional, and always try and use existing artwork that others have created.

Ipad Portrait Portal Screen Finished
After meeting with some test users, they approved the initial screen and I have finished all of their changes (and it didn't take me 100s of hours :) ). The biggest change was that in the prototype the UI was blocked until all of the thumbnails were finished. Once they approved the design, I rewrote that code to not block the UI while the documents are being thumbnailed. In the shot below you can see that it finds all of the documents (in about 2 seconds) and displays the clock/busy symbol on the thumbnail and then fetches them while leaving the UI active. Previously 30 document so would take around 25 seconds to complete. Now the user can start working immediately.

Remote Logins with VNC and NX
I am still testing various remote connection software on the iPad and as I mentioned the VNC Software called Remoter is working very well for us. The developer has been very open to ideas and the response time over VNC is crisp and instant. When I touch the stylus pen to tablet running Xournal, the pens and markers are as fast as a local computer and very fluid. The next step is to work with our networking and security guys and get this working from the outside so that performance can be tested over EVDO and Internet connections.

Concurrently, the Nomachine guys have released their next alpha release of NX which finally supports iPads. The software continues to have an excellent design and installs easily, but the iPad and browser performance still suffer in comparison to using a native client. My testing indicated about half the speed of VNC. While promising and interesting, right now it's not something that could be deployed. Hopefully performance will improve during the beta and release candidate stage. We certainly have no objections to a native client for iPad if it's faster; even though it would require installing additional software.

Ipad Landscape Login
The Xsession file is configured so that if you log in in portrait mode the above UI is used, but if you log in in landscape mode the server thinks you are just a regular 1024x768 device and gives you a full desktop. There is a bug in shinyswitcher in avant-window-navigator that causes it to crash when using VNC/NX and they guys on #awn are cooking a patch. Response time is excellent over VNC and it's a very usable roaming desktop with plenty of real estate. Around 50% of our LCD monitors are still 1024x768 so this resolution is still fully tested and QA'd and it's working great on the tablet. I'm pondering some ideas right now in allowing users to grab control of their already running desktop from the tablet. That should prove interesting.

Servers Are Offline
In this case, that's good news. The old NFS server and browser/Firefox3 server have been powered off after running as an emergency backup server for 1-2 months. No problems with all of the new 64bit servers and things are running very well.

New GNOME Desktop Deployment
It's always a challenge to consider how much information to give users when you deploy new software. Most don't read what you send them, no matter how carefully crafted. I have noticed that when sending email they will read a bulleted message with very short explanations versus long paragraphs. I also have found that boxed, quick concise snippets of information will catch their eye. So with that said, the document below is the current handout that is being readied for the upgrade.

I'm Back & Project Updates

2011-04-14T10:14:00.003-04:00

I was off for almost two weeks, the first time that I have taken an extended vacation and it sure helps recharge the batteries. Some things took place in my absence and this week has been super busy.

Groupwise/Evolution Replacement
While I was on vacation the IT Director and Infrastructure Manager continued to review options for moving email into the cloud and met again with Google. There are a lot of considerations concerning security and bandwidth and based on all of your comments many you agree that deploying email and calendaring to hundreds of users is never pleasant. :) The IT Director has not been satisfied with milestones in regards to Evolution improvements. Evolution is a work horse for us, thousands of messages a day go through and for most people it runs 8 hours a day. But those with more messages and huge calendars seem to have more crashers; and these people tend to be in management positions.

GNOME Desktop Upgrade To OpenSuse 11.4
Everything was working very well on our beta desktop server running OpenSuse 11.3 that it was tempting not to upgrade. But we decided that we didn't want to be a version behind when pushed to all of our users and performed the upgrade. For the most part all of our customizations came forward and after installing the new desktop-agnostic I was able to sync avant-window-navigator with newer code. A few addon programs needed some changes and patches and we now everything is running as it did before.

The one oddity that I have seen is that gsf-office-thumbnailer now creates thumbs with transparent backgrounds. :\ Does anyone know why this was changed? Possibly to make it blend in better with themes? It's really not working well for us. Users use the thumbnails all the time to locate documents based on the first page.

iPad Testing Continues
We are still finalizing ideas and a scope of work for exactly how to use iPads as thin clients on out network. After meeting with a few end users, a few minor changes were requested to the UI and that's almost done. We should be able to allow some limited testing of these ideas in the next 1-2 weeks. There are opportunities for very real dollar savings by decreasing printing costs.

GNOME Shell
Now that it's out, I'll spend a bit of time testing it again over remote display. We also have some Intel based thin clients (HP t5745) which might produce different results than the ATI models. Initial testing a year ago did not demonstrate this to be a viable option for us.

iPad Thin Client Work Almost Done

2011-03-24T16:22:00.002-04:00

The project to use an iPad as a thin client is nearly ready to push out to beta testers and the current prototype UI is below. Nice big buttons for fingers, and only showing them the bare minimum they need to quickly access their documents. Handwriting and highlighting is working great in Xournal. These solutions are not going to techies, but to employees of various computer skills. I'm hoping by next week to start getting some of their feedback.

And Now Testing A Fullscreen iPad UI

2011-03-23T17:14:00.003-04:00

The iPad work flow that I mentioned the other day is working well. The original design was going to put documents into a 7/8ths screen mode and have a panel sitting below. The overwhelming consensus was it was far better to be able to view and markup documents in full screen mode. This gave me a lot more real estate to experiment with some designs. Please, no UI nazi comments. :) The shot below is what is being tested. Widgets and art are not final and the layout is not complete. But this clearly displays all of the documents the users have put into their MobileDocuments folder and allows them to easily copy documents that others have marked as being shared. The buttons and spacing are for fingers and a stylus. The user taps the thumbnail once and a larger preview displays, they can then edit the document or send it out via email. They can also create empty PDFs (Xournal) and OpenDocuments (OpenOffice) with just a single click. The software preselects a file name and saves it once, all they have to do is begin the construction process.

Some new art, a nicer theme and a bit of alignments and we might have something that can be pushed to some beta testers. Documents are 100% always on the server and can never be lost.

Dragon Naturally Speaking testing is going well too, and text is pasting into documents with just a few taps. Fun stuff.

iPad Testing Going Very Well

2011-03-21T13:48:00.015-04:00

If you have been reading my blogs through the years, you know that I consider it to be our top priority to reduce waste, run computers as efficiently as possible, and save the taxpayers as much money as possible. The steps that we are taking into the area of tablet computing is not being done because it's a "toy"; it's being done because we have some inefficiencies in regard to using printers. The culture started in the 1980s and just has gotten worse each year. Each software package that we buy that is supposed to be "paperless" seems to create more print jobs than ever.

So the iPad is being reviewed in a manner to allow for more of a paperless office. One idea being kicked around is looking at only deploying them from the cost savings of reduced departmental print jobs. Reduce first, then get the device, and then maintain the reduced printing. The dollars spent on consumables for printing could easily pay for devices in the first year, and then each year thereafter would provide cost savings. The worst part of this process will be mind share, habit and desire. People WANT to hold paper, people LOVE paper. They very often have no concern about how much it costs, and how much support is absorbed by IT.

So the perspective that I am testing is not using it for a "computer", but to use it to replace the binders that are being carried to meetings. I want them to feel like they are using paper. I want the documents to be easy to find and easy to edit. I don't want them to get a full GNOME desktop and then have to click and navigate to find their documents. I also want shared documents to be available for the meeting, and easily absorbed into the users folders as a personal copy and then easily editable.

The other big issue for me is that I don't want any documents stored on the local device. Moving documents back and forth is 10+ year old design. Users tend to make islands of data, and documents are always lost. Users having to go back to their desk and dock and then copy files will not succeed, file management is always the part that they struggle with the most.

We purchased a stylus for testing from Amazon, the iPad thinks it's a finger on the glass. This allows for fewer smudges, and also allows you to write notes and highlight. It works great!

So these ideas that have been in my head are finally being tested, and feedback of the design has been very positive. It's fast, easy and removes any "techie" steps from their work flow.

On the new/beta GNOME server is a folder right on their Desktop that houses what are considered "Mobile Documents". These are documents that they wish to have available on the iPad. It's simply a staging area. It also holds documents that are created on the iPad; easily located and then moved into project folders at a later time. In the shot below, clicking on the icon displays these staged documents. All of the MIME bars that launch when they double-click on files allows them to easily add to this folder as well:

For this initial test, I'm using VNC. This replicates the functionality exactly, and we will do a full review of products when we are closer to deployment. Hopefully the NX upgrade product will be available at that time. In the meantime, for one dollar we purchased the "Remoter" application and found it to be excellent. It's got a nice interface, allows for a paste of the local clipboard (more on that later) and allows Linux applications to run very similarly to how they would if they were on the local iPad. In the shot below, the iPad is turned on and the icons appear.

Once the remoter software is touched, a nice interface comes up and shows you your available connections. I have created a Landscape connection (which gives them a full GNOME desktop), and then a Portrait connection (which gives them this document editing mode).

Selecting this option gives you good old GDM, identical functionality to the users thin clients. Note the translucent buttons that hover over the screen. These give you the ability to define how to react to your finger and touching the glass and work great! For instance, you can select to make your fingers simulate the mouse wheel. This allows you to scroll up and down through the document by sliding your fingers on the glass. You can also simulate left mouse or right mouse clicks. When you go into full screen mode, these buttons are hidden. You can go back and forth between this mode and full screen mode by touching three fingers on the glass. Very clean.

The UI appears after authentication and only shows them what they need for document processing on the tablet. The documents they have placed into $HOME/MobileDocuments appear as thumbnails on the bottom. The most recent document is always on the far left side. Touching the document once gives you the file name and file type. Touching it a second time opens it. Note the floating buttons provided by Remoter that allow you to exit fullscreen mode and bring up a keyboard. (( no UI nazis please, just testing ideas right now! :) ))

In the case of PDF files, the Xournal application is invoked and placed into full screen mode. You can beautifully see the entire page and write notes over the top and highlight. These are saved as another layer in the PDF document. Response time on the pen over VNC is fluid and very usable. Pepp cooked a nice patch for us that allows PDFs opened to default back into PDF format, no more XOJs to contend with.

When the user returns to their full desktop and GNOME session, all of these edits are in the folder and instantly available for filing. In the shot below, the document that I edited on the iPad is the same on the desktop. It never left the server; simple and elegant.

Some users have also requested a way to use Dragon Naturally Speaking. This application is free for the iPad and seems to work fairly well. In the workflow being tested, the user speaks into the software and once they are done place the "note" into the local clipboard. In the shot below I have created a simple document by speaking and am doing a copy:

I can then double-tap the HOME button on the iPad and flip back to my Linux session and use the Paste button of the Remoter application and the text is pasted into any software running on the server (including inability to spell Linux) :) . Once saved, this document is available from their MobileDocuments folder.

The coming weeks will be interesting and exciting. Hopefully some cost savings and better ideas will fall out of these designs.

Google Mail/Calendar & iPad Testing

2011-03-17T12:30:00.005-04:00

Email and Calendaring is still being reviewed to build a list of all available options and ideas. We had our first high level conference call with Google to see their product and get their pricing. There is no silver bullet with running email in the cloud. It solves some problems and gives you new ones. One example is that our users have a large set of available ICS files available to them on our intranet. If the calendar UI is running in California, that means we have to put them on the Internet. And that also means we have to push them over our circuit upstream to appear in the UI. So there are a lot of issues like this being discussed and reviewed. More testing and meetings to come, I'm sure. Other products are being reviewed too.

As I have mentioned previously, we have some cost savings ideas in regards to using iPads to display documents and provide some remote connectivity. These savings would come in the form of reducing printing and replacing some laptops that are used in City vehicles which are higher than the 500 dollar price tag of the iPad.

We were kind of waiting for the release of NX 4, but delays in that area have required that we look at other technologies. So I enabled VNC on our new GNOME server and have started using a vnc client on the iPad to at least simulate this work flow. All of our custom screens and UI popups have been designed with 1024x768 in mind and everything is working fine. Now that I have logged into a full GNOME desktop, I'll spend some time testing the custom iPad Glade UI that I built to expedite file management and for use in meetings. I'll post a summary of that analysis in my next blog.

So the shots below are:
- Logging into GNOME, avant-window-navigator and custom MIME UI all fitting in footprint
- Opening Firefox from the GNOME session. What's interesting about running Firefox from a server is that if you are in the field with a EVDO card you will have access to a much faster 50Mb pipe inside our building. EVDO is then only used for screen repaints.
- GNOME Activity Journal running, thanks to Cando for a quick fix in trunk to allow it run correctly on the Xserver that is used by Vnc-server.

Evolution (And Groupwise) No More?

2011-03-07T07:58:00.004-05:00

Just typing those words reminds me of the classic comic book frame from the late 1960s:

Sadly, this week one of my projects is to create a list of ideas and alternatives for moving off of Evolution and possibly Groupwise. Both of those products are under support contracts, and after many years of tugging they are perpetually staying at a grade of "B-" for enterprise users. Patches are slow in coming and require constant pinging by our staff to get them moving. And we are having problems with regressions. I think that sometimes companies remember the old days when patches and upgrades would come in 1 or 2 year cycles, and just have not adapted well to Internet time and how to turn open source software into a positive. There are people out here that would gladly download and test upgrades that come from a build service; which would eliminate the issue of regressions and running a "one of" type build as it is now. One really expects better service when paying 25K+ a year.

So I'm going to review the current status of the web interface to Groupwise (bad) and then the Java Groupwise client (worse, SLOW, leaks memory). One idea that I'm going to test is running the Java client locally on the thin client as a way to get around the fact that this software is impossible to scale and run on a multi-user server because of how badly it leaks memory.

There is a strong trend now to just farm out email and put into the cloud and be done with it. One would think that this would put email vendors on their toes and increase the quality of their products, right? In many ways I'm running out of arguments to counter this trend.

Finalizing The Beta Desktop

2011-03-01T13:58:00.003-05:00

Lots of technology was pushed live in the last two months, a lot of my time was consumed by this process and then debugging and tuning issues. We had some very bad/odd issues with NFS4 on the Firefox server communicating with our NAS disk storage server. Under heavy loads it would lock up and suffer from slow performance. We disabled NFS4 and dropped everything back to NFS3 and it seems to be working better. I don't have the hours to debug this problem, and because NFS3 is fine for us...it's going to just have to wait. If any of you are working on this code, feel free to find me on the IRC and I'll give you more detailed information.

I'm back on the project of deploying the GNOME desktop on OpenSuse 11.3. Everything is coming together nicely. The Avant guys are packing up a new tarball for me so that I can get the most recent panel changes. This really is one of the final parts of this project. The only other major piece of technology is waiting for NX/Nomachine 4 to be released.

We are changing the design of our desktop servers. Previously we had two servers with identical functions, both capable of running a full load of users. This was done in case of failure, the second one would always be available. In order to save money, and reduce staff load we are going to make the backup server virtual. Once a week, the primary GNOME server will replicate itself to a virtual instance: boot, and automatically change its IP addresses. There were a few reasons why this made sense for us; 1) the hardware is rarely failing and we haven't yet had an instance where we needed to run the whole City on one computer. 2) Budget and staff cuts require that we do more with less 3) Having this desktop tested and certified as running virtually will give us better disaster recovery.

So I'm testing the virtual copy of our GNOME server and it's working well. Not as fast as real hardware, but certainly fast enough as a backup.

I'm closing in on the final design of the user interfaces and so far everything is working well. The screenshot below shows the MIME UI screens running. Users can now opt out and use a more traditional approach to file management (as mentioned in a previous blog). What's interesting is that very few people did so, this functionality is pretty popular and helpful.

(screenshot shows the helper UI that comes up for pictures, PDFs and documents)

And the shot below shows all of the functionality that I have bound to hotkeys: Windows+F5 brings up Beagle. Windows + F6 brings up Activity Journal (with fresh trunk UI fixes!). Windows + F9 brings up the weather applet which has undergone some upgrades and changes. The weather applet downloads all of these maps ONCE for the whole city and they are shared. This reduces bandwidth requirements greatly.

I saw some posts that I need to re-read carefully concerning the removal of the minimize and maximize buttons. We have to be very very careful with this decision. My gut reaction is that this would have major negative feedback from our end users. We are getting dangerously close to making decisions about what technique people should use, and very often people have barely figured out *one* way to do things---it might be poor in our eyes but it works for them.

GNOME Activity Journal + OpenOffice 3.3

2011-02-08T14:54:00.006-05:00

It's been far too long coming, but after some pestering on #zeitgeist(sorry guys :) ), I have gotten the Activity Journal working for our users. Multiple users and documents being on NFS always makes these things more interesting, but I got it working well enough to push to testers.

Right now it's only showing documents created with OpenOffice via .recently-used.xbel; but this is the piece that helps us the most. Because we have a multi-server environment, there is separate instances of this file on different computers. I need to tinker with combining them together either over NFS or with some kind of merging process. That project is on the back burner though.

Here is the journal running from our NAS server. .recently-used.xbel is copied over from the OpenOffice server, loaded into $HOME , daemons started and the GUI launched. I'm sure I'll make improvements to the design as it's tested. So beagle is now bound to Windows+F5, and Journal is bound to Windows+F6. Documents searches have never been easier for our users.

Speaking of OpenOffice, I upgraded us to 3.3 this morning. It was the usual process: Close all running instances, block them from running it in the script, run the upgrade, QA that it worked and then open the one script to our 800 employees. So far no major issues and they are starting to test the new features.

Up next: Two days of debugging and testing Evolution. There are some problems and I want to create proper bug reports.

Beagle Returns

2011-02-04T10:50:00.004-05:00

I mentioned that a week ago we installed a new NAS server to hold all of our documents. Now that a few tuning projects are over, I had some time to restore the functionality of having Beagle available for users to search their documents. We continue to use Beagle because it offers the ability to build the databases in a manner suited for multiple users, and multiple departments. You can tell it to create a database of all documents in the IT directory, and then grant permissions for only IT staff to see those files. Each individual user in IT doesn't then have to crawl and find the documents; it's only done once.

Beagle had changed a bit since the old 32bit server was deployed, so I had to check out some new backends and command line arguments. In looking at the old code and clunky UI, I decided to try a new approach to make it easier. This only took a short period of time, it's simply some new bling on top of the old infrastructure.

The old beagle-search UI had their available sources (folders) in a drop down menu along the top that they could toggle on and off. Users rarely found them and didn't know what they were selecting. Since all employees have access to three folders, I made a quick glade/python UI to connect them to the right document databases. The shot below shows the current design for testing. I added this UI to metacity/Windows+F5 so that it can be opened with a simple keystroke. UI pops up and you tell it if you want to search your personal documents, your departmental documents or citywide documents and then beagle-search comes up pointed to that static database. Simple, easy and stable.

Now all I have to do is install our standard MIME document bars on this server and the project is complete.

PS: Looks like Beagle-search really needs some new artwork, pretty fuzzy icon.

OpenOffice/Oracle Help, Firefox, OpenFire

2011-02-02T10:31:00.004-05:00

Some experimental work has been done to start checking on how well OpenOffice can connect to the Oracle database. The database is outside of my division, so I'm hoping someone just knows how to solve this issue:

We can connect to Oracle cleanly using the jar file driver and everything works. However, for some reason it's displaying what looks like system tables along with the tables we desire. Has anyone experimented with this connectivity and resolved this issue? It seems odd to me that this is being exposed over ODBC.

New Browser Server

After going live on 64bit Firefox we had a few days with some brownouts and issues that required some changes and tuning. Two things seemed to have corrected the problems and it's running very well now. The original deployment used a NFS directory to be the area to hold temporary files instead of /tmp. This worked fine until a certain number of people and combination of plugins and then the disk io started to bottleneck. With 600 users that can access the Internet, there are thousands of sites they visit each day and it's hard to simulate that during beta testing. I moved the temporary storage back to /tmp and then only copied files to NFS when they were needed by external applications. I stumbled on "iotop" which is a beautiful piece of software. Those of us that used SCO Unix back in the day were always spoiled with "iohog" which performed a similar function.

We also had tuned Firefox with some custom tweaks available on the Internet to increase performance. These too worked in beta testing and under smaller loads, but it looks like when we got over a certain number of concurrent users it flooded the Barracuda, firewalls and eventually the Internet circuit with too many requests at once. When these settings were de-tuned things worked better. Even without them, response time is excellent.

So we have settled in around 80-100 concurrent users in Firefox and here is the load:

The City has "cyber cafes" set up in various places to allow employees to get on the Internet. This was done so that non-work related sites could be visited on breaks and eliminate this being done at their desks. I pointed the thin clients to the new server and have created a point release to our internal build with this feature. I'm testing it this very second and it will go live tomorrow. At that point nearly all services are disabled on the old server and we can begin the process of putting it into retirement.

We use OpenFire for Jabber on our internal instant messenger network. I brought over this software from the old server to discover that it was crashing on 64bit Linux. By default it runs its own included Java, which is only 32bit. All I had to do was modify the openfire rc.d file and add a line to point it to JAVA_HOME. It then ran the 64bit version and is working perfectly. Another item off my list.

Projects on the horizon for me: Upgrading to OpenOffice 3.3 (next week), working with Vincent and Hans on some GNOME issues, getting avant-window-navigator recompiled with all of the latest features, watching for updates to NX 4 to allow us to log into the servers with an iPad.

MIME Helper Interfaces & Opt Outs

2011-01-26T12:36:00.003-05:00

A few people commented to me when I blogged about our MIME helper interfaces that this intermediate step would drive them crazy. As those of you know that follow our work, when you double-click on a file a dialog comes up and then offers you several options on what exactly you want to do with the file.

What's interesting is that we just deployed a new Firefox server a week ago, and there were a few file types that were not configured to open these helper interfaces and we got calls from some users that had no idea what to do when a download manager opens and asks them where to save a file. I then added those file types to the mimetypes file and they were content. So, I know that a vast majority of our users have benefited from this module being in place. Instead of having to worry about WHERE, they can focus on WHAT.

But, there are a few advanced users that want a traditional interface on the new desktop. When they double-click on a file, they want the default application to just open. They are perfectly content with moving the files around manually and do not like the additional step in the middle. So after some conversation with my Director, I wrote a quick Glade/Python UI that allows people to opt out of the helper interfaces and to just launch an application.

I broke this down into three different settings. One of them is for the Desktop, which is defined as interaction of files with Nautilus. The second is Evolution, which is defined as the interaction with files that are attached in email. The third is with Firefox, and defines how to handle files that are downloaded from the Internet. This offers a great deal of flexibility and those users that wish a more traditional approach to file management is able to work in this manner.

In the shot below is the opt out UI, along with a view of the current helper interface that opens when you double-click a file. Changing the setting to "No" disables this feature.

We're Hiring

2011-01-21T10:45:00.002-05:00

This job opening is not in my division, and I'm not involved with the hiring process. But I wanted to pass it along to anyone interested. The City is hiring a Programmer to write business software. Currently most in-house applications are written with a toolkit called Panther. It doesn't get much simpler than this tool; connect to the database and drop and drag the fields to a screen and then hook up the SQL queries, buttons and logic.

There are pros and cons to working on Government, but it's a very stable work environment and it's fun to be a part of working outside the box with Linux and save lots of money in the process. The job would require living in the area. Largo is located close to Tampa/St Pete Florida.

Here is the job posting, and good luck to anyone that wishes to apply.

64Bit Firefox Deployment Complete

2011-01-20T10:26:00.005-05:00

There is always an unknown when you deploy new software to hundreds of people. All of your testing never simulates their techniques and a full load. But yesterday the cutover to 64bit Firefox 4 on the new server went beautifully. We announced the upgrade and at noon I just killall'd their processes on the old 32bit server and moved the binary out of the way to halt it from running. I then tar'd up all of their .mozilla directories, and extracted them on the new 64bit replacement server. I changed the global launch scripts and was done. The total process took about 15 minutes.

The shots below are the 'top' windows with about 65 users with Firefox sessions running. Watching the load split on the threads is a thing of beauty and scrolling is as crisp as it was with just our beta testers. There are a few spikes in the 5-8% busy area, but the bulk of the time it sits under 1-3%. I believe this server could easily get 300-400 users on it concurrently. Very nice. Another item off my list. :)

Here is the link to the server.

Firefox Goes Live + Thanks + Possible Job Opening

2011-01-18T13:46:00.003-05:00

As I have mentioned previously, lots of technology has been awaiting the new year when we are fully staffed in order to deploy. Tomorrow we are going live with the new 8 CPU, 6 Core server for deploying Firefox sessions. Everything is all compiled cleanly and natively on 64bit OpenSuse 11.3, stable and fast. Between noon and 1pm when the users are lunch, I'll do a backup of all $HOME/.mozilla directories and push them over to the new server. Then I just change the launch scripts to point to the new server and everybody is live. This technique works out well; if we have an issue that requires a rollback, all we do is change the launch scripts back to the old server. Today I'm giving the new Internet circuit and the plugins a good shakedown. Frame rates are good and sound quality over PULSE is excellent.

I'm expecting an afternoon tomorrow of minor tuning issues and helping people that didn't read the announcement message; but no major problems. In the shot below you can see I'm giving FF a good Flash test with Youtube.

Many thanks for Vincent Untz who has been working with me on getting a few upstream patches merged into OpenSuse 11.3 to fix a few showstoppers on our GNOME desktop upgrade project. A few more patches to GDM, Nautilus/GIO and avant-window-navigator and we will be ready to move the desktop into wider beta testing. Right now I have about 20 people testing and using it fulltime, and feedback is very positive. I've spent a good amount of time squeezing every second of time from our customized Xsession login script. I'm firing some of the code into background processes as appropriate to allow the script to get to the point that gnome-session starts as quickly as possible.

There is a chance that we will be hiring one programming position in the next few weeks. When I have an exact job description and notification, I'll post it here in case anyone is interested.

Finalizing USB Sticks & Cameras

2011-01-12T14:01:00.002-05:00

I have blogged about parts of this topic in the past, but the project has developed and I wanted to update the status.

I have been working with management and end users to come up the best way to handle USB sticks around the City. One wants to allow users to upload their photos. However, using USB sticks as a 'sneakernet' needs to be discouraged. The whole point of having a centralized environment and using thin clients is so that the data remains on the server. Some people still fight the concept of centralized files and want to squirrel their data away. This usually ends up with disaster. Drop the USB stick in the parking lot and it's gone! We also want to avoid the situation where someone lassos entire sections of City documents and puts them on a USB stick and walks out the door.

In regards to word processing documents, we are a proponent in keeping them in OpenDocument format to try and ensure that they can always be opened in the future. There also is the issue that when Microsoft file formats are used, it's a downgrade format when used with OpenOffice. If a certain feature is used that has no comparable feature in MS Office, the information is not retained. So we always tell people to use odt,ods,odg formats when not collaborating with people on the outside. And now that OpenDocument is starting to be included in MS Office, the case for using this format is even stronger.

So with all of these things in mind, we have developed our first release of the new design. I'm sure there is a better UI design, but for right now we continue to focus on the work flow. In the shot below the UI on the right side opens for all City employees when they insert a USB stick or camera. It does a quick scan of the device and generates thumbnails of the most recent 36 photos. To insert these into any City applications, all they do is click on the photo; this places it into the clipboard at 1024x768. On the right side you can see Draw, and the right-mouse Paste command has inserted the photo. No file management skills required, no lassos and all they can do is move the pictures upstream to the server. This eliminates the need of having to give USB access to employees that might want to take a few pictures a month for posters or flyers.

If the software detects any Microsoft Office documents, the button [ Convert To OpenDocument ] lights up. If the user clicks on it, their documents are sent to the Linux server in a non-privileged user account and converted to OpenDocument on the command line (pyuno) and then copied back to the stick in a separate folder and without touching the original documents. The red arrow indicates the button was pressed, and then all MS Office docs are converted. Those users with USB stick access can then use Nautilus to bring them over to the server. The original MS Office are hidden from view so no chance of them grabbing the wrong files.

Our Director is going to take this approach to the other Directors and then we will get this code into the hands of end users for testing and feedback.

One last bit of information: All of this is unrelated to the users ability to send and receive MS Office documents via email as part of collaboration and sharing information with outside people. These steps are being done with USB sticks to protect our data; and try and break their habit of working using 1980s and 1990s file handling skills.

Project Updates

2011-01-06T12:21:00.006-05:00

The New Year is here and lots of technology was waiting in the wings for our staff levels to be back to normal levels now that the Holidays are complete.

I finished the software module that allows our support group to support remote users. We also can obtain information about their Xserver (depth, dimensions) from this UI too. It's been live for a few days now and they are reporting that's been very helpful and saving us valuable time. The software it replaced took far more steps.

Support staff types in part of the users name into the Filter area and then a search is performed. Once they find the right user, they click on the user and it displays technical information about their session.

If Support clicks on the [ Request Remote Control ] button, the other users receives the dialog below. This allows them to grant permission before the session is viewed. Instead of running VNC on the end device, I'm using x11vnc which is just an excellent approach. We have users logging in with thin clients, NX/Windows and NX/Mac and x11vnc works no matter what end device is used. It's also 100% host based and can be upgraded and altered once instead of having to touch individual devices. If we attempted to put VNC on Linux, Windows and Mac there would be operating system nuances to deal with as well. The other positive thing that I discovered is that x11vnc is much faster on the new 64bit server. Support gets quicker UI responses and repaints; very nice indeed.

Here is what the remote user sees when we send them a remote control request:

The new MIME bars that we have been testing for many months are stable and working well, so I have backported them to some of our production servers and will be integrating them into our technology. In the shot below, opening a PDF from Evolution opens the new UI. Our Firefox upgrade is nearly ready to go live too, and this UI is used there as well. They also will see them when the new GNOME desktop server goes live later this year.

LibreOffice installed cleanly on our production server without disrupting OpenOffice, so I added an icon to the desktop for some beta tester users to "kick the tires". No decision has been made to transition at this time, but it's prudent to monitor this issue with an eye to the future.

Some Project Updates

2010-12-22T15:21:00.005-05:00

I wanted to blog one more update before the Holidays. I'll be off the next week enjoying time with friends and family.

We received a patch from Barracuda that allows our web filter to work in a manner better suited for our network design (all users coming from the same IP address). This was the last hurdle before deploying our new 64bit Firefox server. It will be nice to see the 48 hyper-threading CPUs in production. It should be a nice upgrade for our users and make their sessions more responsive. We also got a circuit upgrade and more bandwidth. This should go live the second week in January.

Now that my iPad prototype login is finished, I spent time bringing over more shortcut icons from the production GNOME server and testing them on the new one. Mostly all that remains is Windows apps now, and they are being QA'd as we move from using Citrix to RDP. My coworker has also been experimenting with running them in seamless windows, which would be a nicer user experience.

We had a utility on the old GNOME server to allow people to share their desktops with one another using VNC (x11vnc). What I saw when lingering in the support area is that they spent as much time showing people how to initiate this request as they did actually solving their issue. Users have a hard time finding new icons. So in my new design, the originator sends a vnc request and once accepted their desktop remotely displays. All the users have to do now is wait for a popup window, accept it and then get their support from our staff. I have also experimentally added a few more options to the UI, and having a fun time learning to use Glade and Python. The shot below shows the screen. You perform a search for the user, it matches their name and alerts you that they are online (green button) and then gives you options to take over their screen. It also gives you information about their resolution, color depth and thin client OS release number.

Nomachine released a preview of NX 4 last night, and being that this will be used for several of our projects I took the time today to get it installed and test it. The install process was quick and easy, and I simply typed in the URL and authenticated and the GNOME desktop appears right inside Firefox. I am waiting for some networking to be altered to allow my iPad to connect to the City network so that I can test this from that device using Safari. This technology seems to be working pretty well and promise. I really need to give this all a good shakedown (and maybe read a few manuals ;) ), and will report further as it's deployed. In the shot below, I'm logged into GNOME and then using Firefox opened another session with a different account name.

First Tablet UI And Work Flow Nearly Finished

2010-12-10T15:40:00.004-05:00

This week went quickly, along with my various projects I have been improving my python skills so that the language wasn't a hindrance to creating the ideas that were in my head.

When logged into the new GNOME desktop, the MIME bars were updated to offer another selection to Save 'To MobileDocuments'. While this is a simple concept to computer people, moving files around the network is very difficult to regular users. With a single click, the document that is being displayed will now appear on the iPad/Tablet device.

The first iteration of the user interface that displays when you log into the network in portrait mode from an iPad is finished (below). All of the documents in your MobileDocuments folder display on the lower panel as thumbnails. When you click on them once, the drop shadow changes color and the status line displays information about the file. Touching the thumbnail button again or hitting the [ Open ] button opens the document in the appropriate software application. I'm designing with the consideration that users will be holding a stylus pen. I'm not sure how easy a right mouse click will be with a pen, so I'm coding with this limitation in mind. The UI supported PDFs and OpenOffice documents, and I just added photos. There might be circumstances where employees need to take photos with them into the field, and it only took a few minutes to add this feature. Being that everything is running on the server, if the iPad is lost, stolen or fails, no City documents are ever lost.

When I am sure that we are feature complete for the first release, I'll spend some time on the UI. Focus was on work flow, eye candy can come later. Here is a shot of a 768x1024 window, displaying documents and pictures and then opening a photo with eog.

With the coding nearly complete, I'll be at a stand-still now until NX 4 is released. Right now it's impossible to log in from the iPad. With this sub-project nearly finished, I'll focus on delivering the rest of the software packages on the new GNOME desktop and await patches to fix the last of our show stoppers.

Up next: Expanding use on the new desktop/GNOME server, putting the new Firefox server live, testing the new thin client changes and creating a large NFS drive for our documents. December is looking to be very busy.

Already A Good Morning

2010-12-08T08:29:00.002-05:00

Yesterday I blogged about some ideas we are experimenting with to reduce printing costs. It was one of those days where you walk out the door knowing that pieces are just about ready to work.

I wrote a few python lines of code for the Glade UI to copy a pdf template from a skel directory into the MobileDocuments folder of the individual user and then fire Xournal against that document. The results are below. I then simulated the stylus pen being used for note taking.

The documents created are 100% on the server and never on the local iPad/tablet (simulated) and immediately show up in the users full GNOME desktop. Double-clicking on the file brings up the PDF MIME bar which shows the document already available on the network.

I just have to learn a little about using the paste clipboard signal in python and that feature should be working here pretty quickly. That will allow Dragon Naturally Speaking documents to be pasted and immediately placed on the City servers.

Then I just need to write some code to create a listing of files in MobileDocuments and generate thumbnails. Not a bad start to my Wednesday.

Tablet Footprint & Cost Savings

2010-12-07T12:05:00.005-05:00

Sometimes it's very hard to pack many months of conversations and needs analysis into blogs. My current project is just such an issue; I could type paragraphs describing the specifications but will try and be concise. We have tried to close up a number of open requests with one design.

There is a growing demand for mobile solutions that allow users to move around the building and gain access to their files. Our centralized design makes this easier, and many different concepts were discussed. The first inclination is to move in the direction of getting a laptop footprint device. Those of you that have seen my previous posts know that we already have a mobile laptop thin client that is being tested. The laptop footprint is not the best for meetings. Sitting in a room with 10 screens flipped up is not really ideal, and then you have to consider power needs, cords and batteries. They also are not suited for what people want to do most: annotate over the top of previously created documents. It would be wonderful if regular users worked as we do in the computer field with Wikis and sharing information, but that's not how they work at this time. They still want to annotate over the top of document with their own notes. Currently this is being done pen to paper. Technology changes are sometimes done in baby steps.

So if your IT department is like ours, you are not as staffed as you would like and the hours of the day are filled. So how do you add a new type of technology and still maintain a quality design? Replacing the thin clients at the users desktops with something they can pick up doesn't make sense. Our desktop cost is 600 dollars (400 thin client + 200 monitor) with a projected 10 year duty cycle. Annual costs are minimal. Laptop footprint devices (even thin client laptops) would then have lots of docking and undocking, snapping of wires, cords being moved, monitors plugged in, USB devices plugging and unplugging...all very expensive to support and problematic. Why change one of the most stable parts of your network? It was clear that our efficiency and cost savings should then be obtained from our printing infrastructure. We don't have desktop printers which are very expensive to maintain, but instead run departmental laser printers. Our printing costs and printed page counts are still too high in my view. There are people that print pages during document construction and then hand write notes on the pages; and some people print email messages in order to read them. Nearly every meeting involves pages being printed and handed out to employees. There has got to be better ways of working, and this for sure is the area that needs attention.

So what I am developing now with the IT Director is a way to integrate an iPad footprint device into our design as a way to reduce printing costs. The goal is to give them the power of the devices, but not allow them to store documents locally.

In prototype form I have accomplished the design below. I'm simulating this work flow on a thin client until NX 4.0 is relased. NX/Nomachine 4 will allow you to log into our servers with Safari directly from an iPad. The server will then detect your Xserver is 768x1024 and know that you are on a tablet and then bypass starting GNOME in favor of a UI designed just for this footprint. This will allow you to be logged into the server with a full GNOME desktop and also a tablet device at the same time. The UI opens in a split second, and will offer much faster response time than waiting for a desktop to start. The image below shows Xsession passing you to the two environments based on Xserver size criteria.

The workflow that we have worked out so far is: When users log into GNOME they always have the following icon on their desktop. This opens a Nautilus folder into which they can drop and drag PDF and OpenDocument files. Anything in this folder is staged then for the iPad devices.

The shot below shows the current UI that is displayed in the 768x1024 footprint, exactly what will be seen on iPads. The panel at the bottom appears and thumbnails are generated of everything in $HOME/MobileDocuments sorted in reverse order newest to oldest. Clicking once on the document will display information about the document on the status line. Pressing Open will open it in either Xournal or OpenOffice. Xournal will allow them to use a stylus pen and mark up the meeting notes in their own handwriting. These documents are always on the server, and when they return to their desks they are available for immediate reference and further editing.

The iPad also has Dragon Naturally Speaking, and I have created a small multi-line text widget that will accept a paste from the local device. Once pasted, the button "Save Dragon Document" will become active. When they click this button, the text will be converted into OpenDocument (with perl modules) and a unique date stamped file name auto generated. The document will then display on the left of the panel as a thumbnail and be available for editing.

To answer the questions that I think will be coming :) -- ** Yes other tablets will be reviewed and the market is always monitored. This design is elegant in that it does not lock you into a certain vendor. ** Yes the UI is very basic and will evolve and probably sux. I'm more focused right now on this work flow. ** Yes iPads are $500 a piece, but even buying 10-20 of these would be but a small dent in our printing budget. A 10-20% reduction in printing might pay for the hardware in the first year.

I'm looking forward to testing this workflow and it's proceeding as a back burner project along with my primary projects.

Portrait Feature Merged Into Thin Clients

2010-11-24T10:06:00.004-05:00

A conversation that has taken place over the last few months is the idea of adding portrait mode as a supported thin client workstation feature. This is being done for two reasons: 1) to promote electronic document construction and review for those users that do this often as part of their jobs. We still have a good number of employees that use paper as part of the construction process. They print the document, hand write their notes on the paper and then submit it back and someone re-types their changes. (sigh!). One bad thing we have done in the technology arena is widen our screens for beautiful landscape views....and yet continued to design documents in portrait mode for printing on paper. From this perspective we kind of have promoted not working electronically. Hopefully this change will reduce these older techniques. 2) With tablet devices getting more popular this footprint needs to be tested and accounted for in design. Note: I'll be blogging about an offshoot of issue number 2 in the next week or two.

Since I was updating the thin client operating system anyway to make picture management easier (see earlier blogs), I decided to use the short week to just get this feature deployed. After a few mods to our standard xorg.conf file, our ATI cards are working great with xrandr -o and rotating cleanly with no artifacts. Nice.

gdm on GNOME 2.30 plays nicely with rotated screens and I'm happy with the results. Authentication cleanly worked, GNOME started and avant is working well in this mode. In the shot below you can see 1050x1680 in all of its beauty. OpenOffice looks great and for the most part everything is working as anticipated.

One issue that is being discussed is whether or not we will allow users to flip their monitors day to day or if this will be a setting that is locked down. As anyone knows that supports lots of users: anything that moves or can change will have failures and increases support. Something as simple as a power cord popping out produces calls to our help desk.

I made a quick change to our login chooser (below), and will get get this out to a few beta testers in the coming weeks. Initially we will allow the users to flip the screen on their own, and see how it works.

In my next blog (probably week of December 6th), I'll write about some tablet/mobile ideas that are a part of this portrait project. We have some ideas on how to save money in this regard.

Another Week, More Progress

2010-11-19T09:48:00.007-05:00

The weeks go very quickly around here, and I have made lots of progress on my projects.

The Firefox 4 sever is basically all done and ready to deploy. There are some policy issues related to content blocking that are being resolved. I'm not really involved in that decision, and it's being handled at an Administrative level. Once that is settled, I'll create a script to backup all of the users $HOME/.mozilla directories and move them over. We have done this before and it's very simple and easy. 800 users moved to a new version of Firefox with just a tarball and script change. :)

I finished writing the first release of our USB stick user interface idea. The beauty of this design is that we will be able to give this to employees and they will be able to insert a USB stick with photos (or camera) and the photos will display and once clicked go into the clipboard. This will allow us to eliminate the need for many users to have full access to the sticks. Those employees that do need full access to read and write USB sticks will use Nautilus to transfer files, just as they did before.

Regardless of the megapixel size used on the camera, the UI converts them to 1024x768 before placing the pictures into the clipboard. Currently we have a lot of people that shoot 10 megapixel and then insert that photo into a document and resize it to 1 inch by 1 inch with the resize handles of OpenOffice and then wonder why their resulting PDFs are so big. For many documents and email messages around here, 1024 is plenty big. The shot below shows the thumbnail screen that comes up when you insert a stick. It loops through the files and summarizes the file types and then sorts the photos into reverse date order and displays thumbnails of the newest pictures. The black lines demonstrate the thumbnail being pressed, and then a control-v in OpenOffice and Evolution and the photo is inserted. I gave the UI a nice icon (green line) which then displays in the lower panel cleanly.

The functionality of the MIME bars is pretty well locked now and finalized so I have been adding the final changes to make them look nicer. I found a nice script that allows you to use convert to dynamically create a drop shadow for any thumbnail. So now all of the bars look like the shot below. I just need to give the UI an icon and do some final spacing changes...but I'm pleased with how they work.

One of my ongoing projects is to try and get as much speed as possible running Microsoft Windows applications over RDP. In the last few months we moved the RDP client off of the server and made it run locally on the thin client. We also have started to trickle in the new HP t5745 thin client to the users that use graphics heavy Windows applications because its Gig networking and faster CPU is yielding about a 20% speed increase. Running RPD locally on the thin client also got us about 20% increase in speed, nice.

One other technique that can be used is to use cache to increase performance. On a computer with a hard drive, this works well and does in fact make things run faster. However, the thin clients have solid state Apacer flash memory in place of a physical hard drive. The write time of these drives was not a fit for trying to use it to hold cache from running software. Also, whenever the thin clients would perform a disk flush (sync) the software would sometimes slow for a few seconds. What I did in the new release of our thin client OS is make a 50MB ram drive, and then mount it to $HOME/.rdesktop of the 'user' account. This will allow cache to be sitting in memory and should yield better performance. I'll be pushing this change to beta testers later today.

The shot below shows the RAM file system which is then mounted into the cache directory.

Got a nice comment concerning the Rdesktop fork called FreeRDP. I'll be doing QA work on that in the coming weeks and see how it works. Very nice to see patches being merged and new releases.

Happy Friday Updates

2010-11-12T07:51:00.010-05:00

I have been busy working on various projects and making progress. Our architecture group also had a meeting and have made some decisions about some open design goals.

Compiz No More

At this time we are going to not bring over Compiz to the new GNOME desktop that I am building. This technology will probably be tested in a back burner type project, but is not on the design goals that must be met before we go can go live. There are users that love Compiz and I too used it every day, but a few issues were taken into consideration. 1) In order to have a consistent thin client build on all HP models (5725, 5735, 5745) we upgraded to HPs Debian Lenny build. On our ATI cards, the 3D desktop is crashing the Xserver periodically. For some people it never happens, and for others it happens often. I haven't gotten any updates to my bug report and don't expect that anyone will fix this issue. When HP releases a Squeeze update, I'll review the updates and try again. 2) We wanted to have a consistent desktop now on all devices. If you enabled Compiz here at City Hall it worked great. If you went to a remote site on lower bandwidth the regular GNOME desktop appeared by design. Sometimes people had been on Compiz so long, they didn't remember how the other desktop works. 3) We are anticipating a future where people will be on more and more mobile devices and carrying devices with them; so a simple, fast and consistent interface is desired. You aren't going to be pinching your fingers and turning a cube on an iPad type device right? :)

On a technical note, I compiled Compiz 0.9.0 on the 64bit server and the speed was excellent. With some patches from Debian on the thin clients and more time I am confident it would have worked great. Sadly, time is my biggest enemy sometimes.

Firefox 4 Updates

We have about 15 people using Firefox 4 (64bit) fulltime and all plugins are working and the server looks great. Pages are loading faster than ever and scrolling is crisp and fast. I'm very pleased with the results. Having 48 CPUs kind of helps too. :) I expect this server to run for a good 4-5 years and it's the most cost effective way to deploy browser and video technology to our users and our thin clients. Adobe Acrobat, Flash and Firefox are touched once and the updates go to everyone instantly. This server should go into wider deployment in the coming weeks...almost done.

Desktop MIME UI Status

As I have been adding users to the beta desktop, I have been hearing consistently that they really like MIME UI bars that open when you double-click on files. It's as simple as it can be to move files around the network while sheltering them from the physical processes. I have said many times that a file picker that opens in any application is the kiss of death for productivity. Navigation even in 2010 is one of the hardest concepts for regular users to understand.

We have been working on promoting the usage of the insert link/file:// technique in Evolution to email a link of a document to other employees instead of doing a physical attach. This reduces the instances of duplicated documents and also helps with quota space in the post office. I experimentally added a feature to do this automatically from the GUI and it's working well. When they double-click on a PDF for instance, one of the options is [ Link To Evolution Message ] (below). This builds a small HTML file which is attached to a new email message and contains a bit of href "magic".

Below is the resulting Evolution message generated. This eliminates them having to use the Insert Link feature, and mess around with spelling things correctly and understanding where documents are stored. Feedback was positive on this feature. I think once we have more people using this feature, it will be used often.

The other feature that I added was a checkbox to help resolve one of our improvement areas: users don't understand how big the pictures are that are coming from their cameras. Shooting 10 megapixel is great, right? :)

When [X] Email Friendly Size is pressed the picture is reduced to 800x600 and compressed into JPG.

USB Sticks, Viruses And Functionality

Those users with permissions are able to gain access to USB sticks on our current deskop and it's working well. A ftp daemon is running on the thin client and allows the GNOME server to gain access to the device. FTP is jailed into /media and also certain types of files are disallowed from view. Pictures and OpenDocument files can be moved to/from the sticks. Microsoft Office and PDFs can be moved TO the sticks, but not FROM the sticks. This is because of their virus risk, and also the fact that this work flow is not in our business practice. Documents should not be constructed off network and especially with non-City standard software packages. All of our software writes documents into open document format. If documents are to be received from the outside, they should be emailed whereby they are fully checked for viruses. It also is not a good practice to use USB sticks to house data. The risk of data loss (loss, theft, physical failure) is very high.

A review of the logs indicates that primarily users are using USB sticks to upload pictures, and very often just a few pictures at a time for inclusion into documents and presentations. I have designed a mockup of a dialog that will come up when a USB stick is inserted into the thin client. For many users, this will provide a simple UI to handle their needs with no file managers. The UI will build thumbnails of the last 32 photos taken along with a summary of files contained on the stick. If the user clicks on the image it puts it into their copy buffer at which time it can be pasted into any other application. This also will allow me to shrink the image to 1024x768 prior to putting it into the clipboard and reduce the huge pictures that are being taken on the cameras. The UI also gives them the option of emailing PDF and MS Office documents to their email account.

Of course the users will continue to have access to the full file manager to manually drop and drag files as they did previously. But at least now a simple interface which covers 80% of our needs will open automatically when usb insertion is detected.

Coming up for me next week:
- Begin moving services live on the new Firefox 4 server.
- Continue testing the new GNOME server, and installing updates and patches.
- Await the next avant-window-navigator tarball to fix a few issues.
- Experiment with making the USB process easier
- Begin testing some ideas to speed up rdesktop/xfreerdp with the use of a RAM file system on the thin clients.
- Being engineering the replacement for the server that holds the bulk of our documents and pictures. Use beagle again or try and install Zeitgeist?

Desktop Progress Continues

2010-11-04T15:20:00.002-04:00

Problems can be solved, they just need time. :)

I had been kind of putting off a few technical issues while I explored options to see if better solutions existed. Our online friend "Pepp" has been poking at getting Helix/Real Player to compile natively in 64bit. The 32bit version runs, but the problem is that I was having problems getting padsp working to redirect the sound to PULSE. This is because you cannot wrap a 32bit application with 64 bit pulse libraries (padsp). I used cpio to extract the 32bit version of libpulsedsp.so from the i586 RPMs and installed it into /usr/lib and created a padsp & padsp.64 to run both types of applications. Everything is working as it should now. Circled in purple below is real player running seamlessly from 64 bit Firefox. CPU usage is excellent, and sound and video are synced well. Seems like it's going to work just fine. I still would love a native port however! :)

I also was having some problems with the mail-notification popup applet, it apparently is still expecting the icons to be in the layout of older versions of GNOME and certain pieces of art were not displaying (stock_unknown and stock_mail and others). I went into the source code and forced it to read these icons from /usr/share/mail-notification/ as a quick hack and now everything is working. I also wrote a few lines of code to always force working settings into the mailboxes.xml file which houses the email account information to ensure it's working each time users log into the server. The ability to get popups when email arrives has proven very popular with the end users and it was important to bring this over to the new server.

The last major piece of testing is with our Microsoft Windows applications. I mentioned in an earlier blog that on the production desktop we were using three different connection methods to launch Windows, and we are standardizing them all now with RDP/Rdesktop. So some QA and testing days ahead for me.

Weather Applet Re-Wrapped

2010-10-28T10:43:00.003-04:00

For a number of years we have used a global weather applet which has become very popular. The reason why this application is worth the time to develop is because otherwise you have hundreds of people during the day navigating around the Internet to look for information; this is especially true during inclement weather. Instead just one daemon runs and downloads all of this information ONCE and saves it to a shared directory. Then when users activate this applet, they load in these pictures and get current information. This greatly reduces Internet usage and helps our bandwidth.

I re-wrapped this application with greater flexibility and made it consistent with the new desktop helper MIME bars that I wrote. One of the biggest problems that users have is getting information around the network, "I'm looking at this image, but I have no idea how to get it to email or to print it". So there are now options to take the currently displayed image and save it, copy it to the clipboard for later paste, print it or email it directly from the UI.

I just pushed this live and bound it to the Super-F9 key, let's see what my early beta testers say. :)

Up next: Cleaning up other dialogs and screens from the older desktop, installing patches and poking for bugs and installing more icons with links to our internal software. Still on target for wider beta deployment in November.

Desktop Cleanups Continue

2010-10-25T14:27:00.003-04:00

Still working on prepping GNOME 2.30 to replace our very aged GNOME 2.16 server and also on getting Firefox 4 ready. I have been testing different cache lockdowns in gnome-mplayer to find a good middle ground between the clip starting and having enough cache to play without interruptions. Obviously this depends greatly on the remote server, but testing it to find a good sane setting that will work most of the time. I don't want users tinkering with these settings; because they will randomly increase and decrease values and then wonder why things are broken. :)

In testing the workflow of the new Picture Options bar that comes up when you double-click on a picture, I decided to add two very basic features. EOG doesn't have a way to put the image into the clipboard, and it's a shame to have to wait for GIMP to start in order to perform this step. So I added a feature to send the image to the clipboard. It's very common for users to want to rotate pictures they have taken, and having to start EOG to perform this step was clunky. Since EOG doesn't copy to the clipboard, after rotating the photo they then have to save the photo and then double-click on it again. A simple 90 degree clock-wise rotation was added and works great. (seen in screenshot below). The python Image library is awesome and easy to manipulate photos. Did I mention something about clipboards twice? :)

With some help from the avant guys, I was able to get software to better sit under their corresponding launcher icon. This combines the launcher icon with the window list functionality nicely and provides more space on the panel. Next up is pondering the fact that all rdesktop instances come back with a WM_CLASS of "rdesktop". (current panel seen in shot below).

At this point the functionality is mostly in place and it's time to bring over the rest of the software packages and create artwork and icons.

I had to do something clunky to application-browser to get around a bug. In previous versions of GNOME if the user selected this software process again, it would connect to the already running process and reuse it. In the current release there is a bug and it keeps adding another instance to the process list. If a user clicks on this menu system 20 times during the day, they would have 20 processes. :| So the code below resolved that issue. Hopefully a patch is forthcoming; this is some serious nastiness.

Current desktop screenshot:

Panel, Browser Stress Test & Thanks

2010-10-19T17:13:00.003-04:00

With some tinkering of lockdowns and gconf settings, I have gotten avant-window-navigator working well enough to push to some early beta testers. Very nice applets and a clean modern look. Let's see what kind of feedback I get back tomorrow. :)

Thanks to Kevin for your help with understanding that xrdb is a g-s-d plugin. Testing Motif applications right now to see how the themes are working.

So with that in place I'm giving mplayer, Firefox/Flash and Pulse Audio a stress test to try and get them to crash. So far sound is strong even running remotely over the network, frame rates are excellent. I think the user community will be pleased with these upgrades.

64 Bit Firefox + GNOME 2.30 Continues

2010-10-18T15:19:00.009-04:00

Happy Monday,

Work continues on the new servers and things are making nice progress. I brought down the latest release of avant-window-navigator along with Extras to see how the project has advanced in the last few years. One of the cool things is that it now works on a regular 2D desktop. I'm testing the bare minimum of applets we need to go live and will push it to some beta testers later this week. One possibility is using it as replacement for the regular gnome-panel. I'll see what kind of feedback that I get in the coming days from the 'regular users'. (screenshot follows)

Sadly one possibility is that we won't be bringing the 3D (Compiz) configurations with us to the new server. 3D has helped users conceptually understand how to use multiple desktops to avoid opening and closing windows all the time. However my initial tests indicate that it's crashing the Xserver on the thin clients a great deal, and being that I have gotten no feedback whatsoever on my Debian bug report...I'm not sure that anything is coming in that regards. Trying to upgrade Mesa and all of its hooks into X doesn't seem pleasant. I'll continue to tinker and see if I have some ideas in this area. This is not a high priority feature; users can certainly work fine with the 2D desktop.

It seems like someone pulled out the code in GNOME that set traditional X resources to match your theme. I'm sure the thinking was that very few applications were still running in this regard, but actually we still use them heavily. The Citrix manager uses Motif, and our in-house applications still run this way too. I hacked a quick ksh script to try and read your current theme and get the *background and *foreground colors; but it's not pretty and the themes don't store these colors consistently. If someone wants to hack a little C app to dump the colors of the current theme to standard out, that would really help a great deal.

One idea that came up was to use the five seconds during login to display current calendar information about City events. So I built a quick mockup that displays during authentication. In theory we could cron the City global calendars to dump an image once a day with updated information and lay it over the top of this wallpaper. (shot below)

I finished the MIME helper applications on the GNOME desktop and have copied them over to the new Internet server and hooked them into Firefox. So the users get a consistent interface when interacting with content on the Internet. This helps them quickly and easily print, save and view such content. (shot below).

Since it seems like 32bit Acrobat Reader isn't going to wrap inside 64bit Firefox well, I have once again pulled it out and made it a helper application again. I have been testing those (nasty!) 3D PDF animations to ensure that a few of them being opened concurrently isn't going to clobber our server. Thankfully Reader doesn't seem to be using multiple-CPUs at once. In the shot below, the video is playing (propellers are spinning) and it's only using one of the cores.

Lots of progress and very happy with the results. A few more pieces being finished and I can widen the beta program and solicit more feedback.

Dear Adobe

2010-10-06T16:13:00.004-04:00

Dear Adobe:

Thank you for the 64bit beta of the Flash player, it's starting to work better. Thank you too for the just released 9.40 version of Reader. But unfortunately it didn't put a smile on my face; the lack of 64bit support for Reader is absolutely making server upgrades miserable. As you can see from the message below, there is no ETA on making 32bit Reader work on 64bit Firefox 4.0. The Mozilla developers don't have the resources to debug it, nor should they have to. Your help in this matter is appreciated.

Disclaimer: Yup, I know about the open source readers...they aren't robust enough to use at this time.

MIME Update, Work Continues

2010-10-04T15:54:00.002-04:00

I got a quiet day in today (no meetings!) to work on the new desktop design. I feel like the last major pieces are falling into place and once a good work flow is finalized, I can install the rest of the icons and increase the number of beta testers.

I'm testing some new ideas on the MIME bars that launch when you double-click on files. I wanted to try a "one stop shop" type approach. Everything you need to perform your task is on one screen. No further dialogs or hidden screens. I added a gtkcombobox with system printers. Once a printer is selected it 'sticks' each time a new document is double-clicked. This lets the user clearly see which printer will get their documents. With nearly 60 printers, people send things to the wrong printers *all* the time. I also have added a gtkentry widget to house a quick technique to send the file directly to the email post office. Very often we have users that have to email files to the same people over and over again. One click, and it's delivered; without having to wait for the Evolution composer and enter all of the details. I also connected the areas at the bottom that indicate the size of the file as a visual cue of file sizes.

I'm testing this dialog right now, and hope to have the rest (Pictures, Documents, Video) all finished by tomorrow.

Lockdowns, Flash & Text Files

2010-09-29T14:40:00.005-04:00

Work continues on getting GNOME 2.30 ready for our employees and making great progress. One of the benefits of this project is that we will be able to save some money and resources by reducing our server count. Right now we have two identical GNOME servers and users log into one or the other. Both servers can run our highest concurrent load (350 users), and are in place so that if we have a failure the other server is always ready to go. We have to have a backup of some type -- You can't have hundreds of people sitting around because of hardware problems. But the fact is that in all of these years, the hardware hasn't had a single failure.

Our goal on the next upgrade is to only have one server on a nice multi-core, multi-CPU server with 64bit Linux. We then will create a physical-to-virtual copy once a week and have it ready to go. If the physical machine fails, we will put the users on the VM server until the problem can be resolved. As I an designing the desktop, I'm going to write code so that if it's running on VMware certain features will disable so as to provide the best possible speed. An example will be compiz, when running virtually, that feature will disable and drop back into 2D. Another example might be to turn off thumbnailing and so on. Hopefully the VM server will never need to be used, but we will be ready and save a good amount of money by reducing one physical server replacement every 3-5 years.

Lockdowns

Very often you find a great piece of open source software to use, but it allows the users to tinker too much and change settings. Often these settings will cause the software to fail or work in a manner that we don't want. Such was the case with using gnubiff. It works great, but one bad click while tinkering and it will no longer connect to the post office. Or another issue is where users think it's wonderful to check for new email every 1 second; which does wonderful to your post office. :) So what I do is get everything working as expected and then create a template and replace the things that are specific to users with variables. In this case their account name and password are the only two things that need to be unique. The shot below shows how I then implement this design. Grab from their current .gnubiffrc the stuff I need, and then use sed and merge it into my template and give them a fresh working configuration file before the software launches.

Flash & Firefox
There are days when you feel that all of your bug reports and efforts will never advance and then suddenly you start getting updates that actually work. Such was the case yesterday when I installed Firefox beta 6 running 64bit with the just released preview 2 of 64bit Flash. Preview 1 and the earlier 10.1 64bit beta were really pretty bad on our configuration. Frames and sound were choppy, playback would block the browser from scrolling. I was actually getting better results with a wrapped 32bit Flash release, but that's so clunky to me and I'm trying to run 100% 64bit. I'm happy to report that preview 2 is working MUCH better now. Sound is strong and frames are better, even with remote display and remote sound. This gave me a chance to test the volume control. It's working great...it detects the sound card on the remote thin client from the GNOME server and has no idea it's running over the network. Excellent.

In the shot below, you can see the video playing along with a zoom of the volume slider. SiS is the sound card on the thin clients.

Text Files
One of the things that users loved on the last desktop release was the use of thumbnails for all of their documents. It really does aid in finding what you are seeking. I have an eye for consistency, and always hated that text files (.txt, .ics and .csv) have never really been appealing in Nautilus. So it turns out that 'convert' very nicely can change a text file into a picture. I created a new schema for text/plain and pointed it to 'convert' and thought it would work. It turns out that Nautilus sends %o without a PNG file extension. So convert has no clue what file type to convert into, and it didn't work. So I whipped up a quick script called text-thumbnailer. This script also takes care of the issue that if the file does not end with .txt, convert doesn't understand the file type. The script softlinks a .txt file in /tmp and generates a PNG and then passes it back to Nautilus as expected.

In the shot below, you can see the thumbnail schema, the text-thumbnail script, and a Nautilus window which contains the resulting thumbs. Scrolling through the folders now is much cleaner and consistent.

So along with various other projects, this development will continue. I'm getting positive feedback on the results and am looking forward to the coming weeks and ongoing improvements.

Beta Testers On GNOME 2.30, Making Progress

2010-09-24T13:25:00.005-04:00

Lots of things keeping me busy, but making excellent progress on building a new server to run 64bit OpenSuse 11.3 and GNOME 2.30. Now that gdm is working with XDMCP, I was able to get the logins working. Halfline (thanks!) once again helped me with turning off the animations that were displaying when the authentication screen appears and also then as it goes away and the users wallpaper appears. This type of animation is too slow over remote display, and caused you to have to wait while the screen blinked multiple times. I'm sure it looks great on a local video card, but not suitable for here and not needed in a business/work environment.

Very nice to see enable_esd no longer required. PulseAudio is hooked right into GNOME and once I selected a sound theme with a login clip, it worked immediately. The pulseaudio volume slider bar is working great, even though the sound daemon is running remotely on the thin client. Excellent.

I was able to get beta testers logged in Wednesday and they are using this server fulltime. It's obviously not fully functional, but enough icons and applications are working to do their work.

Previously we were using 3 different techniques to connect to the various Microsoft applications around the City. Some where using Citrix, some were using RDP/Rdesktop and others were using an old product (UIS, Unix Integration Services) that has been discontinued for years. The other System Administrator has been working magic and working on standardizing everything around RDP. This makes it easier for him...just one connection package. It also allows us to fully deploy the new connection method that I had blogged about previously. This has some disadvantages, but is MUCH faster and provides much better sound.

The old technique had rdesktop running on the GNOME server itself. Connection would then be made to Windows. Windows would communicate with rdesktop via RDP, and then we used X11 and Pulse to provide the screen and sound. 100% centralized, stable, easy to upgrade and make changes (blue boxes below). The way we are deploying on the new server, the GNOME server signals the thin client to start a rdesktop session and that session then talks to Windows with RDP. The user presentation is probably at least twice was fast (green boxes below).

So as he continues to get applications ready for RDP, I'll add more and more icons and get them to the beta testers. This is providing the same integrated Linux/Windows desktop we had previously, but with lowered network usage and far better user experience.

Always have to give a screenshot, right? So here is a shot of the current status. MIME handlers are working great for the various file types. Application-browser is starting to fill in nicely with our various City applications. And I launched Internet Explorer to demonstrate how cleanly Windows apps integrate and work right off the standard GNOME icons.

Huh? You are probably wondering why we run Internet Explorer. 99% of all Internet usage is done on Firefox/Linux. But many other Governmental agencies (state and federal mostly) designed pages that are hard coded to only work with Internet Explorer. Our tax dollars at work. :)

Thanks For Ideas Guys

2010-09-16T10:53:00.003-04:00

Yesterday when I posted the MIME bar prototype, I was given some ideas to make the dialog easier to use and fit better into current UI designs. I was mostly thinking about the desktop flow, but UI ideas are always welcomed. I whipped up a new version in Glade and connected the signals to the existing code and it's already working. It's not something that I could release because it's hard coded for our software packages, but might be a good idea for a GNOME project in the future. I guess in theory some more X-Categories could be added so it knows where to place these items. I'll be creating similar ones now for word processing files and for Pictures.

Shot below, I like that it has a similar appearance to the main-menu.

Update: A quick dupe of the screen and change of the thumbnails and the Picture and Audio/Video dialogs are starting to work. Thanks again Kevin for the awesome 'nailer' program for the videos.

MIME Ideas Continue, Fulltime GNOME 2.30

2010-09-15T13:59:00.002-04:00

Many things are happening at once, but I have made some changes to the new MIME bars to test ideas with our users. I'm now logging on GNOME 2.30 (OpenSuse 11.3) fulltime from a thin client. I have gotten all basic applications (Evolution + Firefox + OpenOffice) running. I can start the other applications manually easily. After I installed another group of icons for our beta testers, I'll start getting feedback.

I have always wished that Nautilus gave you a hover mouse page preview larger than the thumbnails that currently appear. If you have hundreds of thousands or millions of documents many of them are very similar, especially those on letterhead. So experimentally I have created a tabbed dialog that appears when you double-click on a file. The "Options" tab allows you to select your next step. (shot below):

The Details tab shows you a big preview of the first page of the document, along with the owner of the document and the file size. Very few people discovered the "properties" found in the right-mouse of nautilus. They call all the time to find out the owner of files; so hopefully this will help. And believe it or not, but a lot of people don't understand K, MB and GB acronyms; so I'm testing the idea of displaying the file size fully expanded with lots of zeros. :)

I'm looking forward to getting feedback, and think these minor tweaks of the UI will make the system easier than ever for our employees.

Testing GNOME 2.30 & Updates

2010-09-14T14:12:00.004-04:00

One of the benefits of starting to poke at a desktop upgrade to GNOME 2.30 is being able to take what I learned the last time and making improvements. Certain types of specialized launch scripts that were developed are being merged into a core library which will run when any icons are triggered. One such example is the gnome-panel, which still apparently treats a double-click as two single-clicks. I thought this was changed at some point, but it's still working in this manner. Users tend to double-click (or more!) on icons and are launching multiple occurrences of the same application. So I brought over and improved the code from the old desktop. Gnome-panel now detects that someone is repeatedly clicking and disregards them for a few seconds. The popup below displays:

The MIME bars that were deployed on the previous desktop release have worked well. When users double-click on files in the file manager, it asks them what step they want to perform. The previous bars had buttons that were stacked left to right, and on certain monitor resolutions we were running out of real estate. I also wanted to better categorize them into middle level functions (Edit/View, Save, Print & Email). The screen below is my first attempt and I'm installing the MIME types this very second.

This design might seem odd at first glance, but think of this situation: You get a PDF file and you know you aren't going to read it online and for sure are going to print. With a single click, it's queued and already finished. Users like having steps 1-2-3 that are consistent and simple.

I hope to have some early beta testers on this server by Thursday or Friday of this week.

Still poking at the slow printer dialog on SLED 11/Evolution to figure out what's happening. My attempts to hack out certain polling areas of the code were not successful. I'm also looking at f-spot for use in a multi-user environment, working on making it easier to pass files back and forth to USB sticks, beta testing OpenOffice 3.3 and checking into connectivity between OpenOffice and the Oracle database.

Gnome 2.30 & main-menu

2010-09-09T16:46:00.002-04:00

It's good to see GNOME 2.30 running on our thin clients (below). My thanks to those that helped me get this working and squash a few bugs. I have been removing all of the software that is automatically installed on OpenSuse and won't be used by our users. Next up will be connecting our core group of applications (Firefox, Evolution, OpenOffice) so that I can get some beta testers logged in. I also have been pondering some updated MIME bars based on feedback and things we learned from the last upgrade. These bars will launch when you double-click on a file in the file manager and ask you what task you want performed. Everyone has gotten used to them and people really like the desktop to flow in this manner. Sure would be nice to be able to put these into Nautilus as a single click.

One very interesting usability issue that I found when reviewing how we currently have the main-menu grouped is that most of the regular users never figured out that they can type software applications into the Filter....even though the cursor focus is there when it opens and it's blinking waiting for you to type. :| The users also didn't really figure out that you can click on the Groups on the left edge to reduce the number of packages displayed. Instead, most of them are scrolling down the entire list of software and looking for what they want. I really need to ponder this more, but sure sounds like an improvement area in the design.

I'm also excited to see NX 4.0 is coming out soon, and will allow you to log into our desktop servers from an iPad. How cool is that?

Project Updates

2010-09-07T08:53:00.005-04:00

Just back to the office after a long weekend and I thought that I would create an update on the various projects.

Recreation System
The developer and business analyst are making progress on new recreation system. Now that IT got a signoff on the receipts, the final design has been implemented. A screen was designed to allow them to add coupons on the far right side of the printout. Following our internal goal of never having users obtain a file manager, this step is performed inside the GUI itself. All available coupons display and the user just has to click on it and hit the arrow which loads it into the printout page. They are beginning the process now of implementing more robust accounting principles at this time.

(shot below, sorry Steve Perry isn't coming to Largo) :)

Evolution Print Dialog
We have had some problems with libgnomeprint which will require me to change the source. We have about 60 printers, all HP using JetDirect. They are always in various states of function; paper jams, paper outages, toner shortages, waiting on paper tray, etc. Some of them are turned off, and some of them have latency from the Internet. When you hit the print button in Evolution the dialog will very frequently gray out and then block Evolution which grays out too. After a few seconds the dialog returns and everything works. Jeff Steadfast happened to be on IRC and gave me a few areas to try to improve this situation. I'm going to comment out the call for it to poll the printers. We don't really care about their statuses and users will just send the print jobs and await their completion. Sounds like a nice gconf key idea to me for future releases.

Debian Crashes
Does anyone want to backport some packages for me? This bug is really causing us some grief. It's causing our thin clients to crash for 3D/Compiz users and some people have gone back to 2D because they are annoyed by losing the Xserver. The last instruction was to upgrade Mesa, which will require an extensive upgrade of all of the X packages too which would take a lot of QA work. If anyone has any ideas, feel free to hang them on the bug report.

Halfline XDMCP Hero
XDMCP really hasn't worked for a while now in the latest OpenSuse releases and Halfline cooked a patch which I was able to get compiled and it's working! It's been submitted to Suse/Novell and Vincent already classified the bug report (thanks!). The production "Desktop" server is running Gnome 2.16 so there are many new things for me to test over remote display since that release. I also see the wallpapers now transition/animate with a .xml file and I need to check performance over the network. I built Compiz 0.9.0 and it's crashing the Xserver a good amount of the time (See Debian Crashes) above. There is a lot to review before I can get beta testers on here; but it's very encouraging.

Yup, the final login screen will be a bit more professional. When I am hunting down configuration files, I always change them so radically to leave no doubt that I found the right settings.

Photo Taking Hardware

2010-08-19T12:09:00.002-04:00

In a previous blog I had described the security camera that we selected for taking pictures of our citizens for recreation cards. We found a low sitting tripod for around $15, which pivots the camera at a 90 degree angle. The pictures are taken sideways and then corrected via the 'convert' command line utility in scripts to rotate back 90 degrees and appear correct on the IDs. These cameras are completely self reliant and don't need any attached hardware or PCs to run them. Total cost including tripod + camera should come in less than $150.

Recreation System, Milestone 1

2010-08-19T09:57:00.004-04:00

Our IT/Integration division was able to go live with the first milestone of changes to the recreation system on a new server. I haven't been working on the coding on this project, but have been helping with some technical aspects including cameras, mockups, printing and ideas.

The old recreation system used older Motif widgets and was first built in the mid-1990s. We already had thin clients at that point. In the mid 90s the remote sites were in 8 bit color, 800x600 and only had limited bandwidth. The screens were built with those limitations in mind.
(shot below)

For the first milestone the front page was changed to a 'portal' which displays a lot more information. Motif was upgrade to 2.3 which gave us anti-aliased font. The theming system was upgraded as best as possible to allow GNOME colors to alter the look and feel of the software and match the rest of the applications. All of the code from the old system came over without modification; the screens just had to be touched in order to accommodate the new font metrics.

(shot below, no GUI nazis please :) )

The developers are now starting to alter the code of the screens and add functionality. The biggest new features being implemented are membership cards with photos and bar codes, and better accounting integration. If you are a citizen of Largo, these milestones will slowly increase the quality of your experience at our recreation sites. Be assured that all of this technology is being developed with costs and long term impact in mind.

Java & Sun/Oracle Blaze The Way :)

2010-08-03T14:50:00.004-04:00

I never thought that I would such high marks to anything related to Java, but as it stands right now this is the only piece of the puzzle working natively and well on 64bit Firefox 4.0. I wanted to draw myself a picture of the pieces (below) so that I could work out in my head how this will all work. There are 4 main areas where I need to run embedded plugins based on our users requirements. 4 of the 5 plugins have caveats and are not deployed in the clean manner that I would like. Some of them are still only 32 bit, some of them don't natively use PULSE and some of them are missing features when built natively 64bit.

Green indicates satisfactory techniques, red indicates problems.

I guess for now we tell our parents and grandma to keep installing the 32bit flavors.

64bit Browser Work Continues

2010-08-02T09:19:00.002-04:00

I was pulled off of doing Admin work for a few days to help with some technical projects and assisting in getting some pieces together for our Integration Division. I had previously noted that a system was developed for taking citizens pictures for our recreation cards. I wrote a small ksh and perl script to create the new cards. (shot below). Integration staff will make final changes and add some colors and move the boxes around slightly to fit the perforated paper; but it's all essentially working. The Panther development toolkit we use has no runtime licenses at all on Linux. So that along with the inexpensive cameras and using perl to print the cards has kept the costs to a minimum.

So now I'm back working on the 64bit 'browser' server which will run our Citywide Firefox sessions. I got mplayer compiled natively and it's playing all of our media files correctly. Nice. Now I'll work on getting the multimedia plugins loaded into Firefox. We use the RealPlayer (Helix) to play our commission meetings, but sadly this too seems to be missing 64bit support. Will check for source code, maybe I can build it myself. I guess I have come to the realization that in the short term, I certainly won't have a nice clean 64bit implementation; I'll load as much as I can and upgrade as more packages are released in the future. It's all super fast, the users won't see much of a difference.

I tried loading Chrome as downloaded from Google and there were some dependency problems. I found a release from OpenSuse (11.3) and was able to get it installed natively. What is interesting is that I have found it to be very SLOOOOW. Firefox is very responsive on this server; cold starts in about 2-3 seconds and pages load quickly and spinning the mouse to scroll a page is crisp and works as expected...even over remote display. Chrome feels like page rendering is fighting mouse movement, and spinning the mouse wheel is slow and not very smooth. When you grab the scroll bars, they seem to be jerky and slow. Possibly no one is testing Chrome over remote display, but as it stands right now it's not even close to be usable in our environment.

Work will continue over the next few days with plugin testing and loading more software. I should be able to put on some beta testers in the coming week or so and begin testing printing and printers.

Friday Afternoon Adobe Rant

2010-07-23T16:57:00.002-04:00

I have to join the voices today that are complaining about Adobe. I'm building a multi-year 64 bit server to run our Firefox sessions for hundreds of people. It should take me 5 minutes to install the PDF and Flash plugins. But guess what, it's 2010 and they still don't support 64bit platforms. So now I have to try and wrap them to run in 32bit mode which is uncovering bugs when used with Firefox 4. The beta 64bit Flash player seemed to show some promise, but I can't deploy it because it's too old in terms of security exploits.

So Adobe, you have a big market share with Internet content. You need to step up to the plate and make this content easier to deliver. If you can't, the world is going to move past you quickly.

Note: Yup I know about the open source projects to replace these products. They aren't a fit for us at this time.

Accessing Remote Devices From Centralized Hosts

2010-07-22T12:33:00.007-04:00

What is interesting to me whenever we look at software and hardware to solve problems is the needless level of complexity and cost. Let me explain. When you meet with vendors, their first inclination is to add lots of hardware to your infrastructure....and other Cities must just buy lots of it because they seem shocked that we question their solutions and want to keep it simple.

In this particular case, we are adding the ability for our Recreation department to put photos of citizens on their recreation cards. When you review cameras, you begin to hear, "Well you will just add a PC at each remote site to connect to the camera and process the photos". What?!? Huh?? It's very clear that this is unneeded and adds support and upgrade costs. I think sometimes these vendors fail to understand that if you deployed in their design, you would have 100 extra PCs on your network in no time.

So how do we fit in something like a remote camera into a thin client environment? I'm a strong proponent of centralized software and only having very minimal hardware in the field. I didn't want to get a usb webcam that needed drivers and software on the thin clients. Too many problems, and keeping up with ever changing hardware is very difficult. Very often we buy a few devices, and within a few months that same model isn't available anymore. Touching and QAing thin clients and updating them with drivers is not a good idea. I also didn't want the hosts to communicate via USB to remote sites. We always use stateless connections to keep things simple and stable.

So I honed in on the Cisco/Linksys WVC80N Internet camera. Ok, so some problems were solved. We can use the motion sensors, and our employees at the remote sites won't have to push any buttons. All they have to do is sit down the citizen and the photos are taken automatically. Very cool. The camera snaps the photos on motion, and then uploads them via FTP. Because some sites are on lower bandwidth lines, I decided not to send the photos back to our servers. Otherwise anytime there was movement photos would be uploading constantly and possibly slowing their performance. Ever see how many kids run around at recreation sites? :) I wanted to only send the photos when they are really needed by our host software.

It was a logical progression of thought then to house the photos at the remote site until they are used. But where? It turned out that we could store them on the thin clients very easily. We already use FTP to transfer files back and forth from USB sticks to the server to ensure that all file transfers and connections are stateless. So what I did was create a "fake" USB stick in /media that is really a RAMfs (tmpfs) with 20MB of space. This was to avoid having to have a USB stick plugged into the thin client to house the pictures. It was not desirable to store the photos on the flash drive of the thin client itself, these drives have a fixed amount of writes.

When someone moves in front of the camera, it shoots the pictures and puts them on the thin client located at that site via FTP. When we then need those photos from our host based software, we FTP then off the thin clients and back to the server. Clean, solid state and hardware independent. When this camera is discontinued, all we have to do is buy the newer model and drop it into the same location.

Total cost to take photos of our citizens, $110.

Projects Keeping Me Busy

2010-07-15T10:01:00.003-04:00

Life at Largo never slows down. When you push new technology live, there is always more waiting in the wings. Here is what is going on:

Evolution 64 bit: As I have reported everyone has been moved to release 2.28 on SLED 11 SP1. Still blazingly fast even with hundreds of people working. I moved the bug-buddy binary out of the way and inserted a custom script that dumps backtraces to flat files and into a common directory. I can see how many people, what people and why people are crashing in real time. Looks like we are hovering around 20 crashes a day that dump BTs and a few more that crash at startup. Around 400 unique people check email a day, so most people are having a stable experience. BTs are being shipped to Novell and we hope to start getting some patches soon. Those patches should also land upstream and help everyone else too.

OpenOffice 64 bit: 3.2.1 is working well and stable. It's still running so fast that even under a user load you don't see the splash screen at all. On a multi-user system you never really have a "cold start" but the UI opens in about 2-3 seconds at which point you can immediately start typing. Nice. We have identified that extensions are causing the memory leak, and hopefully Sun/Oracle will be able to work on it soon. In the meantime we just bought more memory (cheap) to keep everyone out of the swap device.

Firefox 64bit: Pavel Janik finished testing our new 48 processor (8 cpu x 6 core) server to try and set some new OpenOffice compilation records. I won't spoil the surprise and you can read his blog when it comes out. I'm downloading OpenSuse 11.3 right now and we will move that server back inside the DMZ and prepare it for our employees. I'm going to begin testing 64bit Firefox 4 and see how it works. Adobe seems to have pulled back 64bit flash, so I'll be experimenting with the various plugins and see how they work. I'll check on the state of the various media players and see which one works. We used mplayer last time and it works great, but I always keep an eye on other solutions.

Debian Lenny Xserver Crash On Thin Clients: I finally packed up a bug report on the crashing Xserver issue. It only happens about every 2-3 weeks, with compiz enabled and happens when you close a child window. Bug report is here

Verizon USB 760 EVDO Modems + Laptop Thin Client: Verizon released a nasty modem, the USB 760. They thought it would be a good idea to put software and a modem into one USB stick, nice. Linux sees it as /dev/sr0 and tries to automount what it thinks is a CD/DVD drive. The workarounds are not pleasant, you have to tell udev to then immediately eject the drive and then attempt to fire up your dialer. I'm poking around at some ideas to make it not so clunky. Note to Verizon: Please don't make hardware devices like this anymore!

GDM + XDMCP: Halfline and I are trying to get synced and have some time to experiment with newer versions of GDM and find out why it won't let thin clients connect with XDMCP anymore. Hopefully that will happen in the coming days.

Update: 2 second cold starts on Firefox on the 48way server. Obviously no user load, but promising. :)

A Word About My Book

2010-07-07T12:11:00.002-04:00

A few years ago I was asked to write a book and some of you have purchased it. Obviously one hopes that a book always meets the expectations of the reader, but there have been some negative reviews. Some of the goals while it was being written were not entirely matched to what people thought they were buying. Let me explain, and tell everyone what is in the book:

Obsolete Technology: When I was changing cubicles a few weeks ago all of the contents had to be packed and moved. I was shocked at the high number of books that were completely obsolete. I had a RedHat 6 book from 1999 and on inspection you find that nearly every technology has completely changed in 10 years. I wanted to create something that had a longer shelf life. In the time since the book was published, HP has already released two new thin client models. Had the book dug deeply into the internals of the model available at that time, it would have been just another technical book sitting around. So I wanted to include some topics that never change.

People Issues: The book deals a great bit about *people*. We as techies sometimes don't like to think about them or talk to them :) , but they are the biggest issue that you have to overcome when you move to thin clients and even more so when you move to Linux. Yeah, it saves lots of money, for sure...but if your administrative people aren't on board any project such as this is doomed. The book also deals with the fact that IT staff will have greatly changed duties post migration. You no longer have lots of people running around keeping client devices running and focus moves to R&D and centralized support. Instead of maintaining, you are improving and that sometimes requires different types of people.

Pictures And Pictures: All of you would be surprised at the number of people that think Linux is "character" and "command line". Many people think that if you move your email to Linux that you have an xterm window open with pine or something. During the editing process they asked me about including the screen shots, and I felt strongly that we needed to SHOW people applications like Firefox, Evolution, OpenOffice and Realplayer running graphically and similarly to how they look and work on Microsoft Windows. People also have this impression that if you have to get to Microsoft Windows applications from the Linux desktop that you have to log out in some manner and then connect to the various software packages. I felt it important to show that everything runs in a consolidated desktop, Windows and Linux apps right fine side by side.

Unique Site Requirements: Every deployment will be completely different, so it would be impossible to give exact instruction on how to setup and configure everything. So the book had to be about 'ideas'. I tried to describe exactly what was possible, and then leave it the reader to come up with a plan. I had no way of knowing the skills of the staff, the skills of the users, the money available and exactly what operating systems would be used. I also felt that more specific details would be available on my blog. I always try and blog about major milestones that we reach and hope that it's helpful.

There are good days and bad days on any computer system, but we have inched our way up to about 700 thin client workstations and things are still lightning fast. This design works, is stable and saves a lot of money.

Time Consuming Upgrades

2010-07-02T09:50:00.003-04:00

Centralized computing makes it so difficult to upgrade hundreds of users to the latest release of the ever exploited Adobe Acrobat Reader. :)

browser:/u # ls -lad acro*
drwxr-xr-x 3 root root 4096 Jan 13 14:08 acrobat_9301
drwxr-xr-x 3 root root 4096 Feb 23 10:29 acrobat_9310
drwxr-xr-x 3 root root 4096 Apr 23 13:32 acrobat_9321
drwxr-xr-x 3 root root 4096 Jun 30 16:06 acrobat_9331
lrwxrwxrwx 1 root root 12 Apr 23 13:33 acrobat_9_live -> acrobat_9321
browser:/u # rm acrobat_9_live
browser:/u # ln -s acrobat_9331 acrobat_9_live

and then push the reader into Firefox

browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_sound/firefox/plugins/
browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_nosound/firefox/plugins/
browser:/u/acrobat_9331/Adobe/Reader9/Browser/intellinux # cp nppdf.so /u/firefox3_noflash/firefox/plugins/

All done. :)

64bit Evolution Is Now Live

2010-07-01T10:23:00.004-04:00

After that small hiccup with running Evolution on 32bit Linux and the memory limitations, we are now live on 64bit SLED 11 and the difference in speed is striking. The per user memory requirements have grown, but it's a small price to pay (and memory is cheap!). The top screen below is 220 concurrent Evolution sessions. CPU is barely used at all. Disks are configured RAID 1 which has proven to be the best for Evolution (RAID 5 is much slower). When using the Groupwise backend, hitting Send/Receive the dialog only appears for a split second. The server is has 4 CPUs, quad core and I think we could easily get 500 concurrent users on here with no performance penalty. I'll have a few days of tuning and minor issues. Next on deck for me: getting gdm fixed to allow XDMCP connections again, and starting to work on that new monster server to run Firefox.

OpenOffice, Evolution & Benchmarks

2010-06-29T16:17:00.005-04:00

The week has been busy and it's only Tuesday. Here are updates on my current projects.

OpenOffice
Migration to 64bit Linux is complete and things are working pretty well. Even with 100 users, OOo opens in about 2-4 seconds, nice. I was able to confirm that indeed there is a nasty memory leak when using the Start Center and have submitted a bug report and valgrind information. The short term work around was just to install enough memory to cover the leak. Hopefully someone will be able to look at it soon.

Evolution
We migrated the rest of the users to SLED 11/SP1 and things appeared to be working well initially. We then started getting reports of Evolution crashing during peak loads. After some investigation it was determined that we are having the same problem that we did with OpenOffice -- having too many people to run the 32bit release. You basically run out of 'low' memory even though when viewing 'top' you perceive of no memory problem. When low memory is exhausted, it begins picking processes and zapping them with oom-killer. I have the servers configured to make them super easy to migrate, so we downloaded 64bit SLED 11 and it's already installed and working on another box. I'm testing Evolution and mime types, and all we have to do tomorrow is just backup /home for everyone and move it over and change IPs. Everyone then instantly is on a new server.

Benchmarks & New Server
The one area of our computing infrastructure that is growing the fastest is the use of Firefox/browsing. More than ever it's being used for data retrieval, videos and downloads. We also are purchasing some new software that might make heavy use of Java and we wanted to ensure the best possible performance; so using budgeted monies we ordered a new server.

The new server finally arrived this week. It's got 8 CPus each with 6 cores. I took a shot of 'top' below so you can see what it looks like. I had to pick 6 point font just to get the CPUs all to display. :)

One of the things that we do when these new servers arrive is hook up with our good friends working on OpenOffice and give them a chance to try and set a new compile world record. This is done before the server is placed into production and when they are finished we move it inside our firewalls and reload the operating system. So in the coming days, hopefully we will get some interesting information concerning just how fast this new server runs.

OpenOffice Gurus?

2010-06-25T12:43:00.002-04:00

So this week we deployed OpenOffice on 64bit Linux for the first time. It's smoking fast and works great especially under user loads; however it's chewing up LOTS of memory. It seems to be using easily 3 times more memory. I have been using lsof to check the documents that the users have open and they are nothing out of the ordinary and some of them are just a few pages with no images.

Anyone have any thoughts on this matter? Could it be leaking this badly? This type of memory usage is not sustainable; so I guess that means I'll have to start looking for it by process of elimination. Possibly it's user settings of some type. Very odd.

(top shot below shows memory usage).

Evolution 2.28 & 64 Bit OpenOffice Going Live Starting Today

2010-06-21T09:48:00.006-04:00

Evolution 2.28
Those of you that have met me and communicate with me know that I'm pretty fair in describing good & bad experiences in working with technology. The road upgrading from Evolution 2.6 on SLED 10 to something newer has not been very fun and took 1.5 years, which is WAY too long. There were a few times in recent months where we had meetings and discussed moving to completely new solutions. We are using the Groupwise backend with very heavy calendaring and had a minimum level of features that we wanted to get included and working. Sometimes I still get the feeling that Linux vendors are somewhat caught off guard when you *REALLY* run desktop software not on MS Windows. Evolution is by far the best solution to deploy on Linux over the Groupwise Java client (which is absolutely horrid), and the web interface. I just wish that more resources were allocated to its advancement.

With SP1 of SLED 11 enough upstream patches have been merged that our features are working well enough to deploy. It's not perfect and some bugs remain, but users can perform their calendaring functions including meeting resends and retracts. I will be moving over the users to the new server and then doing a bit of QA testing on their accounts to ensure that everything is working.

OpenOffice 3.2.1 On 64 Bit Linux
We had an issue last week whereby hardware is failing on our production OpenOffice server. As luck would have it, the replacement computer had already arrived and was sitting on the loading dock. I did a review of all of the packages that we needed and determined that everything was available in 64 bit versions. The server was quickly built and will be put into production this week. Openoffice on this server is FAST. The composer UI of Writer is appearing fully rendered in about 2-3 seconds, and running so quickly that the splash page doesn't even appear at all. Very nice, and thank you to everyone involved with making my job easier. :)

XDMCP & OpenSuse 11.3 Anyone?

2010-05-28T16:42:00.003-04:00

I have been poking at getting indirect XDMCP working on OpenSuse 11.3 with gdm with no success. Has anyone gotten this working? I did see that recently gdm was rewritten and there were some complaints that not all features were brought over to the new version.

Firewall is off on the inside network, gdm.schemas was edited..which looks like it replaced the custom.conf file. I put the lines into custom.conf anyway just to be safe. Xaccess looks good for indirect connections (*), yast2 security has remote connections allowed.

If anyone got it working, step by step instructions are appreciated.

I hate to think that 1975 xdm is going to be greeting our users on shiny new OpenSuse 11.3. :)

Ideas Welcome, Low Level Mouse Events

2010-05-19T15:04:00.002-04:00

My current project is related to Accessibility. Certain parts of the accessibility modules will help me and are being tested.

One requested idea is to make use of a 5 button mouse. By default the 5 button mouse works per the defaults: Thumb1 does a Back, and Thumb2 does a Forward.

The special need that we have is to eliminate the need to use the keyboard in combination with the Left mouse button. I'm trying to map it like this:

Thumb1 = Hold_Shift_L + Regular_Left_Mouse_Button + Release_Shift_L
Thumb2 = Hold_Control_L + Regular_Left_Mouse_Button + Release_Control_L

This would let them make non-continuous selections and shifted selections without the keyboard. It seems like imwheel is the closest software package for this purpose, but it seems incapable of sending a physical mouse click after another mouse click. Can anyone confirm this is true?

What I tried in the .imwheelrc file is this:

".*"
#,Thumb1,H|E|L|L|O
,Thumb1, Shift_L|Left|-Shift_L

The commented line works great, click Thumb1 and the string appears in the current window. You also can issue a Control-A to select all, works great. But when you attempt to simulate a mouse click, it doesn't seem to understand how to do that. The line below the commented line above to me should work, but doesn't work. I also tried Button8 instead of Left and that also does not work.

Anyone know of another program or technique that could provide this functionality?

Thanks!

Client Based RDP & GNOME Icon Launching

2010-05-14T09:12:00.006-04:00

RDP Architecture Change

For many years we have integrated MS Windows applications into our GNOME desktop and delivered them to our thin clients. Various techniques have been used to do so, at first we used UIS (Unix Integration Services) which was wonderful but is no longer available. We used Citrix to deliver applications as well. Because the RDP protocol added high color support a few years ago, this now is becoming our primary delivery technique. It's always been our goal to run as little as possible on physical thin client hardware and keep as much on the server as possible. The more that you run at desktops, the greater the chance that you will have to touch them with updates. It also is harder to troubleshoot and resolve problems when it's offloaded from a server. So what we have done is run rdesktop on the server and this has worked great. Rdesktop talks with Windows and then the presentation is delivered to the thin clients with X. For most applications to date performance has been appropriate. But there are some things that are now making this design not work as well as we would like:

More and more users are getting dual and triple screens to do wide screen work with lots of real estate. This is a lot of data to push over X.
Certain CAD applications are becoming more sophisticated and are doing lots of movements and zoom that should deliver reasonable response times.
More applications are starting to deliver sound and video from Windows.
Products such as Aqua Connect allow you to deliver Mac OS X applications from servers which also require faster response times.

As seen in the diagram below, our current design kind of created a double hop. RDP screens and sound went to the server and then went to the thin client with X and pulse.

Rdesktop runs on the HP Thin clients fine, but obviously when you do so that then creates a more client/server environment and condition where if you want to make a change to that software package you are going to have to touch desktops. While rdesktop isn't changing heavily, an example of what might happen is the users finding a keysym problem after deployment that requires a change to the configuration files. When rdesktop is on the server, such a change takes 2 minutes.

The other problem is how to make an icon launched from the GNOME server initiate rdesktop running locally on the thin clients. The user account doesn't exist on the thin client and is simply running an X server and then using XDMCP to connect to the server.

What has been devised is a technique that seems like it will work well. The thin clients basically have 2 user accounts, 'root' and an account of 'user' with an ID of 1000. I set up an account called 'user' with ID of 1000 on both of the GNOME desktops and then have granted sudo permissions for all other users to have access to the rsh command as 'user'. When they click on the icon, it will sudo the rsh command and pass to the thin client instructions on all of the command line arguments to run and then initiate a local copy of rdesktop to display :0.

Performance is much faster running in this manner; X and PULSE are completely removed from the loop and the Windows servers now have a straight RDP hop right to the thin client as seen in the diagram below:

Full screen video works well with this design,sound is much less prone to breakups and applications are much faster. We continue to test this design, but it's showing promise.

GNOME Icon Launching

During the debugging stage of Evolution I have noticed certain user techniques which have been unknown to me and I have taken steps to improve the user experience. When a user clicks on an Evolution icon our launch script automatically kills any running sessions of that software and starts a new one. The reason it's designed this way is:

Regular users cannot be expected to understand how to kill applications with a GUI or to look at process lists. People have no idea what this means or how to do it.
Users understand reboots, and if they had a local computer would just reboot it. Obviously, they can't reboot the servers. Hundreds of other people would not be pleased. :)
There are 15 hours of the day and weekends when someone from IT is not here to kill processes that are stuck or not responding.
The regular GNOME dialog that comes up when an application is not responding very often does not halt all related processes. Yeah it might get evolution.bin, but it very likely will leave the data-server and alarm processes happy churning away.
Very often users log off without closing Evolution correctly and sometimes just turn off their thin clients. This leaves processes behind on the server.

So halting all prior instances is desirable. They click on the icon, it cleans up the process list and deletes any temp files that might be hanging around and ensures 100% that they will get a running instance. Sounds great, right? Apparently not. Certain techniques that I saw when I was scoping the server were revealed.

Users Click On Icons Multiple Times To "Speed It Up".

I guess the theory is that an icon is like a water pump. The harder you pump, the faster and more likely you are to get water? Users have been double and triple clicking on the icon. Each click was then halting and restarting Evolution. Nice. So what they see is Evolution blinking multiple times until they stop clicking and then finally it opens.

I installed a small script that I called 'gun slinger' that counts the number of seconds since the last click of certain icons and ignores repeated and quick clicking. Here is the popup they see :)

Users Use The Shortcut Icon To Replace The Window List GUI

Instead of finding an already running process in their window list, they just click on the shortcut icon again. If they have something maximized over the top of Evolution, they have no idea how to bring it to the front and have just been clicking on the shortcut again.

I installed a bit of code in the launch script for Evolution that if it detects an already running session, it uses wmctrl to give it a window manager hint. It comes to the front, un-minimizes and flips the cube to the right side to try and get them to run the already running session. They then get this dialog:

All of these techniques are being logged to a file and I'm seeing how many people have been doing this all along. I'm hoping these two steps will eliminate abnormally shutting down Evolution and also reduce the likelyhood of damaged files and perception of instability.