Thin Clients, Groupwise 2012, Alfresco & LibreOffice

Lots of projects, and progress on them.

Our next thin client release was pushed out a wider group of users and they have been testing the new features. The new interaction with USB memory sticks and cameras is making it much easier for those users that just want to shoot a few pictures and insert them into email downsized from their raw size.  One click and a paste and it's done and they can go back to fixing cars and do their jobs and not having to learn how to use GIMP for this basic function.  We also got a few users on the local ICA client which offloads this canvas and transport from the GNOME desktop to their workstation.  They are reporting that it's faster for sure, and those CPU cycles are now off the server.

I mentioned that HP released a new Ubuntu build for their thin clients.  I downloaded it and installed it on the 5745, which is their current model.  The release notes indicate much faster boot time and that was the case for sure.  Within a few seconds of power, X was up running.  In the coming months, I'll move our customizations over to this new OS version and then backport it to the older 5725 model, all of which is easy.  The code is all sitting in one directory and I just need to install the right pieces of software and drivers.  We should see better performance with newer X drivers and it also will make it easier to add more local software if we so desire.  I spent a few minutes and installed the new Skype client on the thin client and it worked.  Lots of ideas to kick around and it should be a nice upgrade for users.

We are once again looking at the web interface for Groupwise 2012, so I spent some time again reviewing the feature set and testing.  Outside connectivity to devices is exploding, and we're exploring options.

Document management is a hot spot on our radar and I was assigned the project of installing and reviewing Alfresco to see what it does.  It's very Linux friendly on the back and front ends and was a great place to start.  I downloaded and installed the community version on a retired server and found it extremely easy to get up and running -- nice work guys.  A few clicks of a wizard and everything loaded.  I then spent some time looking over the design and flow and was impressed.  It certainly would save us untold hours of staff time.  The biggest problem with these types of solutions is mind share and skills.  Hopefully we'll start to see some moving in this direction and we'll be looking at other products too.

The ever helpful Cedric Bosdonnat helped me understand how his experimental CMIS code connects to Alfresco from LibreOffice and I was able to make an initial connection and view documents.  Possibly this connector could be ready by the time we need it and are ready to deploy, so I will be spending bits of my time checking the progress and helping do QA.

The shot below shows LibreOffice making the connection to Alfresco right from the file manager along with Alfresco displaying inside of Firefox; all 100% Linux front and back.  No software licenses anywhere in this shot!

I installed the Alfresco client software on an iPad and tested it there as well.  I had easy access to the documents stored in the various Sites.  Things appear to work quickly and easily right out of the box.  Here is a shot:

Coming up for me:  Final push of the new thin client software patches to our users; testing LibreOffice 3.6 and looking for regressions and prepping to go live; Additional testing of Alfresco, reviewing other products for features and suitability with our goals; Looking at issues related to moving Groupwise 8 to Groupwise 2012 on 64bit Linux; Disk and server requirements for Groupwise 2012.  It all keeps one busy.

Another Busy Week

It's hard to believe that it's Friday; the days go so quickly.  I have been putting the final touches on our next thin client release and it's now in the hands of more users. I have been pushing it to 16 test thin clients to ensure that everything works.   I had a white board of ideas and things that could be improved and nearly everything is crossed off....a sign that a release is almost here.  I finally got a few people to test full time the Citrix client running on the thin client instead of being host based.  The feedback has been a 10-20% increase in speed and those CPU cycles will now be off the server.  So it's a win/win for us. 

I wanted to be able to remotely poll the thin clients to detect all devices plugged into the USB ports.  People sometimes bring in their own hardware without approval and we need to be able to see that it's been installed.  We QA with all known hardware when we do upgrades, but unknown hardware might yield unexpected results.  For now I just made it very simple: I wrote a small wrapper for lsusb that formats the output a bit better and then creates a more friendly output.  (lsusb is not enough information, lsusb -v is too much).   The support portal grabs the output and puts it into a simple treeview (below).  I had some ideas in my head for a graphical screen, but unfortunately the various hardware vendors return their own non standard strings...too hard to parse as part of this project, this will have to do for now.

I have been testing LibreOffice 3.6, found some documents that won't open and UI screens that aren't working and filed bug reports.  Very happy to see how quickly this software is moving.  3.5 has proven itself very stable, and we still have had very few issues moving away from OpenOffice.

HP released a new thin client operating system which is now Ubuntu based.  I downloaded it and installed it on a test device.  It will be nice to have newer underlying libraries to put live, and the boot time is greatly reduced.  Newer Xorg drivers should help us too with various monitor configurations; good stuff.  I installed Skype 4 and it went on cleanly and was peppy on the t5745 device.  Looks like we will be able to offer a lot of nice new features two updates from now, later this year.

We have been looking at Alfresco document management software a bit, and I was asked to install it as part of some early testing.  I'll be testing how well it interacts with Linux, Firefox and how well one can download documents from iPads and LibreOffice.  This is a huge change, and will obviously need a lot of testing...but it's on the radar and assigned to me.

Other issues: Monitoring crash reports, tuning servers, adding a bit of code here and there to help users restart applications on failure. For the number of users logging in, things are very stable and support is happy.

Happy weekend.

User Instructions For USB

User Instructions:  If you support a large number of users, one of the hardest parts of any project is creating documentation.  If it's more than one page, no one will read it ... even if it's required for their job and something they should know.  I'm sure there are many techniques, but I have found instructions work best when they have screenshots and arrows and post it notes over the top.    Here is my first draft on the notes for how to interact with USB sticks and our software.  The left column is the new UI which eliminates file management almost completely.  A few clicks and it's into email and already sized.

I got about 5 people that indicated they would test all the new USB code and should be getting feedback over the next few days.

I also took the time to create a one page cheat sheet related to using the desktop.  There are a lot of keystrokes and shortcuts and if they are not known this lowers productivity; or they say they want another operating system because of memorized technique -- be it Windows or Mac.  The items mentioned are to assist with issues that cause user frustration.  Computers do really cool things, users just often don't find the steps intuitive..  User top issues:

- I find it hard to navigate to folders and remember where I saved my documents  (not one person has ever figured out the GNOME shortcut concept in the file manager without training)
- I need help finding those documents, and also documents that other employees have created.  Where did they save them?
- I saved a document last week, but forgot where it was saved and the file name.
- How do I change the speaker volume?
- We have to plan for the weather, how do I easily see the forecast and radar?
- How do I find other software applications that are not shortcuts?
- How do I check for new email messages quickly?
- I want to just type in the file folder instead of having to click in GUI mode. (not one person has never figured out that you can type in / on the wallpaper to open an entry dialog).
- How do I lock my screen easily?
- How do I take a screenshot on a timer, so that I can capture drop down menus?
- How do I take a screenshot of just one window, to avoid having to print the whole screen?

This one page document went out today and already people are trying new things.

Simpler USB Sticks And Cameras

In the last thin client upgrade, the first UI to assist users with USB sticks was released.  There were some issues in the design and people were using these workstations not always as expected.  So as I mentioned in my last blog, I have been retouching that code to make it work better.  The biggest issues that need to be solved:

1) We don't want everyone to have full USB stick access with drop and drag capabilities.  Why?  Because for most employees they are only working with photos and they are moving UP to the server from cameras and sticks.  In the hands of many, file managers are a dangerous things.  Errant lassos and drags are always present on a multi-user system.  Many workers only use computers for the smallest part of their jobs, and being able to insert a few pictures into a flyer or email message is all they need.

2) Users don't understand file types and sizes, no should they...yet just simply giving them a file manager yields all kinds of problems.  a) Why is this document so large?  b)  I resized the photo with the green handles in LibreOffice, why didn't the document get smaller?  c) I emailed this document and it was rejected on the other side as being too large  d) Why is the printer blinking for 5 minutes waiting for these 10 pictures to print?  e) Why is this taking so long to export to PDF?  --- and for Information Technology   a) Where the &*# did our disk space go?  b) Why did this user upload 500 pictures when they only used 2?

In the last blog, I showed the dialog that appears when a user inserts a USB stick.  Those with full approval can activate a full file manager and drop and drag files as they wish.  But for the rest the updated clipboard manager UI will appear.  The screen gives them the most common photo adjustments that they will want to do before it goes into LibreOffice or Evolution.  The original source files never are uploaded to the server, and everything is passed through the clipboard.  A few clicks and a paste and they are done.  Of course, these can be done in GIMP, but it's way too many steps and beyond the scope of what many can do. 

These new screens are modeled after the photo helper MIME bars that appear when they double-click on a photo.  Which were also designed to make it easier to shrink photos to the right size before inserting them into documents.

The software is not finished, but for the most part it's working as designed.  User inserts USB stick and it does a crawl and finds all pictures and sorts them by date and then by time in reverse order.  The screen below appears and shows the most recent date photos were taken.  If pictures are in sub-folders from the same date, they will appear all merged together.  The concept is not folder-centric, but instead date centric.  ( I remember WHEN I took them, but don't remember WHERE they were saved).  Arrows on the left and right of the thumbnails allow you to navigate up and down through the days with photos taken.  The arrows below the thumbnails allow you to move up and down through that date.   At the bottom, there is a mini navigator that allows you to quickly move to another date without having to wait for the thumbnails to refresh.  When you click on a photo the file size and width and height appear....which for most people means nothing.  As was the case with the desktop UI, it estimates the "LibreOffice Suitability" and "Evolution Suitability".  

If the photo is too big, they can select the appropriate size with the combobox, the thumbnail data changes and the new file size, height, width and suitability appears.

Once the adjustment is made, they click on the clipboard button and it's ready for any GNOME application that will accept the photo with a simple paste.   No file management, no GIMP...it's as simple as it can be.

I still have QA work on the code and will be testing it with real USB stucks here shortly and then will push it to some beta testers who will very probably find some issues.  But, I am pleased with the progress.

The white board of features for the new workstation upgrade only has a few items left and then we go to a more widespread beta test.  It will be great to get this code into production.

Other projects this week: Packing up Evolution crash data for the month of May; testing LibreOffice 3.6 with our documents and the new Corel Draw import filter; Testing Firefox 13 which came out today; looking over Java 1.7.4 and repercussions with web software that might have problems.

Happy 5th, Firefox And USB Progress

We were working on future budgets and were reminded again about the great savings obtained with running thin clients.  It was just about 5 years ago that our thin clients arrived and they are still working great.  We only have been losing maybe about 5 or so a year due to hardware failure since the 3 year warranty expired.   Right now it appears we can get another 5 years of duty.  $400 / 10 years for around a $40 dollar a year physical hardware desktop cost; and they still are very rarely touched by IT staff.  This architecture really does work and really does save lots of money.

We had a very odd situation where HTML 5 videos were only working for my user account and root.  I knew at that point it must have been some kind of file permissions issue.  After some poking around with strace, I found it in /tmp

drwx------    2 drichard drichard       4096 May 29 15:56 mozilla-media-cache

Out of the box, Firefox does not support multi-users with this cache.  The 'fix' is to do this in the script:

mkdir /tmp/$USER.mozilla 2> /dev/null
export TMPDIR=/tmp/$USER.mozilla

and this gives everyone their own private temporary folder and everything works now as expected.

I'm still working on the new thin client upgrade, and my area of focus right now is making the USB experience better for our users.  We have a situation where some users have full access to the USB sticks, but for many others we just want to give them upload capability only for pictures they have taken with cameras; there is no reason for them to be moving files off the server.   If I open the photo manager automatically, that forces people with full USB access to have another unwanted UI.  So now when users insert a USB device they are given a dialog which describes the two paths they can take and lets them pick (below).

Remember too that the files and USB stick are running locally on the thin client.  The GNOME desktop is not aware of the insert; so there are some nuances in that regard.  As the user performs the various functions, the files are moved to the server via the clipboard (copy/paste) or via FTP file transfer and Nautilus.

So the next improvement area is when to do a sync to the stick.  Those users using the photo manger and clipboard won't need it, but those using drop and drag will.  So I wrote quick little tray/notification applet that when activated does a sync and flushes the files.  When they pick "Full File Manager" they are given a dialog which indicates that the applet is going into the panel.

Nautilus does a great job of hiding the complexities of FTP; and generates nice thumbnails.  When they are finished they right-mouse on the applet and select "Save All Files And Prepare For Eject" and a sync is done.

One of the beautiful things about X/GTK and Linux is the transparency of where the software is running.  GNOME and avant-window-navigator are on the server in our computer room.  The USB notification applet is running on the thin client itself and it properly detects that it should sit into the panel.  If one of our users is performing this USB interaction and running Evolution, LibreOffice and Firefox they very well could be running software from 3 or 4 unique servers along with software running on their workstation and it's completely integrated together.   Next up will be improvements to the simple photo manager and clipboard application for those users that do not have full USB access rights.  This is building on the feedback of those that used the last iteration.

Other Projects: Trying to get some NX issues worked out, installing patches and testing.  LibreOffice 3.5.4 is out, will install that tonight from home and it contains 2 patches that affect us -- very nice to see how fast this software is advancing.

Support Portal Updates, Pictures Options And Other Updates

It's already been a busy week, but productive.  I have been hacking on the support portal to add more "actionable" events and cleaning up the UI.  There are still come spacing issues related to names that will be addressed.  But the support group can now see when users are clicking on icons to which they do not have a license.  After the user clicks on it three times in a row we are confident it wasn't accidental and now get a tile stating this fact.  The portal now also displays when users have changed some settings in Evolution that we do not want changed. (email checking less than 10 minute intervals).  We also can now see when a user monitor goes dark or blinks and they power off the thin client and log back in again.  People seem surprised/pleased that we can see these things and that they are contacted almost immediately.

I was doing some UI changes to the picture options UI that appears when you double-click on a photo and that now is live.  Centralized scripts are just a thing of beauty.  I always make the changes in another .py file, test with a few people hard coded and then comment us out and it immediately goes live.  Here is the script that launches the new UI and how the old one was turned off.  Right now I'm sending debugging information to /tmp and after a few days that will be turned off and the commented code removed.  Easy to roll back, easy to put into production, nothing destructive.

Here is the finished UI, similar to what I posted last week with a few small changes:

The reason for the Crop buttons becomes clear when you see the shot below.  Users on dual screens get this when they hit PrintScreen and the aspect ratio is terrible for printing on paper...and very often they want to print what is on 1/2 of the screen:

So they just hit the Crop button and the screen chops in half and is ready to save to their Desktop, print or place into the clipboard:

Another reason for the changes made to the UI is that I wanted to update the UI that appears when users insert USB sticks into their thin clients.  I'll post a more detailed blog about this issue when I am finished.  But in a nutshell all users have access to a simple photo manager that allows them to quickly remove a few photos from a digital camera or USB stick.  Those users with higher access levels can use a full file manager as you would expect from such devices.  I have started to modify this simple UI that trips when the thin clients detect USB insert.  I have built a basic Glade screen fashioned after the one on the main GNOME desktop server; it will mature and look much nicer when finished.  The design goal is to keep those huge megapixel images off the City network that are just bound for LibreOffice and Evolution.  Their higher quality is not needed. As I said, more detailed information will come in a few days.

Other projects: Still working on the new thin client build and making progress; updating LibreOffice to 3.5.3 tonight; very pleased with our drop in Evolution crashes on SLED 11 with the most recent patches--now it's time to figure out some of these deadlocks.

Project Updates & Why Customize?

The topic of desktop customizations has always come up here at Largo.  I do spend time making changes to how the desktop works, but it's not a significant part of my day.  Most of my job entails debugging software and problems, and helping architect future technology.  Over the years my viewpoint has become that when people say they want "Microsoft Windows" what they really mean for the most part that they want to have a certain set of steps they have memorized on that operating system.  When people have their own computers they spend countless hours tinkering with it and then countless dollars buying software to move documents from one file format to another and hobble together something that works.  We in the computer business have done a horrible job with software design in that regard.  The things that are basic to us:  File system layouts, file formats and file sizes are exactly the thing that users struggle with the most.  I've mentioned before that when I was younger I always thought things would get better when the "computer generation" came into the workplace; but that hasn't happened at all.  I cannot overstate how much time users spend on "files".  In our case we also have the issue of users that log on for a few minutes a day and have never used computers in the past and have no desire to become proficient at them.  If the desktop was not modified there would be some issues that would come to the surface 1) Many many man hours spent looking for files, retyping 'lost' documents, etc  2) User customizations and settings that affect the desktop and cause it to fail.  3) An increasing desire to buy more and more software to 'fix' things that are happening because of skill 4) Failure because it's too hard.

The customizations therefore yield lower support calls and increased efficiencies by eliminating issues of file location, file type and file sizes from their work.  The design tries to create an environment where files in the right format and size are moved automatically between applications.

I posted a Glade screen a few days ago with the revamped Picture UI that comes up when you double-click on a photo or image.  It's moved past the vaporware stage and is now being tested by about 5 of us.  Still some issues to work out, but it's showing promise.  Python makes this all a breeze, but for end users these things are just so confusing.  It's not appropriate to try and get people into GIMP for these basic functions required to interact with software packages; they'll never get it -- and it's just silly to expect it.

The screen appears and it clearly shows the file size and number of pixels; which for most people mean nothing.  However the new feature is to estimate "suitability" for use in Evolution and LibreOffice which are the two primary areas that will receive images.  In the shot below the picture has been opened and it's way too big for document construction and for inserting in an email. 

The ResizeTo option is set to "Medium" and the image reduces and the file size drops substantially.  From here it can be emailed, placed into the clipboard or printed.  I disabled GIMP and EOG when it was reduced to avoid users opening the temporary buffer and making changes and then losing it because they don't save it to the right folder location.  The details of the enabling and disabling of buttons is still working through my head.

 There is a gotcha that I found with placing a picture into the clipboard; it seems like the parent application needs to stay open or the buffer is lost if the image is over a small size.  I tried to im.store() it but that doesn't seem to work either.  So for now when they click on [ To Clipboard ] they get a green checkmark indicating that it finished and then an intrusive dialog with instructions on how to continue.  Putting this message into the status line on the bottom would never be seen so I felt this was the best technique in this case.

I expect these changes to be fine tuned and then moved out to users next week.   I'm hopeful this will increase the usability and success of interacting with pictures.  With just a few lines of python, we should see benefit quickly.

Other Projects Updates From This Week:  

LibreOffice is running like a champ and very stable.  We have hardly gotten any support calls and it seems to have slipped right in like a champ and take over for OpenOffice.  I do wish that LibreOffice was hooked into bug-buddy so that I can see how often people are crashing.  We aren't getting calls about crashes, but I like to see them happen and have backtraces.

Novell came through for us and created a big GTK patch for some libraries that were not thread safe interacting with Evolution.  It was a merge of some upstream patches and we loaded them Wednesday.  Previously we would get about 3-5 crashes each day on just this one bug and so far I haven't received one.  As I have mentioned in the past, all backtraces and deadlocks come to me automatically. Very happy to see this improving.

SuseCon will be in Orlando this fall and some of us from Largo will attend. Federico has built a page with an early concept of a site visit to our City so that people can see technology in place.  The link is here.  If you have never seen centralized servers and how software works over remote display it's pretty cool stuff.  It's always nice to show the issues we deal with because I feel they represent issues seen in the enterprise.

Support Portal, Thin Client Updates & UI Updates

Our "support portal" and any thin client updates go hand in hand.  We control features centrally and then they are pushed to thin clients.  My current biggest project now that the GNOME desktop is deployed and LibreOffice is live is merging in all of the features we wanted to deploy in our next thin client update.  I have finished the code to allow Kiosk/POS type connections to Microsoft Windows without requiring a connection first to the City system.  For those at our Recreation sites that move around between workstations, this will be a big help.  I also have added a Help button from the thin clients that allow the users to send us email without first logging in; and they can check on the status of the servers.  We placed our reboot schedules in ICS files for Evolution users, but now the thin clients will be able to display this information as well.  I'll blog more about this with shots when it's all finished.

One feature request that's coming in this next release is the ability to support dual monitors beyond just [ LANDSCAPE ] [ LANDSCAPE].  We'll be adding [ PORTRAIT ] [ PORTRAIT ] and [ PORTRAIT ] [ LANDSCAPE ].  We are all on the same video cards, so once I get it working it deploys to everyone else with the same Xorg files, nice.  The support portal is being modified to understand what combinations work and set the appropriate configuration files for them to download at next OS update. I spent some time hacking on the thin client detail screen which now better understands the monitors, their type and what exactly is displaying on each half.  When requested, the portal breaks their current screen into two pieces and then displays it on the UI on the appropriate monitor.  Mouse over on the monitor displays all supported resolutions, and the new OS build queries their devices and returns the make and model of monitor.  This is going to save us lots of time.

I cleaned up the [ Summary ] tab which displays actionable tiles of user problems.  These are not warnings, but issues where it's mandatory for action to be taken.  I have cleaned up exactly what is displayed and done some alignments of the widgets.  In the shot below it's alerting our support staff that some printers have stuck queues, that some users had some software problems because of missing network permissions and that some users had the power drop on their workstations.  Hovering mouse over these tiles displays the problem along with the most likely way to resolve the issue.

I monitor support calls and have hallway conversations with users all the time.  The intermediate UI that comes up when you double-click on a picture from Nautilus had some issues.  It's always interesting to me to see how users react and use software, and the things that I didn't think of during the design stage.  Very often I have better ideas, but am always trying to work these issues within the hours of the day.  These changes require very few coding changes, it's just mostly about the presentation and UI.  The show below shows the old UI (left) and the new one (right).  The new UI is not yet live and a shell right now in Glade until I feel all the changes look good and then I'll move the code over -- which will be easy.  If you are interested in user interaction, here are the changes:

• Users seem to like having art on each button, they seem to remember steps by artwork and not the words; each button now has art.
• The users were not finding the button to PrintToSelectedPrinter easily, so the button was moved to the right of selected the printer.  No call is made to the GNOME printer UI, it has way too many options for this purpose.
• The users were not seeing easily that you could DeliverWithEvolutionBypass (SMTP dump to Groupwise) based on the entry email box below.  This should be more clear.
• The 'Size of This File' area while a good start; did not really tell them anything they could understand and use.  The new UI will alert them of "suitability" with Evolution and LibreOffice.  Everyone is shooting 10 megapixel now which is not needed for email and document construction.  I'll make generalized statements (Good, Too Big, Too Small, etc) about the pictures.
• The functions that allow you to alter a photo are now under the photo; the functions to do something with the resulting output are all on the right side.
• Users with dual screens were taking screenshots and then wanting to only print 1/2 of the screen.  The current code would print the whole width landscape which is too small.  Buttons are now available to crop the screen.  Going into GIMP and doing this by hand is too many steps for such a basic function.
• Various alignments and layout techniques improved because I'm slowly learning Glade as time allows.

Once the code is done, I'll connect it for my user account only for testing and then release it to beta testers for wider use and then deploy citywide.  These changes are super easy on a centralized server and are literally just commenting in and out a few lines.   I'll be interested to obtain their feedback.

LibreOffice Data And Notes, Let The Computer Do The Tedious

We have been live on LibreOffice now for a few days and things are going well.  In conversations with our support group, the biggest issue that people had was "file location".  Most people have no idea where their documents are saved.  If they customized MyDocuments location or lost any RecentDocuments entries they struggle and assume that everything was lost in the upgrade.  File management still continues to be the biggest problem for users, and it's not ever going to change.  I believe that devices like iPhone and iPad succeed because there is no "file system".  Save a photo, and it's available to all applications.  Users don't have to make any choices in that regard; no file names or folders.  That's why the desktop has been customized to allow for as much drop and drag as possible.

Here is a shot of top running with about 100 open LibreOffice instances.  Looks like we could easily get another 100-200 instances running easily, which is wonderful.  Typing is crisp and fast.

While watching the LibreOffice server run and doing some slight tuning, I have been able to hack in some features that I wanted to merge into our "Support Portal" software.  This software is accumulating and monitoring nearly every click and issue on the GNOME desktop and application servers.  One of my pet peeves about software is when there is a tedious task that the computer can and should do for you and the software requires that you do it manually.  Much of the information we are logging is informational, but some of it is "actionable".  These are things that require a fix or step in order for the users issue/request to be resolved.  So I have begun to develop the [ Summary ] tab.  This section monitors all data that is coming in from various software packages and creates an easily seen tile/button of information.  Someone watching the portal is instantly aware of something that they need to do and it's easy for them to find the offending server or application. 

I know there are people out there that create UI fulltime and I'm sure these rough screens are hard to view.  :) But at this point this is more about fleshing out ideas and trying to create something useful for our staff.  Time does not allow for fulltime software engineering, this is usually hacked along with many other projects concurrently.

The screen is broken into 18 tiles and the last 18 events that require our attention appear.  The following items have been marked as "actionable" (more to come):

* RSH failure, whereby user is trying to run software and they don't have the right permissions
* CALENDAR failure, Evolution has a bug where if it crashes it occasionally drops their Groupwise calendar.  The user therefore does not get alarms for meetings because the calendar is disconnected.  We get about 1-2 of these a week
* MEDIA request, where users have asked us to send them our open source DVD which contains the software we run at the City for them to take home for Windows/Mac personal computers.
* FORCEQUIT - Networking, which means they logged back into the server and indicated they dropped off the server.  This usually is a cable or jack problem
* FORCEQUIT - Electrical, which means the power dipped and they were kicked off.  This means that the users UPS is probably dead, or they are not plugged into the battery side.  All of our employees have a UPS; if you have been to Florida in the summer you know why.
* LOAD, one of the servers has gone over 10% CPU usage, this very often means an errant process
* PRINT, there are print jobs that have not flushed from one of the servers within a 15 minute period.  This usually means paper jam, out of toner, etc.  Support can connect to the printers with a browser and debug what's happening.

When you hover your mouse over the button tile, it indicates which tab contains more detailed information and also provides a FIX which normally resolves the issue.  Clicking on the button brings up a user detail screen.

This new area is underdeveloped, but I'm looking forward to continued progress and testing.  I'm also looking forward to NX client for iPad 3 which will allow us to carry this information around at all times.

LibreOffice Live, Better Crash Experience

Today was the big day, LibreOffice went live.  We had been testing it for many months, and it fits so well into our architecture that I was confident it would go pretty well; but one never knows.  Last night before I left I reset everyone back to defaults one more time and then when I arrived today I put it live.  The migration was painless for City employees because it didn't cause any disruption.  The next time they requested the word processor, they were pointed to LibreOffice.  Those people that were already in OpenOffice continued in their session until it was closed.  All launch scripts are in common scripts, so I only had to make about a handful of changes and it was done.

The helper applications that I have described in the past seemed to work well in testing and have been deployed.  Notify-send is used in the lower right hand corner to give users tips and FAQs.  The lower left hand corner is a popup button on a timer that allows them to remove deadlocked instances from the process list.  If you leave it alone, it counts down to zero and then just goes away. We have had a few people not read it, and just click on it and then nothing opens because all processes were halted.  Technique will improve quickly as people figure it out.  Non-intrusive, intrusion.  This is required to allow users on 24 hour shifts to clear out processes when IT Support is not available.

With about 100 documents open and about 25 users typing, top looks great.  A few of these processes are still OpenOffice from earlier this morning.  All OpenOffice sessions will flush out as users close documents.

For the number of users on the network and the amount of work being done, things are very stable in regards to teh GNOME desktop.  But as is the case with all software - sometimes things crash.  I kind of had a "duh" moment yesterday when I realized that I could improve their experience.  The GNOME server of course calls bug-buddy when software crashes, by design it gives users a box that contains information that means nothing to them.  A while back I configured bug-buddy on our Evolution server to automatically grab their backtrace and dump to a flat file.  My logs indicate that a few people each day are dumping mail-notification and avant-window-navigator and the server was just giving them the bug-buddy UI and leaving them on their own. Shame on me.  With the panel gone, the session is worthless.  So I moved the bug-buddy binary out of the way and inserted a custom script that logs the crash to our tracking software, and then simply restarts the application for them.  I kill -11'd a session next to me for testing and it's working great.  The user sees a short blink and notify-send of restart and their critical pieces come back.  I'm logging all crashes to see what else might be crashing, but so far it's quiet. I'm thinking this is affecting maybe 5-6 users a day.  If other components are having problems, I should be able to do something similar and just get them up and running again.  The experimental script is below.  I'm pondering the potential for an endless loop and will monitor and adjust as needed.

LibreOffice, You're On Deck

After many months of prep work, QA and testing we are finally going live with LibreOffice next week.  There were a few bugs and nuances during early testing but with 3.5.2 we are to a milestone that seems ready for use with our 800+ users.  I have a group of beta testers and they have been very helpful in identifying documents and issues that needed attention and review.  Today I installed code to push them a fresh set of "defaults" each time they launch LibreOffice for a suitability test prior to deployment.  This is exactly what everyone will see next week, and this will allow me to push them the latest updates leading up to that date.  They are given a quick dialog which alerts them that settings are being reset

This dialog will go away next week.  The first time that you launch LibreOffice a dialog is presented which displays a quick summary of all things changed since our last upgrade of OpenOffice.  I have used this in the past with OpenOffice and found it to be very helpful.  No one reads the published articles and email messages about upgrades, but here it is again.  We then have acknowledgement that they clicked through.  The dialog gives them the option to receive a PDF of all release notes.  It also gives the users the ability to have a document emailed to their home address with the download links of LibreOffice; or if they want we will burn them a CD with the software already downloaded.  It's proven very helpful to promote people running these same software packages at home on their computers.

And then LibreOffice launches normally and quickly.  While you don't really ever have a "cold start" on a multi-user server, it's about 4 seconds from splash page to the point that Libre is fully rendered and ready for typing.  Certainly acceptable. 

Typing is crisp and responsive over remote display,and users are indicating that it seems to run faster in many places than did OpenOffice.  Being that OOo and Libre are on the same physical server, it certainly is an apples to apples test.  Both have identical resources, fonts and printers.   We have seen some problems with printing tables in comparison to OpenOffice and our support group has discovered some ways to align the tables for better printing.  Many of these documents have been upgraded several times going from WordPerfect 6, 7 and 8 and then OpenOffice 1, 2 and 3 and now LibreOffice.  It's a wonder they open as well as they do. 

Once this is deployed, I will be spending some time in some known problem areas.  We have seen some performance problems when Anti-Aliasing is enabled for graphic objects.  I want to better understand these problems and work with the developers to make improvements. 

Trapping User Crashes And Eliminating False Positives

When I was working on the new GNOME desktop servers, I wrote all of the launch scripts to go through a central set of calls so that we could trap an unprecedented amount of data.  Every click and login is recorded and we can tell when users are having problems.  One of the unexpected issues noted is the number of people that are "stealing" software packages that are running on other workstations.  We have two GNOME servers that you can log into and from there you can launch software applications such as Evolution and Libre/OpenOffice which are running on other "applications servers".  Both of these software packages cannot run on multiple heads at the same time.  This multiple login architecture was designed with the expectation that it would be used just occasionally and in a pinch; you are already logged in and running Evolution and go to a meeting in another building and need to 'steal' Evolution from the second workstation.  But the data was indicating that people were doing this all day as part of normal business practice.  When you steal Evolution it crashes and when you steal Libre/OpenOffice you are 100% assured that you will get a recovery dialog of some type.  In the dialog below, it shows how Libre and Evolution are running on servers which then can be run from the two GNOME desktops...but only to one at a time.

We had a lengthy conversation with our Director about this technique and sadly it cannot be stopped, because there are really times when users need to do this.  And there are 16 hours of the day when IT does not have support staff available to assist users.  So what we settled on was a consistent dialog to be used in places where users make an attempt to 'steal' software from one location to another.  The dialog warns the users of the severity of what they are doing and then they have to enter a six digit code which is their 'signature' that they accept the risks.  This same dialog is used for LibreOffice too and soon will be used for times when users attempt to log into the same GNOME desktop for a second time from another workstation.

The beauty of this design is that now when they terminate software with this technique it simply does an exit 0 and no longer is logged in our software as a crash and we can now make no attempt to grab a backtrace.  They are on their own and any problems they have are their own.  What they are doing is exactly like having a desktop computer with software running and pulling the cord; of course you'll have problems and very probably lose work.  We have had a huge drop in people using this technique once the dialog was installed. 

Now that false positives have been eliminated, it's now possible for the first time to grab a backtrace from software that has deadlocked. We know it's frozen because they are requesting a kill from the same workstation from which it was started the first time.  In the case of Evolution when they try and kill a running Evolution instance, they now get this dialog.  Once they select an option from the UI below, it finds the 'evolution' and 'evolution-data-server' process and grabs a backtrace and then closes them down.  Whereas previously we only had backtraces of a crash, now we have backtraces of deadlocks.  This will allow me to look for locking trends and work with Novell in improving the software. 

And if you are interested, here is the code that runs prior to starting the Evolution process.  When they click on the Evolution icon it always first tries to raise Evolution to the front of the window stack with wmctrl.  It then looks to see if it's already running.  If it's already running, it determines if it's running on their current workstation or another workstation.  If it's the current workstation it gives them the dialog to pick the last thing they did before deadlock; from this dialog they can obviously cancel and leave it alone.  If they terminate Evolution, it finds their two processes and used gdb to non-interactivity dump a backtrace.   If instead it's running on another workstation, it warns them that shutting down software is dangerous and then if they do so shuts it down and makes no attempt to create crash backtraces or deadlock backtraces...they are on their own.

I'm excited about the quality of data that we will be obtaining now, and this same code will be moved fully into LibreOffice once it's deployed next week.  We're getting complete crash and deadlock backtraces from hundreds of users without them having to mess with bug-buddy; everything is fully automated.

Improving Window Resolutions

Things have been so busy, I have been getting a lot done and been meaning to blog about what's happening. Perhaps some of what we are doing will give you ideas to solve similar problems. The first chunk that I worked on was the user experience in regards to interacting with MS Windows software from GNOME.

Without a doubt one of the hardest parts of deploying technology to users is monitor resolutions. It's hard to explain and it's hard to explain now it relates to font size and screen appearance. Our environment allows users to roam around all of our buildings and use mobile devices to connect to the same GNOME desktop. I was not satisfied with static or hard coded user settings. I wanted them to be able to truly roam around through the City and have the experience as simple as possible and not require them to keep picking settings or have a dissimilar experience on another workstation.

All of the Linux applications work great and require no customizations. However, Microsoft Windows applications that are integrated into GNOME are a challenge. Largo uses mostly RDP/Rdesktop which works well for us. Many applications will run seamlessly and behave just like standard Linux windows and are not a problem. However many other Windows applications are not well suited to run seamlessly and require a "container" around the UI.

The code for a single monitor on the new GNOME server was already settled for how to handle launching a MS Windows application on a single monitor workstation. It subtracts enough room for the lower avant-window-navigator panel (which we do not allow users to resize or move) and then enough room for the metacity window manager. The window consumes about 90% of the screen as seen below. Users can still switch workspaces and they have the largest possible canvas to run their software. Users tend to always want to maximize windows and this gives them the best possible experience without obscuring anything and confusing those that struggle with window management and movement. This code works for all resolutions and monitors.

So of course when you settle on a design, there is always an exception. We have 1-2 applications on Windows that were developed with the expectation that users would run them in maximized mode and need close to 1024x768. (Why would anyone multi-task and use multiple windows, right? ;) ) We have a good number of users still in 1024x768 so what this means is that the lower few pixels of their software was getting chopped off sitting above the panel. After reviewing options, I decided to develop code that allows us to specify WIN_MINIMUM_RESOLUTION in the GNOME launch script and it performs the following logic:

Is their monitor physically smaller than this value? Disallow the software from running and alert the user that their resolution is too low. (Yup, we have just a few 800x600 devices still in play and home users sometimes configure their PCs in 800x600).

Is their monitor big enough to hold this value, but it's obscured by the panel? If so, enable Intellihide in avant-window-navigator and launch the software. The panel then hides only when it's launched and is active only for that session.

Is their monitor big enough to run this software completely above the panel? Just launch it in 90% mode as it did before.

Intellihide is a feature that is hard to explain to users, so only those few people that need to see it, have it enabled. Everyone else has the same experience. This code was deployed and worked well and now the users can see all of the UI.


As long as I was in this code, I have been wanting to clean up the code for those users on dual screens. Some software applications users want to fill both monitors, and some things they want to fill just 1/3 or 1/2. Previously there was an underdeveloped UI that allowed them to pick the size each time MS Windows was launched on a dual screen monitor. This was very simple and worked, but I hate being asked the same question repeatedly and I am sure they did too. So I made the assumption that each unique application is normally run with the same scaling factor. You might always use an engineering application in full screen, and you might always use Visio in 1/3 size. I added a checkbox on the UI that allowed them to store per application settings of 1/3, 1/2, 2/3 and full screen modes. If this checkbox is enabled, it keeps it and uses it automatically the next time. If they log into dual screens @ 1440x900 or @1024x768 it won't matter, everything is dynamic and opens in the same scaling factor.

If this same user logs into a single screen workstation, this code is not used and it just runs in 90% mode. So the infamous question, what if the user wants to change it after it's set. Old school thinking would be of course to make a UI where the settings are stored and allow them to go in and make the change. I have found that such screens are seldom remembered and we get support calls, or they just suffer and are not happy with the experience. So following in the trend of some recent projects, I present this information on a dialog box similar to notify-send. It's on a timer and pops up in the lower left corner. If you leave it alone, the software uses this setting automatically in five seconds. Very clean, and no "hidden screen" to change settings...it's always right there and easy to change.

Screenshots; I love them and maybe they help show how this all looks to the users:

On a single monitor @ 1440x900 Visio is launched and creates a nice workspace with no prompts.



On a dual monitor @1024x768 user is prompted for desired Visio window, checkbox allows them to make this setting "stick"



The dual screen user has selected 1/3 and opted to remember this setting each time. Container opens as requested.



User launches this same application and GNOME finds setting and alerts the user that it's going to be used. Pseudo-notify-send popup is on a timer and allows them to easily change it later without having to hunt for a hidden UI somewhere.



Feedback has been positive and I think it was time well invested. Support calls concerning resolution problems should drop.

Quick Updates: Acrobat Reader 9.5.1 finally released and installed; Flash 11.2 seems to be working well and is now live. Testing Firefox 12.

Other Projects: Significant advances in improving techniques with shutting down software correctly to avoid recovery dialogs. Finally a way for us to grab backtraces on deadlocked software while dropping false positives....I'll blog about this next week. It's really coming together nicely.

Offloading To Thin Clients

When we work with outside vendors, it's always interesting to find out their misconceptions about what is a "thin client" and how it's run. When we call in issues, they always want to know the operating system of the thin client even though for the most part it doesn't come into play. Once you log into a GNOME server with XDMCP all you are doing is remote displaying the software. This concept for some reason seems to baffle people. Debian is on the thin client, but we aren't running Debian based GNOME.

In years past we always ran 100% of the software on the server and it's worked great. But in the last year we offloaded the RDP/Rdesktop client with success. The software to which we connect with RDP is obviously still host based, but now we run the client on the local workstation. In the case of RDP the users got a huge performance gain.

I'm reviewing all possible ideas for what should be offloaded to make use of distributed CPUs, while still maintaining our low costs of centralized servers. I have mentioned in the past that for sure we'll be doing the same thing with the ICA/Citrix client. The shot below was taken from our current GNOME server and you can see that wfica (the citrix client) chews CPU as the canvas is repainted. The server is certainly not taxed and we could get by running it in this manner; but I have a mind that tinkers and tunes and this really should be offloaded. Maybe a part of me wants to see 250 users running at 1% busy. :) For sure, it will increase the speed of the responsiveness and crispness of their UI interaction. RDP will communicate with the server instead of using X11 to deliver the presentation.



A few times a month, our users will stumble into a page that just does not play well over remote display. It's usually Flash and for some reason the player has problems and it just cannot keep up. I have always suspected that the video was encoded at a very high frame rate, but never have taken the time to verify that fact. As part of this whole process of reviewing offloading certain functions I have experimentally loaded Firefox 11 and Flash on the physical thin client. I wondered how it would work having access to the local video card.

I modified the master thin client to accept a request for starting a browser from the server side. Those with this new build can click on an icon on the GNOME server as picture below.



Firefox then runs locally. I didn't really know what to expect and the results were interesting. Firefox was slower in starting on the thin client than over the network from the big servers. I know that many people have complained about thin client speeds at other organizations; and I think this shows why. Software should be tested as both host based and local based and the right fit deployed. The local version of Firefox didn't have the crisp response time in the pulldown menus and UI interaction. It was certainly usable however. So the big test was then playing Flash content. My testing indicates that videos are not faster playing locally, and in fact might be a bit slower. In the shot below you can see "top" running on the thin client and Flash is just hammering the thin client. It's clear that Flash will consume as much CPU as it can get. I was testing on the older 5725 HP thin clients and once I pack up another beta build, I'll test it on the 5745s which I believe will provide a slightly better experience. But all in all I don't see any advantage to this design. Flash and Firefox constantly need upgrades and the devices would probably have to be upgraded every few weeks. I can upgrade the server in just minutes and it's deployed for all users; we'd need a strong case for offloading browsers to the local device and right now I'm not seeing it being worthwhile.



Current Projects: Installed and testing LibreOffice 3.5.2 and one by one crossing off items on my list for the next thin client release.

Software Portal Changes Pushed Live

With increasing workloads, I have been doing everything possible to increase what we can do remotely to eliminate having to send IT staff to users desks. Very often a big part of the problem is not knowing exactly what equipment they have. So I have reworked the "Thin Client Detail" screen in our support portal to give us more information than ever. All of these changes will also allow me to now begin adding new features for the end users.

I've always posted lots of pictures, I think they make it easy to understand how it's working. I'm sure that UI could be better, but it's working and will improve. When you bring up a thin client device it gets the server side configuration files and displays what it *thinks* should be configured on the workstation. This is done because very often the devices are powered off and now we'll be able to see how it's configured even if it's not currently running. The status line indicates that the information is based on configuration files and not the physical device. The REFRESH button is enabled in this case because it has done a ping and detected the device is powered on.



Once REFRESH is pressed, it physically queries the thin client and obtains as much information as possible. It detects version, function and other settings. It also then polls the Xserver and detects the resolution. One great new features is that the EDID is tested and it displays the exact make of the monitor. VG1930wm is an old ViewSonic monitor.



The next tab obtains the rest of the settings and detects their version. If they are running a Beta release of the thin client OS this is clearly indicated; otherwise it just displays a thumbs up symbol indicating they are current.



NX creates an interesting issue, the server thinks they are a thin client but it's really a virtual Xserver. So in this case the UI clearly shows that it's NX and then tries to find their connection IP.



Here is another feature now implemented. Some users have the ability to rotate their monitors as they desire. So when you enter the detail screen it detects that the monitor is configured in this manner (Rotate) but doesn't know how it's currently being used.



However when the REFRESH button is pressed, it does a query and detects the current orientation and displays it accordingly along with color depth and the exact cable (HDMI) being used. Being able to see the cable will allow us to upgrade users from VGA as time allows.



Another new features that is wonderful for us is to know the exact capabilities of the monitor. Are they configured for optimal resolution or running it in the wrong aspect ratio? Does the monitor go into a higher resolution? The portal queries xrandr and gets all of the supported resolutions and displays them as a tooltip when you hover your mouse over the monitor. Previously support would have to check this by physically going to the workstation.



The last major feature that will come is the ability to get a quick thumbnail of their session and a few other cosmetic fixes...but all in all it's working well. Centralized support has been a real time saver for us.

Other projects this week: Prepped Firefox 11 for release, tested Flash 11.2 RC, checked some pages that are having problems with sound, QA'd LibreOffice , reviewed all Evolution crashers and tried to organize them into groups and reviewed log files to look for errant networking cables and bad UPSs

Next week: The next thin client OS feature set will be started now that we have the host infrastructure to support it.