Lots of issues remain, but it's wonderful that we have progressed this far. Back to testing.
Wednesday, June 29, 2011
Lunch With A Side Of GNOME
Sometimes your plans for the day are altered greatly because of external circumstances. Nomachine released the latest NX 4 preview last night. We have been very anxious for this technology in order to deploy iPad/tablets, so this was my primary project for today. Prior releases inched closer to our goals, but were *far* too slow to do any beta testing. VNC testing over EVDO was painful as well. At this time no native client is offered for the iPad, instead it works by just using the Safari browser and then connecting to a web server. X is started inside the browser and your desktop appears. Performance on Firefox/Linux/Wired is very snappy and fast. Safari/iPad/WiFi works fairly well as does Safari/iPad/EVDO. So for the first time ever, I was able to take an iPad to lunch with me and log into our new GNOME server. I present, lunch with a side of GNOME:
Lots of issues remain, but it's wonderful that we have progressed this far. Back to testing.
Lots of issues remain, but it's wonderful that we have progressed this far. Back to testing.
Tuesday, June 28, 2011
Gotta Love Glade + Python
It's wonderful to have ideas in your head and be able to quickly get them prototyped and running. Using Glade and Python, I was able to connect the features that we need to make it easier to support our thin clients remotely. The software is detecting the number of monitors, resolution of the monitors, whether DVI or VGA is in use, color depth, Xserver size, uptime and ping response. It also allows us reboot the device, reset back to factory defaults, request remote control access and get a command line prompt from the Linux OS on the thin client for troubleshooting. (There are local processes that run and might have to be checked: Rdesktop and PulseAudio are two examples). I also have it snapping a screenshot and placing it over the top of the monitor image. I'm just using -gravity north from "convert" right now, and it's going to take some tinkering to get the various aspect ratios to fit correctly...but I'll leave that for a Friday afternoon project. :)
My coworker Brian had great luck with FOG today, wake on LAN is working and we configured all of the thin clients in our training room for PXE boot. Using FOG and the new server side storage of settings we were able to easily push an update to the thin clients, they rebooted and configured all of their settings and then booted right up to our XDMCP chooser. Very cool stuff, and we are looking forward to continued testing.
My coworker Brian had great luck with FOG today, wake on LAN is working and we configured all of the thin clients in our training room for PXE boot. Using FOG and the new server side storage of settings we were able to easily push an update to the thin clients, they rebooted and configured all of their settings and then booted right up to our XDMCP chooser. Very cool stuff, and we are looking forward to continued testing.
Monday, June 27, 2011
Doing More With Less, Improving User Experience
Usually when there are improvement areas in IT, we already know about them and have them in our heads. The roadblock is almost always the number of hours in the day, or money. The use of computer technology is exploding here at the City, and unfortunately we are not really hiring enough people to cover these increases. So we are trying to find every possible way to reclaim staff time. Sometimes in doing so you actually improve the user experience as well, and that is the case with my current project.
When you have 600+ thin clients (or PCs) it's very difficult to know exactly how each is configured. There are also sometimes cases where not all updates have been applied. We also sometimes have circumstances where users request additional features that require a small change to their thin client. Steps are being take to allow all of these tasks to be performed remotely.
These ideas have moved out of my head and into the form of screens. Please, no UI nazi comments. :) I'm sure HIG is being broken, and that a few pixels here and there are wrong. Right now this is just a work in progress to move designs on paper to the real world. Some of this is coded and now working and being tested for functionality. The thin client detail screen offers as many remote features as possible. From this screen we can remote control the user, force all settings back to defaults, reboot their device, see the number/orientation/resolution of their monitors, see which cables (DVI/VGA) are in use and also for the first time store all of their settings on the server. Here is the current (rough) screen. I also plan on splitting a thumbnail of their current screens on the monitors images as well so that support has an instant preview of what the user is seeing. Looks like xrandr can return back a string that indicates the exact monitor model, and I'm checking to see how that works.
We currently configure many settings locally to ensure the best possible user experience. If their monitor won't rotate, that feature is disabled. The monitor is configured for optimal resolution only and does not perform any plug-and-pray type polling for these settings. If you have ever supported users with changing resolutions, you know how important this stability is to your mental health. :) The UI above stores these entries in a flat file sitting in a jailed area of our internal FTP server. When the thin clients are reset to factory defaults or updated with a new operating system they now try and download this file automatically. If the file is found, the configuration UI toggles the right buttons and reboots itself in 10 seconds. So no more human intervention is required. Once it works, it will now always work after upgrades. It's working well, and we are going to try and test this in our training room which contains 16 devices to evaluate a network load.
My coworker Brian is concurrently working on testing FOG to allow for PXE updates. In our initial deployment of these thin clients, PXE updates were tested with the HP provided Altiris and we were not happy with the results so it was never enabled. It's our hope that FOG will allow for easier updates and scale better. Once that is up and running, the technology described above will allow the thin clients to automatically go right back to the tried and true settings that the user was previously using. Getting everyone on the latest code will solve problems for sure and these centralized techniques will make this easier than ever.
There are a few other ideas in my head to increase efficiencies. Another that is being developed is software to make it easier to allow us to see how the servers are performing. One negative aspect of using tools like "top" is that if several of us are monitoring the same server we are all consuming resources obtaining the same data. So what I did was write a very quick background script that checks user counts, CPU usage and number of print jobs and then create charts just *once* and store it in a JPG image. When you go into this monitoring tool, all you are doing is simply loading in the JPG image every 5 minutes. 10 of us looking at this at once will be no problem and we are all sharing the same resources. I put in a bit of code so that if it detects a condition where there might be a problem the chart turns orange and then red to catch your eye. The first iteration of this test UI is below. Once the features are working as expected, the UI will be improved.
So things are busy for sure, but it's fun to be hacking again and the City will for sure easily recoup these hours in staff resources.
When you have 600+ thin clients (or PCs) it's very difficult to know exactly how each is configured. There are also sometimes cases where not all updates have been applied. We also sometimes have circumstances where users request additional features that require a small change to their thin client. Steps are being take to allow all of these tasks to be performed remotely.
These ideas have moved out of my head and into the form of screens. Please, no UI nazi comments. :) I'm sure HIG is being broken, and that a few pixels here and there are wrong. Right now this is just a work in progress to move designs on paper to the real world. Some of this is coded and now working and being tested for functionality. The thin client detail screen offers as many remote features as possible. From this screen we can remote control the user, force all settings back to defaults, reboot their device, see the number/orientation/resolution of their monitors, see which cables (DVI/VGA) are in use and also for the first time store all of their settings on the server. Here is the current (rough) screen. I also plan on splitting a thumbnail of their current screens on the monitors images as well so that support has an instant preview of what the user is seeing. Looks like xrandr can return back a string that indicates the exact monitor model, and I'm checking to see how that works.
We currently configure many settings locally to ensure the best possible user experience. If their monitor won't rotate, that feature is disabled. The monitor is configured for optimal resolution only and does not perform any plug-and-pray type polling for these settings. If you have ever supported users with changing resolutions, you know how important this stability is to your mental health. :) The UI above stores these entries in a flat file sitting in a jailed area of our internal FTP server. When the thin clients are reset to factory defaults or updated with a new operating system they now try and download this file automatically. If the file is found, the configuration UI toggles the right buttons and reboots itself in 10 seconds. So no more human intervention is required. Once it works, it will now always work after upgrades. It's working well, and we are going to try and test this in our training room which contains 16 devices to evaluate a network load.
My coworker Brian is concurrently working on testing FOG to allow for PXE updates. In our initial deployment of these thin clients, PXE updates were tested with the HP provided Altiris and we were not happy with the results so it was never enabled. It's our hope that FOG will allow for easier updates and scale better. Once that is up and running, the technology described above will allow the thin clients to automatically go right back to the tried and true settings that the user was previously using. Getting everyone on the latest code will solve problems for sure and these centralized techniques will make this easier than ever.
There are a few other ideas in my head to increase efficiencies. Another that is being developed is software to make it easier to allow us to see how the servers are performing. One negative aspect of using tools like "top" is that if several of us are monitoring the same server we are all consuming resources obtaining the same data. So what I did was write a very quick background script that checks user counts, CPU usage and number of print jobs and then create charts just *once* and store it in a JPG image. When you go into this monitoring tool, all you are doing is simply loading in the JPG image every 5 minutes. 10 of us looking at this at once will be no problem and we are all sharing the same resources. I put in a bit of code so that if it detects a condition where there might be a problem the chart turns orange and then red to catch your eye. The first iteration of this test UI is below. Once the features are working as expected, the UI will be improved.
So things are busy for sure, but it's fun to be hacking again and the City will for sure easily recoup these hours in staff resources.
Monday, June 20, 2011
SQL To The Rescue
As I continue to close up the last of the minor issues with the new GNOME desktop, I decided to quickly implement a dialog that appears after you log into the server. The print cost database is now loading information completely via cron and the numbers are astonishing. We have departmental printers and costs are high. I can't imagine the costs in Governmental agencies that have desktop printers. Be assured that we are doing everything possible to save tax payer dollars here at Largo. This kind of stuff drives me crazy!
For the first time, when users log in they are made aware of their print consumptions. As you can see, I don't print very often. :)
And the SQL to perform this query took just a few minutes:
We are still settling in on a value for the cost of a theoretical 100% coverage. The cost per page that they quote you is always for 5% coverage. So 2 cents a page @ 5% coverage is around 40 cents per fully printed page. That value is multiplied times the coverage rate and number of pages.
For the first time, when users log in they are made aware of their print consumptions. As you can see, I don't print very often. :)
And the SQL to perform this query took just a few minutes:
We are still settling in on a value for the cost of a theoretical 100% coverage. The cost per page that they quote you is always for 5% coverage. So 2 cents a page @ 5% coverage is around 40 cents per fully printed page. That value is multiplied times the coverage rate and number of pages.
Thursday, June 16, 2011
Fun With Aspect Ratios & Resolution
I'm not sure what it says about a person if this seems fun, but today I worked on improving some of our internal scripts that allow our support staff to remote control end users. :) This was a project brought about because of a desire for increased tablet usage. One caveat of that process is dissimilar resolutions when using VNC. Some of the users have higher resolution and some work in portrait mode and vncviewer would open 1:1 in the resolution on the remote site. This created scroll bars that had to be navigated, and the user would move things around and not realize you couldn't see their whole screen.
So as part of the iPad remote control project, I came up with some additions to the ksh code that checks resolutions on both sides and then always ensures that their screen fully fits on yours. This code is used in both areas: support to remote control end users and also to remote control your desktop session from any device. These shots show the results:
Desktop user (1440x900) is remote controlling an end user who is in portrait mode (1024x1280):
iPad (1024x768) is remote controlling a portrait thin client (1024x1280):
iPad (1024x768) is remote controlling a landscape thin client (1440x900):
The code is really very simple, get the height and width of both devices and figure out the aspect ratios. Then use that to figure out if it's best to use the width or height as your baseline, subtract a few pixels for window manager and lower panel, and calculate the other value based on the aspect ratio.
Fun stuff, and useful for the rest of our staff. It's already live and I should get feedback tomorrow.
So as part of the iPad remote control project, I came up with some additions to the ksh code that checks resolutions on both sides and then always ensures that their screen fully fits on yours. This code is used in both areas: support to remote control end users and also to remote control your desktop session from any device. These shots show the results:
Desktop user (1440x900) is remote controlling an end user who is in portrait mode (1024x1280):
iPad (1024x768) is remote controlling a portrait thin client (1024x1280):
iPad (1024x768) is remote controlling a landscape thin client (1440x900):
The code is really very simple, get the height and width of both devices and figure out the aspect ratios. Then use that to figure out if it's best to use the width or height as your baseline, subtract a few pixels for window manager and lower panel, and calculate the other value based on the aspect ratio.
Fun stuff, and useful for the rest of our staff. It's already live and I should get feedback tomorrow.
Wednesday, June 15, 2011
Testing Remote Control Ideas
As the GNOME desktop project is getting closer to completion, I spent some time testing an idea that I have had for a while: Offer another option to remote control an existing login. On the current server, if you log in and already have a session running (from another thin client) it allows you to 1) exit without killing the other sesson or 2) Kill the other session and log in cleanly from the current device. We of course have talked about running NX or similar technology for all City thin clients which makes it easy to just steal a session. But old school X is still our preferred technology in all of our buildings with fiber optics. It has a different feel, is more crisp and your software looks like "software" and not a "picture". If you have used NX or VNC, you know exactly what I mean. When possible, we go for the best presentation.
So with this design, I wanted to offer a another option: 3) Remote control your existing session. This solves many use cases for us. You might want to gracefully shut down your software or you may want to quickly access your already running software.
I'm using our good friend x11vnc and it's working well. When you log in for a second time, it alerts you of that fact and then offers you the option to remote control your existing session. Once selected it initiates x11vnc on your original thin client and then starts the viewer on the second device. The second device can be another thin client....or an iPad. In the shot below, I logged into the thin client initially, and then attempted to log into the iPad; got the dialog and am doing a remote control.
What I'm looking at now is the best way to use the -scale option. The second login will very probably be in another monitor resolution. In the case of the shot above, original thin client was 1280x1024 and then I -scale'd it down to 1024x768 for the iPad. Lots of aspect ratio issues to consider.
A fun little useful project, and a good sign that the new server is almost finished.
So with this design, I wanted to offer a another option: 3) Remote control your existing session. This solves many use cases for us. You might want to gracefully shut down your software or you may want to quickly access your already running software.
I'm using our good friend x11vnc and it's working well. When you log in for a second time, it alerts you of that fact and then offers you the option to remote control your existing session. Once selected it initiates x11vnc on your original thin client and then starts the viewer on the second device. The second device can be another thin client....or an iPad. In the shot below, I logged into the thin client initially, and then attempted to log into the iPad; got the dialog and am doing a remote control.
What I'm looking at now is the best way to use the -scale option. The second login will very probably be in another monitor resolution. In the case of the shot above, original thin client was 1280x1024 and then I -scale'd it down to 1024x768 for the iPad. Lots of aspect ratio issues to consider.
A fun little useful project, and a good sign that the new server is almost finished.
Monday, June 13, 2011
GDM & Login Cleanups
We are inching closer and closer to going live with the new GNOME desktop upgrade for our thin clients. With all of the complexities of hundreds of icons and the nuances of having hundreds of users, it really does take 6-12 months to prep a server for this type of deployment. Not only do you have to consider the GNOME server, but also all of the remote servers and software applications. We are also weighing upgrades and connection techniques (RDP vs Citrix vs UIS).
We are up to about 45 concurrent users that are testing the new server, and uncovered a nasty issue in GDM related to two users trying to authenticate at the same time. Halfline/Ray was awesome as usual to create a patch and it's already posted. I just now need to get it built on OpenSuse 11.4 and will test.
My focus last week was on optimizing the user experience in authenticating and then the time it takes before avant-window-navigator launches.
GDM Tuning
These are the changes I made to GDM to make it run faster, and produce a dialog as soon as possible after XDMCP request
I have hand selected some gconf settings that are loaded when users log in and found that they were taking a few seconds to load. On a home/personal computer forcing in settings would never be done, but in a production environment where people are working 24 hours of the day...I have to ensure a working session each and every time. These gconf settings remove any customizations that might cause failure. So what I did was break them into smaller groups and then fire them off into the background concurrently. This was able to shave off several seconds from the point that you enter your password and hit ENTER to the point the panel appears. I also reviewed every line of the script and removed any items that were unneeded or obsolete.
This morning I put all of these changes live, they had been tested on a VM clone of the server and worked fine. So far the results are promising. The GDM dialog appears almost immediately upon request and the panel appears in about 4 seconds after the password is accepted.
This week I'll continue to ensure that all icons are brought over from the older GNOME desktop, QA work, and also working on testing the VM copy that is being generated once a week. We are experimenting with techniques to have the VM server detect first boot after copy and transition to new IPs automatically.
We are up to about 45 concurrent users that are testing the new server, and uncovered a nasty issue in GDM related to two users trying to authenticate at the same time. Halfline/Ray was awesome as usual to create a patch and it's already posted. I just now need to get it built on OpenSuse 11.4 and will test.
My focus last week was on optimizing the user experience in authenticating and then the time it takes before avant-window-navigator launches.
GDM Tuning
These are the changes I made to GDM to make it run faster, and produce a dialog as soon as possible after XDMCP request
- I opened the .ui file of the password dialog box with glade-3 and made it a fixed size. Previously the "animation" process was kind of slow over remote display. It would display with no entry widget and then wiggle a bit and resize larger and then complete. My strings and logos won't be changing, so it was ok to test with my art/strings and size accordingly. This improves the user presentation and seems to make it a bit faster.
- I turned off server side GDM wallpapers, hesitantly. I like the idea of keeping as much as possible on the server in case one wants to make changes in the future, but by turning this off I was able to speed up the process of getting the password dialog. I made a change to the thin client build that gives the Xserver a black wallpaper (-br), so the old school gray/white grids are gone. Users only see this for a few seconds at most each day, so comfort has increased in this change.
- The /tmp/orbit-gdm directory basically never deletes any linc* files, and at one point we had a million temporary files (after many, many months of me not knowing about it). I now have a cron running that cleans these out each night with linc-cleanup-sockets. I'm sure that temp file placement is diminished when packed full of old files.
- I removed several .desktop files in the gdm startup group that we didn't need. It was attempting to start another pulseaudio process, even though pulse is already running on the thin client. This was probably creating some nastiness and delays.
I have hand selected some gconf settings that are loaded when users log in and found that they were taking a few seconds to load. On a home/personal computer forcing in settings would never be done, but in a production environment where people are working 24 hours of the day...I have to ensure a working session each and every time. These gconf settings remove any customizations that might cause failure. So what I did was break them into smaller groups and then fire them off into the background concurrently. This was able to shave off several seconds from the point that you enter your password and hit ENTER to the point the panel appears. I also reviewed every line of the script and removed any items that were unneeded or obsolete.
This morning I put all of these changes live, they had been tested on a VM clone of the server and worked fine. So far the results are promising. The GDM dialog appears almost immediately upon request and the panel appears in about 4 seconds after the password is accepted.
This week I'll continue to ensure that all icons are brought over from the older GNOME desktop, QA work, and also working on testing the VM copy that is being generated once a week. We are experimenting with techniques to have the VM server detect first boot after copy and transition to new IPs automatically.
Friday, June 03, 2011
Happy Friday & Updates
I haven't blogged for a while but have certainly been busy. Things I worked on this week:
Print Job Data
The project to generate print job costs is finished and running nearly unattended. I have been QAing the files before the load into the sqlite3 database each morning, but should be able to cron that process in the next day or so. It'll just email me the exceptions if there are any. Printing costs are way too high, now comes the hard part of trying to break 20 year old habits and changing the culture.
GNOME Desktop
I'm adding more and more users and testing features and making tuning changes. There is some lingering NFS oddness still and I have some ideas of changes to make on the NFS remote server. I also found that /tmp/orbit-gdm was holding 1 million temp files, and now implementing a cron job with linc-cleanup-sockets running as gdm to remove the dead files. We are getting some odd delays at the point of XDMCP connect and the screen fully appearing, checking into that too. I have been updating artwork and icons and testing to make sure that the software packages are launching as expected. We are running about 40 concurrent users now and things are working well.
Password Protected OpenOffice Files
We have a City policy that if you password protect files, that you have to fill out a form and give it to your Director. It seems like that wasn't always happening so I was asked to write a script to find the files that were password protected. I found that unzip -l lists the .odt archive regardless of password protection...however the Thumbnails folder was missing in the case of a password. Makes sense, generate no thumbnail for a document that is locked..otherwise people would see the first page. So a scan found about 20 documents that were locked, not terrible.
iPad VNC Testing
NX still hasn't released version 4, so I started testing VNC over EVDO on the iPad. It's really really slow and not something that could be deployed. It only works in 8 bit color and that gives them a pretty messy experience...but even at 8bit color it's too slow. 16bit color took a full minute for the GDM wallpaper to even appear, and then GDM timed out and the connections drop. Completely not usable. I'm going to poke around with other ideas, and hope that a NX client is released at some point.
Happy Weekend!
Print Job Data
The project to generate print job costs is finished and running nearly unattended. I have been QAing the files before the load into the sqlite3 database each morning, but should be able to cron that process in the next day or so. It'll just email me the exceptions if there are any. Printing costs are way too high, now comes the hard part of trying to break 20 year old habits and changing the culture.
GNOME Desktop
I'm adding more and more users and testing features and making tuning changes. There is some lingering NFS oddness still and I have some ideas of changes to make on the NFS remote server. I also found that /tmp/orbit-gdm was holding 1 million temp files, and now implementing a cron job with linc-cleanup-sockets running as gdm to remove the dead files. We are getting some odd delays at the point of XDMCP connect and the screen fully appearing, checking into that too. I have been updating artwork and icons and testing to make sure that the software packages are launching as expected. We are running about 40 concurrent users now and things are working well.
Password Protected OpenOffice Files
We have a City policy that if you password protect files, that you have to fill out a form and give it to your Director. It seems like that wasn't always happening so I was asked to write a script to find the files that were password protected. I found that unzip -l lists the .odt archive regardless of password protection...however the Thumbnails folder was missing in the case of a password. Makes sense, generate no thumbnail for a document that is locked..otherwise people would see the first page. So a scan found about 20 documents that were locked, not terrible.
iPad VNC Testing
NX still hasn't released version 4, so I started testing VNC over EVDO on the iPad. It's really really slow and not something that could be deployed. It only works in 8 bit color and that gives them a pretty messy experience...but even at 8bit color it's too slow. 16bit color took a full minute for the GDM wallpaper to even appear, and then GDM timed out and the connections drop. Completely not usable. I'm going to poke around with other ideas, and hope that a NX client is released at some point.
Happy Weekend!
Subscribe to:
Posts (Atom)