Desktop Home Stretch

After the technical issues of the last few months, the GNOME desktop upgrade is finally on the home stretch. We're very excited about this deployment and the users seem happy with the results. Now that the kernel issues are resolved, the server is VERY fast. Even with 50 users, avant-window-navigator is fully rendered in about 3-4 seconds and ready for launching software. Several months ago I spent time firing all of the processes in certain orders and multi-tasking them to get the fastest possible response. I'm pleased with the results.

We're still waiting on the next NX release so that I can test this over lower bandwidth devices. Sadly that is taking longer than expected, but one can always be hopeful of a release date soon. I'm not expecting any problems with thin clients, the area that will need testing is tablets.

One of the biggest flaws that we have on the older GNOME desktop is clear and provable records of who is using what software. We have so many little MS Windows applications and it's difficult to even know how often they are *really* used. Our Windows apps and Linux apps each go through a common set of launch scripts so it was very simple for me to dump each desktop click to a CSV file (below). We now have an instant audit of all activities. This data can be reviewed in a spreadsheet, and also will very soon be available from our support portal UI.

[ Very simple data log of each users clicking activity]


As time allowed, I have started to hook up all of our flat files to the support portal and move the [ Software ] tab past being vaporware. When the portal is started, it reads all of the files in /usr/share/applications and finds all of those active on the desktop. It then reads in our license files and builds an array with this data. Now our support staff can search for programs, hover their mouse over the icon and see exactly what operating system and IP is being used. It also indicates the users that have licenses to run the various packages. (as seen below). This is a huge time saver for them. Everything is dynamic, and eliminates having to use another tracking software package for this purpose. Start up the portal, and it's reading live data immediately.



I built a quick UI to get my head around the information that I want to display when you click on the software detail button. Please, please...no UI critique. :) The goal in this first pass is to just work out the functionality and then I'll do cleanups later. Using the flat files described above, the UI will render a chart of the number of clicks of the software package in the current month (purple). You'll then be able to use the arrow buttons to navigate through the months and look for usage. It will display buttons for all users that have access in the middle section. The lower area will give more technical information concerning the app, including the .desktop and launch scripts used. It will allow you to click on buttons and edit those files with gedit. When you click on the IP address of the server which is used for the various packages, it will either telnet or rdesktop to that server for review and to make it easier to kill stuck processes. Right now, support has many sheets of paper with this information for 250 software applications running on around 25 servers. This will be a huge time saver. The lower section also will display additional usage information to assist in determining how heavily software is being used.



The last few days have been spent going through all of the icons on the production desktop, figuring out exactly what task they perform and then contacting users to see if they still use the software. I'm then upgrading the icons, testing them with Seamless RDP and updating the artwork. I hope to be finished with this process by Monday.

Everyone is sensing this project is nearing completion, and I think we'll have a deployment to be proud of.

And There Is Java Sound

Sometimes one creates a post just as a way to document something that they do not do that often. In a few months when I have to do this again, it'll be here. That is the case today. :)

We are getting our first application that uses Java to deliver sound, and of course it has no way by default of delivering to the thin clients. After some Googling and poking around, the process is not terrible. I installed Java 1.6.27 into /usr/java and then got the open source version of Java in RPM format and extracted the two PULSEAUDIO files:

rpm2cpio java-1_6_0-openjdk-1.6.0.0_b17-7.3.x86_64.rpm | cpio -ivd ./usr/lib64/jvm/java-1.6.0-openjdk-1.6.0/jre/lib/amd64/libpulse-java.so

rpm2cpio java-1_6_0-openjdk-1.6.0.0_b17-7.3.x86_64.rpm | cpio -ivd ./usr/lib64/jvm/java-1.6.0-openjdk-1.6.0/jre/lib/ext/pulse-java.jar

I then extracted the sound.properties file as an example of how to add the lines for PULSE.

rpm2cpio java-1_6_0-openjdk-1.6.0.0_b17-7.3.x86_64.rpm | cpio -ivd ./usr/lib64/jvm/java-1.6.0-openjdk-1.6.0/jre/lib/sound.properties

You then hand move these files into the appropriate folders in the real Sun release, and merge in the PULSE lines into the sound.properties file. Restart your browser and Java sees PULSE and considers it a sound card. Very nice, we now have sound.

In the shot below, the vendor software is displaying the sound card and the clips play nicely.

Happier OpenSuse Days

This week we scheduled, installed and rebooted OpenSuse 11.4 with the 3.0.4 kernel. We broke the infamous rule of technology and also changed from the kernel-desktop to desktop-default; which means now that it's working beautifully we are not 100% sure which of the two issues resolved the problem. If one had more hours, we'd prove or disprove both theories but at this point we are going to enjoy the success and just move ahead. We've used both kernels on other servers and both have worked fine. Previously on this new server disk performance was terrible after user loads increased and now it's absolutely blazing in speed. Even with 40+ users, avant-window-navigator is starting to open in about 3 seconds even over remote display. You can barely even notice that 40 concurrent people are working -- in other words it's working as I would expect from Linux. I have solicited our users for more beta testers and hope to get us in the 50-60 mark by next week.

Other updates:

No NX updates this week to test, so I'm going to re-install and tinker with the current beta to see how it works on our new Wifi network. Still have those performance ideas in my head, waiting for the new Linux client updates in order to test them.

A few Evolution patches to test were received from Novell. They were compiled for SLED 11 SP2 which is not yet released. So I built a quick server with old hardware and tested them. The patches were not entirely satisfying, but at least now we know their status. Emailed results back to the developers.

We have a requirement for our first Java applet to work with PULSE, which I have never done before. A quick Google indicates it's possible, possibly via the Alsa layer. Time to tinker and better understand how this all works. Right now the sound from Java is very nicely trying to play on the server in the computer room. :)

I spent a little time hacking on the new Support Portal software and adding features that are useful to me (and others). It's now much easier to search for users and display their departments and to search by departments. With many hundreds of employees, it's almost impossible to keep them all straight. Next up, I'm going to write some quick code to keep track of what's in /usr/share/applications/*.desktop and log which users are clicking on which applications and how often. All of our icons pass through two common shell scripts (one for Windows, one for Linux) and it's going to be very easy to just write out data when they are clicked. This will allow us to better understand applications that can be removed from the network because no one is using them anymore.

Started to look at upgrading MoinMoin from 1.8 to the 2.0 (beta) release. Lots of libraries that need to be upgraded, and will require an operating system upgrade as well. Downloaded OS 12.1 (beta as well) for review and testing. Technology churn really keeps us employed, right?

Happy Weekend.

OpenSuse + Support Portal Code Deployed

The OpenSuse guys have been helpful in testing ideas to find the server bottleneck. I really don't stress over these things. Our older GNOME server is running fine and only those people that volunteered to beta test are seeing the performance issues mentioned in prior blogs. What's interesting is that even with the occasional freezes and slowdowns, they still prefer the new technology. All of the UIs are much nicer and easier and they really like avant-window-navigator. I'm sure we'll find it.

One of the things that is nice about hacking code is that you can picture ideas in your head and just write it and make it work. I love to see charts, stats and data. I also love having the computer monitor itself and report to us issues. Our Support staff has the same viewpoint. So I wrote the "Load" detail screen. The main portal displays in green the current user load for all of the big servers, and now when you click on it it gives you a chart with a much longer timeline. It then creates a front end to "top" and shows the top processes. The user name is a button, and you can click on it and the user detail screen which was already coded opens. It then hunts down all of their sessions along with information about their department. One can hone in on the sessions even further if desired and obtain information about the device they are using.

If any of the servers are over 10% busy for more than 10 minutes, a warning notify-send popup is generated and alerts all of us to review the issue.

The processes are GTK toggle buttons (marked in purple) and I'm going to allow our support staff to select processes and click the STOP button to shut them down. All of these features were available from the command line obviously, but this front end simplifies the whole thing greatly.



Up next for me: New NX 4 preview code (hopefully next week), testing the new WiFi with iPads, trying to get our Moin Wiki upgraded, trying to get 2 Evolution patches merged for SLED 11, installing and testing the 3.0 kernel on OpenSuse. But first, a three day weekend to recharge.

OpenSuse 11.4 Reload & Problem Is Back

Tuesday we moved our OpenSuse 11.4 users to a VM copy of the new GNOME desktop and we wiped the physical machine completely. We pulled the drives from the RAID array and put them in a new order, and then re-added a new RAID (1+0) partition. Our customizations are always placed in /u and /u2 so a few well placed tarballs and tweaks and we had a brand new machine with the same functionality as the old one.

Very sadly, the same issues have come back. I appreciated all of the ideas and tips presented in the comments area in prior blogs. But I do have a bit more information and it's very odd. This server should run 200 users easily, but as we get over about 20-30 and especially at 40 we see performance get very poor. Here is the new information: If I vi /etc/services it blinks for a few seconds and then opens. If I copy /etc/services to /tmp/services and /home/services and vi those files...it opens immediately. So there is some kind of contention or lock on the /etc/ directory. This contention seems to be the core of the problem. So many services are constantly looking at those files, and they are somehow bottlenecked. If you have any ideas to assist, the bug report is here.


Networking is still sub-optimal; note the dropped RX packets.

eth0 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:72
inet addr:128.222.99.243 Bcast:128.222.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12191295 errors:0 dropped:73239 overruns:0 frame:0
TX packets:12731836 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5112551787 (4875.7 Mb) TX bytes:10726315924 (10229.4 Mb)
Interrupt:16 Memory:f8000000-f8012800

eth1 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:74
inet addr:172.23.1.235 Bcast:172.23.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:73278 errors:0 dropped:238 overruns:0 frame:0
TX packets:8009 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15639151 (14.9 Mb) TX bytes:827822 (808.4 Kb)
Interrupt:17 Memory:fa000000-fa012800

There are a few more steps that we can take, including installing the 3.0 kernel to see if that helps. We might have to start looking at other distributions if we don't see progress soon.

Other projects continue: Writing some code for the support portal, testing NX, looking at upgrading our Moin Wiki, WiFi upgrades. I also have been poking at some ideas to improve NX performance to our thin clients, will blog about that if it pans out.

Server Reload & NX Testing

Server Reload

I have mentioned in previous posts some problems with our OpenSuse 11.4 server which is going to be used for running GNOME. It's been running slower than expected and is having some very odd disk and networking issues. The OpenSuse kernel guys have been great at reviewing all of our files and data, and right now everything looks like it should work fine...yet performance is not as expected. So we are going to rule out any chance that something failed during the upgrade from OpenSuse 11.3 to 11.4. As everyone that upgrades knows, sometimes what you get after an upgrade isn't the same as a fresh install. So next week we are going to make a Clonezilla backup of the server, and do a fresh install of OS 11.4 and then lay our customizations over the top. Our beta testers will be moved to a VM instance and won't have any down time. We'll know more next week. I keep reminding everyone that this is a normal part of upgrades and testing. Previous GNOME upgrades took time to debug and certify as well. The production machines are all working fine, so only those people that have volunteered to test are seeing these issues.

Testing NX To Anticipate Video Expansion

I've mentioned many times that we run old school X for those people on our fiber option network, which is the vast majority of users. One thing that is happening is that video training and video playback in general is exploding. We have noticed that certain streams (Flash, cough) seem to weigh heavily on our network from time to time. Many videos play fine, but at times higher quality (or frame rates?) cause the network to get more busy. The other issue we are seeing is that users expect a "video" to be a "TV" and are very often stretching the window as big as possible before there is frame loss or it gets too grainy. If we did not anticipate further growth in this area, we probably could continue to use our current design. But we are expecting this to continue to grow.

We use NX technology at our remote and low bandwidth sites. I have been testing it on the high speed network as a method to compress the video and video streams. I made a quick graphic below to demonstrate the difference.


The current design is to use X or XV and Pulse audio. With NX, the physical Xserver is on the system console itself and only a compressed stream is delivered to the thin clients. I made some tweaks to the configuration files and disabled XV and tested and results are not bad.

The shot below is a Flash video playing from youtube to a thin client over NX. This type of compression software is intelligent enough not to hammer your network. I had not tested this in several years and really it's working pretty well.



The one caveat with using NX/RDP/VNC type software is that sometimes it gets confused about screen changes and leaves artifacts. This is why it was never deployed on our high speed network. Remote X feels very similar to being on the console in terms of being crisp and repaints. In the shot below, while the video was playing I changed tabs in Firefox and the video section remains as an artifact. I had to take this shot with my camera because this is what the eye sees on the thin client. However, if you do a screenshot the artifact is not there. X thinks that area is gone, but the compression formulas don't always know to repaint all areas.



This testing was done with NX3.x, and we are awaiting the NX 4 Linux native client to see how it works. Pulse is now supported and will reduce our network overhead even more. We have a long way to go before we decide to make this type of change, but we are trying to say ahead of the curve with trends.

Happy Weekend.

Post Vacation Projects Continue

One returns from vacation with a new sense of energy, so I am working again on the various issues and projects.

OpenSuse 11.4 Kernel Problem

I got a lot of nice ideas to try and get our OpenSuse 11.4 server running better, and sadly none of them have made major improvements. I finally feel like we have ruled out everything except the kernel and have opened a bug report with the kernel guys. None of the admin tools are showing why the machine is running poorly, but it feels as if the whole server is running from the swap device. Copies of this server moved to VM suffer the same results. Unless I can get some kind of movement in this area soon, I am going to have to consider other Linux distributions. A few well placed tarballs and a VM copy will allow me to do that quickly, but it's still not pleasant. Everything is in place and working, we just cannot scale over 35-40 users. This hardware should run 150+ easily.

Evolution Issues Continue

Still communicating with some of the Evolution developers in the hopes of nailing down some nagging issues. Email and calendar events flow and things are working, but it's not yet to a milestone that gives me comfort. Everyone suffers from low resources in the IT field these days it seems.

Firefox 6

It's out already and I have been testing it heavily with various pages and technologies. Another Flash release came out in early August and that is being given the Youtube stress test over remote display with PULSE. While it never will be as robust as a local video card, playback is not terrible and more and more users are playing this type of content. I have some ideas to improve how this works, and will blog about it in the future. Just not enough hours in the day.

NX Testing and Ideas

I was able to connect finally with a thin client to NX 4 and started testing it in preparation of it going live in the coming months. I also had a few product feature ideas that I emailed to them. While we are using NX right now only at remote sites with low bandwidth, we are watching the marketplace and the whole "cloud" thing. If we move anything off site, we'll need NX for compression.

Support Portal Project

One of the best parts about writing software is when people actually use it, and we recoup the time invested in coding. Merging lots of little functions into one UI is saving time for our support staff, and also I feel allows us to provide better service. When there are problems with printers for instance, we know about it within just a few minutes.

Once again I will mention, please no HIG comments. :) I have kind of come to the mindset that projects should not be over-engineered up front. I'm talking about the types of projects where you spend so much time in design and building it "correctly" that the end users don't see any code for months and months. I'm building ideas and testing them with real people. Some of the data is still stored in flat files which is clunky of course; but the end users never see that stuff. Why make them wait for you you to build and optimize a database when just experimenting? If I abandon an idea, only a short period of time was used. I'll clean as I progress and like the results.

I have started to write some code for the handling of printers. You can now select a department and all of the printers in that department appear. Each printer offers three options/buttons: the first button launches Firefox and just goes to the IP of the printer. This connects with the HP web admin page found on all of their printers. The second button will display the number of print jobs currently in queue (over all servers), and when you click on the button it takes you to the UI for the viewing and halting of print jobs. The third button (marked in green) is an idea I had to make use of the under-utilized "lpmove" command. Yup, you can move print jobs to other printers...and no one ever uses it. When you have 60 printers, you can barely remember them all, let alone where they are located. So using lpmove on the command line would require digging out some kind of paper document to remember where each printer is located. So I have started to build a simple screen that will allow us to enter in the closest printers to each printer, along with directions from one to another. That will allow our support staff to take the stuck jobs, and fire them to a printer that might be located 50 feet away from the original. You might be wondering: Why not just cancel it and just have the user resend it? The real world answer is that many users tend to print (and expect it works 100% of the time) and delete and empty trash before they see that it really worked. What I envision then is the end user would get an email message alerting them the print jobs were moved to a new printer, with directions on where it's located. It'll be interesting to see how this all works. Printers really continue to be a nightmare, anything I can do to reduce our time on them seems worthwhile.



I also wrote some of the initial code to allow for searches of our users by first, last or account name. All matching users appear in the form of buttons. Hovering your mouse over the buttons gives you further details. This continues my pet peeve about making people open a detail screen to see information that should be easily found on the primary UI. If you do click on the user button, a detail screen appears with much of the same information found in the tooltip. It then also scans the servers and looks for their logins and lights up a monitor when found. We allow users to log in multiple times, so it's possible they have 1-4 active logins at a time. Other detail information that will appear on this screen (not yet written) includes information about their print jobs, and then a listing of software applications they are allowed to run. Linux software is deployed to nearly everyone because it's mostly free, Windows software has licenses and must be checked prior to launching. We also are looking at tracking how often all software is used, so that we can review it periodically and remove those packages that are no longer needed. If you click on a user login monitor, it takes you to the thin client detail screen and gives you greater detail about the physical device they are using. This allows us to check software versions, color depth, type of connection and the like.



Happy Friday!

OpenSuse 11.4 Woes

I'm leaving for a vacation tomorrow so I thought I'd post a blog concerning the status of our OpenSuse 11.4 GNOME server. We are in the home stretch of the deployment, but unfortunately the underlying Linux is giving us problems. I thought I'd relay the information and maybe someone has some ideas on getting this resolved. When I get back in just over a week, we'll start doing some major changes one at a time to resolve the issue. I'm not concerned about project success, but the best path is not yet clear.

All of the software is installed and for the most part GNOME and avant-window-navigator are working as expected. The hardware is beefy and should support at least 200 users+, but as soon as get to about 20 users - things really start slowing. We are seeing problems with the networking and disk performance. They may be related, but I haven't yet been able to prove it.

We have cloned the server and moved it VMWare and it's suffering the same problems. So I feel pretty confident it's not hardware. We are using this hardware elsewhere with no problems.

Disk IO Problem

For the first time we tried to deploy ext4, all of my Googling seems to indicate no major problems and no complaints about performance. I did not see any "multi user" issues reported. If we have a few (10ish) people on the server things work just fine. Once you get more, the disk becomes VERY slow. If I vi /etc/hosts (shot below), it just sits for 3-4 seconds with a blinking cursor before the file appears.




With a user load, if I run the passwd command it sits for5 seconds after entering a password before it completes. On other servers even with hundreds of people, all of this happens instantly. if I run yast2 and install software with a load, the whole server becomes very slow and non responsive. On other releases of OpenSuse we have never seen this before.

While this might seem a clear IO problem, what also enters my head is that we are having some networking problems too; and I'm wondering if the NFS mounts are having problems which is affecting local disk IO. Unfortunately I cannot dismount the NFS mounts to prove that idea, too many necessary pieces are on remote file systems.

Networking

It seems like others are having OpenSuse 11.4 dropped RX woes too. The first shot below is the desktop server, and note the packets that are being dropped. This number climbs constantly every few seconds.




But note on this shot below from OpenSuse 11.3 how it *should* behave. This browser server has over 100 people using Firefox 5 and you can see it's being hammered on the network side and there are almost no dropped packets. This reflects how our other Linux servers work.




I have noticed that certain NFS activities really slow the machine, when they are in Nautilus and generating thumbnails there is a noticeable slowdown for the other users. But it's not clear if this is "networking" or "disk io"

What's Next?

Aside from someone out there having intimate knowledge of OpenSuse 11.4 and a quick fix, we'll begin making drastic changes when I returned. The first thing I'll do is grab the experimental 3.X kernel that I see on some of the OpenSuse channels. I have one inclination that we are having a kernel/scheduling problem because it's affecting two subsystems. Currently I have upgraded to the latest stable release of kernel-desktop-2.6.37.6-0.5.1.x86_64. If that doesn't work, the next thing that I'm going to do is reformat the server with ext3 and rule out a file system problem. This is a nasty thing, and going to require lots of redoing work that is already complete. But what we have now cannot be moved into production.

One other idea I had was that somehow the -desktop kernel is not suited for multiple users, and I could try moving to the more vanilla version; but it sure seems like this change won't make disk and networking better.

If you work on OpenSuse and have any information, it's greatly appreciated. We are anxious to move this server into production.

Project Updates

It's been a while since my last blog, but as always projects continue to evolve and mature.

Sandboxed/Jailed Firefox Sessions

My last few blogs were concerning us creating sandboxed/jailed Firefox sessions to run certain City applications and that has been fully deployed and working well. I was able to get Firefox 5 pushed live with no problems. Flash 11 was also released and that too has been deployed. I am currently testing Firefox 6 with Java 1.7 on the web and with our internal applications. As is the norm lately with upgrades, 1.7 fails to work with some of our web based software. Wasn't the point of a browser to make it easier to deploy software? :)


Networking Problem, GNOME Desktop

I have been fighting a networking problem on OpenSuse 11.4 where after a certain amount of people log into the server it starts to get odd lags and we see RX errors on the NIC.

RX packets:45731224 errors:0 dropped:307554 overruns:0 frame:0

When you issue a command such as "vi /etc/hosts" it sits and blinks for about 3 seconds before it displays. However, as users log off at night it suddenly begins working again. I'm going to update the kernel tonight and the next step will be to install another type of network card and see if the problem goes away. This unfortunately has prohibited us from adding more beta testers. Very odd.


Portal UI And FOG Thin Client Updates

My coworker Brian has been doing a great job in getting our FOG server (running in VM) configured and working. He has been testing various techniques to push out updates to our thin clients. Many combinations are being tested in terms of performance. He is looking for the sweet spot in how many current thin clients to update at the same time. Sometimes there are circumstances where doing 5 thin clients at a time twice is faster than doing 10 all at once. I should have more information on our final analysis in the coming weeks.

While he has been improving pushing updates to the thin clients, I have been spending time as it's available working on our "Support Portal". Having all of your user sessions on a server is usually pretty wonderful to support, but in many cases it relies on command line tools and tinkering by hand. The tools that come with the distros are more designed for configuration versus server monitoring. I have been trying to think of ways to have the servers do most of the work, all of the information is right there; but it shouldn't take a System Admin to review it and know how to process.

I know those of you that build screens all the time will want to post HIG and UI comments. Please, these are back burner ideas that are still being developed. Our support staff already seems pleased with the features and capabilities and it's barely even started. If we have to do more with less (tm), this is certainly one way to obtain that goal.

The server monitoring screen is giving us graphs that show total number of users, load and total print jobs. The later is probably our most support intensive. I hate having to go into child screens to get information, so the designs are trying to make use of tooltips as much as possible.

Here is live data coming from the servers, user loads are displayed. Hovering your mouse over the load graphs (blue) gives you a tooltip that shows you the user accounts names running that particular software/server.



The lower graph (cyan) turns read when print jobs appear to be stuck. Hovering your mouse over the graph shows you the user and printer that appears to be having problems.



If you do click on a print job graph, it brings up a UI that shows the details of the print jobs on that particular server. The blue area is a toggle button and let's you pick multiple at once. The print jobs will then be capable of being cancelled or moved to another printer. The red area is the name of the printer and when clicked initiates a Firefox session and goes to the IP of the printer. Those of you with HP printers know that they provide an administration console on port 80. The magenta area is the name of the user. Clicking on this area will being up a user detail screen; which is not yet written.



When print jobs are stuck, we now get notify-send popups alerting us of this fact. A similar popup appears when the server seems to be 10% busy after several samples.



The companion piece to the FOG thin client updates was to create a screen to find and maintain files which contain server side settings for the thin clients. Once they get an update, they attempt to download a flat file which contains their settings and then reboot into the right mode. From the thin clients tab you can enter 3 or 4 octets of the IP address and the results will appear. A wrench symbol appears on thin clients that have server side configuration files completed. If you hover your mouse over the thin client entry, it gives you detailed specs on the device. You don't have to open a child screen to view the data.



You can filter and look for thin clients running in many different modes. If someone asks us how many thin clients are still in 1024x768, for the first time we can easily obtain that information. We can also easily query and find out how many thin clients are using two monitors.



We can search for the "function" (purpose) of the thin clients. We are using the same hardware for different purposes around some the city. Some are full featured workstations and others are set into Kiosk modes. Others are configured for low bandwidth sites and use NX.



If you click on a thin client pushbutton, a child screen appears and returns all of the information about the device. We can see who is using it, how many monitors it has and whether it's using HDMI/VGA/DVI cables. We can reset it back to factory defaults, reboot it, we can request a remote control, and we can do a wake on lan and power it on remotely.



At this point enough is working in this code for us to finish configuring the 650+ thin clients around our sites. We hope to have that complete by September. The UI will progress and advance, and I promise will even start looking nicer. :)

We've Got Browsers

I mentioned that I had a project assigned to try and get multiple browsers running to the same user at the same time so that we could run the release supported by the various vendors on our intranet. One doesn't want to allow older browsers and versions of Java to get out to the Internet. Behold, three versions running in one session:



In this case, a user can use the Oracle and SAP software and currently surf the Internet using Firefox 3, 4 and 5.

I had to think about the best way to make this work, so the current version always works normally and makes use of the users bookmarks and settings (obviously). The older versions run as users "firefox3" and "firefox4" and I made use of the -ProfileManager feature in Firefox to allow multiple users to run from the same user account. I then locked down prefs.js and localstore.rdf to settings I want to be used. For instance in order to make it look more like "software" I hid the URL and menu bars so they aren't tempted to try and surf with the sessions. The beauty of this design is that when Firefox 3 and 4 are no longer needed, I simply delete the user account and all of of these settings and infrastructure for that version is deleted.

If you are struggling with the same issue, this technique allows you to run all of your Firefox sessions from the same server. Our server for Firefox is a monster, and it would have been a shame to have to offload this functionality to another server or VM instance.

Many Projects Continue

For those interested, many concurrent projects continue here at Largo.

FOG & Host Stored Configurations & Portal
We have continued working on implementing network updates using FOG and things are going very well. We have a group of about 40 devices configured to network boot and have been testing the wake-on-lan features and configuring FOG to do updates at specific times. Our network guy has been reviewing the logs to see how much these updates impact the network. I completed the UI to allow us to store host based settings for the thin clients (pic below). It's a very simple file that is downloaded at first boot of update and configures the thin client back to the settings we desire. In between things I have been hacking on the portal UI below too and slowly adding features and figuring out ways to make it easier for our support staff to perform repetitive and common tasks that are handled via the command line right now. I'll post a more detailed blog when more is finalized and working. As mentioned, this really is a back burner project.



Firefox 3, 4, 5, 6, 7 & 8

OK, I know we are only up to Firefox 5. :) But the new aggressive version upgrades and feature changes have presented a challenge for us. It's very easy to do upgrades here, it would only take me a few minutes to physically install the new release. The impact is the *software* on the backend. Many vendors are very slow in "supporting" browser and Java changes. One vendor in particular that currently owns Java has software that has to run Java that is 2-3 versions old. This whole issue is a challenge for us on a centralized computer, I can't even imagine the gray hair it's causing on a client/server network. I tend to be the type of person that tries to turn a problem into a strength; and I believe we have a solution. What we are going to do is keep a copy of older Firefox releases for specialized intranet applications and not allow these sessions to get out to the Internet (for security reasons obviously). They will be jailed inside our network. For instance we have one financial package that requires Firefox 3.6, and another that requires Firefox 4. What I am going to do is create user accounts "firefox3" and "firefox4" and when the user clicks on an icon for those apps it will "su" into that account, start the appropriate version of Firefox and then automatically connect to the software authentication screen. These sessions will not be able to "surf" to the Internet. When they click on the Firefox icon, it will launch the most recent release (5 in this case) and use their personal bookmarks and settings. I don't want these older versions of Firefox running as the end user and impacting their settings. The graphic below is what I sent out to our IT staff to illustrate how it will work. Some technical issues remain, but most of the scripting is already in my head.

Virtual GNOME Desktop Working
The other hurdle that we completed was getting the new virtual GNOME desktop server working automatically. The physical server can be copied once a week now. One issue we had was that when rebooted the server would try and use the same IP address as the live server. So a few lines of code at startup detect a VM instance and changes the address of the NICs automatically. So we had a few users log into this second server and they were very pleased that all of their settings and customizations were all the same. This virtual GNOME desktop will allow us to split the load of users and allow for a second host to be available to be available in the event of some kind of hardware failure.

NX iPad Testing Continues

Now that the NX pre-release is working well enough for testing, I have been spending time asking questions and understanding how it works. My expectation was that there would be a place to configure your resolution just as you do in the NX clients. But the way it works is that you log in 800x600 and then use the desktop tools (xrandr front ends or whatever) to change your resolution on the fly. These tools are not exposed to our users, so I had to come up with a way to do this automatically. I'm sure when I have a few days to think about it, I'll have more ideas on how to properly get this deployed.

The code below was the hack-ish thing that I did to make it work for now. The xdpyinfo string returned by NX is different than our thin clients. Xsession checks for 800x600 and then does a secondary check to see if the NX string is returned. In that case it can then kick you into any resolution that you wish using xrandr -s x.



NX then automatically implements something similar to the -scale command of xrandr and fits whatever resolution you pick into the real estate of the tablet. From the code sample, you can see that I tested 1024x768 which gives you an almost 1:1 scale for the iPad. I then bumped it to 1440x900 and it automatically fits the entire tablet and is usable, but a bit grainy. Probably we would not bump it up that high nor give it the wrong aspect ration in this manner. Regardless, it was an interesting test. In the shot below I'm running software in 1440x900:



The speed of the presentation is still not ready for end users, they are impatient and it's hard to explain that tapping icons 50 times in a row doesn't make it faster. :) My tests were over 54Mb WiFi and it seems about 80-85% the speed that I would desire. Much better than early previews, but not ready for prime time.

Next up will be additional testing of NX over EVDO and also testing the webplayer from browsers over broadband connections. If this works well, it might mean we can stop having to give City employees client software to take home to log into the City network. I need to see better performance before that change could occur.

Happy long weekend!

PS: The iPad stand from Amazon is wonderful.

Lunch With A Side Of GNOME

Sometimes your plans for the day are altered greatly because of external circumstances. Nomachine released the latest NX 4 preview last night. We have been very anxious for this technology in order to deploy iPad/tablets, so this was my primary project for today. Prior releases inched closer to our goals, but were *far* too slow to do any beta testing. VNC testing over EVDO was painful as well. At this time no native client is offered for the iPad, instead it works by just using the Safari browser and then connecting to a web server. X is started inside the browser and your desktop appears. Performance on Firefox/Linux/Wired is very snappy and fast. Safari/iPad/WiFi works fairly well as does Safari/iPad/EVDO. So for the first time ever, I was able to take an iPad to lunch with me and log into our new GNOME server. I present, lunch with a side of GNOME:




Lots of issues remain, but it's wonderful that we have progressed this far. Back to testing.

Gotta Love Glade + Python

It's wonderful to have ideas in your head and be able to quickly get them prototyped and running. Using Glade and Python, I was able to connect the features that we need to make it easier to support our thin clients remotely. The software is detecting the number of monitors, resolution of the monitors, whether DVI or VGA is in use, color depth, Xserver size, uptime and ping response. It also allows us reboot the device, reset back to factory defaults, request remote control access and get a command line prompt from the Linux OS on the thin client for troubleshooting. (There are local processes that run and might have to be checked: Rdesktop and PulseAudio are two examples). I also have it snapping a screenshot and placing it over the top of the monitor image. I'm just using -gravity north from "convert" right now, and it's going to take some tinkering to get the various aspect ratios to fit correctly...but I'll leave that for a Friday afternoon project. :)



My coworker Brian had great luck with FOG today, wake on LAN is working and we configured all of the thin clients in our training room for PXE boot. Using FOG and the new server side storage of settings we were able to easily push an update to the thin clients, they rebooted and configured all of their settings and then booted right up to our XDMCP chooser. Very cool stuff, and we are looking forward to continued testing.

Doing More With Less, Improving User Experience

Usually when there are improvement areas in IT, we already know about them and have them in our heads. The roadblock is almost always the number of hours in the day, or money. The use of computer technology is exploding here at the City, and unfortunately we are not really hiring enough people to cover these increases. So we are trying to find every possible way to reclaim staff time. Sometimes in doing so you actually improve the user experience as well, and that is the case with my current project.

When you have 600+ thin clients (or PCs) it's very difficult to know exactly how each is configured. There are also sometimes cases where not all updates have been applied. We also sometimes have circumstances where users request additional features that require a small change to their thin client. Steps are being take to allow all of these tasks to be performed remotely.

These ideas have moved out of my head and into the form of screens. Please, no UI nazi comments. :) I'm sure HIG is being broken, and that a few pixels here and there are wrong. Right now this is just a work in progress to move designs on paper to the real world. Some of this is coded and now working and being tested for functionality. The thin client detail screen offers as many remote features as possible. From this screen we can remote control the user, force all settings back to defaults, reboot their device, see the number/orientation/resolution of their monitors, see which cables (DVI/VGA) are in use and also for the first time store all of their settings on the server. Here is the current (rough) screen. I also plan on splitting a thumbnail of their current screens on the monitors images as well so that support has an instant preview of what the user is seeing. Looks like xrandr can return back a string that indicates the exact monitor model, and I'm checking to see how that works.



We currently configure many settings locally to ensure the best possible user experience. If their monitor won't rotate, that feature is disabled. The monitor is configured for optimal resolution only and does not perform any plug-and-pray type polling for these settings. If you have ever supported users with changing resolutions, you know how important this stability is to your mental health. :) The UI above stores these entries in a flat file sitting in a jailed area of our internal FTP server. When the thin clients are reset to factory defaults or updated with a new operating system they now try and download this file automatically. If the file is found, the configuration UI toggles the right buttons and reboots itself in 10 seconds. So no more human intervention is required. Once it works, it will now always work after upgrades. It's working well, and we are going to try and test this in our training room which contains 16 devices to evaluate a network load.

My coworker Brian is concurrently working on testing FOG to allow for PXE updates. In our initial deployment of these thin clients, PXE updates were tested with the HP provided Altiris and we were not happy with the results so it was never enabled. It's our hope that FOG will allow for easier updates and scale better. Once that is up and running, the technology described above will allow the thin clients to automatically go right back to the tried and true settings that the user was previously using. Getting everyone on the latest code will solve problems for sure and these centralized techniques will make this easier than ever.

There are a few other ideas in my head to increase efficiencies. Another that is being developed is software to make it easier to allow us to see how the servers are performing. One negative aspect of using tools like "top" is that if several of us are monitoring the same server we are all consuming resources obtaining the same data. So what I did was write a very quick background script that checks user counts, CPU usage and number of print jobs and then create charts just *once* and store it in a JPG image. When you go into this monitoring tool, all you are doing is simply loading in the JPG image every 5 minutes. 10 of us looking at this at once will be no problem and we are all sharing the same resources. I put in a bit of code so that if it detects a condition where there might be a problem the chart turns orange and then red to catch your eye. The first iteration of this test UI is below. Once the features are working as expected, the UI will be improved.



So things are busy for sure, but it's fun to be hacking again and the City will for sure easily recoup these hours in staff resources.

SQL To The Rescue

As I continue to close up the last of the minor issues with the new GNOME desktop, I decided to quickly implement a dialog that appears after you log into the server. The print cost database is now loading information completely via cron and the numbers are astonishing. We have departmental printers and costs are high. I can't imagine the costs in Governmental agencies that have desktop printers. Be assured that we are doing everything possible to save tax payer dollars here at Largo. This kind of stuff drives me crazy!

For the first time, when users log in they are made aware of their print consumptions. As you can see, I don't print very often. :)



And the SQL to perform this query took just a few minutes:



We are still settling in on a value for the cost of a theoretical 100% coverage. The cost per page that they quote you is always for 5% coverage. So 2 cents a page @ 5% coverage is around 40 cents per fully printed page. That value is multiplied times the coverage rate and number of pages.

Fun With Aspect Ratios & Resolution

I'm not sure what it says about a person if this seems fun, but today I worked on improving some of our internal scripts that allow our support staff to remote control end users. :) This was a project brought about because of a desire for increased tablet usage. One caveat of that process is dissimilar resolutions when using VNC. Some of the users have higher resolution and some work in portrait mode and vncviewer would open 1:1 in the resolution on the remote site. This created scroll bars that had to be navigated, and the user would move things around and not realize you couldn't see their whole screen.

So as part of the iPad remote control project, I came up with some additions to the ksh code that checks resolutions on both sides and then always ensures that their screen fully fits on yours. This code is used in both areas: support to remote control end users and also to remote control your desktop session from any device. These shots show the results:

Desktop user (1440x900) is remote controlling an end user who is in portrait mode (1024x1280):



iPad (1024x768) is remote controlling a portrait thin client (1024x1280):



iPad (1024x768) is remote controlling a landscape thin client (1440x900):



The code is really very simple, get the height and width of both devices and figure out the aspect ratios. Then use that to figure out if it's best to use the width or height as your baseline, subtract a few pixels for window manager and lower panel, and calculate the other value based on the aspect ratio.

Fun stuff, and useful for the rest of our staff. It's already live and I should get feedback tomorrow.

Testing Remote Control Ideas

As the GNOME desktop project is getting closer to completion, I spent some time testing an idea that I have had for a while: Offer another option to remote control an existing login. On the current server, if you log in and already have a session running (from another thin client) it allows you to 1) exit without killing the other sesson or 2) Kill the other session and log in cleanly from the current device. We of course have talked about running NX or similar technology for all City thin clients which makes it easy to just steal a session. But old school X is still our preferred technology in all of our buildings with fiber optics. It has a different feel, is more crisp and your software looks like "software" and not a "picture". If you have used NX or VNC, you know exactly what I mean. When possible, we go for the best presentation.

So with this design, I wanted to offer a another option: 3) Remote control your existing session. This solves many use cases for us. You might want to gracefully shut down your software or you may want to quickly access your already running software.

I'm using our good friend x11vnc and it's working well. When you log in for a second time, it alerts you of that fact and then offers you the option to remote control your existing session. Once selected it initiates x11vnc on your original thin client and then starts the viewer on the second device. The second device can be another thin client....or an iPad. In the shot below, I logged into the thin client initially, and then attempted to log into the iPad; got the dialog and am doing a remote control.



What I'm looking at now is the best way to use the -scale option. The second login will very probably be in another monitor resolution. In the case of the shot above, original thin client was 1280x1024 and then I -scale'd it down to 1024x768 for the iPad. Lots of aspect ratio issues to consider.

A fun little useful project, and a good sign that the new server is almost finished.

GDM & Login Cleanups

We are inching closer and closer to going live with the new GNOME desktop upgrade for our thin clients. With all of the complexities of hundreds of icons and the nuances of having hundreds of users, it really does take 6-12 months to prep a server for this type of deployment. Not only do you have to consider the GNOME server, but also all of the remote servers and software applications. We are also weighing upgrades and connection techniques (RDP vs Citrix vs UIS).

We are up to about 45 concurrent users that are testing the new server, and uncovered a nasty issue in GDM related to two users trying to authenticate at the same time. Halfline/Ray was awesome as usual to create a patch and it's already posted. I just now need to get it built on OpenSuse 11.4 and will test.

My focus last week was on optimizing the user experience in authenticating and then the time it takes before avant-window-navigator launches.

GDM Tuning
These are the changes I made to GDM to make it run faster, and produce a dialog as soon as possible after XDMCP request

• I opened the .ui file of the password dialog box with glade-3 and made it a fixed size. Previously the "animation" process was kind of slow over remote display. It would display with no entry widget and then wiggle a bit and resize larger and then complete. My strings and logos won't be changing, so it was ok to test with my art/strings and size accordingly. This improves the user presentation and seems to make it a bit faster.
• I turned off server side GDM wallpapers, hesitantly. I like the idea of keeping as much as possible on the server in case one wants to make changes in the future, but by turning this off I was able to speed up the process of getting the password dialog. I made a change to the thin client build that gives the Xserver a black wallpaper (-br), so the old school gray/white grids are gone. Users only see this for a few seconds at most each day, so comfort has increased in this change.
• The /tmp/orbit-gdm directory basically never deletes any linc* files, and at one point we had a million temporary files (after many, many months of me not knowing about it). I now have a cron running that cleans these out each night with linc-cleanup-sockets. I'm sure that temp file placement is diminished when packed full of old files.
  
• I removed several .desktop files in the gdm startup group that we didn't need. It was attempting to start another pulseaudio process, even though pulse is already running on the thin client. This was probably creating some nastiness and delays.
  

Xsession Tuning
I have hand selected some gconf settings that are loaded when users log in and found that they were taking a few seconds to load. On a home/personal computer forcing in settings would never be done, but in a production environment where people are working 24 hours of the day...I have to ensure a working session each and every time. These gconf settings remove any customizations that might cause failure. So what I did was break them into smaller groups and then fire them off into the background concurrently. This was able to shave off several seconds from the point that you enter your password and hit ENTER to the point the panel appears. I also reviewed every line of the script and removed any items that were unneeded or obsolete.

This morning I put all of these changes live, they had been tested on a VM clone of the server and worked fine. So far the results are promising. The GDM dialog appears almost immediately upon request and the panel appears in about 4 seconds after the password is accepted.

This week I'll continue to ensure that all icons are brought over from the older GNOME desktop, QA work, and also working on testing the VM copy that is being generated once a week. We are experimenting with techniques to have the VM server detect first boot after copy and transition to new IPs automatically.

Happy Friday & Updates

I haven't blogged for a while but have certainly been busy. Things I worked on this week:

Print Job Data
The project to generate print job costs is finished and running nearly unattended. I have been QAing the files before the load into the sqlite3 database each morning, but should be able to cron that process in the next day or so. It'll just email me the exceptions if there are any. Printing costs are way too high, now comes the hard part of trying to break 20 year old habits and changing the culture.

GNOME Desktop
I'm adding more and more users and testing features and making tuning changes. There is some lingering NFS oddness still and I have some ideas of changes to make on the NFS remote server. I also found that /tmp/orbit-gdm was holding 1 million temp files, and now implementing a cron job with linc-cleanup-sockets running as gdm to remove the dead files. We are getting some odd delays at the point of XDMCP connect and the screen fully appearing, checking into that too. I have been updating artwork and icons and testing to make sure that the software packages are launching as expected. We are running about 40 concurrent users now and things are working well.

Password Protected OpenOffice Files
We have a City policy that if you password protect files, that you have to fill out a form and give it to your Director. It seems like that wasn't always happening so I was asked to write a script to find the files that were password protected. I found that unzip -l lists the .odt archive regardless of password protection...however the Thumbnails folder was missing in the case of a password. Makes sense, generate no thumbnail for a document that is locked..otherwise people would see the first page. So a scan found about 20 documents that were locked, not terrible.

iPad VNC Testing
NX still hasn't released version 4, so I started testing VNC over EVDO on the iPad. It's really really slow and not something that could be deployed. It only works in 8 bit color and that gives them a pretty messy experience...but even at 8bit color it's too slow. 16bit color took a full minute for the GDM wallpaper to even appear, and then GDM timed out and the connections drop. Completely not usable. I'm going to poke around with other ideas, and hope that a NX client is released at some point.

Happy Weekend!

And One More Thing, Resolutions

Just wanted to pass along a little trick that I use when writing software. About half of our 600 monitors are still 1024x768. We don't replace monitors until they are dead as good custodians of taxpayer dollars, and LCD panels are lasting many years. Usually IT developers have wide screens and some of them have dual monitors. It's easy to forget about the rest of our users, and suddenly software is developed that doesn't fit on their screens. Nothing worse than having to re-engineer a beautiful UI after finding this out during beta testing.

So what I always do is use my handy wallpaper that has a black box drawn at 1024x768 for testing screens.

We Have Print Data

After some short meetings with our Director and other staff, there seemed to be some excitement about analyzing our printing data. They gave me a list of some additional functionality and I had fun learning how to use some new widgets with Python/Glade and how to retrieve data from sqlite. I don't get the chance to write software very much anymore, and I still find it enjoyable after all of these years.

Here is the shot of the UI, simple and easy to use. The users select the data range, the departments and then the various functions they wish to query. The information is returned in three tabs on the right side which appear as "Department", "Printers" and "Users". One of the requested features was the ability to get the data as it appears in the Tree into OpenOffice Calc. The sqlite query is looping to fill the tree, and it was very simple to just dump the data concurrently out to a perl library which allows you to generate an identical .ods file. When the Tree displays, the file is finished and ready for use. I added an area labeled 'Files' which contains the spreadsheets. When clicked, the buttons use the normal MIME popups that we use on the rest of the desktop. The perl library does not generate the thumbnail file that normally appears in an .ods file, so the preview window comes up blank.



In the shot below, I selected OpenOffice - Calc opens the .ods file and will allow them to tinker with the numbers or copy and paste into other applications as needed. It took a bit of Googling to find the right technique to get the numbers over as integers and not as strings. But it's all working.




The primary UI gives them SQL SUMs of the various data, and I knew that people would be shocked at the number. So if you double-click on any of the Tree rows, it opens a child window which performs the same SQL statement except showing the detail records. It's very interesting to be able to see all print jobs by department, printer and user in chronological order. You can really get a better understand of how people are using/abusing the software.



I have a bit of work to do yet to merge in the log files coming over from the MS Windows spoolers which are in a different format. At that point management can review the data and consider ways to reduce costs with the use of other technologies such as portrait monitors, policies, training and tablets.

Paper & Printer Usage Under Review

As part of the GNOME desktop upgrade, a project has been initiated that is near and dear to my heart -- printing. There is a strong culture in many people to fight computerization and continue to work with 10 or 20 year old techniques. The amount of paper and ink used is tremendous and our printing costs are finally coming to light.

CUPs dumps flat files into /var/log/cups/page_log which contains lots of useful information, but is in no way in a format that can be researched. This is especially true when you have multiple servers and operating systems. We have always known that printers are heavily used, because they are a big draw on IT support. However, we have never been able to prove an exact dollar amount nor provide any specifics about poor techniques being used around the City. So I wrote some quick ksh scripts that scan through the page_log each night at midnight and dump the information into a sqlite database. This is yielding very interesting information.

I have done some SQL statements by hand to begin to look at the data, but obviously this isn't going to be the way that the IT Director or analysts will work. So I am writing a quick Glade/Python screen for running the various searches and the current shot is below. This won't be used by end users and won't get much more bling & art. It will serve the purpose of allowing our staff to select a date range, select departments and then include/exclude various desktop functions. The ink coverage is estimated per print job and we will be able punch in the per page cost for HP printers and it will give an estimated cost of the range selected. Initial estimates are that this cost might be north of 150K a year. :\

If we can use the new desktop, portrait LCD monitors and iPads to change peoples work habits, I believe we can increase efficiencies and reduce costs.

A Good Detour Through Lake Worth Florida

The week has been productive. One smaller project that I completed was resyncing avant-window-navigator to the latest versions and deploying this on our Beta desktop. This has a fix in that allows the shinyswitcher to work with VNC and NX connections. I'll be testing that fully this afternoon. This was the last patch required to allow iPads to log into the City and get a fully functional desktop. Cool stuff.

The other project was initiated with a conference call to our IT friends at Lake Worth, Florida. They are exploring ways to implement thin clients, Linux and open source software packages. We offered to send them a copy of our customized thin client build to assist in this process. A customized build has a serious flaw: It's a customized build. :) It was designed for us, and our networking designs were hooked into the scripts which have matured and changed through the years. We have a long term goal of redoing our IP scheme, so making the thin client build a bit more modularized was on our radar anyway. I grabbed some post it notes and went through the code and made a list of all server connections and features that other organizations might want to disable and then built some new UI screens that appear on first boot. The new design makes IP changes a snap, and allows for additional government agencies to piggyback off the project if they so desire. A few hours of design, coding and testing and it's working. There are still some additional areas that need work, but enough is done that we can now ship them a copy of the build and it will work on their network. The UI could probably be better, but it only displays for 30 seconds the first time a thin client is configured and never is seen again. I got feedback from our support division and these screens allow them to touch and update a thin client faster than ever.

On first boot (factory settings), splash page is generic. This provides a visual cue that no settings are configured and the build is currently generic.



The build detects no settings and comes up to a configuration screen (glade/python) that requests that you pick the desired City. (( If you are a Government agency and thinking about HP 5745 thin clients, email me and let's chat about adding you to the build ))



The Site Settings tab has all of the options and IP addresses of servers.



Select the monitor resolution. We hard code the values into xorg.conf to ensure a consistent user experience.



How many monitors does the thin client have?



Does the monitor physically rotate to landscape and portrait? If the monitor doesn't turn, why even allow them to do so?



What is the function of the thin client? Some are are used for desktops, and others are used for Kiosks (human resources, cyber cafe).



After saving the settings, the thin client is now configured for City of Largo.



And users in Lake Worth will get all of their customized artwork.



It's always exciting to hear about other agencies that are exploring ways to save money and as mentioned, these changes improved the quality of code and functionality at our site as well.