More NX & Chrome Books

Here is more information about my big project of late.  We're moving our users to an environment of stateless connections so they can resume between multiple end point devices.  The GNOME desktop and backend software has changed minimally, and focus on this change was making speed optimizations and getting the desktop and software to work on as many devices as possible.

The simplified diagram below shows how end points connect.  A NX Cloud server is the exposed server, and watches for connections over the NX protocol and for logins with a web browser.  A second Node server sits behind the Cloud server and will provide load balancing of users.  In the old "remote X" configuration, Firefox was running on its own server and had an Xwindows hop back to the workstation.  This worked great for many years, but with recent changes in video content and HTML construction, it's gotten too slow.  Now, Firefox is housed on the same server providing the NX transport and this has provided a huge speed gain.  Pages load and scroll immediately and typing is perfectly crisp and fast.  Other software applications that are not as impacted by running over remote X, are still housed on remote servers.  If you click on LibreOffice for instance, it's handed off to another server -- it then hops over Xwindows back to the GNOME/NX server and out to the endpoint device via the NX compression.

Speaking of endpoints, we now are able to provide logins to 6 platforms.  After entering your credentials, the GNOME desktop appears identically on all devices.  And if you are logged into one endpoint, and then log into a second endpoint the session immediately hops to the later device without loss of keystroke.  With the compression of  NX "lighweight" mode, everyone runs quickly over all types of networks and speeds.    Top goals:  Identical software, fast response and resuming.

The most recent end point offering is using a ChromeBook.  This is not yet in production, and being tested mostly by me at this point.  We purchased a HP 14 inch ChromeBook with 4GB memory for around $250.  It boots immediately.  After opening the Chrome browser, you just put in the right URL and credentials and after a few seconds the GNOME desktop appears.  The experience is then the same as the other platforms and this platform will resume sessions started on other types of devices.  This ChromeBook is full 1920x1080 and provides an excellent canvas space for running software.

In the shots below, I am logged into GNOME with just a Chrome web browser.  All software runs natively and doesn't even know it's running inside a browser.  Response is crisp and fast and this integrates nicely with our city workstations. 

Current projects:  Continued testing of Chromebooks, end user training on using NX technology, continued migration of users to NX and starting to look at replacing this older GNOME 2 desktop with GNOME 3.  It will be interesting to see how it runs with NX technology.  Many more blogs to follow in this regard.

Alive And Well In Largo

This blog has been quiet sadly, the last year has been so incredibly busy with changes in technology and high demand for IT services.  So many things have evolved, and I probably should have blogged about them as they were being developed.  This blog will try and get caught up to the biggest advancements.

Thinner And Fatter Workstation Delivery With NX

User requirements have been changing the last few years, and it's been a R&D project to find ways to try and find the right balance between centralized computing and mobility.   Our older GNOME desktop servers were wonderful in the sense that you could log in anywhere in the City and obtain all of your software and files.  But using remote Xwindows as the delivery layer required that you log off in one place to log into another.  Increasingly, users wanted to be able to resume sessions.  Remote Xwindows also is not able to handle certain changing technology needs.  Playing a Flash video over Xwindows will very easily grab 600Mb for just one user on your 1Gb network card -- certainly something that will not scale to many 100s of users.  So the last 1+ year has been spent changing the way we delivery software to the workstations. 

NX/Nomachine technology is a software layer that is installed on a GNOME desktop server that gives you a highly compressed connection layer that can replace remote Xwindows.  We had used NX technology for many years at our remote sites because it solved the issue of bandwidth very nicely, but did not use it for sites with fiber optic lines.  The changes made in NX4 were very suitable for our requirements, and after a lengthy beta period we are moving users to this technology in ever increasing numbers.   This is how we have become thinner.

NX can use a codec to compress the software layer or it can use what is called "Lightweight Mode".  In our testing, we found using a codec to be very expensive in CPU cycles on a multi-user server.  With a requirement to run 300 or more people, the server would have struggled to keep up.   In all but one use case, Lightweight Mode was able to solve this issue.  CPU loads on both server and workstation are extremely low and response time is crisp and fast.  The one caveat of this mode, is that it cannot play many Flash videos inside of Firefox.  Flash can detect latency and will reduce frame rates over remote X and on a stand alone workstation.  But because NX is running on powerful servers, Flash "sees" lots of bandwidth and CPU cycles and plays with no throttling.  The problem happens when the frames are sent down to the workstation -- even in lightweight mode, it just can't keep up.  So everything except videos was working great, how best to solve this?

The concept of running a browser on the local hardware was discussed, and experimentally I installed Firefox into the flash device of our various HP thin clients.  It works, which was expected.  And with a local video card, it plays videos very well.  But with the ever changing versions of Firefox and Flash (it seems like one of these is patched every 1-2 weeks), the update cycles to 600 workstations would not have been pleasant.  So experimentally the concept of launching Firefox over a NFS mount was tested.  When you click on the Firefox icon on the workstation, it NFS mounts our backend server, and starts up a 32bit version of Firefox/Flash and it worked as well and as fast as having it stored locally.  When an update comes out, I install it on the server and all of the workstations immediately pick up the change the next time they run this software.  Video playback is fantastic and we're now able to allow users to play HD videos -- something not possible in high numbers over Xwindows.  This is how we became fatter.

New Workstation Model

Our aging HP 5725 and HP 5745 workstations were nearing the end of their duty cycle, and they were not really powerful enough to handle the requirements of running Firefox locally.  A few of the new HP workstations were tested and based on the pricing and performance, we selected the HP t620PLUS.  They are blazingly fast to run the NX client piece, and also can run Firefox very quickly and well.  Money was available to buy about 180 of them, which would replace about 1/3 of our total number of deployed workstations.  So the last few months were spent receiving, unpacking, and deploying them to the users that needed them the most.  Feedback has been positive so far on these workstations and they are working very well.

Old Worktation RetroFit

400 or so older workstation will not be upgraded for another 12 months because of funding, so time was spent optimizing them to run with these latest advancements.  A workstation build was created that is identical to the one on the t620PLUS model in appearance.  The 5725 model cannot run a local browser, it's just too slow.  But it connects nicely to the NX GNOME server and performance is better than it was with Xwindows.  The 5745 model can run Firefox locally -- not as well as the new workstations, but well enough that it works and videos do work and play.  When users move around through the City on these three models, they look and work almost the same in all regards.  In the coming 30-60 days, these 400 will be moved off of Xwindows and over to NX.

ChromeBook Testing

NX supports Linux, Mac, MS Windows and tablets via a client piece.  All have been tested and are in use in various parts of the City.  Another login method is available.   NX supports logging in with just a browser.  I have been testing this with a Chromebook with success -- it's very fast and all of our software runs well.  The prospect of being able to have a mobile solution while using $250 devices with a laptop footprint is very attractive and potentially will offer a great amount of dollar savings.  I'll blog more about this in the coming weeks.

LibreOffice

Yup, we're still using LibreOffice!  Many thousands of documents a day are touced with this software and it does the job nicely.   The QA guys have been wonderful, and helped teach me how to bibisect bugs.  When a bug is found here that impacts us greatly, I can now do the leg work to find the regression and the developers have been wonderful in creating patches quickly.  About 200 users are now using version 5.0 with no known issues.  The rest of the users will be migrated in the coming weeks as part of receiving upgraded workstation builds and being migrated to the new GNOME server. 

Firefox Delivery

In the past, we had a server running GNOME and when a user clicked on Firefox, it handed that process off to another server and Firefox then remote displayed back to the workstation.  This met our needs for many years.  When using NX as the transport however, having Firefox running on its own server meant that there was an Xwindow hop in the middle.  Because of the network hungry nature of Firefox, this application was moved and now runs directly on the same server as GNOME/NX.  This gives Firefox direct access to the NX/Xserver with no hop in the middle.  Firefox therefore is very much faster, scrolling and typing is far superior.  This also meant that our scaling and loads have changed and required tuning and in the coming weeks some load balancing.  The server version of Firefox is used for all aspects of user requirements, except for video playback which is now handled by launching the Firefox version found on the local workstation. 



In Progress Projects

A lot of the ongoing projects have been mentioned in these prior paragraphs.  My top action items in the coming weeks:

* Continue moving more users to NX technology
* Tune and monitor the servers as the user loads increase
* Upgrade the NX4 technology to NX5
* Install and deploy the NX Cloud server piece, so users can log in with web browsers
* Add a second NX node, so that we have load balancing and can increase user counts
* Work on project to allow for embed of Youtube videos into LibreOffice for our employees and return the source to the community
* Continue working on our in-house support software and adding various features that have been scheduled.

Very kind regards to all of the people that ask me about our deployment even after all of these years.  It's all still working, and continues to provide significant cost savings.

Bring Your Own Device, But Can't Touch Them

The issue of BYOD (bring your own device) certainly has challenges for IT professionals.  Putting on one hat, you can easily see that it's wonderful to allow users to be productive with their personal tablets.  The other hat comes from years of experience, and knowing that they could be a support nightmare.  In enterprise IT much of what you do is work to having a consistent hardware base, to ease upgrades and reduce the difficulties that arise from diverse hardware.  BYOD is exactly the other end of the spectrum, there are thousands of hardware and operating system possibilities and end users often don't understand why their own personal $200-$500 purchase decision doesn't work.

The IT Director has crafted a new City policy, which includes a description of BYOD in great detail.  The overview is that they are allowed, and that no IT resources will be allocated to making them work or troubleshooting problems.

With all of that said, how then to deploy NX technology to tablets?  Users want to use their own tablets to connect to our GNOME desktops, but we cannot touch the hardware.  Users can download the Nomachine/NX client, but do not have the right key pair and there are settings and optimizations that would difficult for them to do on their own.  So we can't touch them, and it's not secure to email them the settings and keypair.  We kicked around some ideas and decided the best approach was to allow users to connect their Apple and Android tablets to City Workstations via USB and then initiate a small amount of software that mounts and then installs the .nxs and .cfg files needed to make the device work as expected.  This process is initiated by them via icon, and they accept the dialog alerting that there is no support in the event of problems or failure.

Once this R&D project was approved, I started to looking at tablets.  Android tablets mount pretty easily with go-mtpfs and Apple tablets can use ifuse.  I was able to then get to the NX settings folders of both types of devices on the command line and built a platform specific tarball.  I then created a simple Glade UI that requires them to ACCEPT the notice statement (UI is seen below).  This software is running on the workstation (not the server) and downloads the tarball and performs the install of the settings.  So far so good, and it's working on all devices that I have on my test bed.

It was simple enough to add a tab that displays a list of tablets that are known to work, and this is downloaded at runtime with the most recent additions. 

When the current settings profiles are built on the server prior to download, they are date stamped (YYYYMMDD) so that users can easily tell the date of their files right from the NX connection manager.  In the shot below, the UI has been used to install our three profiles and they are display correctly.

We have a few users now testing NX technology with Apple iPads and the feedback so far is promising.  In my next blog,I'll describe the user experience of NX and the GNOME desktop via a tablet designed for Touch software.

And Several Months Later....

I haven't blogged in several months, sadly.  It's easy to get caught up in projects and suddenly a good amount of time has expired.  So I'll try and do a quick update on my projects -- and try and publish blogs at better intervals. 

Here are the major things that I touched in the last few months:

LibreOffice

We have continued to use LibreOffice for almost all of our employees, and after a few point releases upgraded to version 4.2  The filters continue to improve, and stability is improving too.  We don't often get reports of hard crashes.  We're down to mostly questions of how to use features, and from time to time issues with imports of the various and different OOXML formats.   What's interesting about LibreOffice is that as the years go by, younger employees are less impassioned for using Microsoft Office.  There seems to be a greater acceptance in just working with different software packages.  I think this might be rooted in the fact that every phone or tablet you pick up has different software and at a certain point you realize they can all help you reach the same destination.  Many thousands of documents are touched each day and work is getting done -- with no license costs!

We changed over to the SIFR theme Citywide, which are all monochrome.  More and more software packages are moving in this direction it seems, and now LO better matches their appearance.

The Alfresco connector works great in LO and we have been testing that as well with success and considering deployment options.  Alfresco is still in a test mode, and not widely used beyond IT employees.

A few weeks ago we received a new monster server (described below), and it's been tradition for me to break in these machines for a short period of time by allowing them to be used to better open source projects.  We loaded Linux on this server and put it on the Internet for the LibreOffice developers.  They used it to do their stress test of various problem documents and test the filters into the different file formats.  Always nice to help as we can.

Firefox

In Firefox 26 a patch was merged that basically broke honoring umask for downloaded files.  This was a really bad bug for us and made it impossible to upgrade.  On multi-user servers it's important the files honor the permissions we want -- to make it easier for users to share files.  We finally ended up putting out a few dollars to pay for a few hours of work to get a patch written and integrated and it's working like a champ once again.  We were able to install and deploy Firefox 31.  The only problem we had with the upgrade was related to changes in the way that bookmarks are retained and work, and once this was understood we made changes on the server and it's working as expected.   Firefox 31 also contains new monochrome artwork which better fits into our consistent desktop look and feel.

Support Portal

Our internally developed support portal software application has advanced further and was fitted with a new monochrome theme.  I have been changing certain aspects of how user information displays to make it easier to understand the device used to connect to our network.  BYOD tablets and laptops are not supported by our staff, and now we can clearly see they are using a personal device. 

Federico was awesome to help me understand how to use pygtkChart from within glade/python to finally use *real* charts.  This has been a huge help and reduced the lines of code greatly. 

The support portal has been fetching information from our Linux servers regarding user counts, load, disk and memory stats.  With the help with our internal Windows Admins, we're now fetching from MS Windows servers too.  This helps us greatly by alerting us when we reach certain thresholds.  Very awesome.

GNOME Desktop To New Hardware

We are moving our production GNOME desktop to new physical hardware.  After some discussions and reviewing work loads, we decided for now to stay with GNOME 2.  The older server was cloned and was finally moved to the new hardware.  The server is 100% solid state drives with 80 hyperthreaded cores.  This increased capacity was needed for the next project:

Moving From Remote X to NX Technology

Using remote X has been a wonderful thing for us the last 20 years with our thin clients, but new requirements from end users are changing that landscape.  Roaming desktops are the top most requested feature, and we have decided to move from X to NX and run sessions fully stateless and server based.  We have been working closely with the fine folks at Nomachine to implement in our enterprise using version 4.2.  We're not quite there yet, but getting very close.   The short summary status:

-- Workstations with NX

We have two workstation thin client models and I have been able to get the NX client running on both.  Logging in and resuming sessions is very stable.  Our older HP thin client is struggling a bit with some Flash content, and I'm working with NoMachine on some optimizations to try and allow these devices to finish their duty cycle.  All other aspects are working very quickly on thin clients using the NX protocol.  I'm typing this blog from the oldest workstation and response is quick and crisp.  NX is especially noticed in Firefox when scrolling through pages.  Certain pages are starting to get slower over remote X -- NX will be a nice upgrade in that regard.

-- Tablets with NX

The iPad client was finally released, and I have been testing this along with Nomachine for Android with good results.  The ability to interact with software not designed for "Touch" is better than expected and our users are very excited.  I rolled out a few pilot tablets just yesterday and have already gotten good feedback. 

So a very brief update.  I have a few more blogs that I want to post in the coming days regarding specifics of these projects in the hopes they assist others.   Next up, is how we are solving the issue of allow users to configure their own personal tablets without IT intervention. 

What's happened to Firefox?

Lots happening in Largo lately and I will get to a proper blog update in the next few days.  I have been working heavily on infrastructure changes to accommodate stateless GNOME sessions and BYOD devices.  Good progress, and interesting things to report.  Very busy hours of the day.

So it was an inopportune time that I have had to work a bug report with the current Firefox.  With all of these new versions, it seems like there is a strong culture now to get it out the door regardless of impact to long time users.  It seems like there is a horse race with IE and Chrome to pack in as much stuff as possible and this seems to be at the expense of less understood features that are critical.

We have been using this technology forever, riding the Netscape wave and jumping over to Firefox around the 1.5 era.  Firefox is very fast and stable for us, even over remote X and thin clients.  Everything just works.  It takes me just minutes to do an upgrade and it's something that just churns for hundreds of concurrent users.  Our email is now web based and this is the backbone of the City.  Most new software and all cloud based solutions work with it...that's just awesomeness.

With a constant barrage of security exploits, it's critical that upgrades come in a timely manner.  And then came the problem:  Somewhere around Firefox 24 the whole download infrastructure was rewritten and now all downloaded files no longer honor umask.  It's been a disaster for us when this code was pushed live.  It was patched in Firefox 25 and now is not working again in both Firefox 26 and 27.  This is horrible for Linux and Mac users that want downloaded files that are world readable, they all default to 644 regardless of umask.

Here is a comparison of the older version vs FF 27:

-rw-r--r-- 1 drichard drichard 30169644 2014-02-05 13:33 ffirefox-27.0.tar.bz2
-rw-rw-rw- 1 drichard drichard 30169644 2014-02-05 13:30 firefox-27.0.tar.bz2

Maybe some of the developers have never seen Firefox running in the enterprise, or on multi-user servers, or in a VM or on a Mac with multiple users and don't realize the importance of this working correctly.  Please come and visit us anytime and we'd be happy to demonstrate these types of deployments!

So we are left now on a version two old with no solution in sight.  Do I need to start testing Chrome?

Quick Network Change Diagram

I made a quick diagram to illustrate to a few people the new design that we are testing whereby NX technology is used to deploy to our workstations.  I thought it might be a good visual tool to clarify my last blog.  Instead of using the Xserver and Pulse daemon on the thin clients, everything is passed through the backend computer server running NX/GNOME.  This allows for a stateless connection that can be resumed on any hardware that runs the NX client and any browser.

Lots of progress in testing this concept, and I'll blog about it next year.

Happy New Year.

Thinning Thin Clients, And Other Projects

I have not published an update in a good while, but things have been busy.  Here are the things that I have worked on since the last blog:

Thinning Thin Clients

How the heck can you thin a thin client?  We'll it's my current project and things are progressing fairly well. Since we started using thin clients in the mid-1990s, we have always used remote X as the transport.  It's elegant and fast for our needs, and consumes almost no bandwidth on a modern network.  Our current design has been wonderful for having roaming profiles.  You can log in anywhere in the City, and because everyone is running from a centralized host all of your software and files are immediately available for use.  If you go home and log in using NX, the same is true. 

However this has one design issue, you have to log out of the first location before starting another session.  If users forget to log out from the first location, they  are able to "steal" the session which severs the X connection abruptly and is not ideal when they damage settings or lose some of their work.  In general all of this is working very well and people move around the City all the time.  With the advent of tablets and more mobility, users are wanting to "resume" sessions over different networks of various speeds, including WiFi.  In order to accomplish this goal, we're testing using NX technology for all sessions.  Using NX thins a thin client, because now it's 100% running in the data center and the workstation is used only for mouse and keyboard; the Xserver is no longer remotely running on the thin client.

What this means for end users is that when a second server instance is started with their user account, they'll be able to "resume" the session from one location or device to another.  You can start typing a document at your desk, walk to a meeting and start up a tablet and resume the session and continue on the new footprint.

This change is presenting some scaling issues.  In the past some load was offloaded to the workstations, especially in the case of memory consumption.  Now all of this must be moved into the data center, which means bigger and faster servers with more cores.  We already have money in our budget to replace the GNOME server in 2014, so the timing could not be better.

We have two HP thin client models in use at the City.  The t5725 and t5745, both of which are discontinued.  We ordered the new t610 model which came with Ubuntu 10 installed.  I started the formal process in recent weeks of staging a build for production use.  Customizations were required to order to accommodate the older models.  The 5745 uses the intel driver and the 5725 is running ati.  So the build was modified to detect thin client model at first boot and set up the xorg.conf appropriately.  I was making great progress, when HP released an upgrade for the t610 to Ubuntu 12.  So I created a tarball of the customizations and in a few hours, they were all working on the new operating system.

More and more users are doing Skype interviews and in the past they would just check out a laptop and use that from their desk.  Since Skype is available for Linux, we're testing the concept of adding that feature right on the thin clients.  So far it's working like a champ.

In a nutshell, all three models of the thin clients boot Ubuntu 12 very quickly, and start up a very basic FVWM desktop which offers them the ability to connect to our servers using the NX Client. I am in the process of tuning the build for the three models and I'm working with NoMachine on some issues to make the NX Client work better in our environment.

Here is a shot of the current alpha build. FVWM provides menu and window management for software that connect to the servers.  Rdesktop, Skype and NX Clients run as siblings.  Full GNOME session is running inside the NX Client.

Firefox

After a quick patch in Firefox 25 to allow files downloaded by users to honor umask, it did not land in Firefox 26 and will return in Firefox 27.  So we're going to skip a version, but the good news is that with Firefox that means only waiting a few weeks.

LibreOffice

We jumped on LibreOffice 4.1 at 4.1.1 to solve some issues and improve file filters versus 4.0.  Out of 800 users, about 20 had to be rolled back for various bugs, which is normal and expected.  With release 4.1.4 we have been able to finally get everyone off of 4.0.  Things seem stable with this release too.  The server reports when users kill their software, and I'm not seeing many with LO.  Very cool.  Looking ahead to 4.2, there is a bad bug for us that prohibits us from testing heavily.  It's here.   We have lots of documents that make use of Nimbus Sans and the font currently is not rendering correctly.