Wednesday, December 23, 2015

Alive And Well In Largo

This blog has been quiet sadly, the last year has been so incredibly busy with changes in technology and high demand for IT services.  So many things have evolved, and I probably should have blogged about them as they were being developed.  This blog will try and get caught up to the biggest advancements.

Thinner And Fatter Workstation Delivery With NX

User requirements have been changing the last few years, and it's been a R&D project to find ways to try and find the right balance between centralized computing and mobility.   Our older GNOME desktop servers were wonderful in the sense that you could log in anywhere in the City and obtain all of your software and files.  But using remote Xwindows as the delivery layer required that you log off in one place to log into another.  Increasingly, users wanted to be able to resume sessions.  Remote Xwindows also is not able to handle certain changing technology needs.  Playing a Flash video over Xwindows will very easily grab 600Mb for just one user on your 1Gb network card -- certainly something that will not scale to many 100s of users.  So the last 1+ year has been spent changing the way we delivery software to the workstations. 

NX/Nomachine technology is a software layer that is installed on a GNOME desktop server that gives you a highly compressed connection layer that can replace remote Xwindows.  We had used NX technology for many years at our remote sites because it solved the issue of bandwidth very nicely, but did not use it for sites with fiber optic lines.  The changes made in NX4 were very suitable for our requirements, and after a lengthy beta period we are moving users to this technology in ever increasing numbers.   This is how we have become thinner.

NX can use a codec to compress the software layer or it can use what is called "Lightweight Mode".  In our testing, we found using a codec to be very expensive in CPU cycles on a multi-user server.  With a requirement to run 300 or more people, the server would have struggled to keep up.   In all but one use case, Lightweight Mode was able to solve this issue.  CPU loads on both server and workstation are extremely low and response time is crisp and fast.  The one caveat of this mode, is that it cannot play many Flash videos inside of Firefox.  Flash can detect latency and will reduce frame rates over remote X and on a stand alone workstation.  But because NX is running on powerful servers, Flash "sees" lots of bandwidth and CPU cycles and plays with no throttling.  The problem happens when the frames are sent down to the workstation -- even in lightweight mode, it just can't keep up.  So everything except videos was working great, how best to solve this?

The concept of running a browser on the local hardware was discussed, and experimentally I installed Firefox into the flash device of our various HP thin clients.  It works, which was expected.  And with a local video card, it plays videos very well.  But with the ever changing versions of Firefox and Flash (it seems like one of these is patched every 1-2 weeks), the update cycles to 600 workstations would not have been pleasant.  So experimentally the concept of launching Firefox over a NFS mount was tested.  When you click on the Firefox icon on the workstation, it NFS mounts our backend server, and starts up a 32bit version of Firefox/Flash and it worked as well and as fast as having it stored locally.  When an update comes out, I install it on the server and all of the workstations immediately pick up the change the next time they run this software.  Video playback is fantastic and we're now able to allow users to play HD videos -- something not possible in high numbers over Xwindows.  This is how we became fatter.

New Workstation Model

Our aging HP 5725 and HP 5745 workstations were nearing the end of their duty cycle, and they were not really powerful enough to handle the requirements of running Firefox locally.  A few of the new HP workstations were tested and based on the pricing and performance, we selected the HP t620PLUS.  They are blazingly fast to run the NX client piece, and also can run Firefox very quickly and well.  Money was available to buy about 180 of them, which would replace about 1/3 of our total number of deployed workstations.  So the last few months were spent receiving, unpacking, and deploying them to the users that needed them the most.  Feedback has been positive so far on these workstations and they are working very well.

Old Worktation RetroFit

400 or so older workstation will not be upgraded for another 12 months because of funding, so time was spent optimizing them to run with these latest advancements.  A workstation build was created that is identical to the one on the t620PLUS model in appearance.  The 5725 model cannot run a local browser, it's just too slow.  But it connects nicely to the NX GNOME server and performance is better than it was with Xwindows.  The 5745 model can run Firefox locally -- not as well as the new workstations, but well enough that it works and videos do work and play.  When users move around through the City on these three models, they look and work almost the same in all regards.  In the coming 30-60 days, these 400 will be moved off of Xwindows and over to NX.

ChromeBook Testing

NX supports Linux, Mac, MS Windows and tablets via a client piece.  All have been tested and are in use in various parts of the City.  Another login method is available.   NX supports logging in with just a browser.  I have been testing this with a Chromebook with success -- it's very fast and all of our software runs well.  The prospect of being able to have a mobile solution while using $250 devices with a laptop footprint is very attractive and potentially will offer a great amount of dollar savings.  I'll blog more about this in the coming weeks.

LibreOffice

Yup, we're still using LibreOffice!  Many thousands of documents a day are touced with this software and it does the job nicely.   The QA guys have been wonderful, and helped teach me how to bibisect bugs.  When a bug is found here that impacts us greatly, I can now do the leg work to find the regression and the developers have been wonderful in creating patches quickly.  About 200 users are now using version 5.0 with no known issues.  The rest of the users will be migrated in the coming weeks as part of receiving upgraded workstation builds and being migrated to the new GNOME server. 

Firefox Delivery

In the past, we had a server running GNOME and when a user clicked on Firefox, it handed that process off to another server and Firefox then remote displayed back to the workstation.  This met our needs for many years.  When using NX as the transport however, having Firefox running on its own server meant that there was an Xwindow hop in the middle.  Because of the network hungry nature of Firefox, this application was moved and now runs directly on the same server as GNOME/NX.  This gives Firefox direct access to the NX/Xserver with no hop in the middle.  Firefox therefore is very much faster, scrolling and typing is far superior.  This also meant that our scaling and loads have changed and required tuning and in the coming weeks some load balancing.  The server version of Firefox is used for all aspects of user requirements, except for video playback which is now handled by launching the Firefox version found on the local workstation. 



In Progress Projects

A lot of the ongoing projects have been mentioned in these prior paragraphs.  My top action items in the coming weeks:

* Continue moving more users to NX technology
* Tune and monitor the servers as the user loads increase
* Upgrade the NX4 technology to NX5
* Install and deploy the NX Cloud server piece, so users can log in with web browsers
* Add a second NX node, so that we have load balancing and can increase user counts
* Work on project to allow for embed of Youtube videos into LibreOffice for our employees and return the source to the community
* Continue working on our in-house support software and adding various features that have been scheduled.

Very kind regards to all of the people that ask me about our deployment even after all of these years.  It's all still working, and continues to provide significant cost savings.

7 comments:

Jared said...

Very interesting. You said you're using NX4 right now. Have you moved to Gnome 3? If so, how has the compatibility been? I know older NX versions had problems with compositing window managers and NoMachine claims to have added support in version 4.

Anonymous said...

Have you considered/tested to use a Intel NUC as a client machine?

Anonymous said...

What about using a Raspberry Pi 2 as a client machine?

Anonymous said...

Missed your posts. Good to see davelargo popping up on pgo again! Happy new year!

Dave Richards said...

@Jared: We are still using GNOME 2 at this time. With all of the infrastructure changes, we didn't want to change the desktop environment as well. We are told that GNOME 3 runs with NX now, but have not tested it. One of my first projects of the new year will be to look at replacing the desktop.

@Anonymous: We watch the market and have looked at NUCs. There is a high level of comfort with buying new HPs because of how well their older models have worked for us all these years. HP supports the Linux OS running on this hardware too. Also, they boot from a flash device and then run as a live distro -- so there are no hard drives to fail. Very stable and very fast.

@Anonymous: Raspberry Pi, similar thought to the previous response. Cool hardware, always under consideration. I suspect that in 2016, we will buy a few devices for R&D. We're also going to try out a Chrome Box from HP to see how they work.

@Anonymous: Thanks for the kind words, we're here and testing new technology all the time. I really will try and post more often.

Anonymous said...

Great to see you're posting again. I suspect I am missing something obvious, but what was the use case for video in Libreoffice?

Albert Jhon said...

HP Z1 G3 Workstation HP introduced world’s first workstation all-in-one: the new HP Z1 G3 Workstation, combining the simplicity of an All-in-One design along with performance typically found in traditional tower workstations.