Lots of progress in testing this concept, and I'll blog about it next year.
Happy New Year.
Tuesday, December 31, 2013
Quick Network Change Diagram
I made a quick diagram to illustrate to a few people the new design that we are testing whereby NX technology is used to deploy to our workstations. I thought it might be a good visual tool to clarify my last blog. Instead of using the Xserver and Pulse daemon on the thin clients, everything is passed through the backend computer server running NX/GNOME. This allows for a stateless connection that can be resumed on any hardware that runs the NX client and any browser.
Lots of progress in testing this concept, and I'll blog about it next year.
Happy New Year.
Lots of progress in testing this concept, and I'll blog about it next year.
Happy New Year.
Thursday, December 26, 2013
Thinning Thin Clients, And Other Projects
I have not published an update in a good while, but things have been busy. Here are the things that I have worked on since the last blog:
Thinning Thin Clients
How the heck can you thin a thin client? We'll it's my current project and things are progressing fairly well. Since we started using thin clients in the mid-1990s, we have always used remote X as the transport. It's elegant and fast for our needs, and consumes almost no bandwidth on a modern network. Our current design has been wonderful for having roaming profiles. You can log in anywhere in the City, and because everyone is running from a centralized host all of your software and files are immediately available for use. If you go home and log in using NX, the same is true.
However this has one design issue, you have to log out of the first location before starting another session. If users forget to log out from the first location, they are able to "steal" the session which severs the X connection abruptly and is not ideal when they damage settings or lose some of their work. In general all of this is working very well and people move around the City all the time. With the advent of tablets and more mobility, users are wanting to "resume" sessions over different networks of various speeds, including WiFi. In order to accomplish this goal, we're testing using NX technology for all sessions. Using NX thins a thin client, because now it's 100% running in the data center and the workstation is used only for mouse and keyboard; the Xserver is no longer remotely running on the thin client.
What this means for end users is that when a second server instance is started with their user account, they'll be able to "resume" the session from one location or device to another. You can start typing a document at your desk, walk to a meeting and start up a tablet and resume the session and continue on the new footprint.
This change is presenting some scaling issues. In the past some load was offloaded to the workstations, especially in the case of memory consumption. Now all of this must be moved into the data center, which means bigger and faster servers with more cores. We already have money in our budget to replace the GNOME server in 2014, so the timing could not be better.
We have two HP thin client models in use at the City. The t5725 and t5745, both of which are discontinued. We ordered the new t610 model which came with Ubuntu 10 installed. I started the formal process in recent weeks of staging a build for production use. Customizations were required to order to accommodate the older models. The 5745 uses the intel driver and the 5725 is running ati. So the build was modified to detect thin client model at first boot and set up the xorg.conf appropriately. I was making great progress, when HP released an upgrade for the t610 to Ubuntu 12. So I created a tarball of the customizations and in a few hours, they were all working on the new operating system.
More and more users are doing Skype interviews and in the past they would just check out a laptop and use that from their desk. Since Skype is available for Linux, we're testing the concept of adding that feature right on the thin clients. So far it's working like a champ.
In a nutshell, all three models of the thin clients boot Ubuntu 12 very quickly, and start up a very basic FVWM desktop which offers them the ability to connect to our servers using the NX Client. I am in the process of tuning the build for the three models and I'm working with NoMachine on some issues to make the NX Client work better in our environment.
Here is a shot of the current alpha build. FVWM provides menu and window management for software that connect to the servers. Rdesktop, Skype and NX Clients run as siblings. Full GNOME session is running inside the NX Client.
Firefox
After a quick patch in Firefox 25 to allow files downloaded by users to honor umask, it did not land in Firefox 26 and will return in Firefox 27. So we're going to skip a version, but the good news is that with Firefox that means only waiting a few weeks.
LibreOffice
We jumped on LibreOffice 4.1 at 4.1.1 to solve some issues and improve file filters versus 4.0. Out of 800 users, about 20 had to be rolled back for various bugs, which is normal and expected. With release 4.1.4 we have been able to finally get everyone off of 4.0. Things seem stable with this release too. The server reports when users kill their software, and I'm not seeing many with LO. Very cool. Looking ahead to 4.2, there is a bad bug for us that prohibits us from testing heavily. It's here. We have lots of documents that make use of Nimbus Sans and the font currently is not rendering correctly.
Thinning Thin Clients
How the heck can you thin a thin client? We'll it's my current project and things are progressing fairly well. Since we started using thin clients in the mid-1990s, we have always used remote X as the transport. It's elegant and fast for our needs, and consumes almost no bandwidth on a modern network. Our current design has been wonderful for having roaming profiles. You can log in anywhere in the City, and because everyone is running from a centralized host all of your software and files are immediately available for use. If you go home and log in using NX, the same is true.
However this has one design issue, you have to log out of the first location before starting another session. If users forget to log out from the first location, they are able to "steal" the session which severs the X connection abruptly and is not ideal when they damage settings or lose some of their work. In general all of this is working very well and people move around the City all the time. With the advent of tablets and more mobility, users are wanting to "resume" sessions over different networks of various speeds, including WiFi. In order to accomplish this goal, we're testing using NX technology for all sessions. Using NX thins a thin client, because now it's 100% running in the data center and the workstation is used only for mouse and keyboard; the Xserver is no longer remotely running on the thin client.
What this means for end users is that when a second server instance is started with their user account, they'll be able to "resume" the session from one location or device to another. You can start typing a document at your desk, walk to a meeting and start up a tablet and resume the session and continue on the new footprint.
This change is presenting some scaling issues. In the past some load was offloaded to the workstations, especially in the case of memory consumption. Now all of this must be moved into the data center, which means bigger and faster servers with more cores. We already have money in our budget to replace the GNOME server in 2014, so the timing could not be better.
We have two HP thin client models in use at the City. The t5725 and t5745, both of which are discontinued. We ordered the new t610 model which came with Ubuntu 10 installed. I started the formal process in recent weeks of staging a build for production use. Customizations were required to order to accommodate the older models. The 5745 uses the intel driver and the 5725 is running ati. So the build was modified to detect thin client model at first boot and set up the xorg.conf appropriately. I was making great progress, when HP released an upgrade for the t610 to Ubuntu 12. So I created a tarball of the customizations and in a few hours, they were all working on the new operating system.
More and more users are doing Skype interviews and in the past they would just check out a laptop and use that from their desk. Since Skype is available for Linux, we're testing the concept of adding that feature right on the thin clients. So far it's working like a champ.
In a nutshell, all three models of the thin clients boot Ubuntu 12 very quickly, and start up a very basic FVWM desktop which offers them the ability to connect to our servers using the NX Client. I am in the process of tuning the build for the three models and I'm working with NoMachine on some issues to make the NX Client work better in our environment.
Here is a shot of the current alpha build. FVWM provides menu and window management for software that connect to the servers. Rdesktop, Skype and NX Clients run as siblings. Full GNOME session is running inside the NX Client.
Firefox
After a quick patch in Firefox 25 to allow files downloaded by users to honor umask, it did not land in Firefox 26 and will return in Firefox 27. So we're going to skip a version, but the good news is that with Firefox that means only waiting a few weeks.
LibreOffice
We jumped on LibreOffice 4.1 at 4.1.1 to solve some issues and improve file filters versus 4.0. Out of 800 users, about 20 had to be rolled back for various bugs, which is normal and expected. With release 4.1.4 we have been able to finally get everyone off of 4.0. Things seem stable with this release too. The server reports when users kill their software, and I'm not seeing many with LO. Very cool. Looking ahead to 4.2, there is a bad bug for us that prohibits us from testing heavily. It's here. We have lots of documents that make use of Nimbus Sans and the font currently is not rendering correctly.
Tuesday, October 29, 2013
First Alfresco Upgrade, LibreOffice Work Around, Firefox 25
We had been testing Alfresco Community 4.2.c for a good while and getting positive feedback from our users. Version 4.2.e came out and appears to be the version that will release as "4.2". Naively, I thought there would be a ./install type upgrade with the latest code. Alfresco upgrades are really more of a dump and reload, which increased the complexity of this greatly. We decided to allocate some better hardware for this pilot project and for me to take the time to teach myself this process. In a nutshell, you dump all of the data from Postgres and then it is restored on the new hardware. You then bring over the alf_data directory and start up the daemons and hope it all works. After a few test runs, I worked out all of the kinks and it came over and has been made available to end users.
The LibreOffice 4.1 CMIS connector sadly no longer works with Alfresco. I brought down LO 4.2 (Alpha) which has been resynced with the latest libcmis changed and it works like a champ, better than ever. I have "upgraded" 5 of us to this release and we are going to begin testing saving and editing documents directly via LibreOffice -> Alfresco. Response time and folder refreshes are much faster and the library changes are noticed and appreciated. Let's see what feedback we get from the end users.
In the shot below, the new icons and art with Alfresco 4.2.e, and LibreOffice 4.2 making direct edits to files stored within the database.
Firefox 25 is out and fixes the issue of not honoring umask from the parent shell. 5 minutes after download, it was installed and live and we were fully patched. 800 users will be very happy about their file permissions.
Current Projects: I'm looking over Ubuntu 10 loaded on the HP t610 thin clients and checking to see if it can be easily loaded onto 5745 and 5725 model thin clients. This will continue with our design of having the same operating system on all workstations. I expect it to work, and easily. Testing NX 4 client and server with the concept of using NX between all workstations and the server.
The LibreOffice 4.1 CMIS connector sadly no longer works with Alfresco. I brought down LO 4.2 (Alpha) which has been resynced with the latest libcmis changed and it works like a champ, better than ever. I have "upgraded" 5 of us to this release and we are going to begin testing saving and editing documents directly via LibreOffice -> Alfresco. Response time and folder refreshes are much faster and the library changes are noticed and appreciated. Let's see what feedback we get from the end users.
In the shot below, the new icons and art with Alfresco 4.2.e, and LibreOffice 4.2 making direct edits to files stored within the database.
Firefox 25 is out and fixes the issue of not honoring umask from the parent shell. 5 minutes after download, it was installed and live and we were fully patched. 800 users will be very happy about their file permissions.
Current Projects: I'm looking over Ubuntu 10 loaded on the HP t610 thin clients and checking to see if it can be easily loaded onto 5745 and 5725 model thin clients. This will continue with our design of having the same operating system on all workstations. I expect it to work, and easily. Testing NX 4 client and server with the concept of using NX between all workstations and the server.
Wednesday, October 16, 2013
What's Happening?
Yet again a good number of days have gone by without a blog posting. Projects and upgrades have consumed most of my time in the last few weeks.
NX/NoMachine 4 Released & Tablets
NX 4 was released. I had installed and looked over various beta builds as it was being developed, and I put the released version on a VM copy of your primary GNOME desktop server. The install process is a piece of cake, and you just edit a few .cfg files to point to the right scripts to run at startup and away you go.
One notable feature is the ability to connect with a browser and get a full GNOME desktop and run all of your software without modification. As seen below, you enter a URL and up came the desktop. Speed on Firefox 24/Linux is very acceptable.
This browser solution has not proven itself to work well on tablets, even over local WiFi. But I was happy to see this on the download page:
Tablets have been interesting to me because one school of thought is that people would log in and get a GNOME desktop and work as they do from their desk. Another thought is that they would work more client/server and download documents and use local software and then upload. By far users have expressed more interest in the former. They don't seem to want to use different/local software and want to use what works here at the City. So the clients for iOS and Android will be greatly appreciated and used.
The biggest issue for us right now with NX 4 is that the clients are not yet ready for deployment in a multi-user environment with "regular users". NX3 clients allowed you to automate the login process and once they entered their account/password...GNOME came right up. NX 4 has a lot of prompts and questions which are better suited for power users. As it stands right now, we would be bombarded with questions and support and there would be a drop in productivity. If a user is attempting to connect, they have work that must be finished. If they fail, that work doesn't get done. There are plans on the roadmap from what I understand to add more of these entperprise features in the future, and we'll have to wait for them before we can deploy.
EDIT: Sarah from Nomachine posted information on this issue on the comments section. We were unaware and are going to test.
Support Portal Has More Features
Our in-house support portal is still helping us monitor all aspects of a thin client and centralized environment, and some modifications have been made. It's wonderful that when you have ideas, you can just throw in a few lines of code and it's done.
We're using the issue tracking module now for all calls received and the data is very interesting. In the shot below you can see that in the last 24 hours our primary support call center got around 20 calls. This is with many hundreds of users logging in and out of the workstations and using software. Our current regular concurrent load is right around 300 users. I have said it many times before, that using thin clients allows your staff to work on maintenance and future projects vs spending lots of resources keeping support intensive PCs running. I marked the general trends of problems
(RED) Linux/GNOME/Desktop problems Only 1 all day!
(GREEN) Phones
(PURPLE) Problems with hosted/cloud apps
(BLUE) Problems with MS Windows on various specialized computers
(BLACK) End user questions about working software or procedures.
I also hacked in a quick screen on the user detail area to show documents edited by the user. Very often people call and have "lost" their document...or sometimes they make edits to a document in /tmp accidentally and then it's lost when this area is purged. Now at a glance we can assist them without having to look on backup tapes and such. Very helpful.
LibreOffice 4.1 , Handling Rollbacks
This isn't unique to LibreOffice 4.1, because it's happened going back all the way back to OpenOffice 1.1. Whenever you get a patch or even more so, whenever you get a major release there are always issues where documents don't work correctly. I keep all of the old versions on the server and can always test and find the regression. Standard procedure then is to rollback a user until we get a patch and then upgrade them again to the latest release. This was done by me, with hard coded scripts. I put this into the support portal and now anyone in IT can either roll their own account back or rollback another user to solve these types of problems. All of this activity is logged, and we're nagged once a day of people on prior versions so that we continue to try and resolve their issue.
We have three bad issues with LO 4.1 right now, affecting about a dozen people. 1) There is a bug with building graphs from spreadsheet cells 2) There is a bug for selecting sections of spreadsheets for printing 3) There is a bug with drop down cells in Calc.
These users are on 4.0 and everyone is happy and working.
Firefox 24 Bug And Quick Response
Firefox 23 had an issue with saving files, the files were saved as 600 on Linux and no longer honored umask. When then happened was that work flow for many people was interrupted because as they shared files with others, they could not be opened. In general users do not understand how to use file permissions nor how to change them. The Firefox developers came through in a pinch and a patch is already landing in time for version 25. Thanks guys.
Alfresco Testing Continues.
We have three non IT people starting to test Alfresco 4.2 as a proof of concept. Deployment is not imminent at this point because we don't have the staffing; but it's gaining some steam and people are starting to see the power of working in this manner. The testers are putting PDF content into the Alfresco database and then seeing that it's immediately available on their tablets. We're meeting today again to discuss how to connect with LibreOffice, and will talk about the concept of using hybrid PDF as our official City file format. Right now LO is a bit clunky with this format, but we see possibilities I can start creating feature requests and possibly funding improvements.
Alfresco supports RSS feeds, so I have been building a proof of concept systray application which would alert users of changes to documents and work flow that need their attention. In the shot below, the icon is sitting in the tray and clicking on the icon shows the last document updates. At our meeting today we'll talk about this idea and whether we want to test notification popups too. The strength of open source: being able to rapidly build prototypes and then deploy solutions quickly with little to no cost.
SuseCon Orlando
It's only a few weeks away, and I'll be attending along with my coworker. Find us and say Hi!
NX/NoMachine 4 Released & Tablets
NX 4 was released. I had installed and looked over various beta builds as it was being developed, and I put the released version on a VM copy of your primary GNOME desktop server. The install process is a piece of cake, and you just edit a few .cfg files to point to the right scripts to run at startup and away you go.
One notable feature is the ability to connect with a browser and get a full GNOME desktop and run all of your software without modification. As seen below, you enter a URL and up came the desktop. Speed on Firefox 24/Linux is very acceptable.
This browser solution has not proven itself to work well on tablets, even over local WiFi. But I was happy to see this on the download page:
The biggest issue for us right now with NX 4 is that the clients are not yet ready for deployment in a multi-user environment with "regular users". NX3 clients allowed you to automate the login process and once they entered their account/password...GNOME came right up. NX 4 has a lot of prompts and questions which are better suited for power users. As it stands right now, we would be bombarded with questions and support and there would be a drop in productivity. If a user is attempting to connect, they have work that must be finished. If they fail, that work doesn't get done. There are plans on the roadmap from what I understand to add more of these entperprise features in the future, and we'll have to wait for them before we can deploy.
EDIT: Sarah from Nomachine posted information on this issue on the comments section. We were unaware and are going to test.
Support Portal Has More Features
Our in-house support portal is still helping us monitor all aspects of a thin client and centralized environment, and some modifications have been made. It's wonderful that when you have ideas, you can just throw in a few lines of code and it's done.
We're using the issue tracking module now for all calls received and the data is very interesting. In the shot below you can see that in the last 24 hours our primary support call center got around 20 calls. This is with many hundreds of users logging in and out of the workstations and using software. Our current regular concurrent load is right around 300 users. I have said it many times before, that using thin clients allows your staff to work on maintenance and future projects vs spending lots of resources keeping support intensive PCs running. I marked the general trends of problems
(RED) Linux/GNOME/Desktop problems Only 1 all day!
(GREEN) Phones
(PURPLE) Problems with hosted/cloud apps
(BLUE) Problems with MS Windows on various specialized computers
(BLACK) End user questions about working software or procedures.
I also hacked in a quick screen on the user detail area to show documents edited by the user. Very often people call and have "lost" their document...or sometimes they make edits to a document in /tmp accidentally and then it's lost when this area is purged. Now at a glance we can assist them without having to look on backup tapes and such. Very helpful.
LibreOffice 4.1 , Handling Rollbacks
This isn't unique to LibreOffice 4.1, because it's happened going back all the way back to OpenOffice 1.1. Whenever you get a patch or even more so, whenever you get a major release there are always issues where documents don't work correctly. I keep all of the old versions on the server and can always test and find the regression. Standard procedure then is to rollback a user until we get a patch and then upgrade them again to the latest release. This was done by me, with hard coded scripts. I put this into the support portal and now anyone in IT can either roll their own account back or rollback another user to solve these types of problems. All of this activity is logged, and we're nagged once a day of people on prior versions so that we continue to try and resolve their issue.
We have three bad issues with LO 4.1 right now, affecting about a dozen people. 1) There is a bug with building graphs from spreadsheet cells 2) There is a bug for selecting sections of spreadsheets for printing 3) There is a bug with drop down cells in Calc.
These users are on 4.0 and everyone is happy and working.
Firefox 24 Bug And Quick Response
Firefox 23 had an issue with saving files, the files were saved as 600 on Linux and no longer honored umask. When then happened was that work flow for many people was interrupted because as they shared files with others, they could not be opened. In general users do not understand how to use file permissions nor how to change them. The Firefox developers came through in a pinch and a patch is already landing in time for version 25. Thanks guys.
Alfresco Testing Continues.
We have three non IT people starting to test Alfresco 4.2 as a proof of concept. Deployment is not imminent at this point because we don't have the staffing; but it's gaining some steam and people are starting to see the power of working in this manner. The testers are putting PDF content into the Alfresco database and then seeing that it's immediately available on their tablets. We're meeting today again to discuss how to connect with LibreOffice, and will talk about the concept of using hybrid PDF as our official City file format. Right now LO is a bit clunky with this format, but we see possibilities I can start creating feature requests and possibly funding improvements.
Alfresco supports RSS feeds, so I have been building a proof of concept systray application which would alert users of changes to documents and work flow that need their attention. In the shot below, the icon is sitting in the tray and clicking on the icon shows the last document updates. At our meeting today we'll talk about this idea and whether we want to test notification popups too. The strength of open source: being able to rapidly build prototypes and then deploy solutions quickly with little to no cost.
SuseCon Orlando
It's only a few weeks away, and I'll be attending along with my coworker. Find us and say Hi!
Thursday, August 01, 2013
HP t610 Thin Client R&D Project
I haven't talked about desktop hardware for a good while. Years ago we bought several hundred HP t5725 workstations. They're still production and working great, and are projected to run their full 10 year duty cycle. HP replaced the t5725 with the t5745 model which offered better performance and 1Gb networking. We have purchased around 50 of the t5745s and given them to users that need faster performance; such as those that use Google Earth and large PDF files. They too should provide a 10 year duty cycle.
In recent months the t5745 was discontinued and replaced with the t610. Specs are here. In short, it's an AMD Dual-Core T56N APU with Radeon HD 6320 Graphics 1.65 GHz running the Ubuntu operating system. We purchased one device for around $400 dollars. My first inclination of course was to open it up and look under the hood. Pictures and comments are below.
In the coming weeks, I'm going to install our custom modifications into the base OS and get it working the same as the earlier models.
I've spoken about thin clients many times, but wanted to mention again how incredibly cost effective and stable they are to run for enterprise use. A small City of 10 employees would not reap benefits, but as you get into the hundreds of users, the savings are clear. We have about 560 of them deployed around various City buildings. $400 purchase price with a 10 year duty cycle yields $40 per year (hardware only) desktop costs. When one of them dies, the users have nothing saved locally and our support group just walks down and replaces the hardware. The user is back up again in just a few minutes with no loss of productivity.
Here is the inside of the t610, the heat from the CPU is moved away with copper and vents. No moving parts, no fans. The blue module in the upper left hand corner is the flash "hard drive". These can be 1GB or 2GB and are cheap and easily upgraded.
Lots of ports on the front, nice design.
More ports and some legacy connections on the back. All of our hardware should connect and work.
The entire case has no screws and pops apart with just clips. Here it is back together.
In recent months the t5745 was discontinued and replaced with the t610. Specs are here. In short, it's an AMD Dual-Core T56N APU with Radeon HD 6320 Graphics 1.65 GHz running the Ubuntu operating system. We purchased one device for around $400 dollars. My first inclination of course was to open it up and look under the hood. Pictures and comments are below.
In the coming weeks, I'm going to install our custom modifications into the base OS and get it working the same as the earlier models.
I've spoken about thin clients many times, but wanted to mention again how incredibly cost effective and stable they are to run for enterprise use. A small City of 10 employees would not reap benefits, but as you get into the hundreds of users, the savings are clear. We have about 560 of them deployed around various City buildings. $400 purchase price with a 10 year duty cycle yields $40 per year (hardware only) desktop costs. When one of them dies, the users have nothing saved locally and our support group just walks down and replaces the hardware. The user is back up again in just a few minutes with no loss of productivity.
Here is the inside of the t610, the heat from the CPU is moved away with copper and vents. No moving parts, no fans. The blue module in the upper left hand corner is the flash "hard drive". These can be 1GB or 2GB and are cheap and easily upgraded.
Lots of ports on the front, nice design.
More ports and some legacy connections on the back. All of our hardware should connect and work.
The entire case has no screws and pops apart with just clips. Here it is back together.
Tuesday, July 30, 2013
Lots Of Updates
I just noticed again that it's been a while since my last blog. Things have been busy at always, some of my projects lately were not so conductive to screenshots and technical thoughts. But a few things might be of interest.
SuseCon 2013
First up I got permission and funding to attend SuseCon in Orlando this fall. It's always nice to meet people and have hallway conversations. Find me and say Hi if you'd like to converse. The conference last year was very productive!
Alfresco Testing & Analysis
I have been working with the IT Director on testing and exploring ways to deploy Alfresco, and content management in general. Historically we have always had silos of data that were not highly shared, and it's our hope that we can move in another direction with this software. We have been exploring ideas on the best way to build the top level Sites and then how content should be stored. If several departments are working on a City project, our information should be shared without duplication of effort and documents. We're making some headway on ideas and believe that we have built the best possible top level structure. We are also considering the staffing requirements of this type of deployment and impact to other City departments. Lots of testing and R&D ahead on this project. The software itself and mobile devices are working great.
Support Portal, Additional Logging
We have not been able to add additional staff for a while, so I have been trying to get more and more data that is logged to flat files on our servers into the hands of our support staff. This allows all of IT to see issues that normally would have meant checking the various files manually and understanding how to scan through logs. The screens are rough, but working and are allowing us to be very proactive on seeing problems. The blue area shows users logging in and out of the servers. The purple area is showing us users that have logged in the most often in the last few days. Usually they are not having problems, but sometimes this indicates someone that is struggling with errant hardware or network connections. Remember, that very often end users will never call for help and will continue to struggle. Seeing this information allows us to call them and offer help. The dark green section is showing users that logged in multiple times without first logging off. This sometimes means they have simply powered off their workstations, or are losing power and have a bad UPS. The red section shows users logging into NX from the Internet. The light green section shows users that had authentication failures with Zimbra. The black section will show users that entered the wrong passwords into the server and those users that entered the wrong passwords into their screensavers.
Support Portal, Creating Issues
The portal is aware of a lot of issues and sees problems on the network. So it was a very simple step to just create a basic issue tracking module. In the various tabs of monitoring, there is a button marked [ Create Issue ] that takes the highlighted item and automatically generates an issue and allows us to add notes. These screens are so simple to create with Glade and python, that even as it stands now it's increasing our ability to assist users. This development is not a fulltime project, so I'm sure the UI will not be scrutinized. :)
The shot below shows issues that were generated automatically from the various log files. The portal knows who, where and what -- We just needed a way to track status and add notes. The red section shows a summary of the open issues. The black section is a very simple plugin system that allows for summary of the data.
And the issue detail screen allows for very basic tracking and note taking.
Looking For Monitor Problems
We have been seeing a few issues a day where users are having problems with monitor resolutions and some issues where software seems to put the Xserver out of sync with the monitor. The first step was to write a little ksh script that fires at login, and basically compares their Xorg file against what xrandr -q reports. In many cases X just kind of fixes itself and works, but in some cases the users will have problems. So we're slowly cleaning up these issues and getting everything configured correctly. Each little 1% improvement that you get counts and reduces user frustration. In the issue detail above, the portal detected that a user had dual monitors that were not configured in the right way; one supported 1920x1200/1920x1080 and the other only supported 1920x1080. Without this centralized software and a server based solution, this would be very time consuming to troubleshoot.
Other projects: Testing NX 4 preview, prepping for Zimbra patches and upgrades, testing LibreOffice 4.1 and prepping for deployment, continued patches and testing for Firefox, Flash and Java,.
SuseCon 2013
First up I got permission and funding to attend SuseCon in Orlando this fall. It's always nice to meet people and have hallway conversations. Find me and say Hi if you'd like to converse. The conference last year was very productive!
Alfresco Testing & Analysis
I have been working with the IT Director on testing and exploring ways to deploy Alfresco, and content management in general. Historically we have always had silos of data that were not highly shared, and it's our hope that we can move in another direction with this software. We have been exploring ideas on the best way to build the top level Sites and then how content should be stored. If several departments are working on a City project, our information should be shared without duplication of effort and documents. We're making some headway on ideas and believe that we have built the best possible top level structure. We are also considering the staffing requirements of this type of deployment and impact to other City departments. Lots of testing and R&D ahead on this project. The software itself and mobile devices are working great.
Support Portal, Additional Logging
We have not been able to add additional staff for a while, so I have been trying to get more and more data that is logged to flat files on our servers into the hands of our support staff. This allows all of IT to see issues that normally would have meant checking the various files manually and understanding how to scan through logs. The screens are rough, but working and are allowing us to be very proactive on seeing problems. The blue area shows users logging in and out of the servers. The purple area is showing us users that have logged in the most often in the last few days. Usually they are not having problems, but sometimes this indicates someone that is struggling with errant hardware or network connections. Remember, that very often end users will never call for help and will continue to struggle. Seeing this information allows us to call them and offer help. The dark green section is showing users that logged in multiple times without first logging off. This sometimes means they have simply powered off their workstations, or are losing power and have a bad UPS. The red section shows users logging into NX from the Internet. The light green section shows users that had authentication failures with Zimbra. The black section will show users that entered the wrong passwords into the server and those users that entered the wrong passwords into their screensavers.
Support Portal, Creating Issues
The portal is aware of a lot of issues and sees problems on the network. So it was a very simple step to just create a basic issue tracking module. In the various tabs of monitoring, there is a button marked [ Create Issue ] that takes the highlighted item and automatically generates an issue and allows us to add notes. These screens are so simple to create with Glade and python, that even as it stands now it's increasing our ability to assist users. This development is not a fulltime project, so I'm sure the UI will not be scrutinized. :)
The shot below shows issues that were generated automatically from the various log files. The portal knows who, where and what -- We just needed a way to track status and add notes. The red section shows a summary of the open issues. The black section is a very simple plugin system that allows for summary of the data.
And the issue detail screen allows for very basic tracking and note taking.
Looking For Monitor Problems
We have been seeing a few issues a day where users are having problems with monitor resolutions and some issues where software seems to put the Xserver out of sync with the monitor. The first step was to write a little ksh script that fires at login, and basically compares their Xorg file against what xrandr -q reports. In many cases X just kind of fixes itself and works, but in some cases the users will have problems. So we're slowly cleaning up these issues and getting everything configured correctly. Each little 1% improvement that you get counts and reduces user frustration. In the issue detail above, the portal detected that a user had dual monitors that were not configured in the right way; one supported 1920x1200/1920x1080 and the other only supported 1920x1080. Without this centralized software and a server based solution, this would be very time consuming to troubleshoot.
Other projects: Testing NX 4 preview, prepping for Zimbra patches and upgrades, testing LibreOffice 4.1 and prepping for deployment, continued patches and testing for Firefox, Flash and Java,.
Tuesday, June 18, 2013
Zimbra, Alfresco, LibreOffice and Firefox
Once again it's been a while since I blogged, but I have been keeping very busy.
Zimbra
Zimbra has been running very well, and aside from a few bugs the migration has gone well. On June 15th, we completely disabled Evolution and Groupwise. We had been on Groupwise since 1994 and used Evolution for many years. In recent years we just could not elevate these products past nagging bugs, even under a support contract with Novell. So with mixed feelings we have now moved to this new software. I have seen some blogs about the future of Zimbra, and it seems very bright.
Firefox
We are having one nagging issue with Firefox. For many users (including me), about once a day it just closes/crashes. I have been looking for a pattern or trend, and it seems to happen when a page is about to be displayed during a refresh; but not always. The normal crash dump screen never appears with useful information. I put a running FF instance on gdb, and it finally crashed. However, it's crashing with a SIGKILL for some reason which so far alludes me. Does anyone know why it would stop with SIGKILL? I looked at the logs and this doesn't appear to be a OOM type issue, the server is running fine and not overly loaded. Nothing appears in the logs at all. Is there code in FF that tries to shut down tabs that are running slowly, and maybe this area is buggy? Any ideas on how to get a proper crash dump with this signal? If I cannot find it, I'm going to start upgrading various pieces including the underlying operating system which is OpenSuse 11.4.
Alfresco (Document Management)
I have been working on setting up an instance of Alfresco for the IT department to use in the coming months. We built a sandbox several months ago for a demo and decided to proceed with this pilot project. Version 4.2c installed easily on Ubuntu Linux and was up and running pretty quickly. I was able to figure out how to get user accounts in LDAP to auto-provision, and am shaking out what appear to be bugs in LDAP group syncing. Once you get used to the design of Alfresco for housing documents, you can see the power of the software. The web interface is very clean and customizable.
There is an application for iPod/iPhone that connects to the database and displays your content on that hardware footprint:
And they have an application for iPad tablet footprint too:
So right now we're testing the workflow and taking steps in the direction of moving our IT content into this data store. I have some ideas for ways to integrate Alfresco with GNOME and will be blogging about that in the future.
LibreOffice
We have hit one small snag with LibreOffice, with workaround. This issue is relevant with OpenOffice too, and it's a bit unique to our setup. The new SVG rendering engine is VERY slow over remote X, which mostly impacts users Impress slide shows. Presentations that used to open in a second now take much longer. The workaround is to open the presentation and save the background in PNG format and then it works as expected. Not a very good end user experience.
As part of the Alfresco project I have been testing CMIS connections from LibreOffice, and they have advanced a lot since my last review. Within the file picker, if you select the Alfresco connection it asks for a password:
Once you pick a file, it now asks if you want to check the file out:
And when you are done, it asks if you want to check it back in again. It also allows you to enter a comment. The document version number is incremented and this comment appears inside of the web interface.
I put a few beta testers on LO 4.1 to check out the new side bar code. This is "experimental" at this point, but looks like it will be a big hit with our end users when it's released for LO 4.2. Very cool.
Other projects: Patches and upgrades to install, already a new Flash upgrade to QA and deploy. Continued Alfresco work, setting up software for our support staff to put users into LDAP groups with a single click.
Zimbra
Zimbra has been running very well, and aside from a few bugs the migration has gone well. On June 15th, we completely disabled Evolution and Groupwise. We had been on Groupwise since 1994 and used Evolution for many years. In recent years we just could not elevate these products past nagging bugs, even under a support contract with Novell. So with mixed feelings we have now moved to this new software. I have seen some blogs about the future of Zimbra, and it seems very bright.
Firefox
We are having one nagging issue with Firefox. For many users (including me), about once a day it just closes/crashes. I have been looking for a pattern or trend, and it seems to happen when a page is about to be displayed during a refresh; but not always. The normal crash dump screen never appears with useful information. I put a running FF instance on gdb, and it finally crashed. However, it's crashing with a SIGKILL for some reason which so far alludes me. Does anyone know why it would stop with SIGKILL? I looked at the logs and this doesn't appear to be a OOM type issue, the server is running fine and not overly loaded. Nothing appears in the logs at all. Is there code in FF that tries to shut down tabs that are running slowly, and maybe this area is buggy? Any ideas on how to get a proper crash dump with this signal? If I cannot find it, I'm going to start upgrading various pieces including the underlying operating system which is OpenSuse 11.4.
Alfresco (Document Management)
I have been working on setting up an instance of Alfresco for the IT department to use in the coming months. We built a sandbox several months ago for a demo and decided to proceed with this pilot project. Version 4.2c installed easily on Ubuntu Linux and was up and running pretty quickly. I was able to figure out how to get user accounts in LDAP to auto-provision, and am shaking out what appear to be bugs in LDAP group syncing. Once you get used to the design of Alfresco for housing documents, you can see the power of the software. The web interface is very clean and customizable.
There is an application for iPod/iPhone that connects to the database and displays your content on that hardware footprint:
And they have an application for iPad tablet footprint too:
So right now we're testing the workflow and taking steps in the direction of moving our IT content into this data store. I have some ideas for ways to integrate Alfresco with GNOME and will be blogging about that in the future.
LibreOffice
We have hit one small snag with LibreOffice, with workaround. This issue is relevant with OpenOffice too, and it's a bit unique to our setup. The new SVG rendering engine is VERY slow over remote X, which mostly impacts users Impress slide shows. Presentations that used to open in a second now take much longer. The workaround is to open the presentation and save the background in PNG format and then it works as expected. Not a very good end user experience.
As part of the Alfresco project I have been testing CMIS connections from LibreOffice, and they have advanced a lot since my last review. Within the file picker, if you select the Alfresco connection it asks for a password:
Once you pick a file, it now asks if you want to check the file out:
And when you are done, it asks if you want to check it back in again. It also allows you to enter a comment. The document version number is incremented and this comment appears inside of the web interface.
I put a few beta testers on LO 4.1 to check out the new side bar code. This is "experimental" at this point, but looks like it will be a big hit with our end users when it's released for LO 4.2. Very cool.
Other projects: Patches and upgrades to install, already a new Flash upgrade to QA and deploy. Continued Alfresco work, setting up software for our support staff to put users into LDAP groups with a single click.
Friday, May 17, 2013
Sharing Files With Employees Via Email
I have been unhooking the last bits of software applications that were connected to Evolution and making them work with Zimbra. I finished all of the places that allowed users to "attach" a file with the drag and drop technique described a few blogs ago. Next up was the feature that allowed them to create a file:// link inside of their email. This was used in order to try and get people to "share" files and not send multiple copies of the same file around the network via email. We're all on the same server, so there isn't ever really a need to send a file to another user inside the City...it's already here and centralized. But many people still are thinking in terms of a PC or USB sticks and feel like that's how you get files from one person to another.
The file:// link was available inside of Evolution, but is now considered an exploit and not really used in browsers anymore. We wanted to replace the functionality however, so I found a solution and it's being tested. I created a new MIME type for files that end with '.lnk" inside of Firefox. Inside of this file is the path to a file located internally. The user is then able to open these linked/shared files without physically sending a copy. This too is very quota friendly.
When users double-click on the file, the normal MIME UI appears. The option circled in green is "Share Via Zimbra"
Using the same UI that was used to solve the LibreOffice temp file issue (described two blogs ago), a link symbol appears. User drags this into their composer window and a .lnk file is attached.
The recipient then clicks on the .lnk file and a box appears alerting them that a file has been shared, and then once opened the appropriate helper application (Evince, Acrobat Reader, LibreOffice, EOG, GIMP) opens based on the file type. I also have circled the new MIME type as it appears in Firefox.
I am logging when users perform this step and it's already being used around the City. This seems to be the simplest way to solve this problem, and hopefully will get more people to begin sharing files until we can move in the direction of a proper document management system.
The file:// link was available inside of Evolution, but is now considered an exploit and not really used in browsers anymore. We wanted to replace the functionality however, so I found a solution and it's being tested. I created a new MIME type for files that end with '.lnk" inside of Firefox. Inside of this file is the path to a file located internally. The user is then able to open these linked/shared files without physically sending a copy. This too is very quota friendly.
When users double-click on the file, the normal MIME UI appears. The option circled in green is "Share Via Zimbra"
Using the same UI that was used to solve the LibreOffice temp file issue (described two blogs ago), a link symbol appears. User drags this into their composer window and a .lnk file is attached.
The recipient then clicks on the .lnk file and a box appears alerting them that a file has been shared, and then once opened the appropriate helper application (Evince, Acrobat Reader, LibreOffice, EOG, GIMP) opens based on the file type. I also have circled the new MIME type as it appears in Firefox.
Tuesday, May 14, 2013
Firefox 21 Changed Plugins Folder
Today was the day to install patches for Firefox and the various plugins, and at first I couldn't get Firefox to see the plugins at all. It turns out they moved the folder and it's not really well documented, and doesn't seem to be in the release notes. Maybe publishing the fix will help someone. Seems like this is going to really break things on distros if users upgrade.
Previously plugins on a multi-user server were in
[base_of_firefox]/firefox/plugins/
Now they are stored in
[base_of_firefox]/firefox/browser/plugins/
Previously plugins on a multi-user server were in
[base_of_firefox]/firefox/plugins/
Now they are stored in
[base_of_firefox]/firefox/browser/plugins/
Tuesday, May 07, 2013
Whew!
I just looked and my last blog was March 27th; The last 1+ month has been very busy. Not a lot of things took place that warranted screenshots or R&D updates. OpenLDAP and Zimbra deployments both occurred and took most of my time. Status updates:
OpenLDAP
The change to centralized passwords went off very well. During my prep work, I created a "Password Sandbox" application that allows you to test various passwords before actually making the change. This proved to be very popular and effective. Part of my time was then allocated monitoring all 800 of our users to ensure they all changed their passwords and sending out daily status reports to work on 100% completion. It's now all done and our passwords have been hardened to current metrics.
Support Portal Application
I have been putting in some patches and features for our support staff. They could see at a glance users with passwords that still had not been changed, and made it easier to create and upload LDIF records. We've reached a milestone with this functionality where it's very simple to create new employees in LDAP. We also can now auto-provision new people in the Zimbra post office with just a single click. Within 10 minutes of checking the button, the user has email.
LDAP Sync To Zimbra.
Zimbra auto provisions email accounts, and I wrote a small python script that runs at 1am to push any updates that occur from OpenLDAP to Zimbra. Phone numbers and job titles change, and all of this information is now synced. Support staff only has to change the information in one place, saving us time.
Zimbra Is Live
After a long ramp up time of testing, we put Zimbra live and it replaced Evolution/Groupwise. It was kind of sad to move away from Evolution, but unfortunately we just could not get bug fixes from Novell/Suse under our support contract in a timely manner. Zimbra was deployed with virtually no problems. We built one server and have about 250 concurrent connections via the web interface and then an equal number of connections from mail-notification(IMAP) on the lower panel to alert them of new messages for a total of 500+ connections. Zimbra is flying and you can barely notice the difference with a full load. Fast and crisp.
In the past our server which provided Firefox would have about 100 concurrent users, now that users are accessing email from a browser this has jumped to about 250. The server took this load without a hiccup and is working very well. With 100 concurrent the machine would be about 10% busy and now it's running about 15%. We scaled this machine with the expectation of greatly increased Firefox loads and it's working like a champ.
And The One GOTCHA
There is always one, right? :) LibreOffice (and OpenOffice) have features embedded in various places to hand off the current document, or the current document converted to another format directly to your email software. This is a great feature because it eliminates the need for users to convert to other formats prior to sending to outside. A document can be created in ODT format and retained here in that manner, and then converted to DOC on the fly...and the DOC is never saved to the local file system. In case you didn't know it did this, here is where the magic happens:
Now the bad part, it basically does a mailto: to your email client and uses &attach which it turns out is not officially in the RFC and is only available in certain email software...including Evolution. But many/most web based email software does not allow or support it. I can certainly see that this could be a bad exploit. Someone could create a mailto tag that requests files of an unsuspecting users desktop.
You cannot just tell users to drag and drop the file out of Nautilus because in many cases, it's generated as a temp file and is never saved for them to be able to find. You can't expect them to open Nautilus in /tmp and *maybe* find their file out of the hundreds of other temp files from the day. Clunky to say the least.
Zimbra supports HTML 5 drag and drop reception, so I worked up a prototype idea. Jasper and Federico were so kind to help me with the drag-and-drop mechanics of GTK and we're testing this concept now. When a temp file is generated and needs to be attached to email from LibreOffice (and other applications if it pans out)...a popup window appears on the left edge with a thumbnail of the document. The end user can then grab the document and drag it into their composer window. In the shot below, user has selected the option to "Email as PDF" and the popup appears with thumbnail and Zimbra composer window opens (PURPLE). The user then drags the file into the composer (RED).
In the coming days, I'll be testing this concept and soliciting user feedback. If it seems to be working, I'll connect it to the rest of our applications that previously auto-attached files to Evolution.
Next up for me is creating LDAP groups and then getting Alfresco to auto-provision users and put people into those groups. We're at the breaking point around here of maintaining documents only on a file system and are in need of a management system. We'll be pilot testing this in IT to see how it works.
Other issues: 64bit HP thin client is sitting here for me to test, various portal changes have been requested, continued Zimbra troubleshooting and support, various Java/Flash/Firefox upgrades.
OpenLDAP
The change to centralized passwords went off very well. During my prep work, I created a "Password Sandbox" application that allows you to test various passwords before actually making the change. This proved to be very popular and effective. Part of my time was then allocated monitoring all 800 of our users to ensure they all changed their passwords and sending out daily status reports to work on 100% completion. It's now all done and our passwords have been hardened to current metrics.
Support Portal Application
I have been putting in some patches and features for our support staff. They could see at a glance users with passwords that still had not been changed, and made it easier to create and upload LDIF records. We've reached a milestone with this functionality where it's very simple to create new employees in LDAP. We also can now auto-provision new people in the Zimbra post office with just a single click. Within 10 minutes of checking the button, the user has email.
LDAP Sync To Zimbra.
Zimbra auto provisions email accounts, and I wrote a small python script that runs at 1am to push any updates that occur from OpenLDAP to Zimbra. Phone numbers and job titles change, and all of this information is now synced. Support staff only has to change the information in one place, saving us time.
Zimbra Is Live
After a long ramp up time of testing, we put Zimbra live and it replaced Evolution/Groupwise. It was kind of sad to move away from Evolution, but unfortunately we just could not get bug fixes from Novell/Suse under our support contract in a timely manner. Zimbra was deployed with virtually no problems. We built one server and have about 250 concurrent connections via the web interface and then an equal number of connections from mail-notification(IMAP) on the lower panel to alert them of new messages for a total of 500+ connections. Zimbra is flying and you can barely notice the difference with a full load. Fast and crisp.
In the past our server which provided Firefox would have about 100 concurrent users, now that users are accessing email from a browser this has jumped to about 250. The server took this load without a hiccup and is working very well. With 100 concurrent the machine would be about 10% busy and now it's running about 15%. We scaled this machine with the expectation of greatly increased Firefox loads and it's working like a champ.
And The One GOTCHA
There is always one, right? :) LibreOffice (and OpenOffice) have features embedded in various places to hand off the current document, or the current document converted to another format directly to your email software. This is a great feature because it eliminates the need for users to convert to other formats prior to sending to outside. A document can be created in ODT format and retained here in that manner, and then converted to DOC on the fly...and the DOC is never saved to the local file system. In case you didn't know it did this, here is where the magic happens:
Now the bad part, it basically does a mailto: to your email client and uses &attach which it turns out is not officially in the RFC and is only available in certain email software...including Evolution. But many/most web based email software does not allow or support it. I can certainly see that this could be a bad exploit. Someone could create a mailto tag that requests files of an unsuspecting users desktop.
You cannot just tell users to drag and drop the file out of Nautilus because in many cases, it's generated as a temp file and is never saved for them to be able to find. You can't expect them to open Nautilus in /tmp and *maybe* find their file out of the hundreds of other temp files from the day. Clunky to say the least.
Zimbra supports HTML 5 drag and drop reception, so I worked up a prototype idea. Jasper and Federico were so kind to help me with the drag-and-drop mechanics of GTK and we're testing this concept now. When a temp file is generated and needs to be attached to email from LibreOffice (and other applications if it pans out)...a popup window appears on the left edge with a thumbnail of the document. The end user can then grab the document and drag it into their composer window. In the shot below, user has selected the option to "Email as PDF" and the popup appears with thumbnail and Zimbra composer window opens (PURPLE). The user then drags the file into the composer (RED).
In the coming days, I'll be testing this concept and soliciting user feedback. If it seems to be working, I'll connect it to the rest of our applications that previously auto-attached files to Evolution.
Next up for me is creating LDAP groups and then getting Alfresco to auto-provision users and put people into those groups. We're at the breaking point around here of maintaining documents only on a file system and are in need of a management system. We'll be pilot testing this in IT to see how it works.
Other issues: 64bit HP thin client is sitting here for me to test, various portal changes have been requested, continued Zimbra troubleshooting and support, various Java/Flash/Firefox upgrades.
Subscribe to:
Posts (Atom)