Beryl Baselines / ATI 9250

The guys at OpenGL and on #beryl-dev were very helpful in increasing my understanding of how Beryl works. There hasn't been a lot of testing done of 3D desktops over remote display, and I volunteered to do what I could to test and improve this type of deployment.

The first step is to build a baseline of how it works now, and then monitor as patches and changes are made. It's my understanding that for a crisp 3D desktop one wants to have around 100 frames per second from the Beryl Benchmark plugin. When you drop below that it works, but feels sluggish.

My baseline is to log into GNOME, start beryl, and have Evolution and one gnome-terminal window open. Based on that benchmark, Beryl runs well for all of the 16 bit color resolutions, and will support 24 bit color in 1024x768. 1024x768 is the standard for 99% of 'Office Workers'. Once you go higher than 1024 in 24 bit color, it begins to slow. The highest I tested was 24 bit color, 1680x1050 which works, but is a bit too slow to deploy. Window movement is not crisp, and applications take a while to render and scroll.

I'll put the Nvidia card back in again after Christmas and perform the same test. Thanks to everyone that is helping me understand how it all works. It's certainly something that could be deployed.

(Benchmarks below, go to my blog if the chart does not appear).

Testing With ATI Radeon 9250

Sometimes things that you think will never work....DO. I have been testing remote displayed 3D presentation from a Fedora Core 6 server, to Fedora Core 6 installed on a thin client. I'm testing the Radeon 9250 card and it seems to be working pretty well. Many thanks for the suggestions and ideas that all of you have sent me. I have already tried Nvidia, and would have liked to have tried an Intel card; but apparently they don't make a PCI based card with that chipset. Contact me if this isn't the case.

A fresh set of Beryl RPMs were just placed under Yum extras and are working pretty well. I was able to compile and install XglSnow for testing and benchmarks. I really thought it would be horribly slow, but it's not running badly at all.

(Shot below, follow link to my Blog if it doesn't display)

Thin Clients And Remote 3D

We contacted two of the big Linux operating system companies and requested some minor technical help in attempting to deploy our next generation thin client solution. Neither had the resources to help, so we are moving ahead and doing the engineering and design in-house. I invite anyone with access to Gartner to read study G00140085.

I believe the mistake being made by these vendors is that they are attempting to install Linux on the personal computer, instead of putting Linux on their desktop. The Gartner study shows that there is a 48% reduction in cost on the Microsoft Windows platform by moving it from an unmanaged PC environment to a centralized design with thin clients. 1/2 the cost, and no change in functionality. Imagine then what the savings would be if companies had the option to move to thin clients *and* Linux at the same. A major part of the cost in the white paper is licenses and software products. Imagine going into companies and telling them that they could save 60-70% on computing costs. Really, trying to shake off Microsoft Windows from their personal computers just isn't enough to warrant a change for most people. It doesn't offer the major cost reductions that are found with a complete, and stable re-design. Centralized computing using thin clients really works. There shouldn't be so few of us implementing and being the voice.

With that, I have been able to make significant headway this week on getting our design goals for our scheduled early 2007 rollout of new thin clients. We are deploying HP 5725 devices, and a new 2GB flash device has become available in the last week. I installed this 'disk drive' into the case and this increased capacity made it a LOT easier to load Linux and GNOME. Many thanks to the email messages and blog responses with ideas. We are going to experiment with some other 3D video cards and also are testing AIGLX instead of XGL. I was able to get Fedora Core 6 to install and with a kernel upgrade and a few packages it was working standalone in 3D.

[ In the shot below you can see the opened case. The flash device is circled. Installed Nvidia card into expansion slot. (Go to my blog if you don't see the images) ]




At this point, it was identical as a personal computer. Everything was running on the thin client. But our design goal is to move this to the server and turn these into thin clients. So we loaded Fedora Core 6 on a computer to simulate a server and then logged in remotely with XDMCP. The server recognized the video card and performance with Beryl is mostly, excellent; even over a network! We were really amazed at how well this works, even in prototype form. We hobbled together a quick new cubecap and the shot below shows a prototype of what our users will have early next year. This increase in capability will come in around 40 dollars extra per user for the video cards, with a projected duty cycle of 10 years -- and no support at their desks. The GNOME session and 3D elements are pushed down from one big server which is then upgraded every 3-4 years for hundreds of users.

[ In the shot below, Beryl cube running over the network via remote display, with Evolution, GIMP and GoogleEarth ]



Up next, testing of other video cards for performance and ease of installation.

Beryl guys: Nice job on what is running on Fedora Core 6. Some issues that seem to need work over remote display: wobble windows and initial startup time. I hope to work with you in the near future on testing over remote display.

Sometimes It's Just The Bling

Two things in the last few days reminded me about how important desktop bling is to the user community. I tend to focus on the technical and bug issues, and sometimes forget this point.

Last week, I did a 2007 technology preview for about 20 City employees, and showed the 3-D rotating desktop found on newer distributions. They were simply amazed and there was a gasp in the room.

Today I took a few minutes to download some free wallpapers for the Christmas holidays. I also compiled the 'oldie but goodie' Xsnow for those people that are helping me test SLED 10 on thin clients. When I first saw it 10 years ago, it was such a server and network hog that it wasn't something that could be deployed. With the technology of today, it barely even shows up on the process list. For some reason I remembered that it didn't work well with GNOME, but obviously this has been fixed. Snowflakes drop down from the top and stack on the top of windows in your current view. (Shot below, follow link to my blog if it doesn't display)



Ok, back to bug reports now :)

Busy Few Weeks At Largo

The last few weeks have been busy at Largo. The week is ending and I thought I would blog about what's happening.

Firefox 2.0
We upgraded the whole City to Firefox 2.0 Tuesday at noon. It went off without a hitch. All of the users bookmarks and settings all worked. To the developers, thank you for a nice, simple upgrade. I just moved the launch script to the new release and everyone just picked it up the next time they launched the program.

OpenOffice 2.0.4
We upgraded the City to 2.0.4 and a nasty little printing bug forced me to roll it back. It's been fixed in a CWS but looks like we have to wait for 2.1 to get it. Upgrade and rollback were handled in seconds and easy, but I hate losing the eyes that would be reviewing the rest of the new code.

Evolution For SLED
Harish and the guys on the other side of the world are doing great work squashing Evolution bugs. I have spent significant time debugging and testing Proxy and other GroupWise features so that we can roll them out to our users.

SLES/SLED/Thin Client Testing
I hacked a GDM splash page for login with our colors and logo. People really like little customized touches like this. Really, I didn't steal the colors from Logitech :) [follow link to my blog if image doesn't display]



We found a nvidia FX5200 video card for testing in the thin client. We needed one that plugged into a PCI slot, and has no fan. The thin clients has no moving parts, an we wanted the card to work the same way. The expansion slot was added to the thin client, and then the card installed. SLES found the card and kicked it right into 1600x1200 and it's working. Below are pics of the expansion slot added to the thin client on the left side, and the yast2 program finding the right video card. This is the first thin client I've seen with 3D support, so for grins I ran GoogleEarth for Linux over remote display. :) Well, it works, but isn't fast enough to deploy. I think that we will probably flash the devices with this binary and it will just run locally. Work continues on the Citrix/XDM chooser and getting X to flip between modes correctly. I'm also still working on remote sound.

USB Access To Thin Clients Via Server

One of the things that has changed since our first thin clients were deployed 10 years ago is the desire to plug in USB memory devices and gain access to them from City applications. After considering various ideas, I have finished implementing this on my experimental SLES 10 device. The biggest issue that we face is different skillsets and the propensity of people to just turn off devices without dismounting them first. NFS mounting would have been a nightmare for that reason. The technical hurdle is that the USB ports are on the local thin clients, yet Nautilus and other City software runs on the server. I wanted to create something as stateless as possible. My solution was to set up a user called 'usb' on the thin clients, and start up vsftpd and jail that user into the /media directory. Ivman then picks up devices, mounts them automatically. Nautilus has the ability to add a 'Place' that is a ftp connection. The end result is a file manager that appears to the user to be working locally on their device. It also in theory would allow users to share files to USB memory sticks to users in different buildings. (On my wishlist: ftp connections from F-spot).

The images below show how it works; (follow my blog link if images do not appear).

[ The usb camera is plugged into thin client. Ivman sees the plugin and mounts /dev/media. Nautius is running on the server, and ftp's over to the thin client and sees the mounted memory sticks or cameras ]


[ ivman working well, mounts and dismounts automatically ]



[ The most important issue, transparent access to these files for the user. This looks no different than it would if they were on their own computer. ]



Still struggling with poor SiS video drivers in SLES. Next up, working networked ESD to get sound from the server.

HP has officially added this product to their website.

Roller Coaster Enthusiast

Between work and buying a new house, I have been very busy the last few months. It was nice to get out of town for a few days and head up I-4 to Orlando. I've gone on a lot of roller coasters around the country, and the The Hulk at Universal is definitely my favorite. I thought I'd share some pics of what it looks around Florida, and the coaster itself. [follow link to my blog if pics don't appear]

Thin Client Work Continues

Concurrently with all of my other projects I still am working on building a working prototype of a SLES based thin client. We are going to be buying around 500 devices, and I want to make sure that everything works before we make the purchase. Getting SLES on 1 GB flash device has been a challenge. I was able to finally get an install down to around 950MB and it loaded. My first test was with reiser FS, and I found it not a good fit for a thin client. Users often just turn off devices, and it was taking a long time to reboot after such an event while the file system cleaned. Ext3 has been working better. Identical packages take 40 seconds to boot with Reiser, and 33 seconds to boot with ext3. Interesting.

I have hobbled together a working prototype, it boots and starts X and presents a chooser for XDM and Citrix connections. My first chooser screen was done with Zenity for testing, and now I used Glade to create something a little nicer looking. [ Shot below ]. I'll get that installed on the boot image next week.

The devices have a SiS video card built in, and the driver isn't working well past 1024x768. There is a lot of screen flicker and blackness flashing on and off. More testing will follow in that area.

Next up is adding the PCI bridge and testing with 3D video cards to see how that works. One wishes for more hours in the day!

Really Efficient Software Deployment

One of the misconceptions about deploying multiple people on a server is this thought process:

"If I am on a computer by myself and sometimes it's barely adequate...how can it possibly work having hundreds of people sharing a server?"

We hear this over and over again as we talk to others and explain our technology. I think they assume that you divide the CPU speed by number of users and that somehow gives you your time slice.

In fact, once people see the speed of the software running at Largo, they are amazed and often say it runs faster than it does on their PCs. There are a few reasons for this:

* The servers are obviously bigger than any computer you would have at your desk.
* Even if you have hundreds of people on, at any given time only a fraction of that total is actually using the software.
* When you have enough memory to hold everyone in RAM, there is no swapping at all.
* You never really have a cold start of software. It's always running by someone else in the City.

I thought I would show a few 'tops' to demonstrate how well Linux handles multiple users. I think we could easily get 2-3 times more people on these servers if we had to do so. The most interesting thing to note is the CPU usage.

This shot is OpenOffice 2.0.3, 85 concurrent users at the time.



This shot is Evolution 2.6, 180 concurrent users at the time:



This shot is Firefox 1.5, 48 concurrent users at the time:

SLES 10 On A Thin Client

The City of Largo is doing some experimental work with testing SLED/SLES on thin clients. Our 10 year old devices are at the end of their duty cycle, and we need to buy new ones to run for the next 10 years.

We want something that can be upgraded to some extent and that we can customize to meet our needs. I was unable so far to get SLED 10 to install because of flash drive limitations, but finally was able to get SLES 10 to install and X to come up. [ Shots Below ]. The devices are from HP and have 1GB of flash disk, and 512MB of memory, and 1GHZ CPUs.

SLES forces you to install some packages that aren't appropriate for a thin client, so next up for me is to delete everything we no longer require. This should reduce the boot time. Then, I'm going to try and install a few things by hand and get it to start X and present some kind of chooser for XDM and Citrix connections.

The next step after that will be to install the optional PCI bridge and try and install a 3D video card and experiment with using Xgl from SLED 10 servers over the network.

A great Friday project, and successful even.

We Like You, Even With RBL & Blocked Addresses

Dave Neary (bolsh) posted an interesting blog about the RBL and blocked addresses. We are a site that blocks all of France, but not because of any "Freedom Fries" issues. :) The City of Largo has tasked us with providing a hostile-free environment, and part of that is very aggressive cleaning of email. People like to complain if they know they can, and we were getting flooded with email that was 'offending' people, which means for some being one step away from a lawsuit. The RBL has helped us block ISPs that are not actively going after spammers on their network. The theory is that the user community will give them so many support calls about email being blocked (like you!), that they will hire enough people to manage the network better. As for blocking entire countries, there is a reason for this. Spam appliances do a forward and reverse DNS check on sites as part of the first check. If this passes, filters are applied and then it's released into our GroupWise server. Our observations indicated that certain countries were sending out a great deal of spam that was getting through more than others. France (.fr) unfortunately was one of the top senders. Possibly, only about 5 out of 750 of us receive email from other countries, so we had to make the decision to block and work out exceptions as they come up.

I did a live snapshot of our spam appliance running, you can see the scope of our problems. I am sure many of you are dealing with the same thing.

GNOME By The Numbers

Federico mentioned in one of his blogs about the high number of users of GNOME from thin clients and on multi-user systems.

As software developers, some of you possibly had not considered a few things when designing, debugging and testing on single user systems:

• Most business environments are open around 250 days a year. (52 x 5 - 10). We have approximately 750 users. Let's assume that a known issue is in a software package, but considered minor because it only happens once a year. If this issue requires a call to our support staff, we instantly have 3 calls a day. (750 / 250)
  
• If a condition is left in a software package that locks a NFS drive, or other issue that requires a reboot, we have to schedule this and then kick people off the server during a time that was not pre-scheduled. It's not just an issue of rebooting your local computer. There are multiple users on servers 24 hours of the day, and it's always a bad time for someone.
• If a software package is leaking memory, and let's assume that it's 50MB a day. Your first thought might be that 50MB is no big deal, and cheap and certainly not a problem. For instance, on our Evolution server, we are running about 210 concurrent users. A 50MB a day memory leak then suddenly becomes a bigger problem.
• File handles can become a problem if they leak. It really helps to consider hundreds of people running a software package rather than just one person. At a certain point there will be problems and tuning issues with Linux with file handles. Often, this shouldn't be required because it's just an issue of them not being closed and released.
  

Largo Application Server Design

I have had the pleasure of meeting Nat and Harish in person, and they were given a quick tour of our network and design. Since I have contact with so many more of you, I thought I would explain our design a bit and how it relates to GNOME and software.

There are 3 basic designs for deploying software to users, Largo has deployed Application Servers.

Desktop Computer
In this design a physical computer is sitting at the desk of each user, each loaded with operating system and software. This is by far the most expensive, and support intensive way to deploy. Technology churn forces upgrades and changes often, and patches need to be applied to each machine. Even when pushed from a central server, it's support intensive.

Departmental Servers In this design, groups of users log in with thin clients. It's very similar to a desktop computer in that most of your software is located on one server, except that multiple people are logging into the same computer. The problem with this design is that similar packages are still on multiple departmental computers and require upgrades. It's also sometimes difficult to have multiple software packages on one machine and be able to upgrade them without library upgrades and patches that cause problems with other software. You also sometimes have problems where a user might lock up a package or NFS mount which forces a reboot at the expense of kicking everyone off the server.

Application Servers
In this design, each unique major application is given a server. This allows you to select the best operating system to deploy, and upgrades can be installed without worrying about other things failing. Upgrades are very simple, you install the new version and then just change the launch script and point to the new version. As people request this package, the new version just goes live instantly until the old one is no longer used.

In our case, we have a big server running GNOME. All that runs on this server is the desktop itself and this allows users to customize their desktop. The basic tasks like fonts, wallpapers, colors and themes are configured on this machine. When a user requests an application like Evolution, a signal is sent to another server which is running only that program and it launches a session and then remote displays it back to the end user. From a users perspective, they cannot tell that they are logging into multiple servers at once, as it's fully automated.

Because the horsepower comes from the servers, the user thin client devices have a long duty cycle. Our current devices were deployed in 1997 and are still working fine. They are being retired because of certain features that have become standard since 10 years ago: the RENDER extension is missing on them, they only support 1024x768 and they do not have scroll wheels. This fall we are installing new thin clients which will once again have a 10 year duty cycle. A 500 dollar thin client therefore will have a 50 dollar per year cost per user.

I have created a simplified image of our design. The user logs into the GNOME server with a thin client. Once there, as they click icons a session is formed on another server and then sent back to the users device. This allows you to scale easily to hundreds of users on the same server and support is lowered greatly. We have been running this design for 10 years and it's the most stable environment I have ever seen.

Problem With HP 5515 Thin Client

I showed a few people this issue, and wanted to get a few more eyes and thoughts. We are testing 3 models of thin clients from HP. One of the models is the 5515 device. Everything works fine on it, except for the color squares to the left of the calendar sources in Evolution, and the wallpaper thumbnails in gnome-background-properties. Both of these work fine on GTK 2.6, but fail on GTK 2.8.

The thin clients have been flashed with the latest release, and cannot be upgraded with normal distributions. XFree indicates:

Section "Device"
Identifier "Videocard0"
Driver "radeon"
VendorName "HP"
BoardName "HP Video Card"
VideoRam 16384
#Option "NoAccel"
Option "NoDDCValue"
Option "XaaNoPixmapCache"
EndSection

This screen shot and shows how these software packages display on this device:

HP 5700 Series Thin Client

I have had conversations with many of you in regards to thin clients. This fall will be the 10th year of deployment of our current devices. We are testing devices to come up with a good solution for the City to deploy as a replacement. It's difficult to know where technology will go in the next 10 years of their scheduled duty cycle. The secret to success of the last device was to keep them simple, and running no local software on the device itself. Servers are then upgraded to accommodate software changes and to increase performance.

I think what we would like to do is get a stripped version of something like SLED running on a device with 3D and Xgl support. Most thin clients do not contain very powerful video cards, so therefore we have been checking out the 5700 series thin client from HP. You can buy an optional expansion cage which adds a single PCI slot that can hold a video card. The device has 256MB of RAM, and a 512MB flash drive and runs at 1Ghz.

Ideally we would like to have it boot, start up the Xserver and then build a XDM login chooser of available hosts.

I have taken pictures of the inside of the device for those interested. It's completely solid state.

[Looking down from the side ]


[Looking up to the single PCI slot]


[ Cover back on, ports on the back including standard PCI ]

Gotta Love Application Servers

All of our city-wide software is run on large dedicated application servers. We all share the same code and fonts and all I have to do is upgrade in one place and everyone picks up the changes the next time they launch the application.

It took me 1 minute to upgrade OpenOffice from 2.0.2 to 2.0.3 for 700 users. In the shot below, users up to 12:20 launched the old version. Users after that time picked up the new one. Those in the old version are not logged off. The two exist at the same time, and 2.0.2 is retired when the last user logs out tonight.

All done. :)

Testing Evolution Calendar Creation

I continue to build scripts to automatically bring down calendar information from various web sites. I'm looking for someone that lives in the US that is interested in running some of the scripts and just get another set of eyes on the project. [ I'm downloading only information so far from US sites like Yahoo and XM Radio ]

You would need to have a bit of expertise with shell scripts, cron and setting up a simple web server. Email me at flbeachlf at yahoo dot com if you are interested and I'll send you a tarball. Happy 4th all!

Here is a shot of Evolution automatically displaying the program guide for XM Radio for the current day.

Making Evolution Cool

I wanted to try and find a project that I could work on in my free time that would help the community, along with helping solve some of my own needs. We have had excellent success here at the City taking time and date material from manual systems and from our financial software and loading it into iCal files which then display in Evolution.

I have been poking at developing a method for downloading time and date related information from the web and then formatting that information into .ics files which then display in Evolution. At this early stage it's just developed with lynx --dump and then a ksh script does the formatting. But I would like to develop a simple scripting language for reading this content and formatting it easily to promote the development of lots of similar plugins.

I have been working on a few simple plugins to determine the scope of fields that need to stored, and also how the scripting language would work. I have been testing plugins to display movie times at local theaters, displaying the lowest rates for car rentals for a block of dates, displaying lowest prices for air travel for a block of dates, and displaying lowest prices for hotels for a block of dates. This would turn Evolution more into a portal, that watches the web automatically and updates itself as the prices change.

Wouldn't it be nice if a plugin system like this was built into Evolution? That would make Evolution very cool. Find me on the IRC or email me (flbeachlf at yahoo.com) if you have any suggestions. It's all at a very early stage, and I'll need some help with the scripting language development. There also would have to be a method to generate front ends very quickly to allow the entry of the basic criteria.

What is a blog without screen shots? The shots show the original source material as it appears in Firefox, and the resulting information that was brought down via the scripts.

Evolution displaying car rental rates:



Evolution displaying show times at a theater:

Benchmarks: Display Of Email Headers

I have been doing some testing for Harish, Sankar and Chen in regards to the time it takes for headers to display in Evolution using SOAP/GroupWise. I found a user that had 430 messages in their Mailbox and did some benchmark testing. The results are fairly flat, and seem to be roughly the same no matter what time of day they are performed. I timed from a cold start, from the point the UI fully displayed up to the time the messages appeared. The results are below:



I'm hoping that we can shave some time off these numbers because that would make Evolution appear to be running "faster". Right now our biggest issue is load time of calendar events, and we seem to be making progress in that area.

Guidance - GNOME Hurricane Tracker

The hurricane season started on June 1st, and Alberto already went right past us. I wrote a program last year to downoad images from Weather.com and our users were starting to run it again this year. It turns out that Weather.com had moved the images. I made a small change and it's working again. I'll pack up another release and move it over to Source Forge.

Here is a shot of the UI. All it does build a list of the hurricanes for the current year, and put them on tabs. It then puts the downloaded images into the appropriate area.

Benchmarks: Evolution Calendar Loading

Harish has been so kind as to put in some experimental Evolution code for testing purposes to resolve one of our issues. We have some users with thousands of appointments that go all the way out to 2008. The old version of Evolution would find all of the meetings and load them in reverse order from 2008. The power users were having to wait 2-3 minutes before the UI would load the current date.

I have configured a test of doing a preload of +- 21 days. This loads 3 weeks into the future and then 3 weeks into the past initially and then does the rest of the calendar after it's finished. The shot below shows how the new code loads appointments:


I tested with a stopwatch the user that has the most appointments in the entire City, easily thousands of them and got the following benchmarks:


I have been able to draw the following conclusions:

• The normal load time should be approximately 6-7 seconds.
• There is a latency/threading/timing issue between Evolution and Evolution-Data-Server which greatly alters the results. It seems like sometimes EDS is not quite ready for a query at startup and then Evolution waits a few seconds and tries again. This possibly could be because of the speed of our server and/or threading issues on 4 CPUs that isn't normally seen on a laptop or workstation. When the two are in sync, it loads in 6 seconds. When they are off, the timeout adds about 6 seconds.
• It seems possible to drop into code which doesn't use the environmental variable for setting the 21 day PRELOAD. Sample #4 followed the old code logic and loaded the calendar in reverse order and took over a minute.
• It seems possible for eds to never sync with Evolution in rare circumstances. In sample #10 data was never returned from GroupWise.

I think we are on the right path with this new code, once these issues are resolved.

GNOME Polish

This widget problem has been around forever, and falls under "GNOME Polish". For years now many applications have had option menus with white space that fills the entire screen and forces the users to scroll up in a very un-natural way. These little things irritate 'regular users'.

[Shot taken from Acrobat, many other apps do the same thing]

Handheld Device Progress

Linux powers our desktop (GNOME), email (Evolution) and email backend (Groupwise). An ongoing project to allow our users to use handheld devices to access our network is nearly complete.

Requirements

1. Plugin for GroupWise must run on Linux and not require another server.
2. Offline viewing of email/calendars must be supported via sync and download.
3. Sync must happen over wireless, because we don't have want/have cradles.
   
4. Installation to handheld devices must not require Microsoft Windows.
5. Citrix client must allow real-time access to our applications in 802.11 hot spots.

We have selected Omni Mobile because they resolved requirements 1-4 very nicely.

We picked devices running PocketPC because it resolved issue 5, and is one of the supported platforms for Omni.

We are rolling this out into production in the next few weeks.

Screenshots

Omni Mobile syncing with and displaying my GroupWise Mailbox.


Omni Mobile syncing with and displaying my GroupWise calendar.


Citrix connection to GNOME & Evolution in 480x640, 16bit, Display Factor 2.


Citrix connection to GNOME/OpenOffice 2 in 480x640, 16bit, Display Factor 2.

Cairo Endian Fixes

After working with Rodo from Novell for about a week, he was able to resolve the issue where GTK applications would appear in the wrong colors when running in 8 bit depth. The issue apparently was related to fact that Metaframe for Unix runs on Solaris and it was having some endian problems. The screenshot indicates his excellent before and after work. Thanks!

I Moved In Too

Thanks to Harish for pointing me to this blog site. I have been wanting to post information for a while now about the City of Largo. Instead of focusing on the development process, I can speak of deployment issues.

I hope to provide information that is useful to other people.