Tell me about your troubles and woes configuring, deploying and maintaining distributed .NET apps

Believe it or not, Microsoft is always trying to improve its products and to do so, I find we do our best work when we actually ask our potential customers what they think.

In this case, I’d like to know what “pain points” you experience when configuring, deploying and maintaining distributed .NET applications. That can be any kind of app, whether it’s a client-side app that phones home for code or data updates or whether it’s a fully distributed grid or anything in between. Please be as specific as possible and, if you’d like one of our courteous technical people to follow-up, make sure to include contact info.

Use this as an opportunity to vent – don’t hold back. Remember: the life you save may be your own…

A couple things off the top of my head...
Deploying: regardless of broadband, cheap disks, etc...there are still a lot of customers out there that don't like the idea of having to download/install the framework (20+ MB download and 40+ MB install) just so they can run my 500 KB application. I try to explain that it's a framework that allows developers to write more robust and secure programs and that it's kind of an extension of the OS and will be built in to future versions, etc. I've talked to several friends who's company's aren't using the .NET framework because they are afraid their customers will turn to competitors (whose install doesn't require the framework). Why doesn't Microsoft make it a required update? It IS a required update if Microsoft wants us all to feel safe about selling apps that use it.
Another major pain for us when upgrading distributed apps is related to call context. When an object living in call context implements ILogicalThreadAffinitive, it will flow across remoting boundaries. If that object can't be deserialized (by the remoting infrastucture) on the other side for some reason, an exception is thrown back to the client. In other words, clients have no way of communicating with servers that don't understand what's in the call context (even if the server could care less about what's in there).

  For a while I was building and releasing apps written by other developers at work. (This both ensured reproducibility and cut down on people with logins to production servers.)
  I ran into problems with a number of solutions with 2 or more projects, where the dependant project couldn't find the project it depended on. Removing and then adding the required project back fixed the problem, but it required checking the project out, which the person doing the build shouldn't have to do.

While not currently working on deployable apps, I have in the past had to manage deployment a typical multi-tier application (web ui / web services / data) with multiple web / web service applications and a couple of databases and the biggest issue was managing configuration. This pain occurred in the following scenarios:
 - across developer machines (depending on the aspect a developer was working on, they may or may not have had a local copy of services and / or databases).
 - from dev to internal test
 - from internal test to customer testing
 - from customer testing to production

The reason this issue became such a pain point was the 4 - 6 configuration files required for each scenario. While I have yet to look deeper into .NET 2.0 to see if there are changes that have made this type of deployment easier, at first glance I have seen little to indicate there is. Reading the previous comment on configuration servers, I was intrigued and think this concept may be great for larger installations where there are numerous applications runing within the same environment, however the overhead for 1 application / 1 config server means this would probably not scale down well.

The solution must enable multiple configuration files (possibly with well known names i.e. dev.web.config / test.web.config etc) so that it is easy to do numerous deployments between environments without requiring extensive modification of configuration files (the less you have to change bwtween deployments the better). This would probably mean tools for generating / migrating / diffing / updating settings from the development configs to test and production configs.

Having recently seen RoR and it's approach to managing dev, test and production databases I think some valuable lessons could be learned from the approaches used - remember, one of reasons Rails was built relates to programmer happiness. =D

Not sure if this is an off the wall rant or not but hopefully there's something useful in all of this.

Include a tool as powerful as Wise at the price of Wix !

The only pain I've got is the 23+ MB framework. Put it as a required Windows Update! Please! I've been screaming this for the last several years. Please, please, please make the .NET framework install by default as part of the OS, and make the framework be a REQUIRED update for Windows Update.

Make ClickOnce actually work with Firefox.

Also, just thought I'd ask: Is there any way to make a ClickOnce application autostart when Windows starts?

Installing apps to a PocketPC is so painful it hurts.

The .NET environment itself is great for developing applications. The installation technologies are what really spoil the development experience. The two big gripes I have with MSI are the lack of support for multilingual installers (without using a bootstrapper) and the lack of managed custom actions.

definitely the installation of the framework. It is the larges obstacle for our company to switch versions.

We use NSIS to install our apps and a custom built updator to run updates. We are going to go live soon :) Dunno if this comments helps...

I realize there aren't that many Winforms controls being run out there which are hosted in web pages. But I'd bet there would be even more if they could be debugged easily while running within IE and the "Internet Permission set". Creating/debugging winforms controls which run in a web page is a bit challenging right now.

First, I'd like to say thanks for seeking input on this.

We have a typical high availability configuration (2 load balanced front end application servers and 2 clustered back end database severs). We built it using VS2003. There are 50 projects spread over 10 solutions.

At one point we considered writing a full blown MSI based installer using a tool like InstallShield, but concluded this wasn't cost effective since the application has only 1 production deployment.

That decision left us with completely manual deployment, which has caused some pain. Another interesting aspect of our environment is what we call instances. This is where the application is deployed multiple times to a set of servers. For example, you have the 4 servers. Spread across those servers could be an instance for testing, training, sales demos, etc., with each instance potentially at a different revision level.

So, where does it hurt?

The biggest problem has been config files. Each instance has 18 config files spread over the 4 servers. Furthermore, much of the config information is duplicated. Database connection string is the best example of this. We've had numerous problems caused by not having the config just right in all the config files.

Hand editing of config files using Notepad has also been a problem. Sometimes the XML isn't well formed. Since there's no schema for what's inside a section, there's no way at the time of editing to know whether the XML is valid (ignoring the fact that Notepad doesn't validate).

One of the things that's driving the editing of config files is upgrades. Right now what happens is that development supplies release notes to the sys admins defining the changes to the config files and during an upgrade the admins edit the config files. Missing from this is a good way to document the config files. Release notes aren't a permanent record of the contents of a config file. Obviously, comments can be put in the config file, but this can make for a difficult reading experience. Also, there's no good place to record in the config file which version of the application it is meant for.

An area of concern has been COTS component. Most of the COTS components we're using are not written using .NET and don't have a side-by-side deployment model (e. g., print drivers, IBM MQ Series). Each version of the application is developed using a particular version of each COTS component. But multiple instance of the application are running on each server, and these have different COTS component version requirements. I say this is an area of concern because in practice running the latest version of each COTS component almost always works.

DLL labeling (setting the version number, etc.) does not work well in VS2003. It makes no sense to me to put the version number in the source code (AssemblyInfo file). Who wants to run around an update all those AssemblyInfo files? I'd much rather see a build process designed for release beyond the developer's workstation where the version number is declared and applied to the build outputs. Maybe MSBuild address this?

What's the solution?

In a perfect world, you'd model a version of an application. This would involve declaring the different types of hardware devices in use (server, workstation, PDA, cell phone) and the attributes/minimum requirements of those devices (e. g., operating system, RAM). You'd then define your high level components (e. g., web service) and the dependencies with other components (some custom, some purchased) and the component versions. Also associated with the version would be a definition of how the components can be configured.

This information would form the basis of an installer that can install one or more high level components, and their dependencies, on a hardware device. The installer would need intelligence about the configuration of all the hardware devices so that the installation can be wired together. The configuration definition would be used at install time to allow configuration and after the fact to adjust the configuration. It should also be possible to inventory the installation to discover any missing pieces or version mismatches. Upgrades should be handled smoothly and the instance concept from above should be directly supported. Upgrades should handle altering the database schema and adjusting the configuration, of course while preserving the data.

It would also be helpful if an instance could be brought on line / take off line with a single command, and done in dependency order (e. g., bring up the database before bring up the web services).

Dave Douglass
david_douglass at ieee dot org

Seperate the frame work into a server and client version. The server version will contain everything while the client will have system.web etc.. removed - that might nmake the framework download smaller for client systems an d make .net more feasable for desktop adoption

Last week I had a 1.1 winforms app on a clean WinXP SP2 VPC with the 2.0 framework on it. When I ran the MSI setup of the program (created in VS2003) it said please install V1.1 of the framework (!) and would I like to go to Microsoft's web site to download it. I said yes and it took me to this page http://msdn.microsoft.com/netframework/downloads/updates/default.aspx
in which there is no hyperlink to the V1.1 of the framework at all - only V2.0 downloads - and there is no obvious hyperlink to the V1.1 framework within 2 clicks of this page.

Some rhetorical questions...

Although the program itself ran under the V2.0 framework, why was the setup program bound to V1.1?

What will happen to V2.0 installation when V3.0 comes out?

Will you fix that web page to make the download to all previous versions of the framework more obvious?

Regards
Dave A

I agree with some of the others who replied earlier. The real pain is managing configurations. A central server that could SECURELY provide the needed config for an app would be a sure winner.

Moving an application from one box to another.

I'd like a 'export website to package' capability.

Then you goto the target box and 'import website'

It would keep the structure intact and advise of security differences

Thanks :)

Most of my issues are actually with the dev IDE (Visual Studio) rather than the actual copying over of the project directory (although there's one right there...a full XCopy gets a lot of stuff that doesn't belong in a compiled/deployed app, but the /bin directory is only the codebehind...I'd rather the build process creates a directory that contains all my aspx & DLL files, but keeps out the .aspx.cs files):
1. VS2003 has a problem locking dependencies, requiring a restart every 5-10 debugging runs.
2. VS2005 gracefully handles a number of errors by flat-out shutting down with no error message.
3. VSWebCache (in Documents and Settings)...let's just say it's too aggressive in what it caches. I haven't noticed the issues so much in 2.0
4. Lack of #regions (or something comparable) in the web.config. As the config file gets larger & larger, I'd really rather collapse large sections without having to collapse the parent node, which would hide siblings that I may want to see. Means more scrolling to find a particular value.
5. I don't even know where to classify this or even how to technically describe this, but the SQL Express integration with 2.0 is flaky as hell. I get timeout errors reading profile properties about as often as VS2003 locks dependency DLLs. Hell, even installing the thing tore up my MSSQL2000 installation to where I was in the registry manually cleaning up entries.
6. Ability to restart a single web app without restarting the entire IIS instance...especially crucial when the application cache (and various singletons) get stale & I need to have the *app* reinitialize (killing the asp worker thread is a common daily task for me)

I agree with posters above. Installing and configuration are the true achille's heel of .net.

When running a ASP.net app connected to an SQL 2005 DB, an error 40 and/or 32 kept re-occuring. The various permissions-related solutions offered on the web did not help.

I eventually ended up trying an obscure solution involving changing around the user account on that particular computer. I still cannot figure out what was wrong or what I had to do to fix it.

lack of good obfuscation tools suck - even after obfuscation the .NET code is too easy to decode - so intellectual property is not preserved so goodcas in Java or in native (non-managed) code.

There's a definite lack of decent obfuscation tools.

Microsoft won't deliver the framework through Windows Update and/or make a linker. The unavoidable fact is that broadband penetration in the US is still poor with declining uptake.

We have an application that relies on IIS on the desktop. If uses install IIS AFTER the .NET framework, the IIS install does not work, because the mappings do not get set. That bites.

Easier way to run multiple version of framework apps on IIS.

First, you have to make the folks that build the WMI/ADSI APIs actually *use* the APIs that they designed. They are way under-documented and very scripting unfriendly (you wind up spending a lot of time writing wrappers so the experience doesn't suck).

This is by far the biggest hurdle to deploying applications to servers. Come up with a decent, unified model for scripting deployment would be the biggest thing to invest money in.

Things that are hard to do:

1) Writing/reading ACLs on files without shelling out to something like cacls
2) Discovering what the regional settings for Windows are.
3) Creating accounts, setting privileges on those accounts, assigning group membership.
4) Controlling WLBS clusters from script.

Reducing friction in:

1) Make it easier to generate web.config files for each deployment environment.
2) Make it easier to sync databases between different environments

Start really supporting Mono. Make it easier for me to use the great technology, namely .Net, on the platforms where I need it the most, (not always Windows). Trying to make windows a requirement for .Net at any level makes me more willing to not use or recommend it.

Make .Net applications work better with !IE. Again, make using .Net an easier decision, and I'll make that decision more easily.

i've not have a single good experience as a user in using a .net app. All the devs' excuses about loading the framework that takes up 30+ seconds before i can even do anything on the ugly winform is a bad experience.

i can do a similar app in RoR, and get it loaded and usable in seconds.

I just cant roll it out though as i'm in a windows/.net world.

It just takes too long to write a "simple" but user friendly setup programme using Windows Installer and I shouldn't have to pay a grand to some third party just to fix all the shortcomings such as:
- severe limitations on number of dialogue boxes that can be used (eg 12 text boxes maximum)
- limitations on amount of boilerplate text on a dialogue (three lines for the whole dialogue)
- inability to put ANY boilerplate text on some of the dialogue boxes (the PROGRESS bar - it would be nice to tell the user that this is going to take quite a bit of time if they selected such and such an option)

I appreciate that the whole thing has to be a bit like polishing a turd because of the underlying "foreign key guids all over the place" tables that actually comprise an msi, but even so....
We're moving to writing our own web installation code (much like Community Server does) but that's a lot of code and a lot of expense for a small software house. I guess my biggest gripe is that the limitations of writing a setup program aren't very well known or publicised and it's only when you get to the "coding" stage you start hitting the brick walls.

Config files, config files, config files - the biggest bane in my life.

They can be brittle, ie type information is very fussy. Sometimes the assembly shouldn’t be provided, sometimes the simple assembly name is required, sometimes it needs to be fully qualified.

Naturally there is different config between Dev, Test, Prod. Config files ensure I can’t simply “xcopy” everything. It’s always xcopy everything apart from the .config file – oh unless it’s a new installation.

A lot of presentational/UI work(ie the fruits of my labour) is delivered in Excel. Naturally, I’ve ended up with an Excel.exe.config file in everybody’s Office installation. I’m just waiting for the day when some other installation replaces my Excel.exe.config file with theirs. It’s only a matter of time, user’s already have an Outlook.exe.config file from a third-party application.

What’s my Config solution? - I’m doing away with the damn things (as much as I can). All my applications have a DB element in them. So I’m putting a connection string in the registry, holding the config in the DB, and writing a few classes to serve it all up. I may support some std. System.Configuration base classes and Interfaces but any over-engineered nonsense from the Patterns & Practises group is going to be ignored.

Whilst I’m venting the spleen. Microsoft Visual Studio 2005 Tools for Office. It’s a start. A hacky patch of a start. But I don’t want to craft a single “Smart Document” of an Expenses Submission form or Sales Stats Capturing spreadsheet. (Do marketing think up these contrived samples? They seem to spend a lot of time submitting expense claims.). I want to deliver access to re-usable libraries of .net code to multiple Excel spreadsheets., ie write an .xla in C# and call the functions from a cell.

And finaly, Microsoft Visual Studio 2005 Team Edition for Software Developers, Microsoft Visual Studio 2005 Tools for Office. Who keeps on coming up with these names??

Please, please, please make the .NET framework a required download on Windows Update.

If Microsoft really wants everyone developing in .NET then Windows Update should support that decision!

Thank you.

what about making an sort of an "prefix" you can enter in the machine.config. and when theres the prefix "test" it uses the "web.test.config" if its there.... think thats one of the easiest solutions. its really silly, to rely on programms that does this for you, so you can easy deploy.

having a separate installation for .NET 1.1 SP1 is a real pain; why can't there be a single installation? I hope this mistake won't be repeated with .NET 2.0

I'm focus on .Net Office apps and on Sharepoint, and the current pain point is that the VSTO and Sharepoint teams seem to be unaware of each other's existence - even though they both occupy the Office-in-the-enterprise space.

Hopefully this will be fixed in the 2007 versions. However, no-one seems to be talking about it at the moment that I can see.

If this doesn't make sense to you, try setting the document template on a Sharepoint Document Library to a Word Template (.dot) with a code-behind dll. GFL!

My reply is a little long, so I've posted it as an entry on my own blog.

http://www.munit.co.uk/blog/archives/2006/04/managing_configuration_setting.html

Thanks.

Mun

Let me state, I have been using DotNet for 5 years, I am a big proponent of MS in general.

First of all, I am using VS.NET 2003 and 2005. VS2005 Crashes ALL THE TIME. Specifically it crashes when working with Oracle and mySQL DB's in Reporting Services during query construction. My work around? I downloaded some mySQL tools (FREE) construct my Queries then copy/paste them to my RS Project. We ARE using MS SQL 2005, but our vendors use everything from Pervasive, mySQL, Postgres. Oracle 8,9,10. So I need better tools in the IDE to use *GASP* non-MS products, THIS IS THE REAL WORLD, NOT EVERYONE USES MS! Where am I going now?
I have been teaching myself ROR (Ruby on Rails) I must say that the SPEED increase I experienced is awesome. I wanted to create a simple task list, I started it ASP.NET and finished it in ROR. ASP.NET I spent 8 hours hooking up the Database etc. I was getting obscure errors about Data types not matching in the Update Query (Auto-Generated by VS2005). I had never used mySQL and ROR. After taking the ONLamp Tutorial written by Curt Hibbs, I created my project (after installing Ruby etc) in about 2 hours. From Zero to Finished! I like Dot.Net I am sure there are many challenges in Ruby. I am no-longer drinking the Kool-Aid, MICROSOFT HAS TO DO BETTER.

By the way I downloaded and used Eclipse (RAD rails) http://www.radrails.org/.

Awesome! Fast and clean. Never used Eclipse before, but Microsoft Drove me to it.

Client-side deployment is our issue as well. Short of making the .NET runtimes a required OS update, how about a MSI compliant MergeModule that at least encapsulates dotnetfx.exe? A full-on MSM would be preferable, but even this normalization would help. Thanks for trying to gather feedback.

Oooh thought of something else more “on topic”. We have the wonderfully finely grained model that is CAS, where I can control all security type .net-ty things. And then… two bloody great big dirty off buttons within IE Internet Security settings.

Naturally this caused a corporate-political stand-off at the corp I was employed with for a while. I was able to dictate the policy for the CAS for the whole organisation – punching slightly above my weight but my team had the .net knowledge and on-one else did at the time. I wasn’t able to stop 3rd party (Reuters if I remember correctly) applications from wiping my carefully managed enterprise & machine config files with generic copies – but I could get round that by having the corporate copies re-deployed every night.

What I could never gain control of … was the two great big dirty off buttons in IE’s Internet Security settings. What are they doing there? Surely it’s all covered by CAS? Needless to say some IE admin team had switched them all to off for all Security Zones. I couldn’t get them to change policy so I couldn’t host/re-use .net controls in our corporate web applications.

Backwards compatibility. Why must we go through this pain every few years to upgrade our ASP.NET apps? Did MS get it so wrong with the last version that they had to reinvent themselves again? Why would any developer invest their time in a framework that changes so much with every new version. It's hard to imagine that an ASP.NET app I write today could be obsolete in a few years.

Take a lesson from RoR. New version = full backwards compatibiliity. Beautiful.

Also, client side reporting is still joke - I know we have reporting services but on the client you just need a WinForms application to generate some nice reports that can be printed and emailed.

I know Crystal is bundled but even after being on the market since 1492 it still contains too many bugs, it is too difficult to use, too difficult to deploy and if you have a non-trivial report to write it just does not come to the party. Also the bundled version is too cut down.

ClickOnce is very close to being a viable option for internal deployment, but not until the following items are addressed:

 - Ability to do silent uninstalls
 - Hooks for install/uninstall custom actions
 - Option to install for "all users" as opposed to per-user (so the apps can be included in a machine image for example)

i tried your example in NET Zero Deployment
Security and Versioning Models in the Windows Forms Engine Help You Create and Deploy Smart Clients (http://msdn.microsoft.com/msdnmag/issues/02/07/NetSmartClients/)

i followed all the steps but unfortunately this error page appears:

The page cannot be found
The page you are looking for might have been removed, had its name changed, or is temporarily unavailable.

--------------------------------------------------------------------------------

Please try the following:

If you typed the page address in the Address bar, make sure that it is spelled correctly.

Open the localhost home page, and then look for links to the information you want.
Click the Back button to try another link.
HTTP 404 - File not found
Internet Information Services


--------------------------------------------------------------------------------

Technical Information (for support personnel)

More information:
Microsoft Support


can you help me out:) thanks

I like the idea of a client version of the framework, or at least some configurable version that you could define what your client needs to run your app. I understand the concept of one big framework that has everything you could ever want, but my clients don't understand that.

Also, what about a central configuration file? I run all of my web apps/services on a farm. When I deploy I have to make sure that if I make one change to a web.config file that all of the apps get updated as well. I wish there was a way that I could define in a section where the actual config was. Sortof like Assmbly Binding Redirect, but its more like Configuration Settings Redirect.

The installation of the framework.

The fact that it has System.Everything.YouCouldPossiblyThinkOf.ToHelpMe.ToEvenScratchMyButt but it doesn't have a good PDF generating library built into it??? WTF? I mean really everything and the kitchen sink but no built in PDF library? What gives?

No fixed point library for small device work. Yes, I can write one and I have but I shouldn't have had to do so. If I am going to use a managed language I want a reason to do so. I can code a fixed point library in C as well as I can in .Net and it will run faster so why should I use .Net on small devices if it doesn't give me basic things needed for small device work? Many small devices say they support float types but the support blows and fixed point works way better and much faster. It is another case of everything and the kitchen sink for small devices in .Net but no fixed point? Again what gives?

I've dug into MSI a bit at past jobs and all I can say is you've made what should be simple (installing an application) really complicated. It's too darn painful. Maybe you need to come up with a new idea of how to do this.

It would be nice if I could write installation code in .Net instead of this cryptic MSI stuff. Give us some installation framework classes we can code against. Maybe even have it generate the default code based on where I say files belong. Let me run it through the VS debugger so I can see what goes wrong. Then just add a new installation project type.

Of course, this solution necessitates that the .Net framework is already installed on the machine (Windows Update?). Trying to get that in an install alone is painful.

I've also done corp. rollouts. Pushing out sofware through AD isn't fun, but we got it to work. We're experimenting with Click Once. Maybe that'll be better, although it has some limitations.

Give us some built-in tools in .Net so we can easily build in common support tasks. For example, if I push a button in the about box, I might want a dialog to come up showing all references assemblies and their versions and location. Let me get that list with a quick API call so I can just bind it to a control and I'm done...or serialize it into an email message to the help desk.

For ASP.Net applications, a similar ability to grab all referenced assemblies would be helpful. It can get very painful figuring out what ASP.Net is binding to sometimes.

The new VS2005 idea of hiding all the partial class information is horrible. Please leave those in the projects. Stop hiding things in obscure directories littered around the computer. We've sometimes spent a lot of time tracking down these things when figuring out problems. I like that Scott Guthrie's new web projects put the partial classes right in the solution where I can find them and see them.

The size of the framework is a big problem, I won't develop for .Net until I can be sure my customers have it preinstalled and I don't see that happening for a long time.

The fact it will come with Vista won't be a solution, I can't ask my customers to upgrade their operating system just so they can run my software.

Off the top of my head:

-Managing multiple app.configs, one for each deployment environment, and keeping them in sync
-Reconciling click-once application sandboxing with admin rights required to install performance counters etc...
-Syncing database schemas
-Remote fault diagnosis: eg getting enough out of a stack trace to repro problems (eg: why aren't 'primitive' value-type parameters included in the Exception.tostring() - save a bucket load of exception-wrapping)

Does anyone have a simple How-to guide for dummies?

My woes: Distributing a ClickOnce deployed application to a customer site.

We're an ISV. We've built a WinForms application to sell to our customers. We want to simplify deployment at the client site and ClickOnce is a great option.

Now the problem: VS2005 only allows deployment to a *specific* website and update URL. We need to create an installer that will setup the ClickOnce deployment site from a CD/download, allowing the customer to choose the location. Currently this is not possible.

And in case you need another vote, making the framework a required update would be great too.

With Framework 2.0 I can host a Winforms control in a web page with code running in the Internet Zone permission set. Everything works fine. However, if I create a Click Once application which also has it's runtime permissions limited to the internet zone it doesn't just deploy. The "Publisher cannot be verified" dialog pops up. Does this mean that "Internet Zone" (Click Once) apps (.exe ) which stay within the "Internet Permission set" must be signed with authenticode but winforms controls hosted in .htm pages do not?

"Please make the framework part of windows update"

WTF!?

Has this guy ever use dial up? Windows Updates take forever at the moment, imagine downloading more than a couple of megs.

That guy has smoked way too much crack.

monk.e.boy

Please ignore monkey boy! The fact that some people living in the stone age with 56k modems does NOT mean we should scale back our efforts as software developers. If that were the case, we might as well be coding everything in small compact ASM.

PLEASE make the .NET framework a required Windows Update and include it as installed by default with the operating system.

My biggest problem right now is having to diagnose strange IIS setup scenarios also Windows Authentication issues... I don't want to become an IIS expert!