Resolve 500 Internal Server Errors

If you receive a 500 (Internal Server Error) when running an application in IIS (SharePoint or otherwise), what can you do to resolve it? The ULS logs won’t show anything because the error happens before SharePoint or your web application has a chance to log an error. However, IIS has its own trace logging which can be easily turned on.

1. Open IIS and click on the node for the site returning the error
2. In the Actions pane on the right, click Failed Request Tracing…
   
3. Check Enable
   
4. In the center pane, click the Failed Request Tracing Rules icon
5. In the Actions pane on the right, click Add…
   
6. In the wizard that opens, click next to select all content
7. Then check Status codes(s) and enter 500, or any other error codes you want to trace
   
8. Click Next to select all trace providers, and the Finished
9. You will now see logs get created for failed request in the following folder:C:\inetpub\logs\FailedReqLogFiles
10. Clicking on one of the log files will display the following information. In this case we had a duplicate handler <add> element in the web.config – very easy to diagnose and correct with tracing enabled!
    

Deleting TFS Work Items

When setting up team projects in TFS sometimes it’s helpful to create a few test issues or bugs to make sure any customizations you have made function correctly. However, you will quickly notice there is no option to delete these work items from the UI or administrative console. It can be done though via the Command Prompt. Here’s how:

1. Open a Visual Studio Command Prompt window
   
2. Enter the following command:
   witadmin destroywi /collection:<team project collection URL> /id:<work item id>
   

For more information on using the Work Item Type Administration console utility see: http://msdn.microsoft.com/en-us/library/dd312129.aspx

Permission Work Item Types in TFS

When tracking work items for a development project you may want to capture issues reported by business users. However, giving business users access to TFS can lead to a lot of bugs getting reported that are really just issues that need to be triaged, broken down, or better categorized. One issue may be resolved by fixing 3 different bugs, 2 tasks being performed, and perhaps a test case developed. An issue can then be turned into a bug once the development team has reviewed it, or several different types of work items and related back to the original issue. But how can we ensure business users only enter issues, not bugs?

You might think work item types can be assigned permissions by user or groups. However, TFS currently doesn’t support this. There is a way though to still prevent a group of users from creating work items by type.

1. Create a global group for business users and add them as members:
   

   

2. Install the TFS Power Tools for VS if you haven’t already.
   http://visualstudiogallery.msdn.microsoft.com/c255a1e4-04ba-4f68-8f4e-cd473d6b971f
3. The Power Tools install the Process Editor which allows you to edit work item types. Open each WIT for your project and go to the workflow tab:
   
4. Select the first transition and set Not to the business user group you just created. The prevents the first workflow step from executing if the current user is in the group.
   
5. Now users in the business group cannot save work items of type Bug:
   

Of course, this approach merely prevents users from creating work item types they don’t have access to. We still need to train business users to enter issues into TFS since the error message is not very informative. And of course security trimming would be nice but at least we have the bare minimum of access control by item type.

October CU Fails to Install

After you install SharePoint cumulative updates you need to run the SharePoint 2010 Products Configuration Wizard (psconfig.exe). However, while installing the October 2011 cumulative update recently, I found it was failing on the last step where the udpate gets applied. I discovered that the SharePoint 2010 Timer service was getting shut down after the first step when running psconfig which in turn made the last step fail. The solution is simple: keep the Services console open when running psconfig and restart the Timer service before reaching the last step!

IIS Timeout When Debugging in VS

You can stop the Visual Studio debugger from timing out and stopping by changing a single setting in IIS. Just select the application pools you want and go to Advanced Settings in the IIS Manager.

There you can set the Ping Enabled property to False which will keep the debugger alive indefinitely.

You should also do this for the service application pool as well so things like the profile service don’t time out. Since it’s identified by a GUID you need right-click and View Applications to find the right pool.

jQuery Selectors with Server-Side ID’s

If you want to bind a client-side function to an ASP.NET control using a jQuery selector you can use the attribute selector id$= to specify what the client id ends with. The leading underscore is added to match only complete server id’s.

Take an ASP.NET control:

Which gets rendered with the following id:

We can bind an event handler in jQuery using the following selector:

This is a handy way to wire up client-side functionality to your server-side ASP.NET controls.

Deployment Conflict Resolution in Visual Studio

If you are provisioning a ListInstance in Visual Studio you may see the following error message:

The URL or name of this list instance conflicts with a list instance already on the server.
The list instance on the server will be deleted before deploying the new list instance.

If you select Resolve Automatically, your list and all of its data will be removed before a new instance of the list is provisioned. To prevent this, you can set the Deployment Conflict Resolution property of the list instance in the Solution Explorer to None. Simply right-click on the list instance project item in VS and select Properties. Then select None from the dropdown list of values:

Logging SPRequest Allocation Call Stacks

When writing code to extend SharePoint functionality it is important to make sure we are disposing SPWeb and SPSite objects carefully. However, no matter how careful you are a few undisposed webs and sites can creep into your code. If this happens you will see entries in the log files that say "An SPRequest object was not disposed before the end of this thread".

This general message doesn’t help though to find the specific problem, you need to see the actual call stack. To do this, you can enable CollectSPRequestAllocationCallStacks using a PowerShell script like the following.

Afterwards, you will see the full stack trace where the web or site was not disposed.

Cross-Site Collection Navigation

Background

SharePoint uses the provider model for supplying data to its navigation controls. The v4.master, for instance, uses an AspMenu control for the top navigation whose DataSourceId is set declaratively to an instance of a SiteMapDataSource. The data source control is wrapped in a delegate control so it can be easily swapped out using a feature.

<SharePoint:AspMenu
  ID="TopNavigationMenuV4"
  Runat="server"
  EnableViewState="false"
  DataSourceID="topSiteMap"
  AccessKey="<%$Resources:wss,navigation_accesskey%>"
  UseSimpleRendering="true"
  UseSeparateCss="false"
  Orientation="Horizontal"
  StaticDisplayLevels="2"
  MaximumDynamicDisplayLevels="1"
  SkipLinkText=""
  CssClass="s4-tn" />
<SharePoint:DelegateControl runat="server" ControlId="TopNavigationDataSource" Id="topNavigationDelegate">
    <Template_Controls>
        <asp:SiteMapDataSource
          ShowStartingNode="False"
          SiteMapProvider="SPNavigationProvider"
          id="topSiteMap"
          runat="server"
          StartingNodeUrl="sid:1002"/>
    </Template_Controls>
</SharePoint:DelegateControl>

The data source control has a SiteMapProvider property which can be set to any of the named sitemap providers declared in the web.config file.

<siteMap defaultProvider="CurrentNavigation" enabled="true">
  <providers>
    <add name="SPNavigationProvider" type="Microsoft.SharePoint.Navigation.SPNavigationProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="SPSiteMapProvider" type="Microsoft.SharePoint.Navigation.SPSiteMapProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="SPContentMapProvider" type="Microsoft.SharePoint.Navigation.SPContentMapProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="SPXmlContentMapProvider" siteMapFile="_app_bin/layouts.sitemap" type="Microsoft.SharePoint.Navigation.SPXmlContentMapProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="ExtendedSearchXmlContentMapProvider" description="Provider for navigation in Extended Search pages" siteMapFile="_app_bin/layouts.sitemap" type="Microsoft.Office.Server.Search.Extended.Administration.Common.ExtendedSearchXmlContentMapProvider, Microsoft.Office.Server.Search, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="AdministrationQuickLaunchProvider" description="QuickLaunch navigation provider for the central administration site" type="Microsoft.Office.Server.Web.AdministrationQuickLaunchProvider, Microsoft.Office.Server.UI, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="SharedServicesQuickLaunchProvider" description="QuickLaunch navigation provider for shared services administration sites" type="Microsoft.Office.Server.Web.SharedServicesQuickLaunchProvider, Microsoft.Office.Server.UI, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="GlobalNavSiteMapProvider" description="CMS provider for Global navigation" type="Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider, Microsoft.SharePoint.Publishing, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" NavigationType="Global" EncodeOutput="true" />
    <add name="CombinedNavSiteMapProvider" description="CMS provider for Combined navigation" type="Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider, Microsoft.SharePoint.Publishing, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" NavigationType="Combined" EncodeOutput="true" />
    <add name="CurrentNavSiteMapProvider" description="CMS provider for Current navigation" type="Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider, Microsoft.SharePoint.Publishing, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" NavigationType="Current" EncodeOutput="true" />
    <add name="CurrentNavSiteMapProviderNoEncode" description="CMS provider for Current navigation, no encoding of output" type="Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider, Microsoft.SharePoint.Publishing, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" NavigationType="Current" EncodeOutput="false" />
    <add name="GlobalNavigation" description="Provider for MOSS Global Navigation" type="Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider, Microsoft.SharePoint.Publishing, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" NavigationType="Combined" Version="14" />
    <add name="CurrentNavigation" description="Provider for MOSS Current Navigation" type="Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider, Microsoft.SharePoint.Publishing, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" NavigationType="Current" Version="14" />
    <add name="SiteDirectoryCategoryProvider" description="Site Directory category provider" type="Microsoft.SharePoint.Portal.WebControls.SiteDirectoryCategoryProvider, Microsoft.SharePoint.Portal, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="MySiteMapProvider" description="MySite provider that returns areas and based on the current user context" type="Microsoft.SharePoint.Portal.MySiteMapProvider, Microsoft.SharePoint.Portal, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="MySiteLeftNavProvider" description="MySite Left Nav provider that returns areas and based on the current user context" type="Microsoft.SharePoint.Portal.MySiteLeftNavProvider, Microsoft.SharePoint.Portal, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
    <add name="MySiteSubNavProvider" description="MySite Sub Nav provider that returns areas and based on the current user context" type="Microsoft.SharePoint.Portal.MySiteSubNavProvider, Microsoft.SharePoint.Portal, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
  </providers>
</siteMap>

Problem

This arrangement works well when you stay within a site collection. However, if you need to show navigation nodes from another site collection you will need another approach. The problem is that the SiteMapProvider is limited to returning only nodes in the current site collection. If you have created, for example, a My Site Host site collection, and want to display the root site collection’s top navigation at the top of the page instead of the default My Site Host navigation, you need a way to read the root site collection’s navigation nodes.

Solution

The solution provided below is for a publishing site and uses the PortalSiteMapProvider. The PortalSiteMapProvider has a method called GlobalNavSiteMapProvider which can be used to get the top navigation nodes as a SiteMapNodeCollection. We can interate over these nodes in a custom control and build up the navigation for our site in whatever way we want.

But, to return a SiteMapNodeCollection from another site collection we need a reference to the PortalSiteMapProvider in the context of the other site collection. One way to do this is to use an HTTP handler that can be called from the Layouts folder (e.g., _layouts/handlers/navigation.ashx) and then add the appropriate site collection’s URL. If we look at the following three site collections, their URL’s all point to the same HTTP handler:

• http://rootsitecollection/_layouts/handlers/navigation.ashx
• http://rootsitecollection/mysites/_layouts/handlers/navigation.ashx
• http://rootsitecollection/searchcenter/_layouts/handlers/navigation.ashx

If we call the first URL from either the Search Center or My Site Host site collections, the PortalSiteMapProvider, if it is referenced in the handler, will return navigation nodes from the root site collection. Note that this method will only work within a single web application. If we attempt to reach across applications this would amount to cross-site scripting which is prevented for security reasons.

The custom handler can then be called asynchronously from client code and a serialized set of navigation nodes returned to the browser and formatted using client script.

Implementation

To implement this solution, add an HTTP handler to your project. The following assumes you have a mapped Layouts folder in a Visual Studio 2010 SharePoint project. Right-click the Layouts folder and Add an ASP.NET Handler from the Web item list.

This will create a class that inherits from IHttpHandler which requires you to implement a single method, ProcessRequest, and a single property, IsReusable.

public void ProcessRequest(HttpContext context)
{
    SPSecurity.RunWithElevatedPrivileges(GetGlobalNav);
}

public bool IsReusable
{
    get { return true; }
}

The IsReusable property allows subsequent requests to reuse the same instance of the handler. ProcessRequest acts as our entry point and, to ensure our method that retrieves navigation nodes has sufficient privileges, we call it with RunWithElevatedPrivileges.

The GetGlobalNav method then gets a reference to the PortalSiteMapProvider which then gets a SiteMapNodeCollection based on the root node. This collection is first passed into a method that converts it into a collection that can be easily serialized and then finally we call SerializeWrite to output JSON to the client.

private void GetGlobalNav()
{
    try
    {
        //Note: PortalSiteMapProvider returns data for the current request's site collection
        var rootNode = PortalSiteMapProvider.GlobalNavSiteMapProvider.RootNode;
        if (rootNode != null)
        {
            var nodes = PortalSiteMapProvider.GlobalNavSiteMapProvider.GetChildNodes(rootNode);
            SerializeWrite(GetSerializableSiteMap(nodes));
        }
    }
    catch (Exception e)
    {
        //log exception
    }
}

Since the generic List<t> can be easily serialized if we use a custom struct, all we need to do in GetSerializableSiteMap is build up a List using the specified SiteMapNodeCollection.

private List<SerializableSiteMapNode> GetSerializableSiteMap(SiteMapNodeCollection nodes)
{
    var serializableSiteMap = new List<SerializableSiteMapNode>();

    foreach (SiteMapNode node in nodes)
    {
        var serializableSiteMapNode = new SerializableSiteMapNode()
        {
            Description = node.Description,
            HasChildNodes = node.HasChildNodes,
            Key = node.Key,
            Title = node.Title,
            Url = node.Url
        };

        if (_siteMapLevel < Depth && node.HasChildNodes)
        {
            _siteMapLevel++;
            serializableSiteMapNode.ChildNodes = GetSerializableSiteMap(node.ChildNodes);
        }

        serializableSiteMap.Add(serializableSiteMapNode);
    }

    return serializableSiteMap;
}

public struct SerializableSiteMapNode
{
    public List<SerializableSiteMapNode> ChildNodes;
    public string Description;
    public bool HasChildNodes;
    public string Key;
    public string Title;
    public string Url;
}

The result of this is passed into a method that handles writing the HTTP response. Fortunately, .NET has a JavaScript serializer which makes our job easier.

private void SerializeWrite(object objectToWrite)
{
    var outputString = String.Empty;

    var serializer = new JavaScriptSerializer();
    outputString = serializer.Serialize(objectToWrite);

    HttpContext.Current.Response.Write(outputString);
}

The custom handler can be called in a user control using the appropriate site collection URL and the response data formatted using jQuery. We have added a user control to the top of the master page in our My Site Host site collection that calls our handler using a relative reference to the root site collection ("/"). So while we display the navigation in the My Site Host site collection, the nodes will be pulled from the root (main) site collection using the URL "/_layouts/handlers/navigation.ashx". The user control loads an empty unordered list element whose list items are built up asynchronously based on the response from our handler.

<script language="javascript" type="text/javascript">
    var handlerUrl = '/_layouts/handlers/navigation.ashx?&Depth=2';
    var level = 0;

    $(document).ready(function () {
        ul = $('#navGlobal');
        ul.children().remove();

        $.get(handlerUrl, function (data) {
            if (data != '') {
                tags = eval(data);
                writeNodes(tags, ul);
            }
        });
    });

    function writeNodes(tagNodes, ulNode) {
        $(tagNodes).each(function (i, n) {
            if (n.Url == null || n.Title == null) return true;

            var a = $(document.createElement('a'));
            a.html(n.Title);
            a.attr('href', n.Url);
            if (level == 0) a.attr('class', 'top_link');

            var li = $(document.createElement('li'));
            li.append(a);

            if (n.HasChildNodes) {
                level++;
                var ul = $(document.createElement('ul'));
                writeNodes(n.ChildNodes, ul);
                li.append(ul);
            }

            ulNode.append(li);
        });
    }
</script>

<ul id="navGlobal" />

This is a simple approach to simply reading and displaying navigation nodes from one site collection in another. We have not tried to merge navigation nodes using this method but I don’t see why one could not.

In addition, we have only used our handler to return data to the client. If you need to return it to server-side code, you can simply create a WebRequest object on the server that calls the handler. The data should then be serialized as XML and not JSON so you can use the .NET XML object model to handle the response.

Creating Profile Properties

The SharePoint User Profile Service comes with a lot of profile properties out of the box. It’s unlikely however that these properties will fulfill all your requirements. Instead of adding custom properties through the UI, it is better to create a feature to provision them to the profile service on deployment.

Before we look at creating a feature to deploy our properties, we want to create a property manager for controlling how properties get added and updated in our profile store. Our manager is based on the one described in this excellent post: Creating profile properties and sections the SharePoint 2010 way.

The manager’s constructor takes an SPSite in order to store references to a series of SharePoint user profile managers.

private SPSite _site;
private SPServiceContext _serviceContext;
private SharePointUserProfiles.UserProfileManager _profileManager;
private ProfileSubtypePropertyManager _profileSubtypePropertyManager;
private UserProfileConfigManager _userProfileConfigManager;
private ProfilePropertyManager _profilePropertyManager;
private CorePropertyManager _corePropertyManager;
private ProfileTypePropertyManager _profileTypePropertyManager;

public UserProfilePropertyManager(SPSite site)
{
    _site = site;
    _serviceContext = SPServiceContext.GetContext(_site);
    _userProfileConfigManager = new UserProfileConfigManager(_serviceContext);
    _profilePropertyManager = _userProfileConfigManager.ProfilePropertyManager;
    _profileManager = new SharePointUserProfiles.UserProfileManager(_serviceContext);
    _profileSubtypePropertyManager = _profileManager.DefaultProfileSubtypeProperties;
    _corePropertyManager = _profilePropertyManager.GetCoreProperties();
    _profileTypePropertyManager = _profilePropertyManager.GetProfileTypeProperties(ProfileType.User);
}

Once the manager is instantiated, we can call methods to:

1. Create or update profile property sections
2. Enable users to override the privacy setting
3. Create or update profile properties by type (date, integer, Boolean, and string)

public class UserProfilePropertyManager
 {
     #region Private Properties

     private SPSite _site;
     private SPServiceContext _serviceContext;
     private SharePointUserProfiles.UserProfileManager _profileManager;
     private ProfileSubtypePropertyManager _profileSubtypePropertyManager;
     private UserProfileConfigManager _userProfileConfigManager;
     private ProfilePropertyManager _profilePropertyManager;
     private CorePropertyManager _corePropertyManager;
     private ProfileTypePropertyManager _profileTypePropertyManager;

     #endregion

     #region Constructors

     public UserProfilePropertyManager(SPSite site)
     {
         _site = site;
         _serviceContext = SPServiceContext.GetContext(_site);
         _userProfileConfigManager = new UserProfileConfigManager(_serviceContext);
         _profilePropertyManager = _userProfileConfigManager.ProfilePropertyManager;
         _profileManager = new SharePointUserProfiles.UserProfileManager(_serviceContext);
         _profileSubtypePropertyManager = _profileManager.DefaultProfileSubtypeProperties;
         _corePropertyManager = _profilePropertyManager.GetCoreProperties();
         _profileTypePropertyManager = _profilePropertyManager.GetProfileTypeProperties(ProfileType.User);
     }

     #endregion

     #region Display Order

     private int GetLastDisplayOrderForSection(ProfileSubtypeProperty section)
     {
         int displayOrder = 0;

         foreach (ProfileSubtypeProperty profileSubtypeProperty in _profileSubtypePropertyManager.PropertiesWithSection)
         {
             if ((section.DisplayOrder < displayOrder) && (profileSubtypeProperty.IsSection))
             {
                 break;
             }
             displayOrder = profileSubtypeProperty.DisplayOrder;
         }
         return displayOrder;
     }

     private int GetLastDisplayOrderForSection(string sectionName)
     {
         var section = _profileSubtypePropertyManager.GetSectionByName(sectionName);
         return GetLastDisplayOrderForSection(section.Name);
     }

     private int GetDisplayOrderForLastSection()
     {
         int displayOrder = 0;

         foreach (ProfileSubtypeProperty profileSubtypeProperty in _profileSubtypePropertyManager.PropertiesWithSection)
         {
             if (profileSubtypeProperty.IsSection)
             {
                 if (profileSubtypeProperty.DisplayOrder > displayOrder)
                 {
                     displayOrder = profileSubtypeProperty.DisplayOrder;
                 }
             }
         }
         return displayOrder;
     }

     #endregion

     #region Sections for Properties

     private ProfileSubtypeProperty GetSectionForProperty(ProfileSubtypeProperty property)
     {
         ProfileSubtypeProperty section = null;

         foreach (ProfileSubtypeProperty profileSubtypeProperty in _profileSubtypePropertyManager.PropertiesWithSection)
         {
             if (profileSubtypeProperty.IsSection)
             {
                 section = profileSubtypeProperty;
             }
             if (profileSubtypeProperty.Name == property.Name)
             {
                 break;
             }
         }
         return section;
     }

     private ProfileSubtypeProperty GetSectionForProperty(string propertyName)
     {
         var property = _profileSubtypePropertyManager.GetPropertyByName(propertyName);
         return GetSectionForProperty(property.Name);
     }

     internal ProfileSubtypeProperty CreateOrUpdateSection(string name, string displayName)
     {
         if ((_corePropertyManager.GetPropertyByName(name) != null)
             || (_profileTypePropertyManager.GetPropertyByName(name) != null)
             || (_profileSubtypePropertyManager.GetPropertyByName(name) != null))
         {
             throw new Exception(String.Format("WARNING! Property with the same name ‘{0}’ exists.", name));
         }

         CoreProperty coreProperty = _corePropertyManager.GetSectionByName(name);
         bool exist = (coreProperty != null);

         if (!exist)
         {
             coreProperty = _corePropertyManager.Create(true);
             coreProperty.Name = name;
         }

         coreProperty.DisplayName = displayName;

         if (exist)
         {
             coreProperty.Commit();
         }
         else
         {
             _corePropertyManager.Add(coreProperty);
         }

         var profileTypeProperty = _profileTypePropertyManager.GetSectionByName(name);
         if (profileTypeProperty == null)
         {
             profileTypeProperty = _profileTypePropertyManager.Create(coreProperty);
             _profileTypePropertyManager.Add(profileTypeProperty);
         }

         var profileSubTypeProperty = _profileSubtypePropertyManager.GetSectionByName(coreProperty.Name);

         if (profileSubTypeProperty == null)
         {
             // we should get the display order before creating our new section
             // otherwise its display order will be returned when trying to get
             // the display order of the last section
             int lastSectionDisplayOrder = GetDisplayOrderForLastSection();
             profileSubTypeProperty = _profileSubtypePropertyManager.Create(profileTypeProperty);
             _profileSubtypePropertyManager.Add(profileSubTypeProperty);

             // set the position for new section by adding 100 to the last section
             _profileSubtypePropertyManager.SetDisplayOrderBySectionName(name, lastSectionDisplayOrder + 100);
             _profileSubtypePropertyManager.CommitDisplayOrder();
         }

         return profileSubTypeProperty;
     }

     #endregion

     #region Privacy Overrides

     public void EnableUserOverridePrivacy(string propertyName)
     {
         var property = _profileSubtypePropertyManager.GetPropertyByName(propertyName);
         if (property == null) return;

         property.UserOverridePrivacy = true;
         property.Commit();
     }

     #endregion

     #region Create or Update Properties

     private ProfileTypeProperty CreateOrUpdateProfileTypeProperty(CoreProperty coreProperty, bool isVisibleOnEditor, bool isVisibleOnViewer)
     {
         var profileTypeProperty = _profileTypePropertyManager.GetPropertyByName(coreProperty.Name);

         if (profileTypeProperty == null)
         {
             profileTypeProperty = _profileTypePropertyManager.Create(coreProperty);
             _profileTypePropertyManager.Add(profileTypeProperty);
         }

         profileTypeProperty.IsVisibleOnViewer = isVisibleOnViewer;
         profileTypeProperty.IsVisibleOnEditor = isVisibleOnEditor;

         profileTypeProperty.Commit();

         return profileTypeProperty;
     }

     private ProfileSubtypeProperty CreateOrUpdateSubtypeProperty(CoreProperty coreProperty,
         bool required, Privacy privacy, bool userOverridePrivacy, bool isVisibleOnEditor, bool isVisibleOnViewer,
         bool isUserEditable, ProfileSubtypeProperty section)
     {
         var profileTypeProperty = CreateOrUpdateProfileTypeProperty(coreProperty, isVisibleOnEditor, isVisibleOnViewer);
         var profileSubTypeProperty = _profileSubtypePropertyManager.GetPropertyByName(coreProperty.Name);

         if (profileSubTypeProperty == null)
         {
             profileSubTypeProperty = _profileSubtypePropertyManager.Create(profileTypeProperty);
             _profileSubtypePropertyManager.Add(profileSubTypeProperty);

             // set the section if specified and differs from the current one
             if ((section != null) && (section.Name != GetSectionForProperty(profileSubTypeProperty).Name))
             {
                 int displayOrder = GetLastDisplayOrderForSection(section);
                 _profileSubtypePropertyManager.SetDisplayOrderByPropertyName(coreProperty.Name, displayOrder + 1);
                 _profileSubtypePropertyManager.CommitDisplayOrder();
             }
         }

         profileSubTypeProperty.IsUserEditable = isUserEditable;
         profileSubTypeProperty.DefaultPrivacy = privacy;
         profileSubTypeProperty.UserOverridePrivacy = userOverridePrivacy;
         profileSubTypeProperty.PrivacyPolicy = (required ? PrivacyPolicy.Mandatory : PrivacyPolicy.OptIn);
         profileSubTypeProperty.Commit();

         return profileSubTypeProperty;
     }

     internal ProfileSubtypeProperty CreateOrUpdateDateProperty(string name, string displayName, string description, bool required,
         Privacy privacy, bool userOverridePrivacy, bool isVisibleOnViewer, bool isVisibleOnEditor, bool isUserEditable, ProfileSubtypeProperty section)
     {
         return CreateOrUpdateProperty(name, "date", displayName, description, required, privacy, userOverridePrivacy, isVisibleOnViewer,
             isVisibleOnEditor, isUserEditable, section, null, null);
     }

     internal ProfileSubtypeProperty CreateOrUpdateIntProperty(string name, string displayName, string description, bool required,
         Privacy privacy, bool userOverridePrivacy, bool isVisibleOnViewer, bool isVisibleOnEditor, bool isUserEditable, ProfileSubtypeProperty section)
     {
         return CreateOrUpdateProperty(name, "integer", displayName, description, required, privacy, userOverridePrivacy, isVisibleOnViewer,
             isVisibleOnEditor, isUserEditable, section, null, null);
     }

     internal ProfileSubtypeProperty CreateOrUpdateBooleanProperty(string name, string displayName, string description, bool required,
         Privacy privacy, bool userOverridePrivacy, bool isVisibleOnViewer, bool isVisibleOnEditor, bool isUserEditable, ProfileSubtypeProperty section)
     {
         return CreateOrUpdateProperty(name, "boolean", displayName, description, required, privacy, userOverridePrivacy, isVisibleOnViewer,
             isVisibleOnEditor, isUserEditable, section, null, null);
     }

     internal ProfileSubtypeProperty CreateOrUpdateStringProperty(string name, string displayName,
         string description, int length, bool isMultiValued, string separator, string termStoreName,
         string groupName, string termSetName, bool required, Privacy privacy, bool userOverridePrivacy, bool isVisibleOnViewer,
         bool isVisibleOnEditor, bool isUserEditable, ProfileSubtypeProperty section)
     {
         TermSet termSet = null;

         if ((!String.IsNullOrEmpty(termStoreName)) && (!String.IsNullOrEmpty(groupName)) && (!String.IsNullOrEmpty(termSetName)))
         {
             var session = new TaxonomySession(_site);
             var termStore = session.TermStores[termStoreName];
             var group = termStore.Groups[groupName];
             termSet = group.TermSets[termSetName];
         }

         Action<CoreProperty> customSettingsOnCreate = coreProperty =>
         {
             // for multivalue and taxonomic properties the length
             // will be set to 3600 chars by default
             if ((!isMultiValued) && (termSet == null))
             {
                 coreProperty.Length = length;
             }

             coreProperty.IsMultivalued = isMultiValued;

             if ((isMultiValued) && (!String.IsNullOrEmpty(separator)))
             {
                 coreProperty.Separator = (MultiValueSeparator)Enum.Parse(typeof(MultiValueSeparator), separator);
             }

             if (termSet != null)
             {
                 coreProperty.TermSet = termSet;
             }
         };

         Action<CoreProperty> customSettingsOnUpdate = coreProperty =>
         {
             if (termSet != null)
             {
                 coreProperty.TermSet = termSet;
             }
         };

         return CreateOrUpdateProperty(name, String.Format("string ({0} Value)", isMultiValued ? "Multi" : "Single"),
             displayName, description, required, privacy, userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable,
             section, customSettingsOnCreate, customSettingsOnUpdate);
     }

     private CoreProperty CreateOrUpdateCoreProperty(string name, string propertyType, string displayName, string description,
         Action<CoreProperty> customSettingsOnCreate, Action<CoreProperty> customSettingsOnUpdate)
     {
         if ((_corePropertyManager.GetSectionByName(name) != null)
             || (_profileTypePropertyManager.GetSectionByName(name) != null)
             || (_profileSubtypePropertyManager.GetSectionByName(name) != null))
         {
             throw new Exception(String.Format("WARNING! Section with the same name ‘{0}’ exists.", name));
         }

         var coreProperty = _corePropertyManager.GetPropertyByName(name);
         bool exist = (coreProperty != null);

         if (!exist)
         {
             coreProperty = _corePropertyManager.Create(false);
             coreProperty.Name = name;
             coreProperty.Type = propertyType;
         }

         if (coreProperty.Type.ToLower() == propertyType.ToLower())
         {
             coreProperty.DisplayName = displayName;
             coreProperty.Description = description;

             if (exist)
             {
                 // call custom settings method if specified
                 if (customSettingsOnUpdate != null)
                 {
                     customSettingsOnUpdate.Invoke(coreProperty);
                 }
                 coreProperty.Commit();
             }
             else
             {
                 // call custom settings method if specified
                 if (customSettingsOnCreate != null)
                 {
                     customSettingsOnCreate.Invoke(coreProperty);
                 }
                 _corePropertyManager.Add(coreProperty);
             }
         }
         else
         {
             throw new Exception(String.Format("The CoreProperty with the specified name ‘{0}’ exists, but is a type of ‘{1}’ not ‘{2}’ ",
                 name, coreProperty.Type, propertyType));
         }

         return coreProperty;
     }

     private ProfileSubtypeProperty CreateOrUpdateProperty(string name, string propertyType, string displayName, string description,
         bool required, Privacy privacy, bool userOverridePrivacy, bool isVisibleOnViewer, bool isVisibleOnEditor, bool isUserEditable,
         ProfileSubtypeProperty section, Action<CoreProperty> customSettingsOnCreate, Action<CoreProperty> customSettingsOnUpdate)
     {
         var coreProperty = CreateOrUpdateCoreProperty(name, propertyType, displayName,
             description, customSettingsOnCreate, customSettingsOnUpdate);

         var profileSubTypeProperty = CreateOrUpdateSubtypeProperty(coreProperty, required,
             privacy, userOverridePrivacy, isVisibleOnEditor, isVisibleOnViewer, isUserEditable, section);

         return profileSubTypeProperty;
     }

     #endregion
 }

We can then use the UserProfilePropertyManager in a feature receiver to provision profile properties when the feature gets activated. In the following code we update an OOB property (WorkPhone) so that it’s privacy can be set later and then call a helper method (ProcessProperties) that creates or updates the properties.

/// <summary>
/// Configures privacy settings for default user profile properties and adds custom IHI
/// profile properties that match or extend the eDefine user profile.
/// </summary>
[Guid("C8069C3C-03B1-4E31-917F-95659402C4E7")]
public class ProfilePropertyFeatureReceiver : SPFeatureReceiver
{
    private SPFeatureReceiverProperties _properties;

    #region SPFeatureReceiver Overrides

    public override void FeatureActivated(SPFeatureReceiverProperties properties)
    {
        _properties = properties;

        try
        {
            var site = (SPSite)_properties.Feature.Parent;

            if (site == null)
                return;

            site.AllowUnsafeUpdates = true;

            //update privacy settings for default profile properties
            var manager = new UserProfilePropertyManager(site);
            manager.EnableUserOverridePrivacy("WorkPhone");

            //create additional profile properties
            ProcessProperties(manager);

            site.AllowUnsafeUpdates = false;
        }
        catch (Exception ex)
        {
            Springs.GAC.Logging.Logger.Log(ex);
        }
    }

    #endregion

    private void ProcessProperties(UserProfilePropertyManager manager)
    {
        var uri = GetConfigurationFile("UserProfileProperties");
        var root = XElement.Load(uri);
        var sectionElements = root.Element("sections").Elements("section");
        var propertyElements = root.Element("properties").Elements("property");

        var sections = new System.Collections.Generic.Dictionary<string, ProfileSubtypeProperty>();

        foreach (var node in sectionElements)
        {
            var sectionName = (string)node.Element("name");
            var sectionDisplayName = (string)node.Element("displayName");

            var section = manager.CreateOrUpdateSection(sectionName, sectionDisplayName);
            sections.Add(sectionName, section);
        }

        foreach (var property in propertyElements)
        {
            //parent nodes
            var type = property.Element("type");
            var privacy = property.Element("privacy");
            var permissions = property.Element("permissions");
            var taxonomy = property.Element("taxonomy");
            
            //property
            var name = (string)property.Element("name");
            var displayName = (string)property.Element("displayName");
            var description = (string)property.Element("description");

            //type
            var typeName = (string)type.Element("name");

            var length = 0;
            int.TryParse((string)type.Element("length"), out length);

            var isMultiValued = false;
            bool.TryParse((string)type.Element("isMultiValued"), out isMultiValued);

            var separator = (string)type.Element("separator");

            var required = false;
            bool.TryParse((string)type.Element("required"), out required);

            //privacy
            var privacyScope = (Privacy)Enum.Parse(typeof(Privacy), (string)privacy.Element("scope"), true);

            var userOverridePrivacy = false;
            bool.TryParse((string)privacy.Element("userOverridePrivacy"), out userOverridePrivacy);

            //permissions
            var isVisibleOnViewer = false;
            bool.TryParse((string)permissions.Element("isVisibleOnViewer"), out isVisibleOnViewer);

            var isVisibleOnEditor = false;
            bool.TryParse((string)permissions.Element("isVisibleOnEditor"), out isVisibleOnEditor);

            var isUserEditable = false;
            bool.TryParse((string)permissions.Element("isUserEditable"), out isUserEditable);

            //taxonomy
            var termStoreName = (string)taxonomy.Element("termStoreName");
            var groupName = (string)taxonomy.Element("groupName");
            var termSetName = (string)taxonomy.Element("termSetName");

            //properties section
            var section = sections[(string)property.Element("section")];

            switch(typeName)
            {
                case "string":
                   manager.CreateOrUpdateStringProperty(name, displayName, description, length,
                        isMultiValued, separator, termStoreName, groupName, termSetName, required, privacyScope,
                        userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable, section);
                    break;
                case "boolean":
                    manager.CreateOrUpdateBooleanProperty(name, displayName, description, required,
                        privacyScope, userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable, section);
                    break;
                case "integer":
                    manager.CreateOrUpdateIntProperty(name, displayName, description, required,
                        privacyScope, userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable, section);
                    break;
                case "date":
                    manager.CreateOrUpdateDateProperty(name, displayName, description, required,
                        privacyScope, userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable, section);
                    break;
            }
        }
    }

    private string GetConfigurationFile(string fileName)
    {
        try
        {
            var featuresDirectory = System.IO.Directory.GetParent(_properties.Feature.Definition.RootDirectory);
            var templateDirectory = featuresDirectory != null ? featuresDirectory.Parent : null;
            var rootDirectory = templateDirectory != null ? templateDirectory.Parent : null;

            if (rootDirectory == null)
            {
                throw new Exception("Unable to find SharePoint root directory.");
            }

            return String.Format(@"{0}\Config\IHI\UserProfile\{1}.xml", rootDirectory.FullName, fileName);
        }
        catch (Exception e)
        {
            Logging.Logger.Log(e);
        }

        return String.Empty;
    }
}

We use an XML configuration file which is deployed to the Config folder in the 14-hive to specify the properties we want to add. The advantage of using a configuration file is that we can update properties just by deploying a new configuration file and activating our feature which has already been deployed to the farm. The file has the following structure:

<?xml version="1.0" encoding="utf-8" ?>
<profileProperties>
  <sections>
    <section>
      <name>Organization</name>
      <displayName>Organization Properties</displayName>
    </section>
  </sections>
  <properties>
    <property>
      <name>UserGuid</name>
      <displayName>User GUID</displayName>
      <description>User ID from authentication provider</description>
      <type>
        <name>string</name>
        <length>50</length>
        <isMultiValued>false</isMultiValued>
        <required>true</required>
      </type>
      <privacy>
        <scope>Private</scope>
        <userOverridePrivacy>true</userOverridePrivacy>
      </privacy>
      <permissions>
        <isVisibleOnViewer>false</isVisibleOnViewer>
        <isVisibleOnEditor>false</isVisibleOnEditor>
        <isUserEditable>false</isUserEditable>
      </permissions>
      <section>Organization</section>
      <taxonomy/>
    </property>
  </properties>
</profileProperties>

A helper method allows us to get a URL reference to our configuration file in the 14-hive using the current feature’s context.

private string GetConfigurationFile(string fileName)
{
    try
    {
        var featuresDirectory = System.IO.Directory.GetParent(_properties.Feature.Definition.RootDirectory);
        var templateDirectory = featuresDirectory != null ? featuresDirectory.Parent : null;
        var rootDirectory = templateDirectory != null ? templateDirectory.Parent : null;

        if (rootDirectory == null)
        {
            throw new Exception("Unable to find SharePoint root directory.");
        }

        return String.Format(@"{0}\Config\UserProfile\{1}.xml", rootDirectory.FullName, fileName);
    }
    catch (Exception e)
    {
        //log exception
    }

    return String.Empty;
}

The ProcessProperties method then reads the configuration file and calls the appropriate methods on the UserProfilePropertyManager to create our properties based on the specified settings.

private void ProcessProperties(UserProfilePropertyManager manager)
{
    var uri = GetConfigurationFile("UserProfileProperties");
    var root = XElement.Load(uri);
    var sectionElements = root.Element("sections").Elements("section");
    var propertyElements = root.Element("properties").Elements("property");

    var sections = new System.Collections.Generic.Dictionary<string, ProfileSubtypeProperty>();

    foreach (var node in sectionElements)
    {
        var sectionName = (string)node.Element("name");
        var sectionDisplayName = (string)node.Element("displayName");

        var section = manager.CreateOrUpdateSection(sectionName, sectionDisplayName);
        sections.Add(sectionName, section);
    }

    foreach (var property in propertyElements)
    {
        //parent nodes
        var type = property.Element("type");
        var privacy = property.Element("privacy");
        var permissions = property.Element("permissions");
        var taxonomy = property.Element("taxonomy");
        
        //property
        var name = (string)property.Element("name");
        var displayName = (string)property.Element("displayName");
        var description = (string)property.Element("description");

        //type
        var typeName = (string)type.Element("name");

        var length = 0;
        int.TryParse((string)type.Element("length"), out length);

        var isMultiValued = false;
        bool.TryParse((string)type.Element("isMultiValued"), out isMultiValued);

        var separator = (string)type.Element("separator");

        var required = false;
        bool.TryParse((string)type.Element("required"), out required);

        //privacy
        var privacyScope = (Privacy)Enum.Parse(typeof(Privacy), (string)privacy.Element("scope"), true);

        var userOverridePrivacy = false;
        bool.TryParse((string)privacy.Element("userOverridePrivacy"), out userOverridePrivacy);

        //permissions
        var isVisibleOnViewer = false;
        bool.TryParse((string)permissions.Element("isVisibleOnViewer"), out isVisibleOnViewer);

        var isVisibleOnEditor = false;
        bool.TryParse((string)permissions.Element("isVisibleOnEditor"), out isVisibleOnEditor);

        var isUserEditable = false;
        bool.TryParse((string)permissions.Element("isUserEditable"), out isUserEditable);

        //taxonomy
        var termStoreName = (string)taxonomy.Element("termStoreName");
        var groupName = (string)taxonomy.Element("groupName");
        var termSetName = (string)taxonomy.Element("termSetName");

        //properties section
        var section = sections[(string)property.Element("section")];

        switch(typeName)
        {
            case "string":
               manager.CreateOrUpdateStringProperty(name, displayName, description, length,
                    isMultiValued, separator, termStoreName, groupName, termSetName, required, privacyScope,
                    userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable, section);
                break;
            case "boolean":
                manager.CreateOrUpdateBooleanProperty(name, displayName, description, required,
                    privacyScope, userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable, section);
                break;
            case "integer":
                manager.CreateOrUpdateIntProperty(name, displayName, description, required,
                    privacyScope, userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable, section);
                break;
            case "date":
                manager.CreateOrUpdateDateProperty(name, displayName, description, required,
                    privacyScope, userOverridePrivacy, isVisibleOnViewer, isVisibleOnEditor, isUserEditable, section);
                break;
        }
    }
}

After activation, you should see the new properties in Central Administration > Manage Profile Service: user Profile Service Application > Manage User Properties.

Once you have created your profile properties, you can make them available for searching by creating features to provision crawled and managed properties as described in these two posts:

Creating Crawled Properties

Creating Managed Properties