Returning a collection of objects from a PowerShell function

If you’re writing a function that returns a collection then don’t forget to include the comma operator in the return statement. If you forget it your function will work correctly when the collection contains multiple objects, but fails when it contains 1 object.

Take the following buggy example:

function GiveMeAllTheThings()
{
    $myarray = @()
    #fill $myarray with results of type String. Assume that
    #run-time conditions determine if it is filled with 
    #0, 1 or more items and that each item is a string
    return $myarray;
}
$result = GiveMeAllTheThings
$result.GetType().FullName

If you execute this code when $myarray has many strings in it, the returned type from the function is System.Object[]. If $myarray has only 1 string in it, then the returned type will be System.String.

The code should have been written like this:

function GiveMeAllTheThings()
{
    ...
    return ,$myarray;
}

Understanding the LINQ nested grouping example

Here’s an explanation of how the default example for LINQ nested grouping actually works. The usual example for nested grouping looks like this:

from student in Students
group student by student.Faculty into Faculty
from dbtgroup in
(
    from student in Faculty
    group student by student.DebtCategory
)
group dbtgroup by Faculty.Key;

The objective of this statement is to first group-by students into faculties and then in each faculty create subgroupings of students by their DebtCategory.

So how does this actually work and whats the equivalent method/lamba syntax? The first step is to groups each student into their faculty. Assume we have the following data

public class Student
{
   public string Name { get; set; }
   public string Faculty { get; set; }
   public int DebtCategory { get; set; }
}

IList<Student> Students = new List<Student>();
Students.Add(new Student { Name = "John" , Faculty = "IT"     , DebtCategory = 2 });
Students.Add(new Student { Name = "Jane" , Faculty = "IT"     , DebtCategory = 2 });
Students.Add(new Student { Name = "Jesse", Faculty = "Finance", DebtCategory = 2 });
Students.Add(new Student { Name = "Linda", Faculty = "Finance", DebtCategory = 1 });

The following query groups each student into a faculty

var query1 = from student in Students
group student by student.Faculty into Faculty
select Faculty;

//The Method syntax for the above query is:
var query1Method = Students
.GroupBy(student => student.Faculty)
.Select ( Faculty => Faculty);

//This gives us the following IGrouping<string, Student> as result
//
// [0]
//    Key   :  IT
//    Values: 
//          [0] John (IT) (2)
//          [1] Jane (IT) (2)
//
// [1]
//    Key   : Finance
//    Values:
//          [0] Jesse (Finance) (2)
//          [1] Linda (Finance) (1)

The next step is to add another level of grouping:

var query2 = from student in Students
group student by student.Faculty into Faculty
from dbtgroup in
(
    from student in Faculty
    group student by student.DebtCategory
)
select dbtgroup;
//This gives us the following IGrouping<int, Student> as result
//[0]
//  Key   : 2
//  Values:
//        [0] John (IT) (2)
//        [1] Jane (IT) (2)
//
//[1]
//  Key   : 2
//  Values:
//        [0] Jesse (Finance) (2)
//
//[2]
//  Key   : 1
//  Values:
//        [0] Linda (Finance) (1)

// The following is the literal translation of the above Comprehension syntax into method syntax. We're ignoring this as explained below
//	var query2Method = Students
//		.GroupBy(student => student.Faculty)
//		.SelectMany(  Faculty =>Faculty.GroupBy(student => student.DebtCategory)
//					, (Faculty, dbtgroup) => dbtgroup);
	
//The final complete query ends with"group dbtgroup by Faculty.Key;" 
// this statement causes the compiler to see that you're refering to the Faculty object from the select many, so instead of 
// "(Faculty, dbtgroup) => dbtgroup" it emits a slightly different projection "(Faculty, dbtgroup) => new {Faculty, dbtgroup}
//structure
var query2Method = Students
.GroupBy(student => student.Faculty)
.SelectMany( Faculty =>Faculty.GroupBy(student => student.DebtCategory)
	 , (Faculty, dbtgroup) => new {Faculty, dbtgroup});

Query2 is close to our desired output, however the grouping is the wrong way around. So the final step is:

var query3 = from student in Students
group student by student.Faculty into Faculty
from dbtgroup in
    (
    from student in Faculty
    group student by student.DebtCategory
    )
group dbtgroup by Faculty.Key;

//The method/lambda syntax is:
var query3Method = Students
.GroupBy(student => student.Faculty)
.SelectMany (
	Faculties => Faculties.GroupBy (student => student.DebtCategory)
	, (Faculty, dbtgroup) => 
		new  
		{
			Faculty = Faculty, 
			dbtgroup = dbtgroup
		} )
.GroupBy( item => item.Faculty.Key, item => item.dbtgroup );

//This gives us the following groups as result
//[0]
//  Key   : IT
//  Values:
//        [0] Key   : 2
//            Values:
//                  [0] John (IT) (2)
//                  [1] Jane (IT) (2)
//[1]
//  Key   : Finance
//  Values:
//        [0] Key   : 2
//            Values:
//                  [0] Jesse (Finance) (2)
//        [1] Key   : 1
//            Values:
//                    [0] Linda (Finance) (1)

Loadtesting SharePoint file uploads with mandatory fields in the content types

When a user uploads a document into a library it can be the case that mandatory fields need to be filled in. If these fields aren’t correctly submitted to the server then the file will remain checked out and other users will not be able to see it.

For an automated load test its important that it correctly supplies the required fields. In a previous post I described all the requests that are involved in the upload. In this post we’ll describe how to correctly POST the fields to the server.

Lets assume you have a required field called “ReferenceId” in your content type. When we want to POST the value of this field to SharePoint we must include SharePoint’s Field ID (a GUID) into the name of the form post parameter.

The first point where we need to intervene is the GET request to EditForm.aspx. Its response contains the ID of each field that’s included in the content-type. Each meta-data field of the content type is described in a JSON datastructure that looks like this:

//This JSON structure is sent as 1 very long line in the response
{
   "ListSchema":{
      "...":{
         "Id":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
         "...":"..."
      },
      "ReferenceId":{
         "Id":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
         "...":"..."
      },
      "...":"..."
   }
}

You can use the following ExtractionRule to extract the Id of your ReferenceId field:

Property Value
Context Parameter Name GUID_ReferenceId
Ends With
Extract Random Match False
Html Decode True
Ignore Case False
Index 0
Required True
Search in Headers False
Starts With “ReferenceId”:{“Id”:”
Use Regular Expression True

On the POST request to EditForm.aspx you need to add an extra Form Post Parameter like this:

Parameter Value Explanation
ClientFormPostBackValue_{{GUID_ReferenceId}}_ReferenceId Insert your desired value for the ReferenceId here See how the name of the form post parameter is constructed from the GUID we extracted earlier

Anatomy of a loadtest for SharePoint file uploads to a document library

When you use Visual Studio’s recorder on a file upload to a SharePoint document library, you will get a test that needs some tweaking before its usable.

Firstly it doesn’t take into account where on your test machine the file is stored. The next time you run the test, it will fail because it can’t find the file to upload.

Secondly I like to make this a generic test that can upload any file to any site/library regardless if we’re running on an agent or on the local development machine.

In this post we will take a look at the structure of the test, then we’ll know how to tweak these tests in the future:

Request 1: GET https://root/sites/yoursite/_layouts/15/Upload.aspx

The first requests retrieves SharePoint’s file-upload dialog.

QueryString Parameters

Parameter Value Url encode
ListId The GUID of the library to upload to. In format {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx} No
RootFolder The ServerRelativePath to upload to. Eg /sites/yoursite/yourlibrary/yourfolder/anotherfolder Yes
IsDlg Set this to 1. No

Extraction rules

The only thing we need to do with the response is to extract all the hidden form fields in the response because the next request needs to POST them back to the server.

Extract Hidden Fields
Property Value
Context Parameter Name 1
Html Decode True
Required True

Request 2: POST https://root/sites/yoursite/_layouts/15/UploadEx.aspx

The second requests uploads the file from the test PC into the correct library/folder.

QueryString Parameters

Parameter Meaning Url encode
Same as previous request Same as previous request Same as previous request

Form Post Parameters

This request POSTs a lot of the extracted form parameters from the previous request back to the server. The complete list of parameters is in the following image. There are a few interesting ones in there though.

Screenshot of form post fields for request 2

Interesting form post parameters
Parameter Value Explanation
__EVENTTARGET ctl00$PlaceHolderMain$ctl03$RptControls$btnOK I’m not sure if this is needed. I just send it.
ctl00$PlaceHolderMain$UploadDocumentSection$ctl05$OverwriteSingle on Determines whether the user/test placed a check in the “Overwrite existing files” checkbox.
ctl00$PlaceHolderMain$ctl04$ctl01$uploadLocation /<yourfolder>/ Determines into which folder the file will be uploaded.
ctl00$PlaceHolderMain$VersionCommentSection$ctl01$CheckInComment The text of the user’s check in comment.
File Upload Parameter
Property Value
Content Type This is the Mime Type for the file that’s being uploaded. Take a look at freeformatter.com for a list of commont types. For .docx you can use application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Name The path of the document on the machine running the test.
File Upload Name The name of the file as it should appear in the library/folder. Do not include folder names
Generate Unique Name Set to True if you want the test framework to automatically insert a unique value into the name of the uploaded file. This value will not be stored in the test context, so if you want unique names AND you also need to remember that name, then you must generate the unique part of the name yourself before sending this request.
Name ctl00$PlaceHolderMain$UploadDocumentSection$ctl05$InputFile
URL Encode True
Use Guids Determines what the format of the unique part of the filename will be. Its only relevant is Generate Unique Name is set to True

Extraction rules

One the file has been uploaded into the library, SharePoint wants the user to provide values for the various meta-data fields. In order to achieve this, SharePoint assigns the uploaded file a temporary ID, communicates it back to the browser and expects requests 3 and 4 to complete the meta-data input process. If you don’t do this properly, then the uploaded file will remain checked out. The ID is hidden in a bit of JavaScript that looks like this

window.location.href = 'https:\u002f\u002f......\u002fEditForm.aspx?
Mode=Upload\u0026
CheckInComment=
\u0026ID=HERE IS THE ID YOU NEED
\u0026RootFolder=\...
\u0026IsDlg=1';               
Extract Text: The following extraction rule retrieves the ID of the uploaded file
Property Value
Context Parameter Name UploadEx_ID
Ends With \u0026
Extract Random Match False
Html Decode True
Ignore Case False
Index 0
Required True
Search in Headers False
Starts With \u0026ID=
Use Regular Expression False

Request 3: GET https://root/sites/yoursite/yourlibrary/Forms/EditForm.aspx


This GET request retrieves the HTML form that users use to edit the metadata on the uploaded file.

QueryString Parameters

Parameter Value Url encode
Mode Upload No
CheckInComments The user’s comment for the check-in. Yes
ID The value of context parameter UploadEx_ID we extracted on previous requests No
RootFolder See request 1 See request 1
IsDlg See request 1 See request 1

Extraction rules


This request has a lot of extraction rules on its response that retrieve information that needs to be POST’ed in the next request.

Extract Text: The following extraction rule retrieves owsHiddenVersion
Property Value
Context Parameter Name OWSHiddenVersion_0
Ends With ,
Extract Random Match False
Html Decode True
Ignore Case False
Index 0
Required False
Search in Headers False
Starts With {“owshiddenversion”:
Use Regular Expression False
Extract Text: The following extraction rule retrieves the ContentTypeId of the first content type in the dropdown list on the form
Property Value
Context Parameter Name ContentType_0
Ends With
Extract Random Match False
Html Decode True
Ignore Case False
Index 0
Required False
Search in Headers False
Starts With “ItemContentTypeId”:”
Use Regular Expression False
Extract Regular Expression: Extracts the GUID for the ContentType drop down list on the form
Property Value
Context Parameter Name CONTROL_GUID0
Html Decode True
Ignore Case False
Index 0
Regular Expression _g_([0-9a-fA-F]{8}_[0-9a-fA-F]{4}_[0-9a-fA-F]{4}_[0-9a-fA-F]{4}_[0-9a-fA-F]{12})_ctl
Required True
Use Groups True
Extract Hidden Fields
Property Value
Context Parameter Name 1
Html Decode True
Required True

Request 4: POST https://root/sites/yoursite/yourlibrary/Forms/EditForm.aspx

QueryString Parameters

Parameter Value Url encode
Same as previous request Same as prevous request Same as prevous request

Form Post Parameters


Most of the form post parameters are the usual hidden fields extracted from the previous step:

Screenshot showing all the form post parameters of request 4

Interesting form post parameters
Parameter Value Explanation
__EVENTTARGET ctl00$ctl43$g_{{CONTROL_GUID0}}$ctl00$ctl02$ctl00$toolBarTbl$RightRptControls$ctl00$ctl00$diidIOSaveItem I’m not sure if this is needed. I just send it.
ctl00$ctl43$g_{{CONTROL_GUID0}}$ctl00$ctl02$ctl00$ctl01$ctl00$ContentTypeChoice Value of context parameter ContentType_0 The value of this parameter determines which of the possible content types the user/test chose from the drop-down list.
ctl00$ctl43$g_{{CONTROL_GUID0}}$ctl00$ctl02$ctl00$ctl06$ctl00$owshiddenversion Value of context parameter OWSHiddenVersion_0 SharePoint needs this value to detect conflicting changes to the SPListITem that represents this file.

Running unittests as part of a loadtest in Visual Studio

Did you know that the loadtest framework isn’t limited to only running webtests or calling web-services? You can also include your unittests! This is very convenient for loadtesting MVC controllers, WebApi controllers or SharePoint’s CSOM interface.

Your unittests should work without any changes. If you need one of the following then your unittest needs some minor modification:

  • Work with loadtest transactions.
  • Use information such as the UserId or AgentName.

Below is a small example class that shows how to achieve the above without sacrificing the ability to run the class as part of a standalone unittest. Just inherit from it and use its methods.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.VisualStudio.TestTools.LoadTesting;

namespace Loadtesting
{
    [TestClass()]
    public class CombinedUnitLoadTest
    {
        /// <summary>
        /// Visual Studio's Unit Test Context. Microsoft's UnitTestFramework sets this member
        /// </summary>
        public TestContext TestContext { get; set; }
        
        /// <summary>
        /// Load Test Context. Only present if the testcase is being run in a loadtest
        /// </summary>
        public LoadTestUserContext LoadTestUserContext
        {
            get
            {

                if (this.TestContext.Properties.Contains("$LoadTestUserContext"))
                {
                    return this.TestContext.Properties["$LoadTestUserContext"] as LoadTestUserContext;
                }
                else
                {
                    return null;
                }
            }
        }
        /// <summary>
        /// The name of the Agent where the testcase is running. Returns "" if the test is not running on an agent
        /// </summary>
        public string AgentName
        {
            get
            {
                if (this.TestContext.Properties.Contains("AgentName"))
                {
                    return this.TestContext.Properties["AgentName"].ToString();
                }
                return "";
            }
        }
        /// <summary>
        /// The Id of the virtual user that is running the testcase. Returns "" if the test is not running in a loadtest
        /// </summary>
        public string UserId
        {
            get
            {
                if(null != this.LoadTestUserContext)
                {
                    return this.LoadTestUserContext.UserId.ToString();
                }
                return "";
            }
        }
        /// <summary>
        /// Returns a number that specifies how many tests the current virtual user has already executed
        /// </summary>
        public string Iteration
        {
            get
            {
                if(null != this.LoadTestUserContext)
                {
                    return this.LoadTestUserContext.CompletedTestCount.ToString();
                }
                return "";
            }
        }
        /// <summary>
        /// Starts a named transaction. The loadtest framework will report the time between BeginTransaction() and EndTransaction()
        /// </summary>
        /// <param name="name"></param>
        public void BeginTransaction(string name)
        {
            if(null != this.LoadTestUserContext)
            {
                this.TestContext.BeginTimer(name);
            }
        }
        /// <summary>
        /// Ends the named transaction. The loadtest framework will report the time between BeginTransaction() and EndTransaction()
        /// </summary>
        /// <param name="name"></param>
        public void EndTransaction(string name)
        {
            if(null != this.LoadTestUserContext)
            {
                this.TestContext.EndTimer(name);
            }
        }
    }
}


What to do when Visual Studio keeps opening your webtests in the wrong type of editor

Is Visual Studio opening your .webtest files in the XML editor instead of the GUI? You can fix this by editing the .csproj file of the project that contains the .webtests.

Find the line for your webtest and remove the following line below it:

<SubType>Designer</SubType>

Restart Visual Studio and the next time you double click on the test, it will be opened in the GUI editor.

a loadtest plugin to generate a GUID per request

Visual Studio has an out-of-the box plugin that generates GUIDs for you. However its a WebTest plugin, so it will only generate them once per webtest. You can’t include that plugin into a loop.

Fortunately we can write a simple WebTestRequest plugin that does what we want:

using System;
using System.ComponentModel;
using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools.WebTesting.Rules;

namespace LoadTestPlugins
{
    [DisplayNameAttribute("Generate GUID")]
    [DescriptionAttribute("Generates a GUID during a request")]
    public class GenerateGUID : WebTestRequestPlugin
    {

        #region Methods
        private void GenerateIt(WebTestContext Context)
        {
            Context[this.ContextParameterName] = Guid.NewGuid().ToString();
        }

        public override void PostRequest(object sender, PostRequestEventArgs e)
        {
            if(this.BeforeRequest == false)
            {
                this.GenerateIt(e.WebTest.Context);
            }
        }

        public override void PreRequest(object sender, PreRequestEventArgs e)
        {
            if(this.BeforeRequest == true)
            {
                this.GenerateIt(e.WebTest.Context);
            }
        }
        #endregion

        #region Properties
        [Description("Name of context parameter to store the number in"),
        DisplayName("Context parameter"), IsContextParameterName(true)]
        public string ContextParameterName {get; set;}

        [Description("Whether to generate the number before or after the request has executed"),
        DisplayName("Apply before request"),
        DefaultValue(true)]
        public bool BeforeRequest { get; set; }
        #endregion
    }
}

A loadtest plugin to clear context parameters (and why we need it)

If a webtest stores temporary data in a context parameter, then you need to make sure that its reset to some initial state when the webtest starts. Otherwise it will fail if that webtest is included multiple times within the same testrun.

Visual Studio’s “Set Context Parameter Value” plug-in won’t help here as it cannot set the value of a parameter to an empty string. If you try this, then during the test run, the request will fail with the error “Request failed: Missing or invalid Context Parameter Name.”

The solution in my case was a very simple custom plug-in. I choose to implement it as a WebTestPlugin instead of a WebTestRequestPlugin to avoid confusing it with Visual Studio’s existing plugin.

using System.ComponentModel;
using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools.WebTesting.Rules;

namespace LoadTestPlugins.WebTestPlugins
{
    [DisplayNameAttribute("Set context parameter value")]
    [DescriptionAttribute("Sets the context parameter to a value")]
    public class ClearContextParameter : WebTestPlugin
    {
        public override void PreWebTest(object sender, PreWebTestEventArgs e)
        {
            e.WebTest.Context[this.ContextParameter] = this.Value;
        }

        #region Properties
        [Description("Context parameter to set"),
        DisplayName("Context parameter"), IsContextParameterName(true)]
        public string ContextParameter { get; set; }

        [Description("Value to place into the context parameter"),
        DisplayName("Value")]
        public string Value { get; set; }
        #endregion
    }
}

SharePoint people-pickers in loadtests

A significant number of the webtests in out loadtest need to simulate a human tying into a people-picker control. This control waits until the first few letters have been entered and after that calls SharePoint’s client.svc web-service every time the user enters data into the control.

I simulate this with the following logic in webtests:

GET https://...
For Each Character In Targetusers display name:
    POST https://.../_vti_bin/client.svc/ProcessQuery
GET https:/...

As Visual Studio doesn’t contain a plugin that iterates over characters in a string, I wrote my own. This plug-in allows you to choose:

  1. Which input parameter contains the display name to iterate over.
  2. Into which output parameter to place the sub-string.
  3. At which index to start iterating. The 1st iteration of the loop includes all characters upto the starting index. All subsequent iterations add one more letter from the input string into the output string.

Some nice-to-have improvements to this plugin are:

  • Add a configurable delay in milliseconds to each loop. This would be nice as think-times on requests can only be configured in intervals of whole seconds
  • Add an option to automatically clear the output parameter once all iterations have finished

Here is the code for the plug-in:

using System.Collections.Generic;
using System.ComponentModel;
using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools.WebTesting.Rules;

namespace LoadTestPlugins
{

    [DisplayNameAttribute("For Each character in context parameter")]
    [DescriptionAttribute("Sequentially copies each character from the source into the destination. You can use this to simulate a user that types a name into a peoplepicker for example")]
    public class ForEachCharacter : ConditionalRule
    {

        #region Methods
        public override void CheckCondition(object sender, ConditionalEventArgs e)
        {
            string Input = e.WebTest.Context[this.InputContextParameterName].ToString();
            string Output;
            try
            {
                Output = e.WebTest.Context[this.OutputContextParameterName].ToString();
            }
            catch (KeyNotFoundException)
            {
                Output = string.Empty;
            }


            //Only do something if the output is not yet complete
            if (Input.Length == Output.Length)
            {
                e.IsMet = false;
                return;
            }
            

            int LengthToCopy;
            if(Output.Length == 0)
            {
                //Copy all characters before the starting index and 1 extra
                LengthToCopy = this.StartIndex + 1;
            }
            else 
            {
                //Copy 1 extra character to the output
                LengthToCopy = Output.Length + 1;
            }

            e.IsMet = true;
            e.WebTest.Context[this.OutputContextParameterName] = Input.Substring(0, LengthToCopy);
            return;
        }
        
        #endregion

        #region Properties
        [Description("Name of context parameter that contains the characters to iterate over"),
        DisplayName("Input Context parameter"), IsContextParameterName(true)]
        public string InputContextParameterName {get; set;}

        [Description("Name of context parameter to store the each successive iteration"),
        DisplayName("Ouput Context parameter"), IsContextParameterName(true)]
        public string OutputContextParameterName { get; set; }

        [Description("The zero-based index to start the iteration from"),
        DisplayName("Start Index")]
        public int StartIndex { get; set; }
        #endregion
    }
}