ContentIterator – Very Large Lists – list-throttling – SPQueryThrottledException

Something interesting we discovered while testing list throttling in combination with large lists…

MSDN: ContentIterator class provides methods to iterate list items, lists, sites to regulate the amount of data that is transferred (i.e., to avoid throwing a SPQueryThrottledException).

We create lists with more than 5000 items, 10000 items, 50000 items and 150000 items, we used both the SPList.GetListItems as the contentIterator methodologies to retrieve data from these lists and there was quite an unexpected error, euhm issue…

First things first…
When using large lists in SharePoint you should make use of the best practices described at the following blog post and the following whitepapers.

list throttling will stop you from querying a list with more than 5K items in it, although you can change this setting in central administration… You don’t want to change this every time you hit this boundary!
In a scenario where you are 100% certain that there will be more than 5K items, there are possibilities to let your list grow to 50 million items and still be able to query the items you need. But this requires an extra property on your Microsoft.SharePoint.SPQuery object. The object I’m referring to is called QueryThrottleMode and allows you to overrule the list (view) throttling limits applied on the Web Application the code is run on.

private int GetItems(SPList list){
    var query = new SPQuery();
    query.Query = "<Where><Eq><FieldRef Name=\"Title\" /><Value Type='Text'>Very Special Item That I Need</Value></Eq></Where>";
    query.QueryThrottleMode = SPQueryThrottleOption.Override;
    query.RowLimit = 1;
    var items = list.GetItems(query);
    return items.Count;
}

This method returns the specific item in less than 2 seconds from a list containing 108 366 items

If we use the following code implementing the contentiterator class like described on the msdn page it crashes with the SPQueryThrottledException.

static int exceptions = 0;
static int items = 0;
private int GetItemsByContentIterator(SPList list){
    var query = new SPQuery();
    query.Query = "<Where><Eq><FieldRef Name=\"Title\" /><Value Type='Text'>Very Special Item That I Need</Value></Eq></Where>";
    query.QueryThrottleMode = SPQueryThrottleOption.Override;
    query.RowLimit = 10;
    ContentIterator iterator = new ContentIterator();
    iterator.ProcessListItems(list,query,ProcessItem,ProcessError);
    var items = list.GetItems(query);
    return items.Count;
}
public bool ProcessError(SPListItem item, Exception e){
    // process the error
    exceptions++;
    return true;
}		

public void ProcessItem(SPListItem item){
    items++;
    //process the item.
}

But while executing this result we get the result as showing in the screenshot above.This asks for more detailed research, maybe we did something wrong?

If you want to read more about this topic, I’ll invite you to the following interesting posts:

Bye!

VN:F [1.9.22_1171]
Rating: 8.4/10 (17 votes cast)
VN:F [1.9.22_1171]
Rating: +6 (from 6 votes)

Firefox 4 compatibility issues with SharePoint 2010 : Content Security Policy

When you install FireFox 4, you instantly receive a new security mechanism called Content Security Policy. This mechanism works behind the scenes to prevent some of the more severe web-based attacks against users and websites…

While using Firefox 4 to access our latest project (custom solution based upon the SharePoint 2010 platform in https context), we noticed that we couldn’t use the Out-of-the-box and often used Date Time Control

The issue occurs when trying to switch to the next or previous month in the dialog that occurs after clicking on the calendar icon. In detail, the HideUnhide(‘DatePickerDiv’,’DatePickerDivP1′,’20110501′); method wasn’t executing as expected.

By using the fire bug plug-in I was able to detect this warning in the console panel after clicking on the next month icon.

the SharePoint DateTime control makes use of an underlying Iframe to display the popup and I think that this might trigger the security guard in FF4 to prevent clickjacking

from developer.mozilla.org

Content Security Policy (CSP) is an added layer of security that helps to detect and mitigate certain types of attacks, including Cross Site Scripting (XSS) and data injection attacks. These attacks are used for everything from data theft to site defacement or distribution of malware.

CSP is designed to be fully backward compatible; browsers that don’t support it still work with servers that implement it, and vice-versa. Browsers that don’t support CSP simply ignore it, functioning as usual, defaulting to the standard same-origin policy for web content. If the site doesn’t offer the CSP header, browsers likewise use the standard same-origin policy.

A secondary goal of CSP is to mitigate clickjacking. Clickjacking happens when a malicious site directs a victim’s mouse click to an unintended target in another site. This is typically done by framing the target site’s content in a transparent element.

CSP lets a site specify which sites may embed resources, thereby helping to prevent this sort of attack.

Note: For security reasons, you can’t use the <meta> element to configure the X-Content-Security-Policy header.

The policy can be delivered from the server to the client via an HTTP response header or an HTML meta element. Both mechanisms indicates that a resource must have the set of restrictions specified in the policy applied to it by the user-agent while rendering the content.

https://dvcs.w3.org/hg/content-security-policy/raw-file/tip/csp-specification.dev.html

VN:F [1.9.22_1171]
Rating: 10.0/10 (1 vote cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)