ContentIterator – Very Large Lists – list-throttling – SPQueryThrottledException

Something interesting we discovered while testing list throttling in combination with large lists…

MSDN: ContentIterator class provides methods to iterate list items, lists, sites to regulate the amount of data that is transferred (i.e., to avoid throwing a SPQueryThrottledException).

We create lists with more than 5000 items, 10000 items, 50000 items and 150000 items, we used both the SPList.GetListItems as the contentIterator methodologies to retrieve data from these lists and there was quite an unexpected error, euhm issue…

First things first…
When using large lists in SharePoint you should make use of the best practices described at the following blog post and the following whitepapers.

list throttling will stop you from querying a list with more than 5K items in it, although you can change this setting in central administration… You don’t want to change this every time you hit this boundary!
In a scenario where you are 100% certain that there will be more than 5K items, there are possibilities to let your list grow to 50 million items and still be able to query the items you need. But this requires an extra property on your Microsoft.SharePoint.SPQuery object. The object I’m referring to is called QueryThrottleMode and allows you to overrule the list (view) throttling limits applied on the Web Application the code is run on.

private int GetItems(SPList list){
    var query = new SPQuery();
    query.Query = "<Where><Eq><FieldRef Name=\"Title\" /><Value Type='Text'>Very Special Item That I Need</Value></Eq></Where>";
    query.QueryThrottleMode = SPQueryThrottleOption.Override;
    query.RowLimit = 1;
    var items = list.GetItems(query);
    return items.Count;
}

This method returns the specific item in less than 2 seconds from a list containing 108 366 items

If we use the following code implementing the contentiterator class like described on the msdn page it crashes with the SPQueryThrottledException.

static int exceptions = 0;
static int items = 0;
private int GetItemsByContentIterator(SPList list){
    var query = new SPQuery();
    query.Query = "<Where><Eq><FieldRef Name=\"Title\" /><Value Type='Text'>Very Special Item That I Need</Value></Eq></Where>";
    query.QueryThrottleMode = SPQueryThrottleOption.Override;
    query.RowLimit = 10;
    ContentIterator iterator = new ContentIterator();
    iterator.ProcessListItems(list,query,ProcessItem,ProcessError);
    var items = list.GetItems(query);
    return items.Count;
}
public bool ProcessError(SPListItem item, Exception e){
    // process the error
    exceptions++;
    return true;
}		

public void ProcessItem(SPListItem item){
    items++;
    //process the item.
}

But while executing this result we get the result as showing in the screenshot above.This asks for more detailed research, maybe we did something wrong?

If you want to read more about this topic, I’ll invite you to the following interesting posts:

Bye!

VN:F [1.9.22_1171]
Rating: 8.6/10 (13 votes cast)
VN:F [1.9.22_1171]
Rating: +5 (from 5 votes)
ContentIterator - Very Large Lists - list-throttling - SPQueryThrottledException, 8.6 out of 10 based on 13 ratings