Sunday, 11 August 2013

Entity framework large data set, out of memory exception

Entity framework large data set, out of memory exception

I am working the a very large data set, roughly 2 million records. I have
the code below but get an out of memory exception after it has process
around three batches, about 600,000 records. I understand that as it loops
through each batch entity framework lazy loads, which is then trying to
build up the full 2 million records into memory. Is there any way to
unload the batch one I've processed it?
ModelContext dbContext = new ModelContext();
IEnumerable<IEnumerable<Town>> towns = dbContext.Towns.OrderBy(t =>
t.TownID).Batch(200000);
foreach (var batch in towns)
{
SearchClient.Instance.IndexMany(batch,
SearchClient.Instance.Settings.DefaultIndex, "Town", new
SimpleBulkParameters() { Refresh = false });
}
Note: The Batch method comes from this project:
https://code.google.com/p/morelinq/
The search client is this: https://github.com/Mpdreamz/NEST

No comments:

Post a Comment