As you might already know, reader, if you hit this page, is that Indexing is and will always be:

• Indexing comes from a simple need, for dev related things or not.
• It helps getting to the data you want FASTER.

“I want to find all people with their first name starting with S”, I can sort that out with that folder sorting.

When it comes to a data storage things are the same:

• File systems indexing the position of files on the actual hard drive and flash drive,
• databases can allows you to create multiple indexes on certain columns, to make sure you can sort it faster.

Why making the point for code then? Well, it comes down to the exact same problem: performance.

Most of the times, people using LINQ, to actually parse data and get subsets of it for processing, in a loop for example:

foreach (var countryCode in _countryCodes)
{
    var countriesPerCode = _entitiesLists.Where(e => e.Country == countryCode).ToList();
    var count = countriesPerCode.Count;
    // Code supposingly doing something with entities.
}

It can be fine if not too much of these loops are run, but why is it so different when it runs on let’s say a million rows?

Same problem occurs with a database: if the sql enginge running the query doesn’t know anything about how to get rows that match your WHERE clause, it will have to run on all rows.

In the case of our loop above, LINQ does the same: how LINQ would know about which items match you lambda function until it tried it on all items in your list?

To solve that issue we are going to use an uncommon LINQ object: Lookup.

The goal is simple: we are going to use it to build out an index out of our data to group it by a given key. Running this only once on our dataset fixes our problem, in the sense that getting back data subset for each loop iteration will be instant with our Lookup.

Here are the performance difference you can get from our test app (output from our console app):

Data: building
Data: done
Test1: start
Test1: processed in 535 milliseconds
Test2: start
Test2: lookup done in 140 milliseconds
Test2: processed in 141 seconds

To summarize our little article (TL,DR):

• The cost of initializing the dictionary first takes some time, but doing it once offers around a x4 performance gain
• Make use of indexing capabilities when you dataset starts to be above a certain number of items (that is to say most of the times!)
• This is a really simple sample that does not reflect other things that could happen around your code, as I have seen performance getting from 45 minutes to 5 (a x9 increase then)
• Repro code is to be found in Github here.

Using Entity Framework with git and having multiple branches updating a model can be quite challenging.

Here is a concrete example:

• develop branch is on code first migration v1
• feature X updates model to add fields to an entity, adding migration v2
• feature Y also updates model  to add fields on an entity, adding migration v3
• feature X is merged into develop
• feature Y is merged, coming on top of previous one

Remarks:

First, both feature migrations are independent and both built on top of v1:

EF will be able to process them, but what will happen is this:

Changes noted here as incoming will bring back migration v2 changes when running “Add-Migration” again, even if they are present.

It basically means EF does not see v2 because v3 was generated with v2 not being present in the branch at the time of adding a migration.

This has to do with data found in migration RESX file, which contains Base 64 binary encoded info to keep track of link between each.

Bottom line: to avoid having this to happen:

1. Rebuild migration v3 on top of v2 in develop branch by deleting v3 after getting v2 from develop
2. Build a model update feature that everyone merge into their branch one updated with their model change
3. merge feature X into feature Y only for db change, then add your fields in feature Y so that new fields rely on migration v2

Happy merging!