travis.io

I Completed the 75Hard Challenge (And You Should Too)

Sat, 04 Jan 2020 00:00:00 +0000

I'd first heard of the 75Hard Challenge in mid 2019. It's a 75-day challenge started by Andy Frisella, founder of 1st Phorm, Arete Syndicate, and the MFCEO and Real AF Podcasts.

A few people I know had completed 75Hard and I was inspired by their progress and the mindset shift they seemed to go through. After talking with them about their experience I was set on tackling it at the start of 2020 to coincide with some of my new year goals.

While out for lunch one day my wife pointed out that the final 75 days of the year was coming up in just a couple of days. With this timing and her support, and a little nudge from some of my more savage Facebook friends, I figured there was no better time to do it.

2 days later, on October 18th, I thrust myself into the challenge (in the midst of CrossFit Open, which I was also participating in.)

What Exactly is 75Hard?

The challenge consists of the following tasks, which must be completed every day without exception:

Two 45-minute workouts (you define what the workouts are)
Drink 1 gallon of water
Read 10 pages of a personal development/business/self help book
Stick to a diet (you define the diet)
No alcohol or cheat meals
Take a progress picture

There are also a few rules:

You must complete all 75 days in a row; if you miss a single day you fail. Failure means any deviation from your plan, in the strictest sense. No substitutions, no catch-up days, no excuses.
If you fail, you must start over on day 1
One of the daily workouts must be outside, no matter what the weather conditions are
Audiobooks do not count toward the reading goal
The day ends when you go to bed

More details about 75Hard can be found on the MFCEO Podcast, episode 290 or at 75Hard.com (I recommend the podcast, but only if you like foul language and intense people, which I do.)

On Completing the 75Hard Challenge

Completing 75Hard was both awesome and difficult, and wasn't without hiccups.

It's no coincidence that I chose to do 75Hard over the "food and drink" holidays—Halloween, Thanksgiving, Christmas break, and New Year's Eve. It's also the coldest time of the year, so the outdoor workouts would be pretty uncomfortable. I had the option to wait until the new year, but I decided I'd prefer to do it during the hardest time of the year. I'm a big believer in the importance of friction when pursuing change and growth, so this was a perfect opportunity.

It was the right choice.

Big Wins in My 75Hard Challenge

I landed a 300lb deadlift PR on New Years Eve while the rest of the world was out celebrating (no judgement here, I wish I was there with you!) I had put off the workouts until the end of the day, and the fireworks turned out to be a fun complement to the final workout.
Having a plan and sticking to it has given me a lot of insight into what works for me and what doesn't. I went into January with a plan, and already have a plan for February and March.
I exercised while sick, while tired, during terrible weather, in the freezing cold, in the middle of the night, with family in town, while traveling, and during 4 major holidays. This showed me that there's always time to exercise if you prioritize it.
I had great support from friends and family, and 75Hard is an interesting topic to talk about with others
I actually have before/after photos of myself, and something exciting to document

The Numbers

Number of days in the challenge: 75
Gallons of water drank: 75+
Number of nights I woke up to pee multiple times: All of them
Number of workouts completed: 150
Number of times I said "I don't feel like working out": 150
Number of times I was glad I worked out: 150
Length of each workout: 45+ minutes
Total workout time: 112.5+ hours
Total pounds lost: 5 (weight loss was not a goal; my focus was on strength training and overall health)
Total books read: About 4
- The 21 Irrefutable Laws of Leadership, by John C. Maxwell
- Choose Wonder Over Worry, by Amber Rae
- Man's Search for Meaning, by Viktor Frankl (first half)
- Launch, by Jeff Walker
- First, Break All The Rules, by Marcus Buckingham and Curt Coffman (first half)
Active calories burned:
- October: 11,672+
- November: 25,143+
- December: 17,587+
- Total: 54,402+
Steps taken:
- October: 323,981+
- November: 385,830+
- December: 403,682+
- Total: 1,113,493+ (Over a million!)
Daily average walking/running distance:
- October: 5.1mi+
- November: 6.2mi+
- December: 6.1mi+
Number of workouts completed in the rain: 2
Number of workouts completed after midnight: 41

75Hard - The Good

Tasks are clear and it's straightforward to complete them
There's a great community of people online participating in the challenge
Each day ends when you go to bed so you can always complete the daily tasks, even if that means going to bed late
Progress pictures showed major changes in the first half of the challenge
Despite the physical component, 75Hard isn't designed to be a fitness challenge; it's designed to be a psychological challenge. Improved physical fitness is a side effect of pushing yourself every day to complete the difficult list of tasks.

75Hard - The Bad

It's long and it's difficult to complete! There will be days when "life happens" and you'll want to sabotage it. Don't do it.
It would be really easy to go straight downhill after completing the plan (tip: have a re-entry plan; I created a 31-day plan for January which I started immediately.)
Unless you are really regimented with your schedule (I'm not!) you will probably find yourself completing the tasks late at night on occasion. I went to bed several times at 2AM or later. This can be hard to recover from because it's painful to wake up early and get a jump on the next day.
Our entire household got hit by a bad virus at the start of December (involving 2 urgent care visits, and late nights taking care of sick kids.) Because of this, I ended up doing a lot of walking in the month instead of strength training or CrossFit. As a result progress pictures showed less noticeable changes, which was frustrating! I wanted to ramp up during the last 30 days, not slow down.

Why 75Hard is the Perfect Challenge

75Hard is designed to be challenging, not necessarily difficult. Completing the plan and starting a new one felt like a natural shift, and because it's so intense any new plan seems relatively easy. The first few days of January have been a breeze, even though they were a big leap from where I was before 75Hard.

You get to define the diet and exercise portions, and you choose which books you will read. This is a double-edged sword; anyone can complete 75Hard, but it's entirely possible to sandbag the entire challenge by making it too easy. The key is to define a plan that's challenging for you, and to stick with it. The harder you make it, the more you'll get out of it.

In a pinch, all of the daily tasks *can* be done at the end of the day, but this is really hard to do. The plan encourages you to "Eat That Frog" and complete the tasks earlier in the day. Great for procrastinators like me!

The act of completing the daily activities is binary—you either complete them, or you don't. If you don't complete them, you fail. Failing is painful, but fully recoverable. You can still complete the program if you fail by restarting on day 1, and you'd actually get more out of it this way.

Regarding the length of the challenge, 75 days feels perfect. It's longer than a 30-day challenge, so it's long enough to help you establish new habits and ideas. It's also shorter than a 90-day challenge, which feels too long for something so intense. To me, it hits that sweet spot of "just over 2 months" which feels achievable.

Lastly, the daily tasks are varied and they contribute to your health, learning, vitality, vulnerability, and strength against addictions and vices (for those who need it.)

All in, 75Hard was an awesome experience and I'm thankful I did it. At the very least I'm considering doing it for the last 75 days of 2020, though I may tackle it earlier.

Thinking of Doing 75Hard Too?

You should! Send me an email () or connect with me on Facebook. I'd be happy to email you and chat with you along your journey to help you keep going. You'll be glad you did.

Agile Is A Strategy, Not A Process

Thu, 14 Mar 2019 00:00:00 +0000

In my experience working with teams ranging in size from a few developers to large enterprise development projects, one of the recurring themes I’ve seen is that of process.

The best teams I’ve worked with have processes in place, but they are streamlined, honed routines without the overhead of decision making or heavy ceremony.

When I first started working in an Agile environment, it wasn’t one I was dropped into, with a well-defined way of managing the process. Rather, it was a company-wide initiative to move to an Agile methodology from a strongly-ingrained waterfall one. The company’s digital arm started a division-wide initiative to certify all of its developers, project managers, QA staff, and leadership with the goal of making SCRUM a first-class citizen in the organization.

As with most large transformations of this sort the change management process was a long, arduous one. SCRUM coaches were hired, every member of staff was trained and certified in all things Agile, and several team sessions were held with our internal SCRUM knowledge expert to practice these new techniques. We were all excited about the promises that Agile would bring—who wouldn’t be? Especially with shiny new Certified ScrumMaster notches in our belts.

Agile’s Broken Promise

Unfortunately, in a company not quite ready for the change, Agile isn’t really all it’s sold to be.

You see, Agile has to be adopted from the top down. Leadership has to get on board with the changes that Agile brings. And when I say leadership, I’m not talking about director-level or even VP level and below. I mean all leadership—all the way up to the president and CEO of the company.

You see, without full adoption, at some point someone in leadership is going to want to see a GANTT chart. Without full adoption someone is going to want to define set-in-stone delivery dates with resource plans, user flow diagrams, and detailed specs before an actual line of code is written. This overhead trickles down to the developers and any level of non-adoption creates an unavoidable rift in what should otherwise be a holistic effort.

Waterfall isn’t perfect; but it’s a hell of a lot better than scrummerfall.

A broken Agile strategy doesn’t just introduce complexity in delivery and tracking. It causes a painful and frustrating disconnect between developers and upper management. It’s a modern-day language barrier that prevents the people building the project from effectively communicating with those sponsoring and paying for it.

But There’s Good News: Agile Works, If You Do It Right

The companies with the worst adoption of Agile are the ones who aren’t willing to embrace the simplicity of it.

Agile is best when you focus on the essentials.

At its core, Agile is an incredibly simple concept, which is why it’s both easy and confoundingly difficult to execute. Where I’ve seen teams try to adopt a simple project management concept like Agile and fail, I’ve noted two diametrically opposite problems:

They try to use too little of the framework, and end up just labeling their work items “stories” and “defects”, and doing obligatory “standup” meetings which provide little or no actual value.
They try to use too much of the framework, and end up exhausting team members with more overhead and ceremony than they care to deal with.

So how does a team find a happy medium and implement Agile project management in a way that works?

The answer lies in dissecting the two points above and finding where the problems of one are solved by the benefits of the other. And no matter what anyone tells you, this will differ by team. Some teams work better together by having longer standup meetings with slightly more discussion and planning, while others benefit most by keeping standups punctuated and focusing instead on swarming outside of meeting time.

First, holding daily standup meetings and calling yourself Agile is a waste of time (not to mention that a traditional standup format really doesn’t provide much value in the first place.) Effective project management requires a set of tools, not just a single one.

Second, there are several tools in Agile that complement the standup, and each one has a special and specific place. But don’t keep doing something just because you’re supposed to follow the framework, especially if it’s not providing real value. Adjust things to work, and drop them if they don’t, but do this with intention and care.

So What’s The Solution?

Agile works best when it becomes a routine. It should be a strategy, not a process.

What does that mean? Let me put it another way:

Seek to uncover the essence of Agile and what works for your team; don’t just do a bunch of shit so you can check off the boxes and say “we do the Agile things.”

Agile is never going to work for a team that holds all the prototypical meetings and never considers whether they’re helpful or not. Every retro meeting should be an opportunity to discuss what elements are working for the team, and how they can be improved. Like in code, the adoption of Agile is a long-term effort and best done iteratively.

One of the big wins built into Agile is its predictability. Team members must be given core time to focus on their work and a limited set of well-defined, scheduled meetings with clear objectives is the best way to grant them that.

Retro meetings at the end of every sprint are invaluable and yet many teams don’t even have them.

Standup meetings should be timely and pared down to the essentials, but more valuable than “this is what I did yesterday, this is what I’m doing today, and I have no blocks” (but please, save technical details and solutioning for a parking lot discussion.)

My point here is that we work best with routines in place. That’s just how the human brain works—always working to optimize and move problems into background processing, and routines allow that to happen.

Decision fatigue and context switching are productivity killers, and a streamlined (read: well-defined, predictable) Agile routine is your best tool to combat them.

Want to see team productivity and individual happiness levels skyrocket? Eliminate wasteful process, create predictable routines, and pare project management down to the essentials.

How To Get ASP.NET Web API to Return JSON Instead of XML in a Browser

Tue, 13 Oct 2015 00:00:00 +0000

One of the more frequently asked questions about ASP.NET Web API is how to force it to return JSON instead of the default, XML. When viewing a GET endpoint in a browser like Google Chrome, Firefox, or Microsoft Edge, the default Content-Type displayed in the browser is generally text/html in XML format. This is by design, and it’s important to understand.

A Quick Summary of Content Negotiation and Media Types

For every HTTP request/response, there is a process called Content Negotiation (sometimes mistakenly called Content-Type Negotiation). In this process, the client (e.g. your web browser) tells the server how it wants the response formatted. The parameters that determine what the Content Negotiation process looks like are defined in HTTP headers like Accept, Referer, Cookie, Host, etc.

Among the headers is one called Accept containing values like text/html, application/json, multipart/form-data, or various other values. These are known as Media Types. There are a lot of different Media Types, but generally you’ll only ever need to know about a handful. As an example of how media types work in the Content Negotiation process, a client basically tells the server “Hey, if you can send data back in JSON format, please do that. If not, I can also read XML.”

The server should reply in one of those formats if possible. The list is also ordered by priority; the server is expected to send data back in the first requested format if possible. If it can’t, it should send in the next requested format, and so on. If the server cannot match any of the Accept headers sent by the client, generally it will return its own default format.

ASP.NET Web API And Media Type Content Negotiation

By default in ASP.NET Web API, you can send various types of Accept headers and automatically receive a serialized response in that format.

Here’s a simplified example request:

GET /api/books/ HTTP/1.1
Accept: application/json,application/xml
[blank line]

With resulting response:

HTTP/1.1 200 OK
Content-Type: application/json

[{
  "id" : "978-0641873245",
  "name" : "Hi, That's a Nice API You Have There",
  "author" : "Snarky Human Person",
}
,
{
  "id" : "978-1411923349",
  "name" : "ASP.NET Web API for Cool Cats",
  "author" : "Captain Stacktrace",
}]

Now, if for some crazy reason we decided to remove the JSON serializer from our ASP.NET Web API serializer collection, but still had the XML serializer, we would automatically get an XML response, like the one below. Note that we would also get the same response if we had defined application/xml first in our Accept request header above.

HTTP/1.1 200 OK
Content-Type: application/xml

<?xml version="1.0" encoding="UTF-8"?>
<books>
   <book>
      <author>Snarky Human Person</author>
      <id>978-0641873245</id>
      <name>Hi, That's a Nice API You Have There</name>
   </book>
   <book>
      <author>Captain Stacktrace</author>
      <id>978-1411923349</id>
      <name>ASP.NET Web API for Cool Cats</name>
   </book>
</books>

Why Does Any of This Matter?

Good question! Unless you’ve already figured it out. As I mentioned above, Content Negotiation is a part of the HTTP request/response process and responses are serialized in specific formats for a reason. Naturally, web browsers want to receive data in a format that they can best understand and work with. Generally, this means text/html. So, that leads us to the most commonly suggested answers to this question, and exactly why they work, but are still wrong.

The Suggested, Not Ideal Answers

A read through this StackOverflow Post reveals a few answers to this problem. Let’s explore them:

Change the default formatter for Accept: text/html to return JSON. This is the top rated answer, and with arguably good reason. This forces ASP.NET Web API to use a JsonFormatter to serialize the response body in the case that a request is made for text/html, the default for most browsers. This will be absolutely fine in 99% of cases and works great if you use a web browser to check HTTP responses. The real problem here is that the Content-Type of the response will still be text/html; for public APIs, this can be misleading. Imagine someone writes an application to query your API and actually wants an HTML-formatted response. I’ve worked with enough APIs to know that this is a real pain.
Change the default formatter for Accept: text/html to return JSON, and also return a valid Content-Type: application/json header. This is better, but we’re still doing the same thing here. The client is requesting text/html, and we’re saying “Nope. Here’s some JSON. You take it and you like it.” This isn’t necessarily the worst and it’s the default behavior if your API simply doesn’t support HTML responses.
Completely remove the XML formatter, forcing ASP.NET Web API to return JSON by default. Okay, I admit I’ve used this technique in the past, and of course it’s fine if you’ll only ever serialize responses in JSON, and I’d only recommend it for internal or highly-controlled APIs. This removes the ability for ASP.NET Web API to return XML, and in turn you limit compatibility or adoption for clients requiring XML. Never do this for a public API without good reason.

The Best Answer

Honestly, when you really distill it down, the best answer is don’t use a browser to test your APIs. Why? Because this isn’t what browsers do best, and there are a ton of tools that allow you to test APIs much, much easier. And they’re all free. And you should be using them because testing APIs in a browser (outside of plugins or the inspector) is a fool’s game.

What I Use to Test APIs

I use a few tools depending on the situation, but here are the ones I most commonly have open:

Free Tools for API Development

Postman. Easily one of the best tools out there for doing API work. Postman has great features and it’s very easy to use once you get the hang of it. You can save sessions and requests to run at the click of a button, sync your data across devices, and even auto generate code snippets to bootstrap HTTP requests in other technologies like JavaScript, Python, Ruby, and even command-line Curl.

There’s a premium version of Postman called Jetpacks which allows you to write unit tests, chain requests into a series, or create collections of API requests that you can run with the click of a button. Postman works on Windows and Mac, and best of all, it’s totally free (unless you decide to upgrade).

If I had to distill this entire article down to one main point, it’s this: You can cover all your bases of testing and working with APIs using only Postman. The other tools presented here, while useful, are best considered for convenience and extended circumstances.

Simple REST Client. A super slim plugin for Google Chrome that does very little, but does it well. Perhaps because it was the first REST client I ever used, I still favor this plugin for most simple API work. It hasn’t been updated in about 5 years (hey, why not submit some updates for it), but let’s be honest, we’re working with HTTP, which doesn’t change often.

Because it’s a browser plugin, Simple REST Client opens quickly, and it’s compatible on Windows, Mac, and probably Linux.

Fiddler. While I don’t consider this the best REST client, it certainly does the job. Fiddler is a full-blown traffic inspector (available only for Windows), basically like the Chrome inspector or Firebug plugin on speed. It has built-in functionality supporting manual HTTP requests, and has some functionality for saving requests to a scratch pad for future use. That said, I don’t use Fiddler much these days given the above tools, but it’s a great tool to have available.

Services for API Development

Runscope. I don’t use Runscope as much as I’d like to, but when I do I’ve been happy with it. While Runscope offers a wide array of features, I primarily use it to verify the requests from third-party APIs before pointing them at my own endpoints. For example, I recently used Runscope on a project to verify that Stripe callbacks were in the format I expected (despite them having amazing documentation), and used them to better understand what data I would be receiving before actually building out my own endpoints. I essentially used Runscope as a bucket to capture all of the HTTP requests so I could inspect them later.

Runscope also has some great tooling for validating responses, monitoring your API for outages, and providing uptime data for your users. It has a great free offering too and it’s something I recommend at least checking out.

I contacted Runscope while writing this article, and they generously agreed to provide this link which grants you additional resources for the free account (3 team members instead of 1, and an upgrade from 25,000 to 100,000 requests per month). Full disclosure, I'm not an affiliate, but I'm happy to be able to provide you this benefit.

Mockable.io. One of my new favorite tools, and one that I discovered in its early days, Mockable.io basically lets you create mocked API responses. This comes in handy for testing how your application or API responds to expected or unexpected data, or error conditions. Since I’ve used it from launch, I’ve watched the team add more features to it and it’s only gotten better. I expect great things from it as it continues to grow. It’s an incredibly handy tool to have, and offers a very generous free plan.

So, That Was Long. What’s The Takeaway?

I wrote this article to help you understand why sometimes the highest-voted answer on StackOverflow isn’t always the best answer. Now don’t get me wrong, it usually is, and at worst it will solve your problem with little fallout. But it’s important to simply ask why sometimes.

There’s often a fine line between something that works and something that is semantically or practically better. This subject is clearly targeted more directly toward developers who create APIs, and I truly believe that API design is an area that takes acute, critical thinking and planning. The smallest nuances can make an otherwise intuitive and delightful API tricky and frustrating. I hope I’ve helped curb one of those issues up front, and likewise introduced you to some tooling that will become everyday staples in your API development toolkit.

Please do me a favor and comment below to let me know what you thought of this article. If you have questions, I’m pretty quick to respond, so ask away!

Migrating from ASP.NET Membership to ASP.NET Identity

Tue, 24 Mar 2015 00:00:00 +0000

One of my websites (a Web Forms application founded in 2007) was created using the ASP.NET Membership provider. For many years, the Membership framework served me well and the site hasn’t changed substantially so major infrastructure-level changes were never necessary. Fast forward to present day. This legacy application is undergoing some major changes and growth, and the logical step to support it in the future is to move away from the ASP.NET Membership provider and toward a newer, more extensible framework.

Enter ASP.NET Identity, a subset of Windows Identity Foundation.

Before I dig in too much, I want to make sure I give credit where credit is due. The following articles helped me through this process:

Migrating an Existing Website from SQL Membership to ASP.NET Identity - Much of the code I provide below comes from this article. The information I provide below is similar, but I tried to clarify some areas that confused me about that article.
Setting Up ASP.NET Identity Framework 2.0 with Database First - Because I was using a database-first approach in my solution, this helped clarify some details for me from the above article.
ASP.NET Identity – User Lockout - Helped clarify the way a user is locked out in ASP.NET Identity vs. ASP.NET Membership. This also discusses the use of SignInManager which I did not end up using (explained later).

My Scenario

In this incredibly short post I’m going to outline the steps that I took to migrate my code and users to the new ASP.NET Identity framework. I’ll provide code samples and some gotchas that I ran into. But first, here’s a summary of my situation, since yours may differ:

The application uses Web Forms and ASP.NET Membership. At the time of this migration, there was no use of MVC. If you do use MVC though, the code is still applicable (I think!)
Entity Framework (v.6) was introduced fairly recently in a database-first approach. This is not largely relevant, but I do use Entity Framework in some optional steps below.
Roles and Profiles were not used in this site, so I won’t be covering them.
I discarded a lot of the Membership data from the [aspnet_Membership] table; I just really didn’t need most of it.
As much as possible, I wanted to preserve the vanilla implementation of IdentityUser, the model that ASP.NET Identity uses, so any of the additional data from the old Membership tables that I did keep is stored in a separate table.
ASP.NET Identity does not, by default, utilize a security question and answer for password reset. I wanted to preserve this functionality since my site uses it, so I will outline how I did that.
My application database used a single Application ID from the Membership database, so I did not cover the case of a multi-tenant Membership database.
My application is split into a few projects, so if you’re not sure where to put the code I provide, just put it somewhere. Refactor and move it once you understand it (which is exactly what I did.)
Many examples online use Database Migrations to create an extended version of the basic Identity models, but again for the purpose of preserving as much about the vanilla integration of ASP.NET Identity I did not use this approach; if you want to, the process is documented in the first article I linked above.

The Fundamentals

ASP.NET Identity uses a class called IdentityUser which it maps to a database table called [AspNetUsers] to persist user data. I’m not sure if the naming of this table is merely convention, but it’s the default and it seems to work. In combination with this, we also have a UserManager which manages the persistence of the user model. This class provides functionality like UpdateUser(...), ChangePassword(...), GetRoles(...), etc. It replaces much of the functionality from the legacy MembershipUser and rolls them into a manager-type class.

We will be extending IdentityUser into our own subclass and calling it ApplicationUser (pretty common, actually), and likewise extending the UserManager into our own subclass called ApplicationUserManager. This does a few things: first, it allows us to extend the default UserIdentity class if we wanted to (we won’t be), and secondly it allows us to override some properties of the default UserManager; the most important property we’ll need to override is the PasswordHasher with our own implementation. Since ASP.NET Identity’s default UserManager has a different hashing algorithm (PBKDF2) than the legacy one used in ASP.NET Membership (SHA-1 by default), we’ll provide our own implementation of it called SqlPasswordHasher, which is capable of validating both types. This allows us to migrate our users as-is and still allow them to log in.

We’ll also need to create a new implementation of IdentityDbContext, which we’ll call ApplicationDbContext. This just gives Entity Framework a context through which to persist the user data. Don’t worry too much about the details of this, because it’s one line of code.

So in the end we will end up with 4 new classes:

ApplicationUser - Model which represents a user
ApplicationUserManager - Manager class which provides all functionality for adding/updating/querying users
ApplicationDbContext - Context class that Entity Framework uses to persist a user model to the data store
SqlPasswordHasher - A magic class that can validate new PBKDF2 passwords, and likewise old SHA-1 passwords for migrated accounts

We’ll also end up with 6 new tables in our database, which I’ll provide the SQL queries to create. I won’t be covering most of these, but you should still create and know about them for the future:

[AspNetUsers] - Stores our users, and maps to the ApplicationUser model above
[AspNetUsersExt] - This is a custom table, and I use it to store additional details from the Membership tables which allows me to keep the [AspNetUsers] table clean and simple. In this example, I will be storing the security question/answer credentials (and only these). You can change the name if you’d like.
[AspNetRoles] - Roles live here (Won’t be covered in this post)
[AspNetUserRoles] - Maps users to roles (Won’t be covered in this post)
[AspNetUserClaims] - Maps users to claims (if you’re not familiar, a claim is a simple but verifiable statement about a user that you define, e.g. “CanDeletePosts” or “PassportNumber”). Claims work alongside roles, but provide more granular assertions about what a user is authorized to do. (Won’t be covered in this post)
[AspNetUserLogins] - This table stores information about other logins that a user has verified, like Facebook or Twitter (Won’t be covered in this post)

Let’s Get Started

Create The ASP.NET Identity Tables
Update Existing Connection String To Add providerName
Add References To ASP.NET Identity Assemblies
Add Code For ASP.NET Identity Functionality
Get Login/Signup Working
Migrate Membership Database Users

Step 1. Create The ASP.NET Identity Tables

The first step I recommend is to create the database tables. Since this just means adding a few new tables, the chance of breaking anything is relatively low. The following script will create the tables described above (sorry, it’s long, but it works on SQL Azure.) I’m not sure why datetime is used instead of datetime2 but it’s working in my project. Feel free to separate these out into individual scripts:

/* Create AspNetUsers Table */
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[AspNetUsers](
  [Id] [nvarchar](128) NOT NULL,
  [Hometown] [nvarchar](max) NULL,
  [Email] [nvarchar](256) NULL,
  [EmailConfirmed] [bit] NOT NULL,
  [PasswordHash] [nvarchar](max) NULL,
  [SecurityStamp] [nvarchar](max) NULL,
  [PhoneNumber] [nvarchar](max) NULL,
  [PhoneNumberConfirmed] [bit] NOT NULL,
  [TwoFactorEnabled] [bit] NOT NULL,
  [LockoutEndDateUtc] [datetime] NULL,
  [LockoutEnabled] [bit] NOT NULL,
  [AccessFailedCount] [int] NOT NULL,
  [UserName] [nvarchar](256) NOT NULL,
 CONSTRAINT [PK_dbo.AspNetUsers] PRIMARY KEY CLUSTERED 
(
  [Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)

GO

/* Create AspNetUsersExt Table */
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[AspNetUsersExt](
  [UserId] [nvarchar](128) NOT NULL,
  [SecurityQuestion] [nvarchar](256) NULL,
  [SecurityAnswer] [nvarchar](128) NULL,
  [SecurityAnswerSalt] [nvarchar](128) NULL,
 CONSTRAINT [PK_AspNetUsersExt] PRIMARY KEY CLUSTERED 
(
  [UserId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
) 

GO

ALTER TABLE [dbo].[AspNetUsersExt]  WITH CHECK ADD  CONSTRAINT [FK_AspNetUserExt_AspNetUsers] FOREIGN KEY([UserId])
REFERENCES [dbo].[AspNetUsers] ([Id])
ON UPDATE CASCADE
ON DELETE CASCADE
GO

ALTER TABLE [dbo].[AspNetUsersExt] CHECK CONSTRAINT [FK_AspNetUserExt_AspNetUsers]
GO

/* Create AspNetRoles Table */
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[AspNetRoles](
  [Id] [nvarchar](128) NOT NULL,
  [Name] [nvarchar](256) NOT NULL,
 CONSTRAINT [PK_dbo.AspNetRoles] PRIMARY KEY CLUSTERED 
(
  [Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)

GO

/* Create AspNetUserLogins Table */
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[AspNetUserLogins](
  [LoginProvider] [nvarchar](128) NOT NULL,
  [ProviderKey] [nvarchar](128) NOT NULL,
  [UserId] [nvarchar](128) NOT NULL,
 CONSTRAINT [PK_dbo.AspNetUserLogins] PRIMARY KEY CLUSTERED 
(
  [LoginProvider] ASC,
  [ProviderKey] ASC,
  [UserId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)

GO

ALTER TABLE [dbo].[AspNetUserLogins]  WITH CHECK ADD  CONSTRAINT [FK_dbo.AspNetUserLogins_dbo.AspNetUsers_UserId] FOREIGN KEY([UserId])
REFERENCES [dbo].[AspNetUsers] ([Id])
ON DELETE CASCADE
GO

ALTER TABLE [dbo].[AspNetUserLogins] CHECK CONSTRAINT [FK_dbo.AspNetUserLogins_dbo.AspNetUsers_UserId]
GO

/* Create AspNetUserClaims Table */
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[AspNetUserClaims](
  [Id] [int] IDENTITY(1,1) NOT NULL,
  [UserId] [nvarchar](128) NOT NULL,
  [ClaimType] [nvarchar](max) NULL,
  [ClaimValue] [nvarchar](max) NULL,
 CONSTRAINT [PK_dbo.AspNetUserClaims] PRIMARY KEY CLUSTERED 
(
  [Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)

GO

ALTER TABLE [dbo].[AspNetUserClaims]  WITH CHECK ADD  CONSTRAINT [FK_dbo.AspNetUserClaims_dbo.AspNetUsers_UserId] FOREIGN KEY([UserId])
REFERENCES [dbo].[AspNetUsers] ([Id])
ON DELETE CASCADE
GO

ALTER TABLE [dbo].[AspNetUserClaims] CHECK CONSTRAINT [FK_dbo.AspNetUserClaims_dbo.AspNetUsers_UserId]
GO

/* Create AspNetUserRoles Table */
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[AspNetUserRoles](
  [UserId] [nvarchar](128) NOT NULL,
  [RoleId] [nvarchar](128) NOT NULL,
 CONSTRAINT [PK_dbo.AspNetUserRoles] PRIMARY KEY CLUSTERED 
(
  [UserId] ASC,
  [RoleId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)

GO

ALTER TABLE [dbo].[AspNetUserRoles]  WITH CHECK ADD  CONSTRAINT [FK_dbo.AspNetUserRoles_dbo.AspNetRoles_RoleId] FOREIGN KEY([RoleId])
REFERENCES [dbo].[AspNetRoles] ([Id])
ON DELETE CASCADE
GO

ALTER TABLE [dbo].[AspNetUserRoles] CHECK CONSTRAINT [FK_dbo.AspNetUserRoles_dbo.AspNetRoles_RoleId]
GO

ALTER TABLE [dbo].[AspNetUserRoles]  WITH CHECK ADD  CONSTRAINT [FK_dbo.AspNetUserRoles_dbo.AspNetUsers_UserId] FOREIGN KEY([UserId])
REFERENCES [dbo].[AspNetUsers] ([Id])
ON DELETE CASCADE
GO

ALTER TABLE [dbo].[AspNetUserRoles] CHECK CONSTRAINT [FK_dbo.AspNetUserRoles_dbo.AspNetUsers_UserId]
GO

Now that we have our tables, we can add the requisite assemblies and classes to our solution which give us the ASP.NET Identity functionality:

Step 2. Update Existing Connection String To Add providerName

If your connection string is missing a providerName attribute, make sure you add it. This is required for the persistence mechanism in ASP.NET Identity to work.

Gotcha Warning: This one really threw me. Long story short, I forgot to update this in my release Web.config transform and took down my production site (only for a few seconds, fortunately!) So, if you use config transforms, make sure you check this!

<add connectionString="Server=SERVERNAME;Database=DB;User ID=USERNAME;Password=PASSWORD;Trusted_Connection=False" name="MyConnectionString" providerName="System.Data.SqlClient" />

Step 3. Add References To ASP.NET Identity Assemblies

You will need to add references to the ASP.NET Identity assemblies, preferably using NuGet as below:

Install-Package Microsoft.AspNet.Identity.Owin
Install-Package Microsoft.Owin.Host.SystemWeb

Optionally if you later want to include Facbeook, Twitter, or other authentication providers, you can do so now (optional):

Install-Package Microsoft.Owin.Security.Facebook
Install-Package Microsoft.Owin.Security.Google
Install-Package Microsoft.Owin.Security.Twitter
Install-Package Microsoft.Owin.Security.MicrosoftAccount

Step 4. Add Code For ASP.NET Identity Functionality

Now that we have our database tables and all of our required assemblies, we need to add the code which ties it all together. If you’re using a single project, you can put these wherever you want. If your application is split into several projects like mine, put them in a deeper assembly like your data tier or authentication layer.

a. Add The ApplicationUser Class

// ApplicationUser.cs
public class ApplicationUser : IdentityUser
{
  public ApplicationUser() {}
}

b. Add The ApplicationDbContext Class

// ApplicationDbContext.cs
public class ApplicationDbContext : IdentityDbContext<ApplicationUser>
{
  // throwIfV1Schema throws an exception if an existing v.1 Schema
  // exists for ASP.NET Identity
  public ApplicationDbContext()
    : base("MyConnectionString", throwIfV1Schema: false) {}
}

c. Add The SqlPasswordHasher Class

This class will be responsible for authenticating old Membership and new Identity passwords. It uses a specific convention which I will explain later, but all you need to know now is that the old Membership password will be stored in a pipe-delimited format in our database after we migrate our users.

// SqlPasswordHasher.cs
public class SqlPasswordHasher : PasswordHasher
{
  public override PasswordVerificationResult VerifyHashedPassword(string hashedPassword, string providedPassword)
  {
    string[] passwordProperties = hashedPassword.Split('|');
    if (passwordProperties.Length != 3)
    {
      return base.VerifyHashedPassword(hashedPassword, providedPassword);
    }
    else
    {
      string passwordHash = passwordProperties[0];
      int passwordformat = 1;
      string salt = passwordProperties[2];
      if (String.Equals(EncryptPassword(providedPassword, passwordformat, salt), passwordHash,
        StringComparison.CurrentCultureIgnoreCase))
      {
        return PasswordVerificationResult.SuccessRehashNeeded;
      }
      else
      {
        return PasswordVerificationResult.Failed;
      }
    }
  }

  // This is copied from the existing SQL providers and is provided only for back-compat.
  private string EncryptPassword(string pass, int passwordFormat, string salt)
  {
    if (passwordFormat == 0) // MembershipPasswordFormat.Clear
      return pass;

    byte[] bIn = Encoding.Unicode.GetBytes(pass);
    byte[] bSalt = Convert.FromBase64String(salt);
    byte[] bRet = null;

    if (passwordFormat == 1)
    {
      // MembershipPasswordFormat.Hashed 
      HashAlgorithm hm = HashAlgorithm.Create("SHA1");
      if (hm is KeyedHashAlgorithm)
      {
        KeyedHashAlgorithm kha = (KeyedHashAlgorithm) hm;
        if (kha.Key.Length == bSalt.Length)
        {
          kha.Key = bSalt;
        }
        else if (kha.Key.Length < bSalt.Length)
        {
          byte[] bKey = new byte[kha.Key.Length];
          Buffer.BlockCopy(bSalt, 0, bKey, 0, bKey.Length);
          kha.Key = bKey;
        }
        else
        {
          byte[] bKey = new byte[kha.Key.Length];
          for (int iter = 0; iter < bKey.Length;)
          {
            int len = Math.Min(bSalt.Length, bKey.Length - iter);
            Buffer.BlockCopy(bSalt, 0, bKey, iter, len);
            iter += len;
          }
          kha.Key = bKey;
        }
        bRet = kha.ComputeHash(bIn);
      }
      else
      {
        byte[] bAll = new byte[bSalt.Length + bIn.Length];
        Buffer.BlockCopy(bSalt, 0, bAll, 0, bSalt.Length);
        Buffer.BlockCopy(bIn, 0, bAll, bSalt.Length, bIn.Length);
        bRet = hm.ComputeHash(bAll);
      }
    }

    return Convert.ToBase64String(bRet);
  }
}

d. Add The ApplicationUserManager Class

// ApplicationUserManager.cs
public class ApplicationUserManager : UserManager<ApplicationUser>
{
  public ApplicationUserManager() 
    : base(new UserStore<ApplicationUser>(new ApplicationDbContext()))
  {
    this.PasswordHasher = new SqlPasswordHasher();
  }
}

Step 5. Get Login/Signup Working

Your implementation of login and signup may differ, but here are a few key points:

User creation is done by populating an instance of ApplicationUser and then passing it to an instance of ApplicationUserManager to persist.
The ApplicationUserManager provides a public property for PasswordHasher. We can directly reference this to utilize our SqlPasswordHasher and log users in whether they provide their legacy ASP.NET Membership password, or a new ASP.NET Identity password. This is why I do not use an instance of a SignInManager, which otherwise doesn’t provide the lower-level interfaces to do this (that I’m aware of–also this works so I didn’t want to change it.)
If a user logs in with their old Membership password, we will take advantage of having their password in memory and reset the user’s password to the new format. This isn’t necessary, but it’s a nice option to have.
In the user migration scripts below, notice how we concatenate our old Membership password with the password format and salt? The SqlPasswordHasher has a method called VerifyHashedPassword(...) that returns one of three results:
- PasswordVerificationResult.Success - All good, the user’s password is valid
- PasswordVerificationResult.Failed - The user’s password could not be validated
- PasswordVerificationResult.SuccessRehashNeeded - The user’s password was validated by our SqlPasswordHasher but we have determined that it was a legacy Membership password. At this point we may choose to re-apply the user’s password.

a. Create A User

Here, we create a user and then add their security question and answer to the data store. Note that after you create the account, you would have to log the user in as well. The details of how to do that are covered next.

var user = new ApplicationUser();
user.UserName = username;
user.Email = email;
user.EmailConfirmed = true;
user.LockoutEnabled = true;
if (_userManager.Create(user, password) == IdentityResult.Success)
{
  // Add the user's security question and answer
  // This uses another Entity Context, and is separate from ASP.NET Identity
  var userExts = new AspNetUsersExt();
  userExts.UserId = user.Id;

  // These 3 properties are up to you; my implementation may be different
  // than what you need
  userExts.SecurityQuestion = securityQuestion;
  userExts.SecurityAnswerSalt = salt;
  userExts.SecurityAnswer = securityAnswer;

  using (var entities = new MyEntities())
  {
    entities.AspNetUsersExts.Add(userExts);
    entities.SaveChanges();
  }

  // All done
  // You would now log the user in
}

b. Log In With The User

Below is quick example of how you could log in a user, with an example of how to re-hash the password. This example uses OWIN context, which admittedly I don’t really understand at this point, so just copy/paste that part. This is a refactored version of my code, but should give you a pretty good idea:

protected bool Login(string username, string password)
{
  // Instantiate a new ApplicationUserManager and find a user based on provided Username
  var userManager = new ApplicationUserManager();
  var user = userManager.FindByName(username);

  // Invalid user, fail login
  if (user == null || userManager.IsLockedOut(user.Id))
  {
    InvalidLogin(); // Do something here to tell the user
    return false;
  }

  // Valid user, verify password
  var result = userManager.PasswordHasher.VerifyHashedPassword(user.PasswordHash, password);
  if (result == PasswordVerificationResult.Success)
  {
    UserAuthenticated(userManager, user);
  }
  else if (result == PasswordVerificationResult.SuccessRehashNeeded)
  {
    // Logged in using old Membership credentials - update hashed password in database
    // Since we update the user on login anyway, we'll just set the new hash
    // Optionally could set password via the ApplicationUserManager by using
    // RemovePassword() and AddPassword()
    user.PasswordHash = userManager.PasswordHasher.HashPassword(password);
    UserAuthenticated(userManager, user);
  }
  else
  {
    // Failed login, increment failed login counter
    // Lockout for 15 minutes if more than 10 failed attempts
    user.AccessFailedCount++;
    if (user.AccessFailedCount >= 10) user.LockoutEndDateUtc = DateTime.UtcNow.AddMinutes(15);
    userManager.Update(user);
    InvalidLogin(); // Do something here to tell the user
    return false;
  }
}

private void UserAuthenticated(ApplicationUserManager userManager, ApplicationUser user)
{
  // Create an instance of an AuthenticationManager and Identity to authenticate and sign in the user
  // If all goes well, redirect the user to either the querystring's return URL, or their account
  var authenticationManager = HttpContext.Current.GetOwinContext().Authentication;
  var userIdentity = userManager.CreateIdentity(user, DefaultAuthenticationTypes.ApplicationCookie);
  authenticationManager.SignIn(new AuthenticationProperties() { IsPersistent = false }, userIdentity);
  user.AccessFailedCount = 0;
  user.LockoutEndDateUtc = null;
  userManager.Update(user);
  Response.Redirect(Request.QueryString["ReturnUrl"] ?? "~/Account/");
}

c. Log The User Out

Fairly straightforward. This kills the auth cookie and logs the user out:

var authenticationManager = HttpContext.Current.GetOwinContext().Authentication;
authenticationManager.SignOut();

Step 6. Migrate Membership Database Users

And now the fun part. Do you have good database backups? Yes? Okay good. Back them up again.

In all truth, this script is pretty simple and should be idempotent, but as always, if you’re running queries on a production database, make sure you have a reliable backup. Is your backup script done yet? Okay good let’s move on.

This last step consists of running a script to find users in the [aspnet_Membership] table, joined with [aspnet_Users], and copy them over into the new [AspNetUsers] and, optionally, [AspNetUsersExt] tables. Remember again, the only thing we’re copying into the extended table are the security question and answer fields. If you’re not using the security question and answer method of password reset in your Membership implementation, you can ignore this table (or use it as an example for migrating any other data you may want to keep.)

In the script below, you can alter the TOP 1 to any number that you are comfortable with. I started with 1, and bumped it up to 1,000, then 5,000, then 10,000. In my SQL Azure database, it took roughly 20-25 seconds to migrate 10,000 users. I ran that batch a few times until all of my users were migrated, about 75,000 in total.

The script will copy the basic user details, but with a few things to note:

The password will be copied as a 3-part delimited string. Again, our SqlPasswordHasher will make use of these pipe delimiters.
Since ASP.NET Identity uses a combination of two fields to determine if a user is locked out ([LockoutEnabled] determines that the user can be locked out, and [LockoutEndDateUtc] actually does the work) we need to set both fields if a user is locked out in the Membership database. In my website, I unfortunately deal with a lot of fake accounts and spammers, so the technique I use is to set [LockoutEnabled] to 1 for all users, and to set [LockoutEndDateUtc] to an arbitrary 1,000 years in the future for locked out Membership accounts. If my site is still up in 1,000 years, they can have their accounts back.
The whole thing happens in a transaction. Smart, right?
The first part of the query migrates a batch of users idempotently. The second part migrates any security question/answers that were not already moved.
Note that we copied the Security Answer Salt from the Membership table. That’s because the PBKDF2 passwords in ASP.NET Identity have a built-in salt, so a separate field is no longer needed in the database to store it. However, we still need it for our security question/answer. I haven’t yet refactored this, but by this point I’m sure you can imagine how.

Gotcha Warning: You’ll notice that the new users table has a field called [SecurityStamp]. This is not a password salt so please do not treat it as such. This is a token used to verify the state of an account and is subject to change at any time. Do not use it as a salt for your application, or for your security question/answer.

BEGIN TRANSACTION MigrateUsers

  /* Migrate users */
  INSERT INTO AspNetUsers (Id,UserName,PasswordHash,SecurityStamp,EmailConfirmed,
  PhoneNumber,PhoneNumberConfirmed,TwoFactorEnabled,LockoutEndDateUtc,LockoutEnabled,AccessFailedCount,
  Email)
  SELECT TOP 1 aspnet_Users.UserId, aspnet_Users.UserName,
  (aspnet_Membership.Password+'|'+CAST(aspnet_Membership.PasswordFormat as varchar)+'|'+aspnet_Membership.PasswordSalt),
  NewID(), 1, NULL, 0, 1, CASE WHEN aspnet_Membership.IsLockedOut = 1 THEN DATEADD(YEAR, 1000, SYSUTCDATETIME()) ELSE NULL END, 1, 0,
  aspnet_Membership.Email
  FROM aspnet_Users
  LEFT OUTER JOIN aspnet_Membership ON aspnet_Membership.ApplicationId = aspnet_Users.ApplicationId 
  AND aspnet_Users.UserId = aspnet_Membership.UserId
  LEFT OUTER JOIN AspNetUsers ON aspnet_Membership.UserId = AspNetUsers.Id
  WHERE AspNetUsers.Id IS NULL

  /* Migrate user question/answer */
  INSERT INTO AspNetUsersExt (UserId, SecurityQuestion, SecurityAnswer, SecurityAnswerSalt)
  SELECT aspnet_Users.UserId, aspnet_Membership.PasswordQuestion, PasswordAnswer, PasswordSalt
  FROM aspnet_Users
  LEFT OUTER JOIN aspnet_Membership ON aspnet_Membership.ApplicationId = aspnet_Users.ApplicationId 
  AND aspnet_Users.UserId = aspnet_Membership.UserId
  LEFT OUTER JOIN AspNetUsersExt ON aspnet_Membership.UserId = AspNetUsersExt.UserId
  LEFT OUTER JOIN AspNetUsers ON aspnet_Membership.UserId = AspNetUsers.Id
  WHERE AspNetUsers.Id IS NOT NULL AND AspNetUsersExt.UserId IS NULL

IF @@ERROR <> 0 
  BEGIN 
    ROLLBACK TRANSACTION MigrateUsers
    RETURN
  END

COMMIT TRANSACTION MigrateUsers

All Done… Almost!

At this point we’ve completed all the requisite steps to move users and make them available to log in. You should be able to log in using any newly-created ASP.NET Identity user, or any legacy migrated Membership user.

Gotcha Warning: One thing I wanted to point out is that the [AspNetUsers] table uses a Guid ID. This table likewise does not have a signup/create date for users. The end result is a table of users with no discernable order nor signup date. I personally prefer to have a signup date and order for users, so I later added a column to the table called [CreatedOnUtc]. You can also use an auto-incrementing integer, but it takes some code and database changes which you’ll need to research.

To add the column to [AspNetUsers] with a default UTC date and time:

ALTER TABLE AspNetUsers 
ADD CreatedOnUtc DateTime2 NULL 
CONSTRAINT DF_AspNetUsers_CreatedOnUtc DEFAULT SYSUTCDATETIME()

To update the values from the Membership database:

UPDATE TOP(1) AspNetUsers 
   SET AspNetUsers.CreatedOnUtc = aspnet_Membership.CreateDate 
   FROM AspNetUsers INNER JOIN aspnet_Membership 
   ON AspNetUsers.Id = aspnet_Membership.UserId
   WHERE AspNetUsers.CreatedOnUtc IS NULL

The [CreatedOnUtc] field does not necessarily need to be added to your ApplicationUser model, though you could add it. I simply wanted to have the data in case I need it in the future.

Please let me know if you have any questions, comments, or concerns below.

RDLC Performance Issues in .NET 4.5

Mon, 27 Oct 2014 00:00:00 +0000

I was recently tasked with investigating some performance issues related to RDLC reporting on a website. RDLC is the local implementation of SQL Server Reporting Services (SSRS) wherein report processing happens on an application server using in-memory data, and doesn’t require the use of an SSRS server. Reports can be embedded directly on a web page using the <asp:ReportViewer> control, which is quite nice.

RDLC support is built into .NET, and it’s an acceptable choice for smaller projects with simple reports. It’s also free, and that makes it an perfect choice for clients who don’t want to invest in a great tool. Commercial reporting tools get expensive, especially when you start looking into all the fun Business Intelligence, number-crunching options they provide.

The Problem

In this particular installation, the client requested a new report with a few more expressions and more levels of grouping than we had in previous reports. This new report also required more sub-total sections, many of which had aggregate calculations for Average and Sum. Unfortunately, the complexity of the report pushed its rendering time up into the tens of seconds on smaller reports, and lol-infinity amounts of time on larger ones. The entire report was unusable. I’m talking 10-15 minutes of processing for a 10-page report. It was that bad.

Searches online yielded very little information, until I eventually stumbled on a clue that Microsoft broke RDLC and failed to find the issue in regression tests:

The problem seems to be a degression that occured with the transition of the report viewer local mode to Net 4.0 running in the sandbox Domain (the Sandbox Domain is used by default in Net 4.0) … Expression Evaluation in Local Mode Unfortunately this problem was not caught in the Beta Testing.

More information can also be found here: Expression Evaluation in Local Mode

Solution Options

As it turns out, there are a few ways to fix the problem:

Force the use of legacy CAS security with the configuration option <trust legacyCasModel="true" level="Full"/>. I don’t even care what this does; it sounds like a terrible idea. Reverting to an older security model? No thank you. That said, I tried this, and it was no help. The use of dynamic variables in the project prevents using this legacy setting, and being an Umbraco project, we use them extensively in many of our Razor views.
Refactor the report. This may have helped somewhat, but the underlying issue would remain. Plus, refactoring the report felt like a giant hack. Given the next option, though, I seriously considered it.
Revert to .NET Framework 3.5. This is the part where you get to LOL and omgwtf, especially if you consider this is what I actually did … sort of.

The Solution

After much consideration, it made the most sense for me to split the reporting areas of the site out into another IIS Application running under .NET 3.5 (on a different subdomain), and add in some simple functionality to link between the two. It ended up being much easier than expected, and is thus far fully transparent to the client. No, it wasn’t fun, in case you’re getting jealous.

The good news is that the application is now able to process a 200-page report in roughly 12-15 seconds, and smaller reports of a few pages render in mere seconds.

Final Thoughts

Overall, RDLC is a decent technology that suffers merely from neglect. Report design is straightforward and relatively flexible. Displaying reports inline on a website is easy with the <asp:ReportViewer> tag, despite the viewer being a bit dated.

I really hope Microsoft will reinvest in RDLC for those of us who want a simple solution without selling family members to pay for it. It has the potential to be a great tool.

Error: ENOENT... When Installing Socket.io using NPM

Fri, 17 Oct 2014 00:00:00 +0000

The Problem

When installing Socket.io on Node.js (Windows) via the following NPM command:

npm install socket.io

You may receive the following error:

Error: ENOENT, stat 'C:\Users\...\AppData\Roaming\npm'

Mmmm, ambiguous error messages. :trollface:

The Solution

Apparently this is a bug in the currently current version of Node.js (v0.10.32) and/or NPM (v1.4.28). The solution is straightforward and just requires creating the above noted “npm” folder within AppData. I’m sure you can figure this out, but copy-pasting is much more fun:

mkdir %APPDATA%\npm

That’s it. You should now be able to execute the NPM command to install Socket.io without any errors. Note you will probably need to run this command with administrative privileges. I didn’t need to run my command line as an Administrator, but I’ve read that you may have to (yes, you, specifically; the Google referred to you by name.)

Deleting Foreign Key Records While Keeping a Historical Association to Them

Tue, 14 Oct 2014 00:00:00 +0000

Developers have a hard time letting go of data. Developers like myself are, for example, likely to keep thousands of photos backed up on a server, half of which are too blurry, overexposed, or underexposed to be usable. I probably have a few that were taken with the lens cap on, and yet I keep them. I’m a byte hoarder, and storage is cheap. At least it’s just data and I still have access to my bathroom, so it could be worse.

The hesitation to delete makes us likely to archive records because it gives us a sense of security, particularly with client applications hosting critical business records. Data retention policies aside, it’s an easy and safe practice provided the means by which data is removed (in this case: “deleted”, “archived” are reasonably synonymous) is documented and easy to follow. Oh, and hard to screw up. We don’t suddenly want old, irrelevant data spilling out into the application.

The Problem

A client recently wanted to delete some Customers from a very custom sales application. Up until this point, the application has been small and pragmatic, and iteratively built. Deleting a Customer is a feature we hadn’t implemented yet (believe it).

Easy enough. In this particular application’s database, we largely followed the pattern used in AdventureWorks (highly normalized, with entities like Customer, Supplier having an “is a” relationship to a BusinessEntity table.) To delete records, I implemented an “Archived” bit in the BusinessEntity table. Arguably, an ArchivedOn field of type DateTime2 would probably have been a better choice, and adding that field to the more specific subtype of Customer may have been more logical, but hindsight, right?

Now a simple flip of the bit “deletes” a record. But, how do we handle the historical foreign key relationship now that the the Customer is “deleted”? If we’re always querying from the Customers table for non-archived records, how do we load the one we need for a Sales Order when we’re editing or viewing it? It’s a deceptively complex problem that some people will naturally understand; perhaps from lack of sleep or overestimating the problem, I was not one of those people! I won’t admit how long I had to chew on this one. :grin:

The Solution

The solution is simple. Like, really simple. When loading the record, we simply need the CustomerID (foreign key) from the SalesOrder record, and we use that in our Customers query to append that single record to the result set. So effectively, our query becomes something like this:

SELECT CustomerID, Name FROM Customers WHERE Archived = 0 OR CustomerID = 56404

That gives us all of the non-archived Customers plus the one we need.

Relevant links

StackOverflow Question

Slow Performance - Windows Performance Counter API

Tue, 30 Sep 2014 00:00:00 +0000

I recently had the ~~pleasure~~ opportunity to work with the Windows Performance Counter API as a data store for a metrics-type service in ASP.NET Web API. We needed to generate a large number of counters (in the thousands) for several partners and properties, and used them to provide data endpoints that partners could consume.

Adding the first couple of partners was a breeze, but as we rolled out the service to more partners, performance quickly began to degrade and our beautiful service became unusable. Requests took minutes to process, and timeouts were rampant. But the Windows Performance Counter system is supposed to be fast, right?

I couldn’t see any bottlenecks in my code, save for the actual Performance Counter API calls. There was a slight delay in getting all counters for a category, as expected, but reading each counter took anywhere from 10-20ms, and the per-counter read time worsened with the addition of more counters. For several thousand counters, that’s a lot of lag! Let’s also consider that many counters are read twice to collect average or timed data.

The Problem

Category overload: To dig deeper into the problem, I created a counter Category for each partner (instead of putting all counters in a single category), which immediately seemed to help. This brought the per-counter read time down drastically. So it was clear that the read time per counter was affected by the total number of counters in a category–an exponential problem. Strangely, even iterating through a small subset of counters from an overloaded category wasn’t much better.

Here are some stats when reading counter values from a single category:

8 total counters, 1-2ms per counter
256 total counters, 15-18ms per counter
512 total counters, 30ms per counter
3,584 total counters (reading all counters), 200ms per counter
3,584 total counters (reading only 512 counters), 50-90ms per counter.

I used System.Diagnostics.Stopwatch before and after the change to verify this:

Stopwatch t;
foreach (var c in counters)
{
    t = Stopwatch.StartNew();
    var r = c.RawValue;
    Debug.WriteLine(t.ElapsedMilliseconds.ToString("000") + " - " + c.CategoryName + ":" + c.CounterName + "(" + c.CounterType + ") = " + r);
}

The Solution

More categories! That’s it. Each partner now has its own category, each having 8-10 counters. Read time is < 5ms total, and everyone is happy. To test, I was able to programmatically spawn a hundred new categories, each having 5-10 counters, and performance was still blazing fast. Crisis averted.

One last thing: When working with the Performance Counter API, the addition of more than just a few counters will require either a registry change or a machine.config update to allocate more memory to the process. More information can be found here: MSDN - performanceCounters Element