Technical Debt: that escalated quickly

First written and published on LITA Blog
http://litablog.org/2017/07/technical-debt-that-escalated-quickly/


If you’re not familiar with the term “technical debt”, it’s an analogy coined by Ward Cunningham[1], used to relay what happens when rather than following best practices and standards we take shortcuts on technical projects to have a quick fix. Debt occurs when we take on a long-term burden in order to gain something in the short term.

I want to note that inevitably we will always take on some sort of debt, often unknowingly and usually while learning; the phrase “hindsight is 20/20” comes to mind, we see where we went wrong after the fact. There is also inherited technical debt, in all of my jobs, current and past, I inherited technical debt, this is out of my control, it happens and I still need to learn how to deal with it. This piece aims to give some guidelines and bits I’ve learned over the years in dealing with technical debt and doing me best to maintain it, because really, it’s unavoidable and ignoring it doesn’t make it go away. Believe me, I’ve tried. 

Technical debt can refer to many different things including, but not limited to: infrastructure, software, design/UX, or code. Technical debt reduces the long term agility of a team; it forces us to rely on short term solution thinking and make trade-offs for short term agility. When done haphazardly and not managed, technical debt can shut down a team’s ability to move forward on a project, their long term agility.

It accrues quickly and often we don’t realize just how quickly. For example, I’d been tasked with implementing single-sign on (SSO) for a multitude of applications in our library. In the process of mapping out the path of action this led to learning that in order to implement the bits we needed for SSO most of the applications needed to be updated and the newer versions weren’t compatible with the version of PHP running on our servers, and to use the version of PHP that would be compatible we needed to upgrade our server and the upgrade on the server was a major upgrade which led to having to do a full server upgrade and migration. Needless to say, SSO has not yet been implemented. This technical debt accrued from a previous admin’s past decisions to not stay on top of the upgrades for many of our applications because short term hacks were put in place and the upgrades would break those hacks. These decisions to take on technical debt ultimately caught up with us and halted the ability to move forward on a project. Whether the debt is created under your watch or inherited, it will eventually need to be addressed.

The decisions that are made which result in technical debt should be made with a strategic engineering perspective. Technical debt should only be accrued on purpose because it enables some business goal, intentional and unintentional. Steve McConnell’s talk on Managing Technical Debt [2] does a good job of laying the business and technical aspects of taking on technical debt. Following that, ideally there should be a plan in place on how to reasonably reduce the debt down the road. If technical debt is left unaddressed, at some point the only light at the end of the tunnel is to declare bankruptcy, analogically: just blow it up and start over.

Technical debt is always present, it’s not always bad either but it’s always on the verge of getting worse. It is important to have ways of hammering through it, as well as having preventative measures in place to keep debt to a minimum and manageable for as long as possible.

So how do you deal with it?

Tips for dealing with inherited technical debt:

  • Define it. What counts as technical debt? Why is it important to do something about it?
  • Take inventory, know what you’re working with.
  • Prioritize your payoffs. Pick your technical battles carefully, which bits need addressing NOW and which bits can be addressed at a later date?
  • Develop a plan on what and how you’re going to address and ultimately tidy up the debt.
  • Track technical debt. However you track it, make sure you capture enough detail to identify the problem and why it needs to be fixed.

Preventative tips to avoiding technical debt (as much as you can):

  • Before taking on debt ask yourself…
    • Do we have estimates for the debt and non-debt options?
    • How much will the quick & dirty option cost now? What about the clean options?
    • Why do we believe that it is better to incur the effort later than to incur it now? What is expected to change to make taking on that effort more palatable in the future?
    • Have we considered all the options?
    • Who’s going to own the debt?
  • Define initial requirements in a clear and constant style. A good example of this is Gherkin: https://cucumber.io/docs/reference
  • Create best practices. Some examples:  KISS (Keep It Simple Stupid), DRY (Don’t Repeat Yourself), YAGNI (You Aren’t Gonna Need it)
  • Have a standard, an approved model of taking shortcuts, and stick to it. Remember to also reevaluate that standard periodically, what once was the best way may not always be the best way.
  • Documentation. A personal favorite: the “why-and” approach. If you take a temporary (but necessary) shortcut, make note of it and explain why you did what you did and what needs to be done to address it. Your goal is to avoid having someone look at your code/infrastructure/digital records/etc and asking “why is it like that?” Also for documentation, a phenomenal resource (and community) is Write The Docs (http://www.writethedocs.org/guide
  • Allow for gardening. Just as you would with a real garden you want to tidy up things in your projects sooner rather than later. General maintenance tasks that can be done to improve code/systems/etc now rather than filed on the low priority “to-do” list.
  • TESTS! Write/use automated tests that will catch bugs and issues before your users. I’m a fan of using tools like Travis CI (https://travis-ci.org/), Cucumber (https://cucumber.io/docs), Fiddler (http://www.telerik.com/fiddler) and Nagios (https://www.nagios.org/)  for testing and monitoring. Another resource recommended to me (thanks Andromeda!)  is Obey the Testing Goat (http://www.obeythetestinggoat.com/pages/book.html#toc)
  • Remember to act slower than you think. Essentially, think through how something should be done before actually doing it.

And my final thought, commonly referred to as the boy scout rule, when you move on from a project or team and someone else inherits what you leave behind, do your best to leave it better than when you found it.


Footnote:
  1. Ward Cunningham, Explaing Debt Metaphor [Video] http://wiki.c2.com/?WardExplainsDebtMetaphor
  2. Managing Technical Debt by Steve McConnell (slides) http://2013.icse-conferences.org/documents/publicity/MTD-WS-McConnell-slides.pdf

Extra Reading/Tools:

How to deal with technical debt? by Vlad Alive https://m.vladalive.com/how-to-deal-with-technical-debt-33bc7787ed7c

Obey the Testing Goat by Harry Percival  http://www.obeythetestinggoat.com/pages/book.html#toc

How to write a good bug report? Tips and Tricks http://www.softwaretestinghelp.com/how-to-write-good-bug-report/

Tools & Services list https://www.stickyminds.com/tools-guide

Don’t take the technical debt metaphor too far http://swreflections.blogspot.com/2012/12/dont-take-technical-debt-metaphor-too.html 

Advertisements

Where I continue to admire the Problem without a complete Solution.

Hi — it’s me again. It’s been a while because I’ve been “stuck” on what I have wanted to populate this space with. I’ve decided on for now as a loud thinking space for problems I am working on at work because writing it out helps.

A problem I run into on a daily basis is knowing what I want to accomplish, the steps to accomplish it but now knowing how to piece it together. This is where I think that fundamental training (have you) in computer science would help. I have all the pieces but I don’t have the glue. Yet. I’m learning what I need to do to solve this but there’s little structure to it, because it’s all in the moment. See also why sometimes the formal education can be helpful.

Current problem I am trying to solve: we send out new hire emails and currently that’s done manually, meaning we receive an email notice with bits of information on a new employee/intern/etc (but not their email because it may or may not have been created at the time the notice was sent). We retain X, Y, Z of that information, put it in an excel spreadsheet and then manually look the person up in our internal directory for their email (if it exists) and drop that into the excel sheet & then do a mail merge on an email template (with a sizable amount of links).

This is a LOAD of work and there is a large back log that exists and to do that manually would be 100% inefficient, expensive & a waste of time.

Current solution I have: Run an ldapsearch query for exchange accounts created equal to or greater than a certain date & are not test/dummy accounts and print the X, Y, and Z variables for all of those accounts. Then convert that data from LDIF to .csv and save to a file on the server, which I can then drop the file into local shared drive (OR send email w/ attached file) where person who does mail merge can then take the csv file and run the mail merge. Goal is to automate the mail merge in the sense of once the file is created, have a job that checks “modified date” when that changes automatically send email and then have a VB Script that can be ran to check for certain emails (this is where instead of email attachment, might be better to have the file in the local drive) or check the local shared drive folder for this file & run the mail merge on it to send the emails.

New problem I have *no* idea how to glue all of this together so it can be executed and ran from a single command. I have the ldapsearch query, I know how to print the output, I have the Perl script to convert the data from LDIF to csv, I know how to email the output file (as an attachment), I don’t have the script all the way together for the mail merge yet because I’d like to focus on and solve the first problem of getting the data because you can forget about the mail merge if you don’t have the data.

Solution? I don’t have one yet.

**UPDATE**
To say I have no idea how to glue all of this together is not completely accurate. I know I want to write a bash script because I can run all of these pieces from the command line, that was purposeful. I know that I will want to use the python-LDAP API. I know that I can (will?) use perl for the data format conversion. I know that I want to automate the emailing of the output file as a cron job. What I don’t know. Yet. is the syntax of gluing these together so that they run seamlessly from minimal effort on my part (in the end).

Resources:
Python-LDAP Applications (using the python-LDAP API) [part 1] [part 2] [part 3] [part 4]

The job I didn’t get

I wrote this the night after my interview with Elon University for the Systems Librarian.

A day of Elon. I flew into Raleigh NC yesterday afternoon where [redacted] picked me up and took me on a beautiful scenic route back to Elon. I sure have missed the east coast.

[redacted] reminds me of [redacted] which is fitting because [redacted] is from Greensboro. [redacted] has been wonderful and really helpful this entire process, I am surprised that she is a cataloging librarian only because she is a delight to be around and I think would do well at the front.

Dinner last night was wonderful. The ladies I ate with had me laughing and smiling the entire time. My office would be right next to [redacted] and she is a hoot! I’ve really missed southern hospitality.

I am staying at this adorable and beautiful BnB called the Burke Manor. The bathroom is huge, the bed is comfy and the service is fantastic. I’m not use to this type of service and catering. Makes me look forward to staying in them more often.

Today was a long but delightful day. I had 4 more mtgs/interviews, a presentation, lunch and dinner with staff and 3 different tours. The staff at Elon is much smaller (30ish ppl) than UCR but they really embrace change and I appreciate and enjoy the direction and initiative the library dean has, the support of staff and the support for professional development and helping and making way for the staff to grow. One thing that I love about Elon is the community and campus involvement the library has. They do really great out reach and everyone holds high respect for each other.

I really felt like I fit in with this library it hits all the points that I want in a position. In one of the interviews the dreaded what do you want accomplish in 5-10 years career wise.  My answer? I want to be established at an institution, preferably one (if not) like Elon.

I attended a musical tonight put on by an Elon honor student and it was great. It struck me on every emotion and I was thrilled to attend it.

I am feeling really good about this, I just hope they see that I am a great fit for the job. Only time will tell and it is now in God’s hands.”

I finally heard that although I had a strong interview that they had selected another candidate for the position. I was crushed to read that. To get so far and feel so good about the interview and not make it; well at least I now have a good failure story. Which made me think ‘If I wanted to use this as my interview response, what would I want to be able to tell them?’ This has helped shaped my reaction to a crummy situation.

There are plenty of things I could have done better in the interview, because you can always do better. However, I put everything out there and although right now I feel that maybe if I interviewed better with the dean or with the associate provost or if I presented more eloquently I would have been selected, I know that I did give a very strong interview, that I did everything that I could to show them that I was their best choice.

So what now? I read the response from the dean at Elon that answers my question, “what was different between myself and the chosen candidate that made them chose them?” I put myself back together, I continue to learn and be involved and I try again and again until I make it.

How do I do this? Well I started another in depth search for positions I found interesting and I saved them. The goal is to begin working through the application process. My in depth search involved reviewing other job descriptions and pulling keywords to create my taxonomy.

Then I pulled up articles on skills that were highly recommended for the job I wanted as well as pulling from job descriptions. I then checked in with Treehouse so see if they had any tracks for training and I saved them. I also looked up tutorials and I used social media to ask for help.

One thing that I did do was allowed myself essentially a time for mourning the loss. I think that it is healthy to allow your body and mind to release that stress. It isn’t a sign of weakness to be sad and even upset about a situation like this. However, the key is to give yourself a deadline in which you will pick yourself back up and you will put it behind you and begin again.

Before diving back into a list of new applications, I also gave myself the day off, where I did everything but check email, get on the computer and access social media. Sort of a cleanse by disconnecting.

Then I start again. I put my best foot forward and I work towards that end goal.