Technical Debt: that escalated quickly

First written and published on LITA Blog
http://litablog.org/2017/07/technical-debt-that-escalated-quickly/


If you’re not familiar with the term “technical debt”, it’s an analogy coined by Ward Cunningham[1], used to relay what happens when rather than following best practices and standards we take shortcuts on technical projects to have a quick fix. Debt occurs when we take on a long-term burden in order to gain something in the short term.

I want to note that inevitably we will always take on some sort of debt, often unknowingly and usually while learning; the phrase “hindsight is 20/20” comes to mind, we see where we went wrong after the fact. There is also inherited technical debt, in all of my jobs, current and past, I inherited technical debt, this is out of my control, it happens and I still need to learn how to deal with it. This piece aims to give some guidelines and bits I’ve learned over the years in dealing with technical debt and doing me best to maintain it, because really, it’s unavoidable and ignoring it doesn’t make it go away. Believe me, I’ve tried. 

Technical debt can refer to many different things including, but not limited to: infrastructure, software, design/UX, or code. Technical debt reduces the long term agility of a team; it forces us to rely on short term solution thinking and make trade-offs for short term agility. When done haphazardly and not managed, technical debt can shut down a team’s ability to move forward on a project, their long term agility.

It accrues quickly and often we don’t realize just how quickly. For example, I’d been tasked with implementing single-sign on (SSO) for a multitude of applications in our library. In the process of mapping out the path of action this led to learning that in order to implement the bits we needed for SSO most of the applications needed to be updated and the newer versions weren’t compatible with the version of PHP running on our servers, and to use the version of PHP that would be compatible we needed to upgrade our server and the upgrade on the server was a major upgrade which led to having to do a full server upgrade and migration. Needless to say, SSO has not yet been implemented. This technical debt accrued from a previous admin’s past decisions to not stay on top of the upgrades for many of our applications because short term hacks were put in place and the upgrades would break those hacks. These decisions to take on technical debt ultimately caught up with us and halted the ability to move forward on a project. Whether the debt is created under your watch or inherited, it will eventually need to be addressed.

The decisions that are made which result in technical debt should be made with a strategic engineering perspective. Technical debt should only be accrued on purpose because it enables some business goal, intentional and unintentional. Steve McConnell’s talk on Managing Technical Debt [2] does a good job of laying the business and technical aspects of taking on technical debt. Following that, ideally there should be a plan in place on how to reasonably reduce the debt down the road. If technical debt is left unaddressed, at some point the only light at the end of the tunnel is to declare bankruptcy, analogically: just blow it up and start over.

Technical debt is always present, it’s not always bad either but it’s always on the verge of getting worse. It is important to have ways of hammering through it, as well as having preventative measures in place to keep debt to a minimum and manageable for as long as possible.

So how do you deal with it?

Tips for dealing with inherited technical debt:

  • Define it. What counts as technical debt? Why is it important to do something about it?
  • Take inventory, know what you’re working with.
  • Prioritize your payoffs. Pick your technical battles carefully, which bits need addressing NOW and which bits can be addressed at a later date?
  • Develop a plan on what and how you’re going to address and ultimately tidy up the debt.
  • Track technical debt. However you track it, make sure you capture enough detail to identify the problem and why it needs to be fixed.

Preventative tips to avoiding technical debt (as much as you can):

  • Before taking on debt ask yourself…
    • Do we have estimates for the debt and non-debt options?
    • How much will the quick & dirty option cost now? What about the clean options?
    • Why do we believe that it is better to incur the effort later than to incur it now? What is expected to change to make taking on that effort more palatable in the future?
    • Have we considered all the options?
    • Who’s going to own the debt?
  • Define initial requirements in a clear and constant style. A good example of this is Gherkin: https://cucumber.io/docs/reference
  • Create best practices. Some examples:  KISS (Keep It Simple Stupid), DRY (Don’t Repeat Yourself), YAGNI (You Aren’t Gonna Need it)
  • Have a standard, an approved model of taking shortcuts, and stick to it. Remember to also reevaluate that standard periodically, what once was the best way may not always be the best way.
  • Documentation. A personal favorite: the “why-and” approach. If you take a temporary (but necessary) shortcut, make note of it and explain why you did what you did and what needs to be done to address it. Your goal is to avoid having someone look at your code/infrastructure/digital records/etc and asking “why is it like that?” Also for documentation, a phenomenal resource (and community) is Write The Docs (http://www.writethedocs.org/guide
  • Allow for gardening. Just as you would with a real garden you want to tidy up things in your projects sooner rather than later. General maintenance tasks that can be done to improve code/systems/etc now rather than filed on the low priority “to-do” list.
  • TESTS! Write/use automated tests that will catch bugs and issues before your users. I’m a fan of using tools like Travis CI (https://travis-ci.org/), Cucumber (https://cucumber.io/docs), Fiddler (http://www.telerik.com/fiddler) and Nagios (https://www.nagios.org/)  for testing and monitoring. Another resource recommended to me (thanks Andromeda!)  is Obey the Testing Goat (http://www.obeythetestinggoat.com/pages/book.html#toc)
  • Remember to act slower than you think. Essentially, think through how something should be done before actually doing it.

And my final thought, commonly referred to as the boy scout rule, when you move on from a project or team and someone else inherits what you leave behind, do your best to leave it better than when you found it.


Footnote:
  1. Ward Cunningham, Explaing Debt Metaphor [Video] http://wiki.c2.com/?WardExplainsDebtMetaphor
  2. Managing Technical Debt by Steve McConnell (slides) http://2013.icse-conferences.org/documents/publicity/MTD-WS-McConnell-slides.pdf

Extra Reading/Tools:

How to deal with technical debt? by Vlad Alive https://m.vladalive.com/how-to-deal-with-technical-debt-33bc7787ed7c

Obey the Testing Goat by Harry Percival  http://www.obeythetestinggoat.com/pages/book.html#toc

How to write a good bug report? Tips and Tricks http://www.softwaretestinghelp.com/how-to-write-good-bug-report/

Tools & Services list https://www.stickyminds.com/tools-guide

Don’t take the technical debt metaphor too far http://swreflections.blogspot.com/2012/12/dont-take-technical-debt-metaphor-too.html 

Advertisements

Update to the Problem: we have a solution

So a few months ago there was a problem Where I continue to admire the Problem without a complete Solution. This has been solved for a few months, but sh!t happens. Am now finally getting around to providing the a solution, one that works for me, but may not work for you. This is pieced together haphazardly, mainly so I have note of it and well so if you’re looking for help on doing something similar you have a much better starting point than I did.

After a hot minute with Python-LDAP I determined it was a beast I was not interested in taming at the moment because well I had another option, mind you at the time we thought this ‘other’ option was going to be easier. I don’t know if it was or not, but it took some serious neuron firing to do.

At one point I dove deep into VBA scripting, where I figuratively lost loads of hair and age 20 years. The script scraped hundreds of emails for a text string (unique ID), to parse out into an excel file (staying within the suite of Microsoft Office deemed significantly easier than going out) and then ran line by line on LDAP to print the output of the request attributes and then convert to csv and email to colleague to do a mail merge.

**confession** I actually signed up for a Stack Overflow account because of this!

One Solution but not THE solution:

VBScript and a Bash script that ran LDAP Queries, Python conversions and used Mutt to send an email attachment.
I’ve changed values to neutral values, you will need to update them to match what you need.
Here is my repo that provides the files you’ll need if you decide to take the VBScript route. https://github.com/whitni/VBScriptandLDAP

Once you set this up in VBA, you can set it as a macro, but I had it set as a rule at first and then switched to doing something different right after I had this solution in place…so didn’t bother.

The Solution I settled on:

I returned to a single bash script that queried LDAP matching on certain attributes, primary change was the decision to send emails out based on start dates and not when we received an email notification from HR. This will soon be turned into a cron job and I can dust my hands of it BUT walking away with significantly more knowledge of LDAP, AD, VBA and all that…

#!/bin/bash
#A simple script

## date format ##
today=$(date +”%F”)
startdate=$(date +”%Y%m”)

##backup path ##
BAK=”/path/that/you/saved/file/newhires_$today”

# LDAP Search query
ldapsearch -W -h your.AD.server.com -x -D ldapusername@email.com -b “dc=$1,dc=$2,dc=$3” -s sub “attributes you need to match or not match on” attributes you want > $BAK

#convert LDIF to csv
python LDIFtoCSV/LDIFtoCSV.py $BAK > /path/that/you/saved/file/newhires_$today.csv

#send email with file attached
mutt -a /path/that/you/saved/file/newhires_$today.csv -s “New Hire Emails” -c persontoCC@email.com — persontoemail@email.com

Resources used:
VBScript Library: https://msdn.microsoft.com/en-us/library/aa227499(v=vs.60).aspx
RegEx parsing: http://www.slipstick.com/developer/regex-parse-message-text/
LDAP Man page: [Linux_terminal] > man ldap
LDIFtoCSV conversion tool: https://github.com/tachang/ldiftocsv
Stack Overflow — my question: http://stackoverflow.com/questions/36752876/why-does-copying-a-string-from-outlook-to-excel-open-a-new-instance-of-excel-for/36754211#36754211

Collaboration work a barrier

Wikis, blogs, IM chats, and cloud drives, all types of collaboration tools. When I think about implementing  new tool first thing that comes to mind (after my “OMG this is going to make things so much easier”) is what old crab is not going to use this? We all work with them – don’t deny it.

This is a collaboration barrier, and not the only one. Other barriers include:

  • Conflict between department schedules, budgets, goals, and interest
  • Learning to balance the roles between the “hired role” and the “collaboration role”
  • Giving the collaboration value- “what will I get out of it?”

How do you solve the barriers? A couple of suggestions pulled from, Hutch Carpenters post “Enterprise 2.0: Culture Is As Culture Does” and Morten T. Hansen’s “When Internal Collaboration is Bad for Your Company” .

1. Eliminate the option to use something different (Carpenter, 2009).
 Carpenter recommends when implementing a new collaboration tool to eliminate the option to use something different. If you want to start using IM chats, instead of responding to emails send them a message via chat and no other way.

2. When introducing the software or change provide a sense of the future with the change. Show them what it will be like if they make the change.

3. Create incentives. It is difficult to change or desire to change and put in extra effort to use something new unless their is an acceptable answer to “what do I get out of it”. Hansen encourages determining the rate of return before jumping in. Is it going to be worth it both financially and efficiently.

For every barrier there is a solution.

-NimbleLibarian