Technical Debt: that escalated quickly

First written and published on LITA Blog
http://litablog.org/2017/07/technical-debt-that-escalated-quickly/


If you’re not familiar with the term “technical debt”, it’s an analogy coined by Ward Cunningham[1], used to relay what happens when rather than following best practices and standards we take shortcuts on technical projects to have a quick fix. Debt occurs when we take on a long-term burden in order to gain something in the short term.

I want to note that inevitably we will always take on some sort of debt, often unknowingly and usually while learning; the phrase “hindsight is 20/20” comes to mind, we see where we went wrong after the fact. There is also inherited technical debt, in all of my jobs, current and past, I inherited technical debt, this is out of my control, it happens and I still need to learn how to deal with it. This piece aims to give some guidelines and bits I’ve learned over the years in dealing with technical debt and doing me best to maintain it, because really, it’s unavoidable and ignoring it doesn’t make it go away. Believe me, I’ve tried. 

Technical debt can refer to many different things including, but not limited to: infrastructure, software, design/UX, or code. Technical debt reduces the long term agility of a team; it forces us to rely on short term solution thinking and make trade-offs for short term agility. When done haphazardly and not managed, technical debt can shut down a team’s ability to move forward on a project, their long term agility.

It accrues quickly and often we don’t realize just how quickly. For example, I’d been tasked with implementing single-sign on (SSO) for a multitude of applications in our library. In the process of mapping out the path of action this led to learning that in order to implement the bits we needed for SSO most of the applications needed to be updated and the newer versions weren’t compatible with the version of PHP running on our servers, and to use the version of PHP that would be compatible we needed to upgrade our server and the upgrade on the server was a major upgrade which led to having to do a full server upgrade and migration. Needless to say, SSO has not yet been implemented. This technical debt accrued from a previous admin’s past decisions to not stay on top of the upgrades for many of our applications because short term hacks were put in place and the upgrades would break those hacks. These decisions to take on technical debt ultimately caught up with us and halted the ability to move forward on a project. Whether the debt is created under your watch or inherited, it will eventually need to be addressed.

The decisions that are made which result in technical debt should be made with a strategic engineering perspective. Technical debt should only be accrued on purpose because it enables some business goal, intentional and unintentional. Steve McConnell’s talk on Managing Technical Debt [2] does a good job of laying the business and technical aspects of taking on technical debt. Following that, ideally there should be a plan in place on how to reasonably reduce the debt down the road. If technical debt is left unaddressed, at some point the only light at the end of the tunnel is to declare bankruptcy, analogically: just blow it up and start over.

Technical debt is always present, it’s not always bad either but it’s always on the verge of getting worse. It is important to have ways of hammering through it, as well as having preventative measures in place to keep debt to a minimum and manageable for as long as possible.

So how do you deal with it?

Tips for dealing with inherited technical debt:

  • Define it. What counts as technical debt? Why is it important to do something about it?
  • Take inventory, know what you’re working with.
  • Prioritize your payoffs. Pick your technical battles carefully, which bits need addressing NOW and which bits can be addressed at a later date?
  • Develop a plan on what and how you’re going to address and ultimately tidy up the debt.
  • Track technical debt. However you track it, make sure you capture enough detail to identify the problem and why it needs to be fixed.

Preventative tips to avoiding technical debt (as much as you can):

  • Before taking on debt ask yourself…
    • Do we have estimates for the debt and non-debt options?
    • How much will the quick & dirty option cost now? What about the clean options?
    • Why do we believe that it is better to incur the effort later than to incur it now? What is expected to change to make taking on that effort more palatable in the future?
    • Have we considered all the options?
    • Who’s going to own the debt?
  • Define initial requirements in a clear and constant style. A good example of this is Gherkin: https://cucumber.io/docs/reference
  • Create best practices. Some examples:  KISS (Keep It Simple Stupid), DRY (Don’t Repeat Yourself), YAGNI (You Aren’t Gonna Need it)
  • Have a standard, an approved model of taking shortcuts, and stick to it. Remember to also reevaluate that standard periodically, what once was the best way may not always be the best way.
  • Documentation. A personal favorite: the “why-and” approach. If you take a temporary (but necessary) shortcut, make note of it and explain why you did what you did and what needs to be done to address it. Your goal is to avoid having someone look at your code/infrastructure/digital records/etc and asking “why is it like that?” Also for documentation, a phenomenal resource (and community) is Write The Docs (http://www.writethedocs.org/guide
  • Allow for gardening. Just as you would with a real garden you want to tidy up things in your projects sooner rather than later. General maintenance tasks that can be done to improve code/systems/etc now rather than filed on the low priority “to-do” list.
  • TESTS! Write/use automated tests that will catch bugs and issues before your users. I’m a fan of using tools like Travis CI (https://travis-ci.org/), Cucumber (https://cucumber.io/docs), Fiddler (http://www.telerik.com/fiddler) and Nagios (https://www.nagios.org/)  for testing and monitoring. Another resource recommended to me (thanks Andromeda!)  is Obey the Testing Goat (http://www.obeythetestinggoat.com/pages/book.html#toc)
  • Remember to act slower than you think. Essentially, think through how something should be done before actually doing it.

And my final thought, commonly referred to as the boy scout rule, when you move on from a project or team and someone else inherits what you leave behind, do your best to leave it better than when you found it.


Footnote:
  1. Ward Cunningham, Explaing Debt Metaphor [Video] http://wiki.c2.com/?WardExplainsDebtMetaphor
  2. Managing Technical Debt by Steve McConnell (slides) http://2013.icse-conferences.org/documents/publicity/MTD-WS-McConnell-slides.pdf

Extra Reading/Tools:

How to deal with technical debt? by Vlad Alive https://m.vladalive.com/how-to-deal-with-technical-debt-33bc7787ed7c

Obey the Testing Goat by Harry Percival  http://www.obeythetestinggoat.com/pages/book.html#toc

How to write a good bug report? Tips and Tricks http://www.softwaretestinghelp.com/how-to-write-good-bug-report/

Tools & Services list https://www.stickyminds.com/tools-guide

Don’t take the technical debt metaphor too far http://swreflections.blogspot.com/2012/12/dont-take-technical-debt-metaphor-too.html 

Advertisements

The library in 5 years. 

In October 2013, I wrote:

Ideally in 5 years, our library will become less of a traditional library and evolve towards an information hub, providing a greater breadth of resources in electronic formats, accessible from around the world via internet access. Our library, like many others, is electronic preferred and we will see physical stacks condense as patron driven acquisitions drive up electronic purchases.  

Instead of physical books, students will check out tablets with subscriptions to core collections of
electronic books loaded on them. The condensing of these stacks will open up spaces for collaboration and technology labs. We will see more “pings” from internet connections. The librarian’s role will shift to internet communication and virtual reference, allowing patrons to “call” in for help.  

Along with the shift in resource format, top priorities of the library will include library management
focusing primarily on funding for special projects and less focus on physical collection building. We will
also see more focus on providing adequate internet access, technology devices and electronic resources. 

As the library space and resources transform we will see departments unite in a way they have never done before; engineering students will create programs that medical and biology students will use for lab research.  

Our library will be evolving towards acting as an information hub and a space for collaboration, I envision our library working like a shoelace, securing the core pieces of one entity by providing the final knot of support and unity the campus will thrive from; encouraging innovation and multi-disciplinary collaboration. 

We have a little over a year left; how are we doing? 

Designing and Building for Ourselves

Originally written and published on LITA Blog, http://litablog.org/2017/04/designing-for-ourselves/ 


I’m in the throes of designing a new help desk for our department that will serve to triage help tickets for approximately 15,000 employees. This has been a major undertaking, and retaining the confidence that I can get it done has been a major challenge. However, it’s also been a really great exercise in forcing me to be introspective about how I design my own ethics and culture into the system.

When we design and build systems for ourselves, we design for what we need, and if you’re like me, you also aim to design for simplicity and the least work possible that still accomplishes your end goal. When I’m designing for myself, I find that I am more willing to let go of a feature I thought I needed because another one will do the job okay, and okay was enough, especially if it means less work for me.

Designing for ourselves in a way is easier than designing for someone else. You essentially know what you need; there’s no guess work or communication gap. Yes, we can get caught up in semantics about how we may not actually understand what we need, and thus you may build something that doesn’t achieve the end goal you had. But hopefully, in the process, you evolve and learn to design and build what you really need.

Also, designing for ourselves forces us to let go on the complex and unnecessary features and build a more simple product that will hopefully be easier to maintain over time. I do not know a time while working in libraries where we (library folk) were not hooting and hollering about the awfulness of the library technology ecosystem. As I mentioned, I’m in the depths of designing a new service desk for my team (in JIRA Service Desk), and I find myself asking “Do we REALLY need this? Can this complex setup be accomplished through a different, simpler method? Can we maximize the use of this setup and use it in more than just one functional way?” When I have to do all the legwork, I think more carefully about essentials and nice-to-haves than when we hired someone else and I was the “ideas person” – and probably much less flexible on the tedious items.

If the load that I carry and my intimate connection to the build force me to think differently about what we do and don’t need, this suggests that maybe we have the wrong people designing library systems. Or at least maybe we don’t have the right people involved throughout the design and build process. Vendors need to include librarians who work in the trenches in the design process. There needs to be representation from the academic, public, corporate, museum, medical, special, etc. communities,  at a level that is more than just “We’re looking for feedback we might incorporate in the future!”  I don’t yet have an answer to how we can accomplish that, but I have ideas on where to start. Stay tuned for “Why you should leave your library and work for the ‘Dark Side.’”

The flip side to this is that maybe my intimate connection with the workload also encourages me to overlook and take shortcuts that seem fine but really ought to be examined carefully. What comes to mind is a presentation I refer to frequently: Andreas Orphanides’ Code4Lib 2016 talk Architecture is politics: The power and the perils of systems design[1]. Design persuades; system design reflects the designer’s values and the cultural context [Lesson 2 in Andreas’ talk].

Fortunately for me, this came to light while I’m still in the middle of the design process. While not an ideal time because I’ve already done a lot of work, the opportunity to step back, adjust and try again sits in perfect reach. I’ve started reexamining our workflows, frontend and backend; it’s going to take more time, had I thought about the shortcuts I was making sooner and the impact they had on the user experience maybe I’d have less reexamining to do.

When we design for ourselves, how often do we make a compromise on something because it makes the build easier? Does our desire to just get the job done cause us to drop features that might have made the design stronger, but leaving it out meant less work in the end? If someone else was building your design, would you demand that that feature be included – even though it’s difficult to do? Does our intimate connection with the system design encourage us to continue to build in poor values? Can we learn to be more empathetic [2] in our design process when we’re designing for ourselves?

I hope I’ve encouraged you to consider what you may be missing when you design a system for yourself; what habits you’re creating that will be an influence when you design a system for another.
Cheers, Whitni


[1] Slide deck: http://bit.ly/dre_code4lib2016  Video of Talk: https://youtu.be/P03kD_Q5qcU?t=38m36s

[2] Empathy on the Edge http://bit.ly/erl17_empathyontheedge

Never Again.

Write a list of things you would never do. Because it is possible that in the next year, you will do them. —Sarah Kendzior

I, Whitni Watkins, hereby commit to the neveragain.tech pledge [pasted below]. Please stand with me and hold me to it.

Our pledge

We, the undersigned, are employees of tech organizations and companies based in the United States. We are engineers, designers, business executives, and others whose jobs include managing or processing data about people. We are choosing to stand in solidarity with Muslim Americans, immigrants, and all people whose lives and livelihoods are threatened by the incoming administration’s proposed data collection policies. We refuse to build a database of people based on their Constitutionally-protected religious beliefs. We refuse to facilitate mass deportations of people the government believes to be undesirable.

We have educated ourselves on the history of threats like these, and on the roles that technology and technologists played in carrying them out. We see how IBM collaborated to digitize and streamline the Holocaust, contributing to the deaths of six million Jews and millions of others. We recall the internment of Japanese Americans during the Second World War. We recognize that mass deportations precipitated the very atrocity the word genocide was created to describe: the murder of 1.5 million Armenians in Turkey. We acknowledge that genocides are not merely a relic of the distant past—among others, Tutsi Rwandans andBosnian Muslims have been victims in our lifetimes.

Today we stand together to say: not on our watch, and never again.

We commit to the following actions:

  • We refuse to participate in the creation of databases of identifying information for the United States government to target individuals based on race, religion, or national origin.
  • We will advocate within our organizations:
    • to minimize the collection and retention of data that would facilitate ethnic or religious targeting.
    • to scale back existing datasets with unnecessary racial, ethnic, and national origin data.
    • to responsibly destroy high-risk datasets and backups.
    • to implement security and privacy best practices, in particular, for end-to-end encryption to be the default wherever possible.
    • to demand appropriate legal process should the government request that we turn over user data collected by our organization, even in small amounts.
  • If we discover misuse of data that we consider illegal or unethical in our organizations:
    • We will work with our colleagues and leaders to correct it.
    • If we cannot stop these practices, we will exercise our rights and responsibilities to speak out publicly and engage in responsible whistleblowing without endangering users.
    • If we have the authority to do so, we will use all available legal defenses to stop these practices.
    • If we do not have such authority, and our organizations force us to engage in such misuse, we will resign from our positions rather than comply.
  • We will raise awareness and ask critical questions about the responsible and fair use of data and algorithms beyond our organization and our industry.

 

Getting your color on: maybe there’s some truth to the trend

Post originally published on LITA Blog http://litablog.org/2016/05/getting-your-color-on-maybe-theres-some-truth-to-the-trend/


FullSizeRender

Coloring was never my thing, even as a young child, the amount of decision required in coloring was actually stressful to me. Hence my skepticism of this zen adult coloring trend. I purchased a book and selected coloring tools about a year ago, coloring bits and pieces here and there but not really getting it. Until now.

While reading an article about the psychology behind adult coloring, I found this quote to be exceptionally interesting:

The action involves both logic, by which we color forms, and creativity, when mixing and matching colors. This incorporates the areas of the cerebral cortex involved in vision and fine motor skills [coordination necessary to make small, precise movements]. The relaxation that it provides lowers the activity of the amygdala, a basic part of our brain involved in controlling emotion that is affected by stress. -Gloria Martinez Ayala [quoted in Coloring Isn’t Just For Kids. It Can Actually Help Adults Combat Stress]

Color Me Stress Free by Lacy Mucklow and Angela Porter

A page, colored by Whitni Watkins, from Color Me Stress Free by Lacy Mucklow and Angela Porter

As I was coloring this particular piece [pictured to the left] I started seeing the connection the micro process of coloring has to the macro process of managing a library and/or team building. Each coloring piece has individual parts that contribute to forming the outline of full work of art. But it goes deeper than that.

For exampled, how you color and organize the individual parts can determine how beautiful or harmonious the picture can be. You have so many different color options to choose from, to incorporate into your picture, some will work better than others. For example, did you know in color theory, orange and blue is a perfect color combination? According to color theory, harmonious color combinations use any two colors opposite each other on the color wheel.” [7]  But that the combination of orange, blue and yellow is not very harmonious?

Our lack of knowledge is a significant hindrance for creating greatness, knowing your options while coloring is incredibly important. Your color selection will determine what experience one has when viewing the picture. Bland, chaotic or pleasing, each part working together, contributing to the bigger picture. “Observing the effects colors have on each other is the starting point for understanding the relativity of color. The relationship of values, saturations and the warmth or coolness of respective hues can cause noticeable differences in our perception of color.” [6]  Color combinations, that may seem unfitting to you, may actually compliment each other.  

Note that some colors will be used more frequently and have a greater presence in the final product due to the qualities that color holds but remember that even the parts that only have a small presence are crucial to bringing the picture together in the end. 

“Be sure to include those who are usually left out of such acknowledgments, such as the receptionist who handled the flood of calls after a successful public relations effort or the information- technology people who installed the complex software you used.”[2]

There may be other times where you don’t use a certain color as much as it should have and could have been used. The picture ends up fully colored and completed but not nearly as beautiful (harmonious) as it could have been. When in the coloring process, ask yourself often “‘What else do we need to consider here?’ you allow perspectives not yet considered to be put on the table and evaluated.” [2] Constant evaluation of your process will lead to a better final piece.

While coloring I also noticed that I color individual portions in a similar manner. I color triangles and squares by outlining and shading inwards. I color circular shapes in a circular motion and shading outwards. While coloring, we find our way to be the most efficient but contained (within the lines) while simultaneously coordinating well with the other parts. Important to note, that the way you found to be efficient in one area  may not work in another area and you need to adapt and be flexible and willing to try other ways. Imagine coloring a circle the way you color a square or a triangle. You can take as many shortcuts as you want to get the job done faster but you may regret them in the end. Cut carefully. 

Remember while coloring: Be flexible. Be adaptable. Be imperturbable.

You can color how ever you see fit. You can choose which colors you want, the project will get done. You can be sure there will be moments of chaos, there will be moments that lack innovation. Experiment, try new things and the more you color the better you’ll get. However, coloring isn’t for everyone, at that’s okay. 

Now, go back and read again, this time substitute the word color for manage.

Maybe there is something to be said about this trend of the adult coloring book. 


References:
1. Coloring Isn’t Just For Kids. It Can Actually Help Adults Combat Stress http://www.huffingtonpost.com/2014/10/13/coloring-for-stress_n_5975832.html
2. Twelve Ways to Build an Effective Team http://people.rice.edu/uploadedFiles/People/TEAMS/Twelve%20Ways%20to%20Build%20an%20Effective%20Team.pdf
3. COLOURlovers: History Of The Color Wheel http://www.colourlovers.com/blog/2008/05/08/history-of-the-color-wheel
4. Smashing Magazine: Color Theory for Designers, Part 1: The Meaning of Color: https://www.smashingmagazine.com/2010/01/color-theory-for-designers-part-1-the-meaning-of-color/
5. Some Color History http://hyperphysics.phy-astr.gsu.edu/hbase/vision/colhist.html
6. Color Matters: Basic Color Theory http://www.colormatters.com/color-and-design/basic-color-theory
7. lifehacker: Learn the Basics of Color Theory to Know What Looks Good http://lifehacker.com/learn-the-basics-of-color-theory-to-know-what-looks-goo-1608972072
8. lifehacker: Color Psychology Chart http://lifehacker.com/5991303/pick-the-right-color-for-design-or-decorating-with-this-color-psychology-chart
9. Why Flexible and Adaptive Leadership is Essential http://challenge2050.ifas.ufl.edu/wp-content/uploads/2013/10/YuklMashud.2010.AdaptiveLeadership.pdf

Update to the Problem: we have a solution

So a few months ago there was a problem Where I continue to admire the Problem without a complete Solution. This has been solved for a few months, but sh!t happens. Am now finally getting around to providing the a solution, one that works for me, but may not work for you. This is pieced together haphazardly, mainly so I have note of it and well so if you’re looking for help on doing something similar you have a much better starting point than I did.

After a hot minute with Python-LDAP I determined it was a beast I was not interested in taming at the moment because well I had another option, mind you at the time we thought this ‘other’ option was going to be easier. I don’t know if it was or not, but it took some serious neuron firing to do.

At one point I dove deep into VBA scripting, where I figuratively lost loads of hair and age 20 years. The script scraped hundreds of emails for a text string (unique ID), to parse out into an excel file (staying within the suite of Microsoft Office deemed significantly easier than going out) and then ran line by line on LDAP to print the output of the request attributes and then convert to csv and email to colleague to do a mail merge.

**confession** I actually signed up for a Stack Overflow account because of this!

One Solution but not THE solution:

VBScript and a Bash script that ran LDAP Queries, Python conversions and used Mutt to send an email attachment.
I’ve changed values to neutral values, you will need to update them to match what you need.
Here is my repo that provides the files you’ll need if you decide to take the VBScript route. https://github.com/whitni/VBScriptandLDAP

Once you set this up in VBA, you can set it as a macro, but I had it set as a rule at first and then switched to doing something different right after I had this solution in place…so didn’t bother.

The Solution I settled on:

I returned to a single bash script that queried LDAP matching on certain attributes, primary change was the decision to send emails out based on start dates and not when we received an email notification from HR. This will soon be turned into a cron job and I can dust my hands of it BUT walking away with significantly more knowledge of LDAP, AD, VBA and all that…

#!/bin/bash
#A simple script

## date format ##
today=$(date +”%F”)
startdate=$(date +”%Y%m”)

##backup path ##
BAK=”/path/that/you/saved/file/newhires_$today”

# LDAP Search query
ldapsearch -W -h your.AD.server.com -x -D ldapusername@email.com -b “dc=$1,dc=$2,dc=$3” -s sub “attributes you need to match or not match on” attributes you want > $BAK

#convert LDIF to csv
python LDIFtoCSV/LDIFtoCSV.py $BAK > /path/that/you/saved/file/newhires_$today.csv

#send email with file attached
mutt -a /path/that/you/saved/file/newhires_$today.csv -s “New Hire Emails” -c persontoCC@email.com — persontoemail@email.com

Resources used:
VBScript Library: https://msdn.microsoft.com/en-us/library/aa227499(v=vs.60).aspx
RegEx parsing: http://www.slipstick.com/developer/regex-parse-message-text/
LDAP Man page: [Linux_terminal] > man ldap
LDIFtoCSV conversion tool: https://github.com/tachang/ldiftocsv
Stack Overflow — my question: http://stackoverflow.com/questions/36752876/why-does-copying-a-string-from-outlook-to-excel-open-a-new-instance-of-excel-for/36754211#36754211

An Open Letter to the #c4l16 Program Committee

Thank you.

As I’ve begun the processing period of what a Code4Lib conference does to you, there has been but one thing that has remained the same, working with you all has been one of my best conference experiences. All the hours spent in analyzing proposals, drafting up many many potential programs setups, it was worth it.

Thank you for the support and willingness to try something new. I’ve been on a few conference committees and none of them, have made it feel like we could try something new, until now. Ideas were not shot down, they were taken into consideration. As an outsider looking in, this is a place where if I’m going to put in large quantities of energy, it would not be wasted. I’d willingly do it again.

Thank you for providing a space where I felt less intimidated to say something. For a space where, if I did decide to say something, you were listening to understand and not to respond. As someone who felt like they didn’t belong at Code4Lib — you’ve helped me find my place.

We tried something new, we did not fail, and in fact in my eyes we succeeded. The program was strong and broad reaching. This was something we wanted to happen and worried (given our excitement when the panel came together) that maybe we were overreaching, but what matters most is that we were given the power to try – that says something about the Code4Lib community in general. Thank you to those who helped in paving the pathway so we could do this.

I want to apologize for my falling short on delivering the lightning talk we all envisioned, but thank you for making it an opportunity to be delivered. For standing behind me, in person and by live stream, so I wasn’t up there alone. Thank you for the constant encouragement.

It’s been an honor to be a part of this but most importantly, it’s been a pleasure. Hard work is less hard when you’re on a supportive team and a team that carries the burden together. Because of my experience working with you all this year, you’ve secured my buy in to Code4Lib, for this I am grateful. Thank you for restoring my faith that we can actually be the change we want to see in an organization.

Sincerely,
Me, a very lucky volunteer.

PS Thank you Ben for the extra nudge to join the committee & for putting my name on the list and for your constant encouragement to speak up.