Analytic Codecraft: What to Do When You Don’t Know What to Do
Imagine this: you are working on four market analysis reports where your data is being looked at in Excel, maps are produced in a GIS system, the report contents are being written in Word, and the final output is a pdf. The whole process is full of chaos as spreadsheets are updated with multiple versions floating around, many people writing in the same document, and certain calculations are needed to be rerun as mistakes are found, and the whole process is being repeated as each of the four projects progress. You typically do the same six to eight tasks for each project and you need a better way to bring it all together, so you do some research and discover Quarto, a powerful technical publishing system where you can do some analysis with code, generate graphs and tables, and publish reports all in one streamlined process.
The next morning, you rush into your boss’s office ecstatic to share your discovery about how you’ve found the perfect solution all the problems in the company’s publishing workflow. Except, when you’re finished explaining things, she’s both utterly confused and absolutely mortified. Then come questions like:
- Why change our process if were able to do what we do just fine?
- How and when are we going to learn to use something like this?
- How are we going to implement this tool without completely disrupting our work?
- Will this really save time?
And since these things might not have crossed your mind, you can’t give a solid answer.
Fast forward six weeks where you have so eagerly taken upon yourself to learn programming so that you can prove yourself to your boss. And yet, it has become an absolute disaster. You’re not entirely sure what you’re doing, things are taking longer because you’re trying to figure out how to just debug your code let along accomplish what you’ve been tasked with, and you’ve hit a total dead end because you don’t know how to even get your desired output in some cases.
Whether you’re a beginner programmer like I was in the above scenario, an advanced analytic developer working on new and experimental methods in the Tidyverse, or somewhere in between, improving and optimizing your workflow can be difficult. However, these five practices can ensure success along the way.
1. Run mental models
Mental models are critical to understanding your problem, especially complex ones. Often times, your issue is not so unique and while you might be able to see one facet of the problem, there’s something you might be missing. Using an aid like The Unstuck Box can help orient yourself on the problem to get to a better understanding. Two of my most common models that I work with as an analyst are Maslow’s Hammer and Gall’s Law which are crucial to consider when building out workflows and finding solutions. Of course, as I get into the solutions, there are supplemental models that can be applied as well.
2. Don’t get caught up on one specific tool
Maslow’s Hammer is a cognitive bias towards the tools that your are most familiar with. Abraham Maslow wrote in 1966, “If the only tool you have is a hammer, it is tempting to treat everything as if it were a nail.”
Quarto is great. I can run analyses and get reports easily so I run it for my internal operations. However, my clients need often need deliverables as slide decks or specific ad hoc reports, and I need information readily available and usually in a pseudo dashboard format. I could easily leverage quarto presentations and shiny apps but I don’t quite have the knowledge, capacity, or bandwidth to do so. So instead, I used R for running heavy and extensive analyses, excel to do quick calculations and ad hoc reporting, and word and powerpoint for big client deliveries.
Yet, that doesn’t necessarily mean that my workflow is disjointed. When I can, I use packages such as officer and readxl and my also write macros. VBA is undoubtedly one of the most underrated skills in the office.
3. Make sure your system always works at all times
Gall’s law states that a complex system that works is invariably found to have evolved from a simple system that worked. What this means is that the workbook that currently cycles though a dropdown cell and prints each sheet of the report into a single pdf which is stored in the proper directory was once just a simple spreadsheet that I updated manually. Continuous integration and continuous deployment are essential not only for software development but also systems in general. While the process is a little more abstract, CI/CD practices allow for operations and R&D to occur simultaneously.
4. Start with what you know
While I have worked with tidyverse packages, I am nowhere near skilled enough to work with them at an industrial scale. Yet, I have been completely successful in my efforts to improve my processes by mastering things that I have already learned. It seemed a little daunting at first but eventually, I completed an entire workflow in base R. As a result, I had fully working systems in place that allowed me to spend more time on learning new things. Additionally, I am able to grasp more complex topics such as data masking more quickly because I fully understand the fundamentals.
5. Make sure your system always has your desired output
An ship without sails is pretty much useless, and a workflow that can’t give you what you need might as well be a ship without sails. I once came up with a beautiful solution that had an input as a list. Except that’s as a far as I got because I’m not that great at working with purrr or lists for that matter. Then, the project deadline was rapidly approaching and my deliverable was mediocre at best. In keeping practice 3 and 4, you will generally always keep this one. Recently I read about a winner’s game vs. a loser’s game in Sahil Bloom’s Curiosity newsletter. It’s also sometimes known as the concept of unforced errors. Basically, 80% of points scored by pro tennis players are by making incredible shots while 80% of points scored by amateur tennis players are made through unforced errors. You need to know what type of game your playing, if your solution is high-risk, and if you can afford to take it. Once your systems are up and running, then focus on implementing new methods.
System integration is not always an easy task. Project workflows are often complex. Solutions often require learning new tools or methods which take time. However, with these best practices problems can be identified, defined, and understood, systems can be built out with the right tools, and solutions and improvements can be implemented continuously with minimal disruption.
Joseph Mohr is a Catholic, Athlete, Craftsman, Writer, Stoic, and, most importantly, a Floral Enthusiast. He graduated from Texas A&M in 2019 with a B.S. in Human Geography but is more so a polymath studying the intersection of politics, economics, construction, public health, and religion through the lense of demographics. He currently works as a data analyst for a non-profit working to improve education in Texas and runs The Night Watch, his personal consulting office. In his spare time, he enjoys studying art, architecture, cartography, psychology, and language, and is a lover of running, lifting, drawing, camping & hiking, hockey, leatherwork, and Texas Two-Stepping.
