Welcome to Incidentally, Allma’s publication. We interview engineering and reliability leaders, founders, and makers on the secrets and tools they use to scale their systems and teams. We share incident collaboration stories and learnings, in solidarity and with openness. Our goal is actionable for you to implement and high-level to be applicable; your feedback is always sought.
Salsify's Director of Engineering on scaling startups, his time at Drizly, and the future of Salsify
From school to startups
Hi, I’m Tanner Burson. I grew up in Oklahoma and went to college at Oklahoma State (Go Pokes!). I was always interested in the intersection of computers and business, but when I was in school the concept of a software engineering or entrepreneurship degree didn’t exist, at least not in Oklahoma. I bounced back and forth between computer science and business about 4 times but wasn’t happy with either because that meant I was either going to be making databases for oil companies or doing math with computers for the rest of my career. While I was in college, I was lucky to work for the university in their first web development department for a few years before switching to working in traditional IT in a different department on campus running software. Once I left the University, I ran IT and e-commerce for a local business but found that as time went by that I missed writing code. So, I got serious about finding an actual software job. At this point, I had been married for a while, and my wife and I wanted to try living outside of Oklahoma. Fortunately, one of my old friends from college worked in tech in Boston and was able to introduce me to a few people, so I ended up taking my first “startup job” in Boston.
The journey to Tapjoy and scaling the team
Getting to Tapjoy was a very interesting process, as I was initially working at a small startup called Viximo that simultaneously went out of business and got acquired by Tapjoy. There were only 12 engineers who made the leap and we immediately became the Boston branch office. We went from running a very small and often struggling startup to being a remote office for a much larger company that was growing incredibly fast. There were 40 engineers in San Francisco, so we became a small part of a big team. We came in with more engineering experience, but we had no understanding of the underlying system, business, or problems.
When it came to growing teams in both the Boston and San Francisco offices, I learned a ton during this time of rapid expansion.
I led more interviews than I could have imagined, and saw, from the inside, many different organizational and technical iterations. I learned so much balancing these facets of scale while figuring out our place within the wider company at the same time.
We weren’t scaling from zero to something, we were scaling from decently sized to even bigger. We went from roughly 50 engineers to 80 in about our first year at Tapjoy. Along with the team growth, the company’s product was scaling incredibly fast. We had to work around established processes and adapt to the status quo instead of building from the ground up. When you have an existing structure, you have existing people in those structures. It's a lot of context to keep track of and learn while trying to build the product itself.
Building out a reassuring hiring process
As we bulked up our team at Tapjoy, we focused the most on developing our hiring process. We took a deliberate approach, constantly refining the process of hiring instead of just trying to hire a lot of people quickly. Sean Lindsay, who had been the CTO at Viximo and then the CTO at Tapjoy, was a driving force behind the process. We were trying new iterations and wanted to make sure the interview was fair and that people didn’t walk out feeling disheartened. While this is tough, it builds better company culture and a better reputation in the community.
Something Sean always said that I hear myself repeating to my team when we do interviews is that you want the candidate to leave feeling good about the experience, regardless of whether or not they did well. You want them to recommend the company and the people involved to their friends.
If they aren’t the right fit, maybe they know someone who might be the right person, and that's a valuable way to grow your network.
Understanding the structure and process at Drizly
I got to Drizly right after they raised a series A and I was the seventh person on the engineering team. Engineering in particular was tiny at the time and most of the company was in sales. We were just “the engineers”- there were no teams, managers, or structures. While I was there, engineering went from a lowercase “e” to a capital “E” Engineering Team. We got to completely build our department, ending with three distinct teams with dedicated team leads and respective backlogs.
Drizly was an interesting, fun place to work with a very unique set of business challenges. It was challenging in the sense that there was an obvious market; you could explain the base idea to anyone in 30 seconds. But under the hood, it’s a series of complicated problems to tackle. Trying to create a new experience in the market, both in ecommerce and on-demand delivery, took a lot of coordination between product and engineering. In the same case as many early stage startups, there were often challenges getting everyone aligned. There are always the classic technical and scale problems to work on on the product side, but there are many avenues and directions to account for when it comes to e-commerce and the delivery of alcohol, specifically.
Blame it on the alcohol
I’d say in my time at Drizly, we spent half the time figuring out how you do online retail for alcohol and the other half factoring in delivery on top of this. Those two experiences are vastly different. When you think of things that do instant delivery, you think of a Domino’s Pizza, but that doesn’t look like traditional e-commerce; you don’t go in and browse through Domino’s and then decide you’re going to buy a pizza, and you don’t google “pepperoni” and end up on the Domino’s Pizza site. On the other hand, it’s not a completely typical e-commerce experience either. No one knows how people shop for alcohol online because no one was doing it! Drizly was the unique blend of this kind of process, traditional e-commerce, and on-demand delivery in an hour (which was still new to the table as well). We were joining two very new markets and business angles and trying to fit them into a single cohesive product. It could get a bit stressful.
On top of that is all the nuance required in alcohol sales. For example, in Massachusetts the retailer had to do the delivery themselves, and in New York some stores could sell beer and wine, but not liquor. This meant that the system had to know how to handle orders that may require multiple deliveries, but only in some places. Every small change was complex to think about, and this made the problem challenging, and fun. At one point we added some stores in Canada, so then we had an entirely different set of challenges, all the way down to things like “How does Canada show a zip code?”. It worked to our benefit that we started in New York and Massachusetts because they both have very arcane liquor rules. It made expanding to places like California and Florida easier because they are comparatively the Wild West.
Shifting to Salsify
Moving to Salsify was a complete shift from what I was doing at Drizly because it was a late-stage company doing B2B SaaS. I was excited because I went from working on, and being responsible for, all parts of the stack to focusing purely on the infrastructure, so I had a chance to narrow my focus within my work. The biggest problem I had to tackle upon arriving at Salsify was a complete migration from Heroku to Kubernetes. Pretty much everything was in Heroku and the project was already 6 months late. My biggest challenge coming in was trying to complete the move of our core platform from one set of infrastructure to another without much interruption, both on the tech side and the company side. I had to dive into the deep end and learn quickly about how projects, especially massive ones like this, work in a different environment, and with different people. How do you jump in and work with existing goals, plans and people, while there are new decisions to make?
The great migration and the last 10%
To finish the migration to Kubernetes, we broke the problem down further than it had been planned initially, and re-thought the timing of many of the existing goals. We found incremental delivery to be the best path forward so we could ship smaller pieces and celebrate these smaller accomplishments as we went through. It’s easy in a big project to delay celebrating while waiting on the final story to be complete, but odds are it’s not going to be done until much later than you expect or desire. I’ve never seen a big project land on time and within the scope it was planned. This means there is always the chance you never reach the desired state that matches your initial expectation of “done”. You make it 90% of the way and then realize the last 10% won’t get you what you want. If you wait for the constantly changing “last 10%” to have a victory party, you can leave yourself with a lack of a real sense of accomplishment. Infrastructure work suffers from this in a more extreme way than many other engineering disciplines. There are constant interruptions and diversions, and priorities are constantly shifting. We broke things down and declared victory, no matter how small it was, at varying stages, to avoid the “all or nothing” feeling. We got it across the finish line. We have a very small footprint in Heroku today, which wouldn’t have looked like success when we started. Ultimately, we landed all the things that mattered, built a process for tracking what was left, and revisit it periodically to make sure priorities are still in line.
We redefined what success looked like, based on what we learned, so that we could move on to new projects without feeling like this one was permanently lingering over our heads. We celebrated the accomplishment, and moved on to new challenges.
The next phase of Salsify
Looking to the future of Salsify, we are growing the infrastructure and operations group as a whole. Through the majority of my time here, we’ve only had about 4 folks in ops or infra to support 80+ engineers. We’ve managed to do a bunch with that squad. However, we want to continue to grow so we can push towards things like SLOs everywhere, better guidelines for individual services, and high-value metrics everywhere. We also continue to decompose our monolithic platform into pieces, even if they aren’t in different services, to better allow us to maintain and measure the scalability and availability of the different components. It’s great we have the chance to take our past learnings from big projects like the migration and apply it to our overall goals in the future of our architecture and our company.
Going from hybrid to fully remote work
I didn’t know how well we would fare when we had to move from being a hybrid team to a fully remote team, especially for a team that had a very established office culture. If someone had asked me this before, I probably would have said you can’t take an existing hybrid team and make them more effective by going full remote. But I was very wrong. We have people in different countries and time zones all fully remote and we’ve still managed to keep velocity up and keep things going. This isn’t to say it’s been easy or that we didn’t have plenty of work to do, but it’s much more accessible than I would have ever thought.
Smoking meat with Tanner Burson
Coming from Oklahoma, barbeque just means food. It’s the ethnic cuisine of the region, so I’ve eaten amazing barbeque my whole life. Sadly, when I moved to New England, everywhere I went that was apparently “the best barbeque” was disappointing time after time. Eventually, I saw a cheap smoker on sale, and I remember thinking that I couldn’t do worse than some of the stuff I’ve been eating. I bought it and started making chicken and ribs, and it turned out I could do better than most of the places around here without a ton of work and I’ve been smokin’ ever since.
If you’re just starting, pellet smokers are easy and can be found everywhere. They’re computer-controlled and fully automated, so you put your meat in it, set the temperature, and wait- they’re amazing! If you want awesome barbeque right in your backyard, that is the best way to go and you can’t screw it up.
The most I’ve gotten to play games recently has been playing Mario Kart with my four-year-old. She’s gotten to the point where she’s aware that video games are a thing that she can participate in, and it’s great because Mario Kart on the Switch has a toddler mode. It goes back to motion steering so she can sit there and wiggle the controller around, and it continually drives forward and you can’t fall off the track. All she can do is weave back and forth across the track, but we're playing together and that's what counts.