ShipTalk Podcast - DevOps in a Highly Regulated Industry

In this episode of ShipTalk, we are joined by Donovon Carter who is a DevOps Engineer at Dexcom, a medical device company. Dexcom is one of the firms leading the charge against diabetes with CGM, constant glucose monitoring technology. 

Donovon graduated from university with a political science degree, which comes in handy with all of the negotiations that a DevOps organization and engineer have to go through. Supporting the pipeline of innovation and regulatory requirements can be tricky.  Donovon walks through how to balance innovation and regulation. 

With any forward-thinking organization, wanting new technology is a constant.  Donovon also helps balance how you should think about keeping your skills sharp with and without adopting new technology. 

Transcript

Ravi Lachhman  00:06
Hi, everybody! Welcome back to another episode of ShipTalk. Today I'm joined by one of our buddies, Donovon Carter, who's a DevOps engineer at Dexcom. For people who don't know you, Donovon, why don't you tell us a little bit about yourself? 

Donovon Carter  00:20
Happy to be on the podcast today. I’m really excited to do this; this is the first time I've done a podcast in a long time. And by that, I mean, I was on a buddy's podcast before it was big. Stoked to be here. As Ravi mentioned, I work for Dexcom, which is a Continuous Glucose Monitoring company, a CGM in the diabetes space. For those that don't know, my journey to Dexcom is a little atypical. I started as a political science major in college, and then kind of bounced around— help desk job, working my way up to network administrator and then stumbling into DevOps, which was not something that I knew existed when I was in college. So bit of an interesting journey.

When I'm not DevOps-ing, or technology-ing, I play guitar, and sing. I make lots of noise and annoy my neighbors. I'm a dad, [and] I've got a two-year-old son who takes up a significant amount of time— most of which would have gone towards guitar playing, but I love it no matter what. And then jeep-ing as well, cooking barbecue out in the jeep when the weather permits. So I'm happy to be here.

Ravi Lachhman  01:31
That's awesome. Actually, Donovon and I don't live too far from each other. You mentioned barbecue— you have my attention now, Donovon!

Donovon Carter  01:37
Come on over! I just went to the butcher store earlier today and got a Boston Butt and a rack of ribs for hopefully this weekend, if the weather stays nice. We don't we don't get very long springs here in Georgia. For those that don't know, we get like a week of really perfect 75-degree weather, and then the heat cranks way up.

Ravi Lachhman  01:58
Everything turns yellow; it's hard to go outside for like a certain period when the pollen comes. I wish this was a barbecue podcast—  [jokingly] I should change the name [to] BarbecueOps. 

But really interesting journey that you’ve had. We had another guest on before, and a very salient point from your background is that in the DevOps world, decisions can take a long time. So my background— I've been a software engineer for forever. Even with this whole Agile revolution, life is two weeks or three weeks at a time in a sprint. But in the DevOps world, change can take years depending on where you are. Let's talk about your political science background— [well], you're in a unique position to be ready for that type of change. 

Donovon Carter  02:50
Yeah, it's amazing. I tell people from time to time that the part that I love the most about my job is the technological side where I'm doing the code and hands-on keyboard. But what I spend most of my time on is the people side— talking to folks, Zoom-ing, Slack-ing, phone calling, whatever it is to get in touch with folks and to try and get points across and enact change. And especially at a company like Dexcom, or any other big company— there are bigger companies and smaller companies from us, I'm sure— it can be even harder. Even at small companies where you're dealing with developers who probably aren't reading the latest Kelsey Hightower tweets or whatever, you probably don't get a lot of the DevOps-y stuff that we want to do. So yeah, a lot of it is using that interpersonal communication.

We were talking about this the other day, and political science to me, and I think to a lot of other folks in the discipline, is the study of how groups of people make decisions. A lot of people think it's like, oh, Republicans, Democrats— and it's sure there's opinions about stuff like that and political parties— but it's really more of a social science about how groups of people make a decision, which is what companies and software development organizations and developers do all the time, all day every day. But you're right, it can take a really long time to get that wonderful shining consensus, where it's like, “Yes, we've gone through and done the spreadsheet evaluation, the point scoring, and all of the requirements line up, and we're picking this tool.” At which point, half of the room says boo, and the other half says hiss. And then there's like two people who are like, “Yes, my tool got picked!” And then the implementation and all of the gory details start to come out. 

So it's been surprisingly helpful. I really thought that I was going to go to law school and become a lawyer because I'm good at arguing— and I kind of like it. [Then I] realized fairly quickly that I didn't like many of my classmates enough to spend three more years with them in law school, and I just really didn't actually want to do that. I like to solve problems and puzzles, and I got into IT. [I] had always been kind of technologically-savvy anyway and the rest is kind of history, so to speak.

Ravi Lachhman  05:04
Yeah, awesome points. For those who haven't dipped into DevOps, or maybe [are] still very siloed in their engineering shell or cocoon, there's a lot of decisions. That’s actually what took me aback when I started getting into my professional career. I used to think that after my group project and university was over, I'd never had to deal with people again. That was far, far from it— in fact, it’s flipped. Software is a bunch of negotiation. Technology in general, too, is a negotiation of decisions before your time, during your time, and after your time. I really like how you talk about point scoring— a lot of people don't experience that.

Donovon Carter  05:47
Yeah, it really is. And it's funny, because we all have those group projects, and we all have those people in the group projects who just come show up at the last minute, like, “Oh, hey, I wasn't able to get this done. Is there any chance you could bail me out here?” We always talk about, “Oh, yeah, I'm glad I don't have to do group projects ever again.” And then, as you mentioned, you join a software organization and become a DevOps engineer and realize those people still exist. DevOps [are] the people [who are like] “Oh, hey, we forgot about TLS termination. Is that something you guys can just do for us real quick? Also, we need to ship tomorrow.” It's just like being back in college.

Ravi Lachhman  06:25
[Jokingly] Just rotate those certs, man, come on.

Donovon Carter  06:28
[Mimicking] “Yeah, hey, we need an SSL cert for this. And also, it needs to be in production tomorrow, or the VP is going to literally shoot someone.” “Okay, yeah, no big deal. Let me just stop what I'm doing.”

Ravi Lachhman  06:38
A story as old as time! Continues today with different variables— but insert story here. 

So digging in a little bit to where Donovon works. So in full disclosure, I'm a user of a CGM— a constant glucose monitor— and it's a bond that we have here. It’s changed my life, [and] it's given me my life back. Personal thanks to everybody at your firm and the other firms that partake in this industry. It's beating diabetes with data; it's amazing. But being [at] the forefront of such innovation, there are two crossroads. And I think that some of our listeners would find this extremely interesting, because a lot of times DevOps engineers usually at software firms don’t have any regulation. But on the flip side, you are under so much regulation because of the medical device that you represent. 

Let's talk about that. How does innovation happen at a medical device company? And how do you keep the DevOps movement alive and moving forward? I know some of the story, and I'm very impressed, but for the listeners, it’d be very foretelling.

Donovon Carter  07:50
Yeah, no, that's a great question. How does innovation happen at a heavily-regulated company? Well, the answer is: very carefully. The CGM community is really amazing, and I don't want to miss that point. I think that any CGM benefits people just so tremendously. I have an uncle who has type one diabetes and uses a CGM, and it's just been amazing the benefit to the quality of life that that's brought not only for him, but for family members. And just I know a lot of folks in the community who greatly benefit from this technology. We did a Superbowl Ad recently that brought a lot of awareness; I think it's a pretty neat time if there ever were one for this type of technology, and to get to see the benefit of it is really cool. 

But the innovation behind it certainly isn't without a lot of scrutiny, as you can imagine. It's a device that goes inside of people's bodies, breaks the skin— we have a little sensor that goes in there and starts reading your blood glucose values from your bloodstream. Or it's not actually your bloodstream, it's the membrane tissue just below the skin— without getting too gory for people who are squeamish. Sorry about that; I didn't mean to go into that level of detail. I'm not a bio-scientist, so I probably got some of that wrong. So make sure you email Ravi, and correct me; I'm sure he'd love to hear from you about that. 

We’re subject to the FDA obviously, being here in the States, but there's also the various different FDAs that exists internationally. We have a presence in Europe and presences around the world, so we're subject to those regulating bodies as well. We're audited both internally as well as externally. One of the fun things about working at a regulated company like this is the amount of trainings on our processes and procedures that we have to do. It can be full story points at times, days-worth of training, making sure we have our processes and process instructions handy for those dawn raids when the FDA or whatever governing body wants to come and do a spot audit— they can do that. It makes life really interesting for us in DevOps, because so much of DevOps and CI/CD (and the CD part is Continuous Deployment). 

We've seen that Spotify Agile video that the Agile coaches like to show folks of, “We're all off in our little spheres. And we're running fast, and we're deploying squads. We’ve got our squads, and our scrums and our tribes— it’s all wonderful, right?” Then you show up, and it's like, “Okay, it takes four weeks to get something through control. And nothing can get done until it's gone through control. And so, put your CI/CD on pause until we get things sorted out.” So we have a really great QA department and a great regulatory  department who all work together with software development to make sure that we're not only getting stuff out, but [that] it's safe and complies with our governing bodies. 

It's a unique challenge. [Well], I know that there's other heavily-regulated industries, so I shouldn't say it's super unique— but being in the medical field, with software in medical devices as a DevOps engineers is a really interesting space. It can be frustrating for those who want to run super fast. But it doesn't mean you can't embrace automation. I think that's been one of the big takeaways for me—is that just because we're not deploying five times a day, like Spotify or somebody else, doesn't mean that we can't use the same CI/ CD, the same principles that other people use. We just have to proceed with caution and remember to take a lot of care and what we do.

So yeah, it's the same but different. To sum it all up, we have a pretty tremendous burden, and it's kind of unique to our industry. And again, kind of circling back to the original point, it makes sense. I mean, I wouldn’t want to be under any less burden as far as regulation and quality and safety, because what we do impacts people's lives— pretty positively, but if we weren't as careful and cautious, that could be a negative. And that's something that nobody wants.

Ravi Lachhman  12:10
Yeah, absolutely. And I really appreciate all the safeguards and trials that the CGM community goes through. I'm not sure which part of the body [it] sticks into too, if anybody knows; it's on my arm, where it gets the data from. 

I think a very good point you brought up before— [well, I’m cheating] on the podcast, [since] we had a chat before— is that a big part of your job in your organization is supporting iteration where you can. And so, some people will be able to run a little bit faster until some sort of regulation. Maybe we can chat about that. The developers— they need to iterate a lot because you can build anything the first time versus when the audits or the trials come in, you have to slow the roll for more finalized product. Maybe talk about that continuum of speed, the slow-the-roll.

Donovon Carter  13:03
Yeah, the velocity side of things. We are a growing company, and it's a competitive space. There is the impetus to keep the pedal to the metal and continue pushing ahead and developing new things, even as we're waiting to release things that are code-complete. When we do you step back from deploying to production, we have various different staging grounds that we are pretty much continuously deploying multiple times a day easily. So, we still have that innovation and that iterative development. 

One of the benefits of being heavily regulated is we have really great test teams who spend a lot of time continually poking at our stuff. So it's like our continuous deployment, that feedback loop, gets moved just out of production into our testing areas. And so developers are constantly deploying new things, new features, trying new stuff out. We as DevOps are constantly rehearsing things and trying to align. It's really like a train conductor— actually, folks that are air traffic controllers are probably a better example. They get the fourth dimension of things, and it's constantly watching things come and go and get shipped and then get completed. 

So there is that element of iteration that's there for our development teams. It's just we don't do the full Agile, “get feedback from the customer”, because we want to make sure before the customer ever sees it, that we know what their experience is going to be. But we still have that principle of continuous feedback with our testing, whether it's automated or even just our R&D folks manually testing things. It is still a part of our process for sure. 

Ravi Lachhman  14:44
I think I was really excited what you just [said] in the last part there. I think it applies to any heavily regulated industry— that you want to get feedback before the customer gives the feedback. And that's the goal of any regulation, whether it be medical or financial or any sort of regulation. It's there for protecting people, and that’s it. I think [it’s] a very modern model, and I'm very honored to talk to you.  You're having a lot of iteration early on right before production. That was very succinct— like, “hey, we are getting it right, rehearsing it, and then shipping it when we feel that it's right.” Awesome stuff. 

Let’s talk a little bit about some of the trends that you've seen as your DevOps career has evolved. In the last three or four years, [has anything] kind of shocked you, blown you way, [or are you] doing things differently than you’ve done a couple years ago? 

Donovon Carter  15:43
Yeah, [it’s] something I’ve really been working on this year. And it's kind of my theme for last year. I don't know if you guys know this, but last year was kind of crazy and typical for some reason. Maybe unprecedented is the right word, I don't know. But it was an interesting year. And so, my goals for last year personally have carried over to this year— and it's actually to write less code. I'm a big fan of Kelsey Hightower, and I've gotten the chance to meet him a couple of times and to hear him speak. Anytime you can, I think you should; he's just got such a great perspective, and he's such a great communicator both technically and just being really pragmatic. 

The time that I got to meet him at a previous moment was about a technical challenge. And he was just like, “Oh, just do this. Boom.” He picked the simple, most pragmatic things like, “That's all you need to do. Sure you could do a lot of other things. But should you do it?” And I was like, that's a really profound point. Especially within DevOps, there's so many new things being announced and launched, so much shiny stuff, like “should we adopt Istio or LinkerD? Or is there another service mesh that we should be considering.” It’s like, “Well, do we need one?” I'm very guilty of falling into the trap of the really cool [and] shiny. Writing less code, for me, [is] just saying, “Don't repeat yourself.” I'm very bad at repeating myself and trying to get things done quickly. And so, the idea of being more pragmatic with the design decisions that I make, and also being more conscious of how [I can] work more efficiently and build things into little blocks that get reused so I don't have to rewrite those over and over again/ [It] has really helped save me a lot of time actually. 

I like to write Go; Go is my programming language of choice. Being able to just go fetch something that I wrote six months ago, import it, and use it without having to reinvent [or] remember how I wrote it, or rewrite worse copypasta, from Visual Studio Code from one repo to another— it’s just safer. It's a safer pattern to work in because I'm not monkeying with the code over again. And it allows me to work faster. So that's kind of a trend personally, that I've been looking at. But it's also something that I'm seeing be applied across the industry. Terraform Modules— we all love our Terraform, we always have. Terraform Modules and being able to pull those building blocks down has been really cool. Some of the stuff that Google's doing with their Cloud Configuration Manager, making things into YAML that you can templatize and then reuse. 

Reusability feels like the big trend right now, and I totally get it. We're all under enormous pressure to get things done and deliver— reducing the amount of time it takes to do that by repeating safe, well known things just makes a ton of sense. And it's super-efficient. So those are really cool things that I see happening more and more, especially as infrastructure-as-code continues to be the thing— as well, it should. Being able to render those things repeatedly without having to go write stuff over again is kind of the jam.

Ravi Lachhman  19:00
Yeah, perfect. This is really great, because it draws in a lot of different disciplines. That was a very experienced person’s answer. It’s probably taken me since I was a entry level engineer all the way to [when] I was a principal engineer—years gone by— to actually say “let me stop chasing the shiny penny. Simplicity has its virtues.” That’s like a page on the Site Reliability Engineering Handbook: simplicity in solution and the ability to staff it has its virtues. 

A more intrinsic question, because it's something that I think everybody struggles with, and I think you had a pretty good answer there— how do you balance learning and simplicity? So clearly, as an engineer, [on] your career journey, you had to [learn] like a sponge for a long time. You have to be learning initial skills, but when you start kind of getting into a more-experienced level, you're taking a step back and saying, “you know what, let's make things more simplistic, let's use more common design patterns.” So you're not chasing the penny anymore as much. How do you, A, just get to that mindset? At what point is it not [about] absorbing like a sponge? 

Donovon Carter  20:13
Yeah, it's certainly a certain amount of maturity, and it's one that I'm working towards attaining. I still have those moments like, “They did just release a new version of Istio. And I really want to know what they changed.” So maybe 10 to 15% of my time, I try and take as whitespace to go, “okay, I've been wanting to do this for a while. I've gotten maybe a use case in mind; let me go spike it out real quick and just see.” But I try and keep in mind developer self-service is the important thing. 

Also, my teammates need to be able to go and work on it too. We have a very diverse team as far as skillsets and backgrounds, so what makes sense to me isn’t just automatically going to make sense to one of my colleagues and vice versa. It's maybe that posture [that] helps me keep me in check when I do want to run after something super shiny. But the temptation is still there.  

There's also the day-to-day, like, “hey, [I’ve] got to ship things, and [I’ve] got to have stuff done.” But that doesn't mean that doesn't come at the risks— like at the expense of never innovating. I think innovation is really important, so it's very tough. I try and evaluate what's the most simplistic thing that I could do with this particular ask. Because a lot of times, I think we as DevOps tend to be both the cook and the maitre D, who's out there, the waiter that's taking the orders. It's like, “hey, what would you like?” “Well, I didn't look at the menu. But I was really hoping you had this.” “Okay, well, let me go ask the chef, go back change hats. Alright, I think we have all the ingredients to make this; I'm going to grab my Nginx proxy that I know. I'm going to throw that out there, and we're going to build this thing real quick.”

And I think as part of that answer, it's about what tools do you have in your tool bag that you're familiar with, and are they flexible enough to get the most things done? I think that's another part of it for me, too. If you have good tools that can do the work that you need them to do, and you're skilled with them— there's really no shortage of what you can do. I'm not a golfer, but if you've got a pitching wedge that you know you can't miss with— hey, when you get to a certain point, you're good to go. And you know it. 

So yeah, it's a good question. And it's a tough one that I think we as DevOps folks, and especially me, have to wrestle with on a daily basis. That's kind of my spin on it, or at least how I try and approach it.

Ravi Lachhman  22:45
That's probably the most pragmatic answer I've heard, and a very realistic answer as technologists. We wouldn't be in technology if you weren't impressed by the shiny penny, right? We’re not so jaded, like, “oh no, change…”. We embrace change. But also make sure you balance change with consistency and confidence. I like that 10 to 15%. 

It's interesting— when I used to be an engineer and development manager— when I would hire people, I would kind of look for more of a skills gap. So if someone had 100% of the skills, I would be like, “they won't be happy, because you're not going to learn anything,” versus “they have 2% of skills, that might be too much of a stretch.” But I kind of look at that 60%— if they need 40%, or a little bit less of the gap, they're going be excited to learn and take things on. And so your particular metric is great. In the role, you're expanding your skill set without totally burning down the house.

Donovon Carter  23:40
Yeah, a lot of folks would get mad if I started tearing things down, and that's been a good check. It also has to do with the folks and the developers that you work with. I've been fortunate to work with some very, very good development cultures about introducing new things. [It’s] kind of like I was talking about— the whole team has to really get behind it, the whole team has to understand it and be able to own it. It's not just a me thing. So that's kind of has helped change my perspective, or at least helped me be, I think, a better engineer about stuff like that.

Ravi Lachhman  24:13
Awesome. My last question for you— I always end all the podcasts with this. Donovon and I went to competing, rival universities; let's say it's graduation day and you graduated from your university and current Donovon walks past Donovon of years gone "Underneath The Arches". [Chuckling] I think [that’s] what they say there, Arches. What would be any advice you would tell your old self with what you know today?

Donovon Carter  24:45
Man, learn to code sooner would certainly be up there. Also, maybe go sit in and audit some logic classes, some philosophy classes. Spend some more time on that, because as greatest political science is, there's still like a logical and philosophical aspect of what we do that's really important. Some of the stuff that we've been talking about is very philosophical. And that philosophy can really be a framework in which you do your work, just like Agile or SAFe or whatever it is— they’re frameworks. Frameworks are germane to software and engineering, and understanding that stuff is very important. 

Be more active and go to the gym more often, and get into good habits there because those are frameworks that are important as well— especially for those of us who spend a lot of time at a desk or standing at a desk. But yeah, I spent a lot of time doing sysadmin, help desk stuff. [I] was concerned that I wouldn't be good coder or developer because I just didn't want to be the guy in the dark room with headphones on, just staring at a screen [and] pounding on a keyboard all day. Little did I know how much time developers spend in meetings, on Slack, and talking to people— and how little time they spend coding. It's actually been a delightful thing for me. 

So yeah, definitely, get into code. Be less opinionated about your tech choices. And try to think more in frameworks— would be some of the advice that I would give my pass self. Now, unfortunately, I don't know how much of my past self would listen to that advice— especially since I didn't have long hair and a big beard when I was in college, so I probably wouldn't recognize myself. For those that can't see, I have long hair and a big beard.

Ravi Lachhman  26:37
That's hilarious, all very good advice. Actually, I'll take some of that advice right now— the current Ravi. With that, Donovon, thank you so much for being on the podcast. Continue to fight the good fight in the CGM community at Dexcom. 

Donovon Carter  26:51
Thanks so much, Ravi. I loved being on the podcast today. I hope we get to do it again soon, and come over for barbecue!