Ten Years of Rails at Phamily: Upgrades, Action Responders, SLA-Named Queues, and Tests as Theory

Welcome to On Rails, the podcast where we dig into the technical decisions behind building and maintaining production Ruby on Rails apps. And I'm your host, Robby Russell. I run Planet Argon and for over 21 years, we've helped teams maintain and evolve their long-lived Rails apps. So, I tend to approach these conversations through that lens.

In this episode, I'm joined by Bryce Harlan, a senior principal engineer from Jaan Health. Bryce has spent the last decade helping evolve Jaan Health's Ruby on Rails platform which supports healthcare providers and care managers as they coordinate patient care between office visits. Along the way, the team has navigated major Rails upgrades, large scale Sidekiq workloads, and the realities of maintaining a long-lived Rails app in healthcare. Bryce joins us from New York in the United States.

All right, check for belongings, all aboard.

Bryce Harlan, welcome to On Rails.

Hi, thanks for having me.

So, Bryce, to kick things off, what keeps you on Rails?

Yeah, it's a good question and thank you. I think for us, Rails is sort of the sweet spot of the ecosystem is really lively and effective. We've been on it for 10 years, so there's a little bit of inertia there. The things that drew us to Rails in the first place, so things, you know, around making products quickly, you know, iterating quickly, spending a lot less time in the weeds of what makes a web app a web app are still things that, you know, keep us staying on Rails.

So I always find it interesting in our previous conversation, for our listeners, we had like a pre-in conversation, like a couple, maybe a month or two ago. And one of the things that kind of stood out about you was that you didn't work with Rails prior to this job, right? And

No.

So, how long have you been with Jaan Health at this point? And can you tell us a little bit about your background there prior to coming to, like, what tech stacks were you working with?

Yeah, so I joined the company about 10 years ago, actually almost like 10 years to the day. And so before that I worked at a few companies where the main web app components were all written in Flask. There's one that had, there were some services in Node.js, but yeah, hadn't touched Ruby, hadn't touched Rails at all. And my background at the time was really more on like the math side of things. So it's kind of a very classic, you know, math programmer where all the variables are like single letters and things like that. So yeah, I think that coming to this company, getting used to Rails, it was especially enlightening because there were a lot of things that felt a little bit obtuse is a good word for it, I think. But, you know, like just dealing with middleware and dealing with all these layers that, you know, as a math person, I'm like, I just wanted to, you know, talk about what matters. And then Rails came and I was like, "Oh, I can actually focus on the thing that I'm being tasked with building, not all these other details."

You know, when you think about working with, I think it was Node and Django and Flask that you worked with prior, what felt familiar when you started working with Ruby and Ruby on Rails? And what do you think felt completely foreign to you?

I think the things that felt familiar were just the overall architecture, you know. So the MVC pattern, you know, that landed pretty quickly with me. I kind of was easily able to map, you know, I was like, okay, yep, this is this thing, this is that thing. Some of the things that felt maybe less familiar, I mean, one, just Ruby, it was a very different looking language, like at sort of a superficial layer, but I was like, where are all the parentheses? What's up with, you know, these block statements and things like that? And so, but that was a pretty welcome change. Yeah, man, I'd have to really dig back into some of the other things, but oh, you know, a really good one is Active Record. I think the ORM is just something, especially, you know, back in sort of 2014, 2015, getting to use something like Active Record just felt like a complete change of pace. You know, I was used to writing SQL and writing lots of it, and going to making that feel a lot more like code was very different.

What stood out compared with other ORMs you might have interfaced with before? Were you primarily just working in scenarios where it was just a lot of raw SQL, or were there

So I mean, I guess to pull from a slightly more recent example, I recently helped our AI team spin up, you know, a web server around some of their Python code, and that was written using FastAPI and SQLModel. So SQLModel was the ORM layer there, and I think the amount of indirection that you put in between, and that it's really your job as a developer to manage that indirection. So having, you know, these dedicated service layers, you end up almost rewriting what feels like, in Rails, you know, big parts of Active Record just in your codebase, just to do basic stuff like listing, you know, sorting, orderings, chaining scopes, things like that.

How has maybe your appreciation for Active Record changed over time then?

I think it's only grown. I think it's one of the things that we probably use or try to take advantage of the most in our app. Especially for doing, you know, some of the stuff around creating views with pretty complicated filters and things like that. Being able to treat those as just atomic scopes and chaining those together is something that the more I use it, the more I'm kind of rewarded by using it. And yeah, going into other places like that SQLModel layer, it just felt like, "Oh, I'm missing a critical tool in my tool belt."

Right. Right. So, before we get too much further, to help our listeners better understand the world that you're working in in particular, maybe set the stage for everybody listening, can you give the listeners a high-level overview of what Jaan Health does?

We are a web app that helps clinical practices. So think doctors, nurses, they buy our tool and we help them with managing between-visit care. So it's a lot of, you know, health care that happens outside of the four days a year where you go to a doctor, or the one day a year, or the no days a year for a lot of patients. Helping with things like medication, you know, symptom reports, stuff like that.

So who are the kind of primary users of the application? Is it the practitioners themselves, care managers? I'm not sure the terminology you might be using for that.

It's mainly sort of nurses and care managers. Practitioners, physicians, they'll use the platform in more of like an overseeing capacity. In certain cases, you know, smaller clients, they might have a more active role, but I would say nurses and care managers are the primary users.

So, what are these users actually doing inside the application?

Yeah, it's a great question. So, they're putting patients on clinical protocols. So, based on who you are, you'll get your own custom pathway. And that person is kind of acting like a quarterback, almost. So, you know, we work with a lot of older patients, and so older patients, they see lots of different doctors and they have lots of different medications, and there's a lot of coordination that needs to happen. And so having one central point of, you know, "Hey, I'm trying to get something approved by my insurance," or "I'm not sure where to look for the results of my latest blood draw," or something like that. They're someone that can receive those questions, and then also doing a bunch of proactive outreach. A lot of times people are fairly, you know, they want to be independent. They don't think they need help. The health care system usually isn't super friendly to, you know, it's not the most welcoming space. So, they're like, "Oh, if I end up trying to get help, I won't get it anyway, so why even bother?" And so, reaching out, creating that relationship and helping. And so, yeah, in terms of what they do on the application, it's a lot of just sending messages back and forth with the patients, you know, reviewing things like those plans, making updates, sharing documentation back to systems of record.

So, is this a safe assumption? There's a lot of HIPAA compliance involved in this.

Oh, yeah. Everything is fairly sacred. So, yeah, the whole app needs to be fairly locked down.

And we've talked to a number of different people on the podcast so far, different guests that have worked in companies like Doxivity, and I think even, well, Gusto I think has more like PII information in their particular, but that does have healthcare related, I don't know. But regardless of that, had you worked in the healthcare space prior to joining Jaan Health?

No. Yeah. So, like I mentioned, I worked at essentially a 3D graphics startup. And so a lot of my work was, yeah, completely different. So, yeah.

What would you think, you know, for those that have never worked in the health care space, is it as daunting as a place to work in when it comes to thinking about a lot of the data protection there, or is it actually a lot more streamlined and kind of known, like it's easy enough to get ramped up and kind of go through training? I'm assuming you've needed to take training courses occasionally. We've had to do it ourselves when we have different clients that work in that space as well, just so we can provide some consulting with them for, you know, a couple months or something like that. But for those that have never gone into that environment, how would you describe that to them?

Yeah, I wouldn't say daunting. I mean, you certainly are going to have to sit through a lot of those training things. I mean, it happens. I do it every year. I spend basically a weekend just, you know, clicking through these little scenarios and signing paperwork that says I understand and whatnot. I mean, that's fine. It's not a big deal usually. I think in terms of the technical side of things, there's definitely more eyes on your work in that kind of critical lens than maybe you would be used to. And so I kind of see that as a good thing, you know, it gives you more of an explicit opportunity to slow down and really consider the work that you're doing and making sure that it's, you know, secure, which oftentimes if you're at a fast-growing startup, it might not be top of mind. We use a company called Aptable. So, and I'm certainly not the person to ask on some of the platform questions, we have a team for that, but a lot of the nitty-gritty of securing the deployment and things like that are managed through a company that we work with. So,

I see, that makes sense. Circling back to your product itself. So, you mentioned there's a lot of patient communication involved in there. So how much of that interaction happens directly through the Rails app itself? And is it primarily one Rails app, or is there kind of a suite of apps you have there? Where does Rails sit in all that, and through that, the communication layer of that?

Yeah. So specific to the communication, I guess the way that we've architected things is Rails is sort of a central hub API. There's a few spoke services, but for the most part, I would say, I don't know, 80% of the magic happens in the Rails layer. We do have a patient web portal. Most of the communication is happening over text. So, it's probably something like 90/10, you know, in terms of what people are sending, talking about over SMS versus the web portal. The web portal is its own mini Rails app, much lighter weight. And so yeah, for the actual SMS messaging, that's all managed through Twilio, but Rails is kind of the orchestration on those Twilio calls.

Out of curiosity, was that an initial assumption, that SMS would be a primary driver for most of the communication initially, if you can recall back to that?

Yeah, so when I joined the company, we were actually an appointment reminder company. So it was a lot of the kind of texts that you get, you know, I think they're fairly common these days; back in the day, less so. So SMS was a natural, you know, it's like getting a text to a link or asking for an app to say, are you coming to your appointment, just is not that viable. But a very, very early pivot that we made, it was probably six months or something after I joined, was towards this more like care orchestration, care management. We started to notice that a lot of these front desk people were essentially developing relationships that we see and look for today with the patients, just naturally. So it was kind of an emergent behavior from our users, and then we sort of leaned into texts a bit more as a result.

Now I can imagine that product teams or folks from maybe a sales perspective there might be thinking, like, whoa, maybe we need mobile apps for this sort of thing. What if we just give the patients a mobile app? Had that ever come up as part of the direction there?

Yeah, I mean it's something most people ask for, both internally and externally, and it's not something that we're closing the door to by any means, but we kind of saw it as a differentiator. You know, it's one of the reasons to go with us, because we kind of embrace this approach to health care. If you ask our CEO, he'll say, like, meeting the patients where they are. You know, it's like people have a phone, they don't want to download an app, they don't need to download an app unless they want to.

Provide the extra tech support on, like, how do I install, I can't log into my, yes, I'll be there this afternoon, but I can't.

Yeah. I mean, I think maybe half of the phone calls I get from my mom are trying to have me troubleshoot some login to some app that she needs.

Yeah. Out of curiosity, with the SMS integration that you have there, what platforms are you using to handle that with? Is that something like a Twilio or something? And what are you allowed to communicate over SMS given the healthcare regulations there?

Yeah, so it's all done through Twilio. All the communication, really any data transfer that's going to a system outside of ours, has got to be covered by what's called a BAA. And that's a legal agreement that just says that various properties of HIPAA are guaranteed. So things like encryption at rest, things of that nature. In terms of the actual SMS, there's an opt-in system. So we do have to get sort of permission from the patient. If we don't have the permission, you know, we sort of direct them to the web app, and that covers a lot of that.

So now you've been working there and working on the application for about a decade. How long ago did the application start getting developed? Was that just a little bit before you joined? And what version of Rails was it on when you got there?

Yeah, I think it was maybe one or two years before that. It was just the founders at that point. So when I joined it was very early on. I forget the minor version, but we were on Rails 4 at the time, and we stayed on Rails 4 for quite some time.

How many engineers are typically touching the Rails app on, say, a week-to-week basis, or each month?

Yeah, so right now we've got a team of about 15 or so, and I would say pretty much all of them are touching the Rails app. There's maybe a couple that it's less frequent for. So, you know, some of the AI folks touch it less often, but yeah.

Do you recall some of those early, if you had started there and it was on Rails 4, were you involved at all in any of the upgrades since then? And where have you been able to get up to? Are you reasonably up to date these days, if you're willing to share that?

Yeah, we're reasonably up to date. So we are now on Rails 7. We're kind of evaluating Rails 8. It's very early in those stages at the moment, but definitely some interesting things there that we're looking at. Yeah, in terms of the upgrade, I would say, so we went a kind of unconventional route. And I don't know that I would recommend the route that we went, but we went straight from Rails 4 to Rails 6 in the spring of 2020 or so.

Yeah. [laughter]

Like there wasn't enough going on in the world.

Yeah. Well, so funny enough, like for us, part of the decision was actually born out of that, because we had only been in existence for three, four years, or I guess a little bit more than that at that point, but we were still a relatively small company. There's a lot of people at the time who were skeptical, to say the least, about the value of virtual care management and virtual care in a clinical setting. And then overnight, essentially literally, we were swimming in business, and I think that was a big part of our decision to make a big change at once.

We were, I think, and this is probably true of a lot of companies in the early stages, we were a bit of a, you know, Frankenstein of gems. You know, we were just trying to ship features, build, trying to find product market fit really. And as a result, we were looking at our app saying, like, we might only have, you know, one chance to do a bigger change like this. You know, it's very quickly going to become the kind of thing where this turns into a six-month project, versus, you know, just rip the band-aid off, deal with the fallout if there is any. And yeah, so that was the direction we decided to go.

So when it came to the upgrade itself from four to six, and maybe you wouldn't recommend necessarily going that path, do you recall why that was a challenge for you, and what sort of things kept you in that four era for as long as you did? Because 2020, trying to remember what was out then, but I feel like seven was kind of close to coming out around then, give or take.

If I remember correctly, I think six had been out for maybe half a year or something. So six was the latest version at the time that we changed, but yeah, I think probably the blog posts had started coming in on seven, you know. But yeah, in terms of what kept us on for so long, I think a big part of it was, I guess, like I said, you know, we were a very small team at that point. There was not a lot of, I guess, appetite for paying down some of these tech debt things. I'm trying to remember what Rails 5 maybe was. Action Cable was like the headliner, if I remember correctly. There weren't a lot of things that we were eager to jump ship for, and it was kind of just the inertia of, if it isn't broke, don't fix it.

How would you have rated your automated testing situation at that point in time? Was that a blocker at all, or a factor at all in—

Yeah, it was certainly poor, and that's one of the reasons why I wouldn't recommend it. Yeah, at the time, like I mentioned, we were using a lot of gems. We used this gem called Mailboxer to represent, you know, conversation state, but it's really a gem designed to handle email and whatnot. And so a lot of the upgrade was kind of, we almost treated it like starting over. You know, it was like we have one opportunity that we're going to have to rewrite things, and so we gutted a lot of the existing application and just wrote it from the beginning in Rails 6, which made it a little bit smoother.

Looking back, what decisions do you feel like aged surprisingly well?

It's a good question. I think some of the decisions that aged, I mean, certainly just getting on to the latest software, it made conversations around security, things like that, go much smoother. So that was a win in general. I think a lot of what has aged well was owning the fact that, like, I think gems are great. We use gems all the time. Once you reach a certain level of maturity as a product, there can be a lot of, not over-reliance, but like the gem becomes, you know, you start to torture it into these shapes that it was never really designed for, if it becomes too load-bearing. So I think the decision to bite the bullet, take this Mailboxer gem, which, we weren't building an email platform, and just design some proper models, you know, take some inspiration from the parts of it that we liked and ship our own architecture. Because, I mean, it's the same architecture we still use today. So I guess it's aged pretty well.

Do you feel like your team now thinks differently about how and when to bring in an external dependency, like a different gem that's not bundled in Rails? How do you think about picking one or not, versus being influenced by one and then being like, let's kind of own this ourselves?

Yeah, honestly, in the age of AI, it's an interesting question to consider, because I feel like that's almost added a whole other dimension to your decisions. The team definitely thinks differently. I think at the time it was very much trying to compose from well-tested, well-understood, well-documented pieces of code. There's still a preference towards using gems when some combination of: the problem that you're trying to solve is truly a commodity problem. Like, I mean, we're not writing our own auth layer or anything like that. And then maybe the other axis being when the gem itself has a decent amount of, you know, either community activity, it's relatively well maintained. You can kind of, I don't know, I'm a big fan of, one of the best ways to learn how to code really any language, but especially Ruby, is to just poke inside, look around, and get a sense of, is this well-written code, and do I like it?

I can appreciate that. So kind of dive a little deeper into your Rails app, you know. From what I understand, you have a primary, is it kind of like a primary API app, and you mentioned you have kind of a portal app that sits in front of it? Is that correct?

Yeah.

So what's the architecture of that look like? Are you relying a lot on just the default Rails API approach, or is that like JSON APIs?

Yeah. So we treat the Rails app as just an API at this point. And we use, you know, API mode and whatnot. All our front ends are built with Vue. But in terms of the data transfer between the two systems, we do sort of an interesting, it's almost a take on the Rails philosophy of, you know, thin controllers and fat models. We took that almost to its logical extreme, I [laughter] would say, where a lot of our APIs will route through a singular API endpoint, and then we created this thin abstraction layer that we call Action Responders. And those are responsible for handling a lot of the data serialization, kind of giving a shared access pattern whether you're calling an endpoint through an HTTP request or even potentially having, you know, code that calls code, like from a service. You're going to call with the same arguments and whatnot, and then that all goes through and returns kind of a standardized, normalized payload to the front end.

I see.

Do you recall any of the conversations around when you decided that you would not also have much of the front end be in Rails itself? You mentioned there's a Rails portal, but if you have a bunch of Vue front ends, what were some of the decisions? Do you recall what that kind of rationale was at the time?

Yeah. It's a good question. I think a lot of it came down to, we were using a lot of HAML. So pretty much all of our views were server-side rendered. We were on Rails 4, so we started to get some questions about reactivity that we didn't have great answers for, because things like Action Cable and whatnot weren't part of the stack. But ultimately, I mean, in a lot of ways, I think we decided on Vue out of a similar reason to our decision to go with Rails, I would say. I liked the approach that Vue was taking, you know, at the time, this idea of single file components. It really resonated with us in a lot of the ways that, you know, some of these principled philosophies to development do. It felt like they had a really strong perspective on how front end should be written, and so that was something that we were excited about.

Do you feel like if you were starting to build this product now, you would approach that differently? Not saying that it was wrong or right, you know, obviously, but right now, do you feel like the tooling feels more appropriate for that sort of requirement? Because I do remember that era of, well, there's a lot of Rails apps that were primarily APIs. We worked on a bunch of those ourselves at our consulting company, and we're trying to figure out, like, oh, this seemed to be kind of in vogue at the time. Understand that that's kind of how things shift around a little bit. Maybe to start off with that initial question: how would you think about that now with what's available in Rails?

Yeah, it's definitely something that I'm curious about. It's another one of those sort of inertia questions. On the front end, I think especially, you know, code can really start to get much bigger quicker. If I were to start today, I mean, I think I would certainly want to try the latest Rails. I mean, the idea of just having one stack, there's an elegance to that that is appealing. So I haven't played around with some of the latest changes to Rails as much, but I think ultimately that philosophy of, you know, declarative programming on the front end seems to have become just industry standard. Like, nobody's really writing jQuery frontends anymore.

Yeah. So I think I would be interested. I don't know if I would go, I would at least try it out, I guess.

Hey kids, are you ready to get on Rails? Now you can bring the power of Ruby on Rails online straight to your home computer with the Ruby on Rails 1.0 Edition. Just pop in our free CD-ROM and get 40 hours of Ruby on Rails delivered right to your door. That's right, 40 hours. Build forms, ship CRUD, generate scaffolds so fast your family computer gets warm to the touch. Featuring classic Rails 1.0 favorites like script/generate, rake db:migrate, and beautiful server-side rendering the way nature intended. And finally, you can build that family budgeting app and tour chart you've been meaning to get to. So don't wait. Pick up your phone, dial the number on your screen, and get your 40 free hours of Ruby on Rails on CD-ROM. Call now. Not compatible with Adobe GoLive or Microsoft FrontPage. Requires compatible computer, compatible patients, and one working phone line. Some assembly required. CD-ROM does not include security.

So let's go back to the API thing for now. I'm curious, is it a fairly REST approach, typical Rails REST API? I've talked to a couple different people that have done some interesting, curious things, that I don't say is wrong or right, but where they've had some more complex approaches to how they've thought about their APIs. But are there external consumers of the API, or is it all primarily just your team? Like, is anyone building their own apps, like customers that are interfacing with your APIs?

Yeah. So not currently, but it's one of our biggest requests, you know, especially from some of these bigger health systems that have their own IT teams and whatnot, and they're like, "Hey, can you just give us an API key and we'll build what we want to build." We haven't gone that way just because it's a whole other can of worms of, you know, security that we haven't prioritized tackling yet, but I think it's in our future for sure.

Are there any other really interesting things about your application? Like, if someone were to start working at Jaan Health tomorrow, and it was also going to be, say, hypothetically, your last day, and you had 30 minutes to give them a quick download of, hey, it's a pretty typical Rails API backend, but you might want to know about these few little weird things before you don't get to ever talk to me again.

Yeah. So I would hope that our documentation is in a place where some of that would not be necessary, but I think if I were to pick one thing, I sort of alluded to it earlier, this concept of an Action Responder. So it's definitely bespoke, but I do think it's built in the spirit of Rails, I guess. So I would sort of explain it like, hey, this is basically a controller. It's just, you know, a controller that lets you make sure that you don't have to worry about data serialization. You don't have to worry about some of this, like, standardizing your parameter parsing and things like that. All of that is, you know, stuffed into this layer that's called an Action Responder.

And one thing that's really interesting, that plays nicely, I guess, with especially some of these declarative frameworks like Vue or React and whatnot, is that because we're using JSON API, the end user can request sparse fieldsets and associations and whatnot, and kind of out of the box, any endpoint that is built using these Action Responders will respect this interface, where if the user just wants a response with, you know, one field. It's a little bit like GraphQL in a lot of ways.

Curious about— Yeah.

Yeah. Which we did evaluate before we committed. It was a big decision of, like, okay, do we want to just, you know, go with GraphQL? And ultimately, I think the decision that we were making there was that it felt like GraphQL was a lot of overhead, and we didn't need a lot of the features, and at a certain level it just didn't feel like the Railsy way to do it in a lot of senses. [laughter] We were like, we don't really need all these things. We really just want to make it so that we can take some of this logic that's sitting in these controllers and, rather than shipping it off to this litany of, you know, service objects or whatever, put them closer to the model.

How do you organize a lot of your business logic in your application, out of curiosity? You mentioned service objects. Do you lean on that much?

So we certainly use service objects. I think what we noticed before we made the shift towards Action Responders was that our controllers had largely become wrappers around the model, and there were a lot of these service objects living around that were more or less responsible for the same types of things, you know.

Can you give us an example?

Yeah, a lot of it was around parameter parsing, some of it around, like, we use Pundit for authorization, so stuff around authorization was getting repeated all the time, and you can imagine authorization's kind of a big deal in our environment. And so we started to see that it was almost like a necessary evil, I guess, of like, you have to do all those things. A lot of times you do them the same, but there's not an abstraction that pulls out, you know, the generic parts from the parts that are specific to a given model. And so that, I think, is what led us down that path initially.

I'm kind of curious about your Action Responders there and this concept. So it's approaching, kind of in a way, like trying to DRY up maybe some stuff in your controllers in that sense. Do you feel like, if someone's joining the company, that abstraction layer is easy enough to wrap your head around, and there's some consistency there? Do you

Feel like it's like, I'm assuming that was like somebody's clever idea and like, yeah, that sounds great. Let's all do this and we don't have to have a bunch of the same things showing up in different parts of the application. How does it feel like actually living with that day in, regardless of whether or not you think that was like a good idea or not? But like, do you feel like it's been a really helpful, benefited added thing that you feel like it's something missing in Rails, or do you feel like this was something that was part of how your team solved an interesting challenge that you were seeing?

Yeah, it's a good question. I think like missing in Rails might be a bridge too far. I do think a gem that I could see a lot of people potentially being interested in using or something like that, that feels more like in that category. But I mean overall it's a decision that has paid a lot of dividends over the years in terms of, you know, executing, building quickly. It's our own thing, so there's always going to be a little bit of overhead of, you know, teaching people the patterns and whatnot like that and maintaining documentation, but for the most part I think it's really solved a few problems.

One of the most interesting ones, I guess, has been in the testing layer actually. So I mean we still use FactoryBot, we write factories, but there's a lot of cases where we noticed that, you know, when we had a test or a situation like a bug that came up, we looked at the test, all the tests looked fine, and part of the reason that the tests were fine was because we were in this like stub hell of just this hyper-idealized state of things. It didn't really reflect all the messy, maybe side effect logic or things like that that get triggered.

And so we were finding ourselves building these bespoke seed files or fixtures or whatever to try and capture some of that. Once we switched to Action Responders, from a seed file you can call just the same, and it almost marries some of these integration tests, right? Where it's the exact same code that would get run if you sent the request over the network. You can just call it directly from the back end. And there's no difference, it's all the same parameters, they get parsed the same way, any side effects that get triggered. So suddenly it became very easy to create pretty complicated, you know, situations just through a series of, okay, then this was called, this was called, this was called, and now I can kind of guarantee I'm in this state of the world.

Sure. What sort of tooling do you have in place for things like observability or monitoring things? How does your team diagnose and debug issues right now that might be popping up in production?

Yeah, so we use Datadog primarily. That's like our main APM. The host provider that we use, Aptable, they have some tooling as well, but I would say, yeah, most of our diagnostic stuff happens in Datadog.

You know, it's like we're recording this in the middle of May 2026, and so there's a lot of AI tooling out there and we can dig a little bit deeper. I'm curious if you've been able to leverage much of that within interacting with Datadog and the tooling that they're providing at this point to help you debug things quicker or anything like that. What's your experience actually been like on the ground?

Yeah. No, that's a good question. I haven't used much of Datadog's native tooling, I guess, but just the ability to take, I don't know, this happens even just during development, you know, like using some tooling to understand stack traces. It feels like the almost ideal case of what AI can do in terms of, you know, taking something that, like, yes, I can do it, it's tedious, I just manually go through and check line by line by line, but then to have that just instantaneously so that you can focus on that higher-order thinking and not just what code called what code, which ended up in the problem state. So definitely having some of those trace deep dives and things like that.

So an application that's been in development for 11, 12 years, and you've been there for over a decade or so. Every application tends to eventually develop a few scars, maybe, to say the least. So curious if you could talk a little bit about, say, the operational reality a bit. When we had a previous conversation, I think there were some topics that came up related to background processing, kind of, you've had some issues over the years. Tell us more.

This is a problem that actually came up kind of recently, but it's been, you know, especially as we've grown, we've seen it, we've been kind of monitoring it for a while. We use Sidekiq for all our job processing. We have a pretty bursty peak of usage, I would say. So our app sends, or people through our app will send, on the order of like 100,000 Twilio messages a day, and those are all managed through our background, you know, those are all on async jobs so that they can do the proper retriability and things like that.

And so the thing that had been coming up recently, the job queues themselves were relatively healthy, with some caveats on our SLA. We gave ourselves a fairly generous SLA, I guess, but we weren't exceeding our SLA. But what we noticed with some of these really big bursty use cases was that the end-user experience of the queue was very different because of the way that we were enqueuing our jobs. What was happening was that one person would log in, they would maybe queue off a set of messages to, call it like 500 patients or something like that, and then the next person would log in and they would queue up their 500 patients. And so the first person, they would see those 500 messages and they'd start getting replies earlier. But while the queue is being processed, and the real numbers are quite a bit bigger than 500, you know, that second person is basically waiting for the entirety of the first person's process to finish.

And so there were a few things that we did, but I think the biggest change was we changed the way we were treating those batches. So rather than having a batch that spawns a bunch of underlying jobs and you have to wait for the batch to get picked up, we immediately moved the batch to all be on the same queue to have the batch enqueued quickly. And then the next thing that we did was we sort of broke up the jobs so that the first hundred of a given batch, regardless of how big the batch was, we put that on a very fast queue so that everyone will sort of get the feeling of responsivity, even though, if you're trying to send 10,000 messages, the 9,000th message might not send for an hour and it's waiting in line, but you're not sitting there waiting because someone else tried to send 10,000 ahead of you.

That's interesting. I mean, is this just a matter of you just needed to have more parallel queues type of thing? How do you think about these things, and does Twilio have, are you bumping up against any constraints that Twilio might have for you? And do they treat those separate for each? I'm assuming, like, if one, I'm just trying to wrap my head around this, but as far as you have one doctor's office that has some people working there and there's another doctor's office, are they all going through Twilio through one account, through your organization, or do they have their own individual things? So could you in theory, do they have any caps or anything like that that also come into play into this?

So, Twilio does have sort of a multi-tenant setup where, so we have a dedicated, you know, each account, each doctor's office will have their own dedicated Twilio instance. I don't know how they do it on the back end, but they call them sub-accounts basically. So in terms of rate limiting and stuff, usually we haven't had many issues because I think the rate limits are really at the, each sub-account has its own rate limit. In terms of how we start to bump up against this, the splitting on our side, I guess maybe this isn't the direction you were imagining, but are you kind of asking about almost like our queue topology?

Well, just kind of a little bit of that, and I think just a matter of, it's an interesting thing if you've got several people at approximately the same time trying to queue up things and then trying to make sure their experience is, like, they're not waiting because someone else just triggered a similar thing. Do you feel like you found a good happy path with that at this point? Or what sorts of things are you still navigating as far as, is there a better way to orchestrate that sort of thing so that it does get through them quicker? But I guess there's also a challenge, maybe from a user perspective, if a nurse practitioner, whoever in the office, is sending out a blast of 500 SMSs, and they're probably maybe not ready to start getting 500 responses right away either.

Yeah. No, exactly. The last point you brought up was actually a big part of our decision, because we had sort of kicked the can of just increasing the number of workers, increasing the capacity of those workers for a while, and then we did reach a point where we started to ask ourselves, is this even the right problem to solve? Are we not thinking about it from exactly that lens of, maybe it's actually better to have these kind of naturally spaced out and not make the user really think about, you know, like, I know I want to message these people today. I don't necessarily need to message them at 10:06 a.m.

And so in terms of our happy path, I think there's a really great article, a blog post by Judoscale about queue naming, and it totally changed our philosophy, because we had been doing sort of just an ad hoc, like we had an import queue and a real-time queue and a low queue and a default queue, with weights that had just been sort of tuned in this ad hoc way. I'm sure probably a familiar experience for a lot of people. But this post talked about naming your queues with the SLA. So now our queues are all, we have a within-5-seconds queue, we have a within-five-minutes queue, we have a within-30-minutes queue, within an hour.

And so it's just really great because you can set up your Datadog to trigger an alert of, oh, your queue SLA is not being respected anymore, and that triggers you as a developer to go investigate, like, oh, are we putting too many jobs on the queue? Do we need to increase the processing power, or is it that there's some job that really doesn't belong on that queue we could shift down a layer? And as a developer, it's great because, I mean, I get so many fewer questions being like, should I put this on the import queue or the default queue, because the question almost answers itself. It's like, well, how fast do you need it to be done? What's the user expecting?

Yeah, I just pulled up this article right now. So for our listeners, it's like isolated. The examples are like 5-second worker, five-minute worker, five-hour worker, and then kind of some scaling config options there as well. No, that's helpful. I'll try to remember to include a link to this article from Judoscale in the show notes for listeners as well. So you're using Sidekiq there, and how often are you needing to debug anything that's happening in the processing?

I wouldn't say that often at this point. It certainly was when there were some early growing pains, I guess, where we just hadn't really given, we never needed to, you know, we had a small enough volume of jobs, so queues, we didn't think about it very much. And especially since making this change, it's been much more straightforward.

How would you say Rails is part of Jaan Health's secret sauce, so to speak?

Yeah, I mean to me, I'm sort of biased. If you ask other people at our company, they might think their job is more important, but because it's my job, I think it's really a big part of our secret sauce. Especially, I mean, a big part of our vision is to not just use AI tooling for our own purposes, but also to start to use it to help assist care managers and whatnot in their daily workflows. And I think that Rails has provided a lot of just a framework that we can rely on and not ask questions that are getting in the way. It's a space that's moving super quick. If you work at an AI company, you always feel like, oh no, someone's gonna vibe code my app in a week or whatever. There's always this sort of background pressure. So knowing that we're not spending our time thinking about how to build something technically excellent and worrying about all the security things, and that we can just take a lot of that for granted and focus on the problem in front of us, the thing that actually makes our company, our app, our product unique, is to me like the whole secret sauce almost.

That's great. How often is your team deploying updates to production?

Yeah. So we do nightly builds, but that doesn't always, we push to sort of release branches, you know, and so we don't always merge everything that's ready nightly. But yeah, we use GitHub Actions to manage a lot of the deployment infrastructure, but I would say probably it depends on what features are the most critical, but usually on the order of, we're shipping stuff probably twice a week.

How does your team approach developer onboarding, and how do you navigate trying to have realistic data to debug and test things in a local developer environment or staging QA environments and stuff like that?

Yeah, I mean it's definitely one of the biggest challenges. So we have a dedicated staging app that is basically a faithful recreation of the true prod environment that's hosted in all of our most secure infrastructure. It has all the protection and whatnot, but it's more of a break-in-case-of-emergency kind of place to go. So in terms of getting newer folks onboarded more quickly, I think that we rely a lot on some of these seeds. I think maybe that's also part of why some of the seeding stuff is so critical for us, because a lot of these relationships are long-lived. It's not very helpful to just get a patient with no data in them. And so it's definitely an area that we're working on improving, I would say, at this point. But for the most part, when we're testing, the last little bit of testing, if it's something where we expect that there's at least some risk of failure just from the fact of the true production data, we'll do our last round of testing and CI on that pre-prod environment.

I see. What does a testing process look like if, not saying primarily, but a large part of it is sending out a lot of SMS messages and feeling pretty reliable? How do you verify that things are working as smoothly, knowing that the jobs are getting closed off? How do you test that before you ship things to production, because you're not able to just push? Are you relying on a lot of VCRs and

We rely on a lot of stubs and mocks, I guess, of certain endpoints outside of our system. We do have a few, so

It's a really great feature of Twilio, but they have kind of dedicated testing phone numbers. So we can hook up things to a sandbox account that has these dedicated testing numbers where, you know, if you text this number it will say like, oh, I'm a landline or something like that, or it'll give you the actual response. So for that layer we sort of centralize our tests into, you know, we have one part that tests all that.

And then we have some integrations. We use Playwright for our full end-to-end integration tests, where usually we'll only run our Playwright against a sort of mocked environment where those external connections are just stubbed out. Occasionally we sort of have it as an environment variable flag to instead run it against an environment that's deployed pointing at that Twilio sandbox.

I see, I see, that makes a lot of sense. So another thing I wanted to touch on quickly with you is, without turning this into an AI episode, I'm trying to not do that necessarily, but a lot of teams are, and we touched on this earlier, trying to figure out how this fits into their day-to-day logistics. So what sort of experiments or conversations is your team having right now around, say, AI-assisted development?

Yeah, it's a big thing we've been working on a lot recently. I think we're still, maybe we're just starting to graduate out of this zone of people doing whatever they want, you know, some people doing things one way, other people trying to do them another way.

I think the part that has really started to solidify at our org is this idea that, it's a favorite phrase of our founder, that our job is to give people superpowers. And normally we had thought about that in terms of the products that we build, but now it's starting to happen in terms of how can we supercharge developers.

And the use case that I guess we've landed on is trying to take a lot of these features that are maybe smaller, the kinds of things that often sit in the backlog for a while. They're a good idea, there's maybe not a strong advocate for them. Giving developers the power to take something that is maybe just a vague idea or a loose set of requirements and turn that into a feature in a way that our product managers would sign off on, with all of the different things that were involved.

And so we've got a set of, we use Claude Code mainly, but a set of skills that go through a pretty rigid pipeline at the moment, where the first step it will do is generate a PRD. You give it a description of what you're trying to do, or if there's a ticket for something, you can copy-paste the ticket, and then it gives you a PRD that you can take to a product manager and be like, "Hey, I think I want to build this. Does this look like the right set of requirements?" or whatever.

And then it goes through this pipeline where it builds an FRD and then a sort of technical document that is an implementation plan. And at each step along the way, our expectation of our engineers is like, "Hey, you guys are smart. Take a read through it, refine it, try and do the same things that your PMs would do, that anybody would do looking at this."

And then it ends up at a test suite. So that's one critical component of how we're choosing to approach this. For small things, if it's a small bug, sure, all the rigor isn't necessarily required. But for these features that are maybe three, four days' worth of effort, we try to make sure that the developer is really signing off on this: these tests make sense, these tests are the correct way to prove this feature, before you actually have the LLM write any of the other code.

Yeah, it's been very powerful in terms of that "give engineers superpowers." We've seen engineers create pretty complicated features, things that touch the front end and the back end, maybe involve a lot of complex requirements, just on their own, with very little involvement from a lot of the other players that you would usually have involved.

What I find interesting about that is I've had a lot of conversations. For some context, I work on the software consulting side, so companies are coming to us, and we're talking with a number of different companies right now where they're trying to move more of that type of work a little bit more upstream, giving their product owners the ability to do a little bit more experimenting within the codebase in their own little sandbox version of it. That way the engineers can focus on the things they need to be focused on, like, oh yeah, you can move down your path. It sounds like maybe there the engineers are taking a little bit more of that initiative and taking on more of that. I don't know, does that speak to the size of your company or just the culture there?

I think it's a little bit of both. I think in this past month almost all of our PMs officially have GitHub access and have their own environments set up. So we're definitely approaching it from both sides.

I think what we have strategically decided is that the more we can have the PMs focus on some of these bigger-picture things, almost feature-level or epic-level work, that's where their time is going to make the biggest difference. That's where you really need to take in a wealth of information from the outside world, from customers, in our case a lot of regulations, things like that. We wanted to put most of their time on that, rather than, hey, there's this UX flow that's sort of clunky and maybe there's a better way to do it, or there's this page that loads kind of slow.

Oftentimes what we found is that work is a natural fit for both, right? It's stuff that the engineers were bugged by the most, and they were often saying, why can't I ship a fix to this thing that I know is broken? And the PMs were saying, why am I spending all this time working on these things that just make the system a little bit better?

So kind of allowing both of those sides to take on a little bit more ownership and turn things around a little bit quicker and potentially experiment quicker. You mentioned that you're using Vue for a bunch of your front end. Are those in separate repositories? How is that all being handled? Are you using a monorepo type of approach, or is that something you're currently re-evaluating at all given LLM usage?

Yeah, so right now they're split out into two different repositories. We do have basically a VS Code workspace, so that if you're running Claude Code or something, you're able to do that full-stack development in one pass. If you're writing something that touches both front end and back end, you don't have to...

All right, so if you're working through a commit and it needs to touch two different repositories, or maybe three, to ship out a new feature, it can orchestrate that pretty seamlessly. I feel like four or five months ago that was more of a challenge for teams.

Yeah. No, and it's funny, all those skills, we have kind of a wrapper, top-level repo that's really just a registry of all the underlying repos, and that's where all those skills live. Anyway.

I was talking with Brian Scanlan from Intercom, now known as Fin. I think they just announced it yesterday, I'm just trying to remember to say that. They were talking about how they've added a lot of observability and monitoring of how those skills are being used. Have you explored anything like that as of yet?

It's possible that someone other than me has, but I would say it's still in the early days of development. I think just in the last couple of months we really handed it over to some of the engineers. For a while we had a few engineers trial it, give feedback and whatnot.

Can you think of a recent thing that you've done with Claude Code where you're just like, wow, this works really well within the Rails workflow?

Yeah. One thing that I worked on pretty recently, because I wouldn't say I was an AI skeptic, but I certainly wasn't an AI booster for all. There were a lot of times where I was a little bit like, it's good for scripts, it's good for SQL queries, but I don't want it to write my application code. That's my baby, you know. [laughter]

And something that I worked on recently is a fairly complicated business-process model. So think Zapier, Flowise, some of these, or Lucidchart, just a way to build flowcharts. It's a big part of healthcare, right? You have these decision points, and you need to be able to generically represent these things.

There was a long evaluation period, and at a certain point I kept thinking, well, I can't build this, I don't want to build this, other people must have built this better than me. And then I wasn't finding the answers I was looking for, and as a last-ditch effort I was like, okay, let me just sit down and really describe exactly what I need, exactly what I do and don't like about all these different options I've looked at, and see, like, "Hey, Claude, let's try and build something on this."

And it was a pretty impressive result. [laughter] The models were pretty clean. It certainly understood Rails patterns, the idiomatic way to write Rails, which I think for a long time was one of the things that I felt was lacking a little bit. So yeah, it's something that I shipped much faster than I think I would have shipped otherwise.

Where do you find that boundary right now? Do you feel like someone that's not a developer would have been able to get there on their own without you being part of that workflow? I'm just trying to speak to, are you worried about your role as a software developer in the community because the tooling is getting better? Or do you feel like because you have that domain expertise, as you're evaluating different options you can tell it can't quite solve these types of problems, at least as of yet?

Yeah, it's definitely an existential question right now. But the way I see it, there's this essay I really love called "Programming as Theory Building." I don't know if you're familiar. It's by Peter Naur. I think it's from the 80s or something, maybe the early 90s. It's this old idea, and it's talking all about why you can't just throw more programmers at a problem to ship it faster.

And the argument that he makes is that the job of the programmer is not to produce code. Source code is not the output of a programmer. It's the theory of the source code. So this idea that your job is to build a theory that is a good theory for handling the kinds of problems, and the code itself is just a consequence, it's just an implementation of that theory.

And for me, as the production of code becomes even more of a commodity, it's just giving you the tools to really do what you were always supposed to be doing anyway, which is crafting that theory, thinking about what that actually entails, what are the interfaces, how can you build this in a way that makes it effective and usable, maintainable over time. And the more that you can give away the actual writing of the code... It's a big part of why we're so in favor of using tests, because the test suite almost becomes the theory of the code. It's really, this is what we're describing.

Yeah, that resonates. I'll look up that article and see if I can include it for our listeners in the show notes. I'm kind of curious, do you know the backstory on why Rails was chosen in the first place there?

Only a little bit, I would say. Our founder had previously worked at some Y Combinator places, and he was someone who was very much in the startup scene around the time that Rails came out and became popular. So I think for him, and just knowing him for 10 years, he really embraces the philosophy that goes into it.

I see. Are they a developer as well originally?

Yeah.

Okay. So it's one of the tooling choices they made at the time, and I know a lot of Y Combinator companies are still choosing to use Rails, so that's great. All right, Bryce, I feel like I've kept you long enough. I'm curious about some advice for our listeners. As you reflect on choices and decisions that your team has made, what's one technical or team decision that you think our listeners would probably benefit from also considering?

Yeah, it's a tough question because it really depends on what your demands are. I don't know if it's a technical one, but one that comes to mind is: you never need as much as you initially think. That's one lesson I've learned a number of times. It's very easy to talk yourself into certain use cases. Programmers are inclined to think about edge cases, inclined to overbuild in a lot of cases. But just trust that what you actually need will emerge from what you ship. And if you have that faith, it will come. [laughter]

That resonates and speaks back to the topic earlier around how we evaluate gem dependencies and things we're going to pull into our app. You mentioned table-stakes things like maybe authorization, where maybe we can rely on the community to provide support for this thing. But with something like that business modeling thing you did, maybe there are things you looked at and evaluated and said, well, we have a very unique thing, let's not try to adapt our business model to fit a generic problem pattern that the community's been sharing. So I think that's helpful. Out of curiosity, is there a software book that you've found yourself recommending to peers lately?

Not as much. I came from the math background, so I wasn't as steeped in the books. I will say, one thing that's maybe a hotter take, especially these days, is that one of the best ways to learn programming is to really sit down and force yourself to learn C at least once in your life. I think it really helps you. Especially in the age of AI, I see it as this layering of higher

order thinking of like going from assembly to managing memory to, you know, these more expressive programming languages to now maybe you're not really thinking as much about even the programming language layer. And I mean, maybe I've never written assembly. Maybe I would feel that way about assembly, but at least going down to the level of C gets you really familiar with like how a computer thinks. And that can be really, really helpful for building out your intuition about debugging, about systems, about what works and what doesn't work.

I think that's some reasonable advice for our listeners there. And maybe with AI, maybe you can do that a little bit quicker now. I don't know, but this may be worth an experiment there. Does Jaan Health have like an engineering blog or anything like that we can direct people to?

No. I think like we post some stuff on LinkedIn every once in a while. So, yeah, people can find us there. Or phamily.com is our sort of marketing website.

That's family with a PH, right?

Yes. Family with a PH. Not the best name for a podcast, I suppose.

Well, we'll include links for that in the show notes for all our listeners. And with that, Bryce, thank you so much for stopping by to talk shop with us on On Rails.

Yeah, thank you so much. I appreciate it.

That's it for this episode of On Rails. This podcast is produced by the Rails Foundation with support from its core and contributing members. If you enjoyed the ride, leave a quick review on Apple Podcasts, Spotify, or YouTube. It helps more folks find the show. Again, I'm Robby Russell. Thanks for riding along. See you next time.