Thirteen Years of Rails at the Auto Shop: How Shop-Ware Maintains a Mature Monolith With a Smaller Team

Open on YouTube ↗
Overview

Shop-Ware makes software for independent automotive service shops. Its codebase started as a Rails app in 2013 and is still more than 90% Rails. On this episode of On Rails, host Robby Russell of Planet Argon talks with Nikky Southerland, lead software engineer at Shop-Ware, which is part of the parent company Vehlo. The conversation asks what it takes to keep a large, decade-old Rails application healthy after the engineering team has shrunk to roughly half its former size.

33 min read

Southerland credits Rails' conventions for much of what keeps the system manageable. The interview also covers the harder parts: a front-end migration still in progress, an API architecture the team regrets, a move from biweekly to daily releases, cautious experiments with LLM tools, Postgres scaling lessons, and an upcoming move off Heroku.

From Perl Forums to Rails

Southerland has worked in Rails for 12 to 13 years and describes the attachment as "pure love." Every time Southerland drifts to other tools, they end up coming back, because nothing else gives "the warm embrace" of a framework that feels cared for and fits the work.

Southerland started in the 2000s on Perl forum software: a CGI-bin application that "probably like three people used." It stored its data in a flat text file rather than a real database. After that came a lot of WordPress and PHP work.

Russell asked whether Perl and Ruby felt similar. Southerland said not really. The biggest difference at the time was that Rails had an actual ORM, while the old Perl forum had no database at all. Southerland still defended Perl, saying it has a bad reputation that it doesn't deserve. In Southerland's view, it is friendly for what it tries to do and was designed with a lot of thought and intent.

What Shop-Ware Does

Southerland calls Shop-Ware one of those categories of software most people never think about. A customer brings a car to a shop. A service advisor inspects it and tells the customer what the repair will cost ("always more than you'd probably like"). The customer approves the estimate, and the job goes to a technician, who does the work and logs labor hours. Shop-Ware tracks that whole job and serves as an all-in-one platform for running the shop.

The platform is mainly for internal operations, but it can also communicate with vehicle owners. Shops can send customers an inspection, which usually includes many photos taken during the inspection, and the platform supports some texting. It does not handle marketing or public websites. Southerland said other products within the parent company cover that. Shop-Ware handles payments, including down payments on large jobs, and estimate approvals. Southerland explained why approvals matter: on a large job, the owner wants to approve the spending before thousands of dollars of parts go into the vehicle. The product is SaaS.

Rails From the First Line of Code

Southerland has been at Shop-Ware for about four years. Southerland understands that the very first line of code was Rails, probably Rails 3, and says lines from that era still survive in the codebase. The company also has some platform services and AWS Lambdas, as most companies of that era do, but Southerland estimates the codebase is more than 90% Rails. It totals "a couple hundred thousand lines of code." About 10 to 12 engineers maintain it, including some contractors.

Southerland describes Rails' opinions as the platform's "secret sauce." If you trust the people behind the framework, you don't have to bikeshed every decision or research and second-guess the choices they already made. Southerland sees another benefit: when you want to do something that strongly goes against the framework, Rails lets you, but it also makes you stop and ask whether the idea is sound. The team uses that as a built-in check. If an approach will take four times as long because it isn't the Rails way, the team asks whether it is worth doing or whether the feature should be designed differently.

Russell asked whether the team starts from "how would Rails do this" or from "what's the best way" and then adapts to Rails. Southerland said it's both. The engineers are "very steeped in the Rails way," and people are shaped by what they have experience with. So when the team designs new architecture, the Rails approach is always in their minds, even if only subconsciously, because they know it works well with the system.

Active Record, MVC, and a Team of Full-Stack Engineers

Southerland named Active Record as the biggest single benefit, as it is for most Rails teams. It gives a well-structured way to work with the data layer, and new developers, including those without Rails experience, pick it up quickly. For a team that rarely hires, that matters a lot.

The model-view-controller separation also matters. Southerland has seen cluttered applications with no clear boundaries between layers. Rails, by contrast, draws a "really bright line," and if you take care to enforce it, the code stays well organized. As a codebase grows, that clarity also shows who the expert is on each part, who maintains it, and where application logic belongs.

Everyone on the team, including Southerland, is a full-stack engineer, and Southerland believes that is the right model. People lean toward the back end or front end, especially as they become more senior. But because of how the application is organized, an engineer can do application-layer work one day and HTML or CSS the next, depending on what the business needs.

Infrastructure is handled by the same team rather than a separate one. Southerland said dedicated infrastructure staff are sometimes better, but the current arrangement works well for Shop-Ware. The senior and senior-plus engineers who have been around a long time understand how servers and application layers behave, so the team monitors, deploys, and hosts the application itself. Southerland also thinks keeping engineers close to production is valuable. They see how a code change behaves in production, they spot performance problems quickly, and they stay aware that careless work can cause downtime later. Southerland called that "the ultimate like accountability tool."

Observability: New Relic and Sentry

Asked about monitoring, Southerland recalled a Rails class around 2013. An instructor installed the New Relic gem and application keys on what was probably Rails 3.2. The students, most of them from PHP and WordPress, "were all just like stunned." They could see application performance data broken down by individual controllers, along with the actual queries and their EXPLAIN plans, and it felt "like magic." Southerland says New Relic has since grown into a much bigger product, but the core is still there. Tools like it keep the team closely connected to how the code behaves in production.

Error reporting also still feels "a bit magical" to Southerland, and Sentry is the tool Southerland has used most. The team sees errors almost in real time. Sometimes they can fix a problem before support hears about it, or they can warn support that complaints are coming in about ten minutes and that the fix has already shipped. According to Southerland, that gives the team the confidence to ship faster and more reliably.

Seeds, Onboarding, and Local Stress Testing

Southerland says the team's database seed strategy "has grown organically over the past decade." Southerland considers a reliable seed file important because a good way to start on a new bug is to wipe the database and rebuild the dev environment from scratch, so everything is known to be clean. The team has specs that verify the seed file runs cleanly. For example, if someone drops a table and forgets to remove it from the seeds, the spec fails and the change can't be merged.

Onboarding is harder. Because the team is small, it doesn't onboard often, and "every time we do, something new and unexpected and fun comes up": a new service, a missing key, a database step that doesn't quite work. The onboarding docs have also grown organically, with each new developer adding to them. At the time of recording, Southerland was going through the process personally after breaking a laptop and having to set up a new dev environment. Southerland said it has been useful to experience the process from scratch and confirm that every step works.

Everyday seeds don't produce production-scale data, so the team has written many stress-testing scripts. Southerland credits Ruby and Rails for making these easy to write. The scripts quickly insert hundreds of thousands of associated records. At that scale, Postgres behaves closer to production: instead of relying on sequential scans, it actually uses indexes. Southerland also praised the QA team for providing good reproduction steps when bugs come in. When asked about more formal processes, Southerland answered plainly: "there's no great processes in place."

The Long Migration From Backbone to React

Southerland said any Rails application started in the past decade faces the same front-end problem: the original framework choice was a best guess about what would survive. Shop-Ware chose Backbone. Southerland still thinks Backbone is "quite cool," especially the Backbone Relational component, which fits well with Rails. But it is unmaintained, it doesn't do what the team needs, and "no developers know what it does anymore."

The team is moving to React, which works well for them, though Southerland stressed that it is quite different from Backbone. Southerland also observed that Rails gave strong guidance on back-end best practices but, for a long time, gave little on the front end. Everyone, Shop-Ware included, went their own way, and getting back to something more coherent has been "a journey." Front-end assets are still compiled as part of back-end releases. The team is working to decouple them, but it is slow going because of assumptions and libraries that are hard to unwind once they are baked in. Both the Backbone and React code live in the same repository.

The team is actively migrating existing screens, not just building new features in React. Southerland explained why it is slow: the application has highly skilled power users whose workflows center on a few core pages. Anything rebuilt in React has to be fast, reliable, and work from day one without disrupting those workflows, because changes can seriously affect these businesses. Southerland said most of the application has moved to React so far, and user feedback has been very good.

Russell asked whether React could end up where Backbone is now in five years. Southerland said this "always keeps me up at night," because the team's veterans have been burned by abandoned frameworks too many times. Still, Southerland doesn't expect React to reach that point for a very long time, pointing to its momentum, corporate backing, and community. Southerland also expects that any future deprecation would come with much more notice. In Southerland's view, React has done well at keeping developers from worrying that the rug will be pulled out.

Could an LLM finish the migration? Southerland said the team had discussed exactly that the previous week. Many Backbone views are simple CRUD screens that an LLM could probably convert just fine, and Southerland believes enough Backbone training data exists for that. The core pages are a different story. The value, as Southerland sees it, is in clearing out the routine conversions so engineers have more time for the core pages that need careful human work.

"Lukewarm Waters": Where LLMs Help and Where They Don't

Southerland said the team has been "dipping our toes into the LLM water," and so far the water is "lukewarm." Part of the difficulty is the codebase's size. Agents have to be told how to navigate it: where helpers and service classes live, and that a method probably already exists somewhere instead of needing to be rewritten.

Southerland named two clear wins. The first is writing specs. Southerland's workflow is to have the tool write specs first based on the intended change to a model or controller. The code is then written, possibly with LLM help. Afterward, Southerland goes back and asks whether any edge cases were missed and adds specs for those. Southerland doesn't worry about having too many unit tests, since they run very fast, and would rather have slightly too many than slightly too few, because low-level bugs tend to be found there.

The second is early pull request review. GitHub Copilot can review a PR, which catches syntax issues, unused variables, and similar slips before a human looks at it. The human reviewer can then focus on larger structural questions instead of misspellings or renaming variables.

Russell asked whether "lukewarm" described the results or the team's enthusiasm. Southerland said the most common and still very useful role is a rubber duck: asking a CLI tool a question is easier than searching Stack Overflow or a search engine. Questions that used to become a standup "parking lot" item now often get answered by engineers on their own when they are stuck, which keeps people unblocked.

The uncertainty is about how much new code should be generated in the main application. Southerland thinks LLMs are great for greenfield work. When the team recently added a new API, Southerland built a simple interface to test it in about ten minutes: code that hit the endpoint and showed the results in the browser. The product team has also used LLMs to build proofs of concept and try out UI ideas.

Inside the Rails monolith, Southerland said code generation "is a hit or miss." It helps engineers new to a framework, such as someone new to the platform services or to Ruby. The team hasn't yet worked out how to generate large amounts of code or re-architect parts of the application without breaking what already exists.

Southerland described the core problem. Rails apps like Shop-Ware's are strictly separated by concern, which is good for humans. For an LLM, though, the files relevant to a request are often far apart, and the tool can't always grasp what a large refactor is trying to do. Test coverage is good enough that the team feels prepared to experiment, and some attempts have worked. Often, though, "you throw it away and you throw it away" until something makes sense. The pattern that works best for now is to use the LLM for a proof of concept inside the monolith, confirm it works and has some tests, and then hand that code to an engineer as evidence that the approach is possible: "this is probably not that great, but it physically shows that it's working. Let's make it better."

The Cultural Side Effect, and Code Review as the Backstop

Russell asked whether self-service answers from LLMs might be isolating people on the team. Southerland said the thought had come up just minutes earlier. Shop-Ware is fully remote and encourages pairing, huddles, and working through problems together, which builds the familiarity that makes people comfortable asking questions. LLMs can have a siloing effect.

Southerland also pointed to a difference in quality. A colleague's answer will usually be highly relevant to the question. An LLM makes an educated guess based on pattern recognition across the codebase. If 80% of the code follows a pattern from ten years ago that the team no longer uses, the LLM doesn't know that, "but your colleague a Slack room away does."

Russell added that this problem existed before LLMs. Consulting clients with 40 engineers already struggle to decide when to refactor toward a new pattern and when to leave old code alone. LLMs might speed that process up, but they also make it harder to keep everyone aligned.

Southerland described several checkpoints the team uses to stay aligned, and said the earlier a problem is caught, the better. For epics and feature work, engineers write the tickets themselves and think through everything from UI to back end to deployment strategy. Someone reviews that plan, usually Southerland, so there's no ambiguity about the intended architecture. Then the code is written, "through whatever means." The final checkpoint is code review.

Southerland considers PR review even more important than it used to be. Ten years ago, a PR was written by a human, copied from Stack Overflow, or perhaps produced with an early tool like Tabnine. Now it can come from anywhere. Copilot handles small syntax issues. For the larger questions, Southerland considers the author and the PR approver equally responsible for any code that reaches production. The review is where someone can say that a service isn't used anymore or that the team doesn't do things that way. Southerland added that reviewers can also catch decisions an LLM made that the author, deep in the work, didn't notice.

The Public API: Two Codebases and One Big Regret

Shop-Ware has a public API used both by customers and by third-party integrators. When the API was built, before Southerland joined ("a little bit of archaeology here"), the decision was made to put it in its own codebase, separate from the main Rails app. "We regret this decision," Southerland said. "I think we regretted this decision almost the minute it was made."

The main cost is duplicated logic. A model change in the main app has to be mirrored in the API. So do validations and any service helper that does a dynamic calculation. Southerland called it "a big morass" that the team is trying to unwind without taking the API down. The team keeps the two in sync partly with scripts that automatically copy over model code and the schema file, which "keeps us somewhat honest." They also copy and paste service-level calculators when needed.

On data flow, the API app reads through read-only database connections. For writes, it acts as a proxy: a POST arrives, and the API app sends a separate request to internal, non-public APIs in the monolith, which enforce data integrity and access rules. The read-back then goes through the read-only connection. No other internal consumers use those internal APIs.

The team is in the early stages of rethinking this design. Southerland said there is broad agreement that the next time a major API effort comes up, the API will move back into the main application.

The API is REST, or "RESTish," because it was built before GraphQL became popular. Southerland said that if they started from scratch they would "surely" choose GraphQL, but the current model works, and there is no pressure or demand in their ecosystem to switch. Still, "we certainly gaze at GraphQL with wistful eyes sometimes" for the flexibility it could offer. Russell joked that LLMs might eventually make merging the API codebase easy, and Southerland agreed that it's possible.

Doing the Same Job With Half the Team

A few years ago, the engineering team was about twice its current size, with several parallel efforts to build new things and launch new applications. It has since been scaled back. Southerland said losing colleagues is never good, and a lot of institutional knowledge leaves with them despite the team's best efforts at documentation, cross-training, and avoiding silos. Rails helps recover some of that knowledge because its structure forces some organization, so engineers aren't completely lost in unfamiliar code. But Southerland said Rails gets them "part of the way but not all the way."

Southerland thinks prioritization has changed mostly because less work gets done overall. The key point is that infrastructure work stays roughly the same whether a team has 10 engineers or 100. A large team can spread the hours for DevOps and on-call duty across more people. A small team has to make infrastructure as efficient as possible, because inefficiency costs the business proportionally more. Southerland expects defects to scale with feature output, so fewer features should mean fewer defects.

Third-party integrations don't shrink with the team, though. Vendors upgrade their APIs and ask Shop-Ware to update integrations built a decade ago, for example replacing an old SOAP interface. Southerland sees this as a natural fit for LLMs: working with APIs is "an LLM's bread and butter." Giving the model an API and asking how it works and what an integration needs gets the team through the first several steps. Russell recommended this use case to LLM skeptics, especially for publicly documented APIs.

Niche Integrations and No Off-the-Shelf Gems

Russell asked about using Ruby gems for integrations. Southerland said gems can become a future blocker, because adding one is a bet that someone will still maintain it when new Ruby versions arrive, or else the team ends up maintaining it. In practice, Southerland couldn't think of a case where an off-the-shelf gem fully covered one of Shop-Ware's integrations. Southerland sees an upside: the team controls its integration code, knows exactly what it does, configures it as they want, and can code defensively in ways that are hard with a gem that doesn't quite fit.

The integrations are unusual. Southerland doubts most Rails developers buy automotive parts from wholesalers programmatically or pull data from vehicle lookups. The auto industry has standardized a huge amount of data keyed to a vehicle's VIN, including identifiers for the engine, transmission, submodel, trim, and transmission type. Southerland finds the amount of data "really incredible." Southerland guessed that maybe 50 people in the world consume this data, and there are only a few providers.

Southerland said Shop-Ware is a relatively new cloud-based entrant in a market that computerized early. One of the first uses of desktop computers in the 1980s was auto shops tracking their work. As a result, many shops still run on pen-and-paper notebooks or on Windows XP- or Windows 95-based terminal applications. Those tools have been discontinued, but shops keep them because they work. Much of the job is migration: handling the nuances and helping shops see the value of moving to a modern system.

Onboarding Shops and Migrating Their Data

Customers include both single shops and entities that own multiple shops. Unless a shop runs on paper, it brings data from another system, so data migration is a large part of onboarding. Southerland said the company believes shops should be able to both import and export their data, and it tries to make bringing data in as easy as possible. But there are many source systems, some requiring custom work, and in some cases a migration is "not even really possible" because the old system is so different. Customers range from sophisticated shops to new ones that intend to become sophisticated. Russell confirmed that the migration and onboarding process is part of the value Shop-Ware offers.

Russell asked whether the codebase contains customer-specific conditionals for large accounts. Southerland said the team works hard to avoid them. When a customer asks for features A, B, and C, the team evaluates each one. Features relevant to many customers become available broadly, and others may go on the roadmap. Southerland added that they aren't closely involved in how product prioritizes. The team generally avoids tenant- or shop-specific code, "because every time you do that you're like doubling your complexity." Southerland admitted it does "happen a little bit." Russell said that from his consulting work, it's oddly reassuring that even polished SaaS platforms make occasional exceptions for a big customer.

Feature Flags With LaunchDarkly, Including Kill Switches

Instead of custom forks, Shop-Ware relies heavily on feature flags. A customer who pushes hard for a feature can get it early through an early-access program. That helps the team check how the feature works before rolling it out to larger groups and then to all shops.

The team uses LaunchDarkly. Southerland said it took a few years to settle on a workflow and get everything working as intended, but it is now very helpful. The product team controls when features are enabled. They can turn them on per tenant or for everyone, and toggle them on and off, completely independently of engineering. Southerland said this is what makes frequent deploys possible, because new UI and logic changes ship behind flags.

Engineering also keeps some permanent flags of its own, notably kill switches for third-party integrations. If an integrator goes down and that makes parts of Shop-Ware slow or unresponsive, the team can switch off that integration with a flag until the integrator recovers.

From Biweekly Waterfall Releases to Daily Deploys

When Southerland joined, Shop-Ware released every two weeks, at least in theory, and "everyone hated" the process. A QA regression pass lasted about a week and sometimes found bugs. Features that had been promised were often pushed in at the last minute, delaying the release candidate and hoping nothing broke. If you missed the window, you waited two more weeks, and bugs went unfixed for two weeks unless they were serious enough for a special release. Southerland said it consumed a lot of engineering time and a very large amount of QA time, product was unhappy with the slow pace, and the waterfall process forced many trade-offs.

A few years ago, after a big discussion of the pros and cons, the team moved to daily deploys. An engineer deploys the code each day. Almost all engineers are deployers, which also means they are on call and close to production. Bug fixes ship faster, and product changes ship incrementally as tickets on a feature are completed. That code runs in production early and can be enabled for early-access or testing tenants, where the product team can verify it on the production environment.

Southerland highlighted a change in developer mindset. The large manual regression pass that used to act as a backstop is gone. Engineers now have to think defensively about what their code does to production queries and third-party integrations. Southerland said people have become much more aware of those trade-offs and more careful about what they push, and considers the change "a big success."

Southerland listed the results: less time spent deploying, more time on targeted QA against things that matter, and, Southerland believes, fewer defects, because code ships when it's ready instead of being rushed to meet a two-week deadline. Rolling back now means undoing a day of changes instead of two weeks. Russell added that fast feedback keeps context fresh: a developer can remember what they did yesterday far better than what they did three weeks ago.

Russell asked whether the smaller team made this easier. Southerland first stressed that Shop-Ware's customers depend heavily on stability. If the application is down, shops can't repair vehicles, see new customers, take payments, locate cars, or order parts. There is "a real human cost," and the team treats incidents seriously. Even so, Southerland doesn't think team size matters much for frequent deploys, as long as everyone buys in. QA checks what an engineer directly touched, the change ships, and engineers stay very aware of the "blast radius," because they're the ones who get paged or asked about it a day or two later. If a new Sentry exception appears the day after a release, the engineer probably knows what caused it. Two weeks later, they may be on a different epic and remember nothing.

Postgres: Unlearning the Fear of Connections

"We love Postgres for a lot of reasons," Southerland said. "We don't love scaling Postgres for a lot of reasons, but less reasons than why we love it." Southerland described the standard advice: relational database connections are precious, especially in Postgres, where new connections are expensive in compute and memory. So you keep connection counts as low as possible and aim for close to 100% utilization. The usual fix is a connection proxy, with PgBouncer the best-known option.

According to Southerland, what nobody tells you is that these proxies can hit CPU saturation themselves, and when they do, it is very hard to observe. Southerland recalled starting on Heroku 10 to 12 years ago, when connection limits were very small, around 20 as Southerland remembered it. The team used every built-in Rails tool and connection pooler it could to keep connection counts down. Over time, they learned not to fear connections so much. The team has decided to "embrace connections again" and return to Rails' own connection management. That may mean somewhat more connections, but the application stays stable instead of overloading middleware that can take the whole application down.

Everything runs in one large Postgres database. Southerland said that works well almost all the time, but tenant sizes vary widely, so indexes can become unbalanced, large, and hard to maintain. The team tried Postgres partitioning by tenant ID, but it ran into problems on Heroku Postgres, where they lack full access to advanced administrative tools. It became "workarounds upon workarounds," and they concluded it wasn't workable on Heroku. Southerland said they hope to do it on AWS, because they are very interested in sharding tenants and isolating them from one another. Russell noted that other guests had split databases for a different reason: they were hitting size limits on AWS Aurora.

Reading the Writing on the Wall With Heroku

The episode was recorded in mid-March. Russell described how Heroku had recently made clear, a few weeks before, that it would not invest much more in developing the platform. Asked whether Shop-Ware is evaluating hosting options, Southerland said "it is very safe to say yes."

Southerland said the team started looking at alternatives the year before. After several incidents over the past couple of years and Heroku's lack of new offerings, "I think we all read the writing on the wall." To Southerland, the platform looks like it is in maintenance mode, keeping what it has without adding features. Southerland also said that if you open Heroku today, it would feel familiar to someone who learned it in 2010. The evaluation is still going, and they hope to move within the next year or so. Southerland said the obvious answer is AWS or something similar, where they can run containers and scale flexibly.

Southerland explained that Shop-Ware's load depends heavily on time of day. Most customers are U.S. shops open roughly 9 to 5 across time zones, so overnight a very large database sits mostly idle, "because nobody's repairing cars at 3:00 a.m. Pacific time." Southerland sees a big opportunity for smarter resource scaling on a more modern platform. The team currently deploys on Heroku's default stack ("we've been spoiled"), and containerizing the app is in progress as the first step.

The Atypical Thing: Async Updates Without Action Cable

Russell asked Southerland to imagine writing a letter to an experienced Rails developer joining while Southerland was away on a six-month sabbatical. What unusual thing about the codebase should they know? Southerland's answer was asynchronous updates. Shop-Ware has them, as many apps do, but they don't use Action Cable. The feature was built before Action Cable existed and relies on a third-party provider, which Southerland thinks is called Stream. This trips up new developers, who assume the app uses Action Cable or has been converted to it.

The flow works like this: third-party code integrated into the app opens the WebSocket and listens for updates from the provider. When the app needs to send an async update, it pushes the update to the provider, which Southerland assumes runs something like Redis, and the provider delivers it to clients. "It's essentially Action Cable, but like we're doing it pre-Action Cable."

Book Recommendation: Digital Apollo

Asked for a non-technical book recommendation, Southerland happened to have one nearby: Digital Apollo by David Mindell. It covers the design and architecture of the Apollo Guidance Computer, built in the 1960s, which Southerland describes as one of the first pieces of computer hardware that was both mission-critical and widely known to the public. Southerland likes it because it looks back at where the field came from and shows how today's decisions closely mirror ones made 30 to 50 years ago. The work of earlier engineers still informs today's.

Asked where listeners can follow the team, Southerland answered: "We have no blog. We have no socials."