Rails Scales for the Rest of Us: How Donorbox Runs a Decade-Old Monolith with a Boring Stack
Ruby on Rails
Donorbox helps nonprofits raise money online and in person, and the platform behind it is a Ruby on Rails application that has been evolving for more than ten years. In this episode of On Rails, host Robby Russell talks with Hussain Akram, an engineering manager at Donorbox, and Antonio Carlos de Souza (A.C.), its chief architect. The conversation centers on one question: how far can a team get by staying close to Rails conventions? Both guests argue it gets them a long way. They describe a deliberately simple infrastructure that handles serious traffic, a CI process moved back onto developers' machines, an ongoing effort to pull older code toward Rails defaults, and agent documentation that steers AI tools toward the patterns the team wants.
Why Both Guests Stayed with Rails
Hussain started using Rails in their third or fourth semester of a bachelor's degree, while also taking courses in object-oriented programming, data structures, assembly, and C. Following Rails conventions let them build term projects quickly without getting pulled into infrastructure questions. They describe the appeal as being able to hold a mental model of a hard problem and solve it in readable Ruby, instead of first deciding how to wire everything together. Hussain was selective about their first job after graduating in mid-2018 and looked specifically for companies using Rails, which was around Rails 3 or 4 at the time. Because they already knew the framework, their onboarding training lasted about two weeks before they joined a project.
A.C. started working in 2002 with Java on the web, after earlier frustration with how non-web applications were rolled out. They also worked with Flash and Java Struts. Their main complaint about the Java world was endless debate over how to build the same thing. The most extreme example they recall was a long argument over which logging framework to use. There were three main competitors, and the discussion was pointless, A.C. says, because dependencies required all three anyway. Around 2009 A.C. was looking for a platform without those debates. They looked at TurboGears and Django from the Python world before finding Rails, which had conventions for exactly the things they were tired of arguing about. A.C. also remembers the Java community around 2008 jumping straight to UML diagrams of design patterns instead of asking whether they had the problem a pattern solves.
What Good Rails Looks Like
Robby asked whether the two agree on what good Rails looks like across codebases. A.C. said Rails builds a sense of familiarity. When you move between applications, teams, or companies, you can guess where things live and what shape they should have. That familiarity is A.C.'s test for whether an approach is good or weird. A.C. has worked at companies that constantly fought Rails and changed every decision it makes, to the point of asking why they chose Rails at all. A.C. suspects some of this comes from boredom: Rails has already solved many problems, so people want the feeling that they made the decisions themselves. A.C. adds that this problem only exists because Rails still lets those teams deliver in the end.
Hussain's advice is to stop reinventing the wheel. Before solving a problem, look at how the community has solved it, because in their estimate 99% of the time a simpler solution already exists. The codebases Hussain has struggled with were the ones that reinvented things and became complex as a result. With Rails, Hussain says, the pieces are already in front of you, and the job is to put them in the right place.
Inside Donorbox: Teams, Kiosks, and Offline Payments
Hussain describes an engineering organization split into five or six teams, each owning one area of the service. Each team has a project manager, a designer, and a QA person. Hussain's team handles in-person fundraising, which covers live kiosks (one stands in the corner of Hussain's home office), ticketed events, text-to-give, and related features. Other teams include online fundraising, which owns the main donation form, and an experience team that builds tools for the other teams. There is also a platform team and an integrations team for Salesforce, HubSpot, and similar systems. A support team rounds this out. Its members are engineers who do high-level debugging, fix and push bugs themselves when they can, and escalate larger issues to the owning team. In total the guests estimate about 50 people.
Robby asked about unreliable internet at events. Hussain said kiosks are used by mosques and churches across the US, and some locations lack a stable connection. Stripe provides a way to process transactions offline, so the kiosk stores transactions in the device's local storage. When the device reconnects, it sends them to the server for processing.
A Monolith with a Few Support Ships
The core product is a Rails monolith. A few separate repositories exist alongside it: Android and iOS code for the kiosk, and a separate integrations application that talks to the monolith through an internal API. A.C. calls this a mothership with some support ships around it. The integrations app is essentially a background job that maps events in Donorbox to third-party tools. It has no user experience beyond that mapping.
A.C.'s test for whether something can live outside the monolith is whether the whole application could be deleted after a business change without anyone noticing. This rarely happens. A.C. joked that a business change that large would probably mean deleting the team as well.
A.C. also explained how compliance shapes architecture. Donorbox holds PCI, SOC, and other certifications, which add double checks to its processes. A.C.'s example is code review. It is easy to explain to an auditor that every pull request needs approval from someone other than its author. It is much harder to explain a homegrown integration process that runs many checks before anyone else sees the code. Weighing time lost in audits against time lost in daily process, the team concluded that a long audit costs more than asking a colleague to approve a pull request. A.C. described Donorbox as a company that didn't follow the expected startup path of breaking and vanishing. Instead it has kept growing, so old habits have to be adapted to requirements nobody expected.
A Founder Who Learned Rails to Build the Company
A.C., who joined in 2019, shared some company history. The founder built Donorbox personally, learning Rails because it was what the cool kids were using at the time. In A.C.'s words, the founder was not a programmer by trade but became one by necessity, "before Claude," which A.C. considers an important difference. Rails offered a clear learning path for someone who had never built a web application, and the founder followed it. Where the founder couldn't build something alone, they managed tasks contracted through Upwork. When A.C. joined, the founder was still doing production deployments. A.C. credits Rails with letting someone passionate build something that was useful, secure, and manageable by one person.
Shipping Fast and Paying Down the Debt Later
Hussain described the events product as an example of building fast and optimizing later. The founder had heard that many nonprofits wanted to run events for their main fundraisers, so the feature started as an experiment and was built in about three or four months, ahead of a gala season. Speed brought refactoring costs later. Because the product served nonprofits well and generated good revenue, the company could then justify investing in it. Today the in-person team of four or five developers manages events along with other products. Hussain has seen the same pattern in the main application and sees no shame in it.
Traffic and Running on Heroku
A.C. gave traffic figures measured at the edge. In the last 30 days Donorbox processed 188 million requests, excluding favicons, JavaScript, CSS, and images. (Robby restated the figure as "almost 120 million," and A.C. did not correct it, so the exact number is unclear from the conversation.) In the last 60 days, the peak was 204 donations in a single minute on July 15. A.C. says nobody in the company noticed; it wasn't a special day for anyone. Hussain added that A.C. regularly posts Cloudflare stats in Slack showing spikes that happened while everyone slept.
All of this runs on Heroku, which A.C. says the team chose for the same reason it chose Rails: the conventions already exist and work. A.C. acknowledged Heroku's reputation for being expensive but noted that Donorbox has no dedicated infrastructure team to run its own servers.
Payment Providers and a Global Donor Base
Besides Stripe, which handles far more than credit cards, Donorbox integrates PayPal and The Giving Block, which processes stock and cryptocurrency donations. According to A.C., most donations come from the US, but about 40% come from elsewhere. Europe is the second largest region, followed by the United Kingdom and India. Those regions bring their own payment methods, such as SEPA in Europe and what A.C. recalled as something like "ACSS debit."
The Life of a Donation Request
A request starts at Cloudflare, which serves the donation form, either embedded on a nonprofit's site or as a hosted page. A.C. notes these served forms were not included in the request numbers above. When the donor submits, the browser asks Stripe or PayPal to tokenize the payment method. Donorbox's infrastructure only ever receives that token. From there, A.C. says, it is essentially Heroku writing a row to Postgres.
The team tries to create a donation as a single insert into a single table, which keeps it fast because fewer locks are taken. Background jobs then extract information from the donation, for example checking whether the donor already exists and creating a record if not.
Keeping the Hot Path Fast
Robby noted that this looks unusual for a Rails app, since a typical create action might write several related records at once. He asked whether it was designed this way from the start. A.C. said the donation is by far the hottest path in the application. It was originally built around the idea of one individual making one donation to one organization. Over time that idea stretched, for instance when a company donates. Years later the team measured that keeping the operation this simple helped them scale.
A.C. was emphatic that "we're not Shopify." This is scale "for the rest of us." A.C. has been in many discussions where people claimed Rails doesn't scale, and once told someone they had more microservices than customers. A.C.'s view is that Rails scales well in most cases as long as you pay attention.
Donorbox did hit a real scaling problem, and A.C. says it was self-inflicted. The app used touch everywhere to keep counter caches updated, so creating a donation set off updates across much of the database. On a hot path that needs to insert donation after donation, stacked-up updates make the database complain, and the team had a number of outages because of it. With 100 customers you don't notice. With a thousand you see spikes if you watch the logs. A.C. said Donorbox now has more than 30,000, in context apparently customers, on one database for writes.
Robby contrasted this with e-commerce, where inventory must be checked before a sale. A.C. agreed that for donations "the more the better," but noted exceptions. Some campaigns, such as political ones, require information that must be validated before the donation is accepted, and those are the most complicated cases. Hussain added that much validation happens at the organization level, by confirming the organization is a compliant 501(c) nonprofit. Fraud prevention blocks people who are abusing the form. Beyond that, the flow is simple: the client sends JSON parameters, a donation job is enqueued, and a couple of tables are written.
Sidekiq Today, Solid Queue Tests, and Right-Sizing Dynos
Background jobs currently run on Sidekiq with Redis. The team plans to move to Solid Queue and is testing it in two of the support ships, one for the communication team and one in the integrations team, before trying it in the main app. A.C. cited storage space for jobs as a main reason. Growing Redis, particularly on Heroku, is expensive. A.C. was careful not to bash Redis, but said a relational database makes it much easier to query jobs that have run. Hussain pointed to job continuations. Currently a restart or deploy stops all running jobs, which then rerun. Most jobs finish in a second or two, but some take longer and occasionally get stuck.
Those stuck jobs are usually caught by Zacarias, whom A.C. calls the team's "server shepherd." Zacarias isn't a dedicated DevOps team but enjoys infrastructure work and watches for queries running longer than 30 seconds, queries missing caches, and queries that load half the database into memory. Hussain said Zacarias recently downsized the Sidekiq dyno from 15 GB of RAM to 2.5 GB because the extra capacity was unused.
A.C. reported that the median web dyno count over the past month was two and a half. The baseline is two, because A.C. is "too paranoid" to run just one, and it occasionally goes to three. A.C. attributes most of the bumps to deploys. A production deploy invalidates the cache, Heroku reads the result as a traffic spike and adds a dyno, then removes it about 15 minutes later.
Review Apps, QA, and a Boring Deploy
Hussain described the deploy process as simple and classic. A developer opens a PR. Each team has its own Heroku review app where the change can be deployed for the team's QA person to test. Hussain says coverage is close to 100%, and some system tests exercise the embedded versions of the donation form across configurations. QA also re-embeds forms in places like WordPress sites to check them. Changes to the donation form are rare and usually tied to major launches, such as a recently relaunched version that required thorough testing across embed variants. After QA comes code review, which any team may join. Then a single command runs all the tests and reports results to GitHub, and another internal shell command, similar to Capistrano, deploys to a staging server used for demos and then to production.
Moving from CircleCI to Local CI
Donorbox used to run CI on CircleCI. Hussain recalls a recurring problem. When two PRs merged at about the same time, coverage on the main branch dropped by fractions, blocking the next deploy and causing arguments in Slack. Around then DHH posted about moving to local CI, which the team took as validation. They built a rake-based tool that runs linting, Brakeman, and the tests, and reports results to the GitHub PR. Since only one PR can merge at a time, the coverage-drop issue went away.
A.C. said they were the one who pushed for the change and framed it differently. On CircleCI the suite took about an hour. A.C. wanted developers to experience the problems they created and to get back the feedback they had given up by outsourcing CI. A.C. also shared DORA metrics, Google's research-based DevOps measurements, which A.C. considers hard to game because the four metrics depend on each other. Last week Donorbox averaged about 10 deployments a day with 15 developers.
The tool, which A.C. calls "integrated box," has 12 steps. It runs RuboCop with autocorrection, using the full Omakase RuboCop configuration "after a lot of dispute." It runs Brakeman, the tests, and a static analysis that enforces Rails directory conventions, which A.C. credits with keeping 15 developers from drifting into different code designs. It checks that the feature branch is up to date and that the bundle is current, and it tests migrations. It requires the GitHub CLI so it can authenticate and mark the PR as checked, and that mark is a merge requirement. A.C. says it is a plain rake task and grows only when a real requirement appears.
Testing Migrations and the Case for Fixtures
To test migrations, the tool drops the developer database and rebuilds it by running every migration from scratch, instead of loading the schema, to confirm the migrations still work from earlier versions. The first version did exactly this, and the first complaint was that it wiped people's local data. A.C.'s answer was a pitch for fixtures, which A.C. jokingly introduced as "our lord and savior." People didn't realize fixtures aren't just for tests. They provide a set of known database scenarios, which A.C. argued beats having every developer learn and recreate each scenario by hand. The catch was that this conversation happened about ten years after the app was built, so the team has had to catch up on fixing fixtures.
Hussain admitted early friction. A server running in another terminal could conflict with the tool and sometimes drop the database. Hussain treats such problems as signals that something needs fixing, not reasons to abandon the process, and says they rarely see them now.
Tests as a Developer Tool and What Coverage Tells You
Running the suite locally also pushed developers to speed it up. It initially took more than 20 minutes on a developer machine. Hussain says developers proactively optimized slow tests and shared fixes, and the suite now takes seven to eight minutes. Seeing deprecation warnings locally also makes it easier to ask a colleague whether a gem needs upgrading. Hussain reports that coverage sat around 86% for a while after they joined, then rose to 100% within six to eight months of the move to local CI, though they note the change came with costs.
Robby recalled the era before hosted CI, when teams assumed everyone ran tests locally and that wasn't always true. A.C. replied that with an hour-long CI run, waiting for checks before review was exactly the problem. The tests had become a checkmark on someone else's checklist instead of a development tool. A.C. compares tests to a code editor: a tool the company not only allows but requires, for the developer's benefit. Tests don't make the PM's or manager's life easier. They spare the developer from waking up to fix a production bug nobody can explain.
On coverage, A.C. said 100% is "more noise than signal." The useful information is the list of code that could change without any automated check catching it. Reaching 100% doesn't guarantee a test meaningfully exercises a line, but it guarantees some test passes through it. If something breaks anyway, that indicates something odd in the tests.
Asked how they got the team on board, Hussain said it was easy because A.C. shared the tool on GitHub, invited opinions, and addressed suggestions, leaving a single command to run. A.C. described the organizational work. They first pitched the idea to engineering managers, arguing that developers had lost track of why they write tests. With an hour-long suite, A.C. admitted, asking for that faith was hard, so they also worked with company management to get time allocated. A.C.'s advice to teams in that position is that it is doable if you commit.
Moving from Slim Back to ERB
Donorbox is gradually converting Slim templates to ERB. Hussain describes it as part of the push toward pure Rails defaults. Slim was causing formatting issues, added complexity for new hires, and in Hussain's view was slower than ERB. Hussain also prefers templates that mirror what the browser renders, with visible structure and indentation, over configuring editors so they don't strip trailing spaces in Slim files. A.C. added that the team isn't committing to Rails defaults rhetorically. This is how they make the commitment tangible, with the eventual goal of removing the Slim gem. The issue isn't Slim itself but that Donorbox didn't need it.
The rule is that before changing a Slim file, you convert it to ERB first. Hussain says automatic conversion is too risky for some delicate parts of the app. Online converters and gems sometimes make mistakes, so converted files are tested, sometimes with QA involved. Hussain has been using Marco Roth's Herb tooling and praised Marco's talk in Amsterdam the previous year.
Keeping a Long-Running Migration on Track
A.C. used the Slim migration partly for team building, aiming for collective ownership without hurting delivery. A.C. asked management to make it a departmental priority announced to the whole company, so developers knew the time they spent on it counted as part of their job. A.C. connects this to Donorbox being remote-first, with people from California to Cambodia and many time zones between, Pakistan being the most represented. The migration was a reminder that they are one team.
The main motivation tool is a weekly post in the developers' channel showing how many Slim files were migrated the previous week and how many were changed without being migrated. A.C. insists it isn't meant as "a knife on people's neck" but as an invitation to ask for time. In the first weeks, some people edited Slim files without converting them, and A.C. reached out to help. After a few weeks that stopped. A.C.'s observation is that once a behavior changes, people forget why they ever did it differently.
Hiring and Onboarding with Rails Conventions
Hussain calls hiring an underrated Rails feature. A candidate with two or three years of experience can be expected to know the conventions, so onboarding mostly means explaining how Donorbox works. The team keeps an onboarding guide that takes six to eight minutes to read. It covers how teams are structured, how the code is organized, the main parts of the app, and tips for adding database columns and doing database-level optimizations. With that, Hussain says, new developers get an overall picture in one or two weeks. Hussain still consults the guide and links relevant sections when new developers ask questions, updating it when something is missing. Hussain contrasts this with stories from Node.js teams, where every project is structured differently and newcomers spend weeks reconstructing the original author's reasoning.
Refactoring Search Filters with ActiveModel
Donorbox needs extensive filtering for donor segmentation, for example filtering donors by number of donations made. Developers wanted to stop users from building impossible filters, such as "more than five donations and less than two," which can happen by accident, for instance by scrolling over an increment control. The original implementation raised exceptions and used flash messages to explain the error, so it was controlling user flow with exceptions.
A.C. shared Rails guides and documentation on ActiveModel without success, so A.C. asked the two developers involved for three hours together. A.C. told them it would feel tedious and they would want to ask why, since the code already worked and was deployed. A.C. stressed that the alternative wasn't necessarily better, that the merge decision was theirs, and that A.C. has never managed to make this kind of point to a large audience, only one on one.
A.C. deleted the filter code from the controllers, which got their attention, and started a plain Ruby class named, as A.C. jokes, SearchDonors. The first filter was a simple date range. When asked how to validate it, the developers suggested raising an error. A.C. instead introduced ActiveModel, which A.C. describes as the nice parts of ActiveRecord without the database, and added a validation. When the errors appeared directly in the form without extra work, A.C. says it "landed different." A.C. went on to implement all the filters in about three hours. Afterward, one developer told A.C. they had spent the whole session wondering why, but came away with the sense that if something takes too long or feels too hard, they might be doing it wrong. After more than 20 years of Rails, they probably aren't the first person with that problem. Hussain added that ActiveModel and ActiveSupport are very underrated.
AI Agents, Agent Docs, and an MCP-Powered Design System
Speaking in August 2026, Hussain said developers use Claude or Cursor depending on preference. A.C. added an AGENTS.md file and related files covering things like how to write comments and commit messages. The experience team built an internal gem of design components backed by an MCP server, so agents can ask how to add a button or a modal. Hussain says this has saved a great deal of time and unified the product, which previously mixed old and new designs. When Hussain's team reviews a Figma design now, they mostly think about the models and associations they need, because the front-end pieces already exist.
On steering agents away from outdated patterns, Hussain wants code to be readable and to follow domain-driven design, which matches most of the existing codebase. The non-negotiables are documented in the markdown files. A quick agent review will flag conflicts and ask whether to proceed, which Hussain calls a soft warning. A.C. said the team documents "the things we don't want to follow anymore."
Asked who decides those rules, A.C. acknowledged that one name keeps showing up in the commit history. Instead of a committee, A.C. reaches out to the people actually working in an area, describes the two approaches, and asks whether agents should help move toward the new one each time they hit the old pattern. A.C. sees LLM agents as amplifiers that magnify both your knowledge and your lack of it. The agent documents are meant to back up one person's missing context.
A.C. regretted not having Claude during the Slim migration. A.C. keeps one central AGENTS.md and, for Claude users, a symlink as CLAUDE.md. A.C. finds it funny that Claude reads the symlink fine but gets confused when trying to edit it. The core file covers what every question about the codebase needs: how to run tests and servers (just rails test and rails server), the tech stack, the Donorbox components gem and how to update it, and the application's directory structure. Specific pieces of the system get their own markdown files.
No Services Folder, and Integrations Wrapped in lib
Hussain said Donorbox has no app/services directory. Business logic classes live in app/models, namespaced where useful, alongside the other default directories.
The lib directory wraps third-party integrations. A.C.'s example is Cloudflare, which caches forms and hosted pages, so the cache must be purged when an organization updates a form. The code reads like a Cloudflare cache purge called with the form's slug, the same identifier used in the hosted-page and embed URLs. The wrapper translates from "Donorbox language to Cloudflare language," so Cloudflare details don't leak into application code and read like any other library. Another example is Plivo, used for text messages. A.C. stressed that Donorbox only sends texts in donor-initiated flows: a donor who has already given through a form can text to request it and give again. Donorbox doesn't send texts soliciting donations.
Hussain added that agents are discouraged from adding new CSS files. The design components should require minimal CSS, kept in the same template rather than in a stylesheet directory. If a component doesn't support what a design needs, that triggers a 15-minute conversation with the designer and the component owners to decide whether to extend the component or drop the requirement.
How Features Get Planned, and How Far to Trust Agents
Hussain outlined the feature workflow. An R&D phase starts with meetings the designer schedules to clarify goals and scope. After design, Hussain brings in a motivated developer, who spends two or three hours checking feasibility, limitations, and the deadline. Then the two hold a short meeting, about 30 minutes, to sketch a development plan, such as the migrations needed. Because the engineers are mid to senior level, an outline is enough.
Asked how close they are to handing a ticket straight to an agent and skipping review, Hussain said they are fortunate that Donorbox developers don't trust their agents too much, at least for now. Hussain uses Claude for code reviews but doesn't pass its comments directly to developers. Hussain reads them, asks why a suggestion was made, and sometimes updates the markdown files so the agent follows the team's conventions next time.
Using AI to Understand Other Teams' Code
Hussain says the biggest benefit so far comes from the codebase's size. Instead of pinging other developers or scheduling long meetings, you can ask AI how payments work, how invoices are charged, or how scheduling is done when your feature touches another team's code.
Robby asked whether this might quietly reduce conversations between colleagues and change how cohesive the team feels. Hussain doesn't think AI can answer everything yet. Many decisions were made under constraints of the time and were discussed rather than written down. For example, only a handful of people know why the events product uses Rails so differently. AI can read comments and PRs but not those discussions, unless the history is fed into the markdown files. A.C. called agents a much-improved Google search. You wouldn't ask a colleague for the exact name of a numericality validation, and you can use AI to do your homework before going to a person to ask what they were trying to prevent, which is information the AI doesn't have.
What They Want to Try Next
A.C. is interested in Lexxy, the recently released rich text editor, as an alternative to the editor Donorbox currently uses. A.C. sees it as part of a move back to "if you're going to write JavaScript, just write JavaScript," away from recreating JavaScript behavior in another language, referring to TypeScript.
Hussain is excited to adopt Solid Queue, especially job continuations. Hussain has also been reading the codebase of 37signals' Fizzy since its release and was struck by its identity model. An identity is essentially an email, and the identity's relationship to a user determines a role. Coming from Devise-style setups where everything lives on the user model, Hussain found this hard to absorb at first. Hussain likens it to a building's security guard who checks your ID without caring who you are, after which different areas of the building are open to you. Hussain thinks it could fit Donorbox, where a person may log in as a fundraiser, on behalf of an organization, or as a donor, and would volunteer to implement it. (Hussain clarified that Donorbox doesn't actually use Devise; it was just an example.)
Robby asked how someone who gives to ten organizations is represented. A.C. said that from Donorbox's perspective a donor is a person's contact record within an organization, separate from the person. Someone giving to ten organizations is ten donors, but they can manage all those donations through their own donor credentials, while each organization sees only its own relationship with them.
Book Recommendations
Hussain recommended Agile Web Development with Rails, which they called the old bible for Rails developers, along with Eloquent Ruby and Sandi Metz's Practical Object-Oriented Design in Ruby. A.C. seconded those and added The Pragmatic Programmer: From Journeyman to Master by Andrew Hunt and David Thomas. A.C. values it because it shows, through many examples, that everything in professional programming involves weighing pros and cons. It treats programming as a tool for achieving something, while acknowledging that sometimes the journey is the goal, and it calls attention to things developers take for granted.
Welcome to On Rails, the podcast where we dig into the technical decisions behind building and maintaining production Ruby on Rails apps. I'm your host, Robby Russell, and I run Planet Argon. And for over 21 years, we've helped teams maintain and evolve their long-lived Rails apps. So I tend to approach these conversations through that lens.
In this episode, I'm joined by Hussain Akram, an engineering manager at Donorbox, and Antonio Carlos de Souza, Donorbox's chief architect. Donorbox helps nonprofit organizations raise money online and in person. And behind that platform is a Rails application that has been evolving for more than a decade. We'll dig into how their team keeps that application relatively simple while handling serious traffic, how they think about Rails conventions and code design, and some of the ways that they're pulling older parts of their application towards patterns they want the rest of their team to follow. Hussain joins us from Virginia, and A.C. joins us from Brazil.
All right, check for your belongings. All aboard.
Hussain and A.C., welcome to On Rails.
Hi, Robby. Thanks for having us.
Hey, thanks for having me.
So let's get things off to a start. I'm going to start with you, Hussain, first. What keeps you on Rails?
I would say typical conventions over configuration stuff, because I started working with Rails when I was in my third or fourth semester in my bachelor's. So it was very easy for me to just follow the conventions rather than going deeper into the stuff, because I was still studying the other courses in parallel, like object-oriented programming, data structures, assembly, C language. So it was so easy for me to build stuff so quickly for my term projects. So that's where I fell in love with this framework and kept using it.
It's so easy for me to solve hard problems rather than solving the infrastructure problems, if that makes sense. So if I have some problem, I don't have to think about how I'm going to go about it infrastructure-wise. So it's just having a mental model and just solving it using Ruby code, which is so much easier to read and write. So those would be the most prominent things that I love about Rails, and that keeps me using it.
Going through the educational process of that and learning to be a software engineer, did you know you wanted to primarily work in the web space, or were you just interested in software engineering in general?
Yeah, I was very picky on selecting my first job after my bachelor's. So I searched around: what are the companies that are using Rails? And the guy who interviewed me was super impressed by the knowledge that I already had about Rails. And it was a smooth experience for onboarding because I already knew the framework. So training was like two weeks, and then I just jumped right into the project.
Wow. And just to provide some context, how long ago was that? What version of Rails do you recall that was?
It was Rails 3 or 4. It was in 2018 when I graduated, and I got the job even before my last semester, so mid-2018.
That's great. It's great to hear people be able to graduate and then land a job getting to use Ruby on Rails right out of school. What about you, A.C.? What keeps you on Rails?
By far, the convention. I was a little bit older than Hussain. So I started working in 2002, and at the time I was working with Java and for the web. I really enjoyed those information systems in the grad school. I was more like trying to get what I can learn. And one thing that always struggled with me was the rollout process of non-web applications, and I was always wondering how people do that in real life. So at some point in my life, I got to know about the web and how it works. So I got in love.
And what happened is I was looking around on things to use for the web. I worked with Flash. I don't know if you know; ask your parents what it means. And I used Java, Struts, and JSF for action. So it's a long time. And one thing that happens in the Java world is that we have infinite discussions about how to build the same thing. The most extreme I got was when we were discussing which logging framework we should be using.
The problem for this conversation is, well, there are three main competitors in the Java landscape at the time: Commons Logging, Log4j, and something else. I think it's the default logging for Java itself. The thing is, you need all of them because you have dependencies, and those dependencies will depend on them. So even the discussion is pointless.
And what happened is, in 2009, I guess, I was looking for a platform that I don't have those kind of pointless discussions so that I can actually do something. And that's when I met Rails. At that point, I was learning about two different things that I don't know if you ever heard about: TurboGears and Django, both from Python. And I learned about Rails, and guess what? They have a ton of conventions. You don't need to discuss that anymore. It's kind of a blessing, having those conversations already settled for this particular environment where I work. So it's more like, yeah, I finally found it, and I keep on that since then.
That's great. I think about that, A.C., my story is probably more similar to yours. I wasn't doing much Java at the time, but there was a lot of these other—it felt like every project, it was like, all right, what are we doing on this project? What do we want to try this time? Because maybe we weren't happy with the choices the last time, or someone was super curious, or you worked on a project someone else built. What decisions did they make? And it was always trying to put together this puzzle of, all right, well, okay, I kind of understand how these things work together, and then next time I start a new project, I'm going to do it this way. And it felt like every project was an experiment a little bit, and in taste.
Yeah, absolutely. And I don't know if you got this as well, but in 2008-ish, the Java world is getting a lot of the design patterns debates. So instead of reading the design patterns and understanding the concept of, well, if you have this problem, you can implement the solution, and these are the cons for using that solution, people jumped directly to the UML models and said, "Well, we need to implement that."
Yeah.
You don't even have this problem. Why are you doing that to you?
That's wild. Do you think you both generally agree on what good Rails looks like when it comes to looking at—I don't know how many different applications you each worked on, and we'll talk in a moment about Donorbox as well. But just in terms of knowing that you both found Rails in very different paths, and A.C., you come from this background of working with different platforms for several years. Not that you weren't, Hussain, but just that you were in the—I'm air quoting "the real world" for audio listeners here—you're working on different projects. Was there also a moment of, once you started working with Ruby on Rails, you saw you removed some of the decisions, but then as you work across different Ruby on Rails codebases, are you getting a sense of, oh, this is a good approach to building and scaling a Ruby on Rails application versus what doesn't look like a good approach to that with Ruby on Rails?
Rails builds, unless you're not paying attention at all what you're doing, Rails builds in you a sense of familiarity. So when you jump from one application to another, even in different teams or even different companies, you kind of can guess where things are, for the shape you expect them to be. And this feeling is what guides me to, that's a good approach to using Rails; that's a weird approach.
I worked in companies that are constantly fighting Rails. I got to a point of asking, why exactly did you pick Rails again? Because every single decision Rails has, they try a way to change it. A lot of people get bored because, well, you don't actually have those problems anymore, so you just need to actually deliver something. Or it's something like they keep saying that they need to build the application and have this feeling of they built it, like they took the decisions.
People problem, maybe, or who knows? I get that sense.
But that's the amazing thing. This problem only exists because Rails allows you to actually deliver at the end of the day. So even after that, you end up still delivering somehow.
What about you?
Yeah, I would just say, if you're using Rails, I think try not to reinvent the wheel every time you solve a problem. And just go around and see the community, talk to the other persons who are using Rails, or just go search online and see how people solve the issue. Most probably, 99%, this specific problem is already solved by a simplest way.
So the codebases I have struggled with, this is the issue that I saw in the codebase. They were just reinventing the stuff, and it just made the codebase so complex, which is hard to—if you're just using the conventions that Rails provides, it's very hard to diverge from that line if you're using Rails, because everything is in front of you. You just need to put the pieces in the right place and you're done. So that's a main point I wanted to mention here.
Yeah, I'm sure we'll probably get to dig into some more, maybe some tangible examples in a little bit. But I think to provide some context for our listeners, Hussain, maybe you can give us a quick high-level tour of Donorbox from an engineering perspective. And what does the Rails application actually have to support day in and day out at this point in time?
Sure. So our engineering team is divided into five or six teams, and those teams surround and have responsibilities on a particular aspect of service that we provide to nonprofit organizations. For example, my team is the in-person fundraising team, which basically includes everything that has to do with in-person. It can be a live kiosk right on my back. It can be ticketing events, text-to-give, and all that stuff.
Other teams are the online fundraising team. They handle the online fundraising form, our main donation form. And then we have our experience team that handles and provides the best tools for us, for other teams to use. And then we have the platform team, integrations team for Salesforce and HubSpot and all other stuff. So this is how we have divided our teams, and each team has one project manager, designer, and QA.
So in-person literally means being at a real live event for people. In the intro, I provided a little bit of an overview of what Donorbox is. But I think about, so you've got online donations. I think people can wrap their head around it. But then you've got—you pointed behind you, for our listeners. There's a kiosk in the back corner behind Hussain in his home office there. So it's facilitating that. Does that have to do anything, maybe get into the weeds of some of the fun details, but I've worked on some event management tools in the past with some different clients. Some of those require having to work around maybe unreliable internet at times in those spaces. Do you have to do any sort of offline processing or anything like that at this point?
We also had a similar issue, because kiosks are being used by mosques and churches across the US, and some of the places, they don't have a stable connection. So we ran into this, and Stripe does provide a way to process transactions in an offline mode. So our kiosk also handles those transactions in an offline mode. So basically, we use the device's local storage, store them, and then when the device gets online, we start processing those transactions and just send them to the server to process.
Nice.
Yeah.
And you mentioned five or six teams. Approximately how many people? I don't know if you—did you say that? Approximately how many engineers are part of that team right now?
A.C., I think about 50?
50, I guess.
Yeah, okay. I didn't mention the support team. Amazing support team. They serve our customers very closely.
Support team.
Yeah. So support agents, they are also engineers, and they kind of do high-level debugging on stuff. So if they can understand, fix a bug, and then just push it without disturbing the relative team. But if the issue is so big or needs a team collaboration, then obviously they escalate it to the respective team.
And is this primarily a monolith Rails app, or is this broken up into multiple things?
It's a monolith app, but we do have some small repos. For example, our kiosk has code on Android and iOS repos. And then our integrations team, they have their own repo because they have to communicate with Salesforce and all other stuff. So they have a very simple repo which communicates through the main monolith using our internal API.
Yeah, you can think about this as, we have this mothership and some support ships around it. The live kiosk, we have the integrations application that is also separated. So it's essentially a background job for different integrations. This thing lives aside.
And how do you think about, maybe A.C., on your side, when you zoom out across the different product areas, what has allowed you to remain primarily in the one Rails application? And how do you think about when you're going to carve off something like this integrations part of it?
It's essentially, can I delete this whole application? If some business change, can I delete this whole application without anyone noticing? Essentially, this is the question.
Have you ever needed to delete anything like that?
In a way, yes. It's not common behavior. So the default answer for this is no. If that happens, we probably need to delete the team as well, because it will be a huge business change.
And what happened is this thing, especially the integration app, for example, it's essentially a background job. So something happens in the platform, and this something needs to be communicated to a third-party tool that our customers log in there to do something. There's no UX, essentially, besides mapping what happens on Donorbox to a third-party integration.
Those are the cases that we think about, okay, maybe this can live in a different application, because Donorbox has a ton of PCI, SOC, and whatever certificates are required from us. And because of those certifications, our process needs some double checks that prevent us to walk faster.
The easiest example we have is, it's very easy to explain to an auditor that we create a pull request. We require someone else that's not the author of the pull request to approve that pull request before merging. That's unbelievably easy to explain to the auditor. If I tell the auditor that, well, we do have this homebrew integration process that runs a ton of checks against your pull request even before you share it with someone else, it requires the auditor to understand what you're talking about. It makes the audit process way more complicated or longer.
Balance between wasting time in audit, wasting time on the process daily. We found out during our time that the auditor is way harder than just asking for someone that is already working with you to approve this pull request.
That's interesting. So there are some scenarios where you're making architectural choices just to make it simpler to work through the auditing process.
Yes. Donorbox has this problem of not following the full and expected path for a startup. Instead of breaking and vanishing in the sands of time, we're
Striving. So we need to accommodate old habits into a new set of requirements that was unexpected.
How has Rails been part of that growth story? Do you feel like Rails has helped enable you, or has it been able to stay out of the way, so to speak? I know when we were prepping for this conversation, it started off with one thing and you've been adding all these other approaches to fundraising and handling donations and things like that through your platform. Where has Rails been really beneficial to you there? Leave it over to you, A.C., to start with that.
We fall back to the story lessons that it was usually QT on the box. I've been at Donorbox since 2019, so I end up getting a lot of context of how things happened in the past. What happened is Donorbox was built by its own founder. He learned Rails because it's sexy at the time and that's what the cool kids are using.
The thing is, he's not a programmer by trade. He became a programmer by necessity before Claude, which is a big difference. And the thing is, Rails has a learning path, a clear learning path for someone that never done a web application before but is willing to learn, and he walked his path. So he managed to build a company through essentially Upwork tasks that he managed himself while he wasn't able to build the stuff. When I joined the company, the founder and owner of the company is still doing product deployments.
So Rails literally enabled the company to be built by someone passionate about what they're trying to achieve and who wants to actually do something that is not only useful but secure and manageable by one person.
That's great. So the folks building the application weren't historically super experienced software engineers, were able to build the initial versions, get this thing bootstrapped, pick Ruby on Rails because that was what the cool kids were using, or at least the other people that they saw were being successful is probably another way to frame that. Like, oh, those people are doing that. I'll just follow in their footsteps.
I meet a lot of people over the years who were musicians. I was a musician myself as well, but I think about people that didn't have a super deep resume of architecture planning, building startups necessarily, but they had an idea, looked and saw what people were using, picked Ruby on Rails off the shelf and were like, all right, I'll pick this up as I go. And not every product like that ends up being successful. That's okay.
But then the journey was that eventually they realized it was kind of outgrowing them, and maybe we should hire some people that have even more experience who can continue moving this thing forward, improve it, make it more stable and help it scale as an organization, which is, I think, a great testament to Ruby on Rails that it can do that.
Was that an interesting transition? We talked a little bit about what was good Rails or not. I'm not saying you need to complain about the state of the codebase, because I'm sure every codebase has their problems, and if it's someone's first major mature Rails app that they're building, I'm sure there's going to be some things you may not be super proud of necessarily in retrospect, but you learn from that, and I think that's okay, too. But what were some of the types of things that you were coming in and being like, okay, this is how we're going to strategize on iterating from this point forward, that you can recall?
Well, a couple of them. Hussain, I actually have a very good story on that. So basically, most of the codebases that I have interacted with, they were built in a rapid fashion because the owner or co-founder wanted to solve the problem as soon as possible, and Rails gives you that power to just build and worry about the optimizations and all the other stuff later.
So one fine example was our events product, which was a kind of experimental feature at that time. Our founder wanted to give it a try because he heard a lot of nonprofit organizations, they want to organize events for their main fundraisers, and this also relates to our fundraising goal and providing services around that area. So the events product was built in a very rapid fashion. It was, I think, built within three or four months, and that came with this cost of doing refactoring and code debt and everything. But still, we were able to solve that issue gradually because we knew that the events product is serving the nonprofit very well. It's generating a good amount of revenue for the company, and now we can invest in the optimizations.
But still, I would say it's a positive side of Rails that it still gave you tools to build something cool in a very small amount of time before a gala season, and then it performed very well, and then we built a whole team around this. So now we have an in-person team of four or five developers which are managing other stuff along with events. So this happened with the product, and I have seen it even with the main projects, which were built in a very fast fashion, and then the owner or the founder wanted to invest in optimizations later on. I don't think there's any shame following that path.
That's great. And then out of curiosity, do you have any approximate stats you can share with us on how many donations are getting processed on an annual or maybe even daily basis, just to provide some context for listeners?
Yes.
Sure. I'll start from the edge until the donations.
Okay.
Two very interesting numbers. We processed, in the last 30 days, 188 million requests.
Okay.
Filter out any assets. No favicons.
Yes. No favicon, no JavaScript, no CSS, no image. That's our volume of requests in a month.
Okay. So about 120, almost 120 million.
Yes.
30 days. Okay. How big of a platform are you hosting this on? Is this spread out across several AWS data centers, or?
Yeah, it's kind of sad because we use Heroku for the same reason we use Rails.
Conventions. They're already there and working.
You know the story about Heroku being very expensive and stuff, but we don't have a dedicated infrastructure team to keep that thing running. In the last 60 days, we have a spike of 200 donations, 204 donations made in one minute on July 15, and no one knows. Literally, no one's noticed. It's not a particular day for anyone in the company. It just happened.
Yeah. A.C. keeps sharing this information in our Slack channel. There would be big spikes and nobody noticed, and he would show the Cloudflare stats that, while you were sleeping, this is what happened on our server.
Exactly.
So you've got your Heroku dynos, and you mentioned background jobs. What does that actual request life cycle look like? So maybe from the web request, someone makes a donation. Is that then immediately hitting a Stripe thing or something? And are you also integrating with several different payment processors as well to make that work? I'm not sure where your target customer base is. Is that very regional, and do you have to set up separate things in different countries because not everybody has a credit card everywhere, or are you primarily focused in environments? I only bring that up because I know that some people might integrate with Stripe in some places, but not everybody has a credit card in all places, and it's not as accessible, and they might need to use PayPal or some other sort of payment methods. I'm assuming that kind of complexity might show up in this kind of platform, right?
Absolutely. So we do have both PayPal and a partner called The Giving Block. The Giving Block helps us with stock donations and cryptocurrencies. And of course, Stripe has more than credit cards, way more than that.
Yeah. Yeah. ACH only.
What we do is, a majority of our donations came through the US, but we have around 40% outside of the US. Europe, United Kingdom, India, we have a lot of donations. Besides the majority of donations coming through the United States, we do have a global presence, and Europe being the second most, and their own payment methods like SEPA, and it's not a—they call it ACH debit or something close to that.
I see.
You've worked hard to build your Rails app, but have you thought about what happens after you call .save? An email gets sent, a job gets queued, three other records update, and somewhere a webhook fires for reasons nobody remembers. That's why there's Callback Insurance. Callback Insurance protects your team from unexpected side effects caused by before_save, after_commit, and the dreaded after_save inside a concern. So when someone asks, "Why did changing their address cancel their subscription?" you can rest easy knowing you're covered. Callback Insurance, because you shouldn't have to know what happens after save. Coverage does not include recursive callbacks, callbacks added by gems, or anything described in the code as temporary. Existing condition may apply.
Back to that other part of my question there, as I was running around. So if someone makes a web request, how does that life cycle look right now? And then maybe we can talk about, from Hussain, your perspective, what happens on the kiosk. You mentioned things might be later synced. When does that all connect from both sides?
Sure. So the request is started on Cloudflare because we have it in front of us, and Cloudflare delivers the form itself. It's either an embedded form or the hosted forms that you have a kind of a web page for the forms. Most of the time it's cached. By the way, those are not counted in the previous numbers.
The user fills in the information, sends the request to make the payment from the user perspective. But for the user browser, we request Stripe for a token of the payment method, and this is the thing that lands on Donorbox infrastructure. So we essentially only have the tokens from Stripe or PayPal. From this moment on, it's essentially Heroku writing down in a Postgres, and that's it. Can't be more simple than that.
We do have background jobs for the things that happen after the donation. We try to create the donation as one insert in one table, and it makes it faster because it's fewer locks to create during this process. And what happens is, after the donation is created, we create background jobs to extract from the donation information like the donor. Do we already have this donor in our database, or is it a new one? So if it's a new one, create it. Those are what those background jobs do.
I see. That's interesting. That feels like if you were starting to build the app up incrementally, you might not think to do it that way initially, right? If someone were to say, "Hey, we're going to be building a thing. Maybe you have one big table with a lot of columns, and then you insert that," and that feels a little atypical from the Rails way. You might try to do that stuff in your initial create statement, and then it might go off and create a couple different objects or something like that, save other data in different tables, so you have this relational thing.
Was that something that you were just seeing as a scaling thing, and that's where we're going to iterate from, and we evolved it that way, or was that an upfront plan? Because I think that's where people start to maybe prematurely do that and think that we're going to deviate from Rails conventions in that way. But how do you wrap your head around that part of things? If anyone listening would be like, well, that doesn't smell like a typical Rails thing, but maybe that actually is a really good use case if there's a high velocity of throughput.
The donation is our hottest path by far. And what happened is the application was built having in mind one individual making one donation to an organization. Through time, what happened is this concept of one individual, we started getting stretched. If it's a company making a donation, does it count as an individual? It's still a donor. The system started growing around this concept of one person making one donation. And what we measured way later on, like years before that, is that keeping things this okay helped us on the scale.
And by the way, we're not Shopify, okay? So we're not.
Yeah.
Scale for human beings, for the rest of us. And I was in way too many discussions where people started by, "Rails doesn't scale." And I was like, dude, you have more microservices in your application than customers. What are you talking about?
Yeah.
Those conversations are really hard, and for the majority of cases, Rails scales really well, and you don't need to do anything. You just need to pay attention. So we got an issue on scale because we used touch everywhere. You create a donation, we start to touch the whole database. Why are you doing that? Because we are trying to update the counter caches.
What is the problem? When you have a really hot path that needs to keep the database inserting one donation after the other, if you stack up a lot of updates, your database is starting to complain. We have a lot of out of the DOS because we created this problem for ourselves. When you have, I don't know, 100 customers, that's fine. You don't even notice. You have a thousand customers, you see some spikes if you're really paying attention in your logs. Donorbox right now has more than 30,000, one database for writing.
Oh, wow. This is interesting. I think about that path, and maybe for people listening, you mentioned you have this really hot path where you need to keep the inserts as streamlined as possible. Maybe you have this benefit of, if you bring up something like Shopify, where a lot of things have to happen because you have to double-check, is there inventory of the product, right? And maybe in a donation thing, I don't know if there's campaign-type things where there's some specific rules, like, well, we can't have this many donations of this tier, or if it's just, no, the more donations the better.
So let's just make sure money can come in as smooth as possible, and we'll make sure that we audit this thing and we can track where it came from so we can properly map this later. The point is, make it easy to get money to the organizations you want to help out, and so we'll figure the rest of the stuff in the post-processing phase of that.
Yeah, we are in the more donations the better, but we do have very specific campaigns, like political ones, that require some information that maybe is not there, and those are more tricky in the sense of, well, we need to run a validation before actually accepting the donation and report back to the donor.
That makes sense. Those are the most complicated cases for this.
And part of this validation has already been done on the organization level, if the organization is valid and they are compliant with 501(c) nonprofit organization. So it's easy to assume that their donors will also be legit, but we do have fraud prevention in place that's working fine. It can easily block donors who are just messing with the donation form. There's not much validation that we need to do. So it's just very simple. You send the JSON parameters, enqueue the donation job, and just write in a couple of database tables.
Yeah.
What are you all using for background job processing at this point?
We're using Sidekiq and Redis for now.
Okay.
We do have plans to migrate to the, for Rails...
Solid Queue.
Yeah, Solid Queue. Thanks. We have two tests with Solid Queue on Donorbox right now. One for the communication team, and we do have one test in the integration team. So essentially the idea is we have those support ships around the mothership and
We try there first. While you're thinking about and running those tests right now, is there anything that you're encountering when it comes to using Redis or Sidekiq? Is it that you're just like, let's get on board with what the general Rails conventions are, this is what's getting shipped by default? Or are there some limitations or some tooling that you aren't seeing, or benefits that you're looking at?
One of the main benefits is space for the jobs. The thing that happens, keep nudging your head on that thing, like, hey, think about next time you need to do this. And increasing Redis is okay in this size, especially on Heroku, is really expensive. While having a proper database—and I'm not bashing on Redis here. Proper database, I mean a relational database, that's all. Not judging anyone. The thing is, having a relational database to do that is way easier, especially when you're starting to query the jobs that run.
Yeah.
One of the other pain points was the job continuations. I think Solid Queue solves that issue. So that's one motivation, because currently, if we restart the server, make a new deployment, all the jobs have to stop and rerun again. There are very few jobs that take a lot of time, but most of the jobs get processed within a second or two. Some jobs get stuck, and then we have an amazing guy as an infrastructure person. So he solves these kinds of issues and notifies the team. His name is Zacarias.
Thank you for your service.
Yeah, sir.
He recently mentioned that we also downgraded our Sidekiq dyno from using 15 gigs of RAM to 2.5 gigs of RAM because we weren't utilizing anything on that scale.
Is that one of those types of things where somebody slid the thing up at one point, like, this seems safe, and nobody complained about it, and someone was able to start gradually moving that down and be like, let's see what breaks? Or they felt like they were confident enough to slide it way down to two and a half, like, no, based off of what we're seeing? You mentioned you don't have a dedicated DevOps team necessarily there, but it sounds like you do have at least one person there that maybe knows a little bit more and saves the day a little bit here and there. What did that workflow look like?
We were talking about Zacarias. He's our server shepherd. Essentially, what happened is Zacarias is really, really good on infrastructure. He likes this. So what happened is that he keeps an eye on what's happening on the database. Do you have any queries running for more than 30 seconds? Do you have queries, I don't know, not hitting the caches? Do we have queries that load half of the database to the memory before trying to do something? He does that because he actually likes it.
Yeah.
So what happened is the whole infrastructure runs on its own. What Zacarias does is making sure we're not overreaching its capabilities. I have the number for the last week, for the last month, of our median usage of web dynos was two and a half.
Oh, okay.
For the majority of the month, we go from two dynos, because I'm too paranoid to have only one, to three dynos and back to two. Most of the time you see the charts, Heroku increases the dynos when you make a production deployment. So you made a production deployment, all the cache is invalidated. Heroku thinks you're getting a huge spike of requests. They add you one more dyno. Fifteen minutes later, oh, okay, sorry, overreacting. They get back to two. That's how we got to two and a half dynos.
So it's your deploys that are triggering that more than anything.
Yeah.
When you think about your hosting infrastructure there, what's your deployment process look like right now? Before you deploy something to production, are you doing a lot of things in a QA environment? Do you have a dedicated team that are testing? You mentioned earlier, I think it was A.C., talked about your platform allows your customers to embed a form in their website. So maybe a nonprofit, maybe you can embed a form so it feels like it's part of their website, but it's an embed. Are these iframes? Are these JS snippets? Maybe a couple different versions of those that you might be doing. You can't really predict how much traffic those places are going to get, right? And that's great that you're not having to think about it that much, but I'm curious about what that QA workflow looks like when you're dealing with these forms elsewhere. Are you using Heroku preview environments or something like that for that, or is it relying a lot on just internal testing there? How do things get shipped, from a new feature or a new button getting added and getting shipped to production?
Hussain, that's all yours.
Yeah. So our deployment process is very simple and classic in terms of we raise a PR, and then we have Jira columns. Every team has their own Heroku review app where you can deploy and notify your QA to test it out. And then we have really good, close to 100, I think 100% code coverage. Some of the tests that we have written are system tests that actually test your embed versions, different types of configurations, and all that stuff. And QA does another excellent job to retest it, re-embed it in different WordPress websites or some other places to see if it all works well.
There are not frequent moments when we make changes on our donation form. It happens very rarely, when we are pushing major features to the donation form. For example, we launched a new version of donation form, which was Omni, which needed thorough testing from QAs and other teams to make sure that everything works fine on different embedded versions.
After the QA is done, we do code reviews. Every team is welcome to review the code of their peers, and then we have one command which runs all the tests and pushes the results to GitHub, all green. You run another command, deploy, similar to Capistrano that we used to do. So it's an internal shell command that we just run, and it deploys to our main staging server, which is used for demo purposes, and on our production. Very quick, very simple and boring.
You mentioned keeping it kind of boring. What do you think that allows you to focus on? I think some teams are struggling, or maybe not struggling, but they're weighing up how much they're relying on their local CI versus doing all the CI on GitHub or stuff like that, GitHub Actions. GitHub Actions has had its reliability issues recently, and hopefully by the time this airs, they'll have solved a lot of that. But it is what it is. Are there any other interesting tooling that your team has built internally, you feel like, as a result, because you've kept your infrastructure relatively boring?
Yeah, so we were using CircleCI before using local CI. With CircleCI, we used to run into this issue where the code coverage would drop if the two PRs get merged simultaneously, and when the code is accumulated on the main branch, the coverage would drop in fractions. People would be fighting with each other on Slack that I'm unable to deploy the next PR. During the same time, DHH posted this post that he also shifted onto local CI. So that was a big blessing for us, an excellent idea for us to use that. And then we developed this rake gem which basically runs the linting, Brakeman, runs the tests and everything, and pushes the results to GitHub PR. Now only one PR can be merged at one time, and we didn't have any issue related to code coverage dropping after that.
Interesting.
I have a couple of numbers about this that maybe sound interesting. We keep track of the DORA metrics in Donorbox. DORA metrics is the effort from Google for measuring the DevOps teams inside Google. Because there are four metrics and they are depending on each other, they are less capable of being cheated or gamified. One of the metrics is the deployment frequency. Last week, we were around 10 deployments a day, considering 15 developers. It's kind of nice.
This integration process that Hussain mentioned, I have a slightly different view on that because I was guilty of introducing this process. What we had before, as Hussain already explained, was an outsourced CI, and it took around one hour to run the tests, and only run the tests. And I have a different view because I was the person pushing to run the tests locally so that the person doing the thing experiences the problems they are creating. Another way, and less sad way, of seeing this is that I'm giving you back the feedback you lost when you put that on CircleCI.
And these integration steps are pretty funny because they are a little bit more than just run the tests. We run the static analysis, so we're enforcing the Ruby on Rails directory compliance. So that's how we can have 15 people without exploding different code designs. We run RuboCop, Brakeman. We make sure that feature branch is updated, bundle style, because things happen. And then we test the migrations.
We drop the developer database and recreate it, running the migrations from scratch. Why do that instead of just using setup? So that we check if the migrations are actually working in the process of migrating from a previous version to a new one. Funny thing is, the first version did exactly that. They dropped the database and recreated it. The first thing people complained about was, well, you dropped my database. I lost my test environment, my local test environment.
Yeah, happened a lot. My first question is, hey, sir, do you have time to listen to our Lord and Savior, fixtures? And it's pretty funny because people didn't understand that fixtures is not only about running tests. It's having a set of known situations, scenarios for a database. I could convince people that fixtures and fixtures:load is a way better solution than every single person needing to learn about this new scenario and create the scenario locally. The problem is that I have this conversation like 10 years after the application was built. So we kind of need to catch up fixing the fixtures.
I would also admit that after doing it locally, it also motivated developers to re-evaluate the tests. Because initially, the tests were taking more than 20 minutes. And with that time, every developer, thanks to all the developers at Donorbox, every developer was proactively working through the slow tests. They were optimizing it, sharing it with peers, and doing it across the project. And now we reduced the time to half. Still, we are continuously working on improving the time to run the whole test. If there are any warnings or any deprecation warnings, because you are running the tests on your machine, it's easier for you to just see and ping the relevant developer. Can you check if there are deprecation warnings? Should we upgrade this gem or something like that? Rather than just outsourcing to CI and just continuing on your work.
What's always interesting about hearing these types of stories is that's how we used to do it. We used to run it all locally, and then there was, oh, great, GitHub provided us a way to run it on their servers, or we had CircleCI or Jenkins. And it was also another backup thing. We assume people are running their tests. We didn't have, I think, enough confidence at the time that our peers were actually running their tests before they would push a change up. Before we had pull requests even, when you get a patch or whatever the process was, I can't even remember exactly how we used to do it, but you would have to pull down the changes, run your tests locally, and you were assuming that everybody was running through the steps before they deployed things. And that wasn't always true.
There were definitely times like, well, we shipped this out really quickly because there was a quick bug. And then that starts to degrade, or that consistency kind of goes away. So it was nice to have a nice little green check mark on GitHub to be like, well, this thing says it's happening. We didn't break anything, hopefully. But then you're saying it was taking an hour to run your tests, and then maybe locally it was taking 20, 25, whatever minutes, and then maybe you cut that down even more, 10 minutes or so. Is that?
Yeah, eight minutes. Seven to eight.
Eight minutes. Okay. And you would have to wait an hour before for CI to do that.
Yeah. So does that even mean that if you sent a PR for somebody, then you're waiting at least an hour before someone's going to review it?
That's exactly the problem. Because the tests are taking so long, they became a check mark you add to the thing, not a developing tool. So bringing back the CI to the developers reminds the developers that, hey, guys, that's for you. The customers completely don't care about automated tests. This feature, the company—because I worked in companies that I needed to fight for writing automated tests. I'm not that old, okay? Just happened to be there.
And the thing is, people forgot that the automated tests are a tool for development and for developers. And more than that, they didn't see that tool as they see their code editor, for example. And this is the main analogy I keep doing over and over: dude, you have a tool that the company not only allows you to create, they demand you to create just to make your life easier. Writing tests doesn't make your PM's life easier. It doesn't make your manager's life easier. It makes your life easier because you are the person that's not waking up to fix a bug that ended up in production and no one has any idea how.
We mentioned the test coverage earlier, that we reached 100%. First, guess who pushed this number up? Of course, it was because he is the person that, in the whole team, cares the most about those details. And the way you see the test coverage is the information is not the coverage itself, but what's not covered. So 100% test coverage is more noise than signal. The signal is the list of things that, if you change it, don't have any way to automatically check if that thing breaks.
That's the way we're using this measurement. Not if you're reaching 100%, but the list of things that break if you change something and you don't have an automated test for checking it out. And the reason for reaching this 100% is, well, I cannot guarantee the test will exercise that thing, but I can guarantee that one of the tests passes through that path. So if even this is breaking, something's really odd in the tests.
I'm curious about when your team was putting more of an effort into making everybody run their tests themselves locally and having to live with that, and learned that it's going to blow away your database. So you need your fixtures or your seeds or whatever your team is using locally to optimize that workflow. Did your team have to go through any training together? Or was there, here, follow what A.C. is doing with this approach, that seems to be the right path? Or were there workshops that you were doing to get everybody on the same page? Or did everybody pick it up over time and at their own pace? What would you replicate, and what would you maybe do differently next time?
I think you'd love to see the different perspectives on this. So, Hussain, if you'd like to go, yours, please.
At Donorbox, getting everybody on board on this was very easy because we, or A.C., built this shell tool on GitHub, shared it with everybody to share their opinions. If there were any suggestions, those were addressed, and there was just
One single command that we asked rather than expecting CI to respond to you. You just run this command and then it will respond. Yes, there were some instances where it was frustrating because if there is another server running on another terminal, it would create a conflict and sometimes drop the database. But it doesn't happen every time you run the test. Now mostly I don't encounter this issue at all. That's a signal that you need to fix something rather than just replacing the whole process.
So that happened initially. Tests were slow, and with time developers were motivated. I think we wouldn't have this time without moving the tests to local if we were just doing it on CI. So now our tests are faster. We improved our coverage, and when I joined Donorbox, the coverage stayed, I think, at 86% for a while. But I saw the spike to 100% very quickly within, I think, six to eight months when I shifted it from CI to local. So that's a positive thing, but it came with some cost.
Yeah. Tell me more about this one shell command. So you mentioned, is it just running a bunch of these different tools at that point, or what else is in there? A.C., you mentioned Brakeman and RuboCop and things like that. Is it running all those types of things? Is it similar to the bin/ci script, but just some other executable?
What happened is that we have that very creative name of integrated box, and what happened is we have 12 different steps before you share your code. So run RuboCop, they fix the things for you. You don't need to worry too much. We don't have crazy requirements on RuboCop. We're running the full Omakase RuboCop version right now after a lot of dispute on what you should run.
We run the test Rake task. We add a bunch of steps there, and we keep evolving this task, but only when we actually have a requirement for that. We don't spend too much time on that. To create the first version, it takes some time to list the things you want to do, but it's a Rake task. Very straightforward, not fancy.
The first thing you do here is requiring the GitHub app installed in your computer. It could be the CLI because we do the authentication to add a mark on the pull request that someone ran this, and we use this as a requirement for the merge. For the process of doing that, I presented to the engineering managers this idea, and my main argument is people are losing track of why you write automated tests. Some developers are getting way too close to see the tests as a check mark for someone else's checklist. To run this test again in your machine, it reminds you that you are the responsible one for that, not somebody else.
And what I had to do was first convince the engineering managers of having faith and try to see what happens. But again, the test run takes one hour. It's really, really hard to convince someone of doing that. And I worked with the company management of requesting time for them to do that. So if you're not there yet in my test which takes one hour to run, it's easier. If you get to this point, it's doable. You just need to commit yourself.
I appreciate that perspective. So I want to pivot the conversation a little bit. And one of the things I've been thinking about was in our prep conversation, I think, Hussain, you mentioned that you're in the process of moving parts of Donorbox from Slim templates back to using ERB. So why, out of curiosity? What's motivating that to start off, and then maybe what's that process look like over on your end?
Yeah. So this is also our effort to go with pure Rails defaults. Also, we started seeing issues with Slim files related to formatting and everything. So we want our new developers, or when we scale our development team, to make it easier for them to just onboard them with pure Rails defaults, easy to convey that this is how we work. So Slim files were just adding another complexity, and it was also slower than ERB. So there was no point of having this extra pretty layer on top of what we already have with Rails. That was the main thought behind it, in my opinion. A.C., do you want to add anything into it?
It also was an effort from the engineering department perspective. We're not saying that you should commit to Rails defaults as a rhetorical commitment, but this is our effort for making this commitment more tangible for everyone. If you need to change a Slim file, convert that to ERB first, and at some point we could finally remove the Slim gem. Again, it's not about Slim itself. It's that on Donorbox it wasn't necessary.
Were you around when they just started using Slim, or was that just a decision that was made very early on? Was it just a choice like, well, that's what the cool kids are doing, so I'll just use Slim because that seems to be what my peers are using? I think about that whole HTML templating abstractions that we've gone through back and forth a couple times, and I was always personally a little like, what are we doing here? Because I always felt like it was easy for me as someone that hires software engineers and front-end developers in particular, what we used to call front-end developers before as much of the JavaScript side of things came into play.
But people that could take designs and make HTML and CSS out of that, and like, great, well, let's also then introduce, now you need to learn this Ruby code to generate the thing that you already know how to work with. It always felt like it was a developer solving a problem of them not wanting to write HTML, even though you needed to understand HTML to use it. So it just felt like an extra layer of, not saying complexity, but it maybe looked cleaner when you're looking at it. I guess some people might be listening, be like, well, it's slower, but maybe it actually looks cleaner to them? I don't know.
Some guy can argue that it's easier. It makes it easier for a developer to write it and understand it. But at the same point, I would argue that we need to replicate everything that we show and render on the browser in our code editor. So it's easier to see the structure with all the indentation and everything rather than configuring different settings for Slim in your code editor to not remove the spaces at the end and all that stuff. So yeah, that was another pain that you need to go through.
Is that not something that you can just run through some sort of conversion script and just call it a day? The workflow right now is just, if you're going to have to make changes in your Slim file, let's convert it first and then make sure things will look okay. Do you feel like the automatic conversion would be a little too risky?
Yeah, I would say that because some parts of the app were very delicate, and we had to involve QA or at least a developer to test it before doing anything or trusting too much on online converter tools. There are some gems as well to convert the Slim files to ERB, but sometimes they make mistakes. It was always a good idea to test it after you convert the Slim file.
Has your team started playing with any of Marco Roth's Herb at all, or anything to—
Yes. Yes. I've been using it. I really, really appreciate his efforts on Herb. It's amazing. I really enjoyed his talk last year in Amsterdam. He's doing an amazing job at that.
Thank you, Marco, on behalf of all of us.
Yes. I'm curious about this particular project that your team's working on, a migration. It's happening over a period of time, and presumably it'll get done. How does your team stay motivated to keep working on that? Is there someone that owns that on your team and reports, hey, we're 60% of the way there? Or what's that look like from a team culture perspective of how you champion these types of things?
This particular migration I wanted to use as a team-building process. So I don't want to affect the things you need to deliver, but I do want that to be a collective ownership of what you do. So what I did is requesting the management to add this as a departmental PR for the rest of the company. The information that the department as a whole is looking for doing this as an internal requirement for the team, it gives them the peace of mind that they're not doing something else not connected to the company. So you're doing your job by doing this.
We announce it to the whole company, like, yeah, the team will reserve some of their time to do this. If they need to change something on the Slim file, they convert to ERB first. The purpose of doing that is remember that we are one team, not different people working in their own home, because Donorbox has this very specific characteristic of being a remote-first company. We have people from California up to Cambodia and very other layers in between, and very other different time zones in between, being Pakistan the most famous one of them, or most populated at this point.
The point of those Slim files is more about, guys, we're working together here. And the way of motivating people is a weekly reminder of how many Slim files were migrated the last week, how many were changed without the migration. I'm not trying to build that as a knife on people's neck, but as, if you need time, let me know.
It's pretty interesting how it worked because in the first couple of weeks when I started to post that on the devs' channel, we'd have people that needed to change their own files without converting them, and I reach out, try to help. After a couple of weeks, this behavior was gone, and people bought the idea of let's update them, and that's it. That's the thing with behavior. Once you change, you don't even remember why you're not doing that in the first place.
It's a useful perspective there. I think about how teams try to, all right, we're going to start working through this migration, or you will it in a way. Or sometimes that team requires an owner or someone championing it. Maybe in this scenario, if you're personally reaching out to people and be like, hey, I noticed you did something, and I'm happy to help provide a little bit, see if there's something we can do to make that easier in the future, what have you. And then that starts to remind people of that process and stuff like that.
I think it's helpful to hear these different perspectives, how teams approach these types of things, because I would imagine if an application's been around for a while, it seems like a good tangible example we can wrap our head around, migrating back to ERB, or maybe just to ERB. Maybe there wasn't an ERB before. Maybe it's not a migration back. But a lot of us deal with a lot of interesting challenges on the front end in our Rails apps because of just the generational things of what was the new hot cool thing of the JavaScript framework at the time. Or you built one part of a thing, and you're like, we're going to build some new thing. We're going to maybe use a new JavaScript framework for this thing to experiment. Maybe that's going to make things better.
And Rails doesn't always get those. It's changed its mind, right? And so we all can't keep up maybe as fast as Rails would like us to at times, or at least it was like, this is where we're going. And so not everybody's rebuilding their app every couple years like 37signals, and they take a new approach, and we get to benefit from seeing that in how they're approaching things.
I'm curious, when you're looking, and maybe a question for you, Hussain, initially, is the more the Donorbox application can be more Rails-like or following Rails conventions, does that make it easier for you to bring on new engineers into your teams and look at other comparison projects, be like, look, let's see how they're doing that, as there's open-source projects or things like that, or other projects you've worked on to weigh that up?
Yeah, hiring is also an underrated feature of Rails. It makes it a lot easier for you to set the expectations because the guy with three, four, or two or three years of experience, you already know that he knows the basic conventions and everything. And onboarding goes very smoothly, in terms of you just need to pass some type of documentation, how we work. We do have our own document. It's, I think, a six- to eight-minute read that you can go through, that this is how we work. And considering that, it only takes one or two weeks for a person to get an overall idea how developers are using Rails in Donorbox.
So for me, at least, it made it so much easier to onboard a new person to the team. They already have the basics, because I have heard stories from different people using different frameworks that it takes a lot of time. Take an example of Node.js. Everybody has their own way of building a project. So for, I think, the first few weeks, you will be just juggling around to get an idea, the founder of this project, how he was thinking, why he did this configuration particularly. We don't have this issue on Rails. We should be thankful for that.
Very similar to what A.C. was describing in Java land before he got introduced to Rails, and we just have to make all these decisions. Or your app needs three logging libraries because of the dependencies. They depend on different logging tooling, and Rails has removed a lot of those decisions for us.
Something else that I wanted to make sure, and being mindful of time, is one of the things we had talked about prior to this conversation was, I think there was something related to, was it search and filter pages in your application, and was that—
Yes.
Tell us more about this. What sort of challenges were you having with that over at Donorbox?
Donorbox requires a lot of filtering for data segmentation. The people working on that particular feature, they're trying to create the filters and creating some ways of preventing a user from creating impossible filters. We have a filter for the amount of donations you received, the amount of donations a donor made. So I want to see all the donors that donated more than five donations and less than two. This kind of logical—
Complex.
Yeah. But do you see that there is a logical failure here? More than five and less than two.
Yep. You need some way for preventing these mistakes because you don't necessarily need to be trying to do something wrong. Maybe you're just writing things down. You roll your mouse over the increment, decrement stuff. So to avoid those problems, we're trying to create a self-validation of the filters. The thing is, the way it was created, we're trying to use the flash message to explain what went wrong.
Oh, I see.
To do that, you raise an error. So essentially they are controlling the flow of the user by raising exceptions. I tried a couple of times sharing Rails guides, sharing the thing about ActiveModel, and it didn't work. So what I did is, I would like to ask you for three hours of your time today. I sit down with you and try to show you an alternative way of implementing the same set of filters.
But I need you to do a leap of faith because it will sound unbelievably tedious, and you have the urge to keep asking me, but it's already working. Why are you doing that? And I was like, yeah, it's not only working, it's deployed. You already delivered the value. I just want to show you there is another way. I never managed to do that in a huge audience. Every time it worked for me was when I sat down with the person and I say, I got what you did. Let me show you another way of doing that. I'm not saying it's better. I'm not saying I like mine more. I'm just trying to say that this is
Different of what you did. I want you to judge. The merge button is yours. You do whatever you want with that. But I need to give you these three hours.
So I sit down, deleted the whole code from the controllers. It kind of calls the attention when you do that. People are like, "Wait a minute." And what I did was implementing the whole filters as one ActiveModel with a very creative name of SearchDonors, because I'm very creative with those names. So given you're searching donors, that's the name of the filter.
Plain Ruby class, and I created a method. I don't remember exactly what was the name of the method. I don't know if it's all operator or something like that. But the thing is, I added the first very, very simple method of filtering by dates, and you add the first date and the last date of the donation, and you filter your donors by that.
So it feels kind of awkward in the beginning of the exercise, but I asked them, "How do you implement the validation of this?" And, "Oh, you can throw an error, yada yada." And I was, "Yeah, yeah, you also have this Rails feature called ActiveModel. ActiveModel is every nice stuff you have on ActiveRecord without the database. So let's start with the validations." So I create a validation using...
And when they saw the errors appear in the form itself without requiring them to do anything, it landed different. It's like, "Oh, wait a minute. How did that do anything? I just follow the results." It's way less effort. And I keep doing that until I implemented all the filters.
The thing is, it took me three hours. And what happened is during this process, I saw people having the same—because it's two guys—they have the same feeling that people keep having when I do that this way. One of them got back to me and said, "You're right. I was actually through the whole process thinking, why are you doing that? It's already working. But you gave me this feeling that if it's taking too long or it's too hard, maybe I'm doing something wrong."
Yeah.
Because it's not possible that in more than 20 years of Rails, I am the first person with this problem. So maybe Rails has already thought about that. You know this feeling.
The power that ActiveModel and ActiveSupport have, it's super underrated. So yeah, you can do a lot of stuff using those two modules.
I think that's probably a common thing that a lot of teams struggle with. You might have, there's already a lot of complexity maybe in a controller action somewhere. I think search is a really good example of that, where probably one of the most common things—I've seen hundreds and hundreds of Ruby on Rails applications—and those really big controller methods. And sometimes people abstract that a little bit and come up with some way to approach that, and it's not always super consistent, but seems to be a little tricky. They can be a little fickle and complicated to debug, like, "Why am I getting this weird issue?" Or you end up with a lot of weird nil messages showing up.
One of the probably most common things I see when it comes to weird things that people are like, "Well, it's probably an issue, but maybe not an issue. I'm not going to look into that particular error today, but we'll look at that at some point." It could be someone just crawling the website and hitting a search page or something, and then it triggers some weird error messages because it's a bot or something.
But knowing when to leverage Rails for these types of things, and if the code is already seemingly working and has already, as you said, provided the value, when do you come back and revisit these things? And so I appreciate Donorbox has an approach in your culture there of, "Let's go back and revisit these things. It's okay. We don't have to be scared to redo this and reimplement this in three hours." And that could unlock things in people's minds.
I'm curious, I think maybe Hussain was mentioning that you have a thing that takes six to eight minutes to read. What's in how Donorbox approaches—are these your team's engineering principles and when to reach for things? Is Rails even mentioned in that?
Yes, basically it handles everything for a new hire to let them...
Yeah, onboarding guide. It mentions the commands that you need to run. Those are all already mentioned on the GitHub repo as well. But overall idea how the teams are structured, how the code works, what are the main parts of the app, some tips and tricks for adding new columns in the database, how to do any kind of optimizations on database levels. A lot of those, and we kind of try to evolve this document with time.
So we revisit it sometimes. It's been like two years, I'm not sure, but I still revisit the same document whenever I'm having any issues. Maybe there's a new developer who raises some questions. I try to see if it has been answered already in our onboarding guide or not, and then I just share the link to that part of the document to this person. So this way we keep evolving this document, and it makes it easier for us to onboard a new guy in a frictionless way.
I'm curious, given that we're having this conversation in—it's August 2026, and this episode will probably come out in the next month or so—but I'm curious about this tooling also that you've been investing in. Are you bringing on agents into your development workflow? Are you using any LLM AI agentic development locally to help you and your engineers work on this? And how is this tooling and the onboarding documentation part of that?
Yeah, so we have put similar efforts into the agent side as well. So we do have Claude and Cursor. Some people prefer Cursor in their development flow, and some people are using Claude. And then we have, thanks to A.C., we added AGENTS.md and another couple of files for an overall view for the agent before writing anything, and how to comment, how to write commit messages and all that stuff.
And thanks to our experience team, we do have our internal gem for adding design components, which is powered by an MCP server. So your agent can ask questions: how to add a new button, how to add a modal, these types of questions. And then it will make it easier for you to not spend too much time on the front end and just solve the main problem on the back end.
So with the design components, it has made a huge difference in saving so much time and making all the products of the same nature. Previously they were a mixture of pages using old design and new design, and now it feels very easy to add a new feature. Right now my team is at the level that when we are going through Figma design, we are just thinking and modeling in our mind, "Okay, I need to add this model, this connection, association, and that's it." And it doesn't take much time to develop as well. So because you have design components in place, you just need to put them in the right place and just focus on the backend strategy and structure, and you're done.
Do you have much tooling or come up with any ways to feel pretty confident that the agent or the PR is using the ideal path going forward? Because a lot of codebases have three or four equally plausible historical ways of approaching similar things, and you might be, as a team, like, "We're going to start moving more in this direction," but the agents might be getting to source just a lot of things. How are you wrestling with that right now?
Personally, I would prefer your first thing as readable code, and it should be a domain-driven way because it's a legacy project and most of the code is built in this way. So those are the two things I want my agent or anybody's agent to focus on: that code should be readable and it should follow domain-driven design pattern. But there could be different ways. The ways that we can't compromise, they are already mentioned in MD files. So if you are asking your agent to do a quick review, it will ask you, "Okay, this is what is mentioned in your MD file. Do you still want me to write it this way, or should we rethink about it?" That's like soft warnings.
Sure, sure.
Yeah. What happened is we're documenting the things we don't want to follow anymore for the agents.
Is that just something that you're kind of, as things pop up, you're like, "Oh yeah, we should add it to the list of things to stop following in that direction"?
Who's deciding that?
Maybe that's a complicated question, but...
I don't know if everyone will have this view that it is collective, because there is one particular name that keeps popping up every time you see the commit messages. But what happened is that I reach out to people before changing these documents. Instead of creating a big committee of what are the decisions we should take for this particular thing, I reach out to the people that are actually working on that and request for comments, like, "We're doing this way in this particular side of the code. We have this other way of doing it. What do you think about requesting the LLM agents to, every time we stumble on that, help us to move to this other direction we're looking for?"
I see the LLM agents as an amplifier of yourself. So it will amplify your knowledge and also amplifies your lack of knowledge. What we're trying to do is having a place for the lack of knowledge or lack of context of one individual has a backup in an agent document. So that's the overall idea on how this is being built.
It's helpful to hear how different teams are navigating that right now, because I hear you on the, "It's a collective thing and we're all participating," but maybe there is one person that's being a little bit more vocal or being like, "Hey, what do you think?" And it sounds like you're inviting people. You're not dictating, "This is how we're going to do our things now. This is A.C.'s way. Everybody follow along." Maybe I can have a side conversation with Hussain later and he can tell me otherwise, but we'll just trust that that's actually the case there. I believe you there.
I'm thinking about, as a team, do you have any recommendations for teams right now that haven't quite figured out—is it just as simple as, in your AGENTS.md, you're annotating certain types of patterns, like, "Hey, we've done it like this," and so when the agents are looking at things, here's how to avoid, or at least to ask the soft question, like, "Are you sure you want us to...?" Maybe as a better tangible example, do you have something in there for the Slim and ERB thing, and if a Slim file is being modified, raise the question in the chat with, "Should we convert first?"
The sad story here is that we didn't have Claude when we were moving from Slim files to ERB, which is kind of sad because it helps so much. And what you're doing here on the agents is trying to describe the pieces of the puzzle in their own MD files. I do have a big AGENTS.md, and for those who use Claude, you can always create a symbolic link, which is pretty funny when you need to update your CLAUDE.md file—air quotes—because Claude tries to update the file. He said, "Oh, wait a minute. This file doesn't exist. It's a symbolic link. Yeah, sorry about that." But for reading, he didn't notice.
So what I do is having in the AGENTS.md file the core things that you need to use in every question you make about the codebase itself. So how do I run a test? How do I run the servers? In our case, it's pretty boring because it's rails test, rails server. So yeah, you got it. And we keep the tech stack, so it helps on building the context for the LLMs. We added the Donorbox components gems and how you update them, because it helps when they're creating something that requires UX. In the same file we have the application structure, like how our directories look and what is there.
Is there a lot of interesting things that are different than a, I'm going to say, a typical Rails app, that you've done differently in your Rails directory structure? Anything for our listeners that stands out, kind of interesting?
For example, we don't have a services folder. So that's one example. We want every class that you want to require, if it's associated with the business logic, it can stay in the models directory. So no need to add another directory in the app folder.
Do you namespace those at all, or...?
Yeah, we do, but it is still in the models directory. So it's just conventional stuff like app, models, views, helpers, and those defaults.
Anything interesting in your lib directory, or...?
We have a lot of different integrations. For example, Cloudflare, because Cloudflare maintains the cached forms and hosted pages. So every time a user—in our case, an organization, our direct customers—they update their form, we need to invalidate the cache. We use lib as a wrapper around those integrations.
So we literally have code like, "Cloudflare cache purge this form right here, this campaign form," and the translation from Donorbox language to Cloudflare language happens in this call of, "I tell you the name of the form," and by this name, the Cloudflare cache purge knows how to get that URL.
The cool thing about this is when you're reading the code inside Donorbox where there's this code, it sounds that Cloudflare is just another library that you're using. When you have the update of the form, you have a Cloudflare cache purge with the name of that form. So we usually call it a slug because you have a donorbox.org/slug of the hosted page, or embed/slug to get the embedded version. The Cloudflare details didn't leak inside Donorbox code, and we have a couple of examples like this.
The other one is Twilio, because we send text messages for donations, and we don't send text messages for requesting donations. We do have this feature of you text Donorbox asking for a form. If you already made a donation in that form, you can make another one through text. But it's always a donor-initiated process, not a...
I see. Any other fun, interesting things in your Rails directory structure there, or...?
Also, I wanted to mention, we do have one thing that we are kind of demotivating agents to do, is adding any new CSS files. So the whole point of having design components is to not write, or at least write minimal CSS, and that CSS should live in the same HTML file. So there shouldn't be a directory or pathway of all the CSS files.
So if there's something missing in design components, then this is a good point to reach out to the team that, "I'm using this design component, but it's not supporting, and the designer built it this way." So it's just a 15-minute talk with you all three, and then you make a decision, "Okay, we need to evolve our design component, or we need to ditch this new requirement."
That's interesting. Is that typically coming up when you're working on some new feature request or something? You're like, "How do we...?"
Mostly new features.
You mentioned you're using Jira. Out of curiosity, when a developer gets a ticket, the engineering team has a ticket, is it already kind of determined, like, "You should be pointing to this in our design library," or is that something the developer is figuring out as they start working on the ticket?
So basically, in the start of the task, we do an R&D phase where a designer would schedule a couple of meetings to get an overall idea what we want to achieve, what we need to skim before we actually start the design process. And after the design process is done, I involve one developer who is motivated to work on that feature or product, and then we would set up a meeting and then ask any questions. So this developer
Would take two or three hours to go through the design, see if there are any limitations, if this is doable, if we need to skim, seeing the deadline and everything, and put it in perspective if it will be done before that. If all good, he has answers to all his questions, then I would do another meeting with him just to devise a development plan. It's not a long meeting like three hours, but it's like a 30-minute meeting. We would just quickly go through, this is the migration that we need to add, like overall idea, because engineers are also mid- to senior-level engineers at Donorbox. So you just need to give an overall idea that this is my approach. If you have ideas, then we share it with each other and get to work.
I see.
Yeah.
And how far away do you feel like you are right now? Maybe we think about Hussain, on your side, where you have your Linear or your Jira ticket or whatever your team is using, and be able to be like, all right, a developer takes that, pops it into their Claude or Cursor, and be like, all right, implement this, and not needing to do a lot of code review themselves. How far away do you feel like you are right now with that culturally at the moment, or personally maybe as well?
No, I think we are blessed in this regard that developers at Donorbox, they are not trusting too much on their agents, for now at least. So I'm not very cautious about this particular issue. I do use it for code reviews and all that stuff, but I become hesitant just sharing that review from Claude to the developer. I need to go through it, ask questions: why did you post this? And maybe update the MD file so the next time it doesn't make that mistake and it considers the convention that we are following or the direction that we are following in terms of code quality or code debt or anything like that. So yeah, I'm not much worried about this particular issue.
Do you feel like you're seeing a lot of throughput improve, like you're able to get through more as a team at this point? I feel like talking to different people on the podcast, and some people are like, oh yeah, the number of PRs has drastically increased, and we have other issues now as a result, but now we have more PRs to review or something like that. Is that showing up yet, or?
I think the biggest advantage for us that we have got so far is because the Donorbox project is very big. So it has empowered you so much to know what other teams are doing just by asking AI rather than pinging other devs. You can just have a conversation with it: how the payments are working, how we are charging the invoices, how we are scheduling everything. Those have empowered you to understand how other teams, if your code or feature is related, is touching some parts of their code. So rather than having long meetings, when AI—
Something I'm curious about with that, do you foresee, fast-forward this progression a couple years, regardless of whether or not we're going to get to a point where we can just assign an agent a task and be like, great, and then it starts sending PRs where you're not even looking at anything. But if your developers are maybe spending a little less time in those longer meetings, do you see there potentially being any interesting things culturally where your team's not checking in with each other more, but themselves, and they're just asking the AI for information on what their peers are doing? And do you feel like that could cause any interesting, maybe not a good or bad thing, but just more of something I'm always curious about when people are like, I'm not needing to ask my co-workers for their input, like how have we done this in the past, because I can just ask Claude or Cursor or whatever. And if those conversations start to get reduced, I feel like there's got to be some sort of interesting unexpected consequence of that. Maybe not a bad or a good thing, but it'll be different. Maybe will it feel like a cohesive team in the same way, or maybe it'll just feel different?
Yeah, I think we haven't gotten to that point that AI can answer every question because there were so many decisions that people made considering the limitations at that time. So AI doesn't have that information. It can go to the comment and PR and all that stuff, but not the discussion that was made. For example, why events had a different way of using Rails? Why it was so much different? So only a handful of people at Donorbox know this story. Unless you feed it into the MD file, AI can't know why it was built this way. So this is kind of a contextual history. I think you need to feed it, and you can't trust it too much with everything. But you can ask it what this class does, or trivial stuff which takes time for a human to understand.
Any thoughts on that, A.C.?
Yeah, the agents are a very enhanced Google search. So the same way you don't approach a colleague to ask for what is the exact name of that validation for numeric values that is positive and only integers, you just search for that. Looking into the future, questions like, okay, where should I look for some information? You can start the conversation with the AI and get the information you need to actually have a conversation with a human being and use that as, oh, now I did my homework. This is what I found. Can you help me understand what the hell you did here? Why is it this way? What are you trying to prevent? Because this is the information that the AI doesn't have.
Is there anything, maybe for each of you, that you've yet to be able to experiment with that you would love to be able to get an opportunity at some point? Something coming down the road or something that's been introduced that you're like, well, we haven't been able to start doing that at Donorbox, but maybe at some point I'd be really interested to be able to have a chance to play with that. Something in the Ruby or Ruby on Rails ecosystem. And maybe we'll start with you, A.C.
Right now we were using this FA editor, and I'm really interested in the MD editor that was released recently, as recently as possible, but you got the point. It sounds interesting to check it out. And it's a move back to the, well, if you're going to write JavaScript, just run JavaScript. Don't try to recreate JavaScript in another language behavior that Trix didn't have. It was that TypeScript stuff. I would love to have the opportunity to test that because it sounds very interesting.
That's Lexxy.
Yeah. What about you, Hussain?
I think I'm excited to use Solid Queue in our codebase very much, especially the job continuations. It's been a while since Fizzy released their codebase online. So it's just so much fun for me to casually read the codebase. And another thing I noticed was how they are using the identity. So identity is basically equal to email. And if you do identity current to your user, and if you do identity dot user, it tells you your role. Basically, this was very unconventional for me to absorb and digest because coming from the background using Devise and session controllers and all that stuff. And I think this can play very well at Donorbox in terms of logging in can be done from different roles. You can be a fundraiser, you can be on behalf of an organization, you can be a donor. Implementing this would be fun for me, and I would raise my hand to do this.
Out of curiosity, I didn't want to go too far down the weeds there into your platform, but if a single person, let's say robbyrussell@gmail.com, is a donor to one organization, but I've donated to 10 different organizations, 10 different customers there, is that linked at all in your system, or are those treated as separate donors per customer type of scenario?
I think you're going to laugh at my answer, but the answer is yes. The thing is, from a donor perspective, a donor is a contact of a person in that organization's agenda. The person itself is another thing.
I see.
If you donated to 10 different organizations, you have 10 different donors to represent yourself. But the cool thing is you can manage all those donations through your donor credentials.
So you can see all of them, but they can only see their own context for you.
Yeah. Okay, that's helpful. I was kind of curious about that, but Hussain, do you feel like there's a scenario where you might look to migrate away from something like Devise?
We're not using Devise. I just gave an example because typically on Devise or any, you try to use everything in the user model, like authentication pairs and everything. But Fizzy gave a very different perspective on how they solve this issue. Using identity, so that was interesting. It's just like an analogy of entering a building. There's a security guy. He just wants to check your ID, doesn't care about who you are, where you work, and he just lets you in. And then there's different types of areas in the building that you can go in.
Yeah. All right. So before I let you both go, here's a question that hopefully is a little less complicated. I'll start with you, Hussain. Is there a programming or software book that you find yourself recommending to other engineers?
I would just recommend the old bible for Rails developers, Agile Web Development. That's a good one. I loved Eloquent Ruby and Practical Object-Oriented Programming by Sandi, I think.
Sandi. Great. What about you, A.C.?
Well, Hussain stole all my recommendations, so plus one. I go with The Pragmatic Programmer: From Journeyman to Master. It's not a new book. It's kind of old, from Andrew Hunt and David Thomas. You probably heard about David before. The discussions in this book are about everything you do in a programming environment professionally. You need to weigh pros and cons, and they give you a thousand examples of that through the book. But the way they describe it is programming is a tool for achieving something you want to do. Sometimes your goal is the journey of programming itself, and that's fair, but they call your attention to things that you just take for granted. So that would be the reason around my book recommendation.
That's a good addition to that. Well, I'll definitely include links to all of those in the show notes for listeners. And with that, I want to thank you both, A.C. and Hussain, for stopping by to talk shop with us on Rails. It's been a delight getting to chat with both of you about all the fun stuff you're doing at Donorbox, and hopefully we'll get to chat again at some point, maybe at a conference or something.
Sure. It's been a pleasure.
By the way, will you be at Rails World in Austin?
Yeah.
Good. We'll look forward to getting to catch up there. I'm speaking at the end of day one, I just learned. So, well, thank you so much, both of you. Thank you again for stopping by and for joining us on Rails.
Thank you so much.
That's it for this episode of On Rails. This podcast is produced by the Rails Foundation with support from its core and contributing members. If you enjoyed the ride, leave a quick review on Apple Podcasts, Spotify, or YouTube. It helps more folks find the show. Again, I'm Robby Russell. Thanks for riding along. See you next time.
Article published · Updated