Scaling Ingestion Without Breaking Customers: How Ascend Runs a Rails Monolith for Insurance Money Movement
Ruby on Rails
In this episode of On Rails, host Robby Russell talks with Eddie Galindo, an engineering lead at Ascend, and Kagen Hearn, a senior software engineer there. Ascend builds software that helps insurance agencies move and reconcile money. Five years in, the whole platform still runs as one Rails monolith. Eddie has been there since the start. Kagen joined a few months before the recording, just as one product line began onboarding much larger customers.
The conversation keeps returning to one question: what happens to a startup-speed Rails app when enterprise-scale customers arrive, and how does a small team find and handle the real limits? The guests describe a pragmatic approach. They stay on Rails conventions, keep infrastructure minimal, and add complexity only once there is a concrete reason for it. The most detailed part covers how they found the limits of their data ingestion pipeline, and why the final limit turned out to be their customers' own APIs.
What Keeps Them on Rails
Eddie started with Rails in the last year of college and just after graduating. He then spent several years away from it. At LinkedIn he did mostly front-end work, and he lived through that company's move from YUI to some jQuery, then to Ember.js, with discussions about React starting as he left. After that he joined a startup that used only Node.js. He said it was nice to come back and find Rails essentially the same. He had bought into the Ruby and Rails philosophy, and its principles still resonate with him.
Kagen came to Rails more recently, at their previous job at Midesk. Before that they had worked with Node.js and Python/Flask on the back end, and with Python, Scala and SQL during a data engineering phase. What Kagen values is that Ruby and Rails "lean into their own design principles rather than trying to hedge against them." In their view, Ruby is a dynamic, object-oriented language that doesn't try to be anything else, and that commitment gives it expressiveness and allows elegant designs.
What Ascend Does and the Scale Behind It
Eddie described Ascend as a company founded about five years ago to build a financial platform for the insurance industry. It focuses on money-related problems. For small and medium-sized agencies, it helps collect and disburse funds. For enterprise customers, it supports workflows for operating bank accounts and reconciling money such as incoming commissions. Asked about the business model, Eddie named a few channels: SaaS subscriptions, a financing product that helps people finance the policies they buy, and revenue tied to money transactions. The engineering team is about 20 people, the largest it has been.
Kagen works on the SaaS product for direct bill reconciliation, which serves accountants at insurance agencies. In direct billing, the carrier that underwrites a policy collects the premium and later passes the commission to the agency. The agency receives the commission deposit asynchronously, and the statement describing it can arrive before or after the money. Accountants then have to match deposits against statement entries to work out where every dollar came from. Ascend's product helps with that matching.
The guests would not discuss dollar figures but did share volumes. According to Kagen, Ascend has recently processed tens of thousands of statements over a few months, and a single statement can contain tens of thousands of entries. The number of reconciled statement entries has more than doubled in the past few months as new customers came on. Kagen expects it to grow by many more multiples, because many of those customers are just starting to ramp up and more are in the pipeline. The main scaling factor is the number of statement entries ingested and processed, which Kagen said is "growing by the millions."
Why Rails, and Whether You Need Rails Developers
Eddie gave a plain answer on why Rails was chosen. One co-founder had used Rails at Instacart, and it was the easiest way to get started, so they bootstrapped the app with it. What mattered early on was speed to a working product. The team didn't want to make "a hundred technical decisions" that would be irrelevant if the company didn't survive. They wanted to focus on the product and on whether they were building the right thing.
Robby asked whether a new startup needs to hire experienced Rails developers. Eddie said Ascend didn't. At the start, perhaps 30% of the team knew Rails. The team formed because people had worked together at a previous startup and wanted to keep doing so. For those new to Rails, the framework was easy to pick up. Eddie attributed this partly to a deliberate choice not to deviate from a typical Rails app and to rely heavily on the official guides and other public material.
Eddie estimated that 80–85% of the work can be done by looking up how the guides do something. Things like validations or returning a JSON response are settled patterns the team no longer has to decide on, so the interesting problems are on the product side. When Rails doesn't cover something, a gem usually does.
Kagen confirmed this from the newcomer side. They were "pretty impressed" by how closely the codebase followed Rails conventions. Things were generally where they expected them to be, which made onboarding easy, especially for someone who already knew Rails.
Why the Platform Is Still One Monolith
Ascend runs one monolith for what it calls its core platform. Eddie explained that the product is meant to be a platform, with customers using different parts of it, and that a monolith has been much easier to manage than multiple microservices.
The different products share a lot at the foundation. Ascend integrates with external systems and normalizes the data it pulls in, and every product benefits from that normalization. User management and organization management are also shared. Eddie also stressed infrastructure and compliance. For a small team handling money, not having "a hundred applications" to monitor closely matters. His stance is to push the monolith as far as possible and revisit the decision once it becomes hard to manage.
Ascend has five teams, one for each of its five products, and they are roughly evenly sized, though newer products have fewer people. Each team owns its product. Shared platform code is a collective responsibility that everyone is expected to keep clean. Eddie acknowledged that this arrangement may break down as the company grows, but said it has worked so far.
Robby asked Kagen whether the shared monolith feels like a superpower or a constraint. Kagen said it currently gives more benefits than constraints. In their view, many benefits of microservices are organizational: teams can deploy their own domain independently and isolate dependencies. Ascend is not at that point, because many models are common across the insurance domain, whether the work is agency bill, direct bill or another accounting task. Policies, lines of business and clients are shared everywhere, and the teams benefit from extending those models in one place instead of updating them across services.
Eddie did name one downside later. Direct bill reconciliation is the product pushing the platform's scalability and performance limits. Because everything lives in one app, the team has to watch infrastructure concerns that only one part of the system really needs.
When Bigger Customers Arrived, Ingestion Assumptions Broke
Kagen said that until recently, direct bill had two relatively large organizations as customers, on a product that had been "fairly hastily put together" a year or two earlier. They described the tension as "trying to ship at startup speeds for large enterprise customers." Once many new customers began onboarding, earlier assumptions started to fail, especially around ingestion.
To serve a direct bill customer, Ascend has to ingest their policies, lines of business, clients and other data from the customer's agency management system (AMS). Kagen said most of these systems have been around a long time, some old enough to use XML and SOAP services. When the wave of new customers arrived, Ascend could process roughly 500,000 to one million new policies per day. The limit was latency. Ingestion means many slow external requests, and each one holds a Sidekiq thread for the whole wait. The number of Sidekiq workers therefore capped how much volume could be processed per day.
Finding the Real Limit: Postgres Connections and PgBouncer
Robby asked why they didn't just add more workers. Kagen explained that the team didn't know the actual scaling boundaries, because it had never needed to ingest more. The first step was to find out. Outside business hours, they gradually increased the number of workers and watched what happened.
The first limit was database connections. According to Kagen, Sidekiq takes a database connection for each thread the first time that thread does database work, and all these workers do database work. As they added Sidekiq workers, they quickly approached Heroku Postgres's connection limit.
The fix was transaction pooling. The team added PgBouncer on the client side as a sidecar, but only for the Sidekiq workers doing ingestion. Kagen explained that transaction pooling rules out some Postgres features, and checking the whole application for code that depends on them would be a heavy lift. Limiting PgBouncer to ingestion workers helped a lot without requiring that audit.
Inside the Ingestion Pipeline: Change Events on a Cron
Kagen then described how ingestion works. A cron job runs every 10 minutes and polls each customer's API for change events, meaning which policies changed within a configurable lookback window. For each change found, Ascend writes a change-event record to Postgres.
A second scheduled Sidekiq worker picks up those change events and pushes them onto processing queues, mainly the policy sync workers. This middle step gives the team one lever for concurrency: they can control how many change events this worker picks up at a time. A backlog can build up and be worked through as fast as needed or possible. Policy sync workers can also trigger secondary work, such as syncing transactions related to a policy. All customers run on the same shared resources simultaneously.
Robby asked what happens if a policy changes just after a poll. Kagen said it's fine for processing to happen later. The lookback window is deliberately longer than the 10-minute interval so nothing is missed. There is a daily sync covering the past day, and the 10-minute polls look back further than 10 minutes. Kagen wasn't sure of the current setting but thought it was about the past hour.
When the Customer's API Becomes the Bottleneck
After transaction pooling fixed the connection problem, a new limit appeared: customers' own APIs could only take so much ingestion traffic. Kagen said that too many concurrent policy requests led to 504s and other server errors. In the worst case, Ascend could disrupt a customer's own service if it wasn't careful. Simple retries were not acceptable, because the goal was to avoid overloading the customer.
A Concurrency Rate Limiter Inspired by Stripe
The solution was a concurrency rate limiter based on a 2017 Stripe blog post about rate limiting, specifically its section on concurrency limiting and the sample implementation Stripe published as a GitHub gist.
Kagen described how it works. Each integration with outgoing requests has a set in Redis. Before a worker makes an outbound request, for example to fetch a policy, it must acquire a slot, which means adding an item to that set. Each organization has a configured concurrency limit, the number of simultaneous requests its system can handle. If the set is at or above the limit, the worker can't get a slot. It retries about every 100 milliseconds, eventually times out, and re-enqueues the job.
This design has consequences for how jobs are written. A job can fail to get a slot at any point, so every enqueued job must be idempotent. In rate-limited tasks, the team tries to put one request in each job so that each job has one side effect. Kagen noted these are general Sidekiq principles but not always easy to follow. A policy sync tries to acquire a slot at the start. If it fails, the job goes to the back of the queue, so other organizations keep moving while one is throttled. The transaction-sync jobs it triggers are rate-limited as well.
The limiter is written as a Ruby class used with a context block, so it is opt-in. Requests from user-facing paths, such as a user clicking a button that triggers an external API call, are not limited and don't use slots. Kagen's reasoning was that user-driven traffic doesn't put customer APIs under the stress that bulk ingestion does.
Eddie added that none of this was designed upfront. The pipeline began as "just a worker that's going to make an API call" to ingest data, and the team kept running it until it hit a limit, then dealt with each limit as it appeared.
Robby asked how the limits are set. Kagen said new customers start with a safe, conservative value. If more ingestion volume is needed, the team estimates what the customer can handle based on its size and whether it runs on-premise or on shared infrastructure. Some customers can only take very little traffic, so their limits have to be lowered. Robby joked about asking customers to just buy more hardware. The guests laughed it off, and the topic ended there.
Feature Flags, Organization Configuration, and the God Object
Kagen confirmed that feature flag and configuration complexity has grown a lot in recent months as large customers arrived. They see this as natural in enterprise B2B software. Each customer wants its own version of the product, with its own accounting workflows and ideas about reconciliation. Some have acquired other agencies, which changes their processes too. The team tries not to keep adding conditionals, but Kagen said a lot of them are unavoidable.
The team uses two mechanisms. Feature flags, managed in LaunchDarkly, gate actual product functionality: which views and UI buttons a customer sees. Organization configurations live in the database and adjust behavior of features every customer uses. Kagen's example is statement matching in direct bill reconciliation. Each statement entry has to be matched to the policy it belongs to, and every organization has its own matching rules, which can differ by context. Organization configuration lets each customer express those rules.
Robby asked whether this becomes a growing god object. Kagen said yes, it currently is one. Each setting is a real column on a database model that belongs to the organization, not a JSON blob. Kagen said "thankfully" to that, and noted that needing a migration for every new setting adds friction that somewhat discourages adding more. Eddie added that the team is considering splitting configuration by domain, for example payment configuration and reconciliation configuration, so it isn't one huge object. The application is multi-tenant, with organizations' data separated but stored in a single database.
Eddie explained the rule for choosing between the two. Feature flags are for short-lived things. With enterprise customers, a feature may need customer approval or a slower rollout, but the goal is for everyone to get it eventually, after which the flag and its branching are removed. Organization configuration is for things that permanently change business logic, such as how granular reconciliation should be. Those settings are part of how the product works and will stay.
When Customer Flexibility Becomes Technical Debt
Robby asked Eddie how to tell when customer flexibility turns into technical debt. Eddie said everyone struggles with this. The team tries to push back and think about the product in general terms instead of building one-off features. It tries to see the request from the customer's side and from Ascend's side: does this belong in the product long-term, or can the customer adopt a different process? He admitted they don't have all the answers. Sometimes a request makes sense and they build it. It is decided case by case, with attention to what makes sense for all customers and why a customer is asking.
This works partly because engineers and product people often join customer calls. That lets them work through the actual need instead of receiving a promise already made by sales. Kagen said the same was true at previous SaaS companies. The largest contracts got more custom work, but before agreeing to anything custom, the team asked whether it would generalize to others. They usually pushed back on very bespoke features that would never apply to anyone else, though there were exceptions.
Convention Over Configuration and Duck Typing
Robby asked whether Rails gives enough tools for this kind of per-customer customization. Kagen disagreed with the common criticism that convention over configuration makes custom work harder. They think that criticism applies more to technical choices than to product work. Because conventions make it quick to build something new, it is also quick to build something bespoke.
Duck typing helps too. In Kagen's explanation, duck typing means an object's type is its interface, the methods it responds to, not declared type information. When a customer needs something odd built on shared functionality, the team can write an adapter or a class that matches the interface an existing module expects, or override a method, and reuse most of the existing logic. Kagen said this helps "with hacky things like that in particular."
Robby raised the concern that this can feel too magical and be hard for newcomers to debug. Kagen agreed it is a real trade-off. You can write such code in a way that hides what's going on, especially if you ignore Rails conventions. Conventions help you find things and reason about how they work. When you drop them, the "magic" can take over. Kagen sees this as part of the cost of Ruby and Rails fully committing to duck-typed object orientation. Eddie had nothing to add.
Heroku's Uncertain Future and Tuning Puma
Ascend runs on Heroku. On Heroku's long-term plans, Eddie said the team is as confused as everyone else. He mentioned seeing on Reddit that Heroku had posted about going into maintenance mode while still releasing features. The team has started looking at options and will move if it has to. It is hitting limits in some areas, but hasn't yet done the work of evaluating alternatives. Eddie mentioned that their autoscaling provider said it would evaluate vendors and publish recommendations for Heroku-like platforms. He didn't know if that had come out yet and said the team will do its own evaluation when the time comes.
Eddie also described a Puma tuning change. Requests were being accepted and then sitting idle, so the team looked at using more of Puma's concurrency. It turned out to be an easy configuration change: raising concurrency improved requests per second. Higher concurrency also used more memory on their Heroku dynos, so they switched to jemalloc for Ruby memory management. Eddie said both changes have worked well so far.
They tested these changes in a Heroku sandbox configured much like production. Eddie said they had learned the hard way: an earlier attempt to turn up concurrency made the application stop working.
Eddie credits sticking close to a standard Rails app with keeping Ruby and Rails upgrades easy. With coding agents, he said, you can ask for a version bump, have the agent run the tests, and then do the team's own verification.
AI Tooling, Testing at Volume, and Local Seeding
The team hasn't standardized on AI tools. Eddie said most engineers use Claude Code, but nothing is required. Robby noted the recording took place in the second half of June, and that the tools change quickly.
On simulating realistic data volumes, Kagen said the team has seeding rake tasks. As far as they know, it hasn't invested heavily in AI-specific guardrails such as setups that let an AI fully develop and verify its own work. Kagen personally uses AI a lot for that purpose, though. Their example: customers can export any list view to CSV, and at current volumes, building the CSV in memory became very slow or even crashed. Kagen rewrote the builder to stream. To test it, they asked Claude to open a Rails console, create a statement with a million transactions, and generate a CSV, to check that memory use stayed stable and that latency dropped reasonably.
Eddie said the sandbox contains only mock data, and the team relies much more on its automated tests than on clicking through a sandbox. The Rails app mainly serves a JSON API. For data-scale concerns, the team writes benchmark scripts and measures memory pressure in different places. Eddie said a test-first approach is what mainly keeps bugs out.
A JSON API, a Next.js Front End, and Over a Billion Sidekiq Jobs
The JSON API is used both internally and by customers through a public API. Responses are built with Active Model Serializers. The customer dashboard, where users do most of their work, is a Next.js application. Eddie said that decision was made early by the engineer who owned the front end. He didn't remember whether Hotwire existed yet, but at the time building dynamic UI in Rails seemed harder, and so did hiring engineers for it. He agreed with Robby that things have improved since.
Sidekiq has processed about 1.3 to 1.4 billion jobs. Eddie called it a very important part of their infrastructure, and the team is on the Enterprise plan.
How the Team Vets Gems and Dependencies
Eddie said the team's approach from the start has been that any gem it brings in is code it owns, even though it lives outside the repo. The team has to understand it during upgrades and debug it when something breaks. So they avoid pulling in a complex gem to solve a small problem. They are comfortable with established, widely used gems. PaperTrail, used for model audit logs, is one he named.
Robby asked about abandoned or blocking gems. Eddie said they have been lucky: when they needed a fix, a PR usually already existed upstream, so they would use their own fork until it was released. He thinks they are down to maybe two forked gems. The team rarely adds new gems now. PgBouncer was the most recent new dependency he could think of, and the application's problems tend to be similar enough that a small set of gems covers them. Keeping existing dependencies from being compromised is something he said is on their minds, especially given the industry they are in.
Running Elasticsearch Alongside Postgres
Ascend has two main data stores, Postgres and Elasticsearch. Kagen explained why. Customers have millions of policies, statements can have tens of thousands of entries, and users eventually want to filter or sort by almost any field on a transaction. Elasticsearch handles that well on large data sets, and the number of supplier statements will keep growing without bound. Postgres serves detail views. Elasticsearch serves list views with filtering, sorting and search.
Keeping the two in sync is an ongoing challenge. Documents are serialized into Elasticsearch with Active Model Serializers, which causes two problems. First, when a document includes values from related models, updating a related model has to trigger reindexing of the parent document, or list views and filters will be wrong. Second, serializers often have N+1 queries, which can load the database heavily and slow the queues.
Kagen said inconsistencies can reach users if they are bad enough. Because list views come from Elasticsearch, a stale document shows an old value, and a new record might not appear at all, while the detail view shows current data. The UI then disagrees with itself. This can happen when indexing queues back up or when someone forgets to reindex from a related model. Kagen said users tend to notice quickly and report it. Model callbacks on save and update that run the related serializers help, but dependencies between models are still hard to track.
Sidekiq Batches, State Machines, and ParadeDB
Some flows are designed for eventual consistency, and the UI is blocked while indexing finishes. Kagen used supplier statement extraction as an example. A PDF statement arrives, AI parses it, and each entry becomes a new record in the database. During this, the statement is marked as extracting or matching. Sidekiq batches coordinate the follow-up work. For example, matching each entry to its policy runs automatically after extraction. When all jobs in the batch are done, the batch callback unblocks the statement for users. Extraction states are managed with a state machine using AASM. Eddie said these patterns are used throughout the application because the team relies so heavily on Elasticsearch.
Robby asked about alternatives to Elasticsearch. Kagen said the team has recently started looking at ParadeDB, a Postgres extension, but hasn't decided anything, and that it is a broader organizational decision. The appeal is that indexing would happen automatically when records are saved to Postgres. Kagen also mentioned that Modern Treasury, which does similar money and accounting work, uses ParadeDB for Elasticsearch-style work.
Do LLMs Work Better in Typed Languages?
Asked about using LLMs with Ruby, Kagen described a growing consensus that LLMs work best with strongly typed languages, because types provide guardrails and automatic checks. Kagen offered a counterpoint. Rails conventions and Ruby's expressiveness also suit AI well, because they give LLMs a ready-made set of rules that keep them "on the rails" at low token cost. In Kagen's experience, LLMs don't create abstractions upfront, plan for scale, or set conventions on their own. They do exactly what they are asked. Existing conventions fill that gap, and fewer tokens spent means better reasoning. Robby said he wants to see studies on this and that for now these are anecdotes, not science.
The Cost of Writing Code Versus the Cost of Good Software
Eddie said AI has made the team somewhat more efficient. One benefit he likes is that people spend more time explaining what they want to do and why, so intent is clearer upfront. He treats AI output as if it came from a colleague he is working with. He opposes committing code and hoping someone else reviews it. Whether you wrote it or a tool did, you are expected to understand it. He compared it to copying from Stack Overflow: you still had to understand the code.
Kagen cited a recent post from Ashby's engineering team on AI and the future of engineering. Its thesis is that the cost of writing code is approaching zero, but the cost of producing meaningful software is not. The skills that set good engineers apart, such as judgment, taste and understanding customers, matter more now. As Kagen summarized it, LLMs make it easy to skip the thinking, and because they produce plausible but slightly wrong code or wrong patterns, engineers need to think harder than before.
Robby said teams vary widely. He has talked to people who barely review AI-generated code, which he finds surprising, and others who are very cautious. Kagen said the answer depends on context. If being wrong is cheap, fully vibe-coding and shipping with little review can make sense, because you can fix it and ship again. If failure is costly and you need it right the first time, it doesn't. The Ashby post describes a spectrum: full vibe coding for low-stakes work like throwaway scripts and internal tools, which Ascend does use for internal tooling, and human-driven work with AI as an assistant when stakes are high. Kagen said the stakes at Ascend are often high because the software handles money for accountants who care about details. So the acceptable level of AI use varies even within one company.
Technical Decisions They're Glad They Made
Asked which decisions besides Rails have served the team well, Eddie pointed to minimal infrastructure. From the start, the team wanted to spend its time on the product. That doesn't mean ignoring infrastructure, but dealing with it when it matters. He summarized it as not doing "engineering for the sake of engineering." Managed services may cost more than running everything yourself, but without a dedicated infrastructure or DevOps person, simplicity is what lets the team keep moving. His example of complexity that is justified: the work on data ingestion and Elasticsearch now makes sense because customer volume grew, but before that it would have been unnecessary.
His second choice was enforcing standards and a testing culture from early on, meaning tests for everything, especially back-end code that handles money. He said this has prevented many production bugs while still letting the team move fast. The test framework is RSpec. Eddie called that choice "probably debatable," but said they have too much invested to switch. Robby, who has long used RSpec and recently tried Minitest on small projects, agreed there's no need for a religious war over it.
Robby, speaking from consulting, said he has seen apps with three developers, five repositories, Terraform, and no customers. Eddie agreed that every piece of complexity needs a reason.
For book recommendations, Kagen chose Structure and Interpretation of Computer Programs, the MIT introductory programming book that uses Scheme, and called it "a real nerds book." Eddie recommended John Ousterhout's A Philosophy of Software Design, a short book he said can be read in a day or a week and which he finds valuable for its ideas on software design.
Ascend doesn't have an engineering blog yet. The guests said they have discussed one but never started it, and Robby encouraged them to begin writing about the work they described.
Welcome to On Rails, the podcast where we dig into the technical decisions behind building and maintaining production Ruby on Rails apps. And I'm your host, Robby Russell, and I run Planet Argon. And for over 21 years, we've helped teams maintain and evolve their long-lived Rails apps. So, I tend to approach these conversations through that lens.
In this episode, we're joined by Eddie Galindo, who's an engineering lead at Ascend, and Kagen Hearn, a senior software engineer at Ascend. Ascend builds software that helps insurance agencies automate the movement and reconciliation of money. Today, we're going to explore how their team has evolved the Rails app from its earliest days into a platform supporting multiple products, increasingly large customers, and billions of background jobs, all while continuing to invest in our shared Rails monolith. Eddie and Kagen join us from San Francisco, California. All right, check for your belongings. All aboard.
Eddie and Kagen, welcome to On Rails.
Hey Robby, happy to be here.
Hey Robby, thanks for having us. Glad to be here.
We've had a chance to talk a little bit ahead of time, you know, over the last couple month or two in prepping for this conversation. So I'm really excited to kind of dig into things. So before we get into some of the topics I wanted to bring you on, first question I want to ask each of you is what keeps you on Rails? How about you, Eddie, first?
Yeah. So I think for me it was actually the first framework I used in the last year of college and I think right after graduating. So it's been sort of part of my whole career. I stopped using it for a few years and it was nice coming back and having it be the same. I think I kind of bought into the philosophy on Ruby and Rails and all of that. So I think it's just been part of how I've been building things forever. So a lot of the principles kind of resonate with me.
What about you, Kagen?
Yeah, for me it's a more recent introduction to my career. I started working with it in my last position at Midesk, but I really fell in love with it. I really appreciate how much Ruby on Rails and Ruby as a language lean into their own design principles rather than trying to hedge against them. It's very much a dynamic object-oriented programming language and it doesn't try to be anything else, and because of that it lends a lot of expressiveness to the language and very elegant designs.
And out of curiosity, what did you work with prior to getting introduced to Rails?
Yeah, I've worked with a bunch of things. I've done Node.js on the back end. I've done Python and Flask on the back end. And I had a little bit of a phase where I was doing some data engineering work which involved Python, Scala, a bit of SQL.
Oh, interesting. And Eddie, you mentioned you had started off in Rails and then you kind of went away for a little bit to do some other things. Where did you wander off to in the tech programming language world?
So I joined the big tech world. I used to work at LinkedIn and I was primarily just doing front-end work there, and then from there, another startup that I joined right after was just a Node.js shop. So it was all JavaScript and stuff.
What front-end frameworks were you working with over at LinkedIn? Was that like Ember there, or?
It was a lot of stuff. So when I first joined they were actually still in YUI, and then from there it was a little bit of jQuery, and then we migrated to Ember.js, and then at the time I was leaving I think there were also some conversations about going over to React. So it was a little bit of everything.
Okay. I think I knew a few people that kind of wandered over to LinkedIn at some point with the Ember crew that I think went over there at the time. So,
anyhow, that's a conversation for another day. So I've got a lot of things I want to talk with both of you about. But before we get into those things, I feel like we should probably provide a little context for our listeners for what is Ascend. Maybe, Eddie, you can give listeners a quick overview of Ascend and some of the problems that you're solving for your customers.
Yeah, so Ascend is a company that started around five years ago with the mission of building a financial platform for the insurance industry. So we specialize in all money-related problems specific to the insurance world. So we primarily work with insurance agencies across the country, trying to solve two money-related issues. We focus a lot on small and medium-sized agencies, helping them collect money and disperse those funds, and then more on the enterprise side we kind of facilitate workflows for them to operate their bank accounts and just money reconciliation, like commissions that they're receiving and stuff. So the big focus of the application is around money and how can we make you more efficient when it comes to dealing with those types of problems.
What's Ascend's business model? Is it like a broker in between that, or how does Ascend make money with that being part of that equation?
So it's a few channels. So one is we're a SaaS platform. So that's part of it. On our transaction product, it's really just on our financing book. So we help people also to finance the policies that they're purchasing. So that's part of it, and more on the money transactions side of things too. So we have a few different channels.
Roughly how large of an engineering team is there at this point in time?
Today we have around 20, which is probably the largest it's been.
Okay, nice. And you said it's about five years in development now as an application. So rails new was five, six years ago or so.
Yeah, it's been Rails since the start and we haven't moved away since.
Were any of you there around that period of time, or how soon after did you join?
I've been there since the start. Yeah.
What about you, Kagen?
So, yeah, I'm fairly new. I joined the team just a few months ago, right as our SaaS side of the product is hitting its hockey stick in terms of customer growth.
Interesting. So, it's kind of fun. You get the two different perspectives for our audience here. You've got someone that's been there at the beginning and someone that's kind of new. So, Kagen, if I recall from our conversation, you work more on the direct bill side of things. So, anything you can add to help further paint that picture for our audience here?
Yeah, so the SaaS side of the product is mostly tailored towards the accountants that work at the insurance agencies. So for direct bill, the insurance carriers that underwrite the policies will collect the premium and then they will pass the commission for that policy along to the insurance agency afterwards. They will receive the deposit for the commission asynchronously, and then they'll get a statement for it either before or after they receive the money, and they need to reconcile where all of the money comes from, from their deposits versus what's in all of the statements, and do some matching, and the direct bill reconciliation team helps with that.
Are either of you allowed to share anything in terms of scale, like how much you're processing on a regular basis in terms of number of transactions, or is that a little off the books at the moment for us?
I think the money stuff, we probably... but I think the number of statements, Kagen probably can mention a bit on that.
Yeah. We are processing, more recently, tens of thousands of statements in the past few months, and some of those statements can have tens of thousands of entries on them also that need to be reconciled. So I think the number of statement entries that we've reconciled has more than doubled just in the past few months, over this period where we're onboarding lots of new customers, and that number is going to continue to grow pretty dramatically, many more multiples, as those customers ramp up, because a lot of them are just getting started and there are also more in the pipeline. So the primary scaling factor is the number of statement entries that we're ingesting, or that we're processing, and that is growing by the millions.
Okay. Well, I know we're going to want to dive into some of those kind of fun scaling challenges you might be navigating. So I'm curious, before we get into that, since, Eddie, you were around at the beginning, do you recall why Rails was considered for this particular platform?
Yeah. So I think a big factor of it, one of the co-founders used to work at Instacart. He had Rails experience. It was like, oh, what's the easiest way to get started? So he bootstrapped the app and that was kind of it. And then from there, I think a few of the attributes from Rails that really helped us at the beginning was just the speed to actually having a product out. It was really easy to get the initial thing going, not having to spend time making a hundred decisions of like, oh, what am I going to use here, what am I going to use over here. At first, it was just really like, hey, what's the fastest way that we can put something out in front of customers and make sure that we're going down the right track, and focus more on the product side of things rather than making a hundred technical decisions that might not be relevant if we don't actually survive.
Sure. So it was a little bit of a familiarity thing, you know. There probably was more than just a familiarity thing, but were you recruited because you specifically had worked with Rails and they were trying to bring in people that had worked with Rails before, or what did that look like? I'm just thinking for anyone listening that might be thinking of starting a new startup venture and they're like, do we actually need Ruby on Rails developers to be able to build this Ruby on Rails application and platform? It's something I talk about with a lot of potential business owners or budding business owners, but what's your take on that? Do you think that's true, like you need to hire people with experience early on, or do you think you can let people from other skill sets come in and pick it up pretty quickly?
I don't think necessarily. If anything, actually, at the beginning it was probably like 30% of the team that actually knew Rails. It was more of like, hey, we used to work together at the previous startup. It was more of a people thing of like, oh, we like working with each other, so we'll continue working together. And then from there, I think Rails, for the people that hadn't used it, was easy to pick up. I think we've also tried to not deviate from your typical Rails app. So we relied heavily on the guides and just stuff that was out there, right? I think there's a lot of content that people can look up to familiarize themselves with the framework.
Sure. Kagen, since you joined more recently and came to an established Rails application, what did your onboarding experience look like? Did it look and feel like a pretty typical Rails application from your perspective?
Yeah, it really did. I was pretty impressed, actually, by how well Rails conventions have generally been followed. It was pretty easy to onboard. Wherever I expected to find things, I would generally find them. So I do think that we have benefited from staying mostly on Rails conventions over time, especially if you're somebody that's already familiar with Rails and you're onboarding.
Okay, that makes a lot of sense. I'm curious, Kagen, how much do you feel like the Rails guides have provided? You mentioned it was really helpful to you too, Eddie. Do you feel like there are aspects to the Rails guides that your team was able to quickly pick up on for most things, or how does your team decide when you're going to do something the Rails way versus coming up with your own innovative solution? Because we know that Rails doesn't necessarily provide all the solutions that an organization like yours might need.
Yeah, I think I can speak more on the things from the start. So I think we use the guides as a way of like, hey, somebody probably already thought through, oh, how do you do this? So this is the place where we start. And we actually haven't run into too many situations where it's like, oh, this doesn't exist in Rails. If anything, you can very easily find a gem. But I would say maybe 80, 85% of the work, you can just go to the guides and see how something is done in terms of the application. I think the interesting pieces are in figuring out the product side of things rather than, oh, how do you do validations, how do you send a JSON response back, right? Those things, we've been doing them the same way and we don't have to keep making that decision. So I would say the majority of things are kind of covered there.
Is this platform primarily a monolith? I know you work on separate teams, so to speak, within the same engineering organization. What does that structure look like today?
So we do have a monolith application for what we call our core platform. Because the product is meant to be a platform and you have customers using various pieces of it, that has kind of kept us on that monolith route, and it has also been way easier to manage than having multiple microservices.
I'm curious, because you mentioned Kagen coming in is working on a different part of the platform, I guess maybe the SaaS part of the platform. Were there conversations around splitting that up and having them separate, or was there enough shared? What's the rationale for not separating it, for anyone listening who's like, well, these seem like different teams, why wouldn't you?
At the foundational level, some of our modeling that we do, we do share a lot of concepts. So for example, we integrate with external systems and we run some normalization of data there, and actually the whole product benefits from that. Aside from that, there are other core components like your user management, your organization management, and all of that that are still shared. I think that's a big reason why they still make sense to be in the same place. And at the same time, I think it's also a lot of the infrastructure side of things. It's been way easier to manage, and I think for a small team that's pretty important, especially in the world that we're in, on the money side of things, where we have to be compliant and all of that. There aren't a hundred applications that we have to keep monitoring closely. So that part is also important. It's just easier to do, and I think we'll try to push it as far as we can. Once it gets hard to manage, we'll probably think about it again.
Sure. I'm sure it'll come up again, but how does your team establish boundaries at this phase of it? Because you're still around 20 people, and I don't know if you're going to be hiring more people in the near future or not, but how do you think about boundaries of which part of those two teams is responsible for what, or is it kind of a
little bit implicit?
Yeah. So at the product level, each team kind of owns their product, and they're responsible for that. More of the platform side of things, and things that are shared across, is really more of a team effort, or like, hey, we all kind of own it together and are responsible for keeping that clean. I think that eventually as we add more people it might break, but I think for now it has worked well for us.
What about for you, Kagen? As someone that's been primarily working within the direct bill side of things, does the shared monolith feel like a superpower or a constraint at this point?
I think it provides a lot more benefits right now than it does constraints. In my opinion, a lot of the benefits of microservices is organizational. It makes it easier to deploy changes to a specific team's domain independently without interfering with anything that other teams depend on, right? Like you can isolate dependencies and deploy independently for each team's domain.
But I think right now we're still at the point where we share a lot of the models, right? Like a lot of things in the insurance industry are common regardless of whether it's agency bill or direct bill or whatever other accounting task that you have going on, such as the policies, any lines of business, clients, etc. And all parts of the shop share those models and continue to benefit from being able to add to them in one place rather than needing to update them everywhere anytime you want to extend functionality.
Will the direct bill team be growing more as this, you mentioned this hockey stick growth type of thing, and is that part of the plans right now, or are you trying to stay kind of in a realm of... And how's the team divvied up? Is it like 50/50 approximately at this point, or is it just a few people working on one side of it?
So we're actually split up into five different teams today.
Okay.
For five kind of products that we have in the market, and the teams are pretty, I would say, evenly split. There might be some that have fewer people right now just because they're newer products, but they're pretty evenly split today. Yeah, I think we're looking forward to continuing to grow each one, because most of these products usually start up just tackling the surface, and I think as you go on and discover new things, there's always areas of more that you can go into.
Sure. That makes a lot of sense. So Kagen, in our prior conversation, you had mentioned that you'd been onboarding new customers on the direct pay side of things. So maybe some of them were dramatically larger customers. What sort of impact did that have on the platform that your team had started building already, and how much larger relative to prior customers are we talking?
Yeah. So on the direct bill side of the house, up until recently we had two relatively large organizations using the product, and the product was fairly hastily put together. I think, Eddie, what was it, about a year ago or so? One or two years.
Are you blaming Eddie?
Yes. No. Of course not.
Don't be so hasty, Eddie.
But there's this interesting dynamic going on where we're trying to ship at startup speeds for large enterprise customers. And as we've started to onboard lots of customers, some of the prior assumptions that were made started to break down under the pressure, especially ingestion scaling.
So in order to service direct bill customers, we need to ingest their policies, their lines of business, any of their clients, etc. from the agency management system, or AMS, that they work with. At the time that we started getting this deluge of new customers, we were able to process maybe 500,000 to a million new policies in a day, and the limit was just latency. It's a lot of external requests that we have to make. They take a lot of time, and each one holds a Sidekiq thread. We use Sidekiq for asynchronous processing. And each one holds a Sidekiq thread the entire time that it's waiting on the request. And so there's just a strict limit, with how many Sidekiq workers we had, to how much volume we could process in a given day. And so we needed to start driving that up.
Out of curiosity, what's an AMS, for anyone out there listening that doesn't know what that is?
Yeah, it's an agency management system.
Okay. And are these more modern platforms that you're integrating with, or are these older?
Most of them have been around for a while.
Okay. We're talking as old as like XML, SOAP services?
Yes.
Okay. So,
But SOAP services, XML, all that.
Okay. Any CSV transfers over FTP or anything like that?
I don't think we're ingesting anything that way.
Okay. Then you're not going too far back in the eras of data transfer between different platforms. So yeah, you've got all these background jobs using Sidekiq and they're happening asynchronously. Well, can't you just add more workers? What's the solution there? I mean, when you say you couldn't scale up more, is it just that it could only handle so many in a day?
Yep. So when we first started facing this problem, we weren't sure actually what the scaling boundaries were, because we had never tried to ingest more, as we had never really needed to. And so the first thing we needed to do was try to understand what the actual scaling limiters were. So during off-business hours, we would just kind of turn the knob up on the number of workers to see what would happen.
And the first limitation that we noticed was just the number of database connections that those would check out. Sidekiq will take a database connection for each thread the first time that it does any database work in that thread. And all of these workers are doing database work. And so we would start pushing pretty close to Heroku Postgres's connection limit pretty quickly when we started trying to up the number of Sidekiq workers.
Are you still hosting this on Heroku Postgres right now?
And did you come up with any interesting solutions to that, or is it something that you just had to figure out how to throttle at this point in time?
Yes. The standard solution to connection limits is just transaction pooling. So we brought in PgBouncer client-side as a sidecar for the Sidekiq workers specifically that are doing ingestion. We didn't apply it to all of our workers because, in order to implement transaction pooling, there are some Postgres features that you actually can't use, and it can be a bit of a heavy lift to actually comb through the application and make sure that none of those places are currently relying on it. So we introduced it in a limited capacity for ingestion workers, which really helped us out.
When you're processing all this, and say I'm trying to just walk through the workflows, let's say you're ingesting for a customer and going through their things. Are you able to then run another one for another customer in parallel, or does it then have to wait for the other one? How is that currently looking, or how did it originally look, and how have you kind of been evolving and morphing that?
Yeah, the way that it looks is we have a cron job that runs every 10 minutes that will poll each of our customers' APIs for any change events. So this will tell us which policies have changed in the last X time; the window is configurable, how far you want to look. And for any of those policies that we find that have change events, we'll create a record in our Postgres database to represent that change event.
And then we have another scheduled Sidekiq worker that runs on a cron that will pick up change events and then push them into the respective processing queues. So the policy sync workers, for example, this is the primary one, will be enqueued by this worker that picks up change events. And that allows us to limit the number of change events that we process at a given time. That's one lever to limit concurrency: how many change events this worker will pick up at a time. So we can stack up a backlog of change events and then process them as quickly as we're able to, or as we need to.
And then the policy sync workers will pick up those change events. Those have some of their own side effects. Those will sync transactions that are related to those policies, for example, as a secondary side effect. But all of the customers can run on the same shared resources at the same time.
When it comes to that workflow, out of curiosity, if you're polling every 10 minutes, what if the policy changes 12 minutes later? Are you having to cycle through? Is there anything about those policies where you're having to then go back and check things as you're actually processing that policy again, to double-check the data? Or do you poll and just know that, and then you fetch it and handle that transaction then, and it's safe enough that it can keep running 12 minutes later and it's not going to be an issue?
Yeah, it's completely fine for it to run 12 minutes later. The window that we look back is not 10 minutes. We look back further than that to give us some buffer room to make sure that we're not missing policies or missing any change events.
Sure. Sure.
We have a daily sync that will look back over the past day. And then over the course of the day, we also, every 10 minutes, will look back at policy change events from, I'm not sure what the latest is, the past hour.
I see. So you kind of have a little bit of a fail-backup situation happening just in case, to catch things. Hopefully catch things.
Yes.
And with you running all this, fetching all this stuff from these different systems, has that all been super smooth in terms of those platforms being able to handle the load that you're requesting from them?
Yeah, that's a great leading question.
Thank you.
What we have observed, after we solved our database connection problem with transaction pooling, the next limiter that we observed is that our users' own APIs are only able to handle so much ingestion traffic that we send at them. If we send too many concurrent requests at a time for policies, we'll start getting 504s, server failures. In the worst case, we might even cause them disruption on their own service if we're not careful.
How have you handled that? Just a lot of retries, or...?
Yeah. Well, we don't want to bring the customer down, so we can't just retry it.
Sure.
So we implemented a concurrency rate limiter, and this is inspired by a Stripe blog post on rate limiting from 2017, specifically one on concurrency rate limiting. They have a really nice GitHub gist that gives you a sample implementation, and we use that.
And the way that works is it puts a set in Redis for each individual integration that has outgoing requests, and each time that we want to make an outbound integration request, say to fetch a policy, that worker has to try to obtain a slot, and obtaining a slot is just adding an item to that set. Each organization has some configuration that tells how much concurrency they're able to withstand, so how much we can send them. And if the number of items in that set is equal to or greater than the limit they have configured, then the worker is unable to acquire a slot. And so it'll keep trying once every 100 milliseconds or so, and then eventually it'll time out and it'll just re-enqueue the job.
Interesting. What's been the kind of behind the scenes? At a high level, what does that look like from within your Ruby code?
Yep. So the one thing this forces us to do is make sure that all of the jobs that we enqueue are idempotent, because a job could fail to acquire a slot at any time in its processing. In any rate-limited task, we try to just put one request in the job, make sure it only has one side effect. These are general Sidekiq principles, but they're not always super easy to follow.
And so at the beginning of a policy sync, for example, we will attempt to acquire a slot. If we're unable to acquire a slot, then we'll just re-enqueue the job at the back of the queue. This helps make sure that other organizations are able to continue processing volume even if one is currently being rate limited. And then that will enqueue some side effect workers of its own, like syncing transactions, for example, that are also rate limited.
And because all of this is implemented using a Ruby class with a context block, it's opt-in. So requests that are executed from production user paths, for example, are not subject to the rate limiting. If we're doing something where the user is clicking buttons and that needs to make external API requests, that's not subject to rate limiting, but ingestion requests are subject to rate limiting.
And do you reserve a certain allocation of slots then for user-driven things?
So user-driven things actually aren't subject to the slots at all, because they tend not to push so much volume that they put the customer's API under stress,
Okay.
as much as the ingestion ones do. So the slots typically are only used for ingestion.
Okay. Interesting. And Eddie, when you're thinking about designing systems like that, how do you think about protecting... Are you using these types of functionality in other parts of the application as well?
No, I think today this product is probably the one that's pushing the boundaries on our application in terms of scalability and performance. So, kind of connected back to one of your initial questions, it is probably one of the cons of the monolith, like, hey, the different products might have different requirements. We're just living with the fact of, hey, we need to pay attention to infra on some side of the system, like we'll have to do it for everything. But yeah, I think this is probably the one that has been pushing it the most.
And I think another important aspect is it's probably things that we didn't consider at the beginning. I think everything that Kagen can describe is like, hey, we didn't start this way, we just kind of slowly got in there. If we take it back to the basics, I think what he described started really as, hey, it's just a worker that's going to make an API call and ingest stuff, and we just kept running and running until we kind of hit the limit.
Have you been able to work out with your customers and figure out what these... Do they make it very obvious what the limits are of how often you can be hitting their platforms? Or is it just trial and error, and you're like, well, we seem to be able to do this many and then we start getting a bunch of errors, so we'll just pencil in this number right now until we see there's an issue again?
Yep. Most customers' limits start at a fairly safe and conservative value. And then if we feel like we need to push more ingestion volume than what that conservative value allows, we will assess based on the size of the customer, whether they have on-prem infrastructure or they're using something that's shared, for example, to try to estimate what we can actually push through. And there's a few customers that are only able to take a very limited amount of volume as well, and so we actually have to scale them down.
Interesting. And it's not been as easy for you to convince them, can you just increase your side of the equation and add some more hardware over there?
Yeah. Just spend more money.
Yeah. You're trying to provide them a service, right? So you're having to do all this stuff to work around their limitations and, anyway,
maybe save that for another day. I can't remember which one of you was mentioning this, it might have been you, Kagen, but I think you mentioned that in the platform there's a lot of feature flag and maybe configuration complexity already. So is that true?
That's definitely true. That has become a lot more true over the past few months as we've been onboarding all of these large customers. This is, I think, natural to enterprise B2B contexts. Each customer wants their own version of the product. There's different things that they need. They all have their own accounting workflows and ways that they think about reconciliation, different processes. Some of them have acquired other agencies and that affects their processes as well. And so the set of features that we expose to each customer and the configuration that we allow them is different depending on which one it is.
So there's a lot of conditionals in your code to do more than maybe just swap out a logo.
We try to avoid continually adding them, but there are necessarily a lot of conditionals. Yes.
How do you see that scaling? If you're months into this process right now, and I've talked to a lot of companies that are years into the process and they find themselves with this tangled mess of, it's really challenging to debug things in different environments. And maybe part of that was because back when they started, they didn't get to leverage a lot of things like feature flags or things like that.
So are you using much in the way of feature flag type approaches to this? Is this just some big config files, or how does your team think about this? Thinking about the long term, but you're obviously trying to solve problems today for the customers you're bringing in, and you're still learning. I would imagine you probably don't have a clear vision of how this would look in two years at this point.
That's right. We have two primary levers, two mechanisms that we can use. We use feature flags pretty extensively, and the idea for those is gating actual product functionality. So what views are available to who, who sees which buttons in the UI, stuff like that.
And then we also have organization configurations, which live in our database, and this is to tweak functionality for products generally that everybody is on. So this is things like, for direct bill reconciliation, for example, we need to match the entries on a statement to the policy that the entry is for. And every organization has different rules for exactly how they want to match those entries. And org configurations allow us to provide a way for customers to say, we want to match it this way in this context and this way in that context, and allow everybody to have it the way they need it.
When you say you have things in your database, is this like one potentially growing god object at some point, or...
Yes, the organization configuration. Exactly. It is currently a large, growing god object.
Are these a bunch of attributes, like actual columns, or is this like a JSON? What does that look like behind the scenes right now, if you don't mind sharing a little bit of the spiderweb?
Yeah. It's columns on a database model. It's not a JSON.
Okay, thankfully.
So one of the things that adds a bit of friction and makes it harder to continuously introduce new organization configurations, discourages it a little bit, is that you need a database migration for each organization configuration that you want to add. And so each one is a database column on a model that belongs to the organization.
I see. Is this a multi-tenant application? And do you keep the data very separate, or are you allowed to kind of share that?
So organizations, their data is split up, but we still have a single database.
Okay, one more thing to add on the configuration. I think we are thinking of actually starting to split it up more by the domain that it's configuring, like, oh, we have, I don't know, payment configuration, or we have our reconciliation configuration, or for other products, so to ease the modeling there and not have one huge god object for configuration. And I think the decision for us on actually putting it in the database is because those ones are meant to actually be part of the product and how it works, and they're going to stay there forever, if that's the case.
That's interesting. How do you decide when to use a feature flag versus putting something in configuration? And what are you using for feature flags across...
For feature flags we're using LaunchDarkly today.
Okay. So yeah, I think it's kind of that principle: whatever is meant to be short-lived. As we're working with more enterprise customers, some things might require approval from them or a slower rollout, right? But eventually the goal is that everybody would get this feature, we clean up the feature flag, and the branching in the code is gone.
Versus the configuration model, where those are actually going to change your business logic. One example is what you were describing earlier, like, oh, you might want different granularity on when you're reconciling something and stuff like that, and that's just part of the actual behavior in your product.
This episode of On Rails is brought to you by Unsafe Save. The fastest way to skip validations and write questionable data with confidence. Are you tired of pesky validations getting in the way of your momentum? Just add validate: false and boom, your record is saved. Empty fields? No problem. Missing associations? Who cares? But wait, there's more. Act now and we'll throw in update_column totally free. Skip validations and callbacks. That's right. Bypass your business logic entirely. Update your database like no one's watching. Why wait for your code to behave when you can just skip the parts you don't care about? Unsafe Save. Because sometimes you just need that record in the database, consequences pending.
When you think about how the application's been evolving over the last few years, how do you decide, and this may be more directed at you, Eddie, when that customer flexibility can start to become, for lack of a better term, technical debt of some sort?
It's a good question. I think it's probably something that everyone struggles with. We do try to push back and think about the product in generic terms, right? Obviously you don't want to be building one-off things. So I think it's more about putting yourself in their shoes and then also looking at it from our perspective: is this something that makes sense in the product long term, and is it part of our process that you can standardize? Or do you actually work with the customer to say, hey, maybe this is another process you can follow, so we don't need to add this feature and stuff.
So I think it's kind of both ways, and having empathy for where they're coming from and why they're asking for something. With that, I think we don't have all the answers. So sometimes they might ask for a feature and we're like, oh, this actually makes sense, we need to add it. So it's probably not the answer everybody would want to hear, but I think it's really on a case-by-case basis, right? You just have to sit there and think through the requests, think about what makes sense for all of your customers, and where your customer is coming from.
No, that's true. I always think about how there's that balance, and I don't know if it's a sales-driven conversation, like, hey, we can land this new customer, and yeah, we're 90% of what they think they need, but there's these few new things, so we've committed to building these things. Well, how is that going to impact the rest of the customers we already have, will they need it or not, and how do you roll that out? Every one of those, I'm assuming, becomes its own interesting set of conversations and things to think about.
But do you have a pretty smooth workflow for, like, do you come in and say, all right, we're going to build out a new... First of all, is that assumption true, that to help bring in a new customer you have to change things a little bit for that customer? Is that true?
Yeah. So I think fortunately for us, most of it is actually a conversation, in some cases with engineering and product actually jumping in with the customer and trying to understand what they're asking for. So we're able to work through the features a little bit more and not just essentially be like, oh...
We promised...
We're trying to sell this, so you have to do it, kind of thing.
Yeah. So the engineers get to be part of those conversations, typically before the ink even gets signed, and you're like, this is what we can do, and we'll make some adjustments to make this work for you if necessary, or, oh, we don't need that right now, but this is coming down the road, this is on the roadmap for the next 12 months and we'll get there with...
Yeah, so a combination of both. We do try as much as we can to actually have engineers in the calls, so that you're close to the customer and you can actually understand what they're asking for.
Out of curiosity, just because I think a lot of people work in certain types of SaaS businesses that don't work with enterprise customers, where maybe you're selling a $30, $50, $100 a month subscription service, and there's maybe some tiers. Presumably you're not doing a lot of high-end customization for specific customers, because the idea is you're trying to get bulk customers, not necessarily larger clients. Did either of you work at SaaS platforms before working at...
I have.
Yep. Most of my experience is at different kinds of SaaS platforms. Yes.
And was it similar, where there was a lot of this kind of per-customer customization in those other environments as well?
Yes, in particular for the largest contracts, we would be especially willing to do more custom things. Although I think something that holds for those companies as much as it does here is that generally, before we go about agreeing to do something custom, we try to think about whether that's going to generalize well to use cases for anybody else in the future, whether this makes sense as part of the product in general. And if it doesn't, we do generally try to push back on super bespoke custom features that won't ever apply to anyone else. There are exceptions.
Sure. Sure. Money is an attractive thing to businesses at times. Depends if they'll pay for it, right? Or pay towards that, I suppose. I can appreciate that. But yeah, I was just always curious what that actually looks like in different environments.
How do you think Rails simplifies that? Do you feel like Rails is giving you enough tools to help work on these types of platforms, where you are customizing it in that sort of fashion, or do you think Rails should think about that a little bit more from the framework itself, to have some more baked-in tooling of that sort?
I think, maybe contrary to the expected criticism that Rails's convention over configuration makes it more difficult to do something custom, that's more true on the tech side than it is on the product side. Because of the ability to spin up something new fairly quickly relying on those conventions, it actually makes it a little bit easier to build out something bespoke pretty quickly as well, and the duck typing also goes a long way in that respect too.
Can you give me an example of where duck typing might come into play in that environment, for our listeners that might not be that familiar with the term?
Yeah, the duck typing helps with taking something that we have already built out, for example, that's shared across a lot of customers, and then we want to do something weird with it that is maybe oddly specific to a particular customer, but we can rely on a lot of the logic that's already been written. Because what duck typing essentially says is that the type of the object is its interface. It's what methods it responds to, rather than statically declared type information. And so if we're able to write some kind of adapter or interface to our already existing code that responds correctly to the pre-existing interface, we can leverage a lot of the work that's already been done on past projects.
And is that as much as just overriding a method to do something, and you can kind of rely on everything else that's already there?
Yeah. Overriding a method or writing a class that matches the interface of some other module that we already have that does most of the work we're looking for and expects that interface, for example. It really helps with hacky things like that in particular.
It kind of sounds like you have a positive favor. I've heard some people say this can feel a little too magical or a little weird to diagnose, for people coming in who don't understand how these things work.
Yep.
What's your take on that?
That's a real trade-off, especially if the module that you're writing is... You can definitely write it in a way that makes it harder to figure out what's going on, especially if you're not following Rails conventions. The conventions exist for a reason. They help you find where things are and reason about how things work when you follow them. If you don't, some of the magic, so to speak, can kind of take over and make it a little bit more difficult to reason about what's going on. But that's just one of the trade-offs that you make when Ruby and Rails are fully leaning into the design philosophy they believe in, which is duck-typed OO.
Eddie, do you have any thoughts on that as well, or...
Nothing to add.
Nothing to add. Okay. Well, let's shift gears a little. I want to pivot over to talk about infrastructure, because you mentioned that you're running on Heroku, and for listeners out there, maybe an obvious question might be: do you think there's a good chance you're going to try to stick around on Heroku as long as they provide support? I know there's a little bit of ambiguity about their long-term support right now that we've been hearing about from their blog posts.
Yeah, I think we've been as confused as everyone else. I think I saw it on Reddit: you released a blog post saying you're going into maintenance mode, but you keep releasing features, and people were confused. So I think we're kind of in the same camp. We've started exploring options, and I think we'll move if we have to. In some areas we're kind of hitting the limits, but we still have to do the homework of actually exploring other options. We use uniscale for autoscaling our Rails application, and I remember those guys said that, oh, we're
gonna look into all the vendors and kind of write up a blog post to suggest you something that's like Heroku, like, so I don't know if that has been posted already, but we're kind of waiting on that, but I think we'll also do our homework when the time comes.
So when we had prep for this conversation, you mentioned that you needed to do some tuning with Puma. Can you speak to that specifically within the Heroku environment?
Yeah, I think we actually took the time to probably take advantage of all the concurrency that you can do in Puma. So we were running into issues where we're kind of taking in requests and just sitting idle and not doing anything in the process. So we started to take a look into that a little bit more and actually leverage what Puma does for you, which is like, hey, you can handle requests more concurrently and scale a little bit better.
And that turned out to be a pretty easy fix. It was just a configuration change, like, hey, increase the concurrency, and all of a sudden you can have a little bit better performance, with the trade-off of, I think we also had to make some memory modifications, because then that would obviously start consuming more memory in, in our case, our Heroku dynos. So we also made a change there to start using, I don't know if I'm pronouncing it correctly, but jemalloc for Ruby memory management. So yeah, I think those two have been working good for us so far, and the change really helped us increase our requests per second processing.
How did you go about testing that out? Did you just tweak it a little bit and watch, or were you able to kind of simulate this a little bit in a different environment before you rolled it out to the production stack?
Yeah, so we were able to simulate it a little bit. So we have a sandbox environment in Heroku that's pretty much configured similarly to our production applications. So we were able to test some of these changes. I think we've made the mistake in the early days of, like, I think we actually tried to turn it on a while back and I think the application just stopped working. So we learned our lesson.
That happens. You learn, you figure it out, and sometimes it works. Quite often it worked. Are you relatively up to date with versions of Ruby and Rails and Puma?
Yeah. I think actually, thanks to the fact that we haven't deviated a lot from just your standard Rails application, I think the upgrades have been pretty easy and we're able to keep up to date. I think now with all the coding agents and stuff, it's pretty easy to, like, hey, can you bump to this version, then have it run tests, then we run our own verification just for our own sanity, and from there they're pretty easy to get through.
And out of curiosity, is your team kind of standardized on AI tooling at this point that you're using, or are you kind of experimental, everybody's trying different things at the moment?
I think more of the trying different things at the moment. I would say the majority are kind of centered around Claude Code, but yeah, I think that's still pretty open for the team, and we're also not enforcing, like, hey, you have to use this or that.
And for our listeners and watchers, we're recording this in the later half of June. So by the time this gets published, who knows where things might be. These things are changing quickly.
So I'm kind of curious. You were talking about your deployment workflows there. And so you're using Heroku, so it sounds like fairly straightforward deployment, and you've got sandbox environments. Have you built any interesting tooling internally to help you simulate realistic data in your local development environment, or what does that workflow look like, so you can have that sort of volume? Are you able to even test locally or in your sandbox? It's a pretty big volume of transactions you're bringing in. You mentioned an account or a customer might have 10,000 transactions in a single, I forget the word you're using.
A supplier statement.
Supplier. Yep. So, do you have tooling locally for that that you've been building out for yourselves, or a bunch of rake tasks, or what does this look like?
Yeah, we have seeding rake tasks. But we haven't, to my knowledge, Eddie, correct me if I'm wrong, we haven't invested tremendously in a lot of guardraily type AI-specific technology, such as self-verification stuff, or some of the things that some other organizations are working on to make it so that AI can fully develop and verify its own work. We haven't gone fully in on that. However, I think on a personal level, I've used it a lot for that purpose.
A good example of exactly just what you were talking about: a little while ago, I needed to migrate one of our CSV builders over to streaming from building a CSV file in memory, because the scale that we were hitting was causing some of the reports, CSV reports we were having customers build, they can click a button essentially to export a CSV of any list view that they're looking at, and the volume was starting to get to the point where building it in memory would become insanely slow or sometimes even crash.
And so I needed to migrate it over to streaming. I went ahead and implemented that and needed to figure out how to test it. And so I was just like, Claude, go open a developer Rails console and just instantiate a statement with a million transactions, and then we'll generate a CSV to see if the memory profile of the generator is stable over time and that we got a reasonable decrease in the amount of latency that it takes to process.
That's kind of a cool use case. What about, are you able to test things out in your sandbox environment before you push things out to production for customers when you're rolling things out? How much does that mirror your production data, if you're allowed to share?
Yeah. No, so all of the data there is just mock data, but we actually rely a lot more heavily on the tests that we write, especially on our Rails application. It's not so much seeding data and clicking around, primarily because the Rails application serves as a JSON API today. And I think when we're testing the boundaries more of data scale, I think we do those more as well, so like, oh, we have some scripts like benchmark this, and I think just measuring memory pressure in different places. But yeah, I think we rely a lot more heavily on making sure that we follow a test-first approach. So that's kind of what prevents us more from the bugs, rather than, oh, let's go into sandbox and test something out.
What are you using on the view layer then? You mentioned it's a JSON API. Are your customers using the JSON, or is that something you're using internally with a different front-end framework or something, or are you using Rails views at all?
So we use it both internally and externally too, because we also have an API that customers can use. So we use Active Model Serializers for that side of the view layer.
And what about when it comes to rendering out HTML and CSS? What's...
So that one, we actually run a Next.js application.
Okay. So you have Next.js involved as well. Is that for the customer-facing side of things, or...
Yes, they have a dashboard where they do most of their tasks.
Since that part of the application, what was the rationale for using that versus using what kind of comes with Rails at the time? Do you recall that?
I think a lot of that was actually during the beginning, from the engineer that was kind of selling the front end of things. It was more of the ease of, like, I think at that point in time, I don't remember if Hotwire was already out or not when we were starting that, but I think a lot of the dynamic work on the UI was a little bit more challenging to do, especially, I think, just finding engineers in the market that were like, oh, I think that the Rails, sorry, at that point on the UI was a little bit more involved. So it's what's easier to speed up and get something.
It's historically been a little complicated over the years, like, what's the new hot thing, and is this going to stick around, and what's the job?
It has gotten a little bit better.
It's true. But you're always kind of wondering, like, well, maybe enough people are using this. Well, hopefully. Yeah. You know, another thing you mentioned when we were talking is you mentioned Sidekiq. Could you recall approximately how many Sidekiq jobs your system has been processing to date, approximately?
Yeah, I think we're more than one billion. I think we're around 1.3, 1.4 billion. So I think Sidekiq is a really important part of our infrastructure. We rely heavily on it, and I think we liked it a lot.
That's great. And I'm assuming you're doing that on a Pro account.
I don't remember. I think we're actually on the Enterprise account by now.
Enterprise. Yep. Sorry. That's awesome. When you're thinking about, you mentioned using gems and stuff like that in the community earlier, does your team kind of have an ethos or approach about how you vet potential gems you might bring in, and do you feel like your thoughts on that have changed now? We're talking about the AI era now, where that becomes a potential risk of bringing in some external dependencies. So historically, what has been your ethos about when you decide to build something yourselves versus bring something into the repository, into your project, and throw it in your Gemfile? And then two, are your thoughts on that changing at all at the moment?
I think at the very beginning we were following more of the approach of, hey, whatever we bring in, it's still code that we own, even if it's not on our codebase. So you have to understand that you cannot bring a gem that's super complex for maybe a tiny little problem that you're trying to solve. So I think that was a big piece of, like, hey, it's still code that we have to maintain. Whenever we have to do an upgrade, we have to understand what's going on. If there is a bug there, we have to be able to figure out what's happening. So I think that has been a big piece of it. But I think with that said, we don't shy away from, especially, things that are already implemented, right? Like we use, for example, PaperTrail for our audit logs on models. Like, hey, we know it's something that has been around for a while, is stable, and we feel comfortable using it. So I think it's probably a combination of that, of like, hey, we're going to own it, and then, hey, is it something that other people are using, is it stable, and do you feel comfortable adopting it.
Have you needed to navigate, in these past five years, many gem migrations where maybe the support was discontinued and you needed to migrate to allow you to take advantage of some new functionality from those gems, or maybe it was a blocker for upgrading Rails or Ruby itself?
I think we've been lucky that every time we try to do something, there's already a PR open on the repository. So it has been mostly like, hey, we need to cut our own version until that gets released. So I think that has probably been the most common situation that we've encountered, rather than, oh, this is not supporting, I don't know, the latest Rails version or whatnot.
Okay, that tracks. Do you have many forked gems right now that you're maintaining? You don't have to disclose which ones they are, but...
I think now we've been able to cut it down to maybe just two.
Okay.
Yeah. I actually don't remember the number of how many we depend on today.
Speaking to that, the point about bringing in dependencies, whether that be, it even sounds like maybe on the npm side, how does your team think about bringing in dependencies? Are you having active conversations about that right now, like, hey, this is a way that we could be potentially at risk? Not that you weren't before, but it feels like things could sneak in a little bit quicker under our nose.
You mean from the existing ones that we use, or from new ones?
Potentially from new ones, or...
I think it doesn't happen that often, because we actually don't add new gems that much at this point in time. I would say the PgBouncer example that we were walking through, that was probably the most recent one. But aside from that, I would say the problems throughout the application are very similar. I think we've been able to kind of constrain ourselves to a handful of gems. I think with that said, with the ones that we have, it is a constant thing that's on our mind of, oh, how do we make sure that those ones don't get compromised. I think that's something that's top of mind, especially in the field that we're in.
I appreciate that. Kagen, from your side of the world, I'm thinking about infrastructure and decisions that are kind of affecting your day-to-day. One of the things in our prior conversations ahead of this was that you mentioned that you're needing to also maintain Elasticsearch, I think, as part of the equation with your Postgres. What are you using Elasticsearch for, and what sort of problems was it solving, and are there any new challenges that it has since introduced as well?
Yeah, that's right. We maintain two primary data layers. So we have Postgres and Elasticsearch. The reason we have Elasticsearch is, as previously alluded to, many customers have millions of policies. Supplier statements can come in with tens of thousands of entries on them. And most of the fields that exist, especially on a transaction, our users at some point will want to be able to filter or sort by that value. And Elasticsearch is really good for that on large data sets, especially because when it comes to supplier statements, for example, the number of those in production is just going to continue to grow unbounded. So we rely on Postgres for more detail view rendering, and Elasticsearch renders our list views and provides filtering, sortability, search.
Are there any interesting challenges with that? Like, if you have a pretty growing data set, how much of that data is getting indexed, and how do you keep that in sync?
Yeah, both of those are big challenges. The first one that's an evergreen, ongoing challenge is, when you're trying to maintain the same data in both Postgres and Elasticsearch, you need to make sure that you keep it in sync. We use Active Model Serializers to serialize the documents into ES.
And a couple of complications there. If some of the fields rely on other models in order to render their values, one, we need to make sure that when those dependent models get updated, the parent's document gets reindexed so that the list view and the filters and sorting are all accurate. And the other is, I'm sure you've seen lots of Active Model Serializers that are not very performant, lots of N+1 queries with relations and such like that. And when those things happen, it can put a lot of strain on the database. It can
slow the queues down. So we have to be very careful.
What would be the impact of it not being totally in sync right away? Like, is this a customer-facing issue or is it just it'll eventually get updated? Is it just primarily a UI issue?
It's definitely can be user-facing if severe enough, because Elasticsearch is used to render the list views. The worst case scenario is if something doesn't get updated, the list view will continue to render an outdated value, or if it's a newly created record, it just won't show it at all. And then when you click into the detail view for that, it'll show the updated information. And so the UI will disagree on the state of the world in two different places. This can happen if, for example, the queues get backed up for processing Elasticsearch indexing. It can happen if we forget to reindex a document somewhere that's important, like on one of those dependent models, for example.
Do you have any ways to convey that to the user if things are a little out of sync? How would you know, or how would they know, unless they raise a support request or something like, hey, I noticed that the dollar amount or whatever, when I did a search, was this and it's showing a different number here? And I would imagine there's a little bit of a versus, like, the data changed and they just happened to search immediately at the same time. Is that pretty obvious, or how would you know that that's happening?
Yeah. I think whenever there is a failure in this regard, it can be visible to end users pretty quickly. They'll notice it, they'll report it to us, and we will know. Because in some contexts, for example, we're extracting a new supplier statement and all of its entries, there's an expectation of eventual consistency. And so there's a little bit of a grace period that we're allowed before items populating into the list view will show up. We'll generally try to block UI elements when we're waiting on something to reindex.
What's that workflow look like? Is it just like it knows that it's going to be rebuilt and so it just waits, or
Yeah. In the case of a supplier statement, when we're extracting, extraction for the record is a PDF document comes in and we use AI to parse the document and extract all of the individual entries that are on that statement and create new models in our database for each one. There's a record for each transaction, etc. And while that's completing, we mark a status on the supplier statement that says that it's currently extracting or matching. And we use Sidekiq batching to orchestrate all of the things that happen as a result of extraction. So matching, for example, will happen automatically after extraction. We'll try to match each of the entries to the policies that they belong to. And after all of the Sidekiq jobs have completed, the Sidekiq batch callback will fire and it'll unblock the supplier statement for user viewing.
Okay. Are you using any sort of state machine approach to processing all that, or
Yeah, we do. For extraction in particular, there's state machine transitions. We use, what's the module called? It's AASM for short, I forget.
Acts As State Machine.
Yes. Okay. How does that kind of match up from your perspective, Eddie, when you think about product requirements and things around architectural decisions? Do you already have patterns like that that you're already using elsewhere in the codebase, or are these kind of newer things that you've been adopting as a team?
No. So I think those patterns are pretty central across our application, especially because we do depend heavily on Elasticsearch. It is something that we keep an eye on so that the UI is actually reflecting what the customer is expecting to see; otherwise it creates a bunch of confusion on their side and on our side as well. So yeah, I think those patterns are just shared across the application. Yeah,
we have some patterns that help us enforce that, like callbacks on our models, for example on save or on update, that will run any related ES serializers. But it's still tricky when the document depends on other models for some of its values to make sure that when those change, any necessary documents that depend on it are also updated.
That's challenging. Out of curiosity, have you looked at alternatives to Elasticsearch since you decided to first use it? I'm not saying I know something off the top of my head, but this seems like an interesting challenge that other teams might be faced with. So if anyone's listening out there and has got ideas and wants to respond or write something in the comments there. But is this something you're actively previewing, or is it just like, no, this will probably be fine for now? I'm just thinking, you've mentioned hockey stick growth, data consumption. So that sounds like a pretty big index to be managing.
We have. Yes. Recently we started looking at ParadeDB, which is a Postgres extension. We haven't made any decision to migrate or anything like that. It's a broader organizational decision, but it's something we started looking at because it's a Postgres extension, and so you get the indexing, the document reindexing, for free as part of just saving records to Postgres in the first place. Something interesting about it is that it's also relied on by Modern Treasury, who do similar money accounting workflows, and they rely on it for Elasticsearch-type work as well.
Interesting. I'm not familiar with that one. I'll have to check that out. You always learn something new. I'm kind of curious, one of the other things I wanted to come back to: you mentioned a little bit earlier around using AI, and Kagen, you specifically had mentioned that you're using AI a bit more. How would you describe it from within the realm of using it specifically working on Ruby in particular, when it comes to using these LLM tooling?
Yeah, I think there's a consensus opinion developing that AI models, LLMs, tend to work best in the context of strongly typed languages because it provides more guardrails and automatic verification for the AI to check its work. But I think on the other side of that, the conventions, the expressiveness of Ruby and Rails also lend pretty well to AI usage because they provide a ready-made set of rules for LLMs that kind of keep them on the rails that are also token inexpensive. As anybody that has worked a lot with LLM coding knows, they are not very proactive about developing abstractions upfront or thinking about how things should scale into the long term or setting conventions. They will do exactly the thing that you ask them to do to solve the problem at hand. And Rails and Ruby's conventions and expressiveness help keep the token expenditure down so that the reasoning stays better, and also provide a good set of conventions that are already made for it to rely on.
We definitely have been hearing that more as of late. I'm looking forward to seeing some more studies on this so we can say this with some confidence, like we know what we're talking about. And I know there's been some useful anecdotes being shared out there, so I want to take that with a grain of salt for anyone listening. This isn't a scientific perspective, but something where we're trying to read the tea leaves as best we can here. And Eddie, what's your take, what's your perspective here? How is AI tooling changing your workflows over at Ascend?
I think it helps with being a little bit more efficient with the task. And I think actually one great output from all of this is that people are kind of spending more time on expressing their thoughts of like, oh, I want to do this, and this is why, and this is what I'm thinking. So I think that part has been great because you have more of that clarity up front of like, hey, this is what I'm trying to do. On the other side, I've been following more of the approach, and my opinion is more like, hey, whatever output comes from AI, I still consider it the output of the peer that I'm working with. I think I'm kind of against, like, oh, I'll just put some code there and pray that somebody else will review it for me. So that part I'm still mindful of: whatever output is coming out, I don't care if you wrote it yourself or if you used a tool to write it, I still expect the person writing it to understand it and know what's going on and all of that. I kind of see it the same way as back in the day when you would copy something from Stack Overflow. You still have to understand it. You can't just copy paste and say, I'm done, and move on.
There's a blog from Ashby about AI usage, actually, that really resonates with me on philosophy about AI usage in software development in general. It's from this month and it's called "AI, Ashby Engineering, and the Future." And the way that it positions it, the thesis is that the cost of writing code is heading towards zero, but the cost of producing meaningful software is not. And that the differentiators for good engineers, which are judgment, taste, customer understanding, those things are becoming more important rather than less important. And we have to think more and think harder than we did before, because LLMs make it very easy to avoid the cognitive burden of doing the work. That's some of the appeal sometimes, being able to offload the thinking part. But when LLMs are producing very plausible but not quite correct, or not quite the right pattern, code, it requires us to think harder and better than we did before. And I really appreciated that framing.
Any follow-ups on that, Eddie?
No, I think it's going to align with my thinking as well.
I think it's an interesting time right now for teams, and I've interviewed a number of people in some other upcoming episodes that'll be published in the near future, or I guess actually before this episode, but I was always trying to track where different teams are at. You talk to some teams and they're fully like, oh, we're just letting the, as Kent Beck is now calling them, the genie... he wouldn't say this, but I've talked to some people where AI is cranking out the code and they're barely even reviewing the code anymore. And I'm just like, that's wild. What kind of guardrails do you have in place that you feel so confident about that? It's just interesting. And some people are like, we've got to be super careful. I'm not saying that that's you, but where is... and maybe everybody's reality is reality? And that's okay. And maybe we're always working a little bit differently. I don't know. I'm trying to be optimistic and skeptical at the same time, and it's an interesting place to be, and
I think there's a lot of nuance in it that's not always acknowledged by either side having the conversation. It's very context dependent. If the cost of being wrong is low, then it makes a lot more sense to just fully vibe it out and have the AI ship the code and barely review it, because if it's wrong, then you can just iterate and ship again. But if the cost of failure is high and you need to get it right the first time, it makes a lot less sense to operate that way. And the article that I mentioned, the Ashby engineering one, also frames AI usage on a spectrum: full vibe coding for low stakes tasks like a throwaway script or some kind of internal tool. And we do this, we use Lovable for internal tooling, because the cost of making a mistake is fairly low. But on the other side, when the stakes are high, I think human-driven with AI as more of a sidekick makes more sense. You can have AI type out some of the code, but you can review it very carefully, think through the problem, make sure you're using the right abstraction, etc. In our context, a lot of times the stakes are high because we're dealing with money. Our customers, our accountants, they care about the details. And so even within the same company, you see the different spectrum of what's acceptable.
I think that's a good place to end the AI topic. I appreciate that. I'll find that link and share that in the show notes, as you said, AI Ashby, or I'll have you send that over to me and we'll include that for everybody in the show notes. I've kept you guys long enough, so I want to start working towards a few last things I'm curious about, and this may be for you, Eddie. You talked earlier about coming back to Ruby on Rails as part of the secret sauce of Ascend's success to date, and its ascent, so to speak. What other technical decisions do you feel like your team's made that you're really glad you did, aside from Ruby on Rails obviously? Is there something else you were able to do early on that you feel has been a really good anchor for the organization on the engineering side of things?
I mean, tied to the Rails stuff, on the infrastructure side we've also tried to keep things minimal. From the very beginning our main focus was, hey, how can we spend more time thinking about the product and building stuff? So we try to minimize the extra stuff as much as possible. And it doesn't mean, oh, we're not going to pay attention to it, it's just more like we'll pay attention to it when it matters. So we operate a lot under that principle of it's not engineering for the sake of engineering. It's more like, we have a product and a customer base to serve, so we want to dedicate our time to that. So a lot of the infrastructure decisions come with that. You have the trade-off that some of them might be more expensive than what you'd pay if you roll out everything on your own, but at the end of the day, you don't have a dedicated infra or DevOps person, so we have to keep things simple so that we can keep moving forward. So we operate a lot under that principle, and also on enforcing standards. I think that was something that we tried to do at the beginning, like, hey, we want things to keep a cohesive approach. And with that also came the testing philosophy of, hey, we want to start writing tests for everything that we do, especially on our back end when we're dealing with money and all of that. So I think that's been great with the team, and it's something that we keep up until this day. And it's something that has also saved us a lot from just having bugs in production and stuff. But I think it just allows you to move with velocity at the same time.
That's great. Out of curiosity, I don't think you touched on this earlier. What testing framework are you using within your Rails app?
Today we use RSpec.
Okay, using RSpec. All right.
I think that decision is probably debatable, but
Okay.
have enough, so we're sticking with
Well, I've heard some people have been using their LLMs to migrate, and putting that like, we'll just offload some of that stuff. I don't know how conf... I guess if you can find some
good parity there, that would be not the bad, not a bad thing necessarily. But I'm a longtime RSpec user and fan myself, but I've been using Minitest more recently on some smaller projects, and like, I could do this, I can probably get behind this as well. But we don't need to have a religious war about tabs or spaces or RSpec and Minitest there.
But no, that's useful, and I think your point there around maybe intentionally not having certain types of roles, maybe around infrastructure and stuff like that, and using something like Heroku and trying to use that as an interesting constraint for your organization. Maybe there's a tendency to kind of like, well, we'll add more complexity to the infrastructure if we have people that know how to do that, and maybe there's a really good reason for it, but then there's the cost of not maintaining that, and maybe people on your team may or may not have the time and bandwidth to learn all those things to also be able to participate in some of the infrastructure management that might get involved in that. Is that a safe kind of read on what you were saying there?
Yeah. No, I think that's right. I feel like all of the things that we do need a reason of, like, oh, why are we doing it? But some of the problems, like for example on the data ingestion, Elasticsearch, is like, hey, we have a reason because we're now getting more customers, so now it makes sense for us to spend time there. Versus before, if you had very little traffic, why would you have a super complex set of infrastructure when it's not actually necessary?
As someone that works in the consulting space where we get brought into projects, there have been many, many projects where there'll be three developers that had worked on an app and there'll be five repositories, Terraform involved, and I'm like, you've got no customers. What are you doing? So that's a thing.
All right. Well, with that, a couple quick questions. We'll start with you, Kagen. It's for both of you, though. Is there a programming book or technical book that you find yourself recommending to peers?
Oh man, tough question. I think I need a second.
I do not, not at the velocity that I wish that I did, but I do read them. One of my favorites is the classic, I'm forgetting actually the title of it, embarrassingly. But it's an introductory programming book through the perspective of Scheme, from an MIT course. I don't know if that rings a bell for you, but that's a personal favorite of mine. Eddie, do you know the title of that one?
I know, but I don't know the title.
What about you, Eddie? Do you have a
I read, I think a few years back already, A Philosophy of Software Design. It's a pretty small book, but has a lot of interesting pieces. One of those that you could probably read in a day or even in a week. It doesn't have a lot, but I think it just has ideas around how to design software that I find pretty valuable.
Structure and Interpretation of Computer Programs. That's the one.
Okay. It's a real nerd's book.
Okay. Real nerd's book. All right. Well, great. We'll definitely include links to both of those in the show notes. And out of curiosity, does Ascend have an engineering blog or anything at this point, or have you talked about that at all, to share what you're cooking up over there?
We've talked about it. We've never done it, but yeah, probably something that we will look into. Yeah.
Maybe you can have one of your AI bots build you one, and then your first post could be like, "Hey, we were on the On Rails podcast, and you can check out the video. Here's some of the things we talked about." So I'm giving you your first blog post right there. [laughter]
But I think, yeah, you folks are doing some really interesting stuff there that I think the community would benefit from hearing more about. So I want to thank you both for taking time to come on On Rails and talk shop with us and share some things that are going on behind the scenes over at Ascend. I really appreciate that. And on behalf of our audience, thank you.
Thanks for having me on.
That's it for this episode of On Rails. This podcast is produced by the Rails Foundation with support from its core and contributing members. If you enjoyed the ride, leave a quick review on Apple Podcasts, Spotify, or YouTube. It helps more folks find the show. Again, I'm Robby Russell. Thanks for riding along. See you next time.
Article published