Agents on Rails: Building a Proactive AI Pipeline from Plain Rails Jobs
Ruby on Rails
Daniela Velasquez's talk asks how to put agents "on Rails," and also how to keep them on the rails once they start producing output. Her position is that you don't need an extra framework. A proactive AI pipeline can be built from ordinary Rails pieces: a background job, a few database records, a prompt, an LLM provider, and some tools. The harder work, she argues, is making the agent's output trustworthy and letting it learn from users over time.
Why go to the user instead of waiting for them
Velasquez starts with the motivation. Many applications hold large amounts of data with interesting patterns in it, but nobody has time to go through all of it and relate it to what is happening in the world around it. Dashboards and chatbots are existing attempts to address this, but both wait for the user to come and ask. She proposes going one step further and taking the insight to the user. Her reasons: people "don't know what you don't know," you don't want users to have to think so hard, and some ideas are hard to track by hand.
Her running example for the talk is a hypothetical app that tracks an athlete's workouts, sleep logs, and heart rate. The goal is to coach the athlete before they ever have to ask for help.
The researcher: a job, a prompt, a provider, and tools
"It's all going to start with a job," Velasquez says. The first piece is a ResearcherJob, a simple Rails job whose subject is an athlete.
The next piece is the prompt, which she calls "the heartbeat of the LLM." Her example prompt tells the model it is a running coach. It should study the athlete's workouts, sleep, and heart rate, propose concrete training changes for next week, and back every change with data it actually looked at. She notes the prompt can be as long or short as needed and should be tailored to the application.
Then comes an LLM provider. She mentions Google, Anthropic, and GPT without recommending one, and suggests picking whatever you want and can afford, "cuz they are expensive." The prompt goes to that provider along with a set of tools: workout history, sleep log, weather forecast, and web search. The agent isn't required to use these tools. It decides when each one is useful. Velasquez points out that the toolset deliberately reaches beyond the app's own data. Weather and web search let the agent look at the world around the athlete and connect it to the internal metrics.
The last two settings are the model and the reasoning level. She describes the model as "the brain you're trying to rent," where bigger brains cost more. For the researcher she argues you don't need a fast model. You want a big, slow one, because it is doing the real research. Reasoning controls how hard the model thinks before answering, and in her framing, higher reasoning gives you more confidence in the result.
The deploy problem and persisting the conversation
That gives you a researcher agent living inside a job, calling tools, reading data, and working toward an answer. Velasquez warns that this job can run a long time: 10, 15, or 30 minutes, depending on the number of tools and the prompt. That creates a practical problem. Most teams deploy continuously, and a deploy restarts the job, so the agent goes "back to zero." Calls to LLM providers are stateless, so nothing from the interrupted run is remembered. You lose time and money, and, she adds, "your bosses are probably not be really happy."
Her fix stays within Rails. The researcher job now receives a conversation record as well as the athlete, and that record tracks every message exchanged with the provider. If the conversation has no messages yet, the job stores two: the prompt with the system role, and a user-role message asking the model to do research on this athlete. The job then enters a loop. It calls the provider with the tools as before, but it now sends the full list of conversation messages instead of just the prompt. Every response is saved back into the conversation. If the job restarts, it picks up with the full history of what was already exchanged. This also preserves the multiple messages the provider returns during its thinking process. The loop needs a termination condition so the job knows when the agent is done and can exit.
What the researcher produced
To show what the working agent might return, Velasquez plays "pretend" with four example suggestions:
- Tuesday runs follow the athlete's worst sleep, so start them an hour later.
- Add two 45-minute bike rides this week to build aerobic base.
- The long run starts too fast and fades, so ease the first kilometers.
- Heavy rain is forecast for Saturday, so move the long run to Sunday morning.
She notes that some of these come purely from the app's data, while others probably come from the web search or weather forecast tool. "That was kind of easy, right?" she says. You let an LLM come up with ideas and show them to users.
Falling off the rails: the "Wawa shampoo" story
Velasquez then turns to trust. LLMs, she says, sometimes find very creative ways to do something other than what you asked, and "half the time I feel like they're just trolling you."
She illustrates this with an experience from when her team first built this kind of researcher at Shopify. Reading through the suggestions it generated, she found one that looked harmless: bundle "Wawa shampoo" and "Wawa conditioner" because they are usually bought together. Then she saw the same suggestion again and wondered if she was missing a trend. When it appeared a third time, she checked what the merchant actually sold. It was bras. Digging further, she found the same suggestion across many shops. The source turned out to be the prompt itself. The Wawa bundle was an example in it, and the researcher had apparently decided that whenever it ran out of ideas, it would hand back that exact example as a suggestion. For Velasquez, this is how agents inevitably "fall off the rails," not in the Rails sense but in the "making sense" sense.
LLMs as judges
Her answer to the trust problem is to "throw more LLMs at the problem," but in a narrower role. These LLMs act as judges of the researcher's output. Each judge is basically a classifier that answers one question your application cares about. In her example there are three: one catches made-up numbers, one checks that the suggestion is kind, and one does a reality check on whether the athlete can actually do what is suggested. She acknowledges the apparent irony of adding three more LLMs after admitting one can't be trusted. She also says the number of judges is flexible: you can have more or fewer than three.
The trust, in her approach, comes from evaluating each judge against a labeled dataset. Using the kind judge as the example, she suggests writing 200 examples of kind suggestions and 200 unkind ones. Her samples include a gentle swap of an easy run for a chill session (kind); "Seven runs in 7 days, stop before you break yourself" (not very kind); "You've run with the same shoes 900 km, no wonder your knees hurt" (not kind); "Your pace drops after 60 minutes, grab a snack at the 45-minute mark" (kind); "Cramps show up past 15 km, carry water on the long ones" (fine); and a version telling the athlete to drink water "like an adult" (not kind).
Since you know the correct label for every example, you can write a judge prompt and measure its accuracy. Her kind-judge prompt says roughly: you review training suggestions before an athlete sees them; answer one question, would a supportive coach say this? Blame, sarcasm, or threats fail; honest data delivered kindly passes; reply with pass or fail and why. Running the prompt over the whole dataset shows how accurate it is. She points out that this setup lets you compare different prompts and different models to find what fits. Judges should be smaller, faster models. You don't need the most expensive model or high reasoning for this.
Running the judges as the pipeline's second stage
Once the researcher finishes, the second stage runs every judge on every suggestion. A judge job takes a suggestion and sends the provider two messages: the judge prompt and the suggestion to evaluate. It uses a small model with low reasoning, because "you want them to be cheap and fast," and saves the verdict.
Applied to the four example suggestions, the kind judge might flag the Tuesday suggestion because telling the athlete it's their worst sleep isn't very kind. The reality-check judge might flag the bike rides because the athlete doesn't own a bike. Two suggestions fail and are withheld, so only the two remaining ones reach the athlete. In her recap, the athlete's metrics go to the researcher, the researcher produces personalized ideas (four in this case), and a second batch of jobs runs the relevant judges on each one. Each judge gives a thumbs up or thumbs down on its own metric, and the user sees only what passes.
Capturing user feedback
Velasquez then adds the user's reaction. Users "are going to have thoughts and feelings, especially after you tell them that this was AI-generated." Her example: the athlete replies, "I work on Sundays from 6:00 a.m. to 3:00 p.m." No amount of tooling would have given the researcher that fact, but it should know it the next time the pipeline runs.
The implementation is a simple ApplicationRecord model for feedback that belongs to an athlete and to a suggestion. The researcher job then adds this feedback to its prompt, telling the agent what the athlete has said before so it can learn from it.
Why raw feedback shouldn't go straight into the prompt
She warns against putting raw user feedback into the prompt. LLMs "really thrive on patterns." The example athlete was polite, but users get frustrated, and some people resort to name-calling with agents. The kind judge would catch hostile suggestions downstream, but you don't want to spend tokens on a researcher that has absorbed the "bad vibes" and produces angry suggestions that only get filtered out later.
Her fix, again, is another agent. Raw feedback goes to an agent that strips out the noise and keeps only the fact. However the user expresses frustration, the system should remember only that they work 6:00 to 3:00 on Sundays. On the Rails side, she uses a callback on the feedback model to enqueue a job, adding that although many people dislike callbacks, she loves them. The job looks like the others. It takes the feedback and sends a prompt saying, in essence, that the athlete rejected this training suggestion, and asks for one sentence on what the coach should remember about this athlete so it doesn't happen again. It also passes the original suggestion. This is another small-model, low-reasoning call. The result is saved as the feedback's processed text, and the researcher job includes that processed text, not the raw feedback, in its prompt.
The open problem: keeping learned facts current
Velasquez ends with a question she doesn't claim to have solved. What happens when the athlete changes jobs and no longer works Sundays? This is where she sees agents entering a "dreaming" kind of phase: deciding whether facts extracted earlier still hold over the long run. She presents it as a challenge everyone building these systems will face as more information gets captured through LLMs.
Her final recap describes the full loop. A researcher proposes workout actions, judges check them against what the application cares about, the athlete sees what passes, and the athlete's feedback is processed and fed back so the agents keep learning. How to keep that learned knowledge up to date remains open, but she says that for the pipeline covered in the talk, "that's it."
In a closing aside, Velasquez says she generated all the slide images with AI. There were some odd results along the way: one image came out scary and weird, a request for "agents" seemed to produce police officers, and the image tool refused to remove a large figure from one background.
[music]
Hello everybody. This is Agents on Rails, and I want to talk about how you can build AI pipelines using plain Rails. We have Ruby on Rails, but how can we put agents on Rails? And before we dive into the how, let's think a little bit about the why.
We know that a lot of applications have tons and tons of data, and within that data there is very interesting things that you can find. But nobody really has the time to sit and look at all this data and make sense of it with what is in the data and what is around us. And we have come up with cool ideas to try to figure this out, like dashboards or maybe chatbots, where you can just connect all that data to your applications and see what happens.
But what if we take a step forward and we don't wait for users to have to come to us, and instead we go to them? Because sometimes you don't know what you don't know. And we don't want them to have to think as much, and sometimes there might be ideas that are hard to track.
We're going to pretend for the exercise of this presentation that we have an application where we're tracking an athlete's workouts and their sleep logs and their heartbeats, and we're going to try to coach them and help them before they even have to come to us.
It's all going to start with a job. We're going to have a researcher job, which is a simple job. And what are we doing research on? An athlete.
Next, we're going to have a prompt, which is the heartbeat of the LLM. In our case, we're going to say, "You're a running coach. Study this athlete's workout, sleep, and heart rate, then propose concrete changes to the training for next week. Back every change with data you actually look at." You can make this as long or as short as you want to, and obviously tailor it to your application.
Then next, you're going to need an LLM provider. There are many out there. There's Google, there's Anthropic, there's GPT. Not trying to promote anything here, but choose whatever you want and whatever you can afford, because they are expensive. And to that provider, you're just going to pass your prompt.
Then you're going to give it access to some tools. In our case, we're going to have a workout history, a sleep log, a weather forecast, and a web search. These tools are not mandatory for the agent to use, so they can come up with ideas on where they might suit you better. They might suit them better to be used. And as you can see, we're not just looking at data within our application, but also giving it access to things outside the application itself, so we can look at the world around this athlete and make connections with that data.
Next, we have a model, which is basically the brain you're trying to rent. And some brains are bigger and more expensive than others. And in this case, you really don't need a big, fast brain. You just need a slow one, but a big one, because you do want it to be really thinking about it, because it's doing all the research.
And then you're going to come up with the reasoning, and this is how hard you want it to think before it starts coming up with answers. And the higher the reasoning, the more confidence you're going to have.
Now we have a researcher agent who's sitting in a job and is doing all these calls, looking through the data, calling tools, and preparing to give you an answer. But this job is going to take a very long time. It can be between 10, 15, 30 minutes, depending on how many tools and the prompt is.
What happens when there is a deploy? As soon as you make a deploy, which I'm assuming most of you do continuous deployment, your agent is going to go back to zero because it lives within a job. And it will go back to zero. And because the calls with the LLM providers are stateless, they won't remember what was happening. So not only did you waste a lot of time, you also wasted a lot of money, and your bosses are probably not going to be really happy.
How can we fix this? Let's go back to the researcher job. And we're going to still have our athlete, but we're also going to have passed a conversation, which is going to help us track all the messages we've exchanged with the LLM providers.
We're going to have a prompt as we did earlier, and we're going to have this. If the conversation has no messages, we're going to store the first message as the prompt with the role of the system, another message as if you were a user and you're telling, "Hey, do research for this athlete."
Next, you're going to go into a loop, and you're going to do exactly as we did before. We're going to call an LLM provider with the tools, but notice that we're not passing just the prompt. Now we're passing all the messages from that conversation, so that if your job needs to restart, it can remember all the messages that had already been passed back and forth. This also allows you to store multiple messages the LLM provider has given you throughout its thinking process.
And obviously, every time you get a response, you're going to save it into your conversation messages. And you need a way to know when you're done so you can come out of the loop and finish your job.
Now that our agent is actually working, let's play pretend what the agent might have come up with. "Tuesday runs follow poor sleep. Start them an hour later." "Add two 45-minute bike rides this week to build aerobic base." "Your long run starts too fast and fades. Ease the first kilometers." And "Heavy rain forecast Saturday. Delay your long run to Sunday morning." This all seems pretty good to me.
As you can see, it's doing some ideas just looking at the data, and some other ideas are coming maybe from the web search or the forecast tool that it was using. That's how you put agents on Rails. And that was kind of easy, right? You let an LLM come up with ideas, and then you show them to your users. But as many of you probably know, it's hard to trust them.
They sometimes come up with very random ways of doing, very creative ways of not the right thing that you asked them to do. And honestly, half the time I feel like they're just trolling you for what you're asking them to do.
When we first started doing this at Shopify, I remember going through the data of the suggestions the researcher was finding, and I found this very interesting one that at first didn't look that interesting, which was, "Bundle your Wawa shampoo and Wawa conditioner because they're usually bought together." That sounds fine. And I continued reading, and I found the same suggestion again, and I thought, "Huh, maybe I'm missing on some sort of trend with this Wawa conditioner and shampoo." Who wants Wawa on their hair? I don't know, but to each their own.
And then I continued reading, and then I found it again. And so I got curious, and I went to look what this merchant actually sold, and they were selling bras. And then I continued looking, and this was a problem throughout the entire researcher doing it for multiple shops.
And then I was trying to figure out how did we come up with a Wawa problem? Why was it continuously asking people to bundle their nonexistent Wawa shampoo and Wawa conditioner? Then I realized that it was in the prompt. It was an example. So the researcher had decided that when it ran out of ideas, it was going to grab an exact example and give it as a suggestion.
That's one way where they just do their own thing. And so inevitably, they fall off the rails. And not in the Rails way, but more in the making sense kind of way. How do we fix it? How can we trust our agents?
We're going to throw more LLMs at the problem. But these LLMs are going to act as judges, and they're going to judge the output of the researcher. And these new LLMs have a much simpler job. They're basically just a classifier, and they're only going to care about answering one question that matters to your application.
Let's pretend that this guy is going to catch made-up numbers. This is going to make sure we're being kind. And this is going to do a reality check, just making sure that what they're telling them to do, they can actually do it.
You must be thinking, okay, we had one LLM that we obviously cannot trust, and now you want to add three more. How does that make any sense? Also, before I continue, just so you know, you don't have to have three judges. You can have more, but you can also have less.
But this is how we're going to trust them. We're going to do the exercise just with the kind judge. And you're going to come up with 200 examples of being kind and 200 examples of not being kind.
For example, "Your hip splitter challenging, swap one easy run for a chill session." That's kind. "Seven runs in seven days. Stop before you break yourself." Maybe not super kind. "Use different shoes. You've run with the same shoes 900 km. No wonder your knees hurt." That's not super kind. Also, I should have used miles. So if you don't know the conversion, just assume it's many, many miles.
"Fueling strategy. Your pace drops after 60 minutes. Grab a snack at the 45-minute mark." That's kind. "Cramps show up past 15 km. Carry water on the long ones." That's fine. "You cramp every run over 15 km. Drink water like an adult." Maybe not super kind. Which reminds me.
Okay, you're going to come up with 200 examples of this. And because you know which ones are kind and which ones are not, now you're going to come up with a prompt for your LLM. In this case, we're going to say, "Hey, you review training suggestions before an athlete sees them. Answer one question. Would a supportive coach say this? Blame, sarcasm, or threats fail. Honest data delivered kindly passes. Reply with pass or fail and why."
Now that you have your prompt, you're going to run that prompt through each example on your data set so you can measure how accurate it is. What's interesting about this is you can try different prompts. You can also try different models and figure out which one fits you better. Also, remember that these are smaller models, and you want them to be faster, so you don't need to choose the most expensive model out there, and you don't need really high reasoning for this.
Once your researcher is done giving you suggestions, you can move to the second step of your pipeline, which is running all these judges for each of the suggestions. We're going to do the exercise again just with the kind judge. What are we judging? The suggestion.
We're going to pass a prompt, same as we saw earlier. And then you're just going to have, again, an LLM provider. You're going to pass your two messages: here's the prompt, this is what you need to judge. And you're going to just have a model and the reasoning. In this case, you really don't need high reasoning. Remember, you want them to be cheap and fast. And once that's done, you just save that information.
Remember what the researcher produced earlier? Once this is done, we're going to run all our judges. And maybe the kind judge will say, "Hey, telling them that it's their worst sleep doesn't sound super kind." The next one, the fact-checking, might tell us that, "Hey, they don't own a bike, so they cannot do this." And you continue doing that through each of the actions. And then you realize that there are two that you cannot really trust to show. One because it's maybe not being kind, and the other one because they simply cannot do it if they don't own a bike.
That's how you build agents on Rails. Let's do a recap. We had an athlete with a bunch of information and metrics, and we're going to give that to the researcher agent. And the researcher is going to come up with personalized coaching ideas based on that data. In our case, it came up with four.
And then after you have all that, your second batch of jobs is going to make sure that you evaluate each and every one of these with the judges that fit your application. And you do that for all of them. And each job is going to give you a thumbs up or thumbs down based on the metric they care about. And then in the end, you're only going to show these two to your athlete. This is ultimately what they see.
However, your users are going to have thoughts and feelings, especially after you tell them that this was AI-generated. What if the athlete says something like, "I work on Sundays from 6:00 a.m. to 3:00 p.m."? That's something that no matter how many tools you give it, you might have not known or didn't have any data. And that's something that your researcher wants to know, so that next time the pipeline is running, it can remember that this happened.
How are we going to do this? We're just going to have a simple class that is an ApplicationRecord to store that feedback. It belongs to an athlete, belongs to a suggestion. And you remember the researcher job? We just put it in the prompt, and we say this is what the athlete has told you before, so that the researcher can learn from it.
But the problem is if you put raw feedback of what your users are telling you, you might run into some issues, because LLMs really thrive on patterns. And in our initial example, she was being nice. But what if she's not nice? Some of us do some name-calling sometimes for agents because it's frustrating. And we have a kind judge, but you don't really want to spend tokens having a researcher just to produce really angry suggestions to your users. And it might pick up on the bad vibes. You don't want to do that.
Any ideas on how we might solve this? Another agent. You're going to pass that raw user feedback and give it to an agent that is just going to take all that noise. And you just want the fact. So no matter how your user tells you the frustrations, you just want to remember that they work from 6:00 to 3:00 on Sundays.
And now if we go back to our feedback class, what we're going to do is we're going to use a callback. I know callbacks are not really loved by a lot of people, but I love callbacks. So you use them, and you send out a job. And that job is going to be very similar to the ones we've seen.
You take your feedback. You pass a prompt and ask it to reject the training suggestion in one sentence, say what the coach should remember about this athlete so it doesn't happen again. And then you do as you were doing earlier to just pass to the LLM provider that message, what the suggestion was. Also, it needs to be a simple model, so small brain, low reasoning, and you're going to save that as the processed text of your feedback. And then if we go back to the researcher job, that's what you really want to show there.
And this is where it also gets interesting, because you have to think about what about when they switch jobs? What if they don't work on Sundays anymore? And this is where you get into the dreaming kind of phase for agents, where you need to figure out that those facts that you just extracted make sense over the long run. And this is just a challenge that we're all going to have to face as we continue to capture information through other LLMs.
Now let's do an actual recap. We had a researcher that produced some suggested actions for a workout. Then we get our judges to make sure that they pass things that our application cares about, and we show that to our athlete. Once the athlete sees them, they might have some feedback to give us, and we're going to put our feedback loop in there, where we process that and we give it back to the agents so they can continuously learn throughout time. And that comes with the challenge of how do we keep that updated over time. But for now, for the pipeline that we're working on, that's it. Thank you.
[applause]
[applause]
And just as full disclaimer, I did all these images with AI, and there were some fun behind the scenes that happened there. Like this one, which was pretty scary and weird. This one where I told it agents, and I think it assumed cops, people. And then this one where it just refused to remove this big guy from the background.
Also, if you did enjoy the little agents, I made a bunch of stickers, so please go find me, and I'll share them with you. Thank you.
[applause]
Brains.
Article published · Updated