Teaching AI Agents and Human Contributors Through Custom Rails Generators
Ruby on Rails
Rachael Wright-Munn is one of four maintainers of RubyEvents.org, an open-source application that indexes Ruby community events. Her Rails World 2026 talk asks how to teach both AI agents and a growing pool of more than 180 contributors to work with a codebase they don't know. Her answer is custom Rails generators, paired with an AI "skill" and later exposed as MCP tools. The talk has three parts: a walkthrough of building a generator from scratch, practical fixes for generator tests, and a benchmark comparing AI approaches across models.
Why RubyEvents Stores Its Data in YAML
RubyEvents describes each event in YAML files. Wright-Munn showed the full set of YAML needed to describe Rails World 2026 and gave four reasons for the format. It is an inherited architectural decision. It lets the maintainers review event changes for spam. It lets any developer update events in the index. Most importantly, if the app goes offline or "is acquired by Meetup.com," the data still persists as part of the Ruby community's historical record.
The format creates a technical problem: every data change requires a commit. Any automation needed a tool that could turn unstructured information into commits, which she described as a hard proposition before AI. Her current workflow uses GitHub Copilot, which turns an issue into a pull request. She can see an event update while scrolling Bluesky, start an agent session, and get a PR she can merge. She said this is how the Rails World 2026 CFP got into RubyEvents.
What Went Wrong When AI Didn't Know the System
That workflow did not work at first. In her words, the AI was "super smart, but it doesn't know our system." It invented keys. It used old or deprecated formats, such as an event_name key that RubyEvents only uses for meetups. It left out to-do comments.
She explained the to-do problem with the videos file, which goes through three phases: when speakers are announced, when the schedule is released, and when the videos are published. Agents struggle with intermediate states like these because they mostly see finished examples. The AI also tended to skip useful keys such as cancelled, or original_title, which is used for talks given in other languages. It also spent many tokens trying to understand examples.
She said contributors have the same problems. They know when events update and have data the project needs, but they mistype keys and spend a long time decoding examples. That led to her central question: how to teach both groups about RubyEvents.
Choosing Skill Plus CLI Over MCP
For AI, she called the problem "kind of a solved problem," with two options. An MCP is an AI-native SDK that fetches context or executes commands for a model. A skill is Markdown documentation that AI can discover, paired with an ordinary CLI.
She acknowledged that AI models prefer MCP tooling and tend to reach for it. A skill plus CLI, though, works for everyone: anyone can read Markdown and anyone can run a CLI. She also mentioned a common argument in the debate, that describing MCP tools consumes context, and context means tokens and money. She chose skill plus CLI, partly for those built-in benefits and partly because it supports the 180+ contributors, and she knows not all of them have tokens to spend.
Why Rails Generators
Rails generators were the natural CLI format. Most Rails developers have run one to create a migration or scaffold an endpoint. Their biggest benefit, she said, was that she didn't have to make the usual design decisions herself. Arguments and options, templating and file generation, documentation, testing, and file conflicts were all already handled. She noted that file conflicts weren't even on her radar when she started. That freed her to focus on the actual tooling. Rails didn't have to make those decisions from scratch either, because Rails generators are built on Thor, a toolkit for building CLI applications.
Generating a Generator: NamedBase vs. Base
Her running example was a generator that produces the YAML file describing an event's CFP. She started by running the generator that generates generators. It produces a CfpGenerator class, a USAGE file, and a CfpGeneratorTest.
The generated class inherits from NamedBase, which explains how the name "CFP" is threaded through every file. NamedBase takes one argument, defined as argument :name, type: :string, and provides many helper methods that output different forms of the name. She looked at how Rails defines them and found they are mostly string interpolation.
She didn't need any of that. She already knew where her file should go, and she didn't need the name helpers. So she switched to Base. Her summary: NamedBase accepts a single name argument, provides many helper methods derived from it, and inherits from Base. Base has none of those features, and she picked it.
Arguments vs. Options
A CLI can take input through arguments or options. An argument is declared like argument :event, type: :string, desc: "event slug" and invoked as bin/rails g cfp rails-world-2026. Arguments can be required or have defaults, and they depend on order, so she compared them to positional method arguments.
Options are declared almost the same way with class_option, but the value is passed with a flag such as --event. They can be required, defaulted, or optional, and they don't depend on order. She compared them to keyword arguments.
She chose options. She felt they make it easier for contributors and agents to avoid ordering mistakes, and they give her more flexibility over which values to pass or leave out. She added all the class options needed to describe a CFP. Some extras, such as aliases and banners, she had no time to cover.
Auto-Generated Help and the USAGE File as Context
Class options do something else she found very useful. Running help on a generator shows documentation, and she admitted she didn't know this existed before the project. The options section of that help output is generated from the class options and updates whenever they change.
The contents of the USAGE file are appended to the end of the help output. She uses this to inject context for AI, because, as she put it, AI is reluctant to read documentation and files she references but is pretty good at running commands.
She showed the current USAGE file for the RubyEvents CFP generator. It has a minimal example, an example for an ongoing meetup CFP (meetups often have no end date because the CFP stays open), an example for lightning talks, and an example for updating or extending an existing CFP, since changing states are hard for AI to capture. All of this context loads automatically when the agent calls help. She joked that humans struggle to read documentation too, but since she herself didn't know this help existed, she isn't sure how much it helps them.
Templates and File Creation
To build the template, she pasted the real Rails World 2026 CFP YAML into a .tt file. TT stands for Thor template, and these templates use regular ERB, so she replaced each value with ERB tags reading from the options. She noted that you can also customize the templates of the default Rails generators by placing replacements in lib/templates, as described in section six of the Rails generators guide. Most of those generators use NamedBase, so the name helpers are available there.
She added two methods to the generator: cfp_file_path and create_cfp_file. The second calls template with the template file and a destination. The template is found through source_root File.expand_path("templates", __dir__), which the generator-generator added at the top of the class. She called this the kind of boilerplate generators are good at capturing. She wrote the destination herself. RubyEvents stores CFPs under data/<event series>/<event>, combined with destination_root, which Rails passes in and which defaults to the directory where the generator was called. Running the generator produced a working CFP file with all the values filled in.
File Conflicts, Conditionals, and Whitespace
Next she ran the generator without start and end dates to simulate an ongoing meetup CFP. A cfp.yml already existed, so she hit a file conflict. template handles conflicts natively and prompted her, and she chose to overwrite.
The output then failed validation. In RubyEvents, if open and close dates are present, they must have values and be formatted as dates. She wrapped those fields in ERB conditionals such as if options[:open_date]. That passed validation but left stray blank lines at the bottom. With HTML ERB, the browser hides that whitespace. In YAML it is visible. She switched to ERB's minus tags to trim it, which left only one extra line at the end.
Testing Generators and the Flakiness Trap
With conditional logic in the template, she wanted tests. Uncommenting the generated test and running it produced a green dot, but the run also reported that no value was provided for the required event option. run_generator takes an array of the same values you would pass on the command line, and she likes to pair each flag with its value on one line for readability. Then she added assert_file, a Rails test helper, with the file path and a content check. She said comparing against a regex is a common pattern.
"Green dots don't inspire much confidence," she said. She found the generated file still sitting in tmp/generators after the run. That raised a question: if the file persists, why didn't the test hit a file conflict? The default test sets destination to tmp/generators, and her cfp_file_path uses destination_root, which explains the location. setup :prepare_destination does the cleanup by deleting the entire destination root and recreating it.
That made her realize that with parallel test cases, a shared directory being deleted and recreated would be flaky. If you just uncomment the default test and move on, she said, you might eventually end up with flaky generator tests. Her recommendation is to set the class's destination to a Ruby temp directory and clean it up in teardown instead of using prepare_destination. RubyEvents has used this approach for a while with no issues.
Debugging Gotcha: Captured Output
With a temp directory she could no longer inspect the file directly, so she dropped in a binding to look at it. The debugger seemed to swallow her session: no green dot, no visible prompt. Control-C didn't work, and she eventually escaped with quit plus Enter, after which the IRB output appeared. The test framework was capturing the debugger's output. According to section 10 of the Rails guide on testing generators, you need to set RAILS_LOG_TO_STDOUT=true for debugging tools to work.
She explained why this isn't the default: if you run the full RubyEvents suite without the output captured, the results get very messy. Her testing roundup has three points:
- Use
RAILS_LOG_TO_STDOUT=truewhen you need to see output. - Replace the destination root with a temp directory.
- Don't use
prepare_destination.
For further learning she recommended Garrett Dimon's blog, the Rails guides, the RubyEvents generators themselves, and the generators in Rails, both open source, to see their testing patterns.
Writing the Skill
The skill half is simple. It is front matter describing when to use the skill, followed by regular Markdown. She includes important notes about things AI commonly gets wrong, stressing that RubyEvents is an index and archive, so accurate descriptions and data matter a lot. She adds to-do list items, which "tend to get loaded into the workflow, you know, if you're lucky." The generator section tells the agent to run bin/rails g cfp --help to load the full context, gives a sample command, and asks it to take a screenshot.
Adding MCP: Generators as Tools
Having built skill plus CLI, she joked about whether "or MCP" could become "and MCP." At RubyConf, Andy gave a talk about abusing schemas with metaprogramming, showing how to use an OpenAI schema to generate a RubyLLM MCP tool. She paired with him to build MCP tooling for the generators, because the detailed class options already describe everything an MCP tool needs. She said the full story is on the Ruby on Rails Podcast.
The result is a script that turns the CFP generator, or any generator, into a RubyLLM MCP tool for their server. Each generator is now both a CLI and an MCP tool. What she called the best part is that merged generator changes show up in the MCP tools when the server restarts. It also let her compare skill plus CLI against MCP directly.
The Benchmark
She started with GPT-5.4. She removed all the improvements she had made to the codebase and added them back one at a time. Each run used the instruction to add Rails World 2026, given a Markdown file of the event details so that different web fetches wouldn't skew results. She validated output with bin/rails validate_all, since validation could also affect results heavily.
No skill, no generator, no MCP. Working from older context, the model used 2023 sponsor logos instead of 2026 ones. It wrote "1Password" in lowercase and capitalized "TableCheck" wrong. It also added a URL for the Palmer Events Center, which told her it had done a web fetch she hadn't authorized. She counted four errors, at about 136,000 tokens. The transcript renders the cost as "$183," most likely $1.83.
Generator only, which she described as where most Rails developers stand with generators today. The prompt asked it to use custom generators. It made two errors: it left out PlanetScale and replaced em dashes with hyphens in descriptions. She quipped that "you can tell the big AI companies are really listening to us." It used 131,000 tokens.
Skill plus CLI. One error: the em dashes again.
MCP. The run had no videos, no schedule, and no sponsors. It quit partway through. It was still the cheapest option with the lowest token usage of all of them.
She also tested the smaller sibling model. With no tools it did worst. It renamed the keynotes, left every talk without a description, and used the lunch-and-break schedule format for all talks. It copied involvements from earlier years, including old MCs and volunteers. It deleted all the to-dos and added the deprecated event_name key. The MCP run was fairly good. It added one involvement, which was a problem, but only the Rails Foundation, and it made the em-dash mistake. The result she found most interesting was a run with this smaller model that cost 39 cents and had zero mistakes.
From this she saw a pattern: on scoped, repeatable tasks like this one, smaller models with better tools outperformed larger models. She said they didn't go off script, didn't try to rewrite her descriptions, and didn't look at other files to second-guess the file the generator produced.
Testing Luna
The Rails Foundation recently published an AI model report that found Luna had surprisingly high accuracy for its very low cost, so she tested it too. Luna struggled with schedules. Earlier configurations produced wrong times, and the MCP run had no schedule. With skill plus CLI, its only schedule error was leaving out day zero. Its other errors involved replacing descriptions with one-line summaries, plus, for MCP, the em-dash substitution. All of the Luna runs together cost less than a quarter.
Conclusions and Takeaways
She drew two conclusions from the experiments. Skills produced fewer mistakes but cost more. And concerns about the cost of MCP tooling may be overblown, since MCP was among the cheapest options in every set of runs.
She closed with several recommendations. Attendees now know enough to build a custom generator. They should remind their agents to use Rails generators for better reliability and lower token costs. They should try a small model with better tools instead of "leaving things to Opus all the time." They should try customizing existing generator templates to match company standards and get closer to the desired result on the first pass. Finally, they should build for humans too: humans and AI both need help understanding codebases, so the tools should serve both.
[music]
Starting off the right foot there. Hello, everybody. My name is Rachael Wright-Munn, and as Michelle said, I am one of four maintainers on the RubyEvents.org team. Now, RubyEvents.org is an open-source application where we index Ruby events from around the community using YAML.
So, this is all of the YAML that's required to describe Rails World 2026. You are here.
Okay, so you're probably wondering why we use YAML, and I'm just going to get it out of the way really quick so we can move on to what we're here for. First, it's an inherited architectural decision. Second, it allows us to review events for spam. Third, it allows any developer to update events in the index. And last, but perhaps most importantly, if our app ever goes offline or is acquired by Meetup.com, our data will still persist and be part of the historical record of the Ruby community.
Now, using YAML introduces an interesting technical challenge in that data changes require commits. And if we wanted any automation, then I needed a tool that would take unstructured data and turn it into commits, and this was a rather challenging proposition prior to the age of AI.
GitHub Copilot is a tool that will take an issue and convert it into a nice little PR for me. So I can scroll Bluesky, I can see an update, and then I can fire off a quick agent session and get a PR that I can merge into RubyEvents. Spoiler alert: this will work and is how the CFP for Rails World 2026 was added. But it didn't work at first.
AI is super smart, but it doesn't know our system. It would invent keys. It would use old and deprecated formats like event name, which we only use for meetups. It didn't include TODO comments. Our videos file goes through three different phases: the first when the speakers are announced, the second when the schedule is released, and the third when the actual videos are released. Intermediate states like that are challenging for AI to capture because they mostly see the finished result.
We also found that useful keys like cancelled or original title, which is used for talks in other languages, would be neglected because it was unaware or it didn't catch those when it was checking our data. The other problem is that it spends lots of tokens understanding examples.
We have this problem with contributors as well. We have 180-plus contributors on RubyEvents.org and increasing all of the time. And they have the exact same problem. They're super smart. They know when events are updating. They have data that we need, but they don't know our system. They would typo keys, and they would spend lots of time trying to understand our examples. So, I had the question: how do I teach them both about RubyEvents?
For AI, it's kind of a solved problem. It comes down to MCP or skill versus skill plus CLI. So an MCP is an AI-native SDK that fetches context or executes commands for a model. And then a skill is Markdown documentation discoverable by AI that's paired with a normal CLI.
Now, MCP tooling is undeniably preferred by AI, and it tends to reach for them. But the skill plus CLI, anybody can use that. We can all read Markdown, and then CLI or CLI, anybody can call them. Now, one of the biggest arguments that I see in the debate between MCP and skill plus CLI is the context that's consumed in describing an MCP tool, and obviously context is tokens is money.
So in the end, I looked at it and I made the decision to go with the skill plus CLI, not just for the benefits that are built into it, but also because it allows me to support those 180 contributors, and I know that not all of them have tokens.
So the next step was picking my CLI format. This is where the generators come into play. Rails generators are a really familiar pattern. Most of us have run a Rails generator to generate a migration or to scaffold an endpoint.
And there was another major benefit, which is that I didn't have to make any of these decisions. Arguments and options, templating and file generation, documentation, testing, and file conflicts, which wasn't even on my radar when I started working on the CLI, were all handled for me, so I could focus on customizing and working with the actual tooling. As a bonus, I learned more about a fundamental piece of Rails infrastructure, starting with Rails didn't have to make these decisions either. Rails generators are based on Thor, which is a toolkit for building CLI applications.
All right, so we're all together on the same page. I want to make a generator to make my YAML files that describe events. So let's start. I'm going to generate a generator because of course there's a generator to generate your generators.
It's going to give us all of these files. It's going to give us the CFP generator. It's going to give us a usage file. And then it's also going to give us a CFP generator test.
Now, you might be wondering how it names things, because it's weaved that CFP through pretty thoroughly. And the first hint is in the class that it generates with for the CFP generator: NamedBase. So NamedBase takes one argument, and then it has all of these methods associated with it that output different versions of CFP.
We can actually take a little peek at that argument and see that it says argument name, type string. And then it gave us even more methods. That's fun. Let's take a cheeky little peek at how those are defined in Rails. They're pretty straightforward, mostly string interpolation. And it gave us even more. Wow. We got so many methods from NamedBase. Let's throw all of that away.
So, I don't need a name. I know where I want my file to be, and I don't need any of those helper methods. So we're going to substitute NamedBase for Base here. Let's summarize the differences between these. NamedBase is going to accept one argument, name. It gives you tons of simple helper methods based on the name, and it inherits from Base. Base has none of that, and it's the one that I picked.
Now, I mentioned that we have that argument name. That's how a CLI will accept input. But there's actually two ways to do it. There's arguments, and then there's options. Now, the arguments look like this. You have argument event, type string, description event slug, and then when you're calling the CLI, it just goes bin rails g cfp rails-world-2026. And the arguments can be either required or default, and it's dependent on order. So you can think of this a lot like positional arguments in a method.
Next, we have options. So the class options is structured almost exactly the same, but with class_option instead. And what you can see is that when we call the CLI, we now have this flag event that we're passing the value to. This means that it can be required, default, or optional, and it's order independent. So you can think of it like the kwargs in a method.
So, if I'm looking at these two options, for me, I'm going to go with the options because I feel like that makes it easier for my contributors and agents to avoid issues with ordering, and it gives me a lot more flexibility when I'm deciding what values I want to pass into it and exclude.
I'm going to go here, and I'm going to add all of the class options that I need to describe a CFP to my generator. There are some little extras in here, like the aliasing and the banner, that unfortunately I don't have time to describe today, but these options do something else that's really, really cool.
Have you ever called bin rails g cfp help, or not CFP, but have you ever called help on one of the default generators? Honestly, I didn't know this existed before this, but there's actually documentation for each of the generators. And if we look at this section right here, our class options have generated documentation based on that. And every time we update those class options, these are also going to get updated. And if you map this back to the description of class options that I had before, you can see that laid out here.
There's something else in here that looks kind of familiar at the bottom. This is our usage documentation. It gets appended to the end of our help call. I've been using the usage file to inject context because one of the things I find is that AI is reluctant to actually call documentation and files that I reference, but it's pretty good at running commands.
So, let's time travel to the present. This is the modern-day help doc or usage file for the RubyEvents CFP generator. We've got a couple of different sections in here. I've got this minimal example, and then I've got an example about how to generate an ongoing meetup CFP. So for our meetups, we oftentimes don't have an end date because they'll just be open and ongoing. I've got one for lightning talks, and then I've got another one that's about how to update or extend the CFP because, again, capturing those changing states is a bit challenging when it comes to AI.
So, all of this context just gets automatically injected when I have it call help. Humans struggle with reading documentation, too. But honestly, I didn't even know this documentation existed, so I'm not sure how much it helps with that.
All right, we've got our inputs, we've got our documentation. It's time to generate some files. So we're going to start with the real-world Rails World 2026 CFP. I'm going to take this, and I'm just going to paste it into our TT file. This is our template file. TT stands for Thor template. And then these Thor templates just use regular ERB. So you can just go in here and replace each of those sections with the ERB tags and then the options. And that's our template.
By the way, you can customize the templates for the default generators as well. There's an entire section for it in the Rails guide for generators, section six, but basically you go into lib/templates, and you can replace those default templates. And most of them use NamedBase, so you have access to all of the helper methods that I showed you before.
All right, so now it's time to actually use the template that we created. So we're going to introduce two new methods to our generator. We're going to introduce cfp_file_path and create_cfp_file. create_cfp_file is pretty straightforward. We have template, and then we pass in our template file, and then we pass the destination we want it to end up in.
Now, you might be wondering how it knows where to find our template file from, and the answer is in the generator that was generated. At the very top, it added source_root and then File.expand_path, templates, and that's one of the things generators are really good at, is capturing glue code like that that needs to be added in.
The destination is something that I added. So in RubyEvents, we store all of our CFPs in the data folder and then event series and then event. And then we have this destination_root, which is passed in by Rails and defaults to where the generator was called.
Let's do that now. Let's actually run the generator that we've just created. All right, so we're going to run this, and it's going to create it, and it's going to work, and it's going to pass in all of the values that we asked for, and this is the output of that. So we have just created a working generator. Yay.
I think next I mentioned that we have those ongoing meetup CFPs. I think we should try calling this generator without that start and end date and seeing what happens. Oh, I forgot that we already had a cfp.yml created. So we ran into a file conflict, but thankfully template has native support for file conflicts by default. So it's going to pop this up, and I can just say yes, overwrite.
Oh, okay. So in RubyEvents, if we pass an open date and a close date, we need to have values in there, and they need to be formatted as a date. So we're going to have to remove those fields from the template. All right, so I'm going to go here, and I'm going to say if options open date, and I'm going to pass that. And again, this is just regular ERB, which you're already familiar with from your templates. And then I'm going to do the same thing for options close date. And it's going to look like this, which is good and passes validation, but it has a bunch of weird extra whitespace at the bottom.
Normally when I'm working with HTML, ERB, all of that kind of gets smooshed out by the browser, and I don't have to see it. All right, so instead, what I'm going to do is I'm going to use ERB minus tags in order to remove that extra whitespace that was added. Okay, that looks much better. And now I only have one extra line at the end. So we trimmed whitespace using these ERB minus tags, but now we have some conditional logic. So I think it's time for us to write a test.
All right, so good thing we created one: the CFP generator test. I'm just going to uncomment the code here and run it and see what happens. And there we go. Uncomment it and run it. And green dot. Yay. It says no value provided for required options event. So I think we should probably pass some options to this.
Okay, so when I'm calling run_generator, I'm going to pass an array that are the exact values that I would have passed to my CLI. And what I like to do is I like to pair them with flag and value just to make it easier to read. All right, let's run this again. Green dot. Not bad.
Maybe we should try asserting the file. So assert_file is another testing helper that's provided by Rails. We're going to put our file path in here, and then we're going to take content. A really common pattern is just checking this against some regex. But you can do whatever you want to assert content, and green dot. Green dots don't inspire much confidence.
Good news, though: if we go to tmp/generators, we can actually see the file. It persists after the test run. Wait, if it persists, why doesn't it run into a file conflict like we saw earlier? That's kind of weird. Let's take a look at what that test is actually doing.
All right, so this is the default generator test that it created. We've got this destination here where we put in tmp/generators. That explains why the file was in tmp/generators. And when I did the cfp_file_path, I passed in destination_root. So that's how it ends up knowing that tmp/generators goes there. Okay, good. And then setup prepare_destination. This must be what does the cleanup. Oh wow, that's so straightforward. We just remove the entire destination_root folder and then remake it. Perfect.
Wait, hang on. If we're removing the tmp/generators directory and then recreating that, wouldn't this be flaky if we had parallel test cases? Yeah, yeah, yep. Yep. If you just uncommented that and rolled forward, then I suppose at some point you might end up with flaky generator specs.
So instead, I recommend this: when you're setting your class's destination, use a temp directory. That way, you don't run into the same errors, and you don't end up running into that same issue. And then the other thing I recommend is cleaning it up in teardown. We've been using this in RubyEvents for a while, and we haven't had any issues with it. So, just a reminder: when you're working with those tests, try and use Ruby's temp directories as the destination and clean up in teardown.
But now I can't see the file, right? And so I'm back to my original asserts and confirmation. I'll just put a binding in here. No big deal. And then I'll go check the file that way. Perfect. Okay, so we can actually see our temp directory. And it worked. That's great. Let's go back to the console. And where did my green dot go?
Where's my binding.irb? Hang on a second. Let me just do a Control-C. Maybe a little quit. Panic. Okay, quit plus Enter got me out of that mess. That's weird. That's so weird. But I can see the IRB output now that I've done quit and Enter. It's almost like it captured my binding.irb output.
And you might be baffled by this if you didn't read section 10 of testing Rails generators very thoroughly. It turns out that you need to pass RAILS_LOG_TO_STDOUT=true in order for your debugging tools to work. All right, let's try that out now. Oh, perfect. Okay, that makes so much more sense, and everything looks fine, and I can see it now. Good.
Oh, wait. Why don't we do that for everything? Well, if I run it on the current test file that we have
In RubyEvents, this is what one of our tests looks like. So if you're not capturing that output, it's going to look really messy when you're running your entire test suite, which is why we need that RAILS_LOG_TO_STDOUT=true.
So the roundup of the feedback for testing generators is: make sure to use RAILS_LOG_TO_STDOUT=true to see output, replace your destination root with a temp directory, and then don't prepare_destination.
I have now taught you enough about generators to be dangerous.
There's a lot more to learn about generators. There's a ton that I couldn't cover. I strongly recommend Garrett Dimon's blog. Obviously, I recommend the Rails guides. They're fantastic. But also, the RubyEvents generators. It's an open source app. You can go look at them. Rails is open source. You can go look at Rails's generators and read them and see what their testing patterns are and how they handle things.
So we have spent an awful lot of time on the CLI. So I think it's time for us to start talking about the skill part of that equation.
Skills are pretty straightforward. We start with front matter about how to use it. Then it's just regular Markdown. I like to include important notes about things that the AI commonly gets wrong. We are an index and archive for Ruby events. So it is very important that our descriptions and our data are accurate.
Next, I recommend adding some to-do list items. Those tend to get loaded into the workflow, if you're lucky. And then here is our actual section on using a generator. We suggest that it runs bin/rails g cfp --help. That way, it loads all of that context and then it actually has a sample so it can run that and create it. And then we ask it to make a screenshot.
So that's the skill part of skill plus CLI.
And now we've done it. We built the thing. We built the skill and we built the CLI. We have handled the "or" part of this equation flawlessly. And really, we're done. It's fine. I don't need an MCP. I have the skill plus CLI.
I said "or MCP." What if it were "and MCP"?
So at RubyConf, Andy did this talk about abusing schemas using metaprogramming, where he talked about how to use an OpenAI schema in order to generate a RubyLLM MCP tool. So I was delighted and excited and I wanted to pair with him, and we did, to build MCP tooling for our generators. We have these really specified class options that describe everything that we need for an MCP tool.
So the full story is on the Ruby on Rails Podcast. If I don't have time for class option options, I don't have time for this story eventually, but we now have a tool generation script that will take a CFP generator, or any generator, and turn it into a RubyLLM MCP tool for our server. It's a CLI and an MCP.
I haven't even got to the best part. Anytime we merge generator changes, it's reflected in MCP tools when we restart the server. It's so nice.
But this also means that I can actually test skill plus CLI versus MCP, which is what I did. So I started with GPT-5.4 and I deleted all of the improvements that I made to the codebase and started rolling them back in slowly.
The first one, I had no skill, no generator, no MCP, just trusting the AI, and I told it to add Rails World 2026, and then I gave it a Markdown file. I wanted to make sure that it was consistent and different web fetches weren't affecting my results. And then I also used bin/rails validate_all to validate, because that could have a major impact on our results as well.
Because it pulled in that old context, it decided to use old sponsor logos from 2023 instead of the 2026 ones. It got 1Password lowercase instead of 1Password, tablecheck instead of TableCheck. And it added a URL for the Palmer Events Center, which tells me that it decided to do a web fetch that I didn't authorize. I'm going to count that as four errors, and it cost me $183 and 136,000 tokens to do that.
Now this next one, I didn't add the skill yet. I just added the generator. So this is the status that you are in with the Rails generators right now. Using custom generators, add Rails World 2026, and everything else is the same. This time it only had two errors. It forgot PlanetScale and it replaced em dashes with hyphens for descriptions. So you can tell the big AI companies are really listening to us.
This one was 131,000 tokens and two errors. And then this last one, we're finishing the skill plus CLI. And this time it only had one error, which was replacing the em dashes with hyphens.
Now, I wanted to see how the MCP went. And no videos, no schedule, no sponsors. It just kind of quit in the middle of the run. So that was a fun thing to discover. However, it was the lowest cost and the lowest token usage out of all of the options.
I didn't just run this with a frontier model, though. I ran it with its smaller sibling too. So in this one, the worst one was no tools. It renamed the keynotes. None of the talks had descriptions. And for the schedule, it decided to use the format we use for lunch and breaks for everybody's talks. They wouldn't link.
Generators but nothing else. It copied involvements, including the old MC and volunteers, it deleted all the to-dos, and it added the event name parameter that I mentioned before.
And for the MCP one, that one was pretty good. It did add an involvement, which was kind of a problem, but it was just the Rails Foundation, so that's better. And then it also did our em dash for hyphens mistake.
The interesting one to me, however, is 39 cents for zero mistakes, because I was starting to see kind of a pattern here, which was that smaller models with better tools outperformed larger models for scoped tasks and repeatable tasks like this one, because they didn't go off script. They didn't try and rewrite my descriptions. They didn't try to look at other files to decide if the file that the generator had created was appropriate.
Which brings us to something very interesting. The Rails Foundation recently published a set of model—they did an AI model report, and they found that Luna, which is right over there, had a shockingly good accuracy rating given its incredibly low costs. So I wanted to test that one as well.
It had a lot of trouble with schedules. At the earlier ones, it would do wrong times. It had no schedule for the MCP option. And for the skill plus CLI, the only error that it had was that it didn't include day zero in the events. And then the other errors were all related to replacing either descriptions with one-line summaries. And for the MCP, it replaced em dashes with hyphens.
So after this—and by the way, all of these together are less than a quarter.
So I got a few conclusions after this, which is that skills had fewer mistakes, but were more expensive. The cost of MCP tooling might be overblown because MCP tooling was one of the lowest cost options for all of these.
And then there are a couple of takeaways I'd like to leave you with. One is you are now dangerous enough to build a custom generator. I'd like you to remind your agents to use Rails generators to get more reliability and to lower your token costs. I think you should try using a small model with better tools instead of leaving things to Opus all the time. I think you should try customizing existing generator templates for your company's standards to see if you can get closer to what you're looking for in a first pass.
And then the last one is to make sure that you build for humans, too. Humans and AI need help understanding our codebases. So let's make sure that as we're building, we build tools for both.
I want to thank you all so much for your time. My name is Rachael Wright-Munn. I go by Chael or Rachael. You can find me at ChaelCodes in all the places. I stream on Twitch and YouTube Sundays at 1800 UTC. Thank you.
Article published · Updated