Glen (Plabayo): This is remastered version of episode 5, originally aired last year 2025. It's an episode with Carl about the origins of Tokio as well as Carl's side project, Toasty. And at the end, we also discuss a bit about AI and the upcoming conference dedicated fully to Tokio. It is the first ever conference dedicated fully to Tokio And for those that cannot attend the Tokio conference, do not worry, Carl confirmed us that all talks will be recorded. And now, we hope you enjoyed this episode remastered with the audio quality that you all deserve.
Elizabeth (Plabayo): This is netstack.fm, your weekly podcast about networking, Rust and everything in between. You are listening to episode five, recorded on the 27th of August, 2025, where Glenn has a conversation with Carl creator and maintainer of the Tokio Runtime, the async foundation powering much of the Rust ecosystem.
Glen (Plabayo): It's another week for netstack.fm and today I welcome to our virtual studio Carl. Welcome!
Carl Lerche: Thank you, super excited to be here.
Glen (Plabayo): Yeah, I'm looking forward to our conversation. So today we will talk a bit about Async in Rust, a bit about Tokio's history and of course Carl his own history. He will also talk about the upcoming Tokio conference, about which I'm also very excited, as well as the future of Rust Async. Now Carl, despite all the impact you've had on the Rust Async ecosystem, or actually in fact that you have 186 crates on crates.io which is quite a lot, Some listeners may not know you yet. Therefore, let us await a bit, pardon the pun, and please start by sharing a bit about your journey and how you first got into technology.
Carl Lerche: Okay, yeah, I'm Carl. Thanks again for having me. ⁓ I've been programming since I was a kid, basically, in the 90s. You know, my family always had a computer. I remember my first computer was an old Zenith machine and it had, ⁓ one of the versions of BASIC. I'm pretty sure it was GW BASIC.
Glen (Plabayo): Wow, nice.
Carl Lerche: Back then you had to number your lines and I remember that you had to do like number all your lines 10, 20, 30 because you wanted to skip numbers so you could add lines later without having to update all of your code to reference different line numbers. So everything was goto ⁓ with that. But it always interested me, know, computers. always loved tinkering and programming and that kind of got me into it. I think I was, I don't remember some probably around high school, the whole internet thing started happening and... I got into, I picked up a PHP book at that point and just kind of started hacking and kept doing it since. didn't think, interestingly enough, I didn't think I was gonna go do computers or programming professionally. ⁓ Not entirely sure why, but it took me a little bit to actually land into software as a profession, but.
Glen (Plabayo): But did you know it was a profession? for example, yeah. Yeah, well, it was because personally, like I also got started with programming quite early, but like, like there were weeks where I was doing it a lot and then other weeks where I wasn't or even a month I wasn't at all. And then I didn't know it was like a profession. I just thought it was something you do, I don't know, to fill time or because you enjoy it.
Carl Lerche: I mean... Yeah, go ahead. Yeah, I mean, totally. think I actually grew up in Belgium and I was in Belgium at that time when I was kind of exploring computers and really, I would say kind of the 90s, there was not really at all a career track in programming. think, you know, going through high school and that early kind of education, I guess you my career track at that point was like going into some sort of civil engineering or that was what you did if you are good with Math. math, kind of like some sort of, you know, and just like, engineering kind kind of thing building bridges, I don't know. But That didn't really, I mean, didn't love that idea. Also around that time, I ended up moving to the United States and to Portland actually, and went to Portland State University, which had a, mean, most US colleges had a good ⁓ software kind of programming track. And I kind of dabbled in some of those courses, but for some reason I still wasn't like, I'm going to go and become a software engineer. there's still, again, while the dot com thing happened, it also kind of just crashed. So right when I was going through college, there was a gut of software jobs as my recollection was. And I kind of left, yeah.
Glen (Plabayo): Interesting and and and before we continue like and I don't want to go too deep into it but It reminds me that maybe the the people are currently at the same stage in their life as you are As you were back then and and given there is now this entire like LLM and AI Well some fear some hope for it and again, I don't want to go too deep into it right now But do you think that how those people also felt like am I? still in the right career, is that something like you also felt in those times?
Carl Lerche: I would say, well, I wasn't in that career yet kind of thing. yeah, yeah, I actually, I graduated with math, with math. So I didn't even graduate computer science. So, but what I would say is there's...
Glen (Plabayo): No, but you were going to study for it, right? And maybe you were wondering, like, is that the future?
Carl Lerche: There's always, I mean, like all things, was a gut then and then people decide, like figure it out. Okay, how do we actually use the internet kind of thing? So, and you mentioned AI and I actually think there's a... a lot of what we're seeing right now with AI is reminiscent of that whole introduction of the internet where kind of everyone like industry had a way of doing things, which was not with the internet. And when the internet was introduced and there things were moving online, there was kind of this big fear that a lot of the jobs, especially retail and all that are going to go away. And, you know, there's going to be far fewer jobs for people. it is true that the internet and technology made a bunch of those kinds of jobs obsolete, but that's not really like, it's not like there's no jobs ever. There's new jobs kind of thing. Like the number of software programming jobs kind of exploded. And I kind of see you alluded to AI. There's a, there's kind of this fear that AI is going to take our jobs, especially now that the newest AI codegen tools are surprisingly good. There, My take is it's not there. It never is they take our jobs and no one can find anything to do. It's the jobs change and you evolve with them. And, you know, there's always a drive to keep building more and to, you know, keep advancing the state of technology and the state of our products and everything we use and make our lives better. And there will be new things and new ways to approach. the same kind of problems. I think AI is definitely gonna change our jobs. ⁓ They're not gonna go away, they're just gonna be different.
Glen (Plabayo): Yeah, cool. I that's very hopeful and I hope that you're there and I do think I agree with it. Now, you said, okay, in Belgium, high school, you were more like civil engineer than in United States. You were finally into college. You were majoring in math. How did you end up then in programming?
Carl Lerche: Well, it turns out that's where the jobs were. Because right then, like right when I started entering the workforce, ⁓ the recovery, like the post DotCom crash recovery was happening. And... the number of software jobs was kind of exploding again. Everyone figured out how to use the internet and figured out what to build. And there was a massive kind of expansion of things to build, mostly internet software and networking software. And my experience kind of like hacking around with PHP and all that was very applicable. And I got a job working PHP ⁓ at a PHP shop. Interestingly enough, also around that time, I think about maybe two years or something into my time just building web apps professionally using PHP. ⁓ That's when the whole Ruby on Rails thing became, well, started catching on. And I do also remember that, so I don't know if anyone remembers what it was like building web apps like. mid 2000s or something like that, but it was painful. I mean, I feel like all software building back then was painful compared to what we're used to now. Yeah, I mean, it was simpler, but ⁓ it was less productive as well. It took longer to build things.
Glen (Plabayo): Well, in a way it was also simpler, And by the way, you said, okay, PHP, but does it mean just PHP? Because I mean, there were also plenty of people, I forgot how it's called, but that you could like mix in C or whatever, native languages on your computers.
Carl Lerche: Yeah, I mean, there was, mean, this is all a long time ago too. There's definitely some of that. I remember what I did was just pure native PHP, like the lamp stack. it Linux, Apache, MySQL, PHP? That's what it was all about back then. But I do...
Glen (Plabayo): Hmm. It was exciting though, like the mix of PHP brings it up. I mean, I do remember it.
Carl Lerche: I mean, it was very exciting. ⁓ There was just kind of a whole bunch of, ⁓ think the whole Web 2.0 thing wasn't coined at that point yet. at that point I would say the phase of... the internet was just introduced and let's just do put everything literally we can think of on the internet without really knowing how to do it ⁓ past. So now things that were being built were more kind of, wasn't so much take existing stuff and shoehorn internet into it. was more now we're re-imagining what we're gonna build like, but on the internet and software, think. ⁓ Software as a service, that concept was also just kind of starting, which was interesting. But I also do remember again, the Ruby on Rails thing, that first experience I saw when you saw the screencast, let's build a blog post in 15 minutes. That was again, nowadays, I feel like you look at something like that and you're like, yeah, it's nothing special. But... when you were coming from the way people did or building software and specifically web apps before you saw that screencast, it was a game changer. And that concept of focusing on just productivity, specifically convention over configuration, these are all things that seem obvious now and are kind of ingrained, but... It wasn't back then. Like that's the thing that's ⁓ funny to think about, like that idea of convention over configuration and just kind of. streamlining the happy path was not really a thing. it was in that kind of revelation, like changed, I think how a lot of people thought about building software and designing ⁓ libraries for kind of productivity and that kind of stuff. So it was, and it influenced a lot. It heavily influenced the way I kind of think about
Glen (Plabayo): Hmm.
Carl Lerche: about software too.
Glen (Plabayo): Yeah and of course it goes without mention but a lot of influential and important people from the Rust community they actually come from Ruby on Rails similar like you so is that for the same reasons where like from the performance was always a mention?
Carl Lerche: Yeah, interesting. there were a lot. I Ruby was big back then ⁓ and when it was taking off. It was also, again, it was taking off really in the early days of the internet where scale was a lot smaller. mean, obviously Google was already big and whatnot, but besides those huge kind of search engines, the scale of building a web app was much, much, smaller. And Twitter like wasn't even around yet, for example. I think Facebook was just starting, ⁓ but... As the internet kind of grew and grew and grew and more and more people got online and the scale, like scale requirements and performance requirements kind of just increase at exponential rates. And I think there was, I mean, Ruby's not known for its performance and there were a lot of attempts and a lot of engineering time put into trying to speed up Ruby, but fundamentally the language itself. was not suited for really optimizing. And there was kind of like a diaspora, I remember, of just like people doing Ruby that spread out to other languages. A lot went to Go as well. Go was kind of like the early one. But I think the Rubyists that you, ⁓ the Rust. developers that came from Ruby didn't go to Go and partly, mean, probably a few reasons. One, just didn't like the language that much. Two, while Go was faster, it wasn't kind of ⁓ like a revolution, I think, because the JVM was also there as well. The JVM was competitive. I think a lot of the Rubyists that went to Go just ⁓ also just had a stronger version to the JVM, which understandable. Rust, when it came about, really was a brand new thing where you got that performance of a native language without the risks of, mean, everyone knows, I read this Rust podcast, you know what the selling point is, but ⁓ that was something very appealing to those of us who never wants to native, like C or C++ because we know we're not good at it. ⁓ So it was an opportunity to kind of jump into that. And the way I got involved, ⁓ I was working, at a startup called Tilde with Yehuda Katz who ⁓ did a bunch of stuff like JavaScript and also was involved early on with Rust. But we were building ⁓ kind of a performance monitoring tool for Ruby specifically. And the gem, the library core part was a Ruby library, but we're pushing a lot of ⁓ traces essentially through that gem and we wanted it to be fast. So a native extension there was the only option, but ⁓ I was like, I remember I was like, let's just do it in C++, YOLO I guess. I'm not very good at it, but I, like many people, I tend to vastly overestimate my capabilities. ⁓ But Yehuda was, and he like, He knew Dave Herman through his work in JavaScript. Dave told Yehuda, I'm working on this language at Mozilla that gives you the pitch, right? Gives you the performance of C++ without the dangers. You should give it a shot. And so Yehuda told me, hey, you should try Rust. ⁓ I pushed back initially pretty hard because it was... Rust was very new, and I mean, if you use Rust back then, you would just be hitting ices constantly. This was, this was at the time where... I don't know if anyone listening remembers the very early days of Rust where it looked a lot like Go. It had a garbage collector. It had green threads, et cetera. ⁓ So this was the point I got kind of got introduced to Rust was when they were doing that work to remove all of that, to turn into the Rust that you know now. So that was just landing. Yeah. Yeah.
Glen (Plabayo): Yeah I joined a bit after that, like not much after that but I was already like without that kind of stuff.
Carl Lerche: Yeah, so that was just landing and I almost, I feel like I remember like when we got introduced, it was literally like, hold up another two weeks, we'll land the last few PRs so that. you actually can run it without the garbage collector. Anyway, I was telling Yehuda hey, this is massively untested, completely experimental stuff. We probably should not use it in our product. That was my argument and to just use C++. I mean, his side was, you can't write C++, you're no good at it. So eventually he won that argument. And I was pretty skeptical at first, but you know, after actually using it, ⁓ getting something working, it's the selling points kind of. It sells itself, I guess. And I guess the rest is history. That was my introduction. I wanted to use Rust more. After that, I like building networking stuff here. And obviously, it was so early, there was nothing there. The only networking APIs were... roughly what the standard library offers now, simple blocking sockets. So I started building MIO, M-I-O, just kind of like those epoll bindings and went from there. Yeah, it started, I first wrote MIO probably 11 or so years ago, that was when I started.
Glen (Plabayo): that story that early? don't know. Because I remember around the time that I was introduced to Rust, which was like 10 years ago, then me and my colleague at work, we were rewriting our first library from C++ into Rust after selling it to our bosses. I know there was, as far as I know, was already Tokio in development, I'm not sure, it probably must be very early days because we definitely didn't use it or have any need for it.
Carl Lerche: 10 years ago, I don't think Tokio was started yet. Maybe I could, I wanna say, what's yours? Actually, I'd have to go look it up. So I mean, when we're talking about like a decade plus or minus, it's hard to remember. I wanna say I started Mio 10, 11 years ago, but. yeah, and.
Glen (Plabayo): Hmm. What was your original goal for mio because for sure it wasn't like to build Tokio.
Carl Lerche: Well, original goal, actual original goal was let's start a side project. ⁓ And I wanted to, know, it a side project I could work on. ⁓ My initial idea was to build, you know, kind of hack on a toy distributed database. Why not? Because that's an easy side project. But... there was no networking libraries, right? So that is what got me started. like, all right, let me just quickly hack together, epoll bindings and go from there. so my, Mio and Tokio and all that was roughly, it was like this big yak shave that I still am not out of, because I never wrote that distributed database. But ⁓ the idea was not Tokio, you're correct. The idea was not Tokio. It was, so Mio was first and then the, There were some early experiments with futures that I made that didn't happen well before Async Await. But the Async Await stuff, I think that was designed by the ⁓ Rust core team. And what they landed on... was quite nice. think it was interesting to be around during that early designing of, well, it was futures before Async Await even, right? So, before even Async Await, there was the future traits and that was built as a external library and seeing the evolution of that was interesting because... ⁓ And it kind of also tied back to how Mio was designed. A lot of async libraries for other languages are built around closures and callbacks, right? And fundamentally, if you're if you're registering a callback, that has to be allocated somehow and moved off the stack because you're saving it all and storing it to be called later. And since it's like, no, I mean, that's how it works in JavaScript. That's roughly how it works, worked in Java. Pretty much every async library at a high level worked roughly like that. And early attempts to design a async I/O system with Rust kind of followed that pattern of let's let's do closures and box them and whatnot. it got, it was obvious that it got, it was really painful. It's just like, you're allocing closures everywhere, you're boxing everything, you're making dynamic trade objects everywhere. Everything has to be, know, send, sync, static, all of that stuff to make a compile. It was pretty obvious early on that that's... didn't work. ⁓ And that's also kind of like, if you look at like, so when I built mio the first thing I was looking at was like libUV in JavaScript for like those, essentially the library backing Node.js. And they also used, you know, function callbacks and whatnot. And I threw that out the window with mio The idea being, let's just model it after epoll and be as close to the metal as possible. ⁓ what I didn't realize after that was that was the path to keep going. And because I was still playing with closures and whatnot. And I think, you know, the Rust core team early Rust Core team that were focusing on what futures would look like kind of came to the realization that you could abstract, you could continue that kind of of polling model and abstract it away. And yeah.
Glen (Plabayo): and were were you like involved? mean, even if just as feedback?
Carl Lerche: I was I was involved like watching in the site involved at a like superficial level, I guess. But then after that, it was like, all right, what what does Tokio look like after that? And they were also experimenting with how to bring that to the networking layer. And what I would say is their early attempts were, their early designs were more... It's just interesting to see that evolution and I've looked at it kind of not that recently, but I have looked back at the evolution just out of curiosity. So their early design for what and ⁓ networking kind of abstraction built on top of futures and the future API at that point is pretty similar conceptually to what is today. But you build a stream, so a stream of network events. essentially, and then you still have your reactor. So the main abstraction there was a future of the IO events. So you would still have your IO loop where you where you receive those events and then call like non-blocking functions. so it's kind of hard to describe, but like the the Waker system was already... kind of there if you're familiar with internals of how futures work but the waker system was there ⁓ it's a it's not a callback per se i would say because it the the framework doesn't call it doesn't call you
Glen (Plabayo): which is still like a callback, right?
Carl Lerche: you're kind of like, ⁓ it's a bit different. I wouldn't call it callback. It's like more of an event system and non-blocking actions. So, but there is kind of like that root point where you await on events in any runtime event loop. There's that place where you wait for events and then you get events and then you process the events. So the Waker, I'm trying to think. I do think there was like a dispatching system where you had the root feature that was quote unquote spawned. And then when the event came, you sent, ⁓ like the runtime called that root future, which is the same as a spawn task. And I believe that their original design for layering IO on top of that was inside of your spawn root future, you then had a stream of IO events that said readable writable, and you are almost building the same mini loop within, like you're building almost like mini reactors within each spawn task yourself that were scoped to one socket, if that makes sense. And I guess the main insight I provided after that like at that point with Tokio was that you could take that same Waker system and apply it directly on ⁓ IO actions. So you didn't have to have a ⁓ future and a stream, because those were the two primitives they were building on. You could just take that entire abstraction and apply it to everything. So my initial version of Tokio was with poll read. I guess introducing poll read for better or worse, if you like it or not, was the Tokio contribution ⁓ where you called poll read and pass in your waker ⁓ and that then registered that point, if that makes sense. That waker for that read was registered and then the spawn future would get called again and you ran your entire state machine. So that was the, I guess, contribution of Tokio to what was non-blocking Rust futures at that point. And then... Further down the line, I don't know if you yourself wrote a lot of code by manually implementing Futures for everything. ⁓ But yeah.
Glen (Plabayo): I did sometimes but definitely I mean and anyway sometimes you have no other option right like let's say for example if you're trying to bind something think to us think or whatever like
Carl Lerche: Yeah, for sure. But now imagine implementing an entire complex like networking system on entirely manually implemented futures. It was doable. You manage your state machine, ⁓ but it was painful and tedious and error prone. And that was when... I so async await was a thing. I don't remember. I honestly don't know who introduced the concept of async await in the language space, but it was already like in a bunch of different languages at that point as syntactic sugar on top of closures. And the... Key thing at that point that the Rust core team did, and I believe, you know, without boats did a lot of this design work. I'm not entirely sure, you know, who did what, but like there was that pin was introduced and then how to take async awaits syntax and translate it to a state machine. That was all, as far as I know, novel work that's... significantly made writing async rust, like it significantly improved the ergonomics of async rust. yeah, for the user of Tokio, well, even for Tokio itself, like if you look at the implementation, we use async await as much as possible and only rely on like...
Glen (Plabayo): That's for the user of Tokio, like for Tokio itself, I mean, I imagine...
Carl Lerche: manual future implementations for what you'd call it, like what I think of as like the leaf node or like the integration points. But most of it's gonna be async await all the way down. And we're gonna push that more complicated state machine code to as little as possible. Yeah. ⁓
Glen (Plabayo): And as well, anything that touches straight, right?
Carl Lerche: Right now probably hopefully one day we'll get I think I mean we're on the path right now getting Async awaits in traits is just turned out to be harder Yeah
Glen (Plabayo): Yeah, mean the trade bounds are tricky and the fact that you cannot assign it to associated types.
Carl Lerche: Yep, I know that improving that is in progress. In fact, we have, know, async function and traits at an early level already, and that's good. And the async traits proc macro by David has also ⁓ been a very key and important contribution. So I do think... the future for async Rust is bright where like, it's been interesting kind of seeing that evolution. Cause initially when Rust was introduced, it was pitched as a systems programming language, right? It was like, this is an alternative for C++, C like you use it to build that infrastructure level. But like, as you get to... use Rust and as the language has evolved to be, it really is a highly expressive language. And when you know it, you can be as productive with Rust as like any other language, I believe. Like my, like, I've written many languages and that is hell true for me. ⁓
Glen (Plabayo): Yeah, fully agreed. Yeah, because like I said, I originally came to it while I was still doing mostly C++. And for me, I also saw it like as a new system language which solved every problem I had to with C++. Not so much problem, but like things that I had to worry about and things that I had to be very careful about and there it was just solved for me. And then at the same time, I discovered Go. So at the very start, I was doing what I would consider like things like web services mostly in Go because... I really saw them as two different things, then more and more I noticed that Rust can handle this also very nicely. And the thing I sometimes say to folks in the Foundation or other people in the Rust community is the thing that they don't mention enough is how you can really fearlessly refactor your code bases For example, I maintain a big network framework called Rama, which is also built on top of Tokio. ⁓ The amount of times I did some refactor, some very big, some almost very big, like, ⁓ and it just works out fine. Like I don't have to worry about it. And that's definitely a power of what Rust and its type system gives to you.
Carl Lerche: Yeah, for sure. that is the like equation that Rust gives you. It's like, I don't think it's ever going to be, I shouldn't say that. I think it'd be hard for Rust to ever be as productive, like to be the most productive language for early development and prototyping, but. On a whole, when you factor in long-term maintenance and like the growth and the longevity of software, Rust, I believe is highly competitive from a productive point of view and I, from a productivity point of view. And kind of my, the way I think about, you know, software engineers is fundamentally. software engineers, programmers, they just want to get their job done as fast as possible, as well as possible. And whatever tool for the job that enables them to do a good job fast is going to be the tool they pick, right? And... Now there's mint and that is, it sounds simple, but they're, you know, that's incredibly complex calculus to pick that tool. But, and there's obviously going to be many biases when picking that tool, but when doing the calculus, mean, but. If you look at it that way, that also kind of explains why Rust exploded for lower level software is because when you're building infrastructure level software, you have fundamental requirements that are like, this has to be fast and this has to be correct and this has to have predictable, reliable tail latencies. And when those are requirements for your software and you look at anything out there, including Go, including the JVM. Rust is the most productive tool for that job. Maybe not because of that initial implementation, but you're gonna get to software that meets those stringent reliability, consistency, predictability, performance goals faster with Rust. And... ⁓ especially once you start learning it, right? Like what is like the hard part, what's crazy is that you can meet, like it's true for Rust, that calculus, that productivity is true for Rust, even if you have a team that doesn't know Rust going in, right? If you have one person that kind of knows Rust and a team of engineers that don't really know Rust, more often than not, obviously it's not a guarantee, but more often than not, if they're building infrastructure software, they're going to be able to learn Rust and build their software and meet their goals faster, you know, with Rust than building with the JVM. Fundamentally because it is really hard to get tail end sees down with a JVM. And I know I'm gonna probably make people upset, like JVM people upset. ⁓ I don't care, but I know they... JVM engineers keep saying we're going to finally get tail encees down with the garbage collector with just one more revision of the garbage collector like there they've been It's I've been hearing that for 20 years. I still remember the pitch in the 90s Kind of late 90s. I was a kid Talking with some older, you know software engineers like I mean older they're probably there late 20s or whatever But I was a kid so whatever and they're like and I was like, hey, what language I pick I wanted to you know I think games is what I said. I want to build games because as a kid, and that seemed fun. ⁓ And I was like, should I learn C, C++? And they're like, no, you should pick up Java because soon Java's gonna be faster than C or C++ because the JVM is able to see more about your code and optimize it beyond what a static ahead of time compiler can do. I'm still waiting for that to be true. But anyway, that was a tangent, sorry.
Glen (Plabayo): I get you. then, ⁓ yeah, mean, so there are a couple of things and later I want to go a bit back, just to continue a bit on this segment is the fact that, so I originally come from C++ and the thing I do notice in Rust is that the entire story around allocation is a bit like under the carpet. So what I mean with that is, so first of all, I don't think we have to worry about all the time. And so also when people learning Rust, were like clone a lot and I think that's fine. And like, again, it's not something we have to worry about that much, but at the same time in C++, we used to worry a lot about allocations and we would make sure we have like our own buffers that we reuse or let's say something like in networking, you would like reuse your buffer or you have certain loops where you often do the same thing. And I feel that arsed. didn't really translate very much from the C++ community and the C communities to Rust I feel. It's a bit in the background while if I then watch over the fence to communities like the ZIC language, their allocations are much more in your face, which has its own disadvantages, but at the same time I feel it's a bit too hidden in Rust sometimes.
Carl Lerche: ⁓ Well, I think that probably, it probably depends on what you're trying to build and exactly that level of ⁓ last performance you want to like kind of like that final bit of juice you want to squeeze out. I don't think for most software use cases, you really want to think about it that much. ⁓ It's kind of like premature optimization.
Glen (Plabayo): I do agree. I'm thinking more like in terms of like, like, so again, I totally agree there and I don't want to like push this as like something very important that the community has to worry about. I'm thinking more in terms of like library authors and framework authors that there is not often enough something where they allow you, let's say you are a user and you do need that often your only choice is then to fork that or to not use it because you have no control over how it allocates, even though everything else is fine.
Carl Lerche: unless... yeah. ⁓ yeah. Yeah, yeah, and that's true. I won't disagree with that. That kind of fell out and you're correct that fell out of Rust's decision to make allocation more built in assuming the default allocator and the custom allocators was added later. And I think it also falls out that Most users of these libraries don't care. ⁓ You're correct that some do. And it's unfortunate that sometimes they have to go and just fork the library or use something else or contribute a change. Though oftentimes that means figuring out how you're gonna contribute and that's... not trivial always, like forking and just hacking it in. I don't mean hacking it in, but just putting it in as a one-off that isn't designed for the entire user base of the library to use plus ongoing maintenance is easier than figuring out how to integrate it with a library for ongoing maintenance for everyone. But... I think it's also a testament that not that many people actually need to customize their allocator in practice. There are use cases for which that is true, but not many in the networking space is my take.
Glen (Plabayo): That's true, but like for example, it's also not very common that people like worry too much about like, I don't know, like for example, okay, I'm each time reading a request and I'm all the time just allocating a new like buffer for it and not really reusing it because it goes down this entire path, even though it's anyway like a very simple part in a way if you look at it very high level. So I don't know like.
Carl Lerche: Yeah. Is that true? Like oftentimes the way people do that is by implementing a buffer pool, right? And then the buffer pools fed around. I mean, I've not looked at hyper in a while, but ⁓ per connection, there are some concepts like per connection, right? So I would guess hyper does allocate per connection and also depends on how long that connection is expected to stay around, right?
Glen (Plabayo): Hmm.
Carl Lerche: and yes, there are going to be potential optimizations, but in practice, like if it's that big of a win, it would be a bigger sell. Right.
Glen (Plabayo): That's true. Yeah, yeah, correct.
Carl Lerche: So, and that is not to say there are not use cases where it does matter, but if you get to the point where it's not entirely true, so take everything with a grain of salt, if you generally, if you're getting to the point where that really, really matters. it might actually make sense for you to start like pulling back hyper for example, and like using the hyper state machine only and managing your buffers and your connections yourself. At some level, libraries are designed for most people to operate quickly and get their tasks done ⁓ without worrying about the details. And If you are building something where it's so critical that you manage that buffer, at that point, perhaps networking management is now key to your product slash application, which means it becomes something that it makes sense for you to manage yourself, right? There's that trade-off there.
Glen (Plabayo): Yeah, correct. And so now that we're talking about these low level details, I think you are the one that created the Bytes crate, right? Or at least you're one of its authors. So how did that get started and what was the idea behind it?
Carl Lerche: I did. That got started early on in my involvement in Rust. when I was in that phase of there's nothing in the Rust ecosystem, let's build it out. And there were two things that motivated the bytes design. I would say one major thing that motivated the bytes design besides just having that crate, right? So it is inspired by libraries that other languages have, bytes management, just byte buffers that are reusable and shared. But the thing that made the reason why bytes went with a reference counting approach and sharing is very much because of what early network async early non blocking networking was in Rust. I without async await with manual futures where everything had to be send and sync and anything that wasn't like rough you could not. Easily share reference. I mean you just couldn't you couldn't share references across await points, right? So That was a big source of friction with designing libraries ⁓ or using buffers or anything. just having something reference counted that you could copy, move around, clone cheaply and move into different things, like move into your futures made life a lot easier. Now it still is true that it's a useful crate and that it's... I just don't know how much, it is still useful, still applicable, but one of the main reasons for that reference counting thing is not really true anymore with async await. So oftentimes like maybe you don't need bytes. So the main reason I have bytes again is if you want to have one buffer with different windows, like different views shared from multiple places in your code base. That's the history.
Glen (Plabayo): Because you do say that and I do believe you and I've seen it happen but at the same time, like most people that example use Tokio, they do it in a work stealing mode, And in that case, you still cannot really deal that good with references, right? mean...
Carl Lerche: Yes. ⁓ you can within one task, right? So within one task, that's going to be pinned to one. Like one task, never references across. And I mean, I do think you have like the task is a abstraction points and you should be very careful in general about sharing any sort of state between tasks. Like it's, I would say it's, it's if you can avoid it.
Glen (Plabayo): Hmm.
Carl Lerche: Don't, just like you don't wanna share state between threads if you can avoid it. And you wanna really think carefully about the, ⁓ like how two tasks interact and how you're gonna manage that synchronization. Like there's a bunch of ways to do it. Channels is probably the best one to reach for.
Glen (Plabayo): Yeah, that's fair enough. I guess the other issue is more related to you earlier saying, how the async support in trades is still pretty limited, it's not always easy to propagate these kind of like send, sync and other properties ⁓ down these trade implementations. mean, once you start to play with the very edgy stuff like implement future return types and stuff.
Carl Lerche: Yeah. Yeah. But fundamentally that's a library problem ⁓ for the most part. Like, and I mean, that's a big problem for libraries, but, and async trait makes it easier. like the async trait proc macro. Like when you're building your application, like when you're building an applications, my kind of rule of thumb is avoid traits as much as possible and prefer enums, right? Like don't, like adding a trait has like cognitive overhead. And if all you have is two implementations of that, something like enum dispatch, ⁓ essentially just make an enum of the two implementations and you decide which one to use. time is fine.
Glen (Plabayo): Yeah, correct. that's kind of what I do indeed in my applications. It makes a lot more sense. Yeah, so to so to move a bit our conversation to a slightly different direction. io_uring is not that new. I mean, it's still a bit experimental, but the experiment but the experimental phase within Tokio has been going on for a while. And I'm not sure what the story there is as an outsider, at least.
Carl Lerche: Yeah, as an outsider, yeah, it's definitely probably confusing. What I would say is there was a lot of excitement about io_uring and there probably is, there probably are use cases for it. But I think based off of what the evolution and how I've seen experiments go and our experiments is that ⁓ like the... It was hyped a bit early on and the wins in practice for your average Tokio application are not. materializing, would say, in a way that we had hoped. So add that to the fact that io_uring has been, I mean, the interface itself has been evolving a lot. Plus there have been security issues with the implementation, CVEs and whatnot. And it is a much more complicated interface. ⁓ it's, I mean, it's for on Linux, it's almost definitely the way to go for files, but because of all those compounding factors and the lack of of real data showing that if we used it, we would get a material win. For most people, it does not make sense to use it. And that also could, yeah.
Glen (Plabayo): And you mentioned Linux, but I know there's a similar system on Windows. Is that also being considered or is that like... Yes.
Carl Lerche: ⁓ are you talking about IOCP on Windows? ⁓ if. I really know. Here's what I would say. I'm thinking about how to phrase this in a ⁓ recording friendly way. ⁓ If you look at the implementation on Windows for Tokio, you will notice that we have gone through great pains to avoid using IOCP.
Glen (Plabayo): I mean, is that part of the story or is it not even considered? you
Carl Lerche: And you will, to the point where we've used, and this isn't just Tokio, this is also Node.js and the whole bunch of other async abstractions. And you will notice that we avoid using it as much as possible in favor of biasing for internal APIs. And there is a reason for that. So.
Glen (Plabayo): And is Is that similar for Windows support in general?
Carl Lerche: ⁓ I mean, Windows support, I mean, we obviously, we try to make it as supportive as possible. has, Epoll is a really great battle tested, simple, efficient API and doing better than epoll is hard. It is true that for file access, pro mixing file access with ⁓ networking, you probably do want to use io_Uring in some cases. you can, the Tokio U-Ring crate shows that you can mix the two and that is probably sufficient for file access. Again, unless you're doing something incredibly focused and performance sensitive, in which case you may want to consider not using Tokio as your main abstraction. Like again, if you are at that point where really tuning it matters, it may be that that tuning is a core part of your product where it makes sense. For Tokio, we want to do the best we possibly can for the majority of people where Where it's as good as it can be out of the box, where these tiny small improvements don't make a material impact on your application. we prioritize, like we try to prioritize correctness, ease of use.
Glen (Plabayo): Hmm.
Carl Lerche: and safety and security, right? And that's obviously hard for a library to do for everybody. Everyone has different requirements. So I'm gonna be the first person to say there may be times where you don't need to use Tokio. I'll also say most engineers and I am a hundred percent in this bucket as well, tend to... underestimate the amount of work it takes to implement yourself and overestimate the benefits of doing so. So yeah, so if it comes a point where, and I think there is some, there's some early investigations. I mean, there's some work to bring io_u ring into Tokio for the file system stuff. ⁓ Again,
Glen (Plabayo): See you around.
Carl Lerche: tying networking and file system, like doing file system access on the same thread as networking in async context is either your file system access is not a core part of that loop, like you're reading config files or you're just reading, like you need to do some file system access like casually, in which case you probably, are fine moving it to a site like another thread, or you need to tightly control how that happens, how the interaction between your file system access and your networking happens, in which case, right now, the best, even if you use IOU ring, It's not that you want to use IOU ring with, you don't want to use Tokio APIs with IOU ring underpinning them. You want to use the full capabilities of io_uring, In which probably means hand rolling a bunch of stuff, right?
Glen (Plabayo): And so... to move the conversation in a slightly different direction is the fact that right now you work at Amazon, I believe. Does that mean they also invest? I mean, do they do it because they want you to work on Tokio and that's what you do there? is like Tokio is now more like something you do outside your work?
Carl Lerche: Yes, I am. ⁓ Yeah, I mean... No, working on Tokio and supporting Amazon's usage of Rust networking libraries is my entire job there. So I think the adoption story of Rust at Amazon, especially AWS has been like pretty crazy to watch. Firecracker, I think Firecracker was the very first introduction to Rust, but it was among that. ⁓ And since then, It's the adoption has been going like crazy at that infrastructure level kind of thing and rust is used extensively like EC2, know, DynamoDB, S3, like all those big services everywhere. Like it is growing really fast at Amazon and a lot. when they're building like at Amazon, when we're building infrastructure level services with rust, it's going to be with Tokio usually not always because sometimes there are cases where it makes sense to hand roll, but the vast majority of use cases it doesn't and Tokio's used that. So my job is to support Amazon's Amazon's usage of it, whether that is answer questions ⁓ or helping debug issues, but also adding features and kind of making sure supporting the ecosystem continuing development as well.
Glen (Plabayo): Okay, and so, I mean, the Tokio project has like a long history, you were there since the start because you created it initially. I mean, how How is maintainership going and the delegation towards a new generation and finding contributors, maintainers, etc?
Carl Lerche: Yeah. I mean... For Tokio, I have mostly taken a backseat for the past few years and Alice has been doing great, a great job with the day-to-day maintenance. And we also have just a team of contributors as well, besides just Alice, but Alice is doing great job leading most of it. And we talk, like we still talk, I still get involved for a lot of the, for the bigger decisions and I guess, but for the most part there, I have taken to step back and my kind of mission, I guess, at this point is to really look at how to bring Rust to higher level use cases. And I wouldn't say higher level use cases, but to make Rust appealing for the general developer population building web apps or network, or just generally networking applications that don't necessarily need stringent performance requirements, because what I see at Amazon, but also generally across the Rust ecosystem is, you know, once Rust is introduced within an organization, it tends to grow out from there and even to places where performance isn't a big consideration because of that reliability, that it becomes, and because Rust is productive, especially once you build organizational knowledge and experience and you start building internal libraries for it, like being able to share code that you build for one service across many without having to write it for many languages is a big productivity multiplier. So if you already have to use Rust for key components, using Rust elsewhere also makes sense. But that's, that's, ⁓ kind of growth story of using Rust elsewhere is hampered by a lack of higher level. libraries. I don't think there is a great ORM or database library for us. That's a really big project and it's a side project for me, but I'm working on hacking on Toasty on the side. ⁓ Though I do hope to get it to the point of it being ready for introduction in real apps by the end of the year. It's somewhat an ambitious goal, but I'm hoping to do that. . that full kind of how do I build a web app or how do I build a backend service that is talking to a like JavaScript front end or a mobile front end. ⁓ That story kind of is not as great with Rust and I'd like to see it better because I do think that there's potential for us to be great in that space. ⁓ Maybe not as productive again, initially for the prototyping as other languages, but over the long run when you're acturing in the entire needs of an organization, it starts making sense there.
Glen (Plabayo): Yeah, true. And so you mentioned Toasty. what is, maybe you can introduce the listeners a bit about Toasty and like what your goal is.
Carl Lerche: Yeah. So, yeah. So, Toasty at... is on the Tokio repo, Tokio GitHub organization, ⁓ Toasty. a, it's not really an ORM, but it kind of fills that need. I think of it more as a application level query engine, because I'm taking a lot of the same design principles of a query engine. ⁓ And kind of the idea is you define your model level schema, very simple, like you were working at that application model. layer and the database is somewhat abstracted, though the goal is to not make it invisible, but not make it in your face when it doesn't matter, if that makes sense. So you can always reach for database specifics, but for the most part, when you're building an application, you don't care. You just wanna like move forward, be productive ⁓ and worry about later kind of thing. So.
Glen (Plabayo): So for example, here at Plabayo we are also working on a couple of new commercial projects and for now I was thinking to just go for SQL and Postgres because that's what I know about. But would choosing Toasty at this point be an option? I even though it's experimental and edgy, would it already be useful?
Carl Lerche: Ha ha ha. that entirely depends on how much you're willing to contribute. I would love, I think it's very close to being at that point. It's if you try to use it, you're going to hit definitely issues and bugs and panics for cases that are not implemented yet that I meant to implement but haven't. And if you are willing to jump in and help fix those, believe it definitely, it won't be as fast as building with raw SQL right now because you're going to have to collaborate. But in the long run, especially if you are excited about the prospects of you using ⁓ higher level kind of database library that focuses on productivity and easy APIs. So one goal of Toasty is also kind of this theory that you can make Rust libraries much easier to use if you kind of focus on that and not that last kind of juice that, like don't focus so much on zero ⁓ cost abstractions. Yes, Rust can do them. ⁓ but they come with a cognitive cost at times and most of the time at that higher level you don't need them. So Toasty is trying to prioritize developer experience first with the theory, the hypothesis that... even with those productive productivity focused APIs, we can make it as like fast enough for most users with escape hatches when you need to, right?
Glen (Plabayo): And just to be clear, it's just a front-end, right? In the back-end, you still use something like Postgres?
Carl Lerche: Yes, it's going to be able to target SQL. ⁓ I'm also aiming to target ⁓ other databases than just SQL. Like think DynamoDB is one early one because I'm at Amazon and also it's a good one to target early ⁓ because of how it's modeled. Most key value kind of like NoSQL databases will also be able to be supported if DynamoDB is. the idea, and this is part of why I mentioned it's a query, it's an application level query engine. And the idea is not to abstract DynamoDB from your model. ⁓ So it's not that you can do any query that you can with SQL that will just run on DynamoDB. It's almost like the opposite. ⁓ But part of my kind of fears, like I've used ORMs and I've used NoSQL libraries and I've used all these things like throughout my career. And I'm always struck by how how 80 % is exactly the same and it's always like that last 20 % diverges. part of the hypothesis and it's still an experiment is can I base the API around that 80 % and then make that last 20 % like opt in via features or just generally dependent on how you use your, how you use the library. And interestingly enough, ⁓ finding that that last 20 % is not actually that different because what you need to really power a good data model with DynamoDB is also useful for power, like for more advanced use cases with a SQL database where maybe you're building application level schema or your application level schema doesn't directly translate to the database schema. Like for For one thing in DynamoDB is you denormalize, right? You have your core models and then for every query, if that doesn't fit nicely with your ⁓ primary key that you have, you need to denormalize and access it via that. But that kind of translates to materialized views or other concepts as well with SQL, with more advanced SQL. what I initially thought would be completely diversion is turning out to interestingly be ⁓ less divergent than I originally thought. as well, ⁓ if you kind of approach it as like this query engine, this application level query engine that knows about your underlying database capabilities, the idea is less to translate the query, but when you issue queries that don't make sense, it flags it. for you. So for example, if you do a query back, not a compile time, not, but at runtime kind of testing time. a lot, so like, my idea is that like, essentially there's going to be a checkpointing API anyway, but there's going to be a step around migrations. So usually when you're working with databases, like build, you do a whole bunch of work. You target your driver, right? So what will probably happen is when you run your tests or you,
Glen (Plabayo): You mean at compile time or just like a runtime?
Carl Lerche: you start implementing a query that doesn't work, it's going to panic at compile time or at runtime. And that's, that's okay. Then what you don't want is you don't want to get back to get to production. Then there's probably going to be an additional step when you commit your migrations and you're working like, okay, I'm done with my iteration. I'm going to generate migrations. That's going to be another step. That's more like a compilation time step, but you're committing it and it analyzes your queries and tells you, okay, this looks good. Or, this is. a query we can't officially run on DynamoDB. eventually you could even say like, don't like it could take, anyway, these are kind of crazy ideas I want to explore later. it's just an interesting observation I'm finding that supporting DynamoDB well and supporting SQL advanced cases with SQL is not that different.
Glen (Plabayo): interesting well sign me up you also yeah go for it
Carl Lerche: So yeah, I was gonna say the flip side is that makes the scope of Toasty very big and part of why it's taking so long.
Glen (Plabayo): Yeah, fair enough, mean, projects can take a while, I don't mind. So...
Carl Lerche: My superpower that I joke about is I constantly, I mean like everyone underestimates, but my superpower is drastically underestimating and then just jumping ahead first and powering through. Which is kind of like how we ended up building up all this kind of like rust ecosystem as well.
Glen (Plabayo): Yeah, mean, someone has to do it or rather some folks have to do it and yeah, and as long as you enjoy it, why not?
Carl Lerche: Yeah.
Glen (Plabayo): You also mentioned that recently, I mean, you didn't mention the podcast, but we talked about it before over chat and you were mentioning that you were dabbing into AI and not so much that you were on the hype, but you want to keep evolving and yeah.
Carl Lerche: yeah yeah For sure, and I know AI is controversial and there's lots of reasons why it's controversial and there's a lot of legitimate concerns and I am not going to touch on that. ⁓ That is not really a good focus. going to spend, I'd rather focus on the pure kind of ⁓ engineering questions at the tool level. ⁓ I will say that like, is AI a good tool, effective tool to improve ⁓ my productivity as a software engineer. Because I do think like while there are a lot of issues, I hope that they will be solved as things get better, right? Anyway, not to get too much into it. But as a tool, think, as software engineers, we're... responsible and like part of our job is to evaluate tools as they come from an engineering, like engineering mindsets and try to be as objective as possible. And every tool that's introduced has pros and cons. And I'm not going to say, and there's no silver, you know, silver bullet for everything. But we have to be somewhat objective as well. It's like, is this going to improve productivity where productivity is kind of like, can I get my job done faster within the constraints? Can I meet the requirements and the constraints faster? And the project is responsible for defining those constraints. fundamentally, not every software project has the same amount of stringent correctness requirements. And that's just a fact of life. I know we all Software engineers like enjoying taking pride in their work and doing a good job and that is good. ⁓ Reality is like if you're building a tool. for that's essentially not used that like a side project, not side project, but like a side testing tool or even like a backend for a hobby app or some small kind of thing that doesn't really, doesn't have a stringent requirements. Shipping some bugs occasionally and catching them later is probably okay. I mean, otherwise we'd all be using, let me rephrase. I'm not going, I'm not doing a good job. If never shipping a bug, was always a requirement, we would be using formally formal verification for everything. Right? We don't. Why? Because there's a productivity cost to using that. And there's a real trade off between reliability and productivity. So. If you assume that there's a real trade off, like you can look at, like you can evaluate AI, how much faster can I ship stuff? What is the real correctness trade off or the real trade off for that? And what I'm finding in my experiments of it is I am able to ship code faster within like reliability tolerances I'm okay with. And the reliability tolerance is going to depend on the project in question. And it's like the amount of productivity I'm gaining is such that it's really hard to ignore. is not just a little bit. is multi orders. Like I don't know if it's orders of magnitude. I don't have data, but it feels like multiple I am able to be multiple times faster using modern GenAI tools. So again.
Glen (Plabayo): And I know you're pretty confident because there were also recently studies where the engineers thought that but then if they were objectively checking it, it was turned out to be a lot less than they thought.
Carl Lerche: I think it depends on... ⁓ what you're building and the engineer in question, right? Like I think there are definitely uses of it where it is not adding too much. And that is tends to be existing code bases that are not onboarded. So it is a tool you have to learn. It is not magic, right? And I've not looked at the study. I'm just talking. So I could not talk about that study, but I can say like from experience, looking at other people trying to use it, looking at my own experiences using it. is a learning curve, like if you try to use it without really understanding where and how to use it, it's going to slow you down for sure. Right? So second, there's when you're introducing it to existing code bases, ⁓ there is kind of like you have to almost teach the tool how to work in that code base and figuring out how to do that is not obvious. Right? Third, Every AI tool is not the same. I don't know what AI tools these studies used, but I would tend if it was just on like the state of the art is shifting, is moving so fast that like. If you ran the study like three months ago or four months ago on whatever AI tools were popular, I don't wanna use names or like say what's good or bad, but I would say four months ago, I would not be able to be productive increasing my productivity with them. The most recent iteration of tools, I think are changing that equation. They're just constantly, they're getting better at a rate that is... Surprising, I guess, hard to imagine. It is both. So it is not just the models, because I've used tools that use the same underlying models with vastly different ⁓ results. the tool, it's basically, like, it's gonna be the lowest common in denominator.
Glen (Plabayo): Are you talking about the models? Are you talking about how you integrate it?
Carl Lerche: the model, it's whichever is the weakest link between the model and the tool, that is gonna be the quality of the output. when you, the better the tool with the better, the better tools with the better models are again, again, not a silver bullet. There are gonna be cases where there's lots of situations where they're not the right place to do it. And it's better to do it right now by yourself, like as a human. But when you know when to apply it, how to apply it, which tools to apply, It is a vastly different experience than using the wrong tools initially without that info. kind of my experience has been, I had that initial, like my first like two weeks using it, I would say. it was not proving successful at all. ⁓ But there was almost kind of like this aha moment once I got my flow down where it unlocked a lot. And a big part of that I do think is knowing when to apply it quickly, knowing how to, ⁓ and knowing when not to apply it.
Glen (Plabayo): And so maybe to give some lessons to our listeners, can you maybe tell a couple of... First of all, maybe make it also a bit more real, if possible, and mention at which project did you use it, what was the win there, and also you had to learn it, because you say it's important that you learn it. Can you share some lessons and what people can take away from it?
Carl Lerche: Yeah. So kind of assert struct, which was ⁓ a development kind of like a testing tool, pattern matching, an assertion for... ⁓ complex structures, of like imagine like if you want to do really fancy match statements pass of what Rust lets you do for tests. That's what assert struct is for. That was my kind of initial, let me try to build this from scratch. And my experience there is for the most part, if you have a well-defined requirements and you know how to test it and you know, like you have the well-defined feature set and you're able to express that clearly in a new code base, it works really well. I started having some problems kind of when I wanted to really nail down the... I want to really nail down the error messages and that's pretty fuzzy and that kind of like I had more friction there. ⁓ It was impossible, it's, it was definitely, it's possible that that would have been something that would have been better done by hand because it's a lot more fuzzy what you're actually supposed to Secondly, I do know that like, can't just let it, while it was really building a cert struct quickly, you obviously have to review the code and it does weird stuff sometimes though. I, again, modern tools seem to be, seem to do weird stuff a lot less the time. But I do remember one time, ⁓ a search struct was kind of my experiment of touch, like get in the way, like touch, look at the code and kind of touch it as little as possible and see what happens. ⁓ I noticed early on, like not early on, but like I noticed as was building, it kind of just. was hitting a wall and just hitting bugs and like not able to not generalizing. I was like, what's going on? I finally looked at the code and it was understand what's interesting. Like I saw the, it took the wrong algorithm algorithmic path and it was actually part of the error messages. So I think a combination of not explaining, not being able to clearly define the specification for how error messages should look plus the tool went down the wrong algorithmic path. ⁓ And I will not say that this is a like it's because it was AI. I, this is like, I kind of treat it as like a very eager, fast junior developer. Like this was a mistake that a human could like a less experienced human could make. Once I identified that and kind of was like, okay, step back and this is the algorithm you should take. I described it at a high level. It like we wrote everything. It was a lot smoother from there. So For, I would say as a rule of thumb for new code bases, it tends to work a lot better because there's a question of context. Smaller code bases obviously is gonna work a lot better. So if you're able to structure clean abstraction points, it makes it easier. I have dabbled with it for Toasty. It is harder to get it to use for Toasty because Toasty is such a big project with lots of moving pieces like in the engine that doesn't make it impossible. I will say for making changes, again, this would be an area I don't know. At a general, if I were to use an AI tool 100 % for just Toasty, I don't think I would be more productive. But at least up until my experiences now, I think that it is still an open question and open problem of how do you get these AI tools to onboard with existing complex code bases? So all my experiences with Toasty right now might be completely different in six months as a disclaimer, because I do think there's going to be ways of introducing AI your AI tools to a code base. Don't want to dig into what I've tried so far too much, but suffice to say for some things in Toasty that were more isolated, generating tests, generating like. ⁓ more like changes that are more isolated, any mechanical refactor, like I basically was able to churn through all of the larger refactors that I've been putting off super fast and that felt great. So those kinds of problems it's really good at solving. Anyway, I'm still like, I'm only like four weeks into it. So I don't have any groundbreaking. ⁓ I'm not like the expert yet, but I am very bullish, I will say.
Glen (Plabayo): Hmm, very interesting. think it teaches at the very least that we should all be open-minded and try to use them and go with the flow and see which tools come out, which integrations, models, etc.
Carl Lerche: But... Yes. What I would say is it would be foolish to ignore this tool. I understand why and I understand how. you're going to, like, it's going to feel like you can, I mean, as engineers, always think we do a better job than we can't. Like it's the urge and I'm saying this also, like I'm not, I know, I recognize this in me, right? Like every single problem, I think I can do better myself. And I don't think I'm the only one, I think it's traits. And taking your hands off is hard. in relying on reviews, and it also does take a willingness to not nitpick everything. And those skills of, what's funny is like, I don't wanna say this, it's gonna make people angry, but like the skills I built up, like work on open source and PRs and like reviewing PRs and just generally code reviewing. in general, I've built up kind of like a skill set, a willingness to like pick my battles and some things I'm not 100 % happy with, but I will just accept because it's not like, it's not worth nitpicking. Like if you get a PR from a person that doesn't know the code base super well, they're trying to do a ⁓ good job. You can't... always get everything you want out of that PR, like going back and nitpicking every single thing is going to be discouraging the person, it's going to kind of put them off the project. So I have over the years built up like a tolerance for, well, this isn't perfect, let's just move forward with it because it's acceptable. And that's... skill set, I guess, and acceptance also applies to working with AI tools, because you're not going to get exactly what you want. And it is almost like that 80-20, like you can get 80 % of the way there. Is that 20 % worth nitpicking or not? Oftentimes in general, it's not, or it's something you can come back to later. And it just lets you move faster. And as... Software as a craft, like I get that feels bad. Like it's going to be hard ⁓ to have a tool and put your name almost on something that feel that may not be 100 % what 100 % artisanally crafted software. Because I mean, again, I part of what I enjoy about software is that craft like software writing is ⁓ I mean, take like there's almost like there is like an art to it as well and there's an aesthetic to it. And a lot of code that I enjoy writing myself is I spend time making it look pretty in a way that I think is pretty, right? Maybe I'm the only one, I'm sure I'm not, but. there's like that pride of writing something you are proud of yourself that has more than just, ⁓ does it work and is it good enough to work, but also aesthetics and other such aspects and like, like. nice architecture patterns and everything's clean and I, I mean, I for one like doing that and you're not going to always get that with output from gen AI. And the question is, is that okay or not? And from a, like from a craftsman's point of view, it doesn't, it's not, I guess, like it feels bad from a reality of shipping software and meeting deliverables and business requirements. It probably is and knowing how to move forward with it in a way that is productive, like in a way that works, doesn't doesn't sacrifice quality to a point that it doesn't meet requirements, right? Is going to be the art of using this tool.
Glen (Plabayo): when you are not toasting or vibing, you also found time apparently to organize and start the Tokio conference? Like what's the history there?
Carl Lerche: Yes. So yeah, Tokio conference is going to be the very inaugural one in Portland, Oregon next year, April. I honestly don't remember the dates off my head. It's end of April. ⁓ The reason for organizing it is I. wanted to see, I wanted there to be a Rust conference one that's focusing on networking program, like building network software. So it's, while it's called TokioConf, it's not necessarily uniquely for TokioConf. It's more like async RustConf, but that doesn't have as much of a good ring to it. ⁓ But secondly, I wanted a conference that really focused on the professional software developer or like serious hobbyists, but really on topics to help professional software developers at their job, help them do better, help them grow their abilities, help them, know, ⁓ solve, like think about how to solve their problems, where they can do more, but generally. come, I want software developers that are take come to this conference and leave feeling like they it was they have a lot to bring back to their job. And I don't, I've not really felt like there was a conference in the Rust space that really focused on that combo of targeting the professional software developer in the networking space, because and it's a big space, part in part because a lot of the rest conference are like community focus conferences, which are also great. ⁓ Those are great conferences. There's nothing, I don't wanna say like there's anything wrong with them. Those are needed and they're high value and I like going to them, but they're less about ⁓ really providing value to professional software developers and more about community and like open source contributions and advancing the state of the Rust project. So that kind of, ⁓ the goal of TokioConf was to build that space for bringing value and talking more talking about how to do our jobs as professional software developers better. and less, it's less going to be about the future of Tokio. It's not about, it's not going to be about where should Tokio go from here. Like that is not the main point though that is going to be part of it because how can we do, like how can we bring value to professional software developers that might also be specifics around ⁓ how to improve Tokio or the Tokio ecosystem in general. So the CFP for, this is going to be for Tokio Conf will be opening very soon in September so I think it might actually be open when this airs I don't know putting any dates is tough for things that air in the future But that is going to be the goal and the topic and the kind of talks we hope to bring out. And so if you want to talk, share how you have been effective and what you've learned about using Tokio or Async Rust in your company, you've introduced it is gonna be a big thing. There's also like, where does it make sense? Where does it not? especially what I want to hear as well is if you have found that story of introducing for performance sensitive aspects and trying to introduce it to other aspects, what didn't go well there? Like what, maybe you abandoned that, maybe you ⁓ didn't abandon it, but things didn't go as well, but you've found ways to make it work. Anyway, those kinds of stories ⁓ are going to be, I think, what the kind of talks that we want.
Glen (Plabayo): And so, yeah, given this day and age, it's not easy always for everybody to reach Portland. Is it also like remote accessible and then both in terms of speakers as well as visitors?
Carl Lerche: You Yeah. So for. this conference we're going to make it in person only though recorded talks will be available after. Part of that is I know like Portland is kind of off on the west coast. The main reason it's here is it makes it easier to organize and I am available to do it. I will say that if you're able to attend that would be great if you don't totally understand it and hopefully if this is pulled off successfully we will have future editions of Tokio Conf globally and for other demographics. So yeah.
Glen (Plabayo): understand and so as we are wrapping up the episode is it anything more you wish to plug or mention?
Carl Lerche: you I think I mentioned everything so far, or have we gone on for a bit? I tend to ramble, it seems like. Hopefully you all ⁓ found it interesting, got value out of it. Don't flame me too hard for my thoughts on AI. Keep an open mind. ⁓ And I think it'd be worth just having that conversation, right? Because I will say I want to have the conversation of AI and software in an objective fashion, in a hopefully ⁓ in a way that is open because I see a lot of parallels. ⁓ with different inflection points in the software industry. I know I'm going back, but I'm rambling. Hopefully, don't, hopefully that's ⁓ okay. But .com as well, the shift to internet software, there was a big, there was a big pushback against writing internet software because real, I don't want to be dismissive because there was a feeling that software had to be run locally for it to work well. Then even before that, like you can look, I mean, this is before my time. but you can find trends of like compilers even when compilers were introduced right like there was a well compilers can't do as good of a job as a human that is very versed in this in like how to build software for platforms and that is true like I bet although I don't know that's true today I shouldn't say but initially it was true compilers were not as good as humans for building machine code, but they were far, more productive, opened up software programming for a much wider audience and resulted in building more and solving more problems. And I think... The exciting thing to me about this new tool, and again, there's a hype cycle. It is still too early to know where it's going to land, but I'm excited about what it will unlock people to build. It's going to change the calculus and the economics of software. not in a way, everyone worries it's going to make software engineers obsolete or companies are going to be able to do less with people are gonna be able to do more with fewer people that is true companies will be able to do more with fewer people but that's in practice not what happens like it means what in practice happens is push like lean into building more even more faster like solving more problems it's gonna be an exciting time I think
Glen (Plabayo): Alright, with that positive note I would like it to end. I love exciting times. And I want to thank... yeah... yeah, okay, even better, interesting and exciting. So thank you for showing up and having this nice conversation.
Carl Lerche: Interesting times no matter what. Yeah. No, thanks for having me again.
Glen (Plabayo): and take care.
Carl Lerche: Alright, have a good one.
Elizabeth (Plabayo): Netstack.fm is brought to you by Plabayo building secure, open, and resilient infrastructure with Rust protocols, and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcome sponsorships to keep building and supporting its ecosystem. The theme music of this podcast was composed by DJ Mailbox. If you enjoyed this episode, don't forget to subscribe on your favorite podcast platform and leave a five-star review. It really helps others discover the show. Thanks for tuning in. We'll see you next time for the next handshake.