{"version":"1.0.0","segments":[{"speaker":"Elizabeth","startTime":12.0,"endTime":77.0,"body":"Welcome to netstack.fm, your weekly podcast about networking, Rust and everything in between. You are listening to the first episode of our second season recorded on the 9th of July, 2026. In this episode, Glen talks with Jon Gjengset Jon is a principal engineer at Helsing. He is the author of Rust for Rustaceans and one of the best-known teachers in the Rust community. His open source code is used by projects across the Rust ecosystem. They talk about how to build software that stays fast under heavy use, how Jon decides what to build, and what it means to maintain code that other people depend on. Jon explains why even read only operations can cause lock contention. How to measure the slowest moments in a system and how to avoid latency spikes. When data structures need to grow."},{"speaker":"Glen (Plabayo BV)","startTime":77.0,"endTime":112.0,"body":"Hello, welcome to another week of Netstack.FM. It has been a while as we were working on our new platform for a podcast which is now live. In case you are also a subscriber to our RSS feed, you can also point already your URL that you are subscribed too to the new one, as we are now fully self-hosted with all our media and you can also with the new website use the web player. and follow the live transcripts that we manually edit. So with that being said, welcome in our virtual Studio Jon."},{"speaker":"Jon","startTime":112.0,"endTime":113.0,"body":"Thank you."},{"speaker":"Glen (Plabayo BV)","startTime":113.0,"endTime":122.0,"body":"Yeah, it's very nice to speak to you tonight. And I think I would like to first start with you giving a short introduction of yourself, if you want."},{"speaker":"Jon","startTime":122.0,"endTime":177.0,"body":"Sure. my name is Jon. I guess I do a lot of random different things, but I'll start with I'm Norwegian, and my journey has taken me around the world quite a bit. So I did my bachelor in Australia, where I ended up with a double general in computer science because I couldn't decide what I wanted to be doing. And so I didn't specialise on anything. I just took every subject. And then I did my master's in London, where I studied network computer systems. Then I moved to do my PhD at MIT in Boston, where I studied distributed systems and parallel and distributed operating systems and the like. and I in particular wrote my thesis on databases and database accelerators. then I joined Amazon, where I essentially built out the Rust build infrastructure at the company. so sort of the Rust build tooling at AWS in particular. and now for the past couple of years, I've been a principal engineer at Helsing, which is a European software defense company."},{"speaker":"Glen (Plabayo BV)","startTime":177.0,"endTime":195.0,"body":"Okay, that's quite a lot. And the fact that you took so many different directions, is that always because you were looking for that? Or it's more like you were looking for where can I best study this subject and that's where you went, or is it some relationship with university? I don't know, like how did that work?"},{"speaker":"Jon","startTime":195.0,"endTime":370.0,"body":"You know, for me, it wasn't all that intentional. So I had the intention to go to MIT since I was like 12, but at the age of 12 I had no idea what that meant, or that it was difficult, or that the like none of those thoughts entered my 12-year-old mind. It was more like, huh, there's some cool stuff and it has MIT's name on it. I wanna do those things. And so the first thing I did after graduating high school in Norway was I applied to MIT and I got rejected. and then I was like, well, I guess I should study or something. And so I started a bachelor's degree and then I applied again after a year. I got rejected. And then I moved to Australia and I didn't finish my bachelor's there. And then I applied again and then I got rejected. And then I moved to the UK and I got my master's from there. And only when I applied then the fourth time is when I got accepted. So it It was more of a well, I guess I have to do something else until the next time I apply. was at least the driver for the education part of it. And I and I think for work, you know, I've I've always wanted to do things that I think could be really interesting and challenging, but that I don't fully know from before. Like I want to step into things that are slightly uncomfortable. because I think it's The way that I learned the most, but also the way that the chance that your impact is bigger is also higher. and so for me, you know, trying to trying to build out the Rust infrastructure at Amazon certainly felt ambitious, yes, but also something that I didn't know how to do. I'd d I'd done a bunch of work on like, you know, building random software because that's what you do when you're in open source maintenance. and so I had some of those skills and I've it was very familiar with the command line, with build tools, with the Linux kernel, with distros. And so I felt like I had many of the pieces and obviously very familiar with Rust. And so I felt like I think I could do well at this, but there is going to be a big learning curve. But that was part of the attraction. And I think for Helsing it was the same thing where, you know, I was like, well I'm moving to Europe now and I working for Amazon when I moved was not necessarily trivial, especially because Amazon was in their return to office wave at the time. and I was looking for where's somewhere where I can do something that I I don't think it's obvious that I know how to do, but I think I have some of the skills that would come together to let me do this really well. combined with the fact that especially at the time, and I think still, the situation in Europe and the world was such that I felt like this was a place where I could make a a difference outside of just software engineering as well. and so those things came together to say that's where I want to be next. I don't know what that looks like, you know, down the line eventually. I like it's it's more that my, you know, choice function is things that I'm slightly uncomfortable with, but where I think I can really grow and make a difference. and you know, combined with interesting things to work on."},{"speaker":"Glen (Plabayo BV)","startTime":370.0,"endTime":382.0,"body":"Yeah, yeah, but that makes sense because if you want to be passionate about it, if you're hungry for learning, you will also really apply yourself. So I think as an employer you also want a new employee. So I totally get that. Now your Yeah."},{"speaker":"Jon","startTime":382.0,"endTime":515.0,"body":"I mean I think this is also actually how I started doing the education bit, which I suppose I forgot to mention in the intro, but I also do Rust education of various kinds. but the way that started was during my PhD, was sort of when the Rust community started evolving, because I started my PhD in 2014, and I started looking at Rust in 2015, which is the release of 1.0. and you know, as a result, I I find programming languages interesting. I like experimenting with them and see what they can do. And so when I was starting a new iteration of my research project, I went, hmm, the a database has a bunch of concurrency in it and it needs to be safe and it needs to be fast and it needs to be low level and everything. So how about I try this wacky new language from Mozilla? And then I started working with it, I really enjoyed it, and then I built this whole database in it, including a lot of, you know, the the early days of Tokio where I helped build part of part of that runtime and I got more and more involved with the community. And then I think it was in the 2017 or something, State of Rust survey, where one of the conclusions of that survey was there are not enough intermediate teaching resources for us. We have a lot of early like beginner resources, but not a lot for people who know the language now. They want to use it, you know, in anger, so to speak. And I went, well. I think I could do that. again slightly uncomfortable with it, but I feel like, you know, I I have the necessary skills. I've been doing some I've been like a teaching assistant, I've been enjoyed teaching for a long time. I've used the language a lot, so I know those pieces. And then I went, well, all right, let's figure out what people want. And I did a just a like a Twitter poll, and I think I got maybe ten answers or something, right? Where people to my great surprise, wanted like live streamed video content, which to me is weird because that's not how I learn. that's what people wanted. And so I was like, I guess I'll order a microphone. and again, I think it's it's because I saw it as a sort of opportunity for doing something that's slightly uncomfortable. But if it goes well, it'll be a really great learning opportunity. And I mean, lo and behold, it certainly has ended up being that."},{"speaker":"Glen (Plabayo BV)","startTime":515.0,"endTime":536.0,"body":"Yeah, yeah, and of course I imagine as you research for these different videos and as you do them, I imagine you also learn a lot on the way because it's only when you teach that you're really challenged like how well do I really know this? And probably quite often you realize, man, I know so few about this. How will I explain this?"},{"speaker":"Jon","startTime":536.0,"endTime":577.0,"body":"Yeah, it's very true. certainly for some of the videos that I do that are much more sort of deep dives in a particular topic. So I have some where I just build things and and building things is very fun, but it's also something I'm the most comfortable with. but when I do the deep dives, like I have some of these videos called Decrusting videos where I take a crate like Tokio or Serde and I try to figure out like explain how do they work on the inside and what what are all the APIs for. And that even though they're crate These are crates that I know well. The amount of extra depth that I need to know in order to be able to explain the things that are not as deep is a great learning experience for me for sure."},{"speaker":"Glen (Plabayo BV)","startTime":577.0,"endTime":581.0,"body":"how much prep time do you take for some of these Decrusting videos?"},{"speaker":"Jon","startTime":581.0,"endTime":645.0,"body":"so it really depends, usually very little, because I actually think that me sort of learning on the fly in the video is part of the learning experience. Like, for example, even in a Decrusting video where I do prepare a little bit more in the sense that, you know, I've used these crates for a long time, but I go read some of the source before I do the video and everything. But even there, if I, you know, realize This API, I know I know how to explain what it does and roughly how it works, but then I get a question, for example, in the chat that asks for something more specific about its behavior. And I go open up its source and dig a little bit through. And I think that exploration pattern of how I figure out how to answer that question is itself a useful thing for people to learn from. To see how I explore a code base, how I reason about its abstractions, and how I sort of unpack an explanation. from the sequence of of code that I see, I think is is part of what takes you from beginner to intermediate to to eventually a sort of expert programmer."},{"speaker":"Glen (Plabayo BV)","startTime":645.0,"endTime":655.0,"body":"Yeah, and your videos are very helpful to many and it's even appreciated by the maintainers I see because I think I believe your Serde video is even on their official documentation page. So seems to"},{"speaker":"Jon","startTime":655.0,"endTime":657.0,"body":"Yeah, it's true. It's true."},{"speaker":"Glen (Plabayo BV)","startTime":657.0,"endTime":678.0,"body":"explain it very well. now you were telling, okay, I'm very comfortable building things, but were you always comfortable exposing yourself like this? Because in the end you're built in the open. you really can not hide. You will see your flaws. If you can call it that, you will see that you don't know certain things. Were you always comfortable with this?"},{"speaker":"Jon","startTime":678.0,"endTime":694.0,"body":"I think so, and I don't really know why. Because e even if you if you search far enough back, you'll find like forum posts from ages ago, many of them in a region where I'm like, I remember there was one where I was really against CSS way back in the day. I was like in"},{"speaker":"Glen (Plabayo BV)","startTime":694.0,"endTime":696.0,"body":"Ha ha ha."},{"speaker":"Jon","startTime":696.0,"endTime":797.0,"body":"favor of table-based layouts. I'm like, I've always been someone who, you know, I want to have the discussion, I want to Like learn if I'm wrong. Like if if I have an incorrect perception or I do something incorrectly, I'd rather be pointed out so I can learn from it. and so I think that's been part of it, that that sort of mentality I've had from early on. and then I think the other part of it is like there's something desensitized in my brain somehow where I'm just like not that worried about being I don't know, criticized or or made fun of or or embarrassed, maybe. and I think part of this actually, I so for many, many years, I worked at a summer camp for kids here in Norway. And at that camp, one of the things that you shed very early on is your your sense of embarrassment. Because when you're around these kids all the time and you try to keep them entertained and whatever, and as part of that, you make a fool of yourself because that is one of the ways that you entertain kids. And you just have to like check your embarrassment at the door. And so I think I've built up a lot of resiliency of like. I will be laughed at or, you know, I've will make mistakes and that's part of the fun. And that's okay. and I think that has been a sort of accidentally helpful lesson in my professional life as well. So that could be part of where it comes from as well. But I think I've I've always felt very strongly this need almost to share the things I do. Because I and maybe this is related to I have an aversion to other people having to redo work that I have done. Right. And so if I build a library for myself, I want to make sure no one else has to ever do that. and the only way to do that is to share that code."},{"speaker":"Glen (Plabayo BV)","startTime":797.0,"endTime":832.0,"body":"Yeah, that makes sense and and talking about libraries, if people would look at your crates.io profile, I mean I've never seen someone having so many crates and especially so many useful ones and a lot of them are used by a lot of people and I don't really find a particular common theme except that they are really focused on one topic and they are done well and often it's around I would say high performance because 'Cause you have like a lot of like lockless things or collections or maybe some things around runtimes, I don't know. like I don't know, like how how do you"},{"speaker":"Jon","startTime":832.0,"endTime":834.0,"body":"Yeah, it"},{"speaker":"Glen (Plabayo BV)","startTime":834.0,"endTime":839.0,"body":"get to these? Is is it's just like, let's make this or or is it because you need it?"},{"speaker":"Jon","startTime":839.0,"endTime":933.0,"body":"i it's a little bit of a collection of things. So so some of them are I just found the problem interesting. Like you'll find some that I built over the course of streams, for example, just because it was interesting. like for example, there's there's one on there called ordsearch, which I don't think I built on stream, but it was basically I read a research paper about a new algorithm someone had found for If you have a collection of numbers and you want to be able to do the query of find me the largest value that is less than or equal to some query value. don't remember why I read this paper, but I read it and I was like, that's a cool algorithm. And it it claimed that it was significantly faster on modern hardware than a lot of the previous ones, but it had all of its code was in C. And I went, well, how about I try to implement it in Rust? That sounds like fun. and so I went ahead and implemented and wrote all these tests and benchmarks and ran a bunch of experiments and then it ended up not being faster. I think now it is because someone found some like bugs in the stuff I put out there. But it was an interesting learning experience for me, not only in porting the research paper and learning the algorithm, but also because I learned more things about performance optimizations on modern CPUs. Like Why was the algorithm designed in this way? Well, it was to make use of these parts of the CPU processing pipeline and these assembly instructions are available here. And I wouldn't have exposure to those kinds of things if I didn't end up implementing something that used them, if that makes sense."},{"speaker":"Glen (Plabayo BV)","startTime":933.0,"endTime":946.0,"body":"does make sense. But then also what I wonder is okay, at that point you implemented and and also because they are so well defined in scope you seem to be pretty disciplined in keeping it well within scope, it"},{"speaker":"Jon","startTime":946.0,"endTime":984.0,"body":"I think it's partially that and just and partially that I just really enjoy making building blocks. I like to complete a crate, which is easier said than done, but and you can only do this if you scope them down a lot. but I try to pick crates where I can build the thing and sort of declare that it's done now. And I haven't perfectly managed this. There are some crates I maintain that are not like this and that that sort of have to keep evolving. but in general, that is one of the things that I look for. and I consider it a sort of feature, not a bug, if I have a crate that I have not touched in a few years, then I've then I think I've done well with that crate."},{"speaker":"Glen (Plabayo BV)","startTime":984.0,"endTime":998.0,"body":"What I sometimes wonder is because sometimes people look okay, this crate is maybe not updated in a couple of years. Would it be maybe useful that there would be some kind of way as a maintainer that you can label okay, this is feature complete so that people like yeah."},{"speaker":"Jon","startTime":998.0,"endTime":1040.0,"body":"Maybe. I mean, this is arguably what David Tolnay did with the serde_yaml, right? Was sort of this is complete now. I'm not doing anything more with it. It works perfectly fine. There are feature requests and everything, but none of those are gonna happen. And you know, the tricky thing there is maybe something around, well, what happens if there's a critical security vulnerability found, for example? and I think that's one of the reasons people had a sort oh, serde_yaml isn't maintained anymore. reaction, but the intent I think was it's done. Like this crate is done. There isn't anything more I'm gonna add to it. There aren't any features I'm there's no major known bugs to it. Like it is just this is the state of the crate and it will remain like this in perpetuity. and I think there's"},{"speaker":"Glen (Plabayo BV)","startTime":1040.0,"endTime":1041.0,"body":"but of course"},{"speaker":"Jon","startTime":1041.0,"endTime":1042.0,"body":"some attraction to that."},{"speaker":"Glen (Plabayo BV)","startTime":1042.0,"endTime":1044.0,"body":"I fully agree. course the but it it what yeah."},{"speaker":"Jon","startTime":1044.0,"endTime":1050.0,"body":"But it is true, you're right, that there's no way to mark a crate as being done in that way. And I think that's partially"},{"speaker":"Glen (Plabayo BV)","startTime":1050.0,"endTime":1051.0,"body":"Exactly."},{"speaker":"Jon","startTime":1051.0,"endTime":1060.0,"body":"because there are very few crates that would meet those criteria, right? Most crates are going to keep having at least some kind of evolution over time."},{"speaker":"Glen (Plabayo BV)","startTime":1060.0,"endTime":1067.0,"body":"yeah, for sure. I and you mentioned the YAML crate and I don't know, is it is it a consortium or is it like I don't even know who's behind that, but I"},{"speaker":"Jon","startTime":1067.0,"endTime":1068.0,"body":"Yeah, I'm not sure actually."},{"speaker":"Glen (Plabayo BV)","startTime":1068.0,"endTime":1079.0,"body":"I usually with these things is they cannot leave it alone and you will get some new versions and people add stuff to it and then everybody who maintains a YAML crate probably has to I mean that's very different than all of your crate. I mean a less"},{"speaker":"Jon","startTime":1079.0,"endTime":1080.0,"body":"Yeah. No, it's true."},{"speaker":"Glen (Plabayo BV)","startTime":1080.0,"endTime":1081.0,"body":"yeah."},{"speaker":"Jon","startTime":1081.0,"endTime":1166.0,"body":"It's true. but you will find some my crates that are more like this too. Like if you look at Fantoccini is the you know one example I know we'll talk about later. Faktory is another one where so the maintainer of a tool called Sidekiq, which is like a job queue runner, very widely used. And the author of that made a basically a new program that does much of the same called Faktory. and this one has like an enterprise tier and everything, but like there's an open source version which is factory, and then there's an enterprise version that has a bunch of fancy features. And I saw it and I went, like, this looks kind of cool. And then I went to look at the protocol and I was like, this doesn't seem that bad. How about I implement a protocol for it? I had no need for it. There was nothing, I didn't use it for anything. I didn't have any service I wanted to use it for. I just went, This is an interesting protocol, it's not that bad to implement, and it seems like a useful service. Let me go implement it. and that crate is now fairly stable. but at this point, like, new features keep getting added to factory itself, and so therefore, that crate also needs to evolve with time. It can't quite stay stagnant. and that does make them more annoying to maintain, right? It makes it harder for me to hold on to quite as many crates as I do. Like, arguably to the point where it's now becoming a problem that I have. so many crates that each only require a little bit of attention, but that means I can't give the attention to all of them that they need."},{"speaker":"Glen (Plabayo BV)","startTime":1166.0,"endTime":1234.0,"body":"Yeah, yeah, I get it. And then That brings us to an interesting point, I think, is is you have some crates where you clearly had a purpose and then some were just curiosities. And then if I look at my own developer experience and if I look at projects that I also have like that, okay, some are like passion projects and then some are like okay, I have a real need for it. So I guess passion project is like a wrong name because everything is a passion, but still so you have these things no need, the things do need. And what I what I find for myself is When I have like real production use from one or several companies for something, I really get into these mental warps and hoops because they they run into all kind of scenarios that I didn't even like imagined and it forces you really almost like a rock through time to really form and shape and and rethink and then I wonder if you reflect on these crates that you didn't ever need for do you find ways to anyway get through those hoops or is it like it is whatever it is and if ever somebody really cares, they will find it."},{"speaker":"Jon","startTime":1234.0,"endTime":1451.0,"body":"So I think maybe a good example here is actually the IMAP crate. So the IMAP crate was not my original invention. It was made by someone who Forget their last name, but Matt and something. anyway, so this was the IMAP crate was was owned by this person who I had some early interactions with, and I contributed some things to the to the crate, and then they just sort of vanished. I don't know what happened. They just stopped replying to things, and they disappeared off the face of the earth, as far as I could tell. I had their email, didn't get any reply to that. The only thing I got was at one point they left a comment on a GitHub issue saying. Something along the lines of like, I'm willing to let Jon take over this crate. And then I asked some follow-up questions like, can we can you hit the buttons here and here to grant the access? Never heard anything back. and at that point I was like, Well, I at that point I was using the IMAP crate. And I was like, Well, I want this to keep working, so let me I guess I'll take it over now. and I managed to go through the crates.io team and do the whole ownership transfer process. It was a whole ordeal. And what ended up with was basically me sort of Adopting the IMAP crate in the form that it was, I didn't have enough time to rewire it the way that I wanted it to be. I it arguably should be completely rewritten. So what happened was I sort of maintained it over time and made sure that it worked for the use cases I had for it. But I didn't quite do the whole like mold it into the thing it should be. and that wasn't because I didn't have the need or didn't see it, but simply because. if it's a side project, if it's not something that you're doing for a client, for example, it's very hard to motivate yourself to say, I'm gonna do a from scratch rewrite of this thing in a way where it's now significantly better, because it takes so much time and effort, as as you're correctly pointing out. and I simply didn't have that time to invest. But the downside of this is now that create has gone to the point where I know it needs to be rewritten, but I cannot do it. And I also can't really review the things people want to do with it because the IMAP spec evolves and the extensions to it evolve. And so there's quite a lot of traffic coming to that repository. and everything, all the contributions become more painful because the architecture of the crate is wrong. But I can't fix that without investing the time. And I don't want to land more things in it because they shouldn't land in this form. and so the net result there was basically I sort of declared bankruptcy of the on that crate and I started a thread that basically said maintainers needed and l pinged in a bunch of the people who'd sent PRs who discussed things on on issues and said someone else needs to take this over now. And I tried gently for a little while and then eventually I was like, no, this need needs to just stop. and I eventually got some people who picked up on it and was like, they'd either maintain their own fork or they'd started to write a thing from scratch for themselves. and at this point I think the answer really is for someone else to just write An IMAP crate, and then take the IMAP crate name and just issue a new major release of it, and it's a completely different crate. And I think that's the path we're gonna end up taking. There's actually someone who recently posted to that thread, and I really owe them a reply, saying we now have a production-ready IMAP crate that we use for like this product at work, so it's like a real thing. It's a completely different architecture and it uses some patterns that I don't necessarily think are the right ones. But it doesn't matter. It is still significantly better that it is maintained by someone who's able to do that low, slow, steady sort of pressure cooking and shaping that's needed to make this the crate it really deserves to be."},{"speaker":"Glen (Plabayo BV)","startTime":1451.0,"endTime":1467.0,"body":"I can already say well done. I do think you handled that very well. You came to a reflection, you realized what had to happen, you communicated, you dared to take this choice because when something is like that, used by so many people, a not a lot would dare to do that. I"},{"speaker":"Jon","startTime":1467.0,"endTime":1515.0,"body":"But it's tough, right? You end up in a sort of you end up trapped between a rock and a hard place because you feel like you owe it to the uses of the crate to keep maintaining it. But you also know you're not maintaining it the way it should be. and you can't afford to put more time in it because then your other commitments go off the side. And thIs is no solution to it except to say, you know, put your hands up and say help. and ultimately that's what I did with that crate, is I simply Cannot be the maintainer of that crate anymore and and sort of do it justice. And it it was painful. Like there was certainly a barrier to doing it. But I think I, you know, where I landed was it is worse for everyone if I don't do this. Like if I don't raise my hands and say I won't do this anymore. It is worse for everyone involved. And so"},{"speaker":"Glen (Plabayo BV)","startTime":1515.0,"endTime":1515.001,"body":"Yeah."},{"speaker":"Jon","startTime":1515.0,"endTime":1517.0,"body":"therefore I would rather do that."},{"speaker":"Glen (Plabayo BV)","startTime":1517.0,"endTime":1551.0,"body":"I totally agree with your choice there. And yeah, let's see what happens with it. It's indeed a a commonly used crate. another, maybe even more used crate, even though I didn't check the numbers, is a crate like arcstr. And then because in theory you have something like &'static str, that's most easy case. You have sometimes what people call like a Box<str> because they they just need a string but they don't really need to modify it, so like a read only string. And then you have something like like an arcstr and that's in a way you created also once it's also a read but you can share it which is different than a Box<str> because you would need to clone it each time you want to yeah."},{"speaker":"Jon","startTime":1551.0,"endTime":1556.0,"body":"It's it's actually it's actually not even arcstr. That crate is called arccstr because it's"},{"speaker":"Glen (Plabayo BV)","startTime":1556.0,"endTime":1557.0,"body":"look at that."},{"speaker":"Jon","startTime":1557.0,"endTime":1589.0,"body":"actually a C string, so it doesn't store the length, it only stores an Arc that is a pointer to the first letter, and the assumption is that it's null terminated like C strings are. And the reason for this is because I wanted it to be as small as possible. And the this one I actually had a use case for it, which is I needed to store lots and lots and lots of strings. and they were all very short strings, and so every byte for the storage of every string mattered. And this was the smallest possible string representation I could come up with."},{"speaker":"Glen (Plabayo BV)","startTime":1589.0,"endTime":1593.0,"body":"Yeah, I see. and what are then the trade offs, because that's very different than a regular string."},{"speaker":"Jon","startTime":1593.0,"endTime":1668.0,"body":"Yeah, so so so it has trade-offs like if you want to look up the length of the string, the only way to do so is to count it. So the length method is linear in the length of the string. Whereas in you know a standard Rust string, for example, looking up the length is constant time because the length is stored. it also means you can't store strings that have null bytes in them, because null bytes terminate the string and so That's not valid. This is the same limitation that C strings have. And they're not mutable because they're stored behind an Arc. And so that sort of naturally limits their mutability. But I think those are the three primary things. The crate documentation, I think, is a decent breakdown of like when might you consider using this and then all of the reasons why you shouldn't. but I like building things that have that kind of flavor of trade-off where There are some very specific cases where this will be a great choice. And there'll be lots of cases where you should not use this. so I think left-right is another example of this, which is a concurrency primitive that gives you really, really fast concurrent reads to a data structure, but it makes writes very slow and it doubles your memory use."},{"speaker":"Glen (Plabayo BV)","startTime":1668.0,"endTime":1679.0,"body":"Mm-hmm. and for example another crate in this ecosystem for that same problem is arc-swap. And how do they compare? Like what is the different trade-offs that they took there?"},{"speaker":"Jon","startTime":1679.0,"endTime":1735.0,"body":"arc swap is a little bit different because arc-swap does not keep just two copies of your data. What it does is effectively, when you want to write, it does an atomic swap of where the arc points. And then anyone who was using the previous arc keeps having an Arc to that, and anyone using the new one is now following the new pointer and using the thing it's now pointing to. but what that can mean is you can have As many copies of your thing in memory as you have readers. Because in theory, each one could have read in between when there was a swap, and all of them get kept around. the other problem it has is that every time you do a write, you need to copy the whole data structure that you store behind the arc. Because you basically need to clone it and then make that the new arc. but the upside that it has is You can keep around multiple versions, right? that's a a big win. and"},{"speaker":"Glen (Plabayo BV)","startTime":1735.0,"endTime":1736.0,"body":"We know."},{"speaker":"Jon","startTime":1736.0,"endTime":1776.0,"body":"the other thing that it gives you is it gives you pretty easy memory reclamation. So the the API can be pretty straightforward for using this thing, precisely because it's effectively an Arc. left-right has some properties around the readers where you need the sort of you need a kind of guard around your access because otherwise the memory reclamation wouldn't work. so the ergonomics of an Arc swap is also a little bit easier. But the read performance of left-right should be significantly higher because you don't even have readers contending on the reference count when they drop and grab new copies."},{"speaker":"Glen (Plabayo BV)","startTime":1776.0,"endTime":1778.0,"body":"Hm, interesting. But"},{"speaker":"Jon","startTime":1778.0,"endTime":1784.0,"body":"Well, this this actually depends a little on your exact usage pattern. but they they do have different trade offs here."},{"speaker":"Glen (Plabayo BV)","startTime":1784.0,"endTime":1796.0,"body":"Yeah, that makes me think that in arc-swap I guess you can you can really get into some kind of invisible branching isn't that what you're saying? Because the reader can basically write and then it becomes it's almost like a Cow (clone-on-write), no? Like where you"},{"speaker":"Jon","startTime":1796.0,"endTime":1839.0,"body":"Yeah, it's all almost like a Cow. I haven't looked enough at the implementation details of arc-swap to tell you what the exact algorithm they use is. But I could certainly imagine that you could end up with this sort of branching where every reader could end up having their own copy of the data. I think in practice that's unlikely because application patterns are such that you're more likely to have readers all branch off from one place. And so the writes all happen in one place, and then every time a reader comes, they'll sort of peel off the current state of the data structure and carry that arc along with them until they're done processing. They're unlikely to branch further after the initial fork off, because they're unlikely to do writes afterwards. That would at least be my guess. but but"},{"speaker":"Glen (Plabayo BV)","startTime":1839.0,"endTime":1839.001,"body":"I see."},{"speaker":"Jon","startTime":1839.0,"endTime":1860.0,"body":"it it also depends on how they implemented this, right? Whether if you do a write on a already cloned or or forked arc, does that end up with another fork or does that end up swapping every arc that this was ever cloned from? I think it's the latter. I don't think you actually get this sort of forking tree, but I'm not"},{"speaker":"Glen (Plabayo BV)","startTime":1860.0,"endTime":1860.001,"body":"Mm."},{"speaker":"Jon","startTime":1860.0,"endTime":1861.0,"body":"sure."},{"speaker":"Glen (Plabayo BV)","startTime":1861.0,"endTime":1863.0,"body":"Yeah, yeah. that would be otherwise."},{"speaker":"Jon","startTime":1863.0,"endTime":1877.0,"body":"But I like the I like these kinds of trade offs, right? Like data structures where whether you choose A or B is a very, very nuanced question, but if you choose correctly, it gives you really good performance characteristics, for example, or ergonomics or whatever it might be."},{"speaker":"Glen (Plabayo BV)","startTime":1877.0,"endTime":1896.0,"body":"Yeah, I mean that makes me think, let's say you have some kind of really heavy data structure and you're only ever updating it, meaning writing from one background thread, then I guess something like left-right is the proper choice because because you have a lot of readers and only one writer. So there's there's not even any contention possible there."},{"speaker":"Jon","startTime":1896.0,"endTime":1899.0,"body":"Yep."},{"speaker":"Glen (Plabayo BV)","startTime":1899.0,"endTime":1906.0,"body":"and then how does it exactly work that the fact that you Okay, you're right. How how do the readers then know, okay, now I"},{"speaker":"Jon","startTime":1906.0,"endTime":1909.0,"body":"There there's we we could spend an hour talking about this. And I know"},{"speaker":"Glen (Plabayo BV)","startTime":1909.0,"endTime":1910.0,"body":"Okay."},{"speaker":"Jon","startTime":1910.0,"endTime":1913.0,"body":"this because I've given an hour a lecture on this data structure. So so so I don't yeah, the"},{"speaker":"Glen (Plabayo BV)","startTime":1913.0,"endTime":1916.0,"body":"look at that. We we should link to it."},{"speaker":"Jon","startTime":1916.0,"endTime":2019.0,"body":"we'll put a link in the description or in in the show notes for this. But very basically what it does is every reader keeps a a sort of counter that tracks how many times it's accessed the database or access sorry, accessed the data structure. And anytime it access the data structure, it increments that counter by one. And what the writer will do is when it wants to write, so there's you keep two instances of the data structure. imagine to begin with that all of the readers are in the left copy of the data structure, and the writer has exclusive access to the one on the right. When it wants to expose the writes that it's done to the one on the right, it swaps an atomic pointer at the bottom that points to one or the other that all the reads go through. And the reads now start funneling to the right data structure instead of the left. But there's still some readers that are accessing the left one. So now you have some on the left and some on the right. But over time all of the readers will start to be in the one on the right. When the writer wants to do another write, what it does is it reads the counters of every reader that's known and it waits until it's seen all the counters tick up by one. Because that way it knows that the reader must have read the updated atomic pointer, which means that it must no longer be looking at the left one. It must now be looking at the right one. As when all of them have ticked over, then it knows that the left one is now empty of readers. And therefore it knows that it can now mutably access the left one. and then now you've swapped. And then and then when you want to do it again, you swap back. So you only ever have those two same copies you keep updating. has a bunch of nuances to how to get this right and how to like keep them in sync and everything, but that's the very basic algorithm."},{"speaker":"Glen (Plabayo BV)","startTime":2019.0,"endTime":2041.0,"body":"Okay. and for example with a crate like arccstr. I understand okay, that's super scoped. So I can imagine once you are done you're pretty much done because it's also very simple. But then something like left-right, I imagine that you after a month or six months or a year, you might realize maybe this could be better. I don't know, like I feel yeah, or not."},{"speaker":"Jon","startTime":2041.0,"endTime":2087.0,"body":"kind of. So so with left-right, it's more it's it's very sporadic. So it's like every now and again, and it's it's it doesn't happen on a fixed cadence. It's like effectively random. there will be either I or someone else will realize, wait, either this is slightly nuancedly wrong, like the concurrency doesn't match up, or there's a performance optimization we can make here to make the concurrency better, and then we'll go update the crate. But it's it's actually quite rare. like the crate it has been stable for a while now. I know actually there's some open issues and PRs on it right now that I haven't had a chance to look at yet. So maybe someone has thought of another either thought of another potential bug, like a potential concurrency race, or have thought of an optimization or both. we'll find out when I go look at them."},{"speaker":"Glen (Plabayo BV)","startTime":2087.0,"endTime":2094.0,"body":"I see. And how easy is it for you to not look at these tickets? Because I would be very curious."},{"speaker":"Jon","startTime":2094.0,"endTime":2225.0,"body":"I am very curious as well. The big problem is time, right? And this is the same reason why I ended up sort of declaring bankrupt bankruptcy on the IMAP crate was I simply don't have the time to maintain this many things well. I could maintain them poorly, but that that's not I don't want to do that. I'd rather give it over to someone else and hopefully they can maintain it well. my GitHub notifications list, it it's now at a manageable number, but Even just like a week ago, I think it was at 300 notifications. And that's after getting rid of the ones that are like, random watch things. and it spans a ton of different projects. So it's not like, if you are a contributor to the Rust compiler, for example, I could easily imagine you get to a similar number just because of the amount of traffic that you get there. But for me, it's like, I have Three notifications in this repo, and five in that one, and two in this one, and three over here. and so it's it's a lot of context switching for any one of them. And so I kind of need to prune them down by saying, today I will look at this crate, today I will look at that crate. and it doesn't really work to say, that looks fun. Sometimes I will do that, but I need to sort of ration my time a little bit. and not to mention it's it's both the ration between projects. But also to ration between the open source work that I do and just like the rest of my life, where I could easily just sort of be stuck in front of my computer only doing the open source things that I've already published. But then I would never make educational videos. I would lose out on social life and you know, my partner and my family. but also things like I wouldn't be able to do as much educational content, I wouldn't be able to write new crates that I have ideas for. I wouldn't be able to keep up with the board games I want to play or the videos games I want to play. Like I also need to do things that are fun. and balancing all of these is tricky. And the only way I've really found to do it is to time box the open source efforts. And if I'm going to time box them, I need every every time box to be relatively structured, which effectively ends up being today I will look at this crate or these two crates."},{"speaker":"Glen (Plabayo BV)","startTime":2225.0,"endTime":2225.001,"body":"And then"},{"speaker":"Jon","startTime":2225.0,"endTime":2231.0,"body":"Or or this type of issue, right? Like today I will look at Dependabot updates."},{"speaker":"Glen (Plabayo BV)","startTime":2231.0,"endTime":2235.0,"body":"I see. Yeah, I really don't like the Dependabot."},{"speaker":"Jon","startTime":2235.0,"endTime":2237.0,"body":"I have opinions on that too."},{"speaker":"Glen (Plabayo BV)","startTime":2237.0,"endTime":2247.0,"body":"what I just usually do is I I just update them regularly myself and then I look at the change logs and I don't know, I find them otherwise very noisy because yeah, I don't know."},{"speaker":"Jon","startTime":2247.0,"endTime":2272.0,"body":"Yeah, I've I've spent a bunch of time trying to customize my Dependabot config to make it not be as noisy. Like if I have a library crate, I don't want it to ever update my Cargo.lock file. So I ignore all patch and minor updates on those repositories. So I have it only file pull requests when there's a new major version of a dependency, which significantly reduces the amount of of traffic that you get on things that are library crates."},{"speaker":"Glen (Plabayo BV)","startTime":2272.0,"endTime":2272.001,"body":"yeah, okay."},{"speaker":"Jon","startTime":2272.0,"endTime":2275.0,"body":"So there's bunch of tricks like that that you can pull."},{"speaker":"Glen (Plabayo BV)","startTime":2275.0,"endTime":2307.0,"body":"Look at that. We learn the tricks one by one. Okay, very cool. And so okay, we already talked a bit about a couple of crates and then other crates that you maintain or develop are often around lockless data collections, which in a way is like a dark art because it's the thing that we all want is like fast and sharing and being able to mutate it yet as little to no contention at all, like not having to ever wait for a lock. yeah, I don't know. Is it always based on papers or is it like a mixed bag of okay, somewhere just ideas based on field experience?"},{"speaker":"Jon","startTime":2307.0,"endTime":2478.0,"body":"No, it's it's a mixed bag. I mean, left-right I built because the database I was implementing for my PhD, the bottleneck for application performance, which is like one of the big things that I was making a claim about in the thesis, main bottleneck was the how long it took to take a read lock, read one entry in a hash map, and then serialize it out. Like that was the the hot loop. There's not a whole lot you can do to optimize in there, right? Like the HashMap lookup, you can't do very much about. the serialization you can't do a whole lot about. The reader lock, though, you can because it turns out the way reader-writer locks work, and I recently gave a talk on the nuances of locks. but the way a reader-writer lock works, even if you take the read side of the lock, is that you first take an exclusive mutex, but only for like a split second. And then you release it again. And it's not really a mutex, it's like a compare and swap instruction, but it it has the same semantic of every reader ends up contending on one value in order to take the read lock. that means that you effectively have serialized the acquisition of the lock. Not the holding of the lock, but the acquisition of the lock is serialized. And in the situation I just described, where you take the lock, read a hash map. Serialize and end the acquisition of the lock is like a decent chunk of the work of those three operations. And so it turns out that serializing that basically serialized the performance of the entire system. So I had to find something else. And then I just spent a bunch of time at a whiteboard being like, what data structure has the properties or what concurrency algorithm has the properties I want for specifically this access pattern. And that's how I ended up with left-right. So that one did not come from a paper, although I had admittedly read a bunch of papers that probably inspired me to write it, but it was not something that came from a paper. whereas other things are because I'm interested in this topic and in distributed systems and concurrency and performance, a lot of the papers in that area interest me. And so I'm more likely to read them, which makes me more likely to want to implement what they put in there, or have the sort of Idea of, well, in this crate, I need something that does X. Wait, I read a paper that told you how to do X well and then implement them as a result. And sometimes it's not even based on need or that I found the paper really interesting. Sometimes it's I want to do another educational stream, and one of the things I want to teach people about is concurrency algorithms. So I need to implement something concurrency related. Let me find a paper that implements an interesting concurrency algorithm, and let's implement that one. So it's not driven by a need for me at all, except a need for a topic that teaches the things I want to teach."},{"speaker":"Glen (Plabayo BV)","startTime":2478.0,"endTime":2498.0,"body":"I see. and let's say Rust didn't appear in radar or even it didn't exist, do you feel you would still be working on so many different what I call dark art, like really on the edge of of what's safe and what's not, because I feel in Rust it almost empowers you to dare to take that step. which yeah."},{"speaker":"Jon","startTime":2498.0,"endTime":2567.0,"body":"I think so because I think otherwise I would do like C or C++ or something in that vein where you have that low level control. or Zig maybe, like who knows, like something in that in that area because those things fascinate me so much. And I actually think this relates to my desire to build building blocks, because I think the higher up in the stack you are in terms of like level of abstraction for programming languages, the less the things you implement are about building blocks and the more they are about larger ecosystems. Like if you're operating in Node.js, for example, the likelihood that you have a library that's actually like a self-contained small building block, I think is lower because the kinds of applications people build are one level of abstraction higher. and so therefore the things they need are also one level of abstraction higher. And a lot of things are provided already by the language, for example. Like you don't need or you're unlikely to need a data structure like left-right in the JavaScript ecosystem. And that means that I'm less attracted to the language because I'm in interested in those kinds of algorithms."},{"speaker":"Glen (Plabayo BV)","startTime":2567.0,"endTime":2586.0,"body":"Yeah, yeah. I understand. But it's because and maybe it was just different time but I remember and I came into Rust also very early, like before the 1.0 days, even though I discovered Rust in phase. I mean I I sometimes I I joke to people that I learned Rust three times because I learned it like in the pre 1.0 days"},{"speaker":"Jon","startTime":2586.0,"endTime":2587.0,"body":"Yep."},{"speaker":"Glen (Plabayo BV)","startTime":2587.0,"endTime":2604.0,"body":"and I made my first production grade there for back in the game company then and then yeah maybe faded away because I couldn't use it for the job anymore and then I started using it again, which I also meant I had to relearn it again because I didn't master it yet at that time. And then the third time is only when I really mastered Rust"},{"speaker":"Jon","startTime":2604.0,"endTime":2653.0,"body":"Yeah. I've heard that journey, I think, multiple like from many different people that have the same experience. And for some it's the pre-1.0 to post 1.0 days. For some it's pre-async and post-async, where they feel like they basically had to learn the language again. and then I think for some it is just a they learned it around 1.0 they felt like the language had too many things missing from the ecosystem. And so they were like, I can't. I enjoy the language, but I can't build useful things in this because there are too many gaps in the ecosystem. And then they come back to it now and they go, Well, I still remember the language, but the whole ecosystem I might now also need to learn how Tokio works and how Serde works and how tracing works and how Axum works. and as a result, they end up feeling like they have to learn at least the ecosystem all over again. so that seems to"},{"speaker":"Glen (Plabayo BV)","startTime":2653.0,"endTime":2654.0,"body":"Mm. And and"},{"speaker":"Jon","startTime":2654.0,"endTime":2655.0,"body":"be a recurring story."},{"speaker":"Glen (Plabayo BV)","startTime":2655.0,"endTime":2666.0,"body":"Yeah, because for example the first two times I was around, I never had heard of your name before. And then the third time I appear on the surface and like it's like, I see your name all over the place, like where you come from"},{"speaker":"Jon","startTime":2666.0,"endTime":2742.0,"body":"Well, but I think that is because like I entered the sort of the the more public scene of Rust, I guess, early 2018, I think, is when I did my first stream. And that was sort of the start of let's just build things and see if people enjoy me talking through building things. And then I think over the course of 2018 that sort of escalated to being, I think, a useful resource that people started sharing. And then around 20 19, 2020, I forget. Or somewhere around there I started writing Rust for Rustaceans. And and that was driven in part because people I think continued to feel the need for intermediate teaching resources for people who don't care about videos or who don't want to sit through ten hours of programming, which I don't blame them, right? Like that's not for everyone. And so I figured, well, let me do a a written version that I feel like captures at least some of the things I wanted to teach. And I think between those two resources, people I guess felt like those were useful enough that they started sharing them pretty aggressively. And then you get this sort of like this network effect where no matter where you look, someone has referred you to one of the resources. and you know, it it is very fun to see that that it It seems like I've made a difference to the ecosystem."},{"speaker":"Glen (Plabayo BV)","startTime":2742.0,"endTime":2780.0,"body":"I'm sure you did. for example what I always have liked is also okay there is an open source project I can try to look under the hood I can learn how how it works. But then yeah I start having kids and there's less and less time for that and it's very fun if if someone like you does something like a Decrusting video and I can I can basically instead of me doing it I can watch someone doing it kinda like a colleague and get the same benefits though because like you said you go through the process you think through it but I don't know if it's just you but because if I was me I would sometimes be stuck and like okay maybe tomorrow I'm gonna go check it back but you seem to always get there like no matter what"},{"speaker":"Jon","startTime":2780.0,"endTime":2826.0,"body":"Well, I I think this is I don't know where I've gotten this I guess skill from, but I think it's because I've done so much debugging of distributed systems and concurrency problems where you really need to like get in your head the order of how everything ties together and this like whole web of dependencies and how things all interact. And so then looking at many, not all, but many software projects, I'm like, yeah, these these pieces that all fit together, they just sort of construct as a graph in my head. And then I can use that graph to sort of query my way around. and so I think it like I built up muscle memory for basically keeping keeping graphs in my head. And that's turned out to be really useful for that kind of exploration."},{"speaker":"Glen (Plabayo BV)","startTime":2826.0,"endTime":2834.0,"body":"Interesting. And do you take notes in between you refer to them or it's like for the entire live stream it's just all in your head?"},{"speaker":"Jon","startTime":2834.0,"endTime":2860.0,"body":"no, I mean the the only notes I take are the ones that people see on screen. I don't I don't have a separate note-taking thing on the side. No, I mean there's not really anywhere to hide, right? Because the camera's on my face, so the only place I can look is at the screen. I guess the only thing I have on my screen that people can't see is the chat. which like I mean chat is actually actually now that I think about it, chat is arguably my notes. Like sometimes I'll"},{"speaker":"Glen (Plabayo BV)","startTime":2860.0,"endTime":2860.001,"body":"Okay."},{"speaker":"Jon","startTime":2860.0,"endTime":2865.0,"body":"be like, where was that function again? And chat will know. Because there'll be some person"},{"speaker":"Glen (Plabayo BV)","startTime":2865.0,"endTime":2866.0,"body":"Ha ha ha."},{"speaker":"Jon","startTime":2866.0,"endTime":2888.0,"body":"who remembers that bit of information. And I genuinely think it's gotten to the point now where, and this is partially a property of scale, right? But if like for many of my streams now I have two to three hundred people watching, and so the chance that one of them remembers the thing that I ask for is pretty high. So it becomes a sort of pretty resilient cache that I can draw things out of."},{"speaker":"Glen (Plabayo BV)","startTime":2888.0,"endTime":2893.0,"body":"Yeah, yeah. I mean that's a very extreme form of what they could mob programming, no? Like"},{"speaker":"Jon","startTime":2893.0,"endTime":2894.0,"body":"Yeah, exactly."},{"speaker":"Glen (Plabayo BV)","startTime":2894.0,"endTime":2910.0,"body":"And that's getting very much bigger than that. I wonder if it would be ever fun of you doing that like on a on a stage on some conference, I don't know, because you give many conference talks. And at home you do these kind of Decrusting videos, this or that. I wonder if there's ever like some kind of I don't know, mix of that possible."},{"speaker":"Jon","startTime":2910.0,"endTime":2959.0,"body":"I mean, it it's interesting because one of my professors at university would do these sometimes in lectures where he would basically pull up a terminal and just start like debugging a distributed system or something live. And I think you need to be pretty bold to do that. But I do also think it's really useful. I mean it's similar to what I do in some of the live streams. I think the challenge I've had is doing it on stage is a little bit different because Everyone in the off in the in the audience, it's like a little harder for them to engage with me productively. Because people can't just like stand up and scream. They can't just talk to me because it no one would hear them. And so I don't know what the interaction model looks like in such a way that it's actually interesting and engaging for most of the people there. But maybe"},{"speaker":"Glen (Plabayo BV)","startTime":2959.0,"endTime":2960.0,"body":"Yeah, yeah."},{"speaker":"Jon","startTime":2960.0,"endTime":2963.0,"body":"if someone has an idea, then send it my way. I'd be up for trying it."},{"speaker":"Glen (Plabayo BV)","startTime":2963.0,"endTime":2986.0,"body":"yeah. Look at that. You heard it here first. And then the one odd thing that that jumps out for me among all your crates is then something like Fantoccini like you said before. And also because it's the first time we even touch the protocols that it touches. maybe it could also be good if you can like kinda introduce the protocol it touches and also then why did you get started on this."},{"speaker":"Jon","startTime":2986.0,"endTime":3079.0,"body":"Sure. So Fantoccini is a an implementation of the WebDriver protocol. WebDriver is essentially the sort of web standards bodies trying to standardize what Selenium was doing for browser automation. So the ability to programmatically tell the browser, you know, click here, scroll here, select this element, input this text, press this key, navigate here, close this tab, and so on. so Selenium had this kind of orchestration for browsers quite a long time ago, but it was all built on relatively ad hoc integration with the browsers. and then, you know, I think people realized, and I think this was driven by Selenium actually, to say, let's build out this, let's standardize this protocol and have multiple browsers support it so that you can drive any browser with the same protocol so that you can do, for example, automated browser testing. And I think the WebDriver protocol was like Almost a copy-paste of the Selenium protocol, but not quite. There's like a bunch of ways in which it's subtly different, and it's really annoying when you try to implement the spec, because some browsers only support the Selenium version, and there's a there's a whole there's a whole a whole backstory there. But effectively the WebDriver protocol is a relatively simple HTTP-based protocol between The driver, which is the name of the thing that sits with either within the browser or next to the browser. So for example, for Firefox it's called geckodriver. For Chrome it's called ChromeDriver. for Safari, I think it's called WebKit Driver [safaridriver]. I forget. And it"},{"speaker":"Glen (Plabayo BV)","startTime":3079.0,"endTime":3086.0,"body":"And and and did they always have it, like this driver? Was it there and and and WebDriver is just a glue in between or did they make it specifically for"},{"speaker":"Jon","startTime":3086.0,"endTime":3113.0,"body":"No, so these drivers are basically the imp the implementation of the browser side of the WebDriver protocol. And then the driver knows how to actually tell the browser to do things, often using like proprietary protocols or they like directly link against the browser libraries or whatever. they basically use an IPC mechanism for communicating with the browser. But the driver process or program is the one that implements the the actual spec, the actual protocol."},{"speaker":"Glen (Plabayo BV)","startTime":3113.0,"endTime":3113.001,"body":"Okay."},{"speaker":"Jon","startTime":3113.0,"endTime":3396.0,"body":"And so what you will usually do is you'll start, let's do Firefox. So you start Firefox, then you start geckodriver, and then you connect to the port the geckodriver is running on. you can also usually tell these to say, you don't need to start Firefox yourself. You start geckodriver and you say, point it at this Firefox binary, and you say, whenever you need to start a new browser session, start this binary and then instrument it in however you do that behind the scenes. And the interaction model is pretty straightforward. So it's basically you send a little JSON payload over HTTP and the spec dictates like should be a GET or a POST or a PUT or DELETE or like the HTTP verb. And then it dictates the method you should use for example, d delineate between do you want to do a click or do you want to do a scroll or do you want to do a key input? and then there's a protocol the other way that gives you both. responses to things you've sent, but also events that can happen like page loaded and stuff. and all of them are relatively simple JSON objects. So for example, for actions you might have additional properties you include with a request saying like when you click, click at this XY coordinate. and so that's that's the whole protocol. And the WebDriver protocol has evolved a lot over time. So basically the way it evolves is new I forget what they call them. I think it's methods. Like new methods get added over time. so that now you have the ability to like close alerts. Now you have the ability to execute JavaScript. Now you have the ability to execute async JavaScript. Now you have the ability to open the file picker and so on. so keep gaining more or screenshot was one that was added somewhat recently, I think. And each one basically appends to the standard another set of. HTTP verbs or HTTP methods you can send one way and HTTP responses you might get the other way. And Fantoccini is actually a fairly dumb crate. I designed it that way on purpose. So it's not intended to be a super ergonomic like browser automation crate. It's intended to be a fairly direct mapping of the client side of the WebDriver protocol. So it has you know it has structs for all the different methods you can send and for their argument types rather. It has an HTTP client that knows how to speak the protocol. It has methods on that client that let you send the various kinds of requests or receive the various kinds of responses, maps them to the right types, knows how to translate those into HTTP requests and multiplex them, knows how to serialize and deserialize. But apart from that sort of gets out of the way. And so it doesn't have, for example, a way to Write nice test cases for how to chain operations. That's not a thing it does. It just has the the almost like direct mapping of the operations you can perform in WebDriver. And the intent was always other people will write crates on top of Fantoccini that will do the ergonomic thing, and Fantoccini will just implement the protocol. Because that way you can have one person whose job it is to map the protocol really well. And one person who can iterate on the ergonomics, which tend to change a lot more rapidly. You could even have multiple different ergonomic implementations depending on what kind of interface people like. And so Fantoccini was never intended to be that ergonomic layer. It's only intended to be the protocol implementation layer. That said, I've actually found a decent number of people do end up using Fantoccini directly because the WebDriver protocol is pretty straightforward. Like you just want to open a page and then Find an item using a CSS selector and then click that thing and then grab the page. Like it tends to be a pretty simple interaction model. And so the the direct mapping of the protocol works okay. but there are other crates like there's one called thirtyfour, I think, that tries to be a much more ergonomic mapping. That one used to use Fantoccini under the hood, then they changed because I forget the exact rationale. There was something they wanted about the API they wanted to provide that I didn't want to do because it would I think it would require giving up some of the type safety of the crate in order to allow them to do that. Like it they needed to be able to have like owned handles to the connection or something or something along those lines. and so they were like, we're just gonna do the parts of the protocol that we care about ourselves, which is fine. I don't need people to go through there if the protocol is simple enough that other people can implement it. but Fantoccini is actually pretty low maintenance for from my end, because the WebDriver protocol is relatively stable. And when things get added, there's a pretty predictable way to add new things. And it doesn't happen very often. And the API that it presents is straightforward enough that it doesn't, it's not like I have a deep feeling that it needs to be completely rewritten."},{"speaker":"Glen (Plabayo BV)","startTime":3396.0,"endTime":3398.0,"body":"Mm-hmm. Yeah, I mean."},{"speaker":"Jon","startTime":3398.0,"endTime":3411.0,"body":"And it's it's like it's not quite feature complete, right? But it it's as close to feature complete as I want it to be, which is I'm not planning to do anything to it except for add new methods over it or add new operations over time as they get standardized."},{"speaker":"Glen (Plabayo BV)","startTime":3411.0,"endTime":3415.0,"body":"I see. And what was your use case for it?"},{"speaker":"Jon","startTime":3415.0,"endTime":3512.0,"body":"So for me, I wanted to use it to watch TV online without the ads. so basically there were there are all these like streaming sites that let you watch various video clips. and it was really annoying to open them and like try to click through to watch this episode of that thing because there were lots of ads and then it would do pop-up windows, and then you had like scroll down to find the right thing, and then You had to like find the video player, but anytime you hit play or pause, it would pop up an ad. And so what I did was basically for several different streaming sites, I wrote like the automation required to grab just the video stream URL. And so I had a program where I could input like this show, this season, this episode, find all the streaming sites that have this particular episode of that show. navigate to it, use the Fantoccini thing to extract the video link, and then dump that out to standard out. And what you would end up with is a program like the command line tool. You just give like show season episode and it prints after a few seconds this link and then you're just mpv that or play it with VLC and now I can just watch it directly with with not doing anything. so that was the the original thing. That thing I had originally implemented in bash. But eventually the bash grew so annoying that I was like, there must be a better way and then I implemented the WebDriver protocol and then drove it through that and through Rust rather than like through greps and regexes and pup and jq."},{"speaker":"Glen (Plabayo BV)","startTime":3512.0,"endTime":3555.0,"body":"Yeah, yeah, yeah. Okay, I see. Very interesting. And then you do a lot of this open source work, you also have the educational work. Did you never consider okay, especially now that you have a name and fame and I imagine you can probably pull it off? Like, what if I make a job of this? Like I I ask companies like, okay, what is your most challenging problem or your most difficult thing? And I will bang my head against and you can even do that combined with your educational videos on that problem and make you the best crate possible for that topic. Kinda like almost like a researcher for hire. And but but someone who actually delivers like a functional crate given how anyway you like these things."},{"speaker":"Jon","startTime":3555.0,"endTime":3583.0,"body":"I've thought of doing that. I think there's a couple of challenges to it. One of them is I think one of the things that's really interesting about software engineering is to observe how it develops over time. So I like building something and then being forced to live with the choices I made. I think as a if you work in that more of a consulting mode, you build a thing and that's interesting, but then you sort of toss it over the fence and you don't get to see what happens to it. You don't get to see"},{"speaker":"Glen (Plabayo BV)","startTime":3583.0,"endTime":3584.0,"body":"The the yeah."},{"speaker":"Jon","startTime":3584.0,"endTime":3596.0,"body":"I mean, they might come back to you and say, Well, these things didn't work, do it again, or it needs to change in these ways. But it it's a little bit different to like Sitting there with the users of your thing day to day. being being"},{"speaker":"Glen (Plabayo BV)","startTime":3596.0,"endTime":3598.0,"body":"yeah."},{"speaker":"Jon","startTime":3598.0,"endTime":3685.0,"body":"in the place where like I also work with the people who use the thing I built, and they now need to build things on top of it. And I need to work with them to make those things possible. And at some point you end up just being an employee of the company, at least for a while. and I mean, you could you could work that way instead. So basically join a company as a consultant for a while and then leave when you think it's widespread enough. But I think even then you lose some of the longevity. the other is I would have much less ability to choose the things to work on. What I mean by that is at both at AWS and at Helsing, I feel like I can look for the interesting problems and go solve the ones that I think I'm best suited to do. I think if I did this as a sort of more of a consulting thing where jobs came my way, I think I would probably have to take what came my way. And that might not be the thing that's most interesting for me to build, or the one that's best suited for like you know, my skills or expertise or even interest. and so I do think I would end up feeling a little forced to do everything that came my way. But I don't know, right? Like this is part of the part of the things that makes it difficult to take the jump into something like this, is also you don't know what the reality is going to be. Am I going to get lots of things I could do so I can choose freely and have no problems like, you know, earning an income, doing the things I find fun, or is it gonna be like I'm gonna scrape by with small projects that are like not quite what I wanted to spend my time doing?"},{"speaker":"Glen (Plabayo BV)","startTime":3685.0,"endTime":3704.0,"body":"I see, yeah. and just to be clear, I didn't mean to throw it over the fence. I meant more like you so maybe you propose the problems like okay, you know like okay within the Rust ecosystem or within this doesn't even have to be Rust, but I guess it would be Rust, like clearly there is a need for this. And you you kinda like look for funds and and"},{"speaker":"Jon","startTime":3704.0,"endTime":3718.0,"body":"I mean th I think that's interesting because to me, the way you find out what is needed is by working at a place where you have needs. Right? So if I'm just sort of sort of free-floating radical atom, right?"},{"speaker":"Glen (Plabayo BV)","startTime":3718.0,"endTime":3718.001,"body":"Ha ha"},{"speaker":"Jon","startTime":3718.0,"endTime":3770.0,"body":"And I'm just like looking for gaps, I think it's very hard if you don't have the context of what people are trying to build. You can go talk to lots of people and see what they need, but that's not quite the same as sitting there and feeling the pain yourself. And a lot of the crates that I wrote came out of a need that I had. Some of them came out of the, you know, working on that database and being like, well, I need a crate that does X. There is no crate that does X, so I will write a crate that does X. And I need a crate that does Y. I will go do the same. Or at AWS I did the same thing. And at Helsing, I do the same thing. And so it's very driven by the other things that I'm trying to build. And it becomes this sort of not house of cards, but it becomes a a pyramid that you have to build one stone at a time. And you need to start at the bottom and allow yourself to build higher. and I think that is part of the joy is figuring out what is the right thing to build."},{"speaker":"Glen (Plabayo BV)","startTime":3770.0,"endTime":3785.0,"body":"I see. and then at Helsing, which is like your current job, do you also do there these kind of highly difficult concurrency challenges or is it more like similar like to AWS where you really put out your comfort zone and and you work on something completely different?"},{"speaker":"Jon","startTime":3785.0,"endTime":3900.0,"body":"I'd say both of them challenge my comfort zone in different ways. So at AWS it was more about build systems and build infrastructure and developer ergonomics, right? Because I was building tools that are going to be used by lots and lots and lots of engineers. At Helsing, it's more around the extreme needs for correctness, right? You need to build systems that they really cannot fail, at least in certain ways they should never fail, because of the domain that we're in. but also you have very particular constraints about how you build software, the hardware they get deployed, on the ecosystem they get deployed in, the threat model that you have. and part of it is also that you end up working with like not necessarily legacy systems, but other systems and other standards that are sort of imposed on you that you need to operate within the confines of or cooperate with. And I think that Poses a whole different set of things that are interesting to think about and make the work challenging. But I'd say there's less sort of, at least for the work I'm I'm currently doing, is less sort of focused on deep, low-level concurrency, for example. I'd say there's more stuff around distributed systems. I did some work on, especially for the first year, year and a half I was there. I did a lot of distributed systems design. I've talked some about this crate we open source called Decent. which is an implementation of a particularly cool kind of CRDT. so this is very like distributed systems stuff. and then I've worked on for a while for example I was working a lot on coordinate system transforms and rigid body transforms for spatial mapping. And so for that I built a crate called Sguaba, which is type safe coordinate system transformations in Rust. Because I needed them to be really, really safe. Like I wanted to write code where the compiler could tell me if I got the conversions wrong. and so I built a crate that enabled me to do that. And so again, it's an example of you the things I needed to build told me what the right building blocks to build were."},{"speaker":"Glen (Plabayo BV)","startTime":3900.0,"endTime":3907.0,"body":"Hmm, I see. Those are interesting problems and so well very well scoped. Probably probably like quite a pleasure to work."},{"speaker":"Jon","startTime":3907.0,"endTime":3934.0,"body":"But that's also because that's what I look for, right? I don't look for, I don't I don't look for like, I want to build the next ORM for web systems in Rust. Like, that's not the kinds of problems that drive me. The ones that drive me are the ones where I'm gonna like, this crate that does this thing really well is the thing I want to build. And then I want to complete it so that I can now rely on it going forward. And so I specifically looks for that shaped holes."},{"speaker":"Glen (Plabayo BV)","startTime":3934.0,"endTime":3944.0,"body":"And then does that mean that like more and more over time you're basically just having projects that's pretty much all your own dependencies or how that works?"},{"speaker":"Jon","startTime":3944.0,"endTime":3967.0,"body":"no, it it I think there's enough stuff in the Rust ecosystem that is in a good shape where I'm happy to take dependencies on them. But I do think I end up like pulling in HdrHistogram, for example, more than most people would, because I know how useful it is and I also know that it's good, right? so I know of more weird things maybe because I've built some of them."},{"speaker":"Glen (Plabayo BV)","startTime":3967.0,"endTime":3973.0,"body":"Yeah. maybe it's your chance to put it a bit in the spotlight. Like why would people have to care about this crate because I certainly haven't heard of it."},{"speaker":"Jon","startTime":3973.0,"endTime":4173.0,"body":"so HdrHistogram is great. HdrHistogram is actually it's a data structure that I did not invent. so I think it was originally implemented in Java. let me see if I can find actually the original author, just so I don't misquote who they are. but it was originally written in Java by Jill Teen. And so you can find it on hdrhistogram.org is the sort of website for the whole project. And it is a data structure that is a high dynamic range histogram. So imagine that you have a bunch of data that comes in over time that you want to sample. And you want to keep track not just like the, you know, the average and the min and max. you want to keep track of what is the median, what is the 95th percentile, what is the tail latency or whatever it might be. There are a couple ways you could do this. One of them is you just stored every sample you ever make. This obviously takes up a lot of memory space or storage space, and so it's not ideal. The next thing you could do is keep a histogram. So you keep a histogram that's like if the let's take HTTP requests, like processing latency. If the latency is between zero and five milliseconds, then you increment the zero to five bucket by one. If it's between five and ten, the increment of five by ten bucket by one, and so on. The problem with this approach is If you have some values that are really large or really small, they end up meaning that you would need to have an extreme number of buckets because the number of buckets you need is like linear in the delta between the maximum value and the minimum value you want to keep. or you just lose a lot of fidelity, right? You have one bucket that's like thousand and above, and you don't know if it was a thousand or ten thousand or a hundred thousand, or like anything that the difference between .01 or .0001 is lost in the noise between because your bucket was zero to five. So what HdrHistogram does is instead of having linear buckets, it has logarithmic buckets. So it keeps buckets of like this is simplifying it slightly, but you'll have one bucket that's like from 0.1 to 0.05. And you'll have 0.05 to 0.002. And like You you end up spacing them such that with a relatively small number of buckets, you can keep decently fine-grained information across the entire spectrum. Because the buckets will get smaller the smaller the values are, and larger the larger the values are. And so this means you do lose fidelity here too, but you can keep in very concise memory space. You can keep track of high fidelity of your data. or of your samples of whatever you're trying to sample. and it's also very fast because you have this very small memory footprint and so everything fits in cache at least often and the operations are pretty fast. So it's just a really cool data structure that is used a lot in the Java world. And so I was like, well, there are r there are ports of this crate to a lot of different languages, but there's none for Rust yet. How about I build it for Rust? and now it's ended up being picked up by a lot of the Rust metrics infrastructure. There's like a couple of different crates in Rust that do metrics. And I think many of them now actually use HdrHistogram under the hood because it is such a convenient way to collect metrics without giving up the fidelity over those metrics."},{"speaker":"Glen (Plabayo BV)","startTime":4173.0,"endTime":4179.0,"body":"Yeah, it makes sense. I was just about to suggest how come they don't use it and now I see they do. Even dial9. Look at that"},{"speaker":"Jon","startTime":4179.0,"endTime":4190.0,"body":"Yep. Yep. Because it's it's a great data structure, but a lot of people don't know about it because it's how would you know that this was even a thing that existed?"},{"speaker":"Glen (Plabayo BV)","startTime":4190.0,"endTime":4211.0,"body":"Yeah, yeah, for sure. But of course maybe also sometimes you don't really have a choice. I mean let's say you wanna hook into something like which are pretty famous about being pretty bad with their histograms or things like I would say Grafana or OpenTelemetry and then yeah, I mean there's not a lot you can do anyway because it somehow has to work with the server side, so what you wanna do there?"},{"speaker":"Jon","startTime":4211.0,"endTime":4228.0,"body":"Yeah, you don't always have the option, but if you do, then this can be super helpful, right? So even just for tracking internal stats in your applications, throw them in an HdrHistogram instead of either just keeping very coarse-grained stats or keeping massive amounts of memory or having to store them on disk."},{"speaker":"Glen (Plabayo BV)","startTime":4228.0,"endTime":4234.0,"body":"Yeah, because like you say, you have like one, ten, fifty, hundred, more than thousand. It's like everything."},{"speaker":"Jon","startTime":4234.0,"endTime":4249.0,"body":"Yeah, arguably arguably the example I should have done instead of my weird floating point numbers is like you have one bucket that's one to ten, and one that's ten to a hundred, and one that's a hundred to a thousand, and so that the buckets are not equally sized, but they are spaced such that you capture the tail as well."},{"speaker":"Glen (Plabayo BV)","startTime":4249.0,"endTime":4252.0,"body":"Very cool. Okay. now we're giving it some spotlight. Maybe more people will discover it."},{"speaker":"Jon","startTime":4252.0,"endTime":4255.0,"body":"Heh. Let's hope so."},{"speaker":"Glen (Plabayo BV)","startTime":4255.0,"endTime":4267.0,"body":"Yeah. And then we talked about a lot of different things already. Do do you feel we we have still something that we didn't cover or that you wanna specifically talk about before we round off?"},{"speaker":"Jon","startTime":4267.0,"endTime":4284.0,"body":"I don't think so. I mean I have I have too many crates to be able to really get at all of them and you know be able to meaningfully speak about them. I think maybe the one that the largest number of people know about that is like a slightly interesting to talk about is Inferno."},{"speaker":"Glen (Plabayo BV)","startTime":4284.0,"endTime":4285.0,"body":"Hmm."},{"speaker":"Jon","startTime":4285.0,"endTime":4346.0,"body":"so Inferno is a Rust port of FlameGraph. FlameGraph is a very common tool used for building Profiling data from applications. And FlameGraph was originally written in Perl. And then Inferno is the Rust port of FlameGraph. And then now there's also cargo flamegraph that has existed for a few years now that people are pretty familiar with. And cargo flamegraph is a combination of running, it runs the perf program for you over Cargo binaries. And then it takes that output and it runs it through Inferno to actually render the SVG that you can explore your performance data in. And so Inferno is the crate that I built to visualize flame graphs in this way. So it generates the flame graphs instead of you having to shell out to Perl. and that that was actually built on stream. So the entirety of Inferno from start to finish was built on stream. I think it was over the course of three or four videos, maybe. and that's a tool that I think a huge number of people are using, mostly through cargo flamegraph, without knowing that they're using the Inferno crate."},{"speaker":"Glen (Plabayo BV)","startTime":4346.0,"endTime":4347.0,"body":"Mm."},{"speaker":"Jon","startTime":4347.0,"endTime":4371.0,"body":"But it is like it was a really fun thing to port because you know th there's a quite a lot of things you port from Java to Rust or C to Rust or maybe Go to Rust, but Perl to Rust, there's not a whole lot of porting experience with. it's a thing I recommend. Like if people are curious about what Perl is like, that's a it's a good one where you get some exposure to all the weird all the weird Perl in the world and how it translates to Rust."},{"speaker":"Glen (Plabayo BV)","startTime":4371.0,"endTime":4372.0,"body":"Yeah, yeah."},{"speaker":"Jon","startTime":4372.0,"endTime":4374.0,"body":"For something that like actually is used."},{"speaker":"Glen (Plabayo BV)","startTime":4374.0,"endTime":4381.0,"body":"Yeah, yeah. Well I mean Perl used to be used a lot more and then of course depends on which Perl version, like the most modern one is is very different."},{"speaker":"Jon","startTime":4381.0,"endTime":4382.0,"body":"Also true."},{"speaker":"Glen (Plabayo BV)","startTime":4382.0,"endTime":4390.0,"body":"It's kinda it's kinda like PHP, like I think many of us did PHP, I don't know, twenty years ago or whatever. But then if you"},{"speaker":"Jon","startTime":4390.0,"endTime":4394.0,"body":"Yeah, I I also started out like some of my early work was also in PHP."},{"speaker":"Glen (Plabayo BV)","startTime":4394.0,"endTime":4398.0,"body":"Yeah, yeah. But then if you look at modern PHP, I it it looks quite different. So"},{"speaker":"Jon","startTime":4398.0,"endTime":4399.0,"body":"Yeah, that's true."},{"speaker":"Glen (Plabayo BV)","startTime":4399.0,"endTime":4405.0,"body":"okay, very cool. I really didn't know that that flame graphs came from Perl. I mean that's what you're saying, no? Or is it is more like your implement okay."},{"speaker":"Jon","startTime":4405.0,"endTime":4429.0,"body":"Yep. Yeah, yeah. The the original FlameGraph program, I mean it's still maintained. it's by Brendan Brendan Gregg, I think wrote flamegraph.pl. and that is still sort of the authoritative version of FlameGraph that ships in every distro and like that is FlameGraph. and then Inferno is just the the Rust port of that. Although the Rust port is a lot faster, perhaps unsurprisingly."},{"speaker":"Glen (Plabayo BV)","startTime":4429.0,"endTime":4448.0,"body":"Yeah, yeah, yeah. And then did you never try to consider okay, I have a really useful data structure, some really useful primitive. Did you never consider maybe I should try a Rust RFC for this and try to maybe push it into the standard library because it could be pretty useful?"},{"speaker":"Jon","startTime":4448.0,"endTime":4489.0,"body":"No, not not usually. And the the reason for this is because the Rust standard library is a very deep but narrow standard library. The goal is not to have the kitchen sink, it's not to have all batteries included. It's to have things that you must have for basically every program and then do them very well and very thoroughly and not do anything else. And so getting like putting something new in there, you would need to have a very strong rationale for like. Most programs need this thing. or that they are a perfect complement to something that's already in there. And almost nothing of what I've built fits that bill."},{"speaker":"Glen (Plabayo BV)","startTime":4489.0,"endTime":4500.0,"body":"Yeah, it makes sense. I was thinking maybe something internal, they use a primitive and clearly yours is better and it would be easy to swap because it's not really exposed or or let's say in the in the Rust compiler."},{"speaker":"Jon","startTime":4500.0,"endTime":4521.0,"body":"Yeah, so ordsearch had that possibility for a little while that maybe the algorithm from ordsearch is something that could be used in the standard library somewhere, but we never quite found the right fit for it. plus the fact that it didn't actually seem like it was all that faster, but it was quite a lot more complicated, meant that it it's like not really quite the right thing."},{"speaker":"Glen (Plabayo BV)","startTime":4521.0,"endTime":4525.0,"body":"Mm, I see. Very cool. I mean not very cool, but it's"},{"speaker":"Jon","startTime":4525.0,"endTime":4526.0,"body":"Yeah."},{"speaker":"Glen (Plabayo BV)","startTime":4526.0,"endTime":4535.0,"body":"good that you come to the the same agreement there. of course another part could be that they maybe like adopt some crates, to make it look at least more official. but yeah."},{"speaker":"Jon","startTime":4535.0,"endTime":4663.0,"body":"Yes, yes and no. I think there's a there's a pretty big danger to the blessing crates in that way, because the reality is that the ecosystem changes over time and you kind of want you don't really want to ever unbless something. That is usually that would be a problematic path. But it's kind of tricky to choose winners because suddenly like people's life situation change and now someone else is maintaining, or a new pattern emerges that's just way better. And there's a new crate that does a new pattern, and that's really the one you should use now. I mean, arguably, we saw this with Serde because originally there was a thing called rustc-serialize that was like, this is the way you do serialization in Rust. And that's the thing everyone was using. That's the thing that like almost was gonna go into the standard library. And then Serde came along and was like, Nope, no one is gonna use that anymore. That's now been fully deprecated. Like, this is this way is way better. The whole ecosystem adopted it. And if we had blessed crates, rustc-serialize would have been blessed. There's like almost no doubt in my mind that it would have. I think it was even under the rust-lang org. but it clearly was not the right thing because Serde swept the field afterwards. And so then the question becomes, should we bless Serde now? I'm not so sure. I mean, we'd I there are plenty of complaints people have about the sort of Serde architecture, and that maybe there exists something better out there. We don't know exactly what that is yet. There's nothing else that's sort of won out, and that's clearly the better choice. But I'm also not convinced to say that, there never will be, which I think is the conviction you need to have to sort of bless something enough to say, if not to put it in the standard library, to at least say, this is the Rust project stamp of approval. This is the thing we tell everyone to use. Because then that judgment is likely to change over time. it would also put more maintainership responsibility on the sort of core Rust community, which is already stretched pretty thin. And I think it's actually better to have a somewhat distributed ownership of those projects rather than than trying to centralize them under what is effectively a volunteer organization."},{"speaker":"Glen (Plabayo BV)","startTime":4663.0,"endTime":4678.0,"body":"Yeah, yeah, true. and I fully agree actually with it certainly shouldn't be all under the Rust Foundation, but maybe that you have more like islands but collective items. For example that or the the RustCrypto organization, that's the this that's that"},{"speaker":"Jon","startTime":4678.0,"endTime":4699.0,"body":"I mean, you see some of this already, right? Like RustCrypto has ended up as a sort of island. Tokio and its ecosystem has ended up as a sort of island. Serde and its ecosystem is certainly a bit of an island. I think we see the same in embedded, right? The sort of an embedded island that has a bunch of the the embedded HALs and everything. You see an island over by Bevy. So I do think we have some of this the sort of island"},{"speaker":"Glen (Plabayo BV)","startTime":4699.0,"endTime":4699.001,"body":"Yeah."},{"speaker":"Jon","startTime":4699.0,"endTime":4723.0,"body":"infrastructure. But even there, it's not like David Tolnay is gonna own and maintain. anything that is implements a Serde data structure or Serde convenience tool or utility that would be crazy. And I don't think we want that either because you want the ability for people to experiment and do something new. Maybe it works, and maybe then you want that person to own that thing because they came up with it and they know it really well. so I"},{"speaker":"Glen (Plabayo BV)","startTime":4723.0,"endTime":4724.0,"body":"Yeah, yeah."},{"speaker":"Jon","startTime":4724.0,"endTime":4732.0,"body":"don't think the goal necessarily should be this sort of everything gets pulled together to a core of people that own it whether here or there."},{"speaker":"Glen (Plabayo BV)","startTime":4732.0,"endTime":4761.0,"body":"I see and then given you care so much about performance because Serde it's nice and you have this contract and it works very well because you know you separate these different steps or different concerns and it's like kind of protocol agnostic but then what I often wonder is it's not really made for example for streaming, let's say, like l you you have large data structures and you can work around it. But sometimes I wonder it wouldn't be better if you first saw the streaming thing and then anything that's not streaming you can just block on it."},{"speaker":"Jon","startTime":4761.0,"endTime":4767.0,"body":"Quite possibly. I mean Serde has the same problem with async, where it doesn't work super well with with async readers, for example,"},{"speaker":"Glen (Plabayo BV)","startTime":4767.0,"endTime":4768.0,"body":"No."},{"speaker":"Jon","startTime":4768.0,"endTime":4795.0,"body":"where you basically need to buffer them all in memory and then call Serde synchronously. Whereas it does work okay with synchronous readers. and so there's a bunch of these kinds of things that I think are some of the questions around is Serde the right thing forever? Certainly right now, I'm not telling people not to use Serde. Like right now it is the thing to use. but should it be a blessed crate? I I think the answer is no. And I think the answer is no for almost every crate, to be clear."},{"speaker":"Glen (Plabayo BV)","startTime":4795.0,"endTime":4797.0,"body":"Well yeah. plus Serde would be super giant."},{"speaker":"Jon","startTime":4797.0,"endTime":4800.0,"body":"Except of course for mine. You can bless all of mine. That's great."},{"speaker":"Glen (Plabayo BV)","startTime":4800.0,"endTime":4802.0,"body":"Yours are all fine. I mean for sure. Yeah,"},{"speaker":"Jon","startTime":4802.0,"endTime":4805.0,"body":"Yeah, exactly. Mine will stand the the test of time forever."},{"speaker":"Glen (Plabayo BV)","startTime":4805.0,"endTime":4808.0,"body":"I mean some for sure, like some are so simple that I cannot see it changing."},{"speaker":"Jon","startTime":4808.0,"endTime":4809.0,"body":"It's true."},{"speaker":"Glen (Plabayo BV)","startTime":4809.0,"endTime":4818.0,"body":"Okay, very cool. It was very nice to talk to you tonight. I imagine, if you're in Norway you're in pretty similar time zone as me, so must be pretty late."},{"speaker":"Jon","startTime":4818.0,"endTime":4822.0,"body":"Yeah, it's I mean we still have the sun, but it's it's gonna"},{"speaker":"Glen (Plabayo BV)","startTime":4822.0,"endTime":4822.001,"body":"Wow."},{"speaker":"Jon","startTime":4822.0,"endTime":4823.0,"body":"get darker soon."},{"speaker":"Glen (Plabayo BV)","startTime":4823.0,"endTime":4828.0,"body":"I'm not sure how far north you are, but I guess you have maybe too much sun in the in the in the summer. I don't know."},{"speaker":"Jon","startTime":4828.0,"endTime":4833.0,"body":"Yeah, it's true. It's true. no, it's been great. Thanks for having me. It was a fun conversation."},{"speaker":"Glen (Plabayo BV)","startTime":4833.0,"endTime":4864.0,"body":"Yeah, and of course it's a bit normally we talk a lot more about networking but the fact is that within complex enough network systems, it is the kind of crates like yours, and I specifically mean the topic, but of course also some of yours that really make a difference because you're working with high concurrency, where you have to try to get everything out of your machines and then you really start to look for these edges and obvious ones are allocation, obvious ones are I/O, lock contention those kinds of things"},{"speaker":"Jon","startTime":4864.0,"endTime":4923.0,"body":"Yeah, like at the tail end here, like I have a crate called atone, which is a data structure that is I'll see if I can find what I wrote as the description for it. it provides a VecDeque and a Vec that spreads the resize load across pushes. so normally the way the vectors behave is when you push on them, push is a constant-time operation, except when you get to the capacity. Because then you need to double the size and move all the items. And so you get these spikes in latency that grow over time, but they also get longer and longer between. So amortizes constant time. But when you're doing network programming, for example, those latency spikes really bite you. And so this is a data structure I wrote specifically for that kind of use case where when it does a resize, it actually keeps both the new and the old vector around. And every insert it moves It also moves some of the old elements so that you get to actually spread that load over time rather than they get these spikes."},{"speaker":"Glen (Plabayo BV)","startTime":4923.0,"endTime":4933.0,"body":"Amazing. I mean what happens you solved. Maybe I should really take a time one day, like take a full day and just inventory everything you made because I feel there is there is some stuff I still need to describe a"},{"speaker":"Jon","startTime":4933.0,"endTime":4935.0,"body":"I mean you'll you'll find a lot of junk in there too."},{"speaker":"Glen (Plabayo BV)","startTime":4935.0,"endTime":4941.0,"body":"bit. Yeah, how to know? Do you dare to label it?"},{"speaker":"Jon","startTime":4941.0,"endTime":4949.0,"body":"I don't think I have anything labeled as junk, you know. But I think if you I think you you should be able to tell pretty quickly by looking at the state of the documentation."},{"speaker":"Glen (Plabayo BV)","startTime":4949.0,"endTime":4969.0,"body":"Look at that. Okay, that's the tell sign. Well, I I cannot thank you enough, not only for the the many useful crates, the inspiration you give to people, the the educational material you do and even then your brave courage to still try to catch up with the Rust releases in in your podcast. Like you you are"},{"speaker":"Jon","startTime":4969.0,"endTime":4977.0,"body":"Ha yeah, I mean I think Ben Ben and I are perpetually behind there, but but w one one day one day we will catch up. We were so close. We were so close earlier"},{"speaker":"Glen (Plabayo BV)","startTime":4977.0,"endTime":4979.0,"body":"Yeah, you work."},{"speaker":"Jon","startTime":4979.0,"endTime":4982.0,"body":"this year, and then and then we didn't."},{"speaker":"Glen (Plabayo BV)","startTime":4982.0,"endTime":4984.0,"body":"You will you will keep time travelling I guess. But"},{"speaker":"Jon","startTime":4984.0,"endTime":4987.0,"body":"Yeah, it's true. It's true."},{"speaker":"Glen (Plabayo BV)","startTime":4987.0,"endTime":4991.0,"body":"look at that. Well thank you very much. I wish you a nice evening and take care."},{"speaker":"Jon","startTime":4991.0,"endTime":4996.0,"body":"Likewise, thank you."},{"speaker":"Elizabeth","startTime":4996.0,"endTime":5001.0,"body":"Netstack.fm is brought to you by Rama, an open source framework for moving and transforming network packets. Rama is built and maintained by Plabayo, a company focused on secure, open and resilient infrastructure. The theme music for this podcast was composed by DJ Mailbox. For more episodes like this, subscribe. And if you know someone who might benefit from this conversation, share it with them. We love to hear from people who have built or maintained technologies that many others rely on, especially in networking, protocols, Rust or infrastructure. Have a story or insight to share? Reach out at hello@netstack.fm. Thank you for listening. Until next time, at the next handshake."}]}