episode 22 — Rust URL with Simon Sapin.
In this episode of Netstack.fm , Simon Sapin discusses building the rust-url crate based on the WHATWG URL specification. He also highlights that IDNA support was integrated to handle internationalized domain names, which includes critical security checks to prevent phishing from "confusable" characters that look identical to users.
If you like this podcast you might also like our modular network framework in Rust: https://ramaproxy.org
00:00 Intro00:50 Get to know Simon Sapin05:35 Origin of rust-url07:19 Implementing Specifications in Rust10:05 Understanding IDNA and Its Importance11:17 Origins of form-url / percent encoding crates12:36 The Role of Percent Encoding in URLs13:21 IDNA Crate Origin15:41 Challenges with IDNA and Usability30:13 unpacking the rust-url crate39:25 Understanding URL Specifications and Compatibility45:10 Url Crate Parser52:41 Surprising Insights into URL Implementation01:01:41 Final words01:14:07 Outro
Music for this episode was composed by Dj Mailbox. Listen to his music at https://on.soundcloud.com/4MRyPSNj8FZoVGpytj
Elizabeth (Plabayo BV)
0:14 | 🔗
This is netstack.fm, your weekly podcast about networking, Rust, and everything in between. You are listening to episode 22, recorded on January 9th, 2026, where Glen has a conversation with Simu Sapin one of the engineers behind Servo Together, they look at real world URL implementation in Rust. and explore where the most important design boundaries appear. Hello everybody, welcome to another week of Netstack.FM This week is the first time we will touch some amazing servo technology and I have the perfect guest to help us unpack all the URL crates in this project. So welcome Simon. Hello, thank you for having me. So Simon was one of the original developers on the Rust URL crate and the related sub crates. But before we dive into the technicalities, can you maybe introduce yourself a bit Simon? Sure. I'm a software developer based in France and I've been doing Rust for quite a long time. I started in 2013 when I joined Mozilla to work on Servo. So before that, my first job was working on Weezy Print, which is a project that turns web pages into PDF. So if you want to print things or archive things, PDF file format can be helpful. ⁓ But the company I at, they really wanted to use web technologies. It's a tiny company and they hired me basically to make what is mostly a web rendering engine. So it's implementing CSS. And ⁓ that got me involved with web standards and CSS implementation. And ⁓ later I applied to Mozilla and joined the server team. So that's how I got involved and I happened to work on the URL creates because we needed that at a time for servo. But my background is mostly CSS layout and rendering, more than networking. so you you said that your background was originally in more like rendering and yeah and i know in servo there are also several like That's right, and the CSS layouts. I mean, that's one of the big things it had in beginning was around CSS ⁓ engine and those kind of things. Is that also how you were involved in Servo? Yes, exactly, actually. ⁓ So there was a time where Servo used an existing library for CSS computation from the NetServ project. NetServ is, I guess, browser project, I'm not sure exactly, written in C, and they have some components available as separate libraries. But that was limited for a bunch of reasons and several projects wanted to have something in Rust. And my previous job, I had written a CSS parser before in Python. So I said, I thought to myself, that's a good opportunity to learn Rust and I'm gonna learn Rust and write a CSS parser in Rust. And so I applied to Mozilla. and came to a job interview with an implemented parser. So that was ⁓ a bonus for me, I guess. very nice and I know they also have like ⁓ things like selectors like you I'm not sure what the great name is but I'm pretty sure they also have yeah to select based on CSS selectors and I guess export and that kind of stuff That's right. That's right. And ⁓ so I guess my approach was successful because Mozilla hired me. And after I joined the team, we realized that ⁓ the parser is good, but it's not enough because the NetServ CSS library does also style computation, dealing with selectors and cascading and all of those aspects of CSS. And so ⁓ we got to work implementing the rest of it. including selectors and that became what we sometimes call stylo. And after some development, it ended up also being used inside Firefox. And that project was also called Quantum CSS. ⁓ around 2017, you may have heard of Firefox Quantum, which was a push to add several components written in Rust inside Firefox. and they are still in use today. Yeah, very cool. So there is definitely anyway too much like servo quality stuff to unpack in one episode. And I definitely want to one day touch on the on the CSS bits, especially related to the selectors. Is it by the way related? Is it like the same crate doing the I mean, does the crate implements the selector stuff also is that used in the in the rendering bit or is it like totally separate? Yes. So the selector's crate is used by server and Firefox today, ⁓ but it's not directly involved in rendering. It's more upstream of the computation where you have a bunch of style sheets and you have a dumb document tree and you want to apply style sheets to the documents and get ⁓ computed CSS values based on all the selectors and style sheets. So that's a bunch of computation that... happens kind of ahead of rendering. And then based on that, the CSS layout is gonna create a rendering tree. Very cool. So when you joined, it was basically all about the CSS and the rendering, et cetera. It's very exciting. But what I want to now narrow down a bit to is like Rust URL. And I wonder what was already started at point or that came much later. ⁓ It was a bit early in my history, I guess, with the server projects. I don't remember what year exactly, but we can look it up, the early commits. And the way it happened is basically the Rust ecosystem was fairly young at the time. So there wasn't yet... a library dealing with URLs in Rust and we need it for servo for networking, also dealing with the links and hash ref inside documents, that kind of things. And also the WHAT Working Group (WHATWG) which is the standardization body that came up with HTML5. They also have other specifications ⁓ for web related things, including one for URL. And so there was a specification already available oriented for the web. basically telling us everything we need to know about how to do things. So basically I took that specification and started writing Rust code doing what it says to do. And how was that experience like, like based, mean, let's say you implement the spec and based on that, imagine your scope of test cases was fairly straightforward or did it collide a lot with the real world, let's say. the implementation was fairly straightforward because of the style of WHATWG specification. They tend to like to have algorithms written in English in sort of pseudo code, but ⁓ explicit steps with ⁓ fully details about what to do. And in particular, ⁓ they also tend to define error handling. So for any inputs byte sequence or string, unicode string, basically the spec should tell you what to do. That's the design goal of the spec. Even if it's invalid input, it will tell you exactly which inputs are invalid, a lot and what kind of error. And that's a design philosophy that came from HTML5. where previous to that, HTML was de facto a permissive language where some things are valid or invalid, but if you make mistakes, the browsers will still try to recover and still display the document as best as they can. But different browsers were doing different things, so interop wasn't great until... the HTML5 project started to fully define HTML parsing, including error handling. And that was the foundation of the WHATWG Okay perfect and then are we at that point only talking about the the core URL create or what was exactly implemented at that point? and the project of Rust URL started as take the spec and write code to do what it says. ⁓ And it was a single crate to begin with, but over time some components were separated out, ⁓ like person-to-encoding, because something, I think it might be forms or something else wanted to use person-to-encodings. outside of URLs. ⁓ Then at first there wasn't any IDNA support. ⁓ But when we added that, it made sense to make it a separate crate because it's a fairly well-defined boundary of API. And for IDNA, it's the Unicode Consortium, which is a different stoneless body that defines IDNA and algorithms to implement. Okay perfect and let's say okay you have the relting. What year are we talking then by the way? I don't remember exactly, but would have been between 2013 and maybe 16. There's a commit in 2016 which is about split IDNA into separate crates. So guess initially it was inside the crate. Okay. And so now that we have this, okay, so we had the URL core crate but then right now, if you would look at the REST URL ⁓ repository, there's also things like ⁓ percent encoding, which is used for the encoding of data that is not like URI compatible, let's say, like if you have special character or whatever. And... and then you have like a form you're all encoded which is like for example if you're posting form data like how that's encoded when did these came were they also already there and they were later separated out or what's the story there? That's right. So these two are both parts of the URL parsing algorithm in the sense that ⁓ if you have, let's say, HTML with a link and you use non ASCII characters with accents or Chinese or whatever in the path component. ⁓ when turning that into a network requests, ⁓ the browser has to percent encode it to, because the HTTP does not accept Unicode non-ASCII characters. So that is part of the URL parsing algorithm. So it was there from the start and... but maybe later split into separate crates for users who wanted to use that independently. And similarly for form URL encoded that in URLs it is used for the query components of the URL which is after the question mark. Yeah and I mean the okay let's say for example the presenting coding was it always part of the initial spec of your error that came later? It is part of the Webworking Group specification because for the browser point of view, if you have a form with method get, that will be turned into a URL where the different components of the form are part of the query component of the URL after the question mark. And that is the format with name equal fields and then the ampersand for other fields. And then, yeah, guess the formula and code that's pretty crucial to use here as well because you're in the end all writing browse technology and that's used a lot. But then we come to one that is relatively new and that's like IDNA Can you maybe like describe that a bit? Like what is IDNA about and why it came into life? Right, right. So, Idna or IDNA is ⁓ coming from the Unicode Consortium and its aim is to lift the limitation of which characters you can use inside domain names. because historically DNS is limited to ASCII letters and digits and the dash sign. And I think that's it. I'm not sure exactly. And that's mostly fine if you speak English, but in many other languages there are ⁓ either accented letters or completely different alphabets. And... ⁓ in the internationalization efforts that Unicode supports. It's nice to also be able to use all those alphabets inside domain names. So that's what IDNA tries to do or does rather. And the way it works is that you have the quote real domain name in quotes. which can use a number of Unicode characters. I think there are some limitations, but we can get to that. And that gets encoded into an underlying DNS name, which is only using as key letters and digits and dash, et cetera. And that encoding is called punicode. And ⁓ it looks like... like X, N, dash, dash, and then some other random looking letters and digits. And that's how you can layer on top of the existing DNS infrastructure with limited characters and add support for more characters. Okay, that's pretty interesting. And I get that it might be nice, but also, yeah, I guess it can make it a bit complex to even type. Well, I guess it will work. Let's say you're in that alphabet and you're used to it. I suppose you will anyway write it correctly, but... I know that even with just like let's say ASCII like people already mistype URLs like plenty of times. So yeah, if you start to involve like more complex names, I wonder how that will make the usability in one way or another. Right, that is a concern and it is something that Unicode Constitution has thought about and ⁓ considered when designing IDNA. And that's why IDNA is not just punicode, the encoding, because the ⁓ encoding itself, it's just mapping numbers, which is the Unicode character numbers to some ASCII strings. and it's a bit of math and the algorithm, ⁓ it's not super difficult and it can end up relatively small. ⁓ But then there are various concerns such as ⁓ different characters in Unicode that look like each other. They are called confusables and that can be a security concern ⁓ for phishing. Like, If you go to your website, mybank.com and ⁓ some evil attacker registers the domain name that looks the same but uses a Cyrillic A instead of the Latin A for the A in bank, it could potentially be a domain name that looks very similar or maybe even looks exactly the same in some fonts. but it's a different website. And if you're not careful, you might get fished into giving sensitive information. And to avoid all of that, ⁓ the Unicode provides a sort of database of known, confusable character pairs. And part of IDNA is looking for that kind of characters and in some cases rejecting. ⁓ So some inputs are just invalid in IDNA. ⁓ I don't remember the details, but I think some of it is if you mix characters from different scripts, or that's what we call different alphabets, that is suspicious. And probably an attacker may not be allowed. I'm not sure of the details exactly, but it's this kind of thing. But that means that implementation of IDNA. needs to carry around a bunch of data about many characters in Unicode. And you need to ship the database, sort of. And that means in particular the... for users of the crates like URL. ⁓ If it includes IDNA, your binary size might get significantly larger to include of this Unicode database. And for some use cases, that's a concern. And in some of those cases, ⁓ they don't necessarily care about having IDNA or about... untrusted input that could intentionally use Confusable or ⁓ they might prefer not to have this big database of characters. And that means that there's like... I mean that's similar concern as UTF-8 in general, no? That you would need like an entire set of glyphs or whatever with you if you want to support like the raw parsing of UTF-8 or something. Well, UTF-8 itself is pretty simple in terms of, ⁓ it's just a bit of math of if you have a character ID that is above, I don't remember what XS symbol number, then instead of one byte, you will need two bytes or three or four bytes to encode one character or one technically unique code scalar value or code points. So UTF-8 itself is relatively simple, but then some other Unicode related things can require a database of characters. For example, if you want to have something in uppercase or lowercase, if you want to change something to uppercase or lowercase. ⁓ In the ASCII world, it's pretty simple. There's just one range to map to another. But if you want to do it correctly for any language, it requires more data to know the different mappings. And annoyingly, it's actually different depending on the language, like some characters map differently. ⁓ So this kind of higher level operation ⁓ on Unicode text, some of them require database of characters that increase code size. And then, yeah, so I can understand why it's important again to be within the rest URL space because in the end it's still for browsers and now people will start to use these domains and so you need to be able to parse them and interpret them correctly. Yes, so for a time we had ⁓ an amount of people who were saying like, we don't really care about IDNA and it's an incosys for us, but we would still like to use URL. And for a time it was, well, sorry, but it's not really a solution to just remove it for everyone. And later this was after I was not really involved in the project anymore, but... ⁓ These days it's Henry Sibonen who I think is still at Mozilla. ⁓ kind of to cover that part of the project, the IDNA crate. And Henry introduced a mechanism where there's an intermediate crate called IDNA underscore adapter, I think. And what it does is let the end user of the crate at the application level choose between different implementations of IDNA ⁓ using cargo feature flags. ⁓ maybe not feature flags, but different versions. ⁓ I don't remember the details, but there is a documentation that explains all of it and how to use it. And... ⁓ There is a choice of implementation between, I think two different sets of libraries dealing with Unicode that have different trade-offs. But also there is a search choice, which is like the do nothing implementation, which does not actually implement IDNA. And that means you don't have to pay for the code size of shipping character databases for use cases that, where that's not relevant. But the trade off of course is that you can't deal with Unicode characters in DNS and you also don't implement some of the security checks related to Confusible or other phishing concerns that... that IDNA prevents. I'll try to prevent. Yeah, but of course for some use cases that might be okay, especially if no human is involved in the... Right, if all of your inputs is trusted, for example. Yeah, and then ⁓ I know URLs, okay, one thing, but they also contain domains and then domains we enter DNS territory. It was there, and as far as I know, the Watt working group is mostly concerned around browsers, but I guess they also deal with DNS RFCs. Was there like a big cooperation in there, like how to deal with it from a DNS point of view? Right. No. So there is a sort of separation of concerns where the URL specification written by the watch watching group defines passing and error handling for many parts of URLs. But as far as DNS and in particular, IDNA is concerned, they... the white working with spec does not duplicate everything Unicode does, but instead they reference the specification from Unicode. So if you want a full implementation of the white working with URL specification, you also need by reference to go look into Unicode specifications and implement some of those for IDNA. Okay, and so what does a registrar see? I mean, what if I want to register in a domain register like an IDNA compatible domain, do I enter there also? Like let's say, because most people do if they have App ⁓ GUI or whatever, will they enter like just even a code variant and it will translate them to this ASCII format and that's all that will be in the records or how do I see that? Yeah, so IDNA has been around for a number of years now. I would guess almost every register supports it. And they make it so that in the human interface, when you register your domain, you type whatever you want with Unicode character. It will tell you if that character is allowed or not. And... I think there may be restrictions based on which top level domain you choose as well, but I'm not sure. That's a decision of the organizations that manage each top level domain. And if your domain is acceptable and compatible, then the register will take care of translating that into puny code and doing the necessary steps. But then if you look under the hood and run ⁓ some diagnostic tools like the host command or dig command, ⁓ most likely you will see the uncoded version in puny code. And unless those tools themselves, they implement IDNA to decode. Yeah, yeah, I see. And then you set up restrictions. For example, Unicode also contains emojis. Can I register emoji domains like that? That's a great question and I don't know You'd have to check I don't remember seeing any emoji domain so I probably not maybe not Ha ha ha ha. Yeah. Yeah, that's true. But at the same time, even though you it's around since a couple of years, I also don't really see any other like what I would consider IDNA territory kind of domains. like now that I know about it, I started looking for it and yeah, I can find some examples, but I must say it never, I never saw it on my radar before. rates. Right, I would guess that's because at the register level, this is their job. So they are well prepared now to deal with it. ⁓ But in the rest of the world, many pieces of software don't always deal with it very well. And... ⁓ I don't have much experience with that either, but it's possible that, especially in the early years, if you do use an IDNA domain, ⁓ you might encounter bugs in various different software that tries to handle your domain. ⁓ Maybe some of them will reject your domain saying that's not valid because they are only filtering and accepting the ASCII characters. And... I would guess at least at first, because of those limitations in other software, people tend to avoid doing it and they stick to what is compatible with everything, which is ASCII. And that slows the adoption. ⁓ I would think or I would hope at least that over the years, those kind of bugs tend to be fixed or limitations and more and more. ⁓ software implements IDNA and does it correctly. So if you do have domain name with Unicode characters and IDNA, ⁓ it's probably fine nowadays, but ⁓ I guess the adoption wasn't that great. And another factor is that ⁓ you and I speak English and other languages that use the same alphabets. So if we want to write names in ASCII, it's mostly okay. And it's not as much of a difference from how we use it to write. But compared to other languages, like if you write Arabic every day, having to convert your name into a Latin character that's kind of out of place, more so than English, I guess. Yeah, yeah, I understand. So maybe in those territories already used more as well, I imagine. Okay. And so then if we step a bit back and ⁓ start to dive into the core URL crate, because like I said, we... We talked with Larry Masinter before and we got like a nice like philosophical point of view in it and the background but we didn't get a chance yet to look into an implementation and for sure Rust URL is not their own implementation like another known implementation is the one from Hyper like they also have their own URL implementation which I guess came later. And so, yeah, even though I do wonder, but I guess that's not for you to answer unless you do have an answer, it's like, I wonder why they then wouldn't have maybe started with the rust-url crate of Servo, which seems fairly compatible anyway, but I'm sure there is a good answer there. Yeah. So, ⁓ I think the root of those differences is kind of the design philosophy or scope of the white working group specifically. And what working group is concerned with the web and specifying what web browsers need to do to be compatible with each other. And so there is a very specific point of view of ⁓ passing a URL means passing the href attribute of an a element in HTML. And that can turn into an HTTP request. ⁓ But it's not so much concerned about what HTTP servers need to do, for example. And... ⁓ That means there are some use cases that are not served ideally by the what working group spec of URL. And the scope of the project of Rust URL is least initially defined by implementing that specification from what working group. And that means... dealing with everything for browser needs, but if someone else needs something that is different or not quite compatible, there is tension there on how to deal with that. And in some cases, if it's just an additional API that can be added on top, that's fine. But if you want different behavior, for example, it's more tricky like how to deal with that in one project. Do you have a feature flag that switches between different behaviors? ⁓ Besides behavior, there may be differences in design of what a URL even is, ⁓ what is part of it or what is not part of it. And I don't know all of the details, but I think that is part of the reasons why Hyper did not choose not to use the URL create and they went with something else because they have constraints from HTTP. ⁓ When... trying to be compatible with every other HTTP client or server in the world. There's different set of constraints in terms of what is valid or what is invalid. Yeah, I understand. Yeah, it's also what Larry said is like he said that nobody can really agree anyway what the URL is and we tried to unpack it a bit and because you have the data URL for which you also have a crate which is is that the same as the blob scheme because I can see in your code that you also have like scheme blob is that also the data URL or is it something else? Right. Yep. That's different. So the data URL, the idea is that ⁓ everything you need is contained inside the URL itself. And instead of making a network request, you just have some algorithm that decode stuff. the contents of the URL, the data, ⁓ you put it literally in there, either encoded as base64 or ⁓ not encoded at all, but then the URL itself has percent encoding. So if you don't use base64, you effectively use percent encoding instead for characters that don't normally appear in URLs. ⁓ So you have the data directly in there. And that means if you have a lot of data, you have a very long URL, ⁓ which can be okay up to a few kilobytes. But if you have many megabytes of URLs, some systems don't deal with that very well. Or maybe it can be slow or whatever. And Blob is a separate thing where I... I'm not super familiar with it. I don't know all the details of how it works, but I think the core idea is that you don't have everything inside the URL. You don't have megabytes of data directly in there, but instead you have unique identifiers. I think it may be a UID, I'm not sure. And that refers to some blob of data somewhere in the software that is created by different API. but refer to bad idea basically. okay that's interesting because it's also something he was referring to as something he wants to be done so i guess it's partially there already ⁓ and then of course yeah yeah go for it Yeah. And earlier you said nobody can agree what a URL even is. And I think that's true, but it's also a philosophical difference, I would say, where the white working group tries to come up with an answer. And they would say, a URL is this, at least in some context. And maybe that's not the right answer for everybody, but it is an answer. And that answer is well defined and precise. And ⁓ as an implementer, what I find frustrating sometimes with ⁓ specifications or RFCs from a ETF, especially the older ones, is that they describe, I'm paraphrasing. This is what a URL looks like. and maybe there is a grammar for example, in BNF, something like that. And the grammar itself is precise, but then what happens if you are passing inputs that does not match the grammar? How do you deal with that? And in many RFCs, that's kind of not really considered or it's left up in the air. And that's where the what working group specifications try to be different and give you an answer on how to deal with stuff. But often the answer is just try to be as graceful as possible. I mean, like it's basically, maybe ignore it or somehow like, yeah, because in the end it's browses, right? So they're very forgiven. Yeah, so in the world of standard on interoperability, there is something, let me just check that I'm not confusing the name. Right, something called Postel's Law, also known as the robustness principle. And ⁓ the idea is that you're trying to have different software talk to each other and sometimes there's bugs. And to deal with those situations and be compatible with the... The idea of Poe's Tales law is try to be permissive in what you accept, but strict in what you emit. So when you're passing ⁓ stuff coming from someone else, try to recover from errors. ⁓ But when you serialize or emit... stuff in different file formats or protocols, try to be as strict as possible and only stick to valid documents or items or things. The problem with that idea is that ⁓ try to recover from errors. is not well defined and if you just say try and everybody tries independently they come up with different things that may or may not be compatible with each other. If you are also permissive in what you accept, that means ⁓ even stuff that is nominally invalid according to the spec, ⁓ people are gonna start relying on that because that's how ecosystems work basically. something... ⁓ if the software behaves in a certain way or accept some inputs, some people are gonna start relying on that, even if it's officially wrong. And if you change it later to say, that's wrong and we're gonna break that in that case, real world, stuff in the real world is gonna start breaking. And in a lot of cases, it means you can't really do that. So the invalid stuff ends up becoming de facto valid and the way people do things. And in my opinion, it's better to try to specify from the start how errors should be handled by everyone instead of living up to every implementation to figure out different ways to do it and ways that may be ⁓ incompatible with each other. Yeah, yeah, I do like that and that's totally fair. And then I see that there is also ⁓ FTP and file support where like file is just your, there is no original part, but it's just your file part. And then is there much difference between an FTP URL and let's say an HTTP URL? Right. No, FTP and HTTP are very similar to each other. It's just the default port number is gonna be different. But in terms of passing, as far as I remember, they should be identical, but there may be some special cases. Does FTP also support query and fragment parts or that's not there? At the parcel level, I think yes, but at the protocol level, I don't think there is anything like query in the FTP protocol. So my guess is would be it's silently dropped when you make the request. But the fragments however, is client side. It's in a browser where after making the request and getting a document, the fragment designates a particular part of the document. So that's independent of the protocol. ⁓ But these days that's kind of moot because every brother has removed support for FTP. ⁓ So I guess HTTP is the only protocol universally supported by brothers. yeah and of course like websocket that's that's htp URLs anyway so just a different scheme Yes, that's right, that as well. And file is a more special case because different operating systems have different syntax for file paths like the backslash versus forward slash. So there is a bit more of a special casing in the parser for today with all of that. Yeah, indeed. But cool. And so that's, guess, the more interesting aspect for me to when I mean like what's the URL even about, what my main concern mostly is, is let's say I'm the implementer of some network library and I want to support a URL kind of thing as a concept. What I would like to avoid is for each protocol to have like a separate implementation because yeah in the end they all look very similar but there are details which are different here of course they are quite similar I mean there are three distincts here like blob, HTTP and then file okay but from a network point of view you mostly care about the HTTP point of view but then as far as I know there I mean because then I had a conversation in the podcast with ⁓ Daniel from Curl and they as a client library as well as the CLI tool and they deal with basically everything that ⁓ has a URL and that's like a whole lot of protocols and so I just wonder like that's for me more the harder question like like what's similar there and what's different because in the end if you don't look at close enough they all look so alike Yeah. Yeah, so. I think in general there is a desire to have a mechanism that is generic and works for every protocol. ⁓ the URL spec at one correctly group and the URL grades, they try to do that in having, if you have a URL with a scheme, that is the protocol part. If it's unknown, there is a code path, path in a specification to handle generic protocols and. That means we don't need to have baked-in support for every protocol on earth in the library. ⁓ And ideally, every protocol would work the same way, but ⁓ for compatibility with existing systems, there are some special cases we need to deal with with file, for example. I see. And so is there also a principle in the URL crate to try to, I don't know, like share as much data in a single blob as possible and just work with kind of like ranges where you say like, I don't know, maybe the origin is this size, maybe the port is in this location, or do you also already have the parse version available? ⁓ Right, so. Part of what the URL parser does is taking a single string as input and splitting the different components. And which components exist is kind of the fixed data model of what a URL even is. And that means you get to have a scheme, you get to have a domain, a path, a query, a fragments, and... If you are designing a new protocol and you want URLs, it's a good idea to make your URLs look like that. There is a scheme that's gonna be the name of your protocol and the domain probably goes to DNS and the path maybe you use it or you don't use it and similarly, maybe you want parameters in the query. ⁓ And if you're designing something new, trying to fit into that model means you can use existing libraries. ⁓ Yes. And then, ⁓ okay, so we have that. have the, we have, okay, because. I think last time we already kind of went over what a URL is. We talked about it as a scheme. It has like the username part, like maybe also the password, but that's like rarely used except for some proxies. You have the host, which can be a domain or an IP, IPv4, IPv6. You have the port number, then you have the path, then behind the question mark, if there is any, there is the query. And then behind the hashtag, there is the fragment, which is as far as I know, a client only thing. It's not really... yeah is a like is there what are the rules by the way for what what can be after the fragment is that also like like you're all encoded I guess so That's a good question, I don't remember. I think the... if you're parsing the href inside HTML, depending on which characters you use, some of them may be Euroland coded by the parser, ⁓ but then there is also an API to, in a real-life to decode it and get the decoded string that you would use to match against the document. find the actual piece corresponding to the fragment. Okay, and then you said, okay, if there's not a scheme found, which means like the protocol, that we might treat it as like ⁓ an opaque, like URL, like some generic thing, but there is also the concept of the port. Like what if it's defined, would you also consider that as a valid alternative for the scheme? Let's say it's 443, which you then say like it's an HTTPS thing, or you wouldn't go that far. Yep. Yeah, so the port number is also a thing that exists in the data model in the sense of what is a URL that is one component. And for... the Russia Alcrate at least in the generic case for an unknown scheme, you can have a part number, but we don't know the default value of the part number. in terms of Rust API, it's gonna be an option that ⁓ it will return if you ask for the part number. ⁓ But compare with HTTP for example, where we know that for HTTP the default part, if you don't specify it is 80 and if for HTTPS it's 443. I don't remember exactly. I think there is a different API to tell you specified part number or the default, something like that. Okay and so then if we zoom in on the on the part ⁓ one which I find very interesting what does the the what specification tell about like case sensitivity I imagine it says it's case insensitive or am I am I wrong there That's a good question. I think for the path, it's gonna not change anything. So whatever you put in the input is gonna be left as is. ⁓ And that is probably different for DNS. ⁓ Yeah, but that's of course the host part, is a different part. Even though I'm pretty certain that domains are also case-sensitive. I don't think you can just register, I don't know, google.com in capital letters. I don't think that's a thing. I'm pretty certain it's a case-sensitive thing as well. Yeah. I think you can, it's insensitive in the sense that it's all going to be folded into the same. Yeah, yeah, that's what I mean. But I mean, it's not like I can register some capital variation of that. ⁓ yeah. Right, because it's gonna be considered the same name. And ⁓ for the path components, as far as I remember, it's not gonna be converted to lowercase or anything like that. But then it's up to the server or other components to decide whether they consider it the same path or not. Yeah, yeah, yeah. Okay, so that is left up to the service, what I thought, because many service will just see it as strings and so they might give you a 404 if you have the wrong casing. Yeah, it's... Yeah. I think it's usually not considered the same pass. So if you have different case, it's gonna be four of all, ⁓ but some servers may just pass the string to whatever file system they use. And if the file system happens to be case insensitive, then they will do that. Yeah, okay, too bad. Because like, for example, with HTTP, let's say headers used to be case sensitive, and then only since HTTP2, they said like, no, everything is lowercase. And I think that was a huge benefit because also before you could have like, I don't know, some proxy which like was having like, I don't know, maybe the... some kind of like first worst ⁓ first letter casing of every word or every part instead of the original casing and you could get into weird issues where the server certainly didn't recognize your headers and then since HTTP 2 everything was lowercase I would have thought that it might make sense to also just force lowercase there yeah but then of course yeah maybe not because again what is the URL Yeah, so that goes back to the design philosophy of how you want to design protocols and whether you should be permissive in what you accept or not. And SCP2 decided to be more strict. Hmm. Okay, fair enough. And so as you dove quite deep into the protocols of URL, because you implement them, I'm sure several iterations, are there like some things you learned about URLs that maybe other people don't know about and like things that surprised you and that maybe some gems of knowledge you can share there? ⁓ I don't know about gems of knowledge, but ⁓ I guess there's surprising amounts of ⁓ special cases and things that need consideration. ⁓ On the surface, a URL that looks simple, but then if you actually want to be compatible with everything on the web, ⁓ there's a bunch of things you have to deal with and some of it is annoying, but that's life. We make the software that takes care of this so that hopefully our users don't need to. Yeah, Fair enough. And yeah, do you remember something that seriously caught you off guard that you thought like, OK, I didn't know this was a thing or possible or should even be a thing, but it's something I need to support somehow. I guess IDNA was the bit surprising in terms of size. Like if you think about what is a URL and how much work is it to implement a library that deals with URLs, you can have some ideas initially. And even if you look at the RFC or specification, you can look how big it is and get some idea. but then you get to the parts about now you should probably deal with Unicode and IDNA. And what is a small paragraph inside this document is actually a link to another document that is huge. And there's a whole scope that you might not have anticipated with lots of complexity. I see. And if we, if you go to URL encoding, is that something? No, well, percent encoding, So if you go to percent encoding, is it something that you should always, what does the specification say? it, let's say you're writing a web router and you want to route on a path. Should it always first present encode or there are cases where you might say no, here it should be checked as without being decoded. the WHAT working group spec defines a passing algorithm where during passing, it will always encode ⁓ some characters. Whereas if I remember correctly in the IETF RFCs, ⁓ ⁓ it's kind of undefined where encoding is a thing that exists, but when you should encode or decode is not really talked about. And that's a choice that the what working group made for how browsers should behave. But if you're making something that's not a browser, maybe you would want to make different choices, I'm not sure. But there's also something that... maybe surprising, which is we think as URL encoding as one format, but it turns out inside the URL, the set of characters you want to encode or not encode is different depending on which component you're looking at. Like for example, if you want to have a question mark inside of your path, Well, normally the question mark marks the end of the path and the start of the query components. So you need to encode it if you want to have it be part of the actual path instead. But if you are inside the query itself, then the question mark, if you have another question mark, it doesn't have any special meaning. So you can just use it as is and you don't need to encode it. And you can encode it and it will work out. And would it matter if you anyway encoded? Probably, but you don't need to. And that's why in the crate for persons encoding, in the API, when you use it, it will take a parameter for, I think it's encode sets, something like that, to basically choose which characters you want to encode on it. I Okay, that's indeed interesting. And then, ⁓ yeah, I'm also surprised by how many different URL encoded libraries there are in Rust. And of course, there's also a crate for that in Rust URL, but I guess yours is purely focused on, yeah, again, the browser point of view, right? I think the explanation for that is that URL encoding itself is very simple as a format. And so if you are looking at the library that does it, but not exactly in the way you want it, it may be easier to just make another library or just implement a module, private module in your own code rather than... ⁓ make a super generic library. ⁓ That's my guess for why the MediaLibrary is doing it. Yeah, that's fair enough I guess. It's very cool. I I like the the crate a lot. It looks very simple and I would suggest anyone that wants to learn a bit more about parsing. I mean it's definitely a beautiful chord base to look through. Yep. And that's a debate we have in in the REST ecosystem, but in programming in general as well about ⁓ using a library versus implementing it yourself. And I think there's a lot of values in having libraries that are shared, used by different projects and maintained by different people together. And especially for more complicated stuff. sharing the implementation is good. But in some cases for very simple stuff, like the overhead of having a separate project with a separate repository, with its own CI, with its own API, and its own semantic versioning compatibility concerns. That's a lot of things you need to deal with. And if it's just 50 lines of code that you could copy and paste in your code, sometimes that's a good solution too. Glen (Plabayo BV)
1:00:46 | 🔗
Yeah, that's true. But then again, I think you did a... I mean, you and your colleagues did a good job because anybody is already like several crates, like you have the IDNA crate, you have the percent encoding crate. And then even within the URL crate. ⁓ Actually, that's not true. thought you, I mean, can you use the URL create just for parsing or does it always end up that it only exposes the API for the actual URL? Because, because that's where I would think that you might start to see different preferences where my, okay, maybe as the raw URI parsing, you can all agree because it's basically just, want to find out which part of the string is what, but then how you want to represent it from there in memory. I think that's where you will start to see differences because they might have their own, I don't know, data types and their own, I don't know, whatever way they want to represent these things. And I think, but the parsing itself, would say that that should be fairly, yeah, I guess people could share that, right? Right, so the Rust URL create provides both. It provides a parser and it provides a struct, a restruct that is the parse URL where you can access each component. And ⁓ if that's in memory representation works for you, then great, you can use it. And ⁓ if it's not exactly what you want, then maybe a solution is to use it to parse, use the library to parse, and you get that representation, which is not the one you want, but then you can convert. That's a possible solution. Glen (Plabayo BV)
1:02:31 | 🔗
Yeah, I think that's a very fair solution. And that's also why I wanted to, first of all, this create is a very good way to learn more about URLs. And we learned a lot from you today. And secondly, I think it's also a create that's not yet known by everybody because especially people who come to RESTful Networking, they will often move to, for example, the hyper ecosystem because it's so closely tied with Tokyo. And so they will see URL there and it's used by a lot. And it's also a nice implementation, nothing wrong there, but I think it's nice to also learn about these other gems and especially like servo as it's also so crucial to the Rust history. I think they did a good job there and so yeah I would like to thank you very much for your work and also from your colleagues on this open source work like it's greatly valued. I also wonder now that this thing is there and okay, we had the, was developed originally for the Servo project in its first, I would say phase, but then there were the huge Mozilla layoffs and then it was taken over again by another company. And now we are very happy that these things are maintained again. But at the same time, is there like a lot of work on these kind of crates? Like once they are like, kind of like finished? Because I imagine, I wonder. If you take away things like IDNA, because maybe that's like a bit more like still a moving target I would think that the other parts are fairly yeah, I don't know They could reach a point of being finished right like I cannot really see that they would ever That they would still change that much Yeah, that's fair that I think in every project there is an initial development phase that with a lot of activity and then at least for some projects you reach a plateau where basically things work the way you want to so you'll need to do as much evolution all the time and probably the ratio of all the great is there. and has been for some time. I haven't followed it very closely for a while, but that doesn't mean it's completely frozen, because there's still some amount of complexity and subtleties, and you can have bugs or improvements, but they are probably fairly minor compared to... I don't expect a complete redesign of the crate anytime soon. So it's not quite the case of, for a smaller project, you might imagine calling it completely done and basically not expect any change ever because it's so small, as you can see, it does exactly what it needs to do. I think Ural is a bit more complex than that. it's good to still have maintenance and be able to respond to reports. Glen (Plabayo BV)
1:05:43 | 🔗
Yeah, fair enough. That's very cool. so with that work behind you, like, where are you now at that your life and the kind of, I don't know, let's say, let's say what are your professional activities these days? Yeah, so recently I was working at Apollo GraphQL, ⁓ developing ⁓ GraphQL Federation router in Rust. ⁓ And I left that job recently, ⁓ but it was a very good team and I really enjoyed working with these people. So I'm looking for work now and... ⁓ If you have a Rust team who is hiring and is working in European time zone, I would be interested in hearing from you. Glen (Plabayo BV)
1:06:35 | 🔗
Very cool. And I would think there are, so I wish you the best of luck with that. And funny enough, had like a last episode, we had a guest from GraphBase, where they also work on a GraphQL Federation. So I guess your work overlapped a lot there as well. Glen (Plabayo BV)
1:06:57 | 🔗
That's very, very cool. For example, because it was a very enlightening episode for me because it was around GraphQL. And I learned about GraphQL when it was introduced and all these hyphers around it. But then I kind of like... didn't really follow anymore that space and so I was pleasantly surprised to learn ⁓ from him about this entire concept of federation and it made like a lot of sense to me where yeah I think that's something you really want because it's a beautiful concept but to do it all as a single team yourself or to to write one big like I don't know API server that does GraphQL very well, it's almost like an entire full-time activity in itself, which is very different from writing your typical REST API. And so this concept of a federation server, which does all the planning and optimizations and caching and whatever else you might want to do, I mean, that's a beautiful concept. Yeah. Yeah, Federation is pretty cool and there is a lot going on in implementing it, yes. Glen (Plabayo BV)
1:08:04 | 🔗
Yeah, and I also find it interesting how then, yeah, for example, for Apollo, I learned from him that, and you can correct me because you worked at there, is that he said, okay, there are some companies that first started the federated server in something like TypeScript, and I think that's what Apollo did. And then they transitioned it to Rust. Yep. That's right. So at Apollo, they had a project called the Apollo Gateway, which was the initial implementation of Federation in TypeScript. And that was useful for a bunch of people who were successful with it. ⁓ But they ended up with huge deployments with many machines dealing with the traffic. going through this TypeScript code base. And so some number of years ago, not long before I joined, ⁓ Apollo decided to start building things in Rust. And so the Apollo router is a rewrite of the gateway, but in Rust instead of TypeScript. And the team I joined was formed around that project. Glen (Plabayo BV)
1:09:05 | 🔗
Yeah. starting to use REST at the company. Glen (Plabayo BV)
1:09:14 | 🔗
Okay, and I cannot stop myself from observing that you went from like working on rendering engines and on things like CSS, which is like related to there. Then you moved to always like more like network-based stuff, which was then you, when you work on GraphQL and Federation stuff, it's still that kind of world, which at some point ever want to move back to. to some kind of rendering because it's a very different kind of way to program altogether, Yeah, it's different ecosystems and different domains. And ⁓ I joined Apollo knowing basically nothing about GraphQL. So that was interesting, learned a lot. ⁓ But ⁓ I guess what I brought was more the experience with Rust. And... ⁓ Even though it's different domains, there's a lot of things that end up being similar, like writing parcels, writing libraries with nice APIs that are used by different components, ⁓ or what we call system level ⁓ code or systems. That's a bit vague definition, but it's something. Glen (Plabayo BV)
1:10:42 | 🔗
Yeah, it's fair enough. And actually you hit like a pet peeve of mine because yeah, I mean, I used to do a lot more on parsing compilers. And it's funny because at some point I wanted to join Mozilla and I could, but they couldn't hire me because I in some really weird region in the world at that time that they couldn't hire me from. that was... that would have been for the JavaScript engine team. And what I often tell people is, yeah, you should really learn to write parsers because what I see often people like grabbing too is like just trying to do a... maybe some kind of, yeah, a very naive like string lookup or they might ⁓ use regular expressions even though they don't even master regular expressions. And I think for some things and I would include in their URLs and things like emails and there are many of these examples, you really just wanna have a parser be it one that you can grab off the shelf of or write yourself because it makes it so much easier, especially if all you care about is a specific part of that kind of string because your parser will be relatively small because you can focus on just the first stage of the parser. It's a very valuable skill to learn, I think. Yep, that's right. And if you've never written a parser before, it maybe looks a bit scary, but I promise it's not too difficult. And especially if you look into something called recursive descent, that is a style of parser that I like to write. And it's very flexible and can do basically anything. And it's funny you mentioned compilers because one of the projects I worked at Apollo is called actually Apollo compiler. And it's kind of the foundational library for not only parsing, but everything around GraphQL, like having an in-memory representation that can be modified. And it's used by several different projects inside Apollo. Glen (Plabayo BV)
1:12:57 | 🔗
Yeah, I mean, these are very nice products to work on because they're usually pretty contained and you can test them very well. And you kind of have like clear boundaries, like here's my input, which I totally do not trust. And you go to these different stages where you trust more and more. like, I mean, it's a very nice waterfall kind of system where like, if you do it with care and you do not rush it, like it's a very nice thing to work on. Glen (Plabayo BV)
1:13:26 | 🔗
yeah okay very cool so that said we discussed a lot like i learned a lot about URLs obviously we couldn't like discuss every Single line of these creatures neither is the intent, but I will definitely link it in the show notes if you have some resources For people to learn more about this. It could be like the the what spec but also some other things Also things like let's say maybe if people want to learn this like Parsing thing they can look up this kind of sets. You can you can send them my way ⁓ Is there something else you would like to plug in the show before? before we round off. if you haven't looked into servo, it's pretty cool. Glen (Plabayo BV)
1:14:07 | 🔗
Very cool. And is there like a shout out you want to do to the Rust community? I really enjoy being part of the Rust community and ⁓ the Rust project in particular ⁓ is something I've participated in and I enjoy that as well. ⁓ Kind of like writing parser, it may look scary, but ⁓ if you want to get involved, I would encourage to do it and ⁓ people there are very friendly and will help newcomers get started. Glen (Plabayo BV)
1:14:42 | 🔗
Yeah, I couldn't agree more. Thank you very much for your time today Simon and all the expertise you put in these projects. Elizabeth (Plabayo)
1:14:51 | 🔗
Netstack.fm is brought to you by Plabayo building secure, open, and resilient infrastructure with Rust protocols, and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcome sponsorships to keep building and supporting its ecosystem. The theme music of this podcast was composed by DJ Mailbox. If you enjoyed this episode, don't forget to subscribe on your favorite podcast platform and leave a five-star review. It really helps others discover the show. Thanks for tuning in. We'll see you next time for the next handshake. On this page On this page