It's the wrong architecture from a dependency management perspective. Directly importing a table into Iceberg allows analytics consumers to take dependencies on it. This means the Postgres database schema can't be changed without breaking those consumers. This is effectively a multi-tenant database with extra steps.
This is not to say that this architecture isn't salvageable - if the only consumer of the Iceberg table copy is a e.g. view that downstream consumers must use, then it's easier to change the Postgres schema, as only the view must be adjusted. My experience with copying tables directly to a data warehouse using CDC, though, suggests it's hard to prevent erosion of the architecture as high-urgency projects start taking direct dependencies to save time.
Eh, as long as it isn't life or death, I think allowing direct consumption and explicitly agreeing that breakage is a consumer problem is better for most business use cases (less code, easier to maintain and evolve). If you make a breaking schema change and nobody complains, is it really breaking?
I have spent way too much life maintaining consumer shield views and answering hairy schema translation questions for use cases so unimportant the downstream business user forgot they even had the view.
Important downstream data consumers almost always have monitoring/alerting set up (if it's not important enough to have those, it's not important) and usually the business user cares about integrity enough to help data teams set up CI. Even in these cases, where the business user cares a lot, I've still found shield views to be of limited utility versus just letting the schema change hit the downstream system and letting them handle it as they see fit, as long as they're prepared for it.
> it's hard to prevent erosion of the architecture as high-urgency projects start taking direct dependencies to save time.
IME, it feels wrong, but it mostly does end up saving time with few consequences. Worse is better.
If the LLM was as smart as a human, this would become a social engineering attack. Where social engineering is a possibility, all three parts of the trifecta are often removed. CSRs usually follow scripts that allow only certain types of requests (sanitizing untrusted input), don't have access to private data, and are limited in what actions they can take.
There's a solution already in use by many companies, where the LLM translates the input into a standardized request that's allowed by the CSR script (without loss of generality; "CSR script" just means "a pre-written script of what is allowed through this interface"), and the rest is just following the rest of the script as a CSR would. This of course removes the utility of plugging an LLM directly into an MCP, but that's the tradeoff that must be made to have security.
* A media player with playlist of local media that executes in a web browser.
* proxies for http and WebSockets. Apache made this challenging and I thought I could do it better. I can now spin up servers in seconds and serve http and WebSockets on the same port
* tools to test dns, http, WebSockets, hashes, certificate creation, and more
I think OP was saying those three things were surprisingly dangerous. Kids have a natural fear of heights and falling, while the three on the list not so much.
Babies will happily crawl off of the edge of whatever they're on. I'm not sure if it's because they aren't afraid, or if it's because they're so used to being carried that they don't grasp the concept of gravity, or both.
My toddler recently went out on our roof to retrieve a football. I expected her to be a bit nervous, but she walked right up to the edge, no fear apparent at all. I had to desperately shove my instinct to yell for her down so I didn't scare her and distract her.
There's research (and my own anecdotal evidence) supporting my initial claim, so I don't see how it's an obvious mistake. Do you have anything to back up your point?
This is like arguing with a wall. Right back from Gibson & Walk’s 1960 visual‑cliff experiment, there's endless research showing that babies don't have an innate fear of falling. It's so uncontroversial that it's now taken as undisputed fact in medical documentation and research.
You linked to an article about the visual-cliff experiment (apparently having not read it?) as it is what kicked off the avenue of research that came to this conclusion and which has been confirmed and uncontroversial since the mid-2010's.
It's also the lived experience of billions of parents.
There is no currently viable counter-arguments presented anywhere globally. There is more consensus about this issue than, for example, anthropogenic climate change or pangea or any number of other issues than reasonable people aren't expected to defend due to their overwhelming acceptance.
Where this kind of API design is useful is when there is a user with an agent (e.g. a browser or similar) who can navigate the API and interact with the different responses based on their media types and what the links are called.
Most web APIs are not designed with this use-case in mind. They're designed to facilitate web apps that are much more specific in what they're trying to present to the user. This is both deliberate and valuable; app creators need to be able to control the presentation to achieve their apps' goals.
REST API design is for use-cases where the users should have control over how they interact with the resources provided by the API. Some examples that should be using REST API design:
- Government portals for publicly accessible information, like legal codes, weather reports, or property records
- Government portals for filing forms and other interactions
- Open data initiatives like Wikipedia and OpenStreetmap
Considering these examples, it makes sense that policing of what "REST" means comes from the more academically-minded, while the detractors of the definition are typically app developers trying to create a very specific user experience. The solution is easy: just don't call it REST unless it actually is.
> Where this kind of API design is useful is when there is a user with an agent (e.g. a browser or similar) who can navigate the API and interact with the different responses based on their media types and what the links are called.
The funny thing is, that perfectly describes HTML. Here’s a document with links to other documents, which the user can navigate based on what the links are called. Because if it’s designed for users, it’s called a User Interface. If it’s designed for application programming, it’s called an Application Programming Interface. This is why HATEOAS is kinda silly to me. It pretends APIs should be used by Users directly. But we already have that, it’s called a UI.
The point is that your Web UI can easily be made to be a REST HATEOAS conforming API at the same time. No separate codepaths, no duplicate efforts, just maybe some JSON templates in addition to HTML templates.
You're right, pure REST is very academic. I've worked with open/big data, and there's always a struggle to get realistic performance and app architecture design; for anything non-obvious, I'd say there are shades of REST rather than a simple boolean yes/no. Even academics have to produce a working solution or "application", i.e. that which can be actually applied, at some point.
> Where this kind of API design is useful is when there is a user with an agent (e.g. a browser or similar) who can navigate the API and interact with the different responses based on their media types and what the links are called.
It's also useful when you're programming a client that is not a web page!
You GET a thing, you dereference fields/paths in the returned representation, you construct a new URI, you perform an operation on it, and so on.
Consider a directory / database application. You can define a RESTful, HATEOAS API for it, write a single-page web application for it -or a non-SPA if you prefer-, and also write libraries and command-line interfaces to the same thing, all using roughly similar code that does what I described above. That's pretty neat. In the case of a non-SPA you can use pure HTML and not think that you're "dereferencing fields of the returned representation", but the user and the user-agent are still doing just that.
> Where this kind of API design is useful is when there is a user with an agent (e.g. a browser or similar) who can navigate the API and interact with the different responses based on their media types and what the links are called.
> Most web APIs are not designed with this use-case in mind.
I wonder if this will change as APIs might support AI consumption?
Discoverability is very important to an AI, much more so than to a web app developer.
MCP shows us how powerful tool discoverability can be. HATEOS could bring similar benefits to bare API consumption.
It's unclear to me why the "god object" pattern described in the article isn't a good solution. The pattern in the article is different from the "god object" pattern as commonly known[1], in that the god object in the article doesn't need to be referenced after initialization. It's basically used once during initialization then forgotten.
It's normal for an application to be built from many independent modules that accept their dependencies as inputs via dependency inversion[2]. The modules are initialized at program start by code that composes everything together. Using the "god object" pattern from the article is basically the same thing.
The God Object needs to be referenced at least transitively by the wrappers you hand out, and its API surface needs to grow constantly to encompass any possibly security-sensitive operation.
Dependency injection doesn't help here much, at least not with today's languages and injectors. The injector doesn't have any opinion on whether a piece of code should be given something or not, it just hands out whatever a dependency requests. And the injector often doesn't have enough information to precisely resolve what a piece of code needs or resolve it at the right time, so you need workarounds like injecting factories. It could be worth experimenting with a security-aware dependency injector, but if you gave it opinions about security it'd probably end up looking a lot like the SecurityManager did (some sort of config language with a billion fine grained permissions).
What is "injector"? Is this the Service Locator anti-pattern?
The application's entry point takes the God Object of capabilities, slices it accordingly to what it thinks its submodules should be able to access, and then initializes the submodules with the capabilities it has decided upon. Obviously, if some submodules declare that they need access to everything plus a kitchen sink, the choice is either a) give up and give them access to everything; b) look for a replacement that requires less capabilities.
A useful error message would have made this a 1-minute investigation. The "fix" of trying to detect this specific program is much too narrow. The right fix is to change Yarn to print a message about what it was trying to do (check for a Sapling repo) and what happened instead. This is also likely a systemic problem, so a good engineer would go through the whole program and fix other places that need it.
> I found that there’s a slight aversion to creating new types in the codebases I work in.
I've encountered the same phenomenon, and I too cannot explain why it happens. Some of the highest-value types are the small special-purpose types like the article's "CreateSubscriptionRequest". They make it much easier to test and maintain these kinds of code paths, like API handlers and DAO/ORM methods.
One of the things that Typescript makes easy is that you can declare a type just to describe some values you're working with, independent of where they come from. So there's no need to e.g. implement a new interface when passing in arguments; if the input conforms to the type, then it's accepted by the compiler. I suspect part of the reason for not wanting to introduce a new type in other languages like Java is the extra friction of having to wrap values in a new class that implements the interface. But even in Typescript codebases I see reluctance to declare new types. They're completely free from the caller's perspective, and they help tremendously with preventing bugs and making refactoring easier. Why are so many engineers afraid to use them? Instead the codebase is littered with functions that take six positional arguments of type string and number. It's a recipe for bugs.
> I've encountered the same phenomenon, and I too cannot explain why it happens.
I think that some languages lead developers to think of types as architecture components. The cognitive cost and actual development work required to add a type to a project is not the one-liner that we see in TypeScript. As soon as you create a new class, you have a new component that is untested and unproven to work, which then requires developers to add test coverages, which then requires them to add the necessary behavior, etc.
Before you know it, even though you started out by creating a class, you end up with 3 or 4 new files in your project and a PR that spans a dozen source files.
Alternatively, you could instead pass an existing type, or even a primitive type?
> But even in Typescript codebases I see reluctance to declare new types.
Of course. Adding types is not free of cost. You're adding cognitive load to be able to understand what that symbol means and how it can and should be used, not to mention support infrastructure like all the type guards you need to have in place to nudge the compiler to help you write things the right way. Think about it for a second: one of the main uses of types is to prevent developers from misusing specific objects if they don't meet specific requirements. Once you define a type, you need to support the happy flows and also the other flows as well. The bulk of the complexity often lies in the non-happy flows.
> I think that some languages lead developers to think of types as architecture components
It's not any languages doing that; it's their company culture doing that.
Java-style languages (esp. those using nominative typing, so: Java, C#, Kotlin, Swift, but not Go, Rust, etc) have never elevated their `class` types as illuminated representations of some grandiose system architecture (...with the exception of Java's not-uncontroversial one-class-one-file requirement); consider that none of those languages make it difficult to define a simple product-type class - i.e. a "POCO/POJO DTO". (I'll pre-empt anyone thinking of invoking Java's `java.beans.Bean` as evidence of the language leading to over-thinking architecture: the Bean class is not part of the Java language any more than the MS Office COM lib is part of VB).
The counter-argument is straightforward: reach for your GoF Design Patterns book, leaf through to any example and see how new types, used for a single thing, are declared left, right and centre. There's certainly nothing "architectural" about defining an adapter-class or writing a 10-line factory.
...so if anyone does actually think like that I assume they're misremembering some throwaway advice that maybe applied to a single project they did 20 years ago - and maybe perhaps the company doesn't have a meritocratic vertical-promotion policy and doesn't tolerate subordinates challenging any dictats from the top.
> Think about it for a second: one of the main uses of types is to prevent developers from misusing specific objects if they don't meet specific requirements.
...what you're saying here only applies to languages like TypeScript or Python-with-hints - where "objects" are not instances-of-classes, but even then the term "type" means a lot more than just a kind-of static precondition constraint on a function parameter.
> But even in Typescript codebases I see reluctance to declare new types.
The current Typescript hype / trend is to infer types.
Problem is at some point it slow things down to a crawl and it can get really confusing. Instead of having a type mismatch between type A and type B you get an error report that looks like a huge json chain.
This can't be the explanation for everything, but I do know that once upon a time just the sheer source-code size of the types was annoying. You have to create a constructor, maybe a destructor, and there's all this syntax, and you have to label all the fields as private or public or protected and worry about how it fits into the inheritance hierarchy... and that all still applies in some languages. Even the dynamic scripting languages like Python that you'd think it would be easy tended to need an annoying and often 100% boilerplate __init__ function to initialize them. You couldn't really just get away with "class MyNewType(string): pass" in most cases.
But in many more modern languages, a "new type" is something the equivalent of
type MyNewType string
or
data PrimaryColor = Red | Green | Blue
and if that's all your language requires, you really shouldn't be afraid of creating new types. With such a small initial investment it doesn't take much for them to turn net positive.
You may need more, but I don't mind paying more to get more. I mind paying more just to tread water.
And I find they tend to very naturally accrete methods/functions (whatever the local thing is) that work on those types that pushes them even more positive fairly quickly. Plus if you've got a language with a halfway modern concept of source documentation you get a nice new thing you can document.
>Even the dynamic scripting languages like Python that you'd think it would be easy tended to need an annoying and often 100% boilerplate __init__ function to initialize them.
For this reason, I very much appreciate the dataclass decorator. I notice that I define classes more often since I started using it, so I'm sure that boilerplate is part of the issue.
> But in many more modern languages, a "new type" is something the equivalent of
You don't even need a modern language for that kind of thing, plenty of languages from a half century or so ago also let you do that. From Ada (40+ years old):
type PrimaryColor is (Red, Green, Blue);
Or if you're content with a mere alias, C (50+ years old) for your first example:
It is true that modern is strictly speaking not the criterion. But what I was referring to is that the languages that were popular in, say, the 90s, generally did have that degree of ceremony.
C has its own problem. First, typedef isn't a new type, just an alias. But C's problem isn't the ceremony so much as the global namespace. Declaring a new type was nominally easy, but it had to have a crappy name, and you paid for that crappy name on every single use, with every function built for it, and so forth. You couldn't afford to just declare a new "IP" type because who knows what that would conflict with. A new type spent a valuable resource in C, a namespace entry. Fortunately modern languages make namespaces cheap too.
My initial interpretation of the title was that the TS team was adding support for another, faster, target such as the .NET runtime or native executables. The title could use some editing.
