> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci.
Agreed. I've got a crazy idea that I think might help...
Most tests have a step where you collect some data, and another step where you make assertions about that data. Normally, that data only ever lived in a variable, so it is not kept around for later analysis. All you get when you're viewing a failed test is logs with either exception or a failed assertion. It's not enough to tell a full story, and I think this contributes to the frustration you're talking about.
I've been playing with the idea that all of the data generation should happen first (since it's the slow part), it then gets added to a commit (overwriting data from the previous CI run) and then all of the assertions should run afterwards (this is typically very fast).
So when CI fails, you can pull the updated branch and either:
- rerun the assertions without bothering to regenerate the data (faster, and useful if the fix is changing an assertion)
- diff the new data against data from the previous run (often instructive about the nature of the breakage)
- regenerate the data and diff it against whatever caused CI to fail (useful for knowing that your change will indeed make CI happy once more)
Most tools are uncomfortable with using git to transfer state from the failed CI run to your local machine so you can just rerun the relevant parts locally, so there's some hackery involved, but when it works out it feels like a superpower.
Although I'd slightly rephrase that to "if you don't change anything, you should end up running pretty much the same code locally as in CI".
GitHub Actions is really annoying for this, as it has no supported local mode. Act is amazing, but insufficient: the default runner images are huge, so you can't use the same environment, and it's not supported.
Pre-commit on the other hand is fantastic for this kind of issue, as you can run it locally and it'll fairly trivially run the same checks in CI as it does locally. You want it to be fast, though, and in practice I normally wind up having pre-commit run only cacheable tests locally and exclude any build and test hooks from CI because I'll run them as separate CI jobs.
I have to disagree about Act, my experience is that it only works for extremely simple workflows, and even then it’s easy to run into differences between Act and GitHub Actions. I’ve raised many bugs but AFAIK there’s like one guy working on it in his own time.
It’s terrible that the community has had to invent something like this when it should be provided by GitHub. I suspect the GitHub Actions team is a skeleton crew because nothing ever seems to get done over there.
I think we're in agreement, generally -- and GitHub have no incentive to help, because Actions are a moat and if it's too easy to run outside their environment then it's too easy to move away from their environment.
It's no shade on the author of Act (or on the quality of the code they've written) that they can't keep up with GitHub.
> AFAIK there’s like one guy working on it in his own time
That may very well be true of nektos/act itself, but it now has at least two (depending on how one counts such things) ongoing forks over in the Gitea and Forgejo projects as they foolishly try to squat on it instead of using a real CI tool
I've been using brisk to run my CI from my local machine (so it runs in the cloud from my local terminal). The workflow is just a drop in replacement for local running. They've recently changed their backend and it seems to be working pretty smoothly. It works very well with AI agents too that are running in the terminal - they can run my tests for me if they make a change and it doesn't kill my machine.
Yeah, and I will never understand why developers accept anything less. GitHub CI is really bad for this. GitLab is a lot better as you can just run the exact thing through Docker locally. I like tools like tox too that automate your whole test matrix and do it locally just as well.
This is fine until your run into differences between your machine and the CI one (or you're writing code for a different architecture than the one you're using), but I agree, this is definitely the first step.
Double plot twist, I need to VPN into a remote network and now all the CI network activity goes through the VPN as well, and some of it gets blocked (true story)
I've poked at this a few times, and I think it breaks down for CI that needs/wants to run integration tests against other services. Eg it wants to spin up a Postgres server to actually execute queries against.
Managing those lifetimes is annoying, especially when it needs to work on desktops too. On the server side, you can do things like spin up a VM that CI runs in, use Docker in the VM to make dependencies in containers, and then delete the whole VM.
That's a lot of tooling to do locally though, and even then it's local but has so many abstractions that it might as well be running in the cloud.
nix-ci.com is built with this as one of the two central features.
The other is that it figures out what to do by itself; you don't have to write any YAML.
I tried out Dagger hoping to tidy up some of our CI pipelines. I really wasn’t impressed to be honest. I initially just wanted to build a basic Docker image using BuildKit and even that was nightmare that the Dagger guys basically told me just isn’t supported.
Very recently. It may run on BuildKit but it doesn't expose it as a build runtime. IIRC Dagger implements its own way of building images that lacks BuildKit features and doesn't work with Dockerfiles.
Hello! I'm the co-founder of Dagger. It seems that there was a misunderstanding, sorry about that.
Dagger certainly can build a container image natively, without requiring a Dockerfile. It also can build a docker image from a Dockerfile, thanks to a built-in compatibility bridge.
Dagger doesn't expose a raw buildkit socket, but it exposes a cleaner API that has 99% feature parity. Hitting the remaining 1% is rare, but it does happen :) Is there a specific buildkit feature you were looking for? Perhaps we can add it.
Here's an example from the CLI (keeping in mind that there are SDKs for Go, Python, Typescript, PHP and Java with 1-1 feature parity to the CLI):
Thanks for your reply. We use BuildKit's cache + SSH socket mounts in most of our Dockerfiles so this was a blocker I came up against immediately when trying Dagger. I'd rather not share my username here but there's a conversation on Discord last month you can dig up (search buildkit in:general) to see what was discussed and proposed solutions.
RE the part about not supporting Dockerfiles - sorry, that was incorrect, I misremembered.
Ah, I see the discussion thread on Discord, thanks.
It looks like you encountered a gap in the Dockerfile compatibility bridge, specifically lack of support for `--mount=type=ssh` [1]. It's true that our Dockerfile compat layer has more gaps than the native Dagger API itself. In effect, beyond very simple Dockerfiles, to get the full benefit of Dagger you need to convert them to code targeting the Dagger API. While there are many benefits to this, it does add significant friction to getting started.
It seems like we need to beef up our compatibility story!
Thanks for taking the time to share all this feedback, and sorry again that your experience didn't work out.
The whole point we started using automation servers as an integration point was to avoid the "it works on my machine" drama. (Have watched at least 5 seasons of it - they were all painful!).
+1 on running the test harness locally though (where feasible) before triggering the CI server.
Not necessarily. For one, the local dev environment may be different or less pristine than what's encountered in the CI. I use bubblewrap (the sandboxing engine behind flatpak) sometimes to isolate the dev environment from the base system. Secondly, CI often does a lot more than what's possible on the local system. For example, it may run a lot more tests than what's practical on a local system. Or the upstream Repo may have code that you don't have in your local repo yet.
Besides all that, this is not at all what the author and your parent commenter is discussing. They are saying that the practice of triggering and running CI jobs entirely locally should be more common, rather than having to rely on a server. We do have CI runners that work locally. But the CI job management is still done largely from servers.
> For example, it may run a lot more tests than what's practical in a local system.
yes this is what i was taking about. If there are a lots of tests that are not practical to run locally then they are bad tests no matter how useful one might think they are. only good tests are the ones that run fast. It is also a sign that code itself is bad that you are forced to write tests that interact with outside world.
For example, you can extract logic into a presention layer and write unit test for that instead of mixing ui and business logic and writing browser tests for it. there are also well known patterns for this like 'model view presenter'.
I would rather put my effort into this than trying to figure out how to run tests that launch databases, browsers, call apis , start containers ect. Everywhere i've seen these kind of tests they've contributed to "it sucks to work on this code" feeling, bad vibes is the worst thing that can happen to code
It does suck when those large scale integration tests fail but sometimes that's the only real way to test something. E.g. I have to call a service owned by another team. It has a schema and documentation so I can mock out what I think it will return, but how will I truly know the API is going to do what it says or what I think it says without actually calling the API?
> I truly know the API is going to do what it says or what I think it says without actually calling the API?
what if the API changes all of sudden in production? what about cases where api stays the same but content of response is all wrong? how do tests protect you from that?
edit: they are not hypothetical scenarios. wrong responses are way more common than schema breaking. tooling upsteam is often pretty good at catching schema breakages.
wrong responses often cause way more havoc than schema breakages because you get an alert for schema failures in app anyways.
Tests can't catch everything; it's a question of cost/benefit, and stopping when the diminishing returns provided by further tests (or other QA work) isn't enough to justify the cost of further investment in them (including the opportunity cost of spending our time improving QA elsewhere).
For your example, the best place to invest would be in that API's own test suite (e.g. sending its devs examples of usage that we rely on); but of course we can't rely on others to make our lives easier. Contracts can help with that, to make the API developers responsible for following some particular change notification process.
Still, such situations are hypothetical; whereas the sorts of integration tests that the parent is describing are useful to avoid our deployments from immediately blowing up.
That's exactly the point, isn't it? If the schema or response format changes, we want to catch that quickly. Canary, staging, or prod, take your pick but we need to call the API and run some assertions against that to make sure it's good.
That's fine. If they're too slow, run them post deployment. But do run them at some point so you can at least catch it quickly without waiting for user complaints.
It is a tradeoff, e.g. running tests with a real database or other supporting service and taking longer vs. mocking things and having a test environment that is less like reality.
https://testcontainers.com/ is not quite the solution to all your problems, but it makes working with real databases and supporting services pretty much as easy as mocking them would be.
I'd really recommend against mocking dependencies for most tests though. Don't mock what you don't own, do make sure you test each abstraction layer appropriately.
I moved from VSCode to Emacs with the help of Doom Emacs. You can check it out online. It’s a distribution and configuration management solution.
Generally people like to recommend writing your config from scratch but that didn’t work for me.
It comes with pretty much all you requested out of the box.
One thing I liked more than with the default Emacs setup is that I could “activate” support for a Language just by un-commenting a line in the well organized config.
Ooh, thanks! That sounds like something that will help me. I tend to do better when I can get something running like someone else has done in the past, then slowly tweak things as I learn more about it. My current VSCode setup and all my dotfiles have been slowly cribbed and edited from various sources who knew more than me about those things.
Seconded, Doom Emacs is such a huge QoL improvement over the DYI solution with a basic Emacs.
(I did use the upstream vanilla Emacs, but the pain of LSP support, etc. etc. was just too much until I tried Doom Emacs... and suddenly Emacs is usable again!)
> and on some days, i feel there is nothing to write
This is what I was trying to fix for myself.
And I found that creating a new text document is just an additional friction.
> i'd be tempted to answer with a multi page essay
This is what I visually restricted the input field for as I also feel like some questions would otherwise suggest a long answer. Trying to answer them in a brief way is a feature for myself.
it's not just that the question suggests a long answer, but it is that i like writing and i have a story to tell, so i want to tell that story. i would not even want to write a short answer because i already know what the short answer would be, and it's not very interesting so there is no point to even bother to write it down.
what i am trying to say is that, for me at least, the prompt itself would have to limit the scope to today or maybe this week. (but then i would not be able to write anything for this particular prompt)
maybe prompts like: what made you happy this week?
what made you sad?
what was a challenge you faced this week?
if you come up with a dozen questions like that, and then randomly rotate through those, one each day, i think that might produce interesting results.
There is not yet an open feature as you suggested but I feel like this could be nice. Not only for journaling but also for todo lists or similar, just starting every day with a blank sheet.
I saw that “1 Day” as suggested in another comment seems to be basically that.
I guess an about page is a great idea, I'll create one that answers the questions here later.
My setup for the questions is the following - I have a questions file with one question per line and a cron job that takes the top one, builds the static page by including it in the template and then appends the question in the end.
That way I can always add more questions but if I run out of questions it would just start from the beginning.
Do you have an idea on how to save to a git repo from a static page? I don't want to make it any more complicated than it needs to be.
Do you have an idea why you could not maintain it?
I think another component why I open the page every day is that I only have the chance to fill this question that day - something that you wouldn't have in the paper one.
A fixed prompt likely was why I couldn't maintain it. The prompt was generic and open ended, but I tired quickly of coming up with new stuff to the same questions.
But the likely real reason is that it just became a chore. I probably should have reduced it to once a week or something. I have kind of tried that as well and still failed.
Another reason is quality. The first draft of anything sucks really bad, but I don't want to write badly. It's a lot more effort to write something present or future me will be happy with.
Ultimately, I think it's a matter of perspective and priority. I like the idea of journaling, but it is somewhat labor intensive (especially if the "first draft" is not good enough for you). The reality is I often choose to do something else with that time.
I can just open it in other tools and for example merge two sources. E.g. I do this on my laptop and on my phone.
Download both, open with Excel and sort by date.
I also considered more questions in the future or tracking numerical values.
There is a great community driven list [0] of 1 on 1 questions that I like to pick a couple questions as conversation starters or as backup when the conversation goes stale.
It is also available as JSON so you can script the random selection.
locally running CI should be more common