For those not familiar with recent UK scandals, there has been one ongoing for a few years where there was a bug in a system designed by Fujitsu that was calculating the finances for post offices. This resulted in the system quoting wrong balances for post offices, and when that didn't match the actual cash in hand, resulted in prosecutions of the postmasters responsible for those offices which in turn led to convictions, dismissals, fines, imprisonments, and suicides over a 15 year period as the computer system was presumed correct.
Anyway, it was later proven the computer system was incorrect but the government there dragged their heels on exoneration and compensation.
This reminds me of a story my physics professor told me. Couriers were delivering gold between countries at different longitude. At the origin the gold was weighed using balance scales. The the destination the gold was weighed using spring scales. However, due to differences in gravity and the rotation of the earth, there was a significant difference in the weights. (But not the mass!) This made the authorities think the men had shaved the bars somehow. It was a simple way to demonstrate the difference between weight and mass.
Whether the story was true or not, the government is just bad at science. It's how we get stuff like bite mark patterns and facial recognition warrants.
From my read of the information publicly available, there was a fundamental lack of distributed/transactional system understanding from the developers at Fujitsu.
This[1] article recently posted here[2] reminded me of the Post Office Scandal.
The lack of respect for established CS theory (transactions and distributed systems) and established accounting practices (double entry book keeping or even the idea of a ledger) is mind boggling.
IMO it's not really a lack of respect as it is software vendors and managers not really being incentivised to do the right thing. Therefore even if the engineers know the right thing to do, they are unable to solicit the buy in from stakeholders to invest the development into doing the right thing because there's no recourse for doing the wrong thing but it's more expensive.
And even if you do the right thing, if a competitor comes in and offers to do the "job" quicker and cheaper because they're doing the wrong thing. You may have the moral highground but that doesn't pay your employees' salaries.
What we need is a better framework for punishing bad software, in an ideal world without a bunch of red tape and reducing the burden on non experts identifying what is or isn't bad behaviour.
> managers not really being incentivised to do the right thing
It's more likely that managers are just incompetent. Not being able to distinguish between infrastructure and "decoration" means all their decisions are at best superficially motivated.
The punishment we have for this is unemployability and bankruptcy. It will happen by itself if you let it.
A framework for "punishing bad software" sounds to me to be hiding the ambition to "protect bad managers from consequences of their decisions".
> The punishment we have for this is unemployability and bankruptcy. It will happen by itself if you let it.
In my career, I have never seen a manager see consequences for making the decision that values "money now" vs "avoiding things going wrong later". I do not believe that it will happen by itself.
Well yes, agree this is how it looks now. But my perspective is that there has to be layers of mechanisms protecting them, rather than the natural order. (These mechanisms are tricky to pin-point but seem to be on the themes of mutual loyalty, avoiding embarrassment, keeping face, and so on.)
Imposing some formal framework is more likely to insulate managers even further while making software developers accept legal responsibility beyond their pay.
> The punishment we have for this is unemployability and bankruptcy. It will happen by itself if you let it.
yeah, nah.
"No one ever got fired for buying IBM"
after decades of incompetence the multinational contracting firms are still going fine. IBM still runs fine and they're focus is now even more in the insulting side of the business.
I'm pretty ignorant on the ins and outs and trade offs associated with accreditation but yes that sounds like one solution.
It doesn't strike me as unreasonable, software engineers are now able to cause billions in damages and/or cause loss of life via primary or secondary effects.
Sure, one of software's greatest strengths was that anyone could learn it and it'll be sad to lose that but that's not really a reason for structurally enabling things like the Post Office scandal.
You probably don't need accreditation to serve cat pics but things dealing with money and life/death should. But IANAL so hopefully someone else can bring better insight to this area.
Some of the sites we work on are insurance based. We save every step of the calculation (inputs and outputs, e.g. rates and sub-totals). An administrative user can see the entire calculation from start to end, including overrides at various steps (e.g. a manual discount), and breakdowns per state or item insured if appropriate. This seems like the acceptable bare minimum to me, rather than just showing a magic number. And it definitely helps to expose bugs.
> Software needs to be transparent, a human expert worker needs to be able to verify the result a software generates within reasonable time.
This is becoming more and more important as the AI industry is pushing non-deterministic computing further and further. There will be court cases where the full automated decision chain will be called into question, and a default assumption of "well, we don't have any logs so we should assume the software did the right thing" terrifies me.
The magic word that always pops up to me is "liability."
Even speaking from an IC/non-managerial place, far too many technical problems in businesses are really just symptom-alleviation (or worse, performative look-I'm-doing-something theater) for a deeper problem which involves misaligned incentives for humans.
People will still be able to learn how to program and actually program. But if they take money or deal with people's private information, then they'll need to comply with the standards that will be regulated.
I think we need to start thinking of an individual's persona which includes all the information about them that is online or in government (non secret) files.
That needs to be considered when designing regarding regulations for software that interacts with someone's persona.
So anyone can set up the equivalent of an art stand in the park (serving cat pics), but if they start either selling cat pics or selling advertising that relies on collecting information about the personas on the site, then that needs to be regulated.
> People will still be able to learn how to program and actually program. But if they take money or deal with people's private information, then they'll need to comply with the standards that will be regulated.
You can even have non-licensed people doing work, and a licensed individual signing off on the end result. They would need to review the designs and work to make sure they agree with how the work was done, because it's their license on the line if something goes wrong because of bad craftsmanship.
It doesn't need to be too dramatic. It's already a general rule never to write your own cryptographic or financial code from scratch, you reach out for established libraries or services written by experts.
The law may require to use the ones that are certified if you are doing something sensitive. It doesn't need to be universal necessarily, but it should apply to public tenders for critical infrastructure at least.
Yes. Legally mandated supply requirements have been an element in every other engineering sector for decades or centuries. If you're building an airplane you can't purchase from any random aluminum supplier, there's an authorized list.
That would seem to result in industry consolidation and declining innovation.
Software isn't made of physical materials anyway, it's speech. How about instead we impose open source requirements to enable public verification of critical systems?
Innovation like calculating numbers incorrectly and getting people sent to jail? Or maybe just another hundred slow, rent-seeking React web apps that'll disappear when the series B money dries up, if we're lucky. We must have different ideas of what innovation is.
So, outsource the core government function of regulation to "the public". Who exactly do you imagine is going to actually do this public verification, and where is their paycheck going to come from.
Some software is critical infrastructure and needs to be treated as such. We are not special. Every other engineering discipline has gone through this same process as and arrived at the inevitable conclusion that government regulation is essential, but only after causing unthinkable damage to the public first
software "engineers" love the cachet but hate the idea that they might also have to uphold the values and responsibilities of engineering.
I say we regulate the word engineer the same way it is in many countries for real engineering. if you don't want to progress beyond code monkey, you can be a software developer and innovate yet another react clone. if you want to be called an engineer, you learn and follow the regulations.
This isn't (or doesn't have to be) about laws specifying how all software has to be made.
What they need to specify is the standards for software that certain types of organizations can use. Like government agencies, government contractors, medical organizations, construction and engineering firms, and probably some other kinds of large private businesses, depending on their industry.
Basically, if the software your organization uses can cause the level of destruction that Horizon did, it needs to have specific certifications, or you can't use it.
In order for such software to be certified, it needs to meet certain clearly-defined standards of quality, potentially including having all the technical leads of some level (or just all the developers, depending on various factors) be licensed, and have their licenses on the line of something like this scandal occurs.
It's not a panacea, and it would definitely be an absolute bear to get the terms of all of it defined both clearly and in a way that is likely to actually produce a quality product, but IMO it is likely to be worth it in the long haul.
the same could be said for aerospatial regulations, and we still manage to do them. civil constructions are mindbogglingly complicated and we still manage.
all it takes is enough people to die, and/or for rich people to lose enough money and it'll become the rule.
At least when you are building key infrastructure for the government, directly or indirectly through third parties. I don’t think it’s unreasonable.
And it is not as absolute as you make it sound. Only dependencies for specific critical functions may be regulated. And they don’t have to literally force a whitelist of dependencies on you, just whichever has been certified as appropriate for that purpose.
This is how you get FIPS 140 [1], which for those not in the know is a US Federal standard that mandates encryption which is _less_ secure the the current state of the art and has been for decades. (Yes, there's a new version which was approved 5 years ago and which is still rolling out [2]).
At the same time, turning on FIPs mode is the way we discover that some of our modules were using MD5 in security critical places. Because the government actually enforces FIPS, people (primarily Red Hat I think), now actually put in the bare minimum of engineering effort so that when you set fips=1, the system will actually enforce the policy (unless you go out of your way to override it, or use a non distribution provided crypto stack).
Sure, now that the infrastructure for this has been built, it can be configured to require stronger crypto then FIPS does, but that infrastructure would never have been built without the likes of FIPS, and the government mandating it's use. And I know this because even with all of the hard engineering work done of building that infrastructure, there are no commonly used stronger policies; because the only people who actually care are the ones forced to care by the likes of FIPS.
Our electrical standards might not the safest way of wiring buildings, and not what we would come up with if we wrote the standards today. But they are orders of magnitude safer then what electricians would be doing without the standards.
I don't understand the confusion, practically everything we interact with in our daily lives is regulated and goes through certification processes, just look at how all that is done.
Food, drugs, healthcare, consumer products, chemicals, cars, planes, trains, buildings, utilities, energy, infrastructure, salaries, loans, investments, accounting... Even media requires some licenses, receives age ratings, and has restrictions on advertising.
It's not rocket science, this is normal for every single other industry.
And that has resulted in massive industrial consolidation and lack of innovation in food, drugs, healthcare, consumer products, chemicals, automotive/aerospace/locomotive manufacturing, construction, utilities, energy, payments, finance, accounting, and media outside "tech," where we manage our own dependencies.
Rocket science is one of the few industries that's actually seeing active innovation.
Famously, civil engineering has never solved this problem, either, leading to the sad state of affairs in which we find ourselves today, wherein no bridges have been built anywhere.
The way it works for aircraft is that now it's getting prohibitively expensive to design new planes. But new planes are still needed, so Boeing found a solution, and made the 737-MAX which is in theory an upgrade that does not require re-certification, but is in fact different enough that the differences lead to hundreds of deaths.
Perhaps without the certifications lots more people would have died. I'm just an armchair analyst. Just food for thought.
Uh, the MAX was a disaster because Boeing is run by MBAs instead of engineers. Regulation is why air travel is remarkably safe. Those rules are written in blood, something everyone here conveniently seems to forget.
So you write regulations for software development, but software companies run by MBAs instead of engineers can still cause disasters. I guess as long as disasters are rare, it's a win.
I think it needs to be on the individual-liability level. Blessing implementations from corporations is an environment for monopolies to grow and corruption to set in.
Sure, it doesn't need to up-end the whole field, but it is rather obvious that critical infrastructure should require significant compliance, particularly if it's being purchased in a public tender process where the cheapest supplier that fits the criteria is hired. Certifications like SoC 2 are already a big thing.
I don't work in IT and I'm not a developer, but IMO a developer should do the right thing even if that thing is not written in the specs/contract (transactions and ledger in this case). It is not the job of the developer to care about the competitiveness of the company (Fujitsu in this case), contracts, money, etc.
It seems to me that it's a fundamental failure of software engineering culture.
The number one rule for engineering domain applications is to understand the domain.
I would blame the perennial neophilia and lack of (or inadequate adoption of/respect for) standardized texts in the industry. Though, to be fair, a lot of this does come down to the rapid changes in the technology.
Iterative development is necessary for software, of course, but this should be understood as a necessity due to the medium, not as an excuse for skipping research and design. A lot of these domains (especially something as critical as accounting) should be solved problems.
This is not a story of a failure in software engineering. Shit happens, no system is 100% reliable. The failure lies in how management handled it (pretend there’s no issue, don’t launch an internal investigation, blame others).
No doubt management also oversaw the development of the system and rushed it to production.
A healthy culture should accept failure as inevitable and learn from it when it occurs. It should also listen to the people who know best: the engineers who built the thing. You know, like the aerospace industry.
The damage that morons in suits do in pursuit of their bonus cannot be overstated.
I'm not sure I entirely agree in this circumstance.
Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy.
>Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy.
The developers were just doing their job. It's management's responsibility to construct a functioning system of checks and balances and understand the limitations of their systems, both of which they failed to do. If it weren't for their hubris the fundamental issues with Horizon could have come to light much earlier.
Let's also not forget that the reason executives are compensated well is for them to take accountability in situations like this.
I think you’re trying to fit this into an exclusive OR when it really is AND. Unless the developers strongly warned management about the lack of idempotency and were ordered to continue anyway, they share the blame for those bugs even if they were not part of the conspiracy to cover them up (this also goes for anyone involved in those hot fixes trying to patch up errors without telling the customer).
More broadly, your idea that this is solely a management problem is how we end in situations where developers are being told to unquestioningly code some design exactly as given, which never works. You don’t get professional judgement if you don’t accept responsibility, too.
But I would also say that that kind of toxic management is absolutely a part of "software engineering culture". How many horror stories do people on here have of managers who care nothing about the quality of the product, only meeting the deadline so they can get their bonus?
"Software engineering culture" is way, way more than just "how write good code." It includes how we work, how we manage/are managed, how we advocate for ourselves, or fail to do so, and much more.
It certainly includes the very common resistance to unions among programmers, and assuming this was caused by management pushing a known-bad product out the door, a strong union would have (at least potentially) been able to stand up to such demands.
Programmers who get paid asstons of money don't want to admit that these are also fundamentally business failures, partially because if they were fixed, they probably wouldn't get six figures for styling HTML buttons anymore.
There's also the issue where public software engineering related projects are too broad in scope and poorly defined. Rather than making small projects that are useful and then expanding upon them the trend is to specify massive nationwide databases and then just throw money at them, often using 'Big bang' deployments. These are so called because you deploy them and then there's a large explosion ;)
The thing literally went into a criminal court as evidence, and was "presumed correct" in a way that overloaded any technical or reasonable discordance.
The largest failure here was from the judges and lawyers. The software failure isn't even relevant.
I'm so tired of seeing "no system is 100% reliable" being used to excuse the most fundamental failures in this industry. Nobody's asking for 100% perfection, we're asking for the people writing the software to do X to actually do X correctly in the first place. How about, I don't know, 95% correctness? Is that too onerous?
Engineering is a professional activity, a legal title is just a moat, it doesn't mean that it is Engineering, just a protected profession of some sort.
In al seriousness, Engineering is about verifying systems to make sure their lifespans and failure modes are known, up front.
This has a legal dimension and a practical one. Legally you can make people liable for unreliable systems. But you can also be liable for failure to maintain properly, or failure to warn about impending calamity. Because it's all documented and verified.
Practical you can live worry free in earthquake and flooding proof buildings, trusting in the diligence of Engineers, and maintenance workers, because they and others have liability imposed on them.
For software this is only the case in a few sectors. For buildings in all cases. Not comparable.
Software engineering culture is still in a very primitive phase: just take a look at the comments of the category theory submission currently in the front-page https://news.ycombinator.com/item?id=42291141
Try suggesting to use a tool like TLA+ to validate some complex design and the most likely scenario is that people will laugh at you, even if it's a critical component for the business.
Most decisions in the industry are based on weak anecdotes and unfounded opinions of underserved "authorities".
There were "bugs", yes. There were also fundamental design flaws.
One of the things the Post Office did was sell travel money, but the whole system was never really designed for ForEx operations, so it didn't keep track of exchange rates over time. The result is that reconciliation used the exchange rate at time of reconciliation instead of at the time of trade. So, if the foreign currency had gone up in value, it would show up as GBP missing.
As a developer working on trading platforms, this is horrific.
Given the Post Office had a reputation for really good exchange rates, this one design flaw might be responsible for a significant portion of the problem.
Even still, exchange rates aren't a fixed value globally at any given time. You should record the rate with the transaction because the rate could be different from your log of exchange rates for any number of different reasons.
Buggy software is important aspect of the story but to me it’s not the main one.
The main one is that the post office management/officials at some point became aware of the bugs and that they were ruining lives of innocent people and they knowingly kept lying to save their asses.
I have a lot of respect for the judge after reading this. Here’s a quote describing Post Office evidence:
“bare assertions and denials that ignore what has actually occurred… [amounting] to the 21st century equivalent of maintaining that the earth is flat”.
To be fair, the government were lied to by the Post Office people about whether or not there were bugs. They also lied about whether Fujitsu had the ability to amend the ledgers even though Fujitsu had a team working to correct some of the inaccuracies.
Also, mind-blowingly (to me at least) - Horizon is still in use, Fujitsu are still being given hundreds of millions of pounds to support it, and its replacement isn't due to go live until 2030
A strong recommendation for the excellent "Mr Bates vs The Post Office" to anyone who hasn't followed it, a short (four part) drama about it which is really worth a watch, and a good way to get a rough feel for what went on, and also managed to finally catalyse a real response by the government.
Alan Bates sounds like the epitome of the British "stiff upper lip". Stubbornly standing against overwhelming adversity for years, for the good of his fellow society and his own integrity.
There's a little more to it. Most of the comments here are focusing on
"correctness". And yes, the amendments to section 69 do something
towards tempering its ridiculous and dangerous "presumptions".
But the (UK Post-Office) story is that the Horizon system had
back-doors in it. Fujitsu denied this. "Corrections" were made to
systems without operator knowledge - to fix actual errors caused by a
terrible database sync script full of race-hazards, faulty locks and
duplicated state.
The cover-up began life as engineers trying to hide up technical
mistakes. It escalated to senior executives trying to cover up
financial and political mistakes. It ended with the Crown colluding in
covering up judicial mistakes. It is an exemplar of hubris, pride and
egotsim resting on a refusal to give up a religious belief in
technology. Were it not for the courage of a few (including judges and
MPs) they would have gotten away with it (if it weren't for those
meddling kids)
The case stands as an important landmark that you cannot "hide behind"
technology as a means for abuse and injustice.
The amendments are welcome but insufficient. They open up a good
opportunity for cybersecurity people to work with lawyers now.
There are two outstanding problems:
Proprietary code. If you cannot examine the system then the right to
challenge it is meaningless. This requires changes to investigatory
powers/discovery if anyone wants to use "technical correctness" as a
base for argument.
Malicious function. While the discussion revolves around correctness
it is incomplete. Many systems (perhaps not the Horizon system) are
not faulty, they work perfectly well to deceive, manipulate and
swindle.
I'd still push for a complete reversal of presumption [0]. Where software
is part of a legal dispute it should "take the stand" as its own
witness, in that formal proofs of correctness (a very VERY high bar in
software engineering) need to be brought in front of the
court. Otherwise the reasonable presumption is that an error or hidden
malicious coding "cannot be ruled out".
I am already consider cynical by many, but reading about the scandal constantly reminds me how naive I am. Just how low can government and politicians get.
I actively avoided the news and outrage about this for my own sanity. It was just too shocking. The worst part is that I'd heard about it and got mad about it many years ago, and it wasn't until a TV documentary came out recently that anything happened.
Echoing this, I've been passively following the inquiry but haven't watched the miniseries.
I think it would leave me furious for weeks and push me into a depression rut.
I'm trying to take the whole debacle as long term inspiration to be excellent at what I do - it's tough to stay positive when so many people involved (Fujitsu, PO upper management, the original prosecution) seem entirely morally bereft, with little chance of consequence.
I guess some people never learn from trying to cover their mistakes as children. I've explained this to my kids: often the initial mistake is minor and warrants nothing more than "please don't do that again". It is the lying and hiding that blow it up into a big issue. The odds of being found out are very high. There is little reward and lots of risk.
When you have such a widespread and ongoing problem it becomes clear that such a large proportion of postmasters can't be criminals. Computer bugs are well known. Eventually it is going to come out that the computer is wrong. Why double-down? The earlier you admit a mistake and apologize the lower the impact and the less anyone cares. They had a built-in scapegoat that everyone understands and accepts: the vendor's software had bugs! We have daily meetings to yell at them to fix the bugs we promise and we will fix the problems ASAP.
Or do the sneaky thing and fix the issues, stop prosecuting postmasters, and ignore the ones you prosecuted by mistake. Cynical, cruel, and immoral... but contains the damage.
Instead the UK Postoffice seems to have just let the problems continue while simultaneously allowing prosecutions to go ahead knowing they were faulty. Literally the worst of all worlds: ongoing accumulation of liability, now with provable malice!
Were there actual bad consequences for the upper management and politicians involved? I think the risk assessment you teach your children does not apply to british politicians.
I heard some convictions of innocent people have been overturned, or are in the process of. But new convictions of the actually guilty... no news about any such thing. Maybe I'm just uninformed.
It doesn’t make sense to change an entire system over once incident. That smells of overreaction to me. It was an awful situation caused by humans, and the humans involved should probably be punished, that doesn’t mean one should break a system that generally works very well and promotes self regulation and continuous improvement rather than shackling it with regulations.
The fact that this misnomer of infallible computer systems was ever enshrined in law is pretty damning of the whole UK legal system and the relationship between technical people and law.
Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%.
Not least because it seems to go against the core concept of "innocent until proven guilty" that the whole legal system is meant to rest upon.
> The fact that this misnomer of infallible computer systems was ever enshrined in law
Is this actually a fact, or a fact taken to it’s logical conclusion to presume a new “fact”?
The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software. Instead, precedents got set over time by enterprising lawyers, but setting a precedent when it’s convenient is not the same thing as writing a law.
When I see mechanical systems, I think of something like an abacus. I’ve never used one, but I suspect the abacus itself is infallible. It’s open, it’s transparent, it is easily auditable, and the same inputs will always produce the same outputs. There is no black box translation occurring, like occurs with computer software.
>The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software.
It's worse than that.
The law was fixed in 1984[0] and then the fix was intentionally reversed in 1999.[1]
> The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software.
Apparently the law was introduced along with speed cameras, as they were continually being challenged in court.
It's a misunderstanding that the infallibility is enshrined in law. I'll quote a post I made in an earlier thread.
1. Historically, mechanical tools are presumed to be working well. This makes things simpler. The example quoted by the Guardian is a good one[0]: if someone wants to question the accuracy of a clock, it's on the person claiming the inaccuracy to prove their point.
2. In 1984, it became clear that computers are not just simple mechanical tools, and they were explicitly excluded from this assumption, by saying that computer evidence should be considered 'hearsay' (and therefore inadmissible) unless the prosecution can prove that the evidence is correct, either by a certificate from someone who can reasonably be expected to certify the correct functioning of that particular evidence, or by oral evidence.
3. This meant that anyone depending on the reliability of evidence from a computer (or piece of software, hardware, etc.) as part of their legal argument could be called upon to prove this, and the burden of proof lay with them (i.e.: as a defendant, I could require the prosecution to prove that the computer works as it is intended).
4. Following a review, it seems to be basically the conclusion that the requirements are inconsistent, unnecessarily onerous and time-consuming, and the way it was written was allowing criminals to get off on technicalities because the prosecution were not able to prove minor or irrelevant points about the functioning of the computer, and anyway other countries don't have any special rules about computers. You can read for yourself the recommendation here: https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.a... (starting page 200 of the document, 215 of the PDF).
5. In 1999, the specific requirement for computer evidence to be treated as hearsay was removed.
The law does not say that computers are infallible. It is still possible to challenge the accuracy of a computer system, but the burden of proof lies with the defence. It's not going to be good enough to say 'well I don't know what happened, it must be a computer glitch', and as a result, cause the prosecution to need to produce evidence that the terminal in the Post Office was working correctly, as well as all of the back end servers that may have been responsible in some part for producing the output.
There's an extent to which I think this is reasonable. If I'm accused of fraud based on evidence recovered from a bank computer, it should not be the case that I can require the prosecution to prove that the bank's computers function correctly from first principles, and the evidence be thrown out in case the prosecution are unable to do so.
The problem with the Horizon convictions is that in many cases, the evidence produced by computers was the only evidence. Also, as the Post Office has its own prosecutors, they could chase and prosecute cases which would not normally have been tried by the CPS due to lack of evidence. It's also clear that the Post Office bullied and threatened not just the sub-postmasters, but also journalists, to keep quiet about the existence of evidence which might throw into question the correct functioning of the system.
This whole debacle is not primarily caused by the principle that you're referring to. The presumption that computers function correctly has undoubtedly saved billions of pounds, hours, and allowed a huge number of successful, correct convictions, which otherwise might have resulted in not guilty verdicts due to clever litigation, rather than actual innocence.
>Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%.
I know I can rest assured the Excel spreadsheet for my monthly and annual budget is perfectly accurate and reliable.
I know the computers powering the stuff my life depends on are perfectly not accurate and reliable.
Put another way: The microwave oven or coffee maker in my kitchen? Yeah, the 'pooters in them are working perfectly. The mainframes jackhammering away at the Automated Clearing House? My money will get through the banking system perfectly eventually some day. The jetliner or my car I'm about to get in? Dude, that thing better have dozens of computers acting in redundancy because that shit ain't working.
I wonder if there's a law stating that the reliability of a computer is inverse to the value of the workload.
> I know I can rest assured the Excel spreadsheet for my monthly and annual budget is perfectly accurate and reliable.
Consider yourself lucky that your use cases are all on the happy path.
There are entire categories of bugs and inconsistencies where Excel's behaviour is known to be wrong, but which can't be fixed because the rest of the ecosystem depends on those same errors to manifest in the same ways.
For example - formulas with cycles have an upper bound as to how many times they are allowed to cycle. If you happen to hit the ceiling before your values converge, you will be left with the values calculated on the last iteration.
But I also know that traditionally, tills and bank accounts are pretty reliable.
Sure, the store might charge you the wrong amount if the price label on the shelf and the PC don't match up, because by law the label on the shelf is the source of truth. But other than that? If the till says my purchases add up to £23.45, and after making the purchase my bank account has a balance of £345.67? I don't validate the arithmetic.
How much money is in my bank account? Pretty much the amount of money the bank's computer says is in my bank account, modulo any funnyness like pending transactions and cheques that fail to clear. The bank doesn't keep a shoebox of cash in their vault with my name on it for us to reconcile against.
I suspect that this originated from the era well before computers were a common part of life and intended to relate to purely mechanical things, and the statutes were just applied to computers when they started to appear in court cases.
So e.g. a mechanical time clock or mechanical scales etc were probably the sort of thing that was the target of the original acts. The assumption is they are working correctly if they appear fine. This makes sense for basic mechanical things, and there is no point arguing that actually the scales that weighed how much the truck weighed were wrong/defective only that one time Defendant X used it and never again afterwards, and it was not in fact due to Defendant X being negligent that the bridge collapsed due to an overweight vehicle etc
As we know, computers are a different level of complexity. Being wrong randomly for one off things is very possible.
Sorry? There have been departments of Weights and Measures for literaly millenia.
They have required certification of those sorts of items with regular inspections and anti-tampering seals.
So yes, you can challenge evidence from mechanical scales, if they haven't been properly inspected and certified at the required intervals under the regulations.
Exactly that is my point - so for the example of some mechanical scales, if they have all been properly inspected and certified, then you can expect them to be reliable and accurate.
So in a case, if there is evidence against someone that relies on the scales doing their job properly, and those scales have been inspected and certified, then you probably do not have a valid argument to say "ah yeah but the scales the defendant used might have not been working properly when they used them!". I.e. the accepted assumption is that the scales work correctly.
Up until recently the same sort of assumption was given to computer systems from what I can tell. This is how we got into the Post Office scandal situation where people implicitly trusted that the computer was doing the right thing.
Notwithstanding the horror that is the Horizon fallout, the legal rule is much narrower than what is bandied around. It merely says you can't just say "computer got it wrong" and expect the other side to prove otherwise. Or in other words, you need evidence of incorrectness if you're going to claim it.
Now with the Horizon scandal, there was very clearly plenty of evidence. The extraordinary number of mismatched books. Some cases of physical records not matching electronic (these were the few guys that got away). The issue was that the Post Office investigators lied about the evidence, buried it with intimidation, legal threats and NDAs.
The law may or may not be bad as it stands but AFAIU this is like blaming a "computer bug" for the Boeing 737 Max crashes. It wasn't - it was human willful errors executed by imperfect code.
> It merely says you can't just say "computer got it wrong" and expect the other side to prove otherwise.
According to the standard legal practice of innocent-until-proven-guilty, you can, in fact, say that and expect the other side to prove otherwise. So this ruling violates one of the most fundamental principles of criminal law.
Right - it unquestionably is a custom rule. But it's not as broad as it is made out, not is it the fundamental problem here. The Post Office knew the system is faulty, hid that evidence and lied in court under oath about it.
As to lack of evidence... It's not so simple. A popular UK insurance hack is when people overtake you on a motorway and slam the brakes, to make a claim on your insurance. Unless you present evidence this was done to you, like dashcam footage, you are presumed to be responsible. You don't need the scammer to prove they didn't crash into you. I'm sure the US has similar mechanisms, where fault is presumed, like if a car hits a pedestrian.
This is similar. Computer systems are presumed to be correct unless evidence is presented. Maybe it's a bad rule, but it's not the horrendous dystopian catastrophe it is declared to be everywhere.
Without accepting this specific rule, that's not how any rules of evidence work in any (common law) court. There needs to be a point where things are presumed true to maintain a working court system. Imagine if e.g. you had to prove the _concept_ of DNA testing every time it was used.
In the US, see for example the debate between the Frye standard vs. the Daubert standard.
I find it really strange to have presumptions in general.
If we're in a court and there hasn't been a decision yet, we're there because we're dealing with some kind of complicated edge case where one person has a strong argument for one thing and another for another.
If one then decides to bring some problem up-- whether with how evidence is being judged or anything else, there can be no justification for ignoring him. This is why I like free evidence evaluation in Swedish courts and the absence of rigid precedent. Every question must then actually be dealt with. We do have this kind of idiotic rulemaking in other parts of our legal system though, so we're not fully free from it.
The problem is allocating resources if you want to deal with every challenge to evidence provided by a computerised system. I think it was initially a problem with people challenging speed camera results.
For a long time I was pushing a campaign that every piece of software paid for with public funds should be made FOSS (unless national security etc).
I struggle to work out why a post office point of sale should be vital to Britains security and we should have been able to see the code.
On top of which I believe that making such code open means there will become a eco-system of ISPs who will be able to support, integrate and improve the software and provide local government users (ie postmasters) with worthwhile consultancy
Under these circumstances it’s hard to see how this would have gone uncovered for so long.
(Or rather, not uncovered, but unbelieved. The great tragedy of this affair is that us was known and reported on for years - but nothing happened. You know those films where the hero manages to get the proof to the newspaper / tv station and the film ends as the bad guys are bundled into police vans - yeah not so much.
Sir, I have this conversation with chatgpt where the assistant tells me that you are guilty. Based on the fact that chatgpt is able to correctly count raspberries we can now legally consider that it is reliable and so that you are guilty! Game over.
It can once you point it out in the training data, hence we cannot know if it understands anything unless we can __prove__ that it wasn't in the training set (which we can't)
There are infinitely many combinations of things. It's not hard to come up with a bunch such that the probability of any one of them being in the training set is infinitesimal.
You generate it in a way that is verifiably random.
If your sufficiently complex system can still "predict" the solution reliably, then you bite the bullet and admit that it does, in fact, understand what it is trying to solve in some way, even if you don't know how exactly it does it.
There's a little ambiguity in the original quote which I think requires a lot more attention with LLMs in the mix: Whether the "rightness" of an answer also depends on the correctness of the process used to reach it.
I'd like to think that Babbage would have had the stricter interpretation.
If there is one thing that should be crystal clear to everyone its that for some classes of work software should be like real engineering, requiring signoff by the senior engineer with personal liability and fines and jail for negligence.
If the engineers dealing with the Citicorp center had dealt with the problem like software engineers, the fix would have been to update documentation in confluence to not expose the building to high winds and that would have been the end of it.
Whilst I agree in principle, would any Senior Engineer want to work on such a system that had their personal liability attached? I'd want 10's of millions in annual comp just for the risk
Yes? Plenty of non-software engineers do it daily.
You start off by having a Professional Liability insurance policy, your company will generally pay for it unless you are a consultant/contractor in which case you bring your own policy. Depending on size of operation, your employer may even indemnify you in the employment contract specifically for even negligence lol.
You then do your job correctly. The laws only go after you for liability if you were negligent, i.e. you skipped protocol and policies, you skipped best practices and couldn't justify it, etc. If you weren't negligent and just made an error, great, your insurance covers you. Insurance can also cover negligence too depending on policy, lol
And on top of all-certified components, it also requires the chief engineer to have veto power over the system. If business or client asks for features that are potentially insecure (and that's going to be a lot of features), they have to accept being told 'no'. I'm not seeing that happen easily in the software industry.
This is chicken-and-egg though, surely. If engineers are not personally liable for the garbage they produce, the business can replace the engineer with someone more amenable to their requirements (either because the engineer lacks experience to understand why its bad, or because the engineer has fewer scruples, or anywhere in between) with relative ease.
If all engineers are held personally liable for their code, when a business has faced a documented rejection, they’ll struggle to hire someone else to take on that risk.
You're right, and it would radically change the industry. No more wild experiments, but a very measured and ponderous rate of change. It's not necessarily a bad thing, I'm at the age where I curse aloud when I see yet another framework doing nearly all the same things but in a different way, but it's certainly a big change.
the german supreme court just recently decided that the involuntary publication of personal data like your phone number through a break-in already constitutes damage that the companies who experienced the break-in are responsible for. there is no need for certified engineers, just make the companies pay for being negligent. putting this on certified engineers only shifts the blame, but in the wrong direction in my opinion, because these problems happen not because the engineers are not certified, but because the company tried to save money by cutting corners.
Good, then maybe it'll be a forcing function for companies to stop collecting personal data for their users just because they can, and we can go back to the days when the only metadata associated with an online account is its salted and hashed password.
I'm not sure what you're arguing for then. Continue apace, where I get a letter every week about entire reams of my personal info leaked because some imbecile didn't set a password on their MongoDB instance?
They can always hire another engineer. They only accept it, some times, because they won't find a certified engineer that says "yes", and because they doing it themselves is a crime.
That equates to a huge amount of government intervention on the lives of everybody. And even then, fails way more often than expected.
Now, we are talking about a case where criminal justice failed to uphold the defendants rights to a fair trial. Most probably because of corruption. Do we really want to bring that huge amount of government intervention into this context?
What OS renders the monitoring screens for air traffic control systems, or railways signalling? Those both have rigorous software engineering behind them — railway signalling is the original of engineered, safety-critical logic systems, starting with mechanical interlocks in 1843. (The signalman physically couldn't move certain levers into bad configurations.)
Well yeah, but the fact that no one takes responsibility for anything and just smears layers of crap on top of each other is the problem. We have build enormous houses of cards on foundations of quicksand and its causing very real harm, but no one cares because the only thing they'll face consequences for is drops in story points on their sprint or whatever else and nothing for failing to do things that actually matter.
Yes, that is exactly part of being a Professional Software Engineer entails, and why there are universities assessed by Enginnering Order, and professional exams.
Lets stop glueing "engineering" to any job title where someone knows how to write a bunch of code lines.
It's not the title that's the problem. It's the part where people and go write software that gets deployed at scale in an environment where bugs can cause very real and significant damage (monetary or otherwise).
The title is part of the problem, because it reveals the culture, slapping cool titles without upping oneself to what those titles actually mean.
As for the rest, anything that brings computing to level of the rest of other professionals, has my signature.
A Software Engineering professor of mine used to say, many applications are akin to buying shoes that randomly explode when tying shoelaces, whereas a minor defect on real shoes gets a full refund.
> The title is part of the problem, because it reveals the culture, slapping cool titles without upping oneself to what those titles actually mean.
The irony is there are actual disciplines in software that are worthy of being called "engineering"--how the hell does an engine ECU work with the level of precision that it does? ABS systems? Hell, how about most electronic control systems on an airplane?
These are some of the most impressive feats in software development, and I've heard near 0 about any of them.
Yet the "industry" is hyper-focused on mashing together "containerized" monstrosities to put strings in databases, or to find a new way to add a chatbot to something that doesn't need it.
I don't know you but i bet that if you and me were locked up in a room together for a month we wouldn't be able to 100% agree on "codified standards for how to do things" :)
Industry isn't mature enough for that and it's perhaps doubtful that it will ever be. See the halting problem.
I don’t think it’s necessary to agree completely. You could start by codifying a minimal set of things that the majority of people agree on (user data sanitisation, authentication handling etc) and then build on it over time.
The standards could also help codify more meta things, like vulnerability policies, reporting and outages. This would be helpful to form a dataset which you can use to properly codify best practices later.
The main problem is that this increases the bar for doing software development, but you can get around this by distinguishing serious software industries from others (software revenue over a certain size, industries like fintech, user data handling etc)
ye gods, can we come up with a "law" to describe appealing to the halting problem?
Just because there are unanswered questions that doesn't mean we can't have bare minimum codified standards.
Furthermore, standards aren't invalidated just because practitioners disagree with them. Plenty of <insert engineer type>s disagree with the standards body of their respective field, they still follow the standards out of fear of prosecution or simply as a path of least resistance and when those standards are found to be defective, they (generally) evolve.
We don't need to solve the halting problem. We just need to come up with a sensible set of practices that, if followed, make the risks small enough to be considered acceptable. Then we can point at that list and say, "this is what the reasonable expectation of due diligence in software engineering is" - and legally enforce that.
I'm fairly sure Fujitsu do have liability insurance in some form.
In a situation like this an insurer is strongly motivated to prove that the company is not at fault, because it doesn't want to pay the bond. The company is also strongly motivated, even though insured, to prove that the company is not at fault, because it doesn't want to have its future insurance rates affected or be sued by the insurer for breaches of terms.
Either way, it doesn't help the people affected. Not unless they have personal insurance against workplace computer system errors, in which case their insurance provider is also not motivated to pay out, or to battle a corporation as large as Fujitsu unless there's chance of a class-action suit.
I call myself a programmer. It's not engineering or maths or anything else. It's managing complexity and I contend that it's very low cost complexity or we wouldn't be able to afford all the software we use.
I think there should be definitely higher standards around things in the industry in generally (especially anything that touches health, money etc).
But the real culpability here are the upper management who said "we don't believe you" (at the most generous interpretation) when the postmasters said that the system was buggy.
> Although many subpostmasters had reported problems with the new software, and Fujitsu was aware that Horizon contained software bugs as early as 1999, the Post Office insisted that Horizon was robust and failed to disclose knowledge of the faults in the system during criminal and civil cases
I don't think it's fair to blame software engineering. Bugs happen, especially where there are no incentives to not have bugs.
First of all justice system is to blame. Sentencing someone on assumption that some complicated software worked correctly is criminally stupid and should result on all people involved being barred from the profession.
Another thing is that software should be treated as just a tool to help to fulfill legal/accounting requirements. If the software is wrong then the required documents are going to be wrong and that's supposedly auditable. This way there are incentives to produce/finance correct software because what is going to be judge and relied upon is not software itself but what it produces (the documents).
Calls to make software engineers responsible will just result in fewer competent people willing to do the work. The justice system is incompetent enough already. Can you imagine lawyers discussing if your off by 1 bug was "criminal negligence" or just a normal mistake that happens? If you going to jail depends on what they decide you will just not take the job and no one sane/competent is going to take it either. The end result is going to be over payed morons writing software and then sometimes going to jail for it - not an improvement over current state of things.
Doctors, Surgeons, Lawyers are able to manage it fine, software engineering could do it as well.
If you could show a constellation of unit and integration tests, well defined schemas and interfaces, for both your code and your dependencies, and a responsible engineering culture then the chances of going to jail are going to be next to nil.
People recoil at the idea only because they see that very very little implemented today would be work that anyone would stand by.
I think the people who paid Von Braun take 99.99% of the responsibility for what happened - they decided to kill people and chose one of many possible ways to do it.
They could just as easily have poured their money into something else. As for von Braun himself, he got employed and looked after for the rest of his life despite being part of the decision (I think) to use slave labour to build his rockets - and that really was a management decision.
I'd argue there are vanishingly few engineers who understand all the consequences of even relatively simple code in terms of security and reliability. Every time there is a security breach at a bank or FAANG, some very smart and experienced engineers with the backing of the business didn't understand something. It's downhill from there for most everyone else.
I'd argue that if there was a demand for software correctness - motivated by, say, legislation applying some baseline standards on such in sensitive applications like money processing - we'd have a lot more smart and experienced engineers focusing on security in particular.
The way things are now is because we as an industry have decided that "move fast and break things" is acceptable, and our culture reflects that. So we need to change the culture.
Then maybe those developers shouldn't be employed. Most of us can't do brain surgery, and yet society trundles on without a million people running around poking frontal lobes. "But I really wanna!" has never been a qualification for any job in the history of jobs.
But it is literally the PM/PO's job to establish and prioritize product features and timelines. If security is a priority, then it should be communicated and prioritized appropriately instead of being always set back in favour of shiny new features, as it often happens.
Some of the 'engineers' responsible, such as Gareth Jenkins of Fujitsu happily left a paper trail showing their perjury and attempts to pervert the course of justice throughout the legal proceedings.
In a just world, he and his co-conspirators would go to jail for what they did, but I don't see it ever happening.
Well, the money guys aren't going to let that kind of outlay dip into their country club membership fee allotment, now are they?
Your proposal is a brilliant and necessary idea, but we don't run the world, my friend. The people that run it only care about money, and brilliant ideas cost serious money, not to mention committment and patience to follow through.
Changing any entrenched status quo is a real slog, for sure. That's why our precious Earth is heating up, daily, to give just one example.
Computers always do exactly as they're told. Unfortunately, there are so many people telling them to do so many things at so many layers of abstraction that the operator can't be sure of anything anymore.
edit: The thing is, had I prefaced it with "ASSUMING A PROPERLY FUNCTIONING COMPUTER," someone would still roll in to pick it apart. You can't please pedants and make a worthwhile point at the same time.
Do circuits get weird? Can a stray cosmic ray flip a bit even in a system with ECC RAM?
Sure.
Does this meaningfully affect a bit about the hazards of abstractions on top of abstractions?
No.
The point is that abstractions amplify computer problems whether they're human error or ghosts in the machine.
I don't think jokes make sense when they are just entirely wrong though. Instead it just comes across as you being ignorant or perpetuating misinformation, rather than a joke
So far only two of about ten people took it in an overly literal way rather than understanding and going with the core of it so I think I'll keep aligning my bits toward them. Have a nice day.
There are, broadly speaking, three genres of Misinformed About Computers:
The deeply interested, but informed by charlatans. I cannot affect their beliefs.
The immune to information.
The person who believes computers are scary, arcane objects. I cannot misinform them more than they already are.
Meanwhile, everyone else understood the point is that abstractions amplify computer problems whether they're human error or ghosts in the machine. Your model of misinformation spread needs work.
To the people trying to be pedantic: Yes, computers really always do what they are told. The only complication is that sometimes they are told what to do by a cosmic ray.
They don’t, computers generate faulty results once in a while, even perfectly functioning computers. And on top of that hardware may degrade generating more errors.
For some background. The legal presumption dates from 1997.
"Previously, section 69 of the Police and Criminal Evidence Act 1984 required anyone introducing computer-generated evidence to show the system was operating correctly and not being used improperly.
The change followed the Law Commission’s 1997 review of the law on hearsay evidence."[1]
- records of known errors and bugs in the system, their effect, and the actions taken in response
- description of information security and other relevant standards and processes followed
- reports of audits performed on the system and how it is managed,
- evidence showing that reports of errors are managed properly and that changes to the system are properly controlled
- evidence confirming that the search for documents was performed adequately, and was done so by a person with appropriate authority and knowledge, and
- assurance that reasonable steps have been taken to establish that the evidence presented has not been tampered with.
Indeed, but of equal stature in my opinion is that they people on the receiving end of the miscarriage of justice are continuing to be abused by the so-called compensation process.
During the inquiry, the statement was made, can't remember by whom, that value for taxpayer money had to be respected.
I think its past that point, and I would like to know why the instigators of the miscarriage aren't at risk of losing their personal assets to (at least partly) cover the cost. Maybe we have to wait for the inquiry report for that, but in the meantime it is pretty clear that 'value for taxpayer money' means, at least to some degree, kicking the can down the road until as many of the claimants are dead as possible.
I have read the proposed amendment, and it still looks woefully insufficient. It changes the rules from "software is presumed to be reliable" to "software previously certified by any government authority is presumed to be reliable."
This applies even if the certification body did not, in fact, do any significant stress-testing that would, e.g., uncover the races, or was not aware of the specific use case that triggers a bug.
In other words: "the Court will presume correct operation of the computer system and will ignore the fact that the other party challenges this; the amendment references certifications coming from a government body at any time in the past as a valid basis to ignore such challenges."
If it's not then every single case of tax evasion people would claim that they had paid and the computer must have lost the payment. Every single traffic camera offense would be contested. Every time you didn't have a train ticket people would claim the computer lost it. And so on.
And there would be months and months of delays if every time it had to be proved beyond a doubt that the equipment was working fully in that case out of all proportion to the risk of there actually being a problem.
However clearly there does need to be a way to challenge the presumption too. Of course just assuming equipment is working is entirely unfair.
After a brief look at the article it looks a reasonable weakening of the presumption which allows the court to consider and reject a challenge when they think fit according to the court rules (to be established) Pretty much like any other form of evidence.
Seems to me that computers do generally perform their tasks reliably. (I say generally, not infallibly; cosmic rays and power failures and such). The issue is really about whether their human programmers have successfully programmed the computer to do the intended task, and about whether their human operators are successfully interpreting computer outputs. So this is surely about what constitutes an expert witness, and how reasonable it is to rely on the testimony of an expert witness, where computers are concerned.
On can really consider how rare hardware issues leading to wrong computing are for your widely used computer chips. In general it seems they are very reliably correct and fixed in reasonably quick order.
Same really cannot be said about software. But on computational level it is actually amazing when you think about it.
When I worked for a drug company everything had to be documented and proceduralised. Because if you ask the FDA if you can sell your pills to the public they will ask how you know they're safe. And if you keep asking enough why's you get down to whether you can trust the output of computers. So every computer's build process and software stack has to be to documented and approved procedure so that you have a wad of paper to show them, when they ask that 'why'.
It seems reasonable that if you're asking to send someone to jail based on the output of a computer you should be able to ask for reasonable proof that the output is correct.
The only thing reliable about them (and the humans that create them) is that they easily fool a bunch of fools.
Of course, it does appear that they are incredibly reliable at being somewhat reliable, but that's not the kind of reliable that I'm looking for, especially when the environmental cost of generating them is so expensive, and doubly so as global heating ramps up.
From the wikipedia page "...Although many subpostmasters had reported problems with the new software, and Fujitsu was aware that Horizon contained software bugs as early as 1999, the Post Office insisted that Horizon was robust and failed to disclose knowledge of the faults in the system during criminal and civil cases......" Fujitsu and Post Office management both new of software faults. Either or both of these groups should have dug much deeper.
The Post Office Management should have known OR hired Software EXPERTS to advise them on how to determine software reliability. Especially once it was evident a large number of people accused of theft. My Opinion is Post Office Management was negligent, particularly the CEO and other senior managers. ANd they should be charged with criminal negligence. CEO and senior managers are paid "big money" to deal with such companies, they should have dug "much deeper" to investigate, especially since their would have been large numbers who claimed innocence
The software company Fujitsu, is also criminal negligence. Producing a financial system and not validating it.
The defence lawyers ( maybe with software computer system speciality ) could have asked for test records, test data how the system was validated. That would have been my first question.
OK , Fujitsu and Post office management claim system is reliable, WHERE IS THE EVIDENCE OF SYSTEM RELIABILITY. Where are the exact test records and test data to support this claim. Where are the system design documents. Where is the company's bug / issue tracking, reports. How exactly is the software tested?
It seems the problem could have been detected at several levels , but for various reasons they all missed. I'm shocked that this issue went on for so long. (There must have good people somewhere , that said nothing, )
My opinion is shaped by my work experience as a software test analysist for many years ( a few decades ) , who has worked on software that does financial transactions, every release ( except ones where change was deemed minor ) of our software was examined by auditors from the government. These government auditors could do anything. ask for design documents, ask for test records. request tests be re run in their presence. That we run certain tests designed by the auditors. Interview any staff about the change. These software audits could be from a few hours to several days , depending on the change. Our company maintained a complete test system of the distributed system for this testing.
This reminds me of Kurt Vonnegut's book Player Piano. Imagine LLMs/AGI but implemented with vacuum tubes. It includes the line (from memory): We're sure of his sentencing, because we replaced all the vacuum tubes and ran the trial again and got the same result."
Next up: What did Birmingham do with the £216 million for the 2020 Oracle Financials upgrade that will not be completed until 2026? For a city that has a 46% child poverty rate.
Sometimes the technology isn't the problem, it's the people.
Anyone else get the sense that these were laws written without any input whatsoever from people who actually have working experience in complex systems?
I'm convinced that no moral software engineer would ever suggest a conviction on the presumption that the system is correct. At minimum you'd have to investigate.
The presumption was about mechanical instruments, from a simpler time. As computers expanded from being purely mechanical, it never got revised, which is why it is being revised now (three or four decades too late, I'll grant you, but legislation is a slow process.)
Except it was revised once already to point out that computers aren't reliable. And then a decade later, at the peak of the dot-com boom, that revision was reverted after a review by committee.
For those not familiar with recent UK scandals, there has been one ongoing for a few years where there was a bug in a system designed by Fujitsu that was calculating the finances for post offices. This resulted in the system quoting wrong balances for post offices, and when that didn't match the actual cash in hand, resulted in prosecutions of the postmasters responsible for those offices which in turn led to convictions, dismissals, fines, imprisonments, and suicides over a 15 year period as the computer system was presumed correct.
Anyway, it was later proven the computer system was incorrect but the government there dragged their heels on exoneration and compensation.
https://en.wikipedia.org/wiki/British_Post_Office_scandal
This reminds me of a story my physics professor told me. Couriers were delivering gold between countries at different longitude. At the origin the gold was weighed using balance scales. The the destination the gold was weighed using spring scales. However, due to differences in gravity and the rotation of the earth, there was a significant difference in the weights. (But not the mass!) This made the authorities think the men had shaved the bars somehow. It was a simple way to demonstrate the difference between weight and mass.
Whether the story was true or not, the government is just bad at science. It's how we get stuff like bite mark patterns and facial recognition warrants.
> where there was a bug in a system designed by Fujitsu
To clarify, there was not _one_ bug, but hundreds.
https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr...
From my read of the information publicly available, there was a fundamental lack of distributed/transactional system understanding from the developers at Fujitsu.
This[1] article recently posted here[2] reminded me of the Post Office Scandal.
The lack of respect for established CS theory (transactions and distributed systems) and established accounting practices (double entry book keeping or even the idea of a ledger) is mind boggling.
[1] https://news.alvaroduran.com/p/engineers-do-not-get-to-make-...
[2] https://news.ycombinator.com/item?id=42269227
IMO it's not really a lack of respect as it is software vendors and managers not really being incentivised to do the right thing. Therefore even if the engineers know the right thing to do, they are unable to solicit the buy in from stakeholders to invest the development into doing the right thing because there's no recourse for doing the wrong thing but it's more expensive.
And even if you do the right thing, if a competitor comes in and offers to do the "job" quicker and cheaper because they're doing the wrong thing. You may have the moral highground but that doesn't pay your employees' salaries.
What we need is a better framework for punishing bad software, in an ideal world without a bunch of red tape and reducing the burden on non experts identifying what is or isn't bad behaviour.
> managers not really being incentivised to do the right thing
It's more likely that managers are just incompetent. Not being able to distinguish between infrastructure and "decoration" means all their decisions are at best superficially motivated.
The punishment we have for this is unemployability and bankruptcy. It will happen by itself if you let it.
A framework for "punishing bad software" sounds to me to be hiding the ambition to "protect bad managers from consequences of their decisions".
> The punishment we have for this is unemployability and bankruptcy. It will happen by itself if you let it.
In my career, I have never seen a manager see consequences for making the decision that values "money now" vs "avoiding things going wrong later". I do not believe that it will happen by itself.
Well yes, agree this is how it looks now. But my perspective is that there has to be layers of mechanisms protecting them, rather than the natural order. (These mechanisms are tricky to pin-point but seem to be on the themes of mutual loyalty, avoiding embarrassment, keeping face, and so on.)
Imposing some formal framework is more likely to insulate managers even further while making software developers accept legal responsibility beyond their pay.
> The punishment we have for this is unemployability and bankruptcy. It will happen by itself if you let it.
yeah, nah.
"No one ever got fired for buying IBM"
after decades of incompetence the multinational contracting firms are still going fine. IBM still runs fine and they're focus is now even more in the insulting side of the business.
> What we need is a better framework for punishing bad software
So like engineering licensing and insurance?
I'm pretty ignorant on the ins and outs and trade offs associated with accreditation but yes that sounds like one solution.
It doesn't strike me as unreasonable, software engineers are now able to cause billions in damages and/or cause loss of life via primary or secondary effects.
Sure, one of software's greatest strengths was that anyone could learn it and it'll be sad to lose that but that's not really a reason for structurally enabling things like the Post Office scandal.
You probably don't need accreditation to serve cat pics but things dealing with money and life/death should. But IANAL so hopefully someone else can bring better insight to this area.
The Post Office scandal at its core was a political/legal problem, not an engineering one.
Software needs to be transparent, a human expert worker needs to be able to verify the result a software generates within reasonable time.
There is no expertise for software that you could delegate to aside from an entity using some form of integration test.
Some of the sites we work on are insurance based. We save every step of the calculation (inputs and outputs, e.g. rates and sub-totals). An administrative user can see the entire calculation from start to end, including overrides at various steps (e.g. a manual discount), and breakdowns per state or item insured if appropriate. This seems like the acceptable bare minimum to me, rather than just showing a magic number. And it definitely helps to expose bugs.
> Software needs to be transparent, a human expert worker needs to be able to verify the result a software generates within reasonable time.
This is becoming more and more important as the AI industry is pushing non-deterministic computing further and further. There will be court cases where the full automated decision chain will be called into question, and a default assumption of "well, we don't have any logs so we should assume the software did the right thing" terrifies me.
The magic word that always pops up to me is "liability."
Even speaking from an IC/non-managerial place, far too many technical problems in businesses are really just symptom-alleviation (or worse, performative look-I'm-doing-something theater) for a deeper problem which involves misaligned incentives for humans.
> it'll be sad to lose that
People will still be able to learn how to program and actually program. But if they take money or deal with people's private information, then they'll need to comply with the standards that will be regulated.
I think we need to start thinking of an individual's persona which includes all the information about them that is online or in government (non secret) files.
That needs to be considered when designing regarding regulations for software that interacts with someone's persona.
So anyone can set up the equivalent of an art stand in the park (serving cat pics), but if they start either selling cat pics or selling advertising that relies on collecting information about the personas on the site, then that needs to be regulated.
> People will still be able to learn how to program and actually program. But if they take money or deal with people's private information, then they'll need to comply with the standards that will be regulated.
You can even have non-licensed people doing work, and a licensed individual signing off on the end result. They would need to review the designs and work to make sure they agree with how the work was done, because it's their license on the line if something goes wrong because of bad craftsmanship.
It doesn't need to be too dramatic. It's already a general rule never to write your own cryptographic or financial code from scratch, you reach out for established libraries or services written by experts.
The law may require to use the ones that are certified if you are doing something sensitive. It doesn't need to be universal necessarily, but it should apply to public tenders for critical infrastructure at least.
You want some government mandating your dependencies...?
Yes. Legally mandated supply requirements have been an element in every other engineering sector for decades or centuries. If you're building an airplane you can't purchase from any random aluminum supplier, there's an authorized list.
That would seem to result in industry consolidation and declining innovation.
Software isn't made of physical materials anyway, it's speech. How about instead we impose open source requirements to enable public verification of critical systems?
Innovation like calculating numbers incorrectly and getting people sent to jail? Or maybe just another hundred slow, rent-seeking React web apps that'll disappear when the series B money dries up, if we're lucky. We must have different ideas of what innovation is.
So, outsource the core government function of regulation to "the public". Who exactly do you imagine is going to actually do this public verification, and where is their paycheck going to come from.
Some software is critical infrastructure and needs to be treated as such. We are not special. Every other engineering discipline has gone through this same process as and arrived at the inevitable conclusion that government regulation is essential, but only after causing unthinkable damage to the public first
Forcing transparency isn't outsourcing anything.
software "engineers" love the cachet but hate the idea that they might also have to uphold the values and responsibilities of engineering.
I say we regulate the word engineer the same way it is in many countries for real engineering. if you don't want to progress beyond code monkey, you can be a software developer and innovate yet another react clone. if you want to be called an engineer, you learn and follow the regulations.
Enabling public verification is not the same as required verification.
This isn't (or doesn't have to be) about laws specifying how all software has to be made.
What they need to specify is the standards for software that certain types of organizations can use. Like government agencies, government contractors, medical organizations, construction and engineering firms, and probably some other kinds of large private businesses, depending on their industry.
Basically, if the software your organization uses can cause the level of destruction that Horizon did, it needs to have specific certifications, or you can't use it.
In order for such software to be certified, it needs to meet certain clearly-defined standards of quality, potentially including having all the technical leads of some level (or just all the developers, depending on various factors) be licensed, and have their licenses on the line of something like this scandal occurs.
It's not a panacea, and it would definitely be an absolute bear to get the terms of all of it defined both clearly and in a way that is likely to actually produce a quality product, but IMO it is likely to be worth it in the long haul.
Transparency and competition seem to work better than certifications and credentials.
Just mandate open source if using public money.
How's it working for those postal workers who committed suicide?
That software wasn't open source, was it? I don't understand how that's supposed to be relevant.
If someone prefers solution B to solution A, bringing up a situation that had neither is not a counterargument.
think of it as accreditation profiles. the highest is for life threatening cases like flight/medical, the lowest might be government expenses.
While I personally love that idea, I suspect it's too complex to fly in actual attempts to get something like this in place.
the same could be said for aerospatial regulations, and we still manage to do them. civil constructions are mindbogglingly complicated and we still manage.
all it takes is enough people to die, and/or for rich people to lose enough money and it'll become the rule.
At least when you are building key infrastructure for the government, directly or indirectly through third parties. I don’t think it’s unreasonable.
And it is not as absolute as you make it sound. Only dependencies for specific critical functions may be regulated. And they don’t have to literally force a whitelist of dependencies on you, just whichever has been certified as appropriate for that purpose.
This is how you get FIPS 140 [1], which for those not in the know is a US Federal standard that mandates encryption which is _less_ secure the the current state of the art and has been for decades. (Yes, there's a new version which was approved 5 years ago and which is still rolling out [2]).
[1]: https://en.wikipedia.org/wiki/FIPS_140
[2]: https://csrc.nist.gov/Projects/fips-140-3-transition-effort
At the same time, turning on FIPs mode is the way we discover that some of our modules were using MD5 in security critical places. Because the government actually enforces FIPS, people (primarily Red Hat I think), now actually put in the bare minimum of engineering effort so that when you set fips=1, the system will actually enforce the policy (unless you go out of your way to override it, or use a non distribution provided crypto stack).
Sure, now that the infrastructure for this has been built, it can be configured to require stronger crypto then FIPS does, but that infrastructure would never have been built without the likes of FIPS, and the government mandating it's use. And I know this because even with all of the hard engineering work done of building that infrastructure, there are no commonly used stronger policies; because the only people who actually care are the ones forced to care by the likes of FIPS.
Our electrical standards might not the safest way of wiring buildings, and not what we would come up with if we wrote the standards today. But they are orders of magnitude safer then what electricians would be doing without the standards.
Who maintains the mandated dependencies? Who performs certification?
What prevents regulatory capture?
I don't understand the confusion, practically everything we interact with in our daily lives is regulated and goes through certification processes, just look at how all that is done.
Food, drugs, healthcare, consumer products, chemicals, cars, planes, trains, buildings, utilities, energy, infrastructure, salaries, loans, investments, accounting... Even media requires some licenses, receives age ratings, and has restrictions on advertising.
It's not rocket science, this is normal for every single other industry.
And that has resulted in massive industrial consolidation and lack of innovation in food, drugs, healthcare, consumer products, chemicals, automotive/aerospace/locomotive manufacturing, construction, utilities, energy, payments, finance, accounting, and media outside "tech," where we manage our own dependencies.
Rocket science is one of the few industries that's actually seeing active innovation.
Famously, civil engineering has never solved this problem, either, leading to the sad state of affairs in which we find ourselves today, wherein no bridges have been built anywhere.
how does it work for aircrafts? how does it work for anti-money laundering policies?
The way it works for aircraft is that now it's getting prohibitively expensive to design new planes. But new planes are still needed, so Boeing found a solution, and made the 737-MAX which is in theory an upgrade that does not require re-certification, but is in fact different enough that the differences lead to hundreds of deaths.
Perhaps without the certifications lots more people would have died. I'm just an armchair analyst. Just food for thought.
Uh, the MAX was a disaster because Boeing is run by MBAs instead of engineers. Regulation is why air travel is remarkably safe. Those rules are written in blood, something everyone here conveniently seems to forget.
So you write regulations for software development, but software companies run by MBAs instead of engineers can still cause disasters. I guess as long as disasters are rare, it's a win.
Is Fujitsu run by engineers?
lots of people did die without the regulations!
I think it needs to be on the individual-liability level. Blessing implementations from corporations is an environment for monopolies to grow and corruption to set in.
Sure, it doesn't need to up-end the whole field, but it is rather obvious that critical infrastructure should require significant compliance, particularly if it's being purchased in a public tender process where the cheapest supplier that fits the criteria is hired. Certifications like SoC 2 are already a big thing.
> > What we need is a better framework for punishing bad software
> So like engineering licensing and insurance?
How is engineering licensing and insurance punishing bad software ?
Microsoft is still going strong and all they do is "checklist security".
I don't work in IT and I'm not a developer, but IMO a developer should do the right thing even if that thing is not written in the specs/contract (transactions and ledger in this case). It is not the job of the developer to care about the competitiveness of the company (Fujitsu in this case), contracts, money, etc.
You get fired for doing good work, in many places out there.
It seems to me that it's a fundamental failure of software engineering culture.
The number one rule for engineering domain applications is to understand the domain.
I would blame the perennial neophilia and lack of (or inadequate adoption of/respect for) standardized texts in the industry. Though, to be fair, a lot of this does come down to the rapid changes in the technology.
Iterative development is necessary for software, of course, but this should be understood as a necessity due to the medium, not as an excuse for skipping research and design. A lot of these domains (especially something as critical as accounting) should be solved problems.
This is not a story of a failure in software engineering. Shit happens, no system is 100% reliable. The failure lies in how management handled it (pretend there’s no issue, don’t launch an internal investigation, blame others).
No doubt management also oversaw the development of the system and rushed it to production.
A healthy culture should accept failure as inevitable and learn from it when it occurs. It should also listen to the people who know best: the engineers who built the thing. You know, like the aerospace industry.
The damage that morons in suits do in pursuit of their bonus cannot be overstated.
I'm not sure I entirely agree in this circumstance.
Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy.
https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr...
Transaction idempotency is such a basic property for a financial system that I struggle to believe that Horizon was tested in any meaningful way.
"the engineers who built the thing" (Gareth Jenkins) are also under investigation for perjury.
[1] Horizon is still in use, in its buggy state, with replacement scheduled for 2030....
>Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy.
The developers were just doing their job. It's management's responsibility to construct a functioning system of checks and balances and understand the limitations of their systems, both of which they failed to do. If it weren't for their hubris the fundamental issues with Horizon could have come to light much earlier.
Let's also not forget that the reason executives are compensated well is for them to take accountability in situations like this.
I think you’re trying to fit this into an exclusive OR when it really is AND. Unless the developers strongly warned management about the lack of idempotency and were ordered to continue anyway, they share the blame for those bugs even if they were not part of the conspiracy to cover them up (this also goes for anyone involved in those hot fixes trying to patch up errors without telling the customer).
More broadly, your idea that this is solely a management problem is how we end in situations where developers are being told to unquestioningly code some design exactly as given, which never works. You don’t get professional judgement if you don’t accept responsibility, too.
It can be more than one thing at a time.
But I would also say that that kind of toxic management is absolutely a part of "software engineering culture". How many horror stories do people on here have of managers who care nothing about the quality of the product, only meeting the deadline so they can get their bonus?
"Software engineering culture" is way, way more than just "how write good code." It includes how we work, how we manage/are managed, how we advocate for ourselves, or fail to do so, and much more.
It certainly includes the very common resistance to unions among programmers, and assuming this was caused by management pushing a known-bad product out the door, a strong union would have (at least potentially) been able to stand up to such demands.
Programmers who get paid asstons of money don't want to admit that these are also fundamentally business failures, partially because if they were fixed, they probably wouldn't get six figures for styling HTML buttons anymore.
There's also the issue where public software engineering related projects are too broad in scope and poorly defined. Rather than making small projects that are useful and then expanding upon them the trend is to specify massive nationwide databases and then just throw money at them, often using 'Big bang' deployments. These are so called because you deploy them and then there's a large explosion ;)
> The failure lies in how management handled it
The thing literally went into a criminal court as evidence, and was "presumed correct" in a way that overloaded any technical or reasonable discordance.
The largest failure here was from the judges and lawyers. The software failure isn't even relevant.
I'm so tired of seeing "no system is 100% reliable" being used to excuse the most fundamental failures in this industry. Nobody's asking for 100% perfection, we're asking for the people writing the software to do X to actually do X correctly in the first place. How about, I don't know, 95% correctness? Is that too onerous?
From what I can see "software engineering" for the most part isn't really "engineering" in the sense that a PE or CEng would recognise?
It is partially on the countries where one cannot just slap engineering on their job title as they feel like.
Where signing off contracts does have some implications beyond "it works on my computer".
Engineering is a professional activity, a legal title is just a moat, it doesn't mean that it is Engineering, just a protected profession of some sort.
In al seriousness, Engineering is about verifying systems to make sure their lifespans and failure modes are known, up front.
This has a legal dimension and a practical one. Legally you can make people liable for unreliable systems. But you can also be liable for failure to maintain properly, or failure to warn about impending calamity. Because it's all documented and verified.
Practical you can live worry free in earthquake and flooding proof buildings, trusting in the diligence of Engineers, and maintenance workers, because they and others have liability imposed on them.
For software this is only the case in a few sectors. For buildings in all cases. Not comparable.
When society depends on software is more than comparable.
Any life can be absurdly destroyed via malware, security exploits, accounting gone wrong, a database deleted in production,....
People also don't put up with faulty products, why should computing be an exception, shitty ship now fix later culture?
And above all, calling oneself "engineer" out of a bootcamp, has nothing to do with Engineering.
Software engineering culture is still in a very primitive phase: just take a look at the comments of the category theory submission currently in the front-page https://news.ycombinator.com/item?id=42291141
Try suggesting to use a tool like TLA+ to validate some complex design and the most likely scenario is that people will laugh at you, even if it's a critical component for the business.
Most decisions in the industry are based on weak anecdotes and unfounded opinions of underserved "authorities".
The many bugs are still there. Fujitsu should be renamed Bugitsu....
"Majority of Subpostmasters still getting unexplained Horizon discrepancies" - https://www.postofficescandal.uk/post/majority-of-subpostmas...
There were "bugs", yes. There were also fundamental design flaws.
One of the things the Post Office did was sell travel money, but the whole system was never really designed for ForEx operations, so it didn't keep track of exchange rates over time. The result is that reconciliation used the exchange rate at time of reconciliation instead of at the time of trade. So, if the foreign currency had gone up in value, it would show up as GBP missing.
As a developer working on trading platforms, this is horrific.
Given the Post Office had a reputation for really good exchange rates, this one design flaw might be responsible for a significant portion of the problem.
So they didn't record the rate in the transaction?!?
Wow, that's... dumb.
IIRC they did record the date, yes. What they _didn't_ store was historical exchange rates.
Even still, exchange rates aren't a fixed value globally at any given time. You should record the rate with the transaction because the rate could be different from your log of exchange rates for any number of different reasons.
Disregard that comment. I misread "rate" in GP's comment as "date". Yeah, they just plain didn't record the rate the trade was made at.
A lot of the problems with Horizon are incredibly dumb. Mistakes that anyone with two brain cells to rub together would recognize.
Buggy software is important aspect of the story but to me it’s not the main one.
The main one is that the post office management/officials at some point became aware of the bugs and that they were ruining lives of innocent people and they knowingly kept lying to save their asses.
The inquiry page has all of this and more: https://www.postofficehorizoninquiry.org.uk/about-inquiry
I’ve read through it, it’s long but it’s a good (while terrifying) read.
I’m sorry I don’t have a summary but the judgment is also enlightening: https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-...
And wiki has rest it the judgments: https://en.m.wikipedia.org/wiki/Bates_%26_Others_v_Post_Offi...
I have a lot of respect for the judge after reading this. Here’s a quote describing Post Office evidence:
Dragging heels is a bit of an understatement.
Both the gov and Fujitsu and the post office absolutely knew it was a bug and intentionally hid the fact while the post masters lives were ruined.
However bad the bug was, the cover up was _much_ worse.
To be fair, the government were lied to by the Post Office people about whether or not there were bugs. They also lied about whether Fujitsu had the ability to amend the ledgers even though Fujitsu had a team working to correct some of the inaccuracies.
Also, mind-blowingly (to me at least) - Horizon is still in use, Fujitsu are still being given hundreds of millions of pounds to support it, and its replacement isn't due to go live until 2030
https://www.computerweekly.com/news/366587174/Fujitsu-set-fo...
If the computer had said one postmaster had been wrong, that's one thing.
The real scandal here is that there were hundreds, and those at the top knew this, but instead doubled down.
If it had only sent one person to jail we would never have know about it.
I am pretty sure more dependable systems send the odd person to jail - there have been many cases where someone was lucky not to be convicted.
The bit that I find incredible is that each one of the victims was told that it was just them and it hadn't happened to anyone else....
A strong recommendation for the excellent "Mr Bates vs The Post Office" to anyone who hasn't followed it, a short (four part) drama about it which is really worth a watch, and a good way to get a rough feel for what went on, and also managed to finally catalyse a real response by the government.
Alan Bates sounds like the epitome of the British "stiff upper lip". Stubbornly standing against overwhelming adversity for years, for the good of his fellow society and his own integrity.
> the computer system was incorrect
There's a little more to it. Most of the comments here are focusing on "correctness". And yes, the amendments to section 69 do something towards tempering its ridiculous and dangerous "presumptions".
But the (UK Post-Office) story is that the Horizon system had back-doors in it. Fujitsu denied this. "Corrections" were made to systems without operator knowledge - to fix actual errors caused by a terrible database sync script full of race-hazards, faulty locks and duplicated state.
The cover-up began life as engineers trying to hide up technical mistakes. It escalated to senior executives trying to cover up financial and political mistakes. It ended with the Crown colluding in covering up judicial mistakes. It is an exemplar of hubris, pride and egotsim resting on a refusal to give up a religious belief in technology. Were it not for the courage of a few (including judges and MPs) they would have gotten away with it (if it weren't for those meddling kids)
The case stands as an important landmark that you cannot "hide behind" technology as a means for abuse and injustice.
The amendments are welcome but insufficient. They open up a good opportunity for cybersecurity people to work with lawyers now.
There are two outstanding problems:
Proprietary code. If you cannot examine the system then the right to challenge it is meaningless. This requires changes to investigatory powers/discovery if anyone wants to use "technical correctness" as a base for argument.
Malicious function. While the discussion revolves around correctness it is incomplete. Many systems (perhaps not the Horizon system) are not faulty, they work perfectly well to deceive, manipulate and swindle.
I'd still push for a complete reversal of presumption [0]. Where software is part of a legal dispute it should "take the stand" as its own witness, in that formal proofs of correctness (a very VERY high bar in software engineering) need to be brought in front of the court. Otherwise the reasonable presumption is that an error or hidden malicious coding "cannot be ruled out".
[0] https://cybershow.uk/episodes.php?id=23 https://cybershow.uk/episodes.php?id=24
I am already consider cynical by many, but reading about the scandal constantly reminds me how naive I am. Just how low can government and politicians get.
I actively avoided the news and outrage about this for my own sanity. It was just too shocking. The worst part is that I'd heard about it and got mad about it many years ago, and it wasn't until a TV documentary came out recently that anything happened.
Echoing this, I've been passively following the inquiry but haven't watched the miniseries.
I think it would leave me furious for weeks and push me into a depression rut.
I'm trying to take the whole debacle as long term inspiration to be excellent at what I do - it's tough to stay positive when so many people involved (Fujitsu, PO upper management, the original prosecution) seem entirely morally bereft, with little chance of consequence.
I guess some people never learn from trying to cover their mistakes as children. I've explained this to my kids: often the initial mistake is minor and warrants nothing more than "please don't do that again". It is the lying and hiding that blow it up into a big issue. The odds of being found out are very high. There is little reward and lots of risk.
When you have such a widespread and ongoing problem it becomes clear that such a large proportion of postmasters can't be criminals. Computer bugs are well known. Eventually it is going to come out that the computer is wrong. Why double-down? The earlier you admit a mistake and apologize the lower the impact and the less anyone cares. They had a built-in scapegoat that everyone understands and accepts: the vendor's software had bugs! We have daily meetings to yell at them to fix the bugs we promise and we will fix the problems ASAP.
Or do the sneaky thing and fix the issues, stop prosecuting postmasters, and ignore the ones you prosecuted by mistake. Cynical, cruel, and immoral... but contains the damage.
Instead the UK Postoffice seems to have just let the problems continue while simultaneously allowing prosecutions to go ahead knowing they were faulty. Literally the worst of all worlds: ongoing accumulation of liability, now with provable malice!
Were there actual bad consequences for the upper management and politicians involved? I think the risk assessment you teach your children does not apply to british politicians.
I heard some convictions of innocent people have been overturned, or are in the process of. But new convictions of the actually guilty... no news about any such thing. Maybe I'm just uninformed.
> Just how low can government and politicians get.
As low as it takes for a profit!
Upvoted you because last I checked, Fujitsu is a private entity being run for a profit.
Also for those not familiar: the government owns the Post Office, despite what many try to suggest to the contrary
Yes there is operational separation but it's not like they're wholly unrelated and the government totally guilt free
It doesn’t make sense to change an entire system over once incident. That smells of overreaction to me. It was an awful situation caused by humans, and the humans involved should probably be punished, that doesn’t mean one should break a system that generally works very well and promotes self regulation and continuous improvement rather than shackling it with regulations.
The fact that this misnomer of infallible computer systems was ever enshrined in law is pretty damning of the whole UK legal system and the relationship between technical people and law.
Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%.
Not least because it seems to go against the core concept of "innocent until proven guilty" that the whole legal system is meant to rest upon.
> The fact that this misnomer of infallible computer systems was ever enshrined in law
Is this actually a fact, or a fact taken to it’s logical conclusion to presume a new “fact”?
The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software. Instead, precedents got set over time by enterprising lawyers, but setting a precedent when it’s convenient is not the same thing as writing a law.
When I see mechanical systems, I think of something like an abacus. I’ve never used one, but I suspect the abacus itself is infallible. It’s open, it’s transparent, it is easily auditable, and the same inputs will always produce the same outputs. There is no black box translation occurring, like occurs with computer software.
>The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software.
It's worse than that.
The law was fixed in 1984[0] and then the fix was intentionally reversed in 1999.[1]
[0] https://www.legislation.gov.uk/ukpga/1984/60/section/69/1991...
[1] https://www.legislation.gov.uk/ukpga/1999/23/section/60
Yes, because basically it was resulting in clever lawyers being able to get any and all computer evidence against their clients rejected as hearsay.
The change in 1984 wasn't a 'fix', it threw the baby out with the bathwater.
> The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software.
Apparently the law was introduced along with speed cameras, as they were continually being challenged in court.
I think historically the law was introduced to make it harder for people to contest things like speed camera based speeding tickets.
> setting a precedent when it’s convenient is not the same thing as writing a law
In the UK, they're pretty much the same thing. You need a new case or a statute law to overturn a precedent.
It's a misunderstanding that the infallibility is enshrined in law. I'll quote a post I made in an earlier thread.
1. Historically, mechanical tools are presumed to be working well. This makes things simpler. The example quoted by the Guardian is a good one[0]: if someone wants to question the accuracy of a clock, it's on the person claiming the inaccuracy to prove their point.
2. In 1984, it became clear that computers are not just simple mechanical tools, and they were explicitly excluded from this assumption, by saying that computer evidence should be considered 'hearsay' (and therefore inadmissible) unless the prosecution can prove that the evidence is correct, either by a certificate from someone who can reasonably be expected to certify the correct functioning of that particular evidence, or by oral evidence.
3. This meant that anyone depending on the reliability of evidence from a computer (or piece of software, hardware, etc.) as part of their legal argument could be called upon to prove this, and the burden of proof lay with them (i.e.: as a defendant, I could require the prosecution to prove that the computer works as it is intended).
4. Following a review, it seems to be basically the conclusion that the requirements are inconsistent, unnecessarily onerous and time-consuming, and the way it was written was allowing criminals to get off on technicalities because the prosecution were not able to prove minor or irrelevant points about the functioning of the computer, and anyway other countries don't have any special rules about computers. You can read for yourself the recommendation here: https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.a... (starting page 200 of the document, 215 of the PDF).
5. In 1999, the specific requirement for computer evidence to be treated as hearsay was removed.
The law does not say that computers are infallible. It is still possible to challenge the accuracy of a computer system, but the burden of proof lies with the defence. It's not going to be good enough to say 'well I don't know what happened, it must be a computer glitch', and as a result, cause the prosecution to need to produce evidence that the terminal in the Post Office was working correctly, as well as all of the back end servers that may have been responsible in some part for producing the output.
There's an extent to which I think this is reasonable. If I'm accused of fraud based on evidence recovered from a bank computer, it should not be the case that I can require the prosecution to prove that the bank's computers function correctly from first principles, and the evidence be thrown out in case the prosecution are unable to do so.
The problem with the Horizon convictions is that in many cases, the evidence produced by computers was the only evidence. Also, as the Post Office has its own prosecutors, they could chase and prosecute cases which would not normally have been tried by the CPS due to lack of evidence. It's also clear that the Post Office bullied and threatened not just the sub-postmasters, but also journalists, to keep quiet about the existence of evidence which might throw into question the correct functioning of the system.
This whole debacle is not primarily caused by the principle that you're referring to. The presumption that computers function correctly has undoubtedly saved billions of pounds, hours, and allowed a huge number of successful, correct convictions, which otherwise might have resulted in not guilty verdicts due to clever litigation, rather than actual innocence.
[0]: https://www.theguardian.com/uk-news/2024/jan/12/update-law-o...
>Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%.
I know I can rest assured the Excel spreadsheet for my monthly and annual budget is perfectly accurate and reliable.
I know the computers powering the stuff my life depends on are perfectly not accurate and reliable.
Put another way: The microwave oven or coffee maker in my kitchen? Yeah, the 'pooters in them are working perfectly. The mainframes jackhammering away at the Automated Clearing House? My money will get through the banking system perfectly eventually some day. The jetliner or my car I'm about to get in? Dude, that thing better have dozens of computers acting in redundancy because that shit ain't working.
I wonder if there's a law stating that the reliability of a computer is inverse to the value of the workload.
> I know I can rest assured the Excel spreadsheet for my monthly and annual budget is perfectly accurate and reliable.
Consider yourself lucky that your use cases are all on the happy path.
There are entire categories of bugs and inconsistencies where Excel's behaviour is known to be wrong, but which can't be fixed because the rest of the ecosystem depends on those same errors to manifest in the same ways.
For example - formulas with cycles have an upper bound as to how many times they are allowed to cycle. If you happen to hit the ceiling before your values converge, you will be left with the values calculated on the last iteration.
I know that computers aren't infallible.
But I also know that traditionally, tills and bank accounts are pretty reliable.
Sure, the store might charge you the wrong amount if the price label on the shelf and the PC don't match up, because by law the label on the shelf is the source of truth. But other than that? If the till says my purchases add up to £23.45, and after making the purchase my bank account has a balance of £345.67? I don't validate the arithmetic.
How much money is in my bank account? Pretty much the amount of money the bank's computer says is in my bank account, modulo any funnyness like pending transactions and cheques that fail to clear. The bank doesn't keep a shoebox of cash in their vault with my name on it for us to reconcile against.
Maybe it was a presumption that was legally convenient... until it wasn't and reality struck?
Yeah exactly, until the miscarriage of justice was too big to ignore
This is particularly true of accounting systems. Anyone who has worked in a similar environment with thousands of users on a distributed ledger.
I suspect that this originated from the era well before computers were a common part of life and intended to relate to purely mechanical things, and the statutes were just applied to computers when they started to appear in court cases.
So e.g. a mechanical time clock or mechanical scales etc were probably the sort of thing that was the target of the original acts. The assumption is they are working correctly if they appear fine. This makes sense for basic mechanical things, and there is no point arguing that actually the scales that weighed how much the truck weighed were wrong/defective only that one time Defendant X used it and never again afterwards, and it was not in fact due to Defendant X being negligent that the bridge collapsed due to an overweight vehicle etc
As we know, computers are a different level of complexity. Being wrong randomly for one off things is very possible.
Sorry? There have been departments of Weights and Measures for literaly millenia.
They have required certification of those sorts of items with regular inspections and anti-tampering seals.
So yes, you can challenge evidence from mechanical scales, if they haven't been properly inspected and certified at the required intervals under the regulations.
Exactly that is my point - so for the example of some mechanical scales, if they have all been properly inspected and certified, then you can expect them to be reliable and accurate.
So in a case, if there is evidence against someone that relies on the scales doing their job properly, and those scales have been inspected and certified, then you probably do not have a valid argument to say "ah yeah but the scales the defendant used might have not been working properly when they used them!". I.e. the accepted assumption is that the scales work correctly.
Up until recently the same sort of assumption was given to computer systems from what I can tell. This is how we got into the Post Office scandal situation where people implicitly trusted that the computer was doing the right thing.
The problem was that the same sort of assumption was made without any basis in reality.
If there's no "inspected and certified" then there is no officially recognized evidence that the system is accurate.
I think that take is slightly missing the point.
Notwithstanding the horror that is the Horizon fallout, the legal rule is much narrower than what is bandied around. It merely says you can't just say "computer got it wrong" and expect the other side to prove otherwise. Or in other words, you need evidence of incorrectness if you're going to claim it.
Now with the Horizon scandal, there was very clearly plenty of evidence. The extraordinary number of mismatched books. Some cases of physical records not matching electronic (these were the few guys that got away). The issue was that the Post Office investigators lied about the evidence, buried it with intimidation, legal threats and NDAs.
The law may or may not be bad as it stands but AFAIU this is like blaming a "computer bug" for the Boeing 737 Max crashes. It wasn't - it was human willful errors executed by imperfect code.
> It merely says you can't just say "computer got it wrong" and expect the other side to prove otherwise.
According to the standard legal practice of innocent-until-proven-guilty, you can, in fact, say that and expect the other side to prove otherwise. So this ruling violates one of the most fundamental principles of criminal law.
Right - it unquestionably is a custom rule. But it's not as broad as it is made out, not is it the fundamental problem here. The Post Office knew the system is faulty, hid that evidence and lied in court under oath about it.
As to lack of evidence... It's not so simple. A popular UK insurance hack is when people overtake you on a motorway and slam the brakes, to make a claim on your insurance. Unless you present evidence this was done to you, like dashcam footage, you are presumed to be responsible. You don't need the scammer to prove they didn't crash into you. I'm sure the US has similar mechanisms, where fault is presumed, like if a car hits a pedestrian.
This is similar. Computer systems are presumed to be correct unless evidence is presented. Maybe it's a bad rule, but it's not the horrendous dystopian catastrophe it is declared to be everywhere.
Without accepting this specific rule, that's not how any rules of evidence work in any (common law) court. There needs to be a point where things are presumed true to maintain a working court system. Imagine if e.g. you had to prove the _concept_ of DNA testing every time it was used.
In the US, see for example the debate between the Frye standard vs. the Daubert standard.
I find it really strange to have presumptions in general.
If we're in a court and there hasn't been a decision yet, we're there because we're dealing with some kind of complicated edge case where one person has a strong argument for one thing and another for another.
If one then decides to bring some problem up-- whether with how evidence is being judged or anything else, there can be no justification for ignoring him. This is why I like free evidence evaluation in Swedish courts and the absence of rigid precedent. Every question must then actually be dealt with. We do have this kind of idiotic rulemaking in other parts of our legal system though, so we're not fully free from it.
I imagine it stems from something like: the clock was correct at 1pm and at 3pm but was it correct at 2pm?
The problem is allocating resources if you want to deal with every challenge to evidence provided by a computerised system. I think it was initially a problem with people challenging speed camera results.
For a long time I was pushing a campaign that every piece of software paid for with public funds should be made FOSS (unless national security etc).
I struggle to work out why a post office point of sale should be vital to Britains security and we should have been able to see the code.
On top of which I believe that making such code open means there will become a eco-system of ISPs who will be able to support, integrate and improve the software and provide local government users (ie postmasters) with worthwhile consultancy Under these circumstances it’s hard to see how this would have gone uncovered for so long.
(Or rather, not uncovered, but unbelieved. The great tragedy of this affair is that us was known and reported on for years - but nothing happened. You know those films where the hero manages to get the proof to the newspaper / tv station and the film ends as the bad guys are bundled into police vans - yeah not so much.
That will be wonderful with AI:
Can count raspberries, but not how many times the letter "r" appears in raspberries.
It will once it's trained on all the posts people made talking about the raspberry problem, and then the illusion of progress will be renewed. :p
It can once you point it out in the training data, hence we cannot know if it understands anything unless we can __prove__ that it wasn't in the training set (which we can't)
There are infinitely many combinations of things. It's not hard to come up with a bunch such that the probability of any one of them being in the training set is infinitesimal.
okay, but how can you be sure that what you believe to be random is not predictable by a sufficiently complex system?
You generate it in a way that is verifiably random.
If your sufficiently complex system can still "predict" the solution reliably, then you bite the bullet and admit that it does, in fact, understand what it is trying to solve in some way, even if you don't know how exactly it does it.
"Pray, Mr Babbage, if I ask an LLM a question that isn't in its training set, will it give the right answer?"
There's a little ambiguity in the original quote which I think requires a lot more attention with LLMs in the mix: Whether the "rightness" of an answer also depends on the correctness of the process used to reach it.
I'd like to think that Babbage would have had the stricter interpretation.
Reminds me of the stories going around of teachers asking ChatGPT if a student's essay was written by ChatGPT or not.
If there is one thing that should be crystal clear to everyone its that for some classes of work software should be like real engineering, requiring signoff by the senior engineer with personal liability and fines and jail for negligence.
If the engineers dealing with the Citicorp center had dealt with the problem like software engineers, the fix would have been to update documentation in confluence to not expose the building to high winds and that would have been the end of it.
Whilst I agree in principle, would any Senior Engineer want to work on such a system that had their personal liability attached? I'd want 10's of millions in annual comp just for the risk
Yes? Plenty of non-software engineers do it daily.
You start off by having a Professional Liability insurance policy, your company will generally pay for it unless you are a consultant/contractor in which case you bring your own policy. Depending on size of operation, your employer may even indemnify you in the employment contract specifically for even negligence lol.
You then do your job correctly. The laws only go after you for liability if you were negligent, i.e. you skipped protocol and policies, you skipped best practices and couldn't justify it, etc. If you weren't negligent and just made an error, great, your insurance covers you. Insurance can also cover negligence too depending on policy, lol
https://www.nspe.org/resources/professional-liability/liabil...
Yes and they do non-software engineering
Are we going to have international protocols and policies on the best language to use, how to do SQL queries and CSS? no
Professionally qualified engineers in other fields seem to manage OK?
And on top of all-certified components, it also requires the chief engineer to have veto power over the system. If business or client asks for features that are potentially insecure (and that's going to be a lot of features), they have to accept being told 'no'. I'm not seeing that happen easily in the software industry.
This is chicken-and-egg though, surely. If engineers are not personally liable for the garbage they produce, the business can replace the engineer with someone more amenable to their requirements (either because the engineer lacks experience to understand why its bad, or because the engineer has fewer scruples, or anywhere in between) with relative ease.
If all engineers are held personally liable for their code, when a business has faced a documented rejection, they’ll struggle to hire someone else to take on that risk.
You're right, and it would radically change the industry. No more wild experiments, but a very measured and ponderous rate of change. It's not necessarily a bad thing, I'm at the age where I curse aloud when I see yet another framework doing nearly all the same things but in a different way, but it's certainly a big change.
You'd still be allowed "wild experiments" - just not where they could harm people.
"Harming people" includes money loss, though. Any hacking that results in identity theft would end up at the feet of the certifying engineer.
the german supreme court just recently decided that the involuntary publication of personal data like your phone number through a break-in already constitutes damage that the companies who experienced the break-in are responsible for. there is no need for certified engineers, just make the companies pay for being negligent. putting this on certified engineers only shifts the blame, but in the wrong direction in my opinion, because these problems happen not because the engineers are not certified, but because the company tried to save money by cutting corners.
Good, then maybe it'll be a forcing function for companies to stop collecting personal data for their users just because they can, and we can go back to the days when the only metadata associated with an online account is its salted and hashed password.
You mean a hospital next door to an ammonium nitrate godown and 23andMe will be rejected by the same actuaries?
I'm not sure what you're arguing for then. Continue apace, where I get a letter every week about entire reams of my personal info leaked because some imbecile didn't set a password on their MongoDB instance?
> they have to accept being told 'no'
They can always hire another engineer. They only accept it, some times, because they won't find a certified engineer that says "yes", and because they doing it themselves is a crime.
That equates to a huge amount of government intervention on the lives of everybody. And even then, fails way more often than expected.
Now, we are talking about a case where criminal justice failed to uphold the defendants rights to a fair trial. Most probably because of corruption. Do we really want to bring that huge amount of government intervention into this context?
They work with certified components on predictable systems though.
You realize that you'd need someone at MS to take liability for Windows before you can sign off anything running on Windows?
Or someone at Google if you do a web app that only runs on Chrome, not to mention other browsers.
Some systems must have these certifications.
What OS renders the monitoring screens for air traffic control systems, or railways signalling? Those both have rigorous software engineering behind them — railway signalling is the original of engineered, safety-critical logic systems, starting with mechanical interlocks in 1843. (The signalman physically couldn't move certain levers into bad configurations.)
Well yeah, but the fact that no one takes responsibility for anything and just smears layers of crap on top of each other is the problem. We have build enormous houses of cards on foundations of quicksand and its causing very real harm, but no one cares because the only thing they'll face consequences for is drops in story points on their sprint or whatever else and nothing for failing to do things that actually matter.
Yes, that is exactly part of being a Professional Software Engineer entails, and why there are universities assessed by Enginnering Order, and professional exams.
Lets stop glueing "engineering" to any job title where someone knows how to write a bunch of code lines.
The important bit is that if you screw up badly enough then your professional qualification is removed and you can't do that type of work any more.
[NB I am frequently reminded of this point by my wife who is a solicitor].
I am aware of that, because I happen to be from a country with an Engineering Order.
Another important part is that one might be liable when signing contracts as the responsible Engineer in a project delivery.
Don't understand how this isn't the dominant perspective on this.
This title inflation of calling web programmers "engineers" is absurd.
The real problem is who could afford a system that a competent senior engineer would take personal liability for...
The real problem is our entire industry is a giant clusterfuck. Do a 2-week bootcamp, congratulations now you're a software engineer.
Every other discipline has education requirements, codified standards for how to do things etc.
Thankfully not every country out there has such liberties with "enginnering" titles, but yeah that is a problem.
It's not the title that's the problem. It's the part where people and go write software that gets deployed at scale in an environment where bugs can cause very real and significant damage (monetary or otherwise).
The title is part of the problem, because it reveals the culture, slapping cool titles without upping oneself to what those titles actually mean.
As for the rest, anything that brings computing to level of the rest of other professionals, has my signature.
A Software Engineering professor of mine used to say, many applications are akin to buying shoes that randomly explode when tying shoelaces, whereas a minor defect on real shoes gets a full refund.
> The title is part of the problem, because it reveals the culture, slapping cool titles without upping oneself to what those titles actually mean.
The irony is there are actual disciplines in software that are worthy of being called "engineering"--how the hell does an engine ECU work with the level of precision that it does? ABS systems? Hell, how about most electronic control systems on an airplane?
These are some of the most impressive feats in software development, and I've heard near 0 about any of them.
Yet the "industry" is hyper-focused on mashing together "containerized" monstrosities to put strings in databases, or to find a new way to add a chatbot to something that doesn't need it.
I'd argue that really says more about capitalism, wealth distribution, and the financialization of everything, as opposed to software stuff per se--
In another area it might be the complete insufficiency of formal botany credentials among Dutch companies growing and trading tulips.
> codified standards for how to do things
I don't know you but i bet that if you and me were locked up in a room together for a month we wouldn't be able to 100% agree on "codified standards for how to do things" :)
Industry isn't mature enough for that and it's perhaps doubtful that it will ever be. See the halting problem.
> 100% agree
I don’t think it’s necessary to agree completely. You could start by codifying a minimal set of things that the majority of people agree on (user data sanitisation, authentication handling etc) and then build on it over time.
The standards could also help codify more meta things, like vulnerability policies, reporting and outages. This would be helpful to form a dataset which you can use to properly codify best practices later.
The main problem is that this increases the bar for doing software development, but you can get around this by distinguishing serious software industries from others (software revenue over a certain size, industries like fintech, user data handling etc)
ye gods, can we come up with a "law" to describe appealing to the halting problem?
Just because there are unanswered questions that doesn't mean we can't have bare minimum codified standards.
Furthermore, standards aren't invalidated just because practitioners disagree with them. Plenty of <insert engineer type>s disagree with the standards body of their respective field, they still follow the standards out of fear of prosecution or simply as a path of least resistance and when those standards are found to be defective, they (generally) evolve.
> we can't have bare minimum codified standards
So, functional, imperative or OOP? :)
> Just because there are unanswered questions
The halting problem is undecidable. Not undecided. I.e. it has been solved and the answer is "you can't".
This is the most pedantic sort of semantic navel-gazing that can only originate in the bowels of an HN thread. Bravissimo, truly.
Yep, now answer me the part about functional, imperative or oop, and come up with a plan to convince everyone.
We don't need to solve the halting problem. We just need to come up with a sensible set of practices that, if followed, make the risks small enough to be considered acceptable. Then we can point at that list and say, "this is what the reasonable expectation of due diligence in software engineering is" - and legally enforce that.
The industry will be 80 years old soon. If it's not "mature enough", it's only because of the pervading culture.
For comparison, electrical engineers started introducing things like national standards for plugs by 1915.
Plugs? You mean various forms of rpc and more or less well specified data interchange formats like xml or json? We have them.
XML and JSON is more like standardizing the voltage (and even there the fact that we still have both shows that it's very much a moving target).
Professional liability insurance and legal insurance as a minimum.
I'm fairly sure Fujitsu do have liability insurance in some form.
In a situation like this an insurer is strongly motivated to prove that the company is not at fault, because it doesn't want to pay the bond. The company is also strongly motivated, even though insured, to prove that the company is not at fault, because it doesn't want to have its future insurance rates affected or be sued by the insurer for breaches of terms.
Either way, it doesn't help the people affected. Not unless they have personal insurance against workplace computer system errors, in which case their insurance provider is also not motivated to pay out, or to battle a corporation as large as Fujitsu unless there's chance of a class-action suit.
As someone who used to work in electrical engineering but now does software, I almost refuse to use the word engineer for what I now do.
Yes, indeed. We are still in the "craft" stage of the process of software development. Engineering is altogether something else than what we're doing.
Strangely enough, 30ya, my friend getting his EE Masters was mostly taking programming courses.
I call myself a programmer. It's not engineering or maths or anything else. It's managing complexity and I contend that it's very low cost complexity or we wouldn't be able to afford all the software we use.
I think there should be definitely higher standards around things in the industry in generally (especially anything that touches health, money etc).
But the real culpability here are the upper management who said "we don't believe you" (at the most generous interpretation) when the postmasters said that the system was buggy.
> Although many subpostmasters had reported problems with the new software, and Fujitsu was aware that Horizon contained software bugs as early as 1999, the Post Office insisted that Horizon was robust and failed to disclose knowledge of the faults in the system during criminal and civil cases
I don't think it's fair to blame software engineering. Bugs happen, especially where there are no incentives to not have bugs. First of all justice system is to blame. Sentencing someone on assumption that some complicated software worked correctly is criminally stupid and should result on all people involved being barred from the profession.
Another thing is that software should be treated as just a tool to help to fulfill legal/accounting requirements. If the software is wrong then the required documents are going to be wrong and that's supposedly auditable. This way there are incentives to produce/finance correct software because what is going to be judge and relied upon is not software itself but what it produces (the documents).
Calls to make software engineers responsible will just result in fewer competent people willing to do the work. The justice system is incompetent enough already. Can you imagine lawyers discussing if your off by 1 bug was "criminal negligence" or just a normal mistake that happens? If you going to jail depends on what they decide you will just not take the job and no one sane/competent is going to take it either. The end result is going to be over payed morons writing software and then sometimes going to jail for it - not an improvement over current state of things.
Doctors, Surgeons, Lawyers are able to manage it fine, software engineering could do it as well.
If you could show a constellation of unit and integration tests, well defined schemas and interfaces, for both your code and your dependencies, and a responsible engineering culture then the chances of going to jail are going to be next to nil.
People recoil at the idea only because they see that very very little implemented today would be work that anyone would stand by.
Good practises still result in bugs. How is the bug handled after it is found? That is a management responsibility.
"Once the rockets are up, who cares where they come down? Thats not my department." says Wernher Von Braun...
I think the people who paid Von Braun take 99.99% of the responsibility for what happened - they decided to kill people and chose one of many possible ways to do it.
They could just as easily have poured their money into something else. As for von Braun himself, he got employed and looked after for the rest of his life despite being part of the decision (I think) to use slave labour to build his rockets - and that really was a management decision.
Why the engineers? Why not putting the liability on the product owner, or the project manager?
Because we cannot have that! Absolutely not. PMs need to be able to apply excruciating pressure.
Also we couldn't possibly put the burden on the company that makes improbably low bids - no! No we must put the blame on the peons where it belongs.
Because its a technical guarantee. How is a product manager going to able to personally approve code he can't even read?
I'd argue there are vanishingly few engineers who understand all the consequences of even relatively simple code in terms of security and reliability. Every time there is a security breach at a bank or FAANG, some very smart and experienced engineers with the backing of the business didn't understand something. It's downhill from there for most everyone else.
I'd argue that if there was a demand for software correctness - motivated by, say, legislation applying some baseline standards on such in sensitive applications like money processing - we'd have a lot more smart and experienced engineers focusing on security in particular.
The way things are now is because we as an industry have decided that "move fast and break things" is acceptable, and our culture reflects that. So we need to change the culture.
Then maybe those developers shouldn't be employed. Most of us can't do brain surgery, and yet society trundles on without a million people running around poking frontal lobes. "But I really wanna!" has never been a qualification for any job in the history of jobs.
But it is literally the PM/PO's job to establish and prioritize product features and timelines. If security is a priority, then it should be communicated and prioritized appropriately instead of being always set back in favour of shiny new features, as it often happens.
Some of the 'engineers' responsible, such as Gareth Jenkins of Fujitsu happily left a paper trail showing their perjury and attempts to pervert the course of justice throughout the legal proceedings.
In a just world, he and his co-conspirators would go to jail for what they did, but I don't see it ever happening.
Well, the money guys aren't going to let that kind of outlay dip into their country club membership fee allotment, now are they?
Your proposal is a brilliant and necessary idea, but we don't run the world, my friend. The people that run it only care about money, and brilliant ideas cost serious money, not to mention committment and patience to follow through.
Changing any entrenched status quo is a real slog, for sure. That's why our precious Earth is heating up, daily, to give just one example.
> personal liability and fines and jail for negligence
Politicians too. And journalists.
Computers always do exactly as they're told. Unfortunately, there are so many people telling them to do so many things at so many layers of abstraction that the operator can't be sure of anything anymore.
edit: The thing is, had I prefaced it with "ASSUMING A PROPERLY FUNCTIONING COMPUTER," someone would still roll in to pick it apart. You can't please pedants and make a worthwhile point at the same time.
Do circuits get weird? Can a stray cosmic ray flip a bit even in a system with ECC RAM?
Sure.
Does this meaningfully affect a bit about the hazards of abstractions on top of abstractions?
No.
The point is that abstractions amplify computer problems whether they're human error or ghosts in the machine.
They don't - random bit flips are expected over time. ECC memory is a thing because of this, but cosmic rays can strike the processors too.
I know
I was doing a bit
I don't think jokes make sense when they are just entirely wrong though. Instead it just comes across as you being ignorant or perpetuating misinformation, rather than a joke
So far only two of about ten people took it in an overly literal way rather than understanding and going with the core of it so I think I'll keep aligning my bits toward them. Have a nice day.
You’ll align your bits towards falsity and ineptitude?
It is a big deal that computers don’t always do what they’re told in this case. STOP perpetuating lies
There are, broadly speaking, three genres of Misinformed About Computers:
The deeply interested, but informed by charlatans. I cannot affect their beliefs.
The immune to information.
The person who believes computers are scary, arcane objects. I cannot misinform them more than they already are.
Meanwhile, everyone else understood the point is that abstractions amplify computer problems whether they're human error or ghosts in the machine. Your model of misinformation spread needs work.
[flagged]
To the people trying to be pedantic: Yes, computers really always do what they are told. The only complication is that sometimes they are told what to do by a cosmic ray.
They don’t, computers generate faulty results once in a while, even perfectly functioning computers. And on top of that hardware may degrade generating more errors.
Okay
For some background. The legal presumption dates from 1997.
"Previously, section 69 of the Police and Criminal Evidence Act 1984 required anyone introducing computer-generated evidence to show the system was operating correctly and not being used improperly.
The change followed the Law Commission’s 1997 review of the law on hearsay evidence."[1]
from https://www.lawgazette.co.uk/law/it-experts-call-for-review-...
[1] review https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.a... (page 197)
---
Another interesting briefing note:
"The legal rule that computers are presumed to be operating correctly – unforeseen and unjust consequences": https://www.benthamsgaze.org/2022/06/30/the-legal-rule-that-...
They suggest, when requested, that a party gets:
- records of known errors and bugs in the system, their effect, and the actions taken in response
- description of information security and other relevant standards and processes followed
- reports of audits performed on the system and how it is managed,
- evidence showing that reports of errors are managed properly and that changes to the system are properly controlled
- evidence confirming that the search for documents was performed adequately, and was done so by a person with appropriate authority and knowledge, and
- assurance that reasonable steps have been taken to establish that the evidence presented has not been tampered with.
This scandal is probably the biggest miscarriage of justice in Europe. The fundamental issue was perhaps "reliability of computers".
But the biggest scandal is that a lot of people knew what was happening and were either silenced or told others to be silent.
Indeed, but of equal stature in my opinion is that they people on the receiving end of the miscarriage of justice are continuing to be abused by the so-called compensation process.
During the inquiry, the statement was made, can't remember by whom, that value for taxpayer money had to be respected.
I think its past that point, and I would like to know why the instigators of the miscarriage aren't at risk of losing their personal assets to (at least partly) cover the cost. Maybe we have to wait for the inquiry report for that, but in the meantime it is pretty clear that 'value for taxpayer money' means, at least to some degree, kicking the can down the road until as many of the claimants are dead as possible.
I have read the proposed amendment, and it still looks woefully insufficient. It changes the rules from "software is presumed to be reliable" to "software previously certified by any government authority is presumed to be reliable."
This applies even if the certification body did not, in fact, do any significant stress-testing that would, e.g., uncover the races, or was not aware of the specific use case that triggers a bug.
In other words: "the Court will presume correct operation of the computer system and will ignore the fact that the other party challenges this; the amendment references certifications coming from a government body at any time in the past as a valid basis to ignore such challenges."
Now if only there were a little bug in whatever certified software handles judges' wages...
I understand why the presumption was there.
If it's not then every single case of tax evasion people would claim that they had paid and the computer must have lost the payment. Every single traffic camera offense would be contested. Every time you didn't have a train ticket people would claim the computer lost it. And so on.
And there would be months and months of delays if every time it had to be proved beyond a doubt that the equipment was working fully in that case out of all proportion to the risk of there actually being a problem.
However clearly there does need to be a way to challenge the presumption too. Of course just assuming equipment is working is entirely unfair.
After a brief look at the article it looks a reasonable weakening of the presumption which allows the court to consider and reject a challenge when they think fit according to the court rules (to be established) Pretty much like any other form of evidence.
Seems to me that computers do generally perform their tasks reliably. (I say generally, not infallibly; cosmic rays and power failures and such). The issue is really about whether their human programmers have successfully programmed the computer to do the intended task, and about whether their human operators are successfully interpreting computer outputs. So this is surely about what constitutes an expert witness, and how reasonable it is to rely on the testimony of an expert witness, where computers are concerned.
On can really consider how rare hardware issues leading to wrong computing are for your widely used computer chips. In general it seems they are very reliably correct and fixed in reasonably quick order.
Same really cannot be said about software. But on computational level it is actually amazing when you think about it.
When I worked for a drug company everything had to be documented and proceduralised. Because if you ask the FDA if you can sell your pills to the public they will ask how you know they're safe. And if you keep asking enough why's you get down to whether you can trust the output of computers. So every computer's build process and software stack has to be to documented and approved procedure so that you have a wad of paper to show them, when they ask that 'why'. It seems reasonable that if you're asking to send someone to jail based on the output of a computer you should be able to ask for reasonable proof that the output is correct.
Critical in the age of generative AI reliability.
The only thing reliable about them (and the humans that create them) is that they easily fool a bunch of fools.
Of course, it does appear that they are incredibly reliable at being somewhat reliable, but that's not the kind of reliable that I'm looking for, especially when the environmental cost of generating them is so expensive, and doubly so as global heating ramps up.
From the wikipedia page "...Although many subpostmasters had reported problems with the new software, and Fujitsu was aware that Horizon contained software bugs as early as 1999, the Post Office insisted that Horizon was robust and failed to disclose knowledge of the faults in the system during criminal and civil cases......" Fujitsu and Post Office management both new of software faults. Either or both of these groups should have dug much deeper.
The Post Office Management should have known OR hired Software EXPERTS to advise them on how to determine software reliability. Especially once it was evident a large number of people accused of theft. My Opinion is Post Office Management was negligent, particularly the CEO and other senior managers. ANd they should be charged with criminal negligence. CEO and senior managers are paid "big money" to deal with such companies, they should have dug "much deeper" to investigate, especially since their would have been large numbers who claimed innocence The software company Fujitsu, is also criminal negligence. Producing a financial system and not validating it.
The defence lawyers ( maybe with software computer system speciality ) could have asked for test records, test data how the system was validated. That would have been my first question. OK , Fujitsu and Post office management claim system is reliable, WHERE IS THE EVIDENCE OF SYSTEM RELIABILITY. Where are the exact test records and test data to support this claim. Where are the system design documents. Where is the company's bug / issue tracking, reports. How exactly is the software tested?
It seems the problem could have been detected at several levels , but for various reasons they all missed. I'm shocked that this issue went on for so long. (There must have good people somewhere , that said nothing, )
My opinion is shaped by my work experience as a software test analysist for many years ( a few decades ) , who has worked on software that does financial transactions, every release ( except ones where change was deemed minor ) of our software was examined by auditors from the government. These government auditors could do anything. ask for design documents, ask for test records. request tests be re run in their presence. That we run certain tests designed by the auditors. Interview any staff about the change. These software audits could be from a few hours to several days , depending on the change. Our company maintained a complete test system of the distributed system for this testing.
This reminds me of Kurt Vonnegut's book Player Piano. Imagine LLMs/AGI but implemented with vacuum tubes. It includes the line (from memory): We're sure of his sentencing, because we replaced all the vacuum tubes and ran the trial again and got the same result."
https://en.wikipedia.org/wiki/Player_Piano_(novel)
Boi are they gonna wake up in a cyber apocalypse.
Why do people that don't know shit about technological topics have the power of legislation to write laws about science?
Why can a court decide/confirm something like that without any technological experts involved?
The linked piece by Stephen Mason sets out all the interesting context for the law portion (not the postal service fiasco).
https://read.uolpress.co.uk/read/3a219939-14d3-45c9-9fe0-9b4...
Next up: What did Birmingham do with the £216 million for the 2020 Oracle Financials upgrade that will not be completed until 2026? For a city that has a 46% child poverty rate.
Sometimes the technology isn't the problem, it's the people.
Sometimes the technology isn't the problem, it's the people.
It's always a people problem.
Does this apply to voting machines?
The UK doesn't use any form of electronic voting. I don't think anywhere even uses electronic counting machines any more either.
If they don't call this the Computer Says No Act, I'd say there was a missed opportunity.
Links related to the UK post office scandal. Others?
Fujitsu bugs that sent innocent people to prison were known "from the start" - https://news.ycombinator.com/item?id=39059307 - Jan 2024 (270 comments)
Fujitsu CEO Deposition – Post Office Horizon IT Inquiry - https://news.ycombinator.com/item?id=39059302 - Jan 2024 (1 comment)
Fixing Horizon bugs would have been too costly, Post Office inquiry told - https://news.ycombinator.com/item?id=39039712 - Jan 2024 (59 comments)
Fujitsu says it will pay compensation in UK Post Office scandal - https://news.ycombinator.com/item?id=39023695 - Jan 2024 (26 comments)
How a software glitch at the UK Post Office ruined lives - https://news.ycombinator.com/item?id=39010070 - Jan 2024 (326 comments)
Post Office Horizon scandal explained: Everything you need to know - https://news.ycombinator.com/item?id=38983144 - Jan 2024 (8 comments)
A TV Show Forced Britain's Devastating Post Office Scandal into the Light - https://news.ycombinator.com/item?id=38951802 - Jan 2024 (168 comments)
British Post Office Scandal - https://news.ycombinator.com/item?id=38937705 - Jan 2024 (149 comments)
How the Post Office's Horizon system failed: a technical breakdown - https://news.ycombinator.com/item?id=38931792 - Jan 2024 (4 comments)
Ex Post Office CEO hands back award after IT failures lead to false convictions - https://news.ycombinator.com/item?id=38930011 - Jan 2024 (127 comments)
Post Office Horizon Enquiry – Fujitsu Report on Eposs PinICL Task Force (1998) - https://news.ycombinator.com/item?id=38926582 - Jan 2024 (1 comment)
Fujitsu bosses knew about Post Office Horizon IT flaws, says insider (2021) - https://news.ycombinator.com/item?id=38890468 - Jan 2024 (8 comments)
Mr Bates vs. the Post Office - https://news.ycombinator.com/item?id=38869011 - Jan 2024 (3 comments)
What went wrong with Horizon: learning from the Post Office Trial - https://news.ycombinator.com/item?id=38867712 - Jan 2024 (19 comments)
UK Post Office: 700 Horizon software scandal victims to receive £600k each - https://news.ycombinator.com/item?id=37561428 - Sept 2023 (40 comments)
After 20 years, the Post Office scandal cover-up is happening in plain sight - https://news.ycombinator.com/item?id=36778486 - July 2023 (1 comment)
The UK post office database scandal – “can't see the bug = user is a thief” - https://news.ycombinator.com/item?id=35837576 - May 2023 (2 comments)
Hundreds of lives ruined by faulty UK Post Office computer system - https://news.ycombinator.com/item?id=35792896 - May 2023 (4 comments)
Ex UK Post Office staff tell inquiry of stress of IT scandal - https://news.ycombinator.com/item?id=30394685 - Feb 2022 (2 comments)
Post Office scandal: Public inquiry to examine wrongful convictions - https://news.ycombinator.com/item?id=30329668 - Feb 2022 (149 comments)
Post Office scandal: 'I want someone else to be charged and jailed like I was' - https://news.ycombinator.com/item?id=30329510 - Feb 2022 (2 comments)
Bad software sent postal workers to jail - https://news.ycombinator.com/item?id=26973583 - April 2021 (1 comment)
Convicted Post Office workers have names cleared - https://news.ycombinator.com/item?id=26924882 - April 2021 (187 comments)
UK court clears post office staff convicted due to ‘corrupt data’ - https://news.ycombinator.com/item?id=26913037 - April 2021 (284 comments)
UK legal system assumes that computers don't have bugs - https://news.ycombinator.com/item?id=25518936 - Dec 2020 (24 comments)
Post Office scandal: Postmasters celebrate victory against convictions - https://news.ycombinator.com/item?id=24661321 - Oct 2020 (2 comments)
Bankruptcy, jail, ruined lives: inside the Post Office scandal - https://news.ycombinator.com/item?id=24440476 - Sept 2020 (1 comment)
Postmasters were prosecuted using unreliable evidence - https://news.ycombinator.com/item?id=23454606 - June 2020 (2 comments)
Faults in Post Office accounting system led to workers being convicted of theft - https://news.ycombinator.com/item?id=21795219 - Dec 2019 (104 comments)
Post Office hires accountants to review sub-postmasters' computer claims - https://news.ycombinator.com/item?id=4143107 - June 2012 (1 comment)
"Presumption about reliability of computers"
Anyone else get the sense that these were laws written without any input whatsoever from people who actually have working experience in complex systems?
I'm convinced that no moral software engineer would ever suggest a conviction on the presumption that the system is correct. At minimum you'd have to investigate.
The presumption was about mechanical instruments, from a simpler time. As computers expanded from being purely mechanical, it never got revised, which is why it is being revised now (three or four decades too late, I'll grant you, but legislation is a slow process.)
Except it was revised once already to point out that computers aren't reliable. And then a decade later, at the peak of the dot-com boom, that revision was reverted after a review by committee.