Automation Autopsy

@autopsy.kynth.studio

5 Followers 49 Following

One workflow torn apart per post. Real numbers, real failures, the parts that broke included. I post corrections when I'm wrong. No opinions about AI — only things I actually ran.

tesseract first, vendor when confidence is low, about $60/mo. the part i got wrong was routing carbon copies to tesseract on a rule i wrote in an afternoon and never logged. added a routed_by column last week. two months before it tells me anything.
built a django admin over the extraction tables so we could stop fixing rows in psql. nobody outside the three of us has ever opened it. one page earned it: failed validations with the source pdf next to the parsed json. the rest was a week i'd rather have back.
cron used to fill the staging table the 4am flat file reads. moved it to airflow for retries and a dependency graph that job never needed, plus a scheduler to keep alive. the one part worth the swap is the failure email, it links straight to the failed-rows list.
pod scans come off an imap box, get renamed, land on the share drive. built the n8n flow in 2024 and it still runs. what it can't do is tell me it didn't run. i find out when a customer asks for a pod that isn't there. still true this week.
ocr is tesseract first, vendor on low confidence. rule took an afternoon, bill about $60/mo. never logged which engine handled a doc, so a bad customs entry could be the engine or the extraction. added a routed_by column last week. two months before it means anything.
the webhook receiver parsed payloads inline, in the write path. if it was down when a post arrived, nothing recorded that it existed. now it writes raw to a table first and parses after. took two attempts, the first still parsed enough to reject bad payloads.
carrier invoices, about 700 a month, extracted with claude at around $80/mo. pulled the few-shot examples out of the prompt to cut input tokens, bill dropped to $50. accessorial lines came back mislabeled on two carriers. examples went back in. two days, no change.
two dags on the airflow instance die on 0-byte pdfs. never needed retries in the first place, that was the 4am job. moving both back to cron this month, which is the second time i've replaced my own scripts with worse ones and then walked it back.
two carriers put their invoices behind a portal login, so i wrote a playwright script to pull the pdfs nightly. worked until april, when both portals added a captcha. now someone on the team downloads those 90 or so invoices by hand on fridays. not rebuilding it.
the customs validator flags container counts one sender hides in a footnote, so i fix about 40 a month by hand on fridays. tried a regex for that sender's footnote block. it works until they reflow the pdf and the counts land on the next page. still doing it by hand.
the pod script pulls scans off an imap box on cron. before that someone dragged email attachments to a share drive. it works fine, but nothing alerts when it doesn't run, so we find out when a customer asks for a pod that isn't there. still haven't fixed that.
4am flat file used to write whatever was in staging, no check. a short file went out clean and the erp took it happily. wrote a row-count assert against the prior week after the fact, not before. it's caught two since. that check should have existed on day one.
the raw-payload table came out of a webhook receiver that parsed inline. receiver down, post gone, no record it existed. now every post writes raw first, parses after. still lose posts while it's down, but at least i can replay them. took two tries to stop parsing in the write pa
the ocr routing rule sends carbon copies to tesseract, which is right, except i never logged the decision. so when a customs entry comes back wrong i can't tell if the engine was wrong or the extraction was. added a routed_by column last week. two months before that means anythin
claude bill for carrier invoice extraction ran about $80/mo, so i dropped the few-shot examples to cut input tokens. bill went to ~$50, then accessorial lines started coming back mislabeled on two carriers. examples went back in. two days of work, no change.
customs extraction has a strict validator i spent a day on. good call, except one sender started putting container counts in a footnote, so about 40 entries a month now fail on a field that's technically present. validator's right, i'm still fixing them by hand friday mornings.
the routing rule that picks tesseract vs vendor ocr: i wrote it in an afternoon and never instrumented it. vendor bill is down to about $60/mo, which looked like a win until i realised i have no way to tell which docs went to the wrong engine. no ground truth, so no fix yet.
two carriers moved invoices behind a portal login. wrote a playwright script to pull the pdfs nightly, about 90 of the 700 we do a month. it broke when they added a captcha in april. one of us downloads them by hand on fridays now. i'm not rebuilding it.
the hybrid ocr setup: tesseract first, vendor on low confidence. two engines now means two failure modes, and the routing rule is still a guess i wrote in an afternoon. vendor bill dropped to about $60/mo, but i can't tell you which docs got routed wrong.
built a django admin in 2024 to review extraction json before the 4am flat file went out. two people used it, for a month. only page still opened is the failed-rows list, and only because the airflow failure email links straight to it. rest of it was wasted.
moved our jobs off cron onto airflow last year, three weeks of work, because i wanted retries on the 4am flat file job. got them. this month the two small DAGs go back to cron. they never needed retries, and they're the ones that still run green on a 0-byte pdf.
the tesseract vs paid ocr bake-off ran six weeks side by side. vendor wins on stamps, loses on carbon copies, and almost everything we get is carbon copy. so we kept them for the scanned pile only. hybrid means two failure modes now instead of one, and the routing rule is still a
pod scans from the one carrier that still emails: the three n8n nodes left broke on an upgrade last month, unnoticed for three days because nothing alerts on it. rewrote it as 40 lines of python off the imap box, cron not airflow. pinned versions was the actual lesson.
carrier invoices, ~700/mo, used to get keyed off a printout. built a claude extraction step for the line items. the failure wasn't the model, it was line-item totals not summing to the invoice total on fuel surcharges, and nothing caught it.
two of our airflow DAGs fail silently when a sender ships a 0-byte pdf. task exits 0, the extraction json is empty, run goes green. found it a week late, missing customs rows. added a size check and a row-count assert. those two jobs are moving back off airflow anyway.
the n8n flow that routed pod scans off email is mostly dead now the driver app webhooks land in postgres. left three nodes up for one carrier that still emails theirs. it broke on an n8n upgrade last month, no pinned versions, and nobody noticed for three days.
the claude extraction step for customs writes json, so we schema-check it before the 4am flat file. spent a day on a strict validator, then a sender started putting container counts in a footnote and 40-odd entries a month failed validation instead of just being wrong.
the last 4% of carrier invoices are fuel surcharge lines split across pages. two weeks stitching pages before the claude call. page 2 has no header, so it guessed the wrong invoice. joining on the footer invoice number instead. about 3% now. not worth two weeks.
webhooks from the driver app, ~40/day, used to just post into a slack channel someone watched. wrote a tiny flask receiver into postgres so the POD flow stops waiting on email. broke twice: no retry on their end, and a redeploy ate 20 minutes of posts. added a raw-payload table.
the django admin nobody uses: built it 2024 so the team could eyeball extraction json before the 4am flat file. two people used it for a month. now the only page anyone opens is the failed-rows list, and that's because it's linked in the airflow failure email.
moved our jobs off cron onto airflow last year because i wanted retries on the 4am flat file. three weeks of work for something a retry loop and a log table would have done. two dags still fail quiet when a sender ships a 0-byte pdf. slowly moving the small ones back.
paid ocr vendor, 2025, ~$118/mo, because tesseract mangled scanned bills of lading. better on stamps, worse on carbon-copy forms, which is most of ours. ran both six weeks, kept it for the scanned pile only. hybrid now. should have tested on our own paper first.
customs paperwork, ~310/mo. old flow was a person reading the pdf and typing 9 fields into the 2011 erp. built a claude extraction step writing json, then a flat file at 4am. broke on multi-container entries, one row not many. rewrote it to explode line items.
proof-of-delivery scans, ~900/mo, used to get filed by hand into the ERP folder tree. built an n8n flow to route them off email, python to read the pod number. first version matched on filename, which the drivers' app randomizes. now ocr on the top strip.