Nick Radcliffe

@njr0.bsky.social

83 Followers 64 Following

Sceptic

LEFT: The conventional testing Pyramid has a wide base of Unit tests, with a smaller second layer of functional tests, an even smaller third layer of unit tests and a top comprising system tests. RIGHT: Dodds's Test Troph has a small base of static static tests, a slightly wider layer of unit tests above that, followed by integration tests as the widest layer, topped by a smaller layer of end-to-end tests.
An analytical process is shown as a circle with inputs ("Record as reference inputs—datasets, parameters,...") and outputs ("Record as reference outputs—datasets, numbers, graphs, models, decisions etc."). The process should be captured as a script, ideally. The idea is then to develop a verification procedure (diff) and periodically re-execute the process on the inputs to check whether the same inputs still produce the same, or equivalent, outputs.
LEFT: TDDA's DISCOVER machine is shown as having an input hopper, an output chute, and gears. Training data is fed into the input hopper and a set of constraints (shown in braces) comes out of the output chute. RIGHT: TDDA's VERIFY machine is also shown as having an input hopper, an output chute, and gears. In this case, both operational data and constraints are fed into the input hopper and the machine delivers verifications from the output chute (shown with a check mark and a cross).
Left: A sample profile and audit for the CompanyName field from the Companies House Data. It has stats on null, whitespace-only and empty strings, then on most common names, then a frequency distribution for name length. Underneath are sample strings showing long, short, first, last, min, max and random names. Right: A profile and audit of the Price Delta field, which is real-valued, in pounds. It starts with counts for positive, zero and negative values and mean, median and mode. It then has a horizontal bar graph and table with a coarse binning of values, followed by a finer bar graph with broken bars. There is then a table of percentiles and finally a table of sample values (first/last, random, minima, maxima). ^D $ xattr -p description#S /Users/njr/books/tdda/image/ProfileAndAuditExamples.png Left: A sample profile and audit for the CompanyName field from the Companies House Data. It has stats on null, whitespace-only and empty strings, then on most common names, then a frequency distribution for name length. Underneath are sample strings showing long, short, first, last, min, max and random names. Right: A profile and audit of the Price Delta field, which is real-valued, in pounds. It starts with counts for positive, zero and negative values and mean, median and mode. It then has a horizontal bar graph and table with a coarse binning of values, followed by a finer bar graph with broken bars. There is then a table of percentiles and finally a table of sample values (first/last, random, minima, maxima).
Title Slide: TEST-DRIVEN DATA ANALYSIS (TDDA) & TEST-DRIVEN DOCUMENT DEVELOPMENT (TDDD) EdinbR 2026–06-11T18:00:00+01:00 2.55 Bayes Centre, Edinburgh Nicholas J. Radcliffe Stochastic Solutions Limited & Department of Mathematics, University of Edinburgh
Python code and output: import unicodedata def show(word): print() for c in word: print(f'{ord(c):04x}', unicodedata.name(c)) cafe = 'café' cafe_c = unicodedata.normalize('NFC', cafe) # normal form composed cafe_d = unicodedata.normalize('NFD', cafe) # normal form decomposed print(cafe_c, len(cafe_c), cafe_d, len(cafe_d), cafe_c == cafe_d) show(cafe_c) show(cafe_d)
LIGHTNING TALKS STARTING AT 9AM SHARP 1. Yanya Widen: LitelLMM supply chain attack 2. Kally Chang: PyData Amsterdam 3. Semyon B: ProbaViz 4.John Sandall: Do you want to joining the PyData London engineering team? 5.Matthew Barber: Hypothesis in 5 minutes 6. Winifred Njoaguani: PyData Community Organising 7.Jessica Walkenhurst: Building a digital cookbook using Claude and Obsidian 8. Katirina Riehl: Accelerated PyData 9. Mariel Kanene: NUMFocus Presents "GIVEMES" 10. Jasan Chao: Making USD $1k in two hours 11. Ituny Ijilia: from n8n to PyDanticAl, what no code developers can teach python developers 12.Marc Wouts: Jupytext + ITables 13. Ruby Walller: Paying to See Ourselves
Rachel-Lee Nabors At PyData London 2026
Sam Colvin speaking at PyData London 2026
Diagram showing four boxes superimposed on a curve that rises to a peak at the third box and then falls steeply so that the extreme right of the curve is lower than the left. There is a vertical axis labeled `AWFUL' at the bottom and `EXCELLENT' at the top. The first box is labeled `GARBAGE IN $\bullet$ GARBAGE OUT'. Above this box, a further label reads `NORMALITY, A.K.A. NEGLIGENCE'. This box is placed fairly low on the graph. There is a single cross below this box. Moving right, and upwards, the second box is labeled `GARBAGE IN $\bullet$ WARNING/ERROR OUT'. It has a label above saying `ERROR DETECTION' and a label below with a single tick. Continuing right, the third box, which is the highest (at the peak of the curve) is labeled `GARBAGE IN $\bullet$ GOLD OUT'. A label above says `ERROR CORRECTION' and there are two ticks below. Finally, the fourth box, on right, very low down, is labeled `GOLD IN $\bullet$ GARBAGE OUT'. The label above it reads `CORRUPTION'' and there are two crosses under this box. A horizontal arrow marked `TDDA' spans the right-hand side of the first box, all of the second box and the left-hand side of the third box.
TDDA book cover. Test Driven Data Analysis by Nicholad J. Radcliffe. A Chapman & Hall Book. CRC Press. Taylor & Francis Group. Data Science Series. The cover is black with a 3-row, four-column grid of squares. Each square has a number of white dots. In the top left, there are 32x32. Moving left to right, top to bottom, the number of dots roughly halves in each successive square. The numbers of dots on the bottom row are 6, 2, 2, 1.
Ed Hawkins climate stripes as a bar graph from 1850 to today. All the bars until about 1980 are blue and negative (w.r.t. the overall average). All the bars from about 1980 are increasingly positive. Each of the last 10 years is higher than all the years before 2013 and 2023 and 2024 are highest and deepest red. The caption reads Global temperatures have risen by 1.3 degrees celcius.
Picture of iPad with infographic labelled 'Strategic Storytelling: Working with Data' consisting of various hand-written text and three hand-drawn (or possibly faux-hand drawn) graphs.
Screenshot containing this text: Whatever you think about Dominic Cummings, he takes nuclear weapons very, very seriously. And he absolutely believes it is a subject that not only publics don't spend enough time thinking about, but politicians don't spend nearly enough time thinking about. And he has written about his shock and outrage at just the ignorance of British politicians about the strategic questions, the technological questions. They don't understand the technology, they don't understand the strategy, and they don't understand the history. The story is that when Boris Johnson became Prime Minister, his chief advisor, Dominic Cummings, was horrified by Johnson's ignorance. And so he arranged a three-hour briefing in Downing Street with leading military figures, strategists, historians and others, trying to fill him in. You are now responsible for this arsenal, and you need to pay attention. And to signal to Johnson that you need to pay attention, he took his phone off him, because he knew that Boris Johnson couldn't spend 10 minutes without checking WhatsApp. And he thought the only way to get him to pay attention was to take his phone off him. So he took his phone and he locked it outside the room, and he made him sit through three hours of this. And according to Cummings, at the end of this meeting, Johnson was white, the blood had drained from his face, not with horror or shock, but with fury at Cummings for taking his phone off him. And he came out of the room and he said, ‘Never fucking do that and never fucking waste my time like that again’. Apologies for my language, but if you want to be scared about the world we live in, there's a story to make you scared. — David Runciman, Past Present Future Podcast, Where are we going? Nuclear War, Part 2. With S. M. Amadae. https://podcasts.apple.com/gb/podcast/where-are-we-going-nuclear-war-part-2/id1682047968?i=1000750268760
Suggestions | Accepted | True Positive Rate Claude + ChatGPT | 500-1000 c. 100 | 10-20% Single Human Proofreader* | 296 | 284 | 96% * After Claude's & GPT's corrections had been applied
Claude code at startup after /context command showing 32% of tokens used and 22.5% reserved for autocompactification.
Cover of the UK Hardback edition of Blue Machine, by Helen Czerski. Feaures the oceans, as foreground, in blue, with the major currents shown in gold. The land is shown as background, in black. The projection is chosen to keep the oceans as the central focus, with the land split in various ways, as the ocean more often is, to allow the shape of the oceans to be more clearly seen.
Screenshot of paragraph features "varaintstens" (highlighted). Content is: In practice, the most recent versions of these systems depend not only on massive caches of text and images generated and curated by humans but also on human judgment and knowledge in other forms. In particular, the systems rely on reinforcement learning from human feedback (RLHF) or its variantstens of thousands of human employees provide ratings of model outputs. They also depend on prompt engineering—humans must use both their background knowledge and ingenuity to extract useful information from the models. Even the newest ‘chain of thought’ models regularly begin from dialogue with their human users.
Stats from my Kagi trial account. 52/100 web searches used. 0/50 AI interactions used.
Last few paragraphs of Simon's post: If we want people with decision-making authority to make good decisions about how to apply these tools we first need to acknowledge that there ARE good applications, and then help explain how to put those into practice while avoiding the many unintiutive traps. (If you still don’t think there are any good applications at all I’m not sure why you made it to this point in the article!) I think telling people that this whole field is environmentally catastrophic plagiarism machines that constantly make things up is doing those people a disservice, no matter how much truth that represents. There is genuine value to be had here, but getting to that value is unintuitive and needs guidance. Those of us who understand this stuff have a duty to help everyone else figure it out.
Heatmap showing every year from 1BCE to 2024 in a 45x45 grid. Everything is blue, representing values close to 270 parts per million, until the early 1900s. The second top row, repesenting 1935 to 1979 shows gradually increasing levels. The top row, representing 1980 to 2024, starts green (representing about 320ppm) and ends up bright red in 2024, representing just over 420 parts per million. Every year from about 2000 is orange or red, representing carbon dioxide concentrations over 350ppm. No year before about 1990 is.
Screenshot from The Guardian. Refers to the Italiam PM as Guilia Meloni.
Two squares, each with horizontally and vertically centred text. The top square says "Hey diddle diddle". The bottom one says "The text's in the middle". Underneath these is the full web page, with the surprisingly simple CSS to achieve it two different ways, both featuring CSS's display:grid, one using place-content: center and the other using justify-content: center and align-content: center.
Map of the World with country sizes scaled by population (2018). From Our World in Data. Country positions and shapes are preserved where possible. Hard to summarise in words, but inevitably low density countries, like Russia and Canada appear much smaller than their geographic size. Densely population countries such asIndia, Indonesia, Japan, Nigeria, D. R. Congo and even UK are shown much larger than on equal area maps (and common non-equal area projections).
Notebook Best Practices. (all text). HTML version at https://www.tdda.info/best-practices-for-notebook-users. PDF version at https://www.stochasticsolutions.com/pdf/nbp.pdf Text is: NBP1 □ Ensure that your Notebook runs correctly after completion □ Develop the Notebook □ Take a temporary copy of the Notebook □ Clear the Notebook □ Run and confirm the results match the temporary copy □ Fix (if this is not the case) □ Clear the new Notebook again □ Commit the new Notebook to version control □ Rerun (so the Notebook contains the results) □ Delete the temporary copy NBP2 □ Store the (cleared) Notebook in version control (see NBP1) NBP3 □ Parameterize inputs and outputs at the top of the Notebook □ e.g. Set and use variables such as INPATH and OUTPATH □ Write the most important outputs to file (if not already done) NBP4 □ Replace some individual cells or groups of cells with functions □ Move them into an importable file, import and use □ Prioritize potentially re-usable code and code requiring testing NBP5 □ Write some tests □ Create a reference (regression) test for the whole process □ Create unit tests for the individual functions NBP6 □ Consider restructuring/extracting the code as a standalone script NBP7 □ Allow the parameters to be set from the command line □ Alternatively,readfromaconfigurationfile(e.g..jsonor.toml) NBP8 □ Consider using safer alternatives like Marimo and Quarto
Quote: If debugging is the process of removing software bugs, then programming must be the process of putting them in. — Edger W. Dijkstra'
Not a great one for alt text, but two Observation Windows (a training one with a 12-month "pre-"-period and a 3-month outcome period) and a deployment one with only a (later) deployment period. Both, sadly, are labelled "Observation Widow".
A graph showing the incomes in the table on a y-axis with a log scale and tick lines at $100, $200, $500, $1k $2k $5k and $10k. The seven countries are displayed on the x-axis as an orange dot for the 10th percentile income and a blue dot for the 90th percentile income. There is a vertical line between these tow number for each country, annotated with ratio of the two incomes.
Extract from (in-progress) book: In his novel Stranger in a Strange Land, Robert Heinlein (1961) introduced the word grok. The definition that seems to me most closely to match its usage is: grok (transivitive verb) 1.To understand profoundly through intuition or empathy. grok (verb) 1. To have or to have acquired an intuitive understanding of; to know (something) without having to think (such as knowing the number of objects in a collection without needing to count them: see subitize). 2. To fully and completely understand something in all its details and intricacies. 3. To get the meaning of some-thing. — The American Heritage Dictionary of the English Language 2022 (5th ed.)
Two graphs showing the same data from the Luxembourgh Income Study as shown in the original post from Our World in Data, but on linear scales. The data compares the income of the 10th percentile and 90th percentile of after-tax income in seven countries The top graph runs from $0 to $8000 and shows income differentials for South Africa of 22x (c. $2470 to c.$110), Brazil 8.4x (c. $1650 vs c. $200), China 7.8x (c. $1900 vs. c. $250) Uraguy 5.6x (c.$2200 vs. c. $400) UK 3.8x (c. $4100 vs. c. $1100) UK 6x (c. $6830 vs. c. $1170) and Norway (c. $5120 vs. $1688). The bottom graph shows the multipliers on a linear scale, aligned with the top, with each country shown as a vertical line from 1x to its country multiplier.
Screenshot of ffs utility. Shows ls command with files with spaces, ampersands etc. in the filenames. Run ffs. Suggests mv command but a further ls shows it has not executed them. Run ffs -f and this forces the renamings. Final ls shows cleaned-up filenames.